|
| Recent
Articles |
Make Your Content Del.icio.us
Del.icio.us is the most popular bookmarking service on the web. By getting on the Del.icio.us popular or hotlist page you could get thousands of visitors coming to your website within minutes. Here are some ways that...
Blogs - The Emotional Description
One topic that often comes up in conversations about blogging is how do you define a blog. Answering the question invariably includes a description of the...
Creating Rich Content
We have all heard the term content-rich, but what does content rich really mean? Content rich means different things for different individuals, because what one person finds useful, another may not. Content rich is all...
Design, SEO, And Usability Laws?
You read and hear it all the time. Somebody tells you to do something a certain way because it's the only correct way. So, you do it, only to learn later that your...
Social Web SEO
American sculptor Horatio Greenough coined the phrase "Form follows function," which was later popularized by architects Louis Sullivan and Frank Lloyd Wright. The principle behind the phrase has long been...
|
|
|
01.29.07
Using Robots.txt To Prevent Search Indexing
 By
Jason Lee Miller
Sometimes there are parts of your website you don't want accessed by the search engines - for any number of reasons, like sensitive private data, articles that require subscriptions - whatever.
Google recently posted a tutorial about how to use the robots.txt file to block search engines from indexing specific parts of the
site.
Though we've talked about robots.txt at length, it often good to remember the newbies (hi, newbies!). So it was good timing that this tutorial was posted, even if product manager Dan Crow apologetically butchered up a certain brilliant paragraph by Douglas Adams in the process.
He redeems himself by providing the following easy to decode example of how to implement one of these necessary files:
Here is a simple example of a robots.txt file.
User-Agent: Googlebot
Disallow: /logs
The User-Agent line specifies that the next section is a set of instructions just for the Googlebot. All the major search engines read and obey the instructions you put in robots.txt, and you can specify different rules for different search engines if you want to. The Disallow line tells Googlebot not to access files in the logs sub-directory of your site. The contents of the pages you put into the logs directory will not show up in Google search results.
Preventing access to a file
If you have a news article on your site that is only accessible by registered users, you'll want it excluded from Google's results. To do this, simply add a META tag into the html file, so it starts something like:
<html>
<head>
<meta name="googlebot" content="noindex">
This stops Google from indexing this file. META tags are particularly useful if you have permission to edit the individual files but not the site-wide robots.txt. They also allow you to specify complex access-control policies on a page-by-page basis.
If you understand that, you're far, far ahead of Belgian newspapers and their courts.
About the Author:
Jason L. Miller is a staff writer for WebProNews covering technology and business. |
