Adding a basic robots.txt file to your website

Oct 302009

By adding a robots.txt file to your website’s main directory, (where your homepage is located), you can direct all the good robots like Googlebot, ASK, MSNbot, and Yahoo Slurp, search engine bots from indexing certain parts of your website. So let’s say you have a webpage that has a bunch of your email addresses on it, and you don’t want that page to show up in any search engine searches. You add that page to a robots.txt file, and the good robots will not index that page, but instead skip over it.

Now I say good robots will skip the pages you have in your robots.txt file, but the bad robots like email harvesters (spam bots), will most of the time bypass the robots.txt file because they are looking for email addresses to steal.

The easiest way to make a robots.txt file is to use a program like Notepad, and save it as a robots.txt file, typing in the pages you don’t want indexed, (as shown below), and putting it in the main directory of your website where your homepage is located.

——-Robots Text File Example—-

User-agent: *
Disallow: /folder/
Disallow: htmlpage.html

—-End Example———————-

User-agent: * designates which search engine you are giving commands to. In this case, the asterisk designates all search engine robots.

Where /folder/ is put any folders you do not want indexed, you can add more folders by using the Disallow: / / command putting the folder name between the two forward slashes.

Where htmlpage.html is put the name of the page (with its extension), that you do not want indexed. You can add more pages by using the Disallow: command followed by the file name and its extension. So .html if it is an HTML file, .php if it is a PHP file, and so on.
If you do not want a robots.txt file on your website, but you still want to keep certain robots out of certain pages, then you can add the following tag to the header of every webpage that you want to keep robots out of.

Enter the following between the <head> and </head> tags of the top of your page.

Share on these Social Networks: