Correct and Incorrect Robots.txt files

The robots.txt file is simply a file that tells the search engines which directories it should not read. You should disallow the search engines reading parts of the site you don’t want to have in the natural listings.

It is easy however to incorrectly edit the robots.txt file and accidently block parts of the site you want to get indexed by the search engines. Correct use of the robots.txt file can help your site hoard the Page Rank of your important pages ,and improve their search engine listings.

Today I thought I would provide you with some examples of incorrect and correct implementation of the robots.txt file

Correct Implementation

• Look, got the directories in your site and disallow the sections you don’t want search engines to index. Good examples are the following directories /wp-admin/, /scripts/ /cart/ etc.
• Disallow duplicate content sections of your site such as print versions, you should only allow the search engines to index the first version of the content
• Ensure you don’t disallow search engines from indexing the main content areas of your site
• Stop search engines indexing sensitive data such as email addresses and phone numbers

Incorrect Implementation

• Don’t add the line disallow: /, as this will disallow the search engines read any of your site and you will not appear in Google’s index
• Don’t list individual files in the robots.txt file as this lets users know which files you don’t want them to find
• Robots.txt files only have the ‘disallow’ command, don’t use the ‘allow’ command.If you want the whole site to be read simply add the line disallow: This allows the search engines to read the whole site.

Paul
SEO Project Manager

Listen to this podcast

No Comments

No comments yet.

RSS feed for comments on this post
TrackBack URI

Leave a comment