While Google’s web crawlers (Googlebots) are becoming increasing efficient at indexing web pages, it certainly can’t hurt for a webmaster to provide them with a helping hand. Particularly on very large websites where some important pages may be buried deep within the navigation, Google may have problems crawling this deeply and indexing those pages.
This is where XML sitemaps come in extremely handy!
XML sitemaps are basically files with a list of pages within a website that search engines can read. Written in the XML language (hence the name), they are built to a specific standard that all the major search engines – Google, Yahoo, Live and subsequently most other search engines – read and index.
To create an XML sitemap for your website you will likely need to have an understanding of the XML sitemap syntax, details of which can be found at: http://www.sitemaps.org/protocol.php. If you’re not familiar with the syntax, there are plenty of XML sitemap generators available online for free (an online search for ‘XML sitemap generator’ will likely turn up a plethora of choices!)
Once created, you will need to let search engines know about the existence of your sitemap, which can be done in a number of ways; you can link to the sitemap on your website, link to the sitemap through your robots.txt file, or even submit directly to the major search engines through their webmaster tools.
Rik Weber
Senior Web Developer