Yahoo Blocks Bots From Delicious Bookmarks

Yahoo’s Delicious (del.icio.us) social bookmarking property has been reportedly blocking spiders and bots from non-Yahoo search engines from visiting the site and spotting new sites, web pages and bookmarks. Experts believ that it is not a simple robots.txt exclusion. Instead it is a 404 response that is being served on the basis of the requesting User-Agent. They also found the Delicious pages that are listed in Google are lacking a title, cache, description and various other information.

Yahoo surely has a competitive advantage over MSN, Google, MSN and Ask.com as it is able to recognize web pages and other content through human bookmarking on Delicious much before the search engine bots can do so. Yahoo also has the ability to categorize web documents with the help of human descriptions and tagging, lending external meta data to these documents. This can result in more suitable web results as well as intent targeted rankings.

Yahoo has incorporated Delicious into its search results. Thus it is quite obvious that Delicious has a very significant position in Yahoo Search. Yahoo is now taking full benefit of its property by restricting its competition from crawling any such information. It is really a brave thing for Yahoo to do as Delicious is user powered.

1 Comment

1. I don’t believe that there is a real chance for Yahoo. del.icio.us is a nice relict from the time when Web 2.0 started. But it is not state-of-the-art at all. So if Yahoo would like to use this Social Bookmarking data for itself, I believe no other Search Engine has a problem with it. The market share of Yahoo is too small that this blocking could be important. Perhaps this is one of those tries of Yahoo to show more current search results, but they will not catch Google with it.

   CommentbyWulffy February 28, 2008 @ 1:08 pm

RSS feed for comments on this post
TrackBack URI

Leave a comment