October 29, 2003

The Politics of robots.txt

When standard web crawlers are indexing sites, they begin by checking for the existence of a robots.txt file [definition]. The file specifies which directories a well-behaved crawler should or should not include in its index. So, for example, irrelevant files or things that are semi-private can, in theory, not show up in search engines. In practice, they still tend to show up for a variety of reasons, including crawlers that ignore robots.txt and via links from other pages. Still, robots.txt files do have an effect and hiding pages and directories with rules in that file tends to make the pages not show up as frequently in search engines.

At some point last summer, the White House changed the robots.txt files in White House websites to include most of the directories with "Iraq" in them to the robots.txt file, meaning that search engines would no longer list pages from White House websites if the pages contained the word "Iraq". As the DNC's blog ("Kicking Ass") describes it,

Why would the White House do this? Those pages are still public, and the White House search engine itself does index those pages, so users can still get to them.

It's easy enough to understand the reasoning if you look at past White House actions. Earlier this year, the White House revised pages on its website claiming that "combat" was over in Iraq, changing them to say "major combat."

One of the reasons some alert readers noticed the change — and were able to prove it — was that Google had archived the pages before the change occurred. Now that all of the White House pages about Iraq are no longer archived by Google, such historical revisionism will be harder to catch.

Seth Finkelstein's Infothought blog offers a more detailed and wide-ranging analysis, including extensive links, comparisons, and ongoing discussion (including what Dan Gilmor describes as the Occam's Razor" explanation).

[via Dan Gillmor's eJournal] Posted by johndan at October 29, 2003 03:15 PM | TrackBack