Bush, Google and censorship
An interesting report on how the Obama government intends to open up the White House website to search engines. Under Bush, they “blocked” search engines from indexing much of the site (although the data was still there) by using the “robots.txt” file. This file is commonly used to tell robots not to index things like your CGI directory, which you might use to administer your website. If you use a program like WPoison, a script which generates pages full of fake e-mail addresses and links to other pages also generated by it so that spammers’ address lists end up with lots of false addresses, “robots.txt” tells Google and Yahoo not to get caught up in them as well.
However, Google and Yahoo are under no obligation to actually honour the file; if Google did not index the White House website because of it, then they were colluding with the Bush government’s censorship. It is bad enough that they collaborate with the Chinese government’s censors and even betray their users to their Gestapo; in the USA, there is no Gestapo and they don’t have to collaborate, so why do they? Presumably none of what was hidden was classified information, otherwise it would not have been on the site at all. An exception could easily have been written into the spider programs so that the content would have been indexed. They are American companies after all, and guess who funds the White House website.
Possibly Related Posts:
- Garmin’s four-day outage reflects incompetence
- Guardian Daily: nice new app, shame about the upgrade
- Riots don’t start; people start them
- The Stallman affair and what it means for Open Source
- Yes, we need our hands-free phones.