Yahoo, MSN Censor More than Baidu

China unblocked many usually censored web sites following intense international pressure and scrutiny after having promised uncensored access during the Olympics. Five days later (August 6, 2008) I tested the search engines that Google, Yahoo! and Microsoft customize for the Chinese market as well as the leading domestic search engine Baidu. I found that all of the search engines were still censoring content that was unblocked by China. one interesting find was that Yahoo! was censoring less than all the others and Baidu (and Google) were censoring much less than Microsoft.

For purposes on comparison Google and Microsoft make a good match because both have to de-list web sites form search results while Yahoo! and Baidu index form within China and thus do not (usually) index sites already censored by China. (For more read my report on search engine comparison.)

Now over a month later things have changed. While these sites remain accessible in China some are still censored by the search engines. Google has dropped to only censoring two sites and is now censoring the least amount of content. Baidu is next with three censored sites. Microsoft remained steady, but Yahoo! has shifted from censoring the least amount of sites to the most!

The divergence between Yahoo! and Baidu is very interesting. If both crawl from within China and are subject to China’s filtering why is Yahoo! censoring so much more than Baidu? It could be that the conclusion that Yahoo! and Baidu do not de-list content is not fully accurate. If the sites are accessible in China then Yahoo! is likely de-listing the sites. Because of the suboptimal method of censorship notification employed by Yahoo! (a standard disclaimer on every page regardless of whether any of the results are censored or not) I cannot fully distinguish between sites that are de-listed and sites that have not been indexed (e.g. because China blocks them).

I’m still struck by the fact that over a month later sites that are available and uncensored in China are still censored by these search engines.

Free Expression Principles

Major technology companies, including Google, Yahoo! and Microsoft, have agreed, in principle, to a voluntary set of principles designed to “guide businesses when they encounter laws and practices that may contravene international human rights standards or be at odds with law or culture in their home jurisdiction.” The objective is to protect and advance freedom of expression and privacy. Included in this initiative are mechanisms to provide for ongoing learning as well as the monitoring of compliance.

Google, Yahoo! and Microsoft sent letters to Sen. Durbin announcing the agreement. The letters re-state each company’s commitment to freedom of expression and highlight the core components of the initiative including the principles, the implementation guidelines and the accountability and learning framework.

Google’s letter draws on my report that compared Google, yahoo! and Microsoft’s search engines along with the domestic Chinese company Baidu. The most significant point centers on the impact of engagement. I found that the presence of foreign search engines resulted in an increased amount of information being available to Chinese Internet users. More specifically, I found that:

When the results from Google, Microsoft and Yahoo are combined, 20% of the sites censored by Baidu are available. However, individually they provide more information, especially Google and Microsoft which provide, on average, 51% and 55% more content (content not available in Baidu) while Yahoo! averages 25% more.

Since the search engines were censoring different content mixing searches across multiple search engine resulted in the ability to find sites censored by the other search engines.

Also, I noted that Baidu, the leading Chinese search engine, had introduced a censorship notification following the lead of the foreign search engines. Unlike foreign search engines under pressure from the home governments Baidu is not. While a still a small step, it shows that engagement can make a difference and that industry standards are important. that is why I think the principles for free expression and privacy are so important. They present a united effort and set an industry standard.

Engagement certainly presents a series of hard choices, but is a better choice than disengagement when it comes to information and communications technologies. These technologies build the bridges that connect diverse people and places, putting up barriers is what the censors do. I find it hard to believe that the promotion of free expression is served in Iran by denying Iranians access to the Java programming language.

The catch here is that this agreement and these principles are not an end point but a starting point. As I noted in my report the overall level of transparency is low — there is work to be done in this area. The process for determining what to censor is still unclear and supports the secrecy and unaccountability of China’s censorship policies. Even within a restrictive environment such as China I believe there is much more that can be done. (See below). I also showed that while the total amount of censorship may not be high, the significance of the censored sites is important.

These censored sites are often the only sources of alternative information available in the top ten results for politically sensitive search queries. Moreover, even the uncensored versions of these search engines highly rank content that is hosted in China or ends in the domain suffix .cn, both of which China retains control over and are thus unlikely to present alternative information.

China recently unblocked many censored web sites after intense international pressure and scrutiny after having promised uncensored access during the Olympics. Andrew Lih tested a sample of websites normally censored in China and found them to be accessible. The web sites of human rights groups such as Human Rights Watch, Reporters Sans Frontiers and Amnesty International are all now accessible.

Andrew posted his test results on August 1st, 2008, five days later search engines are still censoring sites that are not unblocked in China. For example, Yahoo! Microsoft and Baidu are still censoring www.amnesty.org while Google is not. Google, Microsoft and Baidu are still censoring www.hrw.org while Yahoo!s not. (Yahoo! has only one result, www.hrw.org/russian, I’m not sure how many Russian speakers there are in China, anyone know?) Only Microsoft is still censoring www.rsf.org — even Baidu is not. In fact, Microsoft is censoring more of these newly unblocked websites than the Chinese company Baidu! Another noteworthy observation is that Yahoo! is censoring the least of these newly unblocked sites.

* If at least one result was returned for a “site:” search on a domain, it was marked as OK.

To be fair, it does take time for search engines to respond. They have multiple servers, it may take time for them all to be updated. Also, there differences in implementation between those that crawl and index the web from behind China’s filtering system and those that do not and thus have to “de-list” results. (See the report for details on this.)

Still, I find it difficult to accept that sites that are unblocked in China remain censored in these search engines.

Search Monitor in the Press

China’s Overeager American Censors – Forbes

Practically every U.S.-owned search engine has caved to the Chinese government’s demands that they censor political Web sites in China. But none of them seem to agree on just what sites need censoring.

Google, at times, blocks Chinese users’ access to the BBC while Yahoo! permits it. Yahoo! sometimes filters out Voice of America–Google doesn’t. And Microsoft removes entries from the Chinese version of Wikipedia from its results while every other search engine includes them–even the dominant Chinese search engine Baidu.com.

Confused?

Search Engines’ Chinese Self-Censorship – Technology Review and ABC

A report released last week by the Citizen Lab at the Munk Centre for International Studies at the University of Toronto found that different search engines are blocking fairly different content. “The low overlap means that companies are choosing the exact content to censor or, alternatively, to not censor,” says Nart Villeneuve, a senior research fellow at the Citizen Lab and the author of the report. “That doesn’t mean that they’re not getting guidance from the Chinese government in other ways,” he notes. But Villeneuve says that if search engines are interpreting Chinese policies to decide what to censor, that introduces the possibility that they may block more content than is strictly necessary.

Read more about the Search Monitor Project here and here.

Search Monitor: Toward a Measure of Transparency

Citizen Lab Occasional Paper #1, “Search Monitor Project: Toward a Measure of Transparency“, (mirror) has been released today. This report interrogates and compares the censorship practices of the search engines provided by Google, Microsoft and Yahoo! for the Chinese market along with the domestic Chinese search engine Baidu. It is based on tests conducted between November 2007 and April 2008 focused on uncovering web sites that have been censored from search engine results.

The report finds that although Internet users in China are able to access more information due to the presence of foreign search engines the web sites that are censored are often the only sources of alternative information available for politically sensitive topics. In addition to censoring the web sites of Chinese dissidents and the Falun Gong movement, the web sites of major news organizations, such as the BBC, as well as international advocacy organizations, such as Human Rights Watch, are also censored.

The data presented in this report indicates that there is not a comprehensive system – such as a list issued by the Chinese government – in place for determining censored content. In fact, the evidence suggests that search engine companies themselves are selecting the specific web sites to be censored raising the possibility of over blocking as well as indicating that there is significant flexibility in choosing how to implement China’s censorship requirements.

This report finds that search engine companies maintain an overall low level of transparency regarding their censorship practices and concludes that independent monitoring is required to evaluate their compliance with public pledges regarding commitments to transparency and human rights. The lack of clarity in the process and the unwillingness of companies to disclose this information acts to bolster China’s current censorship policy that thrives on secrecy and unaccountability.

It is becoming increasingly clear that technology companies face a dilemma when attempting to penetrate the Chinese market. A failure to comply with China’s censorship policies can result in the wholesale blocking of a company’s entire service or significant levels of interference due to China’s filtering system. Companies that have a physical presence in China face the challenge of obtaining proper licensing and their Chinese employees may face legal threats for the foreign company’s failure to comply with China’s censorship policies. However, it is also clear that compliance with China’s censorship policies is also an unattractive option. Google, Microsoft and Yahoo! are all facing tough criticism from governments, human rights groups and civil liberties advocates as well as their shareholders for their complicity in China’s censorship policies.

While foreign search engines do provide more content than domestic search engines, the greatest benefit of having foreign search engines in China may not be increased access to information but is the potential contribution that these companies can make to further transparency and accountability in the process of censorship.

Since this report was finalized, the domestic Chinese search engine Baidu, following the foreign search engines, introduced a censorship notification indicating that it is possible to make progress through engagement. While this development may seem negligible to some and it is certainly no reason to become complacent, it is a small first step toward lifting the veil of secrecy and unaccountability that permeates China’s censorship policies.

Microsoft: Censorship Notification Returns

Microsoft now has a censorship notification in the censored version of the search engine live.com that they provide for the Chinese market. The notification appears when search are made for particular keywords, however, the notification is not displayed when searches are restricted to censored domains. (See Degrading Transparency: Comparing Google, Yahoo and Microsoft for past reports).

* Microsoft provides notification when searching for particular “key words”, however, no message appears when restricting the search to a censored web site.