by Kingsindian
[Second in a two-part series.]
In Part 1, we looked at how the mechanisms behind Wikipedia are totally inadequate for the purpose of fighting “fake news”. But perhaps, as the communication director of the Wikimedia Foundation (WMF) Juliet Barbara says: “It’s a good thing Wikipedia works in practice because, in theory, it’s a total disaster.” So let’s look at how Wikipedia does in practice.
Self-image
Image: Wikimedia Commons
License: CC BY-SA 4.0
Katherine Maher, the executive director of the WMF, in defending Wikipedia’s content, points to two studies:
“A 2012 study commissioned by Oxford University and the Wikimedia Foundation, for example, showed that when compared with other encyclopedic entries, Wikipedia articles scored higher overall with respect to accuracy, references and overall judgment when compared with articles from more traditional encyclopedias. Wikipedia articles were also generally seen as being more up-to-date, better-referenced and at least as comprehensive and neutral. This study followed a similar 2005 study from Nature that found Wikipedia articles on science as reliable as their counterparts from Encyclopedia Britannica.”
Let’s look at the studies in turn.
The Oxford University study is not, and never pretended to be, an assessment of the quality of Wikipedia articles. It bills itself as a “preliminary comparative study” and a “pilot study” aimed at “explor[ing] the viability of the methods used … for a possible future study on a larger scale.” There is a disclaimer in the “Results” section: “All of the results outlined below are based on a small sample studied for the purposes of piloting the study’s approach and methods, and these results cannot therefore be generalised to the wider output of the online encyclopaedias referred to.” As far as I can determine, no follow-up study was undertaken, possibly because it proved difficult to surmount several obstacles identified in the study. Since it has been five years since then, one can conclude that assessing article quality is not high on the WMF’s priorities. Certainly, nothing in any of the WMF’s past few annual budgets featured a line-item for the process of assessing article quality.
The criticism of the Nature study can be outsourced to Andreas Kolbe’s excellent Wikipediocracy blog post, which demonstrates why the hype about the study is unwarranted and misleading. Also, see this contemporary post by Nicholas Carr, which makes several cogent points. Leaving aside the Nature study’s validity, an obvious question presents itself: why is a study from 2005 is being quoted in 2017? Everyone recognizes that Wikipedia has changed a lot in the meantime, so what is the relevance of a study from that far back? But it seems that the WMF and scores of Internet pontificators just love the thesis of the study, so it’s just too tempting to not keep quoting it for more than a decade now.
Reality
So what is the actual state of Wikipedia content? This is a large topic and many different aspects have been studied. A 2014 paper by Okoli et. al. reviews a large number of studies assessing Wikipedia content. Many of the studies are somewhat old, but no more than the studies quoted above. The review looked at four aspects of content: breadth of coverage, reliability/accuracy, up-to-dateness, and readability. In the following, I give a general outline of their findings about the first two aspects.
Breadth of coverage: Wikipedia, by its very nature is well suited to cover a wide variety of topics, both niche and popular. Wikipedia famously has articles for all episodes of Doctor Who and The Simpsons, as well as obscure English football players and US politicians. The studies reviewed buttressed this popular perception: most of them find that Wikipedia is reasonably comprehensive in its coverage of topics ranging from science, health, history and philosophy. The research typically finds that older topics have more omissions than newer ones, a case of “recentism” which Wikipedia is known to suffer from. Regarding the “gender gap” on Wikipedia, the studies find that Wikipedia has a higher number of women biographies, and the same proportion of notable women biographies as several other datasets; however, biographies of notable women are more likely to be missing than male ones.
Reliability/accuracy: Of course, coverage of topics does not imply that they are presented accurately and completely. Here, the record of Wikipedia is quite mixed, with some studies evaluating quite favourably compared to professional sources, while others pointed to deficiencies, sometimes quite serious ones. The review found that the Wikipedia articles in the health domain (which has been most comprehensively studied topic, due to the obvious life-and-death implications) have mixed evaluations of Wikipedia content. In other domains, Wikipedia was found to be “generally” or “reasonably” accurate — however, Wikipedia articles frequently contain “sins of omission” where relevant and important information is missing from the article.
Studies conducted more recently follow this trend of mixed evaluations. A 2014 study in the Journal of the American Osteopathic Association concluded: “Most Wikipedia articles representing the 10 most costly medical conditions in the United States contain many errors when checked against standard peer-reviewed sources. Caution should be used when using Wikipedia to answer questions regarding patient care.” On the flip side, a 2014 study of pharmacology-related articles found that “Wikipedia is an accurate and comprehensive source of drug-related information for undergraduate medical education.”
Summarizing the evidence above, it’s fair to say that the picture of Wikipedia as being “roughly of the same quality as other reference works” is misleading at best. Wikipedia is decent for some things and unreliable for others. Moreover, even within the same category, there is a lot of variability in articles.
Neutrality
One suggestion in the communications audit was to position Wikipedia as a provider of neutral information and tweet out such information when a controversy breaks. This suggestion is seriously faulty, which will become clear from an example.
In May 2016, Google marked the 95th birthday of Yuri Kochiyama with a Google Doodle. Kochiyama was a longtime activist for various causes and her views on many topics were controversial. At the time, there was a big furor in the media about the Google Doodle; among other things, Senator Pat Toomey of Pennsylvania demanded an apology from Google because of Kochiyama’s views on Mumia Abu Jamal and Osama bin Laden. Without commenting on the merits of various sides, this “natural experiment” nevertheless provides some insight into how Wikipedia operates.
In the immediate aftermath of the controversy, the Wikipedia page was very heavily edited. In fact, it received more revisions in a single day, 19 May, than it had received since the article was started (back in 2005). In a matter of days, the article was rewritten drastically (compare the version on 18 May with the version on 1 June). At the very least, the experience suggests that the idea of Wikipedia as a repository of “neutral” content during times of high controversy is seriously flawed.
A broader view of neutrality on Wikipedia
Did I just pick an unrepresentative example, or is the case described above typical? Here the work of Greenstein and Zhu is useful. In a series of papers, they investigate the bias of Wikipedia articles on US politics. Their work covers a lot of ground, and I’ll only mention a few relevant parts here.
Greenstein and Zhu use the following technique (originally due to Gertzkow and Shapiro 2010) to investigate bias. They look at phrases occurring in the Congressional Record and associate a number of “code words” with Democrats and Republicans. For instance, “war on terror”, “illegal alien” and “Saddam Hussein” are used more often by Republicans, while the phrases “war in Iraq”, “Rosa Parks” or “national wildlife” are used more often by Democrats. The authors then use the frequency of the code words to determine the “slant” of articles on Wikipedia and Encyclopedia Britannica.
The authors find that Wikipedia articles, on average, lean slightly to the left as compared to Britannica, but this effect disappears entirely when controlling for article length (Wikipedia articles tend to be much longer than Britannica articles). Again, however, this average conceals a lot of variability depending on how many revisions the article gets. Articles in the upper quartile of their sample get enough revisions to be more or less neutral, but the median article does not, and shows “considerable difference in slant and bias from their Britannia counterparts”.
Finally, a comment about the overall approach. Wikipedia, as we have seen in part 1, is primarily edited by people in relatively rich, English-speaking countries. Greenstein and Zhu confirm this: they find that both Republicans and Democrats are represented well on Wikipedia, with Democrats outnumbering Republicans by some amount. This pattern is typical of many fields like journalism and academia. On many other topics, say India/Pakistan, or Israel/Palestine, even this rough parity of representation likely does not hold. This is reflected, for instance, in terrorism-related articles on Wikipedia.
Image credit: kthypryn on flickr CC BY 2.0
Toward a more realistic view of Wikipedia
The primary innovation of Wikipedia was to produce a large supply of “sorta” correct information for no charge at all. I do not want to disparage this accomplishment — Wikipedia would not be a top 10 website if people didn’t find it useful. I have used Wikipedia for more than a decade, and have been an on and off contributor for roughly the same time. Many Wikipedia contributors like myself are acutely aware of its flaws, which are many and big. The relationship between the image of Wikipedia I have and the image which the WMF propagates is akin to the relationship between a photograph and the image in a funhouse mirror.
I’ll end with two quotes about my own attitude toward Wikipedia.
The first comes from Andreas’ earlier WO article: “Wikipedia is free. It’s understandable that people don’t like to look a gift horse in the mouth. But no one should fool themselves into thinking that the nag they got for free is an Arabian racehorse.”
The other comes from Amílcar Cabral, one of Africa’s erstwhile anti-colonial leaders. Cabral’s motto for his own organization, which the WMF would be wise to adopt, was: “Tell no lies, claim no easy victories.”