Is Wikipedia up to the job?

If you use the Internet, you know and likely love Wikipedia. But have you ever asked yourself how reliable an effectively "crowd-sourced" information point is? Epic, a bespoke e-learning company, and the University of Oxford recently released their comparative pilot study on the efficacy and integrity of content published by Wikipedia. A sample of 22 Wikipedia English-language articles was compared to corresponding articles in Encyclopaedia Britannica. The study also looked at Spanish Wikipedia vs. Enciclonet and Arabic Wikipedia vs. Arab Encyclopaedia.



Source: Wikipedia.org

Before I go into the report’s findings itself, here’s a brief re-cap of how Wikipedia operates and what its chief English-language competitor – Encyclopaedia Britannica – thinks.

Anyone with access to the Internet is encouraged to contribute. Readers of Wikipedia items “can edit the text as he or she sees fit, anonymously or with a user account” (Wikipedia, 2012). How does this work in practice? Wikipedia operates around the concept of consensus editing based upon open-community derived policies and guidelines. The fundamentals are summarised in the form of five pillars as follows: Wikipedia is an encyclopaedia; Wikipedia is written from a neutral point of view; Wikipedia is free content that anyone can edit, use, modify, and distribute; editors should interact with each other in a respectful and civil manner; Wikipedia does not have firm rules.

Source: Wikipedia.org

The last point of “no firm rules” is important in the sense that entries on Wikipedia revolve around the idea of an open dialectic. Knowledge creation through Wikipedia is open for all, under continuous scrutiny and subject to active stress testing by anyone out there engaging with the subject matter at hand.

Compare this approach to Britannica, where contributions are exclusively received from commissioned experts and scholars. Entries are subject to tight editorial control.

For obvious reasons (such as the lack of disciplined editorial review and the fact that entries can be made by anyone), Britannica vehemently disputes the idea that Wikipedia should be considered a viable alternative as suggested by Nature back in 2005. Nature’s study looked at 42 pairs of scientific articles from Wikipedia and Encyclopaedia Britannica respectively, all of which were of similar length representing broad disciplines (see details of how the survey was conducted). Each pair of selected entries was reviewed by experts who looked out for factual errors, critical omissions and misleading statements. 123 were found out for Britannica, 162 for Wikipedia. Based on those results, Nature argued that Wikipedia comes close to Britannica in its coverage of scientific topics.

Britannica criticised Nature’s findings as fatally flawed based on the argument that its methodological approach and conclusions were unsound (you can read the full demolition of Wikipedia by Britannica here). In return, Nature dismantled Britannica’s charge of flawed analytics in a point-by-point rebuttal (see here).

Assessing the veracity of Wikipedia, which in essence is the embodiment of an inclusive social project built around the idea of empowering an open, knowledge-based society, is complex and by no means a simple task. That’s why Epic/Oxford decided to repeat the exercise, albeit using a more complex methodological approach.

The Epic/Oxford pilot study (2012) aims to:

1. Explore the opinion of expert reviewers regarding attributes relating to the accuracy, quality and style of a sample of Wikipedia entries across a range of languages and disciplines.
2. Compare the accuracy, quality, style, references and judgment of Wikipedia entries as rated by experts to analogous entries from popular online alternative encyclopaedias in the same language.
3. Explore the viability of the methods used in respect of the first two aims for a possible future study on a larger scale.

Emphasis is put on testing a methodological approach that will stand up to scrutiny as the foundation for a large scale study. For this purpose, a special feedback tool was devised that provides reviewers (all experts in their respective scientific fields) with a wide range of quality criteria, drawn from extensive previously published research (Epic/Oxford, 2012). A sample of twenty-two articles was subject of this pilot study; five dimensions for assessment were devised: i) accuracy, ii) references, iii) style/ readability, iv) overall judgment (including citability), v) overall quality score.

The bottom line is that the pilot indicates that Wikipedia scores rather well in relation to the majority of the five variables above when compared to rival encyclopaedic entries. However, considerable challenges were noted concerning the identification of appropriate articles fit for comparison, anonymising articles, and recruiting a sufficient range of suitable reviewers if a large scale study was to be embarked upon (Epic/Oxford, 2012). For this reason, Epic/Oxford recommends that more research could be done on how users interpret and make sense of content from online encyclopaedias in general and from Wikipedia in particular (Epic/Oxford, 2012).

This conclusion is an important one as it highlights the complexities involved in objectively assessing crowd-sourced information points, such as Wikipedia. This important pilot also opens up the discussion on the validity of information and research acquired via non-traditional means. Despite the challenges noted above, is it time for academic and perhaps commercial content producers to start re-assessing their guarded relationship with Wikipedia compared to their open embrace of commercial alternatives?

See also Konieczny's Wikis and Wikipedia as a teaching tool: Five years later