Bad Science?

I came across this infographic via the excellent Communicate Science blog, and it certainly makes for interesting reading. A couple of claims (taken from a variety of sources including a PLoS One Systematic review) in particular stand out:

• Misconduct rates are highest among clinical, medical & pharmacology researchers
• In a sample of 281 clinical psychology papers, 15% contained an error which would have changed the conclusion

As a medical librarian, the former is obviously a cause for concern. Promoting critical appraisal is a key element in evidence-based practice, but how can you tell if the original data is falsified or fabricated? You can't of course, unless all raw data is made publicly available and archived in a transparent way (and even then, it may still be difficult). The Dryad consortium offers a hopeful model for the future.

Given that a number of the statistics are based on surveys and self-reporting, it is likely that if anything, some figures understate the true extent of the problem. If one in three scientists admit to using questionable research practices, I wonder what the total really is, including those who don't admit to doing so when surveyed about such a sensitive topic?

If it is the pressure to publish the 'right results' in the 'right journals' to obtain funding which is in part fuelling this misconduct, then high impact journals like NEJM may be particularly vulnerable to researchers who massage data to further their career. In the world of medical research, recommendations and practice change rapidly. Last year a study in Archives of Internal Medicine examined the frequency of medical reversal by taking a sample of the articles published in the NEJM in 2009. The authors reviewed the conclusions of original research studies published during the year and found that:

• 49% reported a new practice that was better than current practice and
• 13% reversed a current practice

These figures underline the importance of new research in the field of medicine, and the very real impact it has on clinical practice. However if - as 'Bad Science' indicates - a proportion of such studies (For clarity, I am referring to studies generally, rather than those relating to the NEJM sample) are likely to be predicated on false data or questionable research practices, how do we really know when practice should be changed?  Where do you draw the line in investigating the rigour of what we already apparently 'know' to be true at the expense of innovation? These issues are complex enough without the threat of bad science hanging over the results as well.


Created by: Clinical Psychology