Unitended consequences: Open data leading to biased reanalyses
Last week, Stephen Colbert interviewed Leon Wieseltier, editor of the New Republic. Ever the provocateur, Colbert immediately challenged Wieseltier to state his critique of modern culture in 10 words or less. This is what Wieseltier came up with on the spot:
Too much digital, not enough critical thinking, more physical reality.
Ten words exactly. Colbert was duly impressed.
I encountered this critique a day after reading a number of articles with issues of data access (open data) and data reanalysis at their core. One of these also touches on the seemingly endless attempts to link vaccines and autism, despite repeated results showing no link, including said reanalysis itself. Overall, early signs indicate that there are significant responsibilities coming our way as we make data more accessible, and we will need to carefully consider how to proceed.
In August, the BioMed Central journal Translational Neurodegeneration published a paper by engineer-turned-biologist Brian Hooker, in which Hooker reanalyzed 2004 CDC data and found a purported link between measles-mumps-rubella (MMR) vaccines and autism diagnoses. His reanalysis showed no link, except in a subgroup of African-American children. Unfortunately, it was a case-control study, and he reanalyzed it as a cohort analysis, just the beginning of the flaws in the paper.
The story that has emerged since then is long and sordid, involving a strange and sensationalistic video of a supposed confession of data suppression (by a “senior CDC researcher” with no trace of existence on Google and other factual problems), an admission from Hooker that he is not an unbiased participant due to his own son’s autism, a panicked takedown of the paper by the publisher shortly after publication, and the reappearance of Andrew Wakefield as narrator of the purported video confession. You may recall that Wakefield is the British physician whose paper started all this, and who was subsequently disgraced when the paper was found to be fraudulent and was retracted.
Click here to read more from this October 16, 2014 Scholarly Kitchen article by Kent Anderson.