I am pleased to announce that Anish Koka and I will start a regular program on YouTube (and possibly later as a podcast) in which we cover topics of mutual interest in medicine and healthcare policy, including the latest scientific or policy brouhahas.
In our first episode, we discussed a recent editorial by John Ioannidis in which he considers the pros-and-cons of lowering the P-value threshold for statistical significance from the currently commonly accepted 0.05 to a lower threshold of 0.005. Bizarrely, Ioannidis thinks this could be a good idea (at least, that’s my interpretation of the paper. Anish gives him the benefit of the doubt).
Ioannidis, of course, made a big splash in 2005 with a paper entitled “Why most published research findings are false,” subsequent to which he rose to prominence beyond academia. (He was featured in a front-cover article in The Atlantic and in The Economist, and now leads the impressive-sounding Meta-Research Innovation Center at Stanford (METRICS). Here is his Wikipedia entry.) The meaning of false in that paper was not reproducible, so the paper marked a milestone in what is now commonly known as the “replication crisis.”
In the course of my conversation with Anish, I mentioned that the real crisis may very well be that the idea of “replication” as a standard of truth is not as sound as it may seem. In particular, I mentioned a recent editorial by Ioannidis and colleagues Goodman and Fanelli in which the authors acknowledge that replication can refer to different things, including replication of methods, replication of results, and—get this—replication of interpretation!
They give replication of interpretation the more technical sounding name of inferential reproducibility, which they define as “drawing out qualitatively similar conclusions from either an independent replication of a study or a reanalysis of the original study.” To Ioannidis et al., the lack of inferential reproducibility may be the “most important” and “under-recognized” dimension of the reproducibility crisis.
I’ll let you draw your own conclusions as to the significance of that definition, but if you want to hear our take on it, here it is: