On the day that we taped our podcast episode with Brian Nosek about the replication crisis, the renowned statistician Andrew Gelman published an essay in the New York Times on precisely that topic.
Gelman echoes some of the remarks made by Nosek. In particular, he draws attention to the point we discussed regarding what to believe when the results of an experiment are contrary to what is expected. Commenting on a recent study (co-authored by Nosek) in which statisticians were asked to predict, ahead of a replication experiment, whether the findings of the original study would be replicated or not, Gelman says:
Here’s where it gets really weird. The lack of replication was predicted ahead of time by a panel of experts using a “prediction market,” in which experts were allowed to bet on which experiments were more or less likely to — well, be real.
We tried to press Brian on that point. Do certain subjects impose essential limitations on empirical science? It seems to me that certain questions regarding human behavior are essentially answered by human judgment rather than by “mindless” data analysis, despite the fact that human minds can be prone to bias. Brian agreed but also brought up some counterpoints. Don’t miss listening to that episode.