Why is it so difficult to reproduce the results from so many academic research papers, and what does that mean for a scientific culture that is in part founded on the notion that when experiments are repeated they should yield similar results?
The Economist takes a look at the phenomenon, which has been most acutely observed in psychological priming studies, which look at subconscious thought triggers to see how they impact other thoughts and behaviors.
For example, a recent paper found that nine separate experiments failed to reproduce the results of a famous 1998 study which showed that if subjects are asked to think about a professor before taking an intelligence test they score higher than those who were asked to think about football hooligans.
It is easy to see how such studies with some inherent vagueness can be hard to reproduce, but the problem is not limited to priming projects.
Researchers at Amgen tried to replicate 53 studies that they thought were landmarks of cancer research, and even through the co-operated with some of the original researchers to get match their experiments they were only able to reproduce the original results in just six of those experiments.
Another group at Bayer HealthCare tried to reproduce the results of 67 studies and was only successful in about a quarter of them.
A range of factors may be at work when research results prove stubborn to reproduce.
Statistical mistakes in complex data analyses are common, professional competition and pressure may hurry investigators and drive them to publish results before they are completely vetted, and peer reviewers are not as good at spotting errors and inconsistencies as everyone thinks they are.
“There is no cost to getting things wrong,” Brian Nosek, a psychologist at the University of Virginia told the magazine. “The cost is not getting them published.”