A 1962 study by NYU psychologist Jacob Cohen showed that researchers were not publishing unsuccessful studies, and even suggested that some researchers were reporting false positives in order to look successful, The Economist reports. But while it may be tempting to say that was a long time ago, a recent paper from Paul Smaldino of the University of California, Merced and Richard McElreath at the Max Planck Institute for Evolutionary Anthropology showed that the problem Cohen warned about has persisted.
"Not only are dodgy methods that seem to produce results perpetuated because those who publish prodigiously prosper — something that might easily have been predicted," The Economist says. "But worryingly, the process of replication, by which published results are tested anew, is incapable of correcting the situation no matter how rigorously it is pursued."
Smaldino and McElreath calculated that the average power of papers they reviewed was about 24 percent, not much higher than Cohen's 20 percent despite the years that have passed, The Economist says.
When they tried to figure out why this was happening, they found that labs that garnered funding or gained a good reputation because of studies they had published tended to pass their methods on to other labs. But these labs also tended to publish more false positives than others, The Economist reports. "Their methods were good at detecting signals in noisy data but also, as Cohen suggested, often mistook noise for a signal," the article adds. "More thorough labs took time to rule these false positives out, but that slowed down the rate at which they could test new hypotheses. This, in turn, meant they published fewer papers."
And the more a lab gets incentives for publishing, the more its methods are reproduced, whether good or bad. The Economist compares this cycle to Charles Darwin's description of natural selection.
Smaldino and McElreath conclude that if publication volume determines whether a lab is successful, then labs will continue to cut corners in order to publish more and be considered successful.
To solve this problem, The Economist suggests, "universities and funding agencies [should] stop rewarding researchers who publish copiously over those who publish fewer, but perhaps higher-quality papers."