The US National Institutes of Health focused its spotlight on reproducibility in science last January with an editorial in Nature in which Director Francis Collins and Deputy Director Lawrence Tabak proposed a number of initiatives to tackle some of the underlying issues like poor experimental design and data transparency.
Michael White at Pacific Standard notes that NIH is the only concerned organization. The Reproducibility Initiative, spearheaded by Science Exchange, PLOS, Figshare and Mendeley, is also emphasizing reproducibility. It has put together a consortium that will, for a fee, attempt to reproduce experiments researchers submit to them. Researchers, in return, receive a seal saying the work has been independently validated. The Reproducibility Project similarly aims to replicate results from highly cited papers.
But, White, who is a systems biologist at Washington University School of Medicine in St. Louis, asks: how much of an issue is reproducibility?
It is, he says, not clear, as much of the data is anecdotal and "not particularly convincing."
Additionally, White cites an argument from Chris Drummond, who is a research officer at the Canadian National Research Council. Drummond says that an exact replication of a study's results is often a waste of time and money. Instead, a reproducible result is one that can be repeated under slightly different conditions.
Of course, White notes that in some instances, such as clinical trial, exact replication is necessary.
"So by all means, scientists should push each other to do better and to avoid sloppy statistics and flawed experimental designs. But we should be careful not to take the drive for reproducibility too far," he says. "Not all published research will be reproducible, especially on the frontier of a new field, and we shouldn't always expect it to be."