A new report released this week by the National Academies of Sciences, Engineering, and Medicine addresses concerns about reproducibility and replicability in scientific research that have been expressed in scientific and popular media. "As these concerns came to light, Congress requested that the National Academies of Sciences, Engineering, and Medicine conduct a study to assess the extent of issues related to reproducibility and replicability and to offer recommendations for improving rigor and transparency in scientific research," the report states.
The report defines reproducibility and replicability and examines the factors that could lead to non-reproducibility and non-replicability. "While reproducibility is straightforward and should generally be expected, replicability is more nuanced, and in some cases a lack of replicability can aid the process of scientific discovery," the authors write.
The report doesn't go so far as to say that the replication and reproducibility concerns have reached crisis levels, Gizmodo reports. But it does recommend wide-scale improvements in how scientists, academic institutions, journals, funders, and even journalists do their work.
For years, some researchers have called for improvements in the overall quality of published research. They've highlighed common issues such as fraudulent, poorly done, or overhyped studies, with embellished findings based on small sample sizes; statistical manipulation of data; and studies with negative conclusions being suppressed, Gizmodo says.
The problem with these issues isn't just that they make scientists look bad. They've also resulted in unreproducible research. In some surveys, a majority of scientists have agreed that science is facing a legitimate problem, and some groups have started to fact-checking widely accepted hallmark studies, Gizmodo says. But the researchers whose work is being questioned have lashed back, accusing their critics of malicious motives.
The National Academies report is trying to find a middle ground. It does state that there are serious systemic gaps in how scientists conduct and relay their research, Gizmodo says, but it doesn't necessarily agree that there is a true "crisis" threatening science. It provides guidelines for better data transparency and rigor in original studies; criteria for when these studies might merit a reproduction or replication; and recommendations for how journalists should cover and report on these studies.
For journalists, the National Academies recommends reporting on scientific results "with as much context and nuance as the medium allows."
The National Academies report also concludes that scientists shouldn't worry too much about replicating any one individual study, but should rely on "reviews of cumulative evidence on a subject" to understand the state of scientific knowledge on a given topic, Gizmodo adds.