As COVID-19 devastates both public health and the worldwide economy, it has fallen to scientists to come up with solutions, such as diagnostics, therapeutics, and a vaccine. Many researchers and clinicians have gladly risen to this challenge.
Unfortunately, the barriers to scientific progress that existed before the pandemic may only be magnified now. In particular, research reproducibility is essential to rapidly provide solutions and support global health.
A recent study, commissioned by ATCC, sheds new light on the reproducibility problem. Pressure to publish, inadequate study design, and bias toward publishing positive results are all major contributors to the problem. Fortunately, identifying these issues is an important step towards improving them. To some degree, this is a cultural issue within the scientific community, and an attitudinal shift is necessary to solve it.
The ATCC survey queried 415 researchers in biological sciences, chemistry, and medicine from North America, Europe, and Asia. Around 60 percent of respondents worked in academia and government, while the other 40 percent represented pharmaceutical and biotech companies. Roughly 34 percent of respondents were younger than 40; 56 percent were between 40 and 60 years old; and the rest were older than 60.
Respondents were particularly frustrated by the wasted time and resources associated with poor reproducibility. Around 20 percent feared the inability to reproduce their work could mar their reputations. Approximately 33 percent expressed loss of trust in colleagues and collaborators and 25 percent were becoming disillusioned with the scientific process.
When questioned about potential causes for poor reproducibility, 57 percent felt there was too much pressure to publish; 42 percent believed the bias towards publishing positive results is too strong; 26 percent felt scientists receive inadequate training in research methods; and 23 percent felt journals need to do more to emphasize methods in their publication requirements.
The vast majority of survey respondents, around 90 percent, would like to see granting agencies and publishers do more to enforce requirements that could improve reproducibility.
Methods was an important recurring theme. Scientists who had trouble reproducing studies believed part of the problem may be insufficient information about methods in the original papers. In general, 34 percent felt published studies did not include enough details about research methods to accurately reproduce that work.
"The vast majority of survey respondents, around 90 percent, would like to see granting agencies and publishers do more to enforce requirements that could improve reproducibility," said Mindy Goldsborough, PhD, chief scientific officer at ATCC. "They feel the system needs help."
A Generational Divide
Overall, the survey found that 65 percent of scientists feel reproducibility is an urgent problem. These concerns were significantly higher for academic researchers (72 percent) than industry scientists (59 percent). There was also a significant generational divide, as 71 percent of researchers under 40 felt the problem is urgent, while only 62 percent of those over 40 agreed.
"Younger scientists believe the current scientific culture is a major part of the problem and noted the concept of publish or perish puts additional, unnecessary pressure on everyone," said Goldsborough.
Around 87 percent of respondents borrow cell lines from other labs, while only 29 percent of scientists using cell lines in their work re-authenticate them.
Scientists who borrow cell lines without authenticating them have more replication issues. However, 9 in 10 scientists continue to use cell lines from colleagues and/or other labs. Those who primarily use vendor-supplied cells reported fewer problems. They also expressed less disillusionment with the scientific process, highlighting vendor-supplied lines as a possible solution.
In addition to the effect on the quality of research and reputations, poor reproducibility has a quantifiable monetary impact. Over a three-year period, industry scientists estimate they lose $45,000, while academic scientists estimate $16,500, as a direct result of reproducibility issues.
This is not the first survey to highlight the challenges of research reproducibility. Articles published in Nature and eLife found similar results. The hope is that the current findings will encourage scientists to pursue solutions.
The study points to several possibilities. Respondents report a number of ways they follow up when a study cannot be reproduced: 54 percent run the same experiment again; 42 percent consult other experts; 20 percent inform the original authors; 9 percent try to publish their failure to reproduce; and 6 percent report the reproducibility failure to the journal editors for the original article.
Other solvable issues revolve around transparency and study design. "Reproducibility suffers because studies are often not well designed," said Goldsborough. "We need to redesign studies with that in mind, building the necessary elements into the study rather than just doing what’s always been done."
This ultimately circles back to methods. Providing young researchers with better training on methods could mitigate some of these issues. That goes beyond the actual methods in the lab to transparently communicating them in papers. Increased reliance on open source tools could also support more reproducible results. Finally, moving to a system that publishes negative results could do much to increase research transparency and help enhance reproducibility.
More information about the ATCC reproducibility study is available here.