Cancer researchers tend to overestimate how reproducible preclinical studies may be, according to a new analysis.
Researchers from McGill University and York University asked basic and preclinical cancer scientists to predict whether mouse experiments from six prominent preclinical cancer studies were likely to be reproducible. These six studies were the first ones for which the Reproducibility Project: Cancer Biology conducted replication studies, and the five replication studies that had results at the time had not been able to replicate either the significance or the effect size, though those studies have been the subject of some criticism.
As they reported yesterday in PLOS Biology, the Canadian team found that the cancer scientists said they thought there was a 75 percent probability of replicating the statistical significance of those studies and a 50 percent probability of replicating the effect size. The researchers also noted that scientists with a higher h-index — a measure of productivity and citations — tended to be better at gauging reproducibility, while scientists with more specialized knowledge has less accurate estimates.
"This is the first study of its type, but it warrants further investigation to understand how scientists interpret major reports," study author Jonathan Kimmelman says in a statement. "I think there is probably good reason to think that some of the problems we have in science are not because people are sloppy at the bench, but because there is room for improvement in the way they interpret findings."