Irreproducible Results

Why are many research results so hard to reproduce?

Full-text access for registered users only. Existing users login here.
New to GenomeWeb? Register here quickly for free access.

There certainly is a cost to

There certainly is a cost to getting things wrong, particularly now that our publications are tracked for number of times they are cited by other publications.

That said, probably some types of research are inherently harder to reproduce than others. Genetic findings are often highly replicated. As long as clinicians agree about medical diagnoses (which is not always the case), then ultimately there either is or is not a sequence variant in a gene, so ultimately the readout is digital. In research where the readout is much more quantitative, or even continuous, and where one studies complex systems prone to non-linear response, then reproducibility is intrinsically much more difficult as lab experimentation is more sensitive to local conditions. This should not be surprising, although we may tend to overestimate how reproducible our results really are. Still, science (and the attendant engineering) would be much less successful than it is if nothing was reproducible.

The costs of such shoddy

The costs of such shoddy science - and shoddy science funding - is enormous. Many clinical trials are based on university-derived data, now known to be wrong 80% of the time. Hundreds of billions of dollars down the drain each year. This same errant data is the fodder for implementation of Personalized Medicine as well - another arena into which billions of dollars are flowing, public and private.

Garbage IN - Garbage OUT.

In the meantime, fewer new drugs and diagnostics that cannot work. A high price to pay in money, unmet medical needs and personal suffering.

I and several others in the

I and several others in the Microarray Quality Control Consortium (MAQC) examined this very phenomenon and described the results in a paper http://www.biomedcentral.com/1471-2105/9/S9/S10. Microarrays (and now, next-gen sequencing) are commonly used genomic tools for investigating a wide range of diseases and biology including cancer. When one measures thousands of analytes simultaneously, a poorly-understood phenomenon occurs when trying to replicate the results of such an experiment that is strictly due to statistical methods associated with "cherry-picking" the final summarized results and is not necessarily due to poor laboratory methods or a bad experimental design. Yes, there are irreproducible results that are due to bad protocols, mistakes, as well as outright fraud. A separate and prominent class is due to selection bias (only the significant results are published--we don't see the results of the same or similar studies that are not significant, leading to the selection or publication bias). However, my hunch is that many of the irreproducible results in genomic science are related to the statistical methods that summarize them and how we measure similarity between studies.

I still remember in the years

I still remember in the years of burgeoning expression analysis, seeing scientists use (even to this day) a mix of tissue and a terrible control gene to do CT comparative studies. I was proved without the shadow of a dough to a scientists that his control gene, which he had derived from literature rather than valid bench validation was incredibly poor... for the love of god... 10 years of crappy data down the drain... all published in reputable papers. How many PhD did I meet like that ? Scary just thinkin about iLinkage study is another scary sad story...

Re: rayperkins, "Many

Re: rayperkins, "Many clinical trials are based on university-derived data, now known to be wrong 80% of the time." Do you have a citation to support this? Is this based on your research, or on the research of others? If this is indeed the case, having a reputable and reproducible source/study to cite to would be helpful.

Oh give me a break man, you

Oh give me a break man, you bore us... Go do another autism linkage study please...