NIH Seeks Reproducibility

NIH is taking small steps to address the "complex" problem of irreproducible results.

Full-text access for registered users only. Existing users login here.
New to GenomeWeb? Register here quickly for free access.

Has this always been an issue

Has this always been an issue or is it a relatively new development?

The problem has been around

The problem has been around for a long time. I can't recall which Biopharma produced a report several years ago about reproducibility, but they were only successful in less than half of the many experiment they tried to duplicate.

We live in a world of

We live in a world of physical laws, not one that is random and capricious. Other than the quantum chemistry/physics effects seen at atomic level, we know that when you do a test a number of times exactly the same way, you will get exactly the same result. If we have a thousand chicken eggs and a hammer, and we take each egg one at a time and swing the hammer at the egg intending to hit the egg, we expect to break each egg. If we do not get that result then we know something unusual happened (we missed the egg with the swing of the hammer, or the egg was not really a chicken egg). If something is not reproducible, then it is not being done exactly the same way. One source of variation is error/variation in doing the test exactly the same way but that can be accounted for with statistics, and mechanizing or automating the method. So, the data is the data, as long as it was honestly collected, recorded and reported. However, if someone does the test again and fails to get similar data, then something has changed. Not the laws of physics but something else. Furthermore, that kind of result, something different from the expected, is always a possible outcome. It is always going to be that way. It should be expected. It should not be a surprise. It should not be a concern. It is part of what makes science and research difficult. But also what makes it fun, exciting, important and informative. When you get a result you don’t expect, something interesting is going on. The explanation may end up being trivial but there will be a reasonable explanation. Without acknowledging this fact, it seems to imply fraud and purposeful deception on the part of researchers and that’s just unfair. Yes, there will be some of that too (fraud and deception), but let’s acknowledge how amazingly difficult science and research is at the very sophisticated levels we find ourselves exploring these days.

variable experimental design,

variable experimental design, inconsistent assumptions, unintented (or unrecognized) confounders, variable sample sizes and often small sample sizes, incompatable statistical analysis, and mostly RANDOM factors that CANNOT be controlled are all part of the non-reproducibility problem.

Nothing to add

Nothing to add

The issue of data

The issue of data reproducibility is not a problem when basic science experiments are conducted at for profit pharmaceutical companies. There, experiments are conducted under design-reviewed processes, with point of process completion quality system review as the data is obtained to assure that what was intended, was in fact done and done per process requirements. These are the principles that must be followed if the for profit company desires FDA approval of their test articles to proceed into human clinical trials. Without these design-reviewed processes being documented, and demonstration that effective quality control and quality assurance systems were in place, the FDA will not view the data as credible. Those familiar with cGMP (current Good Manufacturing Practices)are aware that such rigorous processes assure the quality of the product made. If such systems were required with NIH funded research, the entire circumstance of data acquisition would be known, and removal of any unknown factors. For those interested in how these processes have been applied to not only basic research, but also to clinical research with order of magnitude greater efficiencies and superior data quality, please view the article at published in the Food and Drug Policay Forum at http://www.fdli.org/products-services/publications/~/resources/resources...

This is a huge and

This is a huge and under-appreciated problem that can set fields back decades due to the presence and then precedence of non-reproducible data. The Nature paper on this topic by Begley and Ellis (March 2012) is just the tip of the iceberg.