Only a handful of randomized clinical trials have been re-analyzed by independent researchers, report researchers led by Stanford University's John Ioannidis in the Journal of the American Medical Association.
He and his colleagues scoured Medline to find studies that re-examined data from randomized clinical trials, uncovering only about three dozen such studies housed in the database. Most of those studies, the researchers further noted, came from the labs that performed the original work — only five re-analyses were performed by independent groups. Still, they noted that more than a third of the re-analyses interpreted the data differently than the original reports did.
“One would like to have trusted that the results and conclusions of randomized trials are reproducible and not subjective or dependent on who analyzes their data and how the data are analyzed," Ioannidis tells Ed Silverman at Pharmalot. “The fact that we found a third of the published re-analyses led to different conclusions on how many and which patients should be treated raises concerns.”
Silverman also notes gaining access to clinical trial data for re-analysis may be challenging because of patient privacy and trade secret concerns.
In a related editorial in JAMA, Yale University School of Medicine's Harlan Krumholz and Eric Peterson from Duke University Medical Center and an associate editor at JAMA write that replication is an important aspect of the scientific endeavor.
"The recognition that one trial can potentially lead to different findings and conclusions depending on many discretionary decisions that are made about the data and re-analyses almost mandates that those choices are transparent and described in detail — and that others have the chance to replicate them," they say. "Rather than the rare exception, open science and replication should become the standard for all trials and especially those that have high potential to influence practice."