What a Result! Wait, a Second...

Highly-cited biomarker studies tend to report larger effect estimates for associations than subsequent meta-analyses do, says a literature review published in the Journal of the American Medical Association. John Ioannidis and Orestis Panagiotou combed through the ISI Web of Science to find studies that had been cited more than 400 times and had been published in a highly cited journal, and they searched for meta-analyses of those studies. From this, they evaluated 35 highly cited associations. "For 30 of the 35 (86%), the highly cited studies had a stronger effect estimate than the largest study; for 3 the largest study was also the highly cited study; and only twice was the effect size estimate stronger in the largest than in the highly cited study," Ioannidis and Panagiotou write. Ioannidis tells news@JAMA that "the key message is that results that seem spectacular are very interesting, but researchers need to wait for further validation from independent research teams and large-scale evidence."


It is also a function of

It is also a function of publication biases. Highly cited papers are often the first in their field, and are cited preferentially by temporal precedent not actual scientific long-term validation. But to get the first publication in the field, it's easier if findings seem stronger. So there is a bias on the first papers to come out, followed by a bias in the way citation works. Eventually however, citation rates probably become more reflect of the most important not the first papers in a field, but for a lot of GWAS studies (as one kind of biomarker) this has not yet happened.

The Loannidis and Panagiotou

The Loannidis and Panagiotou meta-analysis study confirms what most of us in the biomarker discovery field have long suspected so their results are not surprising. The apparent specificity of most biomarkers seems to decline from the moment of their discovery as they are subjected to further scrutiny. The same could be said for drugs that are supposed to be specific for their intended protein targets. This problem of specificity associated with new biomarker tests and therapeutic drug candidates underlies the general decline in the annual rates of government regulatory approvals for these types of products over the last decade and a half.

In view of this, personal genome-wide sequencing of SNP's and mutations is highly unlikely to provide a reliable assessment of the disease potential for most individuals who choose to purchase such services. On the one hand, as meta-analyses become possible with larger numbers of individual genomes and their disease associations, it is most probable that the vast majority of SNP's and mutations will turn out to be very weak biomarkers. On the other hand, some very general conclusions for disease predisposition might emerge from some sort of statistical analyses of thousands of potential biomarkers collectively - essentially a "death by a thousand cuts" assessment. Of course, it may take a few decades between the sequencing of one's genome and the final proof that the prognostic implications had any validity.

It is a well kept secret that

It is a well kept secret that most if not all papers in the field are packaged in hype. You will not find a paper that will not have a last sentence saying something like: our results provide a new cancer biomarker and new insights needed to improve cancer diagnostics and treatment...