A pair of surveys examining more than 1,700 peer-reviewed publications whose authors used quantitative real-time PCR has revealed a lack of transparent and comprehensive reporting of essential technical information — and that the most prestigious scientific journals are the biggest offenders.
The surveys also showed that the number of papers citing MIQE — a set of qPCR experiment guidelines devised by concerned researchers — is still far outnumbered by those not citing the guidelines; however, those that do cite MIQE show the improved transparency and reporting of technical details necessary for high-quality, reproducible results.
As qPCR has grown ubiquitous as a research and molecular diagnostic tool for detecting and quantifying nucleic acids and gene expression, many researchers have been guilty of taking the technique for granted. Multiple publications have demonstrated that qPCR results are only meaningful if a number of complex technical steps and quality control provisions are taken; however, there have been many instances of published, peer-reviewed papers that shirked these necessary steps, resulting in ambiguous or downright meaningless qPCR data.
In order to help address this issue, a group of concerned scientists working in the qPCR space in 2009 published the Minimum Information for Publication of Quantitative Real-Time PCR Experiments, or MIQE, guidelines.
These guidelines address the most important criteria for determining the quality of qPCR- and reverse transcription qPCR-based data. Some of the parameters are more critical than others, but four categories are especially important: RNA quality, reverse transcription conditions, PCR assay details (such as specific primer and probe information), and data analysis methodologies.
As PCR Insider reported in late 2009, in the early days of MIQE, the guidelines were only just beginning to be taken seriously by the research community (PCR Insider 12/31/2009). Several months later, PCR Insider reported that quality control for publication of qPCR data was still severely lacking, and that many of the most prestigious scientific journals were guilty (PCR Insider 6/3/2010).
Some four years later, the tide is starting to turn, but according to the results of the recent surveys, much work remains to be done.
A group of more than 80 scientists working in the qPCR field conducted the surveys, the results of which were published in the November issue of Nature Methods.
In the first survey, the researchers each chose 20 papers published between 2009 and 2011 in any journal. The researchers selected papers based only on whether they were pertinent to their area of research, and did not select the papers based on compliance or expected compliance with MIQE criteria. This resulted in an evaluation of papers from 80 scientific journals with impact factors ranging from 1.9 to 32.2.
Fourteen key MIQE criteria were selected for analysis, and the data were stratified into three groups: an impact factor of less than 5; between 5 and less than 10; and 10 or higher.
All of the journals examined provided authors with the option of providing supplementary information, and the use of this option was significantly positively associated with impact factor. In contrast, the higher the impact factor of a journal, the less information about the qPCR assays was provided – resulting in a significant negative correlation between the amount of relevant qPCR-specific technical information provided and the journal impact factor.
A second focus of the first survey was to examine reporting of RNA purity and integrity, which has been demonstrated to be of "critical importance for obtaining meaningful and reliable gene expression data and for ensuring reproducibility of results," the authors wrote. The specific aim, they noted, was not to identify which methods were used to assess RNA quality, but to see if the publication made any mention of these methods at all.
Again, the survey found that reporting of RNA purity and integrity was "exceedingly poor," and that there was a significant negative correlation with journal impact factor.
In an email to PCR Insider, Stephen Bustin, a professor of medicine at the UK's Angela Ruskin University, a co-author of the report, and one of MIQE's earliest advocates, noted that "the assurance provided by a paper appearing in a high impact factor journal has been shown to be misplaced, since the higher impact factor journals are the worst offenders against transparency. It really is time that journal editors acknowledge this challenge and place more emphasis not just on adequate reporting procedures but also send manuscripts to reviewers that have the technical knowledge to assess the techniques used to generate results and conclusions."
Another ancillary finding of the first survey was that most surveyed publications indicated the use of just a single, unvalidated reference gene for qPCR experiments, data which echoed results of a 2005 survey of qPCR user practice despite the fact that the advantage of using multiple, validated reference genes was demonstrated as early as 2002 by Jo Vandesompele and colleagues at Ghent University. Once again, papers published in the highest impact factor journals were the biggest offenders of this MIQE criterium.
Noted Bustin, "[One does] wonder how it is possible that most researchers still use a single unvalidated reference gene when Jo Vandesompele showed back in 2002 that this was not appropriate."
The report authors' second survey was intended to determine whether citation of MIQE guidelines — which, they noted, have been extremely well-publicized since their introduction — is correlated with improved transparency of reporting.
For this survey, they analyzed 178 publications from 2012 and 2013 from three categories: those that cited MIQE guidelines, those that did not, and those published in three high-impact-factor journals — Nature, Science, and Cell.
The most notable conclusion from this survey, they noted, was that the MIQE guidelines did in fact have a significant impact on the quality of data reporting, with a consistent and significant increase in the comprehensiveness of reporting the 14 parameters in papers that cited MIQE, especially with regard to RNA quality, PCR efficiency, and data normalization procedures. However, they also found that papers that did cite MIQE guidelines were still vastly outnumbered by those that did not.
The survey also revealed that there was no improvement in the transparency of reporting, relative to the 2009-2011 publications, in the 2012-2013 publications whose authors did not cite the MIQE guidelines. Further, when the authors analyzed MIQE compliance in papers published in Nature, Science, and Cell, with no selection for citation of MIQE, the quality and completeness of reporting in those journals was significantly lower than that of publications selected for analysis on the basis of MIQE citation.
"Whilst it is great that we are seeing an increasing number of publications that cite the MIQE guidelines, these are still a small percentage of the papers that use qPCR," Bustin said. "Extrapolating from our results that clearly demonstrate that publications citing MIQE are more transparent in their reporting and use more appropriate analysis methods, it is obvious that most current publications report their data inadequately and present conclusions that are not supported by the data."
Bustin added that he believes the surveys' findings are unambiguous. "We have a problem with what has already been published and it is not unreasonable to assume that this problem extends to many other papers using molecular techniques such as microarrays and [next-generation sequencing]," he said. "These problems are due to inadequate quality control of samples and templates, poorly designed assays, and inappropriate data analysis."
Overall, the group concluded in their report that "the integrity of the scientific literature that depends upon qPCR data is severely challenged, and … the MIQE guidelines are useful for improving these data. We call upon journal editors to implement more stringent quality-control measures for publication."