This week the Microarray Quality Control Consortium — an effort spearheaded by the US Food and Drug Administration to evaluate the reliability of DNA microarray data — published its results in a special issue of Nature Biotechnology.
The publication of the collection of articles, available here, made good on an earlier timeline set by the MAQC to release its results by the end of the summer (see BAN 3/28/2006). According to the MAQC, the project proved that microarray data from different platforms can be reproducible and comparable and will provide the microarray community with metrics and thresholds to compare different gene expression platforms.
Still, while MAQC authors have stressed that the results put to rest questions about the reliability and repeatability of microarray experiments — as well as concordance across different array platforms — some in the array space unaffiliated with the study are approaching the findings with caution.
Over the past year, MAQC has run two "standard" RNA samples — from Ambion and Stratagene — across eight different microarray platforms at six separate testing sites to identify the QC controls. The eight array platforms used were Applied Biosystems, Affymetrix, Agilent, GE Healthcare, Eppendorf, Illumina, Telechem, and an array using oligos made by Operon and printed by the National Cancer Institute.
The project compared multiple whole-genome gene expression profiles across platforms. The main study carried out 60 hybridizations on each of the platforms. Around 1,300 microarrays were used during the entire project.
According to a Nature Biotech editorial summarizing the results of the project, the study confirms that “with careful experimental design and appropriate data transformation and analysis, microarray data can indeed be reproducible and comparable among different formats and laboratories, irrespective of sample labeling format.”
The study also found that fold change results from microarray experiments “correlate closely with results from assays like quantitative reverse transcription PCR,” and that variation between microarray runs was “relatively low.”
“The data demonstrate that arrays do provide repeatable results within labs, reproducible results between labs, and that different platforms can generate similar sets of differentially expressed genes,” Laura Reid, director of R&D at Expression Analysis and an author on the MAQC study, told BioArray News last week.
The Nature Biotech papers represent the first formal results of the study to be published. The data from the main study is also now available publicly through the National Center for Biotechnology Information’s Gene Expression Omnibus database, the FDA’s ArrayTrack database, and the European Bioinformatics Institute’s ArrayExpress database.
Towards Better Experiments?
Now that the results have been published, those involved say that the benefits for the array community are three-fold. First, MAQC authors say that the results will help all microarray users design better experiments. Second, they say it puts to rest issues related to concordance. Finally, they see the availability of the data as useful to the array community.
According to Janet Warrington, vice president of emerging markets and
molecular diagnostics R&D at Affymetrix, the results “will contribute to a better understanding of the important decisions that one makes in designing and executing an experiment.”
molecular diagnostics R&D at Affymetrix, the results “will contribute to a better understanding of the important decisions that one makes in designing and executing an experiment.”
“We hope that the results will build confidence that when scientists design adequately powered experiments, use good laboratory practices, and select and implement algorithms carefully that they will obtain highly reproducible, robust results,” she told BioArray News in an e-mail this week.
Expression Analysis’ Reid said that microarray users should use careful annotation and probe mapping when comparing the same transcript across platforms. She added that the results of the project “also showed that some of the apparent lack of concordance in previous studies can be attributed to the method of data analysis.”
Wendell Jones, a senior research statistician at Expression Analysis and MAQC author, noted that concordance has been “a point of contention” in the microarray community “because some previous published papers reported that they could not get reproducible results.”
“The MAQC study resolves this controversy,” he said. “It is also more comprehensive than the studies that were done before in that the MAQC project involved multiple platforms, multiple labs for each microarray platform, and multiple alternative techniques.”
The Cam Paper
While Reid and Jones did not specify which previous study or studies they were referring to, Richard Shippy, MAQC co-author and GE Healthcare R&D scientist, said that the MAQC results resolve questions about concordance between array platforms raised by an October 2003 paper in Nucleic Acids Research authored by Margaret Cam, the director of the Microarray Core Lab at the National Institute of Diabetes and Digestive and Kidney Disorders, and others [Nucleic Acids Res. 2003 Oct 1;31(19):5676-84].
“The Cam paper showed poor correlation across platforms,” he said. He pointed out that some of the paper’s results could be explained by unrefined probe mapping techniques, an issue that the MAQC study addressed.
|
“Concordance has been a point of contention because some previous published papers reported that they could not get reproducible results. The MAQC study resolves this controversy.” |
Cam’s study was similar to the MAQC project, but on a smaller scale. The study compared results of gene expression measurements from identical RNA preparations using three commercially available microarray platforms.
“RNA was labeled and hybridized to microarrays from three major suppliers according to manufacturers' protocols, and gene expression measurements were obtained using each platform's standard software,” the Cam paper’s abstract states. However, “correlations in gene expression levels and comparisons for significant gene expression changes … showed considerable divergence across the different platforms, suggesting the need for establishing industrial manufacturing standards, and further independent and thorough validation of the technology.”
But despite the apparent similarity, Cam said that the results of her paper and the papers published by the MAQC project are incomparable because the MAQC study did not use a “real biological study” to gauge concordance.
“I believe the MAQC project is an important one in that the level of cross-platform concordance in this study represents the ceiling of what to expect from the technology at this point in time,” Cam told BioArray News this week.
“It's all about signal to noise, and the current study has demonstrated that with large signal to noise ratio, using technical replicates from very different samples, one can get the sort of concordance observed in this study,” she said.
“However, it is incomparable to previously published cross-platform comparisons that have used the framework of a real biological study — using biological replicates with relatively small differences between groups — to measure concordance rates,” Cam added.
Cam said that, from her perspective, researchers will benefit primarily from the public availability of the MAQC data.
“I am very encouraged by the fact that the data has been made public, and am looking forward to seeing this data analyzed by others,” she said.
Roger Bumgarner, the director of the Center for Expression Analysis at the University of Washington in Seattle who is unaffiliated with MAQC, agreed with Cam’s assessment.
“It is quite nice to have a large body of mostly well-designed and published research to which we can point others,” Bumgarner told BioArray News.