Accurate phenotypic definitions and experience in bioinformatics are among the most essential prerequisites for the development of accurate classifiers, according to leaders of the second phase of the Microarray Quality Control project.
Leming Shi, a researcher at the US Food and Drug Administration's National Center for Toxicological Research, and Federico Goodsaid, associate director for operations in genomics at the FDA's Center for Drug Evaluation and Research, told BioArray News last week that the project should provide scientists with guidance on how best to analyze microarray data when predicting clinical outcomes.
The FDA-hosted MAQC-II published a dozen papers earlier this month in Nature Biotechnology and The Pharmacogenomics Journal on sources of bias in array-based studies (BAN 8/3/2010). According to Shi and Goodsaid, the papers provide a number of take-home messages for researchers constructing classifiers for use with array data.
One was that experienced bioinformaticians performed better in the study than less experienced ones. "Given the same data sets, different data analysis teams showed varying degrees of proficiency in developing good-performing classifiers," said Shi and Goodsaid, who provided joint comments for BioArray News via e-mail. "Experienced industrial participants performed better than less-experienced academic participants."
Shi and Goodsaid also remarked that the "performance of a classifier in blinded validation is largely dependent on the nature of the problem or endpoint and can be reasonably well estimated during training, if 'good practices' of modeling are followed." They noted that "different combinations of analytical approaches can lead to classifiers with comparable performance; simple approaches usually perform as well as more complicated ones."
According to Shi and Goodsaid, "accurate phenotypic definitions for the populations in a study are essential for the development of accurate classifiers" and "exhaustive record-keeping in classifier development is essential to understand the rationale behind the development of a classifier."
This is particularly true for researchers engaged in genome-wide association studies. Going forward, Shi and Goodsaid advised that those conducting GWAS should recognize that "analyses of complete datasets with no batch effects minimize variability in the results." In addition, "different analytical algorithms result in the identification of different sets of genetic markers to associate with the clinical phenotype," and "accuracy in phenotypic definitions is essential to identify GWAS associations."
While team experience, batch effects, and definition of phenotype can impact classifiers, MAQC-II did not determine that choice of array platform can affect results. Shi and Goodsaid said that platform choice should have no impact "if the platform has been validated and used properly." They said this finding "applies to platforms for identifying differentially expressed genes, as well as to GWAS associations."
"MAQC-I and –II show that it’s the sequence sampled, and not the technology behind a specific platform that determines the information generated by a genomic platform," Shi and Goodsaid said.
They also said that the poor performance of some classifiers did not mean that other technologies might be more reliable in making accurate, reproducible predictions.
"If the original phenotypic definitions are not accurate, or the analytical protocol is sub-optimal, then the outcome is a classifier which may not survive a blinded confirmatory study," Shi and Goodsaid noted. "Other technologies beyond measuring RNA expression levels may be helpful to make better clinical predictions, but those technologies will need to undergo scrutiny just like microarrays."
Indeed, Shi and Goodsaid confirmed that a third phase of the MAQC project is underway.
"Working with data from novel sequencing technologies," the project aims to identify analytical issues which may help in the accuracy of associations determined from these platforms. "This task is fairly complex because of the fluid nature of novel sequencing platforms," Shi and Goodsaid said.
The MAQC was created in February 2005. The first phase of the project evaluated the reproducibility of microarray experiments across different labs and platforms using two RNA reference samples. Results from those studies were published in a special issue of Nature Biotechnology in September 2006.