With preliminary data in hand, the second phase of the MicroArray Quality Control Consortium project is taking a closer look at quality control issues that might impact the use of microarrays in the clinical setting.
During MAQC’s seventh face-to-face project meeting, held May 24-25 on the campus of the SAS Institute in Cary, NC, consortium participants presented some initial results of the second phase of the project, called MAQC-II. While MAQC-I evaluated the reproducibility of microarray experiments across different labs and platforms, MAQC-II, which officially kicked off last fall, is focusing on the prediction of biological outcomes based on microarray data.
Specifically, the initiative is addressing the challenges of developing and confirming predictive models that use gene expression profiles to predict disease recurrence, prognosis, response to treatment, and other outcomes of clinical interest. Participating groups are independently analyzing several clinical data sets with the goal of identifying “best practices” for developing classifiers that are reliable enough for use in a clinical setting.
Many of the participating groups in Phase II have just started looking at the data in the study, but these initial experiences have already led to some useful insights — and raised some important questions (see BAN 12/19/2006).
One question raised by the group’s experiences so far is how different research groups conduct their internal QC. Wendell Jones, director of statistics and bioinformatics at Expression Analysis and a coordinator of MAQC-II’s Clinical Working Group, noted that this issue is more critical in the clinical setting than in the research setting, because if an array is determined to be aberrant or faulty in a clinical study, “you must then decide whether to extract another sample from the patient, and that would not be your first choice.”
The issue, Jones said, is that there are no definitive criteria for evaluating the quality of arrays for classification purposes. As an example, he noted that for a data set from MD Anderson Cancer Center — 178 Affymetrix GeneChips from a breast cancer study — MDACC “pre-identified” 17 arrays that it considered to be failures. When those same arrays were distributed to 14 different MAQC groups to perform their own QC validation, however, there was noticeable disagreement regarding the poor quality arrays. While 80 percent of the groups agreed that 13 arrays were unacceptable, only one array was considered poor quality by all 14 institutes, and more than half the arrays were deemed “unacceptable” or “aberrant” by one of the groups.
As a result, Jones said that the Clinical Working Group has formed a QC subgroup that will conduct a “rigorous” evaluation of “quality assessment thresholds” and will also explore the impact that aberrant arrays might have on study outcomes if they are used in developing a predictive model.
Jones said that the QC working group also plans to create a set of “simulated” aberrant data from good quality arrays to represent the impact of bubbles, black holes, gradients, 5’ loss, random error, and batch effects on statistical results, especially predictive outcomes.
In a follow-up interview with BioArray News, Wendell Jones said that the QC group consists of about 50 people, roughly one quarter of the CWG. He said that array users all have their own rule of thumb on what arrays are considered aberrant based on their own needs and environments, but that the CWG hopes to arrive at a consensus on what those technical parameters are for classification and prediction.
“We'd like to be more decisive in terms of which arrays should be considered suitable for this kind of application,” he said this week. “We feel it is very important to assess more stringently what the potential impact a poor quality array has to an accurate prediction on each array.”
Jones said that ultimately the group will seek to publish its results.
Breast Cancer Tests Being Examined
Fraser Symmans, associate professor of pathology at MDACC, raised another issue related to microarray quality control in clinical applications during the meeting, though one that is not yet being addressed by the MAQC.
Symmans proposed an effort to “establish the reproducibility of microarray-based test for breast cancer” across different labs, citing the rapid development of commercial array-based breast cancer prognostics from companies like Agendia and Genomic Health.
While both of those firms run their tests in their own labs, Symmans noted that the community will eventually need to ensure reproducibility of array-based tests in multiple reference labs.
And even though MAQC-I proved that microarrays are reproducible for research use, Symmans said those findings are not quite good enough for the clinic.
Symmans proposed a study in which RNA would be profiled in several different reference labs using both the Agilent and Affymetrix platforms in order to “establish reproducibility between cytologic samples and tissue samples from the same tumor.”
Symmans noted that he has already received “verbal commitments” from eight medical centers to collect samples with patient consent. He proposed that all the data from the reference labs be provided to a centralized data-analysis center that will evaluate the results.
Expanding Into Genotyping
The MAQC is also taking a step beyond gene expression analysis with the launch of a new workgroup that will identify best practices for genome-wide association studies.
If an array is determined to be aberrant or faulty in a clinical study, “you must then decide whether to extract another sample from the patient, and that would not be your first choice.”
Leming Shi, a researcher at the US Food and Drug Administration’s National Center for Toxicological Research and coordinator of the MAQC initiative, announced the new group during the meeting after several presentations arguing in favor of its creation.
The new workgroup will be the fifth under MAQC-II, which previously comprised four: the Clinical Working Group, which is analyzing patient data from large-scale clinical studies; the Toxicogenomics Working Group, which is doing the same for toxicogenomics experiments; the Titrations Working Group, which is following up on titration samples from MAQC-I; and the Regulatory Biostatistics Working Group, which is advising the Clinical and Toxicogenomics groups on ways to evaluate the performance of predictive models and classifiers.
However, according to several speakers, the recent rise of array-based genome-wide association studies has presented a number of issues that the consortium would be better off addressing sooner rather than later.
Nick Xiao of SAIC Frederick said during the meeting that “MAQC has successfully proven that microarray technology can be used for biomarker discovery,” and noted that the group can apply many of the lessons learned from MAQC-I to show that “genotyping technology can be just as trusted and just as robust.”
In a similar presentation, Federico Goodsaid, senior staff scientist in the genomics group at FDA’s Office of Clinical Pharmacology, said that the FDA has been getting “a number” of genome-wide association studies under its Voluntary Exploratory Data Submissions guidelines. These GWAS experiments have “enormous sources of variability at each analysis step,” Goodsaid said, yet there is currently “no framework to tell us why [submitters] are doing this or that.”
These problems, he said, are “amazingly parallel” to the issues MAQC-I explored for gene expression analysis.
In a follow-up interview with BioArray News last week, Goodsaid said that the GWAS working group will focus on defining “best practices” before it starts writing any manuscripts.
“I wouldn’t go as far as to characterize the goal of this work to be a “best practices” paper,” he said. “Let’s see first whether we can conclude from the data and analyses that we have a chance to find best practices.”
At the MAQC meeting, Shi announced the formation of the GWAS working group and named Goodsaid and Xiao as coordinators. The first task for the group, he said, will be to identify experts in academia, industry, and government who will be willing to assist with the project.
Shi noted that GWAS experiments present several “new challenges” for the microarray community, “but a lot of similarity with what we saw in gene expression.”
Ultimately, he said that the objectives of the new working group align with the broader goal of MAQC-II, which is to predict health outcomes based on microarray measurements of biological samples.
“For personalized medicine to be realized, we have to be able to make a prediction for each patient,” he said.
— Justin Petrone contributed to this article