By analyzing the power of two widely used statistical methods to evaluate the association between certain genetic abnormalities and diseases, researchers have concluded that consumer genomics companies may have rushed to market with their service offerings, and may be inflating the value of personalized medicine.
In the latest issue of PLoS Genetics, Daniel Weeks, a professor of human genetics and biostatistics at the University of Pittsburgh, and colleagues used risk-based logistic regression and receiver-operating characteristic-curve analysis to study the clinical validity of genetic tests for several complex diseases, including age-related macular degeneration, type 2 diabetes, prostate cancer, cardiovascular events, and Crohn’s disease.
Using these two statistical methods to review the data from their own research in age-related macular degeneration and published literature on the other disease states, the researchers concluded that while a “strong association” between certain genes and diseases may be “very valuable for establishing etiological hypotheses, [it] does not guarantee effective discrimination between cases and controls.”
The scientific community “should be cautious to avoid overstating the value of association findings in terms of personalized medicine before their time,” the study authors write in the paper, “Interpretation of Genetic Association Studies: Markers with Replicated Highly Significant Odds Ratios May Be Poor Classifiers.”
Weeks and his team's findings suggest that currently published data from genome-wide analyses shows it is more important to understand the biology of a disease than to try to predict an individual’s risk of developing it.
“I think there is a need for well-designed longitudinal studies to define true risk and to understand how genetic susceptibility may interact with known environmental and lifestyle risk factors,” Weeks told Pharmacogenomics Reporter last week. “In the meantime, we need to take a step back and proceed with caution while the insights gained from these exciting new association findings are used to explore the basic biological causes of these important diseases.”
The research comes at a time when healthcare regulators, scientists, and doctors are debating the clinical value of genetic risk data sold by consumer genomics firms, such as 23andMe, Navigenics, and Decode Genetics, and questioning whether it is ethical to market genetic testing services directly to consumers.
Weeks specifically called out consumer genomics firms and the popular media for generating “hype” about consumer-targeted personalized medicine.
“There was a lot of hype in the media about personalized genomics and the advent of direct-to-consumer genetic-testing companies promising to provide risk profiles for a variety of human diseases,” Weeks said. “So we were interested in the question, ‘How well could highly associated genetic-risk factors predict individual-level risk?’”
The answer, according to Weeks' research, is not very well.
The Study
Weeks and his colleagues have been studying age-related macular degeneration since the early 1990s, and thus it became a springboard for looking at the clinical validity of currently available genetic risk data. The authors note that they are inventors in a patent filed by the University of Pittsburgh for the LOC387715/ARMS2 locus in age-related macular degeneration.
The other four diseases – prostate cancer, type 2 diabetes, cardiovascular events, and Crohn’s disease – were chosen because there was a large body of published genetic risk data on them.
23andMe, Navigenics, and Decode offer genetic risk data on age-related macular degeneration, prostate cancer, type 2 diabetes, Crohn’s disease, and a variety of cardiovascular ailments. According to Weeks, the SNPs his research team looked at and the SNPs tested for by consumer genomics firms may not always match exactly, but they are located close enough to be considered “surrogate risk predictors.”
[ pagebreak]
In the study, investigators first applied the two statistical methods – risk-based logistic regression and receiver-operating characteristic-curve analysis – to investigate the how well the genes linked to the five disease states are able to predict at-risk patients from not at-risk patients.
In a cohort of age-related macular degeneration patients, the researchers used an additive model of the CFH, LOC387715, and C2 variants and found odds ratios of 2.9, 3.4, and 0.4, with p-values of 10−13, 10−13, and 10−3, respectively. Using the ROC model, although the area under the curve is 0.79 assuming prevalences of 15 percent, 5.5 percent, and 1.5 percent in 80-year-old, 65-year-old, and 40-year-old patients, only 30 percent, 12 percent, and 3 percent of the group, respectively, was classified as high risk.
The same approach using a classification model of 12 SNPs found in the literature to be “strongly associated” with type 2 diabetes revealed an AUC of only 0.64 and two SNPs associated in the literature to prostate cancer achieved an AUC of only 0.56.
Meanwhile, the researchers’ analyses found that the discrimination power of nine SNPs liked in the literature to predict the risk of cardiovascular events was not an improvement over non-genetic predictors of cardiovascular events.
Finally, in Crohn's disease, a model of five SNPs, which included one SNP with an especially low odds ratio of 0.26, has a combined AUC of only 0.66.
“The results of these four examples, although somewhat disappointing, are not surprising given … that achieving a high AUC requires a much larger number of genetic variants than we have to date,” the study authors concluded.
According to published literature, “on average 80 common variants with [odds ratios] of 1.25 each were needed to develop a model useful for identification of high-risk individuals (AUC>0.80).”
In response to the article, Decode Genetics’ Chief Scientific Officer Jeffrey Gulcher noted that “ROC curves are indeed useful for assessing tests attempting to make a definitive diagnosis.”
For instance, “HIV status or whether a plane coming across the English Channel is German or British (the latter diagnosis based on radar signals was what Receiver Operator Characteristics was designed for during WWII),” Gulcher told Pharmacogenomics Reporter over e-mail. “They are not very useful when it comes to assessing risk factors (as has been pointed out by many others smarter than me).”
According to Gulcher, risk factors based on ROC curves are more useful when applied to reclassifying “average risk” patients as “higher risk.”
Risky Calculations
Weeks and his team note in the article that although most genetic testing companies use odds ratios to estimate an individual’s lifetime risk of developing a disease, odds ratio by itself cannot be used in diagnostic testing.
“In retrospective studies, the relative risk or risk ratio (RR) cannot be estimated unless the prevalence is known, and therefore, the OR is used as a proxy,” the authors write. "Theoretically, the odds ratio will give a good approximation for the relative risk if the prevalence is low, but otherwise it tends to overestimate the RR.
“Statisticians should easily understand this relationship between OR, RR, and risk, but a person not trained in statistics (or science in general) may not make the same distinction as easily,” the team added.
The three consumer genomics firms all use odds ratios to estimate their customers’ risk for various diseases.
In a statement to Pharmacogenomics Reporter, a spokesperson for 23andMe noted that the company “does not aim to diagnose customers with diseases or to classify our customers as cases or controls.
[ pagebreak]
“Rather, we provide information about how a customer's genotype affects his or her risk for developing certain diseases. … [Weeks and his colleagues] seem to question the utility of such risk information, which we believe is misguided,” the 23andMe spokesperson said.
The spokesperson noted the low predictive value of smoking for developing lung cancer to point out that people commonly consider various risk factors when managing their health.
“The positive predictive value of smoking for lung cancer over one's lifetime is only between 10 percent and 20 percent. So even though the PPV appears low, this does not mean that people should not consider smoking to be a risk factor for lung cancer,” the 23andMe spokesperson pointed out. “Similarly, if the PPV of a SNP for risk of age-related macular degeneration were only 30 percent, people should still be aware that they possess a factor that increases their risk for [the disease].”
Likewise, Navigenics’ Vance Vanier said that "no current screening risk factor used in clinical practice can definitively guarantee distinction between controls and cases.
“Genomic risk factors used today favorably compare with what we use in the practice of medicine,” Vanier told Pharmacogenomics Reporter. “To diminish their value because they do not guarantee outcome denies patients and physicians access to important additive clinical information and is an inconsistent assessment of medical usefulness.”
Decode’s Gulcher similarly said he felt that the study’s authors were “confus[ing] risk assessment with diagnosis.
“The genetic variants we have discovered for major common diseases are risk factors, not determinative gene variants like for Huntington's,” he said.
Cautious Claims
Weeks acknowledged that consumer genomics firms are “appropriately cautious about providing warnings about the accuracy and usefulness of the genetic risk estimates that they provide.”
23andMe, for example, states on its website that the genetic risk information provided by the company is for research and education use only. Navigenics urges customers to discuss their test results with doctors, noting that the information provided by the company does not constitute or supplant the doctor-patient relationship. And Decode warns customers to use its website and any data offered by the firm at their own risk.
Despite these cautionary statements, Weeks maintains that these companies, in concert with the popular press, are overselling personalized medicine. The most salient example of this, according to Weeks, is when Time Magazine lauded 23andMe’s “Retail DNA Test” as the Invention of 2008.
Weeks points out that the Time story claims 23andMe’s “$399 saliva test that estimates your predisposition for more than 90 traits and conditions ranging from baldness to blindness” will bring about “a personal-genomics revolution that will transform … how we take care of ourselves.”
However, the Time story “does not really discuss how accurate these disease risk predications might be,” Weeks notes.
According to Weeks, one of his study’s limitations is that the example classifiers were developed using case/control samples, which can be biased and which may not represent the general population. “These classifiers should be evaluated in an independent sample, [since] one can have over-fitting in the model-building data set,” he noted.
This study was funded with an NEI grant, by the Steinbach Foundation, Research to Prevent Blindness, the Eye and Ear Foundation of Pittsburgh, the American Health Assistance Foundation, and the Jules Stein Eye Institute. “The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript,” the study authors state.