This article has been updated with information from Genomics Ltd.
NEW YORK – Although polygenic risk scores for coronary heart disease (CHD) have attracted keen interest and are moving towards becoming clinically available, new research suggests that a great deal of uncertainty remains with respect to how well they assess risk at the individual level.
A study published over the weekend in the Journal of the American Medical Association assessed the performance of, and agreement between 46 CHD risk scores and found that while all of them performed equivalently at the population level, they varied widely at the individual level.
Led by investigators at the University of Pennsylvania, the research reveals a need for better statistical methods to quantify uncertainty and strategies to communicate this uncertainty to physicians and their patients.
To explore the individual-level agreement of CHD PRSs that perform equivalently at the population level, the scientists identified 46 CHD PRS found within the Polygenic Score Catalog, an open access online repository where researchers can deposit and publicly share PRSs. These risk scores showed "practically equivalent" population-level performance when compared across 265,380 individuals from the All of Us Research Program, the Penn Medicine BioBank (PMBB), and the University of California, Los Angeles ATLAS Precision Health Biobank (ATLAS).
The population-level consistency of these PRSs largely disappeared at the individual level, however.
Approximately 52 percent of individuals analyzed had at least one risk estimate in the top five percent, a commonly used cutoff for classifying someone as high risk. However, all of these same individuals also had at least one score that rated them below this cutoff and 39 percent of these individuals –– 20 percent of all participants –– had at least one score that placed them in the bottom five percent. When expanding the definition of high risk to the top twenty percent or quintile, another commonly-used cutoff, 80 percent of all participants had at least one risk estimate placing them in the top quintile for CHD risk, as well as one score placing them in the bottom quintile.
Differences between individual estimates narrowed somewhat when comparing scores that were constructed using the same underlying genome-wide association study.
"To us, this suggests that one driver of score variability may be differences in the underlying genetic datasets used to develop the scores," Scott Damrauer, vice chair of clinical research at UPenn and the study's corresponding author, said via email. "However, this doesn't appear to fully explain the variability in individual-level risk predictions."
For example, many of the PRSs used in this analysis differed substantially in terms of when they were developed, the statistical methods used to derive them, the number of variants included, and the diversity of training datasets.
An editorial accompanying the publication noted that the study is timely in light of the rapid growth of data available for risk modeling, as well as advances in statistical methods and artificial intelligence.
"The explosion of available risk assessment tools also raises the need for growing emphasis on how risk thresholds are defined and risk communication strategies to guide clinicians and patients on how to contextualize the uncertainty and probabilistic nature that is inherent to all risk prediction models," the editors wrote.
Seamus Harrison, VP and head of medical at UK precision health company Genomics, called the study "a neat demonstration" of the instability seen at the individual level for risk prediction models developed and tested on large populations.
"In a sense, this isn't surprising and has been demonstrated in other, non-genetic, clinical risk prediction models used in cardiovascular disease," he said. "What's becoming clear, is that even simple biomarkers used in populations have individual-level contexts that can impact the precision of the information provided."
Harrison also commended the study for flagging this issue with polygenic scores in order to increase awareness in the community.
Genomics, which recently changed its registration from a publicly limited company (plc) to a limited company (Ltd), is conducting its own research into CHD PRSs in an effort to make them commercially available in clinical settings. Earlier this year, the UK-based company published a study demonstrating the feasibility of clinically implementing a PRS for cardiovascular disease. That study also highlighted several cases in which a PRS triggered changes in care plans, suggesting clinical utility.
Harrison said that as PRS move beyond research to clinical settings, PRS developers need to demonstrate robust standards with regards to data, polygenic score calculation, and reasonable levels of uncertainty around individual risk estimates, as well as how to communicate this uncertainty.
As the authors of the present study state in their manuscript, one of their motivations for conducting this analysis is that at present, no standard approach to calculating an individual's genetic risk for CHD – or any other condition – exists, making it impossible to reliably characterize their accuracy.
Denver-based Allelica is another company working to advance clinical PRSs. The firm has developed its own PRS for coronary artery disease, which it says can save the healthcare system money by identifying higher-risk individuals earlier.
Allelica recently presented a new method for modeling human genetic diversity without predefined labels, aimed at making PRS more accurate and predictive across diverse populations. The method, called 8 Billion, is designed to generate a unique risk distribution for any individual tested, using genetically similar individuals from Allelica's proprietary database. The company is currently integrating 8 Billion into its own PRS tests and software.
Damrauer and his colleagues hope that their study motivates stakeholders to improve the framework for evaluating CHD PRSs, agree on guidelines for their development and use, and design better investigations into CHD PRS in the future. Improvements in these areas, they wrote, are needed to help clinicians and their patients interpret and understand these divergent estimates of polygenic risk.
"PRSs must be recognized for what they are," the UPenn team wrote. "Estimates of genetic risk that come with an inherent uncertainty."