NEW YORK (GenomeWeb) – Are public variant databases a collective good or are they a source of confusion and uncertainty that can harm patients?
Two new papers published last week added fresh data to a contentious debate over the use of public variant repositories, which are increasingly supported by regulators, the National Institutes of Health, guidelines bodies, peer-reviewed publications, and payors. And while a number of labs and pharma companies are openly sharing variant information in such databases, many are not, citing competitive reasons, patient privacy concerns, and the limitations of the repositories.
In a paper published by Invitae and collaborators last week in the Journal of Clinical Oncology – Precision Oncology, the authors demonstrated that among seven submitters of BRCA1 and BRCA2 gene variant classifications to an NIH-funded public repository, called ClinVar, their determinations agreed for 98.5 percent of around 2,000 variants. When they worked together to resolve the remaining discrepancies in rare variants, agreement was achieved for 99 percent of variants.
Pathogenic mutations in BRCA1/2 genes are known to increase risk for hereditary breast, ovarian, and other cancers, but the two genes are highly variable, and not all variations are potentially harmful. Working collaboratively to classify variants and resolve discrepancies improves knowledge and is ultimately good for patients, according to lead and senior authors Stephen Lincoln and Robert Nussbaum, who are both employed at Invitae, a commercial lab that has contributed more than 41,000 interpreted variants to ClinVar as of mid-March and has been a strong proponent of data sharing. "ClinVar is a tool for careful and responsible laboratories to use to reach the best interpretations they can," said Nussbaum, Invitae's chief scientific officer.
In another paper, published online in The Oncologist last week, a team led by William Gradishar from Northwestern University compared 4,250 unique BRCA1/2 variant classifications in ClinVar against those in the proprietary repository of Myriad Genetics, the leading provider of hereditary cancer testing, and found that 14.5 percent were classified differently between Myriad and the database; 12.3 percent had some agreement between at least one entry in ClinVar and Myriad's determination; while 73.2 percent were in agreement.
Since nearly 27 percent of BRCA1/2 variant classifications don't match up between Myriad and ClinVar, according to this analysis, the authors, most of whom are employed at Myriad, questioned whether there is much benefit to using the public database, particularly since most commercial labs submitting to ClinVar don't provide the underlying evidence for their determinations. "As many ClinVar users may not have the specific expertise to independently evaluate the literature evidence," the authors wrote, the lack of supporting evidence for classifications in ClinVar "will likely introduce uncertainty into medical management decisions."
The debate over the utility of public variant databases has been going on for some time, but the battle lines were solidified once the US Supreme Court decided that gene sequences as they occur in the body are not patent eligible. That 2013 landmark decision broke Myriad's nearly two-decades long monopoly on testing the BRCA1 and BRCA2 genes, and other labs immediately started to sell tests for them.
But competitors were at a disadvantage since Myriad had much more experience testing for these genes and had amassed a proprietary database containing results from some 2 million patients. The strategy of some newer providers of BRCA testing has been to submit to ClinVar and band together to work through instances where labs have different classifications for the same variant, or in cases where a variant has been seen in one or two patients and its link to disease is unclear.
This is happening not just in the case of BRCA1/2 variants, but across other disease-linked genes as well. Yet, since the discovery of the BRCA genes and their role in breast and ovarian cancer risk in the 1990s, there's been bitter competition and controversy, and that history has made data sharing difficult. Myriad — which still holds the largest market share, though competitors like Invitae would say not for long — has not contributed to ClinVar, because it says it wants to protect patients' privacy, and it has pointed to the discordance in the repository as evidence that it is unsuitable for clinical decision-making.
On the other hand, Invitae and other labs regularly contributing to ClinVar see it as a way to highlight variant classification discordances, so they can prioritize which ones need attention and updating. For example, seven of the 30 variants with discrepancies in the Lincoln et al. analysis were due to outdated information in ClinVar, there were two data submissions errors, and for four cases, there was key evidence that impacted the classification but the information hadn't been updated in ClinVar. The rest of the discordance, the authors believe, are "legitimate differences in judgment of laboratory experts" after they've weighed the evidence.
"ClinVar is a database that needs to be used in an intelligent and sophisticated way," said Nussbaum. "There have been a lot of brickbats thrown at ClinVar by people who don't really understand how it should be used and what its real value is." The public repository shouldn't be treated like a Bible or a Golden Rule, he said, and the design of the Lincoln et al. study provides an example of how users might critically use ClinVar.
For example, in the study, not all BRCA variant submissions in ClinVar were analyzed. Researchers only considered data from commercial labs that had submitted at least 200 classifications to ClinVar, more than 50 percent of which were made less than five years ago. As such, this study tried to look at discrepancies between labs that are possibly producing higher quality classifications than other submitters of BRCA data to ClinVar, since they are performing BRCA testing for patient care and have submitted fairly recent interpretations to a repository, minimizing errors due to outdated information.
In this way, when Lincoln et al. investigated variants deposited by seven submitters — six labs and the Sharing Clinical Reports Project that gathers variants from Myriad's test reports — they were able to resolve discordances for 99 percent of variants. Because all discordant variants were rare, they estimated that one in 500 patients would receive a result that would significantly change their clinical management.
The researchers estimated that the more than 2,000 variants investigated in the study represent the test results received by approximately 22,000 patients. After working to resolve discrepancies, Nussbaum and colleagues were able to achieve a per-patient concordance of 99.8 percent.
The high frequency of rare and private variation is challenging for variant classification and significantly impacts patient populations.
The impact of discordance
Myriad is of the view that in the current environment, where labs far less experienced than itself are performing BRCA testing, the chance that patients are getting erroneous results is higher than the estimate from Lincoln et al. If patients get a pathogenic BRCA test result in error, they could have needless mastectomies and oophorectomies, while a wrong benign result could mean the patient isn't screened properly for cancer.
The limited number of discordances Lincoln et al. found between seven ClinVar contributors in rare variants could impact the clinical management of a significant number of patients, according to Susan Manley, VP of medical services at Myriad Genetic Laboratories and senior author on the Gradishar et al. paper. "Approximately 60 percent of our database for BRCA1/2 contains variants that have been seen one [or] two times, and Myriad continues to see new BRCA1/2 variants each week," Manley said. "The high frequency of rare and private variation is challenging for variant classification and significantly impacts patient populations."
In estimating the discordance in ClinVar, Gradishar et al. cast a wider net than the Invitae team. They considered all unique BRCA variant submissions to ClinVar, including from large and small commercial labs, consortia, and academic labs, without any effort to distinguish the quality or expertise of the data submitter, and reported a far higher discrepancy rate. The authors found that in most cases (14 percent), discrepancies happened when a variant was classified as a VUS in ClinVar and as a definitive variant by Myriad. Less frequent were the 0.3 percent of variants where Myriad had a VUS and ClinVar had a definitive classification.
Even rarer were the 0.1 percent of variants, or five in total, where Myriad and ClinVar had opposite classifications, pathogenic versus benign. However, these five variants can impact the care of many more patients that harbor them. As an example, one of the five variants (BRCA1 c.301+1G>A), classified as pathogenic/likely pathogenic in ClinVar and deemed likely benign by Myriad, has been detected by the firm in 22 patients.
Gradishar et al. laid out the reasoning for why Myriad classified this and several other discrepant variants a certain way, and the implication is that the lab is correct and ClinVar is wrong. However, labs like Invitae, sharing their variant data and interpretation evidence in ClinVar, would take issue with a blanket assumption that Myriad's classification is correct in all instances, since it's not publicly sharing variant data.
Patients can also be negatively impacted when there is discordance about whether a BRCA variant is a VUS or has a more definitive association to cancer, Manley noted. The American College of Medical Genetics and Genomics guidelines state that VUS "should not be used in clinical decision making," though this is not always how it plays out in the real world.
In a recently published study in JCO involving 600 women diagnosed with breast cancer who received genetic testing, for example, Kurian et al. reported that about half of 22 patients with an average risk of having a BRCA mutation got a bilateral mastectomy based on VUS results in the BRCA genes. "All genetics organizations agree that VUS are not actionable, like benign or likely benign [variants], but not all clinicians adhere as Kurian et al. concluded," Manley said. At the same time, the ACMG has five variant classification categories — benign, likely benign, VUS, likely pathogenic, and pathogenic — and she noted that "calling these two answers the same is not consistent with the views of clinicians, researchers, patients, or regulators."
Lincoln et al. evaluated discordances by grouping likely pathogenic and pathogenic variants as a "positive" result, and likely benign, benign, and VUS as a "not positive" result, per ACMG guidelines that VUS shouldn't be used to make clinical decisions. Gradishar et al., meanwhile, categorized classifications into three groups — likely pathogenic and pathogenic classifications were "pathogenic"; likely benign and benign classifications were "benign"; and VUS were "uncertain".
They carved out VUS as a separate category, according to Manley, because even though the ACMG advises clinicians to not act on VUS, "it represents an unanswered question for the healthcare provider and patient." Although the majority of VUS turn out to be benign, she noted there is some evidence that clinicians can overinterpret uncertain variants and make recommendations that are more appropriate for patients with pathogenic mutations.
If the goal is to measure the patient impact of variant classification discordances, then Lincoln et al. makes a methodological error by merging likely benign, benign, and VUS variants into one category, Manley argued. Nussbaum challenged that assertion and noted that many guidelines bodies advise treating VUS as not actionable like benign variants, and other researchers have used the same categories as the basis for variant comparisons.
"This is not to say that VUS are not a problem because they could be misinterpreted as being actionable when they are not," he said, also citing the Kurian et al. paper. But he noted that the odds of a breast cancer patient with a BRCA1/2 VUS getting a bilateral mastectomy are small in the study. Nussbaum also pointed out that the study focused on women who had breast cancer, and there are many reasons for choosing to have a mastectomy, beyond BRCA testing results, which is a reality that Kurian et al. also acknowledged.
"Our findings raise concern that average-risk patients in particular may not have understood their VUS results; alternatively, their desire for bilateral mastectomy may not have been affected by test results," they wrote in the JCO paper. However, they also recognized that there is a growing demand for genetic testing among patients who are not at high risk for a BRCA mutation, and there needs to be effective communication strategies for these patients.
[It's] a matter of professional ethics. It's not just a matter of data.
Difficult history
Myriad flagged a number of "serious methodological errors" in Lincoln et al.'s analysis that Manley said rendered the group's "conclusions [to be] of dubious clinical value." In turn, Nussbaum accused Myriad of continuing to spread misinformation and characterized Gradishar et al.'s analysis as "a thinly veiled attempt to confuse rather than enlighten."
Any discussion of public databases and specifically BRCA1/2 variant classification is inevitably weighed down by the history of the Supreme Court's gene patenting decision against Myriad. The wounds of that landmark case are still fresh and the battle lines are still palpable.
Nussbaum was sympathetic to the researchers and patients that sued Myriad, he was a principal investigator for ClinVar, he spearheaded the Sharing Clinical Reports Project to encourage doctors to share deidentified BRCA1/2 results from Myriad with the public database, and now he works at Invitae, one of Myriad's main competitors in the hereditary cancer space. Data sharing "is a matter of professional ethics," Nussbaum said. "It's not just a matter of data."
In the JCO paper, Nussbaum and colleagues called Myriad out for not submitting to ClinVar and restricting ordering doctors in its terms of service from sharing deidentified variant data. Myriad's policy runs counter to the practice of leading labs in the US that have received legal and institutional review board approval to share deidentified variant data in public databases, he added.
Myriad defended and countered. "Invitae inaccurately describes Myriad’s policy and motives regarding data sharing," Manley said, highlighting the company's commitment to patient privacy as one of the earliest supporters of the Genetic Information Nondiscrimination Act, a law to protect people against genetic discrimination. "We believe that only patients, not companies, have the right to share peoples’ genetic information with public databases," she said. Myriad encourages participation in patient-driven registries, she added, such as the Prospective Registry of Multiplex Testing, which is collecting data on cancer-linked gene variants other than BRCA1/2 from labs performing mutli-gene panel tests.
Nussbaum didn't buy it. "I suspect that Myriad’s policies on sharing data are aimed at maintaining high test prices and a (failing) monopoly," he said in an e-mail. "They are contrary to the recommendations of the leading professional organizations. Their claim of protecting patient privacy is a shameless attempt to cloak their practices in false altruism."
But those involved in ClinVar are also sensitive to criticism. In the JCO paper, Nussbaum and colleagues reanalyzed past studies that had tried to characterize the discordance in the public database to show that these analyses exaggerate discrepancies in ClinVar between commercial labs. Specifically, in reanalyzing a study by Balmana et al. using more recent data in ClinVar, Lincoln et al. highlighted that all clinical labs agreed on the classification of 9 of the 19 discordant variants in non-BRCA1/2 genes.
Balmana et al. and experts from Invitae have already exchanged civil, but strongly worded, editorials debating their respective methodologies for estimating discordance in ClinVar. "We were surprised by the depth of feeling created by our paper," said Susan Domchek, executive director for the Basser Center for BRCA at Penn Medicine and senior author of the Balmana et al. paper.
She said the intent of the paper was to simply point out to practicing oncologists in the community, who are not interpreting variants regularly, that there are some discrepancies in ClinVar, but not to criticize any labs or even the database. "It wasn't supposed to be an endorsement or a condemnation of anyone," she said.
In fact, Domchek and her colleagues use ClinVar and are supportive of the effort. "If we're going to use this information to maximize benefits for patients, we need to minimize discrepancies," she said. "In our mind, that means more intensive data sharing, not less."
[A]t some point, they're going to have to give up the ghost.
Using ClinVar
Ultimately, the methodological decisions made by Lincoln et al. and Gradishar et al. reflect their divergent motives. Nussbaum's group wants to demonstrate the value of data sharing among labs that are trying to use ClinVar in a responsible way, while Gradishar et al. want to capture the overall discordance when it comes to BRCA variants, to show that patients might be getting the wrong results from less robust sources in ClinVar. The first group believes that if used correctly, ClinVar could bring much needed transparency to the variant classification process, and collectively improve knowledge of variants, which is good for labs, doctors, and patients. The second group is of the view that in its current state, ClinVar isn't of much value to doctors or patients.
"The high degree of discordance observed here is likely reflective of the fact that public databases cannot currently accommodate the consistent standard of variant classification required for clinical use," Gradishar et al. wrote. "In addition, the time and expertise needed for busy clinicians to research discordant classifications call into question the practicality of checking all test results against a database."
Genetic testing, once the province of academic labs with specialized knowledge about specific genes, is big business now, which means more people are getting tested, and some ordering physicians may not know that some sources of data in ClinVar may not be as robust as others — though the database tries to reflect the quality of classifications with a four-star system. "Myriad may have a point that there are a lot of people that aren't using ClinVar or thinking about these test results, but then, one might ask, should they be?" posited Domchek, who references ClinVar for VUS, and for challenging missense and spice-site variants.
Andrea Forman, senior genetic counselor at Fox Chase Cancer Center's Department of Clinical Genetics, believes that many OB-Gyn's may not even know about ClinVar. "I don't know that they care about ClinVar, and I don't know that they have time to dig into it if they did," she said.
As head of the medical genetics division at the University of California San Francisco's cancer center, Nussbaum also deals with variant classification from the other side. He and his colleagues visit ClinVar regularly to see what other labs have to say about a patient's variant, and in doing so, they take into consideration how old those determinations are, and whether the labs have backed up their classifications with evidence to support it.
Transparency about the underlying evidence should be part of best practices for commercial labs submitting data to ClinVar, he said, though this is not yet widely done. Of the six labs included in the JCO study, for example, only GeneDx and Invitae share the underlying evidence for their variant classifications in ClinVar. If the evidence isn't there in ClinVar, "we think nothing of calling up another laboratory," he said, and asking why they've classified a variant a certain way.
Forman herself doesn't look up every test result that crosses her desk, but references ClinVar when a patient's specific circumstances raise a red flag. For example, she'll consult the database when testing identifies a pathogenic or likely pathogenic variant for a patient but this doesn't align with their family history, or when patients do have a strong family history of cancer but the lab reports a VUS that could fit the phenotype.
Because not all labs explain in test reports why they interpreted a variant a certain way, Forman finds it useful that firms like GeneDx and Invitae do provide the underlying evidence for a particular classification in ClinVar. She'll also call up labs to ask for their rationale. A discordant classification in ClinVar will not typically be reason enough for Forman to reinterpret a result that a lab has provided, although she has done it for one patient, who had breast cancer at 31 years old and where testing through Myriad identified a VUS in the TP53 gene.
Mutations in the TP53 gene are associated with Li Fraumeni syndrome, an inherited disorder that significantly increases the risk of cancers at a young age. While this patient had no family history of the syndrome, her early breast cancer made Forman suspicious. In consulting ClinVar, she noted that three other "legitimate" labs were reporting the same VUS as likely pathogenic, but she didn't stop there.
She called Myriad, and the lab stood by its VUS classification, reasoning that the likely pathogenic classifications in ClinVar were based on a study involving an enriched population and limited information was available. Then she checked with a National Society for Genetic Counselors discussion forum and found counselors who had seen this variant in other women with early-onset breast cancer and in one child with a type of cancer, an adrenal cortical tumor, that occurs commonly in families with Li Fraumeni syndrome.
Her investigation into the VUS revealed information that Forman felt she had to convey to the patient. "I explained that there was some discrepancy between labs and that we needed more data to really get a clear answer," she said. "But the safest option was to proceed with screening as if she tested positive for Li Fraumeni."
In another example, Forman's team had a patient who received a result that was reported as a VUS by labs submitting to ClinVar. They called Myriad and found out the lab considered that variant benign based on seeing it in 18 patients. The labs reporting to ClinVar had each seen the variant only once. "Even though this variant didn't look suspicious, they weren't ready to commit [to a benign classification] because they'd only seen it the one time," she said. "And it was like, Myriad, if you had just shared this, our numbers would have jumped up."
This might be a matter of maintaining market share for Myriad, Forman suspected, "but at some point, they're going to have to give up the ghost, … and we're all going to have to work together, hold hands, and sing Kumbaya."