As advanced diagnostic technologies probe deeper into the human genome, revealing previously uncharacterized markers in patients and families, so-called variants of unknown significance (VUS) are becoming an unavoidable challenge for clinical labs using genomic data to diagnose patients.
"VUS are the number one game you find when you do whole-genome sequencing," Howard Jacob, director of the Human and Molecular Genetics Center at the Medical College of Wisconsin, said at a personalized medicine conference in Mountain View, Calif., last month. A few years ago, Jacob and his team at MCW famously identified the genetic mutation that was causing six-year old Nicholas Volker's severe inflammatory bowel disease, for which he had undergone more than 100 surgeries. Researchers at the medical college sequenced his exome and found a mutation in the XIAP gene associated with an inherited immune disorder.
The culprit mutation was initially deemed to be "likely pathogenic," which means that at the time there was scant evidence that this gene variant was associated with gut disease. But after months of analysis, MCW researchers were able to pinpoint that the mutation was responsible for Nicholas' condition. As a result, Nicholas' doctors were able to perform a cord-blood transplant. He is doing well now, and turned nine in October.
But not all the cases Jacob sees have a happy ending. His lab makes a diagnosis in only around 25 percent of the cases that use WGS, largely because of the uncertainty surrounding the clinical significance of many of the variants that are identified. "There are a lot of these variants of unknown significance, and what do you do with them?" Jacob posited. "If we're thinking about how to change the practice of medicine, we have to think about this.”
Jacob and his team at MCW and Children's Hospital of Wisconsin recently received $1 million from the W.M. Keck Foundation to "go after VUS" and develop a system using gene editing and functionality testing to rapidly determine whether uncertain variants are linked to a patient's disease. Jacob hopes that at the end of this three-year project his group will have developed a system that other labs can use to end diagnostic odysseys like Nicholas' more quickly.
In thinking about VUS as a collective problem for the life sciences community, the US government has taken on the Herculean task of building a public, annotated database of variants across the human genome using standardized classification methods, called the Clinical Genome Resource (ClinGen). A number of labs and organizations have already submitted variant information to the database, which currently has $25 million in NIH funding. The database at the moment lists nearly 80,000 submissions on more than 18,000 genes.
Well-annotated public databases can also help speed new genomic medicine products to market. For example, reviewers at the US Food and Drug Administration used information from Johns Hopkins University's publicly available cystic fibrosis database to rapidly clear for marketing the first next-generation sequencing-based CF test on Illumina's MiSeqDx platform. Elizabeth Mansfield, director of personalized medicine at FDA's Center for Devices and Radiological Health, noted in a recent interview with PGx Reporter that because of JHU's CFTR2 database, the agency was able to "clear a test with many more mutations than we had ever been comfortable doing before." She added that this demonstrated how "the collection of evidence in a well-curated way can really help on the regulatory front."
Despite the growing enthusiasm for collaborative data sharing and variant annotation to advance the field of genomic medicine, open-access repositories are often criticized for being poorly maintained and error prone. The American College of Medical Genetics and Genomics, in its 2013 guidelines for next-generation sequencing lab standards, advises against depending only on disease-specific databases to assess variants, noting that few of these resources are curated to a standard necessary for clinical use. This has kept some labs from contributing to such efforts.
Myriad Genetics is one diagnostics firm that sees more harm than good coming out of public variant databases at the moment, and has not submitted genetic variant information from its tests to such resources for a decade. "The public databases as they exist today were never intended for clinical use on patients. They were intended for research," said Myriad spokesperson Ron Rogers. Chief Medical Officer Richard Wenstrup stressed the need for standards before labs make use of open-access resources to diagnose and treat patients based on WGS and exome sequencing data.
"There's been a lot of excitement around next-generation sequencing, and the development of highly powerful sequencing platforms has allowed for lots of people to do sequencing. But I think what hasn't gotten enough attention … is variant classification," Wenstrup said. "In the public databases now there is a substantial amount of heterogeneity with respect to the methods used and the quality control used to classify variants ... It's widely recognized that there needs to be standards developed for variant classification or reclassification so that public databases have a measure of quality control."
Consider the liabilities
At a session for investors and analysts at the San Antonio Breast Cancer Symposium in December, Myriad officials spoke out strongly against the use of "unregulated public databases" to reclassify VUS. In a video, Karla Bowles, director of Myriad Genetics Laboratories, warned anyone submitting variant data to or using information from such repositories to consider how managers of such databases inform patients of changes to classifications, which methods they use to classify variants, how they ensure only qualified experts are submitting and editing variant information, and how disagreements about classifications are resolved.
Given these uncertainties, Bowles said that "submitting data to public variant databases and using this data without verification has the potential to be associated with significant liability." As a lab director who signs off on each test result that Myriad issues, Bowles emphasized that she would not approve a report based solely upon information in public variant databases. "There is far too much at stake for our patients," she said.
Although whole-genome or exome sequencing increasingly find high rates of uncertain variants in seldom-studied genes, BRCA1 and BRCA2 genes are well researched and annotated. As such, the identification of a VUS result through BRCA testing is relatively uncommon, but because Myriad tests thousands of patients' samples in its labs for variants in these genes, the company comes across a handful of uncertain results per week.
For its flagship BRACAnalysis test, which gauges women's risk of hereditary breast and ovarian cancer, Myriad boasts a VUS rate of around 2 percent, the lowest in the BRCA testing market. Myriad achieved this with the help of its IP position that enabled it to be the sole US provider of BRCA testing since launching BRACAnalysis in 1996, and a proprietary database of 1 million test results. Myriad stopped submitting variant data to the open-access Breast Cancer Information Core (BIC) in late 2004, and has maintained that it did so mainly because of the poor data quality in such repositories.
For example, according to Rogers, when Myriad analyzed the Leiden Open Variant Database (LOVD), it found that 14 percent of the mutations flagged as deleterious in LOVD were benign when compared against its own methods. Overall, Myriad says that 66 percent of variants in LOVD are misclassified when compared to its own data. Myriad is planning to publish this analysis and would not provide more detailed data about the differences between its variant calls and the ones in LOVD.
Although Myriad has lately been the most outspoken about the risks associated with submitting data to open-access resources, it is certainly not the only company wary of public variant databases. Ambry Genetics, a company competing with Myriad in the BRCA testing space, announced last July that it had submitted its first set of variant information to the BIC. “Ambry Genetics is the only commercial US lab to contribute data since 2006,” the company said in a statement at the time.
"Everyone doing clinical interpretation of genomic data, whole-exome sequencing, or whole-genome sequencing or even testing of known individual or few-gene clusters, has to filter out lots and lots of errors and grotesquely incomplete data in the Human Genome Mutation Database and other public databases," Robert Cook-Deegan, research professor of genome ethics, law, and policy at Duke University, told PGx Reporter over email. "Myriad's proprietary database would not be very valuable if the public/nonprofit/payor network were sharing data effectively. This is an area we really want to explore, because it's a super-hard problem to solve."
In an effort to encourage more labs to submit to public databases, expert groups are starting to clean them up, develop standardized ways of tracking submissions, and manage data discrepancies. The International Society for Gastrointestinal Hereditary Tumors (InSiGHT) published their work to establish a system of classifying Lynch Syndrome-associated genes in Nature Genetics last year.
They accepted thousands of unpublished data submissions and applied a microattribution method for tracking entries. After review of all the submitted data by an expert committee, InSiGHT altered classifications for 66 percent of some 12,000 entries on Lynch Syndrome variants, and issued clinical recommendations for more than 1,300 variants. InSiGHT also detailed their variant classification algorithm and laid out a five-tier classification system in the paper. Around 32 percent of the variants, most of which occurred in one family, InSiGHT deemed VUS.
InSiGHT's colon cancer gene variant database is hosted by LOVD, the same resource that Myriad compared against its own database and found around 66 percent to be misclassified. Bryony Thompson of the Queensland Institute for Medical Research, lead author on the InSiGHT paper, acknowledged in an e-mail that before she and her colleagues “cleaned up” InSiGHT's colon cancer variant database, many of the markers were classified based on minimal or unavailable evidence.
“Standardized classification criteria [for variants] and their application by experts in the field is very important,” Thompson said, adding that it also makes scientists very nervous when they have to take a company at their word about variant classification. “With InSiGHT we wanted to make sure that all the information we use to classify variants is out there in the public domain, so that anybody using the classifications understands how they were derived.”
According to Heidi Rehm, director of the Laboratory of Molecular Medicine at Harvard's Partners Healthcare Center for Personalized Genetic Medicine, the majority of public variant databases do not contain high quality information because they often don't adopt a consistent evidence-based standard for variant interpretation and aren't kept up-to-date. This is because most are developed by researchers with inconsistent funding and because many researchers submit data solely for the purpose of publishing a paper and never go back to update the information. Moreover, the poor accuracy of variant repositories reflects problems with variant interpretation in the published literature itself, which submitters have entered into public databases, she said.
"What we need to do is get more data from clinical labs that are routinely performing systematic evidence-based reviews of variant data and classifying variants according to a standard method," she said. "As opposed to what's out there right now, [which is that variants are] often derived from papers that just list variants found in patients, without systematic interpretations, and then databases just take those and stick them in a database labeled as pathogenic."
ClinVar – the centralized repository of genotype-phenotype information for ClinGen -- encountered this very problem. During a week in October, a single lab retracted 80 percent of its submitted variant data on BRCA1/2, bringing the total number of the lab’s submissions down from more than 5,000 to around 1,000. According to Rehm, what happened was that the clinical lab had gathered variant information by curating the literature, before launching a commercial test. The lab then voluntarily shared these interpretations from the literature into ClinVar but a lot of those classifications were wrong.
When these mistakes came to light, the unnamed lab retracted the initial submission from ClinVar and then re-entered the variant information without the pathogenicity claims drawn from the literature. While Myriad and other industry observers quickly jumped on this retraction as glaring evidence of the dangers of open-access repositories, Rehm views the incident as an example of the benefits of greater transparency within clinical labs' interpretation work.
"This is an extremely strong argument for the improvement of patient care by sharing these variant interpretations, exposing the differences, allowing them to be adjudicated," she said. Rehm and her team of researchers recently received an $8 million grant from the NIH for collecting variant data from labs and clinics, and developing standards for depositing this information into ClinVar.
As a lab director, Rehm doesn't believe that submitting to public databases like ClinGen would increase liability for labs any more than what they already face. "Every lab harbors liabilities for poor work. We're all at risk at some level if we do a bad job of interpreting variants, if we don't research them, and therefore, we mis-classify variants," she said. "It’s possible that in making your interpretations transparent to the world, one could argue that people are more likely to find out if you're doing a bad job, but it doesn't change the fact that you're accountable for your work either way. In the end, labs must prioritize patient safety and improving our understanding of genomic variants, and there is no question that this will improve with data sharing and transparency."
Furthermore, she stressed that labs shouldn't use public databases like ClinGen as their sole source for variant classification, and there are disclaimers clearly stating this. Indeed, when visiting the website for the ClinVar repository, one is immediately greeted with the following: “The information on this website is not intended for direct diagnostic use or medical decision-making without review by a genetics professional. Individuals should not change their health behavior solely on the basis of information contained on this website. NIH does not independently verify the submitted information.”
To share, or not to share
The tension between whether clinical labs should invest in improving public resources or build a proprietary database is playing out in the BRCA testing space.
Myriad markets its BRCA tests as the most accurate, claiming it has reduced its VUS rate from 13 percent in 2002 to 2 percent in 2013. In December, the company published in Clinical Genetics its first paper describing its variant classification methods. According to the paper's authors, led by Myriad's Julie Eggington and Bowles, when assessing a variant picked up by a Myriad test, the company uses a number of proprietary and standard industry techniques to discern its association to patients' health.
Myriad makes use of recommendations from the American College of Medical Genetics, as well as internal guidelines put forth by its New Mutation Committee, which draws on the expertise of lab directors, clinical variant specialists, genetic counselors, statisticians, and structural biologists. The company classifies 60 percent of the variants that come across its lab using two proprietary classification procedures, Mutation Co-occurrence (MCo) and Pheno, and 40 percent using standard methods available to the industry. (Myriad's variant classification methodologies are described in greater detail in PGx Reporter's sister publication BioInform.)
The paper comes as the company is facing increasing competition following the US Supreme Court's decision last summer in Association for Molecular Pathology et al. vs. Myriad. In that case, the court deemed several of Myriad's patent claims around BRCA1/2 isolated gene sequences patent ineligible, but also found eligible a number of its claims on cDNA, or synthetic DNA. This two-part decision resulted in much confusion in the diagnostics space as to what types of genetics-related discoveries and technologies can and can't be patented.
Myriad has sued a number of companies, who are performing BRCA testing, such as InVitae, Ambry, GeneDx, Quest Diagnostics, and LabCorp. All of these labs have submitted or plan to submit variant information to ClinGen, and not just on BRCA variants.
Yet, the situation in the BRCA testing market demonstrates that amid IP uncertainty, labs will compete on their ability to quickly and accurately interpret increasingly larger gene panels. In a crowded space, while some labs might see public data repositories as a way to improve their tests, others might decide that not sharing their variant analysis will give them a competitive edge. Many industry observers have opined that in the wake of the Supreme Court decision, diagnostics firms will increasingly rely on trade secrets, like Myriad's proprietary variant database.
Myriad makes no apologies for deciding to keep its variant data behind the closed doors of its lab. "We did spend $500 million developing our database and getting it as accurate as it is today," Myriad's Rogers said. "And any other company is free to make that same investment that Myriad made. No one is stopping them from doing that."
Although Myriad says it decided to stop submitting to BIC due to its poor quality, the firm certainly doesn't deny that its proprietary database is a key asset. "Our large database that correctly identifies harmful mutations versus benign mutations is a major competitive advantage for Myriad," CEO Peter Meldrum told analysts during a recent earnings call. So far, while Myriad is facing some competition in the BRCA testing space – which impacted 15 percent of its revenues last quarter – the company still holds the most market share.
Myriad's leaders believe they can recover any short term losses over time. "Mutation classification is a numbers game," Meldrum said. "And since we test hundreds of thousands of patients every year, we believe we will dramatically expand our mutation classification advantage and further widen the gap between Myriad and its competition."
According to the recently published paper, the company does consider information in public databases and published literature, but in a recent interview, Myriad's Wenstrup deemphasized the importance of open-access repositories to the firm's variant classification process. "Myriad has contributed to public databases in the past, particularly the BIC, but Myriad's classification program is not really significantly dependent upon public databases," he told PGx Reporter.
He noted that while NIH's investment in ClinGen was encouraging, it's too early to say if Myriad would ever contribute data to it. "We do go through publicly available literature [and] that's part of the algorithm," Wenstrup said. "But we primarily use our own database … that we've created with the testing of now 1 million patients."
Myriad has estimated that its competitors' VUS rates for BRCA testing are sometimes as high as 30 percent using public databases. But diagnostics firms performing BRCA tests, such as InVitae, Ambry Genetics, and Quest Diagnostics, claim VUS rates in the low single digits. This can't be confirmed until these labs publish their methodology and variant data.
In this regard, there is no way to check Myriad's claim that its VUS rate is 2 percent, even with the published methodology information, as long as its variant data and algorithms remain proprietary. "So, they've reduced their VUS 'call rate' from around 13 percent in 2002 to around 2 percent in 2012. … And do they know how often they are right and wrong?" Duke's Cook-Deegan questioned after reviewing Myriad's methods paper. "How many women developed cancer who had variants classified VUS, leaning benign, or benign? How many who had 'deleterious' mutations lived through age of risk without cancer? Call rate is one thing. Biological validity is one more step and there's not a word about this" in the paper.
Cook-Deegan has been a vocal critic of Myriad's decision to stop contributing to the BIC and of its proprietary database. In a 2012 paper published in the European Journal of Human Genetics, he, along with James Evans of the University of North Carolina, Chapel Hill, and lawyers John Conley and Daniel Vorhaus of Robinson, Bradshaw & Hinson, asserted that by keeping its variant classification data and algorithms a trade secret, Myriad is denying researchers, physicians, and patients access to "basic scientific and medical information" that would advance the life sciences field and help patients receive the most accurate diagnoses.
Wenstrup acknowledged that Myriad doesn't follow patients for outcomes after reporting a result to track if, after receiving a report for a benign mutation, they actually develop cancer, for example. This type of long-term follow-up would require informed consent, he said. "There are large registries that are publicly funded that do [long-term outcomes studies]. … We don't use those registries," Wenstrup added. "It's not that we may not do that in the future, but we haven't done it at this time."
However, the company does have processes in place to update a patient if her variant classification changes during her lifetime. Additionally, "for families with a [VUS] we offer testing for up to 10 individuals, depending on the family configuration, to help us classify variants," Wenstrup added.
"That's not happening with public databases today," Rogers said. "There's no one there to make sure if the database gets updated with new information, if they get curated, and once that information is updated, if it gets communicated back to the patient's healthcare provider."
From Myriad's perspective, by publishing on its methods, it has done something its competitors have not. "No other labs in industry – and it's not very common for academic laboratories – have published the methods of their variant classifications," Wenstrup said. "So, we actually think that we're leading the way in publishing the methods for variant classification."
Recognizing the increasing challenge VUS data poses to clinical diagnoses, labs and research groups are starting to publish the methods they use to classify variants. Rehm's lab at Harvard recently did so in Clinical Genetics around the same time Myriad published its paper. However, her lab has also submitted data on approximately 7,000 variants into ClinVar.
"Each variant [submission in to ClinVar] includes our classification, as well as any citations or descriptive arguments for why we classified the variant as such and how many cases we’ve seen it in," Rehm said, noting that this information allows others to verify the accuracy of the variant calls the lab is making.
Changing landscape
The Multiple Myeloma Research Foundation launched a public database last year for which it plans to collect detailed clinical and genomic data from 1,000 patients over a five-year period from initial diagnosis of their disease. The information in the database, collected as part of the CoMMpass trial, will be accessible to industry partners and the research community in the hopes of accelerating development of multiple myeloma treatments.
"I wouldn't be surprised if this is the model for all of cancer," Eric Lander, founding director of the Broad Institute, said at an event last fall in New York announcing the launch of the open-access research portal. "I wouldn't be surprised if this is the model for all of medicine."
Genospace, the software company that built the MMRF database, develops systems for storing genomic and health information in formats that enable experts to collaboratively manage and analyze the data. John Quackenbush, CEO of GenoSpace and professor of biostatistics and computational biology at Harvard University, believes that open-access data repositories will not only be critical in life sciences research but also reshape the clinical diagnostics market.
“I think that a lot of companies are going to try to amass private databases, but before some variant becomes an established standard of care, the variants and their association with clinical outcome or response to therapy or some other meaningful endpoint are going to have to be shared and validated,” he said.
While he believes it's a company's right to keep its variant classifications as a trade secret, he thinks it a short-sighted strategy. "Even if all the private genetic testing companies in the country or around the world were holding on to their own data, which they might do, I think there are going to be enough public resources that this data is going to be available," Quackenbush said. "I firmly believe that patients will be reporting their mutational status and other information."
Free the Data is a consortium of organizations managed by Genetic Alliance and the University of California, San Francisco. The entities in this consortium have agreed to submit de-identified information from BRCA test reports to ClinVar and are encouraging labs, cancer clinics, and patients to do the same. Proponents of data sharing and public repositories, such as InVitae, GeneDx, Geisinger, Rehm, and Cook-Deegan, are all on the steering committee of Free the Data, and are working on building a platform for annotating VUS in a crowd-sourced fashion.
InVitae CEO Randy Scott described this annotation platform as a kind of Wikipedia for classifying variants. Wikipedia is imperfect, he acknowledged, but "the idea that one company would be the arbiter of what's pathogenic or not, and not put that out there for peer review" is of far less value than an environment where experts can openly discuss and decide if information on a particular variant is correct, he said.
"The reality of being at the cutting edge of medicine today is that we don't understand what the vast majority of human variation is or exactly means," he said. "And we will only get there by sharing the data on variants … just like surgeons share their surgical techniques and physicians share their experience in grand rounds."
As the molecular diagnostics market moves from very niche service models, where labs specialize in a handful of genetic tests, to doing whole-genome and exome sequencing, industry players will have to pool their resources, many are predicting. Gone will be the days where a lab like Myriad can be the sole proprietor of testing two genes. Rehm foresees that in a few years, "every lab is going to have to be able to interpret all diseases and all variants.
"And you just can't maintain that deep level of knowledge across every gene and every variant," she said. "In my opinion, you will be left in the dust if your lab thinks it is going to survive by maintaining its knowledge in one area and not be able to support other areas."
Cook-Deegan expressed tepid appreciation for Myriad's Clinical Genetics paper, while criticizing the firm for doing little to advance the life sciences industry's collective knowledge of genomic variants. "Myriad seems to do really good work; it's too bad they are doing it behind closed doors with a business model that perpetuates their isolation," he said. "If this becomes the business model for other genes and clinical conditions, we face a system of very expensive tests whose interpretation is dramatically suboptimal because other firms will not share with Myriad any more than [Myriad] is sharing with others now."
Myriad's Wenstrup refuted the charge that the company hasn't contributed to advancing the public knowledge about BRCA variants and hereditary cancers. "There are at least 50 active [projects] with research institutions that Myriad is undergoing across a wide variety of fields. We are actively involved in contributing to the scientific literature," he said.
Rogers highlighted that the latest publication shows Myriad's contribution to the field in reducing VUS rates across different ethnic groups. For African-American patients, for example, Myriad reports that it has reduced the BRCA VUS rate from nearly 40 percent in 2002, to 3 percent currently. "There will be a lot more from Myriad on the methods we use [for variant classification] in the future," he added. "This is just the first in a series of publications around our methods."