NEW YORK (GenomeWeb) – Last week, researchers at Ambry Genetics published a study in the Journal of Molecular Diagnostics that found that Sanger confirmation is still needed for next-generation sequencing tests to ensure optimal specificity and sensitivity.
There has been significant debate about the use of Sanger confirmation in NGS testing to determine variant calls for years now. While Sanger sequencing is still considered by some to be the gold standard in genetics testing, NGS testing has become increasingly accurate, faster, and cheaper. Sanger confirmation adds to the cost of genetic testing and increases turnaround time by approximately two days, so many labs have not chosen to implement Sanger confirmation, though they may have implemented other types of confirmation, such as cross-validation with similar NGS assays.
"It is dangerous to have no Sanger confirmation at all," Aaron Elliott, Ambry CEO and co-author on the study, told GenomeWeb. It's possible to set conservative thresholds for calling variants from NGS data to limit the need for Sanger confirmation, however, it requires a large initial sample size in the tens of thousands, he added.
Laboratory policy guidance surrounding NGS testing confirmation is contradictory. The College of American Pathologists recommends that individual laboratories performing NGS assays determine whether confirmation testing is appropriate. While the American College of Medical Genetics and Genomics recommends secondary confirmation testing for all NGS-reported variants. But neither of these recommendations specifically call for confirmation via Sanger sequencing.
In the study, the Ambry researchers attempted to clarify why it's important to do NGS testing confirmation and provide better informed guidance on how to do it.
The research team took a series of 20,000 patient samples referred to Ambry Genetics for NGS-based multigene hereditary cancer testing using the company's BreastNext, ColoNext, PGLNext, RenalNext, PancNext, GYNplus, OvaNext, and CancerNext. They extracted DNA from either whole blood or saliva samples using the Qiagen QiaSymphony instrument. The team then designed a customized target-enrichment oligonucleotide library to capture 49 hereditary cancer-related genes. The researchers purified, quantified, and hybridized the samples before performing PCR amplification on a Bio-Rad T100 thermal cycler. Then the researchers sequenced the samples on an Illumina HiSeq2500 or NextSeq500 instrument.
They aligned sequence reads to a reference human genome using NovoAlign software version 3.02.07. The researchers generated variant calls using the Genome Analysis Toolkit version 3.2.2 and gave each variant call a Q score. The team then confirmed their findings for nonpolymorphic variants in each sample using Sanger sequencing.
Of the 7,845 nonpolymorphic variants identified, 98.7 percent of them were concordant between NGS and Sanger sequencing and 1.3 percent (99 tests) were identified as NGS false positives. These false positives happened in complex genomic regions such as A/T-rich regions and G/C-rich regions, Elliott said.
Some laboratories get around the issue of false positives by adjusting variant-calling thresholds to a higher specificity, which can eliminate false positives in NGS testing, Elliott said. To verify this, the researchers gradually adjusted the variant-calling thresholds for testing to reflect 100 percent specificity, while still performing Sanger confirmation.
"We simulated a zero false-positive threshold," Elliott said. By doing that you would miss about 2.2 percent of the variants, he said.
In the paper, the researchers reported that 176 Sanger confirmed variants were missed under a zero false-positive threshold. These false-negatives were largely concentrated in 13 genes — ATM, BARD1, BRCA1, CDH1, CHEK2, MSH2, MUTYH, NF1, PMS2, POLE, PTEN, RAD50, RAD51C, and RAD51D. Therefore, the researchers concluded that labs sacrifice sensitivity if they select for specificity in variant calling without any type of confirmation.
The Ambry researchers' biggest takeaway was that laboratories need to run thousands of tests before they can set reliable specificity and sensitivity thresholds for NGS testing that might eliminate the need to confirm every test with Sanger sequencing. "A lot of labs go off their validation data," Elliott said. "Validation data never has enough samples. If you'd done a 1,000 [sample validation study], which is a pretty big validation, you would be looking for about five false-positives [in the data set] if you sequenced the same 47 genes."
False-positives tend to occur in very specific regions, and genes in these regions are more difficult to sequence than others, he said, adding that laboratories need larger sample size data sets to determine what thresholds make sense for particular genes of interest.
While many labs would say they agree that NGS testing confirmation is important, they don't all agree that Sanger confirmation offers the best solution. "Sanger confirmation will detect false-positive variants identified by NGS," Benjamin Roa, senior vice president and lab director at Myriad Genetics, told GenomeWeb in an email. "However, Sanger confirmation will not detect variants that are missed by NGS as false negatives."
"The issue goes beyond Sanger confirmation per se, since it is imperative [for] each lab to ensure the highest quality results," Roa said. Myriad employs orthogonal sequencing methodologies, such as cross-validation with another NGS assay, but also employs additional quality control features, such as adding long-range PCR amplification followed by Sanger sequencing of specific genes to the lab's workflow, to guard against false positives and false negatives, he explained.
"Such decisions need to be based on empirical data from thousands of samples," Roa said. "Each lab should perform rigorous testing on large data sets to make decisions on how their tests should be run."
"It should be up to the lab to set their own policy on Sanger confirmation," Elliott said. However, it should be dependent upon the laboratory having run enough samples, he added.
That being said, some question whether running thousands of tests will provide the quality of data needed to make decisions about how best to design a workflow for an NGS assay. The Ambry Genetics study criticized previous work for not having larger samples sizes, but if you look at the data more closely there isn't a huge difference in the actual number of unique, high-quality variants analyzed between studies, Linnea Baudhuin, a clinical molecular geneticist at the Mayo Clinic, told GenomeWeb. Baudhuin is also an associate professor of laboratory medicine and pathology and an author on a paper published last year in the Journal of Molecular Diagnostics that also looked at the effectiveness of Sanger confirmation.
In that study the researchers only looked at 77 patient samples, but they were able to study approximately 2,000 genetic variants. The Ambry study looked at 20,000 patient samples and only studied about 8,000 genetic variants.
In addition, Baudhuin pointed out that while the study she worked on did question the need for Sanger confirmation in all cases, the researchers came to a conclusion that mirrored the finding of the Ambry study — that the key to determining the necessity of Sanger confirmation is based on the set quality thresholds, which were similar between the studies.
"[The Ambry researchers] included a lot of data from low-quality NGS reads," Baudhuin said. "They included a whole host of artifacts and called them false positives. In many labs, those artifacts would not even be seen because they would be filtered out by the quality thresholds, or the labs would perform upfront Sanger on those regions," she said in an email to GenomeWeb.
Many labs when faced with regions that are known to be difficult to sequence using NGS won't even bother to do so, she added. "They'll just do up-front Sanger." This method of developing a workflow can be more streamlined and efficient, Baudhuin added.
However, Elliott, Roa, and Baudhuin all agreed that the most important consideration when creating a workflow is to do enough development testing as is necessary to ensure accuracy.
"[There are] necessary quality control steps," Roa said. "Shortcuts to try and lower testing costs only end up hurting patients and wasting substantial healthcare dollars."
"It's important for laboratories to find the most efficient way to perform their analysis," Baudhuin said. "It's not necessarily about the number of samples run, but rather about the amount of quality data that you are generating. Proper test design and development should give sufficient data to understand where the problematic areas are, and proper test validation should ensure the highest quality test possible," she said.
Elliott, Roa, and Baudhuin also agreed that there needs to be some sort of quality confirmation in place, especially for those difficult-to-sequence regions, to ensure that patients and their physicians are getting the most accurate testing results.
Yet, they said it's also clear that each laboratory should be allowed to create their own internal policies. Ambry researchers, though, seem to be advocating for a policy that these internal policies must be shown to have been developed on a large quantity of data, which they believe helps ensure the highest accuracy in testing.
"We were getting a lot of samples that were coming to Ambry from other laboratories where mutations were either being missed or not confirming," Elliott said. While he didn't comment on individual laboratories, he did indicate that this trend was what pushed Ambry to analyze the data it has been collecting from its testing experiences and publish it.