By Monica Heger
The US Food and Drug Administration last week hosted a one-day workshop to discuss next-generation sequencing in clinical diagnostic applications, with a particular focus on the best way to assess the analytical validity of the platforms.
While FDA made no decisions and set no timeline for when — or whether — it intends to issue guidance on the matter, stakeholders agreed that broad technical criteria such as a sequencer's error rate or the amount of coverage would not be sufficient measures for assessing the value of a given test. Rather, they argued, the focus should be on the outcome.
"Ultimately, you're trying to [answer], 'How confident am I in this variant call?'" said Brad Ozenberger, program director for the National Institutes of Health's Cancer Genome Atlas.
Likewise, David Dimmock, a clinical geneticist at the Medical College of Wisconsin, said, "The key question that most clinicians really care about is, 'Did you get the right answer?'"
The FDA faces a number of challenges as it explores its options for regulating the use of next-gen sequencing in the clinical setting. First of all, there are many different platforms that can be used in a number of different applications — from whole-genome sequencing to exome sequencing and targeted gene sequencing.
Additionally, the bioinformatics analysis and sample-prep steps all play a role in the accuracy of the outcome. Furthermore, the fact that the same platform can be used for a number of different tests and applications does not fit with the way the FDA currently regulates diagnostic tests, said Elizabeth Mansfield, director of personalized medicine in the FDA's office of in vitro diagnostics.
"Typically someone will come forward with a device that has a specific intended use in a specific population. Meaning, this device is going to detect this in this population and then we have performance metrics around it," Mansfield, who served as moderator of the meeting's panel discussion, explained. "With something that can sequence an entire genome, I think that idea has to be considerably modified because the people who make the platform aren't actually going to know what everybody is going to be using it for, and everybody will be using it in slightly different ways."
The goal, she said, "is to find the appropriate questions to ask and that can be answered with a reasonable amount of clarity so [when] future platforms … are brought to the FDA for clinical diagnostics use, we can ask the right questions."
While the FDA is just beginning to dip its toe into these uncharted waters, there was some consensus at the workshop regarding criteria the agency should not consider when regulating sequencing-based tests. Panel members included representatives from both the clinical and technology sides of academia, the FDA, the College of American Pathologists, the Association for Molecular Pathology, the US Centers for Disease Control, AdvaMed, the National Institute for Standards and Technology, and the NIH.
All of these participants agreed that requiring specific metrics of accuracy or depth of coverage from sequencing platforms would essentially be meaningless because the platforms themselves vary so much. In addition, the amount of coverage or accuracy requirements would change depending on the type of disease being diagnosed and the sequencing protocol — whether whole-genome, whole-exome, or targeted gene sequencing.
Rather than a set of requirements applied evenly across the board for all tests, the panelists suggested a broad framework that would allow for flexibility in test design.
Systemic vs. Random Error
Because different sequencers all produce different types of errors, it will be difficult for regulators to specify a required raw read error rate, or a minimum read length, or even a certain level of coverage, the panel members said.
[ pagebreak ]
For instance, pyrosequencing, such as Roche's 454 GS FLX, has errors in homopolymer regions. So requiring a raw read error rate of less than 0.1 percent, for example, will be meaningless if the target variant is in a homopolymer region. Increasing sequencing coverage won't help either. "You can't sequence your way out of that problem," said Dimmock. While 454 tends to have homopolymer errors, other sequencers are known for errors in areas of high GC content.
On the other hand, the Pacific Biosciences system has a much higher raw read error rate, but the errors are all random, so can be corrected for by increasing sequencing coverage because the machine is less likely to generate the same error more than once.
Certain regions of the genome are also more difficult to sequence than others, so tests that sequence those regions may require more coverage and may never have the predictive power of other sequencing tests.
Madhuri Hegde, a clinical lab director at Emory University who has launched several sequencing-based diagnostic tests through the university's CLIA lab, suggested looking at each individual test, rather than creating a broad requirement for accuracy.
She suggested that the FDA consider the requirements for CLIA certification and CAP accreditation, which require documentation about a test's specificity, sensitivity, reproducibility, and robustness. Limiting the discussion to accuracy and depth limits it to sensitivity and specificity, "and those things can't really be separated when looking at clinical diagnostic application," she said.
Cancer vs. Constitutional Disease
The FDA will also need to consider specific diseases when weighing requirements for accuracy. For example, there is a big difference between sequencing tumor samples and sequencing samples from patients with other types of diseases, said John Pfeifer of the Washington University School of Medicine, who was representing the College of American Pathologists at the workshop.
When considering error rate, sequencing for mutations in the germline is very different than sequencing for mutations in a tumor, added Dimmock. The required error rate for sequencing germline DNA would not need to be as stringent, because "you are assuming that 50 percent of the alleles are going to be one type and 50 percent are going to be the other type. But when you're sequencing a tumor, you can't make that assumption," Dimmock said.
[ pagebreak ]
For instance, with a 1 percent error rate, the "ability to detect a variant that's present in the tumor at about 5 percent is just not there, because of the intrinsic error rate of the technology," he said. "The tolerance for error rate is going to be dependent on application."
"Accuracy could take different forms, depending on how the platform is used. For example, gene panels and targeted sequencing might have one accuracy measure, and exome sequencing and whole-genome sequencing might have others," Mansfield said.
Rather than requiring specific metrics for the platforms themselves, some panel members suggested that the agency focus on variant calls. For instance, Tina Hambuch, a scientific liaison from Illumina who was representing AdvaMed at the meeting, said that in the company's clinical lab, medical professionals know what types of variants they can detect with high confidence.
"In our clinical lab we define our test to say we can detect these kinds of variants, but not these ... We're good at indels of up to this size, but not that ... And, I think that's what's important — knowing enough to measure how good you are, so you can say, this is what we do well and the rest we can't really try to do," she said.
Emory University's Hegde agreed, noted that "what's being missed is important to document, not just what you can detect."
During the workshop's public comment section, Steve Lincoln, vice president of scientific applications at Complete Genomics, suggested that one approach might be to assign confidence scores for each call, or lack of a call.
Illumina's Hambuch agreed. "If you assign a confidence score to every base, based on raw read quality, how well it will align, how many times you've sampled that position ... you can draw a threshold and say, 'We won't report anything with a quality score less than this." She said that Illumina has been doing that with known positions, testing how well the final score represents that known position, which has allowed the company to establish cutoffs and has led to a way of "trusting what you measure."
Last week's meeting was just the first step down the long road of establishing a regulatory framework for sequencing-based diagnostics. As evidenced by the wide range of issues that panel members and the public raised, establishing a framework will be tricky.
However, Eric Kolodziej, senior vice president and head of global regulatory affairs at Roche Diagnostics, noted that creating a regulatory framework should not be impossible, citing as evidence the fact that the European Union has already established a loose framework by which sequencing-based tests can receive CE marking and be reimbursed by payors.
"We need a solution sooner rather than later," he said. Tests that sequence panels of genes are already becoming as common in academic labs as laboratory developed tests, and "sequencing the whole genome is on the horizon."
Have topics you'd like to see covered by Clinical Sequencing News? Contact the editor at mheger [at] genomeweb [.] com.