NEW YORK (GenomeWeb) - In a public workshop last week, the US Food and Drug Administration asked lab directors and pathologists performing next-generation sequencing cancer panels to weigh in on the preanalytical, analytical, and clinical validity characteristics the agency should consider in regulating such tests when used to guide treatment decisions.
"The trend these days is NGS," Reena Philip, director of the division of molecular genetics and pathology in FDA's device division, said at the workshop last week. "But that introduces challenges to the current companion diagnostics regulatory paradigm."
Companion diagnostics are tests that the FDA considers necessary for the safe and effective use of a drug. To date, the agency has only approved companion tests that gauge a single analyte or a set of prespecified markers and has indicated them for use alongside a single drug. But labs are increasingly developing and performing NGS panels, which are multiplex assays that can interrogate a large number of genes, to personalize treatment decisions for cancer patients.
The FDA hasn't yet approved an NGS test for companion diagnostic use, but will have to in coming years. Drugmakers have acknowledged the impracticality of the one-drug-one-test CDx paradigm and are exploring NGS panels for identifying best responders to their drugs. Some drugmakers have inked alliances to develop so-called universal companion diagnostics, which also would use NGS.
At the workshop, the FDA offered a glimpse into how it is thinking about regulating NGS-based oncology panels. The agency asked experts at the workshop to consider the requirements for approving or clearing oncopanels in kit format that labs can perform in a standardized fashion, based on specifications included in labeling.
The agency also proposed for workshop attendees to consider an "intended use" statement, in which the NGS device would be characterized as a tool for detecting sequence variations using a specific platform, gauging certain mutations, and assessing certain types of specimens. The test label would also list markers on the panel in two tables — one that lists the genetic variants intended for CDx use and associated with response to targeted treatments, and another that lists variants that can be used to guide treatments for patients who have run out of treatment options.
At the workshop, experts from cancer centers and academic labs performing NGS panel testing detailed to FDA the numerous preanalytical, analytical, and clinical validation considerations when developing and performing an NGS oncopanel. Speakers often disagreed with one another as to what quality metrics and requirements the FDA could prespecify for NGS panels, and when labs and pathologists should be left to define best practices and tweak procedures to the clinical scenario at hand.
The FDA held the workshop precisely because it is working to find the middle ground in this regard. For example, during the workshop, the FDA considered how it might evaluate a lab's ability to gauge representative sets of variants as a proxy for supporting the analytical validity of a broader oncopanel. Since NGS panels that sequence entire genes will pick up variants that they may not have intended to, "how do you address the analytical validation of variants that you weren't looking for in the first place?" posited Elizabeth Mansfield, deputy office director for personalized medicine at FDA's device division.
Based on the information gathered at the workshop and written comments submitted to the FDA by March 28, the agency said it will likely issue guidelines on the regulation of NGS oncology panels, but did not specify a time frame.
"We know [oncopanels] are important for cancer patients," Mansfield said, noting how there is often scant tissue to allow evaluation through multiple, single marker tests. "We think panels are a really great way to move forward in oncology where you can get all the information you need out of one test … FDA actually supports this idea."
Practice of medicine or best practices
In the preanalytical setting, before patient DNA is even sequenced, the FDA wanted to know from experts whether it could perform certain evaluations, use representative sample sets, and apply quality control metrics as a way to demonstrate the safety and efficacy of NGS oncopanels, instead of requiring test makers to check off all parameters across all sample types.
John Pfeifer from Washington University School of Medicine pointed out all the ways in which preanalytical procedures — tumor sampling, tissue processing, histopathological review, tumor enrichment, DNA extraction, and library preparation — can vary between labs and impact whether a test identifies the gene variation driving a patient's disease.
For example, he noted that depending on the number of PCR cycles performed, a lab could start with a sample with low tumor cellularity, and extract enough nucleic acid for testing, but this would produce a very low complexity library. Hybrid capture-based assays would be able to pick up on this, but this would be harder to measure using amplification-based assays. "Now we're no longer talking about four- or five-fold differences in assay sensitivity based on histopathological review” between labs, Pfeifer said. "We're talking about log orders of differences based on library complexity."
Dara Aisner from the University of Colorado also acknowledged the many sources of variability between labs in the preanalytical stage, but cautioned the agency against being too prescriptive. In the pathologist community, for example, there has been a lot of effort to standardize handling of breast cancer samples before testing them for HER2, ER, and PR status. Although her lab has had to "turn systems upside down" in this regard, there are still samples that fall outside the preanalytical parameters.
Labs are ... free to tweak. It is completely understandable that they need to modify the protocol, that they need to deal with each patient situation uniquely.
"So, no matter how much we want to stipulate these things, we really can't," Aisner said. "Overprescribing these variables will lead to lack of access to testing for substantial proportions of patients."
On the other hand, the absence of guidelines could also negatively impact patient care, some experts observed. "Even the practice of medicine has best standards," Pfeifer noted. He suggested the FDA establish minimum thresholds for certain preanalytical aspects of the testing process, especially when those procedures can impact whether or not a lab is able to identify a variant in a patient sample.
Things to think about
The FDA also invited experts to weigh in on the "appropriate level of analytical validity" that test manufacturers should demonstrate for NGS oncopanels that they want to sell as kits to labs. Although most NGS panels are currently not provided as kits, the agency believes this is where the market is heading. "As this technology grows, people are going to be looking for more convenient and readymade devices," said Donna Roscoe, chief of FDA's molecular genetics and pathology branch. "So, it's absolutely critical for us to have an objective playing field."
Based on the intended use of a test, manufacturers would have to demonstrate the oncopanel's analytical performance in terms of accuracy, precision, limit of detection, and specificity. FDA wants manufacturers to document testing protocols from getting a specimen to reporting a result, so when a lab purchases a test, they can expect a certain level of performance. "However, labs are free to do what they want. They are free to tweak," Roscoe noted. "It is completely understandable that they need to modify the protocol, that they need to deal with each patient situation uniquely."
Still, experts seemed to caution FDA against getting too specific with its requirements, while advising manufacturers to closely document all testing procedures and validation steps so labs could figure out the strengths and weaknesses of a test. "Knowing that there are already well-validated, clinically actionable alterations that fall into buckets, such as point mutations, indels, copy number [changes], and fusions, and if your test is claiming you can detect all those things, you have to at least start there, [while] recognizing that the long tail of mutations in any given patient tracks to infinity," Eliezer Van Allen from Dana Farber Cancer Institute said. "You'll never be able to find all of them," he noted, but test makers should demonstrate that the test can reliably identify the clinically significant markers.
On the bioinformatics side, Van Allen further pointed out that the field's capabilities in calling variants in NGS testing is better for certain types of alterations than others. When it comes to calling point mutations, studies show that bioinformaticians are pretty confident in coming up with similar results, regardless of the algorithm applied. In contrast, when it comes to detecting gene fusions, "we haven't quite hit that point yet, as a community," while the field's confidence in detecting other variations, such as copy number changes and indels, "is somewhere in between,” Van Allen said.
Additionally, the agency asked experts to weigh in on the types of requirements test makers would have to meet when adding to or subtracting variants from an already approved or cleared panel. Roscoe wondered if FDA should request additional validation in this scenario, or review granular characteristics of the changed assay, such as the quality metrics, alignment, and mapping.
Robert Klees from the New York State Department of Health reflected that without running a test, it would be hard to for the developer to determine whether any changes had diminished assay performance. “Any wet bench modification would have to be analyzed not to the extent of the original validation, but [by taking] samples from the original validation and mak[ing] sure there's not a significant change in coverage, sensitivity, reproducibility, and things like that,” Klees said.
Madhuri Hegde from Emory University suggested the agency allow in silico approaches, given the challenge of accessing sufficient samples for all the validation work.
Van Allen's lab has applied in silico approaches to model the false positive rate of tumor-only sequencing with a 300-gene panel. Using different germline sequence databases to filter the data, they reported a 14 percent false positive rate. Depending on the database used, Van Allen's team found that the risk of a false positive was higher in non-Caucasian patients. "These are things to think about analytically as we are releasing this technology into the world," Van Allen said.
Classification is a dynamic, not a static process.
A dynamic process
Given the range of genetic variants that could exist in a particular NGS oncopanel, the FDA is considering the evidence test developers will have to submit for markers intended to be used in a companion diagnostic setting and for non-CDx uses, as well as what evidence is needed to clear “follow-on” tests that have the same intended use as an already approved test.
Currently, labs have their own processes in place for determining the clinical significance of a variant detected by NGS oncopanels, involving literature reviews that include gene association studies, algorithms to determine the functional impact of a variant, and external variant classification databases. After performing such evaluations, a lab will rate the variant's actionability, indicating its use in prognosis and in therapy selection and monitoring.
“Classification is a dynamic, not a static process,” said Shashi Kulkarni from Washington University. “Two, three months down the line, there might be a big study from a cooperative group which might change that classification.”
Although oncology is undergoing a “paradigm shift” due to the availability of NGS tests, Kulkarni said that the field had “huge challenges” when it comes to variant classification, since databases aren't always accurate and cell lines can gain mutational artifacts that can distort accurate classification. Lauding NIH funding for ClinGen, a broad effort to share, curate, and identify genetic variants important to human health, Kulkarni said that he was part of a working group that is developing a similar process and database that doctors, labs, and researchers can use to make clinical grade classifications of somatic variants.
The American College of Medical Genetics and Genomics, along with the Association for Molecular Pathology and the College of American Pathologists, last year issued guidelines on variant classification and terminology in the context of hereditary cancer testing, but these guidelines are not applicable for classifying somatic variants. Kulkarni said he is part of an ACMG/AMP/CAP committee to develop variant classification guidelines for somatic variants, which the group plans to release in the coming months.
While these resources should help labs, the FDA wants, through regulation, to bring standardization to a market where some labs have lots of experience performing NGS tests while other labs do not. Although the experienced labs are free to make tweaks to the lab process, the FDA would like for the less experienced labs to run the test with an agency-approved NGS kit according to the manufacturer's specifications and produce expected results.
Several speakers cautioned that currently, some test makers are using data from in vitro models and animal studies as evidence that drugs are interacting with certain genomic targets, and noted that prospective, randomized-controlled trials are still the gold standard for establishing that a marker is predictive of treatment response. However, given the rarity of certain tumor mutations, prospective randomized-controlled trials may not be possible.
In such situations, panelists emphasized the need for sharing n-of-1 experiences. A number of peer reviewed journals, such as Cancer Genetics and Molecular Case Reports are collecting n-of-1 cases for publication, while the American Society of Clinical Oncology's health IT platform, CancerLinQ, also offers a medium for sharing such information.
Collecting and sharing n-of-1 experiences could also help inform the clinical validity of genomic variants that FDA has approved for one cancer setting but not in others. For example, the FDA has approved Zelboraf to target BRAF mutations in melanoma, but there is anecdotal evidence to suggest that some lung cancer patients with such mutations also respond to the drug.
Regulators can use registries to evaluate the clinical validity of markers on NGS tests, but these tests have to have some standardization before the data can be useful, said Dane Dickson, CEO of MED-C, a non-profit currently trying to build an NGS registry for non-small cell lung cancer.
By locking in assays to the point that we cannot apply science to improve them implicitly states that we think the test is more meaningful than the biology.
Don't mention those three letters: LDT
Although at the workshop FDA officials specifically instructed expert panelists and members of the audience to not bring up the agency's intent to regulate laboratory-developed tests (LDTs), the controversy nonetheless weighed over the discussion. The agency has said it plans to finalize guidelines on the topic this year.
NGS oncology panels are currently offered as LDTs, and so would be regulated under FDA's risk-based oversight plan for such tests. Moreover, in recent months, health technology firm NextGxDx has tracked rapid growth in the number of commercialized panel tests in the US, and estimates there are approximately 1,400 genomic test panels for profiling cancer and about 900 panels for hereditary cancer testing on the market.
And while the agency wants to somehow separate the discussion on regulating NGS oncopanels from the larger LDT oversight debate, laboratory professionals at the workshop could not demarcate their overarching concerns about FDA's intrusion into what they believe to be the practice of medicine.
University of Colorado's Aisner noted that the FDA seems to want to regulate NGS oncopanels through a "sample in, answer out" type of framework, but doing so would harm patients. Quality control metrics in the lab are meaningless without the "flexibility to act up on them," Aisner said. "Assays which are locked in are ones in which identified deficiencies cannot be corrected and can lead to patient harm. By locking in assays to the point that we cannot apply science to improve them implicitly states that we think the test is more meaningful than the biology.”
She cited the example of an FDA-approved EGFR test, which her lab at the University of Colorado performed in a non-FDA-approved manner in order "to see the data." If the assay had been run in an FDA-approved manner, her lab would have missed a critical EGFR mutation in a patient's tumor and the patient would have received standard chemotherapy.
"However, because we were able to view the data correlated with the percentage tumor, we were able to alert the clinician [to the fact] that we thought there was a high probability that there was actually an EGFR mutation in this case, which there was," Aisner said. "The patient is now on therapy and responding." According to Aisner, this is just one of many such examples.
Similarly, the Association for Molecular Pathology, a group staunchly opposed to FDA oversight of lab-developed tests, proposed the agency focus its limited resources on ensuring that NGS instrumentation, informatics, and reagents are performing accurately and reliably in gauging a representative set of markers across common specimen types. But according to AMP, FDA regulation into other aspects of performing NGS testing, would encroach on the practice of medicine.
"FDA's proposed intended use language [for NGS oncopanels] illustrates the agency's continued adherence to an obsolete companion diagnostic test approach that is inconsistent with the nature of massively parallel sequencing technologies and their current and future capabilities," said Roger Klein, chair of AMP's professional relations committee and a molecular pathologist at the Cleveland Clinic, during the workshop. "FDA must recognize that the clinical interpretation of accurately called bases is an integral part of professional medical practice and does not lie within the agency's purview."
Meanwhile, the Personalized Medicine Coalition advised the agency to create an oversight framework for NGS oncopanels that allows for flexibility. Even if an NGS test is locked down, the clinical application of the markers on the panel might change as the science advances. “Moving forward, we recommend that FDA continue developing discussion drafts and/or draft guidance documents describing current thinking on these issues and incorporating feedback from this workshop and other forums,” the PMC said in a statement.