Skip to main content
Premium Trial:

Request an Annual Quote

Test Developers Discuss Challenges of Validating NGS Panels; Pros and Cons of FDA-cleared Products


SAN FRANCISCO (GenomeWeb) – Developers of next-generation sequencing-based tests said this week that despite improvements to sequencing technology, there are still significant challenges in developing clinical NGS-based gene panels.

These challenges include, but go beyond, the sequencing technology itself, and include differences in laboratories' bioinformatics pipelines, DNA extraction and sample prep protocols, as well as differences in the sample itself.

In addition, test developers reported mixed feelings on the potential of increased US Food and Drug Administration oversight of laboratory developed tests, and discussed the pros and cons of using cleared products.

The developers made their comments at Cambridge Healthtech Institute's Molecular Medicine Tri-Conference, held here this week.

Despite these challenges, "NGS can and should be used as part of clinical molecular diagnostic testing menus," Josh Deignan, associate director of the molecular diagnostic lab at the University of California, Los Angeles, said in a presentation.

He said UCLA has access to multiple sequencing instruments and "has benefitted from them all." The lab runs exome tests on the Illumina HiSeq, somatic cancer panels on Thermo Fisher's Ion Torrent PGM, and is currently developing a panel for myeloid disorders on the Illumina MiSeq.

Test development vs. validation

One key thing to keep in mind, Deignan said, is to make sure that the test development process is separate from the validation process. The purpose of test development is to "generate a protocol for the entirety of the assay," which includes determining what genes should be included, sequence coverage, analysis thresholds, and optimizing all those components. Validation, meantime, should follow the College of American Pathologists' recommendations.

In addition, while sequencing technology is not perfect, some of the main challenges of test development and validation have nothing to do with the sequencing itself.

For instance, when developing a somatic cancer panel, working with DNA derived from formalin-fixed paraffin-embedded samples can pose a challenge. The DNA is fragmented, so amplicons that are part of a panel should be no more than 150 to 200 bp in size, Deignan said. In addition, tumor heterogeneity may cause discrepant test results when trying to validate between laboratories. "Two FFPE slides can yield different results and that may say nothing about the test, but just that the tumor is heterogeneous," he said.

An issue, he added, is that "even if two labs are doing the same combination of genes, no lab is doing the same test" since laboratories use different technologies and different bioinformatics pipelines. This issue is especially seen when comparing clinical exome sequencing pipelines. He said he sent one sample to five labs, all with validated protocols. The five labs agreed on 18,000 variants, but each also had their own set of up to several hundred variants that none of the other labs called.

Running cleared tests

FDA-cleared in vitro diagnostic tests would address some of the challenges of developing and validating LDTs and require much less investment by individual labs. Currently, however, the only FDA-cleared NGS-based tests are Illumina's two cystic fibrosis assays that run on its MiSeqDx system.

According to Jamie Platt, a vice president of genomic solutions at Molecular Pathology Laboratory Network who has been performing next-gen sequencing within a CLIA setting for the last 10 years, there are advantages and disadvantages to running FDA-cleared tests. MPLN acquired the MiSeqDx last November and now has experience running both Illumina's 139-variant cystic fibrosis assay and its cystic fibrosis clinical sequencing assay, which sequences the entire CFTR gene.

One main advantage of running cleared assays, she said, is that laboratories incur no development and validation costs. Instead, they must verify that the assay performs according to the FDA label.

One major downside is that cleared products are much more costly to run. For instance, she said, the 139-variant assay kit has a list price of $7,200, which includes two runs on the MiSeqDx. Up to 45 patient samples can be multiplexed per run, which translates to $80 per patient, but if "you're in a low-volume lab or the turnaround time needs to be fast," waiting to fill a run may not be feasible and costs could be as high as $3,600 per patient.

By contrast, a cystic fibrosis LDT can be run for less than $50 per patient, although she acknowledged that does not include development and validation costs.

Illumina's full-gene assay is even more expensive. The list price on that is $60,000, which includes six runs and the ability to multiplex between six and eight patients per run, for a per-patient cost of $1,666. However, if the lab does not have sufficient patient volume, costs could soar to $10,000 per patient, compared to an LDT that could sequence the entire CFTR gene for as little as $200.

Another potential problem is that when using cleared products, the lab is totally reliant on the vendor. "You're relieved of some responsibility, but dependent on the vendor for trusting their validation and that they will be able to supply you with reagents," Platt said. By contrast, clinical labs will often validate a second set of reagents to use as backup in case there are supply issues, she said.

Increased FDA oversight

Some clinical labs are concerned that increased FDA oversight of LDTs will stifle their ability to develop tests and incorporate both new technology and biological advances in a rapidly changing field.

"Technology is often outdated by the time it goes through the IVD process," Platt said. "I'm not historically a fan of IVDs for that very reason."

In addition, Seth Crosby, director of partnerships and alliances at Washington University's Genome Technology Access Center, raised the issue that increased regulations could hamper laboratories' abilities to develop tests quickly and to update tests with more current information.

For instance, he said, in his experience, cardiologists at Wash U approached test developers at the school's Genomics and Pathology Services lab about co-developing a gene panel. "We were able to do this in a matter of a few months," Crosby said, which enabled the physicians to deliver the test to waiting patients. In addition, GPS also runs a cancer panel, which he said is now on its third iteration because "medical knowledge concerning these genes is evolving so rapidly."

He said he is concerned that FDA guidelines may "impact our ability to create panels and to re-version them" in a timely fashion.

Jennifer Morrissette, the clinical director of the Center for Personalized Medicine at the University of Pennsylvania School of Medicine, voiced similar concerns about labs' ability to tweak tests or to combine elements of different technologies and protocols to solve problems.

For instance, she said, her lab runs a lung cancer panel on samples that often contain as few as 50 to 100 cells. "We had to develop a panel directly for that," she said. Existing technologies could not work with such small amounts of input material, and the resulting panel does not use any off-the-shelf kits. Rather, it combines different elements of different types of technologies for preparing sequencing libraries. She said that one of her concerns about regulation of LDTs is that tests like their lung cancer panel would not be allowed.

Other clinical labs have also been expressing concerns about FDA's intention to regulate LDTs. FDA will hold a workshop on the topic on Feb. 20, titled "Optimizing FDA's Regulatory Oversight of Next Generation Sequencing Diagnostic Tests," and has published a discussion paper on the topic to its website.