A workgroup led by the US Centers for Disease Control has published guidelines for laboratories developing clinical tests based on next-generation sequencing.
The Next-generation Sequencing: Standardization of Clinical Testing (Nex-StoCT) workgroup, which included researchers from the CDC, clinical laboratories that have already developed sequencing-based tests, industry representatives, and representatives from the College of American Pathologists and the Centers for Medicaid and Medicare Services, published their guidelines in Nature Biotechnology earlier this month.
The guidelines address issues of validation, quality control, proficiency testing, and reference materials that are specific to next-generation sequencing technology.
According to Ira Lubin, who leads the genetics team at the CDC, separate guidelines for clinical next-gen sequencing are necessary because such tests have the potential to be much more complex than other types of diagnostic testing, even Sanger sequencing, and current guidelines and regulations do not adequately address those complexities.
"The performance characteristics that are traditionally defined through [the Clinical Laboratory Improvement Amendments] and proficiency standards, such as accuracy, precision, specificity, and sensitivity, don't really translate to how things work in the next-generation sequencing world," he told Clinical Sequencing News.
Lubin said that the guidelines aim to help guide clinical laboratories that are just starting to implement next-generation sequencing.
"One of the greatest values of the guidelines is for folks who are less experienced and have less of a history with next-generation sequencing, to provide them some guidance for how to implement it, for issues to consider to ensure that they can do this with good quality," he said.
The CDC guidelines come a few months after a CAP workgroup published an NGS-specific checklist for labs seeking CAP accreditation (CSN 8/1/2012).
While the CAP checklist includes 18 requirements that clinical labs must meet to receive CAP accreditation, the CDC guidelines differ in that they offer guidance to labs about processes and methods that will help ensure that their tests meet such requirements and provide high-quality, robust, accurate results.
Additionally, the CDC guidelines can be used by regulators like CAP or the Centers for Medicaid and Medicare Services to "inform their decisions about how to regulate these tests," Lisa Kalman, coordinator of the Genetic Testing Quality Control Material Program at the CDC, told CSN.
One key issue that the guidelines address is validation, which includes validation for the platform, the test itself, the informatics pipeline, and alternative methods like Sanger sequencing for confirmation.
For platform validation, test developers must demonstrate that the sequencing platform can provide reliable sequence analysis across the genomic regions targeted by the test. Meanwhile, test validation refers to the ability of the test to identify disease-associated variants in those targeted regions.
The workgroup also recommends that an orthogonal technology, such as Sanger sequencing, be used to confirm results from the next-gen test, and that technology must also be validated.
"That was a really highly discussed issue," said Lubin. "The consensus among our group was that in the majority of cases today, because the technology is still new, experience is not that significant and there are errors that exist. … So it's a good idea to confirm your findings."
Additionally, the group recommends that parameters of the test — such as the reagents and informatics used — cannot be changed once a laboratory has started the validation process.
Once performance specifications are established and the test has been optimized, developers should "lock it down," Lubin said. "That is the test you apply to clinical samples, and when you're doing that, you don't touch those settings."
If there is an upgrade in the future, the test will have to be revalidated and performance specifications will have to be re-established.
The group also redefined metrics such as accuracy, precision, sensitivity, specificity, reportable range, and reference range, which are used by CLIA to evaluate test performance, so that the definitions were more suited to next-gen sequencing tests.
"The CLIA regulations mostly refer to very simple analytes like glucose or sodium, where you're just looking at one molecule, whereas the DNA molecule could be three billion analytes" Kalman said. "The CLIA regulations don't really address that complicated of an analyte, so we had to translate what it was trying to say for a simple analyte to a more complex analyte."
For instance, in its definition of precision, the workgroup addresses variability that could occur both from sequencing the same sample under the same conditions, so-called within-run precision; as well as variability that could occur from multiple operators sequencing the same sample using more than one instrument, or between-run precision.
The group also tackled the issue of validating testing across large numbers of samples. Due to the high cost and data analysis required by next-gen sequencing, determining the test's precision across large numbers of samples could prove cost prohibitive for some labs. Instead, the workgroup recommended an alternative approach where "additional metrics such as the depth and uniformity of sequencing coverage would be incorporated to limit the number of samples required to establish precision," the authors wrote.
For quality control, the group addressed NGS-specific metrics useful for monitoring the assay, including depth and uniformity of coverage, quality scores for base calling and alignment, allelic read percentage, strand bias, GC bias, and decline in signal intensity.
Including these specific metrics was important, said Lubin, because "some of the technologies have specific error profiles that make it challenging to detect certain types of variants, or the ability to accurately detect a variant in certain regions of the genome will vary," he said.
Moving forward, Lubin said, the guidelines will likely have to be amended due to new technology coming online or to address sequencing that is performed in the absence of asking a specific clinical question.
For instance, while the current guidelines take into account coverage as an important factor in establishing accuracy and precision, future sequencing technology may move to single molecule and longer reads where depth of coverage will be less relevant, said Lubin.
"The whole way people do things will probably change, such as the mapping and alignment," added Kalman.
The guidelines also focused on the detection of variants associated with heritable human disorders, rather than the use of sequencing in the absence of a clinical question, such as in a healthy person to predict future disease risk.
Lubin said that the guidelines were not meant to put restrictions on NGS-based tests that are already being run out of CAP-accredited and CLIA-certified laboratories.
In fact, he said, researchers from many of those groups, including Emory University, Washington University, and ARUP Laboratories, were part of the workgroup that developed the guidelines. These groups already have a lot of experience with next-gen sequencing, he said, while the guidelines are aimed at those with less experience with next-gen sequencing.
Up until now, early adopters that already had both strong clinical and research frameworks and that had early experience with next-gen sequencing were the ones applying the technology in the clinic, he said, and those groups have largely had to develop their own approaches for test validation and defining performance metrics.
But now, next-gen sequencing is increasingly being considered and adopted in clinical settings, and it is becoming ever more important to have consistent guidelines, Kalman said. "By providing very detailed guidelines, it will inform a lot of different efforts and everything will be harmonized," she said.