NEW YORK (GenomeWeb) – Quest Diagnostics, while an adopter of clinical next-generation sequencing, has published research suggesting that laboratories should be careful when validating off-the-shelf kits for clinical use as they are likely not sufficient without substantial modifications.
In the study, published in PLOS One this week, Quest researchers described the validation of the firm's NGS-based BRCAvantage test for analyzing the BRCA1 and BRCA2 genes for variants associated with increased breast and ovarian cancer risk.
The firm launched the test in 2013 with a list price of $2,500, but had not disclosed the details of the assay.
In the study, the firm described testing both the Illumina MiSeq instrument and Thermo Fisher's Ion Torrent PGM; a bait-based capture protocol; and analysis pipelines provided by the sequencing manufacturers as well as its own internally developed pipeline.
The main take-home message, according to Charles Strom, VP of genetics and genomics at Quest Diagnostics, is that "neither of the machines' software/hardware combinations was sufficient for this particular application."
To develop the assay, Quest used 27 stem cell lines with known BRCA1 and BRCA2 variants as well as 67 consented control samples from Coriell with 352 benign variants.
In its first test, Quest noted that the MiSeq system with the supplied MiSeq Reporter software missed two pathological BRCA1 variants — a 10-bp deletion and a 40-bp deletion. As a result, the Quest team designed its own alignment and variant calling pipeline, Quest Sequencing Analysis Pipeline (QSAP). The PGM was able to detect those deletions, so Quest continued to use the Torrent Suite for analysis of the PGM data.
Strom explained that the MiSeq missed those variants because of its alignment process. When there is a deletion or insertion 10 bases or longer the read will not align, and the aligner throws it out.
Another key to the assay, said Strom, is the use of bait tile capture instead of PCR-based targeted amplification. Allele dropout is more likely to occur in PCR-based target amplification, resulting in false negatives. In addition, some regions amplify preferentially, increasing the likelihood of false positives.
Strom said that Quest uses baits from Agilent and while the company did not have to make any changes to reagents or the wetlab work, part of assay development was optimizing the baits for the specific test.
In the validation set, both the MiSeq/QSAP and PGM/Torrent Suite versions yielded 100 percent sensitivity, identifying all known 379 variants. However, the PGM had lower intra- and inter-assay precision, at 96.2 percent and 96.7 percent, respectively, compared to the MiSeq at 100 percent and 99.4 percent, respectively. The inconsistencies in the PGM were all false positives.
The Quest team next moved forward with clinical testing. Strom said that when the firm first launched BRCAvantage it ran samples on both the MiSeq and the PGM, but the PGM continued to have issues with "accuracy of reads and ease of use" compared to the MiSeq.
From the first 521 clinical samples analyzed on both platforms, there were initially 35 discrepancies between the two. Errors from the PGM and/or Torrent Suite pipeline accounted for 34 of those discrepancies. The MiSeq error was a false negative result for a benign polymorphism due to a combination of strand bias and low sequence coverage, which the Quest team was able to fix by adjusting the QC parameters. After that adjustment, the MiSeq made no further errors and achieved 100 percent sensitivity and specificity, including detection of a 64-bp deletion and 10-bp insertion, both of which were pathogenic BRCA1 variants, and both of which the PGM missed.
After those first 521 clinical cases, Quest decided to stop using the PGM for BRCAvantage. Instead, for the next 1,000 cases, it ran the MiSeq assay in duplicate. "With more than 5,000 variants detected, there were no discrepant results between duplicate analyses," the authors wrote.
Strom said the company now just runs a single MiSeq assay. The entire assay development and validation process took around one year, he said.
Quest has previously discussed challenges in developing NGS-based clinical grade tests, noting that test developers need to take into consideration much more than the sequencing box itself, such as sample prep and bioinformatics.
At a conference last year, for instance, the firm discussed sample prep challenges in the development of its HIV tropism test, which can reduce sample volume by so much that the relevant low-frequency variant may not even be present in the sample being analyzed.
Nonetheless, the company has been a technology-agnostic adopter of NGS, inking multi-year agreements last year with both Illumina and Thermo Fisher to use their respective sequencing technologies for clinical testing.
And while the firm opted to not use the PGM for its BRCAvantage test, Strom said the platform may still have utility for applications in oncology tests for evaluating somatic mutations.
Strom said that Quest has a number of clinical NGS tests in development in the areas of oncology and neurology, although he declined to provide details on the types of tests and when they would be available.
In addition, he said that the company is in the midst of converting all of its Sanger-based assays into NGS tests. "That's a major project," he said, "but we're convinced that the sensitivity and specificity of an NGS assay with appropriate quality metrics and software is equal to or greater than a Sanger assay."
He said the company hopes to complete that project by the end of 2016.
And, rather than a one-to-one conversion, Quest plans to combine its Sanger tests into larger multi-gene NGS panels and will then "filter bioinformatically what the doctor has ordered," Strom said.
Converting to NGS will likely save both time and money, Strom said. The bioinformatics, while complex, are more automated with NGS, so once it is done, it is much easier for the reviewer to interpret, he said.
"I think this technology is ready for prime time, but proceed with caution," he said.