Skip to main content
Premium Trial:

Request an Annual Quote

Multi-Site Study Tests Power of Commercial Reference Materials in Comparing Liquid Biopsy Assays


NEW YORK (GenomeWeb) – Using a set of liquid biopsy reference materials from commercial firm SeraCare, investigators led by a team at the National Institute of Standards and Technology have published a multi-center study providing new evidence for the power of such tools in quality assurance and cross-platform comparison.

In the study, which appeared last week in the Journal of Molecular Diagnostics, the group applied SeraCare's 40-variant product to next-generation sequencing assays employed in seven participating labs with the goal of assessing the suitability of the commercial kit to benchmark multiple steps including DNA extraction, quantitation, preparation, instrument analysis, and bioinformatic analysis within the various assays.

From their inter-lab results, the group concluded that the reference materials do offer an "appropriate tool" to determine the analytical parameters of different tests. Authors also highlighted some of the insights into test variability gleaned from their comparison, including differences in DNA extraction yield seen with different methods and some variation in mutation detection at low variant allele frequencies. Overall, though, they said that they saw largely equivalent results among NGS tests used, at least where direct comparison was possible.

The speed at which liquid biopsy tests have emerged and entered the clinical market had, for a time, outpaced the creation of reference and quality control materials to help the field assure itself of tests' accuracy and comparability. But efforts have accelerated in the last few years, both among academicians, and commercially, to fill this gap.

SeraCare has been providing contrived samples for liquid biopsy validation and quality testing for a few years now. The company has said its references are produced using a unique technology that creates a DNA fragment-size distribution that more closely mimics native cfDNA than other shearing methods.

The product set used in the NIST team's study covers 40 clinically-relevant cancer variants in 28 genes, each spiked into a background of wild-type DNA at three different variant allele fractions (VAF): 2 percent, 0.5 percent, 0.125 percent, as well as a 0 percent control. The overall panel includes single nucleotides, insertions, deletions, and two structural variations "selected both for their clinical importance and to challenge the performance of next-generation sequencing (NGS) methods," the study authors wrote.

Kenneth Cole, group leader of the NIST's bioassay methods, biosystems, and biomaterials division, said in an interview that he and his colleagues are engaged in a variety of studies of reference materials – including comparisons of other commercial products, as well as an effort to develop their own reference samples in partnership with the National Cancer Institute's Early Detection Research Network.

"It's very hard to do this type of focused comparison with patient samples," he said, to drill down into the performance of each aspect of an assay. "But with [synthetic] references, we can," he explained.

In their study each participating lab — which included several labs from the EDRN, Frederick National Lab's Molecular Characterization Laboratory, and some other outside volunteers — extracted DNA from the SeraCare sample set and characterized it with their in-house method using their own extraction, quantitation, analytics, and bioinformatics methods.

The sequencing approaches included four that employed the Archer Reveal ctDNA Targeted NGS protocol — two using Illumina MiSeq instruments, one using a Hi-Seq, and one using the NextSeq 500. Another lab performed Digital NGS on Thermo Fisher Scientific's Ion Torrent PGM, another used a method called SiMSen-Seq on the MiSeq, and the seventh used a technique called DEEP-seq (Deep Error Eliminating Plasma Sequencing), also on the MiSeq.

Among the insights gleaned from the comparison was, first, that extraction methods used by different labs showed significant differences in yield. Five labs used manual extraction methods, three of which employed the QIAGEN QIAamp Circulating Nucleic Acid kit that SeraCare used to develop the reference materials being studied. Two labs used automated instruments.

Authors noted, for example, that for the 0.125 percent VAF samples, lab C, using an automated extraction method, had statistically different results compared to two of the manual-process labs, while lab D, also automated, had statistically different results compared to a third, different manual-process lab. Similar discrepancies were also observed between lab results using 0.5 percent samples and for the 2 percent samples.

Although limited, authors wrote that the results suggest that "automated methods tended to have lower yields compared to some of the manual procedures." However, the comparison was also able to show that certain protocols could help resolve this. For example, "the addition of carrier RNA in the manual method lysis buffer appears to help improve the recovery of the DNA," they added.

Overall, Cole and his coauthors concluded that the seven NGS assays used in the study showed a "good correlation," although the ability to directly compare was limited by the fact that different panels included varying numbers of targets — from just four for the SiMSen-Seq assay to 32 for the Digital NGS Ion Torrent method.

All the methods detected their respective target variants in the 0.5 percent and 2 percent VAF samples, the group wrote. The 0.125 percent and 0 percent sets were more of a challenge.  SiMSen-Seq detected all four of its target variants at the 0.125 percent level.

The Digital NGS Ion Torrent assay was able to detect 28 of 32 target variants in the 0.125 percent samples, missing calls for APC, ATM, BRAF, and PIK3CA alterations that the same test detected at 0.5 percent.

The study authors wrote that four of the 36 Digital NGS variants (primarily mononucleotide repeats) also yielded false-positive reads in the VAF 0 percent reference material, a limitation that was recapitulated in in-house normal patient samples and is consistent with the known limitations of the Ion Torrent technology.

Such findings illustrate the value of reference materials to help in the validation of NGS assays and benchmarking of different platforms, the authors added.

The Archer Reveal assays were able to detect the 23 target variants in the 0.125 percent samples, albeit with adjustments to the bioinformatic detection algorithm. DEEP-Seq also detected all seven target variants at the 0.125 percent level.

Interestingly, three of the laboratories using the Archer Reveal ctDNA NGS assay also identified variants in the wild-type, or 0 percent sample. Orthogonal ddPCR assays performed by two of the labs showed the same results, suggesting that these "false positives" may reflect that these variants are actually present in the background DNA used to construct the SeraCare product, rather than reflecting errors in detection or informatic methods used.

In an email, SeraCare wrote that the GM24385 background DNA used in its ctDNA positive and wild-type reference standards is well-characterized and widely studied. As a result, the fact that it contains variants across the genome that may be detected by certain assay protocols should not be an issue when the materials are used for their primary purpose: as positive sensitivity controls to determine the ability to detect specific variants, rather than as a negative control. The firm includes relevant information on this in the package inserts of its products.

Cole added that another interesting finding in the comparison was that, though digital PCR is increasingly being used as a type of gold-standard to validate NGS liquid biopsy findings, it's accuracy isn't iron clad and seems to be dependent on how an assay is designed.

"SeraCare used digital PCR on practically all the variants, and we did it on about nine, and we saw some differences because of how we designed the assay," he said.

According to Cole, his group at NIST is also expecting to have results soon from larger studies they are engaged with that are investigating multiple liquid biopsy reference products. With the EDRN, the team is hoping to well characterize these commercial products and develop new reference materials that can help ensure that the new generation of cancer early detection assays now on the horizon can be confidently validated and monitored for quality.