NEW YORK (GenomeWeb News) – A new paper is questioning the sensitivity of many published viral and bacterial genetic signatures.
In a paper appearing online today in the Annals of Clinical Microbiology and Antimicrobials, researchers from Lawrence Livermore National Laboratory used a data analysis technique to predict false-positive and false-negative hits for more than a hundred recently published genetic signatures. Their analysis suggests that the signatures tested are generally very specific, but often lack the ability to pick up all strains of interest — results that could impact RT-PCR-based tests relying on these signatures.
“[C]urrent methods for real-time PCR assay design have unacceptably low sensitivities for most clinical applications,” the authors wrote.
But those involved in designing commercial RT-PCR-based diagnostic tests say they are already aware of the issues raised in the paper and routinely take steps to avoid such pitfalls.
Senior author Shea Gardner told GenomeWeb Daily News that she and her colleagues have been interested in using genetic signatures for bio-threat detection since before the Salt Lake City Olympics. “As you can imagine, false-negatives and false-positives can’t be tolerated — neither one,” she said. Consequently, she’s interested in coming up with the most effective ways to design genetic signatures that are false-positive and –negative free.
While she said that computational methods have proven useful for signature design and evaluation, Gardner said she was also curious about how published signatures would perform in their analysis. She and co-author Gordon Lemmon, currently a graduate student at Vanderbilt University, selected 112 genetic signatures and used a BLAST search combined with other computational approaches to match these against public sequence data and determine false-positives and false-negatives for each.
Their results suggested that signatures tested tended to have high specificity, with few-to-no false-positives. The sensitivity, on the other hand, was generally very low, with many false-negatives. Although specificity was an issue in a few cases, Gardner said, this sensitivity or robustness was the biggest issue, since their analysis suggests that many genetic signatures wouldn’t pick up all existing sequences.
Ideally, Gardner explained, a single signature would pick up all strains of an organism of interest. But that just isn’t possible for some species. To combat that problem, the duo used a technique called Minimal Set Clustering to come up with sets of signatures for organisms with published signatures predicted to perform poorly — including influenza A HA serotypes, foot-and-mouth disease virus, Norwalk, Crimean Congo hemorrhagic fever, Ebola, hepatitis A, and other viruses.
In general, Gardner stressed that researchers should try to find as much sequence data as possible for targets and their near neighbors and to design signatures accordingly. She and Lemmon also urged researchers to continually re-evaluate old assays as more sequence information becomes available. “It’s a process,” Gardner said. “A signature isn’t an endpoint.”
“We were pointing out that [the assays that were tested] really weren’t appropriate for the clinic,” Lemmon told GenomeWeb Daily News.
But reaction to the work was mixed amongst those who design RT-PCR-based tests specifically for diagnostic purposes.
Martin Lee, technical manager at the Porton Down, UK-based firm Enigma Diagnostics, told GenomeWeb Daily News that the paper points out that it’s inappropriate to transfer research tests directly to the clinical field. As such, he noted, the paper could potentially apply to some home brew tests. But, Lee added, “The subject matter doesn’t reflect at all on the quality of diagnostic tests that are out there.”
Enigma is developing RT-PCR-based diagnostic tests for influenza as well as organisms causing several sexually transmitted infections.
Similarly, Steve Visuri, chief science officer for Milwaukee, Wisconsin-based biotechnology firm Prodesse, drew a sharp distinction between the genetic signatures evaluated in the paper — which were designed for research purposes — and those developed for clinical or diagnostic purposes.
This January, Prodesse secured US Food and Drug Administration approval for its ProFlu plus test, an RT-PCR-based test for detecting influenza A and B viruses and the respiratory syncytial virus. And last week the company filed a 510(k) application with the FDA for its RT-PCR-based test for detecting human metapneumovirus.
Whereas RT-PCR-based assays designed for laboratory research may suffer from some lack of sensitivity, Visuri told GenomeWeb Daily News, diagnostic tests seeking US Food and Drug Administration approval undergo exhaustive verification and validation.
Both Lemmon and Gardner agreed that their analysis did not look specifically at genetic signatures used by diagnostic companies. As such, they said, the work doesn’t necessarily reflect signatures used for such purposes. Indeed, Lemmon said that it’s possible that diagnostic companies are implementing approaches suggested in the paper.
And while Enigma’s Lee conceded that there isn’t a good common set of tools for designing diagnostics, he emphasized that researchers designing diagnostic tests for a specific pathogen — for instance, the flu virus — use software as one of many tools in the design process. Such efforts rely on diverse sources of sequence information, he said, including proprietary sequence information, collaboration with expert reference labs, and other information derived from isolates in epidemiological studies.
Researchers designing genetic signatures to detect viruses and bacteria in a diagnostic setting already use some of the approaches described in the paper, Lee said, such as checking sequence databases and designing signatures with more than one primer set — including, for example, multiple targets and controls for internal reagent validation.
Visuri, likewise, said that Prodesse is following many of the approaches proposed in the paper. For instance, he said, he and his colleagues comb databases for new flu sequences at least every year, if not more. “I think they give some good advice in there that we already follow,” Visuri said.