NEW YORK (GenomeWeb) – A large, prospective study of pediatric gastroenteritis patients in Canada recently discovered higher-than-expected false-positive and false-negative results for Salmonella spp. in the Luminex xTAG Gastrointestinal Pathogen Panel (GPP) multiplex molecular test, a problem that the company has since solved.
Luminex said it had discovered the problem independently, through customer complaints received prior to the study's publication, and recently issued a software adjustment to fix the problem.
Published earlier this month in the Journal of Clinical Microbiology, the study prospectively evaluated stool samples and rectal swabs from 3,089 children with acute gastroenteritis between December 2014 and March 2018. It looked at the agreement between the Luminex GPP bacterial targets and standard-of-care microbial culture with MALDI-TOF-based pathogen identification, using specific qPCR tests to verify positive results.
The Luminex multiplex GI panel was initially cleared by the US Food and Drug Administration in 2013 with 12 targets, and the firm added three more targets the following year. The bacterial targets and toxins detected include Campylobacter, Clostridium difficile Toxin A/B, Escherichia coli O157, Enterotoxigenic E.coli (ETEC), Shiga-like Toxin producing E.coli (STEC), Salmonella, Shigella, and Vibrio cholerae. A Yersinia enterocolitica bacterial target is approved only in Canada. It also detects viruses, including adenovirus, norovirus, rotavirus A, as well as Cryptosporidium, Entamoeba histolytica, and Giardia parasites.
A 2014 evaluation found Luminex GPP results to be highly reproducible across clinical and public health laboratories and touted the improved turnaround time and a more efficient workflow than conventional culturing methods.
However, although multiplex testing for gastroentric pathogens is being increasingly adopted — with panel-based testing from GenMark, BioFire, Applied BioCode, and Becton Dickinson also being taken up by labs — there have not been many studies of GI panels specifically in children or in high-income countries, according to senior author Stephen Freedman, a pediatric gastroenterologist at Alberta Children's Hospital.
"It's really important that we understand the potential benefits and potential pitfalls of using this technology, particularly because the cost can be exorbitant," Freedman said. "If it also leads to inaccurate or inappropriate anti-microbial use, inappropriate isolation, or inappropriate public health investigations, there can actually be a lot of downsides."
Freedman's colleagues on the research team — otherwise known as the Alberta Provincial Pediatric EnTeric Infection TEam, or APPETITE — included clinicians and scientists affiliated with the University of Alberta, Edmonton and the University of Alberta, Calgary medical schools, as well as Calgary Laboratory Services and the Alberta public health services labs, and Washington University in St. Louis School of Medicine.
The team found a greater than 99 percent overall agreement between stool culture and the Luminex GPP for Campylobacter spp., E. coli O157, Shigella spp., and Salmonella spp. targets.
However, while the overall diagnostic accuracy is very good, that is perhaps partly related to the fact that most of the targets in the panel are rare and typically yield negative results, Freedman said.
"The question is, of the positives, how many of those are accurately reported?" he said.
Indeed, there were a number of false-negative GPP results in samples where pathogens grew in culture that were not detected by the GPP.
There were also false-positive GPP results where pathogens were detected by the GPP that neither grew in culture nor were detectable in the in-house PCR.
The most frequently detected pathogen, Salmonella spp., was found in 64 patients out of more than 3,000 samples, but only 43 patients had agreement between the testing methods, with 12 detected by culture only and nine by GPP only.
All of the Salmonella spp. results that were considered GPP false negatives had higher median cycle thresholds, suggesting a lower concentration of pathogen, and later tested positive on isolate suspensions from the cultures. These also tended to come from younger children with less diarrhea and more vomiting.
The kids with false positive Salmonella spp. GPP results didn't have classical symptoms of infection, and retesting with PCR-based Salmonella assays didn't yield anything, strongly suggesting the GPP result was not correct. The team suggested there might be an issue with the test cross-reacting with something else that resembles Salmonella spp. and also found similar results for some GPP-positive, culture-negative E. coli O157 specimens.
The sample type and test population have their own unique issues, Freedman noted. "Stool is full of bacteria … and you can shed pathogens for weeks to months after infection," he said, with children being more likely to be asymptomatic carriers of pathogens. In fact, public health labs are moving away from testing for pathogens to confirm that a gastrointestinal infection has cleared, he said, in part because otherwise people might be overtreated and quarantined for too long.
A false-positive Salmonella test would probably not result in treatment, since children infected with Salmonella generally do not require antibiotic therapy, Freedman said, unless they are high-risk or have severe disease. However, there could be unneeded public health investigations of false positives, as well as inappropriate isolation, exclusion from school, additional testing, stigma, and unnecessary worry for caregivers. Also, a false-positive Salmonella test could perhaps distract from the real pathogen or cause, Freedman said.
False negatives, on the other hand, could result in increased transmission, with a chance of transmitting Salmonella to someone who is immunocompromised or very young, as well as leading to ongoing investigations seeking a cause of illness that would be unnecessary if the test had been accurate.
Although the vast majority of children don't need treatment for typical gastroenteritis, and the study was for research only, if clinicians are testing, they want accurate results, Freedman said.
The GPP is not approved for rectal swabs, but Freedman noted that the APPETITE group has previously shown a high concordance of swabs and stool samples. They have also previously published some issues with the GPP norovirus target.
In general, for multiplex panels, "diagnostic accuracy needs to be deeply explored," he said, recommending increased post-market surveillance for these tests.
"Although people are moving over to culture-independent diagnostic technology, it is important that the ability to culture continues for all bacterial isolates – for resistance, susceptibility patterns, strain and typing, and to identify outbreaks," Freedman said.
Internal monitoring
According to Sherry Dunbar, Luminex's senior director of global scientific affairs, the company had already been aware of the problem with the Salmonella target.
Indeed, it had issued a fix prior to the publication of the JCM article. Specifically, the APPETITE study was published earlier this month but Luminex changed its software for the test in March.
Still, "I thought it was a really good study and very well done," Dunbar said.
The company became aware of problems with the target through post-launch success or surveillance meetings, she said. These start as soon as a product is launched, and she believes most companies have them.
"In these meetings, every call that comes into tech support is reviewed and gone through for troubleshooting," she said. "This is what we [at Luminex] do, but I'm pretty sure that the other diagnostic manufacturers do this as well," in order to maintain ISO certification and comply with FDA.
"Based on customer feedback … we'd been reviewing threshold settings and call logic, and in March of 2019, we implemented new thresholds and a new call algorithm for Salmonella to address the issue of the false positives," she said.
It is a particularly difficult organism, Dunbar said, because it requires multiple targets to get adequate sensitivity, which can come at the cost of specificity. Dunbar herself worked with Salmonella as a postdoc, and she said there are thousands of serotypes. Furthermore, "there is a lot of similarity between different organisms, so it can be challenging to come up with a sequence that all the Salmonella has but nobody else has," she said.
Also, according to the FDA submission, the Luminex GPP Salmonella target has the lowest agreement with samples that had been characterized using bacterial culture, about 90 percent.
A change in software is not too much of an undertaking — the company is now sending out a new CD to update the instrument with all new shipments of GPP tests — but any change in the target itself would require a new multi-site clinical trial, Dunbar explained.
"If we were to go in and make changes, it would essentially have to be re-cleared, which is why most companies don't do that very frequently," she said.
Furthermore, the GPP was the first GI panel to be cleared by the FDA and is cleared as a "presumptive positive," meaning customers are recommended to verify positive results by another method, Dunbar said, noting that labs often handle this by doing a study for a period of time to prove concordance to themselves rather than test everything twice.
Luminex is now working on a sample-to-answer assay with its Verigene system, she said, noting that the GPP can only be run in a high-complexity lab, but panels have a lot of advantages overall, in that they basically consolidate syndromic testing. Still, more studies of outcomes and clinical utility would be very valuable.
Issues with targets in multiplex panels from other manufacturers have also cropped up recently.
BioFire, for example, hit a threshold of customer complaints this year that triggered an internal evaluation, and the firm stopped shipping the BioFire FilmArray GI panel for five days in March while it evaluated its nonspecific amplification of Campylobacter and Cryptococcus targets, according to Wade Stevenson, senior vice president of global marketing at BioFire Diagnostics, a BioMérieux company.
"We're back up and providing product to our customers again, with a bit of an injunction that for positives for a couple of targets on our panel, you have to confirm with a secondary test," he said, adding that this could be a second BioFire panel or another assay.
He agreed that this type of post-market review is typical for panel makers. "It's hard to say as a company when you become aware [of an issue] — you hear a little bit of rumbling, there are lists and discussion groups and you monitor them all — but the main metric is the rate of customer complaints," Stevenson said.
BioFire is now working on a software fix that may essentially improve the efficiency of the hot start amplification for the two problem GI targets.
Stevenson said that the firm also recently issued a similar information recall — also not a product recall — for its blood culture identification (BCID) panel, which has been detecting contaminating nucleic acids from dead pathogens present in certain lot numbers of blood culture bottles.
Freedman and the APPETITE group said that it is important the new modifications by both Luminex and BioFire "be evaluated and demonstrated to have real-world accuracy."