This story has been updated to include additional commentary from test developers.
NEW YORK – The US Food and Drug Administration on Tuesday published the first results of a limit of detection comparison of Emergency Use Authorized molecular diagnostic tests for SARS-CoV-2.
Reported as nucleic acid amplification test-detectable units (NDU) per milliliter, the limits of detection for the first batch of 55 tests reported differed by 10,000-fold.
In a virtual town hall on Wednesday, Timothy Stenzel, director of the FDA's Office of In Vitro Diagnostics and Radiological Health, said the agency hopes the reference panel information will be helpful.
Stenzel said the FDA assessed relative LoDs for tests that use viral transport media (VTM) and are extracted, as well as for tests that run from dry swabs and, separately, saliva tests. The data from the three groups is presented in three different tables.
"Within a table, you can look at relative LoD and you can understand, I think to a great degree, based on how we asked the testing be done and our analysis, you have a pretty good rank order from a low LoD to a high LoD," Stenzel said, but added that making comparisons between tables is more difficult.
FDA grew its own SARS-CoV-2 virus in culture to create the panel, then inactivated it, and tested that it was inactivated, Stenzel said. The agency began shipping the reference panel to diagnostics developers in May.
In a statement on its website FDA said that as of Sept. 10 the agency had contacted developers of 176 EUA assays for shipping information and had sent the reference panel to developers of 152 assays.
"The FDA is reviewing results as they are returned, and continues to send the reference panel out to additional developers," the agency said.
Stenzel noted that the agency is working on finalizing LoD determinations for more tests and will post the next batch as soon as they are ready.
The agency said that developers who received the reference panel were asked to conduct testing and return results within two weeks of receiving the panel.
FDA noted that many developers returned data to the agency by Aug. 31, 2020, but in some cases, it did not receive the data, or the data was uninterpretable, or is still under interactive review.
At the lowest end, or best-performing of the LoD results is the PerkinElmer New Coronavirus Nucleic Acid Detection Kit, with a reported limit of detection of 180 NDU/ml.
A test from ScienCell Research Laboratories called the SARS-CoV-2 Coronavirus Real-time RT-PCR Detection Kit reported 540 NDU/ml, while tests from Hologic, DiaCarta, BioCore, and Seasun Biomaterials had limits of detection of 600 NDU/ml.
The LoD listing then jumps to 1,800 NDU/ml. Tests from CirrusDx Laboratories, Euroimmun, Helix OpCo, LabGenomics, Quest Diagnostics, Rheonix, and Roche fell into this category, as did one of the tests offered by Becton Dickinson.
The US Centers for Disease Control and Prevention 2019-nCoV Real-Time RT-PCR Diagnostic Panel reported an LoD in the evaluation of 18,000 NDU/ml.
At the high end of the spectrum, three tests — the SARS-CoV-2 PCR Test from Diatherix Eurofins Laboratory, ARIES SARS-CoV-2 Assay from Luminex Corporation, and QIAstat-Dx Respiratory SARS-CoV-2 Panel from Qiagen — reported limits of detection of 180,000 NDU/ml.
The QIAstat-Dx test is the only syndromic panel on the list, a spokesperson for Qiagen said in an email, adding that it has consistently demonstrated high sensitivity in different settings.
In validation studies for the EUA, the test demonstrated 100 percent sensitivity, while a reference hospital in France using the test in an ER setting recently showed 100 percent sensitivity in an assessment of 69 clinical samples, the spokesperson said. AÂ test of labs by the American Proficiency Institute also reported 100 percent sensitivity for five labs using the QIAstat-Dx that were challenged with a sample of containing 5,175 SARS-CoV-2 viral copies per milliliter.
The Qiagen spokesperson pointed to a disclaimer from an FDA statement that notes while a lower LoD represents a test's ability to detect a smaller amount of viral material in a given sample, "signaling a more sensitive test," the LoD data "does not indicate how sensitive a particular test is."
The Qiagen representative commented that the validity of low viral load as a marker of disease "remains questionable." Nevertheless, the QIAstat test is the only multiplex syndromic respiratory panel capable of providing access to Ct values and amplification curves.
Two direct-from-swab tests were reported separately. The Abbott ID Now had an LoD of 300,000 NDU/ml, while the Quidel Lyra Direct SARS-CoV-2 Assay had an LoD of 540,000 NDU/ml. There was a single saliva-based test in the dataset, the Advanta Dx SARS-CoV-2 RT-PCR Assay from Fluidigm, which had an LoD of 54,000 NDU/ml.
Alex Greninger, a virologist at University of Washington, said that at first glance the results don't completely square with his group's published evaluation of the Hologic, DiaSorin, Roche, and Cepheid assays, which found Cepheid's test to be the most sensitive.
The PerkinElmer test's secret sauce may have something to do with the extraction, Greninger speculated. "They're not extracting a ton of material but they use all of what's extracted, so the PCR sees a good amount of VTM," he said. In addition, "Their orf1ab primer looks pretty decent," Greninger added.
Masoud Toloue, vice president and general manager of diagnostics at PerkinElmer, commented in an email that the firm's achievement of the best LOD performance with the FDA reference panel is "no coincidence," but directly related to three key factors.Â
The test uses "highest-quality magnetic bead and PCR raw materials, a stringent manufacturing and quality control system for RT-PCR reagents, and, most importantly, years of our team's experience and expertise in developing NAT tests for blood screening," Toloue said.
In addition to listing LoD results, the agency is also listing tests as "Data not returned," "Data not interpretable," "Under interactive review," and "Did not provide shipping information."
One lab and one firm in the "under interactive review" category commented that this likely means the FDA is still reviewing the data that was already provided rather than that there is some back-and-forth process going on.
Fred Turner, CEO of Curative, a firm that had previously acquired KorvaLabs, said his team submitted the KorvaLabs reference panel data "some time ago," and he commented that "current review times are still substantially backlogged."
A spokesperson at Thermo Fisher Scientific said the firm has already submitted data on its TaqPath COVID-19 Combo Kit test. "Evaluation of our SARS-CoV-2 EUA test with the agency's updated reference panel is a condition of the EUA issued to Thermo Fisher and other test developers," the spokesperson said, adding, "The company has fulfilled this post-EUA commitment, and the agency is reviewing results."
GenMark's ePlex SARS-CoV-2 Test, meanwhile, is among the assays listed as "data uninterpretable."
A spokesperson for the firm said that GenMark did submit data for this program, but the FDA requested that they re-run the study using negative patient specimen VTM rather than blank VTM. "GenMark has requested the FDA update its status to 'Under interactive review' based on these discussions," the spokesperson said.
And, BGI is listed in the FDA's table as "Did not return results," but a spokesperson for the firm said this is not the case. "We have returned our results and report to the FDA in time," the spokesperson said, adding they are not sure why the data is not listed, but referred to the FDA's site suggesting the term means that no data was received prior to Aug. 31.Â
Atila Biosystems is also listed as not having returned results, but the firm's CEO, Youxiang Wang, commented in an email that this was because the project manager was on sick leave, and that Atila will be submitting the data this week.
Ithaca, New York-based Rheonix, meanwhile, had an LoD comparable to tests from BD, Quest Diagnostics, and Roche. Richard Montagna, senior vice president for scientific and clinical affairs, commented that the firm is pleased that FDA has created a reference panel that all manufacturers can test against, so that laboratories can evaluate the relative analytical sensitivities of the currently available tests.Â
"Up until now, it has been difficult to compare tests on an 'apples-to-apples' basis," Montagna said. "Rheonix has known that our test performs well in our customers' hands, and we are delighted to see that the FDA process has validated their feedback."
Chris Mason, a genomics researcher at Weill Cornell Medicine and scientific advisor at Resilience Health, said his group will incorporate the FDA panel results into its analyses of EUA tests.
The team published results of its comparisons of FDA Instructions for Use documents in Nature Biotechnology, with the top assays lining up fairly well with those in the FDA's evaluation. Namely, the PerkinElmer test had the lowest limit of detection in this comparison, as well, and the top results included tests from BD, Roche, Quest, and DiaCarta, as well as ViraCor, Ezno, Abbott, and BGI.
"There is clearly a wide range of performance for various SARS-CoV-2 tests on the market, including multiple orders of magnitude difference in the LoD, plus varied methods for sample collection, extraction, priming sites, and overall processing," Mason said.
"The FDA work, our own work, and related benchmarking efforts at [the National Institute of Standards and Technology (NIST)], X-Prize, and the FDA will help adjudicate the limitations of any tests and help confirm that primers in use for diagnostics are valid," he said. Mason also commented that evolutionary selection pressure on the primer sites can lead to loss of primer reliability, and so, "Frequent validation is helpful to ensure that our target sites for primers are still efficacious."
In the town hall, Stenzel commented that as soon as an international standard for molecular testing is available, the agency will "anchor the FDA reference panel to that, and we will be able to translate the results we posted to international units."
Stenzel said, "We think that will be a huge advancement on top of what we've already done."