NEW YORK – A team of UK researchers recently began using machine-learning tools provided by the company diagnostics.ai, and assessed them in a prospective clinical trial to determine whether artificial intelligence could perform better than specialists in clinical test interpretation.
After comparing manual methods with the firm's pcr.ai tool in more than 20,000 cases, they found that the use of AI improved test accuracy and reliability. The results of the study were published in the Journal of Clinical Virology last month.
Rory Gunson, co-author on the paper, said that the researchers were "surprised by how effective and efficient" the system was compared to manual approaches. Gunson is director of the West of Scotland Specialist Virology Center in Glasgow, which carried out the study. The center is based at NHS Glasgow & Clyde Hospital and is one of the UK's five specialist reference laboratories that advise authorities on regional and national issues.
Gunson said that based on the findings, the application of AI could enable laboratories to "achieve greater standardization and accuracy" as well as to reduce costs. He declined further comment.
Diagnostics.ai's tool, called pcr.ai, automates and standardizes quantitative PCR test analysis, enabling real-time monitoring on a test-by-test basis. The London-based company developed the machine-learning system in part to address shortages of trained laboratory staff, as well as to help labs improve internal efficiencies, both in terms of time and money saved.
As outlined in the paper, the researchers undertook the study to evaluate the pcr.ai tool's accuracy as well as impact when automating the manual data analysis and quality control steps associated with routine clinical pathogen testing using quantitative real-time PCR. They evaluated the use of the tool when used as a final interpretation and verification step for routine in-house qPCR tests for respiratory pathogens and for norovirus for a total of 22,200 interpretations. The tests were run over the course of a month.
As part of the evaluation, they compared their results using the tool to existing manual interpretation, to assess accuracy and hands-on time savings. There was 100 percent concurrence between the AI-based and the manual approaches. They also saved, on average, 45 minutes per respiratory panel run and 32 minutes per norovirus panel run. They concluded that pcr.ai presented them with a tool that reduced the complexity of analysis, lessened the need for specalists, and enable users to get results quickly and at a lower cost with less of a risk for errors.
According to diagnostics.ai CEO Aron Cohen, the firm approached the Glaswegian team with its tool as a potential way to reduce their reliance on specialists for interepreting lab results. "Given that we have a method that we feel reduces need for this expertise, we thought it would be something [that] could resonate with them," he said.
Diagnostics.ai, formerly called Azure PCR, was founded in 2013 and currently employs around 20 people. The firm rolled out pcr.ai to select customers two years ago and is gearing up for a broader launch next year, Cohen said. "These early customers help us to better understand what the typical lab needs, how they work," he noted. "That's why we started working with collaborators like those in Glasgow, to see what it is they need and desire, and how that will drive and improve test turnaround time."
Diagnostics.ai's system is based on its internally developed AI-based learning machines. The algorithms study how experts interpret test results and then use the knowledge to automate future analyses. The pcr.ai tool was developed specifically to carry out and standardize qPCR test analysis and to overcome shortages in clinical laboratory staff.
Typically, the company works with a lab's existing data, which is fed to the learning machine during a training phase. Any inconsistencies are flagged and dealt with during an interactive phase. Once completed, pcr.ai is handed off to users to run in real-time on a standalone server that is not accessible to the firm. Revenues are derived via a service and support model, influenced by volumes and support needs. The firm's emphasis, however, is on reducing costs within labs.
"The model, especially in the UK, tends to be savings driven," said Cohen. "I think that's an important approach we have taken, that in addition to improving quality and accuracy, turnaround time, and getting better results faster, reducing costs is a vital piece of that puzzle."
As it prepares for a broader launch, the company is also investing internally in support so that the company can troubleshoot any problems with the pcr.ai tool should they arise. "Some places have six- or seven-day weeks including night shifts," Cohen noted. "We need to be able to provide support for that," he said. "It's important to have the resources to be able to cope." He added that the results of the recent paper could support broader adoption of pcr.ai by labs.
"I think taking in over 20,000 results, and proving we could do it faster, with no errors, and even picking up mistakes made in the manual process are really impressive results for any clinical or AI product," said Cohen.
'Significant' findings
Ayazali Nazafi, a senior biomedical scientist at Viapath, similarly said the new study was "quite significant," as it demonstrated to potential users that not only does an AI-based approach reduce costs and time, but also produces accurate results.
"I think all PCR interpretations in the future will be handled this way," said Nazafi. "As the number of specimens increases, as the number of assays go up, a person sitting and reviewing the standard curve might be a thing of the past," he said.
Nazafi added that adoption within the UK will likely be based on hubs, where labs send their data off to a centralized system for interpretation, and results are provided back within minutes rather than hours. "That's probably how it will roll out," he said.
Viapath is a London-based pathology services provider owned by British public services provider Serco, Guy's and St. Thomas' NHS Foundation Trust, and King's College Hospital NHS Foundation Trust. Nazafi is based at King's College Hospital and has been working together with diagnostics.ai since last year to implement pcr.ai in Viapath's South London Specialist Virology Center for respiratory, sexually transmitted disease, and gastrointestinal tests.
Viapath and diagnostics.ai have also added features that will allow users to trace specimens at every step of the analysis process so that they might be able to inform better decisions, akin to integrating laboratory information management systems (LIMS) with pcr.ai.
"This way, we would need to access just one system, rather than multiple softwares," Nazafi said. "With PCR, specimens are extracted and may be inspected on multiple platforms," he elaborated. "We are hoping that pcr.ai can string all of those processes into one software, so that all of those specimens are traced all the way through." According to Nazafi, Viapath is now trialing these functions making its work different from what was trialed in Glasgow.
"They primarily looked at the interpretaton power of the system and the time-saving aspects," Nazafi said. "We think time can be further reduced with intergation into a LIMS system," he added. Normally, he noted, these are two different entities. Instruments provide results, then they are transferred to a LIMS system.
Simon Bengen, chief product officer at diagnostics.ai, said that customers stand to benefit from greater integration of LIMS-like features together with the pcr.ai offering.
"This is a general trend across industry of consolidating data streams into a single platform," Bengen said. While the performance of pcr.ai to date has been "brilliant," he stressed that the company continues to innovate. "There is a lot more we can do going forward," he said.