NEW YORK (GenomeWeb) – Seeking to improve the quality of next-generation sequencing in clinical testing laboratories, the European Molecular Genetics Quality Network (EMQN), a global provider of external quality assessment (EQA) services, in collaboration with its UK partner, has been developing a pilot scheme that will eventually allow laboratories to assess their NGS performance.
Last month, EMQN and UK NEQAS (National External Quality Assessment Service) for Molecular Genetics wrapped up their 2015 pilot scheme for next-gen sequencing with a final report. The scheme focused on the ability of labs to correctly call variants in a test sample, and on the general quality of their sequence data, but it did not assess the clinical interpretation of the results. Overall, 236 laboratories from 40 countries participated, with the UK, Germany, and France providing the largest number, though there was also significant participation from labs in Australia, Spain, Switzerland, the Netherlands, Denmark, and Italy.
EMQN and UK NEQAS are currently developing the 2016 NGS assessment, which labs will be able to enter in September. Like the one in 2015, which was the third NGS scheme the partners conducted, this year's will be a pilot. But the hope is to offer a fully-fledged scheme with performance measurements in a couple of years or so, Simon Patton, the director of EMQN, told GenomeWeb. Over the next few months, he and his colleagues plan to publish results from the first three pilots in a peer-reviewed journal, he added.
"Our model works really well," Patton said. "We just need to ensure that it can handle the huge demand from the laboratories." For the 2016 NGS scheme, the organizers expect that almost 300 labs will participate, and they are currently developing mechanisms for handling the data analysis. Traditionally, EMQN has primarily been working with diagnostic laboratories, but because the NGS scheme focuses purely on technology and not on clinical interpretation, it would also work for research labs wanting to assess their NGS performance, Patton said.
The 2015 scheme came in two flavors: one for germline sequencing, in which 211 labs participated, and the other for somatic testing, for which 26 labs submitted data. In each case, laboratories sequenced a DNA sample that had been extensively characterized by the organizers and their collaborators.
Labs analyzed these samples using whatever NSG platform and analysis pipeline they had established but were asked to run their smallest panel if they offered NGS panel testing, and their largest single-gene NGS test if they didn't offer panels. "We wanted something that would fit just NGS as a process, irrespective of how they were doing the process or what type of field they were working in," Patton said.
The sequencing data were collected and stored by a third-party contractor, Euformatics of Finland, which also helped with the data analysis.
Participants had to submit a list of the genes they analyzed and genetic variants they identified but were also encouraged to send in their raw sequence data and alignments.
EMQN provided labs with individualized reports showing how their variant calls compared to consensus calls the organizers had previously established. For the germline sample, these calls were based on genome and exome sequencing data from four partner laboratories who used either an Illumina or an Ion Torrent platform, and they included all variants that were detected by at least two of the four labs. "There are only Life Technologies and Illumina, really, in the diagnostic field," Patton said.
For the somatic sample, the organizers used a well-characterized DNA sample from Horizon Discovery, a company that provides reference materials.
EMQN and UK NEQAS need to establish new references for their NGS scheme every year, "but we now have developed a mechanism that we think is pretty robust and allows us to quickly develop these reference materials," Patton said, adding that his team plans to publish details about this process within the next six months or so.
In addition to the variant analysis report, participants also received a report with quality metrics derived from the data they submitted, including a comparison with other laboratories.
The new NGS pilot assessment is just one of EMQN's many offerings, all aimed at improving the quality of molecular and genetic testing. In addition to providing a variety of EQA schemes — better known in the US as proficiency testing — the nonprofit organization helps other institutions develop best practice guidelines by hosting and organizing meetings.
Founded in 1997 as an offshoot of the UK NEQAS and initially funded through a grant from the European Union's 4th Framework Program, EMQN has grown into a global provider of annual external quality assessment schemes for inherited disease genetic testing, molecular pathology testing, and certain molecular technologies, serving about 1,800 member labs in 67 countries.
Membership is open to any laboratory, not just clinical labs, and is a prerequisite for participation in the assessment schemes. Besides diagnostic service labs, members include, for example, commercial kit manufacturers, who use the schemes as part of their validation process, Patton said, as well as commercial sequencing laboratories.
While many member labs hail from Europe, "that is definitely shifting," Patton said. For example, membership is high in China among labs that focus on molecular pathology cancer testing, and EMQN is currently the main provider of genetics EQA schemes for Australia, he said.
EMQN, which is housed at the Manchester Centre for Genomic Medicine at St Mary's Hospital in Manchester, UK, is also no longer relying on grants, financing its activities entirely from membership fees, which run at £45 ($63) per year, and participation fees for EQA schemes, which range from about £100 to £400, depending on the type of scheme.
The network started out with assessment schemes for genetic tests for individual inherited diseases, such as fragile X syndrome. "That's still the bread and butter of what we do," Patton said, adding that about 70 percent of EMQN's work comes from individual inherited disease EQA schemes.
In addition, EMQN provides assessment schemes for molecular pathology testing, including for solid tumor testing in lung, melanoma, and colorectal cancer.
Both the inherited disease and the molecular pathology assessments look not only at the technical process of testing but also at the clinical interpretation of the results and the diagnostic reporting, he said.
About 10 years ago, EMQN realized that it could no longer provide single schemes for every testable disease, Patton said, so it added technical EQA schemes, which do not assess the clinical interpretation of the data. The largest technical scheme to date remains Sanger sequencing, with more than 300 participating labs. "NGS is sexy, but Sanger sequencing is routine day-by-day testing, and there is still plenty of Sanger sequencing going on," he said.
Besides the Sanger scheme and the pilot NGS scheme, EMQN offers a technical scheme for arrayCGH testing and one for oncogene panel testing, and it is developing new schemes for noninvasive prenatal testing. In addition, in collaboration with the International Quality Network for Pathology (IQN Path), EMQN is working on a scheme for liquid biopsy cancer testing.
"We have not tried things like virology or infectious diseases," Patton said. "I think the questions are slightly different in those. In principle, the mechanism would work, but the bioinformatics side of it is probably more complex, and that's the side we have not really explored."
Participation by labs in external quality assessments is often required by the organizations that accredit them, Patton said, which differs widely between countries. "In the US, it's heavily regulated, and in some European countries like the UK, Germany, France, and the Netherlands, it's the same, but for most countries in the world there is still no accreditation system up and running," he said.
The market for providers of proficiency testing services is competitive, he added, but EMQN is the biggest global provider for genetics. Many countries organize their own national assessment schemes, like UK NEQAS, he added, and some providers focus on certain aspects of the testing but not others.
Unlike EMQN, not many organizations provide their services globally, he said, with a few exceptions, including the College of American Pathologists. In the US, CAP sets laboratory standards and also conducts quality assessments, which is why the US market has been "pretty much closed" for EMQN, he noted, with few US labs participating in its schemes.
Four years ago, CAP published an accreditation checklist for clinical labs performing NGS, and according to Patton, CAP has been working on its own NGS assessment scheme.
On the whole, laboratories do not tend to advertise their participation in EMQN's schemes, he said, unless they performed exceptionally well, though commercial providers sometimes do mention their participation because they see it as a competitive advantage. "But most of the diagnostics around the world goes on not in commercial companies but in public sector laboratories," he said.