NEW YORK (GenomeWeb) – Roughly three years into PrecisionFDA, a cloud-based next-generation sequencing platform developed by the US Food and Drug Administration as part of the Precision Medicine Initiative, the agency has adjusted its approach to the initiative by engaging with the biomedical community through a series of computational challenges.
At the launch of PrecisionFDA in 2015, the FDA tapped DNAnexus to develop a data platform, which it said at the time would be used to share NGS datasets, benchmark bioinformatics approaches, and generate reference standards for evaluating the efficacy and quality of NGS-based tests. The FDA began testing the platform as part of a closed beta program in November of that year.
"Initially, the concept was to provide basic capabilities and resources and reach out to the community directly to help build the library of resources and engage in discussions," said Elaine Johanson, director of the FDA's office of health informatics. However, "what we quickly learned is that the experts were busy and, while supportive and excited about the forum, hesitant to engage," she said. To build the community and increase engagement, the agency decided to launch a number of computational challenges, which started in 2016 and continue today.
Far more people than the FDA initially anticipated have signed on to participate in the PrecisionFDA initiative since its launch. Currently, there are more than 3,000 users on the platform, up from roughly 100 in December 2015, Johanson said, exceeding the developers' estimates of around 1,000 users. The list of participants includes researchers from the National Institute of Standards and Technology, the National Institutes of Health, Illumina, Roche, the Broad Institute, and the US Centers for Disease Control and Prevention.
"What began as a research effort has evolved into what is now a globally used and important scientific collaboration and testing platform" that provides a "virtual multiomics lab for experts to work collaboratively with FDA scientists to advance regulatory science," she said. One of the benefits of this expanded network is that the agency now has significant engagement from the community to better understand the needs in different areas.
In 2016, the FDA launched the first community challenge for PrecisionFDA, called the Consistency challenge, which focused on assessing the reproducibility of pipelines for processing NGS data. Participants were expected to use their informatics pipelines to process datasets provided for the challenge for mapping and variant calling, and then rerun their pipelines on one of the datasets to get a second vcf file for the same sample. They performed pairwise comparisons of the data to evaluate the reproducibility of their pipelines using the same sample as well as with a benchmark dataset. That same year, PrecisionFDA partnered with researchers from the Sync for Genes initiative and other organizations to implement an application interface for the platform that allows community members to download variant call file comparison results in a standard Fast Healthcare Interoperability Resources (FHIR) format.
The second challenge, called the Truth challenge, was conducted in 2016 in collaboration with NIST's Genome in a Bottle (GIAB) consortium and the Global Alliance for Genomics and Health. It offered the community an opportunity to test their variant calling pipelines on a previously uncharacterized sample, called HG002, and then publish the results on PrecisionFDA for comparison to the truth dataset. The third challenge, called the Hidden Treasures, asked participants to investigate the accuracy of their pipelines by using them to find single nucleotide variants, as well as insertions and deletions that were added to fastq files from exome sequencing of reference cell lines.
The first two challenges were intended to help the FDA better understand the questions that are important for assessing the reproducibility and accuracy of NGS tests, and to obtain better benchmarking datasets for NGS test development and validation, according to Živana Težak, associate director for science and technology and personalized medicine policy in the FDA Center for Devices and Radiological Health.
Specifically, the Consistency challenge aimed to provide "a common frame of reference for measuring some of the aspects of reproducibility, and to a lesser extent accuracy, of participants' pipelines," she explained. "It was helpful to learn and distinguish the performance of pipelines that were deterministic [versus] the non-deterministic ones."
For its part, the Truth Challenge aimed to assess the accuracy of informatics pipelines for analyzing unknown human samples. "Participant entries were evaluated against the consensus 'truth' data released by GIAB team … using a new version of data comparator developed by the GA4GH team as the challenge was unfolding," Težak said. "This was a more granular comparison methodology than [the one] previously used."
The emphasis of the third challenge, conducted in 2017, was on understanding the performance of NGS technology for difficult-to-detect or rare variants, or for low variant allele frequencies (VAFs) that are harder to find in clinical samples. Specifically, it focused on the detection of SNVs and indels at certain pre-determined VAFs. This challenge "helped assess the performance of different NGS pipelines not only to detect specific variants, but also to understand their properties in calling specific VAFs," Težak explained.
The next few challenges focused on identifying infectious agents. The so-called Center for Food Safety and Applied Nutrition (CFSAN) Pathogen Detection challenge, which took place earlier this year, asked participants to test the efficacy of their pipelines for detecting Salmonella in shotgun metagenomic samples gleaned from contaminated cilantro. The CFSAN challenge "really made it clear that we certainly still have work to do on our reference databases and … on improving the pipelines for detecting and hunting these pathogens," said Errol Strain, senior science advisor for the FDA Center for Food Safety and Applied Nutrition. But what also became clear, he added, was that the pipelines can work well in the hands of experts.
The next step is to translate what was learned in the challenge into resources that can be deployed in federal field labs and state public health labs, Strain said, adding that he expects this to involve both bioinformatics and laboratory changes.
"Participants were able to do an exceptional job on synthetic datasets, [however] real datasets still have some challenges," he said. There is also work to be done on target enrichment techniques to increase the presence of pathogens in samples earmarked for metagenomic testing. Furthermore, "the quality of the reference databases used for typing is important," he added. "The only way that we can really advance those is through better collaboration and sharing."
Another competition, the Biothreat challenge, which wrapped up on October 18, provided datasets and references for comparing the performance of bioinformatics tools used by the biothreat and infectious disease diagnostics community. Developers were asked to benchmark their algorithms for identifying and quantifying emerging pathogens on blinded mock-clinical and in silico metagenomics samples. This challenge used genomes from the FDA-ARGOS database, which publishes quality-controlled microbial reference genomes for regulatory use. The challenge was jointly sponsored by the FDA and the U.S. Army Medical Research Institute of Infectious Diseases.
"We really expect the results of this challenge to include the development of novel computational algorithms for identifying pathogens in a clinical matrix, for example, the Ebola virus," said Heike Sichtig, a principal investigator in the microbiology division of the FDA’s Center for Devices and Radiological Health.
"We also want to create a lasting and documented and independent evaluation of every NGS computational algorithm, a fixed reference database to aid future developers, and we want to generate greater public and private engagement in infectious disease detection," she said. Sichtig noted in an earlier conversation that one of the challenges of comparing assays and algorithms for analyzing pathogenic samples is that companies and groups developing tests use a mixture of both public and proprietary databases as references, and that the makeup of these databases changes.
Last month, the PrecisionFDA team launched another computational challenge, in partnership with the National Cancer Institute and the Clinical Proteomic Tumor Analysis consortium, dubbed the NCI-CPTAC Multi-omics Enabled Sample Mislabeling Correction challenge. It focuses on sample and data mislabeling – accidental swapping of patient samples and patients omics data, respectively. According to the PrecisionFDA website, the objective of the challenge "is to encourage development and evaluation of computational algorithms that can accurately detect and correct mislabeled samples using rich multi-omics datasets."
The challenge, which wraps up in December, is also discussed in a Nature Medicine paper published by researchers from the FDA, the National Cancer Institute, Baylor College of Medicine and elsewhere in September. According to the paper, it will comprise two sub-challenges: the first, scheduled to finish on November 4, will require participants to develop computational models that can distinguish between samples of matched and non-matched clinical and proteomics data. For the second sub-challenge, participants will be asked to develop algorithms that model relationships between clinical, proteomic, and RNA profiling data. They'll apply those models to identify instances of mislabeled data among the three data types. This portion of the challenge is expected to end on December 18. The expected outcome, according to the paper, are algorithms that can be incorporated into an analysis pipeline and used as part of a quality management system designed to reduce errors.
Meanwhile, some of the metrics used for the PrecisionFDA challenges have begun making their way into broader use. For example, researchers involved in the UK Biobank initiative specified performance on the PrecisionFDA's truth challenge data for demonstrating the quality of informatics pipelines. Also, in a preprint published on BioRxiv earlier this year, researchers from Illumina, Seven Bridges and University of California San Diego used data from the Consistency and Truth Challenges to assess the germline calling performance of their software.
In addition to the challenges, the FDA also sponsored a so-called app-a-thon in 2016 that encouraged researchers to contribute their NGS software to a library of apps. "One of the nice parts about PrecisionFDA is that datasets and pipelines aren't going away, so people can continue to collaborate and talk after the challenge is finished," Strain noted. Furthermore, "there may be academic researchers with good ideas that don't really have high-performance computing infrastructure to test their pipelines and PrecisionFDA gives them that ability."
Moving forward, the FDA plans to run additional challenges. Researchers can also expect new developments on the platform, although the organizers declined to disclose what these are. "We are always looking for ways to build and expand the community, with our ultimate goal of having a self-sustaining community that regularly interacts and engages with each other for the benefit of all participants," Johanson said.