Skip to main content
Premium Trial:

Request an Annual Quote

ABRF Assesses NGS Platforms, Library Prep, Bioinformatics in Effort to Create RNA-seq Standards


This article was originally published Aug. 25.

NEW YORK (GenomeWeb) – The Association of Biomolecular Resource Facilities has published the results of its comprehensive assessment of commercially available next-gen sequencing technologies, library prep methods, and bioinformatics for transcriptome sequencing applications.

The ABRF-NGS study on RNA-seq evaluated five sequencing platforms across 15 laboratory sites using reference RNA standards to test four RNA-seq protocols. The goal of the study, the results of which were published online this week in Nature Biotechnology, was to "establish really high, even the beginning of clinical grade, RNA-seq standards … and serve as a resource for people who want to set up their own lab or test their own lab's techniques in RNA-seq and have something to compare to," Chris Mason, a senior author of the study and an assistant professor at Weill Cornell Medical College, told In Sequence.

He added that the study is not intended to be a "bake-off" between the sequencing platforms, but instead is an attempt to establish a reference data set for each platform.

Overall, the authors found high correlation among gene expression quantification both between labs testing the same platform and between different platforms. "This result suggests broader inter-study analyses and data mining can be successfully carried out across multiple platforms despite intrinsic differences between technologies, methods, and aligners," the authors wrote.

The results of the ABRF study were published in conjunction with several other related and independent RNA-seq studies in an issue of Nature Biotechnology that focused on transcriptome sequencing. Included in those studies was another large-scale cross-platform RNA-seq study that was led by researchers at the US Food and Drug Administration, constituting the results of the Sequencing Quality Control consortium for RNA-seq.

George Grills, director of operations at the core facilities in the life sciences at Cornell University and a senior author of the ABRF study, said that although the studies are very similar and looked at the same set of reference materials, there are still some key differences.

For one, the two studies evaluated slightly different platforms. Both looked at Illumina's HiSeq and Roche's 454, but the FDA study also evaluated the Thermo Fisher Life Technologies SOLiD system, while the ABRF study assessed the Ion Torrent PGM and Proton as well as Pacific Biosciences' RS system.

In addition, the two studies address different sets of questions, Grills said. Some of the defining characteristics of the ABRF study are the variety of conditions the group evaluated, "looking at upstream sample preps, the effect of size selection, sample degradation, and the effects of the different analysis algorithms on the results," he said.

Mason, who was a senior author of both studies, agreed that the main difference between the two was that there was no overlap of the sample prep methods or specific models of a system that were evaluated.

He added that the ABRF and FDA groups have now created a website that will host all the data from both studies.

The ABRF group used reference standards to test four types of RNA — poly-A–selected, ribo-depleted, size-selected, and degraded — on the HiSeq, Ion Proton, PGM, PacBio RS, and 454 GS FLX+ at 15 different laboratory sites. The groups tested RNA from a cancer cell line, pooled normal human brain tissues, and two different predefined mixtures of the samples. The samples also all contained synthetic RNA spike-ins from the External RNA Control Consortium.

While the results yielded high inter- and intra-platform concordance, they differed in their abilities to detect splice junctions, as well as in their cost and efficiency. "For the vast majority of global gene expression measurements, the platforms were fairly similar," Mason said.

One somewhat surprising finding was how well RNA-seq performed on degraded RNA, especially when a ribo depletion step was done, Mason said. "Even severely degraded or ancient RNA can recapitulate the expression profile of intact RNA."

Grillis added that "it was striking just how reliable data can be from degraded RNA."

To test the effect of degradation, the researchers used heat, sonication, or RNase-A on both the cancer line and normal brain tissue samples. Next, they ribo-depleted the samples and then prepared sequencing libraries to be sequenced on the HiSeq at multiple sites. Sequencing fully covered the genes, and similar to ribo-depleted libraries from intact RNA, more reads mapped to intronic areas of the genome. There was high correlation with intact RNA samples for detecting genes and differentially expressed genes, with a correlation coefficient of 0.96.

Another notable finding was the importance of longer reads for splice junction identification. "Despite lower read depths and higher costs, the longer read NGS technologies have the best ability to efficiently catch the vast majority of known splice junctions, indicating that they can be an effective means to annotate splicing complexity," the authors wrote.

Mason added that the longer read lengths also better enabled the reconstruction of full-length cDNAs. The PacBio also demonstrated the most even coverage through the entire length of the transcript, while the other systems had slightly lower coverage at the ends.

Looking at the QV scores for each system, most systems saw a drop in QV as the read length extended. The HiSeq had the highest QV scores, above 30, but read lengths were only 50 bp long. The largest gains in QV scores were seen when circular consensus reads were used on the PacBio RS.

Bioinformatics needs

One notable cause of variation was the specific bioinformatics tools used. For instance, the PGM and Proton had the highest mismatch rates, but they were greatly influenced by the bioinformatics that were used, ranging from an error rate of around 2.5 percent to greater than 10 percent.

Specifically, the group found that the aligners had an impact on mismatches. The group tested different aligners including vendor-recommended aligners and a universal platform-agnostic aligner. The platform-specific aligners produced "better mapping rates [and] gene-body coverage evenness" and correlated better with PCR quantification, with the exception of the HiSeq-specific algorithm, ELAND.

However, Mason noted that although some aligners produce higher mapping rates, sometimes they also have more errors. "You may get a lot more reads that map with one aligner, but also more errors," he said. "Some people have the impression that more mappable reads are better, but if they all map to the wrong place, that doesn't help."

DNA-seq standards

Mason said that the ABRF is now tackling a study to evaluate all commercially available NGS platforms on DNA. He said the study will likely not include the 454, since Roche has stopped production of that system, but all other platforms evaluated in the RNA-seq study will be included as well as some newer platforms like Illumina's HiSeq X Ten and Oxford Nanopore's MinIon. In addition, he said the group will be open to including additional sequencing technologies if they become available, such as the Proton II, Qiagen's Gene Reader, or Genia's system.

The ABRF is collaborating with the National Institute of Standards and Technology's Genome in a Bottle Consortium for this project, and it will include whole human genome sequencing, as well as whole-genome sequencing of smaller genomes like microbes or bacteria, sequencing of FFPE tissue, and sequencing to detect base modifications.

Grills added that in an ideal world, the group would start generating data for the first phase of this project toward the end of this year or beginning of next year, would be able to present some preliminary data at the annual Advances of Genome Biology and Technology meeting in February or at ABRF's annual meeting in March, and would be able to submit a publication by the end of next summer or early fall.