Skip to main content
Premium Trial:

Request an Annual Quote

FDA SEQC Study Shows RNA-seq is Reproducible for Relative Expression, Can ID New Splice Junctions


NEW YORK (GenomeWeb) – RNA sequencing is able to accurately measure relative expression levels, irrespective of sequencing platform and user site, but no gene expression technology provides accurate absolute measurements, according to a new study by the SEQC/MAQC-III consortium.

The study also found that RNA-seq is a powerful tool for the discovery of novel splice junctions, though their biological significance remains to be elucidated.

The goal of the Sequencing Quality Control project, coordinated by the US Food and Drug Administration with a large number of participants from academia and industry, was to assess the performance of RNA-seq across laboratories and to test different sequencing platforms and data analysis pipelines.

The results were published online in Nature Biotechnology this week. A number of related RNA-seq studies, including a multi-platform assessment by the Association of Biomolecular Resource Facilities, appeared in the same issue of the journal.

The SEQC project, formally launched in 2011, follows in the footsteps of two earlier projects, MicroArray Quality Control or MAQC projects I and II, which tested the performance of gene expression microarrays across sites and platforms, as well as sources of bias.

For the study, the consortium sequenced two well-characterized RNA samples, Universal Human Reference RNA and Human Brain Reference RNA, which contained spike-ins of two synthetic RNA samples from the External RNA Control Consortium (ERCC). They also sequenced two combinations of the RNA samples, mixed in 3:1 and 1:3 ratios. To assess dynamic range, the ERCC samples were also sequenced separately.

Six official sites, designated by the platform vendors − at BGI, Cornell University, and the Mayo Clinic for the Illumina HiSeq 2000; and at Northwestern University, Penn State University, and GE Healthcare's SeqWright for the Life Tech SOLiD 5500 − profiled the six samples in replicates of four or two. In addition, data generated on either platform by four unofficial sites were incorporated in some analyses. To assess gene models, three separate sites sequenced the two well-studied RNA samples using Roche's 454 GS FLX platform.

In total, participants sequenced close to 200 RNA-seq libraries for the study, generating more than 100 billion reads, or 10 terabytes of data. They compared the results to those from microarrays and quantitative PCR.

The majority of reagents for the study were donated by the platform vendors, and the FDA shouldered some of the costs, according to Chris Mason, a senior author of the study whose lab at Weill Cornell Medical College was one of the official sites for HiSeq 2000 sequencing.

He said the aim of the study was to "create really high standards for RNA-seq" and to provide physical standards, protocols, and data for RNA-seq users and developers of bioinformatics tools.

One of the key results of the study is that relative or differential expression levels are reproducible across platforms and sites, provided the right filters are used.

According to David Kreil, chair of the bioinformatics research group at the University of Natural Resources and Life Sciences in Vienna and co-senior author of the study, two filters are needed. "When we exclude genes that are weakly expressed and we exclude genes that change only very little, then we get differentially expressed genes that are highly reproducible across sites that also agree well across platforms," he told In Sequence.

However, it is probably not a good idea to mix data from different platforms if it can be avoided. "You can compare them if you need to, but it's still good experimental design to keep things on a single platform," Mason said.

Another important finding is that there is no gold standard for absolute expression levels, including quantitative PCR, and that absolute expression cannot be compared between platforms. "At the beginning, there was the hope that qPCR would be right and everything else can be compared to that, but it turns out that when we take two different qPCR protocols, PrimePCR and TaqMan, the variation between those is just as high as the variation between qPCR and arrays, and qPCR and RNA-seq, and RNA-seq and arrays," Kreil said. "You see differences, and you don't know who is right."

Different technologies and platforms also showed gene-specific biases. "Effectively, every platform has a blind spot," Mason said, so validating results with orthogonal methods will be important.

Illumina's HiSeq 2000 and Life Tech's SOLiD 5500, for example, provided different results for some genes, "but they are similarly different between qPCR and RNA-seq, or between two different qPCR methods, so we cannot per se say one is more accurate than the other," Kreil said.

Compared to arrays, RNA-seq appears to do better in some areas but not others. "At the gene level, if you go deep enough, say to 50 or 100 million reads, RNA-seq, with the right filters, can be more specific at the same sensitivity level than typical microarrays, or more sensitive at the same specificity level," Kreil said.

"But when you go to alternative transcripts, which people thought might be a strength of RNA-seq, it actually turns out that because of the noise that you get when you have few reads that cover a particular junction, RNA-seq really has trouble getting precise results," he said. "So there, the arrays are doing better. That came as a surprise."

The researchers also assessed the ability of RNA-seq to discover novel splice junctions – a capability that sets the technology apart from arrays – and found that no matter how deeply they sequenced, even up to 12 billion reads, they still found new junctions. They proceeded to validate 173 of those junctions by qPCR, including some that were supported by many reads and others that were supported by few reads, and were able to validate more than 80 percent of them. "So there is clearly a large set of new junctions there that is really in the sample," Kreil said. "Of course the question is, do they do anything? Are they biologically relevant? And no one knows. This is, we think, a really interesting puzzle."

One way researchers could use the SEQC data is to estimate how deeply they need to sequence in their own RNA-seq experiments. "If the subset of their genes is highly expressed, they probably don't need to generate more than 5 to 10 million reads per sample. But if they're looking for rare isoforms or lowly expressed genes, they do need to sequence more deeply," Mason said.

Researchers could also use the ERCC spike-ins in their own experiments. "They are a good sanity check, so a good part of the experimental design," he said.

Another significant result for users is the importance of the analysis tools. "Bioinformatics is a key differentiator," Mason said. "Depending on what tools you use, you will need more reads versus less reads to get the same result."

According to Mason, the SEQC study is complementary to the ABRF study, which he also helped coordinate. That study sequenced the same RNA samples but tested additional platforms − the Ion Torrent PGM, Proton, and the Pacific Biosciences RS − as well as different sample prep protocols.

In November, participants of the SEQC study are scheduled to meet at Fudan University in Shanghai to further discuss the results, as well as potential follow-up studies. According to Mason, new studies will likely focus on personalized medicine.

The Scan

Billions for Antivirals

The US is putting $3.2 billion toward a program to develop antivirals to treat COVID-19 in its early stages, the Wall Street Journal reports.

NFT of the Web

Tim Berners-Lee, who developed the World Wide Web, is auctioning its original source code as a non-fungible token, Reuters reports.

23andMe on the Nasdaq

23andMe's shares rose more than 20 percent following its merger with a special purpose acquisition company, as GenomeWeb has reported.

Science Papers Present GWAS of Brain Structure, System for Controlled Gene Transfer

In Science this week: genome-wide association study ties variants to white matter stricture in the brain, and more.