A new comparison between desktop next-gen sequencers suggests that the performance gap between Illumina's MiSeq and Life Tech's Ion Torrent PGM is narrowing, while Roche's 454 GS Junior remains largely unchanged.
The analysis, published by a consortium of researchers in Germany and Austria in Nature Biotechnology this week, is an update to and extension of a similar comparison published a year ago by a UK group in the same journal (IS 4/24/2012). The new study takes into account several software and chemistry improvements the manufacturers have made to their platforms.
It follows several other performance reviews of next-gen sequencers that were published last year, including one by the Wellcome Trust Sanger Institute (IS 7/31/2012) and another by BGI (IS 8/14/2012).
Last year's UK study, which compared the three platforms by sequencing the same bacterial genome, had drawn criticism from Life Technologies and others because the researchers had used data generated by Illumina for the MiSeq platform, which was not commercially available at the time, while generating their own data for the PGM and GS Junior. Life Tech also claimed that the group did not apply the same read filtering and trimming standard to all three platforms (IS 5/15/2012).
To avoid this, the German and Austrian teams generated all data in their own laboratories, which were experienced with the respective platforms.
They sequenced the same E. coli strain − from a 2011 disease outbreak in Germany – as the UK team and used the same analysis scripts, software, and reference genome.
For their study, they tested the performance of several sequencing kits as they became commercially available − 2x150-base and 2x250-base paired-end kits for the MiSeq and 100-base, 200-base, 300-base and 400-base kits for the PGM, with the 400-base kits under early access at the time of the study.
They analyzed insertion and deletion errors, as well as substitution errors, both at the single-read level and at the consensus sequence level. The data for their study were generated between June of 2011 and November of 2012.
As reported by the UK group, at the read level, the MiSeq had the lowest indel error rate – about 0.0009 per 100 base pairs – while the GS Junior and the PGM with 100-base and 200-base kits had higher indel error rates – between 0.35 and 0.4 per 100 base pairs. Indel errors were highest for the PGM with 300-base or 400-base sequencing kits, around 0.7 per 100 base pairs.
The substitution error rate was the lowest for the PGM with 200-base kits – 0.03 per 100 base pairs – followed by the GS Junior, at 0.05 per 100 base pairs. It had similar levels for the MiSeq datasets and for the PGM data with 100-base, 300-base and 400-base kits – between 0.08 and 0.09 per 100 base pairs.
The researchers also generated de novo assemblies from the different data sets with the MIRA assembler and analyzed their contiguity and consensus accuracy.
To measure contiguity – the length of contigs and number of gaps and unsolved ambiguities – they examined about 4,700 coding genes in each assembly. The two MiSeq assemblies scored best, followed by the 400-base and 300-base PGM assemblies and the GS Junior assembly, while the 100-base and 200-base PGM assemblies came in last.
They also found that the GS Junior assembly was the least fragmented, resulting from its long reads, but the assemblies of the MiSeq 2x250-base and the PGM 300-base data had longer N50 sizes.
Next, they assessed the consensus accuracy of the de novo assemblies by comparing them to the reference genome and validated 98 of the discrepancies they found by Sanger sequencing.
The PGM 300-base assembly contained a single substitution error but 526 indel errors, while the MiSeq 2x250-base assembly had nine substitution errors and no indel error, and the GS Junior assembly had 40 substitution errors and 936 indel errors.
According to Dag Harmsen, a professor in the periodontology department at the University of Münster in Germany and the senior author of the study, it is consensus accuracy and contiguity that most researchers care about for practical applications, not single-read accuracy. "Read error is something of interest to developers of assemblers but not for typical end users," he told In Sequence.
At the read level, the substitution error rate "doesn't look extremely favorable for the MiSeq" in their study, but "you don't see them in the consensus anymore if you have enough coverage, and therefore, we don't regard it as very important," he said.
Overall, the MiSeq "made an extremely strong debut" in the comparison, he said, and is, "in certain aspects, still better than the PGM."
However, "the PGM has really rapidly evolved and is a strong competitor for the MiSeq," he added. "Except read length, it's now better in every aspect than the GS Junior," which only had a "minor software update" since the UK team's comparison.
In the UK-led study, "the PGM looked really bad," Harmsen said. "The gap is now much more narrow than before."
His own laboratory continues to use the PGM "for many applications" but also uses a colleague's MiSeq for some projects.
According to Nick Loman, a bioinformatician at the University of Birmingham and the lead author on last year's study, the overall results have remained the same, though. "The numbers have changed, but the relative ranking of the instruments is unchanged," he told In Sequence via e-mail.
The MiSeq still has the highest throughput per run and the PGM the highest throughput per hour, he said, while the GS Junior still provides the longest reads, although the other two platforms are catching up.
Also, the MiSeq continues to have the lowest overall error rate, and the PGM and GS Junior still have higher indel error rates than the MiSeq, owing to their flow-based chemistries. All three instruments have "similar rates" of substitution errors.
"It has become a two-horse race between the MiSeq and PGM, with no significant improvements in the 454 [GS] Junior specifications since our study," Loman said.
The comparison "certainly shows that any technical gap that the Ion platform had relative to MiSeq is essentially closed right now as we continue to push our accuracy up," Mike Lelivelt, director of bioinformatics and software products at Ion Torrent, told In Sequence. The SNP calling performance of the PGM "is actually better than on the MiSeq," he said.
The results "maintain that MiSeq generates the highest quality data when compared to competing platforms," said Jeremy Preston, Illumina's director of product marketing for systems and consumables, in an e-mail. The MiSeq had no indel errors in the consensus sequence, compared to "multiple errors" by both the GS Junior and the PGM, "reflecting the homopolymer error issue that both of these platforms suffer from,” he added.
Real-Time Performance Updates
While Harmsen's study remains up to date for the moment – no vendor has introduced major updates that are not reflected in the paper – it will "also be outdated quite shortly, no doubt," he said. Ion Torrent's Lelivelt, for example, pointed out that a new version of the Ion 316 and 318 chips, which the company plans to launch shortly, would have improved the results for the PGM 400-base kits.
Harmsen's paper went through four revisions and came out four months after the last data were generated, a significant lag regarding the rapid pace of next-gen sequencing technology development.
Like Loman's group, his team made all their read data and analysis scripts publicly available, "so the community can reproduce our results and extend them for future studies," he said.
In the meantime, Loman and a colleague from the University of Oslo, Lex Nederbragt, have built an online resource called "Seqbench" that aims to provide continuous updates on sequencing instrument performance.
Loman, who presented the open-source project at the Advances in Genome Biology and Technology meeting in February, said he plans to release the first set of benchmarks "very soon" and hopes it will help the community "to get the most reliable view of all the platforms as they evolve."