Life Technologies' Ion Torrent has submitted a letter to the editors of Nature Biotechnology claiming that a recent study published in the journal that compared three desktop sequencing platforms was not a fair comparison.
The study, led by bioinformatician Nick Loman from the Centre for Systems Biology at the University of Birmingham, compared the Ion Torrent PGM, Illumina MiSeq, and Roche GS Junior platforms. The comparison was based on sequence data for the Escherichia coli strain that was responsible for an outbreak in Europe in the summer of 2011 (IS 4/24/2012).
Maneesh Jain, Ion's vice president of business development and marketing, told In Sequence that the company identified two main problems with the study. First, he said, the sequencing runs on the MiSeq were done by Illumina at its Cambridge, UK, facilities, while the runs on the PGM and GS Junior were done by outside laboratories. Second, he said, the study did not control for different filtering and trimming of sequence reads between the platforms.
There "wasn't a uniform standard that was applied to trimming and filtering," he said.
Ion Torrent's letter to the journal, signed by Alan Williams, vice president of research and development at Life Technologies, outlines these issues and notes that "aggressive filtering and trimming of sequence reads" after data generation "can significantly distort the summary accuracy metrics."
Loman countered these claims, writing to In Sequence in an e-mail that at the time the study was performed, the MiSeq was not commercially available. "Of course there's a risk that using manufacturer data could introduce bias. We trust the readers to make a judgment about that," he said.
Additionally, he added, there is now MiSeq data available that has been generated by other labs, "which shows the performance is at least as good as reported in our paper."
With regard to Ion's issue with filtering and trimming, Loman said that the data was filtered and trimmed as recommended by each of the manufacturers and that these steps are detailed in the methods section of the paper.
Life Technologies has published the letter to Nature Biotech on its website, along with internal data for the same E. coli strain that it is has generated on the PGM using its latest chemistry, as well as newer chemistry that that it plans to launch in the second half of this year.
In the Nature Biotech study, Loman used the Ion 316 chip with 100 base pair reads, and sequenced and assembled the E. coli strain that was responsible for the German outbreak in June 2011, generating an assembly with 276 contigs and an N50 of 41,831 base pairs.
According to Ion Torrent, using new kits that it plans to launch this month, the company was able to reduce the number of contigs to 230 and increase N50 size to 85,110 base pairs.
Kits that the company plans to launch in the second half of the year, meantime, will enable the genome to be assembled in 192 contigs with an N50 of 109,940 base pairs, the company reported.
By comparison, the Loman study reported that the MiSeq E. coli assembly had 214 contigs and an N50 size of 81,730 base pairs. Illumina has said that it plans to launch improvements to the MiSeq platform by mid-year. According to Rob Tarbox, manager of product marketing at Illumina, the company does not plan to reanalyze the E. coli strain reported in the study.
Tarbox told In Sequence in an e-mail that the company has heard from multiple customers with both Ion Torrent and MiSeq instruments and they have said that "regarding relative performance, their own experience is consistent with the authors' assessment in the paper."
"Given that the feedback from our customers is aligned with the study authors, we do not feel the need to reanalyze the sample," he added.
The Ion PGM kit that will be launched this month will also improve the platform's indel and homopolymer accuracy, Jain told In Sequence. According to the company, the number of indels per 100 base pairs will be reduced from 1.44, which Loman achieved in July 2011, to 0.32 with the May 2012 kits.
Homopolymer accuracy will increase to around 99 percent for homopolymers five bases long, up from the 95 percent accuracy that Loman reported.
The May kits will also increase mean raw accuracy from 97.2 percent on the 316 chip for reads of 100 bases to 99.6 percent on the 318 chip with reads up to 250 bases, according to Ion Torrent.
Additionally, Ion Torrent is planning to launch a long-read chemistry kit in the second half of the year that will increase read lengths to 300 bases, which it says will further improve the platform's accuracy and also help with de novo assembly.
Williams notes in his letter to Nature Biotech that Ion Torrent hopes the scientific community "will consider these potential sources of bias and the current post-light sequencing data sets when drawing conclusions" from the Loman study.
Some researchers have already weighed in with their thoughts. For example, several participants in a discussion thread on the Ion Torrent Community site noted that the technology described in the paper was outdated and asked the company to provide data demonstrating the performance of the current version.
Meanwhile, Lex Nederbragt, a research fellow at the University of Oslo's Norwegian Sequencing Center, wrote on his blog that Ion's criticism of the paper is "not a fair critique" because Loman had no choice but to use the technology that was available at the time.
Nederbragt acknowledged that any peer-reviewed comparison would end up producing "old" data, but argued that such studies are necessary to give researchers an impartial picture of the platforms' overall performance.
In addition, he noted that many researchers are likely using the same generation of kits as those used in the Loman paper, and they will "absolutely want to know about the different error types and accuracy levels" for those products.
"At the very least," he wrote, "the data presented in the paper give an overview of the relative performance at the time of studying, which might reflect on today's performance."