Despite the ability to generate millions to billions of bases of sequence data in a single run, a single next-generation sequencing instrument is not enough for many existing customers, as 40 percent of respondents to a recent In Sequence survey said they have two or three next-generation sequencers installed. In addition, 30 percent of survey participants said they currently run more than one instrument type.
One-third said they definitely plan to acquire additional next-gen sequencers in 2008, and almost a quarter more said they might do so if demand for the platforms grows at their institution, if their grant applications are successful, or if a new platform seems worth a try.
At the end of 2007, three next-generation sequencing platforms were fully commercialized: Roche/454’s Genome Sequencer FLX, which succeeded its first model, the Genome Sequencer 20; Illumina’s Genome Analyzer; and Applied Biosystems’s SOLiD sequencer.
These platforms, as well as others that are poised to enter the market, stand to win more business in 2008 as more research groups plan to adopt ultra high-throughput sequencing. But potential customers are debating which system will be right for them.
In order to get feedback from existing customers, In Sequence asked approximately 50 identified users of the three platforms to rate their experience with their chosen systems, describe their strengths and weaknesses, suggest improvements, cite challenges, and advise new customers (see sidebar below for details of how the survey was conducted). Seventeen users responded.
Users of 454’s Genome Sequencer had the most experience with their systems as the technology has been on the market for almost three years, the longest of the three platforms.
The 10 454 users who responded reported having their instruments between six months and more than two years, with an average of about 14 months.
The nine responding Genome Analyzer customers said they had their instruments between a month and a year, with an average of about seven months. Illumina, which acquired the instrument’s developer, Solexa, in January 2007, fully launched the system around the same time.
Four SOLiD users responded, saying they had their instruments between less than a month and two months. ABI fully commercialized the platform in early October 2007.
Asked what two features they considered most important when assessing a next-generation sequencing platform, 12 of the 17 respondents cited cost, which includes acquiring an instrument and generating sequence data, and 10 said data quality.
Read length and instrument reliability were each mentioned by three users. Trailing in importance were criteria such as throughput, ease of use, new applications, potential for improvements, and paired reads.
Asked what research projects the new sequencing platforms enabled them to perform, users mentioned a variety of applications, including de novo genome sequencing, gene expression profiling, ChIP sequencing, small RNA analysis, metagenomics, and resequencing.
Microbial genome sequencing was the application mentioned most often for the 454 platform, while resequencing and gene-expression analysis topped responses for Illumina’s instrument. No particular application was dominant among SOLiD users.
Users had many good things to say about their sequencers but they also pointed to weaknesses and areas that could be improved.
454’s GS FLX
454 users cited the system’s long read length of more than 200 bases most often as something they like about the platform. Two users also pointed out the excellent technical support from Roche.
Customers also mentioned a high base accuracy, short run time, the fact that base calling “is done in real time,” and reliability of the platform. According to one survey respondent, new plate configurations and kits for amplicon sequencing, paired-end sequencing, and bar coding have helped to “expand applications and flexibility.”
In terms of the instrument itself, one respondent said that “occasional hardware problems” at the beginning were “resolved promptly with minimal downtime,” while another stated that “the machine has performed better than specifications” from the start in terms of the number and length of sequence reads.
One customer stressed that the instrument is “easy to use” and presents “no problems in experienced groups,” but mentioned that upgrading from the GS20 to the FLX was a “disaster” because of mistakes in the protocols.
The greatest complaint about 454’s system, voiced by six of the 10 system users who responded, related to the high cost of sequencing reagents, “considering the output [the instrument] generates.”
“The capacity and throughput of the 454 FLX system (concerning mappable bases) is quite similar to the Solexa system, if one can afford to run it twice a day,” one user pointed out. “If run at maximum capacity, it consumes about €5,3 million [$7.8 million] per year and generates about 75 gigabases” of data, he added.
Half of 454 users complained about sample preparation. “The procedures for preparing samples are tedious and not amenable to any kind of automation,” one user noted, adding that “the labor required for operating this instrument is considerable.”
“Too much upfront manipulations – basically almost three days worth before loading” the instrument, added another user. “The handling is much more tricky than originally described by Roche and other users in some international meetings,” another customer said. One user mentioned his group is working in house on automating the sample prep process.
Users had a few more complaints and suggestions. “There are numerous ‘hidden’ parameters that control the way the instrument performs, and these should be made more accessible to the user,” one customer said. “The protocols still need some improvements,” another mentioned.
Software issues were rarely mentioned, but one user said 454’s assembly software has “difficulties with repeat regions” and proposed better software for “faster, more accurate assemblies and comparisons with known sequenced genomes.” Another user said his “number one need” for handling large projects is a multi-threaded assembler.
Just one user brought up homopolymer errors, in the past a commonly cited problem of the 454 platform.
Asked about improvements, several users requested longer reads, which Roche has promised to deliver later this year (see related story, this issue). Other users asked for lower consumable costs and higher sequence yields per run, which Roche has also said it will deliver this year.
Several Illumina customer respondents said that a benefit of the system is the large numbers of reads and amounts of sequence data the instrument generates. In addition, several characterized the platform as “simple to use” and “very easy to work with,” citing the “ease of sample prep.”
Respondents also said they liked the “broad range of applications,” the “cost efficiency,” speed, and “pretty stable output” of the instrument. The short reads are “good for re-sequencing applications,” according to one respondent.
“The system is a big step forward to reduce sequencing costs in cases where ultra-short reads are sufficient and helpful,” another user commented.
“Don’t expect next-gen technologies to be out-of-the-box solutions.”
However, six of the nine responding Illumina users had issues with software, complaining, for example, about a “lack of informatics to process the data” and problems with the “data analysis pipeline software.”
One user said that the data-analysis software interface is “not intuitive and lacks features,” and stated that “software for downstream analysis is not in place” and that “software packages to handle the data are necessary.”
Two users said they wish for longer reads, which Illumina in the past has said it plans to increase from 35 bases to 50 bases.
Two respondents said they received bad batches of consumables, but noted the company replaced them. Another customer said he had had trouble with “kit instability.”
Some customers had problems with the sequencer itself. One said the instrument runs hot, apparently affecting the sequence quality. “Having to modify the instrument environment to initiate cooling is a drawback,” he wrote in his response. “The instrument should have been better engineered in this respect.” Another respondent said he had “problems with focusing” the optics.
One customer, who was among the early-access users, said he initially had “several stability problems, resulting in off-time of more than 50 percent.” He also complained about the system’s base accuracy, which led him to discard data beyond 30 bases in most runs, and about problems with air bubbles in the cluster station.
Two users said they had experienced long delivery times: One complained that “the company must improve the delivery time of reagents” while the other said he did not like the instrument order backlog.
One respondent did not like that the “base calling is not done in real time” and pointed out the platform’s “big computing requirement.” Another respondent said he wished the instrument would run at higher speeds and have longer mate-pair reads.
Several SOLiD user respondents said they did not have enough experience with the system to rate its pros and cons. One said he liked the amount of sequence and cost per run, the “uniform read length,” the dual base-calling scheme that reduces the error rate, the “ease of mate pairing,” and the fact that part of a sequence run can be re-run if there is a problem.
However, he complained about the lack of speed and having to deal with “color space,” where two colors represent each DNA base, instead of reading base sequences directly.
Advice for All
What advice do next-gen sequencing users have for peers who are pondering which instrument to buy, and what are the greatest challenges they can expect?
Almost all survey respondents pointed out the considerable computational and bioinformatics needs that the new platforms require. “Anyone thinking of getting these instruments needs a strong IT/informatics group,” wrote one Illumina user.
“Our greatest challenge is the lack of bioinformatics support,” another said. Another advised: “Make sure you’ve budgeted adequately for compute infrastructure.”
“If your user community has bioinformatics resources and experience, then you will be fine, but if not, consider carefully the extent of this requirement and be sure you can recruit necessary help in this area,” said another Illumina GA user. “IT needs are also significant in terms of data storage, data transfer, and data analysis/processing requirements.”
“Invest in file servers, computer platforms, and computational biologists,” a 454 user said. “Also learn how to use real databases.” Another one said that the “post-instrument data analysis software” was one of the challenges his lab has faced with the sequencer.
An ABI SOLiD user said the greatest challenge for his group has been “data management, interrogation, and visualization.”
Secondly, users should make sure they know what their needs are before they choose a platform. “Decide exactly what you want to accomplish and then shop accordingly,” said one 454 user. For example, some people might want to do de novo sequencing, while others are more interested in comparative genomics, possibly resulting in different platform choices.
The 454 and Illumina systems “are designed for completely different purposes and the cost of operation, and consequently the price per base, is completely different,” commented one customer who owns both instruments.
“No current sequencer is perfect,” said an Illumina GA user. “Choose the one that fits your needs best and expect limitations.”
However, betting on one platform may not be the best solution. “One system might not be enough, as the combination [of several platforms] might offer better options for the project,” said a service provider, suggesting that it might be “more effective to use a service provider instead of buying a machine.”
“If possible, I would still wait a bit longer before ‘jumping in’ but if you must, due to competitive research pressures, then be sure to do extensive due diligence on all the available platforms as they are constantly being enhanced and upgraded,” advised one Illumina GA user.
“Competitors are beginning to place instruments, so be sure to investigate the alternative platforms,” the user added. “Most importantly, talk to people who have instruments in their lab and who have been running them for a while. There is nothing more valuable than unbiased, first-hand information from someone actually running one of these instruments.”
Sticker shock is something else new customers should consider. “The greatest challenge is the cost of the technology,” wrote a 454 user. “Even though the cost per base is cheap, $500,000 for the machine and $10,000 per run is intimidating to most researchers.”
Another 454 customer proposed potential buyers “get [used] to the price of a single experiment,” including the price of failure of such an experiment.
“The greatest challenge was to get the money” to purchase the instrument, agreed a commercial service provider and Illumina user.
Also, not all upfront costs are instrument-related. “Installing the 454 [platform] required massive rebuilding and reconstruction,” a 454 user wrote. Besides time, “you’ll need the money for this, in addition to the cost of the machine.”
Finally, customers must not expect too much of their platform. A 454 customer who provides services to researchers at his institution suggested that next-gen sequencing providers should learn “to manag[e] the expectations of our user base and educat[e] them to the strengths and weaknesses of the technology.” He added that “the technology sounds so simple that everyone expects their sample will be done in a week. … It won’t.”
In the words of an ABI SOLiD user, “Don’t expect next-gen technologies to be out-of-the-box solutions.”
How the Survey Was Done
In early December, In Sequence sent out an e-mail message to approximately 50 identified users of next-generation sequencing systems, asking them to answer five questions.
We asked about their next-gen platform, what users liked about their system, what problems they have seen, what could be improved, what advice they have for future customers, what research projects the platforms enabled them to do, what their two most-important criteria for assessing a next-gen sequencer are, and whether they plan to acquire additional instruments this year.
Seventeen users, who were promised anonymity, responded to the survey.
Owning a total of 26 next-generation sequencers between them, respondents include research groups and core facilities at nine universities and five independent research institutes, and three commercial sequencing providers.
Ten own a 454 FLX, nine bought an Illumina Genome Analyzer, and four chose an ABI SOLiD, with some owning more than one platform.
Eleven are located in the US, four in Europe, one in Asia, and one in Australia.
Six large-scale genome centers, which have the longest experience with the new platforms, did not respond to the survey.