Skip to main content
Premium Trial:

Request an Annual Quote

Guide to a Beta Life

Premium

Gary Siuzdak’s last experience beta testing a new product turned out to be a dream come true. Early last year the Scripps Research Institute proteomics researcher was looking for new biomarkers in plasma, and through a relationship with Agilent Technologies got ahold of the SpectrumMill mass spectrometry software and a front-end immunodepletion column at least six months before their general release. In previous experiments, Siuzdak had had trouble observing anything but the most abundant plasma proteins, but with the new products, “the results we were getting were infinitely better than what we had before,” he says. “As a result of being a beta-test site, we had a tremendous amount of success.”

Granted, Siuzdak might have generated data just as good had he waited until Agilent made the products generally available, but beta testing the new technology provided a shortcut — in terms of time, equipment costs, and access to Agilent engineering support. Many academic and industry scientists say that working with vendors to test and provide feedback on new products has become an important part of staying abreast of new technology, particularly in a time when technology for life sciences research advances so rapidly.

The stakes can be quite high. In many industry labs, where early access to superior research tools can translate into more useful data, faster, the right vendor relationships can contribute substantially to competitive advantage. There’s another perk: getting early access to technology can be what makes working in the lab fun.With the proper forethought — and a bit of luck — more and more researchers are realizing that early access programs and other mechanisms for helping shape the development of new products have become an increasingly necessary element of running a successful lab.

Of course, even for those labs that have established relationships with many vendors, there are bound to be some instances in which the product is underdeveloped, not related to a researcher’s work, or just plain shoddy. The experiences one can have when beta testing “run the whole gamut,” says John Hogenesch, who heads up a genomics lab at the Genomics Institute of the Novartis Research Foundation (GNF). In many cases, he adds, it makes the most sense to decline to beta test a product if it appears the benefits from early access don’t justify the time required to test it. Nevertheless, of the early access programs his lab does undertake, “the net product is positive,” Hogenesch says.

So how do researchers hoping to expand their repertoire of tools — or those just starting out in the field — go about developing the kinds of relationships necessary to gain preferential treatment from vendors? And once a vendor is willing to work with a lab, how does one choose between the various types of early access arrangements? In speaking with vendors and the researchers who often participate in their early access programs, Genome Technology discovered a number of useful pointers. Read on.

For any genomics-related technology, be it reagents for RNAi, gels for separating proteins, or database software, vendors offer a range of options for contributing to the product development process — or just getting early access to new technology. At the upper end of this range, approaching the limit of the most significant contribution on the part of the researcher, lies the relatively early stage process of alpha testing.

Alpha testing, says Andy Brooks, the director of the functional genomics center at the University of Rochester, typically involves the vendor providing financial support to the lab performing the studies that help establish the technology as a useful tool, such as identifying new applications or comparing its performance with existing technologies. Under this scenario, the lab sometimes receives product development funding from the vendor — or at the very least free or significantly discounted technology — and the vendor typically has dibs on the data for development and marketing purposes.

Hogenesch at GNF says alpha testing new products is a relatively rare occurrence in the 30-employee lab he runs because he tends to alpha test only those technologies related to his work that are completely new, rather than incremental improvements on existing products. “If it’s revolutionary, rather than evolutionary, then we would do it,” he says.

By contrast, beta testing is much more common. Under this scenario, a vendor typically has already completed much of the development work, and is instead looking to farm out the technology to relatively high-profile users as a means of working out the minor kinks and as a source of data for product promotion. At Spotfire, beta testing often involves sending out updated versions of software to a select group of established customers — or what David Butler, Spotfire’s vice president for product strategy and marketing, calls a “users’ group.” These customers get early releases of software “that they would get to play with, and use and apply, and maybe they would say, ‘Yeah, gee, this would be great if you could put feature XYZ into it,’” Butler says.

At the Wellcome Trust Sanger Institute, bioinformaticist Martin Widlake finds his group most often engaged in beta testing as a way of evaluating several hardware or software systems before deciding which to purchase. With the advantage of working at a high-profile institution involved in the Human Genome Project and the Cancer Genome Project, Widlake says his group beta tests on average about one new system a month, providing vendors with feedback on performance and other pros and cons of the respective hardware or software.

Private-sector researchers often have other options. In one example, industry (and a few academic) researchers actually pay the vendor for the privilege of participating in the product development process. In return for an investment on the order of $65,000 to $105,000, researchers can take part in Accelrys’ consortia, which include groups focused on high-throughput crystallography and functional proteomics. In return, participants receive early access to new software on the order of six months to a year prior to market release, and the opportunity to trade ideas with Accelrys, as well as among the members of the consortium. In addition, Accelrys late last year established a “science council” of large pharmaceutical companies and biotechs, a group that voluntarily provides ideas and input to help guide Accelrys’ future software development. “The intent is to influence our development, and give us a deeper understanding of the problems [our users] face,” says Accelrys CSO Scott Kahn. “We do see this as a win-win situation.”

When seeking out partners to beta test new technology, vendors typically look for well-established researchers who will have occasion to use the product often. A company beta testing a new product is looking for visibility, marketing opportunities, and feedback, says Steve Gullans, who serves as CSO for startup US Genomics while on sabbatical from his Harvard Medical School neurodegenerative diseases lab.

Gullans, who says his company is in the process of beta testing a new instrument, adds that the trick from the vendor’s perspective is to find a researcher with enough credibility and visibility to impress potential customers, who at the same time is diligent enough to provide adequate feedback to the vendor on product performance. Alternatively, Gullans adds, the vendor could rely on two researchers — one prestigious, one diligent. “Also, a fairly gregarious personality is helpful,” he says.

Researchers who quite often find themselves the target of vendor attention cited similar reasons for their popularity. Siuzdak at Scripps says his lab appeals to vendors looking for beta testing sites because it operates partly as a core facility for Scripps, and partly as a straightforward RO1-driven research lab. Operating a core facility allows Siuzdak to expose a product being beta tested to a wide variety of problems, and the fact that his group is also involved in its own research means his lab has a “higher competency in solving problems,” he says. “Companies are more interested in people working with a lot of different problems.”

Suizdak’s view is not universal, however. Rochester’s Andy Brooks, who directs AMDeC’s microarray resource center at the University of Rochester, acknowledges that, for example, a lab with a high-throughput microarray facility might appeal to a vendor looking to test out a new scanner, but he doesn’t think size is the only factor in the equation. “Not all companies are driven by the need to seek out a high-volume lab,” he says. “They’re mostly looking for specific expertise, so you don’t have to be a high-throughput, huge lab to engage in these kinds of endeavors.”

Researchers familiar with beta testing also say a crucial factor in deciding whether to participate in an early access agreement involves knowing what’s expected of you — and knowing what to expect from the vendor. Thoroughly vetting the amount of time and effort required to test the technology can help determine whether your lab has the time and money to participate in the early access program. “You need to understand the rules of engagement,” says Brooks.

As an example, Gullans cites a time in his lab at Harvard when a vendor became irritated that no one was testing a particular reagent. As it turned out, testing the reagent required an inordinately long protocol, and “nobody in the lab wanted to do it,” Gullans says. “If you suddenly open a kit and it says, ‘Here’s the next four days of your life,’ you’re not going to get through the first step.”

Similarly, old hands at beta testing say researchers should make sure they know what’s expected of them in terms of reporting the data they collect. Siuzdak says he typically expects only to provide regular feedback, or produce a poster to present at a meeting, but his experience working with Agilent was so positive that he agreed to present a talk at an Agilent users’ meeting. Gullans says he once agreed to take phone calls from potential customers asking about a product he had beta tested. (“Speak honestly — that’s the key,” he says.)

It can sometimes make sense to write a contract detailing the various obligations, adds Hogenesch at GNF in San Diego. A contract can also help protect the beta tester from getting locked into a relationship that discourages working with other technology vendors. “You don’t want to get stuck working on something that becomes obsolete,” says Hogenesch. However, if a contract is appropriate, make sure to have a lawyer or technology transfer office read through the legalese, says Brooks.

Perhaps the easiest and most obvious way to expand your access to new technology is by making an effort to cultivate relationships with vendors through their sales forces. “Nothing works like a ground-up collaboration,” says Hogenesch. Even in situations where large institutions have signed huge partnership deals with technology companies, the individual researchers have had to take the initiative to develop fruitful interactions at the grass-roots level, he adds.

Widlake at Sanger advocates participating in vendors’ user groups as another method of drumming up early access partnerships with vendors. His group’s relationship with Oracle began by participating in the company’s life sciences user group, which then led to Sanger’s beta testing of Oracle’s 10g database software. “It helps to be involved in the community,” Widlake says. “If you’re seen as willing to try things out, and you’re seen to be responsive to vendors,” these types of relationships follow, he adds.

Increasing one’s visibility to vendors can also take other forms, says Gullans. If a researcher makes an effort to attend industry conferences where small-scale industry or academic users are the minority, opportunities to cultivate technology relationships with vendors abound, he says. Getting quoted in prestigious science journals like Science and Nature certainly doesn’t hurt a researcher’s name recognition, Gullans adds. “The other thing was to have my postdocs badger [vendors] for free kits.”

In fact, many researchers say that once you begin beta testing for one company, others will soon come calling. Literally. As this reporter spoke with Hogenesch by telephone from New York, he briefly took a call on the other line. No surprise — it turned out to be yet another vendor.

   

The Scan

Fertility Fraud Found

Consumer genetic testing has uncovered cases of fertility fraud that are leading to lawsuits, according to USA Today.

Ties Between Vigorous Exercise, ALS in Genetically At-Risk People

Regular strenuous exercise could contribute to motor neuron disease development among those already at genetic risk, Sky News reports.

Test Warning

The Guardian writes that the US regulators have warned against using a rapid COVID-19 test that is a key part of mass testing in the UK.

Science Papers Examine Feedback Mechanism Affecting Xist, Continuous Health Monitoring for Precision Medicine

In Science this week: analysis of cis confinement of the X-inactive specific transcript, and more.