With criticism about the lack of proteomics success stories continuing to reverberate, and at least the perception that funding for proteomics is beginning to dry up, a team of researchers has established a campaign with the goal of restoring credibility to the field by pushing for greater reproducibility of experimental results.
The campaign, called Fixing Proteomics, was started a year ago by a small group of researchers and industry executives to recharge a field that had begun to buckle under the weight of what some consider unrealistic expectations and underwhelming results [See PM 11/15/07].
Its initial effort was to help a project initiated by the Industry Advisory Board of the Human Proteome Organization aimed at testing the reproducibility of 2D gel technology [See PM 11/15/07]. With that study finished and a paper being prepared on the results, Fixing Proteomics is turning its attention to a second phase of the study to include more laboratories, Paddy Lavery, a spokesman for the campaign, told ProteoMonitor last week. Lavery is also a spokesman for UK bioinformatics firm, Nonlinear Dynamics, which has several executives who are members of Fixing Proteomics.
The campaign also is in discussions with members of HUPO to do a similar reproducibility test for mass spectrometry-based techniques, piggybacking off work done by HUPO on protein standards, Lavery said.
The goal of the campaign is generally to promote proteomics and more specifically to push for better research by encouraging work that can be reproduced across different labs, a longstanding dilemma that has in no small part hobbled the field.
On its website, the campaign says that for any proteomics research, “reproducibility must be built into experimental design from the start or experiments are almost guaranteed to be irreproducible.” It adds that if a lab’s results cannot be reproduced, “then they should not be published.”
The need for Fixing Proteomics “is that proteomics is not perfect,” said one of its members, Kathryn Lilley, Facility Group Leader at the Cambridge Centre for Proteomics at the University of Cambridge. “There are plenty of people who have either struggled with the technology or even published data which is perhaps not as robust as it might be.”
The campaign results from conversations that executives of Nonlinear Dynamics had with researchers as it was developing new software for the proteomics market. While the company knew that there were problems in the field concerning experimental design and reproducibility — its software, after all, is directed at helping researchers improve both areas — it did not know the extent to which such issues was grinding on the researcher community.
“What got us started was this low morale,” Lavery said of the impetus behind Fixing Proteomics. “People were saying ‘I’m so down. I don’t know what to do about it.’
|
“Ten years ago, we all started out by promising lots of new disease targets and markers generated from these expression proteomics, and that has not yet materialized.” |
“We found that a lot of the core issues … for quantitative proteomics weren’t being addressed [and] saw evidence that funding was becoming harder to justify, and [there was] a general lack of faith,” he added. “And while we were talking to thought leaders and other people … there was a strong undercurrent of frustration around the same issues that we discovered.”
Indeed, the field has recently been engaged in existential self-reflection, asking itself what its purpose is and what value it is adding to scientific discovery. Some of the answers coming back from within the field have been unflinching.
At the 2007 annual HUPO conference, speakers said that much of the research has been riddled with sloppy science, bad protocols, and questionable data. While the field remains fertile with potential, the ultimate goal of taking proteomics to the clinical setting is still not within sight [See PM 10/11/07].
At this year’s conference, during discussions about HUPO’s efforts to jumpstart a 10-year, $1 billion proposal to do a first draft of the human proteome, some speakers said proteomics’ poor track record will be an obstacle to the plan, while funding agencies said such an effort would receive financial support only if the scientific benefits could be made obvious, suggesting that despite all the work that’s been done in proteomics, the results have had limited use [See PM 08/21/08].
In particular, pharma has taken a much more skeptical view of proteomics in recent years and where the industry once rushed to build up it proteomics research capabilities, it now has drastically scaled back such operations.
In an e-mail to ProteoMonitor Johannes Voshol, a member of Fixing Proteomics and a senior research investigator at Novartis Institutes for BioMedical Research, said European pharma has moved away from expression proteomics, “without a doubt because of the lack of success stories.”
But if proteomics has stumbled in fulfilling early promises it’s because people underestimated how hard the work is, he said.
“Ten years ago, we all started out by promising lots of new disease targets and markers generated from these expression proteomics, and that has not yet materialized,” Voshol said.
On its website, Fixing Proteomics acknowledges the skepticism faced by researchers in the field, saying that while proteomics is “an exciting field to be in … there is a growing sense of frustration that proteomics hasn’t delivered on its promise.”
And even as improved technology is providing greater insight into the proteins that make up biological systems and their functions, the challenges faced by the field haven’t disappeared, the group contends.
“They just become different — and often increase,” the campaign said.
Who Are These People?
The campaign has around 10 members, and it remains something of a mystery among proteomics researchers. During a recent thread on the discussion forum of the Association of Biomolecular Resource Facilities, some expressed concerns that the campaign might be just a marketing ruse for companies such as Nonlinear Dynamics, whose CEO, Will Dracup, is a founding member of Fixing Proteomics.
Other members include Lilley; Voshol; David Bramwell, technical director of Nonlinear Dynamics; Mark Baker, director of the Australian Proteome Analysis Facility; Mike Pisano, CEO of NextGen; Jan van Oostrum, head of business development for Zeptosens; Lennart Martens, group coordinator of the PRIDE database at the European Bioinformatics Institute; Mike Dunn, a professor at UCD Conway Institute of Biomolecular and Biomedical Research; and Jules Westbrook, postdoc fellow at UCD Conway Institute of Biomolecular and Biomedical Research.
Other posts on the ABRF thread took issue with its name, saying proteomics isn’t broken so doesn’t need to be fixed. And others threw their support behind the campaign’s goal of helping develop better protocols and standards.
According to Lilley the benefits of the campaign are two-fold: One is to heighten awareness of the issues that face all proteomics researchers. The other is that it acts as a resource for researchers who may not be “savvy with the best experimental designs or data analysis. …
“OK, you want to know more about [something], here’s a place to go to, so it’s almost like a sorting house for ideas and thoughts,” she said.
To that effect, it has created a four-step approach to designing an experiment to optimize reproducibility: Step 1, check the lab’s system to ensure the same results can be achieved time after time; Step 2, run pilot experiments; Step 3, confirm the results achieved from the pilot experiments; Step 4, reproduce across labs to ensure that the results are biological in nature and not influenced by specific protocols in the lab.
While Fixing Proteomics did not initiate the 2D gel test done by HUPO’s IAB, the campaign’s members helped analyze the data, which is available on Fixing Proteomics’ website. But as it moves forward, the campaign is taking a more active role in getting other projects started.
Fixing Proteomics is pushing for a follow-up to the 2D gel study by recruiting additional labs to participate, designing the study, and analyzing the data, Lavery said. The aim is to bring the total number of participating labs to about 20 to 30.
“One of the comments made about the [IAB] study was that it was by five labs that were all experts, and it was quite a niche group. That’s an important comment to address,” Lavery said. “Certainly, the first test showed where the major sources of variability were, and so that’s a good handle for the next group to say, ‘OK, we know what they are now. We’ve got ways to minimize and control that, which are fairly simple. Does that work? Does it allow us to reproduce on a larger scale?’”
Voshol of Novartis said that he hopes that in a year Fixing Proteomics will have created at least one 2D PAGE standard “with web-based analysis tools to point at potential issues with the methodology.”
The campaign is also in early discussions with HUPO executives about doing a reproducibility study with mass spec-based technology, though nothing has been decided yet, Lavery said.
Fixing Proteomics, however, is not looking to initiate projects that will be exclusively run by its members, Nonlinear’s Dracup said in an e-mail to ProteoMonitor.
“It would be duplicating effort; instead we can do what we do best,” he said. “Fixing Proteomics helps facilitate bringing the right people together to help make these projects a success and promote that.”
While part of the campaign’s goal is similar to what other organizations and individuals are attempting to do — HUPO and its Proteomic Standards Initiative, as well as ABRF, are deep in efforts creating standards — Lavery said that Fixing Proteomics may have more success in getting the message across.
“I think we’re trying to do it in a more generally accessible way and to spread the message to as many people as we can,” he said. “I think Fixing Proteomics is trying to distill it all down, or if you can only get your head around certain things, this is what you need to know.”