A study by the Association of Biomolecular Resource Facilities' Proteomics Standards Research Group found that out of a variety of mass spectrometry-based protein-identification approaches, no single approach performed better than others at correctly identifying 49 human proteins in a mixture.
Though the laboratory that had the best results used an ion trap and a shotgun proteomics approach, researchers' skill probably affected the outcome of the study more than the kinds of methods or technologies used, regardless of whether it was a government, academic, pharmaceutical lab, or vendor.
"In this study, method was probably not as important as the skill of the operator in getting good results," said Jeff Kowalak, the chair of the sPRG who presented results of the study at last week's ABRF conference in Long Beach, Calif. "There was no separation approach that led to a higher number of identifications, and there was no indication that any type of mass spectrometry instrumentation did better than the others" in this study.
The idea to create a protein standard mixture originated in 2004 during a two-day workshop sponsored by the National Institutes of Health, the National Institute of Standards and Technology, and the Institute of Systems Biology, said Kowalak, who is also a staff scientist at the National Institute of Mental Health.
|
"Overall, we can conclude that good results are achievable by labs that don't have the latest instrumentation, and that many labs could reliably identify a large fraction of proteins with few false positive results." |
"There is no effective way to evaluate independent sequence library search algorithms without having a dataset that is highly defined, and the only way to generate a highly defined dataset is to have a highly defined sample," Kowalak told ProteoMonitor last September (see ProteoMonitor 9/16/2005). "There are no biological samples that you can purify, and once you do the constituents, the sample is highly well defined. So we decided to make one."
The sPRG joined with Sigma-Aldrich to produce the protein standard mixture, which was designed to emulate a "real world sample." Sigma ended up producing five-picomole aliquots of each of the 49 human proteins, which were either purified from their biological source or recombinantly expressed. The aliquots were combined into a mix that was sent out first to sPRG members, and then to all laboratories that participated in the sPRG study.
Sigma produced the sPRG aliquots for the ABRF study for free. The company plans to begin selling the protein standard mixture in the summer, according to a Sigma sales representative. The price for the product has not yet been disclosed, the representative said.
The sPRG sent out 129 protein-standard mixtures to laboratories. Seventy-eight laboratories, or 60.5 percent, returned results. Fifty-three of the participating laboratories were academic; 11 were government; seven were manufacturers or vendors; and seven were from the biotechnology or pharmaceutical industry. Most of those laboratories were in the US; a smaller number were in Europe; and a few were in Canada, Australia, Japan, and Argentina.
About 46 laboratories used an online 1D LC-MS/MS approach to analyze the protein mixture. Out of those, 33 did not use any protein separation prior to mass spec analysis, while 13 used 1D PAGE to separate out proteins prior to mass spec analysis.
Other methods of analysis included online 2D LC MS/MS, peptide mass fingerprinting TOF, off-line capillary reverse phase high performance liquid chromatography combined with MALDI TOF/TOF, PMF TOF/TOF, and off-line capillary HPLC combined with static nanospray.
A wide variety of mass spectrometers were used. About 23 laboratories used electrospray ionization with a quadrupole TOF; 13 labs used a tandem TOF; 13 labs used a linear ion trap; and 13 labs used a 3D ion trap. Other mass specs used include FTICR; linear ion trap-FTICR; hybrid quadrupole-linear ion trap; linear ion trap-Orbitrap; MALDI-linear ion trap; MALDI-quadrupole-TOF; and reflectron-equipped TOF.
A percent accuracy was calculated for each participating laboratory. The percentage was equal to the number of correct identifications divided by the number of correct identifications plus the number of incorrect identifications.
|
"There was no separation approach that led to a higher number of identifications, and there was no indication that any type of mass spectrometry instrumentation did better than the others." |
The laboratory that performed the best used an ion trap and a shotgun proteomics approach. However, not all laboratories that used ion traps and shotgun approaches performed well. Each method appeared to have a range of accuracies, and no one method appeared to perform better than the others. Overall, accuracies ranged from 5 percent to 100 percent.
"There was no separation approach that led to a higher number of identifications, and there was no indication that any type of mass spectrometry instrumentation did better than the others," said Kowalak.
One unexpected result was the apparent presence of 58 proteins that were not planned to be in the mixture. The 58 proteins were each described by two or more laboratories.
"Those 58 proteins were not intended to be there, but they clearly are," said Kowalak. "They're there as 'bonus proteins' or as contaminants."
Kowalak noted that the issue of "bonus proteins" would have to be dealt with in producing future protein standards.
A single lab identified 362 proteins, but "we believe that these constitute incorrect identifications," said Kowalak.
About 35 percent of participating labs identified more than 40 proteins. Those laboratories tended to have the fewest number of incorrect protein identifications. Laboratories that detected fewer correct proteins tended to identify more incorrect proteins.
Participating labs used a variety of search engines for protein identification, including Matrix Science's Mascot, Thermo Electron's Sequest, Ronald Beavis' X!Tandem, Waters' ProteinLynx Global Server, Agilent's Spectrum Mill, and Bioinformatics Solutions' PEAKS. There was no apparent correlation between search engines used and the number of proteins correctly identified.
The sPRG survey did not ask participating labs directly how much experience they had in identifying proteins. They did ask how much time it took them to prepare and analyze the samples. Most laboratories said it took two or three days to prepare and analyze samples, but answers ranged from one to four hours to more than a week.
Members of the sPRG did not compare time taken to prepare and analyze samples with performance, but Kowalak said such an analysis could be done at a later time.
"Overall, we can conclude that good results are achievable by labs that don't have the latest instrumentation, and that many labs could reliably identify a large fraction of proteins with few false positive results," said Kowalak.
Kowalak added that Sigma-Aldrich's standard protein mixture should have broad utility in the future for a wide variety of proteomic studies.
"I see this [standard protein mixture] being used by any lab that is new to proteomics that wants to be able to evaluate how it stands relative to its peers," said Kowalak. "It was my hope in the first place to provide people with an objective standard that they could use to evaluate their performance relative to others."
Results from the sPRG study can be seen on the sPRG website at www.abrf.org/sprg. A written report of the study will also be published later this year in the Journal of Biomolecular Techniques, which is the official journal of the ABRF.
— Tien-Shun Lee ([email protected])