MEMPHIS, Tenn. — While proteomics may have a tarnished reputation at detecting protein biomarkers of interest in clinical samples, clinical laboratories may also have inconsistent results, according to the results of a survey released this week at the annual conference of the Association of the Biomolecular Resource Facilities.
The study was conducted by ABRF's Proteomics Research Group, and the results were just one of several previewed by several of ABRF's research groups at the conference. In addition to the PRG, other proteomics related groups that previewed their survey results were the Proteome Informatics Research Group, the Proteomics Standards Research Group, the Protein Expression Research Group, and the Edman Sequencing Research Group.
The groups will be posting preliminary results from their studies on ABRF’s website during the next week.
For this year's survey, the PRG chose to tackle a growing request being fielded by proteomics laboratories: the targeted detection of a protein in a complex mixture, and the relative quantitation of that protein.
Immunoaffinity assays such as Western blots have been traditionally used for such purposes, according to PRG, but mass spectrometry-based alternatives are beginning to show up in the literature. As a result, PRG chose as this year’s study an evaluation of the different approaches used by the proteomics community to determine the relative abundance of target proteins of interest.
For its study, each participating laboratory was shipped six lyopholized plasma samples, each containing 40 microliters of plasma from female subjects. The samples were spiked with four proteins at varying concentrations: prostate-specific antigen; beta-chorionic gonadotropin; and two glycogen phosphorylase proteins that had been phosphorylated at different sites.
The samples were sent to 67 laboratories in pairings – there were a total of three pairs – with the laboratories blind to the groupings. Twenty-seven returned their data to PRG. Each lab was assigned three tasks: to provide the relative quantification of the four proteins; describe the methods it used to do so; and describe its experimental design.
Among the labs that participated, academic labs, at nine, made up the largest portion, followed by pharmaceutical/biotech/industrial labs, with six. Fifteen participants said they conduct both core and non-core functions and laboratory research; 43 percent of the participants said they do not offer services that are similar to what they had to do for the PRG survey; 38 percent said they do; and 19 percent said they don't but plan to.
A variety of sample-preparation methods were used by the participants, including ion exchange, and the use of RP-HPLC, and most prevalently the use of depletion columns.
Even more wide-ranging was the use of different mass-spec platforms. In total, 10 different mass-spec technologies were used: the most common platform used with eight lab saying they used it was electrospray ionization, quadrupole time-of-flight instruments. ESI, triple-quad with six labs was the next most-popular technology used. Seven different protein-acquisition methods were used and seven mass spec-quantitation techniques were used.
In one pairing of the samples, 11 labs achieved the correct results. In another pairing, 10 labs had the right results, and in the final pairing, a dozen labs were correct with their results, figures that Michael MacCoss, an assistant professor at the University of Washington and chair of PRG, described as "encouraging." Identification of PSA was especially challenging, MacCoss said during the group's presentation this week, adding that that may be attributed to researcher experience.
As a control, two sample sets of PSA and beta-HCG were sent to two clinical labs that use ELISA techniques to achieve their results. While the results from these labs were "largely in agreement" with PRG's "theoretical values calculated using quantitation value from amino acid analysis," they noted difficulty in analyzing PSA levels in the samples. One lab was unable to detect the biomarker at all, which MacCoss suggested may be a vindication to a degree of proteomics.
"Proteomics labs are failing miserably, but we have here a clinical lab that had trouble as well," he said.
Among the conclusions reached by MacCoss and his PRG colleagues is that quantitative proteomics is achievable since a handful of survey participants succeeded and reported excellent results. But doing so is complex and requires many factors for success, including expertise and experience.
MacCoss also said that the survey did not bear out any technique as being more successful than others since similar techniques carried out by different labs achieved different results.
[ pagebreak ]
Do-Over
Meanwhile, sPRG also took a quantitative approach to its survey this year, but had to delay it after deciding its original study "had too many challenges," according to the group.
Building off a 2007 survey that sought to develop a mixture of standard proteins that contained appropriate stable isotope-labeled peptides that "could be seen as a model for quantitative plasma proteomics," sPRG this year wanted to select 50 proteins from 350 that were to be evaluated. The 50 would be distributed over five orders of magnitude.
However, the results the group received in a pilot study were "well off" what they should have been, said James Farmar, an associate professor at the Albert Einstein College of Medicine and chair of sPRG. "We had to go back to the drawing board to figure out what was off," he said.
Indeed, during last year’s conference, when sPRG first announced its intent for this year’s study, some audience members expressed concern that the study could be too ambitious [See PM 02/14/08].
The study had to be redesigned to develop a simplified standard "based on human plasma [that] would be suitable for use in assessing a laboratory's capabilities for absolute quantitation analysis," the group said in a poster.
To do so, 10 candidate proteins were chosen from the original 350 that had been evaluated. The sPRG members digested and analyzed each protein, then made a list of prospective proteins from which corresponding unlabeled and stable isotope-labeled peptides were synthesized. For each protein, individual samples containing synthesized peptides, labeled and unlabeled, were analyzed.
Further digestion and analysis of the protein mixture was performed. "It was concluded from these experiments that better standardization was needed for the proteins and [stable isotope-labeled] peptides before a study sample could be prepared," sPRG said.
Sigma-Aldrich and a member of sPRG did so, optimizing a digesting protocol and achieving maximum sequence coverage and reproducibility, Farmar said. Seven members of the group analyzed the proteins in a so-called "Prototype 1" of the sample and determined "there was quite close agreement for the results obtained by a variety of scanning and analysis methods."
A "Prototype 2" sample will be sent later this month with results due in March, Farmar said. Results from its study are expected to be presented at the annual conference of the American Society for Mass Spectrometry in June, he added.
Dataset Differences
iPRG this year decided to conduct a study to determine different proteins between two complex samples in order to determine the differences between mass spectrometry datasets from biological samples — what it called "one of the major challenges for proteome informatics."
In a poster, the group added that "accurate and reproducible protein quantitation in complex samples in the face of biological and technical variability has long been a desired goal for proteomics."
iPRG set out to test how effective current protein-differentiation tools are for mass spec. Among their objectives was to uncover whether a spectrum-counting method is reliable, or whether intensity-based quantitative methods are required; and whether researchers can accurately determine the differences in proteins.
Each participant received datasets representing five technical replicates of each sample. In creating the samples iPRG loaded 25 micrograms of E.coli onto eight lanes of a 1D gel, and grouped them according to their concentrations. They were reduced, alkylated, and digested in-gel, then spiked with bovine serum albumin.
An answer key was created and then participants had to identify and rank proteins enriched in specific groups based on concentrations. The results were graded against the answer key.
[ pagebreak ]
Based on the results, iPRG said this week that both spectrum counting and intensity-based methods were successful at achieving correct results, "and there was no clear best methodology."
Median spectrum-counting users, however, did 30 percent to 60 percent better than the median intensity-based user at all error-rate levels, which members of iPRG said may be attributed to an inability of those who did intensity-based methods to use their "software to its fullest potential," suggesting room for growth in the area.
While experience is often cited as a factor in many study participants' ability to achieve the right results, in its test iPRG found that novices did "just as well or better" than more experienced researchers, Brian Searle, a staff scientist at Proteome Software and chair of iPRG, said during his presentation. However, intensity-based methods required more experience for correct results.
The group also noted that despite the success of the participants, further work is needed to show whether intensity-based or spectrum counting is capable of identifying protein changes in real-world studies as the proteins in the iPRG study had "much larger" differences than in real biomarker-discovery experiments.
The group has created a publicly available dataset to serve as a benchmark for researchers to use for testing, which will be available on ABRF's website next week, Searle said.
Edman vs. Mass Spec
ABRF's Edman Sequencing Research Group this week acknowledged the decline of Edman sequencing when it announced that the name of the group has been changed to the Protein Sequence Research Group, reflecting the alternative strategies that are supplanting Edman sequencing for protein sequencing research. Group members at the conference also put out the word for new members and opened up membership to the group to mass spectrometrists.
For this year, PSRG decided to test the ability of researchers to obtain N-terminal amino-acid sequence information from two samples. In a poster, the group said that though Edman sequencing has been the method of choice for determining the N-terminal amino acid sequence of proteins, one major limitation of the technology has been its inability to obtain amino-acid sequences from N-terminally blocked proteins.
Mass spec-based approaches effectively allow for such information to be obtained, but "unequivocal determination of protein N-termini on a routine basis has been elusive."
According to Wendy Sandoval, chair of PSRG and scientific manager at Genentech, one motivation for this year's study was the withdrawal of Applied Biosystems from the Edman sequencing space last summer [See PM 06/12/08], which left researchers in the US without a vendor for the instrument. However, Shimadzu last month said it is making its Edman sequencing instruments available in North America for the first time [See PM 01/29/09].
"We knew we have to look at alternatives" to Edman sequencing technology, Sandoval said during her presentation.
For the study, participants were allowed to use any technology of their choice to obtain as much N-terminal amino-acid sequence information from the two samples.
One sample was of E. coli transformed with a pTrcHis TOPO plasmid containing a cDNA encoding the yeast alcohol dehydrogenase protein. The second sample was glyceraldehydes-3-phosphate dehydrogenase from rabbit muscle.
Of the 30 participants in the study, half used Edman sequencing to get their results and half used in-source decay, digestions, and other techniques.
PSRG found that Edman sequencing remains a "reliable means" of determining the N-terminal sequence of an unblocked protein. Participants using the technology were able to "easily" sequence both proteins" but no one using the technique could identify the first of the two test samples because of its long vector sequence, the group said.
A top-down approach "shows great promise for determining the N- (and C-) terminal sequence of a protein in solution," though low mass ions were poorly resolved.
A bottom-up approach, PSRG said, works well a mass fingerprint can be matched to a protein in a database.
Those using mass spec-based approaches, it added, had to rely on protein identification from a database to fill in certain information, including amino acids with isobaric masses, regions with poor ion signal, and missing low-mass ions.
Express and Purify
Finally, PERG set out to study researchers' ability to express and purify proteins. Samples of a plasmid containing the gene for a poly-histidine tagged yeast alcohol dehygrogenase were provided along with instructions on how to express and purify the recombinant protein, though participants were free to follow their own procedures.
Four laboratories participated in the study. Because of the low participation rate and high number of variables, a statistical conclusion about the differences in protocols could not be obtained, Richard Heath of St. Jude Children's Hospital in Memphis and a PERG member said.
However some important facts could be seen from the returned data. For instance, despite being sent a protocol, each lab did the expression and purification differently, and each method succeeded in generating pure, active protein. So, for a protein that expresses well in E. coli, "satisfactory" expression can be achieved under a range of conditions as long as conditions hostile to the protein are avoided, the group said.
"This is good news for the core facility, looking to purify many different proteins from different sources using standardized conditions," PERG said in a poster. More sensitive proteins probably have narrower ranges of conditions suitable for active material, it added, "so a one-condition-fits-all approach is never going to be applicable to protein purification cores."