This story originally ran on Nov. 19 and has been updated.
By Tony Fong
While mass spectrometry platforms can consistently identify "inventories of proteins even in complex biological samples," whether this translates to clinical utility remains to be determined.
That is the conclusion from the National Cancer Institute's proteomics group in a study published this week in the Journal of Proteome Research evaluating the repeatability and reproducibility of LC-MS/MS systems.
The study is the third to be published by the NCI's Clinical Proteomic Technology Assessment for Cancer initiative recently and follows two studies last month on a yeast reference standard and 46 performance metrics for LC-MS [See PM 11/06/09]. The work described in all three studies is part of CPTAC's efforts to evaluate technology platforms for proteomics applicable for cancer.
The current study also harkens to one published by the Human Proteome Organization earlier this year in which it determined that mass spectrometry technology is robust and can achieve reproducible data [See PM 05/21/09].
In proteomics, reproducibility has long been one of the largest stumbling blocks holding back the field, and in clinical proteomics, in particular, questions surrounding the reproducibility of proteomics experiments is viewed as a major barrier in the adoption of the science in the clinical environment.
Neither the NCI CPTAC study this week nor HUPO's earlier study completely puts to rest questions concerning reproducibility in proteomics. But they do represent the first step in a systematic exploration of the issue and provide a foundation for continued research and improvement in that area.
Repeatability and Reproducibility
The CPTAC initiative was created in September 2006 as part of the NCI's Clinical Proteomic Technologies for Cancer initiative, which is aimed at addressing problems in proteomics as it applies to cancer-related research. As part of CPTAC, five teams of researchers share a five-year, $35.5 million grant to assess proteomics technologies.
Shortly after the five teams were named, they began a series of studies establishing reference standards and sources of variability. Those studies form the underpinning of the JPR article published this week as well as the two earlier studies.
In the current study, the CPTAC researchers looked at interlaboratory datasets to examine both repeatability and reproducibility in peptide and protein identifications. The data they analyzed spanned 144 LC-MS/MS experiments on Thermo Fisher Scientific mass spec platforms — four LTQs and four Orbitraps.
Those instruments were chosen because all of the participating labs had one or both instruments.
They defined repeatability as variation in repeated measurements on the same sample analyzed by the same instrument by the same researcher. Reproducibility was defined as "the variation observed for an analytical technique when operator, instrumentation, time, or location is changed."
For their work, the CPTAC researchers used as samples a set of 20 human proteins developed by the National Institute of Standards and Technology, which they called the NCI-20; Sigma Aldrich's UPS 1 mixture of 48 human proteins; and a protein extract of S. cerevisiae. In some instances, the yeast extracts were spiked either with bovine serum albumin or the Sigma Aldrich mixture.
According to the researchers, while some of their findings fell in line with "conventional wisdom," such as repeatability and reproducibility being higher for proteins than peptides, there were also some subtle, but important, lessons to be gleaned from their analysis.
For one thing, Orbitraps demonstrated better repeatability and reproducibility than the LTQs — for the yeast samples, Orbitraps achieved 9 percent to 18 percent greater peptide repeatability than LTQs — "but aberrant performance occasionally erased these gains," the researchers wrote.
[ pagebreak ]
Also, they found that, counter to what they had expected, sample complexity had no effect on peptide repeatability. On the NCI-20 proteins, there was a median overlap of 44 percent in the peptides identified between pairs of replicates. In the more complex Sigma Aldrich mixture, the overlap for peptides was nearly identical at 46 percent, and for the even more complex yeast sample, peptide overlap was 45 percent.
"Observing these consistent but low repeatability values conveys a key message: all digestions of protein mixtures are complex at the peptide level," the researchers said.
Similarly in yeast, differing concentration levels of the sample had no effect on repeatability. "Though larger numbers of identifications resulted from a higher sample load, repeatability for peptides and proteins was unchanged by the difference in concentration," the authors wrote.
Instead, they identify three factors that do appear to correlate with peptide repeatability — trypsin specificity, peptide ion intensity, and protein of origin.
In terms of trypsin specificity, the scientists write that "allowance for semi-tryptic matches (peptides that match trypsin specificity only at one terminus) has been shown to improve identification yield," but in their analyses, semi-tryptic peptides were less likely to appear in multiple replicates than fully tryptic peptides. In yeast, for example, 62 percent of semi-tryptic peptides appeared in only one replicate. By comparison, 45 percent of fully tryptic peptides appeared in only one replicate.
In the Sigma Aldrich sample, 65 percent of semi-tryptic peptides appeared in only one replicate, while 38 percent of fully tryptic peptides appeared in only one replicate.
Because "ion intensity drives selection for MS/MS" the researchers wrote, they expected that ion intensity also would correlate with peptide identification repeatability. And in an analysis of five replicates of the NCI-20 sample set, they found that for both LTQs and Orbitraps, "repeatability positively correlated with precursor intensity, with peptides identified in only one replicate yielding much lower intensities. We also observed that low intensity peptides were less reproducible across instruments."
They also said that, intuitively, a peptide that results from the digestion of a "major" protein, or a protein in which more than 32 distinct peptide sequences were observed, is more likely to repeat across replicates than peptides resulting from a "minor" protein, or a protein in which only one peptide was observed.
And in their analysis of the yeast sample, they found that 40 percent of all peptides appearing in six replicates matched to "major" proteins, while "almost none" of the peptides matching to a "minor" protein appeared in all six replicates. On the other hand, peptides matched to "major" proteins appeared in only one replicate just 18 percent of the time, the researchers said, while 13 percent of proteins matched to "minor" proteins were observed in only one replicate.
The researchers then moved on to try to address what factors may govern both peptide and protein reproducibility. In some yeast samples, data was generated following standard operating procedures while others did not follow any SOP. However, they found that reproducibility "observed with and without an SOP was unchanged," they wrote.
They add, though, that while median reproducibility was unaffected by following an SOP, "the range of reproducibility broadened when no SOP was employed" in the yeast sample.
Their results on reproducibility, they acknowledge, are limited and "an examination of retention time reproducibility or elution order might reveal considerably more detail about the reproducibility gains achieved through" the SOP that was used in their experiment.
Further, data generated from the experiment in which no SOP was employed by participating labs used "lab-specific protocols [that] may have been altered to incorporate some elements of the SOP, thus diminishing any observable effect."
In conclusion, they said, a "standardized analysis platform yields a high degree of repeatability and reproducibility (approximately 70-80 percent) in protein identifications, even from complex mixtures. Taken together with the accumulation of identifications with replicate analyses, this indicates that LC-MS/MS systems can generate consistent inventories of proteins even in complex biological samples."
What this means in terms of the systems' ability to detect differences in disease versus healthy states in samples is still unclear, however, the authors stressed.
"How this translates into consistent detection of proteomic differences between different phenotypes remains to be evaluated and will be the subject of future CPTAC studies," they wrote.