Last week, when AIDS researcher David Ho published a retraction in Science of some results from a proteomics experiment describing the discovery of a group of proteins with purported anti-HIV activity, he blamed the mistake squarely on sample contamination. Still, Ho’s retraction holds lessons for the proteomics world, observers and involved researchers say. In addition to highlighting the oft-noted need for thorough front-end sample preparation and back-end validation, Ho’s experiment is an example of how the unique sensitivity of mass spec provides both dangers and opportunities for researchers.
Ho’s original discovery, published in the November 2002 issue of Science, and conducted in collaboration with researchers from Ciphergen Biosystems, used Ciphergen’s ProteinChip and SELDI technology to search for the elusive factor that allows some HIV patients to remain healthy for years without ever progressing to AIDS. This factor, which Jay Levy of the University of California, San Francisco first described in 1986 and named CAF — for CD8 antiviral factor — has long been presumed to originate in CD8 immune cells. Following on this presumption, Ho ran preparations of stimulated and unstimulated CD8 cells from HIV progressors and non-progressors through the ProteinChip and SELDI platform and identified differentially expressed defensins — alpha defensins 1, 2, and 3 — which several groups had already described as having anti-viral properties.
Ho verified the identification with tandem mass spec and showed that when the defensins were removed, the anti-HIV activity observed was also removed. He thus concluded that defensins were a CAF.
The findings made a splash, and Ciphergen’s CEO Bill Rich has often held the paper up as the pinnacle of a successful application of the company’s technology (see PM 9-26-03).
But Ho stated in his retraction that the defensins come from neutrophils rather than from the originally postulated CD8 cells, and that the misinterpretation likely arose from his group’s failure to completely remove traces of the neutrophils — which he used to stimulate the CD8 cells. “[E]ven minor degrees of neutrophil contamination could result in the detection of alpha defensins in the culture supernatant of other cell populations,” Ho wrote in the letter.
Ho later verified that CD8 cells did not produce defensin mRNA. But he stuck to his findings that the defensins did have anti-HIV activity, a discovery that has only been previously described once — by a group of Japanese scientists in 1993 — and which was largely ignored at that time. “The anti-HIV-1 activity of alpha defensins we have described is not in doubt, and the mechanism of their antiviral effect should be pursued,” Ho concluded.
Ciphergen, for one, intends to pursue that mechanism. “Now that we know that defensins have this antiviral activity, we can look at the mechanism of how they work and ‘is it possible to design small molecule drugs that work the same way?’” said Rebecca Caffrey, business development manager at Ciphergen and an author on the original paper.
Caffrey said that Ciphergen has expanded its relationship with the New York-based Aaron Diamond AIDS Research Center, of which Ho is director and CEO, and is now working with researchers there to look for defensin receptors on CD4 cells and to understand the mechanism by which the active defensin proteins work.
A Lucky Break?
In Caffrey’s view, the sample contamination that Ho picked up was less an embarrassing error and more a lucky — and not uncommon — mistake that only Ciphergen’s technology was able to detect. “Everyone thought that this CAF was made by CD8 cells. We believe they were a contaminant derived from neutrophils all along,” she said. If that’s the case, the only reason they hadn’t been found before, she said, is because nobody had yet applied the right technology. Others in the past “looked at it with gene chips, they looked at it with gels, they looked at it with all kinds of protein technologies, including MALDI, and they didn’t find it. So the SELDI technology was so sensitive that we were the only ones that were able to see them.”
While other scientists disputed that SELDI was the key, there seemed to be a general consensus that mass spec was indeed the technology these defensins were waiting for. “Defensins are a dream for analysis by MALDI-TOF MS or SELDI-TOF; they are about 6 to 7 kDa and super basic,” Paul Tempst, who has worked on defensins, and who directs the Protein Center at Memorial Sloan-Kettering Cancer, Center wrote in an e-mail. “We have worked on similar antibacterial peptides in my lab and they behave so well in mass spec that we still use them as calibrants.”
Emanuel Petricoin of the NCI-FDA clinical proteomics program agreed that mass specs were well-suited to pick up the defensins. “These are all low molecular-weight entitities — there was no other tool — you couldn’t have run them on gels to see it,” he said. That did not mean, however, that Ciphergen’s tools were the only ones that could have done it. The researchers “could have done classical fractionation, and even perhaps have done ICAT labeling, and been able to do differential display,” he said. “So it’s wrong to say there’s anything special and magical that Ciphergen brought to the table on this.” In fact, Petricoin said, the low resolution of SELDI may have contributed to Ho’s misinterpretation, simply by limiting the amount of information he has at his disposal. “If he used an FT instrument, for example, perhaps they would have been able to drill down to even lower abundant proteins, and may have found even a further archive of molecules that were shed or secreted,” Petricoin said.
Of course, since Ho didn’t know what he was looking for, he didn’t know that the molecule that inhibited HIV would also produce beautiful peaks on a mass spectrum and would be present in his sample in such small quantities that only mass spec could find it. That took the kind of luck that only happens once in a great while. “I suppose the story of Ho’s ‘discovery’ that defensins have anti-HIV-1 activity is not unlike Fleming’s discovery that some fungi have antibacterial activity after he forgot to close the lids on his Petri dishes,” said Tempst.
But Bob Lehrer, who originally discovered alpha defensins in neutrophils in the mid-1980s, said that Ho’s discovery should not be chalked up only to chance. “There’s a wonderful aphorism of Pasteur: ‘Chance favors the prepared mind,’” said Lehrer, a professor of medicine at the University of California, Los Angeles. “The history of the discovery is that there is literature — very contentious literature — about CAF, and they looked for soluble factors in the CD8 T-cells that might inhibit HIV, and they did an unbiased search using some instrumentation, and they came up with defensins.”
CAF = Contamination?
For years there has been disagreement over whether CAF really exists. The disagreement continues.
Caffrey believes that Ciphergen and Ho’s group have essentially found CAF with their defensins discovery. “The defensins fit the definition of CAF,” Caffrey said, ”and they have anti-HIV activity against both subtypes of the virus. Coincidence? I don’t think so.”
Petricoin is not so sure. “I don’t think you can say that” defensins are CAF, he said. “There may be many other keys that are out there; there might be this elusive factor. It remains to be seen whether it’s really there. The hunt is on.”
Lehrer, though, also sees it as a matter of definitions. “If you define CAF as something that is made by and secreted by CD8 T-cells, clearly this is not CAF. The CAF is generally assumed to be one that is synthesized within CD8 cells,” he said. “He’s found something more subtle.”
Lehrer noted that Ho had ventured into a very hot topic that was not likely to go away with his discovery. “[T]he area of CAF is highly competitive with a couple of labs that are very vocal and very active in trying to identify this compound or group of compounds,” he said. “I think there’s a lot of heat in this area.”
Lessons Learned
Whatever the final results, Ho’s misinterpretation was based on a false assumption — that he had removed all of the neutrophils and their components before conducting the mass spec analysis. So what lessons should scientists doing proteomics take away from Ho’s experience? According to Jorge Leon — who heads a molecular diagnostics consulting firm, Leomics, in Princeton, NJ — not much of anything unusual. “This is a very difficult set of experiments that requires that you have very pure populations when you are selecting cells for a proteomics experiment,” Leon said.
But Petricoin said this incident “highlights the need for making sure that you have as rigorous as possible a system in place on the front end.” And, he said, “the [other] lesson learned is, Make sure you validate what you discover.’ … You’ve got to validate on large clinical sets and here, obviously if they had gone back and done the thorough validation beforehand, they may have discovered this.”
But Lehrer said the real lesson is that it’s not shameful to reinterpret your results. “I think the word ‘retraction’ is truly unfortunate,” he said. “He’s reinterpreting it and saying, ‘in our system, the CD8 T-cells imported defensins from other cells that actually made them.’ And if you think about that, how do we know that doesn’t happen in vivo?” In Lehrer’s eyes, nothing at all went wrong with Ho’s experiment. “The instrumentation did what it was designed to do — it found peptides. I don’t see a problem with the instrumentation, I don’t see a problem with the science. And personally, as someone who’s been doing experiments for 35 years, I don’t see anything wrong with reinterpreting data in light of better information.”
— KAM