NEW YORK – Researchers from Amgen's Decode Genetics have corrected a 2023 Nature paper comparing data from large-scale proteomics studies performed using Olink's Explore 3072 platform and Standard BioTools' SomaScan v4 platform.
While the original study found that the Olink measurements were more precise than those made on the SomaScan platform, the corrected version found the opposite, that the SomaScan measurements were more precise.
The error was due to a mistake by the authors in which a proportion of the samples they used to calculate coefficients of variation (CVs) for the SomaScan platform were not duplicate samples but were actually samples taken from the same individual at different time points.
The correction was issued on May 22, roughly eight months after the paper's original publication on Oct. 4 of last year. In the meantime, the incorrect CV data has provided Olink with ammunition in its competition against the SomaScan platform, said Stephen Williams, chief medical officer of Standard BioTools and former chief medical officer of SomaScan developer SomaLogic, which merged with Standard BioTools last year.
The episode is the latest development in the ongoing competition between the Olink and SomaScan platforms, which until recently have been the only feasible options for researchers interested in population-scale experiments measuring thousands of plasma proteins across tens of thousands of samples.
The two platforms use different affinity-based approaches. Olink features the company's proximity extension assay (PEA) technology, which uses pairs of antibodies linked to DNA strands that are brought into proximity when the antibodies bind and are then extended by a DNA polymerase, creating a new sequence that can be used as a surrogate marker for the target protein. SomaScan measures proteins using the company's Somamer reagents, a form of aptamer. Olink's current Explore HT platform measures more than 5,300 proteins, while the current version of SomaScan measures roughly 11,000 proteins. The Decode Nature paper used older, smaller versions of both platforms.
In the study, the scientists looked at data from 48,684 UK Biobank participants, comparing the protein quantitative trait loci (pQTLs) identified by Olink's Explore 3072 platform with those found in a study of 35,892 Icelandic individuals profiled with the more than 4,900 aptamer-based assays encompassed in the SomaScan v4 platform. They also compared the platforms' precision, using a metric called CV ratio — the ratio of an assay's CV to the overall variance for that assay in the cohort being studied. In the original study, they found that Olink performed with a median CV ratio of 0.35 and SomaScan with a median CV ratio of 0.50, indicating higher precision for the Olink platform.
In the corrected study, the researchers did not use CV ratio but instead calculated traditional CV figures and determined that Olink had a median CV of 16.5 percent versus 9.9 percent for SomaScan, meaning the SomaScan assay was more precise.
The error was caused by the researchers mistakenly calculating a portion of their CVs using SomaScan samples collected from the same individual at different time points instead of true replicate samples, and was "a very unfortunate mistake," said Daníel Guðbjartsson, Decode's VP of applied statistics.
Peter Ganz, professor of medicine at the University of California, San Francisco and a SomaScan user, also raised potential issues with the original use of CV ratio as opposed to a traditional CV figure.
He noted that use of the CV ratio metric had a logic to it in that "you can tolerate a larger CV [for a particular protein] if the variability of that protein in the population is large."
However, Ganz added, different populations have different levels of variability, and at a given level of precision, an assay's CV ratio will score better in a high-variation population versus a low-variation population. This makes it difficult to do cross-population CV ratio comparisons like that presented in the Decode group's Nature paper. Ganz suggested that the UKB cohort run on the Olink platform likely had higher variability than the Icelandic cohort run on the SomaScan platform, which would have skewed the CV ratio in Olink's favor.
Ganz also said he would expect the corrected SomaScan CV figures to be even better had the Decode researchers applied the commonly used data normalization steps that are recommended by Standard BioTools.
Guðbjartsson said that different levels of variability between the UKB and Iceland cohorts had only a slight effect on the CV ratios he and his colleagues presented in the original version of the paper, and that, in fact, this effect had favored the SomaScan platform as the Icelandic cohort actually had more variability than the UKB cohort. He attributed this to the fact that the UKB cohort was generally healthier than the Icelandic cohort. Ultimately, Guðbjartsson said, it was the mistaken use of samples from the same individual at different times that produced the incorrect conclusions regarding the platforms' relative precision.
Decode CEO Kári Stefánsson said he and his colleagues still believe CV ratio is a more informative measure for the work presented in the Nature study but said that in light of their mistake, they decided to present traditional CV scores in their correction.
"We felt really bad about the error," he said. "We felt that our right to be very firm on our opinion [on CV ratio] had gone away [due to] the mistake we made."
Standard BioTools' Williams said the erroneous CV ratios had given Olink a talking point in conversations with customers and suggested that following the correction his company plans to highlight SomaScan's advantage in precision.
In an email, Michael Gonzales, VP of global marketing at Olink, said that "while there was some revision of the CV findings," the Nature paper still shows that Olink's platform "delivers superior specificity" and added that this "is of paramount importance for obtaining biologically meaningful results."
Researchers have typically assessed the specificity of the Olink and SomaScan platforms (meaning whether they actually measure the proteins they claim to) by looking at the pQTLs — links between genetic variants and plasma protein levels — that they identify.
PQTLs have emerged as a major focus of research done on the Olink and SomaLogic platforms as such linkages can help researchers identify promising drug targets or patterns of protein expression linked to the development of disease, for instance. PQTLs are typically characterized as either cis — meaning that the pQTL is located close by the gene that encodes that protein — or trans, meaning it is located further away from the gene encoding the protein. While both are potentially meaningful, detection of a cis-pQTL provides strong indication that the platform's assay for that particular protein is specific — that it is, in fact, measuring the target it is meant to.
In the Nature study, 71 percent of Olink assays and 43 percent of SomaScan assays were linked to cis-pQTLs, indicating higher specificity for the Olink platform. Looking at only the 1,848 proteins measured by both platforms, 80 percent of Olink assays were associated with a cis-pQTL versus 58 percent of SomaScan assays.
A 2022 BioRxiv preprint published by the Decode team comparing largely the same datasets similarly found a higher percentage of cis-pQTLs associated with the Olink platform. That study drew criticism from SomaLogic and outside researchers, with both noting that had the researchers normalized the SomaLogic data according to the company's recommendations it would have improved cis-pQTL detection. The preprint was later withdrawn, though the authors did not attribute any research errors to the withdrawal but said instead that it had been "posted prematurely in advance of a UK Biobank Pharma Proteomics Project consortium effort."
Guðbjartsson said that Olink's platform may be more susceptible to epitope effects, which might also contribute to a higher rate of cis-pQTL association. Epitope effects occur when a genetic variant changes the binding of an affinity agent to its target protein. Such cases produce what appear to be a change in protein expression, but, in fact, this change is simply the result of altered affinity agent binding to that protein. Epitope effects can be mistaken for pQTLs. Guðbjartsson said that because the Olink platform uses two antibodies that bind to a larger stretch of target proteins than do SomaScan's aptamer-based reagents, it is more likely to bind to epitopes impacted by genetic variants and, therefore, is more likely to suffer from epitope effects.
"Just because of that, Olink is going to have somewhat more cis [pQTLs]," he said.
Stefánsson said that it is difficult to distinguish between cis-pQTLs caused by true changes in protein expression versus those caused by epitope effects. He added that in cases where a protein with a missense variant exhibits decreased expression, he and his colleagues assume that this is due to decreased affinity agent binding as opposed to a true decrease in expression.
A recent effort led by researchers at Weill Cornell Medicine-Qatar, Brigham and Women’s Hospital, and Harvard Medical School used mass spectrometry combined with Seer's Proteograph plasma protein enrichment platform to look at the impact of epitope effects on pQTL data generated by the Olink and SomaScan platforms. The study, detailed in a BioRxiv preprint, found that of the 200 "strongest associated cis-pQTLs previously identified using the SomaScan and the Olink platforms," roughly a third may be impacted by epitope effects.
Maik Pietzner, a bioinformatician at the MRC Epidemiology Unit at the University of Cambridge School of Clinical Medicine, said that beyond questions of the relative precision or specificity of the Olink and SomaScan platforms, there is a more general concern among many in the field about the lack of correlation between the two systems.
"What the field needs to move forward are some large-scale gold-standard measures to compare both affinity-based platforms against to make some headway," Pietzner said. "For example, even if precisely measured, it still sometimes remains unclear whether the correct protein target is measured, or only one out of many isoforms."
A recent MedRxiv preprint published by researchers at the University of Oxford compared data for 2,168 proteins measured by the Olink and SomaScan platform in 4,000 Chinese individuals, finding only a modest correlation between the two with a large proportion of proteins showing essentially no correlation. Correlations between the two platforms were stronger for higher-abundance proteins, suggesting that the discordance between them may in part be due to the challenge of reliably measuring low-abundance targets.
Stefánsson said this is likely another factor contributing to the higher percentage of Olink assays linked to cis-pQTLs. Because SomaScan measures more proteins overall, it measures more proteins in lower-abundance ranges where the platform may struggle. He added that he and his colleagues have similarly found that the proportion of Olink assays linked to cis-pQTLs declined as they moved from the 1,500-protein to 3,000-protein to 5,000-protein versions of the platform.
Stefánsson said, however, that despite the limitations of the Olink and SomaScan platforms, "they are giving the world an absolutely spectacular insight into the biology of disease."
"Just having the opportunity to screen this large number of proteins is truly amazing," he said, noting that this has given "us a lot of insight into the part of human diversity that is not directly related to the sequence of the genome. For this, these methods have turned out to be absolutely golden. But they have limitations, and it's extremely important to understand the limitations."