NEW YORK – A recent benchmarking study by researchers from Guangzhou National Laboratory in China and their collaborators evaluated and compared the performance of close to a dozen sequencing-based spatial transcriptomics technologies.
Described in a Nature Methods paper last month, the study aims to shed light on the performance of mainstream spatial transcriptomics methods, hoping to inform researchers of the pros and cons of each technology and to establish benchmarking criteria for the field.
“We have a lot of collaborators, and they always ask us which method we think might bring us pretty good results,” said Yue You, a postdoctoral researcher at Guangzhou National Lab and the first author of the paper. “We thought it might be a good chance for us to do a benchmarking study.”
The study examined 11 sequencing-based spatial transcriptomics technologies, which can be broadly grouped into four categories based on their spatial indexing strategies. They include microarray-based methods, the 10X Genomics Visium (v1 and v2) and DynaSpatial from Chinese company Infinity Scope Multi-Omics Biotechnology. Also included were high-definition spatial transcriptomics (HDST), a bead-based method developed by Joakim Lundeberg's group at Sweden's Science for Life Laboratory; BmkManu S1000 Spatial Transcriptome kit from China's Biomarker Technologies; Slide-seq v2, developed at the Broad Institute; Curio Seeker, a commercial product from Curio Bioscience that is based on Slide-seq; and Slide-tags, which was also invented by Broad researchers.
Additionally, the study evaluated polony- or nanoball-based technologies including Stereo-seq, developed by researchers at China's BGI Group; polony-indexed library-sequencing (Pixel-seq) from Liangcai Gu's lab at the University of Washington; and the spatial transcriptomics kit from Chinese company Salus. Lastly, the study included microfluidics-based DBiT-seq, which was developed by Rong Fan's lab at Yale University.
To assess the performance of these technologies, the researchers selected a reference tissue set with well-characterized morphological architecture that included adult mouse brain, embryonic mouse eye, and adult mouse olfactory bulb. “We put a lot of effort into having the tissue to be sequenced by different technologies to be quite similar,” You said.
After standard tissue preparation and sectioning, the authors applied the 11 methods to the tissue slides, generating data for comparing the platforms in terms of spatial resolution, molecule capture sensitivity, molecular diffusion, and marker gene detection.
Overall, the study showed that the spatial methods exhibited “really different” sensitivities, as indicated by their varying molecule capture efficiencies, You said. Her team assessed capture efficiency by examining sequencing depth, looking at either total sequence reads per unit area or normalized read count between samples.
This showed that none of the methods reached read saturation, indicating there is room for improved sensitivity across the board. While Stereo-seq and Slide-tag showed better capture efficiency at raw sequencing depths, the authors noted, Slide-seq v2 and DynaSpatial had better capture efficiency with normalized sequencing depths. Probed-based 10x Visium (v2) also performed well in both scenarios.
To further evaluate the technologies' sensitivity, You's team looked at marker genes known to be expressed in specific regions of the reference tissues. There, the team reported an “unexpected” and “systematic” gene capture bias in the polyA-based 10x Visium (v1) technology, which consistently missed marker genes detected by other technologies.
When looking at the lateral diffusion of molecules, which is linked to spatial accuracy of messenger RNA (mRNA) detection, the researchers found that tissue type exerted considerable influence on the diffusion profiles of the technologies.
For instance, although BGI's Stereo-seq had “pretty good sensitivity,” according to You, it showed notable lateral diffusion in the mouse olfactory bulb and brain tissue but not in the eye tissue, indicating tissue-specific bias.
Furthermore, the study noted that molecular diffusion “considerably impacts” the resolution of spatial methods. Although some technologies have subcellular spot sizes, the authors noted, they did not achieve that resolution due to limited sensitivity and high levels of diffusion. Meanwhile, Slide-tags was the only technology that achieved true single-cell resolution in this study.
“We are pleased the authors included our foundational technologies, Slide-seq v2 and Slide-tags, in their thorough and insightful article,” Neil Kennedy, chief commercial officer of Curio Bioscience, wrote in an email, adding that the researchers' benchmarking “align with” the company's own observations. According to him, Slide-seq v2 and Slide-tags underpin the company's Curio Seeker and Curio Trekker products, respectively, which are both commercially available in kit format with various configurations.
In an email, a spokesperson for BGI Group said the benchmarking study “provides a comprehensive evaluation” of some of the current mainstream spatial transcriptomic methods. She also noted that BGI researchers have further improved sensitivity and diffusion coefficient for the latest version of Stereo-seq.
BGI Group has established a new subsidiary named STOmics, which the spokesperson said has commercialized the Stereo-seq technology. BGI affiliate company MGI Tech became a distributor of Stereo-seq products in June, she added.
“Overall, we commend the authors' efforts,” a 10x spokesperson wrote in an email, but noted several limitations of the benchmarking study. For instance, it did not include the company's latest technology, Visium HD, which became commercially available in March, and did not highlight different technologies' FFPE compatibility.
Additionally, the spokesperson said it is “unclear” whether the study authors used optimal permeabilization conditions consistently across all methods and tissue types, potentially impacting the results of the comparisons. She also noted that later versions of 10x's technologies, such as Visium v2 and Visium HD, do not require permeabilization optimization, a strength of the company's platforms.
Furthermore, she said the study did not “adequately address” RNA integrity across samples and platforms, which might hinder a fair comparison.
As for the gene capturing bias from the company's polyA-based 10x Visium (v1) platform, she said 10x is taking this observation “seriously” and is “committed to further investigating this across various tissue types and taking steps to address and resolve it.”
You said her team commenced the study around two years ago and submitted the paper for publication in September 2023, prior to the commercial launch of Visium HD. However, her team has established an open database for this benchmarking effort and has been depositing more data as new technologies come out, including data generated by 10x Visium HD.
At the same time, You acknowledged that the study has several limitations. For one, she said the permeabilization conditions for the target mRNAs for the evaluated technologies might not be fully optimized, despite the team's best effort in this study.
In addition, You said the study did not evaluate metrics beyond technical performance such as cost, product availability, and simplicity of the workflow, which are all important aspects to consider when choosing a platform.
Despite these limitations, You said she hopes the study, and others to follow, can help inform researchers to make better decisions when deciding on a platform.
“All of those technologies will always say that they perform really well and give you a good result but is that true?” she said. “If you are a researcher who has not used any of the spatial technologies before, and you want to choose one, that can be quite difficult.”
“Sometimes the data looks good, but then the entire wet lab portion is a lot more complicated, then is it a better system?” said Joe Yeong Poh Sheng, a spatial immuno-pathologist at Singapore General Hospital who was not involved in the study.
Sheng's lab runs a slew of spatial platforms, including Stereo-seq and Visium HD. He said cross-platform benchmarking studies are important, but researchers also need to evaluate technologies based on the target tissue type and desired application.
“Bringing any type of benchmarking methods to the field is highly important at this point because it is just such a cutting-edge field,” said Jasmine Plummer, a spatial omics expert at St. Jude Children's Research Hospital who was not involved in the study.
Calling the study “a great starting place,” Plummer said the field as a whole still needs more samples of various tissue types to continue systematically evaluating technologies in this fast-evolving field.
“A field grows when people start benchmarking and establishing what we think we should move forward with,” Plummer said. “This shows that this whole spatial omics boom is here to stay.”