RPTU Benchmark Pits 15 AI Tools Against Old Stats and Old Stats Win
A benchmark of 15 imputation methods across 30 datasets finds that older statistical approaches usually beat modern deep learning ones.
PRO- Benchmark of 15 scRNA-seq imputation methods across 30 datasets, 10 protocols, and 6 downstream tasks.
- Traditional methods (MAGIC, scImpute, WEDGE) outperform diffusion, GAN, GNN, and autoencoder approaches overall.
- Strong numerical recovery does not guarantee biological signal preservation in downstream analyses.
- In many datasets, imputation performs worse than simply using the masked baseline data.
- Task-specific winners: WEDGE for numerical recovery, scLRTC for clustering consistency, MAGIC for annotation.
- Full paper on arXiv; code available from authors on request.
A new large-scale benchmark takes fifteen popular tools for cleaning up single-cell RNA sequencing data, throws thirty datasets at them, and comes back with a result some deep learning enthusiasts will find uncomfortable: the older, simpler statistical methods generally win. The study from researchers at RPTU Kaiserslautern-Landau and DFKI systematically compares model-based, smoothing, low-rank matrix, diffusion, GAN, GNN, and autoencoder approaches across six downstream biological tasks.
The dropout problem, briefly
Single-cell RNA sequencing (scRNA-seq) measures gene expression one cell at a time, but the technology is noisy. Technical limitations lead to dropout events, where genes are observed as zero despite being expressed at low levels in the cell. These false zeros distort downstream analyses like clustering and trajectory inference, which is why the field has spent years building imputation methods that try to recover the true expression values.
Existing benchmarks evaluate only a limited subset of methods, datasets, and downstream analyses, and newer diffusion and GAN-based approaches were largely missing from prior comparisons. This paper closes that gap.
How the benchmark was built
The team assembled 15 representative methods spanning 7 methodological categories, covering both traditional approaches (model-based, smoothing-based, and low-rank matrix-based) and recent deep learning ones (diffusion, GAN, GNN, and autoencoder). Evaluation ran across 30 datasets (26 real and 4 simulated) sourced from 10 distinct protocols including 10x Chromium, SMART-seq2, CEL-seq2, Drop-seq, and Fluidigm C1.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
