Eindhoven's RARE25 Exposes How Cancer AI Fails at Real-World 1% Prevalence
A public benchmark for detecting Barrett's neoplasia at realistic 1% prevalence exposed how AUROC hides the false-positive burden of surveillance CADe.
ProTakeaways
- RARE25 challenge benchmarks CADe systems for Barrett's neoplasia under realistic 1% clinical prevalence (paper).
- Public training set of 3,095 images plus locked test set of 25,000+ images across 12 medical centers.
- Eleven teams from seven countries submitted, all using fully supervised classification pipelines.
- Top team IMSY hit AUROC 0.920 but PPV@90%RECALL only 0.035 under bootstrapped prevalence-matched evaluation.
- No team submitted anomaly-detection or one-class approaches despite the extreme class imbalance.
- Dataset and evaluation framework released via Grand-Challenge.org under CC-BY-NC-SA.
Medical AI benchmarks have a dirty secret: most are built on artificially balanced datasets, and models that shine on them can drown clinicians in false alarms the moment they hit real surveillance data. The RARE25 challenge from Eindhoven University of Technology and Amsterdam UMC targets one painful case where this matters: catching early cancer in patients with Barrett's esophagus, where the actual prevalence of neoplasia sits below 1%.
The prevalence trap
Barrett's esophagus is a precancerous change in the lining of the esophagus, and patients enter lifelong endoscopic surveillance to catch neoplasia early. Caught in time, patients can be treated with minimally invasive endoscopic procedures with long-term success rates above 90%; if the disease progresses to advanced cancer, five-year survival drops to around 15%. Visible neoplasia, however, is exceptionally rare in surveillance imagery.
That rarity breaks conventional computer-aided detection (CADe) pipelines. Traditional supervised models often require artificially balanced training data that fails to reflect deployment conditions, limiting generalization. A model that looks great on a 50/50 split can produce so many false positives at 1% prevalence that clinicians stop trusting it, a phenomenon known as alarm fatigue.
How the challenge was built
The organizers released the largest public Barrett's dataset to date and locked a private test set that mirrors clinical reality. RARE25 pairs a publicly released training set with a hidden test set reflecting the incidence during real-world surveillance, and evaluates methods using operating-point-specific metrics that emphasize high sensitivity while explicitly accounting for prevalence.