Skip to main content
Back

CXR-LT Challenge Exposes How Chest X-ray AI Collapses on Rare Diseases

A radiologist-annotated, multi-center chest X-ray benchmark exposes how far current AI still is from reliably spotting rare thoracic diseases.

Paper, Full Access
  • CXR-LT 2026 combines PadChest and NIH ChestX-ray into a 145,000-image multi-center benchmark with radiologist-annotated test sets.
  • Two tasks: 30-class long-tailed multi-label classification and zero-shot recognition of 6 unseen rare diseases.
  • Top mAP scores: 0.5854 on Task 1 and 0.4315 on Task 2, showing large gaps for open-world generalization.
  • Winning solutions used ConvNeXtV2 with Distribution-Balanced loss and WhyXrayCLIP for zero-shot prompt matching.
  • Head-to-tail analysis shows sharp performance drops on rare and very rare disease categories across all teams.
  • Full overview available on arXiv with leaderboards hosted on Codabench.

Chest X-ray classifiers look great on paper until you show them the diseases they rarely saw during training. That gap between polished benchmarks and messy clinical reality is what the third edition of the CXR-LT challenge tries to measure, and the results are a useful reality check on where vision-language models actually help and where they still stumble.

Why single-hospital leaderboards keep lying to us

Chest X-ray interpretation is hindered by the long-tailed distribution of pathologies and the open-world nature of clinical environments. Existing benchmarks often rely on closed-set classes from single institutions, failing to capture the prevalence of rare diseases or the appearance of novel findings. In plain terms, most public CXR benchmarks train and evaluate on the same hospital, with labels scraped from radiology reports, so models learn to be great at frequent findings from one imaging protocol and mediocre at everything else.

The CXR-LT 2026 challenge, run alongside ISBI 2026, was built to break that pattern. It combines two datasets, introduces cross-institutional test data, and replaces noisy report-derived labels with radiologist annotations on the evaluation sets.

What is actually in the benchmark

This third iteration introduces a multi-center dataset comprising over 145,000 images from PadChest and NIH Chest X-ray datasets, with two core tasks: Robust Multi-Label Classification on 30 known classes and Open-World Generalization to 6 unseen out-of-distribution rare disease classes. Training uses 142,928 PadChest images with weak labels parsed from Spanish radiology reports. Development and test sets are drawn from PadChest-GR, where multiple radiologists annotated each image, and an external NIH subset was added where each candidate was reviewed by three radiologists under a majority-agreement rule.