Skip to main content
Back

MMRareBench Exposes How Medical AI Fails Rare Disease Diagnosis

A new benchmark exposes how multimodal LLMs handle rare diseases, and why medical fine-tuning may hurt more than it helps on multi-image cases.

Pro

Paper, Full Access
  • MMRareBench is the first benchmark for rare-disease multimodal, multi-image clinical reasoning in MLLMs.
  • Contains 1,756 QA pairs and 7,958 images from PMC case reports, aligned to Orphanet ontology.
  • Four tracks: diagnosis, treatment planning, cross-image evidence alignment, examination suggestion.
  • Evaluated 23 MLLMs; treatment planning was universally poor across all models tested.
  • Medical-domain fine-tuned models trailed general MLLMs on multi-image tasks despite similar diagnostic scores.
  • Authors identify a capacity dilution effect where medical fine-tuning erodes compositional multi-image reasoning.

Rare diseases are the toughest terrain in medicine. Thousands exist, most clinicians have never seen a single case, and the diagnostic evidence is almost always scattered across photos, scans, histopathology slides, and lab reports. A new benchmark called MMRareBench is the first to systematically test whether multimodal large language models can handle that mess.

The results are unflattering. A systematic evaluation of 23 MLLMs reveals fragmented capability profiles and universally low treatment-planning performance, with medical-domain models trailing general-purpose MLLMs substantially on multi-image tracks despite competitive diagnostic scores. The models fine-tuned to be doctors are worse at reasoning across multiple images than the general-purpose ones.

Why existing medical benchmarks miss the point

Most medical vision-language benchmarks look at one image at a time and stick to common conditions where the model has likely already seen thousands of similar cases during pretraining. Rare-disease workups do not happen that way. Clinicians often lack prior clinical knowledge for these cases, forcing them to rely strictly on case-level evidence for judgments.

Prior benchmarks leave a wide gap. GMAI-MMBench provides comprehensive evaluation for general medical AI, MTBBench targets sequential clinical decision-making, and MedFrameQA explores multi-image VQA. None jointly evaluate multimodal and multi-image evidence integration under the rare-disease long tail. That is the space MMRareBench targets.