MMRareBench Exposes How Medical AI Fails Rare Disease Diagnosis
A new benchmark exposes how multimodal LLMs handle rare diseases, and why medical fine-tuning may hurt more than it helps on multi-image cases.
ProTakeaways
- MMRareBench is the first benchmark for rare-disease multimodal, multi-image clinical reasoning in MLLMs.
- Contains 1,756 QA pairs and 7,958 images from PMC case reports, aligned to Orphanet ontology.
- Four tracks: diagnosis, treatment planning, cross-image evidence alignment, examination suggestion.
- Evaluated 23 MLLMs; treatment planning was universally poor across all models tested.
- Medical-domain fine-tuned models trailed general MLLMs on multi-image tasks despite similar diagnostic scores.
- Authors identify a capacity dilution effect where medical fine-tuning erodes compositional multi-image reasoning.
Rare diseases are the toughest terrain in medicine. Thousands exist, most clinicians have never seen a single case, and the diagnostic evidence is almost always scattered across photos, scans, histopathology slides, and lab reports. A new benchmark called MMRareBench is the first to systematically test whether multimodal large language models can handle that mess.
The results are unflattering. A systematic evaluation of 23 MLLMs reveals fragmented capability profiles and universally low treatment-planning performance, with medical-domain models trailing general-purpose MLLMs substantially on multi-image tracks despite competitive diagnostic scores. The models fine-tuned to be doctors are worse at reasoning across multiple images than the general-purpose ones.
Why existing medical benchmarks miss the point
Most medical vision-language benchmarks look at one image at a time and stick to common conditions where the model has likely already seen thousands of similar cases during pretraining. Rare-disease workups do not happen that way. Clinicians often lack prior clinical knowledge for these cases, forcing them to rely strictly on case-level evidence for judgments.
Prior benchmarks leave a wide gap. GMAI-MMBench provides comprehensive evaluation for general medical AI, MTBBench targets sequential clinical decision-making, and MedFrameQA explores multi-image VQA. None jointly evaluate multimodal and multi-image evidence integration under the rare-disease long tail. That is the space MMRareBench targets.