Skip to main content
Back

MBZUAI's Fetal-Gauge Exposes GPT-5 Failing Prenatal Ultrasound at 55%

A new benchmark of 42K fetal ultrasound images and 93K questions reveals the best vision-language model manages just 55% accuracy.

  • MBZUAI released Fetal-Gauge, the first VLM benchmark for fetal ultrasound: 42K images, 93K QA pairs.
  • Best model GPT-5 hits only 55% accuracy; most open VLMs perform near the 26% random-chance baseline.
  • Five tasks: plane ID, view conformity, fetal orientation, diagnosis, and bounding-box visual grounding.
  • LoRA fine-tuning boosts Llama-3.2-11B-Vision from 33% to 85%, outperforming every medical-specialized model tested.
  • Models fail on small structures and phantom images, exposing weak spatial reasoning and domain shift.
  • Benchmark and code available at github.com/BioMedIA-MBZUAI/FETAL-GAUGE.

Prenatal ultrasound is one of medicine's most demanding imaging tasks. The pictures are grainy, the anatomy moves, and interpretation depends heavily on the operator holding the probe. A team from MBZUAI set out to measure how well the current wave of vision-language models actually handles this modality, and the answer is: poorly. Their new benchmark, Fetal-Gauge, shows that even GPT-5 tops out at 55% accuracy, while most open models barely beat random guessing.

The gap nobody was measuring

Ultrasound can detect up to 85% of major fetal anomalies, and it remains the primary tool for monitoring fetal health across billions of pregnancies. Growing reliance on the modality has intensified the global shortage of trained sonographers, and producing more of them is slow and expensive. Vision-language models (VLMs) that can look at a scan and answer clinical questions are an obvious candidate to help, either as a decision-support tool or as an interactive training aid for novices.

No public fetal ultrasound VLM dataset exists, despite the modality being the workhorse of prenatal care worldwide. Existing medical VQA benchmarks focus on adult CT, MRI, X-ray, and histopathology, so nobody actually knew where frontier models stood on fetal scans. Fetal-Gauge fills that gap.

Inside the benchmark

The authors aggregated 13 publicly available fetal ultrasound datasets into a benchmark of 42,036 images and 93,451 question-answer pairs, all reformatted as multiple-choice questions to keep evaluation objective. Everything is structured around five clinical tasks a sonographer actually performs: