Skip to main content
Back

Georgia Tech's SGMRI-VQA Catches AI Faking MRI Knowledge Without Pointing to Findings

A new 41,307-pair benchmark forces vision-language models to localize findings across MRI slices, exposing a large gap between describing anatomy and pointing to it.

  • SGMRI-VQA: a 41,307-pair benchmark for multi-frame, spatially grounded MRI reasoning across brain and knee studies. Paper
  • Built from fastMRI+ with radiologist bounding boxes, frame indices, and clinician-reviewed chain-of-thought traces.
  • Five hierarchical tasks per volume: detection, localization, counting, classification, captioning.
  • Zero-shot VLMs describe anatomy well but score near zero on volume-level pixel IoU.
  • Fine-tuned Qwen3-VL-8B hits 99.18% detection and 5.97 V-Score, beating GPT-4o and Gemini 2.5.
  • Benchmark, evaluation, and outputs open-sourced on GitHub.

Ask a modern vision-language model where a lesion sits in a brain MRI and it will confidently name the frontal lobe. Ask it to draw a box around that lesion on slice 14 of a 40-slice volume, and things fall apart. That gap between talking about anatomy and pointing to it is what a new benchmark called SGMRI-VQA, from researchers at Georgia Tech, Harvard, and Georgetown, was built to measure.

Spatially Grounded MRI Visual Question Answering (SGMRI-VQA) is a 41,307-pair benchmark for multi-frame, spatially grounded reasoning on volumetric MRI, built from expert radiologist annotations in the fastMRI+ dataset across brain and knee studies. Each QA pair includes a clinician-aligned chain-of-thought trace with frame-indexed bounding box coordinates. The team also fine-tuned Qwen3-VL-8B on it and showed that a modest amount of bounding-box supervision drastically improves how well an open model can localize pathology in a 3D scan.

Why single-slice benchmarks miss the point

Most medical VQA benchmarks treat radiology like a photo album: one image, one question, one answer. An MRI does not work that way. A single knee or brain scan produces dozens to hundreds of contiguous slices, and a finding like a meniscus tear or a small lesion might only appear on two or three of them. Prior grounded benchmarks such as GEMeX for chest X-rays or NOVA for brain MRI evaluate spatial grounding only on isolated 2D images, overlooking the volumetric nature of clinical imaging, where findings can span multiple frames or appear on only a few slices.

MedFrameQA nudged the field toward multi-image evaluation, but its inputs were 2 to 5 loosely related YouTube video frames rather than the tightly stacked slices of a real MRI volume. SGMRI-VQA is the first benchmark that asks a model to reason over a full sequential stack of 12 to 60 knee or brain MRI slices and produce both textual reasoning and bounding boxes tied to specific frame indices.

Six-panel figure showing detection, localization, diagnosis, classification, counting, and captioning tasks on MRI slices

Inside the dataset