Skip to main content
Back

GazeVaLM Catches What AI Misses When Spotting Fake Medical X-Rays

A new benchmark tracks how 16 radiologists visually inspect real versus AI-generated chest X-rays, and pits their perception against six multimodal LLMs.

Pro

  • GazeVaLM releases 960 gaze recordings from 16 radiologists on 30 real and 30 RoentGen-generated chest X-rays.
  • Two tasks are captured: naive diagnostic reading and an explicit Visual Turing Test with a 10-day washout.
  • Radiologists hit 80.1% real-vs-fake accuracy; best LLM (GPT 5.2 Thinking) reaches only 71.7%.
  • All six evaluated multimodal LLMs show systematic overconfidence, with DeepSeek 3.2 performing at chance.
  • Pupils constrict more on synthetic images even when observers do not know fakes are present.
  • Release includes raw gaze samples, saliency and fixation maps, scanpaths, and structured LLM JSON outputs on arXiv.

Synthetic medical images increasingly train clinical AI, yet the way we judge whether they look real has been stuck on distributional metrics like FID that miss subtle anatomical mistakes. A new dataset called GazeVaLM takes a different angle. It records exactly where expert radiologists look when trying to tell real chest X-rays from diffusion-generated ones, then puts frontier multimodal LLMs through the same test.

Where computational metrics fall short

Generative adversarial networks, variational autoencoders, and diffusion models now produce chest radiographs of striking visual fidelity, and researchers lean on them to paper over data scarcity, privacy limits, and class imbalance in medical AI. Validation is the sticking point. FID and IS measure distributional similarity but do not capture whether images are clinically authentic. A radiograph may score well on standard metrics while containing subtle artifacts that an experienced radiologist would immediately flag as unrealistic.

Eye tracking offers a direct probe into that expert judgment, showing where clinicians attend, in what order, and for how long. Pixel-level scores cannot recover any of that. The Visual Turing Test, which asks experts to classify images as real or synthetic, operationalizes clinical authenticity in a measurable experimental paradigm. GazeVaLM is the first public dataset to combine both.

Inside the dataset

GazeVaLM dataset pipeline showing radiologist and LLM assessors performing diagnosis and real-or-fake classification

The paper describes a four-stage pipeline. Synthetic chest X-rays were generated using RoentGen, a vision-language diffusion model conditioned on free-text radiology reports. Reports were pulled from MIMIC-CXR that met RoentGen's 77-token input limit, ensuring each synthetic image has a paired real counterpart with identical report content. Thirty images were randomly drawn from the remaining pool, yielding a final stimulus set of 30 real and 30 synthetic chest X-rays across five pathology categories: normal, atelectasis, cardiomegaly, pleural effusion, and pneumonia.