Skip to main content
Back

Zhejiang University's MedVeriSeg Teaches Medical AI to Refuse Hallucinated Tumor Masks

A training-free framework catches hallucinated masks in medical segmentation models by reading the geometry of similarity heatmaps between [SEG] tokens and image features.

  • MedVeriSeg adds training-free query verification to LISA-like medical segmentation models to reject hallucinated masks. Paper
  • Core insight: the [SEG] token similarity heatmap looks compact for real targets and diffuse for absent ones.
  • Scores each heatmap on strength, compactness, and purity, then routes ambiguous cases to visual agents.
  • Final decision is a weighted score of three evidence sources, not a direct MLLM call, keeping it auditable.
  • Hits 83.93% accuracy on MedVeriSeg-Bench (6,022 samples), close to medical-student specialists at 88.23%.
  • Code available on GitHub; works as plug-in for MediSee, MedPLIB, and vanilla LISA.

Medical segmentation models built on multimodal LLMs have a quiet reliability problem: ask them to segment a tumor that isn't there, and they will happily draw one anyway. A group from Zhejiang University and Southern Medical University has proposed MedVeriSeg, a training-free verification layer that lets these models say no when the queried target doesn't exist in the image, without retraining a single weight.

The hallucination hiding inside LISA

Most recent MLLM-based medical segmenters follow the LISA blueprint. LISA employs LLaVA to output a text response based on the input image, text query, and a [SEG] token, and the embedding for that customized token is decoded into a segmentation mask via a SAM decoder. In medical variants like MediSee and Med-PLIB, the vision backbone is swapped for MedSAM and the language side for LLaVA-Med, but the mechanism is the same.

These models were trained almost exclusively on valid query-mask pairs, so existing LISA-like methods typically generate masks regardless of whether the target specified in the query is present, leading to hallucinated segmentation. Ask a chest CT model to outline a clavicle in an abdominal slice and you still get a confident mask. In a teaching setting that misleads students; in a clinical setting it risks misdiagnosis.

Comparison of segmentation with and without MedVeriSeg on a brain MRI

Retraining with negative samples is expensive and tends to overfit. Feeding the query into a general MLLM like Qwen-3-VL as a gatekeeper also underperforms, because general models struggle with fine-grained medical visual cues.

The geometry of a real answer