Skip to main content
Back

HKUST's DeP Cuts AI Image Hallucinations by 10 Points Without Retraining

A training-free method probes multimodal models with paraphrased prompts to catch and correct hallucinations driven by language priors, without touching the image or the weights.

Paper, Full Access
  • DeP is a training-free decoding method that mitigates multimodal hallucinations without altering images or weights.
  • Core insight: hallucinations show up as visual attention drifting when prompt wording changes but meaning does not.
  • Uses three paraphrase strategies picked adaptively via an Effective Sample Size uncertainty score.
  • Splits the image into stable evidence regions and drifting suspicious regions using attention variance across paraphrases.
  • On POPE Adversarial with InstructBLIP, F1 jumps 8.3 points; MMHal hallucination rate drops from 59.4 to 49.0 on LLaVA-1.5.
  • Costs scale linearly with the number of paraphrases (M=3 recommended); struggles on generic prompts lacking semantic anchors.

Multimodal large language models have a well-known bad habit: they confidently describe things that are not in the image. Ask the same question two slightly different ways and you can flip the answer from correct to nonsense. A new paper from researchers at HKUST (Guangzhou), Decoding by Perturbation (DeP), turns that fragility into a diagnostic tool, and uses it to steer the model back toward the pixels.

The core observation is simple. If the meaning of a prompt stays the same, the model's attention to visual evidence should stay stable even when the wording changes. When attention drifts anyway, that shift is a fingerprint of the model leaning on language statistics rather than what it actually sees. DeP treats hallucination as this hypersensitivity and builds a decoding-time fix around it.

Example showing how small changes to a prompt cause a multimodal model to shift its visual attention and produce different hallucinated answers.

Why prior fixes leave something on the table

Training-free hallucination mitigation has mostly gone down two roads, and both carry real costs. Visual-degradation methods add noise or corruption to the image and contrast the outputs, which pulls the model away from the natural image distribution it was trained on and makes it hard to localize what actually triggered the false answer. Attention-intervention methods reach into the decoder to rebalance vision and language, but the intrusive edits tend to hurt fluent generation.

DeP sidesteps both routes. The image stays untouched, the weights stay untouched, and the intervention happens on the text side and in the output space where the prior actually lives.

How the probe works

At each decoding step, DeP generates a small set of paraphrased prompts that preserve meaning but vary the surface phrasing. Given the original prompt, it constructs a perturbation set of M variations and runs them through the model to collect logits and cross-modal attention. The perturbations come from three progressively aggressive strategies: