Skip to main content
Back

Dream-7B Hallucinates 98.5% of the Time Where AR Models Don't

A controlled study finds diffusion language models hallucinate more than autoregressive peers, with three unique failure modes tied to parallel decoding.

Pro

  • First controlled study shows dLLMs hallucinate more than matched AR baselines across three tasks.
  • Dream-7B posts 92.54% hallucination rate on PreciseWikiQA versus 89.06% for its Qwen2.5-7B parent.
  • Dream fails to refuse non-existent entities 98.50% of the time, worse than every AR baseline tested.
  • LLaDA-8B saturates with more denoising steps; Dream-7B improves monotonically thanks to non-sequential decoding.
  • Three unique failure modes identified: premature termination, incomplete denoising, and context intrusion into math or code.
  • Code and HalluLens-based evaluation released at ZeroLoss-Lab/Lost-in-Diffusion.

Diffusion large language models have been closing the gap with autoregressive systems on standard benchmarks, but a new controlled study suggests they pay for that parallel decoding trick with a much shakier grip on facts. Lost in Diffusion is the first head-to-head hallucination benchmark of dLLMs against AR baselines built from the same weights, and the result is not flattering for the diffusion side.

Same weights, different decoders

Autoregressive LLMs generate tokens one at a time, left to right. Diffusion LLMs like LLaDA and Dream instead start from a fully masked sequence and iteratively denoise all positions in parallel. That bidirectional view should, in theory, help them avoid the classic AR failure modes: the snowballing of early errors and the reversal curse where a model that learned A is B cannot answer B is A.

Most prior work on dLLMs focused on speed and architecture, leaving faithfulness unexamined. The authors set out to answer one question: does diffusion mitigate hallucination, or make it worse?

Controlling for the confounders

Raw benchmark numbers between arbitrary models are noisy because pretraining data and scale dominate. The paper works around this with two carefully matched pairs:

  • Architectural alignment: LLaDA-8B against LLaMA-3-8B, similar backbones and scale.
  • Parametric alignment: Dream-7B against Qwen2.5-7B. Because Dream is initialized directly from Qwen weights, this pairing controls for parametric knowledge.