Skip to main content
Back

CUHK's MemDLM Fixes Diffusion LLMs, Boosting Long-Context Retrieval by 32%

A bi-level training scheme writes local denoising trajectories into fast weights, boosting long-context retrieval by up to 17 points on 8K benchmarks.

Pro

  • MemDLM adds a bi-level training loop that writes local denoising trajectories into fast LoRA weights.
  • Bridges the gap between static single-step MDLM training and multi-step progressive inference denoising.
  • Boosts RULER Variable Tracking at 8K from 78.8% to 95.8% on LLaDA-MoE.
  • Gains persist even when fast weights are discarded at inference, changing what the base model learns.
  • Training overhead is roughly 2.1x wall-clock; inference cost unchanged unless inner loop stays on.
  • Code available at github.com/JarvisPei/MemDLM, built on top of dllm.

Diffusion language models keep inching toward being a real alternative to autoregressive LLMs, but they still suffer from an awkward mismatch. During training they see a single masked snapshot of text, while during inference they progressively denoise their own noisy predictions across many steps. A new paper from researchers at CUHK and Huawei proposes MemDLM, a training framework that closes that gap by embedding a simulated denoising trajectory into a bi-level optimization loop, and it delivers sizable gains on long-context retrieval tasks.

Where standard diffusion training leaks performance

Masked Diffusion Language Models (MDLMs) work by corrupting text with a mask token and training a network to predict the original tokens back. Standard MDLM training minimizes the expected negative log-likelihood of masked tokens over uniformly sampled timesteps, which is essentially a single-step reconstruction objective. Inference doesn't look like that at all. DLMs generate text through a multi-step progressive denoising trajectory, starting from a fully masked sequence and iteratively unmasking the highest-confidence predictions until all tokens are decoded. At each step, the model is conditioned on its own noisy predictions from previous steps rather than pristine ground-truth context.

The authors call out a second, subtler problem. Dense softmax attention must distribute probability mass across all tokens in the sequence, so as the number of tokens grows, each token receives a smaller share of attention, making it harder to focus on task-relevant information. In long-context settings, standard training gives the model no help beyond token-space attention to preserve what matters.