Skip to main content
Back

Meta's SPG Fixes Broken RL for Diffusion LLMs, Jumping 27 Points on Sudoku

Meta's new reinforcement learning method fixes a fundamental bias in training diffusion language models, delivering up to 27% accuracy gains on reasoning benchmarks.

  • SPG proposes a sandwiched policy gradient using lower and upper likelihood bounds for masked diffusion LLMs.
  • Fixes gradient bias from using ELBO alone by applying an evidence upper bound (EUBO) to negative-reward traces.
  • Delivers +3.6% GSM8K, +2.6% MATH500, +18.4% Countdown, +27% Sudoku over previous state-of-the-art RL methods.
  • Block-wise masking during Monte Carlo estimation aligns training with semi-autoregressive rollout distributions used by LLaDA.
  • Mixing ELBO and EUBO for negative traces provably reduces gradient variance versus either bound alone.
  • Code available at github.com/facebookresearch/SPG, built on LLaDA-8B-Instruct with LoRA fine-tuning.

Diffusion large language models decode multiple tokens in parallel, promising a faster alternative to the token-by-token generation of autoregressive models. Teaching them to reason well through reinforcement learning has been stuck on a mathematical wall: their log-likelihood, the quantity every standard RL algorithm needs, cannot be computed. A team from Meta Superintelligence Labs, MIT, USC and UCLA has published Sandwiched Policy Gradient (SPG), and the fix is elegant enough that it delivers double-digit accuracy jumps on hard reasoning tasks.

Why diffusion LLMs break standard RL

Masked Diffusion Language Models corrupt clean text by progressively replacing tokens with a special [mask] token, then train a neural network to reverse the process by predicting the originals from a corrupted version. During training, the model learns to fill in the blanks. During generation, it starts from an all-mask sequence and gradually reveals tokens in parallel.

Parallelism is the selling point, but it comes with a cost. In an autoregressive model, the probability of a sequence is the product of per-token probabilities, so you can differentiate it directly. In a diffusion model, the same probability requires marginalizing over every possible denoising trajectory, which is intractable. That intractable log-likelihood is exactly what accurate policy gradient estimation depends on.

Prior work papered over this with the Evidence Lower Bound (ELBO), the same objective used during pretraining. Substitute ELBO for log-likelihood in GRPO or DPO and you get something that runs, but with a hidden flaw. The ELBO is only a lower bound on the true log-likelihood, so the RL objective is only a valid lower bound on the true expected reward when all rewards are non-negative. The moment you try to penalize a bad output by pushing its likelihood down, minimizing a lower bound tells you nothing about what happens to the actual likelihood. You might push the bound down while the true probability of the bad answer stays flat, or even rises.