EPFL's LAW Beats Discrete Diffusion Baselines by Carrying Soft Token States
A new continuous language model called LAW ditches discrete token sampling between steps, propagating expected embeddings instead and outperforming diffusion baselines on reasoning tasks.
- LAW is a continuous language model built purely on Self-Conditioning, decoding tokens only at the final step.
- It propagates the Expected Embedding under the denoiser's prediction rather than sampled discrete tokens between steps.
- On GSM8K from TinyGSM, LAW scores 42.2% versus 34.5% for uniform Loopholing Discrete Diffusion.
- Fine-tuning the pretrained Duo model with LAW raises GSM8K accuracy from 17.2% to 48.1%.
- LAW solves 96.5% of hard Sudoku puzzles and 90.9% of 20x20 mazes with only 4 function evaluations.
- Code, scripts, and checkpoints are released on GitHub and Hugging Face.
Researchers from EPFL and collaborating institutions propose LAW, short for Loopholing All the Way, a self-conditioning method that carries a probability-weighted token embedding from one denoising pass to the next. Discrete decoding happens once, after the final pass. Across the paper’s Sudoku, maze, GSM8K, and OpenWebText experiments, LAW improves on discrete-diffusion and flow-matching baselines in several compute-matched settings.
Discrete diffusion language models learn to recover clean sequences from corrupted ones. During generation, each forward pass updates many token positions in parallel, allowing the model to revise an entire sequence. Many implementations then sample hard tokens from the predicted distributions and feed those tokens into the next pass, discarding confidence scores and alternative candidates.
Hard tokens discard useful uncertainty
A sampled token looks the same whether the model assigned it 51% or 99% probability. The probabilities assigned to every other candidate also disappear, leaving the next denoising pass with less information than the previous pass produced.
Self-conditioning partially addresses that loss by feeding an earlier prediction into a later denoising pass. Recent discrete diffusion models generally use it as an auxiliary signal. Loopholing Discrete Diffusion Models, or LDDM, also carry a hidden representation across passes, although the model still receives discrete token inputs.
LAW makes self-conditioning the main state transition:
- The denoiser produces logits over the vocabulary at every sequence position.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
