ALoDLM gives difficult tokens more compute

Researchers from Amazon AGI, the University of Illinois Chicago, and Korea University have released ALoDLM, a diffusion language model that adjusts computation for each token during generation. The release includes 1.7B and 8B parameter checkpoints, training code, and an optimized inference stack.

Standard diffusion language models predict multiple masked positions during each denoising step, creating more opportunities for parallel execution than left-to-right autoregressive models such as Qwen and Llama. Their quality has generally trailed comparable autoregressive models because every unresolved token receives the same processing depth, regardless of its difficulty.

According to the authors’ evaluation, ALoDLM outperforms all diffusion and autoregressive baselines they tested at both model sizes. On GSM8K, ALoDLM-8B delivers about 2.7 times the throughput of Qwen3-8B served through vLLM at comparable accuracy.

Fixed depth spends compute blindly

A diffusion language model begins with masked tokens and repeatedly fills or refines multiple positions. Within the same sequence, some predictions require little context, while others depend on several reasoning steps.

Function words often become predictable after nearby words appear. A number inside a multistep arithmetic solution may depend on calculations elsewhere in the sequence. Applying an identical transformer depth to both positions spends excess compute on the first and may leave the second underprocessed.