Google DeepMind's DiffusionGemma Generates Text 4x Faster by Ditching Sequential Tokens
Google DeepMind's DiffusionGemma generates 256 tokens in parallel, hitting 1,100+ tokens/sec on a single H100 — a new architecture for speed-first local inference
PRO- New architecture: DiffusionGemma generates 256 tokens in parallel via discrete diffusion instead of token-by-token autoregression.
- Speed: 1,000+ tokens/sec on H100 and 700+ on RTX 5090 — up to 4x faster than standard Gemma 4 in single-user local inference.
- Efficient footprint: 26B total parameters but only 3.8B active per pass (MoE); fits in ~18GB VRAM when quantized.
- Quality tradeoff: Trails Gemma 4 on every major benchmark; Google explicitly recommends Gemma 4 for production quality use cases.
- Unique strengths: Bidirectional attention enables code infilling, inline editing, structured output, and non-linear generation tasks autoregressive models struggle with.
- Available now: Apache 2.0, open weights on Hugging Face with day-zero support in vLLM, Transformers, MLX, SGLang, and Unsloth.
Every LLM you have ever used works like a typewriter: predict one token, commit it, predict the next. That sequential constraint is baked into the architecture. Google DeepMind's DiffusionGemma throws that constraint out entirely. Instead of generating tokens one at a time, it starts with a block of 256 random placeholder tokens and iteratively refines the entire block in parallel , the same way Stable Diffusion turns visual noise into a coherent image, but applied to text.
DiffusionGemma is Google DeepMind's first open-weight text diffusion language model, released under Apache 2.0. It uses a 26B Mixture-of-Experts architecture and generates up to 4x faster than autoregressive models of comparable size by refining 256-token blocks in parallel instead of one token at a time.
The Printing Press vs. the Typewriter
To understand why this is fast, you need to understand what slows down standard LLMs on local hardware. Nearly every LLM today is autoregressive: generate token t, then t+1, each depending on all prior tokens. Decoding becomes memory-bandwidth-bound , the GPU waits on KV cache reads more than it computes. A single user on a dedicated GPU leaves most of the chip idle between those sequential reads.
Autoregressive models emit one token at a time, bottlenecked by memory bandwidth. DiffusionGemma instead starts each 256-token block as random placeholders and iteratively refines the whole block in parallel, saturating compute instead of waiting on sequential memory reads. That is what produces the speed.
By shifting the decode bottleneck from memory-bandwidth to compute, DiffusionGemma delivers up to 4x faster token generation on GPUs , 1,000+ tokens per second on a single NVIDIA H100, 700+ tokens per second on an NVIDIA GeForce RTX 5090.
Under the Hood: How Block Diffusion Works
DiffusionGemma is engineered to reduce the sequential bottlenecks of standard causal language models. It employs an encoder-decoder architecture specifically optimized for inference speed. The encoder operates in a prefill capacity, processing the initial prompt and generating the KV cache. The decoder then utilizes bidirectional attention to process an input block (a 'canvas') of tokens, accessing the cached context via cross-attention.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
