Google DeepMind's DiffusionGemma Hits 1,000+ Tokens per Second on a Single GPU
Google's DiffusionGemma generates entire 256-token blocks at once, hitting 1,288 tokens/sec on H200 — but trades benchmark accuracy for raw speed
DiffusionGemma is Google DeepMind's first open-weights diffusion language model (dLLM) , and it flips the fundamental assumption behind how LLMs generate text. Instead of producing one token at a time, it generates and refines an entire 256-token block in parallel. The result is a model that hits speeds no autoregressive system can match on a single GPU, at the cost of some accuracy on standard benchmarks.
DiffusionGemma is built by Google DeepMind on the 26B A4B Mixture-of-Experts Gemma 4 architecture, generating tokens using discrete diffusion. The open-weights model is multimodal, handling text, image, and video inputs to generate text output. It was released under Apache 2.0, with weights available on HuggingFace at google/diffusiongemma-26B-A4B-it.
The bottleneck nobody was talking about
Every GPU-based LLM deployment has a dirty secret: the GPU is mostly waiting. Standard autoregressive models generate one token per forward pass, which means the GPU must reload the model's weights from memory for each token. At small batch sizes , exactly the scenario for local or interactive apps , the GPU's compute cores sit idle while memory bandwidth does all the work.
DiffusionGemma bypasses this limitation by shifting the bottleneck from memory bandwidth to compute, generating and refining a 256-token canvas in parallel. By providing the GPU with a large parallel workload, it utilizes tensor cores that would otherwise sit idle during local serving. The payoff is dramatic: up to 700+ tokens per second on an NVIDIA GeForce RTX 5090 and 1000+ tokens per second on a single NVIDIA H100.
The vLLM team independently validated even higher numbers. The FP8 diffusion model reaches 1,288 generation tokens per second on H200 , roughly 6x a standard autoregressive baseline and about 3x one using multi-token prediction , and 1,008 tokens per second on H100.
How the canvas actually works
The core mechanism is called Uniform State Diffusion. Think of it like a crossword puzzle being solved all at once rather than letter-by-letter. The model starts with a canvas of random placeholder tokens and iteratively refines them over multiple passes.

Over multiple denoising passes, highly confident tokens help resolve adjacent positions, causing the entire sequence to snap into focus. Confidence is measured by entropy: positions where the model is very sure about its prediction get locked in first, and those anchors propagate context to neighboring positions. The process stops early when the canvas stabilizes , either because all positions are confident, or because predictions stop changing between steps.
For outputs longer than 256 tokens, the model uses Block Autoregressive Diffusion: once a 256-token block is fully denoised, the model processes and commits it to the KV cache, then transitions to the next block, initializing a fresh 256-token canvas conditioned on the previously committed history. This chains the parallel speed of diffusion with the sequential stability of autoregressive generation.
A key ingredient making this stable is self-conditioning: between denoising steps, the model feeds back its own previous probability distribution (not just the hard token choices) into the next pass. This gives every step a memory of what the model believed before, so even positions that get re-randomized carry forward information rather than starting from scratch.
What's inside the architecture
The model shares the same weights for two distinct operating modes:
- Encoder mode , uses standard causal (left-to-right) attention to process the prompt and write to the KV cache. Runs once to prefill the initial context, then once per block to commit finished tokens.
- Decoder mode , uses bidirectional attention over the 256-token canvas. Every position can attend to every other position simultaneously, enabling global constraint propagation and self-correction.
Designed as a 26B Mixture of Experts model, it activates only 3.8B parameters during inference, allowing quantized deployment within 18 GB VRAM limits. The sparse MoE design uses 8 active experts out of 128 total, giving it strong reasoning capabilities without the memory footprint of a dense 26B model.

Where bidirectional attention actually matters
The Sudoku demo in the developer guide isn't just a party trick , it illustrates a genuine architectural advantage. Traditional autoregressive models struggle with strict, multivariable constrained problems like Sudoku because they generate text strictly left to right and cannot evaluate future placeholders or backtrack.
While the base DiffusionGemma model is not specifically trained to solve Sudoku puzzles (roughly 0% success rate), applying a simple JAX SFT recipe on a Sudoku dataset raises correctness to 80% success, while decreasing the overall inference step count. The fine-tuned model solves puzzles in 12 denoising steps versus 48 for the base model. This same property , the ability to reason about all positions simultaneously , is what makes the model well-suited for code infilling and inline editing, where the surrounding context constrains what goes in the middle.
The benchmark reality check
The experimental model trails standard Gemma 4 on every benchmark , a tradeoff Google says is worth it for tasks like code infilling and in-line editing. Here's how it stacks up:
| Benchmark | DiffusionGemma 26B A4B | Gemma 4 26B A4B |
|---|---|---|
| MMLU Pro | 77.6% | 82.6% |
| AIME 2026 (no tools) | 69.1% | 88.3% |
| LiveCodeBench v6 | 69.1% | 77.1% |
| GPQA Diamond | 73.2% | 82.3% |
| MATH-Vision | 70.5% | 82.4% |
| HLE (no tools) | 11.0% | 8.7% |
The one benchmark where DiffusionGemma edges ahead is HLE (Humanity's Last Exam) without tools , a notoriously hard reasoning benchmark where the bidirectional context may actually help. But the gaps on math competition problems (AIME) and coding (Codeforces ELO: 1429 vs. 1718) are significant. Google is honest: DiffusionGemma is not and will not be "Gemma 4 quality at 4x speed." It's a different tradeoff.
Getting it running
The model is available today through multiple paths. For serving, vLLM has native support , the first dLLM to be integrated into the framework. Here's the minimal vLLM serve command:
vllm serve google/diffusiongemma-26B-A4B-it \
--max-model-len 262144 \
--max-num-seqs 4 \
--gpu-memory-utilization 0.85 \
--attention-backend TRITON_ATTN \
--generation-config vllm \
--hf-overrides '{"diffusion_sampler": "entropy_bound