Google DeepMind's DiffusionGemma Hits 1,000 Tokens per Second by Ditching Left-to-Right Generation

Google's DiffusionGemma ditches token-by-token generation for parallel diffusion, hitting 1,000+ tokens/sec on a single H100 -- at the cost of benchmark quality

·
·
·
Google DeepMind's DiffusionGemma Hits 1,000 Tokens per Second by Ditching Left-to-Right Generation
Read7 min
TypeNews
SubtopicSmall Models
  • What it is: DiffusionGemma is a free, Apache 2.0 open-weights 26B model from Google DeepMind using text diffusion instead of autoregressive decoding.
  • Speed: Generates 1,000+ tokens/sec on a single H100 (FP8) and 200-400 tokens/sec on a consumer RTX 4090 -- 4-6x faster than autoregressive Gemma 4.
  • How it works: Refines a 256-token canvas in parallel using bidirectional attention and iterative denoising, enabling self-correction mid-generation.
  • Quality tradeoff: Underperforms standard Gemma 4 on most benchmarks (e.g., AIME 2026: 69.1% vs 88.3%); Google recommends it for speed-critical, not quality-critical, workloads.
  • Hardware: Activates only 3.8B of 26B parameters during inference; fits in 18GB VRAM when quantized, runs on RTX 4090/5090 and H100/H200.
  • Get it: Available now on Hugging Face; supported in vLLM, Transformers, SGLang, and MLX.

Google DeepMind just released DiffusionGemma, an experimental open-weights model that throws out the fundamental assumption every major LLM has been built on: that text must be generated one token at a time, left to right. Instead, it borrows the core idea from image diffusion -- start with noise, iteratively refine -- and applies it to language. The result is a model that can hit over 1,000 tokens per second on a single H100, a speed tier that no autoregressive model of comparable size can touch on the same hardware.

It's free, Apache 2.0 licensed, available right now on Hugging Face, and it runs on a consumer RTX 4090. But there's a real quality tradeoff, and Google is unusually candid about it.

Why autoregressive models have a speed ceiling

To understand what DiffusionGemma does differently, it helps to understand why standard LLMs are slow in the first place. Autoregressive models -- GPT, Llama, Gemma -- generate text by loading the full set of model weights from GPU memory, predicting one token, then loading the weights again for the next token. At low batch sizes, this process is memory-bandwidth-bound: the GPU spends most of its time loading weights from high-bandwidth memory to compute units, one token at a time. The compute cores sit mostly idle.

Generating 256 tokens autoregressively requires 256 separate forward passes on sequences growing from 1 to 256 positions. DiffusionGemma does the same in as few as 20 forward passes on a 256-position sequence -- a 12x reduction in the number of passes. That's the core of the speed story.

How diffusion actually works for text

Text diffusion borrows from image generation. Instead of predicting the next token, the model starts with a canvas of random placeholder tokens and iteratively refines them all at once. DiffusionGemma generates and refines a 256-token canvas in parallel. By providing the GPU with a large parallel workload, it utilizes tensor cores that would otherwise sit idle during local serving.

DiffusionGemma architecture diagram showing encoder with causal attention and decoder with bidirectional attention over the token canvas

The key architectural innovation is bidirectional attention during the denoising phase. Standard LLMs use causal (one-directional) attention -- each token can only look backward at what came before. DiffusionGemma's denoiser lets every token on the canvas attend to every other token simultaneously. This matters because:

  • Constraint propagation works both ways. A token at position 1 can be influenced by what the model is generating at position 200, enabling better global coherence.
  • Self-correction is possible. If confidence drops during a denoising pass, the sampler replaces tokens with random ones, allowing for continuous self-correction -- a capability autoregressive models lack since they are stuck with a token once it is generated.
  • Non-sequential tasks become tractable. All tokens attend to all others, which Google says is especially helpful for use cases such as inline editing, code infilling, working with amino acid sequences, and mathematical graphs.

For sequences longer than 256 tokens, the model uses Block Autoregressive Diffusion: once a 256-token block is fully denoised, it gets committed to the KV cache (the stored context), and the model starts a fresh canvas for the next block. This lets it handle up to 256K token context windows while keeping the parallel speed advantage per block.

Comparison of autoregressive vs diffusion token generation showing bidirectional constraint propagation in Sudoku solving

The numbers

At batch size 1 on a single NVIDIA H100, the FP8 version reaches 1,008 tokens per second. On H200, it hits 1,288 -- roughly six times a standard autoregressive baseline, according to vLLM benchmark results. On consumer hardware, an RTX 4090 or 3090 running DiffusionGemma at 4-bit delivers potentially 200-400 tokens/s -- a speed tier no autoregressive 20B+ model can touch on consumer hardware. For comparison, Qwen 2.5 32B at Q4 runs at roughly 40-60 tokens/s on a 4090.

The model itself is a 26B Mixture-of-Experts (MoE) architecture -- meaning it has 26 billion total parameters but only routes through a fraction of them per token. Since the model uses the mixture-of-experts technique, it doesn't have to keep the full 26 billion parameters in memory; instead, it activates only 3.8 billion during inference. This means it can run on a GPU with 18GB of VRAM.

The quality tradeoff -- and Google's honesty about it

Despite the speed gains, Google did not oversell the release. The company's launch post acknowledged directly that DiffusionGemma's overall output quality is lower than standard Gemma 4, adding "For applications that demand maximum quality, we recommend deploying standard Gemma 4."

The benchmark gap is real and worth understanding before you commit to it:

BenchmarkDiffusionGemma 26B A4BGemma 4 26B A4B
MMLU Pro77.6%82.6%
AIME 2026 (no tools)69.1%88.3%
LiveCodeBench v669.1%77.1%
GPQA Diamond73.2%82.3%
HLE (no tools)11.0%8.7%

The one benchmark where DiffusionGemma beats its autoregressive sibling is HLE (Humanity's Last Exam) -- a notoriously hard, open-ended reasoning benchmark. For short-form generation and summarization, quality is competitive. For complex reasoning, long-form writing, and code, autoregressive models win.

When to use it -- and when not to

The speed advantage is real but conditional. Local inference, single-user applications, and low-concurrency serving are where it shines. In those conditions the GPU has spare compute and memory bandwidth is the bottleneck -- DiffusionGemma's parallel block generation fills that gap.

At high batch sizes (32+), autoregressive models pull ahead through KV cache reuse across concurrent requests -- an advantage DiffusionGemma's bidirectional attention architecture cannot replicate. If you're running a high-throughput cloud API serving hundreds of concurrent users, stick with standard Gemma 4.

The sweet spots for DiffusionGemma:

  • Real-time interactive applications where latency matters more than peak quality
  • Code infilling and inline editing where bidirectional context is genuinely useful
  • Synthetic data generation at scale, where you need volume over per-sample reasoning depth
  • Constrained generation tasks like structured output, where self-correction helps
  • Edge and local deployment on consumer GPUs where memory bandwidth is the bottleneck

Getting started

DiffusionGemma is the first diffusion language model natively supported in the open source vLLM inference platform. You can serve it with a standard OpenAI-compatible endpoint:

haskell
vllm serve google/diffusiongemma-26B-A4B-it \
  --max-model-len 262144 \
  --max-num-seqs 4 \
  --gpu-memory-utilization 0.85 \
  --attention-backend TRITON_ATTN \
  --generation-config vllm \
  --hf-overrides '{"diffusion_sampler": "entropy_bound", "diffusion_entropy_bound": 0.1}' \
  --diffusion-config '{"canvas_length": 256}' \
  --enable-chunked-prefill

Or load it directly in Transformers:

makefile
from transformers import DiffusionGemmaForBlockDiffusion, AutoProcessor
MODEL_ID = "google/diffusiongemma-26B-A4B-it"
processor = AutoProcessor.from_pretrained(MODEL_ID)
model = DiffusionGemmaForBlockDiffusion.from_pretrained(
    MODEL_ID, dtype="auto", device_map="auto"
)

Beyond vLLM and Transformers, the model is supported in SGLang and MLX. For fine-tuning, Google released official JAX training recipes via Hackable Diffusion, and third-party support is available through Unsloth and NVIDIA NeMo. Cloud deployment is available via Google Cloud Model Garden and NVIDIA NIM.

The bigger picture

DiffusionGemma is the most significant public release testing whether diffusion can replace autoregression as the dominant paradigm for text generation. While the quality gap currently limits its production readiness, the 4x speed improvement on commodity hardware signals a potential architectural shift for latency-sensitive applications -- particularly edge deployment and real-time interactive systems where token throughput is the binding constraint.

By open-sourcing a model with a genuinely novel generation mechanism rather than another incremental autoregressive improvement, Google is staking a claim on the research frontier while giving the developer community a new axis of experimentation beyond scale and benchmark scores. The field has been debating diffusion vs. autoregressive for text for years -- this is the first time a major lab has shipped a production-grade, fully open model to settle the question empirically.

The assumptions that need updating: memory bandwidth is not the only bottleneck worth optimizing for, left-to-right generation is not the only viable decoding strategy, and quality vs. speed is no longer a fixed tradeoff determined purely by model size. DiffusionGemma is explicitly experimental -- Google says so -- but it's the kind of experiment that could reshape how the next generation of inference infrastructure gets built.

Trending
  • No trending articles

Comments

avatar

Next Reads