Google's DiffusionGemma Generates 256 Tokens at Once, 4x Faster Than Any LLM

Google has released DiffusionGemma, an experimental open model that abandons the token-by-token generation approach used by every major LLM today. Instead of predicting one word at a time, it drafts an entire block of 256 tokens simultaneously and refines them through iterative passes , the same core idea that powers image generators like Stable Diffusion, now applied to text at production scale.

DiffusionGemma is an experimental open model that explores text diffusion, an exceptionally fast approach to text generation. Released under an Apache 2.0 license, this 26B Mixture of Experts (MoE) model moves beyond the sequential token-by-token processing of typical autoregressive LLMs, generating entire blocks of text simultaneously and delivering up to 4x faster text generation on GPUs.

The problem it's solving

Local inference has always had a dirty secret: your GPU is mostly idle. In the cloud, sequential generation is efficient because servers can batch thousands of user requests together to share the hardware load. But when run locally for a single user, this word-by-word process leaves your dedicated GPU underutilized , it spends most of its time simply waiting for the next "keystroke."

Because DiffusionGemma processes the full block in parallel, the weights are loaded once per refinement pass and applied across 256 tokens simultaneously. This shifts the inference bottleneck from memory bandwidth to raw computational throughput, turning the memory wall into a non-problem for single-user workloads. The result is a fundamentally different hardware utilization profile , one that happens to be ideal for local, low-concurrency deployments.

How text diffusion actually works

Text diffusion (also called a discrete diffusion language model, or dLLM) is a generation paradigm borrowed from image synthesis. Instead of committing to tokens left-to-right, the model starts with noise and refines its way to coherence. DiffusionGemma uses a specific variant called Uniform State Diffusion. Here's the process in three steps:

1. Canvas initialization: The model starts with a block of random placeholder tokens , essentially a blank canvas of noise.
2. Iterative denoising: The model makes multiple forward passes, locking in high-confidence tokens and using them as context to refine the remaining uncertain ones.
3. Convergence: After enough passes (controlled by a parameter T), the full 256-token block resolves into coherent text.

Integrating this into existing serving infrastructure required supporting a fundamentally different decoding pattern. Diffusion LLMs do not fit cleanly into the standard autoregressive serving path: they require bidirectional attention, iterative refinement, block-based generation, and custom sampling behavior at each denoising step.

The bidirectional attention is the key structural difference from standard transformers. DiffusionGemma generates a 256-token block in parallel rather than sequentially, with every token position attending to every other. This is impossible in autoregressive models, where causality forces each token to only look backward. That global context is what enables tasks like code infilling, where the model needs to see both what comes before and after the gap it's filling.

Architecture: built on Gemma 4, with a diffusion head

DiffusionGemma is a 26B Mixture of Experts (MoE) model that activates only 3.8B parameters during inference. It is built on the Gemma 4 backbone (26B-A4B) with a diffusion head added. The intelligence base comes from Gemma 4, while the generation technique derives from the Gemini Diffusion research program. It supports multimodal input , text, image, and video , generating text output, with a 256K context window and 140+ languages, released under Apache 2.0.

When quantized to FP8 or NVIDIA's 4-bit floating-point format, DiffusionGemma fits comfortably within the VRAM of consumer GPUs like the RTX 4090 and 5090, making full local deployment practical without specialized server hardware. Google also worked with NVIDIA to support NVFP4 kernels on Hopper and Blackwell architectures for enterprise deployments.

The numbers

The speed claims are real, but come with important context:

• 1,000+ tokens/sec on a single NVIDIA H100
• 700+ tokens/sec on an NVIDIA GeForce RTX 5090
• 4x faster than comparable autoregressive models on dedicated GPUs
• 18GB VRAM required when quantized , within reach of high-end consumer cards

One important caveat: this approach is strictly aimed at consumer-facing, low-concurrency local use; because its parallel decoding offers diminishing returns under high-QPS cloud workloads, the throughput advantage is strongest at low-to-medium batch sizes on a single accelerator. If you're running a cloud API serving hundreds of users, standard Gemma 4 will likely be more cost-effective. Also worth noting: Apple Silicon users should not expect the same gains, since unified-memory architectures remain memory-bandwidth-bound rather than compute-bound during inference.

The quality trade-off

Google is unusually candid about the limitations here. Google is very direct about the trade-off: DiffusionGemma prioritizes speed and parallel layout generation, and its overall output quality is lower than standard Gemma 4. For maximum quality production work, Google still recommends autoregressive Gemma 4.

On all benchmarks, the DiffusionGemma model underperforms when compared to Gemma 4 26B A4B , something Google itself acknowledges. There's no technical reason why a diffusion model couldn't perform just as well as a more traditional LLM, but the focus here is on speed. The quality gap is most visible on complex reasoning and instruction-following tasks. It is viable for simpler code generation tasks like boilerplate, templates, and straightforward functions. For complex algorithmic problems or precise instruction following, autoregressive models like Gemma 4 27B still have an edge.

The number of denoising steps T is a key dial. Output quality is lower than autoregressive Gemma 4, particularly at low denoising steps (T=10-20). More steps means better quality but slower generation , you tune this tradeoff for your specific use case.

There's also a UX consideration: DiffusionGemma outputs a 256-token block at once rather than streaming individual tokens. This affects UX for chat interfaces that display tokens as they generate. Workarounds exist (post-processing the block into a simulated stream), but they add complexity.

Where it shines

The bidirectional attention unlocks genuinely new capabilities for certain task types. Completing a line of code in the middle of a file, rewriting a paragraph while keeping the surrounding context, iterating quickly on a draft: in these scenarios a model that produces and self-corrects in parallel delivers a smoother experience than one that spells things out letter by letter.

A concrete example: Unsloth fine-tuned DiffusionGemma to solve Sudoku puzzles , a task that autoregressive models struggle with because each cell depends on cells that haven't been filled yet. DiffusionGemma's global context makes constraint-satisfaction problems like this much more tractable.

The bigger picture

Text diffusion techniques themselves are not a new concept; the AI research community has explored them for years. However, applying them to large-scale models has long posed significant challenges. The release of DiffusionGemma marks a measurable step by Google toward practical implementation of this research direction. Prior academic work like LLaDA, MDLM, and Plaid laid the theoretical groundwork, but none reached production scale with the ecosystem support DiffusionGemma ships with on day one.

DiffusionGemma is the first diffusion language model natively supported in the open source vLLM inference platform. That's a significant milestone for the paradigm , it means the dLLM serving path is now a first-class citizen in the most widely used open inference stack, not a research prototype requiring custom infrastructure.

The field assumption that needs updating: speed and quality are not permanently coupled in autoregressive models. DiffusionGemma proves text diffusion works at scale with open weights. The 4x speed improvement is real and meaningful for production workloads where latency matters. The quality gap is a current limitation of this generation of the technology, not a fundamental ceiling , and the Apache 2.0 license means the community can iterate on it openly.

Getting started

The weights are available now on Hugging Face. Supported serving frameworks include:

• vLLM , first dLLM natively supported
• MLX , for Mac users (though speed gains may be limited)
• Hugging Face Transformers , standard integration
• Unsloth and NVIDIA NeMo , for fine-tuning
• NVIDIA NIM and Gemini Enterprise Agent Platform Model Garden , for cloud deployment
• llama.cpp , support arriving soon

For fine-tuning experiments, Google released Hackable Diffusion, a modular JAX toolbox designed for composability. If you want to understand the mechanics before diving in, Maarten Grootendorst's visual guide linked in the official blog post is an excellent starting point.