NVIDIA's DiffusionGemma Hits 161 Tokens per Second on a Desktop Machine

NVIDIA's NVFP4 quantization squeezes Google's text-diffusion model into 18GB VRAM, hitting 1,100+ tokens/sec on a single H100 — a new speed ceiling for local inference

·
·
·
NVIDIA's DiffusionGemma Hits 161 Tokens per Second on a Desktop MachinePRO
  • New model: NVIDIA released DiffusionGemma 26B A4B IT NVFP4, a 4-bit quantized version of Google DeepMind's first open-weight text diffusion model.
  • Speed: Generates 1,100+ tokens/sec on a single H100 and 700+ tok/s on an RTX 5090 — up to 4x faster than autoregressive models at low batch sizes.
  • How it works: Instead of predicting one token at a time, it initializes a 256-token "canvas" of random tokens and iteratively denoises them all in parallel, shifting the GPU bottleneck from memory bandwidth to compute.
  • Hardware footprint: NVFP4 quantization cuts memory from 50GB+ (BF16) down to ~18GB VRAM, fitting on a single RTX 4090 or 5090 — no server required.
  • Quality tradeoff: Benchmark accuracy is nearly identical to the BF16 baseline after quantization, but the base model scores below Gemma 4 12B on reasoning — it's a speed-first release.
  • Access: Free under Apache 2.0; available on Hugging Face, NVIDIA NIM, and build.nvidia.com with vLLM serving support.

When a tweet reads "My first reaction: How is that possible?" and racks up 520 likes, something genuinely surprising just happened. The post was about running a 26-billion-parameter model at 161.9 tokens per second on a personal DGX Spark , not a data center rack, a deskside machine. The model in question is DiffusionGemma 26B A4B IT NVFP4, NVIDIA's quantized release of Google DeepMind's first open-weight text diffusion model, and it's rewriting what "fast" means for local inference.

Not your typical LLM

Every major language model you've used , GPT, Claude, Gemma 4 , generates text the same way: one token at a time, left to right. Diffusion language models generate text through iterative denoising rather than sequential token prediction, a fundamentally different architecture from standard autoregressive LLMs. DiffusionGemma takes the same core idea that powers image generators like Stable Diffusion and applies it to text.

The model initializes a blank sequence of 256 random tokens , called a canvas , and iteratively evaluates and refines the entire canvas simultaneously. During the decoding phase, standard Transformer models are memory-bound rather than compute-bound: most of the generation time is spent loading model weights from hardware memory into the processing units, rather than performing the actual mathematical calculations. DiffusionGemma bypasses this limitation by shifting the bottleneck from memory bandwidth to compute, generating and refining a 256-token canvas in parallel , providing the GPU with a large parallel workload and utilizing tensor cores that would otherwise sit idle during local serving.

For long outputs, because a single canvas is fixed to 256 tokens, DiffusionGemma chains diffusion and autoregression together: it runs diffusion cycles to generate a full 256-token block, appends that completed block to the prompt context, updates the encoder's KV cache, and kicks off a brand new 256-token canvas diffusion cycle.

The architecture underneath

Built on the Gemma 4 26B A4B Mixture-of-Experts architecture, it has 25.2B total parameters and 3.8B active parameters, balancing capability with efficient inference. MoE (Mixture-of-Experts) means the model has many specialized sub-networks but only routes each token through a small fraction of them , so you get the knowledge capacity of a 26B model at roughly the compute cost of a 4B one.

Pro article

This story is for Pro members

You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.

Trending
  • No trending articles

Comments

avatar

Next Reads