Google DeepMind's Gemma 4 Beats Models 10x Its Size on Math and Coding

Google DeepMind just published the Gemma 4 Technical Report, pulling back the curtain on the architectural decisions behind one of the most capable open-weight model families available today. The report covers everything from the encoder-free 12B variant to quantization-aware training and multi-token prediction drafters, giving the research and engineering community a rare, detailed look at how Google is pushing efficiency without sacrificing quality.

Five models, one family

The Gemma 4 family is not a single model but a carefully tiered suite designed to run everywhere, from a Raspberry Pi to a data center. Gemma 4 is a new generation of open-weight, natively multimodal language models featuring dense and Mixture-of-Experts architectures ranging from 2.3B to 31B parameters. Here is how the lineup breaks down:

• E2B (2.3B effective / 5B total): Built for ultra-mobile and browser deployment. Supports text, image, and audio.
• E4B (4.5B effective / 8B total): Same modalities as E2B, more capable. Targets phones and edge hardware.
• 12B (Unified): The architectural wildcard. Encoder-free, processes raw image patches and audio directly.
• 26B-A4B (MoE): 26B total parameters, only 3.8B active per token. Designed for high-throughput reasoning.
• 31B (Dense): The flagship. All parameters active on every forward pass.

All Gemma 4 models are available for download from Kaggle and Hugging Face. They are released under an Apache 2.0 license, which removes the legal friction that has historically made deploying open models in production complicated.

The encoder-free bet on the 12B

The most architecturally interesting model in the family is the 12B. Most multimodal models bolt on a separate vision encoder (like a ViT) and a separate audio encoder, then project their outputs into the language model's embedding space. This works, but it adds memory fragmentation, latency, and complexity. Gemma 4 12B bypasses heavy multi-stage vision and audio encoders entirely, feeding multimodal data straight into the LLM backbone, reducing multimodal latency.

Concretely, the 12B unified variant replaces vision and audio encoders with direct linear projections of the input. For vision, the 550M encoder is replaced by a single large matrix multiplication with only 35M parameters. For audio, raw 40ms chunks at 16kHz are projected directly into the LLM's embedding space. The result is a dramatically leaner pipeline. The 12B unified variant drops the separate vision and audio encoders and projects raw image patches and audio waveforms directly into the LLM's embedding space.

This is not just a memory win. The report shows that competitive audio-text performance can be achieved without a dedicated audio encoder at all, which opens the door for future architectures to rethink the encoder-decoder split entirely.

How the architecture actually works

All Gemma 4 models follow a decoder-only Transformer with several key design choices that separate them from a standard architecture:

• Hybrid attention: The main architecture characteristic is alternating local sliding-window and global full-context attention layers. Smaller dense models use 512-token sliding windows; larger models use 1024. The ratio is 5 local layers for every 1 global layer (4:1 for E2B). This keeps memory costs linear with sequence length for most tokens while still allowing the model to attend globally when needed.
• p-RoPE and KV cache sharing: Long-context memory limitations are alleviated via better local-to-global attention ratios, positional encoding, and KV cache sharing. Specifically, the global KV cache footprint is reduced by up to 37.5% by reusing keys as values in global attention layers.
• Per-Layer Embeddings (PLE): One of the most distinctive features in smaller Gemma 4 models is Per-Layer Embeddings (PLE). In a standard transformer, each token gets a single embedding vector at input. PLE adds a parallel, lower-dimensional conditioning pathway alongside the main residual stream. This gives each layer its own token-specific signal without the cost of a full-sized embedding per layer, and is key to why the E2B and E4B models can punch above their parameter count.
• MoE routing (26B-A4B): The 26B-A4B variant is a Mixture-of-Experts model with 128 small experts, activating 8 routed experts plus 1 shared expert per token. Only 3.8B parameters fire per forward pass, achieving roughly 97% of the dense 31B model's MMLU Pro quality at ~12% of the dense FLOPs.

Speculative decoding and quantization built in

Two inference-time innovations ship alongside the models themselves, and both matter for anyone deploying these in production.

Multi-Token Prediction (MTP) drafters are small autoregressive heads trained alongside each model. The idea, called speculative decoding, is to have a tiny model draft several tokens ahead, then verify them in parallel with the main model. Compute efficiency is increased via QAT and memory efficiency via MTP drafters. The drafter cross-attends to the KV cache of the main model, so it does not need a separate prefill pass and supports any draft length.

Quantization-Aware Training (QAT) means the models were trained knowing they would be quantized, rather than quantizing after the fact. This preserves quality at lower precision. The pre-training dataset is a large-scale, diverse collection of data from a wide range of domains and modalities, including web documents, code, images, and audio, with a cutoff date of January 2025. The memory savings are dramatic: the 31B model goes from 64GB in bf16 to 19.2GB quantized. The E2B drops from 4.6GB to just 0.8GB in mobile quantization format.

QAT was also applied to the encoders. On the audio encoder, the team achieved a 78% reduction in on-disk footprint (from 390MB to 87MB) while actually improving performance over Gemma 3n.

Thinking mode

Gemma 4 models come with a thinking mode in which they generate reasoning traces prior to responding, improving overall performance. This is activated via a special token in the system prompt (<|think|>) and is available across all model sizes. The thinking traces are wrapped in <|channel>...<channel|> tags and are not shown to the end user by default. All benchmark numbers in the report are reported with thinking enabled unless stated otherwise.

The numbers

Gemma 4 establishes a leap in performance across STEM, multimodal, and long-context benchmarks, and rivals larger, frontier open models in human-rated tasks. On the LMArena text leaderboard (blind human side-by-side evaluations), the 31B is ranked as the top open dense model with an Elo score of 1451, outranking models like Qwen 3.5 397B-A17B (which activates 17B parameters per token) and DeepSeek V4 Flash Thinking.

Key benchmark highlights from the report:

• MMLU Pro: 85.2% (31B), 82.6% (26B-A4B), vs. 67.6% for Gemma 3 27B
• AIME 2026 (math competition): 89.2% (31B), 88.3% (26B-A4B), vs. 20.8% for Gemma 3 27B
• LiveCodeBench v6 (coding): 80.0% (31B), vs. 29.1% for Gemma 3 27B
• GPQA Diamond (PhD-level science): 84.3% (31B), vs. 42.4% for Gemma 3 27B
• MRCR v2 8-needle at 128k (long context): 66.4% (31B), vs. 13.5% for Gemma 3 27B

On vision, Gemma 4 models demonstrate a leap in performance compared to Gemma 3 across benchmarks, and human evaluations demonstrate that Gemma 4 performs comparably to significantly larger open models. Notably, the E4B equals or outperforms Gemma 3 27B on all vision evals despite being a fraction of the size.

On complex multi-step reasoning chains, frontier closed models retain an edge. The 31B's Humanity's Last Exam (HLE) score of 19.5% without tools is strong for an open model but still far behind the best closed systems. For autonomous software engineering tasks (SWE-bench), Qwen 3.5 27B holds an advantage.

What this is actually good for

The practical use-cases split cleanly by model size:

• E2B / E4B: On-device audio transcription and translation, real-time image understanding on phones, browser-based AI via WebGPU (ONNX checkpoints are available). The audio-capable E2B and E4B variants are the strongest open-weight option for phones today.
• 12B (Unified): Multimodal pipelines where you want to minimize memory fragmentation and encoder overhead. Good for document parsing, OCR, and audio-text tasks on mid-range hardware.
• 26B-A4B: High-throughput serving where you want near-31B quality at a fraction of the compute cost. The 4B active parameter footprint makes batching much more efficient.
• 31B: Math, coding, agentic reasoning, and long-context tasks where you have the hardware. Gemma 4 31B's AIME and Codeforces numbers are exceptional for an open-weight model in this size band.

Ecosystem and availability

The Hugging Face launch post details day-0 support across the full inference ecosystem: transformers, llama.cpp, MLX, transformers.js (WebGPU), and mistral.rs. Fine-tuning is supported via TRL, with an example training script that teaches the E2B to drive in the CARLA simulator using multimodal reinforcement learning. GGUF quantized checkpoints are available for local use with lmstudio, Jan, and compatible coding agents.

For those who want to run the model programmatically, the simplest entry point via transformers is the any-to-any pipeline:

from transformers import pipeline
pipe = pipeline("any-to-any", model="google/gemma-4-e2b-it")
messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": "https://example.com/photo.jpg"},
        {"type": "text", "text": "What is in this image?"}
    ]
}]
output = pipe(messages, max_new_tokens=100, return_full_text=False)

The full model collection, including base and instruction-tuned variants for all five sizes, is available on the Google Hugging Face collection. The technical report itself is on arXiv and the official developer docs cover model-specific deployment details.