Google DeepMind's Gemma 4 Beats Models 10x Its Size on Math and Coding

Google DeepMind just published the Gemma 4 Technical Report, pulling back the curtain on the architectural decisions behind one of the most capable open-weight model families available today. The report covers everything from the encoder-free 12B variant to quantization-aware training and multi-token prediction drafters, giving the research and engineering community a rare, detailed look at how Google is pushing efficiency without sacrificing quality.

Five models, one family

The Gemma 4 family is not a single model but a carefully tiered suite designed to run everywhere, from a Raspberry Pi to a data center. Gemma 4 is a new generation of open-weight, natively multimodal language models featuring dense and Mixture-of-Experts architectures ranging from 2.3B to 31B parameters. Here is how the lineup breaks down:

• E2B (2.3B effective / 5B total): Built for ultra-mobile and browser deployment. Supports text, image, and audio.
• E4B (4.5B effective / 8B total): Same modalities as E2B, more capable. Targets phones and edge hardware.
• 12B (Unified): The architectural wildcard. Encoder-free, processes raw image patches and audio directly.
• 26B-A4B (MoE): 26B total parameters, only 3.8B active per token. Designed for high-throughput reasoning.
• 31B (Dense): The flagship. All parameters active on every forward pass.

All Gemma 4 models are available for download from Kaggle and Hugging Face. They are released under an Apache 2.0 license, which removes the legal friction that has historically made deploying open models in production complicated.

The encoder-free bet on the 12B

The most architecturally interesting model in the family is the 12B. Most multimodal models bolt on a separate vision encoder (like a ViT) and a separate audio encoder, then project their outputs into the language model's embedding space. This works, but it adds memory fragmentation, latency, and complexity. Gemma 4 12B bypasses heavy multi-stage vision and audio encoders entirely, feeding multimodal data straight into the LLM backbone, reducing multimodal latency.

Concretely, the 12B unified variant replaces vision and audio encoders with direct linear projections of the input. For vision, the 550M encoder is replaced by a single large matrix multiplication with only 35M parameters. For audio, raw 40ms chunks at 16kHz are projected directly into the LLM's embedding space. The result is a dramatically leaner pipeline. The 12B unified variant drops the separate vision and audio encoders and projects raw image patches and audio waveforms directly into the LLM's embedding space.