Google's Gemma 4 Gets 70% Faster Prefill and Fixes Its Laziness Problem

Google has pushed a batch of improvements to Gemma 4 across all model sizes, addressing a set of pain points that the community had been vocal about since the model's April launch. This isn't a new model, but it's a meaningful quality-of-life update that touches inference speed, vision quality, tool calling reliability, and response completeness.

The laziness problem, finally addressed

One of the most consistent complaints from Gemma 4 users was the model cutting answers short or refusing to complete tasks mid-way. Google says it has significantly reduced these edge cases, leading to more complete responses. This was a real blocker for agentic workflows: users found the model particularly prone to incomplete outputs at higher context lengths, and in coding agent setups, it would sometimes fail to call tools consistently, leading to minutes of trial and error.

Alongside this, the chat template has been overhauled. The previous template had several structural bugs that caused crashes and malformed conversations:

• Unmatched tool_call_id in tool responses now falls back to 'unknown' instead of crashing
• Consistent .get() access prevents StrictUndefined errors for optional message keys
• The backward scan for model-turn continuation is now O(1) instead of O(n) per message

Google has also provided new Jinja chat templates for multiple Gemma 4 variants (31B, 27B, E4B, and E2B) specifically aimed at improving tool-calling behavior. If you're running Gemma 4 locally via llama.cpp, it's recommended to use the newly updated templates and apply them via the --chat-template-file command-line argument.

Flash Attention 4 lands on Hopper

The headline performance improvement is uniform Flash Attention 4 (FA4) support on NVIDIA Hopper GPUs (H100, H200). FA4 is an attention kernel that rewrites how the GPU computes the attention operation, overlapping matrix multiplications with memory loads to keep all hardware units busy simultaneously. The result for Gemma 4 users: prefill throughput jumps by 25-70% and time-to-first-token (TTFT) drops by up to 31%.

Getting FA4 working with Gemma 4 was non-trivial. Gemma 4 introduces a hybrid attention design where 26 out of 30 layers use sliding window attention with head_dim=256, while 4 out of 30 layers use global attention with head_dim=512 -- a configuration that exceeded what earlier FA versions could handle. Gemma 4 is likely the first widely-used open model to require head_dim=512 with multiple attention heads in production. Resolving this required upstream work in the FlashAttention library itself before it could be enabled uniformly.

Vision: more resolution when you need it

Gemma 4's vision system has always supported a configurable token budget, but the default of 280 tokens was causing measurable accuracy regressions on fine-grained tasks. Google's own documentation explicitly recommends tuning this budget for OCR tasks, and the default 280-token budget is insufficient for fine-grained visual tasks.

This update clarifies and promotes the full range of options. You can control how many tokens each image consumes: 70, 140, 280, 560, or 1120 tokens, giving you a direct knob to trade off between visual detail and inference speed. To understand the practical difference:

• At 70 tokens, the model gets a rough sketch of the image: enough for scene classification or high-level question answering.
• 280 tokens is the default, suitable for most general vision tasks.
• At 1120 tokens, the model receives enough detail for OCR, small object detection, and document layout analysis.

Google has also launched an interactive Space on Hugging Face that lets you upload any image and preview exactly how it looks at each token budget level before committing to a setting in production. The Space shows resized versions at all five budgets while preserving the original aspect ratio.

The key design difference from previous Gemma releases is that Gemma 4 processes images of different sizes using a fixed-budget number of tokens, keeping the image's natural aspect ratio rather than squashing every image into a fixed square. To set the budget in code:

from transformers import AutoProcessor, Gemma4ForConditionalGeneration
processor = AutoProcessor.from_pretrained("google/gemma-4-27B-it")
model = Gemma4ForConditionalGeneration.from_pretrained(
    "google/gemma-4-27B-it",
    device_map="auto"
)
# Bump to max detail for OCR / document parsing
inputs = processor(
    images=image,
    text=prompt,
    max_soft_tokens=1120,  # default is 280
    return_tensors="pt"
).to(model.device)

What's still worth watching

The FA4 gains are currently gated to NVIDIA Hopper GPUs (H100/H200). FA3 and FA4 are gated to compute capability >= 8.0, excluding SM 8.6 and 8.9, so Ada Lovelace GPUs like the RTX 4090 and L40S won't benefit from this path. Users on those cards will need to wait for a separate FA2 extension or a dedicated kernel for head_dim=512.

Gemma 4 models handle text and image input (with audio supported on E2B, E4B, and 12B models), feature a context window of up to 256K tokens, and maintain multilingual support in over 140 languages. All updates are available now on Hugging Face across the full model family: E2B, E4B, 12B, 26B-A4B, and 31B.