Skip to main content
Back

Red Hat Shrinks Gemma 4 Memory in Half Without Losing Accuracy

Red Hat AI shrinks Google's 26B MoE Gemma 4 to FP8 with per-token dynamic quantization, cutting memory in half while keeping over 99% accuracy on most benchmarks.

Pro

Model, Full Access
  • Red Hat AI released an FP8-dynamic quantization of Google's Gemma 4 26B A4B MoE model
  • Cuts disk and GPU memory by roughly 50% versus the BF16 original
  • Recovers 99%+ accuracy on MMLU-Pro, GSM8K, IFEval, MATH-500, AIME, GPQA, and LiveCodeBench
  • Built with data-free PTQ via LLM Compressor, no calibration set required
  • Vision tower, embeddings, output head, and MoE router kept in original precision
  • Ships ready for vLLM, SGLang, and Docker Model Runner under Apache 2.0

Red Hat AI has released an FP8 checkpoint of Google’s instruction-tuned Gemma 4 26B A4B model. The data-free conversion roughly halves the memory occupied by quantized weights and requires no calibration or fine-tuning. Red Hat’s published evaluation tracks the BF16 model closely, giving developers a smaller checkpoint that can run through vLLM on a single 40 GB or 48 GB accelerator at moderate context and batch sizes.

Gemma’s sparse architecture

Google DeepMind’s upstream model is a sparse Mixture-of-Experts system with 26 billion total parameters and about 4 billion active parameters per token. Each token passes through eight routed experts selected from 128, plus one shared expert. Sparse activation reduces arithmetic inside the expert layers, while serving still requires loading the full set of weights into memory.

Base model and checkpoint details
Property Specification
Total parameters 26 billion
Active parameters About 4 billion per token
Expert routing 8 of 128 routed experts, plus 1 shared expert
Maximum context 256,000 tokens
Inputs Text and images
Model features Thinking mode, function calling, instruction tuning
Quantization FP8 weights and dynamic per-token activations

Where the bytes disappear

Red Hat converts weights in the transformer’s linear layers from 16-bit floating point to 8-bit floating point, using precomputed per-channel scales. Activations are quantized dynamically with a separate scale for each token during inference. The embeddings, output head, vision tower, and MoE router retain their original precision.

Selective quantization protects components that are sensitive to numerical drift, especially the router that assigns tokens to experts. It also means the total footprint will exceed the simple estimate of one byte per parameter because unquantized tensors, scaling metadata, the KV cache, and runtime allocations still consume memory.