Red Hat Shrinks Gemma 4 Memory in Half Without Losing Accuracy
Red Hat AI shrinks Google's 26B MoE Gemma 4 to FP8 with per-token dynamic quantization, cutting memory in half while keeping over 99% accuracy on most benchmarks.
Takeaways
- Red Hat AI released an FP8-dynamic quantization of Google's Gemma 4 26B A4B MoE model
- Cuts disk and GPU memory by roughly 50% versus the BF16 original
- Recovers 99%+ accuracy on MMLU-Pro, GSM8K, IFEval, MATH-500, AIME, GPQA, and LiveCodeBench
- Built with data-free PTQ via LLM Compressor, no calibration set required
- Vision tower, embeddings, output head, and MoE router kept in original precision
- Ships ready for vLLM, SGLang, and Docker Model Runner under Apache 2.0
Red Hat AI has released an FP8 checkpoint of Google’s instruction-tuned Gemma 4 26B A4B model. The data-free conversion roughly halves the memory occupied by quantized weights and requires no calibration or fine-tuning. Red Hat’s published evaluation tracks the BF16 model closely, giving developers a smaller checkpoint that can run through vLLM on a single 40 GB or 48 GB accelerator at moderate context and batch sizes.
Gemma’s sparse architecture
Google DeepMind’s upstream model is a sparse Mixture-of-Experts system with 26 billion total parameters and about 4 billion active parameters per token. Each token passes through eight routed experts selected from 128, plus one shared expert. Sparse activation reduces arithmetic inside the expert layers, while serving still requires loading the full set of weights into memory.
| Property | Specification |
|---|---|
| Total parameters | 26 billion |
| Active parameters | About 4 billion per token |
| Expert routing | 8 of 128 routed experts, plus 1 shared expert |
| Maximum context | 256,000 tokens |
| Inputs | Text and images |
| Model features | Thinking mode, function calling, instruction tuning |
| Quantization | FP8 weights and dynamic per-token activations |
Where the bytes disappear
Red Hat converts weights in the transformer’s linear layers from 16-bit floating point to 8-bit floating point, using precomputed per-channel scales. Activations are quantized dynamically with a separate scale for each token during inference. The embeddings, output head, vision tower, and MoE router retain their original precision.
Selective quantization protects components that are sensitive to numerical drift, especially the router that assigns tokens to experts. It also means the total footprint will exceed the simple estimate of one byte per parameter because unquantized tensors, scaling metadata, the KV cache, and runtime allocations still consume memory.