Google's Gemma 4 26B Shrinks to 17GB and Runs on a Single GPU
A community 4-bit AWQ build of Google's Gemma 4 26B A4B compresses the vision-capable MoE to 17GB while keeping the 256K context and reasoning intact.
ProTakeaways
- AWQ 4-bit quant of Gemma 4 26B A4B IT shrinks the MoE model to 17.2GB.
- Keeps 256K context, vision input, function calling, and configurable thinking mode.
- Base model activates only 3.8B of 25.2B parameters per token, giving 4B-model speed.
- Benchmark highlights: MMLU Pro 82.6%, AIME 2026 88.3%, Codeforces ELO 1718.
- Serves via vLLM, SGLang, and Docker; Apache 2.0 licensed for commercial use.
- Over 2.5M downloads on Hugging Face, calibrated on a STEM and agentic dataset.
A trending Hugging Face upload from user cyankiwi has shrunk Google's Gemma 4 26B A4B to a size that fits on a single consumer GPU. The gemma-4-26B-A4B-it-AWQ-4bit repository packs the instruction-tuned multimodal model into a 17.19 GB checkpoint using Activation-aware Weight Quantization, and it has already crossed 2.5 million downloads.
Squeezing a Mixture-of-Experts into 17GB
Gemma 4 26B A4B IT is Google DeepMind's newest open instruction-tuned Mixture-of-Experts release. Despite 25.2B total parameters, only 3.8B activate per token during inference, delivering near-31B quality at a fraction of the compute cost. The MoE layout routes each token through 8 active experts out of 128, plus 1 shared expert.
AWQ drops the weights to 4-bit precision while preserving the channels that matter most to the model's outputs. In practice that means running the 26B MoE at roughly a third of its original memory footprint. The size differences across quant formats:
| Format | Size | Best for |
|---|---|---|
| BF16 (original) | ~50.5 GB | H100/A100 servers |
| Q8_0 GGUF | ~26.9 GB | Dual-GPU workstations |
| AWQ 4-bit | 17.2 GB | Single 24GB GPU with vLLM/SGLang |
| Q4_0 GGUF | ~14.6 GB | llama.cpp on laptops |
What survives the compression
The quantized build keeps the full feature surface of Gemma 4: multimodal input across text, images, and video (up to 60s at 1fps), a 256K token context window, native function calling, a configurable thinking/reasoning mode, and structured output support. The vision encoder (~550M params) is included, so it handles the same image-text-to-text tasks as the base checkpoint.