Skip to main content
Back

Google's Gemma 4 26B Shrinks to 17GB and Runs on a Single GPU

A community 4-bit AWQ build of Google's Gemma 4 26B A4B compresses the vision-capable MoE to 17GB while keeping the 256K context and reasoning intact.

Pro

  • AWQ 4-bit quant of Gemma 4 26B A4B IT shrinks the MoE model to 17.2GB.
  • Keeps 256K context, vision input, function calling, and configurable thinking mode.
  • Base model activates only 3.8B of 25.2B parameters per token, giving 4B-model speed.
  • Benchmark highlights: MMLU Pro 82.6%, AIME 2026 88.3%, Codeforces ELO 1718.
  • Serves via vLLM, SGLang, and Docker; Apache 2.0 licensed for commercial use.
  • Over 2.5M downloads on Hugging Face, calibrated on a STEM and agentic dataset.

A trending Hugging Face upload from user cyankiwi has shrunk Google's Gemma 4 26B A4B to a size that fits on a single consumer GPU. The gemma-4-26B-A4B-it-AWQ-4bit repository packs the instruction-tuned multimodal model into a 17.19 GB checkpoint using Activation-aware Weight Quantization, and it has already crossed 2.5 million downloads.

Squeezing a Mixture-of-Experts into 17GB

Gemma 4 26B A4B IT is Google DeepMind's newest open instruction-tuned Mixture-of-Experts release. Despite 25.2B total parameters, only 3.8B activate per token during inference, delivering near-31B quality at a fraction of the compute cost. The MoE layout routes each token through 8 active experts out of 128, plus 1 shared expert.

AWQ drops the weights to 4-bit precision while preserving the channels that matter most to the model's outputs. In practice that means running the 26B MoE at roughly a third of its original memory footprint. The size differences across quant formats:

FormatSizeBest for
BF16 (original)~50.5 GBH100/A100 servers
Q8_0 GGUF~26.9 GBDual-GPU workstations
AWQ 4-bit17.2 GBSingle 24GB GPU with vLLM/SGLang
Q4_0 GGUF~14.6 GBllama.cpp on laptops

What survives the compression

The quantized build keeps the full feature surface of Gemma 4: multimodal input across text, images, and video (up to 60s at 1fps), a 256K token context window, native function calling, a configurable thinking/reasoning mode, and structured output support. The vision encoder (~550M params) is included, so it handles the same image-text-to-text tasks as the base checkpoint.