Skip to main content
Back

Hugging Face's Gemma 2 9B Squeezed to 6 GiB Hits 800K Downloads

A community INT4 quantization of Gemma 2 9B Instruct crosses 800K downloads, shrinking the model to around 6 GiB of VRAM for high-throughput serving.

Model, Full Access
  • Community AWQ INT4 build of Gemma 2 9B Instruct passes 800K downloads on Hugging Face
  • Needs roughly 6 GiB of VRAM to load, around 75% less than the BF16 original
  • Uses AutoAWQ with GEMM kernels, zero-point quantization, group size 128
  • Supported out of the box in Transformers, AutoAWQ, TGI 2.3+ and vLLM
  • Requires transformers>=4.45.0; NVIDIA GPUs on Linux or Windows only
  • Released under Google's Gemma license, commercial use permitted with conditions

The Hugging Face repository for hugging-quants/gemma-2-9b-it-AWQ-INT4 has crossed 800,000 downloads on the Hub’s displayed counter. Published by the community group hugging-quants, the checkpoint converts Google’s instruction-tuned Gemma 2 9B from 16-bit weights to a 4-bit AWQ representation.

Hugging Face counts qualifying file requests rather than unique users or production installations. The deployment benefit is more concrete: the compressed checkpoint requires about 6 GiB of VRAM to load, bringing a 9-billion-parameter instruction model within reach of 12 GB consumer GPUs and increasing model density on server cards.

AWQ’s four-bit trade

Activation-aware Weight Quantization, or AWQ, uses calibration activations to identify weight channels where rounding errors would have the greatest effect. It scales those channels before packing the weights into 4-bit groups. The stored weights shrink substantially, while runtime activations remain at the higher precision selected by the inference backend.

Property Value Practical effect
Source model google/gemma-2-9b-it Google’s instruction-tuned Gemma 2 9B
Quantizer AutoAWQ Weight-only activation-aware quantization
Weight precision INT4 Four bits per quantized weight
Group size 128 Each block of 128 values shares quantization metadata
Zero point Enabled Each group receives an offset as well as a scale
Packing GEMM Optimized for matrix multiplication and batched serving