Hugging Face's Gemma 2 9B Squeezed to 6 GiB Hits 800K Downloads
A community INT4 quantization of Gemma 2 9B Instruct crosses 800K downloads, shrinking the model to around 6 GiB of VRAM for high-throughput serving.
Takeaways
- Community AWQ INT4 build of Gemma 2 9B Instruct passes 800K downloads on Hugging Face
- Needs roughly 6 GiB of VRAM to load, around 75% less than the BF16 original
- Uses AutoAWQ with GEMM kernels, zero-point quantization, group size 128
- Supported out of the box in Transformers, AutoAWQ, TGI 2.3+ and vLLM
- Requires
transformers>=4.45.0; NVIDIA GPUs on Linux or Windows only - Released under Google's Gemma license, commercial use permitted with conditions
The Hugging Face repository for hugging-quants/gemma-2-9b-it-AWQ-INT4 has crossed 800,000 downloads on the Hub’s displayed counter. Published by the community group hugging-quants, the checkpoint converts Google’s instruction-tuned Gemma 2 9B from 16-bit weights to a 4-bit AWQ representation.
Hugging Face counts qualifying file requests rather than unique users or production installations. The deployment benefit is more concrete: the compressed checkpoint requires about 6 GiB of VRAM to load, bringing a 9-billion-parameter instruction model within reach of 12 GB consumer GPUs and increasing model density on server cards.
AWQ’s four-bit trade
Activation-aware Weight Quantization, or AWQ, uses calibration activations to identify weight channels where rounding errors would have the greatest effect. It scales those channels before packing the weights into 4-bit groups. The stored weights shrink substantially, while runtime activations remain at the higher precision selected by the inference backend.
| Property | Value | Practical effect |
|---|---|---|
| Source model | google/gemma-2-9b-it |
Google’s instruction-tuned Gemma 2 9B |
| Quantizer | AutoAWQ | Weight-only activation-aware quantization |
| Weight precision | INT4 | Four bits per quantized weight |
| Group size | 128 | Each block of 128 values shares quantization metadata |
| Zero point | Enabled | Each group receives an offset as well as a scale |
| Packing | GEMM | Optimized for matrix multiplication and batched serving |