Alibaba's Qwen2.5-3B Shrinks to 2.5GB so Anyone Can Run AI Locally
A 4-bit quantized build of Qwen2.5-3B-Instruct runs chat, coding and long-context tasks on roughly 2.5 GB of VRAM.
- Official 4-bit AWQ build of Qwen2.5-3B-Instruct, 3.09B params, 32K context, 8K generation
- Runs in roughly 2.5 GB VRAM at short context, 5.4 GB at 30K tokens on A100
- vLLM throughput: 165 tok/s at short input, 127 tok/s at 30K-token input
- Minor accuracy drop vs FP16 on general reasoning; math reasoning takes the biggest hit
- Supports Transformers, vLLM, SGLang, and Docker Model Runner out of the box
- Licensed under Qwen Research; 820k+ downloads make it a popular small-model default
Alibaba’s Qwen team published an official AWQ checkpoint of Qwen2.5-3B-Instruct, compressing its linear weights to 4-bit precision. Its Hugging Face counter has passed 820,000 downloads. The model’s roughly 3.1 billion parameters can run within a few gigabytes of GPU memory, making it practical for local applications and single-GPU services.
Activation-aware Weight Quantization uses calibration activations to identify weight channels that contribute most to model outputs. AWQ scales those channels before quantization, reducing the error introduced when weights are represented with 4-bit integers. The process compresses an existing model after training and requires no new training run.
The 4-bit label applies primarily to quantized linear weights. Embeddings, activations, normalization layers, runtime buffers, and the key-value cache can remain in FP16 or BF16, depending on the inference engine. Consequently, total VRAM use exceeds the packed weight size and grows with context length, batch size, and concurrent requests.
Inside the 4-bit checkpoint
The repository applies AWQ compression directly to the base checkpoint. It retains the original architecture, tokenizer, instruction tuning, and context limits.
| Specification | Value |
|---|---|
| Parameters | 3.09B total; 2.77B excluding embeddings |
| Transformer layers | 36 |
| Attention | 16 query heads; 2 key-value heads |
| Context window | 32,768 tokens |
| Maximum generation | 8,192 tokens within the context budget |
| Weight format | 4-bit AWQ |
| Core components | RoPE, SwiGLU, RMSNorm, QKV bias, tied word embeddings |
| Language coverage | More than 29 languages |
| License | Qwen Research License |
Grouped-query attention shares two key-value heads across 16 query heads, reducing key-value cache memory during generation. That saving becomes increasingly useful with long prompts or multiple concurrent requests.
Qwen describes the 2.5 series as improved in coding, mathematics, instruction following, long-form generation, table comprehension, and structured output. Supported languages include Chinese, English, French, Spanish, Portuguese, German, Italian, Russian, Japanese, Korean, Vietnamese, Thai, and Arabic. Applications that require valid JSON should still use schema validation or constrained decoding.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
