QuantTrio Shrinks Alibaba's Qwen3.6 From 70 GB to 24 GB for Workstation Deployment
QuantTrio shipped a 4-bit AWQ quantization of Qwen3.6-35B-A3B that compresses the MoE down to roughly 24GiB for vLLM deployment.
- QuantTrio released a 4-bit AWQ quantization of Qwen3.6-35B-A3B, roughly 24GiB on disk.
- Uses data-free quantization with no calibration dataset, Apache 2.0 licensed.
- Base model is a 35B MoE with 3B active, hybrid DeltaNet plus Gated Attention, 256 experts.
- Reported 73.4% SWE-bench Verified and 92.6% AIME 2026 on the full-precision model.
- Ships with vLLM config for expert parallelism, Qwen3-coder tool parser, and MTP speculative decoding.
- Requires vLLM 0.19.0, transformers 5.5.4, CUDA 12.8 or 13.0; official blog.
QuantTrio releases a 4-bit AWQ build of Qwen3.6-35B-A3B
QuantTrio has published an AWQ repository for Qwen3.6-35B-A3B, Alibaba’s open-weight mixture-of-experts model. The 4-bit release cuts the weight files from roughly 70 GB in BF16 to about 24 GiB, making deployment practical on a multi-GPU workstation, a pair of 48 GB cards, or a comparable server.
The repository has recorded more than 788,000 downloads on Hugging Face. That counter measures file requests rather than unique users or completed deployments, but it indicates substantial early interest in a version that requires less memory than the original weights.
The base model card describes the first open-weight release in the Qwen3.6 family. It contains 35 billion total parameters while activating about 3 billion for each token, and Alibaba distributes it under the Apache 2.0 license.
Four-bit weights cut the deployment cost
AWQ, short for Activation-aware Weight Quantization, stores supported linear-layer weights at 4-bit precision while retaining higher precision where the runtime requires it. This reduces weight memory and bandwidth, with a possible quality loss compared with BF16.
QuantTrio generated this build with a data-free quantization process, so it did not run a calibration corpus through the model to choose quantization scales. That removes calibration data from the build pipeline, although a carefully selected calibration set can sometimes produce better accuracy.
| Item | Published configuration |
|---|---|
| Weight files | Approximately 24 GiB |
| Serving runtime | vllm>=0.19.0 |
| Transformers version | transformers>=5.5.4 |
| CUDA environment | 12.8 or 13.0 |
| Recommended context cap | 32,768 tokens |
| License | Apache 2.0 |
The 24 GiB figure covers stored weights. A running server also needs memory for the KV cache, activations, CUDA graphs, temporary buffers, and concurrent requests. A 24 GB card therefore cannot be assumed to hold the complete service merely because the files are close to that size.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
