Skip to main content
Back

Lorbus Fixes Qwen3.6-27B to Run 2x Faster on a Single Consumer GPU

A community INT4 quantization shrinks Qwen3.6-27B from 54GB to 18GB while keeping its Multi-Token Prediction head alive for 2x faster decoding.

Model, Full Access
  • INT4 AutoRound quant of Qwen3.6-27B shrinks weights from 54GB to 18GB (3x reduction)
  • Fix dequantizes mtp.fc to BF16 so vLLM's MTP speculative decoder actually loads
  • Reaches 80-90% MTP draft acceptance, delivering ~2x throughput (32 to 60 tok/s on RTX 5090)
  • Base Qwen3.6-27B is dense multimodal, 262K context, beats a 397B MoE on coding benchmarks
  • W4A16 group-128 symmetric, quantized with Intel AutoRound, Apache 2.0 licensed
  • Runs on single 32GB consumer GPU via vLLM, SGLang, or Docker Model Runner

Running a 27B multimodal model with 262K context on a single consumer GPU used to mean choosing between fitting the weights and keeping the model's fast-decoding tricks. A new INT4 AutoRound quantization of Qwen3.6-27B from Hugging Face user Lorbus does both, and the fix that makes it work is worth understanding if you serve quantized VLMs.

Qwen3.6-27B is Alibaba's flagship dense open-weight release, a 27-billion-parameter multimodal model under Apache 2.0 that posts agentic coding scores beating the team's previous-generation 397B Mixture-of-Experts flagship. It fits into a 16.8 GB Q4_K_M quantization on a single consumer GPU, accepts text, image, and video inputs, and supports a 262,144-token context window.

Three times smaller, tricks intact

The quant uses W4A16 (4-bit weights, FP16 activations) with group size 128, produced with Intel's AutoRound. The resulting checkpoint weighs 18 GB, down from roughly 54 GB in BF16. That 3x reduction fits comfortably in a 32 GB consumer card with room left for KV cache.

The interesting part is not the compression ratio, which is standard for AutoRound. It's that the model's native Multi-Token Prediction head survives the process intact. MTP is supported out of the box on Qwen3.6 for low-latency decoding, and it delivers roughly 1.4-2.2x faster generation with no accuracy hit, which matters most for local models where latency floors are painful.