Skip to main content
Back

Qwen3.6-27B Community Build Hits 1.74x Faster Decoding on Blackwell

A community NVFP4 quantization of Qwen3.6-27B fixes broken speculative decoding on Blackwell GPUs, delivering 1.74x higher throughput than the baseline release.

  • NVFP4 quantization of Qwen3.6-27B with the MTP head restored in bf16 for working speculative decoding on Blackwell.
  • Uses modelopt format, vLLM's native fast path on SM120, instead of the slower compressed-tensors fallback.
  • 1.67 to 1.74 times the throughput of the earlier NVFP4 baseline, reaching 207 tok/s in 2-parallel aggregate.
  • Runs on a single RTX PRO 6000 Blackwell at 256K context, 14-15 GB VRAM at load.
  • Apache 2.0, text-only, vision tower stripped; use the sibling repo if you need multimodal input.
  • Reproducible via the open-source GGUF-to-NVFP4-SM120 pipeline including the MTP graft recipe.

A new community quantization of Qwen3.6-27B is picking up serious download volume for one reason: it actually makes multi-token prediction work on NVIDIA Blackwell cards. Qwen3.6-27B-Text-NVFP4-MTP is a text-only, NVFP4-quantized sibling of Qwen/Qwen3.6-27B that restores the Multi-Token Prediction head in bf16, letting vLLM run speculative decoding at the promised speeds instead of collapsing to zero draft acceptance.

The result is roughly 1.67 to 1.74 times the throughput of the earlier compressed-tensors NVFP4 release from the same author, on the same RTX PRO 6000 Blackwell, using the same vLLM build. The model is Apache 2.0 and already past a million downloads.

Two bugs, one repo

The repository exists because of two specific complaints filed against the author's original NVFP4 build. The first: the model ran slower than the official FP8 checkpoint on Blackwell, because the compressed-tensors NVFP4 path is slower than modelopt on Blackwell SM120. Switching to modelopt format puts inference on vLLM's native fast path for that hardware.

The second issue was more subtle. AutoModelForCausalLM.from_pretrained does not load the MTP head, so it gets dropped during quantization, leading to 0% draft acceptance. Multi-Token Prediction is a small auxiliary head that predicts several tokens ahead, letting vLLM use it as a draft model for speculative decoding on the same base network. If the head is silently missing, you pay the memory cost of the model with none of the speedup.