Huihui-Qwen3.8 Fits on Two Budget Blackwell Cards at 368 tok/s
A community NVFP4 quant shrinks the 27B uncensored Qwen3.8 to 20 GB, hitting 367.9 tok/s at concurrency 8 on four Blackwell cards.
Takeaways
- Huihui-Qwen3.8-27B-abliterated-NVFP4 compresses a 27B uncensored Qwen from 55.6 GB to 20 GB.
- Uses NVFP4 W4A4 with FP8 KV cache and MTP speculative decoding on Blackwell GPUs.
- Reports 367.9 tok/s aggregate at concurrency 8 on four RTX PRO 2000 Blackwell cards.
- Prefill hits 3,250 tok/s on a 14.5k prompt with 556,063 tokens of KV cache.
- Runs on vanilla vLLM 0.22.0 with no custom quantization flag needed.
- Ablation narrowed to layers 18 to 51, fixing a xhigh reasoning runaway bug.
A new community quantization is pushing what you can run on a pair of modest Blackwell cards. Huihui-Qwen3.8-27B-abliterated-NVFP4 takes the uncensored 27B Qwen3.8 variant, squeezes it through NVIDIA's new 4-bit floating point format, and reports serving numbers that make it usable as a real backend rather than a curiosity.
The headline compression is aggressive. Weights drop from 55.6 GB to 20.0 GB, small enough to fit on two 16 GB cards with real KV headroom. The model is served through vanilla vLLM with no custom fork, and it ships with a Multi-Token Prediction draft head preserved in bf16 so speculative decoding works out of the box.
Why NVFP4 changes the math
NVFP4 is NVIDIA's 4-bit floating point format that landed with Blackwell. INT4 forces the runtime to dequantize weights back to 16-bit before the matmul. Blackwell Tensor Cores handle NVFP4 operations directly, so tensors stay in their compact 4-bit format throughout inference as long as both weights and activations are quantized. That is the W4A4 part of the spec here, and it is why this release targets Blackwell exclusively.
Against plain FP8 or BF16 checkpoints, NVFP4 shrinks weight storage by roughly 3.3x versus BF16 and 1.5 to 1.8x versus FP8, which is what lets a 27B model realistically fit on consumer-tier Blackwell boards.