Alibaba's Qwen3-14B-FP8 Slashes Weight Storage in Half for 24 GB GPUs
Alibaba's Qwen3-14B gets an official FP8 release with fine-grained block quantization, cutting memory use while keeping dual thinking modes intact.
Takeaways
- Qwen released an official FP8 checkpoint of the 14.8B-parameter dense Qwen3-14B model.
- Uses fine-grained FP8 with block size 128, preserving near-full bf16 accuracy on most benchmarks.
- Keeps Qwen3's toggleable thinking and non-thinking modes via enable_thinking flag or /think, /no_think tags.
- Native 32K context, extendable to 131K tokens with YaRN rope scaling.
- Works with vLLM 0.8.5+, SGLang 0.4.6+, Transformers 4.51+, Ollama, llama.cpp, and MLX.
- Apache-2.0 licensed; see the Qwen3 technical report for training details.
Alibaba’s Qwen team has released an official FP8 checkpoint of Qwen3-14B, its dense reasoning and chat model. The first-party build cuts raw weight storage from roughly 30 GB in BF16 to about 15 GB before runtime overhead, making single-GPU deployment practical at modest context lengths while preserving Qwen3’s switchable thinking mode.
FP8 cuts the weight footprint
Qwen3-14B-FP8 quantizes the original 14.8-billion-parameter checkpoint rather than changing its architecture or training. The model retains grouped-query attention, a native 32,768-token context window, tool-calling support, and Apache 2.0 licensing.
| Specification | Value |
|---|---|
| Total parameters | 14.8 billion |
| Non-embedding parameters | 13.2 billion |
| Layers | 40 |
| Attention | 40 query heads, 8 key/value heads |
| Native context | 32,768 tokens |
| Extended context | Up to 131,072 tokens with YaRN |
| License | Apache 2.0 |
Qwen’s fine-grained recipe stores weights in FP8 using 128-by-128 blocks and calculates activation scales dynamically at runtime. Smaller blocks localize quantization error more effectively than a single scale for an entire tensor, helping the checkpoint retain the BF16 model’s quality.
FP8 reduces weight storage by about half, although total GPU memory also includes activations, attention caches, CUDA kernels, and serving overhead. Actual speed gains depend on the accelerator and inference engine because older GPUs may lack native FP8 execution paths.
Reasoning lives in the template
Qwen3’s chat template selects between thinking mode for math, coding, and multi-step reasoning and non-thinking mode for lower-latency dialogue. Developers can set enable_thinking when applying the template or place /think and /no_think in user prompts for turn-level control.