Qwen Drops Qwen3.8-Flash-Next, a 125B Open Model That Activates Only 6B Parameters

Qwen has released an FP8-quantized checkpoint that doubles as a preview of the architecture powering the upcoming Qwen4 generation. Qwen3.8-Flash-Next-FP8 is a 125B-parameter Mixture-of-Experts model that activates only 6B parameters per token, ships with a 262K native context extendable to 1M, and handles text, images, and video in a single stack.

Four architectural changes are bundled into an openly downloadable checkpoint you can serve today on vLLM or SGLang, each one slated to carry forward into Qwen4. The size is almost beside the point; the plumbing is what makes this worth examining.

What changed under the hood

• Hybrid attention with QSA. Rather than selecting individual tokens for processing, QSA operates at the micro-block level, which cuts long-context latency. Layers interleave Gated DeltaNet (a linear-attention variant) with QSA blocks, so most layers use cheap linear attention and only a fraction pay the full attention cost.
• Gated Residual. Information flows through widened residual streams via an element-wise, data-dependent read gate and a per-branch scalar write gate. Four branches with a bottleneck rank of 320 add expressiveness per layer without degrading training stability.
• N-gram embedding. A 20-million-entry table at layer 2 indexes short bigrams and trigrams, adding a parameter-scaling axis that requires less computation and is more amenable to offloading than Mixture-of-Experts. The extra 51B parameters live here and can sit on CPU or slower memory.