Unofficial Qwen3.8-27B Retrofit Runs 2.92x Faster at 261K Tokens
A community retrofit grafts Qwen Sparse Attention onto Qwen3.8-27B, nearly tripling prefill speed at 256k context while matching dense quality.
- Community retrofit adds Qwen Sparse Attention to Qwen3.8-27B, previewing a core Qwen4 technique on an existing checkpoint.
- At 261k tokens, QSA delivers 2.9x faster prefill and 1.8x faster decode than dense.
- Uses a small indexer scoring 4-token blocks with a fixed 2048-token attention budget per query.
- BF16 perplexity ratio vs dense is 0.99978 on 32k tokens; broad quality unverified.
- Ships as Q8_0 GGUF plus a patched llama.cpp runtime; stock llama.cpp cannot run it.
- Prebuilt binary targets Linux x86_64 and NVIDIA sm70; other hardware needs source build.
An independent developer has released an unofficial Qwen Sparse Attention retrofit for Qwen3.8-27B. The model repository contains a quantized multimodal checkpoint that converts the model’s 16 full-attention layers to QSA. In the author’s test at 261,120 input tokens, it processed prompts 2.92 times as fast and generated tokens 1.79 times as fast as the dense baseline. Two narrow BF16 checks produced effectively unchanged results.
The release packages 29.3 GB of Q8_0 weights and a prebuilt runtime from a patched llama.cpp fork. Upstream llama.cpp lacks the required QSA execution path. The author describes the project as an early implementation of a technique expected to appear in Qwen4.
QSA caps the expensive attention work
QSA debuted with Qwen3.8-Flash-Next, which combines 48 Gated DeltaNet layers with 16 global-attention layers. Gated DeltaNet carries history in a fixed-size recurrent state, while the attention layers retrieve specific earlier tokens from the key/value cache. Dense attention reads every available entry, so its compute and memory traffic rise as the context grows.
A compact indexer narrows those reads before grouped-query attention runs. The indexer uses 4 heads, a dimension of 128, and 1 KV head. It pools keys into four-token blocks, scores them, and retains the highest-ranking blocks within a selection budget of 2,048, plus the incomplete current block. The indexer still scans the pooled history, but the costlier grouped-query attention path reads a bounded selection.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
