Empero Distills Qwen3.8 Reasoning Into a Lean 35B Open Model
Empero distilled Qwen3.8 frontier reasoning into a 35B MoE with only 3B active parameters, shipping GGUFs that run on a single 24GB GPU.
- Empero released Qwen3.8-35B-A3B-Distill, their first MoE distillation, distilling Qwen3.8 reasoning into Qwen3.6-35B-A3B.
- 35B total parameters, only 3B active per token, deployable on a single 24 GB GPU.
- ARC-Challenge climbs from 0.548 to 0.591; MMLU holds flat at 0.834 (within noise).
- GGUF quants from Q4_K_M (21.7 GB) up to BF16 for llama.cpp, Ollama, LM Studio.
- Hybrid architecture: 30 Gated DeltaNet layers, 10 full-attention layers, 256 experts routed 8-per-token.
- Apache-2.0 licensed; every response opens with a distilled
<think>block from the teacher.
Empero distills Qwen3.8 reasoning into a 35B MoE
Empero has released Qwen3.8-35B-A3B-Distill, its first Mixture-of-Experts model. The community release uses the sparse Qwen3.6-35B-A3B architecture and supervised training on reasoning traces generated by Qwen3.8 teachers. Ready-made GGUF files support local inference through compatible versions of llama.cpp, Ollama, LM Studio, Jan, and KoboldCpp.
35B weights, 3B active
Knowledge distillation trains a student model on outputs from larger teacher models. Empero says it curated and filtered traces covering mathematics, code, general reasoning, instruction following, and tool use. The resulting student activates about 3 billion parameters for each token, reducing computation relative to a dense 35B model.
The architecture contains 30 Gated DeltaNet layers and 10 full-attention layers. Gated DeltaNet is a linear-attention design intended to reduce the sequence-length cost of standard attention. The model also has 256 experts, with the router selecting eight for each token.
Deployment requires a recent llama.cpp build with Qwen3.6, Gated DeltaNet, and MoE support. Older builds cannot interpret the architecture and may reject the model during loading. Applications built on Ollama, LM Studio, or another frontend also need a current bundled inference backend.
ARC rises; MMLU holds
Empero evaluated the base and distilled models with lm-evaluation-harness in bfloat16, using the same settings and random seed. Higher scores are better.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
