Empero Shrinks a 2.4T Parameter Teacher Into a 2.78 GB Laptop Model
Empero distilled a 2.4 trillion parameter teacher into a 4B model that runs on laptops with 3GB of RAM, with a big MMLU jump.
Takeaways
- Empero released Qwen3.8-4B-Distill GGUF, a 4B reasoning model distilled from Qwen3.8 2.4T A95B.
- Q4_K_M build is 2.78 GB, runs on 4-6 GB GPUs or a decent CPU, Apache-2.0 licensed.
- MMLU CoT jumps from 0.354 to 0.553 versus the Qwen3.5-4B base, a 20 point gain.
- GSM8K regresses slightly from 0.850 to 0.785; code is weaker, use Qwen3.8-9B for coding.
- Trained on ~45,000 curated teacher traces as full-parameter SFT, not LoRA.
- Needs a recent llama.cpp with Qwen3.5 Gated DeltaNet support; 262K native context inherited from base.
Empero has released GGUF files for Qwen3.8-4B-Distill, a reasoning model based on the Qwen3.5-4B architecture. Its smallest recommended build occupies 2.78 GB on disk, making local inference practical on many laptops and consumer desktops.
Empero says it trained the 4B student on reasoning traces from Qwen3.8 2.4T A95B, a mixture-of-experts teacher with 2.4 trillion total parameters and 95 billion active per token. The release offers developers a compact, Apache 2.0-licensed model for testing local reasoning, long-context workloads, and function calling without a hosted inference API.
Five files, different memory bills
The repository provides five builds for llama.cpp-compatible runtimes, including Ollama, LM Studio, Jan, and KoboldCpp. Empero recommends Q4_K_M for general use.
| Build | File size | Intended use |
|---|---|---|
| Q4_K_M | 2.78 GB | Recommended balance of size and quality |
| Q5_K_M | 3.16 GB | Higher fidelity with a modest size increase |
| Q6_K | 3.56 GB | High-fidelity quantized inference |
| Q8_0 | 4.61 GB | Largest quantized build |
| BF16 | 8.67 GB | Unquantized reference weights |
The listed sizes cover model files only. Runtime memory also includes context buffers, backend overhead, and GPU allocations. A 4 GB GPU may require a shorter context, smaller batch, or partial CPU offload, while CPU inference needs enough system RAM for the weights and working memory.
Teacher traces shape the student
Off-policy distillation trains a smaller model against outputs produced by a larger teacher. Empero used about 45,000 curated traces covering mathematics, general reasoning, and instruction following, then updated every student parameter through full fine-tuning. The result is a standalone checkpoint rather than a LoRA adapter.