A 90% Pruned Qwen3.6 Scores 15/15 Green While Being Completely Broken
A deliberately broken Qwen3 MoE model passes 15/15 validation checks after 90% of its experts are deleted, exposing a systemic flaw in how compressed models get evaluated.
PRO- A researcher published a deliberately broken Qwen3.6-35B MoE model with 90% of its experts deleted to expose flawed validation practices.
- The 90%-pruned model scored 15/15 on standard validation checks used by real pruned-model cards on Hugging Face.
- When scored for actual correctness, the same model got 1/5 -- it cannot multiply two numbers or name a capital city.
- Standard "pass" checks only verify that the model returned a completion of roughly the right shape, not that the answer is correct.
- The REAP pruning method from Cerebras (published at ICLR 2026) is legitimate at ~50% compression; the criticism targets how the community validates pruned checkpoints, not REAP itself.
- The fix: publish exact prompts, expected outputs, and scoring code -- the repo includes
rubric_bench.pyas a template for honest evaluation.
Someone just published a model to Hugging Face that is, by design, completely useless. It cannot multiply two numbers, name a capital city, or write a function that runs. And yet, it passes every single check on the kind of validation table you see on dozens of pruned-model cards. That is the entire point.
The setup: MoE compression is booming
To understand why this matters, you need to know how Mixture-of-Experts (MoE) models work. Instead of running every parameter on every input, a MoE model routes each input token through a small subset of specialized sub-networks called "experts," while the rest of the parameters sit idle. This means you can have an enormous total parameter count while keeping inference compute proportional only to the active parameter count.
The base model here, Qwen3.6-35B-A3B, is a strong example of this architecture. It has 35B total parameters but only 3B activated, with 256 experts total and 8 routed plus 1 shared active per token. It surpasses its predecessor Qwen3.5-35B-A3B by a wide margin and rivals much larger dense models like Qwen3.5-27B and Gemma4-31B.
Because MoE models are large in memory but cheap to run, there is a growing ecosystem of tools to compress them further by removing the least-used experts. The technique used here is REAP (Router-weighted Expert Activation Pruning), a method from Cerebras Research. REAP is a pruning criterion that considers both router gate-values and expert activation norms to minimize the reconstruction error bound. Across SMoE models ranging from 20B to 1T parameters, REAP consistently outperforms merging and other pruning methods on generative benchmarks, achieving near-lossless compression on code generation and tool-calling tasks even after pruning 50% of experts.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
