Solo Researcher Turns Qwen3.8 Into a Sparse MoE Running on One GPU

A self-funded research preview on Hugging Face is doing something most labs would need a cluster to attempt: turning a dense 27B model into a sparse mixture of experts after the fact, without retraining any expert weights. The Qwen3.8-Whittle-MoE-27B-A17.8B release from author logic65 is a post hoc MoE carved out of Qwen3.8-27B that runs on a single 24GB GPU and has climbed to nearly 15,000 downloads in its first weeks.

What makes it worth a look is the surgical method used to convert the model, and the finding that training the routers alone was enough to bring a broken partition back to coherent output.

Carving experts out of a dense FFN

The recipe reads more like arithmetic than architecture. All 64 layers keep the parent's attention side untouched, including the 3:1 hybrid of gated deltanet layers and full attention, 16 attention layers in all, at hidden size 5120. The surgery is entirely in the feed forward network.

Each layer's dense FFN, 17408 wide in the parent, was cut into 64 routed expert slivers of width 192 plus one always-on shared expert of width 5120. The math is exact: 64 × 192 + 5120 = 17408, the original neuron count, with zero new FFN weights invented. A small router per layer picks 16 of the 64 slivers per token, so each token runs 8192 of the original 17408 FFN width, giving 17.8B active parameters out of 27B total.

Reviving gibberish with router training

Freshly carved, the model was gibberish, scoring 4 of 39 on the author's knowledge battery. Training only the 64 routers with every expert frozen recovered it to 27 of 39, meaning the knowledge never left the slivers. The routers just had to learn which ones to fire.