AgentionAI's Gyro-Coder Squeezes a 125B Coding Model onto a 24GB GPU
A 125B mixture-of-experts coding model squeezed onto a single 24GB consumer GPU through evolutionary expert pruning and 2.125-bit trellis quantization.
- AgentionAI released Gyro-Coder, a 125B Qwen3.8-Flash-Next MoE pruned and quantized to fit a 24GB GPU.
- Each layer keeps 256 of 512 experts, selected by evolutionary search over a 10^7328 combination space.
- Experts stored at 2.125 bits per weight using in-house trellis-coded quantization in the QTIP family.
- 93.1% top-1 agreement with the full model on held-out Python; passes 13 of 16 auto-graded coding tasks.
- Needs AgentionAI's custom llama.cpp build; stock llama.cpp cannot load the file.
- General knowledge degrades noticeably; use Gyro-S or Gyro-M for open-ended tasks.
AgentionAI has released Gyro-Coder weights, a coding-focused derivative of Alibaba’s base Qwen model. The release prunes half of the mixture-of-experts networks and quantizes the remaining expert weights to an average of 2.125 bits, reducing GPU-resident weights to 19.92 GiB.
The cited GPU footprint rises to about 21.1 GiB with a 64k q8_0 key-value cache, which stores attention state for earlier tokens. That budget fits 24GB cards such as the RTX 3090, RTX 4090, and Radeon RX 7900 XTX, although display use and runtime overhead leave limited headroom.
The release ships as a single 70.6 GiB GGUF file under the Qwen community license. Its storage requirements, custom runtime, and experimental format make it more involved than loading a conventional llama.cpp quantization.
Expert pruning cuts the retained model in half
Mixture-of-experts models contain many specialized subnetworks but route each token through only a small selection. Total parameter count therefore describes the model’s storage scale, while the routed experts determine much of its per-token computation.
Qwen3.8-Flash-Next has 512 experts per MoE layer. Gyro-Coder retains 256 per layer, producing a model with roughly 65 billion retained parameters. The router still activates only a subset for each token.
AgentionAI first mapped experts to domains, then used evolutionary search to find combinations that preserved programming behavior. Choosing 256 experts from every layer creates a search space of roughly 107,328 possible masks. The team evaluated about 2,200 candidates across 14 search runs, built the strongest candidates as complete models, and tested them on coding, repetition, and knowledge tasks.
The search must consider the whole network because removing experts in one layer changes the hidden states received by every later layer. Each promising mask therefore requires full-model construction and evaluation rather than isolated scoring of individual experts.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
