Korea University Reveals Why Reasoning AI Fails by Mapping Its Hidden Wiring
New circuit analysis of reasoning models reveals that SFT, GRPO, and think-on/off training each reshape attention heads in fundamentally different ways.
Takeaways
- Circuit analysis reveals post-training sparks new specialized attention heads, not just parameter tweaks. Paper link
- SFT and distillation cumulatively add stable heads in mid-to-late layers, ablating ~10 collapses benchmarks near zero.
- GRPO dynamically activates and prunes a small set of heads that track the reward signal.
- Qwen3 has no dedicated thinking heads; think-off mode recruits many redundant heads to compensate.
- Ablating overactivated think-off heads improves AIME'24 from 30.0 to 36.6 without expensive retraining.
- Strengthened reasoning heads enable hard problems but cause overthinking on simple arithmetic.
Anyone who has fine-tuned a reasoning model has watched benchmark numbers move without really knowing what changed inside the network. A new paper from Korea University and Upstage AI opens the hood, using circuit analysis to show that post-training for reasoning literally sparks new, functionally specialized attention heads into existence, and that different training recipes do this in strikingly different ways.
The authors track these emergent heads across supervised fine-tuning, distillation, GRPO reinforcement learning, and Qwen3's think on/off mode. They then connect the circuit-level changes to the failure modes practitioners actually see, including overthinking, calculation slips, and logic loops on easy problems.
Reasoning as a wiring diagram
The method treats the transformer as a directed graph where nodes are attention heads, MLPs, embeddings, and logits. To find the subgraph responsible for a given task, the authors use edge attribution patching with integrated gradients (EAP-IG), which improves faithfulness by ensuring that ablating all non-circuit edges preserves task performance. In plain terms, they run the model on a clean reasoning prompt and a corrupted one, then score every edge by how much it moves the output between the two. Edges that matter form the circuit.
By comparing circuits of a base model against a post-trained version on the same problem, they can identify "emergent heads," meaning attention heads that only become active after training. To confirm those heads actually do something, they zero out their outputs and watch benchmarks collapse.
SFT and distillation grow a thick new layer of machinery
The first surprise is how heavy-handed supervised fine-tuning is. SFT consistently activates many additional heads, most of which persist to the final checkpoint. About half emerge by step 100, and most are concentrated in middle-to-late layers. Distillation from DeepSeek-R1 into Qwen2.5-Math produces a similar pattern, adding specialized components on top of the pretrained foundation rather than replacing it.