Baidu's SGT Loops Only 2 Attention Heads to Cut Training Overhead by 16x

Making Transformers effectively deeper without inflating parameter counts has become one of the more interesting workarounds to the scaling wall. The standard trick loops entire blocks, running the same weights multiple times to simulate a deeper model. It works, but burns a lot of compute doing so. A new paper from researchers at Baidu and the Chinese Academy of Sciences proposes a more surgical approach: loop only the two attention heads per layer that actually matter, and grow that recurrence structure gradually during training.

The system, called Sparse Growing Transformer (SGT), shaves the added training FLOPs down from the 16 to 20 percent overhead of block looping to just 1 to 3 percent, while matching or beating the block-loop baseline on reasoning benchmarks.

Why looping whole blocks wastes compute

Recurrence has become a popular way to increase a Transformer's effective computational depth without adding parameters. Existing recursive architectures adopt a training-time static topology and uniformly apply block-level reuse at the parameter level, lacking fine-grained differentiation of parameter roles. Every weight in the block gets replayed the same number of times, whether it contributes meaningfully or not.

The authors argue this is wasteful for two reasons. First, not every parameter benefits equally from being reapplied. Second, the network is treated as static across training when in reality different layers mature at different rates.