---
title: "Baidu's SGT Loops Only 2 Attention Heads to Cut Training Overhead by 16x"
subtitle: "A new looping strategy targets only the highest-entropy attention heads, cutting extra training compute from 16-20% down to just 1-3%."
url: "https://alphasignal.ai/news/baidu-s-sgt-loops-only-2-attention-heads-to-cut-training-overhead-by-16x"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "AlphaSignal"
company_url: "https://alphasignal.ai/company/alphasignal"
published: "2026-04-14T00:00:00.000Z"
updated: "2026-08-23T22:24:10.829Z"
type: "Paper"
topics:
  - "Training Infra"
  - "Llms"
subtopics:
  - "Scaling Laws"
  - "Distributed Training"
  - "Mixture Of Experts"
upvotes: 1755
source_url: "https://arxiv.org/abs/2603.23998"
comments: 0
truncated: true
---

# Baidu's SGT Loops Only 2 Attention Heads to Cut Training Overhead by 16x

> A new looping strategy targets only the highest-entropy attention heads, cutting extra training compute from 16-20% down to just 1-3%.

Making Transformers effectively deeper without inflating parameter counts has become one of the more interesting workarounds to the scaling wall. The standard trick loops entire blocks, running the same weights multiple times to simulate a deeper model. It works, but burns a lot of compute doing so. A new paper from researchers at Baidu and the Chinese Academy of Sciences proposes a more surgical approach: loop only the two attention heads per layer that actually matter, and grow that recurrence structure gradually during training.

The system, called [Sparse Growing Transformer (SGT)](https://arxiv.org/abs/2603.23998), shaves the added training FLOPs down from the 16 to 20 percent overhead of block looping to just 1 to 3 percent, while matching or beating the block-loop baseline on reasoning benchmarks.

## Why looping whole blocks wastes compute

Recurrence has become a popular way to increase a Transformer's effective computational depth without adding parameters. Existing recursive architectures adopt a training-time static topology and uniformly apply block-level reuse at the parameter level, lacking fine-grained differentiation of parameter roles. Every weight in the block gets replayed the same number of times, whether it contributes meaningfully or not.

The authors argue this is wasteful for two reasons. First, not every parameter benefits equally from being reapplied. Second, the network is treated as static across training when in reality different layers mature at different rates.

_Preview only. Full text requires an AlphaSignal Pro account._

## Sources

- [Sparse Growing Transformer: Training-Time Sparse Depth Allocation via Progressive Attention Looping](https://arxiv.org/abs/2603.23998)
- [Sparse Growing Transformer GitHub Repository](https://github.com/YaoChen0203/Sparse-Growing-Transformer)
- [SGT full paper HTML on arXiv](https://arxiv.org/html/2603.23998)