---
title: "Tsinghua's SMELT Cuts AI Training Costs 18% by Looping Layers Twice"
subtitle: "New scaling laws show that looping the middle half of a Mixture-of-Experts model twice saves up to 18% of training compute at matched budgets."
url: "https://alphasignal.ai/news/tsinghua-s-smelt-cuts-ai-training-costs-18-by-looping-layers-twice"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "AlphaSignal"
company_url: "https://alphasignal.ai/company/alphasignal"
published: "2026-09-01T14:52:57.000Z"
updated: "2026-09-08T00:09:13.029Z"
type: "Paper"
topics:
  - "Llms"
  - "Training Infra"
subtopics:
  - "Mixture Of Experts"
  - "Long Context"
  - "Scaling Laws"
upvotes: 454
source_url: "https://arxiv.org/abs/2609.01343"
comments: 0
truncated: true
---

# Tsinghua's SMELT Cuts AI Training Costs 18% by Looping Layers Twice

> New scaling laws show that looping the middle half of a Mixture-of-Experts model twice saves up to 18% of training compute at matched budgets.

Researchers from Tsinghua University, ByteDance Seed, M-A-P, and TokenWave.AI have [published a paper](https://arxiv.org/abs/2609.01343) settling a persistent question about looped Transformers: do they actually help when compute budgets are held equal? Their answer, backed by a scaling ladder up to 54 billion non-embedding parameters, is yes, and the advantage grows with model size.

The architecture they landed on is called **SMELT** (Sparse MoE Transformer, middle layers Loop Twice). It repeats the middle 50% of a Mixture-of-Experts layer stack twice while holding per-token FLOPs, total parameters, and KV cache constant against an unlooped baseline. Under that strict matching, SMELT saves 6.8 to 18.0% of training FLOPs on the compute-optimal frontier.

## Why earlier looping numbers were misleading

Looped Transformers reuse a shared block of layers to increase effective depth without adding new parameters. Looping a 12-layer model to 24 executed layers stores half the weights, yet spends roughly a 24-layer model's per-token FLOPs and needs its full KV cache. Prior work compared looped models against baselines with the same stored parameter count, quietly giving the looped model extra compute. When a competing paper controlled for FLOPs on dense models, it found that r recurrences contribute like r^0.46 unique-block equivalents, suggesting looping loses at matched compute. The SMELT authors argue that result conflates the cost of weight tying with the loss of unique parameters.

_Preview only. Full text requires an AlphaSignal Pro account._

## Sources

- [SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers](https://arxiv.org/abs/2609.01343)
- [SMELT paper HTML](https://arxiv.org/html/2609.01343v1)
- [Relaxed Recursive Transformers](https://openreview.net/forum?id=WwpYSOkkCt)
- [Mixture-of-Recursions](https://openreview.net/forum?id=uyTL5Bvosj)