Tsinghua's SMELT Cuts AI Training Costs 18% by Looping Layers Twice
New scaling laws show that looping the middle half of a Mixture-of-Experts model twice saves up to 18% of training compute at matched budgets.
PRO- New paper introduces SMELT, a recipe that loops the middle half of MoE layers twice under strict budget matching.
- Saves 6.8 to 18.0% of training FLOPs on the compute-optimal frontier at matched parameters, FLOPs, and KV cache.
- Scaled across four sizes up to 54B non-embedding parameters with separate Chinchilla-style fits per architecture.
- Downstream benchmark gains exceed what validation loss predicts, with Code showing the largest 20.4% improvement.
- Advantage grows with sample length and number of in-context demonstrations, hinting at stronger ICL.
- Read the full paper on arXiv.
Researchers from Tsinghua University, ByteDance Seed, M-A-P, and TokenWave.AI have published a paper settling a persistent question about looped Transformers: do they actually help when compute budgets are held equal? Their answer, backed by a scaling ladder up to 54 billion non-embedding parameters, is yes, and the advantage grows with model size.
The architecture they landed on is called SMELT (Sparse MoE Transformer, middle layers Loop Twice). It repeats the middle 50% of a Mixture-of-Experts layer stack twice while holding per-token FLOPs, total parameters, and KV cache constant against an unlooped baseline. Under that strict matching, SMELT saves 6.8 to 18.0% of training FLOPs on the compute-optimal frontier.
Why earlier looping numbers were misleading
Looped Transformers reuse a shared block of layers to increase effective depth without adding new parameters. Looping a 12-layer model to 24 executed layers stores half the weights, yet spends roughly a 24-layer model's per-token FLOPs and needs its full KV cache. Prior work compared looped models against baselines with the same stored parameter count, quietly giving the looped model extra compute. When a competing paper controlled for FLOPs on dense models, it found that r recurrences contribute like r^0.46 unique-block equivalents, suggesting looping loses at matched compute. The SMELT authors argue that result conflates the cost of weight tying with the loss of unique parameters.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
