---
title: "MIT's SMT Trains Powerful RNNs Without Unrolling, Matching Transformer Speed"
subtitle: "MIT researchers train nonlinear RNNs without ever unrolling them, replacing BPTT with a Transformer-supervised memory labeling scheme that eliminates vanishing gradients entirely."
url: "https://alphasignal.ai/news/mit-s-smt-trains-powerful-rnns-without-unrolling-matching-transformer-speed"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "AlphaSignal"
company_url: "https://alphasignal.ai/company/alphasignal"
published: "2026-06-04T00:00:00.000Z"
updated: "2026-06-15T01:09:59.039Z"
type: "Paper"
topics:
  - "Training Infra"
  - "Llms"
subtopics:
  - "Distributed Training"
  - "Pretraining"
  - "Long Context"
upvotes: 45
source_url: "https://arxiv.org/abs/2606.06479"
comments: 0
truncated: true
---

# MIT's SMT Trains Powerful RNNs Without Unrolling, Matching Transformer Speed

> MIT researchers train nonlinear RNNs without ever unrolling them, replacing BPTT with a Transformer-supervised memory labeling scheme that eliminates vanishing gradients entirely.

Recurrent neural networks have always carried a paradox at their core: the very mechanism that makes them powerful , maintaining memory across time , is also what makes them a nightmare to train. A new paper from MIT, ["Pretraining Recurrent Networks without Recurrence,"](https://arxiv.org/abs/2606.06479) proposes a fundamentally different training algorithm called **Supervised Memory Training (SMT)** that sidesteps the problem entirely, and the results suggest it could unlock a new generation of efficient, expressive sequence models.

## The 30-Year-Old Problem with Training RNNs

To understand why SMT matters, you need to understand what has always been broken about the standard approach. Training recurrent neural networks requires assigning credit across long sequences of computations. Standard backpropagation through time (BPTT) addresses this problem poorly: it is sequential in time, limiting parallelism, and suffers from vanishing or exploding gradients, making long-range associations difficult to learn.

Concretely, BPTT works by "unrolling" the RNN across every timestep and then propagating gradients backward through the entire chain. The lack of time-parallelism makes BPTT scale poorly, while its gradient instability makes learning long-range associations difficult, as credit must propagate across up to O(T) steps. For a sequence of length 1,000, that means gradients must travel through 1,000 matrix multiplications before reaching the beginning , and they either vanish to zero or explode to infinity along the way.

The field has tried to work around this for decades. LSTMs and GRUs added gating mechanisms to slow gradient decay. Linear attention models like Mamba and S4 achieved parallelism by restricting the transition function to be linear , but because their transition function is linear, the class of functions they can represent is fundamentally constrained, which can lead to failure on important sequential tasks such as state tracking. Transformers solved parallelism and credit assignment simultaneously, but at a steep cost: Transformers store the entire history of past token representations and attend to all of them when processing each new token. As a result, their memory size grows with sequence length, leading to prohibitive computational costs for unbounded sequences.

## The Key Insight: Separate "What to Remember" from "How to Update"

The core insight behind SMT is elegant. Instead of asking an RNN to simultaneously figure out

_Preview only. Full text requires an AlphaSignal Pro account._

## Sources

- [Pretraining Recurrent Networks without Recurrence (arXiv)](https://arxiv.org/abs/2606.06479)
- [Full paper HTML (arXiv)](https://arxiv.org/html/2606.06479)
- [SMT Source Code (GitHub)](https://github.com/akarshkumar0101/smt)
- [Backpropagation Through Time — limitations overview](https://medium.com/@lakshmi.pallempati22/backpropagation-through-time-bptt-7f32037a2699)
- [Gradient-free training of recurrent neural networks (Frontiers in Neuroscience)](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11267880/)