MIT's SMT Trains Powerful RNNs Without Unrolling, Matching Transformer Speed
MIT researchers train nonlinear RNNs without ever unrolling them, replacing BPTT with a Transformer-supervised memory labeling scheme that eliminates vanishing gradients entirely.
PRO- MIT researchers propose Supervised Memory Training (SMT), a new algorithm that trains nonlinear RNNs without ever using backpropagation through time (BPTT).
- SMT uses a Transformer encoder to generate "memory labels" at each timestep, then trains the RNN via simple supervised learning on one-step transitions — eliminating vanishing/exploding gradients entirely.
- The gradient credit path in SMT is O(1) regardless of sequence length, versus O(T) in BPTT, making training fully time-parallel and hardware-friendly.
- SMT→DMT outperforms BPTT across all five synthetic diagnostic tasks and produces dramatically better pixel-sequence generation on MNIST and Sketchy datasets.
- The paper introduces compression as a new scaling axis: given more compute, SMT models achieve the same performance with a smaller memory state — a property Transformers lack entirely.
- Code and project page are available at github.com/akarshkumar0101/smt and akarshkumar.com/smt.
Recurrent neural networks have always carried a paradox at their core: the very mechanism that makes them powerful , maintaining memory across time , is also what makes them a nightmare to train. A new paper from MIT, "Pretraining Recurrent Networks without Recurrence," proposes a fundamentally different training algorithm called Supervised Memory Training (SMT) that sidesteps the problem entirely, and the results suggest it could unlock a new generation of efficient, expressive sequence models.
The 30-Year-Old Problem with Training RNNs
To understand why SMT matters, you need to understand what has always been broken about the standard approach. Training recurrent neural networks requires assigning credit across long sequences of computations. Standard backpropagation through time (BPTT) addresses this problem poorly: it is sequential in time, limiting parallelism, and suffers from vanishing or exploding gradients, making long-range associations difficult to learn.
Concretely, BPTT works by "unrolling" the RNN across every timestep and then propagating gradients backward through the entire chain. The lack of time-parallelism makes BPTT scale poorly, while its gradient instability makes learning long-range associations difficult, as credit must propagate across up to O(T) steps. For a sequence of length 1,000, that means gradients must travel through 1,000 matrix multiplications before reaching the beginning , and they either vanish to zero or explode to infinity along the way.
The field has tried to work around this for decades. LSTMs and GRUs added gating mechanisms to slow gradient decay. Linear attention models like Mamba and S4 achieved parallelism by restricting the transition function to be linear , but because their transition function is linear, the class of functions they can represent is fundamentally constrained, which can lead to failure on important sequential tasks such as state tracking. Transformers solved parallelism and credit assignment simultaneously, but at a steep cost: Transformers store the entire history of past token representations and attend to all of them when processing each new token. As a result, their memory size grows with sequence length, leading to prohibitive computational costs for unbounded sequences.
The Key Insight: Separate "What to Remember" from "How to Update"
The core insight behind SMT is elegant. Instead of asking an RNN to simultaneously figure out
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
