Michigan Proves Why Training AI on More Tasks Makes It Learn Faster
A new theoretical framework proves why training on more diverse tasks makes transformers learn faster and generalize further — explained through low-dimensional geometry.
Takeaways
- A new paper proves mathematically why training on diverse tasks speeds up ICL learning and enables out-of-distribution generalization in transformers.
- Task diversity is defined geometrically: more non-overlapping subspace dimensions across task families = faster escape from loss plateaus.
- OOD generalization is explained as in-span generalization: models handle unseen tasks that lie within the span of training subspaces, not beyond it.
- Results are proven for single-layer linear attention and validated empirically on GPT-2 and nonlinear function classes.
- Practical implication: mixing structurally different task families during training expands the model's generalization boundary and accelerates convergence.
- No code or models are released; this is a theoretical contribution unifying prior empirical observations. Read the paper.
One of the most counterintuitive facts in modern deep learning is that training a transformer on more tasks simultaneously can make it learn faster on each individual task. Researchers have observed this for years, but the theoretical explanation has remained elusive. A new paper from researchers at the University of Michigan provides the first clean mathematical proof of why this happens, and in doing so, also explains a second mystery: why models trained on one distribution can sometimes generalize to tasks they have never seen.
The Two Puzzles This Paper Solves
In-context learning (ICL) is the ability of a transformer to adapt to a new task just from examples in its prompt, without any weight updates. Think of it as the model reading a few input-output pairs and immediately figuring out the pattern. This capability is central to how large language models like GPT-4 work in practice.
Two empirical observations about ICL training have been well-documented but poorly understood:
- The plateau problem: Training consistently shows long loss plateaus, during which models exhibit minimal improvement, followed by a sudden, rapid surge of learning. Training on multiple diverse ICL tasks simultaneously shortens these loss plateaus, making each task easier to learn. This finding is surprising as it contradicts the natural intuition that the combined complexity of multiple ICL tasks would lengthen the learning process, not shorten it.
- The OOD generalization puzzle: ICL is a remarkable capability of pretrained transformers that allows models to generalize to unseen tasks after seeing only a few examples. As task diversity increases, transformers undergo a transition from a specialized solution, which exhibits ICL only within the pretraining task distribution, to a solution which generalizes out of distribution to the entire task space. But no one had proven why this transition happens.
Prior theoretical work made the task vector definition intractable for analysis, restricting studies to empirical results. While prior works highlight and advocate for task diversity in training transformers, there was no clear theoretical justification for when and how task diversity specifically helps.

The Core Idea: Tasks Live in Subspaces
The key insight is geometric. Instead of thinking of tasks as arbitrary functions, the paper models each task as a vector drawn from a low-dimensional subspace of the full parameter space. Concretely, the authors represent training task vectors as samples from a