EPFL Proves Latent Prediction Closes AI's Trillion-Token Gap Exponentially
A new theory proves that predicting your own latent representations requires exponentially less data than token-level objectives like masked language modeling or diffusion.
- A new paper proves that predicting your own latent representations (JEPA, data2vec) requires O(m³) samples, independent of hierarchy depth, versus O(m^(L+1)) for token-level SSL.
- The exponential gap arises because token-level objectives dilute the learning signal across every level of the data hierarchy; latent prediction avoids this by bootstrapping level by level.
- The result is confirmed three ways: a hierarchical clustering algorithm (ILC), an end-to-end neural network (SLC), and the first sample-complexity analysis of data2vec.
- data2vec implicitly performs hierarchical latent prediction through its EMA teacher, making explicit stacking like H-JEPA largely redundant according to the theory.
- The SLC network achieves the same sample complexity with purely local learning rules (stop-gradients between modules), connecting to biologically plausible predictive coding theories.
- Experiments are on the synthetic Random Hierarchy Model; whether the exponential gap holds on real language/image data is an open empirical question. No code released yet. Paper on arXiv.
Frontier LLMs are trained on trillions of tokens. A human child reaches adult-level language competence after exposure to roughly a billion words. That five-order-of-magnitude gap has long been treated as an awkward empirical fact, but a new paper from researchers at EPFL, Cambridge, and Johns Hopkins now offers a rigorous theoretical explanation for why token-level pretraining is so data-hungry, and proves that a different approach can close the gap exponentially.
The paper, "Learn from your own latents and not from tokens," focuses on a class of self-supervised methods that predict a network's own internal representations rather than raw input tokens. This includes methods like data2vec and JEPA (Joint Embedding Predictive Architecture), which have shown strong empirical results but lacked a theoretical foundation explaining why they work better. The paper provides that foundation, and the answer is stark: the improvement is not incremental, it is exponential.
The token-level bottleneck, explained
To make the analysis tractable, the authors use a synthetic data model called the Random Hierarchy Model (RHM). Think of it as a formal grammar that generates sequences by recursively applying rules down a tree of hidden (latent) symbols. The tree has depth L. The visible tokens you observe are only the leaves; everything above them is hidden. This structure is a reasonable model of how both language and images are compositionally organized.
The key insight is about what it costs, in terms of training samples, to recover each level of this hidden tree. The paper proves the following:
- Supervised learning requires a number of samples that grows as
m^L, wheremis the number of production rules per symbol. Exponential in depth. - Token-level SSL (masked language modeling, diffusion) requires
m^(L+1)samples. Also exponential in depth, and even worse than supervised.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
