LeCun's I-JEPA Beats BYOL and MAE on Robustness in a New SSL Showdown
A new taxonomy splits self-supervised learning into three families and shows predictive latent methods trade raw similarity for stronger robustness.
Takeaways
- New paper proposes Predictive Representation Learning (PRL) as a third SSL category alongside alignment and reconstruction.
- JEPA is positioned as the canonical PRL model, predicting latent embeddings of masked regions from context.
- I-JEPA scored 0.788 on occlusion robustness versus MAE's 0.550 and BYOL's 0.750.
- MAE hit perfect 1.000 augmentation similarity but showed the weakest robustness of the three.
- Published benchmarks: I-JEPA reaches 72.8% Top-1 on ImageNet-1K with a ViT-H/14 linear probe.
- Open problems include theoretical guarantees, long-horizon prediction, and better evaluation of latent prediction quality.
Self-supervised learning has settled into two dominant camps: methods that pull augmented views together in latent space, and methods that mask and reconstruct pixels. A new survey argues that a third camp deserves its own name, and that its poster child, Yann LeCun's Joint-Embedding Predictive Architecture, deserves separation from the first two rather than being lumped in as a variant.
The paper from Dutta, Vyas, and Roy proposes what the authors call Predictive Representation Learning (PRL), reorganizes the SSL landscape around it, and backs the argument with a head-to-head between BYOL, MAE, and I-JEPA. Predictive methods lose on augmentation similarity but win on robustness, which the authors read as evidence they capture structure the other two miss.
The problem with the current split
Most current SSL methods either align representations across augmented views or reconstruct corrupted inputs. Both work well, but both learn from data that is directly available to the model rather than modeling predictive structure over the data distribution. A contrastive loss teaches invariance; a reconstruction loss teaches fidelity to pixels. Neither explicitly trains the model to anticipate what it has not seen.
PRL carves out that third objective explicitly. An unlabeled sample is partitioned into an observed context and an unobserved target. A context encoder embeds the context, a target encoder embeds the target, and training minimizes the distance between the predicted and target embeddings, without negative sampling and without reconstructing the original input.