Yann LeCun's 109M Model Beats 72B Giants at Cross-Modal Alignment

Randall Balestriero and Yann LeCun have given a constrained mathematical form to one of representation learning’s broadest conjectures. In their paper, they prove that Joint-Embedding Predictive Architectures trained on suitable paired observations can recover equivalent representations across modalities, up to an orthogonal transformation. Their 109-million-parameter text encoder also posts stronger vision-alignment scores than 70B and 72B language models on the reported benchmark. The result identifies pair construction as a controllable source of cross-modal alignment and offers a way to connect pretrained encoders without end-to-end multimodal training.

The original hypothesis, introduced by Huh, Cheung, Wang, and Isola at ICML 2024, proposed that models trained with different architectures, datasets, modalities, and objectives often learn similar internal representations. These representations are the vectors models use to encode relationships among inputs. The earlier work supported its claim with empirical observations and left two gaps: a proof and a recipe for producing aligned representations deliberately.

The Theorem Lives in the Pair

The paper calls its result the Platonic Representation Theorem for Joint-Embedding Predictive Architectures, or JEPAs. A JEPA learns by predicting the embedding of one view of an input from the embedding of another. Under the theorem’s exact conditions, JEPAs spanning any number of modalities recover the same geometry through a single orthogonal transformation. That transformation changes the coordinate system while preserving dot products and distances. The approximate case permits bounded differences between modalities and provides a corresponding bound on misalignment.

Good views

Two observations expose the same underlying scene or state while preserving its shared structure. Examples include two captions of the same photograph or nearby frames from the same video.

Bad views

The paired observations contain substantial slice-specific information that the other side cannot recover. The paper’s main text example pairs a document prefix with its continuation.