UC Berkeley Study Shatters the Myth That Vision and Language AI Think Alike
A new paper from UC Berkeley and TU Munich shows the Platonic Representation Hypothesis collapses when tested at realistic data scales, challenging a foundational belief in multimodal AI.
Takeaways
- A UC Berkeley/TU Munich paper challenges the Platonic Representation Hypothesis, showing cross-modal alignment collapses at realistic data scales.
- Mutual kNN alignment scores drop from ~0.135 at 1K samples to ~0.001 at 15M samples, nearly vanishing entirely.
- On ImageNet, vision and language models individually retrieve correct-class neighbors ~50% of the time, but agree on the exact same item only ~11% of the time.
- The original trend that stronger LLMs align better with vision does not hold for 36 recent models tested across reasoning benchmarks (R² as low as -1.75).
- The one-to-one image-caption assumption in the original evaluation is unrealistic; relaxing it to many-to-many further reduces measured alignment.
- A recaptioned WIT-1M dataset using Gemini is released; no model weights or code repo provided.
One of the most seductive ideas in modern AI is that vision models and language models, despite seeing completely different kinds of data, are secretly converging on the same internal map of reality. This idea, known as the Platonic Representation Hypothesis, was put forward by Huh et al. at ICML 2024 and quickly became a touchstone for researchers arguing that modality choice is becoming irrelevant. A new paper from UC Berkeley, TU Munich, and the University of Tübingen throws cold water on that claim, showing the evidence was built on a methodological house of cards.
The core argument of the Platonic Representation Hypothesis is elegant: neural networks, trained with different objectives on different data and modalities, are converging to a shared statistical model of reality in their representation spaces. If true, this would mean a language model and a vision model, trained independently, would end up organizing the world in the same way. As vision models and language models get larger, they measure distance between datapoints in a more and more alike way, converging toward a shared statistical model of reality, akin to Plato's concept of an ideal reality. The hypothesis has been widely cited as justification for language-only learning pipelines, with some researchers taking it to mean that pixels are simply unnecessary.
The Measurement Problem
The original evidence rested on a specific metric: mutual k-nearest-neighbors (mutual kNN). The idea is straightforward. Take a set of image-caption pairs. Embed the images with a vision model and the captions with a language model. For each sample, find its k nearest neighbors in image space and in text space. If the two models agree on the same neighbors, their representations are considered aligned. A score of 1 means perfect agreement; 0 means none.
The experiments in Huh et al. rest on two foundations that warrant scrutiny: the use of mutual kNN alignment on a small evaluation set of only 1024 samples from the Wikipedia Image-Text (WIT) dataset, and the use of data with bijective (one-to-one) image-text correspondences. The new paper, titled "Back into Plato's Cave,"