Source
Huh et al. (2024), LIT-458 — ARXIV-2405.07987.
Two claims, and the record should keep them apart
Both come from LIT-458: the measurement that vision and language models measure distances between datapoints more alike as they grow, and the conjecture, proved only for bijective observations, that they converge on a model of what generated the data.
The convergence is measured. Vision models and language models agree increasingly about pairwise distances between datapoints as they get larger, and the agreement spans architectures, objectives and datasets. Prior results line up with it: a model trained on ImageNet aligns with one trained on Places-365 with performance retained; early layers are more interchangeable than later ones; same-architecture models land in the same weight basin up to permutation, which is why merging works at all.
The endpoint is conjectured. That models are converging on a representation of the world that generated the data — the platonic representation — is a hypothesis with a proof in an idealised setting only. In a world of discrete events with bijective observation functions, contrastive learners converge to the same kernel whichever projection they see. Real observation is neither discrete nor bijective.
A document that ran the two together would be claiming a conjecture is
measured, which is the available mistake here and the reason this is
Proposed rather than Active while its first half is well supported.
What could have gone otherwise
The convergence could have been an artefact of shared training data, shared architecture or shared inductive bias; the interesting cases — a vision model and a language model with none of those in common — are the ones where it persists.
And the authors probe their own assumption rather than around it. Bijectivity predicts that a more informative observation should align better. Varying caption density on fixed images: denser captions align better with the visual representation. That is the prediction the weakest premise makes, measured, in the direction that would have embarrassed them.
What this does not say
It does not say two models converge when they saw different information. Language can state “I believe in the freedom of speech” and an image cannot. The authors propose the nuanced form themselves: alignment capped by the mutual information between the signals and by each model’s capacity. Whether that cap matters in practice is open — CLIP is optimised to capture only the shared information and is nonetheless strong on pure vision tasks.
It does not say convergence is happening everywhere. Robotics has no standardised world-state representation, which the authors attribute to a hardware-driven data bottleneck rather than to anything about the hypothesis. Two modalities is the evidence.
It does not establish the mechanism. Three pressures are offered — task diversity, data diversity, and the simplicity bias of larger models — as arguments, none isolated by intervention.
Alignment is pairwise-distance agreement, which is a weak notion of sameness. Two representations can agree on every pairwise distance and still differ in ways a downstream task would notice, and nothing here rules that out.
And the hallucination implication is not a finding. That scale should reduce hallucination and bias amplification follows from the conjecture and is explicitly conditioned on training data being a sufficiently lossless and diverse set of measurements. It is the claim most likely to be quoted without its condition.
What it underwrites
SOTA-271 — train on a second modality even when the target is
single-modality. That practice follows from the conjecture rather than from
the measurement, which is why it is Proposed too, and why its conditions
name this account’s bijectivity assumption rather than hiding behind it.
Why Proposed
Because the interesting half is the conjecture and the conjecture’s proof is
for a world nobody trains in. What is Active-grade here is “representations
are converging”, which on its own is a survey finding rather than an
explanation. promote_when: asks for the two things that would make the
account carry its own weight: a stronger notion of alignment that survives,
and a third modality entering the regime.