nucleation

THEORY-723: For the final embedding and unembedding of a softmax model, the output distribution fixes the representations only up to an invertible linear map, and whether closeness in distribution bounds representational dissimilarity depends on the distance: not for KL in the saturation regime, but yes for the log-likelihood variance distance

2 min read

Properties7
StatusActive
Tagsrepresentation-learning, mathematics
SourceLIT-253 — When Does Closeness in Distribution Imply Representational Similarity? An Identifiability Perspective, LIT-250 — Duality of Bures and Shape Distances with Implications for Comparing Neural Representations
Extended byTHEORY-002 — Convergence of representations, in the Platonic hypothesis's sense, is convergence of kernels, which fixes representations only up to the symmetry group of what is observed; whether closeness in distribution implies closeness of representation depends on the distance used
PresupposesTHEORY-004 — A representation is determined by its centred kernel up to a translation and an orthogonal transformation, and the rotation-aligned distance between two representations equals the Bures distance between their kernels
SupportsCLAIM-379 — On a finite set of stimuli, regularized linear readouts with ridge b > 0 cannot tell apart two centred representations of the same width exactly when one is an orthogonal transformation of the other, and the unregularized CCA member cannot tell apart representations with the same column space, which differ by an invertible linear map
Cited as “grounds” byCLAIM-193 — A linear probe family can tell apart only realizations whose representations differ in normalized kernel, and kernels fix a representation only up to the symmetry of what is observed: any distinction beyond that is supplied by the analyst, CLAIM-269 — A regularized linear readout can tell apart only representations that differ in normalized kernel, and kernels fix a representation only up to the symmetry of what is observed, so no such readout supplies a distinction beyond that symmetry

Graph View

Source

Nielsen et al. (2025), LIT-253, Thm 2.2 (proof in App. B), Thm 3.1 (constructive proof in App. C.1), Cor. 3.2 and Thm 4.7 (App. E.5–E.6), read in full in NOTE-738, whose claims table rates C1, C2, C3 and C5 “strong”. Harvey, Larsen & Williams (2023), LIT-250, for the kernel-side metric, read in NOTE-753. The orthogonal case is THEORY-004.

What was actually shown

What an observation fixes. For a softmax model p(y | x) ∝ exp(f(x)ᵀ g(y)), with embedding f and unembedding g, two models with the same output distribution have embeddings related by f = A f′, and unembeddings, taken relative to a pivot label, by g₀ = A⁻ᵀ g′₀, for an invertible A (LIT-253 Thm 2.2, a prior result restated with its proof in App. B). The distribution fixes the representation up to the general linear group and no further. A centred linear kernel fixes it up to a translation and an orthogonal map (THEORY-004). So what is observed sets the symmetry group up to which the representation is fixed.

Closeness depends on the distance. KL divergence between the output distributions can tend to zero while the embeddings are not related by any invertible linear map (Thm 3.1, with a constructive proof). The construction needs the unembedding norm to grow without bound, so that both softmaxes saturate. Cor. 3.2 gives the same for deterministic labels. Under the paper’s log-likelihood variance distance the answer reverses: d_LLV ≤ ε implies d_f,g ≤ 2Mε (Thm 4.7). The paper’s conclusion is that the answer “depends critically on the choice of distributional and representational distance”.

These could have gone otherwise. Some distributional distance might have bounded representational dissimilarity uniformly, or none might have. The paper proves one of each.

What this does not say

  • Anything below the final layer. Every result concerns the final embedding and the unembedding. The paper says earlier layers would need identifiability results that do not yet exist (§6).
  • How far apart the KL counterexample’s embeddings are. Thm 3.1 proves only that they are not linearly related. The size of the dissimilarity is shown numerically (Table 1).
  • What the Platonic hypothesis means, or whether it holds. Reading these results as a constraint on that hypothesis, whose kernels are of pooled hidden states, is THEORY-002’s inference, and it stays Proposed.

Backlinks