nucleation

THEORY-004: A representation is determined by its centred kernel up to a translation and an orthogonal transformation, and the rotation-aligned distance between two representations equals the Bures distance between their kernels

3 min read

Properties8
StatusActive
Version8
Tagsrepresentation-learning, mathematics
SourceLIT-250 — Duality of Bures and Shape Distances with Implications for Comparing Neural Representations, LIT-259 — Noncommutative analysis, Multivariable spectral theory for operators in Hilbert space, Probability, and Unitary Representations, LIT-254 — Similarity of Neural Network Representations Revisited, LIT-241 — Gelfand–Naimark–Segal construction (Wikipedia)
Extended byTHEORY-008 — On a finite set of stimuli, what a regularized linear readout can decode from a representation is a function of its normalized kernel, and CKA, CCA and GULP are averages of readout agreement, THEORY-017 — In a Hilbert-space model only unitarily invariant structure is intrinsic; a basis or tensor factorisation, and so any parts or features, is extra data supplied from outside the space, and the record's Hilbert-space works differ in where they get it
Presupposed byTHEORY-002 — Convergence of representations, in the Platonic hypothesis's sense, is convergence of kernels, which fixes representations only up to the symmetry group of what is observed; whether closeness in distribution implies closeness of representation depends on the distance used, THEORY-723 — For the final embedding and unembedding of a softmax model, the output distribution fixes the representations only up to an invertible linear map, and whether closeness in distribution bounds representational dissimilarity depends on the distance: not for KL in the saturation regime, but yes for the log-likelihood variance distance
SupportsCLAIM-148 — A matrix factorization is identified only up to an invertible change of latent coordinates, which weight decay narrows to rotations and nonnegativity to rescalings and permutations plus a data-dependent remainder; such a gauge is not a symmetry of the relational system, and automorphisms of the data act on the factors only when the factorization is identifiable, CLAIM-379 — On a finite set of stimuli, regularized linear readouts with ridge b > 0 cannot tell apart two centred representations of the same width exactly when one is an orthogonal transformation of the other, and the unregularized CCA member cannot tell apart representations with the same column space, which differ by an invertible linear map
Cited as “grounds” byCLAIM-193 — A linear probe family can tell apart only realizations whose representations differ in normalized kernel, and kernels fix a representation only up to the symmetry of what is observed: any distinction beyond that is supplied by the analyst, CLAIM-269 — A regularized linear readout can tell apart only representations that differ in normalized kernel, and kernels fix a representation only up to the symmetry of what is observed, so no such readout supplies a distinction beyond that symmetry

Graph View

Source

Harvey, Larsen & Williams (2023), LIT-250, §2.2, §5.3, Thm 1 (eqs. 14–15), Lemma 2, eq. 25, as read in NOTE-753. Jorgensen & Tian, LIT-259, Remark 1.34, Cor. 1.35, Ex. 4.28. Supporting: Kornblith et al. (2019), LIT-254, as read in NOTE-743: App. D.2 (the Procrustes optimum is ‖YᵀX‖_) and §2.2 (an argument, not a theorem, for orthogonal invariance); the C-algebraic GNS statement, LIT-241, of which LIT-259’s kernel form is the analogue; the article says nothing about kernels, real scalars or orthogonal maps.

What was actually shown

Jorgensen & Tian call the construction of a Hilbert space from a positive-definite kernel “the GNS construction” (Remark 1.34) and state that a kernel’s minimal realizations k(x,y) = ⟨Φ(x), Φ(y)⟩ are unique up to a unitary (Cor. 1.35). The proof is “left as an exercise”; the argument the book proves for Thm 4.35 transfers to it (NOTE-829). Eq. (1.24) as printed, “UΦ₁(x) = Φ₂(x) U”, is a typo for UΦ₁(x) = Φ₂(x). The corollary is over complex Hilbert spaces with a unitary; applying it to real representation matrices with an orthogonal map, or to representations whose rows do not span R^d and so are not minimal, needs a step the book does not state. They also note that commutative states are measures (Ex. 4.28). On a finite sample, two representation matrices with the same number of columns and the same Gram matrix differ by an orthogonal transformation (LIT-250 §5.3, stated there as known, without proof; proved in CLAIM-290, with no assumption that the rows span R^d). Since the centring removes column means, two representations with the same centred kernel differ by a translation and an orthogonal transformation. Harvey, Larsen & Williams prove that the Procrustes distance equals the Bures distance between the centred linear kernels (Thm 1, eq. 14), and that normalized Bures similarity is the cosine of the Riemannian shape distance (Thm 1, eq. 15); both follow from Lemma 2, F(K_X, K_Y) = ‖Σ_XY‖*. So centred kernels are equal iff the Procrustes distance is zero, by Thm 1 and the metric property of B; §2.2 says the like of RDMs, not of kernels. Kornblith et al. had already noted that the orthogonal Procrustes optimum is ‖YᵀX‖* (App. D.2), without linking it to the Bures fidelity. Their §2.2 case that a similarity index should be invariant to orthogonal maps (and isotropic scaling) is argued from training dynamics and symmetry, not proved. Their eq. 25 applies Uhlmann’s theorem: the fidelity is the maximal overlap over all representations consistent with the kernels.

So the kernel is the invariant and the representation is one realization of it — the GNS picture, stated for representations.

What this does not say

  • That orthogonal equivalence is the right notion of sameness for learned models. The Platonic-convergence paper (LIT-302, NOTE-286) adopts it rather than disputing it, by defining a representation by its kernel (§2), and its alignment metric is invariant to more; adopting a notion is not an argument that it is right.
  • Anything for nonlinear kernels on the representation, where what is determined is the kernel’s feature map, not the representation itself.
  • That a model’s kernel is determined by anything short of the model; the uniqueness is of realizations of a given kernel.

Connections

  • THEORY-236. Why the orthogonal group, and not the general linear group. Kornblith et al. (LIT-254, Thm 1) prove that a similarity index invariant to every invertible linear map takes the same value on all representations whose rank equals the number of examples. A comparison under that larger group is degenerate once the representation is wide enough, so the kernel is not traded for an invertible-linear invariant.

Backlinks