nucleation

CLAIM-379: On a finite set of stimuli, regularized linear readouts with ridge b > 0 cannot tell apart two centred representations of the same width exactly when one is an orthogonal transformation of the other, and the unregularized CCA member cannot tell apart representations with the same column space, which differ by an invertible linear map

3 min read

Properties8
StatusActive
Rolethesis
Warrantdemonstrated
Tagsrepresentation-learning, mathematics
Lineformal-apparatus
Rests onCLAIM-290 — A positive-definite kernel's minimal realizations are unique up to a unitary, and two finite representation matrices with the same centred Gram matrix differ by a translation and an orthogonal map
GroundsTHEORY-008 — On a finite set of stimuli, what a regularized linear readout can decode from a representation is a function of its normalized kernel, and CKA, CCA and GULP are averages of readout agreement, THEORY-004 — A representation is determined by its centred kernel up to a translation and an orthogonal transformation, and the rotation-aligned distance between two representations equals the Bures distance between their kernels, THEORY-723 — For the final embedding and unembedding of a softmax model, the output distribution fixes the representations only up to an invertible linear map, and whether closeness in distribution bounds representational dissimilarity depends on the distance: not for KL in the saturation regime, but yes for the log-likelihood variance distance
SupportsCLAIM-269 — A regularized linear readout can tell apart only representations that differ in normalized kernel, and kernels fix a representation only up to the symmetry of what is observed, so no such readout supplies a distinction beyond that symmetry

Graph View

The claim

Let X be an M × d representation of M stimuli, centred, with C_X = XᵀX/M. THEORY-008 gives the decoded signal of a regularized linear readout as K_X z, with K_X = X G⁻¹ Xᵀ/M and G = aC_X + bI. Two representations are indistinguishable to such readouts when K_X = K_Y, and distinguishable when K_X ≠ K_Y, since then some task z has K_X z ≠ K_Y z.

The kernel fixes the Gram matrix when b > 0. Write the thin SVD X = USVᵀ. Then C_X = VS²Vᵀ/M, and

K_X = U diag(s²/(as² + bM)) Uᵀ, while Γ_X = XXᵀ/M = U diag(s²/M) Uᵀ.

So K_X and Γ_X have the same eigenvectors, and each eigenvalue γ of Γ_X goes to k = γ/(aγ + b), with zero going to zero. For b > 0 the map t ↦ t/(at + b) has derivative b/(at + b)² > 0 on t ≥ 0, so it is injective, with values below 1/a. Inverting it eigenvalue by eigenvalue gives Γ_X = bK_X(I − aK_X)⁻¹. Equal kernels therefore mean equal Gram matrices, and the converse is immediate.

Equal Gram matrices mean an orthogonal map. Two finite representation matrices of the same width with the same Gram matrix differ by an orthogonal transformation (CLAIM-290); with centring, by a translation and an orthogonal transformation (THEORY-004). So for b > 0, two centred representations of the same width are indistinguishable to every such readout exactly when one is an orthogonal transformation of the other.

The CCA member sees only the column space. For a = 1, b = 0, K_X = U Uᵀ, the orthogonal projection onto the column space of X. If X and Y have full column rank and the same column space, then Y = XA for an invertible A, and every invertible A preserves the column space. So the CCA member fixes a representation only up to an invertible linear map. That is the freedom THEORY-723 and THEORY-018 find in a softmax model’s final layer.

What it does not say

  • That kernels fix a representation up to the symmetry of the observed data. That is CLAIM-269’s further clause, grounded on THEORY-002, and it is about converging kernels, not the orthogonal freedom of one representation.
  • Anything about nonlinear probes. The family is the regularized linear readouts of THEORY-008.
  • Anything for infinitely many stimuli. THEORY-008 is Active only on a finite stimulus set.

Backlinks