nucleation

LIT-254: Similarity of Neural Network Representations Revisited

3 min read

Properties6
StatusActive — read 2026-10-14 (NOTE-743) in full; worth reading as the source of CKA for comparing representations and of the argument that a similarity index should be invariant to orthogonal maps and isotropic scaling but not to all invertible linear maps. That argument rests on Theorem 1: any index with that wider invariance is constant once rank reaches the number of examples. It also shows R²_CCA, linear regression and linear CKA to be one family, differing in eigenvalue weighting, with canonical ridge interpolating between them. Read it knowing that the case for the orthogonal group is argued from training dynamics, not proved, that it does not state the converse that equal Gram matrices imply an orthogonal relation, and that RBF CKA is scale-invariant only because its bandwidth is set from the median distance.
Version4
Tagsrepresentation-learning, anthology-candidate
SupportsCLAIM-378 — A representational similarity index invariant to every invertible linear map depends on a representation only through its column space, so it takes one value on all column-centred representations of rank n − 1, as it does on all representations of rank n
Cited as “paper” byNOTE-223 — Similarity of NN Representations Revisited (CKA), NOTE-743 — Kornblith et al. — Similarity of Neural Network Representations Revisited
Cited as “source” byTHEORY-004 — A representation is determined by its centred kernel up to a translation and an orthogonal transformation, and the rotation-aligned distance between two representations equals the Bures distance between their kernels, THEORY-008 — On a finite set of stimuli, what a regularized linear readout can decode from a representation is a function of its normalized kernel, and CKA, CCA and GULP are averages of readout agreement, THEORY-236 — A representational similarity index that is invariant to every invertible linear map takes the same value on all representations whose rank equals the number of examples

Graph View

Simon Kornblith, Mohammad Norouzi, Honglak Lee, Geoffrey Hinton (2019), ICML 2019 (Proceedings of the 36th International Conference on Machine Learning, PMLR vol. 97); first appeared as arXiv preprint — ARXIV-1905.00414

Key takeaways

From the reading, NOTE-743.

  • Theorem 1. If s(X,Z) = s(XA,Z) for every invertible A, then s(X,Z) = s(Y,Z) whenever rank(X) = rank(Y) = n. Such an index cannot compare layers at least as wide as the dataset. The proof is four lines (App. A).
  • The right invariances, argued. §2 argues for invariance to orthogonal transformation and isotropic scaling, but not to non-isotropic scaling. Three reasons are given:
    • permutation symmetry;
    • the linear-case gradient-descent result of LeCun et al.;
    • networks from different seeds sharing their large principal components (Fig. 1). No theorem fixes the group.
  • CKA. HSIC(K,L) = tr(KHLH)/(n−1)², and CKA = HSIC(K,L)/√(HSIC(K,K)HSIC(L,L)). Linear CKA is ‖YᵀX‖²_F/(‖XᵀX‖_F‖YᵀY‖_F), the RV coefficient. Centring by H removes any constant added to a kernel, since H1 = 0.
  • One family. R²_CCA and R²_LR are CKA of re-weighted Gram matrices (Eqs. 11–12). Linear CKA is CCA’s overlap Σ⟨u_Xⁱ, u_Yʲ⟩² weighted by the eigenvalues λ_Xⁱλ_Yʲ (Eqs. 13–14). Canonical ridge interpolates between R²_CCA, linear regression and CKA (App. D.1). Orthogonal Procrustes alignment reduces to the nuclear norm ‖YᵀX‖_* (App. D.2).
  • Empirical results.
    • Linear CKA identifies corresponding layers across ten CIFAR-10 CNNs with 99.3% accuracy. Linear regression reaches 45.4%, linear HSIC 22.2%, and CCA, SVCCA and PWCCA 1.4–15.1%.
    • All indexes pass on Transformer encoders.
    • CKA also shows pathological depth, the effects of width, and early layers shared between CIFAR-10 and CIFAR-100.

Standing in the record

Filed on 2026-09-26 while pursuing, at the owner’s request, the connection GNS, unitary equivalence and the convergence of representations (see the curation entry of that date). Read on 2026-10-14 in NOTE-743. It is held as the standard reference for comparing representations through their kernels, and as the place where orthogonal invariance is argued to be the right equivalence for representations. That is the finite-sample counterpart of “determined up to unitary equivalence”. The Platonic hypothesis (ANTH-LIT-458) cites it for that. Its Theorem 1, that an index invariant to every invertible linear map is degenerate at full rank, is filed as THEORY-236.

It is a supporting source of THEORY-004, for the case for orthogonal invariance (§2.2). The case is an argument, and the converse that THEORY-004 needs, that equal kernels imply an orthogonal relation, is in LIT-250 and LIT-259, not here. App. D.2’s Procrustes identity is the closer link to THEORY-004’s Procrustes–Bures equality. It is a supporting source of THEORY-008 only for the family structure: CCA, regression and CKA as one ridge-indexed family. It says nothing about readouts or GULP.

Tagged anthology-candidate (ADR-005): recommending CKA as the default similarity index is a tool for ML analysis. It is kept here by the owner’s decision of 2026-09-26 that new work stays in nucleation until a transfer is judged appropriate (ADR-010).

Backlinks