Source
Johnson, El Hanchi & Maddison (2023), LIT-258, Def. 2.1, Thm 3.2, Prop. 3.1, Thm 4.1, §5 and App. C.3. Supporting: HaoChen et al. (2021), LIT-249, Lemmas 3.1–3.2 and App. F; Pfau et al. (2019), LIT-252, eqs. 6–8; Balestriero & LeCun, LIT-228, on a finite sample; Simon et al., LIT-242, by analogy only.
What was actually shown
K⁺ is a Mercer kernel with an explicit feature map (LIT-258 Def. 2.1). Population kernel PCA under K⁺ returns exactly the L²(p)-orthonormal eigenfunctions f_i of the positive-pair chain p⁺(a′|a), with h_i = λ_i^½ f_i (Thm 3.2). E_p⁺(g(a₁) − g(a₂))² = Σ(2 − 2λ_i)c_i² (Prop. 3.1), so the top eigenfunctions are the most view-invariant directions, and their span is minimax-optimal for linear prediction of approximately view-invariant targets (Thm 4.1). The optimality is for squared L²(p) error, with one p(A) shared between pretraining and the downstream task, and needs d ≥ d*, the multiplicity of eigenvalue 1 (one per communicating class); below d* every representation has unbounded worst-case error, and above it the minimax value is ε/(2(1 − λ_(d+1))) (App. D).
HaoChen et al. show the spectral contrastive loss equals ‖D^(−1/2)AD^(−1/2) − FFᵀ‖²_F plus a constant (Lemma 3.2), so its minimizers are the top eigenvectors up to a positive row scaling and a right factor diag(√γ)R with R orthonormal, invertible while γ_k > 0; Lemma 3.1 says no invertible right factor or positive row scaling changes linear-probe predictions. Johnson et al. identify these eigenvectors with their eigenfunctions scaled by p(a)^½ (§5, App. C.3). Pfau et al.’s Spectral Inference Networks would learn these eigenfunctions if given K⁺: their eqs. 6–7 are for a generic symmetric kernel, and applying them to K⁺, estimated from positive pairs, is the record’s step. They show recovery only for the two-dimensional hydrogen atom, and their argument for recovering the eigenfunctions in order is informal. Johnson et al. App. E.2 names SpIN and NeuralEF as ways to extract them, and applies NeuralEF, not SpIN, to K⁺.
On a finite sample, Balestriero & LeCun (LIT-228) give one spectral method per SSL loss (VICReg as Laplacian eigenmaps, SimCLR as generalized MDS, Barlow Twins as CCA) and say that all of them learn representations whose top singular vectors align with those of the pair matrix G. Under the paper’s constant-degree assumption D = cI, G/c is the transition matrix of the positive-pair chain on the sample and equals HaoChen et al.’s normalized adjacency, so the three select the same eigenvectors and differ in scaling and rank. That reconciliation is the record’s; the paper does not mention K⁺ or the Markov chain.
What this does not say
- That trained networks on real data reach these eigenfunctions. Johnson et al.’s own experiments find recovery depends on parameterization and augmentation strength, on synthetic data. On real data they could not obtain a reliable estimate of the eigenfunctions from a pretrained SimCLR model, and their ImageNet spectral-loss model “does not appear to be a good approximation of K⁺” (App. E.1, F.5).
- Anything for asymmetric methods (BYOL, SimSiam, cross-modal pairs), which fall outside the self-adjoint setting.
- How this relates to Simon et al.’s kernel (LIT-242), which is built from the network’s NTK and the pair structure, not from K⁺, so it supports this account by analogy only. Simon et al. say their final representations differ from those of Balestriero & LeCun, Kiani et al. and Cabannes et al.; of HaoChen et al. they say they complete the account, by identifying which of its degenerate optima training reaches in linearized models.
- Anything about population eigenfunctions or minimax optimality from Balestriero & LeCun (LIT-228). Their support is for the eigenvector identification on a finite sample, for idealized losses, and their downstream result (Theorem 9) concerns zero training-set probe loss only. Johnson et al. note an ISOMAP link without reconciling it with K⁺.
Connections
- THEORY-229 is the dynamical counterpart. This account says which subspace is optimal. Simon et al. (LIT-242) show which of the degenerate zero-loss optima a linearized model reaches under gradient flow on a symmetrized Barlow Twins loss, and in what order: the top eigenvectors, one at a time, in order of eigenvalue. Their operator is built from the model’s kernel and the sample pairs, not from K⁺, so the correspondence is by analogy, as the Source section says.
- THEORY-228 is a second guarantee on the same eigenvectors. At the population optimum of the spectral contrastive loss, HaoChen et al. (LIT-249, Thm 3.8) bound linear-probe error by the probability that augmentation changes the label over the squared sparsest-partition conductance of the augmentation graph. The assumptions on the target differ: label recoverability and conductance there, approximate view invariance here.