nucleation

LIT-228: Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methods

3 min read

Properties5
StatusActive — read 2026-10-14 (NOTE-752) in full; worth reading for its method-by-method dictionary from joint-embedding SSL to spectral embedding on a finite positive-pair graph G. VICReg with a squared variance term is Laplacian eigenmaps (LPP when linear). A Euclidean SimCLR variant is a generalized MDS whose optimum is the top eigenvectors of G scaled by √eigenvalue. Linear Barlow Twins is CCA. All of them align Z's left singular vectors with G's, and they differ in rank. Read it knowing three things. The closed forms hold for modified losses, on a finite sample, with free or linear Z. Theorems 6, 9 and 10 are sketched or unproved. And the practical advice (prefer low-invariance VICReg under a misaligned G) rests on an unproved bound and synthetic experiments with N = 256.
Version2
Tagsrepresentation-learning, anthology-candidate
Cited as “paper” byNOTE-199 — Balestriero & LeCun 2022 — SSL recovers spectral embedding, NOTE-752 — Balestriero & LeCun — SSL Recovers Global and Local Spectral Embedding Methods
Cited as “source” byTHEORY-007 — Kernel PCA under the positive-pair density ratio recovers the eigenfunctions of the positive-pair Markov chain, and their top span is minimax-optimal for linear prediction of approximately view-invariant targets, THEORY-009 — In the positive-pair spectral self-supervised papers, self-adjointness comes from the symmetry of the positive-pair distribution, not from the Riesz representation theorem; elsewhere in the family it comes from a symmetrized loss, a precondition on the kernel, or composing the conditional-expectation operator with its adjoint

Graph View

Randall Balestriero, Yann LeCun (2022), Advances in Neural Information Processing Systems 35 (NeurIPS 2022), pp. 26671–26685 — ARXIV-2205.11508

Key takeaways

From the reading, NOTE-752.

  • VICReg. Its invariance term is the Dirichlet energy 2 Tr(ZᵀLZ) on G.
    • With a squared variance term (in place of the hinge) and α = β, the optimum is the top eigenvectors of I − (1/N)11ᵀ − (γ/α)(D − G), scaled, and only γ/α matters (Theorem 1).
    • With variance and covariance forced to zero and a constant degree, it is Laplacian eigenmaps (Theorem 2).
    • With a linear network it is LPP, and LDA when G is the label graph (Theorem 3).
  • SimCLR. Its softmax similarity is the solution of an entropy-regularized graph estimation (Theorem 4). For the Euclidean comparison ‖G − Ĝ‖²_F, the optimum is Z* = (U_G Σ_G^(1/2))_(:,1:K), a generalized MDS “akin to ISOMAP”, with rank min(K, rank(G)) (Theorem 5, Proposition 2).
  • Barlow Twins. In the linear regime it is CCA: W_a* = top-K eigenvectors of C_aa⁻¹C_abC_bb⁻¹C_ba, with weight sharing when G is symmetric (Theorem 7). The CCA identification is asserted rather than derived.
  • Downstream. Any of the three gives zero linear-probe loss on the training set iff the task’s left singular vectors lie in G’s (Theorem 9, sketched). The ranking “prefer VICReg, then BarlowTwins and finally SimCLR” under a misaligned G rests on Theorem 10, which has no proof, and on synthetic experiments (N = 256–512).
  • Global and local. Contrastive methods map to global spectral methods and non-contrastive ones to local ones. This is an analogy, not a result.

Standing in the record

Filed on 2026-09-26 at the owner’s request, from a list they grouped under the heading Representation learning as a spectral approximation. Read in NOTE-752. It is held as the method-by-method map from SSL losses to named spectral embedding methods, on a finite sample. Its own statement is that all the losses align the representation’s left singular vectors with those of the pair graph G. With the constant degree it assumes, those are the eigenvectors of the positive-pair chain on the sample. That is where it meets HaoChen et al. (LIT-249) and Johnson et al. (LIT-258).

It is a supporting source of THEORY-007. The NOTE answers that theory’s question about which of its dictionaries is right: they pick the same eigenvectors. Its Theorem 7 is an SVD-type (CCA) account of a non-symmetric pair structure, which bears on THEORY-009, though that theory does not cite it. Simon et al. (LIT-242) supply the training dynamics it lacks.

Tagged anthology-candidate (ADR-005): the seed judged it chiefly about machine-learning practice, and its advice on choosing and tuning SSL methods is practice. It is kept here by the owner’s decision of 2026-09-26 that new work stays in nucleation until a transfer is judged appropriate (ADR-010).

Backlinks