nucleation

LIT-242: On the Stepwise Nature of Self-Supervised Learning

3 min read

Properties6
StatusActive — read 2026-10-14 (NOTE-741) in full; worth reading for the dynamics that the optimum-based spectral accounts lack. For a simplified Barlow Twins loss (λ = 1, no batch norm, symmetrized cross-correlation) on a linear or fixed-kernel model, gradient flow from small initialization learns the top eigenvectors of Γ, or of K_Γ, one at a time and reaches the minimum-norm zero-loss solution: kernel PCA. Read it knowing three things. The step for generic initialization (Result 4.6) is an informal derivation. Its operator is built from the model's kernel and the sample pairs, not from K⁺. And its evidence for SimCLR, VICReg and ResNets is experimental: the initial NTK predicts them poorly, and the final empirical NTK captures about half the embedding subspace.
Version3
Tagsrepresentation-learning, learning-theory, anthology-candidate
Extended byLIT-855 — Closed-Form Training Dynamics Reveal Learned Features and Linear Structure in Word2Vec-like Models
Cited as “paper” byNOTE-218 — Simon et al. 2023 — stepwise nature of SSL, NOTE-741 — Simon et al. — On the Stepwise Nature of Self-Supervised Learning
Cited as “source” byTHEORY-007 — Kernel PCA under the positive-pair density ratio recovers the eigenfunctions of the positive-pair Markov chain, and their top span is minimax-optimal for linear prediction of approximately view-invariant targets, THEORY-009 — In the positive-pair spectral self-supervised papers, self-adjointness comes from the symmetry of the positive-pair distribution, not from the Riesz representation theorem; elsewhere in the family it comes from a symmetrized loss, a precondition on the kernel, or composing the conditional-expectation operator with its adjoint, THEORY-229 — For a symmetrized Barlow Twins loss on a linear or fixed-kernel model, gradient flow from a small initialization of common scale aligned with the top eigenvectors of the pairs' symmetrized cross-correlation learns them one at a time, in order of eigenvalue, and ends at a top spectral solution, which is exactly a minimum-norm zero-loss solution: kernel PCA on K_Γ, THEORY-793 — Gradient flow on a symmetrized Barlow Twins loss with a linear or fixed-kernel model, from generic small initialization and not only from initialization aligned with the eigenvectors, learns the top eigenvectors of the pairs' symmetrized cross-correlation one at a time in order of eigenvalue and ends at a top spectral solution, as Simon and coauthors derive informally

Graph View

James B. Simon, Maksis Knutins, Liu Ziyin, Daniel Geisz, Abraham J. Fetterman, Joshua Albrecht (2023), Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202:31852–31876 — ARXIV-2303.15438

Key takeaways

From the reading, NOTE-741.

  • The model. The loss is L = ‖WΓWᵀ − I_d‖²_F, with Γ = (1/2n) Σ_i (x_i x′_iᵀ + x′_i x_iᵀ), symmetrized by hand. This is Barlow Twins with λ = 1 and no batch norm, on f(x) = Wx, trained by gradient flow.
  • Exact steps from aligned initialization (Proposition 4.1). s_j(t) = e^(4γ_j t) / √(s_j(0)^(−2) + (e^(8γ_j t) − 1)γ_j). Mode j waits until τ_j = −log(s_j(0)²γ_j)/(8γ_j), then jumps to γ_j^(−1/2), and the loss drops by about one per mode.
  • Which optimum (Proposition 4.3). Of the many zero-loss solutions, the top spectral W are exactly the minimum-Frobenius-norm ones. For generic small initialization, training follows the aligned path (Result 4.6). That result is derived informally, as the authors say.
  • Kernel PCA (Proposition 5.1). For any fixed kernel, the final embedding is f(x) = U S̃ [b_1 … b_d]ᵀ K̃^(−1/2) [K_xX K_xX′]ᵀ, from the top eigenvectors of K_Γ = K̃^(−1/2) Z K̃^(−1/2). It is the minimum-RKHS-norm zero-loss solution. With identical views, it is the base kernel’s top eigenmodes. With two towers (CLIP), K_XX′ = 0 and the solution still holds.
  • ResNets.
    • Barlow Twins, SimCLR and VICReg on STL-10 show stepwise rank growth, clearest from small initialization, and also in hidden layers.
    • BYOL did not show steps.
    • The initial NTK predicts poorly. The final empirical NTK captures 0.52–0.62 of the embedding subspace, against 0.025 by chance.
  • Speed-ups (App. G). The proposals are untested: “theoretical speculation”.

Standing in the record

Filed on 2026-09-26 at the owner’s request, from a list they grouped under the heading Representation learning as a spectral approximation. Read in NOTE-741. It is held as the dynamical side of that heading. The optimum-based accounts (LIT-249, LIT-258, LIT-228) say which subspace is optimal. This paper says which zero-loss optimum gradient flow reaches and in what order its directions arrive. It is the record’s evidence for any claim about the order or speed of SSL training. Its stepwise result is filed as THEORY-229.

It is a supporting source of THEORY-007, and there it supports by analogy only: its operator Γ, or K_Γ, is built from the model’s kernel and the sample pairs, not from K⁺. It is a source of THEORY-009 for the hand-symmetrized Γ. Its two-tower case bears on that theory’s open question about cross-modal pairs. The NOTE sets out both points. LIT-855 extends the method to word2vec-like models.

Tagged anthology-candidate (ADR-005): the seed judged it chiefly about machine-learning practice, and App. G’s untested speed-up proposals are practice. It is kept here by the owner’s decision of 2026-09-26 that new work stays in nucleation until a transfer is judged appropriate (ADR-010).

Backlinks