nucleation

THEORY-793: Gradient flow on a symmetrized Barlow Twins loss with a linear or fixed-kernel model, from generic small initialization and not only from initialization aligned with the eigenvectors, learns the top eigenvectors of the pairs' symmetrized cross-correlation one at a time in order of eigenvalue and ends at a top spectral solution, as Simon and coauthors derive informally

2 min read

Properties5
StatusProposed
Tagsrepresentation-learning, learning-theory
SourceLIT-242 — On the Stepwise Nature of Self-Supervised Learning
ExtendsTHEORY-229 — For a symmetrized Barlow Twins loss on a linear or fixed-kernel model, gradient flow from a small initialization of common scale aligned with the top eigenvectors of the pairs' symmetrized cross-correlation learns them one at a time, in order of eigenvalue, and ends at a top spectral solution, which is exactly a minimum-norm zero-loss solution: kernel PCA on K_Γ
AnswersQUESTION-175 — From generic small initialization, does gradient flow on a symmetrized Barlow Twins loss with a linear or fixed-kernel model follow the aligned trajectory, learning the top eigenvectors one at a time in order of eigenvalue?

Graph View

Source

LIT-242. Filed by the promotion audit’s narrowing-loss sweep (batch 066, ADR-047 v4) on 2026-10-20, to carry the wider claim that the narrowing of THEORY-229 dropped. THEORY-229 is Active and is this THEORY’s evidence; this one stays Proposed.

What was actually shown

From an initialization of common small scale aligned with the top eigenvectors of K_Γ, each mode obeys a closed equation, and the modes are learned in order of eigenvalue at times τ_j ≈ −log(s_j(0)²γ_j)/(8γ_j) (Prop. 4.1); the top spectral solutions are exactly the minimum-norm zero-loss ones (Prop. 4.3), and the kernel case is kernel PCA on K_Γ (Prop. 5.1). THEORY-229 states these, re-derived in the promotion audit. Result 4.6 argues that a generic initialization W(0) = αW̃₀ follows the aligned trajectory as α → 0, because the off-diagonal parts stay small while each mode escapes its unstable start in turn. NOTE-741 rates that claim moderate: an informal derivation and small numerical checks.

What would settle it

The promote_when is inherited from THEORY-229’s before it was narrowed, and QUESTION-175 asks the same question.

What this does not say

It says nothing about networks whose tangent kernel moves, such as the ResNets in which the paper also sees stepwise rank growth under SimCLR and VICReg; the paper’s own initial-kernel predictions there are poor. It makes no claim about downstream generalization, which the authors leave open.

Backlinks