nucleation

THEORY-740: A two-layer linear network trained by gradient flow on white inputs from a decoupled, balanced start learns each singular mode of its input–output correlation in a logistic rise that waits a time of order its inverse strength times the log of the inverse initial scale, so the modes arrive one at a time in order of strength; a single-layer network on the same data learns every mode on one timescale with no plateau, so under these conditions the stages come from the product of layers and the data's spectrum

2 min read

Properties6
StatusActive
Tagsrepresentation-learning, learning-theory, mathematics
SourceLIT-862 — A mathematical theory of semantic development in deep neural networks, LIT-864 — Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
Extended byTHEORY-186 — A deep linear network trained from small weights learns the singular modes of its input–output correlations one at a time in order of strength, each in a sharp sigmoidal transition, while a shallow network learns them all together; the stages come from the product of layers and the data's spectrum, not from nonlinearity
Cited as “extends” byTHEORY-481 — Under gradient flow averaged over tasks and data, a linear-attention layer trained on in-context regression learns as a two-layer linear network from an aligned start does: when tasks share the input covariance's eigenbasis and the weights start and stay aligned with it, then in the long-context limit each input mode is learned on its own logistic curve whose timescale falls with the square of its variance, so learning is staged by the input spectrum, and the layer converges to the estimator preconditioned by the inverse input covariance; at finite context length the modes decouple exactly only when inputs are isotropic and the modes start equal, and otherwise couple at order 1/N
Cited as “grounds” byCLAIM-360 — The deep-versus-shallow learning results separate three effects that aggregate loss conflates: a larger realizable function class, a larger set reachable within a fixed data and compute budget, and a different internal organization with the same attainable outcomes; the deep linear results show the third, the random-hierarchy results the second, and the depth-separation result for products of inputs the first, at fixed size

Graph View

Source

Saxe, McClelland and Ganguli (2014), LIT-864, Eqs. 6–12 and Appendix A, read in NOTE-667, and the same authors’ 2019 restatement, LIT-862, Eqs. 2–11, read in NOTE-666. Re-derived here as a check.

What was actually shown

Take ŷ = W₂W₁x with white inputs (Σx = I), loss ½E‖y − W₂W₁x‖², and gradient flow with time constant τ. Write Σyx = U S Vᵀ.

  • The deep network. Gradient flow is τẆ₁ = W₂ᵀ(Σyx − W₂W₁) and τẆ₂ = (Σyx − W₂W₁)W₁ᵀ. Start decoupled and balanced: W₁ = R C Vᵀ, W₂ = U C Rᵀ, with R orthonormal and C = diag(c_α). Then W₂W₁ = U C² Vᵀ and τẆ₁ = R C(S − C²) Vᵀ, so the form is kept and τċ_α = c_α(s_α − c_α²); the W₂ equation gives the same. For a_α = c_α², τȧ_α = 2a_α(s_α − a_α), solved by a_α(t) = s_α e^{2s_αt/τ} / (e^{2s_αt/τ} − 1 + s_α/a_α⁰). Mode α reaches half its final strength at t_α = (τ/(2s_α)) ln(s_α/a_α⁰ − 1), and passes from a fraction δ to 1 − δ of s_α in time (τ/s_α) ln((1−δ)/δ), whatever a_α⁰. So, as the initial scale goes to zero, each mode waits a time of order (τ/s_α) ln(1/a_α⁰) and then rises in a transition whose width is a vanishing fraction of the wait. Stronger modes go first.
  • The shallow network. For ŷ = Wx, τẆ = Σyx − W, so W(t) = Σyx + (W(0) − Σyx)e^{−t/τ}. In the SVD basis, b_α(t) = s_α(1 − e^{−t/τ}) + b_α⁰ e^{−t/τ}: every mode relaxes on the one timescale τ, exponentially, with no plateau.

The same data give stages through two layers and none through one, so, under these conditions, the stages come from the product of the layers and the spectrum of Σyx.

What could have failed: a start that is not decoupled. Then the off-diagonal terms couple the modes, and the per-mode equations do not close.

What this does not say

  • Not from random initialisation. That small random, unbalanced weights follow these curves is simulated in LIT-862 (Fig. 3C), not proved. It stays with THEORY-186.
  • Not for correlated inputs. With Σx ≠ I the input and output bases need not align.
  • Not that nonlinear networks learn this way, nor anything about depth beyond two layers, which is THEORY-187’s.
  • Not discontinuities. Each rise is smooth, and sharp only relative to the plateau before it.

Backlinks