Source
Saxe, McClelland and Ganguli (2019), LIT-862, Eqs. 2–11 and the Supplementary Material’s derivations, as read in NOTE-666. The solution first appeared in the same authors’ 2014 ICLR paper, Exact solutions to the nonlinear dynamics of learning in deep linear neural networks (arXiv 1312.6120), LIT-864, Eqs. 6–12 and Appendix A, as read in NOTE-667. The 2019 paper does not cite it. The 2014 paper also has the coupled equations with repulsion between modes, the unbalanced solution for decoupled starts, and arbitrary depth (THEORY-187); the 2019 paper adds the shallow comparison, the data models and the minimum-norm result.
What was actually shown
Take ŷ = W₂W₁x with white inputs (Σx = I), squared error, and gradient descent slow enough to average over an epoch. In the SVD basis of the input–output correlation Σyx = Σ s_α u_α v_αᵀ, start the weights diagonal and balanced between the layers. Then each mode’s strength a_α obeys τ ȧ_α = 2a_α(s_α − a_α). The solution, a_α(t) = s_α e^{2s_αt/τ} / (e^{2s_αt/τ} − 1 + s_α/a_α⁰), is a sigmoid that reaches s_α at a time of about (τ/s_α) ln(s_α/ε). The transition takes a vanishing fraction of that time as the initial scale goes to zero. A single weight matrix trained on the same data follows b_α(t) = s_α(1 − e^{−t/τ}) + b_α⁰e^{−t/τ}: every mode on the timescale τ ln(s_α/ε), exponentially, with no stages and with every individual prediction monotone. The derivation is exact under its conditions, and the difference between the two architectures is proved.
What could have come out otherwise is the random-initialisation case. From small random weights the paper simulates the network and finds the trajectories on the decoupled curves (Fig. 3C). Off-diagonal couplings could have persisted or reordered the modes; in these simulations they do not.
The data enter only through the spectrum. For features diffusing down a tree, the singular values fall with depth, so the schedule is coarse to fine. For a ring the modes are Fourier modes, learned from low frequency up. Between the stages of the deep network, single item–feature predictions can move the wrong way for a time of order the gap between the singular values.
What this does not say
- Not that random initialisation provably follows the solution. The decoupled, balanced start is assumed. The random small-weight case rests on simulation of datasets with a handful of items.
- Not that nonlinear networks learn this way. The paper’s evidence is a visual match between two MDS plots (Fig. 2). Depth here buys no expressive power; the claim concerns the training trajectory of a linear map.
- Not for correlated inputs or prior knowledge. With Σx ≠ I the input and output bases need not align. The SI says the solution does not describe learning that begins with substantial knowledge already in the weights.
- Not that the order is always coarse to fine. The order is by singular value. Fine distinctions come first whenever the data make them stronger; anticorrelated siblings at an intermediate level are the paper’s own example (Fig. 8).
- Not that stages are discontinuities. Each transition is smooth. It is sharp only relative to the plateau before it, and only for small initial weights.
- Not an explanation of children’s development. The paper offers it as a qualitative account and fits no developmental data.
- Not an instruction. It says what such a network learns and when, not how to train one.
Where it sits
THEORY-182 is this dynamics applied to a symmetric factorisation, in a quadratic proxy for word2vec (LIT-855). LIT-859 obtains it in a linear-attention layer, where the product is of two attention blocks rather than two layers. THEORY-183’s Fourier geometry has the ring case as its precedent. THEORY-039 gains a derived kind of phase. THEORY-086 is the contrast: in the kernel regime a network fits eigenspace by eigenspace at exponential rates, which is this account’s shallow case.
Note of 2026-10-13: the balanced-start condition, under gradient flow and SGD
This account is exact for decoupled, balanced initial weights. Two readings filed on 2026-10-13 bear on that condition.
- Under gradient flow, balance is kept. The difference of squared norms between adjacent layers is conserved (Neural Mechanics, LIT-1012, Theorem 1). A balanced start stays balanced, which the exact solution relies on.
- Under SGD, balance is approached. Minibatch noise drives the difference toward balance (THEORY-211). For a two-layer linear network, Ziyin et al. find uᵢ² = wᵢ² at stationarity, with probability 1.
So balance is less special under SGD than under gradient flow. But the attraction acts over the long run, at stationarity. It does not supply balance during the stage-like transitions this account describes, which still need the balanced start, or something close to it.
Note of 2026-10-29: what holds it back
No held result proves that a two-layer linear network started from small random, unbalanced weights converges to the decoupled solution as the initial scale goes to zero; LIT-862 shows it in simulation (Fig. 3C), and LIT-864 solves unbalanced starts only when already decoupled. A simulation run at the fifteenth status review agrees (four modes, five seeds per scale: as the scale falls from 10⁻¹ to 10⁻⁴, the off-diagonal part of the product falls from 15% to 2% of the largest strength, and the deviation from the aligned logistic from 33% to 0.1%), which the promote_when rules out as settling it.