Goldt et al. (2019) — ARXIV-1906.08632
Key takeaways
Derives exact ODEs for the learning dynamics of a two-layer ReLU student network trained on synthetic data generated by a teacher network (teacher- student setup). The ODEs track overlaps Q (student-student: W_i · W_j / ||W||^2) and R (student-teacher: W_i · W_k / (||W|| ||W||)) through time. Shows that these scalar order parameters fully characterize the MSE and alignment trajectory in the thermodynamic limit (N→∞, n→∞, n/N → const). Introduces the “hidden manifold model” where data lies near a low- dimensional teacher manifold.
- Theorem (informal). The test MSE and all overlaps Q, R satisfy a closed system of ODEs in the thermodynamic limit. These ODEs have a unique fixed point corresponding to the global minimum (perfect student-teacher alignment) when H ≥ H*. Holds when: Requires H ≥ H* (number of student neurons ≥ teacher neurons)
- Empirical. The ODE predictions match stochastic simulation results for d ≥ 50 to within 2–5%. Holds when: Gaussian data; erf activation; H* = 2, H = 4
What the evidence does not cover
- Thermodynamic limit analysis — finite-d, finite-H corrections may be significant.
- Online SGD only; minibatch version requires additional noise analysis.
- Activation: erf or similar; not exact ReLU.
- Teacher and student both two-layer; does not cover deeper architectures.
Standing in the anthology
Read — the reading is NOTE-110. Arrived in the imported batch, which brought in the teacher-student line, which solves the learning dynamics exactly in a solvable model.
The reading this note carries arrived labelled Modelling the influence of data structure on learning in neural networks: the hidden manifold model, which is a different Goldt paper (ARXIV-1909.11500). The reading itself is of this paper; the title and authors above are arXiv’s.