Saad and Solla (1995) — DOI:10.1103/PhysRevLett.74.4337
Key takeaways
Solves in closed form the coupled ODEs for Q and R derived in the companion PRE paper, demonstrating that the student-teacher overlap R(t) follows a characteristic S-shaped trajectory: extended plateau at low R, sharp transition, exponential approach to R_∞. The generalization error inherits this shape as ε(t) ∝ (1 - R(t))². Learning rate η controls the overall timescale (τ_sp ∝ 1/η) but not the fundamental shape. This is the paper that establishes the S-curve as the universal learning curve for two-layer teacher-student networks.
- Closed-form solution for R(t). R(t) = R_∞ - (R_∞ - R_0) · exp(-t / τ_sp) where τ_sp = 1 / (η · |F’(R*)|) and R* is the unstable fixed point of the ODE F(R) = dR/dt. For the realizable case R_∞ = 1; for unrealizable R_∞ < 1. Holds when: Symmetric subspace (all R_{jn} equal); erf activation
- S-curve in generalization error. ε_gen(t) = C(1 - R(t))² = C(1 - R_∞)²
- 2C(1 - R_∞)(R_∞ - R_0) exp(-t/τ_sp) + C(R_∞ - R_0)² exp(-2t/τ_sp) This is a sum of two decaying exponentials plus a constant floor. It is NOT a single exponential. The ratio of the two rates is exactly 2 (from the square). Holds when: Realizable case; symmetric subspace
- Learning rate controls timescale, not shape. Doubling η halves τ_sp, uniformly compressing the time axis. The ratio (plateau duration) / (specialization duration) is invariant to η. To change the shape, one must change d, K, M, or the activation. Holds when: Moderate η range
- Capability threshold. Realizable (K ≥ M): R_∞ = 1, ε_gen → 0. Unrealizable (K < M): R_∞ < 1, persistent floor. Over-complete (K >> M): plateau is shorter because redundant students break symmetry faster. Holds when: General K, M
What the evidence does not cover
- Symmetric subspace solution (all R_{jn} equal); general case requires numerical ODE integration.
- Thermodynamic limit only.
- Erf activation; ReLU requires numerical ODE solution.
- Online SGD; minibatch introduces additional terms.
Standing in the anthology
Read — the reading is NOTE-143. Arrived in the imported batch, which brought in the teacher-student line, which solves the learning dynamics exactly in a solvable model.