Rotskoff and Vanden-Eijnden (2018) — ARXIV-1805.00915
Key takeaways
Reinterprets SGD training of wide two-layer neural networks as an interacting particle system and establishes a Law of Large Numbers and Central Limit Theorem for the empirical distribution of parameters. Shows that training descends gradient-like on a convex functional in 2-Wasserstein metric, converging at a rate independent of n, and that the approximation error universally scales as O(1/n). Also quantifies the SGD noise and gives guidance on scaling step size and batch size with n.
- Proposition 3.5 (LLN / global convergence). As n→∞, μ_t^(n) ⇀ μ_t solving ∂t μ = ∇·(∇V(θ,[μ])μ), and lim{t→∞} ∫φ(·,θ)μ_t(dθ) = f (dynamical universal approximation) Holds when: n-independent convergence timescale; requires μ_in regularity and discriminating kernel
- Proposition 3.7 (CLT). n^{1/2}(f_t^(n) - f_t) → g_t in law, where g_t is a zero-mean Gaussian process; fluctuations O(n^{-1/2}) at finite t Holds when: Finite t, assumes LLN holds
- Proposition 3.9 (self-healing at long time). For any ξ < 1, lim_{n→∞} n^{2ξ} E ∫ |f_{a_n}^(n)(x) - f(x)|^2 ν(dx) = 0 along suitable time a_n Holds when: Requires positive-definite kernel M([μ_t],x,x’) along the trajectory and supp μ̂* = D̂
- Proposition 4.x (SGD scaling). With σ = Δt/P = a n^{-2α}, α ∈ (0,1], SGD admits LLN toward same μ_t and CLT with fluctuations O(n^{-α}) Holds when: Choose Δt and batch size P such that Δt/P ~ n^{-2α}; α=1/2 during training, α=1 for final quench
Standing in the anthology
Read — the reading is NOTE-093. Arrived in the imported batch, which brought in the mean-field account of wide networks, where the object that moves is the distribution of neurons.