Nitanda et al. (2022) — ARXIV-2201.10469
Key takeaways
Provides a concise, self-contained convergence rate analysis of mean field Langevin dynamics (MFLD) for KL-regularized objectives in both continuous and discrete time, using a novel “proximal Gibbs distribution” p_q that makes the analysis parallel classical finite-dimensional convex optimization. Establishes linear convergence in continuous time for any regularization strength, and a discrete-time rate for noisy gradient descent on mean field neural networks. Additionally shows p_q exactly characterizes the duality gap in empirical risk minimization.
- Theorem 1 (continuous time). L(q_t) - L(q*) ≤ exp(-2αλ t) (L(q_0) -
L(q*)): linear convergence of MFLD in continuous time
Holds when: Requires uniform LSI constant α > 0 on p_q; holds for any λ
0
- Corollary 1. inf_{s∈[0,t]} KL(q_s || p_{q_s}) ≤ exp(-2αλ(t-1))/(2αλ^2) · (L(q_0) - L(q*)) Holds when: Same LSI assumption
- Theorem 2 (discrete time). L(q^(k)) - L(q*) ≤ δ_η/(2αλ) + exp(-αλη k)(L(q^(0)) - L(q*)) Holds when: δ_η is discretization error O(η) for MF neural nets under bounded gradients
- Corollary 2 (iteration complexity). O((1/(ε α^2 λ^2)) log(1/ε)) iterations to reach ε-accurate solution Holds when: Exponential dependence on 1/λ through α
- Theorem 3 (duality gap). L(q) - D(g_q) = λ KL(q || p_q): exact duality gap in ERM Holds when: L2 regularization, smooth convex per-sample losses
What the evidence does not cover
- Uniform log-Sobolev constant α has exponential dependence on 1/λ in the general bounded-perturbation analysis, degrading rates for weak regularization.
Standing in the anthology
Read — the reading is NOTE-138. Arrived in the imported batch, which brought in the mean-field account of wide networks, where the object that moves is the distribution of neurons.