anthology-of-the-sota

LIT-298: On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport

2 min read

Properties4
StatusActive
Tagsanalysis-and-evaluation
Cited as “paper” byNOTE-139 — On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport
Cited as “source” byTHEORY-009 — A wide two-layer network's training dynamics are a gradient flow on the distribution of its neurons, and that flow is convex

Graph View

Chizat and Bach (2018) — ARXIV-1805.09545

Key takeaways

Establishes asymptotic global convergence of particle gradient descent for a class of non-convex measure optimization problems (including sparse deconvolution and two-layer neural network training) in the many-particle limit. The key technical tool is recasting particle gradient flow as a Wasserstein gradient flow in the space of probability measures, exploiting homogeneity of Phi and V. Provides results for both 2-homogeneous (ReLU) and partially 1-homogeneous (sigmoid, sparse spikes) settings.

  • Theorem 3.3 (2-homogeneous case). If initialization support separates r_a S^{d-1} from r_b S^{d-1} and mu_t converges in W_2, then lim_{t,m->infty} F(mu_{m,t}) = min F. Holds when: Requires ReLU-type 2-homogeneity, Sard regularity, convex smooth R.
  • Theorem 3.5 (partially 1-homogeneous case). If initialization support separates {-r}xTheta from {r}xTheta (e.g., supported on {0}xTheta) and h^1(mu_t) converges weakly, then lim F(mu_{m,t}) = F*. Holds when: Sigmoid/sparse-deconvolution setting; requires boundary conditions on phi.
  • Theorem 2.6 (many-particle limit). Empirical measures mu_{m,t} from particle gradient flow converge (in W_2) to the unique Wasserstein gradient flow of F as m -> infty. Holds when: Under Assumptions 2.1 with mu_{m,0} -> mu_0 in W_2.

What the evidence does not cover

  • Results are asymptotic and non-quantitative: no rate in m or t.
  • Requires convergence of the Wasserstein gradient flow as an assumption; no proof of convergence given.
  • Sard-type regularity is assumed, hard to verify in practice (especially the boundary condition at infinity for sigmoid NNs).
  • Only shallow (two-layer) networks; multi-layer extension left open.
  • ReLU requires a reparameterization (signed-square) to ensure differentiability.
  • Experiments are on synthetic low-dimensional or teacher-student setups.

Standing in the anthology

Read — the reading is NOTE-139. Arrived in the imported batch, which brought in the mean-field account of wide networks, where the object that moves is the distribution of neurons.

Backlinks