Jacot et al. (2018) — ARXIV-1806.07572
Key takeaways
Shows that in the infinite-width limit, the function learned by a neural network trained with gradient descent converges to a kernel method with a deterministic kernel called the Neural Tangent Kernel (NTK). The NTK stays constant throughout training in this limit (no feature learning), turning the dynamics into a linear ODE in function space. Proves convergence and characterizes generalization in terms of the NTK’s spectrum.
- Theorem 1. The NTK converges to a deterministic kernel at initialization as H → ∞. Holds when: Any depth, any smooth activation
- Theorem 2. The NTK remains constant throughout training in the infinite-width limit. Holds when: Requires: η = O(1/H), width H → ∞
- Theorem 3. For positive-definite NTK, gradient descent achieves zero training loss at rate e^{-λ_min t}. Holds when: Loss is quadratic in function space (least-squares / MSE targets)
What the evidence does not cover
- Infinite-width limit: practical finite networks deviate, especially for feature-learning tasks.
- Constant NTK assumption fails at practical learning rates and widths.
- Does not model the specialization/alignment dynamics seen in teacher- student networks.
- Full-batch GD only; stochastic dynamics require separate analysis.
Standing in the anthology
Read — the reading is NOTE-108. Arrived in the imported batch, which brought in the mean-field account of wide networks, where the object that moves is the distribution of neurons.