nucleation

LIT-1011: The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusion

2 min read

Properties4
StatusActive — read 2026-10-13 (NOTE-711): main text, and Appendices A–C and H; the derivations in Appendices D–G were not checked. Held also in the Anthology of the SOTA as ANTH-LIT-454, read there for what the loss curve hides; read here for what a converged run is, under ADR-013.
Tagslearning-theory, loss-landscapes, thermodynamics, anthology-candidate
Cited as “paper” byNOTE-711 — The Limiting Dynamics of SGD
Cited as “source” byTHEORY-209 — A converged network trained with SGD is not at a point and does not sample its training loss: under a quadratic approximation it samples a Gibbs distribution on a modified loss, isotropic in the Hessian's column space, with probability currents that keep it oscillating, so its speed and spread are set by the learning rate, momentum and batch size, THEORY-779 — In Kunin et al.'s underdamped Langevin model of SGD with momentum on linear regression, with gradient-noise covariance σ²H independent of the parameters, the stationary state is Gaussian with parameter covariance ησ²H(H + λI)⁻¹/(2S(1 − β)) and velocity covariance σ²H/(S(1 − β²)): for small weight decay it spreads equally along every nonzero Hessian direction and not along the null space, each Hessian eigenmode is a damped oscillator with damping ratio (1 − β)/√(2η(1 + β)(ρ + λ)), and the expected squared step is η²σ² tr(H)/(S(1 − β²))

Graph View

Daniel Kunin, Javier Sagastuy-Brena, Lauren Gillespie, Eshed Margalit, Hidenori Tanaka, Surya Ganguli, Daniel L. K. Yamins (2021) — ARXIV-2107.09133

Key takeaways

  • Converged is not stationary at a point. Resumed training of five pre-trained ImageNet convnets moves at a near-constant step size, and the squared distance from the pre-trained weights grows as k^c with c between about 1.07 and 1.62 (Fig. 1).
  • What SGD samples is not the training loss. For linear regression with gradient-noise covariance proportional to the Hessian, the stationary distribution is Gibbs on a modified loss Ψ(θ, v). Its parameter part has curvature H⁻¹(H + λI)/(η(1 + β)), which is isotropic in the Hessian’s column space when weight decay is negligible. Its velocity part, v⊺H⁻¹v, penalizes motion along low-curvature directions.
  • Broken detailed balance. The phase-space stationary state carries a non-zero probability current, so the dynamics rotate in each eigen-plane of the Hessian at rates set by its eigenvalue. With many incommensurate rates the paths are space-filling, not limit cycles.
  • The hyperparameters set the transport. Instantaneous speed is η²σ²tr(H)/(S(1 − β²)), matched across a sweep of learning rates, batch sizes and momenta. A larger learning rate makes the oscillators underdamped and lowers the diffusion exponent: faster learning rate, slower global travel.

Standing in the record

Read for the learned-structure line’s question of what training reaches (QUESTION-069): a converged run is a nonequilibrium stationary distribution whose shape is set by the optimizer as much as by the loss. The anthology reads the same paper for practice, as one of three ways an aggregate loss curve hides what training does (ANTH-THEORY-031). The reading here is NOTE-711, and the account it supports is THEORY-209.

Backlinks