nucleation

LIT-1012: Noise Balance and Stationary Distribution of Stochastic Gradient Descent

2 min read

Properties4
StatusActive — read 2026-10-13 (NOTE-714): §§I–IV in full; the proofs in Appendix A not checked. Cited by LIT-970 as Ziyin et al. (2023a), "Law of balance and stationary distribution of stochastic gradient descent"; retitled in v2. The source of THEORY-211.
Tagslearning-theory, loss-landscapes, thermodynamics, anthology-candidate
Cited as “paper” byNOTE-714 — Noise Balance and Stationary Distribution of Stochastic Gradient Descent
Cited as “source” byTHEORY-210 — Continuous parameter symmetries stretch a network's minima into connected manifolds along which the loss is flat, and give gradient flow conserved quantities that fix where on such a manifold training ends; finite steps, gradient noise and weight decay break them, so the position along a symmetry orbit is set by initialization and optimizer, not by the loss, THEORY-211 — In the stochastic-differential-equation model of SGD, with a gradient-noise covariance that depends on the parameters, minibatch noise drives the difference of squared norms along a rescaling symmetry, at a rate proportional to learning rate over batch size, to the point on the symmetry orbit where the gradient-noise traces of the two parameter groups are equal, whenever both groups carry noise; so at stationarity SGD neither keeps the difference fixed, as gradient flow does, nor lets it diffuse, as isotropic noise does, and differs from gradient descent by a finite amount however small the noise, THEORY-790 — In deep networks trained by SGD on real data, minibatch noise drives parameter groups related by a rescaling symmetry toward the point where their gradient noise balances, at a rate set by learning rate over batch size, so that at stationarity they sit neither where gradient flow would leave them nor spread out as under isotropic noise, as Ziyin, Li and Ueda propose

Graph View

Liu Ziyin, Hongchao Li, Masahito Ueda (2023; revised 2025) — ARXIV-2308.06671

Key takeaways

  • Law of balance. With a rescaling symmetry, ‖u‖² − ‖w‖² evolves by an ODE, not an SDE, at a rate proportional to T = η/S (Theorem 1). Its stationary point balances the gradient noise of the two groups, Tr C(w) = Tr C(u), and fixes a unique norm ratio. Weight decay γ adds to the rate.
  • Not a perturbation of gradient descent. The t → ∞ and T → 0 limits do not commute, so at stationarity SGD and GD differ by O(1). Solutions of equal loss are not visited with equal probability.
  • Depth changes the stationary distribution. In the diagonal linear network, depth 1 conserves ratios between subnetworks while depth above 1 balances them (Theorem 3). Above T_c = β₂/α₃ the distribution collapses to zero. Variance falls as T rises. The tail exponent depends on depth only.
  • SGD is not a Bayesian sampler here. Its stationary distribution lives on a measure-zero set, so its KL divergence from the posterior is infinite.

Standing in the record

Read for the null space that THEORY-209 leaves out. The reading is NOTE-714, and the account filed from it is THEORY-211. It makes first-hand the result LIT-970 reported as “different from predictions from a Langevin model”. The Langevin model it means is gradient descent with isotropic noise.

Backlinks