nucleation

THEORY-790: In deep networks trained by SGD on real data, minibatch noise drives parameter groups related by a rescaling symmetry toward the point where their gradient noise balances, at a rate set by learning rate over batch size, so that at stationarity they sit neither where gradient flow would leave them nor spread out as under isotropic noise, as Ziyin, Li and Ueda propose

2 min read

Properties5
StatusProposed
Tagslearning-theory, loss-landscapes, thermodynamics, anthology-candidate
SourceLIT-1012 — Noise Balance and Stationary Distribution of Stochastic Gradient Descent, LIT-1013 — Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics
ExtendsTHEORY-211 — In the stochastic-differential-equation model of SGD, with a gradient-noise covariance that depends on the parameters, minibatch noise drives the difference of squared norms along a rescaling symmetry, at a rate proportional to learning rate over batch size, to the point on the symmetry orbit where the gradient-noise traces of the two parameter groups are equal, whenever both groups carry noise; so at stationarity SGD neither keeps the difference fixed, as gradient flow does, nor lets it diffuse, as isotropic noise does, and differs from gradient descent by a finite amount however small the noise
AnswersQUESTION-164 — Does the law of balance hold in a deep network trained on real data during its limiting dynamics: for parameter groups related by a rescaling symmetry, does the difference of squared norms move toward the value where the groups' gradient-noise traces are equal, at a rate that scales with learning rate over batch size?

Graph View

Source

LIT-1012, LIT-1013. Filed by the promotion audit’s narrowing-loss sweep (batch 066, ADR-047 v4) on 2026-10-20, to carry the wider claim that the narrowing of THEORY-211 dropped. THEORY-211 is Active and is this THEORY’s evidence; this one stays Proposed.

What was actually shown

Within the stochastic-differential-equation model of SGD, with noise that depends on the parameters, Itô’s lemma gives a deterministic drift of the difference of squared norms along a rescaling symmetry, toward the point where the two groups’ gradient-noise traces are equal, at a rate proportional to η/S. THEORY-211 states that result, re-derived in the promotion audit. The authors call the law general (NOTE-714, Limitations), and they read edge-of-stability behaviour and the claim that the minimal model’s phenomena “exist uniquely in deep networks” from it. Their tests are a two-layer ReLU network on Gaussian data and toy and tanh models. Whether a deep network on real data, in its limiting dynamics, shows the drift is what this THEORY claims.

What would settle it

The promote_when is inherited from THEORY-211’s before it was narrowed, and QUESTION-164 asks the same question.

What this does not say

It does not say that the SDE approximation holds at the learning rates deep networks are trained with; where it fails, the law is limited, not refuted. It does not carry the authors’ edge-of-stability reading, which is argued from the minimal model only.

Backlinks