nucleation

LIT-1013: Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics

1 min read

Properties4
StatusActive — read 2026-10-13 (NOTE-713): main text in full, and Appendices F, I and J; Appendices A–E, G and H not checked. The first-hand source for the conservation laws and their breaking that THEORY-210 took from the LIT-970 survey.
Tagslearning-theory, loss-landscapes, anthology-candidate
Cited as “paper” byNOTE-713 — Neural Mechanics
Cited as “source” byTHEORY-210 — Continuous parameter symmetries stretch a network's minima into connected manifolds along which the loss is flat, and give gradient flow conserved quantities that fix where on such a manifold training ends; finite steps, gradient noise and weight decay break them, so the position along a symmetry orbit is set by initialization and optimizer, not by the loss, THEORY-211 — In the stochastic-differential-equation model of SGD, with a gradient-noise covariance that depends on the parameters, minibatch noise drives the difference of squared norms along a rescaling symmetry, at a rate proportional to learning rate over batch size, to the point on the symmetry orbit where the gradient-noise traces of the two parameter groups are equal, whenever both groups carry noise; so at stationarity SGD neither keeps the difference fixed, as gradient flow does, nor lets it diffuse, as isotropic noise does, and differs from gradient descent by a finite amount however small the noise, THEORY-790 — In deep networks trained by SGD on real data, minibatch noise drives parameter groups related by a rescaling symmetry toward the point where their gradient noise balances, at a rate set by learning rate over batch size, so that at stationarity they sit neither where gradient flow would leave them nor spread out as under isotropic noise, as Ziyin, Li and Ueda propose

Graph View

Daniel Kunin, Javier Sagastuy-Brena, Surya Ganguli, Daniel L. K. Yamins, Hidenori Tanaka (2021), ICLR 2021 — ARXIV-2012.04728

Key takeaways

  • Symmetry constrains geometry. If L(ψ(θ, α)) = L(θ), the gradient is orthogonal to the generator ∂_α ψ, and the Hessian obeys a matching constraint (Eqs. 1–2).
  • Three symmetries, three conservation laws. Under gradient flow, translation conserves the parameter sum, scale conserves the norm, and rescale conserves the difference of squared norms (Theorem 1, Eqs. 11–13).
  • Weight decay and finite steps break them. With weight decay λ and learning rate η, each quantity is an exponentially decaying memory of its initial value plus an integral of gradient norms (Eqs. 18–20). For scale-symmetric weights, weight decay pulls in and discretization pushes out, so at stationarity the angular speed is √(2λ/η).
  • Noise lies off the symmetry directions. Batch gradients obey the same orthogonality, so the noise covariance annihilates the generators (§5, App. F).

Standing in the record

Read for the null space that THEORY-209’s limiting dynamics leave out, and as the first-hand source for THEORY-210. The reading is NOTE-713. Its stochastic term (App. F) is the mechanism Ziyin et al. (LIT-1012) develop into a law of balance (THEORY-211).

Backlinks