nucleation

THEORY-781: Strong gradient noise in SGD, like weight decay above a threshold, makes the subspace fixed by a mirror symmetry of a network's loss attracting even where it is a saddle, so that networks trained with SGD at large learning rates or small batches reach sparse, low-rank and identical-neuron solutions, as Ziyin proposes

2 min read

Properties5
StatusProposed
Tagslearning-theory, loss-landscapes, anthology-candidate
SourceLIT-1014 — Symmetry Induces Structure and Constraint of Learning
ExtendsTHEORY-213 — Every mirror symmetry of a network's per-sample loss makes its symmetric subspace invariant under gradient descent and SGD, and when the loss's Hessian is bounded below by −c, weight decay γ‖θ‖² with γ > c/2 puts every minimum in that subspace; rescaling symmetries that hold at λ = −1, rotation symmetries and permutation symmetries contain such mirrors, whose subspaces are zeroed weights, reduced rank and identical neurons
AnswersQUESTION-206 — Does stochastic gradient noise drive training into a mirror-symmetric subspace where it is a saddle, beyond the second-order, diagonal-Hessian argument, and does a trained network's distance to that subspace fall to zero past a threshold set by noise and curvature?

Graph View

Source

Ziyin (2024), LIT-1014, §§2.4 and 3, read in NOTE-716, with Appendix C’s proofs unchecked.

What is asserted

That gradient noise, not only weight decay, pushes training into the subspaces a network’s mirror symmetries fix. Near such a subspace, SGD’s random kicks shrink the off-mirror component on average when the Lyapunov exponent is negative, roughly when the learning rate exceeds −2E[ξ + γ]/E[(ξ + γ)²]; that can hold where the subspace is a strict saddle and gradient descent would leave it. Large learning rates, small batches and noisy data favour it. Rescaling, rotation and permutation symmetries then give sparse weights, low rank and identical neurons in SGD-trained networks.

What the Active account establishes toward it

THEORY-213 holds the exact half of Theorem 1, re-derived: the subspace is invariant under gradient training, and weight decay above half the lower bound on the Hessian puts every minimum in it. What it does not hold is this entry’s claim, which rests on the approximation and on Fig. 3: sparse solutions in a reparametrised regression, low rank in a matrix factorisation, and homogeneous correlation blocks in a ResNet18 on CIFAR-10, which appear with weight decay and rarely without.

What would confirm or reject it

The promote_when above, QUESTION-206’s test. Rescaling symmetry counts here only where it contains a mirror (λ = −1); ReLU’s positive rescaling does not (THEORY-213, history of version 2).

Provenance

Filed on 2026-10-20 by the narrowing-loss sweep (ADR-047 v4, batch 068), from the gradient-noise clause THEORY-213’s title carried before it was narrowed on 2026-10-20.

Backlinks