nucleation

LIT-1014: Symmetry Induces Structure and Constraint of Learning

1 min read

Properties4
StatusActive — read 2026-10-13 (NOTE-716): main text in full; the proofs in Appendix C and the experimental details in Appendix B not checked. Cited by LIT-970 as Ziyin (2024). The source of THEORY-213.
Tagslearning-theory, loss-landscapes, anthology-candidate
Cited as “paper” byNOTE-716 — Symmetry Induces Structure and Constraint of Learning
Cited as “source” byTHEORY-213 — Every mirror symmetry of a network's per-sample loss makes its symmetric subspace invariant under gradient descent and SGD, and when the loss's Hessian is bounded below by −c, weight decay γ‖θ‖² with γ > c/2 puts every minimum in that subspace; rescaling symmetries that hold at λ = −1, rotation symmetries and permutation symmetries contain such mirrors, whose subspaces are zeroed weights, reduced rank and identical neurons, THEORY-781 — Strong gradient noise in SGD, like weight decay above a threshold, makes the subspace fixed by a mirror symmetry of a network's loss attracting even where it is a saddle, so that networks trained with SGD at large learning rates or small batches reach sparse, low-rank and identical-neuron solutions, as Ziyin proposes

Graph View

Liu Ziyin (2024), ICML 2024 — ARXIV-2309.16932

Key takeaways

  • Mirror symmetries unify the common ones. Rescaling, rotation and permutation symmetries each imply reflection symmetries ℓ(θ) = ℓ((I − 2OOᵀ)θ).
  • Every mirror symmetry gives an absorbing constraint. Once Oᵀθ = 0, the gradient keeps it there for any minibatch and weight decay (Theorem 1, part 1; Corollary 1).
  • Weight decay and noise make it favoured. Above a critical weight decay, every minimum satisfies the constraint (part 4). With large learning rate or noisy gradients, the symmetric set is attractive for SGD even when it is a strict saddle for GD (§2.4, Eq. 5).
  • Structure as a corollary. Rescaling gives sparsity, rotation gives low rank, permutation gives homogeneous neurons. The structures vanish when the symmetry is removed (Fig. 3).

Standing in the record

Read as a follow-up to the law of balance (THEORY-211), for the discrete symmetries that law does not treat. The reading is NOTE-716, and the account filed from it is THEORY-213. It supplies the asymptotic half that THEORY-184 leaves open, and it offers a symmetry account of the collapse in THEORY-083.

Backlinks