nucleation

LIT-970: Symmetry in Neural Network Parameter Spaces

2 min read

Properties5
StatusActive — read 2026-10-13 (NOTE-712): §§1–5 and 7–8 in full, §6 in part. A survey, so its claims about the works it cites are secondhand here. It bears on QUESTION-058 and QUESTION-069, and on the null space that THEORY-209 leaves out. The account filed from it is THEORY-210.
Version2
Tagsloss-landscapes, learning-theory, anthology-candidate
Cited as “paper” byNOTE-712 — Symmetry in Neural Network Parameter Spaces
Cited as “source” byTHEORY-210 — Continuous parameter symmetries stretch a network's minima into connected manifolds along which the loss is flat, and give gradient flow conserved quantities that fix where on such a manifold training ends; finite steps, gradient noise and weight decay break them, so the position along a symmetry orbit is set by initialization and optimizer, not by the loss

Graph View

Bo Zhao, Robin Walters and Rose Yu (2025), Transactions on Machine Learning Research — ARXIV-2506.13018

Key takeaways

  • A catalogue. Linear layers have a GL(h) symmetry. Homogeneous activations such as ReLU have positive rescaling, tanh has sign flips, and batchnorm and other normalizations have scaling. Softmax has translation, and pointwise activations have permutation. Attention has GL(d_k) per head, plus head permutation and GL(d_v). A network inherits the symmetries of all its subnetworks (Table 1, §2.2).
  • Completeness. For tanh and Gaussian RBF networks the known symmetries are complete. ReLU networks can have hidden symmetries beyond permutation and scaling, and functional dimension can vary within one fiber (§2.4).
  • Landscape. Continuous symmetries make minima connected manifolds, which is one explanation of mode connectivity. The survey notes that “not all points in a minimum are related by a symmetry transformation” and that connectivity may have other sources (§3.1). Discrete symmetries replicate minima instead (§3.2).
  • Dynamics. Continuous symmetries give gradient flow conserved quantities, layer imbalance being the standard one. These label trajectories and their endpoints, which formalizes part of implicit bias. Finite steps, noise and weight decay break them, and the cited results find SGD drifting toward balanced solutions (§5).

Standing in the record

First filed for QUESTION-058, from the abstract. Read on 2026-10-13 (NOTE-712), after Kunin et al.’s limiting dynamics (THEORY-209) were found silent on the Hessian’s null space, where these symmetries put the flat directions. The account filed from the reading is THEORY-210.

Backlinks