Why it is a question
THEORY-213 holds the exact part of Ziyin (2024). Every mirror symmetry makes its subspace invariant under GD and SGD. Strong enough weight decay makes it the location of every minimum. The paper also argues that SGD noise makes the subspace attractive even where gradient descent escapes it, from the sign of a Lyapunov exponent. It derives the threshold only after a second-order expansion and a diagonal approximation of the sample Hessian, and calls the result qualitatively correct, not exact. Its network evidence (ResNet18, transformers) was not checked. Until 2026-10-20 this was THEORY-213’s promote_when.
What would answer it
- A derivation of the SGD threshold without the diagonal approximation.
- A network with a known permutation, rotation or rescaling symmetry, trained with SGD across learning rates, batch sizes and weight decays, with a measured distance to the symmetric subspace (neuron coincidence, rank, sparsity). Past a threshold predicted from the noise and curvature, that distance should fall to zero. It should not when the symmetry is removed by reparametrization.
Against it: no dependence of the outcome on noise where the threshold predicts one. What cannot settle it: more toy factorizations.