Contribution
A unified account of the geometric constraints that architectural symmetries put on gradients and Hessians, with conservation laws under gradient flow derived from them. The paper then works out what weight decay, momentum and finite learning rates do to those laws, giving exact expressions for the dynamics of a large number of parameter combinations in a real network.
Key insight
The parameters of a network are chaotic, but some combinations of them, picked out by symmetries, are simple. A symmetry direction is invisible to the loss, so gradient flow cannot move along it, and the coordinate along it is conserved. What does move it is everything gradient flow leaves out. Weight decay pulls it toward zero. Finite steps push outward, because each step is tangent to a curved orbit.
Assumptions
- Differentiable symmetries of the training loss: translation, scale and rescale, generated by softmax, batch normalization and homogeneous activations.
- Theorem 1 requires ⟨θ, [∂_α ∂_θ ψ]g⟩ = 0, which holds for the three symmetries treated.
- The exact dynamics of §6 use a deterministic continuous model: weight decay plus the modified loss, without momentum or noise. Noise is treated separately (App. F).
Key results
- Constraints (§3). Translation: ⟨g, 1_A⟩ = 0 and H1_A = 0. Scale: ⟨g, θ_A⟩ = 0 and Hθ_A + g_A = 0. Rescale: ⟨g, θ_A1 − θ_A2⟩ = 0.
- Conservation (Theorem 1, Eqs. 11–13). Under gradient flow, the parameter sum, the norm, and the difference of squared norms are conserved for the three symmetries.
- Broken conservation (Eqs. 18–20). Translation: ⟨θ(t), 1_A⟩ = e^{−λt}⟨θ(0), 1_A⟩, independent of η and the data. Scale: |θ_A(t)|² = e^{−2λt}|θ_A(0)|² + η∫e^{−2λ(t−τ)}|g_A|²dτ. Rescale: the same form, with the difference of gradient norms, whose sign is not fixed.
- Scale balance (Eq. 21). Weight decay’s centripetal pull −2λ|θ_A|² against discretization’s centrifugal push η|g_A|². At stationarity the angular speed is √(2λ/η), consistent with Wan et al. (2020).
- Rescale. “For large enough weight decay, the regularization dominates any complexity introduced by the gradient norms and the difference in parameter norms decays exponentially to zero.”
- Momentum (§6, App. G). The quantities follow driven harmonic oscillators. They are overdamped for standard β, and visibly underdamped at β = 0.99 on VGG-16.
- Noise (§5, App. F). Batch gradients are orthogonal to the generators too, so Gᵀ∂_α ψ = 0: the noise is low-rank along symmetry directions. Under an Itô model, the scale-symmetric norm still gains a deterministic forcing term from the noise variance, (η/S)Tr[GᵀG].
- Count. A VGG-16 with batchnorm on Tiny ImageNet has 12,751 such combinations (App. J, Table 5).
Claims
- The analytic expressions “match nearly perfectly with empirical results” on VGG-16. See Limitations for what that match tests.
Concepts
- Generator: ∂_α ψ at the identity, the direction a symmetry moves the parameters.
- Modified loss: L̃ = L + (η/4)|∇L|², whose gradient flow tracks gradient descent at finite η (Barrett & Dherin 2020).
Connections
- THEORY-210 took “imbalance decays exponentially with large learning rates” from the LIT-970 survey, which attributes it to this paper. This reading did not find that claim in the main text, or in Appendices F, I and J. The main text attributes the exponential decay of the norm difference to weight decay “large enough” to dominate. The η term is a gradient-norm integral whose sign for rescale symmetry is not fixed.
- THEORY-209 (Kunin et al.’s limiting dynamics, the same group a year later) assumes noise covariance proportional to the Hessian. That shares the property used here, zero noise along symmetry generators, but also makes the covariance independent of the parameters. That removes the Itô forcing term App. F derives.
- LIT-1012 (Ziyin et al.) builds that Itô term into a law of balance for rescale symmetry, where the noise drives the norm difference to a balanced value.
Bearing on the record
It is the first-hand source for THEORY-210’s conservation laws and their breaking, and corrects the attribution noted above. With Ziyin et al., it fills the null-space half of THEORY-209.
Limitations
- The match is with recorded gradients. The “theoretical” curves in the figures compute the integral terms from the batch gradients recorded during training (App. J). The discrete versions of Eqs. 18–20 are exact identities given those gradients (App. I). So the agreement tests how well the continuous model approximates the discrete updates. It does not test a prediction made without the trajectory. The translation result, which needs no gradients, is the exception.
- One architecture and dataset for the experiments: VGG-16, with and without batchnorm, on Tiny ImageNet.
- Noise in §6 enters only through the recorded batch gradients. The Itô treatment is sketched in App. F for scale symmetry only.
Open questions
- What is the stochastic counterpart of Eq. 20 for rescale symmetry? That is what Ziyin et al. answer, as a law of balance.
Corrections
None to the paper. One to the record: see Connections on THEORY-210.