anthology-of-the-sota

LIT-457: When do spectral gradient updates help in deep learning?

2 min read

Properties6
StatusActive
Version2
Tagstraining-optimization, signal-structure
Cited as “introduced by” bySOTA-274 — Before adopting a spectral optimizer, measure the stable rank of each block's incoming activations against its gradient's nuclear rank
Cited as “paper” byNOTE-209 — When Spectral Updates Help
Cited as “source” bySOTA-274 — Before adopting a spectral optimizer, measure the stable rank of each block's incoming activations against its gradient's nuclear rank, THEORY-032 — A spectral update wins where the incoming activations are low stable rank and the gradient spectrum is spread out

Graph View

Davis and Drusvyatskiy (2025) — ARXIV-2512.04299

Key takeaways

  • One inequality, two measurable quantities. Compare the guaranteed one-step loss decrease of a Euclidean step and a spectral step on a single matrix block. Spectral wins when the gradient’s nuclear rank — the squared ratio of nuclear to Frobenius norm — exceeds the stable rank of the activations feeding that block. Both sides are computable from quantities a training run already has.
  • The condition is blockwise, and that is the practical content. For an MLP the incoming matrix is the previous layer’s post-activation; for a transformer it is the RMS-normalized hidden states entering the attention projections, or the post-activations entering the second MLP matrix. The rule of thumb the paper states is that spectral updates are most advantageous on the blocks whose incoming features are low stable rank and whose gradients have spread-out spectra.
  • Low stable rank is proved, not assumed. At Gaussian initialization the stable rank of the token-indicator matrix is exactly the inverse frequency of the most common token — around 20 for a Zipf-like corpus — and the paper shows this survives RMSNorm, attention and the MLP up to constants, giving a depth-quadratic, width- and sequence-length-independent bound.
  • The advantage is not only an initialization effect. In spiked random-feature regression, after a short burn-in the gradient’s nuclear rank grows with the data dimension while the activations’ stable rank stays bounded, so the predicted speedup scales with dimension rather than decaying.
  • The measurement holds in a real run. In NanoGPT training the intermediate activations stay low stable rank throughout and the gradients keep large nuclear-to-Frobenius ratios — the regime the condition predicts spectral updates should win in, which is the regime Muon is used in.

Standing in the anthology

The second account this record now holds of why matrix preconditioning works, and the one that names quantities a practitioner can measure rather than a geometry to adopt. It reaches the same class as THEORY-024 from a different direction, and it is not the same claim: this says when, not what the update is. Filed as THEORY-032, with the diagnostic as SOTA-274.

Backlinks