Source
Davis and Drusvyatskiy (2025), LIT-457 — ARXIV-2512.04299, read as NOTE-209. Accounted for by THEORY-032.
What to measure
For each trainable matrix W that acts on an incoming feature matrix X,
compute two numbers:
| quantity | expression | what it says |
|---|---|---|
| stable rank of the input | ‖X‖_F² / ‖X‖_op² | how many directions the incoming features actually occupy |
| nuclear rank of the gradient | ‖∇W‖_*² / ‖∇W‖_F² | how spread out the gradient’s singular values are |
A spectral update’s guaranteed one-step decrease exceeds the Euclidean one by their ratio, so the spectral step is favoured on the blocks where the gradient’s nuclear rank is the larger number. That inequality is Davis and Drusvyatskiy’s (LIT-457), derived from the one-step descent bounds of their §1.2, and the measurement originates there: in a NanoGPT run they track both quantities and find activations staying low stable rank and gradients keeping large nuclear-to-Frobenius ratios throughout training. In a transformer the incoming matrices are the token embeddings, the RMS-normalized hidden states entering the attention projections, and the post-activations entering the second MLP matrix.
Both quantities are norms of tensors a training step already materializes. Neither needs an SVD: the Frobenius and nuclear norms are traces, and the operator norm is a few power iterations.
Why bother, if the answer is usually yes
Because “usually” is doing load-bearing work there and nobody has checked where it stops. The paper proves transformer activations have low stable rank at Gaussian initialization — bounded by the inverse frequency of the most common token, around 20 for a Zipfian corpus, and preserved through RMSNorm, attention and the MLP independently of width and sequence length — and measures that it holds through a NanoGPT run. That is a strong prior that a language model is in the favourable regime.
It is also a prior derived entirely from language. A workload whose inputs are not Zipfian, or whose activations are deliberately spread — a well-conditioned embedding table, a heavily regularized vision encoder, anything with an orthogonality penalty — has no such guarantee, and the measurement is what tells you which side you are on before you pay Muon’s overhead.
Conditions
- This is a measurement, not a routing rule. The paper states the rule of thumb and does not run the ablation that would test it: spectral updates on the blocks that pass, Euclidean on the blocks that fail. Using the condition to select per block goes beyond the evidence, which is why the practice stops at measuring. ADR-017 is the reason the line is drawn here.
- It compares bounds, not outcomes. The inequality is between guaranteed one-step decreases. A better guarantee is not a demonstrated speed-up.
- The propagation results are initialization-time, under Gaussian weights. That they survive training is one measured run.
- NanoGPT scale, plus synthetic regression. Nothing at the scale where SOTA-165 and SOTA-121 are evidenced.
- It does not replace the fair comparison. SOTA-165 rests on ten optimizers tuned per-optimizer at four scales. This predicts which side of a comparison you are on; it does not excuse you from running one.
Known implementations
- None reported. The measurement is stated in the paper and, as far as the record knows, has not been used as a decision procedure by anyone.