Source
Loshchilov, Hsieh, Sun and Ginsburg (2024), LIT-472 — read as NOTE-221.
What it explains
| practice | what it says to do | what this says it is |
|---|---|---|
| SOTA-282 | put every matrix and hidden state on the unit hypersphere | removing a degree of freedom nothing was steering, which had been drifting somewhere bad |
The account
In an ordinary transformer, the direction of every weight vector is trained and its magnitude is not — not directly. Magnitude is a residue of initialization, the learning rate, and weight decay pulling against the gradient. Nobody chose it; it is where the tug-of-war settled.
The claim is that where it settles is not benign. Measured on trained models:
- Input embeddings spread into a hyper-ellipsoid. Norms vary widely, and the covariance eigenvalue spectrum has a high condition number — markedly worse at 1B than at 0.5B, so it worsens with scale.
- Attention matrices lose rank. Singular-value distributions in the third and fourth layers are consistent with degeneration toward lower-rank matrices, which bounds what those blocks can compute.
- It is not only the norms. Renormalizing a trained GPT’s matrices after the fact reduces the condition numbers without bringing them down to the constrained model’s. Some of the ill-conditioning is in the directions, and fixing the norms during training is what prevents it — after the fact is too late.
That last point is what makes this an account rather than a tautology. If normalizing afterwards fixed the numbers, the constraint would just be changing what you measure. It does not, so the constraint is changing what gets learned.
The measurements are LIT-472’s, a by-product of its nGPT comparison: a post-hoc spectral inspection of the trained GPT baseline against the normalized model at matched parameter count, beside the headline result that nGPT reaches the same loss in 4×, 10× and 20× fewer tokens at 1k, 4k and 8k context. The conditioning numbers are where this account comes from; the speed-up is what it is offered to explain.
The accompanying reading — that a normalized transformer is a variable-metric
optimizer on the sphere, each block proposing a direction and its learned
α supplying the step size — is a frame rather than part of this account. It
is why the paper’s parameters have the names they do, and it carries no
evidence of its own.
Why Proposed
Because the evidence is correlational and the paper does not pretend otherwise. Unconstrained models are worse conditioned. Unconstrained models also take more tokens to reach a given loss. Nothing measured connects the two — no intervention isolates conditioning, and no result shows that a badly conditioned model is slow because it is badly conditioned.
What would settle it: an architecture that fixes the conditioning without fixing the norms, or the reverse, with the training-speed comparison held otherwise equal. SOTA-122’s learnable multipliers are close to such a probe — they manage norms explicitly without constraining them to one — and nobody has reported conditioning numbers for a model trained that way.
What this does not say
It does not say normalization layers were doing nothing. They keep activations in range; this is about the weights, and the two are different quantities with a confusingly shared word.
It does not say weight decay is harmful. It says that in an architecture where norms are fixed by construction there is nothing left for weight decay to do. On an unconstrained architecture SOTA-120 stands unaffected.
It is measured at 0.5B and 1B, at layers 3 and 4, on OpenWebText. The scale-worsening observation rests on two points — 0.5B and 1B — which is a direction rather than a trend.