Velikanov et al. (2026) — ARXIV-2601.04890
Key takeaways
- Under weight decay, stochastic gradient noise inflates a weight matrix in a Brownian-like way and the decay pulls it back, so the layer settles at a norm determined by the learning rate and the decay coefficient. The paper treats that equilibrium norm as an artefact of the procedure rather than a property of the solution, and shows it is suboptimal.
- The remedy is to give the scale its own parameter: a learnable scalar multiplier on each weight matrix, and then learnable per-row and per-column multipliers so that individual row and column norms are learned as well.
- Freeing the scale yields consistent gains, generalises Maximal Update Parametrization (LIT-148) — the forward multipliers µP fixes by hand become learned — and reduces the cost of tuning multipliers by hand.
- The size of the gain: on a 0.5B model trained for 200 GT, against a well-tuned Muon baseline, the average over seven benchmarks moves 31.88 to 32.98, +1.10 points. It is not spread evenly — BBH +3.39 and GSM8K +2.61 carry most of it, HellaSwag, ARC-C and MMLU-Pro move by fractions of a point, and MMLU is flat at 47.98 against 47.96. The Adam gain is the same size, +1.21, which is the paper’s own framing: the improvement from adding learnable multipliers matches the improvement from switching Adam to Muon.
Standing in the anthology
The mechanism behind SOTA-122, which stays Proposed until a result from outside the authors’ group is in the record. LIT-119 is the first application at a released model’s scale.