Vyas et al. (2024) — ARXIV-2409.11321
Key takeaways
- The result the method falls out of: Shampoo implemented with the 1/2 power is equivalent to running Adafactor in the eigenbasis of Shampoo’s preconditioner. A formal connection rather than an analogy, and it makes the design question obvious — if that basis is the useful part, run the better optimizer in it.
- SOAP is Adam in that eigenbasis. Because it is Adam in a rotated space, it adds exactly one hyperparameter over Adam: how often to recompute the preconditioner.
- The obvious efficiency fix for Shampoo — eigendecompose less often — degrades performance, and worse the less often you do it. SOAP avoids that by continuing to update the second-moment running average in the current, slowly-rotating basis, which is what Adam does anyway.
- At 360M and 660M, in the large-batch regime: over 40% fewer iterations and over 35% less wall-clock than AdamW, and about 20% better than Shampoo on both.
Standing in the anthology
The second-order line the record’s Muon practices are implicitly contrasted with, and the reason the contrast is interesting: SOAP and Muon reach the same family from opposite directions — SOAP by simplifying a full preconditioner until it is affordable, Muon by orthogonalising a momentum matrix — and LIT-156 finds them both in the fastest group under fair tuning, with the matrix preconditioner as the common factor.
Two practices rest on it, and the split is the point. SOTA-168 is SOAP
itself, Proposed: the record recommends Muon (SOTA-121) on production
evidence at trillion-parameter scale that SOAP does not have. SOTA-165
is the class — precondition with matrices rather than entrywise — and this
paper’s equivalence result is what makes that class a real object rather than
a grouping, which is the argument the note had been making in prose.