Source
Vyas et al. (2024), LIT-157 — ARXIV-2409.11321; with the fair comparison in Wen et al. (2025), LIT-156.
The result the method falls out of is an equivalence, not an analogy: Shampoo implemented with the 1/2 power is exactly Adafactor run in the eigenbasis of Shampoo’s preconditioner. Once that is established the design question answers itself — if the basis is the useful part, run the better optimizer in it. SOAP is Adam in that basis.
Because it is Adam in a rotated space, it adds exactly one hyperparameter over Adam: how often to recompute the preconditioner.
The failure mode it avoids, which is the practical point
The obvious way to make Shampoo affordable is to eigendecompose less often. That degrades performance, and worse the less often you do it — so the efficiency knob and the quality knob are the same knob, turned opposite ways.
SOAP escapes it by continuing to update the second-moment running average in the current, slowly-rotating basis, which is what Adam does anyway. The stale basis stops being a problem because the statistics keep moving inside it.
Reported at 360M and 660M in the large-batch regime: over 40% fewer iterations and over 35% less wall-clock than AdamW, and about 20% better than Shampoo on both.
Against Muon, which is compared_against and not a rivalry
LIT-156 tuned both fairly and put both in the fastest group, with matrix preconditioning as the common factor. That is a comparison somebody ran, which is why the relation is declared rather than argued.
The record recommends Muon (SOTA-121, with SOTA-131’s QK-Clip riding along) on production evidence at trillion-parameter scale that SOAP does not have. This is the same family reached from the other direction — SOAP by simplifying a full preconditioner until it is affordable, Muon by orthogonalising a momentum matrix. Both are members of the trunk practice they extend, and what SOAP supplies to that trunk is the argument that the win belongs to the class.
Conditions, and why this is Proposed
360M and 660M, one group for the algorithm, and no production deployment anywhere in the record. The large-batch regime is also a condition and not a detail: that is where the reported gains are measured.
Known implementations
- None in the record.