Chen et al., Google (2023) — ARXIV-2302.06675
Key takeaways
- Optimizer design posed as program search, with the selection and simplification strategies that stop a proxy-task winner from being an artefact of the proxy.
- What it found is Lion (Evolved Sign Momentum): keep only the momentum — half of Adam’s optimizer state — and take the sign of an interpolation between momentum and gradient, so every parameter’s update has the same magnitude. It is a rule simple enough to read, which is the point of searching program space rather than tuning a parametrized family.
- Results across domains: ViT up to +2% on ImageNet and up to 5× less JFT pretraining compute; 88.3% zero-shot ImageNet on vision-language contrastive learning; a better FID with up to 2.3× less training compute on diffusion models; on autoregressive and masked language modelling, similar to or better than Adam.
- Two consequences of the sign update that are recipe, not trivia: the gain grows with batch size, and the learning rate must be smaller than Adam’s because the sign function makes the update norm larger.
- The paper states its own limits — scenarios where the improvement is small or not statistically significant — and reports production deployment on Google’s search-ads CTR model.
Standing in the anthology
The retirement here is inherited and should be re-examined. The migration filed this paper twice — once under its real title and once as
Lion: Learning with Implicit Optimization (LIT-105, now retired as the duplicate) — and retired this copy as “a research direction without demonstrated practical impact”. That reason was written against a four-bullet description of the paper as automated discovery and theory. It is not what the paper is: Lion is a concrete optimizer with cross-domain results and a named production deployment.
The status is left as it was found rather than reversed here, because whether the anthology wants a practice for Lion is a curation decision and not a citation repair. What has changed is that the decision can now be made against the paper. The record’s live optimizer advice has moved elsewhere regardless — SOTA-121 recommends Muon — and Lion’s sign update is the nearest neighbour to Muon’s orthogonalised one, which is an argument for reading them together.