OLMo Team, Ai2 (2025) — ARXIV-2501.00656
Key takeaways
- Dense models at 7B, 13B and 32B released with weights, the full training data, the code, the logs and thousands of intermediate checkpoints — which is what makes the stability claims below checkable by someone else, and is the reason this note exists rather than a shorter one.
- The architecture and recipe changes are presented as being for stability and per-token efficiency, not as accuracy wins that happen to be stable.
- Dolmino Mix 1124: a specialized late-stage mixture introduced during the annealing phase of pretraining, reported to improve downstream capability substantially. This is the decay stage used as a place to spend data quality, which is the same argument SOTA-140 records from the WSD side.
- Post-training follows Tülu 3, ending in reinforcement learning with verifiable rewards — the stage SOTA-129 names.
- Positioned at the Pareto frontier of performance against training compute, matching or beating open-weight-only models of the period at fewer FLOPs.
Standing in the anthology
The record’s second Ai2 note, and the predecessor of LIT-130. Filed for the stability recipe: it is the most completely documented account in the corpus of the interventions that keep a mid-scale run alive, and unlike the frontier reports it ships the checkpoints that would let a reader verify them.
Its annealing result is the same observation SOTA-140 rests on, reached from a data-mixture direction rather than a schedule-shape one, and worth reading next to it.
No practice of its own, and the reason is altitude rather than adoption. The stability recipe is the most completely documented account in the corpus of what keeps a mid-scale run alive, and it is a recipe: a bundle of interventions reported together, with the released checkpoints that would let someone take it apart. Nobody has. Until somebody isolates which intervention does what, filing it would recommend a bundle on the authority of its outcome, which is the failure ADR-017 is about.
Its annealing result is a different matter and is not orphaned: it reaches the observation SOTA-140 rests on from a data-mixture direction rather than a schedule-shape one, and is worth reading beside it.