Olmo Team, Ai2 (2025) — ARXIV-2512.13961
Key takeaways
- A family of fully open models at 7B and 32B, targeting long-context reasoning, function calling, coding, instruction following, chat and knowledge recall. The release is the model flow: every training stage, intermediate checkpoint, dataset and dependency, not only the final weights.
- Pretraining on Dolma 3; post-training on the Dolci datasets. The Think variants are built by the standard reasoning curriculum — a base model, then supervised fine-tuning on reasoning traces, then preference tuning, then reinforcement learning with verifiable rewards — and Olmo 3 Think 32B is described as the strongest fully open thinking model at release.
- The RL-Zero track runs RLVR directly on the base model, skipping the SFT stage, with four domain-focused series (math against a rule-based verifier with reference answers, code against test cases, instruction following, and general chat against an LLM judge). The authors frame it as an experimental pathway and an open benchmark for RL algorithms and for the effect of pretraining data on RL, not as the production recipe.
- The RL-Zero data is aggressively decontaminated, verified with negative controls trained against random rewards.
Standing in the anthology
Here as the clearest open statement of the recipe LIT-119 departs from. It is the source for both the three-stage curriculum (SOTA-129) and the
Zero-RL variation that drops the middle stage (SOTA-130); the tiny-scale anti-curriculum (SOTA-123) collapses the first two stages instead. Keeping all three lets a reader follow the sequence rather than only its current end.
Its architecture has since been tested twice by its own laboratory. Olmo Hybrid (LIT-799) keeps this model’s 7B architecture, recipe and evaluation and replaces the three-in-four sliding-window layers with Gated DeltaNet layers. Against this model it reaches the same MMLU in 49% fewer tokens and leads in every domain after mid-training. RULER at 64K is 76.9 against 70.9 with YaRN for both. Data mix, schedule and mid-training batch also changed, so the comparison is close but not exact. The long-context ablations (LIT-801) pretrain this architecture for 140B tokens beside 25 variants and extend them identically. With layerwise QK norm, post-norm and sliding windows it scores 47.5 on HELMET at 32K, against 52.4 for the Llama 3 architecture on the same data, and dropping QK norm and post-norm recovers about 6 points. The authors offer this as why Olmo 3 Base is harder to extend than Llama 3 Base.