anthology-of-the-sota

SOTA-129: Build a reasoning model in three stages: pretrain on general data, SFT on reasoning traces, then RL with verifiable rewards

2 min read

Properties6
StatusActive
Version2
Tagsadaptation-and-tuning
SourceLIT-130 — Olmo 3, LIT-172 — Tulu 3: Pushing Frontiers in Open Language Model Post-Training, LIT-128 — Falcon-H1R: Pushing the Reasoning Frontiers with a Hybrid Model for Efficient Test-Time Scaling
Introduced byLIT-130 — Olmo 3
Compared againstSOTA-130 — Skip the reasoning SFT stage and run RL with verifiable rewards directly on the base model

Graph View

Source

Olmo Team (2025), LIT-130 — the Olmo 3 model flow. Lambert et al. (2024), LIT-172 — Tulu 3, where the final stage was named. Falcon-LLM Team (2025), LIT-128 — the same recipe on a different base.

The standard reasoning curriculum, stated as Olmo 3 runs it in the open: a base model pretrained on a general mixture; supervised fine-tuning on reasoning traces so the model produces long chains of thought in the expected format; a preference-tuning stage; then reinforcement learning with verifiable rewards — rule-based checkers for math, test cases for code — to push accuracy past what imitation reaches. GRPO (LIT-127) is the usual RL algorithm. Falcon-H1R (LIT-128) is the same recipe on a hybrid 7B base, and the 135M write-up in LIT-129 is a small instance of the SFT-then-preference half.

Where the last stage comes from. Olmo 3 is the open, end-to-end statement of the curriculum, but it is not where reinforcement learning from verifiable rewards was introduced. Tulu 3 (LIT-172) named RLVR as a stage and published the recipe in full — data, code, decontamination and the negative results — and this practice’s third step is that stage. The note was sourced only to Olmo 3 until #58, which had the effect of attributing a two-lab result to the lab that wrote it down last.

Conditions: this is the recipe at 7B and above, where the reasoning corpus is small next to the model’s memorization window and a from-scratch mix could not carry it (LIT-119’s own arithmetic gives a 100B model a window near 5000 GT against a 5 GT SFT set). Each stage costs a separate run and a separate set of hyperparameters, and the RL stage is the sensitive one.

Successors and variations

  • Drop the SFT stage (SOTA-130): RL with verifiable rewards directly on the base model, Olmo 3’s RL-Zero track. Filed as Proposed because its source frames it as an experimental pathway.
  • Collapse pretraining and SFT (SOTA-123): for tiny models, pretrain from scratch on the reasoning or SFT mixture and keep only the RL stage. LIT-119 shows it beating this recipe at 90M and 0.6B; the same source says it should not hold at 100B.
  • Collapse all three: LIT-119 lists merging the RL phase into pretraining as future work. Nothing in the record does it yet.

Backlinks