anthology-of-the-sota

SOTA-123: For tiny specialized models, pretrain from scratch on the target SFT or reasoning data instead of pretrain-then-finetune

2 min read

Properties4
StatusActive
Tagsadaptation-and-tuning, tiny-models
SourceLIT-119 — Falcon-H1-Tiny: A series of extremely small, yet powerful language models redefining capabilities at small scale
Introduced byLIT-119 — Falcon-H1-Tiny: A series of extremely small, yet powerful language models redefining capabilities at small scale

Graph View

Source

Falcon-LLM Team (2026), LIT-119 — the Falcon-H1-Tiny technical blogpost.

The argument

The authors’ anti-curriculum, a departure from the three-stage recipe (SOTA-129) that Olmo 3 states in the open and that its Zero-RL track varies

from the other end (SOTA-130). The argument: a model’s memorization window

(SOTA-124) scales with its size, and at 90M it is about 5 GT — no larger than an SFT mix — so the constraint that forces SFT into a short final stage disappears, and the high-quality data can be present from the first token.

The recipe originates in LIT-119, the Falcon-H1-Tiny blogpost, which compared the two strategies head to head at 90M — SFT data mixed into pretraining from scratch against a base run followed by a separate SFT stage, and the same for reasoning traces — and adopted the from-scratch version for its instruct and reasoning releases. Both measurements below are that comparison.

The evidence

At 90M over 800 GT:

  • Instruction following: a mix with 25% SFT data, trained from scratch, against the same base recipe followed by a tuned 10 GT SFT stage. IFEval 50.1 vs 40.8 before DPO, 66.1 vs 53.5 after; other benchmarks within noise. The SFT-25% checkpoint scored as well as SFT-0% without the chat template, so it serves as both base and instruct model. An extra SFT stage on top of it gave no measurable gain. Mixes above 50% SFT were worse overall — pretraining data is still needed to develop instruction following.
  • Reasoning: pretraining on reasoning traces alone against pretrain-then- reasoning-SFT. AIME24 pass@16 6/30 vs 3/30, AIME25 9/30 vs 2/30, MATH500 0.4 vs 0.2; the from-scratch run scaled better with budget throughout.

Conditions, and where it did not hold

The 100M multilingual model tied between the two strategies, which the authors attribute to SFT data quality and to capacity; tool calling tied as well, and looked bounded by what a 90M model can represent rather than by when it saw the data. The recommendation is for the regime where the target data fits inside the memorization window — tiny models, or large models with very large SFT corpora — not a replacement for post-training in general.

Known implementations

  • Falcon-H1-Tiny-90M-Instruct, Falcon-H1-Tiny-R-0.6B, Falcon-H1-Tiny-R-90M

Backlinks