anthology-of-the-sota

LIT-119: Falcon-H1-Tiny: A series of extremely small, yet powerful language models redefining capabilities at small scale

4 min read

Properties5
StatusActive
Tagstraining-optimization, tiny-models, adaptation-and-tuning, model-stability
Cited as “contested by” bySOTA-171 — Repeat a data-constrained corpus for up to about four epochs; past that, added compute stops paying
Cited as “introduced by” bySOTA-123 — For tiny specialized models, pretrain from scratch on the target SFT or reasoning data instead of pretrain-then-finetune, SOTA-125 — At a fixed tiny parameter budget, spend parameters on depth and SSM state width before MLP width, SOTA-126 — Run DPO on tiny models for one epoch only, SOTA-127 — Filter chain-of-thought traces out of the training data of tiny specialized models, SOTA-128 — Train fill-in-the-middle data without masking the loss on non-FIM tokens
Cited as “source” bySOTA-121 — Use Muon with decoupled weight decay and AdamW-matched update RMS in place of AdamW, SOTA-122 — Attach learnable per-row and per-column multipliers to weight matrices so their norms are learned, not set by LR and WD, SOTA-123 — For tiny specialized models, pretrain from scratch on the target SFT or reasoning data instead of pretrain-then-finetune, SOTA-124 — Repeat high-quality data freely when its epoch size exceeds the model's memorization window, SOTA-125 — At a fixed tiny parameter budget, spend parameters on depth and SSM state width before MLP width, SOTA-126 — Run DPO on tiny models for one epoch only, SOTA-127 — Filter chain-of-thought traces out of the training data of tiny specialized models, SOTA-128 — Train fill-in-the-middle data without masking the loss on non-FIM tokens, SOTA-131 — When training with Muon at scale, rescale query and key weights whenever attention logits exceed a threshold (QK-Clip), SOTA-145 — Estimate the RL baseline from a group of samples for the same prompt instead of training a critic, SOTA-240 — Apply dropout where the model can memorize what it is shown, and not where it cannot

Graph View

Falcon-LLM Team, Technology Innovation Institute (2026) — tiny-h1-blogpost. Not an arXiv paper; the release’s own citation is @misc{falcon_h1_tiny, author={Falcon-LLM Team}, year={2026}}.

Key takeaways

  • Fixing the model at 90M parameters and varying everything else shows that data strategy, architectural hyperparameters and the optimizer are three independent levers, each of which moves the result at this scale.
  • Muon, with weight decay applied and the update RMS scaled to match AdamW as in ARXIV-2502.16982, trains stably at nearly the same optimal learning rate as AdamW and evaluates better; every released model uses it.
  • Learnable multipliers on each row and column of every weight matrix (ARXIV-2601.04890) let the layer’s norm be learned instead of being fixed by the LR/WD equilibrium. Two otherwise identical 200 GT runs — Muon alone against Muon plus LRM, same data mixture, WSD with a 50 GT decay — give what the authors call a considerable and consistent boost across every benchmark tested; the blogpost reports it as a plot rather than a table, so the quantified version is the preprint’s (LIT-121), not this one’s.
  • A model forgets training tokens seen long enough ago. The authors call the delay after which the loss gap closes the memorization window — roughly 100 GT, or 500 GT conservatively, for FalconMamba-7B — and argue that a high-quality source whose epoch size exceeds the window can be repeated without overfitting. SFT sources like Tulu3 were repeated 100 or more times across the 800 GT of SFT-pretraining with no visible artefacts, which the authors offer as implicit confirmation rather than a measurement. They call this understanding early and leave the systematic study to future work.
  • Anti-curriculum: because a 90M model’s window scales down to about 5 GT, SFT data can make up the whole mix. Pretraining from scratch on 25% SFT data beat pretrain-then-SFT on IFEval (50.1 vs 40.8 before DPO, 66.1 vs 53.5 after), and one checkpoint serves as both base and instruct model; a further SFT stage on top added nothing. Pretraining on reasoning traces beat a reasoning-SFT stage the same way (AIME24 pass@16 6/30 vs 3/30, MATH500 0.4 vs 0.2). It did not help for multilingual or tool-calling data, where the two strategies tied.
  • Architecture at a fixed 90M budget: a 24–27 layer, 512 hidden configuration beat shallower and wider ones; a large SSM state dimension beat a large MLP expansion at every width tried; doubling depth to 50 layers helped MMLU but halved throughput. Vocabulary 32k, tied embeddings.
  • DPO works at 90M: one epoch at LR 1e-6 to 3e-6 takes IFEval from about 50 to over 65 while other benchmarks hold. A second epoch degrades the model even as the DPO reward keeps rising.
  • Chain-of-thought traces in the tool-calling mix sent the 90M model into repetition loops; filtering them out fixed it. Structured capabilities like tool calling looked capacity-bound rather than exposure-bound.
  • Fill-in-the-middle: train the loss on every token, not just the middle span; PSM format; include samples where the model must produce the indentation itself. Dropout 0.1 after the linear projections recovered HumanEval-FIM under the heavy repetition of a small FIM corpus.
  • A 0.6B model pretrained on reasoning traces alone, plus 60 GRPO steps at LR 3e-6, reaches AIME24 pass@1 75.0 and MATH500 94.0; GRPO is sensitive to the learning rate above all else.
  • Common recipe: LR 256e-5, WD 0.1, batch 4 MT, WSD over 800 GT (SOTA-140) with a 100 GT ×64 exponential decay and a power-law decay of the rate from 100 GT (SOTA-142), µP multipliers (SOTA-143) carried over from Falcon-H1 with the transfer done through forward multipliers.

Standing in the anthology

The first source in this record that is not a paper. It is here because it is the only published ablation of these choices at this scale, and because several of its findings are stated as recipes the reader can follow — the test a practice has to pass. Its evidence is single-run ablations at one size on benchmarks the authors themselves call noisy at 90M (BBH, GPQA, MuSR); every practice drawn from it is scoped to tiny models for that reason. The work it leans on for those decisions is filed alongside it: the Falcon-H1 report (LIT-120), the learnable-multiplier preprint (LIT-121), the Muon scaling report (LIT-122), the looping analysis behind its chain-of-thought filtering (LIT-123), the three FIM papers (LIT-124, LIT-125, LIT-126), GRPO (LIT-127), Falcon-H1R (LIT-128) and the tiny-DPO write-up (LIT-129), and Olmo 3 (LIT-130) for the three-stage reasoning recipe it departs from (SOTA-129).

Backlinks