anthology-of-the-sota

LIT-120: Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance

2 min read

Properties5
StatusActive
Tagsmodel-architecture, data-pipeline
Compared againstLIT-799 — Olmo Hybrid: From Theory to Practice and Back
Cited as “introduced by” bySOTA-124 — Repeat high-quality data freely when its epoch size exceeds the model's memorization window
Cited as “source” bySOTA-124 — Repeat high-quality data freely when its epoch size exceeds the model's memorization window, SOTA-125 — At a fixed tiny parameter budget, spend parameters on depth and SSM state width before MLP width

Graph View

Zuo et al. (2025) — ARXIV-2507.22448

Key takeaways

  • The mixer block runs Transformer attention heads and Mamba-2 state-space heads in parallel rather than interleaving layers of each, so the split between the two is a per-block channel allocation that can be tuned, not a layer pattern.
  • Released at 0.5B, 1.5B, 1.5B-deep, 3B, 7B and 34B in base and instruction-tuned form; the 1.5B-deep variant is the first public data point for the claim that depth, at fixed parameters, buys performance comparable to models several times larger.
  • A customised Maximal Update Parametrization (LIT-148, SOTA-143): the multipliers for learning rate, weight decay and the forward pass are tuned once and transferred, which is what lets LIT-119 reuse 35 tuned values at 90M.
  • Multi-epoch training on high-quality data, made safe by measuring how long the model remembers individual training tokens. The data-rollback experiment on FalconMamba-7B — replay tokens seen earlier and read the loss gap against fresh ones — shows the gap decaying with how long ago the tokens were seen. That figure is the whole empirical basis of the

memorization-window argument in LIT-119 and SOTA-124.

  • The tokenizer splits digits and punctuation and injects common LaTeX tokens; the authors attribute part of the math and code results to it.

Standing in the anthology

Filed as the paper LIT-119 is a continuation of: the architecture, the µP transfer and the repetition argument all originate here. Its own claims about hybrid efficiency at 7B–34B are not yet drawn into a practice — the record’s hybrid-attention practice (SOTA-132) is sourced to the interleaved layout of LIT-133, and the two layouts are contrasted there; the

one practice that rests on it directly is the Proposed SOTA-124.

Olmo Hybrid (LIT-799) lists Falcon-H1 7B as the closest parallel-head hybrid in its table of open-weight models. Falcon-H1 was trained on 12T tokens against Olmo Hybrid’s 6T, and it is ahead on every OlmoBaseEval average except GenQA (Math 65.7 against 55.1, Code 45.3 against 32.4). The data and budgets differ, so the row says nothing about which layout is better.

Backlinks