anthology-of-the-sota

SOTA-125: At a fixed tiny parameter budget, spend parameters on depth and SSM state width before MLP width

2 min read

Properties7
StatusProposed
Version3
Consensusunreplicated
Tagsmodel-architecture, tiny-models
SourceLIT-119 — Falcon-H1-Tiny: A series of extremely small, yet powerful language models redefining capabilities at small scale, LIT-120 — Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance
Introduced byLIT-119 — Falcon-H1-Tiny: A series of extremely small, yet powerful language models redefining capabilities at small scale
Compared againstSOTA-190 — Increase depth before any other dimension when scaling a transformer

Graph View

Source

Falcon-LLM Team (2026), LIT-119 — the Falcon-H1-Tiny technical blogpost.

The ablations

Holding a hybrid Mamba/attention model at 90M parameters and moving parameters between axes, on a STEM-heavy mix:

  • Depth vs width (200 GT): the mid-depth 27-layer, 512-hidden configuration beat the shallow one on commonsense tasks by a wide margin (HellaSwag); the 50-layer configuration gained further on MMLU and MMLU-Pro but cost about 2× in training throughput, and was not chosen. The release uses 24 layers to land on 90M.
  • MLP factor vs SSM dimension (70 GT, loss as the proxy): every variant that grew the MLP by shrinking the SSM state dimension descended slower. “SSM capacity is more valuable than large feed-forward width for a tiny model.” For a fixed SSM dimension there is an optimal hidden-size to MLP ratio, and it sat near hidden 512.
  • KV heads: more KV heads, paid for out of the MLP, helped up to a point; the baseline remained best.

These ablations, and the rule drawn from them, are LIT-119’s. The depth result was not unanimous there: BBH favoured the shallow configuration, and the authors chose to weight English commonsense (HellaSwag) because their STEM-heavy mix would lag on it anyway.

Against depth-first scaling

SOTA-190 states the same preference from Tay et al.’s T5 experiments: at matched quality, deep-narrow configurations need fewer parameters and FLOPs (Small 16L comparable to Base at 134M against 223M parameters), with parallelism as the limit. Nobody ran the two side by side; they are separate measurements of one question, on a 90M hybrid Mamba/attention model here and T5 up to XXL there, and they agree on both the direction and the cost — the 50-layer model’s MMLU gain here came at about half the training throughput, which is the serial-depth trade that practice names as its condition.

Why this is Proposed

The evidence is one architecture family, one scale, loss curves and noisy 90M benchmarks. The depth finding echoes the authors’ Falcon-H1-1.5B-Deep result at a larger scale (LIT-120). The trade the deep option loses on is throughput, which is the reason to state this as a parameter-budget rule rather than a compute-budget one.

Backlinks