Source
Falcon-LLM Team (2026), LIT-119 — the Falcon-H1-Tiny technical blogpost.
The claim
From ARXIV-2601.04890 (LIT-121) and validated in the blogpost at the 90M scale: under a decoupled weight decay, a matrix layer settles into an equilibrium norm determined by the learning rate and the weight decay coefficient rather than by the data. Attaching a learnable scalar multiplier to each row and each column of every weight matrix (the authors’ learnable multipliers, LRM) lets the norm be learned, and lets the forward multipliers that µP would otherwise fix be learned too.
Evidence: two 200 GT runs (50 GT of decay), Muon alone against Muon with LRMs, on the final 90M architecture. Improvements on most benchmarks, up to a 20% relative gain on MMLU, BBH and GSM8K — “from random values”, in the source’s own words, a qualifier this practice used to drop. At 90M those three benchmarks sit near chance, so a fifth of the distance above chance is a much smaller claim than a fifth of the score, and it is the claim the blogpost makes. The authors then used LRMs for every model in the series.
What LIT-119 adds to LIT-121’s claim is that validation and nothing more: one pair of runs on the Falcon-H1-Tiny architecture, reported as improvements on most benchmarks with that one relative figure in the text, and a figure rather than a table of scores. The equilibrium argument and the larger-scale numbers are the preprint’s.
Why this is Proposed, and how the condition changed
One team, one architecture family, and the larger-scale results live in the preprint rather than in this source.
The condition as first written — “an independent reproduction or a result above 1B” — has since been half-met in a way that shows the or was doing too much work. LIT-153 is an independent group, at up to 1.2B, working on exactly the phenomenon this practice rests on: the weight-decay equilibrium that fixes a matrix’s norm by hyperparameters rather than data. They confirm the diagnosis and go further, arguing that equilibrium is why matrix optimizers’ advantage shrinks with scale.
But their remedy is the opposite of this one. Learnable multipliers give the scale its own learned parameter; Hyperball pins the Frobenius norms to constants and removes the degree of freedom. So an independent group above 1B has confirmed the mechanism and declined the fix, which the original condition would read as a promotion and which plainly is not one.
Known implementations
- Falcon-H1-Tiny (all released checkpoints)