Source
Falcon-LLM Team (2026), LIT-119 — the Falcon-H1-Tiny technical blogpost.
What the memorization window is
A late checkpoint of FalconMamba-7B (Figure 9 of LIT-120), shown training tokens it saw earlier, has a loss gap against fresh tokens from the same distribution that decays with how long ago the tokens were seen. The authors define the memorization window as the delay after which that gap has closed — around 100 GT, or 500 GT as a conservative estimate, for a 7B model — and argue that a data source whose epoch size (its size divided by its share of the mix) is larger than the window can be repeated indefinitely without the model overfitting to it, while one with a small epoch size will be memorised. The window is assumed to scale linearly with parameters, giving about 5 GT at 100M.
The recommendation in the form stated here is LIT-119’s, the Falcon-H1-Tiny blogpost: it defines the window from the Falcon-H1 measurement, states the rule that a source can be repeated without limit when its epoch size exceeds it, scales the conservative 500 GT estimate linearly down to about 5 GT at 100M, and calls the whole thing a hypothesis, with a deeper study of repetition and forgetting listed as future work. The blogpost itself says multi-epoch training guided by the window was already explored for Falcon-H1, so the repetition argument is older than the rule as written here, and the record names Falcon-H1, LIT-120, as its origin. That report already makes the recommendation in a looser form: it “employ[s] an aggressive up-sampling strategy enabled by multi-epoch training” so that scarce high-quality corpora keep their influence, defines an epoch per data source, and says that by estimating the memorization window “we found it possible to reuse high-quality samples multiple times without compromising generalization.” What it does not state is the threshold — repeat without limit once the epoch size exceeds the window — or the linear scaling to small models; those are the blogpost’s.
This decouples the ceiling on the high-quality fraction of a mix from the total training length, which is what SOTA-123 relies on. Falcon-H1-Tiny is offered as an implicit confirmation: SFT sources such as Tulu3 were repeated 100 or more times across 800 GT of SFT-pretraining, and the memorisation sweep (0 to 100% SFT, 2 GT epoch size at the extreme) showed no degradation during training.
Why this is Proposed, and what it argues with
The source is explicit that this is a hypothesis with one measurement behind it and that the systematic study is future work. It is filed because the recipes built on it are in the record and this is the reason they give. The standing guidance it argues with is now in the record: LIT-166 finds four epochs nearly free and the value of added compute decaying to zero thereafter, across 400 runs. The claim here is that the relevant quantity is not epoch count but epoch size relative to the window — a distinction that may dissolve the conflict, since that sweep repeats a whole corpus at fixed compute while this is about one source inside a mixture. Nobody has run the experiment that separates them.
LIT-175 is a third position worth reading against both: that autoregressive pretraining overfits severely under heavy repetition, and that the overfitting belongs to the objective rather than to repetition, removable with augmentation. If that is right, a recipe repeating a source a hundred times is either augmenting implicitly or paying an unmeasured cost.
The linear scaling is now half-checked
promote_when: above asks for a measurement of the window at a second model
scale, so the linear-in-parameters assumption can be checked rather than
assumed. Morris et al., LIT-440, get part of the way there.
They measure transformer storage capacity by training on uniform random bitstrings, where generalization is impossible and the information content is exactly computable, and find ~3.6 bits per parameter, linear in parameter count across models spanning three orders of magnitude. That is the scaling shape this practice was taking from a single figure.
It is not the window. Falcon’s quantity is a token count — how much unique
data must sit between repeats of a source before repetition starts being
memorized. Morris’s is a bit count — how much the model can hold at all. The
two are related through the information content of the tokens, and that
conversion is the step nobody has taken. So the conjecture is half-checked,
promote_when: stands as written, and the status does not move.