Source
Muennighoff et al. (2023), LIT-166 — ARXIV-2305.16264.
The regime is the one the field has since entered: extrapolating the parameters-and-tokens trend runs into the amount of text that exists, so the binding constraint stops being compute and becomes data.
Both halves of the result are the practice. With constrained data at fixed compute, training on up to four epochs of repeated data changes the loss negligibly against having that much unique data — repetition is close to free. Past that point it is not: with more repetition, the value of adding compute decays toward zero. The first half licenses repetition and the second bounds it, and quoting either alone gets you a different recommendation.
A scaling law for compute optimality falls out, pricing in the decreasing value of repeated tokens and of excess parameters, validated empirically across a sweep up to 900B tokens and 9B parameters. 400 training runs, with models and datasets released.
The four-epoch bound and the law that prices it are Muennighoff et al.’s own reading of that sweep in LIT-166, which is where this recommendation originates.
Two mitigations for data scarcity are tested alongside: adding code data to the mixture, and removing commonly used filters.
Contested, and the disagreement is about the quantity
SOTA-124 says the quantity governing safe repetition is not epoch count but epoch size relative to the model’s memorization window, and on that basis Falcon-H1-Tiny repeated SFT sources a hundred times or more. The dispute is published in the Falcon-H1-Tiny write-up LIT-119, which reports Tulu3 and similar sources repeated 100 or more times across 800 GT with no visible artefacts — a direct counterexample to a four-epoch ceiling, offered as implicit confirmation rather than a measurement. This practice says four, and it has 400 runs behind it where that one has one figure and an argument its own authors call early.
The two are not quite measuring the same thing, which is the interesting part. This sweep repeats a whole corpus under a fixed compute budget; the memorization-window claim is about one high-quality source inside a mixture, where a small share means a long delay between repeats. That distinction may dissolve the conflict — or it may be the mechanism by which a 100× repetition quietly costs something nobody measured.
LIT-175 is a third position and the most useful of the three, because it makes the other two falsifiable in the same terms: it says repetition overfits severely and the overfitting is a property of the objective rather than of repetition, removable with augmentation. If that holds, a recipe repeating a source a hundred times is either augmenting implicitly or paying a cost nobody measured.
Why Active while contested
status: is this record’s editorial position and consensus: is the
field’s; they are allowed to disagree, which is the whole point of the second
axis. Four epochs from 400 released runs is the best-evidenced number
anybody has published for this question, and a reader who needs a number
today should use it. That it is disputed by two positions with different
methodologies is what contested says.
Reproduced independently, and relocated
Prabhudesai et al., LIT-442, re-ran this study — same corpus, same
recipe, same parametric form — with the training objective swapped from
autoregressive to masked diffusion. The autoregressive arm recovers
R* = 31.93, which is this practice’s four-epoch bound in its source’s own
units.
That is a replication, and it changes what the number is a statement about.
The same fit for masked diffusion gives R* = 512.85. So the bound is a
property of the autoregressive objective rather than of repetition, and a
reader whose corpus is fixed has a second lever — change the objective —
that this practice’s framing does not offer.
Kim et al., LIT-441, point at a third lever from a different direction: with weight decay tuned rather than inherited, the overfitting that bounds multi-epoch training largely goes away (SOTA-255). Neither result disputes the four epochs. Both say it was measured under conditions nobody was varying.
Known implementations
- None reports having used the four-epoch bound as a design rule; the record’s recipes either repeat far more (Falcon-H1-Tiny) or do not say.