Source
Chen et al. (2026), LIT-175 — ARXIV-2606.16246.
The premise is the regime: compute capacity is outrunning the rate at which new high-quality text is produced, so pretraining is moving to a data-constrained, compute-abundant setting where multi-epoch training on a fixed corpus has to be made productive.
The failure it starts from is unambiguous. Standard autoregressive pretraining overfits severely in this setting — it reaches its optimum early and then deteriorates continuously. Not a plateau; a decline.
The three families, which are orthogonal
- Token-level noise — masking, random replacement. Random token replacement is the best single method.
- Sequence permutations — right-to-left prediction, fill-in-the-middle.
- Target offset prediction — predict x_{t+i} for i > 1.
Each individually delays overfitting and lowers validation loss against the baseline; combining categories lowers the minimum further. The claim is hundreds of productive epochs on the same data.
The overfitting baseline, the three families and the claim all come from Chen et al. in LIT-175, which proposes augmenting the objective as the remedy — this practice is that paper’s recommendation, and the only outside tests of it are the two recorded below.
The reframing is worth more than the recipe
The record now holds three positions on repetition and this is the third.
SOTA-124 says heavy repetition is safe when epoch size exceeds the memorization window; SOTA-171 says four epochs and then diminishing returns; this says repetition overfits severely and the overfitting is a property of the objective rather than of repetition, removable with augmentation.
That framing is the useful one, because it makes the other two falsifiable in the same terms: if augmentation is what buys productive multi-epoch training, then a recipe repeating a source a hundred times is either augmenting implicitly or paying a cost nobody measured.
Same transformation, three purposes
Fill-in-the-middle appears here as a sequence-permutation augmentation valued for regularisation. SOTA-174 records it as a capability to be trained for. And LIT-119 reached for dropout when heavy repetition of a small FIM corpus hurt HumanEval-FIM — the same corpus, the same transformation, and a regularisation problem solved by a third mechanism. Three purposes for one transformation, which is worth knowing before treating any one of them as the reason it is in your pipeline.
Conditions, and why this is Proposed
One group, one paper, published this year, and nothing in the record trains under it. The augmentation families are also evaluated as delays to overfitting rather than as end-of-training quality at a fixed budget, which is the comparison a reader choosing between “augment” and “get more data” would want.
Tested from outside its own source, twice
Two results now bear on this practice, and the record should say so plainly: neither refutes it, and neither supports it either.
The augmentation arm was run and did not close the gap. Prabhudesai et al., LIT-442, applied random token masking and attention dropout — two of the three families above — to an autoregressive model under exactly the multi-epoch, data-constrained conditions this practice describes. The model still overfit quickly and still trailed a masked diffusion model trained past 500 epochs. Two families is not three and delay is not removal, so the claim as written survives; but this is the first evidence in the record on it that does not come from its own source, and it points the other way.
A cheaper remedy exists for the same diagnosis. Kim et al., LIT-441, agree that the overfitting belongs to the recipe rather than to repetition, and fix it by raising weight decay roughly thirtyfold (SOTA-255). That is one hyperparameter sweep against three new objective terms.
The reframing this practice contributed is holding up better than its recipe.
Known implementations
- None in the record.