Source
Ye et al. (2024), LIT-201 — ARXIV-2403.16952.
Pretraining corpora are mixtures of domains, the proportions matter a great deal, and they are set by heuristics and qualitative argument — not because anyone prefers that, but because the alternative appeared to require training a model per candidate mixture.
The finding is that it does not. Performance is quantitatively predictable as a function of the proportions, in a fittable functional form. Fit it on a sample of mixtures and you can read off performance on mixtures you never trained.
The nesting is what makes it affordable. Compose the mixing law with the existing scaling laws over training steps and model size, and small-scale runs predict a large model on a large token budget under arbitrary mixtures. The search stops needing runs at the target scale, which is the whole cost argument.
What it bought
The recommendation is LIT-201’s: Ye et al. proposed the data mixing law, argued for fitting proportions instead of setting them by judgement, and validated it with the run below.
Optimising the mixture for a 1B model over 100B RedPajama tokens reached performance comparable to the default mixture trained for 48% more steps. Extended to continual training, the law predicts the critical proportion at which catastrophic forgetting is avoided.
The same move the record already makes elsewhere
SOTA-142 sets the peak learning rate by a fitted power law instead of a sweep, and LIT-145 fits scaling laws from one run’s checkpoints instead of a run per duration. This is that move applied to the data mixture — converting a judgement call into a fitted prediction — and it is filed so the three are legible as the same idea.
Contested: the fitted quantity may not be continuous in scale
Gu et al., LIT-451, measure knowledge acquisition from a knowledge-dense dataset mixed into web text, and find thresholds rather than a curve: below a critical model size, or a critical mixing ratio, the model acquires almost nothing however long it trains. The critical ratio follows a power law in model size.
Fitting a mixing law on small runs and extrapolating assumes the quantity being fitted moves continuously with scale. Where a domain’s transition falls between the proxy scale and the target scale, it does not — the small run is then measuring a different regime, and the extrapolation has nothing to say about the one that matters.
This is not a refutation. The method was measured and works where it was measured, and a mixture whose domains all sit on the same side of their transitions at both scales will transfer fine — plausibly the common case for the large web domains it was validated on. The exposure is to exactly the case the method is most wanted for: small, high-value, knowledge-dense domains at low share, where the transition is most likely to fall in between. SOTA-268 is the practice drawn from that finding.
Conditions, and why this is Proposed
One group, one functional form, no reproduction and no dispute. More to the point: no frontier report in this record says it used a mixing law. They report “domain-specific sampling rates determined by ablation studies on smaller models” (Kimi K3) or say nothing. That is the shape of this method without the functional form, so the honest reading is that the field is doing the expensive version and has not adopted the cheap one — which is a reason to hold the practice provisionally, not a reason to leave it unfiled.
Known implementations
- None reported. The nearest thing in the record is small-model ablation, which is the search this replaces rather than an instance of it.