Source
Zhu et al. (2024), LIT-205 — ARXIV-2412.14689.
The premise is not hypothetical. As AI output proliferates, future models will be trained on a blend of synthetic and human text whether or not anyone chooses that, so the question is not whether to use synthetic data but what happens when you do.
What happens, measured. Pretraining across different proportions of synthetic data yields a negative correlation between the proportion and performance. Statistical analysis locates why: distributional shift, and over-concentration of n-gram features. Generated text is narrower than what it imitates, and training on it narrows the next model further.
The remedy inverts the usual move. The instinct is to generate better synthetic data. Instead: edit human data at the token level to produce semi-synthetic data. Because the result stays anchored to a real distribution, the authors prove the test error is bounded above — collapse is prevented by construction rather than by filtering harder afterwards.
Validated on pretraining from scratch, continual pretraining and supervised fine-tuning.
All of that is LIT-205: Zhu et al. measured the negative correlation, traced it to distributional shift and n-gram over-concentration, and proposed token-level editing of human text as the remedy, so the recommendation originates with the paper that measured the problem it answers.
Both ends of the pipeline, and the record uses both without saying so
- Generation. SOTA-124 and the recipes in LIT-119 and LIT-131 lean on rephrasing — Kimi K3 rephrases its knowledge and mathematics corpora with diverse prompting and verifies fidelity against the source. That is generation conditioned on human text, which is nearer to this paper’s semi-synthetic prescription than to the pure synthesis it warns about. The difference is one of degree, and this practice is what makes the degree visible rather than a matter of taste.
- Filtering. DeepSeek-V4 (LIT-139) cites this paper for the other end: filtering batched auto-generated and templated content out of web data, explicitly “to mitigate the risk of model collapse”.
So the record already had both uses — a reason to filter the crawl, and a constraint on how to generate — and no document saying they are the same finding.
Conditions, and why this is Proposed
One group. The bound is proved for the semi-synthetic construction rather
than measured at frontier scale, and the negative correlation is measured at
this paper’s scale rather than at the scale of the recipes that rephrase. The
practice is a constraint on method, and the record’s adopters satisfy it by
accident rather than by design — which is exactly the situation Proposed
is for.
Read with the filtering-horizon practice drawn from LIT-178, which recommends rephrasing what a filter would discard at long token horizons: that is the same operation, and this is the paper that says why it is safe when generating from scratch is not.
Known implementations
- None by name. Kimi K3’s verified rephrasing and DeepSeek-V4’s collapse filtering are the nearest things, and neither cites the construction.