nucleation

LIT-971: Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data

2 min read

Properties5
StatusActive — read 2026-10-14 (NOTE-815) in full; worth reading for QUESTION-016 as an exact, bounded case of degradation along a reconstruction chain: in a fixed-design linear regression refit to its own labels, keeping all earlier data gives test error σ²d/(T−d−1)·Σ1/i², below π²/6 times the first fit's, where replacing the data gives linear growth. Read it knowing that the error still rises every round, that what is retained is the source sample in the training pool, not an interpreter's side information, and that one squared-error task is measured, not sufficiency. Filed as THEORY-289.
Version2
Tagslearning-theory, probabilistic-modeling, anthology-candidate
Cited as “paper” byNOTE-815 — Gerstgrasser et al. — Is Model Collapse Inevitable?
Cited as “source” byTHEORY-289 — For ordinary least squares on T ≥ d + 2 isotropic Gaussian covariates reused every round, with each round's labels generated by the previous fit, keeping every earlier round's data keeps the expected excess test error below π²/6 times the first fit's, though it rises every round, where replacing the data makes it grow linearly in the number of rounds

Graph View

Matthias Gerstgrasser, Rylan Schaeffer, Apratim Dey, Rafael Rafailov, Henry Sleight, John Hughes, Tomasz Korbak, Rajashree Agrawal, Dhruv Pai, Andrey Gromov, Daniel A. Roberts, Diyi Yang, David L. Donoho and Sanmi Koyejo (2024), arXiv:2404.01413.

Key takeaways

From the reading, NOTE-815.

  • The closed form. Real covariates X reused every round, labels regenerated from the previous fit with noise σ², ridgeless OLS, isotropic features, T ≥ d + 2. Accumulating all rounds’ data gives ŵ_n = w* + (XᵀX)⁻¹Xᵀ·Σ E_i/i and test error σ²d/(T−d−1)·Σ_{i≤n} 1/i² < σ²d/(T−d−1)·π²/6 (Theorems 1–2).
  • Against replacement. Replacing each round’s data gives σ²d/(T−d−1)·n (Dohmatob et al.); a newest-only set as large as the accumulated one gives about σ²d/(T−d−1)·log n. Keeping the earlier data, not the amount of data, is what bounds the error.
  • Bounded, not stopped. The sum rises every round; no model beats the first. Each is the first estimate plus independent noise.
  • Neural models, briefly. Language models on TinyStories: round-4 loss −0.06 to −0.28 nats under accumulation, +0.53 to +0.66 under replacement. GeoDiff holds under accumulation. A VAE on CelebA still degrades under accumulation, more slowly. Short, apparently single runs.

Standing in the record

Filed 2026-10-12 from the owner’s papers-feed for QUESTION-016, and read in full on 2026-10-14 in NOTE-815. The seed body called it “a case of retained side information stopping the loss of sufficiency”. The reading supports a narrower claim, filed as THEORY-289: an exact reconstruction chain in which degradation for one squared-error task is bounded because the source sample stays in the pool. The error still grows every round, every model is a garbling of the real sample, and side information at an interpreter, the case QUESTION-016 holds open, is not what is retained. It is also the record’s first closed-form instance of the bounded regime of CLAIM-267’s recursion, and a contrast to THEORY-155’s chain, which replaces its data and forgets its start. It keeps the anthology-candidate flag: its practical message, accumulate rather than replace, is advice for training, which an anthology topic could hold.

Backlinks