Why it is a question
THEORY-289 now states a theorem for one model: ordinary least squares on T ≥ d + 2 isotropic Gaussian covariates, reused every round, with only the labels regenerated from the previous fit. Accumulating every round’s data keeps the expected excess test error below π²/6 times the first fit’s, because round i’s noise enters with weight 1/i; replacing the data makes it grow linearly. The mechanism, 1/i weighting of each round’s noise, depends on the covariates being shared and the estimator being linear in the labels. Whether the bound survives when covariates are also generated by the model, or for nonlinear and over-parameterised fits, is not shown. The neural runs in the same paper are five to eight rounds and show no divergence over a short run, which is a different thing.
What would answer it
A closed form or bound for accumulated data when covariates are regenerated, or for a nonlinear or over-parameterised class, with its limit computed and checked against simulation over many rounds, would answer it yes for that class. A case in which accumulated data, with the original sample kept, still gives error growing without bound in the number of rounds would answer it no. More short runs of neural models with flat loss cannot settle it.