Contribution
Empirically demonstrates and theoretically supports the “double descent” phenomenon: test error follows a double U-shaped curve as model complexity increases, with a peak at the interpolation threshold where training error first reaches zero. Shows this occurs for random feature models, neural networks, and decision trees. Challenges the classical bias-variance tradeoff by showing that continuing to add parameters beyond interpolation improves generalization. Introduces the term “interpolation threshold.”
Key insight
Classical statistics predicts a U-shaped bias-variance tradeoff: more capacity → lower bias but higher variance → optimal capacity somewhere in between. Modern ML practice (overparameterized neural networks, random forests) violates this: after the interpolation threshold, adding more capacity reduces test error again. This “double descent” occurs because overparameterized models interpolate in a “benign” way — they find the minimum-norm interpolant, which generalizes well. The gossip/sync gap follows the same pattern: at H below the threshold, permutation misalignment is costly; above the threshold, the loss landscape is flatter and averaging is less damaging.
Assumptions
- Empirical observations across multiple model families
- Random feature model theoretical analysis
- IID train/test split
Key results
- Empirical. Double descent peak at interpolation threshold across random features, neural networks, decision trees. Holds when: MNIST, CIFAR-10; multiple architectures
- Theorem (random features). For random feature ridge regression, test MSE has double descent as a function of the number of features. Holds when: Requires ridgeless limit (λ → 0)
Claims
| id | claim | strength | support |
|---|---|---|---|
| C1 | Overparameterized models (beyond interpolation) consistently generalize better than at the threshold. | strong | Empirical across multiple architectures and datasets |
| C2 | The minimum-norm interpolant (implicit bias of GD) explains why overparameterization helps. | moderate | Theoretical for linear models; informal for nonlinear |
Concepts
- Interpolation threshold — The number of parameters P at which training loss first reaches zero; equivalently, where the model can exactly fit the training data.
- Double descent — Non-monotone test error: decreases, peaks at interpolation threshold, then decreases again.
- Minimum-norm interpolant — The solution with minimum parameter norm that achieves zero training loss; found by gradient descent with zero initialization.
Connections
Builds on.
- Advani & Saxe 2017 (High-dimensional dynamics) — Earlier observation of double descent in random matrix theory.
Related.
- Goldt et al. 2020 (LIT-340) — Goldt’s ODE framework describes the dynamics through the interpolation threshold.
Bearing on the record
Double descent is the reason the record’s practices about model size do not have to trade off against fitting the training set. No document here states it, and several practices assume it.
Limitations
- Double descent is not always visible in practice (regularization, data augmentation can suppress it).
- Interpolation threshold is not always sharp for nonlinear models.
Open questions
- Does the gossip penalty track test MSE or just the double-descent peak in MSE?
- Can Polyak-Ruppert averaging suppress the gossip penalty near the threshold?