Belkin et al. (2018) — ARXIV-1812.11118
Key takeaways
Empirically demonstrates and theoretically supports the “double descent” phenomenon: test error follows a double U-shaped curve as model complexity increases, with a peak at the interpolation threshold where training error first reaches zero. Shows this occurs for random feature models, neural networks, and decision trees. Challenges the classical bias-variance tradeoff by showing that continuing to add parameters beyond interpolation improves generalization. Introduces the term “interpolation threshold.”
- Empirical. Double descent peak at interpolation threshold across random features, neural networks, decision trees. Holds when: MNIST, CIFAR-10; multiple architectures
- Theorem (random features). For random feature ridge regression, test MSE has double descent as a function of the number of features. Holds when: Requires ridgeless limit (λ → 0)
What the evidence does not cover
- Double descent is not always visible in practice (regularization, data augmentation can suppress it).
- Interpolation threshold is not always sharp for nonlinear models.
Standing in the anthology
Read — the reading is NOTE-101. Arrived in the imported batch, which brought in the overparameterization line.