Charles H. Martin and Michael W. Mahoney (2017), arXiv preprint — ARXIV-1710.09553
Key takeaways
- Two practical knobs as statistical-mechanics control parameters. Randomizing m_rand of m labels lowers an effective load α = (m − m_rand)/N while the trained capacity N stays tied to m. Early stopping raises an effective temperature τ, by analogy with Langevin dynamics and the fluctuation–dissipation theorem (Claim 1).
- The right limit is the thermodynamic one. Hypothesis class and data grow together with α = m/N fixed, not m → ∞ for a fixed class (Claim 2). In that limit, generalization can change discontinuously at αc, and a temperature above τc can remove the transition (Claim 3, Figure 1).
- The mechanism is entropy against energy. For the continuous perceptron the entropy of hypotheses at error ε behaves as ln ε, giving ε ~ 1/α smoothly (Eqs. 6–8). For the Ising perceptron it behaves as −(π²/2)ε² ln ε, the first-order condition −π²ε ln ε = α has no solution for large α, and the error jumps to zero at αc (Eqs. 9–13). The rigorous version (Haussler, Kearns, Seung and Tishby) bounds error by the rightmost crossing of s(ε) and −α log(1 − ε) (Eqs. 14–18, Figure 3).
- Why “regularize harder” fails. Tikhonov and truncated-SVD regularization can always trade overfitting for underfitting because the problem is linear. A nonlinear system in a spin-glass-like phase has no such global knob, and only τ (early stopping) moves it out (Section 4.5, Conclusions 1–2).
Standing in the record
Filed on 2026-10-03 with the owner’s batch on mode connectivity and model merging (ADR-027), as the statistical-mechanics root of its generalization papers. It is the first paper in the record to argue that the statistical-mechanics limit, not the uniform-convergence limit, is the one in which deep-learning generalization should be studied.
Its descendants in the batch say so. Yang et al. (LIT-663) measure its
phase diagram on real networks, with the same load-like and
temperature-like parameters. Liao, Couillet and Mahoney (LIT-679)
derive an exact phase transition in random-feature regression and cite this
paper for reading double descent as the transition between two phases.
Both declare extends to it.
Against the record’s learning theory. The record’s information-theoretic bound on generalization (THEORY-028) is a worst-case-style bound of the kind this paper argues is coarse. It is smooth in the information the output carries, and it says nothing about phases. This paper’s thermodynamic-limit argument and Watanabe’s singular learning theory (LIT-616) share a premise: the asymptotics that matter keep the model’s structure in the limit instead of letting n run away from it. They are different theories, and this paper does not cite Watanabe.
Spectra. The paper’s remark on later work points to the authors’ heavy-tailed self-regularization (their analysis of weight-matrix eigenvalue spectra with heavy-tailed random-matrix theory). That later work, not this paper, is what Papyan’s Hessian measurements (LIT-617) name as possibly related to the power-law tail of H, the residual of the Gauss–Newton split of the Hessian, which carries its bulk. The record does not hold the heavy-tailed self-regularization paper itself (Martin and Mahoney, “Implicit self-regularization in deep neural networks”, arXiv 1810.01075).
Against the information bottleneck. It names Shwartz-Ziv and Tishby (LIT-324) as concurrent work “with a similar flavor” and does not engage it. The record has since rejected the compression account of generalization (THEORY-035).
Machine-learning theory with practical claims about regularization, so
anthology-candidate.