nucleation

LIT-617: The Full Spectrum of Deepnet Hessians at Scale: Dynamics with SGD Training and Sample Size

2 min read

Properties5
StatusActive — read 2026-10-03 (NOTE-478); worth reading as the measurement that the bulk-and-C-outliers spectrum of the Hessian persists at full scale (VGG11, 28M parameters), that the outliers belong to the Gauss–Newton/Fisher term G and the bulk mostly to the residual H, and, in its second version, that G splits into class means (outliers) and two bulks (cross-class and within-class variation). Its tools are stochastic Lanczos spectral density estimation without reorthogonalization, plus subspace-iteration deflation.
Tagsinformation-geometry, learning-theory, anthology-candidate
Extended byLIT-613 — Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra, LIT-619 — Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians
Cited as “paper” byNOTE-478 — The Full Spectrum of Deepnet Hessians at Scale
Cited as “source” byTHEORY-078 — The top C eigenvalues of a trained C-class classifier's Fisher are carried by the second moment of its class-mean logit derivatives, so a cut at the bulk edge keeps the span of the class means, and they stand clear of the bulk to the extent between-class separation exceeds within-class variation

Graph View

Vardan Papyan (2018), arXiv preprint — ARXIV-1811.07062

Key takeaways

  • Bulk and outliers at full scale. Earlier reports (Sagun et al.) used networks with thousands of parameters. On VGG11 with 28M parameters, train and test Hessians show a bulk and, “arguably”, C outliers, plus negative eigenvalues in the train Hessian after hundreds of epochs (Figure 2).
  • Outliers come from G, the bulk mostly from H. With the Gauss–Newton split Hess = H + G, where G = Ave ∂fᵀ ∇²ℓ ∂f is the generalized Gauss–Newton (for cross-entropy, the Fisher) and H the residual, H has no outliers and its spectrum tracks the Hessian’s bulk (Figure 3). H is “never negligible compared to G”, contrary to Sagun et al.’s suggestion that Hess ≈ G.
  • H is heavy-tailed. Its spectrum is nearly symmetric about zero and fits φ = 1.2×10⁻⁴|λ|⁻²·⁷ with R² 0.99 (Figure 4), which “can not originate from Wigner’s semicircle law”.
  • G has three levels (v2, Eq. 5). With gᵢ,c,c′ the gradient of example i of class c as if its label were c′, G = A₁ + A₂ + B₁ + B₂: A₁ from the C class means (the outliers), B₁ from the spread of cross-class means about them (a left bulk), B₂ from within-pair variation (the main bulk), A₂ negligible (Figure 1). Training longer separates the two bulks; more data draws them together (Figure 7).

Standing in the record

Filed on 2026-10-03 at the owner’s request with Papyan’s ICML 2019 paper (LIT-619) and 2020 JMLR paper (LIT-613). The owner’s citation “Papyan (2019; 2020 JMLR)” names the 2019 and 2020 papers; this preprint is filed as well because its second version carries the same class/cross-class decomposition of G, measures the two bulks the ICML paper could not see, and supplies the spectral-density machinery both later papers use. The two 2019-era papers cite each other: the ICML paper uses this one’s FastLanczos and its attribution of outliers to G, and this one’s v2 adopts, “slightly modified”, the ICML paper’s decomposition, citing it as “Anonymous [2019]”.

It is filed under information-geometry because G here is the Gauss–Newton matrix, equal for softmax cross-entropy to the Fisher, and the paper is about its spectrum; and under learning-theory because the author’s stated motivation is the sharpness–generalization question. A deep-learning measurement an anthology topic could hold, so anthology-candidate.

Backlinks