nucleation

LIT-619: Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians

3 min read

Properties6
StatusActive — read in full 2026-10-03 (NOTE-479); worth reading as the paper that explains the C outliers of the deepnet Hessian: the Gauss–Newton/Fisher term G is a second moment, not a covariance, of logit derivatives indexed by (example, class, logit), and the C outliers are the eigenvalues of G₁ = (C−1)·Ave_c δ_c δ_cᵀ, the C class-cluster centres, computable from a C×C Gram matrix without high-dimensional eigenanalysis. The match is empirical, over 18,000 trained models.
Tagsinformation-geometry, learning-theory, anthology-candidate
ExtendsLIT-617 — The Full Spectrum of Deepnet Hessians at Scale: Dynamics with SGD Training and Sample Size
Extended byLIT-613 — Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra
Cited as “paper” byNOTE-479 — Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians
Cited as “source” byTHEORY-078 — The top C eigenvalues of a trained C-class classifier's Fisher are carried by the second moment of its class-mean logit derivatives, so a cut at the bulk edge keeps the span of the class means, and they stand clear of the bulk to the extent between-class separation exceeds within-class variation, THEORY-731 — For softmax cross-entropy with balanced classes, the Gauss–Newton matrix splits exactly into four positive semidefinite second-moment terms, the class-centre term has rank at most C with eigenvectors in the span of the class centres and the spectrum of a C × C Gram matrix, and so the k-th eigenvalue of the whole is never below the k-th eigenvalue of the class-centre term

Graph View

Vardan Papyan (2019), ICML 2019 — ARXIV-1901.08244

Key takeaways

  • G is a second moment, and that is why it has outliers. Writing the softmax Hessian as (I − 1pᵀ)ᵀdiag(p)(I − 1pᵀ) (Lemma 2.1) gives G = (1/n)ΔΔᵀ with columns δᵢ,c,c′ = √pᵢ,c,c′ × (centred logit derivative) (Eqs. 13–16). No mean is subtracted, so large means make large eigenvalues. G is a second moment of logit derivatives, not of loss gradients: δᵢ,c,c is a scaled gradient, but δᵢ,c,c′ for c′ ≠ c are not.
  • The class/cross-class block structure is a two-way layout. Indexing by class c and logit c′ gives C² groups with means δc,c′ and covariances Σc,c′; the off-diagonal means of each class form a cluster with centre δc and covariance Σc. Then G = Ave_c δc,cδc,cᵀ (G₀) + (C−1)Ave_c δcδcᵀ (G₁) + (C−1)Ave_c Σc (G₂) + (1/C)Σ Σc,c′ (G₃) (Eq. 28). The top-C eigenvalues of G match those of G₁, and λc(G) ≥ λc(G₁₊₂) ≥ λc(G₁) (Figure 7).
  • Cheap principal subspace. The outliers can be read from the C×C Gram matrix of class centres, “much simpler even than the power method”.
  • The structure forms during training. In ImageNet t-SNE plots, group means first cluster by logit coordinate c′, then by true class c, the switch near epoch 18 for ResNet50 and 10 for VGG16 (Figure 5).

Standing in the record

Filed on 2026-10-03 at the owner’s request: the owner’s “Papyan (2019)”, the first of the two papers he names as the “Fisher-spectrum block structure, the object being thresholded”. The block structure it finds is in G, which for softmax cross-entropy is the Fisher of the model’s predictive distribution averaged over training inputs (the paper calls it the Gauss–Newton component; the 2020 paper calls the same matrix the Fisher Information Matrix).

It extends Papyan’s preprint (LIT-617), and could not stand without it: the spectral densities in Figure 6 are computed with that paper’s FastLanczos, and its starting point is that paper’s finding that the Hessian’s outliers belong to G rather than H. The 2020 JMLR paper (LIT-613) extends this one in turn.

What it bears on in the record:

  • K-FAC (LIT-622). This paper finds the dominant directions of G indexed by class, which a per-layer Kronecker factorization does not represent; the 2020 paper turns that into a measured failure of K-FAC.
  • Neural collapse (LIT-618) cites this paper and its preprint as the spectral observation that collapse explains: as within-class variability vanishes, class means dominate and the outliers separate.
  • learning-theory: the paper’s motivation includes sharp-minima and generalization arguments that use the Hessian spectrum. anthology-candidate as deep-learning measurement.

Backlinks