nucleation

LIT-613: Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra

4 min read

Properties7
StatusActive — read 2026-10-03 (NOTE-474); worth reading as the general account of class/cross-class structure in deepnet spectra: features, backpropagated errors, gradients, Fisher, Hessian and weights all show one outlier from the global mean, C−1 from class means, a mini-bulk of about C(C−1) from cross-class means, and a main bulk from within-class variation. Per layer, gradient class means are approximately Kronecker products of feature class means and error class means, which is why K-FAC, pairing every class with every class, misaligns the Fisher's outliers, and CFAC, pairing each class with itself, aligns them. Proved only for multinomial logistic regression on a Gaussian model.
Tagsinformation-geometry, representation-learning, learning-theory, anthology-candidate
ExtendsLIT-619 — Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians, LIT-617 — The Full Spectrum of Deepnet Hessians at Scale: Dynamics with SGD Training and Sample Size
Compared againstLIT-622 — Optimizing Neural Networks with Kronecker-factored Approximate Curvature
SupportsCLAIM-391 — For any classes with weights π_c and per-class factor matrices H_c and Δ_c, the class-wise Kronecker product Σ_c π_c H_c ⊗ Δ_c exceeds the product of the weighted averages H̄ ⊗ Δ̄ by exactly the weighted between-class covariance Σ_c π_c (H_c − H̄) ⊗ (Δ_c − Δ̄), so the two agree when either factor is the same in every class
Cited as “paper” byNOTE-474 — Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra
Cited as “source” byTHEORY-078 — The top C eigenvalues of a trained C-class classifier's Fisher are carried by the second moment of its class-mean logit derivatives, so a cut at the bulk edge keeps the span of the class means, and they stand clear of the bulk to the extent between-class separation exceeds within-class variation, THEORY-083 — In Papyan, Han and Donoho's runs of VGG, ResNet and DenseNet trained with cross-entropy on seven balanced datasets without augmentation, training past zero error shrinks last-layer within-class variability against between-class variability, and moves the centred class means and the classifier toward a common simplex equiangular tight frame and the decision toward nearest class mean, nearly to completion on the four easier datasets and only partly on ImageNet, THEORY-085 — A classifier's class-driven Fisher outliers pair each class's feature mean with that class's own error mean, so K-FAC's single Kronecker product of class-averaged factors adds every cross-class pairing and misplaces them, THEORY-788 — Training a deep classifier past zero training error generally collapses its last-layer features onto their class means, drives the centred means toward a simplex equiangular tight frame aligned with the classifier, and makes the decision nearest class mean, as Papyan, Han and Donoho propose from their runs

Graph View

Vardan Papyan (2020), Journal of Machine Learning Research 21(252):1–64 — ARXIV-2008.11865

Key takeaways

  • One pattern, every object. With C classes, a spectrum shows one large outlier, C − 1 secondary outliers, a mini-bulk of about C(C − 1) eigenvalues and a main bulk (Figure 1). The causes are the second moment of the global mean, the between-class second moment, the between-cross-class covariance and the within-cross-class covariance, shown by knocking each out (Section 3, Table 1). The Fisher decomposes as G = Gclass + Gcross + Gwithin + Gc=c′ (Eq. 6.4).
  • The Fisher is a weighted second moment of “extended gradients” gᵢ,c,c′, the gradient of example i of class c as if its label were c′, with weights pᵢ,c,c′/nC (Eq. 6.2). Per layer gˡᵢ,c,c′ = hˡ⁻¹ᵢ,c ⊗ δˡᵢ,c,c′ (features ⊗ backpropagated errors, Eq. 7.4), so Gˡ = Σ w (hhᵀ) ⊗ (δδᵀ), and the class means of gradients are approximately hc ⊗ δc and the cross-class means hc ⊗ δc,c′ (Section 7.12).
  • K-FAC versus CFAC. K-FAC’s block Hˡ ⊗ Δˡ, a product of an average over all examples with an average over all examples, “multiplies the feature mean of one class by the error mean of another”. Its spectrum’s outliers and mini-bulks are “drastically misaligned” from the Fisher’s (Figures 11–12). The proposed Class-distinct Factorized Approximate Curvature, Gˡ_CFAC = Ave_c Hˡc ⊗ Δˡc, aligns much better at deeper layers; at shallow layers the improvement depends on the dataset.
  • Theory for the simplest case. For multinomial logistic regression on xᵢ,c = t·e_c + z, z ~ N(0, I), with symmetric predicted probabilities, the expected Fisher is (s/C)·blkdiag(U¹…U^C, 0) + I ⊗ Ū (Lemma 8.1) and its spectrum has exactly C outliers, C(C − 2) mini-bulk eigenvalues and a bulk (Theorem 8.1). Raising the signal-to-noise s separates all three, so their ratios predict misclassification.

Standing in the record

Filed on 2026-10-03 at the owner’s request: the owner’s “2020 JMLR”, and with Papyan 2019 the “Fisher-spectrum block structure, the object being thresholded”. This is the paper that says exactly what that structure is, across objects and layers, and connects it to the Kronecker approximations.

It extends Papyan 2019 (LIT-619). It takes the three-level hierarchy of that paper, “slightly modified” to probability-weighted means, as its starting point, and credits that paper with attributing the C outliers to class means; what it adds is the mini-bulk and the main bulk, seen only in log-spectra, and the extension from G to every other object. It extends the 2018 preprint (LIT-617) too, whose Lanczos tools (Appendix C here) and Hessian-to-G attribution (Section 5 here) it reuses.

It was measured against K-FAC (LIT-622). Section 7.10 defines K-FAC’s matrix following Martens and Grosse, computes its spectrum per layer on an eight-layer MLP trained on MNIST, Fashion-MNIST and CIFAR-10, and compares it with the true layer Fisher and with CFAC (Figures 11–12). The comparison is spectral only; no optimizer is run with CFAC. The paper cites Ma et al. (2020) on K-FAC being inefficient for large batches, and says it “does not shed light on this phenomenon”.

What the record should take from it:

  • Block structure. The Fisher’s dominant structure in a classifier is a sum over classes of Kronecker products, not one Kronecker product per layer. Shampoo (LIT-621) is not examined.
  • Representation. Feature class means grow in energy and become closer to orthogonal with depth (Figures 7–8); the error class means do the same toward the output and lose it toward the input (Figures 9–10). This is the layer-wise picture whose last layer neural collapse (LIT-618) describes, though this paper does not cite it.
  • Generalization. It argues that sharpness measures should look at the separation of outliers from bulk rather than at the trace or top eigenvalue (Section 9.4), and that the TIC-based predictor of Thomas et al. is dominated by the bulk (Section 9.5).
  • Singular learning. The paper says nothing about singularity; it does show the Fisher’s informative directions numbering about C² against a bulk and a vast null space, which is the empirical shape of a degenerate Fisher that Watanabe’s theory (LIT-354) is about.

It is ML measurement and optimizer design, so anthology-candidate; the anthology’s curvature holdings (ANTH-LIT-768, ANTH-LIT-746) are where a transfer would land.

Backlinks