Source
- Papyan (2020, JMLR), LIT-613, read in NOTE-474: Eqs. 7.3–7.8, §§7.10–7.12, Definitions 7.1–7.2, Figures 11–12.
- Martens & Grosse (2015), LIT-622, read in NOTE-486: Eqs. 1–3, Figures 2–3, §13.
What was actually shown
The layer Fisher is a sum of Kronecker products. A layer’s gradient for example i of class c, evaluated as if its label were c′, is h^{l−1}{i,c} ⊗ δ^l{i,c,c′}: the layer’s input features times the backpropagated error (LIT-613, Eq. 7.4). So the layer block of the Fisher is G^l = Σ w·(hhᵀ) ⊗ (δδᵀ), a weighted sum over examples, classes and would-be classes (Eq. 7.5). That is exact. Martens & Grosse start from the same fact, vec(DW_i) = ā_{i−1} ⊗ g_i (LIT-622, Eq. 1).
K-FAC’s assumption. K-FAC replaces E[āāᵀ ⊗ ggᵀ] by E[āāᵀ] ⊗ E[ggᵀ], which treats activities and derivatives as independent. Its error is a sum of third- and fourth-order cumulants (LIT-622, Eq. 3), and the authors concede it “likely won’t become exact under any realistic set of assumptions”. They checked it on one partially trained tanh network on 16 × 16 MNIST, and their optimisation experiments are autoencoders, which have no classes.
Papyan’s measurement. Papyan computes K-FAC’s layer matrix G^l_KFAC = H^l ⊗ Δ^l (Definition 7.1), the product of the feature and error second moments each averaged over all examples. He compares its spectrum with the true G^l and with a class-wise alternative, G^l_CFAC = Ave_c H^l_c ⊗ Δ^l_c (Definition 7.2), on an eight-layer MLP trained on MNIST, Fashion-MNIST and CIFAR-10. In his words: “G_KFAC has its mini-bulks and outliers drastically misaligned from those of G” (§7.10). CFAC’s spectrum aligns “much better” at deeper layers. At CIFAR-10 layers 7–8 it aligns “almost perfectly”. At shallow layers the gain depends on the dataset, and at CIFAR-10 layer 2 there is none (Figures 11–12). His account: “In KFAC … the feature mean of one class would be multiplied by the error mean of another class. In CFAC, on the other hand, the feature mean of a class would only be multiplied by the error mean of that class” (§7.12). From this he infers that the gradient class means, which cause the outliers, are approximately g_c ≈ h_c ⊗ δ_c.
The algebra behind it. This step is the record’s own, filed with its proof as CLAIM-391; it is summarised here. Write H_c and Δ_c for the per-class second moments, and H̄ and Δ̄ for their averages. Then K-FAC is H̄ ⊗ Δ̄ = Ave_{c,c′} H_c ⊗ Δ_{c′}, which has all C² pairings, and CFAC is Ave_c H_c ⊗ Δ_c, which has C. Their difference is
CFAC − K-FAC = Ave_c (H_c − H̄) ⊗ (Δ_c − Δ̄),
the between-class covariance of the two factors in the Kronecker sense. K-FAC equals CFAC when either factor’s per-class second moment does not vary with class, and otherwise they generally differ. CFAC is K-FAC’s independence assumption made conditional on the class. The class label is a shared cause of features and errors, so it makes them dependent, and that dependence is the cumulant K-FAC drops. The trained classifiers of THEORY-078 are the case where the per-class means are large and far apart, so the difference is large.
What this does not say
- It does not say K-FAC is a poor optimizer. No optimizer is run with CFAC, and the comparison is of spectra only (LIT-613, §7.10). K-FAC’s results come from damping and from re-scaling under the exact Fisher as much as from its factorisation (LIT-622, §6, Figure 7). Papyan cites the large-batch inefficiency of K-FAC and says his paper “does not shed light on this phenomenon”.
- The measured part is narrow. One MLP architecture, three datasets, alignment judged by eye. That g_c ≈ h_c ⊗ δ_c is inferred from CFAC’s better alignment and not measured directly (NOTE-474, C3).
- It says nothing about Shampoo. Shampoo’s factors (LIT-621) are accumulated gradient outer products at the training labels. That is a different matrix from the model Fisher, and Shampoo needs no statistical model to justify its Kronecker shape (NOTE-480). Whether class structure misplaces its directions is not established by any held paper.
- It does not contradict Martens & Grosse. Their check was on a network without classes, where this source of dependence is absent, and they expected the factorisation to be inexact anyway.
Connections
- Extends THEORY-078. That account locates the Fisher’s outliers in the class means. This one follows the class means into each layer’s Kronecker factors and says what a class-blind factorisation does to them.
Note of 2026-10-20: the identity is a claim
Promotion audit, batches 047 and 057. The identity CFAC − K-FAC = Σ_c π_c (H_c − H̄) ⊗ (Δ_c − Δ̄), with class weights π_c (K-FAC’s factors are averages over all examples), is stated by neither source and holds for any matrices by bilinearity, so it is filed as the record’s own demonstrated claim, CLAIM-391, with the proof. This account no longer states it in its title. What stays here is Papyan’s reading: the class gradient means are approximately h_c ⊗ δ_c, so K-FAC’s cross-class pairings misplace the outliers. That is inferred from CFAC’s better spectral alignment on one MLP, judged by eye, and the promote_when’s two measurements are what would settle it.