The claim
Let classes c = 1, …, C carry weights π_c ≥ 0 with Σ_c π_c = 1, and let H_c (m × m) and Δ_c (p × p) be any matrices, one pair per class. Write H̄ = Σ_c π_c H_c and Δ̄ = Σ_c π_c Δ_c. Then
Σ_c π_c H_c ⊗ Δ_c − H̄ ⊗ Δ̄ = Σ_c π_c (H_c − H̄) ⊗ (Δ_c − Δ̄).
In Papyan’s notation for a classifier’s layer (LIT-613, Definitions 7.1–7.2), H_c and Δ_c are the class-c second moments of the layer’s input features and of its backpropagated errors, π_c is the class frequency, the left-hand product is the class-wise approximation CFAC and H̄ ⊗ Δ̄ is K-FAC’s factorisation of the layer Fisher (LIT-622, Eq. 1–3). K-FAC’s factors are averages over all examples, so they are the π-weighted averages of the class moments, and the identity reads CFAC − K-FAC = the weighted between-class covariance of the two factors in the Kronecker sense. With balanced classes π_c = 1/C and the weights become Papyan’s “Ave_c”.
Proof
The Kronecker product is bilinear. Expanding the right-hand side,
Σ_c π_c (H_c − H̄) ⊗ (Δ_c − Δ̄) = Σ_c π_c H_c ⊗ Δ_c − (Σ_c π_c H_c) ⊗ Δ̄ − H̄ ⊗ (Σ_c π_c Δ_c) + (Σ_c π_c) H̄ ⊗ Δ̄ = Σ_c π_c H_c ⊗ Δ_c − H̄ ⊗ Δ̄ − H̄ ⊗ Δ̄ + H̄ ⊗ Δ̄ = Σ_c π_c H_c ⊗ Δ_c − H̄ ⊗ Δ̄,
using Σ_c π_c = 1 in the last term. ∎
Corollary. If H_c = H̄ for every class (or Δ_c = Δ̄ for every class), every term on the right vanishes and CFAC = K-FAC. Equivalently, H̄ ⊗ Δ̄ = Σ_{c,c′} π_c π_{c′} H_c ⊗ Δ_{c′} contains all C² pairings of one class’s features with another’s errors, CFAC contains only the C matched pairings, and the identity says the cross pairings differ from the matched ones by the covariance term alone.
A check. For C = 4 classes with Dirichlet weights and random positive semidefinite 3 × 3 and 2 × 2 factors (numpy, seed 0), the two sides agree to 6 × 10⁻¹⁵, and the difference CFAC − K-FAC has eigenvalues of both signs (−5.26 and 5.51 at the extremes). So the covariance term is not in general positive semidefinite: CFAC is not K-FAC plus a correction that only adds curvature.
What this does not say
- Not that K-FAC misplaces a classifier’s Fisher outliers. That is THEORY-085’s mechanism, which rests on Papyan’s spectra judged by eye on one MLP and stays Proposed. The identity holds for any matrices and says nothing about which approximation is closer to the true layer Fisher.
- Not a necessary condition. Either factor being class-constant is sufficient for CFAC = K-FAC. The covariance term can also vanish in other ways, for example when the class deviations cancel in the weighted sum; the claim does not characterise when it does.
- Not about the true Fisher. Both sides are approximations. The true layer block is a weighted sum of Kronecker products over examples, not classes (LIT-613, Eq. 7.5), and its distance from either side is not addressed here.