The claim
Let f(x; θ) be differentiable in θ ∈ ℝ^p, and let J be the n × p matrix whose rows are ∇_θ f(x_i; θ) at the n inputs. JJᵀ is the empirical NTK Gram matrix, (JJᵀ)_ij = ∇f(x_i) · ∇f(x_j). Model the outputs as y | x ~ N(f(x; θ), β⁻¹).
- The Fisher is βJᵀJ. The model’s Fisher on the n inputs is βJᵀJ, which is the Gauss–Newton matrix of the squared loss weighted by β, up to the 1/n convention.
- Shared spectrum. JᵀJ and JJᵀ have the same non-zero eigenvalues with the same multiplicities.
- Eigenvectors. If u is a unit eigenvector of JJᵀ with eigenvalue σ² > 0, then Jᵀu/σ is a unit eigenvector of JᵀJ with the same eigenvalue.
So the non-zero Fisher spectrum is β times the empirical NTK spectrum.
Proof
(1) The log-likelihood of one observation is −(β/2)(y − f(x; θ))² + const, so the score is β(y − f)∇f. Its expected outer product under the model is β²E[(y − f)²]∇f∇fᵀ = β∇f∇fᵀ, since E[(y − f)²] = β⁻¹. Summed over the n inputs this is βJᵀJ. The Gauss–Newton matrix of the loss (β/2)Σ_i (y_i − f(x_i))² is also βJᵀJ. With k outputs, J stacks nk rows and the argument is unchanged.
(2) Let J = UΣVᵀ be a singular value decomposition, with rank r. Then JᵀJ = VΣᵀΣVᵀ and JJᵀ = UΣΣᵀUᵀ, whose non-zero eigenvalues are both σ₁², …, σ_r², with equal multiplicities.
(3) JᵀJ(Jᵀu) = Jᵀ(JJᵀu) = σ²Jᵀu, and ‖Jᵀu‖² = uᵀJJᵀu = σ², so Jᵀu/σ has unit norm.
What it does not say
- That “Fisher” is the empirical Fisher. It is the model’s Fisher under the Gaussian likelihood. The empirical Fisher, built from observed residuals, differs from it away from the optimum.
- What the spectrum looks like. That the lazy-regime spectrum of a ReLU network on spherical data inherits the NTK’s harmonic-degree blocks, and where a threshold on it is well posed, is THEORY-084’s, Proposed.
- Anything for cross-entropy. There the Gauss–Newton matrix is JᵀΛJ, and the identity holds for Λ^(1/2)J instead.