Shun-ichi Amari (1998), Neural Computation 10(2):251–276 — DOI-10.1162/089976698300017746
Key takeaways
- The natural gradient is steepest descent under the model’s own metric. If lengths in parameter space are |dw|² = Σ gᵢⱼ dwᵢ dwⱼ, the direction that most decreases L for a fixed small step is −G⁻¹∇L (Theorem 1, Eq. 2.3). For a statistical model the metric is the Fisher information (Eq. 3.5), which Amari, citing Chentsov, calls “the only invariant metric to be given to the statistical model”.
- It is Fisher efficient online. With step size 1/t and a realizable teacher, the online natural-gradient estimator’s error covariance is G⁻¹/t + O(1/t²), the Cramér–Rao bound, so learning from each example once is asymptotically as good as the best batch estimator (Theorem 2, Eqs. 4.4–4.6). The proof uses G(w*) and its inverse; for an unrealizable teacher the expected Hessian K(w) replaces G, which is locally Newton’s method (Eq. 4.9).
- The metric is computed, not assumed, for three spaces. For a single perceptron with Gaussian inputs, G = w²c₁I + (c₂ − c₁)wwᵀ and its inverse is explicit (Theorems 3–4). For the Lie group of nonsingular matrices, the invariant metric gives the natural gradient ∇L·WᵀW (Theorem 6), which turns the blind-separation update into (I − φ(y)yᵀ)W. For multilayer perceptrons the Fisher matrix has one block per hidden unit (Eq. 6.13), but Amari notes that inverting it “is not easy except for simple cases”.
Standing in the record
Filed on 2026-10-03 at the owner’s request, as the root of the Fisher
geometry half of nucleation’s line on model comparison and Fisher geometry
(ADR-026). It is the definition the other works in this batch approximate:
Martens and Grosse’s K-FAC (LIT-622) is an efficiently invertible
approximation to F⁻¹∇h “defined as” Amari’s natural gradient, and Papyan’s
spectra (LIT-613) study the very Fisher matrix whose inverse the
natural gradient needs. It carries an instruction for machine-learning
practice, how to precondition a gradient, so it takes anthology-candidate;
the anthology holds later Kronecker and curvature work (ANTH-LIT-768,
ANTH-LIT-746) but not this paper.
The record holds Rao’s 1945 paper (LIT-349) as an unread seed; Amari cites it, with his own 1985 book, as the source of the Fisher metric (Eq. 3.5).
Where it meets singular learning theory. Theorem 2 needs G(w*) to be invertible and the estimator to converge to a point where the Cramér–Rao bound applies. Watanabe’s book (LIT-354), unread here, starts from the fact that in neural networks, mixtures and hidden Markov models the Fisher information is degenerate at the true parameter. The anthology’s reading of Murfet et al. (ANTH-LIT-541) dates that knowledge to “Amari et al. (2003) and Watanabe (2007)”. So the efficiency result in this paper is stated for the regular case, and the models it is about are, by Amari’s own later work, not regular. The shorter Watanabe statement filed alongside this batch (WBIC, LIT-616) is where the record should look for what replaces the Cramér–Rao bound when G is singular. My own observation, not the paper’s: the multilayer Fisher blocks it writes down already show where that happens, since the wᵢ block carries the factor vᵢ² and so vanishes when an output weight is zero.