information-geometry.
13 of 1538 NOTE documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| NOTE-472 | Natural Gradient Works Efficiently in Learning | Natural gradient ∇̃L = G⁻¹∇L is steepest descent under metric G (Theorem 1); for a statistical model G is the Fisher information. Online learning w_{t+1} = w_t − (1/t)∇̃l is Fisher efficient, Ṽ_t = G⁻¹/t + O(1/t²) (Theorem 2). Explicit G and G⁻¹ for a perceptron (Theorems 3–4) and the natural gradient ∇L·WᵀW on the matrix group (Theorem 6). The plateau claim for multilayer perceptrons is a conjecture resting on Yang & Amari (1997). | Read |
| NOTE-473 | A Widely Applicable Bayesian Information Criterion | The Bayes free energy of a singular model is nL_n(w0) + λ log n − (m − 1) log log n + O_p(1), with λ the RLCT. The posterior mean of nL_n(w) at inverse temperature 1/log n (WBIC) reproduces the first two terms without knowing λ, equals BIC for regular models, and two temperatures estimate λ. In reduced-rank regression WBIC chose the true rank 100/100 times, and estimated RLCTs within about 0.5 of theory. | Read |
| NOTE-474 | Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra | G = Gclass + Gcross + Gwithin + Gc=c′ (Eq. 6.4); knockouts give C outliers to Gclass, a C² mini-bulk to Gcross, the bulk to Gwithin, in log-spectra. Gˡ = Σ w (hhᵀ) ⊗ (δδᵀ) (Eq. 7.8), so gc ≈ hc ⊗ δc. K-FAC Hˡ ⊗ Δˡ misaligns G’s outliers; CFAC Ave_c Hˡc ⊗ Δˡc aligns them at deep layers. For multinomial logistic regression on a Gaussian model the expected Fisher has C outliers, C(C−2) mini-bulk and a bulk (Theorem 8.1). | Read |
| NOTE-476 v2 | Prevalence of Neural Collapse during the terminal phase of deep learning training | NC1 Σ_W → 0 (Tr(Σ_WΣ_B†)/C falls), NC2 centred means → simplex ETF (cosines → −1/(C−1)), NC3 W ∝ Ṁᵀ, NC4 nearest class mean; 7 datasets × 3 nets, 480 models. Median TPT gains: +0.35 points test accuracy, +0.0252 DeepFool robustness. NC1–2 ⇒ NC3–4 for Webb–Lowe and max-margin (Theorems 2, 4). Optimal small-noise error exponent C/(C−1)·1/4, only at simplex ETFs (Theorem 5). | Read |
| NOTE-478 | The Full Spectrum of Deepnet Hessians at Scale | Hess = H + G (Gauss–Newton). At 28M parameters the Hessian has a bulk and about C outliers; outliers are in G, the bulk mostly in H; H is symmetric and power-law, φ ∝ |λ|⁻²·⁷ (R² 0.99), not Marchenko–Pastur or semicircle. G = A₁ + A₂ + B₁ + B₂ (Eq. 5): C class means give the outliers, cross-class spread and within-pair spread give two bulks. FastLanczos costs O(Mnp) time and O(p) memory. | Read |
| NOTE-479 | Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians | G = (1/n)ΔΔᵀ with δᵢ,c,c′ = column c′ of diag(√p)(I − 1pᵀ)∂f/∂θ. C² group means δc,c′, C cluster centres δc; G = G₀ + G₁ + G₂ + G₃ (Eq. 28). Top-C eigenvalues of G ≈ those of G₁ = (C−1)Ave_c δcδcᵀ, from a C×C Gram matrix, with λc(G) ≥ λc(G₁₊₂) ≥ λc(G₁). 18,000 CIFAR/MNIST models plus ImageNet VGG16/ResNet50. Group means switch from clustering by logit c′ to by class c during training. | Read |
| NOTE-480 | Shampoo: Preconditioned Stochastic Tensor Optimization | Matrix case: L = εI + Σ GGᵀ, R = εI + Σ GᵀG, W ← W − ηL^{-1/4}GR^{-1/4}. Lemma 8: εI + (1/r)Σ ggᵀ ⪯ L^{1/2} ⊗ R^{1/2} for rank-r gradients, so the Kronecker preconditioner dominates full AdaGrad. Regret √(2r) D Tr(L^{1/4}) Tr(R^{1/4}) = O(√T) (Theorem 7); order-k tensors use power −1/2k per dimension (Theorem 10). Steps per second within a few percent of SGD/Adam/AdaGrad (Table 1); on LM1B 3.509 against 4.87–4.92. | Read |
| NOTE-486 | Optimizing Neural Networks with Kronecker-factored Approximate Curvature | Fisher blocks E[āāᵀ ⊗ ggᵀ] ≈ E[āāᵀ] ⊗ E[ggᵀ], with error a sum of third- and fourth-order cumulants; the inverse is approximated as block-diagonal or block-tridiagonal over layers. Factored Tikhonov damping (π chosen by trace norms), re-scaling by the exact Fisher, and a two-parameter momentum solved on the quadratic model. On three deep autoencoders it beats tuned NAG-SGD by orders of magnitude in iterations; block-tridiagonal gains 25–40% per iteration over block-diagonal. | Read |
| NOTE-520 | Linear Mode Connectivity in Multitask and Continual Learning | From a shared start ŵ₁, the multitask solution w₂ is linearly connected to ŵ₁ and ŵ₂, but ŵ₁ and ŵ₂ are not; the ŵ₁ → w₂ direction is nearly orthogonal to task 1’s top 50 Hessian eigenvectors, and ŵ₁ → ŵ₂ lies in their span. Different starts (CIFAR), input noise, label corruption and removed classes break the line. MC-SGD averages the loss along both lines with a one-example-per-class buffer: after 20 tasks 85.3 / 82.3 / 63.3% (Permuted MNIST / Rotated MNIST / Split CIFAR-100), multitask 89.5 / 89.8 / 68.8, Stable SGD 80.1 / 70.8 / 59.9. | Read |
| NOTE-521 | Geodesic Mode Connectivity | Length of a path in distribution space = √8 ∫ √dJSD (Fisher–Rao). Minimise the discrete energy Σ JSD(pᵢ‖pᵢ₊₁) over 25 interior models between two weight-matched ResNet-20 (4×, LayerNorm) CIFAR-10 networks, initialised on the line; the result keeps train and test loss low where the line has a large barrier, using unlabelled training images. One pair, no numbers in the text, hypothesis stated for all geodesics. | Read |
| NOTE-755 | Picozzi — The information geometry of large language models is shared, learned, and controllable | The Fisher–Rao distance matrix of a model’s next-token laws over a context battery is a function of the laws alone, so behaviour fixes it, whereas activation geometry has a GL(d) gauge. Across ten models of three architecture kinds it agrees at a mean rank correlation of 0.88, against 0.62 for mid-layer activations. The agreement is proved to follow from each model’s closeness to the same language law, and only about 9% of shared eigenvector structure goes beyond what the shared laws imply. The pullback metric also gives minimum-disturbance steering, whose cost advantage over Euclidean steering it predicts. | Read |
| NOTE-972 v2 | Raju, Machta and Sethna — Information loss under coarse graining | For a Boltzmann distribution linear in its couplings, the Fisher information that coarse-grained data lose is the expected conditional covariance of the energy operators. Under an exact RG step with linear beta functions it is the Hessian of the analytic constant, so the singular part of the metric is kept. In the diffusion model it is exact: the metric scales as n^(1−μ) by Hermite order. The claims that every relevant direction is kept and that irrelevant ones contract at b^−(d−2λ) rest on a scaling argument. On the paper’s own scalings the dividing line is λ = d/2, not 0. | Read |
| NOTE-1018 | Machta, Chachra, Transtrum and Sethna — Parameter space compression | Two physics models are not sloppy at their microscopic scale and become so when observed coarsely. For the hopping model the Fisher eigenvalues of the density at time t scale as t^(3/2−k), proved here (THEORY-774). For a decimated critical Ising model, Monte Carlo shows the field and temperature eigenvalues staying put while the rest fall in bands drawn at b^(2y−d) per step (THEORY-777, Proposed). That rate, the source of LIT-1254’s Eq. 21, is argued by the same scaling argument, read off a figure without error bars, so THEORY-742’s scaling claim stays scaling-only. The supplement already states the d/2 line. The causal claim about emergent theories is not shown. | Read |