Topics — What a work is about. Nucleation holds whatever reading the owner files here, whether or not an Anthology of the SOTA topic could also hold it; instruction for ML practice belongs to the anthology (ADR-048). The words are seeded from the clusters the reading-time triage of 2026-09-25 found, and grow by decision.
Information geometry — the geometry of statistical models — the Fisher information as a metric, natural gradient, curvature and its Kronecker and block structure, and the spectra of Fisher and Hessian matrices in learned models (ADR-026).
14 of 2004 LIT documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| LIT-354 v4 | Algebraic Geometry and Statistical Learning Theory | Watanabe (2009), DOI-10.1017/CBO9780511800474. Per publisher description: develops statistical learning theory for singular models (neural networks, mixtures, hidden Markov models) in which the Fisher information is degenerate, using resolution of singularities to show that the real log canonical threshold governs Bayesian free energy and generalization error. | Unavailable — seeded 2026-09-29 from metadata and published descriptions, not read; no lawful full text could be reached, and it stays Unavailable until a copy is supplied. Its core results are now read in a lawful substitute by the same author, Watanabe’s A Widely Applicable Bayesian Information Criterion (LIT-616; arXiv 1208.6338, JMLR 2013). That paper restates the book’s standard representation theorem, the real log canonical threshold and the free-energy asymptotics F = nL_n(w0) + λ log n − (m − 1) log log n, citing the book for each. Read it for the results; the book holds their proofs, the generalization-error theory and WAIC. |
| LIT-607 | Natural Gradient Works Efficiently in Learning | Amari (1998), Neural Computation 10(2):251–276. When the parameter space carries a Riemannian metric G, the steepest descent direction of a loss is the natural gradient G⁻¹∇L, not the ordinary gradient. For a statistical model the metric is the Fisher information, the only invariant one. Online natural gradient learning with step 1/t is Fisher efficient: it attains the Cramér–Rao bound asymptotically, as well as the best batch estimator. The paper computes the metric and its inverse for a simple perceptron, for the group of nonsingular matrices (blind source separation, giving the update ∇L·WᵀW) and for linear systems, proposes an adaptive learning rate with 1/t convergence, and conjectures that the natural gradient escapes the plateaus of backpropagation. | Active — read in full 2026-10-03 (NOTE-472); worth reading as the source of the natural gradient: steepest descent in a Riemannian parameter space is G⁻¹∇L (Theorem 1), the Fisher information is the metric on a statistical model, and online learning with the natural gradient and a 1/t rate is Fisher efficient (Theorem 2). Its efficiency proof needs the Fisher matrix at the optimum to be invertible, which is exactly what singular learning theory says fails for neural networks. |
| LIT-613 | Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra | Papyan (2020), JMLR 21(252). Cross-entropy indexes every deepnet quantity by example, class and would-be class, a two-way MANOVA layout. In the spectra of features, backpropagated errors, gradients, the Fisher, the Hessian and weights, knockouts attribute the top outlier to the global mean, C−1 outliers to between-class second moment, a mini-bulk of C(C−1) to between-cross-class covariance, and the main bulk to within-cross-class covariance. Feature and error class means separate from the bulk and become orthogonal with depth. Gradient class means are approximately feature class means ⊗ error class means, layer by layer; K-FAC’s single Kronecker product misaligns the Fisher’s outliers and the class-wise CFAC aligns them. For multinomial logistic regression on a Gaussian class model the expected Fisher spectrum is derived exactly. | Active — read 2026-10-03 (NOTE-474); worth reading as the general account of class/cross-class structure in deepnet spectra: features, backpropagated errors, gradients, Fisher, Hessian and weights all show one outlier from the global mean, C−1 from class means, a mini-bulk of about C(C−1) from cross-class means, and a main bulk from within-class variation. Per layer, gradient class means are approximately Kronecker products of feature class means and error class means, which is why K-FAC, pairing every class with every class, misaligns the Fisher’s outliers, and CFAC, pairing each class with itself, aligns them. Proved only for multinomial logistic regression on a Gaussian model. |
| LIT-616 | A Widely Applicable Bayesian Information Criterion | Watanabe (2012), arXiv:1208.6338; JMLR 14:867–897 (2013). A model is singular when the parameter-to-distribution map is not one-to-one or the Fisher information is not positive definite: neural networks, mixtures, reduced-rank regression, HMMs. Its Bayes free energy is nL_n(w0) + λ log n − (m − 1) log log n + O_p(1), with λ the real log canonical threshold (RLCT), not d/2. WBIC, the posterior average of nL_n(w) at inverse temperature 1/log n, has the same leading terms even for singular, unrealizable models, equals BIC for regular ones, and needs no knowledge of the true distribution. Two temperatures give an estimate of λ. In reduced-rank regression WBIC picked the true rank in 100 of 100 trials and recovered theoretical RLCTs closely. | Active — read in full 2026-10-03 (NOTE-473); worth reading as the record’s lawful, short statement of the core of singular learning theory, standing in for Watanabe’s unread book (LIT-354). It states the Bayes free energy asymptotics F = nL_n(w0) + λ log n − (m − 1) log log n + O_p(1), with λ the real log canonical threshold and m its multiplicity (Theorem 2), and defines λ through resolution of singularities (Theorem 1, Eq. 18). Its own result is WBIC, the posterior mean of nL_n(w) at inverse temperature 1/log n, which matches F to the λ log n term even for singular and unrealizable models (Theorem 4), equals BIC for regular ones (Theorem 5), and estimates λ from two temperatures (Corollary 3). The prior results are stated with citations to the book, not re-proved. |
| LIT-617 | The Full Spectrum of Deepnet Hessians at Scale: Dynamics with SGD Training and Sample Size | Papyan (2018), arXiv. Stochastic Lanczos without reorthogonalization (O(Mnp) time, O(p) memory) plus subspace-iteration deflation measures the full Hessian spectrum of VGG11 and ResNet18 on MNIST, Fashion-MNIST and CIFAR-10/100, over 16,000 trained models. It confirms a bulk with about C outliers, attributes the outliers to the Gauss–Newton term G and most of the bulk to the residual H, finds H’s spectrum symmetric with a power-law tail rather than Marchenko–Pastur, and finds H never negligible. Version 2 decomposes G into class means (outliers), cross-class variation and within-class variation (two bulks) and tracks them across epochs and sample sizes. | Active — read 2026-10-03 (NOTE-478); worth reading as the measurement that the bulk-and-C-outliers spectrum of the Hessian persists at full scale (VGG11, 28M parameters), that the outliers belong to the Gauss–Newton/Fisher term G and the bulk mostly to the residual H, and, in its second version, that G splits into class means (outliers) and two bulks (cross-class and within-class variation). Its tools are stochastic Lanczos spectral density estimation without reorthogonalization, plus subspace-iteration deflation. |
| LIT-618 | Prevalence of Neural Collapse during the terminal phase of deep learning training | Papyan, Han & Donoho (2020), PNAS 117(40). Training past zero error (the terminal phase, TPT) drives four linked phenomena in the last layer: within-class variability collapses (Tr(Σ_WΣ_B†)/C → 0), centred class means become equinorm and equiangular with cosines −1/(C−1), a simplex ETF; classifier rows become proportional to the means; and the decision becomes nearest class mean. Measured on MNIST, Fashion-MNIST, SVHN, CIFAR-10/100, STL10 and ImageNet with VGG, ResNet and DenseNet, 480 models. Test accuracy and DeepFool robustness keep improving during TPT. Theorems: NC1–NC2 imply NC3–NC4 for the Webb–Lowe and Soudry max-margin classifiers, and the simplex ETF uniquely maximizes the small-noise error exponent. | Active — read in full 2026-10-03 (NOTE-476); worth reading as the source of neural collapse: in the terminal phase of training, last-layer features collapse to their class means (NC1), the centred means form a simplex equiangular tight frame (NC2), the classifier aligns with them (NC3) and the decision becomes nearest class centre (NC4). Measured on 7 datasets × 3 architectures; NC1–NC2 imply NC3–NC4 for the Webb–Lowe and max-margin classifiers, and the simplex ETF is the unique optimal codebook in a vanishing-noise Gaussian channel. It explains Papyan’s Hessian outliers as class means outgrowing collapsing within-class variation. |
| LIT-619 | Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians | Papyan (2019), ICML. For a C-class network the Gauss–Newton term of the Hessian is G = (1/n)ΔΔᵀ, a second moment (not covariance) of logit derivatives δᵢ,c,c′ indexed by example, true class and logit. Grouping them into C² group means δc,c′ and C clusters with centres δc gives G = G₀ + G₁ + G₂ + G₃ (Eq. 28). The top-C outliers of G are those of G₁ = (C−1)Ave_c δcδcᵀ, because clusters are far apart and tight. Early in training the group means cluster by logit c′, later by true class c. Holds across MNIST, Fashion-MNIST, CIFAR-10 and ImageNet with VGG, ResNet and DenseNet; the outliers exceed their approximations slightly, as random matrix theory would predict. | Active — read in full 2026-10-03 (NOTE-479); worth reading as the paper that explains the C outliers of the deepnet Hessian: the Gauss–Newton/Fisher term G is a second moment, not a covariance, of logit derivatives indexed by (example, class, logit), and the C outliers are the eigenvalues of G₁ = (C−1)·Ave_c δ_c δ_cᵀ, the C class-cluster centres, computable from a C×C Gram matrix without high-dimensional eigenanalysis. The match is empirical, over 18,000 trained models. |
| LIT-621 | Shampoo: Preconditioned Stochastic Tensor Optimization | Gupta, Koren & Singer (2018), ICML. For a parameter tensor of order k, Shampoo keeps one preconditioner per dimension, each the gradient contracted with itself over all other dimensions, and multiplies the gradient along dimension i by (Hⁱ)^{-1/2k}. In the matrix case this is W ← W − η L^{-1/4} G R^{-1/4}, equivalent to preconditioning the flattened gradient by (L ⊗ R)^{1/4}, which upper-bounds the full-matrix AdaGrad preconditioner up to the gradient rank (Lemma 8). Regret is O(√T) in online convex optimization (Theorems 7, 10). On CIFAR-10/100 and LM1B it converges faster than SGD with momentum, Adam and AdaGrad at similar step time. | Active — read 2026-10-03 (NOTE-480); worth reading as the source of per-dimension Kronecker preconditioning: for a matrix parameter, left and right statistics L = Σ GGᵀ and R = Σ GᵀG, update W ← W − η L^{-1/4} G R^{-1/4}, i.e. a step preconditioned by (L ⊗ R)^{1/4}, which by Lemma 8 dominates the full AdaGrad matrix. Its statistic is the accumulated gradient outer product, not the Fisher; its guarantee is an O(√T) regret in online convex optimization; its deep-learning evidence is four training curves. |
| LIT-622 | Optimizing Neural Networks with Kronecker-factored Approximate Curvature | Martens & Grosse (2015), ICML. K-FAC approximates natural gradient descent in neural networks. Each Fisher block between layers i and j, E[āᵢ₋₁āⱼ₋₁ᵀ ⊗ gᵢgⱼᵀ], is replaced by E[āᵢ₋₁āⱼ₋₁ᵀ] ⊗ E[gᵢgⱼᵀ] (activations times backpropagated derivatives), whose error is a sum of third- and fourth-order cumulants. The inverse is then taken as block-diagonal or block-tridiagonal across layers. With factored Tikhonov damping, re-scaling by the exact Fisher, an adaptive momentum and a growing mini-batch, it needs orders of magnitude fewer iterations than tuned SGD with Nesterov momentum on three deep autoencoder benchmarks, and it is invariant to affine reparameterizations of each layer’s inputs and pre-activations. | Active — read in full 2026-10-03 (NOTE-486); worth reading as the source of the Kronecker-factored Fisher: each layer’s Fisher block E[āāᵀ ⊗ ggᵀ] is approximated by E[āāᵀ] ⊗ E[ggᵀ], so it inverts as two small matrices, and the inverse is further taken as block-diagonal or block-tridiagonal across layers. The factorization is justified by small higher-order cumulants, checked on one small tanh network; the speedups are on three deep autoencoders. |
| LIT-651 | Linear Mode Connectivity in Multitask and Continual Learning | Mirzadeh, Farajtabar, Gorur, Pascanu & Ghasemzadeh (2020). When a multitask solution and a continual solution start from the same point, the multitask solution lies on a low-loss line to every earlier continual solution, while consecutive continual solutions are separated by rising loss. Euclidean distance and CKA do not detect the difference. The multitask direction lies outside the top Hessian eigenvectors of the earlier task. MC-SGD trains towards a point linearly connected to both the previous solution and the current task’s solution, using a small replay buffer, and leads five baselines on three 20-task benchmarks. | Active — read in full 2026-10-03 (NOTE-520); worth reading as the first study of where continual and multitask solutions lie relative to each other. Started from the same point (the previous task’s solution), the multitask solution is linearly connected with low loss to every earlier continual solution, while consecutive continual solutions are not. The authors explain it by curvature: the multitask path moves in directions the earlier task’s Hessian sees as flat, within a region where a second-order expansion holds. MC-SGD turns this into a regulariser and beats EWC, A-GEM, ER-Reservoir and Stable SGD on 20-task Permuted MNIST, Rotated MNIST and Split CIFAR-100. The connectivity breaks with a different start, input noise, label corruption or removed classes. |
| LIT-662 | Geodesic Mode Connectivity | Tan, Long, Zhao & Laine (2023), ICLR 2023 Tiny Paper. Treats a network as a distribution p(x, ŷ; θ) on a Fisher–Rao manifold and hypothesises that geodesics between SGD solutions are mode-connecting paths. A chain of 25 models between two ResNet-20s (4× width, LayerNorm), initialised on the straight line after Git Re-Basin weight matching, is optimised to minimise summed Jensen–Shannon divergence between neighbours. The curve is longer in Euclidean terms but shorter in distribution space, and keeps train and test loss low where linear interpolation shows a large barrier. | Active — read in full 2026-10-03 (NOTE-521); worth reading as the short proposal that mode-connecting paths are geodesics of the Fisher–Rao metric on the space of the networks’ output distributions. It approximates a geodesic between two trained ResNet-20s (4× width, LayerNorm), after Git Re-Basin weight matching, by optimising 25 models on a chain to minimise the summed Jensen–Shannon divergence between neighbours, using unlabelled training images only, and the resulting curve has low train and test loss where the straight line does not. One architecture, one pair, one figure; the hypothesis that all geodesics between SGD solutions are mode-connecting is stated, not tested. |
| LIT-936 v3 | The information geometry of large language models is shared, learned, and controllable | Picozzi (2026), arXiv preprint. Each model is given the matrix of Fisher–Rao distances between its next-token laws over a fixed context battery. That matrix is a function of behaviour alone, so it is invariant to output-preserving reparameterisation, which activation geometry is not. Across ten models of three architecture kinds it agrees at a mean rank correlation of 0.88, against 0.62 for activations. This is proved to follow from each model’s small excess risk to a common law, and it is measured to come from model disagreement lying nearly orthogonal to the relational structure. The pulled-back metric gives minimum-disturbance steering whose advantage over Euclidean steering it predicts (median measured/predicted 0.967). | Active — read 2026-10-14 (NOTE-755) in full; worth reading as a measured, coordinate-free relational signature. The matrix of Fisher–Rao distances between a model’s next-token laws over a context battery is fixed by behaviour and invariant to any output-preserving reparameterisation, while activation geometry has a GL(d) gauge. It agrees across ten transformer, state-space and recurrent models at a mean rank correlation of 0.88, against 0.62 for mid-layer activations, and a probe on distances to 40 anchors transfers between models. Read it knowing three things. The signature is identified because it is a function of the observed laws, the in-probe case QUESTION-027 calls automatic. Its agreement is proved only as a consequence of each model fitting the same law, on a shared outcome space, and only about 9% of shared eigenvector structure goes beyond what the shared laws imply. It is a single-author v1 preprint whose code is on request. Most of its length is natural-gradient steering, editing and fine-tuning, which is machine-learning practice. |
| LIT-1254 v2 | Information loss under coarse graining: A geometric approach | Raju, Machta & Sethna (2018). Follows the Fisher information metric of the bare parameters as the data are coarse-grained along an RG flow. Exactly: coarse-graining an exponential family loses the expected conditional covariance of its energy operators, and under an exact RG step with linear beta functions that loss is analytic, so the singular part of the metric survives; for diffusion the metric scales as n^(1−μ) by Hermite order, keeping only the drift. By a scaling argument: irrelevant directions contract as b^−(d−2λ). On its own scalings only directions with eigenvalue above d/2 keep their information, not every relevant one. | Active — read 2026-10-20 (NOTE-972) in full, from arXiv v2; worth reading for the identity that coarse-graining an exponential family loses exactly the expected conditional covariance of its energy operators, for its consequence under an exact RG step (the loss is the Hessian of an analytic constant, so the singular part of the Fisher metric survives; THEORY-742), and for the exact diffusion case, where the metric scales as n^(1−μ) by Hermite order. Read it knowing that “relevant directions are exactly maintained” is true only of the singular part, that on its own scalings the preserved directions are those with RG eigenvalue above d/2 rather than above 0, that the contraction of irrelevant directions (Eq. 21) is a scaling argument backed by the numerics of Machta et al. (2013) and not derived, that the drift is exactly preserved only at the Gaussian fixed point, and that no learning dynamics appear. |
| LIT-1299 v2 | Parameter Space Compression Underlies Emergent Theories and Predictive Models | Machta, Chachra, Transtrum & Sethna (Science 2013). Two physics models are not sloppy at their microscopic scale and become so when observed coarsely. In a 1-D hopping model the Fisher spectrum of the density at time t spreads as t^(3/2−k), ordered particle creation, drift, diffusion constant, then skew and higher moments. In Monte Carlo on a decimated 64×64 critical Ising model with 13 couplings and a field, the field and temperature eigenvalues stay put while the rest fall in bands drawn at b^(2y−d) per step. The rate is argued by scaling, not derived, and the Ising numbers have no error bars. The paper reads the resemblance to sloppy models from biology as a common cause of effective theories, which it does not show. | Active — read 2026-10-20 (NOTE-1018) in full, from arXiv:1303.6738v1 with its nine-page supplement; worth reading for the original numerics behind the claim that coarse-graining contracts irrelevant directions of the Fisher metric (Fig. 3: a 64×64 square-lattice Ising model at its critical point, 13 couplings and a field, up to six checkerboard decimations, in which the field and temperature eigenvalues stay put and the rest fall in bands drawn at b^(2y−d) per step for y = 0, −2, −4), for the supplement’s scaling argument for that rate (Eqs. 28–41), which already states that only directions with 2y ≥ d are carried by the singular part, and for the hopping model of diffusion, whose density at time t carries Fisher eigenvalues scaling as t^(3/2−k) (THEORY-774). Read it knowing that the Ising rates are read off a figure with no error bars, no fitted exponents and no second lattice size, and that the eigenvalues are assigned to y by drawn bands, with y = 0 not an operator of the 2D Ising model (THEORY-777, Proposed); that the derivation of the rate is the same scaling argument Raju, Machta and Sethna (2018) later repeat, and assumes the coarse metric is the Hessian of a coarse free energy in the bare couplings; that the Fig. 3 caption’s rate 2^(−d−y) contradicts the supplement and the figure, whose bands fall as 2^(y−d/2); that the diffusion model is not a coarse-graining of one data set, so its leading eigenvalues grow with t rather than staying put; that the main text’s “‘relevant’ directions in the RG are exactly those whose FIM eigenvalues do not contract” is contradicted by its own supplement for relevant directions with 0 < 2y < d; and that the closing claim that emergent theories are “due to” the sloppy hierarchy is not shown, since the paper traces both to the same separation of scales. |