ADR-026: model-comparison and information-geometry join the topics

Context

The owner asked for a set of readings to be checked against both records and filed here. The anthology was to be kept as well wherever it already held any of them. The set is:

• Savage–Dickey (Dickey 1971; Wagenmakers et al. 2010);
• Bayesian model reduction (Friston & Penny 2011; Friston, Parr, Zeidman et al.);
• structure learning in active inference;
• MacKay’s evidence framework;
• Watanabe’s singular learning theory;
• Amari’s natural gradient;
• K-FAC and Shampoo;
• Papyan’s spectra and neural collapse;
• spectral bias (Bietti & Bach, Cao et al., Basri et al.);
• Gemma Scope.

The only one already held was Watanabe’s book, here, as a Deferred seed. Neither record had the others.

Two of the subjects had no word. Choosing among models by their evidence fell under probabilistic-modeling, which is about building models, not comparing them. Fisher geometry fell under learning-theory, which is about why learning generalizes.

Decision

Add two words:

• model-comparison: choosing between models by their evidence. It covers the marginal likelihood and Bayes factors, the Savage–Dickey ratio, Occam factors, Bayesian model reduction and structure learning, and information criteria and free energies for singular models.
• information-geometry: the geometry of statistical models. It covers the Fisher metric, natural gradient, curvature and its Kronecker and block structure, and the spectra of Fisher and Hessian matrices in learned models.

The readings are held here because the owner asked for them here. Several are machine-learning works that an anthology topic could hold: optimizers, deep-learning spectra, neural collapse, spectral bias and SAEs. Those carry anthology-candidate (ADR-005). Under ADR-013 the anthology may also hold any of them with its own reading, and the two entries then name each other.

Alternatives considered

• probabilistic-modeling and learning-theory alone. Each would then hold two subjects, and the line the owner is assembling, from evidence through singular learning theory to the Fisher spectra, would be invisible as one.
• File the machine-learning works in the anthology only. The owner asked for them here. ADR-013 permits holding them in both, and the anthology-candidate flag records that the anthology may want them too.