nucleation

LIT-623: Bayesian Interpolation

3 min read

Properties6
StatusActive — read in full 2026-10-03 (NOTE-471); worth reading as the source of the evidence framework and the Occam factor in the form the field uses: evidence ≈ best-fit likelihood × Occam factor, the Occam factor being the ratio of posterior to prior accessible parameter volume, (2π)^{k/2} det^{-1/2}A · P(w_MP|H) under a Gaussian approximation (Eqs. 5–6). It also gives the "number of good parameter measurements" γ = Σ λ_a/(λ_a + α), which sets the regularising constant (2αE_W = γ) and the noise level (2βE_D = N − γ) at the evidence maximum. The framework is Gull and Skilling's, as MacKay says, and every demonstration is one-dimensional interpolation. Flagged anthology-candidate for its regularisation and early-stopping arguments.
Version2
Tagsmodel-comparison, probabilistic-modeling, learning-theory, anthology-candidate
SupportsCLAIM-384 — For a network linearised at its initial parameters, with an isotropic Gaussian prior of precision α on the parameter displacement and Gaussian noise of precision β, the log-determinant part of the evidence is −½ Σ log(1 + βλ/α) over the non-zero eigenvalues λ of the empirical neural tangent kernel's Gram matrix on the training inputs, so at most n directions are priced however many parameters the network has
Cited as “paper” byNOTE-471 — Bayesian Interpolation
Cited as “source” byTHEORY-080 — In a linearised network with a Gaussian prior on its parameters, the Occam factor is a sum over the NTK's eigenvalues, so on spherical data each harmonic pays ½ log(1 + βnµ_k/α) and the kernel's eigenvalue decay schedules the complexity penalty across frequencies, THEORY-087 — The Bayesian complexity penalty grows as (d/2) log n in a regular model, where the curvature at the optimum sets the Occam factor, and as λ log n in a singular one, where λ is the real log canonical threshold; with the prior positive on the optimal set, λ falls below d/2 when that set is infinite or has a point of degenerate curvature, and equals d/2 when it is finitely many points of non-degenerate curvature, THEORY-718 — In a linear-in-parameters model with Gaussian noise and a Gaussian prior, the evidence is exactly the best-fit likelihood times an Occam factor set by the posterior curvature, and when the design grows linearly with n that factor costs (k/2) log n

Graph View

David J. C. MacKay (1992), Neural Computation 4(3):415–447 — DOI-10.1162/neco.1992.4.3.415

Key takeaways

  • The evidence is the transportable quantity. Model comparison is a second level of inference, distinct from fitting. Models are ranked by P(D|H), “the Bayesian’s transportable quantity for comparing alternative models”, whether parametric or not. The prior over models is usually overwhelmed by it.
  • The Occam factor. P(D|H) ≈ P(D|w_MP, H) · P(w_MP|H) σ_{w|D}. In k dimensions with posterior Hessian A, the Occam factor is P(w_MP|H)(2π)^{k/2} det^{-1/2}A (Eq. 6). It is “the factor by which H’s hypothesis space collapses when the data arrive”. Its log is the information gained about the parameters. It penalises both many parameters and parameters that must be finely tuned.
  • γ, the effective number of parameters. With eigenvalues λ_a of the data Hessian βB, γ = k − α Trace A⁻¹ = Σ λ_a/(λ_a + α) (Eq. 23). Each term measures how much the data rather than the prior determine one direction. At the evidence maximum, χ²_W = 2αE_W = γ and χ²_D = 2βE_D = N − γ (Eqs. 22, 24): the noise estimate charges one unit of misfit per well-determined parameter.
  • Bayes cannot systematically reject the truth. The expected log-evidence difference between the true model and any fixed alternative, under the truth, is a KL divergence and so non-negative (Gibbs’ inequality). This also gives a test for numerical evidence code.

Standing in the record

Filed on 2026-10-03 at the owner’s request, in the model-comparison batch. It may also belong in the anthology: its argument that evidence should set regularisation constants, and its claim that early stopping is a symptom of a model ill-matched to the data, are about how to train models. The anthology’s model-stability (regularisation) or analysis-and-evaluation (theory) topics could hold it, so it carries the anthology-candidate flag last (ADR-005). The anthology does not hold it.

It is here for nucleation’s question, model comparison as one line with Fisher geometry. The Occam factor is a ratio of volumes measured by the determinant of the posterior Hessian, which for a likelihood-dominated posterior is the observed Fisher information. So the evidence framework already ties model comparison to the curvature of the model at its best fit. It also shows where that tie breaks. The Gaussian approximation assumes a single nondegenerate peak. MacKay notes that relabelling degeneracies need a multiplicity factor, and that multiple maxima “complicate the analysis”. Singular models, in which the Hessian is degenerate at the optimum, are the case Watanabe’s theory (LIT-354, LIT-616) handles and this paper does not.

Where it sits in the batch:

  • Wagenmakers et al. (LIT-609) state the same automatic Occam’s razor (their §2.2.1), citing MacKay’s 2003 textbook.
  • Friston & Penny (LIT-620) optimise prior hyperparameters by evidence, as this paper optimises α. Their Table 1 lists ARD, MacKay’s later method, as one case.
  • WBIC (LIT-616) replaces the Occam penalty (k/2) log n of BIC, which the Gaussian approximation underwrites, with λ log n for singular models.
  • MDL. MacKay calls minimum description length “closely related” and says he “can see no advantage in MDL”. The record holds MDL in Grünwald & Roos (LIT-238), read 2026-10-14 in NOTE-760. Its eqs. (15)–(16) give the same (k/2) log n penalty for NML as for Bayes, so MacKay’s Occam penalty is the description-length penalty too.

Backlinks