Source
MacKay (1992), LIT-623, §2 (Eqs. 5–6), the linear interpolants of §3 (Eqs. 7–11, 16, 19–20) and the Occam hill of §6, read in NOTE-471, whose claim C1 is “strong under the Gaussian approximation (exact for the linear models used)”.
What was actually shown
Take y(x) = Σ_h w_h φ_h(x) with k weights, n observations with Gaussian noise of precision β, and a quadratic regularizer, which is the prior w ~ N(0, α⁻¹I). Write Φ for the n × k design matrix. The negative log posterior is quadratic in w, with Hessian A = αI + βΦᵀΦ. So the posterior is exactly Gaussian, and the Gaussian integral gives exactly
log P(D | α, β) = log P(D | w_MP) + log [P(w_MP) (2π)^{k/2} det^{-1/2} A].
The second term is MacKay’s Occam factor, “the ratio of the posterior accessible volume of H’s parameter space to the prior accessible volume” (Eq. 6). For this model class it is not an approximation (Eqs. 16, 19–20).
The large-n form, by the record’s algebra. Suppose ΦᵀΦ/n → Σ, positive definite. Then log det A = log det(n(βΦᵀΦ/n + (α/n)I)) = k log n + log det(βΣ) + o(1). And log P(w_MP) = (k/2) log(α/2π) − (α/2)‖w_MP‖², which is O(1) when w_MP stays bounded, as it does in probability when the data come from the model with a fixed weight vector. So the log Occam factor is −(k/2) log n + O(1). The curvature of the posterior sets the penalty, and its determinant grows as n^k. MacKay sees the scaling in his polynomial fits, where the right side of the Occam hill falls as “k log N” (§6).
The derivation could have gone otherwise in two places. A design whose ΦᵀΦ grew more slowly than n in some direction would give that direction a smaller cost. A posterior that was not Gaussian would make the formula an approximation. Neither happens under these conditions.
What this does not say
- Anything for nonlinear models. There the Occam factor formula is a Gaussian approximation, which MacKay justifies by the central limit theorem (footnote 2), and for degenerate maxima it needs correction. The general regular case, F = nL_n(ŵ) + (d/2) log n + O_p(1), is cited in the sources and not proved there. It and the singular case are THEORY-087.
- That (k/2) log n is the whole story at finite n. The finite-n factor charges ½ log(1 + λ_a/α) for each eigenvalue λ_a of βΦᵀΦ. Directions the data barely determine cost almost nothing, and γ = Σ λ_a/(λ_a + α) counts the ones that do (Eq. 23).
- That α and β are integrated out. The identity is for fixed α and β. MacKay sets them at the evidence maximum, which assumes a dominant peak (Eq. 18).
- Anything about which model is true. The evidence ranks models relative to the priors given.