Source
- MacKay (1992), LIT-623, read in NOTE-471: §2 (the evidence and the Occam factor, Eqs. 4–6), §5 (γ, Eqs. 22–24) and the Legendre demonstration (the “Occam hill”).
- Watanabe (2012; JMLR 2013), LIT-616, read in NOTE-473: Eq. 14 (regular and singular), Eqs. 18–19 (the RLCT), Lemma 3, Theorems 2, 4 and 5, Table 3 and the remark in §7.1.
- Grünwald & Roos (2019), LIT-238, read in NOTE-760: eqs. (15)–(16), a second source for the regular half only (added 2026-10-14).
The claim, in two halves
Regular models: the curvature at the optimum sets the penalty. MacKay shows that the evidence is the best-fit likelihood times an Occam factor. That factor is “the ratio of the posterior accessible volume of H’s parameter space to the prior accessible volume”, the factor by which the hypothesis space collapses when the data arrive. With a k-dimensional Gaussian posterior it is P(w_MP|H)(2π)^{k/2} det^{-1/2}A, where A is the Hessian of the negative log posterior (LIT-623, Eq. 6). For n independent observations whose likelihood dominates the prior, A is about n times the Fisher information at the optimum, so det A ∝ n^d and the log Occam factor is −(d/2) log n plus a term that does not grow with n. MacKay observes this scaling directly in his polynomial models: the right side of the Occam hill falls as “k log N” (LIT-623, §6). This is BIC’s penalty, and it is set by curvature. The determinant of the Fisher measures how fast the posterior shrinks in each direction. Grünwald & Roos give the same (k/2) log n penalty, with the Fisher-volume constant, for NML as well as Bayes, and show that NML and Bayes under the generalised Jeffreys prior become asymptotically indistinguishable (LIT-238, eqs. 15–16). So the regular penalty is the description-length penalty too. Their WBIC form (eq. 31) is cited, not derived, and adds nothing to the singular half.
Watanabe reaches the same place from the other side. The truth is regular for a model if the optimal parameter set is a single point and the Hessian J(w0) of the average log loss is positive definite there. J(w0) is the Fisher information when the truth is realizable (LIT-616, Eq. 14). For a regular truth with a prior positive at w0, the RLCT is d/2 and its multiplicity is 1 (Lemma 3, proved by one blow-up). The proof of Theorem 5 performs MacKay’s Gaussian integral, with its (nβ)^{d/2} det J^{1/2} factor, and concludes WBIC = BIC + o_p(1). In the regular case, then, the λ log n of singular learning theory is MacKay’s Occam factor at large n.
Singular models: the RLCT sets the penalty. When the optimal set is not a point, or J(w0) is degenerate, no normal distribution approximates the posterior. The Bayes free energy is then F = nL_n(w0) + λ log n − (m − 1) log log n + O_p(1) (LIT-616, Theorem 2). Here λ is defined by resolving the singularities of the Kullback–Leibler function: λ = min over charts of min_j (h_j + 1)/(2k_j) (Eq. 18). In Watanabe’s words in §7.1: “If a prior distribution is positive at the optimal set of parameters, then RLCTs are smaller than d/2 in singular models”. With Jeffreys’ prior, which vanishes at singularities, the inequality reverses and λ ≥ d/2.
A worked case. Reduced-rank regression with M = N = 6 and true rank 3 (LIT-616, §6, Table 3). Models of rank 1 and 2 cannot realize the truth, and rank 3 realizes it without redundancy. Their λ is half their dimension, H(M + N − H)/2: 5.5, 10 and 13.5. Models of rank 4, 5 and 6 realize it redundantly. Their RLCTs are 15, 16 and 17, below their half-dimensions of 16, 17.5 and 18 by 1, 1.5 and 1. WBIC’s two-temperature estimates came out at 14.69, 15.74 and 16.53.
What this does not say
- It does not prove Theorem 2 in general. The theorem is cited from Watanabe’s book (LIT-354) and earlier papers, and the record has read neither proof. Table 3 agrees with the theory, but its estimates come from WBIC, which is built on the same theory. The note of 2026-10-29 checks the theorem’s prediction exactly in two singular models, and re-derives the inequality λ < d/2 and where it fails.
- It does not give λ a volume reading. MacKay’s Occam factor is literally a volume ratio. The statement that λ is the exponent at which the prior volume of {w : K(w) < ε} shrinks with ε is in Watanabe’s book and not in the paper read. So “the penalty is a volume ratio in both cases” is the natural reading, but the record holds only its regular half.
- (d/2) log n is the large-n form, not the Occam factor itself. At finite n MacKay’s Gaussian factor charges ½ log(1 + λ_a/α) for each eigenvalue λ_a of the data Hessian against prior precision α. Directions the data do not determine cost almost nothing, and γ = Σ λ_a/(λ_a + α) counts the ones that do (Eq. 23). Below that scale the penalty is governed by the prior and not by curvature or by λ. THEORY-080 works this out for one structured prior.
- It does not say a smaller penalty is a better model. Watanabe’s remark draws the trade-off himself. A λ below d/2 lets a larger model be used with smaller generalisation error, but it also means weaker consistency in finding the true model.
- It is not about the relabelling degeneracies MacKay mentions. Several equivalent maxima multiply the Gaussian evidence by their number (LIT-623, §2). That is a constant, not a change of the log n slope. The singular case changes the slope.
Connections
- THEORY-082 reads the same boundary from optimisation. Amari’s efficiency theorem needs the Fisher at the optimum to be invertible, which is the regular half here.
- THEORY-079 presupposes this account, for a claim about Bayesian model reduction. It argues that BMR’s Laplace scoring uses the regular half’s penalty in the singular models it is applied to.
- THEORY-080 extends the regular half to a linearised network whose prior is set by its tangent kernel.
Note of 2026-10-19: how much of the regular half is held
Only the linear-Gaussian case is held exactly. There MacKay’s Occam factor is exact, and the (d/2) log n cost follows by algebra. That case is THEORY-718, Active. For nonlinear regular models MacKay’s formula is a Gaussian approximation. Grünwald & Roos cite the (k/2) log n expansion (eq. 16) to Rissanen and to Suzuki & Yamanishi. Watanabe’s Lemma 3 gives λ = d/2 for a regular truth, but the free-energy expansion that turns λ into a penalty is his Theorem 2, which is cited. The body’s “The regular half is derived” holds in this sense: derived exactly for one model class, and by approximation otherwise.
Note of 2026-10-29: the re-derivation and the check
Made by the tenth status review under ADR-047’s ruling on re-derivations.
The regular half, by Laplace’s method. Write the free energy as F = −log Z_n with Z_n = ∫ φ(w) exp(−n L_n(w)) dw. The conditions:
- w0 is the unique minimiser of L in the interior of a compact W, and it is well separated: inf over |w − w0| > δ of L(w) − L(w0) is positive for every δ > 0.
- L_n converges to L uniformly, so the minimiser ŵ of L_n converges to w0; the Hessian of L_n converges uniformly near w0, and J(w0) is positive definite.
- φ is continuous and φ(w0) > 0.
The steps:
- Outside a ball |w − ŵ| < δ, n L_n(w) − n L_n(ŵ) exceeds nε for some ε > 0, so that region contributes a factor e^(−nε) relative to the rest, which vanishes against any power of n.
- Inside it, n L_n(w) = n L_n(ŵ) + (n/2)(w − ŵ)ᵀH_n(w − ŵ) + n·o(|w − ŵ|²), with H_n → J(w0).
- Substituting u = √n (w − ŵ), dw = n^(−d/2) du, the integral becomes φ(ŵ) e^(−n L_n(ŵ)) n^(−d/2) ∫ exp(−uᵀH_n u/2) du (1 + o_p(1)), and the Gaussian integral is (2π)^(d/2) det H_n
- So F = n L_n(ŵ) + (d/2) log n − (d/2) log 2π + ½ log det J(w0) − log φ(w0) + o_p(1). Since n L_n(ŵ) − n L_n(w0) = O_p(1), the same holds with w0 in place of ŵ, up to O_p(1).
That is the regular expansion the promote_when asked for, with its conditions.
λ < d/2, re-derived from Eq. 18. Let ζ(z) = ∫ K(w)^z φ(w) dw. In each chart of Theorem 1, ζ is a sum of integrals ∫ u^(2kz) b(u) |u^h| du, and each coordinate integral ∫ u_j^(2k_j z + h_j) du_j converges exactly when 2k_j z + h_j > −1. So ζ converges for real z > −λ, with λ as Eq. 18 defines it, and has its largest pole at −λ (b > 0, so nothing cancels).
- A volume bound bounds λ. Let V(t) = ∫ over {K < t} of φ. If V(t) ≥ c t^α for small t, then for s > 0, by the layer-cake formula, ∫ over {K < t0} of K^(−s) φ ≥ ∫ from t0^(−s) to ∞ of V(y^(−1/s)) dy ≥ c ∫ y^(−α/s) dy, which diverges when s ≥ α. So ζ diverges at z = −α, and λ ≤ α.
- An optimal set that is infinite. It is a real analytic set, so it has a stratum of dimension r ≥ 1. At a smooth point p of that stratum, K ≥ 0 vanishes on the stratum, so K and its gradient vanish there, and K ≤ C·dist(w, stratum)² near p. The set {K < ε} contains a tube of radius √(ε/C) around a piece of the stratum, and φ is bounded below near p, so V(ε) ≥ c ε^((d−r)/2). Hence λ ≤ (d − r)/2 < d/2.
- An isolated point with a degenerate Hessian. Let v be a null direction of J(p) and write w = p + t v + x⊥. Then K(w) = ½ x⊥ᵀ J x⊥ + O(|w − p|³) ≤ C (|x⊥|² + |t|³) for small w − p. So {K < ε} contains {|x⊥|² < ε/2C, |t|³ < ε/2C}, of volume c ε^((d−1)/2 + 1/3). Hence λ ≤ d/2 − 1/6 < d/2.
- Finitely many points, each with J positive definite. Lemma 3 gives λ = d/2 at each, and λ is the minimum over them, so λ = d/2. Eq. 14 calls this case singular, since the optimal set is not a single point, so Watanabe’s §7.1 remark read literally fails here. The title is narrowed to say so.
This uses a lower bound on the volume of {K < ε}, which is not the volume reading of λ that the bullet above says the record does not hold.
The singular half, checked directly. The promote_when named this check: an exact free energy in a singular model of known λ, growing with log n at slope λ and not d/2. Models y = c(w) x + N(0, 1) with Σ x_i² = n and a uniform prior on [−1, 1]^d have a free energy that depends on the data only through ĉ ~ N(c0, 1/n). F − n L_n(w0) was computed by quadrature over the prior, averaged over ĉ by Gauss–Hermite, for n from 10² to 10¹² (10⁸ for the one-dimensional models).
| Model | d/2 | Theory | Measured slope of F against log n |
|---|---|---|---|
| c = a (control) | ½ | ½ | 0.5000 |
| c = a₁x₁ + a₂x₂ (control) | 1 | 1 | 1.0000 |
| c = sin(3a), truth sin 0.9 (nonlinear regular) | ½ | ½ | 0.5000 |
| c = ab, truth 0 (CASE-152: λ = ½, m = 2) | 1 | ½ | 0.463 at n = 10¹²; slope + 1/log n → 0.5008 |
| c = a³, truth 0 (λ = 1/6) | ½ | 1/6 | 0.1667 |
| c = a², truth 0.25 (two non-degenerate optima) | ½ | ½ | 0.5000 |
For CASE-152’s model, a fit of λ log n − β log log n + const gives λ = 0.497 and β = 0.92, against λ = ½ and m − 1 = 1. The slope there is λ, not d/2, and the Gaussian (d/2) log n overcharges it by about ½ log n + log log n. The scripts were written by the reviewer, re-run by the reviewing session, and are not in the record.