nucleation

LIT-616: A Widely Applicable Bayesian Information Criterion

4 min read

Properties5
StatusActive — read in full 2026-10-03 (NOTE-473); worth reading as the record's lawful, short statement of the core of singular learning theory, standing in for Watanabe's unread book (LIT-354). It states the Bayes free energy asymptotics F = nL_n(w0) + λ log n − (m − 1) log log n + O_p(1), with λ the real log canonical threshold and m its multiplicity (Theorem 2), and defines λ through resolution of singularities (Theorem 1, Eq. 18). Its own result is WBIC, the posterior mean of nL_n(w) at inverse temperature 1/log n, which matches F to the λ log n term even for singular and unrealizable models (Theorem 4), equals BIC for regular ones (Theorem 5), and estimates λ from two temperatures (Corollary 3). The prior results are stated with citations to the book, not re-proved.
Tagsmodel-comparison, learning-theory, mathematics, information-geometry, anthology-candidate
ExtendsLIT-354 — Algebraic Geometry and Statistical Learning Theory
Cited as “paper” byNOTE-473 — A Widely Applicable Bayesian Information Criterion
Cited as “source” byTHEORY-079 — Bayesian model reduction under the Laplace approximation prices the reductions of a singular parent model with a Gaussian Occam factor, so its free-energy differences can be wrong by a term that grows with log n, THEORY-082 — Steepest descent is defined only relative to a metric, and Amari's proof that natural-gradient learning is Fisher efficient needs the Fisher information at the optimum to be invertible, the condition whose failure WBIC calls singular; in a multilayer perceptron it fails wherever an output weight is zero, THEORY-087 — The Bayesian complexity penalty grows as (d/2) log n in a regular model, where the curvature at the optimum sets the Occam factor, and as λ log n in a singular one, where λ is the real log canonical threshold; with the prior positive on the optimal set, λ falls below d/2 when that set is infinite or has a point of degenerate curvature, and equals d/2 when it is finitely many points of non-degenerate curvature

Graph View

Sumio Watanabe (2012), arXiv preprint; Journal of Machine Learning Research 14(27):867–897 (2013) — ARXIV-1208.6338

Key takeaways

  • Singular is the common case. “If a statistical model contains hierarchical layers, hidden variables, or grammatical rules, then it is singular.” In a singular model the likelihood cannot be approximated by any normal distribution, and AIC, BIC and MDL do not apply.
  • The free energy of a singular model (Theorem 2, from the book and earlier papers): F = −log ∫ Π p(X_i|w) φ(w) dw = nL_n(w0) + λ log n − (m − 1) log log n + R_n. Here λ, the RLCT, is a positive rational number, a birational invariant, equal to d/2 for regular models (Lemma 3) and smaller than d/2 when the prior is positive at the singularities (§7.1).
  • The RLCT, defined. Resolution of singularities (Hironaka) gives a map w = g(u) in which K(g(u)) = u^{2k} and the prior measure is b(u)|u^h| du, locally (Theorem 1). Then λ = min over charts of min_j (h_j + 1)/(2k_j), and m counts the coordinates attaining it (Eqs. 18–19).
  • WBIC. WBIC = E_w^β[nL_n(w)] with β = 1/log n (Eq. 4). Theorem 4: WBIC = nL_n(w0) + λ log n + U_n √(λ log n / 2) + O_p(1), so its first two terms match F. Theorem 5: for regular models WBIC = BIC + o_p(1). Corollary 3: the difference of the posterior means at two inverse temperatures, divided by the difference of 1/β, converges to λ. One MCMC run estimates the free energy without knowing λ, which depends on the unknown truth.

Standing in the record

Filed on 2026-10-03 at the owner’s request, as the lawful substitute for Watanabe’s Algebraic Geometry and Statistical Learning Theory (LIT-354), which is in copyright and stays Deferred. The record’s rule for books asks for a shorter work by the same author that states the same results. This one does, for the results the record needs: the definitions of regular and singular models, the standard representation theorem (Theorem 1, the book’s Theorem 6.1), the RLCT and its multiplicity, and the free-energy asymptotics (Theorem 2). It cites the book for each. It extends the book in the plain sense: its new results (Theorems 3–5, Corollaries 1–3) are built on those. What it does not carry is the book’s proofs of Theorems 1 and 2, its treatment of the generalisation error, and WAIC. For WAIC this paper gives only the statement E[WAIC] = E[G] + O(1/n²) (Eq. 39), with a citation.

Anthology. The anthology holds the singular-learning-theory follow-ons in deep learning: ANTH-LIT-541 (Deep Learning is Singular, and That’s Good) and ANTH-LIT-542 (the local learning coefficient). ANTH-LIT-541 already cites this paper by name, as the criterion that replaces BIC for singular models, without holding it. This paper is the statistical source those build on, and an anthology topic (analysis-and-evaluation) could hold it, so it carries the anthology-candidate flag last, as LIT-354 does. Nucleation reads it for its own question: model comparison and Fisher geometry as one line. WBIC is model comparison. The definition of “singular” that it rests on is a statement about the Fisher information matrix.

Where it meets the rest of the batch:

  • MacKay (LIT-623). MacKay’s Occam factor, (2π)^{k/2} det^{-1/2}A under a Gaussian approximation, gives BIC’s (d/2) log n. This paper says exactly when that fails: a degenerate Hessian at the optimum. It also says what replaces it, λ log n with λ ≤ d/2 under a positive prior. The remark in §7.1 that singular models therefore “can employ a larger model” with smaller generalisation error, at the price of weaker model-selection consistency, is the singular counterpart of MacKay’s Occam hill.
  • Natural gradient (LIT-607). Amari’s Fisher efficiency needs the Fisher matrix at the optimum to be invertible. This paper’s “regular” is that same condition, and its “singular” is the case where it fails.
  • Bayesian model reduction (LIT-614, LIT-620). BMR scores reduced models with variational free energies under a Laplace (Gaussian) posterior. That is the approximation that, by this paper’s account, is wrong for the hierarchical and latent-variable models BMR is used on. The papers do not cite each other, and neither tests the other’s claim.

Backlinks