Topics — What a work is about. Nucleation holds whatever reading the owner files here, whether or not an Anthology of the SOTA topic could also hold it; instruction for ML practice belongs to the anthology (ADR-048). The words are seeded from the clusters the reading-time triage of 2026-09-25 found, and grow by decision.
Model comparison — choosing between models by their evidence — the marginal likelihood and Bayes factors, the Savage–Dickey density ratio for nested models, Occam factors and the evidence framework, Bayesian model reduction and structure learning, and information criteria and free energies for singular models (ADR-026).
12 of 2004 LIT documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| LIT-225 v2 | Minimum Description Length Induction, Bayesianism, and Kolmogorov Complexity | Vitányi & Li (1999), arXiv:cs/9901014. Ideal MDL selects the H minimising K(H) + K(D|H), Bayes’s rule with the universal distribution in place of prior and likelihood. By the Fundamental Inequality, when the data are random for H and H is random for the prior, Bayes code lengths and Kolmogorov code lengths agree to within K(Pr(·|H)) + K(P). So ideal MDL and MAP agree to that factor among such hypotheses, and can diverge outside them. For finite-set models they coincide with Kolmogorov’s minimal sufficient statistic. For random sequences, shortest-program prediction converges to the true conditional. | Active — read 2026-10-14 (NOTE-801) in full; worth reading as the stated condition under which minimum description length and Bayes coincide: by the Fundamental Inequality (Theorem 3), Bayes code lengths and shortest-program lengths agree to within K(Pr(·|H)) + K(P) when the data are Martin-Löf random for the hypothesis and the hypothesis is random for the prior, so ideal MDL and MAP agree to a factor 2^{−α} among such hypotheses (Theorem 4), and for finite-set models both reduce to Kolmogorov’s minimal sufficient statistic. Read it knowing that α can be as large as the hypothesis’s own complexity, that admissibility cannot be checked, that “with high probability” (Corollary 1) and the overfitting claim are unproved, and that the arXiv text is the submitted version. |
| LIT-238 v2 | Minimum Description Length Revisited | Grünwald & Roos (2019), ARXIV-1908.08484. A review of MDL since 2007, written without coding theory. Each model is replaced by a single universal distribution with small worst-case regret against its best fit. NML is the minimax choice, with complexity (k/2) log(n/2π) + log ∫ v√|I| + o(1). Luckiness functions generalise priors and penalties. The switch distribution nearly resolves the AIC–BIC dilemma, RIPr tests survive optional continuation, general penalised estimators inherit MDL convergence, and MDL and Bayes can be inconsistent under misspecification. Results are cited, not proved. | Active — read 2026-10-14 (NOTE-760) in full; worth reading as the current map of MDL. A model is replaced by one universal distribution with small worst-case regret, NML is the minimax one with complexity (k/2) log(n/2π) + log ∫ v√|I| + o(1), estimation is penalised ML with a luckiness function, and the decade’s results are collected: the switch distribution, RIPr tests, convergence of general MDL estimators, and inconsistency under misspecification. Read it knowing that it is a review whose results are cited rather than proved, that it sets aside Kolmogorov-complexity MDL, and that its “unification” of AIC with BIC and of cross-validation with Bayes is, in its own words, partial. |
| LIT-354 v4 | Algebraic Geometry and Statistical Learning Theory | Watanabe (2009), DOI-10.1017/CBO9780511800474. Per publisher description: develops statistical learning theory for singular models (neural networks, mixtures, hidden Markov models) in which the Fisher information is degenerate, using resolution of singularities to show that the real log canonical threshold governs Bayesian free energy and generalization error. | Unavailable — seeded 2026-09-29 from metadata and published descriptions, not read; no lawful full text could be reached, and it stays Unavailable until a copy is supplied. Its core results are now read in a lawful substitute by the same author, Watanabe’s A Widely Applicable Bayesian Information Criterion (LIT-616; arXiv 1208.6338, JMLR 2013). That paper restates the book’s standard representation theorem, the real log canonical threshold and the free-energy asymptotics F = nL_n(w0) + λ log n − (m − 1) log log n, citing the book for each. Read it for the results; the book holds their proofs, the generalization-error theory and WAIC. |
| LIT-609 | Bayesian hypothesis testing for psychologists: A tutorial on the Savage–Dickey method | Wagenmakers, Lodewyckx, Kuriyal & Grasman (2010), Cognitive Psychology 60(3):158–189. For nested models, H0: φ = φ0 inside H1, the Bayes factor BF01 equals the ratio of the posterior to the prior density of φ at φ0 under H1 (Eq. 12), if the nuisance parameters’ prior under H1 tends to their prior under H0 as φ → φ0. So a Bayes factor can be read off MCMC samples from the larger model alone, with a nonparametric density estimate at one point. Order restrictions are handled by truncating prior and posterior. Three worked psychology examples, including hierarchical one- and two-sample t-tests. Limitations stated in §8: density estimates in the tails, nested models only, and the Borel–Kolmogorov paradox. | Active — read in full 2026-10-03 (NOTE-484); worth reading as the record’s working statement of the Savage–Dickey density ratio: for nested models, the Bayes factor for a point null φ = φ0 is the height of the posterior for φ at φ0 divided by the height of the prior there (Eq. 12), provided the nuisance parameters’ prior under the alternative, as φ → φ0, equals their prior under the null. Its derivation is Appendix A. Read it for the condition and for the limitations of §8 (density estimation in the tails, nested models only, the Borel–Kolmogorov paradox), not for the psychology examples, which are illustrations. |
| LIT-610 v2 | The Weighted Likelihood Ratio, Linear Hypotheses on Normal Location Parameters | Dickey (1971), Annals of Mathematical Statistics 42(1):204–223. The Bayes factor (“weighted likelihood ratio”) for a sharp hypothesis η(θ) = η_H equals the alternative’s posterior density of η at η_H over its prior density there (3.9), if the nuisance prior under the null is the alternative’s prior conditioned at η_H (3.8). Dickey calls this Savage’s density ratio and credits the proof to Dickey & Lientz. With conjugate priors he gives closed-form Bayes factors for a normal mean, for a linear hypothesis in the linear normal model as a replacement for the F test, and for the Behrens–Fisher problems. | Active — read in full on 2026-10-03 (NOTE-508) from an owner-supplied JSTOR scan, via OCR checked against the page images; worth reading as the original general statement of the Savage–Dickey ratio: for a sharp hypothesis η(θ) = η_H with positive prior mass, the Bayes factor is the posterior-to-prior density ratio of η at η_H under the alternative (3.9), exactly, if the null’s nuisance prior is the alternative’s prior conditioned at η_H (3.8), stated for any likelihood continuous in θ before the normal-theory applications. |
| LIT-614 | Bayesian model reduction | Friston, Parr & Zeidman (2018; v2 2019), arXiv:1805.07092. A review of Bayesian model reduction: given a fitted full model’s prior P, approximate posterior Q and free energy F, the free energy of a model with any other prior P̃ is ln E_Q[P̃/P] + F, and its posterior follows by reweighting Q (Eqs. 9–10). Closed forms are given for Gaussian, Dirichlet, beta, gamma, categorical and multinomial priors. Three worked simulations (sparse regression, cluster pruning in a Gaussian mixture, network discovery in a linear dynamical system) and a review of uses: structure learning, hierarchical (PEB) inversion as a cascade of reductions, and “sleep-like” pruning in active-inference agents. | Active — read in full 2026-10-03 (NOTE-477); worth reading as the reference statement of Bayesian model reduction (BMR): the reduced free energy F[P̃:P] ≈ ln E_Q[P̃/P] + F[P] (Eq. 9), which the authors call a generalisation of the Savage–Dickey ratio “to any new prior”, with closed forms for Gaussian, Dirichlet, beta, gamma, categorical and multinomial priors (Eqs. 11–12, Table 1). The Dirichlet form is what lets active-inference agents prune their own likelihood mappings. Its examples are simulations, and its review of applications is of the authors’ own lab. |
| LIT-615 | An Active Inference Approach to Modeling Structure Learning: Concept Learning as an Example Case | Smith, Schwartenbeck, Parr & Friston (2020), Front. Comput. Neurosci. 14:41. A partially observed Markov decision process with eight animal “slots”, some starting with flat likelihoods, learns new concepts without feedback: a slot is engaged only when its flat mapping explains a feature combination better than any learned concept. Bayesian model reduction on the learned prior over states then resets slots the data did not need. In 100 runs per condition the true number of concepts won in 45–80% of runs for 4–7 concepts and rarely for 2–3, and in every run when the likelihood was given. The agent also generalises to an unseen animal in one shot, and the paper simulates neural and dopaminergic responses. | Active — read in full 2026-10-03 (NOTE-485); worth reading as the record’s worked example of structure learning in an active-inference agent: “model expansion” by spare hidden-state slots whose flat likelihoods win inference only for a truly novel stimulus, and “model reduction” by applying Bayesian model reduction to the agent’s learned prior over states, so that unused slots are reset. It is a proof of concept with eight animals and six features. Reduction recovered the true number of concepts in 45–80% of 100 runs for 4–7 concepts, failed for 2–3, and succeeded every time only when the likelihood mapping was given rather than learned. |
| LIT-616 | A Widely Applicable Bayesian Information Criterion | Watanabe (2012), arXiv:1208.6338; JMLR 14:867–897 (2013). A model is singular when the parameter-to-distribution map is not one-to-one or the Fisher information is not positive definite: neural networks, mixtures, reduced-rank regression, HMMs. Its Bayes free energy is nL_n(w0) + λ log n − (m − 1) log log n + O_p(1), with λ the real log canonical threshold (RLCT), not d/2. WBIC, the posterior average of nL_n(w) at inverse temperature 1/log n, has the same leading terms even for singular, unrealizable models, equals BIC for regular ones, and needs no knowledge of the true distribution. Two temperatures give an estimate of λ. In reduced-rank regression WBIC picked the true rank in 100 of 100 trials and recovered theoretical RLCTs closely. | Active — read in full 2026-10-03 (NOTE-473); worth reading as the record’s lawful, short statement of the core of singular learning theory, standing in for Watanabe’s unread book (LIT-354). It states the Bayes free energy asymptotics F = nL_n(w0) + λ log n − (m − 1) log log n + O_p(1), with λ the real log canonical threshold and m its multiplicity (Theorem 2), and defines λ through resolution of singularities (Theorem 1, Eq. 18). Its own result is WBIC, the posterior mean of nL_n(w) at inverse temperature 1/log n, which matches F to the λ log n term even for singular and unrealizable models (Theorem 4), equals BIC for regular ones (Theorem 5), and estimates λ from two temperatures (Corollary 3). The prior results are stated with citations to the book, not re-proved. |
| LIT-620 | Post hoc Bayesian model selection | Friston & Penny (2011), NeuroImage 56(4):2089–2099. If every model of interest is a full model with a different prior and the same likelihood, the evidence of any reduced model is p(y|m_F) times the full posterior expectation of p(θ|m_i)/p(θ|m_F) (Eq. 3). So the reduced models never have to be fitted. A point-mass reduced prior gives the Savage–Dickey ratio (Eq. 6). Under Gaussian priors and posteriors the reduced free energy and posterior are closed-form in the full model’s means and precisions (Eq. 9). In simulations an exhaustive search over 4,096 linear models finds the four true regressors, and 64 network models find the true three-connection graph. | Active — read in full 2026-10-03 (NOTE-487); worth reading as the source of Bayesian model reduction: if a set of models differ from a full model only in their priors, the evidence of each is the full model’s evidence times the posterior expectation of the prior ratio (Eq. 3), so one inversion scores them all. Under Gaussian (Laplace) posteriors the reduced free energy is closed-form (Eq. 9). The Savage–Dickey ratio is the point-mass special case (Eq. 6), and ARD and model selection are the same evidence maximisation over prior hyperparameters. The demonstrations are simulations, and whether the reduced free energy of a nonlinear model is a good proxy for the free energy of the reduced model is left open by the authors. |
| LIT-623 v2 | Bayesian Interpolation | MacKay (1992), Neural Computation 4(3):415–447. Two levels of inference: fit each model, then rank models by their evidence P(D|H). Under a Gaussian approximation the evidence is the best-fit likelihood times an Occam factor, the ratio of posterior to prior accessible parameter volume, so complex or finely tuned models penalise themselves (Eqs. 5–6). For linear interpolation with quadratic regularisers the evidence sets the regularising constant and the noise level through γ, the number of well-determined parameters (2αE_W = γ, 2βE_D = N − γ). It ranks polynomial, radial-basis, spline, Hermite and neural-network models on two data sets. In expectation, no fixed model’s log evidence beats the true model’s. | Active — read in full 2026-10-03 (NOTE-471); worth reading as the source of the evidence framework and the Occam factor in the form the field uses: evidence ≈ best-fit likelihood × Occam factor, the Occam factor being the ratio of posterior to prior accessible parameter volume, (2π)^{k/2} det^{-1/2}A · P(w_MP|H) under a Gaussian approximation (Eqs. 5–6). It also gives the “number of good parameter measurements” γ = Σ λ_a/(λ_a + α), which sets the regularising constant (2αE_W = γ) and the noise level (2βE_D = N − γ) at the evidence maximum. The framework is Gull and Skilling’s, as MacKay says, and every demonstration is one-dimensional interpolation. Flagged anthology-candidate for its regularisation and early-stopping arguments. |
| LIT-911 | Agents and Devices: A Relative Definition of Agency | Orseau, McGill & Legg (2018). A system is described either as a device (an input-output mapping) or as an agent (the function it optimizes). Bayes’ rule over mixtures of each gives the observer’s posterior probability that the system is an agent. In a gridworld, running in circles and wall-following come out devices, and heading for a goal, even suboptimally or with rare goal switches, comes out an agent. Agency here is explicitly observer-relative. | Active — read in full 2026-10-11 (NOTE-705) from the arXiv PDF (v1). Worth reading as the cleanest formal version of Dennett’s intentional stance: whether a system is an agent or a device is a Bayesian model comparison between a mixture of goal-optimizing descriptions (inverse RL) and a mixture of input-output mappings, made by an observer with a prior. It presupposes an identified system, and its results depend on the prior chosen. |
| LIT-1882 v2 | Correspondence between thermodynamics and inference | LaMont & Wiggins (2019). Treats the Bayesian evidence as a partition function with sample size N as inverse temperature and derives the thermodynamic quantities. The internal energy is the leave-one-out predictive loss, whose excess K/2N is equipartition. The heat capacity becomes a learning capacity, K/2 in a regular model and the learning coefficient in a singular one, lower where the prior or discreteness fixes parameters (“freeze-out”), proposed as the definition of sloppiness. A prior chosen to make the Gibbs entropy zero is Jeffreys-like, depends on N, reproduces AIC and avoids the Lindley–Bartlett paradox in one simulation. | Active — read 2026-10-31 (NOTE-1481; appendices B, C1, C3, C4 skimmed); worth reading as the fullest working of the evidence-as-partition-function dictionary with sample size as inverse temperature: generalization error as equipartition, a “learning capacity” equal to K/2 in regular and to the RLCT in singular models (THEORY-1253), and a sample-size-dependent prior that reproduces AIC. The dictionary is formal; it has no physical currents or dissipation. |