Topics — What a work is about. Nucleation holds whatever reading the owner files here, whether or not an Anthology of the SOTA topic could also hold it; instruction for ML practice belongs to the anthology (ADR-048). The words are seeded from the clusters the reading-time triage of 2026-09-25 found, and grow by decision.
Model comparison — choosing between models by their evidence — the marginal likelihood and Bayes factors, the Savage–Dickey density ratio for nested models, Occam factors and the evidence framework, Bayesian model reduction and structure learning, and information criteria and free energies for singular models (ADR-026).
11 of 1352 THEORY documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| THEORY-079 v3 | Bayesian model reduction under the Laplace approximation prices the reductions of a singular parent model with a Gaussian Occam factor, so its free-energy differences can be wrong by a term that grows with log n | The filer’s synthesis, stated by no source. BMR’s closed form (Friston & Penny, LIT-620, Eq. 9; Friston, Parr & Zeidman, LIT-614, Eq. 11) charges a reduction through log-determinants of Gaussian precisions, which is MacKay’s regular Occam factor. Watanabe, LIT-616, says that models with hierarchical layers or hidden variables are singular, that no normal distribution approximates their posterior, and that their free energy grows as λ log n. BMR is applied to exactly such models: mixtures, hierarchical empirical Bayes and latent states. Neither literature cites the other. The exact identity is untouched (THEORY-081). The claim is about its Laplace form, and a test on reduced-rank regression would settle it. | Proposed |
| THEORY-080 v3 | In a linearised network with a Gaussian prior on its parameters, the Occam factor is a sum over the NTK’s eigenvalues, so on spherical data each harmonic pays ½ log(1 + βnµ_k/α) and the kernel’s eigenvalue decay schedules the complexity penalty across frequencies | The filer’s synthesis, stated by no source. Its exact half is filed as the record’s own claim, CLAIM-384; what stays here is the spherical account. A network linearised at initialisation, with an isotropic Gaussian prior on its parameters and Gaussian noise, is MacKay’s linear-in-parameters model (LIT-623). Its Occam factor is −½ Σ log(1 + βλ_a/α) over the eigenvalues of JᵀJ. These are the empirical NTK’s, which on uniform spherical data form Bietti & Bach’s degree blocks, nµ_k with multiplicity N(d, k) (LIT-608). Degrees whose βnµ_k exceeds α pay about ½ log n per harmonic, and degrees below pay almost nothing. With µ_k ~ k^(−d), the number of paid directions γ grows as n^((d−1)/d), not as the parameter count. Basri et al.’s bound (LIT-612, Eq. 14) is the matching data-fit term. The kernel’s decay exponent is the schedule of the penalty. | Proposed |
| THEORY-081 v2 | The evidence of any model formed from a full model by changing only its prior is the full evidence times the full posterior expectation of the prior ratio, so the Savage–Dickey ratio is its point-mass case and choosing which parameters to keep is evidence maximisation over the prior | Friston & Penny (2011), LIT-620, Eq. 3: if models share a likelihood and differ only in priors nested inside a full prior, then p(y|m_i) = p(y|m_F)·E_{p(θ|y,m_F)}[p_i(θ)/p_F(θ)]. Their Eq. 6 is the point-mass case, the Savage–Dickey ratio that Wagenmakers et al. (2010), LIT-609, derive from the other end, and that Dickey (1971), LIT-610, stated in this form for any continuous likelihood. Friston, Parr & Zeidman, LIT-614, restate it for any new prior and give conjugate closed forms. Active because the identity is three lines of Bayes’ rule that anyone can check. The two Savage–Dickey conditions turn out to be the same condition. The identity is exact. Its Laplace and variational forms are only as good as the full posterior, and with no constraint on the prior family the optimum is maximum likelihood. | Active |
| THEORY-087 v3 | The Bayesian complexity penalty grows as (d/2) log n in a regular model, where the curvature at the optimum sets the Occam factor, and as λ log n in a singular one, where λ is the real log canonical threshold; with the prior positive on the optimal set, λ falls below d/2 when that set is infinite or has a point of degenerate curvature, and equals d/2 when it is finitely many points of non-degenerate curvature | MacKay (1992), LIT-623, writes the evidence as best-fit likelihood times an Occam factor, the ratio of posterior to prior accessible volume, which under a Gaussian approximation is P(w_MP)(2π)^{k/2} det^{-1/2}A. When the likelihood dominates, det A grows as n^d, so the penalty is (d/2) log n. Watanabe’s WBIC paper, LIT-616, recovers that case (Lemma 3, Theorem 5) and states what replaces it when the Fisher information is degenerate at the optimum: λ log n − (m − 1) log log n, with λ the RLCT (Theorem 2). The regular half is re-derived by Laplace’s method with its conditions. The singular half is Watanabe’s cited theorem, checked by exact free energies in two singular models of known λ. The volume reading of λ is in Watanabe’s book, which is unread. | Active |
| THEORY-259 v2 | For a recursive family of likelihoods and the universal prior, the Bayesian two-part code length −log Pr(D|H) − log m(H) equals the ideal-MDL length K(D|H) + K(H) plus the randomness deficiency of D for H, up to a constant fixed by the family; so ideal MDL and MAP selection agree to within a constant among hypotheses for which the data are random, and can part where they are not | Vitányi & Li (1999/2000), LIT-225, read in NOTE-801. With the universal prior m and a recursive family Pr(·|·), −log Pr(D|H) − log m(H) = K(D|H) + K(H) + d(D|H) + O(1), where d(D|H) = log(m(D|H)/Pr(D|H)) is the universal randomness deficiency of D for H; d ≥ −K(Pr(·|·)) − O(1), and d is bounded exactly when D is Martin-Löf random for H. So ideal MDL and MAP agree to within a constant on admissible hypotheses (Theorems 3–5, re-derived by the record) and part where the data are not random for the hypothesis (0ⁿ against the fair coin). The paper’s slack α = K(Pr(·|H)) + K(P) is an upper bound, not the gap. | Active |
| THEORY-564 v2 | A bounded generative model can learn its own structure with no machinery beyond inference and learning: spare hidden-state slots with flat likelihoods get filled when an observation fits none of the learned causes, and Bayesian model reduction on the learned prior over states can return unneeded slots, though in a toy concept-learning task it recovers the true number of causes in only 45–80% of runs with four to seven causes, and never with two or three, unless the likelihood is given | Smith, Schwartenbeck, Parr and Friston (2020), LIT-615, read in NOTE-485. Expansion works in the toy domain over eight runs per condition, and only for novel stimuli. Reduction is the weak half: over 100 runs, the true number of concepts won in 45–80% of runs for four to seven, never for two or three, and in every run when the likelihood was given. Table 1 is the record’s only measurement of how often reduction picks the generating structure when the parent model was itself learned. One initialisation; eight concepts, six binary features. | Proposed |
| THEORY-624 v2 | Orseau, McGill and Legg cast whether an identified system is an agent or a device as Bayesian model comparison between descriptions of the function it optimises and descriptions of its input-output mapping, so that the verdict is relative to the observer’s prior; on seven toy gridworld trajectories goal-seeking comes out agent-like and wall-following device-like, and they argue that under a speed prior everything may look like an agent | Orseau, McGill & Legg (2018), LIT-911, read in NOTE-705. A formal version of Dennett’s choice of stance: the posterior probability that a system is an agent, from mixtures of goal-optimizing (inverse RL) and input-output descriptions. Illustrated on seven gridworld trajectories, with no human study. It presupposes an identified system, and its verdicts depend on the prior; under the speed prior everything may look like an agent. | Active |
| THEORY-718 | In a linear-in-parameters model with Gaussian noise and a Gaussian prior, the evidence is exactly the best-fit likelihood times an Occam factor set by the posterior curvature, and when the design grows linearly with n that factor costs (k/2) log n | MacKay (1992), LIT-623, Eqs. 5–6 with Eqs. 16 and 19–20, read in NOTE-471: for y = Φw + noise with Gaussian noise of precision β and the prior w ~ N(0, α⁻¹I) on k weights, the posterior is exactly Gaussian, so the evidence is exactly P(D|w_MP) times the Occam factor P(w_MP)(2π)^{k/2} det^{-1/2}A, with A = αI + βΦᵀΦ. If ΦᵀΦ/n converges to a positive-definite matrix and w_MP stays bounded, the log Occam factor is −(k/2) log n + O(1). Exact for this model class only. The general regular case and the singular case are THEORY-087, which is Proposed. | Active |
| THEORY-1236 | Agency is relative to an observer: an identified system is an agent, for that observer, to the degree that a description of the function it optimises explains its behaviour better under the observer’s prior than any description of its input-output mapping | The relative definition of agency Orseau, McGill and Legg propose, filed beside THEORY-624 by the twenty-fourth status review (ADR-065 §3; ADR-047 kind 1). THEORY-624, Active, holds what the paper does and argues; whether the comparison is a criterion of agency for real systems is this entry’s question, with THEORY-624’s old condition word for word. | Proposed |
| THEORY-1253 | Treating the Bayesian evidence as a partition function with sample size as inverse temperature, the analogue of heat capacity, the rate at which leave-one-out predictive loss falls with sample size, equals half the parameter count in a regular model, as equipartition, and the real log canonical threshold in a singular one, falling below these where prior or discreteness fixes parameters; with multiplicity above one the approach to the singular value is only logarithmic, and the correspondence has no counterpart of physical current or dissipation | LaMont & Wiggins (2019), LIT-1882, read in NOTE-1481: the learning capacity C = −N²∂_N U and its equipartition values (Table II, eqs. 16–18, 24, 28, 30). Active: the regular entries were checked by differentiation, and the singular case exactly in K = a²b² (CASE-152’s model), where C approaches λ = ½ as ½ − 1/log n, not with the O(1/N) error the paper states. The statistical “temperature” is a sample size, so nothing in the dictionary is a physical heat flow. | Active |
| THEORY-1275 | In the toy model of superposition, SGD’s transitions have Bayesian counterparts whose critical sample sizes rise in the order training meets them, yet SGD was not seen to take the posterior’s main transition and ends in a phase the posterior has left at the training sample size | Chen, Lau, Mendel, Wei and Murfet (2023), LIT-1894, read in NOTE-1462. Their Bayesian antecedent hypothesis is one-way: training’s transitions have Bayesian counterparts. From their numbers the antecedents of the chain 4⁺⁺⁻ → 4⁺ → 5 → 5⁺ fall at n ≈ 190, 370 and 1,600–2,500, rising with training. But the posterior’s transition 5 → 6 at n ≈ 600 was not seen in training, and SGD on 1000 samples ends at the 5- or 5⁺-gon while the posterior at n = 1000 favours the 6-gon. Proposed: one model, one SGD setting, estimated learning coefficients. | Proposed |