model-comparison.
13 of 1538 NOTE documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| NOTE-198 v2 | Minimum Description Length Revisited | MDL, reformulated around universal distributions and luckiness functions and presentable without coding theory, extends both penalized likelihood and Bayes, and unifies methods usually seen as rival — AIC vs BIC, cross-validation vs Bayes — within one worst-case framework. | Superseded — by NOTE-760 |
| NOTE-203 v2 | MDL induction, Bayesianism, and Kolmogorov complexity | Ideal MDL, derived from Bayes’s rule with the universal prior, coincides with Bayesian MAP selection exactly when the data are random with respect to the hypothesis and the hypothesis is random with respect to the prior (the “Fundamental Inequality”). For finite-set models it reduces to Kolmogorov’s minimal sufficient statistic, so compression is almost always the best strategy for both identifying hypotheses and predicting. | Superseded — by NOTE-801 |
| NOTE-471 | Bayesian Interpolation | Evidence ≈ best-fit likelihood × Occam factor, the ratio of posterior to prior accessible parameter volume. At the evidence maximum the regulariser’s χ² equals γ = Σ λ_a/(λ_a + α), the number of well-determined parameters, and the data χ² equals N − γ. Polynomials show an “Occam hill” in k, and the true Hermite model wins on data drawn from it. | Read |
| NOTE-473 | A Widely Applicable Bayesian Information Criterion | The Bayes free energy of a singular model is nL_n(w0) + λ log n − (m − 1) log log n + O_p(1), with λ the RLCT. The posterior mean of nL_n(w) at inverse temperature 1/log n (WBIC) reproduces the first two terms without knowing λ, equals BIC for regular models, and two temperatures estimate λ. In reduced-rank regression WBIC chose the true rank 100/100 times, and estimated RLCTs within about 0.5 of theory. | Read |
| NOTE-477 | Bayesian model reduction | The free energy of any model that differs from a fitted one only in its prior is ln E_Q[P̃/P] plus the fitted free energy, with closed forms for Gaussian, Dirichlet, beta, gamma, categorical and multinomial priors. Greedy pruning by this quantity recovers sparse structure in three simulations. Hierarchical models invert as a cascade of reductions. | Read |
| NOTE-484 | Bayesian hypothesis testing for psychologists: A tutorial on the Savage–Dickey method | For nested models the Bayes factor for a point null is the ratio of posterior to prior density of the tested parameter at the null value, under the larger model (Eq. 12, derived in Appendix A). It needs the nuisance parameters’ prior to be continuous at the null. Worked examples give BF10 ≈ 2.2 where p ≈ .006, and BF01 ≈ 4 in favour of a null of no group difference. | Read |
| NOTE-485 v2 | An Active Inference Approach to Modeling Structure Learning: Concept Learning as an Example Case | A partially observed MDP with spare hidden-state slots learns new animal concepts without feedback, refines coarse categories into fine ones, and generalises to an unseen animal in one shot. Bayesian model reduction on the learned state prior resets unneeded slots: right in 45–80% of runs for 4–7 true concepts, never for 2–3, and always when the likelihood is given. | Read |
| NOTE-487 | Post hoc Bayesian model selection | Models that differ only in their priors can be scored from one fitted full model: reduced evidence is full evidence times the posterior expectation of the prior ratio. Savage–Dickey is the point-mass case, ARD the continuous one. Under Laplace the reduced free energy is closed-form. Simulations recover a sparse regression (4,096 models) and a three-edge brain network (64 models). | Read |
| NOTE-508 | The Weighted Likelihood Ratio, Linear Hypotheses on Normal Location Parameters | States the Savage–Dickey ratio in general, not only for normal models: for a sharp hypothesis η(θ) = η_H with positive prior mass, the Bayes factor L_D(H) equals the alternative’s posterior density of η at η_H over its prior density there (3.9), provided the null’s nuisance prior is the alternative’s conditional prior at η_H (3.8). Dickey calls it Savage’s density ratio and credits the proof to Dickey & Lientz. He then applies it to normal means, the Model-I F test and the univariate and multivariate Behrens–Fisher problems with conjugate priors. | Read |
| NOTE-705 | Orseau, McGill & Legg — Agents and Devices | A formal counterpart of Dennett’s physical and intentional stances. A device is a mixture of input-output mappings; an agent is a mixture of goal-optimizing ε-greedy policies, by inverse RL. Bayes’ rule gives the posterior probability that a system is an agent. It is illustrated in a toy gridworld, with no human comparison. Agency is explicitly relative to the observer’s prior. A clearly identified system is assumed. | Read |
| NOTE-760 | Grünwald & Roos — Minimum Description Length Revisited | A review of MDL since 2007. Each model is replaced by one universal distribution, chosen for small worst-case log-loss regret against the model’s best fit. NML is the minimax choice, with complexity (k/2) log(n/2π) + log ∫ v√|I| + o(1). Estimation is penalised ML with a luckiness function v. The switch distribution nearly resolves AIC–BIC, the RIPr gives tests safe under optional continuation, and MDL and Bayes can be inconsistent under misspecification. Results are cited, not proved. | Read |
| NOTE-801 | Vitányi & Li — MDL induction, Bayesianism and Kolmogorov complexity | With the universal prior, the Fundamental Inequality puts −log Pr(D|H) − log P(H) within α = K(Pr(·|H)) + K(P) of K(D|H) + K(H) when D is random for H and H is random for P. So ideal MDL and MAP agree to a factor 2^{−α} among such “admissible” hypotheses. α can be as large as K(H), and outside the admissible set the two can pick different hypotheses. For finite-set models, MAP, ideal MDL and Kolmogorov’s minimal sufficient statistic coincide. | Read |
| NOTE-1481 | LaMont & Wiggins — thermodynamics and inference | Takes the evidence as a partition function with sample size N as inverse temperature, and reads off the thermodynamic derivatives. The internal energy is the leave-one-out predictive loss; the heat capacity becomes a “learning capacity” equal to K/2 in a regular model (equipartition) and to Watanabe’s learning coefficient in a singular one, and smaller when prior or discreteness fixes parameters (“freeze-out”). A prior making the Gibbs entropy zero reproduces AIC. The correspondence is formal; nothing in it is a physical current or dissipation. | Read |