nucleation

QUESTION-040: When the reification rule prices a candidate found in the same data, among many directions the data only partly determine, what must its Occam term count, and how far does the search inflate the candidate's d'?

3 min read

Properties7
StatusOpen
Version4
Tagsmodel-comparison, information-geometry, learning-theory
Linedistributed-agency
RefinesQUESTION-035 — Relative to a stated target, when is a combined form one system, where does its boundary lie, and how long does it last?
Refined byQUESTION-196 — Does Thresholdout keep every one of m adaptively chosen statistical queries within τ of its true mean, with probability 1 − β, on a holdout whose size grows only logarithmically in m, as Dwork and co-authors' Theorem 11 states?
BearingTHEORY-080 — In a linearised network with a Gaussian prior on its parameters, the Occam factor is a sum over the NTK's eigenvalues, so on spherical data each harmonic pays ½ log(1 + βnµ_k/α) and the kernel's eigenvalue decay schedules the complexity penalty across frequencies, THEORY-318 — A bound k on the max-information between a dataset and an analyst's chosen statistic multiplies by at most 2^k the probability that the statistic overfits that dataset, and an ε-differentially private mechanism on datasets of n points keeps k at most log₂e·εn, whatever the distribution on datasets, THEORY-028 — The expected bias of any data-dependent choice, and the expected generalisation error of any learning algorithm, is at most √(2σ²·I) for σ-subgaussian losses, where I is the information the output carries about the data

Graph View

The question

CLAIM-154 prices one candidate direction at a time, and its Gaussian prior and posterior are fixed in advance. Real candidates are blocks or couplings inside a model with many directions. Most of those directions are weakly determined, and the candidate was found by searching the data it is then tested on. The question has two parts. First, what is the right Occam count for a candidate in that setting? Second, by how much does selecting the candidate bias the d’² that the rule compares against that count?

Why the record needs it

CLAIM-154 itself says the rule “prices a candidate; it does not find one”. CLAIM-188 shows that the Gaussian term overcharges candidates whose model is singular where they are switched off. That leaves the regular but many-directional case open, and it also leaves the effect of search open. CLAIM-152 makes the target an input. If the candidate partition was picked to fit that target, the verdict inherits a selection bias. An answer would turn QUESTION-035’s “with what margin” into a corrected threshold.

What the record holds

  • THEORY-080: in a linearised Gaussian model “the Occam factor is a sum over the NTK’s eigenvalues”. Each direction pays ½ log(1 + βλ/α), and directions the data do not determine cost nothing. So the count is the number of well-determined directions, not the number of parameters. Limit: the model is linearised. The THEORY says this “removes that question”, meaning singularity, “rather than answering it”.
  • THEORY-028: the bias of “any data-dependent choice” is at most √(2σ²·I), where I is the information the choice carries about the data. A stable search, one that strong signal settles, biases little. Limit: the bound holds in expectation, for subgaussian statistics.
  • NML’s parametric complexity comp(M; v) (LIT-238, read in NOTE-760): the log of the number of distinguishable distributions, (k/2) log(n/2π) + log ∫ √|I|. It is a candidate Occam count. Like THEORY-080’s spectral count, it weights directions by Fisher information rather than counting parameters, and models with equal parameter counts differ in it. Limit: it does not address the selection-bias half of the question.
  • THEORY-318 (LIT-964, Supplementary Material only): a bound k on the max-information between a dataset and an analyst’s chosen statistic multiplies by at most 2^k the probability that the statistic overfits, and ε-differential privacy keeps k at most log₂e·εn. Whether a holdout can then validate many adaptively chosen statistics, through Thresholdout, is QUESTION-196. Limit: statistical queries on a discrete universe, and a procedure that prevents inflation, not a correction that says how far a search already run inflated its result.

What would answer it

A rule that combines the spectral Occam count with a selection correction, checked against exact Bayes factors on a coupled-block model whose candidate is found by search.

Backlinks