Contribution
A lecture that sets up statistical decision theory, defines when one statistical model is at least nearly as good as another for every decision problem, and states and half-proves Le Cam’s randomization theorem, which turns that comparison into the existence of a stochastic kernel. It then defines Le Cam’s distance between models and gives four examples of exact or asymptotic equivalence. It is teaching material: one theorem is proved, one sketched, and the asymptotic examples are stated with outlines.
Key insight
“Model M can do anything model N can, up to ε” has two readings: for every decision problem M’s best rule is within ε of whatever N’s rule achieves, and M’s data can be run through a noisy channel to look like N’s data, to within ε in total variation. The theorem says they are the same, and the proof of the hard half is a minimax argument.
Assumptions
- Models share a parameter set. M = (𝒳, (P_θ){θ∈Θ}) and N = (𝒴, (Q_θ){θ∈Θ}) (slide 10).
- Losses take values in [0, 1] (slide 11; the bound underlined in the annotation). With a loss in [0, c] the bound scales to cε.
- The decision definition ranges only over finite Θ₀ ⊆ Θ and finite action sets (slide 11).
- The hard direction assumes Θ finite (“for simplicity”, slide 15). It also needs the observation spaces finite, which the slide does not say: the action set it chooses is 𝒴, and the definition admits only finite action sets; and the minimax swap needs a compact set of rules for M.
Key results
- Risk (slide 4). R_θ(δ) = E_{a∼δ(·|X)} E_{X∼P_θ}[L(θ, a)]. A deterministic rule is the case δ = T(X) (annotation).
- Deficiency (slide 11), Le Cam (1964). M is ε-deficient relative to N if, for any finite Θ₀ ⊆ Θ, any finite action set A, any loss L : Θ × A → [0, 1] and any rule δ_N for N, there is a rule δ_M for M with R_θ(δ_M) ≤ R_θ(δ_N) + ε for all θ ∈ Θ₀. The annotation writes it as sup_{Θ₀} sup_L sup_{δ_N} inf_{δ_M} sup_θ (R_θ(δ_M) − R_θ(δ_N)) ≤ ε. Below the box: “hardness result for model M ⟹ hardness result for model N”.
- Randomization (slide 12). N is a randomization of M if Q_θ = K P_θ for all θ, for some kernel K : 𝒳 → 𝒴; then M is 0-deficient relative to N, via δ_M = δ_N ∘ K (annotated diagram).
- Theorem (slide 13). “Model M is ε-deficient relative to N if and only if there exists a stochastic kernel K : 𝒳 → 𝒴 such that sup_{θ∈Θ} ‖Q_θ − K P_θ‖_TV ≤ ε.”
- Easy direction, proved (slide 14). Take any A, L and δ_N, and set δ_M = δ_N ∘ K. Then R_θ(δ_M) − R_θ(δ_N) = E_{a∼δ_N(·|Y)}(E_{Y∼KP_θ}[L(θ, a)] − E_{Y∼Q_θ}[L(θ, a)]) ≤ ‖Q_θ − K P_θ‖_TV, by the annotated identity ‖P − Q‖TV = sup{0≤f≤1} (E_P f − E_Q f). The bound holds at every θ, for every rule δ_N, with the same kernel. This direction is complete.
- Hard direction, sketched (slide 15). “For simplicity assume that Θ is a finite set.” Deficiency gives, for every δ_N, sup_L sup_π inf_{δ_M} E_{θ∼π}(E_{a∼δ_M(·|X)}E_{X∼P_θ} − E_{a∼δ_N(·|Y)}E_{Y∼Q_θ})[L(θ, a)] ≤ ε, with π a prior. The annotations note that the expression is “linear in δ_M” and “linear in {π(θ)L(θ, a)}”, and that the two sups combine into one over the product π·L. “Swap the inf and sup” gives inf_{δ_M} sup_L sup_π (…) ≤ ε, and the inner sups are labelled ‖δ_M P_θ − δ_N Q_θ‖_TV (the maximum over θ is implicit). The handwritten last line reads “Choose A = [·] and δ_N = identity”; the symbol for the action set is hard to read, and the construction needs it to be 𝒴, so that δ_N Q_θ = Q_θ and the minimising δ_M is the kernel K.
- Le Cam’s distance (slide 16). Δ(M, N) is the smallest ε such that each model is ε-deficient relative to the other; a pseudo-metric; equivalently Δ = max{inf_K sup_θ ‖K P_θ − Q_θ‖_TV, inf_L sup_θ ‖L Q_θ − P_θ‖_TV}. “A convex program, but …” with the annotation “infinite card. of Θ and evaluation of TV”.
- Examples (slides 17–25). Δ(M, N) = 0 for the model induced by T(X) exactly when T is sufficient (slide 18, statement only). For fixed support size k, multinomial and Poissonized sampling are asymptotically equivalent (slide 19), by an explicit randomization (slide 20) and a χ²-based bound on the expected total variation that the annotations carry to 0 for fixed k as n grows (slide 21); the homework asks for the tight Θ(min{1, √(k/n)}). Nonparametric regression, Gaussian white noise and density estimation are equivalent for smoothness above 1/2 (slides 22–25), with the KL bound for regression ⇔ white noise written out and the density case described as relying on “multi-resolutional analysis”.
Claims
| id | claim | strength | support |
|---|---|---|---|
| C1 | If a kernel K has sup_θ ‖Q_θ − K P_θ‖_TV ≤ ε, then for every [0, 1]-loss problem and every rule for N, the rule δ_N ∘ K for M has risk at most ε more at every θ | strong | proved, slide 14 |
| C2 | Conversely, ε-deficiency in the decision sense gives a kernel with sup_θ ‖Q_θ − K P_θ‖_TV ≤ ε | moderate as presented | sketched for finite Θ, slide 15: one minimax swap, conditions unstated; true by Le Cam (1964), not read |
| C3 | Le Cam’s distance is a pseudo-metric and equals the larger of the two inf–sup TV programmes | weak as presented | stated, slide 16, following C2 |
| C4 | A statistic induces a model at distance 0 exactly when it is sufficient | weak as presented | stated, slide 18, with an annotated hint |
| C5 | Multinomial and Poissonized models are asymptotically equivalent for fixed k | moderate | randomization given, bound outlined, slides 19–21 |
| C6 | Regression, white noise and density estimation are asymptotically equivalent above smoothness 1/2 | weak as presented | outlined, slides 22–25, citing Brown et al. 2004 |
Concepts
- ε-deficiency — the decision-theoretic definition of slide 11: uniform in θ, rule by rule, over finite problems with losses in [0, 1].
- Randomization — N = K M for a kernel K, model by model.
- Le Cam’s distance — the symmetrised deficiency.
- Asymptotic equivalence — Δ(M_n, N_n) → 0 along Θ_n ↑ Θ.
Connections
Le Cam’s 1964 paper (cited, with the annotation “deficiency paper”) is the source; Le Cam’s 1986 book is annotated “a hard book”. Blackwell’s comparison of experiments is the case ε = 0, held in the record through LIT-781 (Blackwell’s 1953 paper, read in NOTE-595), whose finite converse is also a minimax argument. CLAIM-381 defines the deficiency between two finite tables in the kernel form of slide 13, as a least maximum over items of a total variation, and shows it a linear programme. van Rooyen’s paper (LIT-1859, read in NOTE-1430) proves the same converse with a prior fixed.
Bearing on the record
- CLAIM-488: partly. Its note of 2026-10-30 says the first step still lacks Le Cam’s randomization criterion for finite experiments: that a deficiency of ε bounds every [0, 1]-loss risk gap by ε, which the record already has, and the converse, which it has neither read nor re-derived. This lecture states the criterion in both directions, uniform in θ, in the form CLAIM-381 computes (slide 13). The first direction is proved (slide 14) and agrees with the record’s two lines. The converse is only sketched (slide 15). The sketch is the right argument, and in CLAIM-381’s finite setting each gap it leaves can be closed. A checked re-derivation would need to state three things the slide does not: (1) the observation spaces are finite, so that the action set 𝒴 is admissible and M’s rules form a compact convex set; (2) the minimax theorem applies, the objective being bilinear in M’s rule and in the product π(θ)L(θ, a), which ranges over the compact convex set {g : 0 ≤ g(θ, a) ≤ π(θ)}; (3) the supremum over [0, 1]-losses and priors of the averaged gap is max_θ ‖δ_M P_θ − δ_N Q_θ‖_TV. With those, taking the action set 𝒴 and δ_N the identity gives the kernel. So the reading supplies the converse’s argument and its statement, not its proof; the claim’s note asks for “a checked re-derivation”, and this reading makes one short but does not do it.
- Le Cam’s distance and CLAIM-488’s “approximate case”. Slide 16 is the symmetric version; the claim’s first step needs only the directed deficiency.
- No instruction for machine-learning practice; nothing for the anthology.
Limitations
- The converse is a sketch. Θ finite is assumed “for simplicity”, the minimax theorem is invoked by the words “swap the inf and sup” with its conditions only hinted by the annotations, finite observation spaces are not stated, and the extension from every finite Θ₀ to the theorem’s sup over all of Θ (which needs a compactness argument when Θ is infinite) is not given.
- The annotation choosing the action set is ambiguous. It looks like “A = Θ” in the handwriting; only A = 𝒴 makes the construction work.
- Teaching material. No proofs for the sufficiency theorem or the asymptotic equivalences beyond outlines.
Open questions
- The written-out finite converse, with the three conditions above, is a page; whether the record wants it as its own CLAIM is for the status review.