Ian Fischer (2020), Entropy 22(9):999 (2020) — ARXIV-2002.05379
Key takeaways
From the reading, NOTE-754.
- The criterion. A representation Z of X for predicting Y meets the Minimum Necessary Information (MNI) point when I(X;Y) = I(X;Z) = I(Y;Z) (Eq. 1). Equivalently, I(X;Y|Z) = I(X;Z|Y) = I(Y;Z|X) = 0.
- The objective is IB, reparametrised. CEB minimises I(X;Z|Y) − γI(Y;Z) (Eq. 5). Under Z ← X ↔ Y, I(X;Z|Y) = I(X;Z) − I(Y;Z), so CEB and IB “are equivalent for γ = β − 1”. What is new is the target, ρ = log γ = 0, and a compression term with a floor of zero.
- A measurable distance to the target. The residual information ReX = ⟨log e(z|x) − log b(z|y)⟩ ≥ I(X;Z|Y) uses a backward encoder b(z|y) where VIB has a marginal m(z). It “tells us exactly how many bits we are from the MNI point, assuming that our current classifier is optimal”.
- CatGen is InfoNCE. Normalising b(z|y) over the K labels in a minibatch gives a decoder “mathematically equivalent to … CPC … after an offset of log K”, by Poole et al.’s proof, and “constrained by log K” (§6.3).
- The evidence is empirical.
- On Fashion MNIST, CEB beats VIB at every ρ in accuracy and targeted PGD robustness, with equal rates.
- At ρ = 0 it is well calibrated and detects OoD inputs by the rate.
- It never learns random labels.
- On CIFAR-10, a CEB Wide ResNet (97.51%) beats Madry’s adversarially trained model on PGD attacks, except L∞ ε ∈ [2, 7], without adversarial training.
- The causal hypothesis has no theorem behind it.
- Not a sufficiency result. Footnote 2 avoids the word “sufficiency” so as not to suggest minimal sufficient statistics. The paper does not relate MNI to them formally, nor to Kolmogorov complexity.
Standing in the record
Filed on 2026-09-26 at the owner’s request, from a list they grouped under the heading Relating K-complexity to minimum sufficient statistics. Read on 2026-10-14 in NOTE-754. It is held as the operational form of the IB point where Z keeps exactly the information about X that bears on Y, and as the IB descendant that ties the IB to InfoNCE. It names the in-batch case of the InfoNCE bound that THEORY-006 leaves open, since CatGen’s negatives are the other labels in the minibatch. THEORY-035 names it as part of the IB programme. It makes no claim about a compression phase of SGD, so it does not bear on the account that theory rejects. Its argument that compressed representations generalise rests on Shamir et al.’s mutual-information bound, which LIT-245 argues is vacuous in realistic settings.
Tagged anthology-candidate (ADR-005). The paper carries a training
instruction (use CEB at ρ = 0), and that part belongs in the anthology.
The causal hypothesis is a separate claim and is not yet filed. It is
kept here by the owner’s decision of 2026-09-26 that new work stays in
nucleation until a transfer is judged appropriate (ADR-010).
Bibliographic access when seeded: Semantic Scholar (CorpusID 86765551,
DOI, PMC7597329) and Europe PMC for the journal record (electronic
publication 2020-09-08). An anonymous ICLR 2019 blind submission with
the same title exists on OpenReview (forum rkVOXhAqY7). It was not read,
and published: uses the arXiv v1 date.