nucleation

LIT-372: On the Information Bottleneck Theory of Deep Learning

2 min read

Properties5
StatusActive — read in full 2026-09-30 (NOTE-320); worth reading as the direct experimental test of all three claims of the IB account of deep learning (two phases; compression causes generalisation; SGD diffusion causes compression), each refuted as a general statement with the original code, and for its Appendix C argument that I(h;X) is infinite in a deterministic network so every finite information-plane value belongs to the analyst's noise model.
Version2
Tagslearning-theory, information-theory, representation-learning
Cited as “paper” byNOTE-320 — On the Information Bottleneck Theory of Deep Learning
Cited as “source” byTHEORY-035 — Deep networks generalise because the diffusion phase of SGD compresses each layer's information about the input, THEORY-039 — The later phases reported in neural-network training differ in kind: after the transition a network may simplify its weights, acquire item-specific information, lose generalisation, confine its tangent kernel or saturate its units, and none of these is a measured compression of I(X;T)

Graph View

Saxe et al. (2018), International Conference on Learning Representations (ICLR 2018), conference paper, OpenReview ry_WPG-A- (no DOI or arXiv id for this version); updated version published as J. Stat. Mech. (2019) 124020, doi:10.1088/1742-5468/ab3985 (verified against Crossref: IOP, published online 2019-12-20, seven authors as above). The doi field is the journal version, which was also read. — DOI-10.1088/1742-5468/ab3985

Standing in the record

Filed on 2026-09-30 at the owner’s request, as evidence on phases of training: whether they corroborate the information-bottleneck account of a fitting then a compression phase (LIT-324), and what phase structure they show. published: is the first appearance (ADR-002).

Also held in the Anthology of the SOTA as ANTH-LIT-509 (ADR-013): the anthology reads it for ML practice, this record for what it shows about phases of training and the information-bottleneck account.

It was filed Deferred, unread. NOTE-320 is the close reading of 2026-09-30, and it placed the work: Active — worth reading as the direct experimental test of all three claims of the IB account of deep learning (two phases; compression causes generalisation; SGD diffusion causes compression), each refuted as a general statement with the original code, and for its Appendix C argument that I(h;X) is infinite in a deterministic network so every finite information-plane value belongs to the analyst’s noise model.

Backlinks