nucleation

LIT-324: Opening the Black Box of Deep Neural Networks via Information

1 min read

Properties5
StatusActive — read in full 2026-09-29 (NOTE-298); worth reading as the source of the IB account of deep-learning dynamics and of the drift/diffusion gradient-SNR phase picture, which McCandlish et al. cite. Its compression and generalisation claims are contested: the anthology's ANTH-LIT-509 reports them refuted, which I have not checked. Read it as the claim, not the result, and never without the critique.
Version2
Tagslearning-theory, information-theory, representation-learning
Cited as “paper” byNOTE-298 — Opening the Black Box of Deep Neural Networks via Information
Cited as “source” byTHEORY-033 — The measured noise scale B_simple tracks the critical batch size within about 10× on six of eight tasks and about 20× on two generative models, so the gradient's noise-to-signal ratio sets the critical batch only as far as the local quadratic model with an optimal step holds, THEORY-035 — Deep networks generalise because the diffusion phase of SGD compresses each layer's information about the input

Graph View

Ravid Shwartz-Ziv, Naftali Tishby (2017), arXiv preprint — ARXIV-1703.00810

Standing in the record

Registered on 2026-09-29 at the owner’s request, because the owner’s prior-art novelty map (a private working document on concepts as projectors, the linear representation hypothesis and their neighbours) cites it: row 12 must-cite: information-bottleneck account of learning dynamics. It is seeded from metadata and published descriptions and has not been read here.

It is also held in the Anthology of the SOTA as ANTH-LIT-508. Holding it here too follows ADR-013: this record holds it for the owner’s prior-art map, the anthology for ML practice.

Identification notes: Held in anthology (ANTH-LIT-508); anthology also holds the critique Saxe et al. 2018 (ANTH-LIT-509) — worth noting since the IB-compression claim is contested.

It was filed Deferred, unread. NOTE-298 is the close reading of 2026-09-29, and it placed the work: Active — worth reading as the source of the IB account of deep-learning dynamics and of the drift/diffusion gradient-SNR phase picture, which McCandlish et al. cite. Its compression and generalisation claims are contested: the anthology’s ANTH-LIT-509 reports them refuted, which I have not checked. Read it as the claim, not the result, and never without the critique.

Backlinks