nucleation

LIT-338: The information bottleneck method

2 min read

Properties6
StatusActive — read in full 2026-09-29 (NOTE-300); worth reading as the short, exact source of the IB functional, its self-consistent equations, the information plane and β as the one trade-off parameter. The phase-transition claim is cited out, not shown here. Nothing in it concerns gradients, SGD or batch size, so it is not the principle behind the gradient-as-channel narrative in the sense row 12 suggests.
Version2
Tagsinformation-theory, learning-theory, anthology-candidate
SupportsCLAIM-092 — Translation can be posed as a context-indexed information bottleneck: compress the source while preserving information about the responses in each measurement context separately
Cited as “paper” byNOTE-300 — The information bottleneck method
Cited as “source” byTHEORY-035 — Deep networks generalise because the diffusion phase of SGD compresses each layer's information about the input, THEORY-161 — In Zhao et al.'s (2025) and Chai et al.'s (2023) semantic rate–distortion frameworks, meaning is a latent variable with a known joint law with the observation, so their limits are indirect source-coding limits; Zhao et al.'s posterior-matching semantic distortion is, with KL divergence, exactly the information bottleneck, and with total variation a bound on the expected lost value of every bounded-loss decision, THEORY-163 — In Zaslavsky, Kemp, Regier and Tishby's analysis, the colour-naming systems of the World Color Survey languages and English lie near the information-bottleneck bound for compressing perceptual meanings into words, at a trade-off parameter just above 1 and nearer than hue-rotated variants of themselves, with the need distribution over meanings estimated from the naming data, THEORY-743 — For finite X and Y, every information-bottleneck optimum at every β gives one encoder to all inputs sharing p(y|x), so it is a stochastic coarsening of the minimal sufficient partition (the causal states of memoryless transduction); as β → ∞ the optima converge to that partition, but in general no finite β attains it, except when Y is a function of X, THEORY-791 — The colour-naming systems of the world's languages lie near the information-bottleneck bound for compressing perceptual meanings into words, and differences in one trade-off parameter place each language along it, as Zaslavsky, Kemp, Regier and Tishby propose

Graph View

Naftali Tishby, Fernando C. Pereira, William Bialek (1999), Proc. 37th Annual Allerton Conference on Communication, Control and Computing (1999); arXiv 2000 — ARXIV-physics/0004057

Standing in the record

Registered on 2026-09-29 at the owner’s request, because the owner’s prior-art novelty map (a private working document on concepts as projectors, the linear representation hypothesis and their neighbours) cites it: row 12: “Tishby IB” — the information-bottleneck principle behind the gradient-as-channel narrative. It is seeded from metadata and published descriptions and has not been read here. Only the year of first publication is known; published: uses 1 January.

Identification notes: Neither record holds the 1999 paper. Anthology holds Tishby & Zaslavsky 2015 “Deep Learning and the Information Bottleneck Principle” (ANTH-LIT-531); nucleation holds IB descendants only (LIT-240 Vera 2018, LIT-226 CEB). Allerton 1999 (Sept) is first publication; page range from recollection.

It was filed Deferred, unread. NOTE-300 is the close reading of 2026-09-29, and it placed the work: Active — worth reading as the short, exact source of the IB functional, its self-consistent equations, the information plane and β as the one trade-off parameter. The phase-transition claim is cited out, not shown here. Nothing in it concerns gradients, SGD or batch size, so it is not the principle behind the gradient-as-channel narrative in the sense row 12 suggests.

Backlinks