anthology-of-the-sota

LIT-531: Deep Learning and the Information Bottleneck Principle

2 min read

Properties3
StatusActive
Tagsanalysis-and-evaluation, representation-and-encoding, model-architecture
Cited as “paper” byNOTE-275 — The trunk states the condition its successors were refuted for dropping

Graph View

Tishby and Zaslavsky (2015) — ARXIV-1503.02406, IEEE Information Theory Workshop 2015. Read as NOTE-275.

Key takeaways

  • The proposal. Quantify any DNN by the mutual information between each layer and the input and output variables, and treat the network as a sequence of points on an information plane.
  • The claims it makes for that frame: optimal information-theoretic limits and finite-sample generalization bounds; the optimal architecture — number of layers, features per layer — related to bifurcation points of the IB tradeoff; hierarchical representations corresponding to structural phase transitions along the information curve.
  • The famous picture is labelled as a hypothesis. Figure 2 is “a qualitative information plane, with a hypothesized path of the layers in a typical DNN”. The authors’ own words, in the caption.
  • And it names the condition. “Getting closer to the optimal limit requires stochastic mapping between the layers.”
  • Five pages, no experiments.

Standing in the anthology

It is the trunk under a dispute this record filed without it. For #220 the record took LIT-508 (Shwartz-Ziv and Tishby’s measurement), LIT-509 (Saxe et al.’s rebuttal) and LIT-507 (Chelombiev et al.’s counter-rebuttal), and produced THEORY-058 and SOTA-312. The paper all three are arguing about was not here.

Reading it changes who is owed what. The 2018 rebuttal’s central objection is that I(X;T) is ill-defined for a deterministic network and that the reported compression is an artifact of the binning. This paper says the optimum requires stochastic maps — so the assumption whose absence the rebuttal turns on is stated in the founding paper, and dropped downstream.

Active, and the reason is narrow. Its proposal is live, its picture is labelled a hypothesis, and what the record disputes is a later paper’s measurement rather than this one’s framing. The claims it makes beyond the framing — bifurcation points fixing the optimal depth, phase transitions on the information curve — are unevidenced here and untested since, and the reading says so.

Backlinks