Chelombiev, Houghton and O’Donnell (2019) — ARXIV-1902.09037, read as NOTE-254.
Standing
Held because leaving it out would misrepresent the dispute, and because what it actually shows is not what it is usually taken to show. Read from its abstract it is a defence of the compression phase against LIT-509; read from its figures it is a third independent demonstration that the estimator determines the answer, pointing the opposite way to the rebuttal’s demonstration and reaching the same conclusion about generalization.
Its technical criticism is specific and lands. LIT-509 binned
ReLU activity over a single global range [0, m], m being the largest
activity anywhere in the network across all of training. This paper shows the
last ReLU layer dominates that maximum, so one global range under-resolves the
earlier layers, and that non-adaptive binning therefore under-estimates
compression.
Its ReLU headline does not survive its own averaging. The ReLU
compression it showcases is one initialization. Over 50, fitting and
compression “mostly cancel out” and I(T;X) is largely flat. The averages
do show compression for other activations: “ELU, Swish and centered softplus
activation functions also compress information, despite being
non-saturating” (Fig. 7), so the claim that fails is the one about ReLU.
It reruns LIT-508’s own setting. The network is the 10-7-5-4-3 fully
connected net on Shwartz-Ziv and Tishby’s 12-bit binary task, with 50
initializations “for consistency with previous studies”, though trained with
Adam. Under adaptive estimators the averaged tanh network still compresses,
the most strongly of the activations tried, but the averaged ReLU network
shows no distinct phase, and hidden-layer compression shows no significant
correlation with generalization — only the last softmax layer’s does. That
contradicts LIT-508’s claim that compression is what explains
generalization.