Shwartz-Ziv and Tishby (2017) — ARXIV-1703.00810.
Standing
The claim, and the record holds it as the claim rather than as a result. Its four stated findings are that most training epochs are spent compressing the input rather than fitting the labels; that compression begins when training error becomes small and SGD moves from drift to diffusion; that converged layers sit on or near the IB bound, by a “generalization through noise mechanism … unique to Deep Neural Networks and absent in one layer networks”; and that adding hidden layers dramatically reduces training time, so the main advantage of depth is computational.
Two of those are refuted and one is contested. LIT-509 shows the compression phase is a property of the nonlinearity and of the binning scheme, not of learning; that compression and generalization dissociate in all four combinations; and that full-batch gradient descent compresses too, so stochasticity is not the cause. LIT-507 partly defends the phase claim under better estimators and agrees with the rebuttal on generalization, finding no significant correlation in hidden layers. The fourth claim — the computational benefit of depth — is addressed by neither and is left standing here.
Not read as a NOTE. This record read its experimental setup and its
stated claims directly, which is what the documents around it cite, and did
not work through its theoretical sections. What a full reading would settle
is whether the IB-bound claim (iii) survives the estimator objection, since a
bound stated in units of a quantity that is infinite without an imposed noise
model inherits every problem LIT-509 raises.
Filed Active, as the record files a paper whose claims moved on. Status
here is standing in the anthology, not whether the claim held — the same
treatment LIT-470 gets after LIT-471.