Saxe, Bansal, Dapello, Advani, Kolchinsky, Tracey and Cox (2018) —
doi:10.1088/1742-5468/ab3985, read as NOTE-253.
Standing
The principal rebuttal, and the reason this dispute is filable at all. It does not argue from a different setup: it replicates LIT-508 using the original authors’ released code, then varies one thing at a time.
Which version was read. This record read the ICLR 2018 conference
paper, supplied by the record’s owner. OpenReview returns 403 to /pdf and
to its API, and the JSTAT page resolves but serves no article text — but an
open route does exist: co-author Artemy Kolchinsky’s publications page
(artemyk.github.io/publications) links PDFs of both the ICLR paper and the
JSTAT version. Corrected 2026-09-30; this read “because no open route to it
exists”. The identifier filed is the JSTAT DOI, which is the later, updated
version and is what ADR-009 prefers; any numbers quoted in
NOTE-253 are from the ICLR version, and that is said there too.
The JSTAT version (2019) changes framing, not results. The re-reading done for the nucleation record compared the two and found no new experiment. The abstract adds that the IB claims “instead reflect assumptions made to compute a finite mutual information metric in deterministic networks”; the introduction and discussion add that the binning/noise assumption “is a distinct question from that of the method used to estimate mutual information”, so the results “do not simply reflect the need for more accurate MI estimation techniques”; the discussion softens ICLR’s “ReLUs do not compress in general” to “do not always compress” and concedes binned MI “could still provide a useful metric that connects to generalization”.
Its date. published: is 2018-02-15, the publication date Semantic
Scholar gives for the ICLR record, and the month the cover of the
ResearchGate copy gives. It was 2018-04-30, the conference’s first day,
which the ADR-062 backfill flagged and left because Crossref
knows only the 2019 journal date. The anonymous submission was probably public on
OpenReview earlier, in late 2017; that could not be verified, since
OpenReview refuses automated reads.
What it leaves standing. It addresses three claims. LIT-508’s fourth — that the main benefit of depth is computational, dramatically reducing epochs to good generalization — is not examined by this paper and is not refuted by it.
Its deepest point is not the one it is cited for. It is usually cited for
“ReLU does not compress”. The more general result is in its Appendix C: the
same tanh network, the same training run, binned evenly in net input
rather than evenly in activity, shows no compression. The true I(h;X)
in a deterministic network is infinite, so every finite number on an
information plane is a property of a noise model the analyst imposed and the
network never had.
The counter-rebuttal, LIT-507, disputes the estimator and concedes the
conclusion. It targets this paper’s ReLU binning: one global range [0, m],
set by the largest activity anywhere and dominated by the last ReLU layer,
under-resolves the earlier layers and under-estimates compression. Rerun with
adaptive estimators on the original study’s small network and task, its own average over
50 initializations still shows a ReLU network with no distinct fitting or
compression phase, and it finds no significant correlation between
hidden-layer compression and generalization. It does not address Appendix C.