Why it is a question
THEORY-266 states Theorem 7 of LIT-233. For a loss in [0, 1] that is L-Lipschitz in a distortion, the average generalisation error along a trajectory is bounded, with high probability, through the rate-distortion function of the trajectory’s law given the sample, a coupling coefficient M and a distortion term 4Lε. In the proof, the trajectory’s information about its sample enters only through M. The paper’s CIFAR-10 experiment drops M and 4Lε. It shows only that the estimated rate and the test gap fall together over learning rates, on separate axes. For FCN4 the plotted rate term is about 3, which is vacuous for 0–1 loss.
So whether compressibility given the data, which is what the experiment compares, accounts for generalisation is open. The term that carries the dependence on the data was left out.
What would answer it
An evaluation of Theorem 7’s bound for a trained network that bounds or estimates M from data and keeps the 4Lε term, with the rate-distortion function estimated by a method whose error is controlled. Co-movement of an estimated rate with test error across learning rates or architectures, with M dropped, cannot settle it.