Source
Tschannen et al. (2020), LIT-247, read in NOTE-749: §3.1 (Fig. 1, App. B), §3.2 (Fig. 3), §3.3 (Fig. 4a–b, Fig. 9) and §4.
What was actually shown
The argument. An invertible map preserves all the information in its input, so every invertible encoder of each half of an MNIST image maximizes the MI between the two halves’ representations. MI cannot prefer one invertible encoder to another. The paper adds a thought experiment: a PNG-compressed image keeps the information and loses the geometry. “The information content alone is not sufficient to guarantee a useful geometry in the representation space.”
Accuracy rises at constant MI (§3.1, Fig. 1a–b). RealNVP encoders trained with I_NCE or I_NWJ stay invertible. The estimate and linear-probe accuracy both rise during training while the true MI does not change.
Accuracy falls at constant MI (§3.1, Fig. 1c, App. B). An encoder of the same architecture, trained adversarially against a linear classifier with no MI estimator involved, stays invertible and ends with classification “significantly worse than using raw pixels”.
The encoder decides at a matched bound (§3.3). Training MLP and ConvNet encoders to the same bound value (|I_EST − t| for t = 2, 4), the ConvNet encoders give “clearly superior classification accuracy, for both I_NCE and I_NWJ”, though both architectures reach about 94% when trained supervised.
A tighter bound can do worse (§3.2). For I_NWJ, a higher-capacity MLP critic gives a higher bound value and worse features. For I_NCE all critics saturate near log K, and only accuracy differs.
What could have come out otherwise: the adversarial invertible encoder could have been unable to fall below raw pixels, which would have left open that information preservation sets a floor on linear-probe quality.
What this does not say
- Not that MI is irrelevant. A representation that discards the information a task needs cannot support it. The claim is that, once the information is kept, MI does not order representations.
- Not that InfoNCE is metric learning. InfoNCE with an inner-product critic is algebraically the multi-class K-pair loss (App. C), but that this explains its success is argued from the experiments, not tested.
- Not a general result about probes. Quality here is linear-probe accuracy on MNIST halves, and CIFAR-10 for two experiments.
- Not that tighter InfoNCE bounds give worse features. That is shown for I_NWJ only.
- Not advice. The paper’s recommendations to co-design critic, encoder and evaluation are machine-learning practice and belong in the anthology.
Connections
- THEORY-006 summarizes these experiments under the InfoNCE bound, as evidence that the bound’s value is not what the objective’s success tracks. This account states the stronger, estimator-free point beneath it, about true MI.
- THEORY-001. The optimal critic is a function of the density ratio alone, so the optimum says nothing about which encoder is good. At matched bound values the encoder’s inductive bias decides.
- CLAIM-193. Linear-probe quality depends on geometry that an invertible map can change while it keeps the information. That fits the claim that a linear probe family separates only what differs in the normalized kernel.