Michael Tschannen, Josip Djolonga, Paul K. Rubenstein, Sylvain Gelly, Mario Lucic (2019), International Conference on Learning Representations (ICLR 2020) — ARXIV-1907.13625
Key takeaways
From the reading, NOTE-749.
- Setup. Representations of the top half of MNIST digits, learned by maximizing I_NCE or I_NWJ against the bottom half, scored by a linear probe (pixel baseline about 85%, supervised about 94%); batch 128, so I_NCE ≤ log 128 ≈ 4.85. §§3.2–3.3 are repeated on CIFAR10 with the same conclusions (App. G).
- True MI constant, accuracy rising. With invertible RealNVP encoders, true MI is the same for every parameter value, yet the estimate and the probe accuracy both rise in training (Fig. 1a–b). An encoder of the same architecture, trained adversarially, stays MI-maximal and ends worse than raw pixels (Fig. 1c).
- Estimators prefer ill-conditioned encoders. From an identity initialization, the Jacobian’s condition number grows as the estimate rises (Fig. 2).
- Tighter is not better, shown for NWJ. An MLP critic reaches a higher I_NWJ and worse features than bilinear or separable critics. For I_NCE all critics sit near log K, and only accuracy differs (§3.2, Fig. 3).
- Encoder over estimator. At matched bound values (t = 2, 4), ConvNet encoders beat MLP encoders for both estimators (§3.3, Fig. 4a–b).
- Non-i.i.d. batches break the bound. If batch pairs are identically distributed but dependent, I_NCE is “neither a lower nor an upper bound on the true MI” (App. D); in a Gaussian toy both I_NCE and I_NWJ exceed it (Fig. 4c). CPC with same-image negatives is cited as such a case.
- InfoNCE as metric learning. I_NCE = log K − E[(1/K)Σ_i log(1 + Σ_{j≠i} e^{f(x_i,y_j) − f(x_i,y_i)})]; with f = φ(x)ᵀφ(y) this is Sohn’s multi-class K-pair loss (§4, App. C). The authors propose this view as the better explanation, by argument.
Standing in the record
Filed on 2026-09-26 while pursuing, at the owner’s request, the connection Radon–Nikodym, density ratios and contrastive objectives (see the curation entry of that date), and read in full on 2026-10-14 in NOTE-749. It is held as the record’s evidence that the optimal-critic and MI-bound reading of contrastive objectives does not explain which encoder features are good, and as the statement of where the InfoNCE bound’s i.i.d. assumption fails.
It is a supporting source of THEORY-006 and is named in THEORY-001’s limits. Its invertible-encoder result is filed as THEORY-227. The reading finds that THEORY-006 inverts the first experiment: it is the true MI that stays constant while the estimate rises, not the estimate. It also finds that THEORY-006’s tighter-bound claim holds as shown for NWJ, not InfoNCE. Its i.i.d. analysis builds on Poole et al. (LIT-246), whose proof it restates.
Tagged anthology-candidate (ADR-005): it carries recommendations for
machine-learning practice (co-design critic, encoder and evaluation; treat
negative sampling as central). It is kept here by the owner’s decision of
2026-09-26 that new work stays in nucleation until a transfer is judged
appropriate (ADR-010).