Source
Poole et al. (2019), LIT-246, §2.3, Eqs. 8–10. Supporting: Belghazi et al. (2018), LIT-256, Eqs. 1, 3–4 and App. 8.2.1 (mutual information as E[log dP_XZ/d(P_X⊗P_Z)], and the Donsker–Varadhan bound tight exactly at the log Radon–Nikodym derivative plus a constant; Belghazi et al. say “whenever”; “exactly” is the converse, which follows from their Eq. 25 but is not stated, and the supremum is attained only when P_XZ and P_X⊗P_Z are mutually absolutely continuous, otherwise it is approached in the limit); Tschannen et al. (2020), LIT-247; Fischer’s conditional entropy bottleneck, LIT-226, which relies on the bound (§6.3): its CatGen decoder is InfoNCE with the other examples of the same minibatch as negatives (NOTE-754).
What was actually shown
With K samples and negatives drawn independently from the marginal, I(X;Y) ≥ I_NCE for every critic, proved exactly as the multi-sample NWJ bound with the batch estimate of the partition function (LIT-246 Eqs. 8–10). The expectation in Eq. 10 is over K i.i.d. pairs from the joint, with each y_i’s negatives the other pairs’ x_j, so the proof covers in-batch negatives drawn this way (NOTE-739). Eq. 10 is printed with its indices transposed: the denominator sums over y_j for fixed x_i, while the derivation, Table 2 and Eq. 12 sum over x_j for fixed y_i. The printed form also holds, by symmetry, but the optimal critic’s free term c(y) is right only for the derived form. By construction I_NCE ≤ log K. That the optimal critic, log p(y|x) + c(y), does not depend on K is stated in the paper on Ma & Collins’s (2018) authority, not proved there. Belghazi et al. state the underlying identity: mutual information is the expected log Radon–Nikodym derivative of the joint with respect to the product of marginals, and the variational bound is tight at that log-derivative plus a constant. That it is tight exactly there is the record’s converse inference, and it holds only under mutual absolute continuity (NOTE-828).
Tschannen et al. show experimentally that representations can improve while the true MI does not change: with invertible encoders the true MI is constant, while the estimate and linear-probe accuracy both rise (§3.1, Fig. 1). They also show that a tighter bound can give worse representations, but for I_NWJ with an MLP critic (§3.2, Fig. 3). For I_NCE every critic saturates near log K, and the critic changes accuracy at nearly equal bound values (NOTE-749). So the bound’s value is not what the objective’s success tracks.
What this does not say
- That I_NCE approaches I(X;Y) at any stated rate as K grows (not proved in what was read).
- That maximizing I_NCE maximizes the representation’s information in a useful sense (Tschannen et al. argue against this reading).
- Anything for batches whose pairs are not i.i.d. With identically distributed but dependent pairs, I_NCE is neither a lower nor an upper bound on MI (Tschannen et al., LIT-247, App. D); Hénaff et al.’s CPC, with negatives from the same image, is their example.
- Anything for symmetric or two-sided losses, such as NT-Xent, that also draw negatives from the same view. Johnson et al. (LIT-258) App. B.1 treats NT-Xent with negatives drawn i.i.d. from the marginal; whether the bound extends to the two-sided loss is not checked here.
Connections
- THEORY-237 places this bound in its family. Poole et al. (LIT-246) derive Donsker–Varadhan, NWJ, MINE’s gradient and InfoNCE as relaxations of one unnormalized Barber–Agakov bound. InfoNCE is NWJ applied to a K-sample problem with the batch mean as normalizer, and its optimal critic is the log density ratio up to a function of y.
- THEORY-227 states the estimator-free point beneath Tschannen et al.’s experiments summarized above (LIT-247). Invertible encoders keep the true MI exactly, yet under a linear probe some do better than the raw input and some worse. So it is MI itself, not only this bound on it, that does not fix the quality of a representation.