nucleation

LIT-246: On Variational Bounds of Mutual Information

3 min read

Properties5
StatusActive — read 2026-10-14 (NOTE-739) in full; worth reading as the single derivation that puts Barber–Agakov, Donsker–Varadhan, NWJ, MINE and InfoNCE in one family, and as the exact proof that InfoNCE is a lower bound on mutual information for every critic when the K batch pairs are i.i.d. from the joint, capped at log K. Read it knowing that Eq. 10 is printed with its indices transposed relative to its own derivation (the bound holds either way; the optimal critic log p(y|x) + c(y) belongs to the derived form), that the InfoNCE optimal critic is stated on Ma & Collins's authority rather than proved, that the optimal critic of the interpolated bound I_α is a conjecture, and that every experiment is on 20-dimensional Gaussian toys or dSprites.
Version3
Tagsinformation-theory, representation-learning, probabilistic-modeling, anthology-candidate
Cited as “paper” byNOTE-233 — Poole et al., variational bounds of mutual information, NOTE-739 — Poole et al. — On Variational Bounds of Mutual Information
Cited as “source” byTHEORY-001 — On a finite augmentation space, InfoNCE, logistic and spectral contrastive losses share one population optimum: the positive-pair density ratio, THEORY-006 — InfoNCE is a lower bound on mutual information for every critic and can never exceed the log of the batch size, THEORY-237 — The critic-based variational lower bounds on mutual information, Donsker–Varadhan, NWJ, MINE's and InfoNCE, are relaxations of one unnormalized Barber–Agakov bound, and, when the joint and the product of the marginals are mutually absolutely continuous, each attains its supremum over critics only at a critic equal to the log density ratio up to a term its normalization leaves free, where every bound but InfoNCE equals the mutual information, THEORY-308 — The neural Donsker–Varadhan estimator of mutual information is strongly consistent as an extremum estimator over compact families of critics, and, for critics bounded by M and L-Lipschitz in their d parameters, its uniform deviation from the population bound needs a number of samples set by d, M, L and the parameter radius, not by the data dimension

Graph View

Ben Poole, Sherjil Ozair, Aäron van den Oord, Alexander A. Alemi, George Tucker (2019), Proceedings of the 36th International Conference on Machine Learning (ICML 2019), PMLR 97:5171-5180 — ARXIV-1905.06922

Key takeaways

From the reading, NOTE-739.

  • One family. With q(x|y) = p(x)e^{f(x,y)}/Z(y) in the Barber–Agakov bound, the entropy term cancels and only log Z(y) is left to bound. Donsker–Varadhan, TUBA, NWJ (a(y) = e), MINE’s bias-corrected gradient (a(y) an EMA) and InfoNCE are different ways of bounding it (§2.2–2.3).
  • InfoNCE is a bound, exactly. NWJ applied to I(X_1, X_2:K; Y) with the batch mean (1/K)Σ e^{f(x_i,y)} as partition estimate gives I(X;Y) ≥ I_NCE for every critic, when the K pairs are i.i.d. from the joint (Eqs. 8–10). CPC’s derivation “relied on an approximation, which we show is unnecessary” (footnote 2).
  • Capped at log K. I_NCE ≤ log K, so its bias grows linearly with MI once MI exceeds log K (Fig. 3). Its optimal critic, log p(y|x) + c(y), does not depend on K; that form is cited to Ma & Collins (2018), not proved here.
  • Eq. 10’s indices are transposed. As printed it normalizes over y_j for fixed x_i; the derivation, Table 2 and Eq. 12 normalize over x_j for fixed y_i. Both are valid bounds; the stated optimal critic fits the derived one.
  • Bias against variance. NWJ is unbiased but high-variance; InfoNCE is low-variance but biased. The interpolated bound I_α (α ∈ [0,1], I_α ≤ log(K/α)) gives the best estimates after a full hyperparameter sweep (Table 3), and no bound gives low bias and low variance at large MI and small batch (§4).
  • Known encoders. When p(y|x) is tractable, Eqs. 12–13 sandwich MI with no learned parameters, and a reparameterized NWJ critic (Eq. 14) only has to learn the marginal q(y).
  • Density ratios to bounds. The optimal NWJ and InfoNCE critics are functions of log p(y|x)/p(y); a log-ratio estimator trained by a Jensen–Shannon objective and evaluated with NWJ (I_JS) gives a bound, not a plug-in estimate (§2.6, App. D).

Standing in the record

Filed on 2026-09-26 while pursuing, at the owner’s request, the connection Radon–Nikodym, density ratios and contrastive objectives (see the curation entry of that date), and read in full on 2026-10-14 in NOTE-739. It is held as the record’s exact proof that InfoNCE lower-bounds mutual information and as its statement of the log K ceiling and of the optimal critic as a log density ratio.

It is the source of THEORY-006 and a supporting source of THEORY-001. Its derivation of the critic-based bounds from one unnormalized Barber–Agakov bound is filed as THEORY-237. The reading confirms THEORY-006’s proof, with the index caveat above, and finds that the proof does cover in-batch negatives drawn from i.i.d. pairs. For THEORY-001 it supports the density ratio as the InfoNCE and NWJ optimum for general densities, but states the InfoNCE case on cited authority, and says nothing of the spectral or logistic losses. Its §2.5 structured bounds are the ones CEB (LIT-226) relies on; that link is not checked from the CEB side. Tschannen et al. (LIT-247) restate its proof to show where the i.i.d. assumption enters.

Tagged anthology-candidate (ADR-005): it carries methodological advice for MI estimation in machine learning. It is kept here by the owner’s decision of 2026-09-26 that new work stays in nucleation until a transfer is judged appropriate (ADR-010).

Backlinks