nucleation

LIT-256: Mutual Information Neural Estimation

2 min read

Properties5
StatusActive — read 2026-10-14 (NOTE-828) in full; worth reading as the paper that states mutual information as the expected log Radon–Nikodym derivative and proves the Donsker–Varadhan bound tight at that log derivative plus a constant. Read it knowing that its consistency theorem is for the exact empirical supremum over a compact network family, that two steps of its proof have gaps, that its gradient-bias fix is a heuristic, and that the finite-sample estimate is not a bound (LIT-246).
Version2
Tagsinformation-theory, probabilistic-modeling, representation-learning, anthology-candidate
Cited as “paper” byNOTE-224 — Belghazi et al., MINE, NOTE-828 — Belghazi et al. — Mutual Information Neural Estimation
Cited as “source” byTHEORY-006 — InfoNCE is a lower bound on mutual information for every critic and can never exceed the log of the batch size, THEORY-237 — The critic-based variational lower bounds on mutual information, Donsker–Varadhan, NWJ, MINE's and InfoNCE, are relaxations of one unnormalized Barber–Agakov bound, and, when the joint and the product of the marginals are mutually absolutely continuous, each attains its supremum over critics only at a critic equal to the log density ratio up to a term its normalization leaves free, where every bound but InfoNCE equals the mutual information, THEORY-308 — The neural Donsker–Varadhan estimator of mutual information is strongly consistent as an extremum estimator over compact families of critics, and, for critics bounded by M and L-Lipschitz in their d parameters, its uniform deviation from the population bound needs a number of samples set by d, M, L and the parameter radius, not by the data dimension

Graph View

Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, R Devon Hjelm (2018), Proceedings of the 35th International Conference on Machine Learning (ICML 2018), PMLR 80:531-540 — ARXIV-1801.04062

Key takeaways

From the reading, NOTE-828.

  • The identity. I(X;Z) = E_{P_XZ}[log dP_XZ/d(P_X⊗P_Z)] (Eqs. 1, 3–4). The Donsker–Varadhan gap for a critic T is exactly KL(P‖G_T), with G_T the Gibbs reweighting e^T dQ/E_Q[e^T] (App. 8.2.1, Eq. 25). So the bound is tight at T = log dP/dQ + C, and only there. A real-valued critic attains it only if the joint and the product of marginals are mutually absolutely continuous; that consequence is the reader’s.
  • Consistency. On a compact domain with Lebesgue densities, the population supremum over some compact network family is within any η of MI, and the empirical supremum converges to it almost surely (Lemmas 1–2, Thm 2). This is for the exact empirical maximizer. The approximation proof truncates the log ratio only from above, and the estimation lemma states a stronger quantifier than the uniform law of large numbers gives.
  • Sample complexity. Uniform deviation ε with probability 1 − δ after 2M²(d log(16KL√d/ε) + 2dM + log(2/δ))/ε² samples, with d the parameter count, for bounded, Lipschitz critics (Thm 3).
  • Not a bound at finite n. The paper treats only the gradient bias, with a moving-average heuristic it does not prove. Poole et al. (LIT-246) show that the estimate is neither an upper nor a lower bound.
  • Applications. MI as a regularizer for GANs, ALI and the information bottleneck. The IB is stated with entropies, not in Radon–Nikodym form.

Standing in the record

Filed on 2026-09-26 while pursuing, at the owner’s request, the connection Radon–Nikodym, density ratios and contrastive objectives (see the curation entry of that date). Read in full on 2026-10-14 in NOTE-828, which supersedes the seeded skim NOTE-224.

It is cited correctly by THEORY-006 and THEORY-237, as supporting, for the Radon–Nikodym identity and the Donsker–Varadhan tightness. THEORY-308 was filed from the reading: the estimator’s strong consistency and sample complexity, held to what the proofs establish.

Tagged anthology-candidate (ADR-005). Its MI-regularization recipes are machine-learning practice. It is kept here by the owner’s decision of 2026-09-26 that new work stays in nucleation until a transfer is judged appropriate (ADR-010).

Backlinks