Source
Kraskov, Stögbauer & Grassberger (2004), “Estimating Mutual Information”, arXiv cond-mat/0305641v1 (Phys. Rev. E 69, 066138), LIT-1163, read in NOTE-938.
What was actually shown
Derived. From E(log p_i) = ψ(k) − ψ(N) for the mass of the k-th neighbour ball (Eq. 17), and a constant density inside it (Eq. 18), the Kozachenko–Leonenko entropy estimate (Eq. 20). Counting marginal neighbours at the joint distance gives I⁽¹⁾ = ψ(k) − ⟨ψ(n_x + 1) + ψ(n_y + 1)⟩ + ψ(N) (Eq. 8); using rectangles gives I⁽²⁾ = ψ(k) − 1/k − ⟨ψ(n_x) + ψ(n_y)⟩ + ψ(N) (Eq. 9), with one correction neglected.
Observed numerically. Systematic errors scale as Φ(k/N) or Φ((k − ½)/N) (Eq. 32), are an order of magnitude below Darbellay–Vajda’s on Gaussians, and vanish for every independent pair tested.
Checked. I⁽¹⁾, k = 3, on unit Gaussians with r = 0, 0.3, 0.6, 0.9 at N = 200 and 2000: within one standard deviation of −½log(1 − r²) throughout, with a small positive bias at r = 0.9 (0.839 against 0.830 at N = 2000), as the paper’s Fig. 4 shows.
What could have come out otherwise: the estimate at r = 0 could have carried a k/N bias like the r ≠ 0 cases. In every case tested it did not.
What this does not say
- That the estimator is unbiased or consistent with a known rate. No bound is proved; the rate is fitted (N^{−0.85} at r = 0.9).
- That exactness for independence is a theorem. It is a conjecture, and the stronger ratio conjecture fails for the ordered Weinman exponential.
- Anything about conditional mutual information or transfer entropy. The paper does not estimate them. Nearest-neighbour transfer-entropy estimators are later extensions, not held.
- That it is invariant under reparametrization. MI is; the estimates are not, and the paper transforms variables case by case.
Connections
- A transfer-entropy estimate built this way (TERM-057) needs a null other than THEORY-641’s χ², which is for parametric and plug-in estimators.
- NOTE-788’s mutual-information estimates use an implementation of this estimator; its transfer entropy uses another.