nucleation

LIT-255: Information Theory with Kernel Methods

2 min read

Properties5
StatusActive — read 2026-10-14 (NOTE-837) in full; worth reading as the rigorous construction of the "density matrix" reading of kernel embeddings: with k(x,x) = 1 the uncentred covariance operator is a quantum state, its relative entropy lies between the KL of kernel-smoothed distributions and the KL itself, the empirical version is the matrix entropy of K/n, and kernel mutual information characterizes independence. Read it knowing that the kernel quantities only lower-bound Shannon ones and approach them at O(σ²) only on the torus, that conditional independence is characterized one way only, that Σ_p is uncentred, and that the paper never mentions fidelity, Bures distance, representation similarity or GNS.
Version2
Tagsinformation-theory, probabilistic-modeling, mathematics
Cited as “paper” byNOTE-229 — Information Theory with Kernel Methods, NOTE-837 — Bach — Information Theory with Kernel Methods
Cited as “source” byTHEORY-313 — With a kernel normalised to k(x,x) = 1, the covariance operator of a distribution is a density operator, and its quantum relative entropy is a lower bound on the Kullback–Leibler divergence that is tight for every pair of distributions only for an orthonormal, hence discrete, embedding, lies above the KL of the kernel-smoothed distributions, and on the torus, with Fourier coefficients proportional to e^(−σ‖ω‖₁) and strictly positive Lipschitz densities, is within O(σ²) of KL as the bandwidth σ shrinks, THEORY-811 — With a kernel normalised to unit diagonal, the kernel quantum relative entropy of two distributions is within O(σ²) of their Kullback–Leibler divergence as the bandwidth σ shrinks, for translation-invariant kernels on the torus generally and for kernels on other compact groups

Graph View

Francis Bach (2022), IEEE Transactions on Information Theory (2023); first appeared as arXiv preprint — ARXIV-2202.08545

Key takeaways

From the reading, NOTE-837.

  • A distribution becomes a quantum state. Under (A1), compact X and k(x,x) = 1, Σ_p is PSD with unit trace. It is injective in p when k² is universal (Proposition 1). It is uncentred: footnote 1.
  • Sandwiched between two KLs. D(Σ_p‖Σ_q) ≤ D(p‖q) by joint convexity, with equality only for an orthonormal, discrete embedding (Proposition 4(d)). D(p̃‖q̃) ≤ D(Σ_p‖Σ_q) by monotonicity under a measurement that smooths p with h(x,y) = ⟨ϕ(x), Σ^{−1/2}ϕ(y)⟩² (Eq. 8). On the torus the gap is at most O(dσ²) for Lipschitz densities (Proposition 5).
  • Computable. tr Σ̂ log Σ̂ = tr[(K/n) log(K/n)] (Proposition 6). It converges at about 1/√n without regularization (Proposition 7). As a Shannon entropy estimator it gets n^{−2/(d+4)}, the square root of the optimal rate.
  • Independence, yes; conditional independence, half. Kernel MI D(Σ_{p₁₂}‖Σ_{p₁}⊗Σ_{p₂}) is zero iff the variables are independent (Proposition 9). Data processing and submodularity hold, but their equality cases characterize conditional independence only one way (Propositions 10–11). Conditional entropies do not decompose.
  • Variational inference. The convex dual of the kernel KL gives upper bounds on log-partition functions. As the temperature goes to zero they become kernel sums-of-squares optimization, and they can be tightened by learning the kernel (§7).

Standing in the record

Filed on 2026-09-26 while pursuing, at the owner’s request, the connection GNS, unitary equivalence and the convergence of representations (see the curation entry of that date). Read in full on 2026-10-14 in NOTE-837. Its sandwich bound on KL is filed as THEORY-313.

The seed’s links, checked against the text:

  • GNS. Bach does not draw the link. The identity ⟨g, Σ_p f⟩ = E_p[fg] is a definition in §2.2, and GNS is not mentioned.
  • Representation similarity. Fidelity, Bures distance and representation similarity are not mentioned either. For a unit-diagonal, uncentred kernel, K/n equals the trace-normalized kernel that THEORY-233 compares. Representation kernels are usually centred and linear, so Bach’s results do not carry over to them as they stand.
  • KL. The link to the Radon–Nikodym/KL strand holds as the seed said.

Backlinks