Source
Li, Pogodin, Sutherland & Gretton (2021), LIT-248, eq. 5, eqs. 6–8, App. B.2. Muandet, Fukumizu, Sriperumbudur & Schölkopf (2017), LIT-260, Thm 2.4, Prop. 2.1, Lemma 3.1, Def. 3.2, §3.5.
What was actually shown
Muandet et al. define an RKHS by bounded evaluation, state Riesz (Thm 2.4) and derive the reproducing property from it (Prop. 2.1). Their Lemma 3.1: if E_P √k(X,X) < ∞, then f ↦ E_P f is bounded, so by Riesz there is a unique μ_P with E_P f = ⟨f, μ_P⟩, and μ_P = ∫ k(·,x) dP(x). MMD is then the RKHS norm ‖μ_P − μ_Q‖ (§3.5, eq. 3.29), a metric on distributions only for a characteristic kernel, and P ↦ μ_P is injective exactly when k is characteristic (Def. 3.2).
Li et al. show that with one-hot image-identity labels, (1/2N²)Σ_ij MMD²(i,j) = (N/Δ_l) HSIC(Z,Y) (App. B.2; the identity is unnumbered there, and eq. 5 is Theorem A.1’s proportional form). So maximizing HSIC maximizes the average squared distance between the mean embeddings of each image’s view distribution: views of one image concentrate, different images spread.
What this does not say
- That InfoNCE is HSIC. Li et al.’s §3.1 link (eqs. 6–8) is a Taylor approximation in a small-variance regime, but the link is not only that: eq. 28 is an exact decomposition of the large-batch loss as −HSIC(Z,Y) plus a log-mean-exp of the centred critic, and Theorem B.1 is a bound for every bounded kernel with no variance assumption.
- That the embedding is injective on a learned feature space. That needs a characteristic kernel there, and every result in the review assumes a fixed kernel.
- Anything about losses that are not kernel dependence measures.
Connections
- THEORY-312, filed from the close reading of LIT-248, holds eq. 28 and Theorem B.1 at the strength they are established: the large-batch InfoNCE loss is −HSIC(Z,Y) plus a nonnegative term, so it bounds a kernel dependence loss from above.