nucleation

LIT-248: Self-Supervised Learning with Kernel Dependence Maximization

3 min read

Properties5
StatusActive — read 2026-10-14 (NOTE-830) in full; worth reading as the place where an SSL objective becomes a statement about kernel mean embeddings: with image identity as a one-hot label, HSIC(Z,Y) is the gap between the mean kernel on positive pairs and on all pairs, (1/2N²)Σ MMD²(i,j) = (N/Δl)·HSIC(Z,Y), and the large-batch InfoNCE loss is exactly −HSIC(Z,Y) plus a log-mean-exp of the centred critic. Read it knowing that Theorem B.1 prints its remainder with the wrong sign, that the InfoNCE it analyses is the large-batch form and not the K-negative estimator, that "InfoNCE works because it estimates kernel dependence" is an interpretation no experiment isolates, and that the experiments are single runs.
Version2
Tagsrepresentation-learning, probabilistic-modeling, anthology-candidate
Cited as “paper” byNOTE-234 — SSL-HSIC (kernel dependence maximization), NOTE-830 — Li et al. — Self-Supervised Learning with Kernel Dependence Maximization
Cited as “source” byTHEORY-003 — Maximizing HSIC between representations and image identity maximizes the average squared MMD between the images' view distributions, whose kernel mean embeddings exist by the Riesz representation theorem, THEORY-312 — The large-batch InfoNCE loss is the negative HSIC between representations and image identity plus a nonnegative log-mean-exp of the centred critic, so it bounds a kernel dependence loss from above

Graph View

Yazhe Li, Roman Pogodin, Danica J. Sutherland, Arthur Gretton (2021), Advances in Neural Information Processing Systems 34 (NeurIPS 2021); first appeared as arXiv preprint — ARXIV-2106.08320

Key takeaways

From the reading, NOTE-830.

  • HSIC with identity labels. On N equally likely images with one-hot labels and a label kernel Δl·1[y = y′] + l₀, HSIC(Z,Y) = (Δl/N)·(E_pos k(Z,Z′) − E k(Z,Z′)) (Theorem A.1). The label kernel enters only as a prefactor.
  • The MMD form. With μ_i the mean embedding of image i’s view distribution, (1/2N²)Σ_ij MMD²(i,j) = (N/Δl)·HSIC(Z,Y) (App. B.2), an exact derivation with no stated integrability condition. It needs the mean embeddings to exist, which holds for the bounded kernels used.
  • InfoNCE, exactly. In its large-batch form, L_InfoNCE = −HSIC(Z,Y) + E_Z log E_Z′ exp(k − μ₁) (eq. 28), an identity.
  • InfoNCE, bounded. For |k| ≤ k_max, Theorem B.1 bounds −HSIC(Z,Y) + γHSIC(Z,Z) by L_InfoNCE plus a remainder of second order in Var_Z′ k. The printed theorem gives the remainder a minus sign; its own proof step gives a plus. With the cosine critic at τ = 0.1, γ = 0.0475. The small-variance Taylor form (eq. 7) is the approximation the main text leads with.
  • Clustering. With linear kernels and centred unit-norm features, −HSIC(Z,Y) is within-image scatter, a k-means objective (eq. 10). BYOL’s loss is read as its two-view case, which depends on the centring assumption.
  • Estimation. The i.i.d. HSIC estimator is biased for SSL batches. An unbiased one (Theorem A.2) and an O(1/B)-biased one (eq. 17) are given, with random Fourier features for an inverse multiquadric kernel (Theorem C.1).
  • Results. 74.8% ImageNet top-1 linear evaluation with a target network, 72.2% without. Single runs.

Standing in the record

Filed on 2026-09-26 while pursuing, at the owner’s request, the connection Riesz, reproducing kernels and spectral representation learning (see the curation entry of that date). Read on 2026-10-14 in NOTE-830.

It is one of the two sources of THEORY-003. The reading confirms the MMD identity THEORY-003 relies on, with the constant corrected to N/Δl, and finds the InfoNCE link stronger than THEORY-003 says: an exact decomposition and a bound, not only an approximation. That stronger statement is filed as THEORY-312, with the sign of Theorem B.1’s remainder corrected and the record’s own Jensen step that makes −HSIC(Z,Y) ≤ L_InfoNCE exact.

Tagged anthology-candidate (ADR-005): the seed judged it chiefly about machine-learning practice, and its method half (the SSL-HSIC loss, its estimator and training recipe) is. It is kept here by the owner’s decision of 2026-09-26 that new work stays in nucleation until a transfer is judged appropriate (ADR-010). The theory half, which is what the record uses, is not practice.

Backlinks