Daniel D. Johnson, Ayoub El Hanchi, Chris J. Maddison (2022), International Conference on Learning Representations (ICLR 2023); first appeared as arXiv preprint — ARXIV-2210.01883
Key takeaways
From the reading, NOTE-751.
- One kernel behind three losses. The exponential and the temperature are counted as part of the model. Over all kernel functions on a finite augmentation space, the NT-Logistic and spectral losses are then minimized by K⁺(a₁,a₂) = p⁺(a₁,a₂)/(p(a₁)p(a₂)). NT-Xent/InfoNCE is minimized by K⁺·b(a₁), with b constant on each communicating class of the positive-pair chain (App. B). K⁺ is a Mercer kernel with an explicit feature map φ⁺(a)_z = p(a|z)√p(z)/p(a) (Def. 2.1).
- Invariance in the eigenbasis. In the L²(p)-orthonormal eigenfunctions of the chain p⁺(a′|a), E_p⁺[(g(a₁) − g(a₂))²] = Σ(2 − 2λ_i)c_i² (Prop. 3.1).
- Kernel PCA is the chain’s eigendecomposition. Population kernel PCA under K⁺ gives σ_i² = λ_i and h_i = λ_i^(1/2) f_i (Thm 3.2). The proof is one SVD: the feature covariance and the symmetrized adjacency D_A^(−1/2)P_(A,A)D_A^(−1/2) are BBᵀ and BᵀB for B = Φ⁺D_A
- Minimax optimality. The span of the top-d eigenfunctions minimizes both the worst-case discrepancy of a unit-norm predictor and the worst-case squared approximation error over targets with discrepancy at most ε (Thm 4.1). The worst case is ε/(2(1 − λ_(d+1))). Below d* (the number of eigenvalues equal to 1), no d-dimensional space has finite worst-case error (App. D.1).
- In practice, partly. On two synthetic tasks with computable K⁺, learned kernels plus kernel PCA approximately recover the eigenfunctions. Recovery degrades with head constraints and with weak augmentation. The authors could not obtain reliable eigenfunctions from real-data models (App. E.1, F.5).
Standing in the record
Filed on 2026-09-26 while pursuing, at the owner’s request, the connection Riesz, reproducing kernels and spectral representation learning (see the curation entry of that date). Read on 2026-10-14 in NOTE-751.
It is held as the most explicit single statement that InfoNCE-family losses target one kernel, and that the downstream-optimal representation is that kernel’s top eigenfunctions. It gives the spectral reading of self-supervised learning an optimality claim, not only a correspondence. It is the principal source of these theories:
- THEORY-001 is confirmed. For InfoNCE the minimum is over symmetric functions and its form is cited from Van den Oord et al.
- THEORY-007 is confirmed on every cited result. Theorem 4.1 needs d ≥ d*.
- THEORY-005. The operator identity it calls its own appears in App. E.2, without an integrability condition.
- THEORY-009 is borne out. Self-adjointness comes from P_(A,A) being a Gram matrix, so both views must be drawn from the same p(A|Z), and Riesz is not used.
It reads HaoChen et al. (LIT-249) as the same decomposition up to a p(a)^(1/2) scaling.
Tagged anthology-candidate (ADR-005). The paper recommends a training
pipeline (a contrastive loss, then kernel PCA, with unconstrained heads),
which is machine-learning practice. It is kept here by the owner’s decision
of 2026-09-26 that new work stays in nucleation until a transfer is judged
appropriate (ADR-010). What this record reads it for is the kernel and
eigenfunction statement.