nucleation

LIT-261: A Generalized Representer Theorem

3 min read

Properties5
StatusActive — read 2026-10-14 (NOTE-836) in full; worth reading as the short proof that any minimizer of an arbitrary cost on the training outputs plus a strictly increasing function of the RKHS norm is a kernel expansion on the training points, by the reproducing property and orthogonal decomposition, with no convexity and no Riesz theorem. Read it knowing that the theorem is stated over countable kernel expansions of finite norm rather than the completed RKHS, that it assumes a minimizer exists, and that its semiparametric Theorem 2 is stated without proof.
Version2
Tagslearning-theory, mathematics
Cited as “paper” byNOTE-230 — A Generalized Representer Theorem, NOTE-836 — Schölkopf, Herbrich & Smola — A Generalized Representer Theorem
Cited as “source” byTHEORY-314 — Every minimizer of an arbitrary cost on the training outputs plus a strictly increasing function of the RKHS norm lies in the span of the kernel functions at the training points, by the reproducing property and orthogonal decomposition alone

Graph View

Bernhard Schölkopf, Ralf Herbrich, Alex J. Smola (2001), Computational Learning Theory (COLT/EuroCOLT 2001), Lecture Notes in Artificial Intelligence 2111, Springer, pp. 416–426 — DOI-10.1007/3-540-44581-1_27

Key takeaways

From the reading, NOTE-836.

  • The theorem (Theorem 1). X is any nonempty set, k a real positive-definite kernel, g strictly increasing, and c any cost on the training triples, coupled, non-convex or ∞-valued. Every minimizer of c + g(‖f‖) over the countable kernel expansions of finite norm (eq. 13) is f = Σ_{i=1}^m α_i k(·, x_i).
  • The proof (eqs. 18–24), checked. The component of f orthogonal to span{k(·, x_i)} does not change any f(x_j), by the reproducing property, and strictly raises g(‖f‖), by Pythagoras. Nothing else is used.
  • No Riesz. The RKHS is built from the kernel (§1.2), so ⟨k(·, x), f⟩ = f(x) holds by construction (eq. 11). The word “Riesz” does not occur in the paper.
  • Hypotheses. Strictness of g is what forces every minimizer into the span; with mere monotonicity some minimizer still is (p. 420, asserted). Convexity matters only for local minima of the reduced m-dimensional problem (eq. 30).
  • Not shown. Existence of a minimizer; that the class F is the whole completed RKHS; the semiparametric Theorem 2, which is stated without proof.
  • Examples. SV regression and classification, a strictly monotone regularizer taken from a risk bound, MAP estimation under a Gaussian process prior, and kernel PCA as a unit-variance hard constraint (eq. 31).

Standing in the record

Filed on 2026-09-26 while pursuing, at the owner’s request, the connection Riesz, reproducing kernels and spectral representation learning (see the curation entry of that date), with a skim (NOTE-230). Read in full on 2026-10-14 in NOTE-836, which supersedes the skim, and Active.

It is the record’s source for the representer theorem, filed as THEORY-314. THEORY-009 cites its proof as the parallel case in which the needed structure is had without the Riesz theorem; the reading confirms that citation. The skim proposed kernel PCA (Example 5) as the bridge to the spectral self-supervised results; the reading finds that eq. 31 constrains one unit-variance functional at a time, so the bridge to an eigenspace is not made here.

No instruction for machine-learning practice; nothing for the anthology.

Access when seeded: Read the full 11-page paper as published (Springer LNAI footer) from the author-hosted PDF at alex.smola.org/papers/2001/SchHerSmo01.pdf, via pymupdf text extraction: §1.1–1.2, Thm 1 and its proof, Thm 2, Remark 1, Examples 1–5, §3. Springer’s chapter page gives the DOI and pp. 416–426 and “First Online: 01 January 2001”, which looks like a placeholder. Crossref gives published-online 2001-09-13, used here. The COLT 2001 meeting itself was presumably earlier in 2001, but its date is unverified.

Backlinks