nucleation

CLAIM-149: A probe family that supplies a transformation's training signal cannot also certify what the transformation preserves: preservation must be shown under probes that were not used to train it

3 min read

Properties10
StatusActive
Version2
Rolethesis
Warrantinterpretive
Tagsrepresentation-learning, model-comparison
Linepragmatic-transport
Rests onCLAIM-011 — Intertwining with an uninformative correspondence is vacuous or lossy, so the source–target correspondence must be anchored to independently measured distinctions and evaluated on tasks other than those it was estimated on, CLAIM-266 — A family of probes that supplies an optimized system's training signal cannot also certify what the system preserves: optimization pushes toward whatever the probes reward, including what they cannot see, so preservation must be shown under probes not used to train it
GroundsCASE-041 — FID and Projected GANs: one ImageNet-trained probe family used both to train a generator and to evaluate it
ComplementsCLAIM-142 — Two communicative realizations can agree on every probe in a restricted family without being structurally identical, so observational equivalence is relative to the probes, and only independently justified probes evaluated on held-out cases are evidence that structure was preserved
Illustrated byCASE-071 — Kovacs, Deutsch and Freitag's metric bias in MBR decoding: translations a metric selects, that the metric and its neighbours rate better and human raters do not

Graph View

The claim

A184 §4, Experiment C: “Evaluate generated samples with independent relational probes and human judgments that were not used to train the generator … success under the training probes should not be taken as sufficient evidence of preservation under independent observations”. A184 §6: “What a system learns to preserve is constrained by what its training process is able to distinguish; what we can verify it has preserved is constrained by the observations used to evaluate it.” A203 §30: “FID measures resemblance through probes. Projected GANs trains through probes.” A218’s Study 6: “Test whether matching projected feature distributions guarantees preservation of independently measured relationships”.

The reason is CLAIM-011’s, one step further. A correspondence estimated on some tasks must be evaluated on others. A transformation trained to satisfy some probes must be evaluated on others, because training pushes it toward whatever the probes reward, including what they cannot see (probe-relative equivalence, CLAIM-142). The documented case (CASE-041) is worse than blindness. A Projected FastGAN, trained against ImageNet features, matches StyleGAN2’s FID, which is also computed in ImageNet features, while doing worse in CLIP space and with human judges.

For the manuscript the application is direct. A translation or reconstruction model tuned against a pragmatic classifier, or against an LLM judge, cannot be certified by that classifier or judge. Its fidelity must be shown under probes it was not tuned on. CLAIM-085 makes the neighbouring point for generation and judging with one model: that measures the model’s preferences.

What it does not say

  • It does not say metrics in the training probes are worthless. It says they are not a certificate.
  • It does not say a generator trained against a probe family will introduce the differences the family cannot see. A184 §3: “This does not prove the generator will make that error”.
  • It does not say any held-out probe is neutral. Held-out probes have their own training data; in the case, CLIP “is not neutral either” (ANTH-NOTE-304).
  • It does not settle QUESTION-005. It names one way the operationalization can build its verdict into the measurement.
  • The documented support is one training case, in image generation (CASE-041), and one selection case, in translation (CASE-071).

Note of 2026-10-13: the general content is filed separately

Under ADR-040, the point that training probes cannot certify is filed on its own as CLAIM-266, in the formal-apparatus line, resting on no claim. This claim now rests on it for the general content, and keeps CLAIM-011, for carrying the remedy from correspondences to training objectives. Claims in other lines that need only the general content cite CLAIM-266, so setting pragmatic-transport aside does not take it from them.

Backlinks