Source
Kynkäänniemi et al. (2022), LIT-563. Read as NOTE-304.
The practice
If an ImageNet classifier touches the pipeline at all, FID and KID are no longer trustworthy for comparing the result with methods that do not use one. The demonstrated case is a pretrained discriminator during training. The authors suspect the same of classifiers used to curate the data or placed in the sampling loop — and provenance is not what matters. A classifier trained from scratch on ImageNet labels has the same feature space as one downloaded; the hazard is the overlap with FID’s features, not where the weights came from.
- Report a Fréchet distance in a feature space not trained on ImageNet classification, such as CLIP. SwAV is partly independent: self-supervised, but on ImageNet images
- Treat a large FID gain that the other space does not show as suspect, and look at samples, or better, run a human comparison
- Do not take FID’s ranking across very different training setups at face value. Within one setup (sweeps, checkpoints, catching failed runs) the authors find it dependable
What was measured
The practice is Kynkäänniemi et al.’s, LIT-563: having shown that Inception-V3’s pool3 features are one affine map from ImageNet logits, they recommend checking FID gains in a non-ImageNet space. Their resampling experiment holds a StyleGAN2 generator fixed and changes only the selection of its samples; across four datasets FID fell by 63–72% while CLIP-FD fell by 4–11%. On FFHQ:
| FFHQ | FID | ResNet-50 FD | SwAV FD | CLIP FD |
|---|---|---|---|---|
| random samples | 5.30 | 6.11 | 1.42 | 2.76 |
| resampled to match fringe features | 1.78 | 3.85 | 1.24 | 2.64 |
The drop tracks how much ImageNet each feature space has seen. In the practical case, Projected FastGAN matched StyleGAN2’s FID (5.28 against 5.30) with CLIP-FD 4.67 against 2.76, and human raters preferred StyleGAN2.
The same pattern continues in diffusion distillation. DMAD (LIT-770) takes its best one-step ImageNet-64 FID from 1.24 to 1.04 by adding a projected discriminator on frozen ImageNet-pretrained VGG16-BN and EfficientNet-Lite0 features, and reports that gain in FID alone, with no non-ImageNet Fréchet distance and no precision or recall. It is an unchecked instance of exactly the case this practice is about, not evidence for or against it.
Conditions
- CLIP is a partial check. It has its own training data and biases
- Complements SOTA-307, which is about FID’s variance across seeds. This practice is about its bias
- Says nothing about which side of a fidelity/diversity trade you are on. That is SOTA-425, and precision and recall are subject to this practice too — not because they share FID’s feature space, which they do not (VGG-16 fc2 against Inception-v3, LIT-703), but because they are a second ImageNet classifier, and their own paper reports that Inception features give “substantially similar” results. Two ImageNet classifiers agreeing is not a cross-check
The sampling-loop case
Classifier guidance (Dhariwal and Nichol 2021, LIT-699) is this practice’s sharpest instance and the reason v2 widened the title. The method takes gradient steps that raise an ImageNet classifier’s log-probability of the target class, during sampling, on every step. FID and Inception Score are computed by passing the results through an ImageNet classifier. This is not a classifier that happens to share a feature space with the metric; it is a procedure whose objective is the thing the metric measures.
The paper addresses it in one clause of its introduction — the scale can be raised “by an order of magnitude without obtaining adversarial examples” — and the word “adversarial” appears nowhere else in it. No test, no held-out classifier, no non-Inception distance. Its memorization check (App. C) is run in InceptionV3 feature space, which is the same problem again.
So for guided sampling this practice is unmet, not satisfied: **nobody has run the CLIP or SwAV Fréchet distance on guided against unguided samples at matched
FID.** THEORY-109 is the open question, and this practice already names the instrument that would close it