Contribution
A demonstration, with a mechanism, that FID contains a large perceptual null space tied to ImageNet classes, and a practical case where a training method falls into it.
Key insight
FID’s features are ImageNet class evidence. The pool3 features are one affine map from the logits. So FID rewards matching the distribution of whatever ImageNet classes the images happen to trigger, whether or not those matter for the dataset.
Assumptions
- Standard FID: Inception-V3 pool3, 50k generated against the training set
- StyleGAN2 generators on FFHQ, LSUN Cat, Car and Places, and AFHQ Dog
- Alternative spaces: ResNet-50 (ImageNet), SwAV (self-supervised on ImageNet), CLIP ViT-B/32
Key results
- Table 1 (Top-1 matching): FID −8.5% to −11.3%. CLIP-FD −0.3% to −2.1%
- Table 2 (fringe-feature matching): FID −63% to −72% on four datasets. CLIP-FD −4% to −11%
- §4: Projected FastGAN against StyleGAN2 on FFHQ: FID 5.28 / 5.30, recall 0.45 / 0.46, CLIP-FD 4.67 / 2.76. Humans prefer StyleGAN2
Claims
| id | claim | strength | support |
|---|---|---|---|
| C1 | FID can be substantially lowered without perceptible change | strong | resampling experiments, five datasets, alternative spaces as controls |
| C2 | The null space is tied to ImageNet classes | strong | the size of the drop tracks how much ImageNet each feature space saw |
| C3 | ImageNet-pretrained discriminators exploit it | moderate | one model on one dataset, with human evaluation |
| C4 | FID is reliable within a setup | weak | the authors’ judgment |
Connections
It contests Projected GAN (LIT-562). It complements the FID Lottery (LIT-501), which is about seed variance.
Recommendations
- R1 — When ImageNet-pretrained networks take part in training, confirm FID gains in a non-ImageNet feature space. Filed as SOTA-337
Bearing on the record
Limitations
- CLIP is not neutral either. It is trained on web image-text data, and the authors propose it as a partial check, not a fix
- One worked example of the accidental case
Open questions
- Which feature space, if any, is free of the training data of the models being compared