Source
Bińkowski, Sutherland, Arbel and Gretton (2018), LIT-615. The account of why is THEORY-095.
When this applies
You are comparing two generative models, or one model against a baseline number from a paper, using a Fréchet distance in some feature space — the Inception FID and its CLIP, SwAV and DINOv2 relatives alike, since all of them use the same plug-in recipe.
Do this
1. State the sample count, and match it. FID estimates computed at
different n are not comparable, and the difference is not a rounding
concern: estimating the distance between the CIFAR-10 train and test sets,
where the true value is 0, the estimate is still about 8.1 at n=10,000 —
the whole CIFAR test set — and still falling. A number copied from a paper
that used a different n is not a baseline.
2. Do not read a small standard deviation as a converged estimate. The bias is systematic, so repeating the measurement does not expose it; the error bar reports the spread of the wrong quantity. In the paper’s construction the reported standard deviations were 0.2 and 0.5 on a gap of 2.5, and the ordering was still wrong in every one of 100 trials.
3. When the gap is small, settle it with an unbiased estimator. KID —
the squared MMD in the same Inception features, under the cubic kernel
k(x,y) = ((1/d)xᵀy + 1)³ — has a simple unbiased estimator and is
asymptotically normal, so a confidence interval on it means what it looks
like. Compute it as a small-n estimate averaged over several draws; the
MMD estimator is O(n²d) where FID’s is O(nd² + d³), which at d=2048 makes
KID the cheaper of the two.
4. Or raise n until the ordering stops moving. The estimator is
consistent, so the bias does go away. In their d=2048 example it had gone by
100,000 samples. This is the option that requires no new tooling and it is a
real one — it just has to be checked rather than assumed.
All four steps come from LIT-615. Bińkowski et al. measured the CIFAR-10
train-against-test curve, where KID’s estimate is essentially 0 by n=2,000
and FID’s is still about 8.1 at n=10,000, built the d=2048 reversal in
Appendix D.2, and proved in Appendix D.3 that no estimator of FID is unbiased
for all distributions. Comparing FID only at equal n, and using KID as a
small-n estimate averaged over several draws, are their recommendations.
Why this and not more error bars
Because they answer different questions, and the record already recommends
the other one. SOTA-307 says to report FID as an error bar over several
training seeds and treat a gap below about 2% of the mean as inconclusive.
That is the right response to seed noise, measured by LIT-501 at ≈1.3%.
It does not respond to this at all: seed variance is spread around the
estimator’s own mean at fixed n, and the bias is an offset of that mean
which every seed shares.
The two terms add, and only one of them shrinks when you average runs.
Limitations
- Half the argument is one group’s construction. The distributions in Appendix D.2 are ReLU-censored normals chosen to resemble Inception codes, not two real generators. Nobody has produced the equivalent reversal between models anyone has trained, and it would be worth having.
- The non-existence result needs a mixture family. It rules out an estimator unbiased on any class containing two-component Gaussian mixtures, and says nothing about a class of exactly normal distributions. The bridge to practice is that Inception codes are visibly not normal — about 2% of components are exactly zero — which is the same observation that undermines FID’s Gaussian fit.
- KID is not a free upgrade in what it measures. It answers the estimator problem and inherits the feature-space problem untouched: a KID in Inception features carries LIT-563’s ImageNet-class dependence exactly as FID does, which is why SOTA-338 says FID or KID alone cannot settle its question.