LIT-611: GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium

Heusel et al. (2017) — ARXIV-1706.08500

Key takeaways

• FID is defined here, in a paper about something else. The title, abstract and three-quarters of the pages are about the two time-scale update rule (TTUR) — a separate learning rate for discriminator and generator, with a stochastic-approximation proof of convergence to a stationary local Nash equilibrium, plus an argument that Adam under TTUR behaves as a heavy ball with friction and so prefers flat minima. FID arrives because the authors needed to evaluate those experiments.

• The definition, with its assumption stated. Take Inception coding units for real and generated images; “we assume the coding units to follow a multidimensional Gaussian”; the distance between the two Gaussians is the Fréchet distance, “also known as Wasserstein-2”:

  d²((m,C),(m_w,C_w)) = ‖m − m_w‖²₂ + Tr(C + C_w − 2(C·C_w)^{1/2})
  

  The Gaussian fit is an assumption the paper makes explicitly and that almost nothing downstream restates.

• The validation is monotonicity under synthetic corruption. Figure 3 shows FID rising monotonically with the level of Gaussian noise, Gaussian blur, implanted black rectangles, swirled images, salt-and-pepper noise, and CelebA contaminated with ImageNet images. That is the evidence offered: the measure orders deliberately degraded distributions correctly. It is not evidence that small differences between two competent models are meaningful, and the paper does not claim it is.

• Its stated advantage is over the Inception Score, which it “captures the similarity of generated images to real ones better than”. A comparison against one incumbent, not a general claim of fitness. FID is built as a repair of LIT-614’s Inception Score. It keeps the ImageNet-trained Inception network, but the score looks only at generated images, and the paper names its drawback as “the statistics of real world samples are not used and compared to the statistics of synthetic samples”. FID compares the coding-layer statistics of generated and real images.

• The monotonicity comparison did not survive a rerun intact. In Appendix A1 (Fig. A8) FID rises with every disturbance, while the Inception Score “fluctuates, stays flat or even, in the worst case, decreases”. Bińkowski et al. (LIT-615) reran those disturbance experiments on the standard CelebA train–test split and on CIFAR-10. They found the Inception Score monotonic on more of the disturbance types than Heusel et al. had, with KID tracking FID, and concluded the property “may not be very robust to small changes in evaluation methods”.

Standing in the anthology

Unit H of #304, filed as the largest substrate defect this audit found: 68 documents in this record mention FID and none held the paper. For scale, LIT-587 (ViT) had 47 and LIT-588 (CLIP) 45.

And the dependency was not background. The record already held two papers about this metric’s failure modes and two Active practices telling readers how to survive them — LIT-563 on FID being movable without improving images, LIT-501 on its ≈1.3% seed-noise floor, SOTA-337 on when its gains cannot be trusted, SOTA-307 on reporting it as an error bar — plus

SOTA-338 turning on it. Four documents deep into the critique, and no note for the thing being criticised. The shape unit E found with MAE, at four times the scale and three levels further along.

Reading it supplies two things the critiques assume without restating: the Gaussian fit, and the fact that the validation was ordering corrupted distributions rather than discriminating good ones. Both are load-bearing for LIT-501, which measures how much of a reported FID is noise, and for LIT-563, which shows it can be moved without improving images.

Measured against human preference on text-to-image, it ran backwards. Pick-a-Pic (LIT-782) measured FID as a ranking of text-to-image models against expert preference on 100 MS-COCO captions and found it anti-correlated, Spearman −0.900 across nine model–guidance settings, because people prefer the higher guidance that moves samples away from MS-COCO’s statistics; its prompt-aware PickScore correlated 0.917. Nine settings and colleague annotators make it a small study, and it ranks whole configurations rather than images.

Filed under analysis-and-evaluation rather than generative-modeling as primary because what the record needs from this paper is the metric, not the TTUR result — and ADR-046 says to tag the subject the document is for.