anthology-of-the-sota

SOTA-338: Train the GAN discriminator on frozen multi-scale pretrained features with fixed random channel and scale mixing

2 min read

Properties7
StatusProposed
Version2
Consensuscontested
Tagsgenerative-modeling, training-optimization, vision-and-graphics
SourceLIT-562 — Projected GANs Converge Faster
Introduced byLIT-562 — Projected GANs Converge Faster
Contested byLIT-563 — The Role of ImageNet Classes in Fréchet Inception Distance

Graph View

Source

Sauer et al. (2021), LIT-562 — Projected GAN. Read as NOTE-303. Contested by LIT-563.

The practice

To make a GAN converge in a fraction of the usual training:

  • Project both real and generated images through a frozen pretrained network and take features at four scales (64² down to 8² in the paper)
  • Mix them with fixed random convolutions: 1×1 across channels (CCM) and 3×3 with upsampling across scales (CSM), Kaiming-initialized and never trained. Without mixing, the discriminator uses only part of the deeper features
  • One small discriminator per scale, with spectral normalization and logits at a common resolution, summed
  • Prefer a compact feature network. EfficientNet-Lite1 beat larger ResNets and ViTs
  • Keep differentiable augmentation, reported as required for the best results

The design is Sauer et al.’s Projected GAN, LIT-562, which proposed it. With a FastGAN generator it passed StyleGAN2’s best LSUN-Church FID after 1.1M images against 88M (Figure 4), and reported lower FIDs than StyleGAN2-ADA on large datasets, 3.39 against 7.32 on FFHQ (Table 3). Its feature-network ablation on Church (Table 2) is the basis for preferring a compact network: EfficientNet-Lite1 reached 1.65 against 4.40 for ResNet-50 and 12.38 for a ViT. All of these are FID.

What is contested

Every number behind this practice is FID. The discriminator is itself ImageNet-pretrained, and FID’s feature space is so close to ImageNet classes that matching their statistics lowers FID without improving images (LIT-563). On FFHQ at equal FID, this model loses to StyleGAN2 in CLIP space and with human raters. The speed-up in reaching a given FID is real. Whether it is a speed-up in quality is what is disputed. Evaluate with SOTA-337 before relying on it.

Adoption has not settled it. DMAD (LIT-770) uses a projected discriminator on frozen VGG16-BN and EfficientNet-Lite0 features in one-step diffusion distillation, and its 1.04 ImageNet-64 FID, against 1.24 without it, is the same kind of evidence: FID only, comparable only to a baseline that copies the same discriminator.

Conditions

  • FastGAN generator, 256² to 1024²
  • ImageNet-pretrained feature network. Randomly initialized feature networks do much worse (per the appendix, not read)

Backlinks