Sun, Jiang, Chen, Zhang, Peng, Luo and Yuan (2024) — ARXIV-2406.06525.
Key takeaways
-
The codebook ablation is the part this record files. At downsample 16, codebook size 16384, on ImageNet 50k validation:
code dim rFID PSNR SSIM usage 256 9.21 18.32 0.575 0.29% 32 3.22 19.98 0.646 20.9% 8 2.19 20.79 0.675 97.0% 4 9.88 19.39 0.593 82.0% A 256-dimensional codebook uses three codes in a thousand. Shrinking the code is what makes the codebook exist at all.
-
It is not monotone, and the caption says it is. “Lower vector dimension (from 256 to 8) improves both reconstruction quality and codebook usage significantly” is true over the range it names and the table’s next row, dimension 4, reverses it — 9.88 rFID, worse than dimension 256. The codebook-size caption is bounded the same way: “from 4096 to 16384”, where 32768 is worse (2.26 vs 2.19) and usage runs 100% / 75% / 97% / 85%, which is not monotone either.
-
rFID is meaningless without the token count, and this paper is where that becomes measurable. Same tokenizer, downsample 16: 256 tokens → 2.19, 576 tokens → 0.94, 1024 tokens → 0.70. At downsample 8: 1024 → 0.59, 2304 → 0.37, 4096 → 0.39. A three-fold range from a knob that is not the tokenizer.
-
The abstract quotes the 576-token row. “An image tokenizer with downsample ratio of 16, reconstruction quality of 0.94 rFID” is true and will be read as
256×256 → 16×16, which is 2.19. The 0.94 comes from encoding a384×384input to 576 tokens and resizing the reconstruction back to 256 for scoring. §4’s text gives the same pair as 2.43 and 0.99, matching neither the table nor the abstract. -
Generation: 2.18 FID on ImageNet
256×256at 3.1B parameters, beating LDM and DiT with no visual inductive bias, and 326–414% inference speedup from standard LLM serving frameworks. -
Entropy loss is deliberately omitted from codebook learning “for simplicity”, unlike MaskGIT and the MAGVIT line. Worth knowing when the utilization numbers are compared across papers.
Standing in the anthology
The measurement half of the tokenizer trunk, and the source for both practices filed in this contribution: SOTA-306 on code dimension and utilization, and SOTA-305 on what a reconstruction FID has to state before it means anything.
It also settles a question LIT-494 raised and could not answer. GaussianToken compares itself to “LlamaGen 2.19” at 256 tokens, which is the honest row — and then adds five continuous parameters per token, so the rate is unmatched in the other direction. Having this paper in the record is what lets that be stated as a fact about two documents rather than an observation about one.
Everything but the codebook and the prior is LIT-496’s. §2.2 uses “the same
architecture as VQGAN”, a convolutional encoder-quantizer-decoder, trained
with VQGAN’s objective — pixel ℓ₂, an LPIPS perceptual loss and a PatchGAN
adversarial term — and it keeps raster-order autoregressive generation over
the token grid. One departure in the loss: the adversarial weight is a fixed
0.5 rather than VQGAN’s adaptive gradient-ratio λ. Its Table 4 puts VQGAN
at downsample 16 and code dimension 256 at rFID 8.30 with 1024 codes and 4.99
with 16384, against its own 2.19.
The design it credits — ℓ₂-normalized codes, low dimension, large codebook
— it attributes to Yu et al. (2021), now filed as LIT-500
and declared here as a second extends. Reading the origin showed this paper
reimplements the idea rather than reproducing it: ViT-VQGAN factorizes,
keeping a wide embedding behind a narrow lookup, where this makes the whole
codebook 8-dimensional. Both work; nobody has compared them.
Semanticist’s PCA-structured tokenizer (LIT-751) measures itself against LlamaGen: its VQ tokenizer (2.19 rFID) and LlamaGen-L (3.80 gFID on 256 tokens) are Table 1 rows, quoted rather than retrained, and LlamaGen’s VQ latent is the one that paper decomposes to show semantic content and spectrum coupled. Its own generator is built from LlamaGen.
STARFlow (LIT-785) retrains a 1.4B LlamaGen on 200M samples at batch 512 as one of two near-matched baselines for its latent normalizing flow, also 1.4B. It plots its own 50K-sample FID below LlamaGen’s at every checkpoint (Fig. 10a). The values are not tabled, and the figure does not say where on its own 400M-image schedule STARFlow’s points sit.