LIT-498: MaskGIT: Masked Generative Image Transformer

Chang, Zhang, Jiang, Liu and Freeman (2022) — ARXIV-2202.04200.

Key takeaways

• Raster order was never justified, and dropping it wins on both axes. MaskGIT’s own framing is that treating an image as a line-by-line sequence is “neither optimal nor efficient”. Bidirectional attention plus confidence-ranked parallel decoding beats the sequential model on quality and takes 8 steps instead of 256.
• The mask schedule is a real hyperparameter with a stated shape requirement. γ(r) must be continuous on [0,1], monotonically decreasing, with γ(0) → 1 and γ(1) → 0. Concave schedules (cosine, square, cubic, exponential) implement less-to-more information flow — few confident predictions early, many forced late — and convex ones (square root, logarithmic) the reverse. Cosine wins in all their experiments.
• BERT’s fixed 15% is explicitly wrong here and the paper says why: a decoder generating from scratch starts at 100% masked, so a constant ratio cannot cover the decoding trajectory.
• The tokenizer is untouched. This is a prior-and-decoding paper; every number it reports rides on a VQGAN tokenizer. That is why it appears as a generation baseline in later tokenizer papers rather than as a tokenizer one. The exception in this record is LIT-494, whose Table 1 lists MaskGIT’s tokenizer at rFID 2.28 on ImageNet 256×256 against GaussianToken’s 1.61 at 256 tokens — a reconstruction comparison in which MaskGIT stands in for its VQGAN tokenizer, and one confounded by the five unquantized floats each GaussianToken token also carries.

Standing in the anthology

Filed as the branch of the trunk that changes what you do with the tokens, against LIT-497 which changes how many of them there are and keeps the order. Both are descendants of LIT-496 and they disagree about the most basic question in the line — whether visual tokens should be generated left to right — with MaskGIT winning on speed by two orders of magnitude and LlamaGen later winning on FID while keeping raster order.

The record holds it at reading depth Skimmed equivalent: no NOTE is filed, because this contribution’s purpose was the tokenizer trunk and MaskGIT’s tokenizer is somebody else’s. Its decoding contribution deserves a reading of its own and has not had one.

Discrete Flow Matching (LIT-792) gives the decoding contribution its first controlled comparison in the record. Its Prop. 6 shows MaskGIT is the masked case of discrete flow matching with a confidence-ordered sampler, and its authors retrain MaskGIT on discrete CIFAR-10 with a temperature search. Confidence-ordered unmasking wins at low NFE and loses at high NFE against their cubic scheduler with a corrector, which reaches FID 3.63 at 1,024 NFE (its Fig. 3).