Source
Rombach et al. (2022), LIT-062 — ARXIV-2112.10752, CVPR 2022.
Ho et al. (2020), LIT-036 — ARXIV-2006.11239. Its §4.3 measures this practice’s premise — that most of a pixel-space model’s codelength describes imperceptible detail — two years before Rombach et al. act on it as an assumption.
Gu et al. (2025), LIT-785 — STARFlow is evidence from outside diffusion. The same deep-shallow normalizing flow moved from pixels to SD-VAE latents goes from ImageNet-256 FID 4.69 to 2.40 (its Table 1). The input space, the patch size and the decoder change together, and the latent model decodes through a GAN-fine-tuned decoder, so the size of the gain is confounded. The direction is the one this practice predicts, for a likelihood model that is not a diffusion model.
Split the problem in two. First train an autoencoder that compresses the signal to a lower-dimensional latent, keeping what a human would notice and discarding what they would not. Then train the generative model in that latent space and decode at the end.
The argument for why this is nearly free is the useful part. Likelihood-based generative models spend most of their capacity on imperceptible high-frequency detail — bits that cost real compute and that no evaluator rewards. Moving to a perceptually-equivalent compressed space removes that spend before the expensive model ever sees the data, so the generative model works in a space whose dimensions it actually needs.
The recommendation comes from LIT-062, which proposed the split for diffusion: an autoencoder with a perceptual objective is trained once and frozen, the diffusion model is trained on its latents with conditioning introduced into the latent-space denoiser, and the result was substantially cheaper training and sampling at a given resolution, making high-resolution synthesis practical. The same paper is where the condition comes from that the latent must be perceptually equivalent, since whatever the autoencoder discards the diffusion model can never recover.
The transferable claim is a compute-allocation one, and it is why this is filed by the kind of claim rather than as a diffusion technique: when the expensive model’s input carries structure a cheap model can strip, strip it first and let the expensive model work on what is left.
(This sentence named model-architecture until v3, from before the practice
was retagged representation-and-encoding — “how the signal is encoded before
the expensive network sees it … tokenizers and learned latents”, which is this
exactly. The argument is unchanged and the topic is now the better fit; the
prose had simply not followed the tag.) The two-stage
shape — learn a compact representation, then model that — recurs well
beyond images.
Conditions
The compression ratio is a real trade rather than a free win: too aggressive and the autoencoder discards detail the task needed, and no amount of generative capacity downstream recovers it. The paper’s regularised autoencoders and their choice of downsampling factor are a tuned answer for images at the scale it reports.
It also introduces a dependency the single-stage version does not have. The generative model can only be as good as the decoder, and errors made in the first stage are invisible to the second stage’s loss. STARFlow (LIT-785, App. B.3) shows the ceiling plainly: reconstructing 50K real images through its noisy-latent decoder gives rFID about 2.73, worse than the 2.40 its generator reaches. At that resolution the decoder, not the flow, sets the FID.
Marked universal for the domain it was shown in: latent-space training is
what essentially every deployed image and video generator does, and doing it
at full resolution is what would need justifying.
Why this is an encoding claim rather than an architecture one
Filed under representation-and-encoding alongside BPE tokenization, and the
pairing is the argument: both fit a cheap encoder that compresses the signal,
then train the expensive model on what comes out. One discrete, one
continuous. A latent autoencoder is a tokenizer that did not round.
That is also the line that keeps the topic from becoming a bag of anything with “latent” in it. SOTA-147 compresses internal state at serving time and SOTA-180 compresses weights; neither touches how the input signal is encoded, and both stay where they are.
Known implementations
- Stable Diffusion and the open image and video generation line generally