anthology-of-the-sota

LIT-621: Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models

4 min read

Properties9
StatusActive
Version2
Tagsgenerative-modeling, vision-and-graphics, adaptation-and-tuning
ExtendsLIT-062 — High-Resolution Image Synthesis with Latent Diffusion Models
Extended byLIT-625 — Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets
Compared againstLIT-635 — Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioning, LIT-661 — Photorealistic Video Generation with Diffusion Models
Cited as “introduced by” bySOTA-393 — Freeze the image model's spatial layers when adding time if checkpoint compatibility matters; do not expect a quality gain from it, SOTA-396 — When a latent video model reuses an image autoencoder or upsampler, fine-tune them on video so they see time
Cited as “paper” byNOTE-341 — Video LDM (Align your Latents)
Cited as “source” bySOTA-386 — Show a video diffusion model images before and alongside video, SOTA-396 — When a latent video model reuses an image autoencoder or upsampler, fine-tune them on video so they see time

Graph View

Blattmann et al., LMU Munich, NVIDIA, Vector Institute, Toronto, Waterloo (2023) — ARXIV-2304.08818

Key takeaways

  • The method. Temporal layers (3D-convolution residual blocks and temporal attention) are interleaved into a pretrained image LDM, blended in with a learned α, and trained on video (§3.1). What it extends is the latent diffusion model of LIT-062: the image LDM supplies the spatial layers and the autoencoder, and only the decoder is given temporal layers, so the encoder and the latent space stay those of the image model (§3.1.1). The driving model starts from an LDM trained on driving frames, and the text-to-video model from Stable Diffusion checkpoints.
  • Image pretraining is the controlled result. On driving scenes, against the same design trained end to end with no image pretraining, FVD is 534 against 1155 and FID 48.3 against 71.3 (Table 1). Attention-only temporal layers, matched on trainable parameters, reach 704. The pretraining images are 1 fps frames of the same driving videos, and the end-to-end arm gets no equivalent of the 73K image-pretraining steps (App. H.1, Table 7).
  • The autoencoder has to learn video too. Fine-tuning the decoder on video with a video discriminator cuts reconstruction FVD from 390.88 to 32.94 on driving scenes (Table 3), and on WebVid from 35.82 to 18.66 (Table 11). Reconstruction FID gets slightly worse on driving (7.61 → 9.17) and slightly better on WebVid (13.89 → 11.68) and mountain biking (20.76 → 18.65). Adding an image discriminator back raises FVD to 51.01.
  • Upsampling must be temporal as well. A video-fine-tuned upsampler reaches FVD 45.39 against 165.98 for frame-wise upsampling, with FID unchanged (Table 3).
  • Text-to-video is built on Stable Diffusion 2.1. It ties Make-A-Video on UCF-101 IS (33.45 against 33.00), is worse on FVD (550.61 against 367.23), and is worse on MSR-VTT CLIPSIM (0.2929 against 0.3049; Tables 4–5).
  • Later systems measure themselves against it. W.A.L.T (LIT-661, Table 5) lists this model’s zero-shot UCF-101 result at IS 33.5 and FVD 550.6, against 35.1 and 258.1 for its 3B model, and sets its own causal image-and-video encoder against frame-level autoencoders like this one, which it says flicker. That charge passes over this paper’s video-fine-tuned decoder. Emu Video (LIT-635, Fig. 2) is preferred on quality in 92.3% of comparisons with this model, judged on the samples this paper’s authors released for their own prompts, not on a shared prompt set.

Where the hedges are

Per DP-010:

  • “Only need to train a temporal alignment model” (abstract) is not what the text-to-video model did. App. H.2 says the spatial layers were first fine-tuned on WebVid, which is “necessary to prevent out-of-distribution problems”. The interpolation models trained all parameters, and the decoder was fine-tuned. The paper counts 2.2B of 3.1B as trained (App. H.2.1): the 656M temporal layers and the 1,509M interpolation model. That count leaves out the 865M spatial layers, fine-tuned on WebVid frames first, and the fine-tuned decoder. Only the encoder and text encoder were never updated.
  • “Up to 1280×2048” comes from Stable Diffusion’s 4× upscaler, trained on 320×320 crops and applied convolutionally.
  • The samples and the metrics come from different models. Most sample figures are from the SD 1.4 model, while the metric tables use SD 2.1. The SD 1.4 model scored worse (FVD 656.49; Tables 9–10).
  • The key ablation is at driving-scene scale on a smaller model, and the pixel-space baseline in it is not matched in channels or steps (Table 7).

Standing in the anthology

This is where the line moves into latent space, and the template later open video models start from: initialize from an image model, add time, and teach the autoencoder about video. Stable Video Diffusion comes from the same authors and extends this architecture directly. It is also the clearest evidence behind SOTA-187 for video: the “Pixel-space baseline” row is a controlled pixel-vs-latent comparison, though an unmatched one.

The abstract describes a lighter method than the one that produced the headline model. The record should cite the ablation, not the abstract.

Read in full as NOTE-341.

Backlinks