Guo et al., CUHK, Shanghai AI Laboratory and Stanford (2023) — ARXIV-2307.04725
Key takeaways
- Freeze the image model and train only motion. Temporal-attention modules with zero-initialized residual outputs are inserted into frozen SD 1.5 and trained on WebVid-10M (§5). Because the spatial weights never change, the modules can be dropped into any checkpoint fine-tuned from the same base, such as DreamBooth (LIT-079) or LoRA (LIT-046) personalizations. That compatibility is the point of the design. The base is Stable Diffusion, the latent diffusion model of LIT-062, taken unchanged: its autoencoder and U-Net are kept, the U-Net’s image layers run frame by frame over SD’s per-frame 2D latents with no temporal compression, and what the paper adds is the temporal layers between them.
- A disposable domain adapter. A LoRA on the spatial layers is trained on video frames to absorb WebVid’s defects (watermarks, blur) and dropped at inference. Lowering its scale improves quality and removes watermarks (Fig. 6, qualitative).
- MotionLoRA. Rank-2 LoRAs on the motion modules learn a camera motion from 20–50 clips in 2,000 iterations. Rank 2 is comparable to rank 128 (Fig. 7, qualitative).
- Transfer is measured once. Against Text2Video-Zero and Tune-a-Video (Table 1), AnimateDiff ranks best on smoothness (2.825) and ties on text alignment (2.210 against Tune-a-Video’s 2.180). Users ranked Text2Video-Zero higher on domain similarity (2.620 against 2.280), though CLIP puts AnimateDiff first there (87.29). It has the lowest CLIP text score of the three (31.39 against 35.98).
Where the hedges are
Per DP-010:
- Freezing is never compared against fine-tuning. Neither version tests whether freezing the image model costs quality relative to fine-tuning it. The design’s premise is compatibility, not quality, and the paper does not measure what compatibility costs.
- The limitation was removed. Version 1 reported failure “when the domain of the personalized T2I model is far from realistic, e.g., 2D Disney cartoon” (v1 §5). Version 2 drops the limitations section, while its abstract claims transfer to “any personalized T2Is originating from the same base”.
- The claim widened with no new evidence. The v1 abstract claimed transfer to “most” personalized models, and the v2 abstract to “any”. No new evidence on the stylized case was added.
- The rest of the evidence is in the supplement. User-study size, the convolution-against-attention ablation and training configurations are in supplementary material that is not in the arXiv PDF.
Standing in the anthology
This is the adaptation end of SOTA-386. The image model is kept entirely and never updated. It supports the practice’s first clause by adoption only, and says nothing about the second: no images are mixed in during motion training.
It adds a reason to freeze that the practice does not mention. A frozen base keeps the ecosystem of personalized checkpoints usable. Emu Video (LIT-635) freezes too, and its controlled result is that freezing beats full fine-tuning only narrowly (55/58). Together they make freezing a reasonable default when checkpoint compatibility matters. Neither shows that it wins on quality.
Read in full as NOTE-339.