Zhang et al. (2023) — ARXIV-2302.05543
Key takeaways
- Adds spatial control — edges, depth, segmentation, pose — to a locked pretrained diffusion model, reusing its encoding layers as a backbone. The trainable branch is a clone of those layers, so the adapter starts already knowing how to encode images
- Zero convolutions are the design: the connections between branch and frozen model are convolutions initialised to zero, so the adapter is an exact no-op at initialisation and its parameters “progressively grow from zero”, ensuring “no harmful noise could affect the finetuning”. Zero is not small — it is exactly neutral, so the guarantee is structural rather than probabilistic
- Works singly and in combination, with or without text prompts
- Robust at both ends of the data range — under 50k and over 1m examples. Adapter methods usually have a small-data floor; this reports not having one
Standing in the anthology
Read — NOTE-071. Retagged to adaptation-and-tuning: the generative model
is frozen and untouched.
The takeaway “zero-shot conditioning” was wrong. ControlNet trains an adapter per condition type; nothing about it is zero-shot. The word that belongs there is zero-initialized, which is a different thing and is the actual mechanism.
Nothing is sourced here, but the mechanism has reach the record does not
capture: initialise an added branch’s output to zero so it is an exact no-op
at the start of training. That is here, in LoRA’s zero-initialised B, and in
zero-init residual branches — three independent places, one idea, and
SOTA-052 carries only the scaling version of the same argument, not the limit.
Audited by #121, and now a source. SOTA-051 had been arguing
from this paper’s zero convolutions for the distinction its own title blurs —
zero is not “near zero” — without naming it in source:. Under ADR-017’s
retraction test that section does not survive the paper’s removal, so the link
existed in prose and not in data. R2 and R3 remain unfiled.