Ruiz et al. (2022) — ARXIV-2208.12242
Key takeaways
- Binds a rare token to a subject from ~3–5 images by fine-tuning the whole model, so the identifier then composes with ordinary prompts
- The interesting part is the regulariser. Narrow fine-tuning causes language drift and diversity collapse, and the fix is to supervise the model with its own generated samples: before fine-tuning, sample the frozen model on “a [class noun]” and include those in the objective as a class-specific prior-preservation loss. No external data — the model before you touched it is the regulariser
- Beats textual inversion on both subject fidelity (DINO, CLIP-I) and prompt fidelity (CLIP-T), by sizeable margins
- The base model caps the method: DreamBooth on Imagen beats DreamBooth on Stable Diffusion, “approaching the upper bound of subject fidelity for real images”
- The super-resolution stages must be fine-tuned too, or they hallucinate high-frequency detail they have not seen for this subject — 64→256 is “essential for most subjects”. A pipeline-level finding a personalization-is-a-loss-function framing would miss
Standing in the anthology
Read — NOTE-036. Retagged to adaptation-and-tuning. Nothing is sourced
here.
The transferable finding is one the record holds only in one domain: when narrow fine-tuning risks destroying a general capability, regularise against the pre-fine-tuning model’s own samples. The post-training practices treat KL-to-reference as an RL mechanism; prior preservation is the same instrument in supervised fine-tuning, invented independently, for the same reason. Stating it once, generally, would cover both.