Source
Zhong et al. (2024), TMLR, LIT-1529, read in full in NOTE-1160. §4.3.2, Fig. 5 and Table 3.
What was actually shown
LoRA Switch activates LoRA i = ⌊((t − 1) mod kτ)/τ⌋ + 1 at step t, so over whole cycles each of k LoRAs acts on 1/k of the steps whatever τ is.
- Interval (Fig. 5(a), Table 3). Composition win rate against Merge, read off the plot: about 38% at τ = 1, rising to about 59% at τ = 5; Table 3 gives 55, 57, 59, 58 at τ = 3–6, and image-quality win rates 29, 33, 34, 33. τ = 7 on the plot is about 60%. In the authors’ words, “overly frequent switching, such as changing LoRAs at every denoising step, leads to distortions in generated images”.
- Dynamic intervals (Table 3). τ rising 3 → 5 over sampling: 57 and 33. Warm-up (rising over the first half, then 5): 58 and 34. τ falling 5 → 3: 54 and 31, the worst.
- Order (Fig. 5(b)). Character first, the rest random, scores highest for composition (about 7.5 of 10 on the plot). Clothing, background or object first is comparable to a fully random order, and style first falls “slightly below a random order”. The authors: “the initial choice of LoRA in the activation sequence clearly influences overall performance, while alterations in the subsequent order have minimal impact.”
What could have come out otherwise: with shares held equal, the schedule could have made no difference, as it would if the trajectory simply summed each LoRA’s contributions.
What this does not say
- Not how large the effects are against noise. The ablation’s sets, number of LoRAs and image count are not given, nor any spread. The order differences on Fig. 5(b) are a few tenths of a point.
- Not that each LoRA’s influence on the image depends on its position. Only overall scores were measured. Whether the first LoRA’s element is rendered better, or the others’ worse, is not reported.
- Not why. “prioritizing core image elements in the initial stage” is the authors’ reading. The record’s diffusion literature on layout being set early (MultiDiffusion’s “the layout is being determined early on in the diffusion process”, NOTE-1158) is consistent with it and does not test it.
- Not beyond this setting. Two SD 1.5 fine-tunes, LoRAs chosen to be compatible, GPT-4V as judge.
Connections
- QUESTION-273. The owner’s third mechanism counted influence as “the number of iterations … a given denoiser can apply”. Here the number of iterations is held equal and the outcome still moves with interval and order, as A33’s “Authority can reside in sequence position, not just frequency” proposed. If promoted, this is evidence that the mechanism does not reduce to a time-averaged vector of influence weights.
- THEORY-948. MultiDiffusion’s early masked phase is a second case of the same allocation used differently early in sampling changing the result.