nucleation

THEORY-1185: Merging models trained on different tasks from different initialisations fails where their features have no counterpart in the other model, and such features grow more common with depth

2 min read

Properties5
StatusProposed
Tagsloss-landscapes, representation-learning, anthology-candidate
SourceLIT-665 — ZipIt! Merging Models from Different Tasks without Training
ExtendsTHEORY-113 — Merging ResNets trained from different initialisations on disjoint halves of a label set by permuting one onto the other and averaging fell far short of the ensemble; their cross-model feature correlation fell with depth, merging only the early layers recovered most of the ensemble's accuracy, and merging features within each model as well as across both helped at 4× width and above; Stoica et al. attribute the failure to features with no counterpart in the other model
Bears onQUESTION-316 — Is it the difference in task, and not depth and initialisation alone, that leaves features with no counterpart in the other model and makes merging fail: at matched width, do models trained from different initialisations on disjoint tasks lose cross-model feature correlation faster with depth than models trained on the same task, with merging failing at that depth?

Graph View

Source

Stoica, Bolya, Bjorner, Ramesh, Hearn & Hoffman (2023), ZipIt!, LIT-665, read in NOTE-523. What its ResNets on disjoint halves of a label set showed is THEORY-113, which this entry extends.

The claim

Stoica et al. explain why permute-then-average fails for models trained on different tasks. Permutation maps each feature of one model to exactly one of the other, which assumes every feature has a counterpart. Models trained on different tasks need not learn the same features, so the permuted average lands outside both task basins (§3; NOTE-523, C1). Such unmatched features grow more common with depth: cross-model feature correlation falls from early to late layers, which is why the late layers should stay separate (C5). Merging a feature with a redundant one in its own model recovers what one-to-one matching cannot (C2, C4).

The evidence has no same-task control. Kornblith et al.’s fall in similarity with depth is a same-task finding, so the depth profile may not come from the task difference at all.

What would settle it

The promote_when, which is QUESTION-316: the depth profile of correspondence and of merge accuracy at matched width for same-task and disjoint-task pairs, and a one-to-one merge after a better alignment than permutation. Further results in which partial zipping beats full merging cannot settle it.

Backlinks