Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel and Mohit Bansal (2023), NeurIPS 2023 — ARXIV-2306.01708
Key takeaways
- Two kinds of interference. Keeping only the top 20% of a task vector’s entries by magnitude loses almost nothing (Fig. 3), so most of the update is redundant, and averaging it with others drags influential values down. Among the influential entries that remain, signs conflict between models, and conflicts rise with the number of models merged (Fig. 4). They occur even among ten checkpoints of the same task (App. B.4).
- Trim, elect, merge. For each parameter, keep values from the top-k% of each task vector, set the merged sign to the sign of their sum, average only the kept values with that sign (“disjoint mean”), and add λ times the result to the pretrained weights (Algorithm 1). Every step matters in the ablation; the disjoint mean and the scale λ matter most (Table 12).
- Results, with care. Tuned on a validation set it beats the strongest baseline in every setting of Table 1: (IA)³ +2.5, T5-base +0.7, T5-large +3.6, ViT-B/32 +1.8, ViT-L/14 +1.5. Without validation, using k = 20 and λ = 1, it wins on T5-large and both ViTs and loses to task arithmetic on T5-base. With the signs of a multitask model it reaches 72.0 against 73.1 for multitask training on (IA)³ (Table 11).
Standing in the record
Filed with the owner’s batch on mode connectivity and model merging, under
loss-landscapes (ADR-027). It is the batch’s merging work that leans least
on connectivity. Its argument is about coordinates: which parameter updates
are kept, and with which sign. Its only landscape premise is the shared one,
that models fine-tuned from one pretrained model “effectively share a part
of the optimization trajectory, and can therefore often be merged without
accounting for permutation symmetry” (§2). It cites Frankle et al.
(LIT-654) for that premise and Entezari et al. (LIT-652) and Git
Re-Basin (LIT-661) for the contrasting case of networks trained from
scratch. It builds on none of their designs and measures none of them, so
no relation is declared. ADR-027 kept model merging under loss-landscapes
until a merging work arrives that does not rest on connectivity; this paper
comes close, and the report to the owner says so.
It is an instruction for machine-learning practice (“trim, elect, merge,
k = 20, λ = 1”), which is the reason for the anthology-candidate flag. The
anthology has model soups (ANTH-LIT-675) and a merging-in-pretraining study
(ANTH-LIT-684) but no task-arithmetic or task-vector paper, so this would
arrive there without its main baseline.
Within the batch, Gueta et al. (LIT-658), from an overlapping author group, is cited among the works showing separately trained weights can be interpolated. ZipIt! (LIT-665) cites it as shared-initialisation merging, the setting ZipIt! leaves. The two reach a similar diagnosis from opposite sides: averaging fails where the models’ parameters do not correspond, whether because features differ (ZipIt!) or because updates conflict in sign (here).