Seyed Iman Mirzadeh, Mehrdad Farajtabar, Dilan Gorur, Razvan Pascanu and Hassan Ghasemzadeh (2020), arXiv preprint, listed at ICLR — ARXIV-2010.04495
Key takeaways
- Multitask and continual solutions from one start are linearly connected; consecutive continual solutions are not. After learning task 1, the model either learns task 2 alone (continual, ŵ₂) or tasks 1 and 2 together (multitask, w₂). The loss of task 1 stays low on the line from ŵ₁ to w₂, and from ŵ₁ to later multitask solutions up to w₅, but rises on the line from ŵ₁ to ŵ₂ (Figs. 3–4). Euclidean distance and CKA do not separate the two, since ŵ₅ is nearer to ŵ₁ than w₅ is (Fig. 2).
- The explanation is curvature. The direction from ŵ₁ to ŵ₂ lies in the span of task 1’s top Hessian eigenvectors; the direction to w*₂ is nearly orthogonal to all 50 top eigenvectors, so it moves through flat directions (§3, Fig. 5). Low curvature at the start is necessary but not sufficient: with more epochs both directions leave the top subspace, but only the continual one forgets (Fig. 5c).
- MC-SGD. Minimise the loss averaged over points on the lines from the candidate to the previous solution and to the current task’s solution (Eqs. 3–5), with a replay buffer of one example per class per task. After 20 tasks: Permuted MNIST 85.3% (multitask 89.5), Rotated MNIST 82.3% (89.8), Split CIFAR-100 63.3% (68.8), ahead of Stable SGD, ER-Reservoir, A-GEM, EWC and naive SGD (Table 1).
Standing in the record
Filed with the owner’s batch on mode connectivity and model merging, under
loss-landscapes (ADR-027). information-geometry is the second topic: the
paper’s explanation runs through the Hessian eigenspectrum of the earlier
task, its bulk-plus-outliers shape, and the alignment of update directions
with its top eigenvectors.
It extends Frankle et al. (LIT-654). The whole design rests on their finding that solutions sharing an initialisation are linearly connected: the multitask and continual runs are started from the same ŵ₁ for that reason (§1, §2.1, App. A), and App. B.1 shows that starting the multitask run elsewhere breaks the line on CIFAR-100. It cites Garipov et al. (LIT-673), Draxler et al. (LIT-653) and Kuditipudi et al. (LIT-671) for curved connectivity between independently started runs, the case it sets aside.
Within the batch, Wen et al. (LIT-681) start from this paper and report that in class-incremental learning the line between consecutive continual solutions crosses a high-loss ridge, and that a non-linear path is needed. Gueta et al. (LIT-658) cite it as the multitask precedent for their regions.
It carries an instruction for practice (MC-SGD), hence the
anthology-candidate flag. The anthology’s nearest holdings are on forgetting and
linear connectivity in LLM post-training (ANTH-LIT-235, ANTH-LIT-237), the
context in which its curation log named this paper.