Haitao Wen, Haoyang Cheng, Heqian Qiu, Lanxiao Wang, Lili Pan and Hongliang Li (2023), Proceedings of the 40th International Conference on Machine Learning, PMLR 202:36940–36957 — https://proceedings.mlr.press/v202/wen23b.html
Key takeaways
- A ridge on the straight line. Between the expanded previous minimum and the new minimum, accuracy on old tasks drops in an interval of the linear path for the first increment; for later increments an interval of the path does better on all tasks than either end (§3.1, Fig. 2, PODNet on CIFAR-100, 10 seeds). How the new classifier is initialised changes the depth of the drop: Xavier, Kaiming and Imprint do better than Uniform and Normal.
- A cheap curved path. OPC writes the path as p_θ(λ) = linear path + δ_θ(λ), with δ built layer-wise from cosines and sines at frequencies (4i − 3)π/2, and coefficients constrained to sum to zero so the path keeps its ends (Eqs. 5–12). Gradient projection keeps the constraint (Eq. 17). The loss before a switching point λ* is the old tasks’ memory loss and after it the new task’s (Eq. 4). Sampling points on a cylinder of radius r around the path pulls it towards flatter regions (Eqs. 15–16).
- Gains as a plug-in. EOPC improves PODNet by 1.21, 1.81 and 2.51 points on CIFAR-100 (5, 10 and 25 increments), by 1.39, 2.22 and 3.73 on ImageNet-1K, and reduces PODNet’s CIFAR-100 forgetting from 19.26, 25.01 and 28.55 to 7.68, 8.64 and 12.02 (Table 1). Against polygonal-chain, Bezier and simplicial-complex paths fitted to the same objective, OPC is best at 10 and 25 increments with far fewer learned parameters, and below Bezier at 5 (Table 4).
Standing in the record
Filed with the owner’s batch on mode connectivity and model merging, under
loss-landscapes (ADR-027). It is the class-incremental half of a pair with
Mirzadeh et al. (LIT-651), which it cites as the pioneer and limits to
task-incremental learning with linear paths. Its own linear-path finding is
consistent with theirs: consecutive continual solutions are not linearly
connected. It differs in what it looks for: not a multitask solution
connected to old ones, but a curved path between consecutive continual
solutions whose middle can stand in for the multitask solution (§4, the
“switching point”).
It is measured against Garipov et al. (LIT-673) and Benton et al. (LIT-656). App. E fits Garipov et al.’s polygonal chain and Bezier curve, and Benton et al.’s simplicial complexes, to its own switching-point objective and reports them in Table 4; its path objective, the expected loss along the path (its Eq. 2), is the one it takes from Garipov et al. It follows Draxler et al. (LIT-653) in starting the path search from the straight line, and Frankle et al. (LIT-654) for the role of a shared start.
It carries an instruction for practice (add EOPC after each increment),
hence the anthology-candidate flag. It has no arXiv version, so the
source is the PMLR URL.