Defazio et al. (2025) — ARXIV-2512.17131
Key takeaways
Introduces Generalized Primal Averaging (GPA), a unified optimizer that decouples Nesterov’s two interpolation constants to produce a smoother, single-loop alternative to DiLoCo and Schedule-Free, achieving 8-10% fewer training steps than AdamW on Llama models while reducing memory overhead and hyperparameter count.
- Theorem 1 (GPA convergence). For any base optimizer with O(sqrt(T)) regret, GPA achieves O(1/sqrt(T)) convergence with negative Bregman divergence correction terms that can accelerate convergence beyond the base optimizer rate. Holds when: Convex setting; fixed mu_x, mu_y; base optimizer satisfying O(sqrt(T)) regret bound.
- Empirical speedup on Llama models (Table 2). GPA-AdamW achieves 8.71% fewer steps to target loss than AdamW on Llama-160M and 10.13% fewer steps on Llama-1B. Holds when: Llama-160M and Llama-1B; C4 dataset; mu_y=0.7 (robust default); mu_x set via DiLoCo heuristic.
- Large-batch speedup on ImageNet ViT (Figure 10). GPA provides a 25.5% step reduction over AdamW in the large-batch regime on ViT-S/16. Holds when: ViT-S/16; ImageNet; large batch size setting; cosine LR schedule.
What the evidence does not cover
- Convergence theory is limited to convex settings; non-convex convergence advantages over the base optimizer are not theoretically characterized.
- GPA requires a learning rate schedule (not schedule-free), adding a scheduling hyperparameter that Schedule-Free avoids.
- GPA incurs O(n) additional FLOPs per step vs. O(n/H) for DiLoCo, which amortizes its compute cost over H steps.
- Validated primarily on dense transformer architectures; compatibility with sparse models, MoE, or non-transformer architectures is untested.
- Single-node experiments; distributed multi-worker GPA design is proposed but not empirically validated.
Standing in the anthology
Read — the reading is NOTE-148. Arrived in the imported batch, which brought in the local-update branch — train apart for H steps, then average.