Large et al. (2024) — ARXIV-2405.14813
Key takeaways
- The norm is defined recursively in tandem with the architecture. That is what makes it “modular” and what lets it cover any network built from atomic modules, rather than the multilayer perceptron the prior distance functions were constructed for
- Learning-rate transfer across width and depth, which is the practical claim. Normalising the updates of any base optimizer in this norm makes the optimal learning rate roughly invariant to both, so the user computes no optimizer-specific scale factors
- It sometimes lets a simpler optimizer do the job — the paper reports training GPT with SGD rather than Adam under the wrapper, at a smaller memory footprint
- The gradient is Lipschitz-continuous in this norm, with a recursive formula for the constant, for any network of well-behaved modules. That is the theoretical half, and the paper’s stated point is that it makes textbook optimization analyses portable to deep networks
- It deliberately avoids limits. The paper contrasts its metrization approach with infinite-width and infinite-depth analyses, arguing those are hard to act on — “from a practitioner’s perspective, these results can be difficult to make sense of”
Standing in the anthology
The middle step: it takes LIT-437’s per-layer spectral condition and makes
it an architecture-wide object, then shows what that buys. The record has
SOTA-140 and a cluster of schedule practices but nothing that says the
learning rate can be made scale-invariant by construction rather than by
sweeping.
From LIT-686 it takes the idea of a distance built from per-layer relative changes. The paper describes that paper’s deep relative trust as combining Frobenius norms across layers and names the two limits it removes: it is constructed only for the multilayer perceptron, and its Frobenius norm “precludes good width scaling”. The modular norm is defined for any network built from modules, and the paper contrasts deep relative trust with LIT-437’s RMS-to-RMS operator norm per layer, which it credits with good width scaling.
Filed from #180 — eight distinct days in the reading feed, joint
second across the whole corpus. extended_by the duality paper.
Unread — no NOTE.