LIT-101: AdaNorm: Adaptive Gradient Norm Correction based Optimizer for CNNs

Dubey et al. (2022) — ARXIV-2210.06364

Key takeaways

• Keeps an EMA e_t of the gradient’s scalar L2 norm and, when the current gradient falls below it, scales the gradient up to meet it: s_t = (e_t/‖g_t‖)·g_t. A floor under the gradient norm
• The correction feeds the first moment only. Applying it to the second moment is ablated and is worse, because v_t sets the step size — the method wants a stronger direction signal, not a longer step
• Generic across adaptive optimizers: instantiated on Adam, diffGrad, RAdam and AdaBelief
• Evidence is small-scale CNN vision — VGG16, ResNet18 and ResNet50 on CIFAR-10, CIFAR-100 and TinyImageNet. The title ends “for CNNs”. Largest reported gain 11.15% (ResNet50/TinyImageNet); γ = 0.95 default

Standing in the anthology

Read — NOTE-025. Proposed because it is promising and unproven, and the note now says what would settle it: any evidence on a transformer, at a scale where the record’s other optimizer claims live. There is none four years on, and the absence is itself information — LIT-156 is the paper that explains why this whole wave of Adam variants did not transfer.

Kept rather than retired because of what reading it surfaced: AdaNorm is the exact mirror of SOTA-035. Clipping puts a ceiling on the gradient norm and this record calls it Active; AdaNorm puts a floor under the same quantity. Nobody has run both, and that question is worth keeping a document for.

Nothing is sourced to this paper, and the reading deliberately creates no practice.

Audited by #121; correctly unsourced. R1 is the paper’s thesis and the reading rates it weak — established only on small CNNs. R2 is the transferable half (correct the direction signal, not the step size: touch the first moment and leave the second alone) and it is a reading of what Adam’s two moments are for rather than a measured result. The paper’s own status is Proposed for the same reason. This is what an unsourced note should look like.