LIT-084: DeepNet: Scaling Transformers to 1,000 Layers

Wang et al. (2022) — ARXIV-2203.00555

Key takeaways

• Post-LN trains well and destabilises with depth; Pre-LN is stable and gives up quality. DeepNorm takes both, with a theoretically derived initialization rather than a heuristic
• The residual becomes LayerNorm(x·α + f(x)), and residual-branch weights are Xavier-initialised with gain β. Query and key projections keep gain 1 — they are deliberately not downscaled
• α and β depend only on architecture and depth, and move oppositely: for a decoder-only model α = (2M)^(1/4) and β = (8M)^(−1/4)
• Model updates are bounded, which is the analytic claim the constants fall out of
• 1,000 layers (2,500 sublayers) trained “without difficulty”, an order of magnitude deeper than prior work
• A 200-layer 3.2B model beats a 48-layer 12B model by 5 BLEU across 7,482 translation directions — depth substituting for width at a quarter of the parameters

Standing in the anthology

The source of SOTA-052.

Read in full for #114; the reading is NOTE-013. The practice is confirmed and much vaguer than its source: “use smaller variance for deep networks” is a true description of β that states neither the exponent, nor that α grows as the same root, nor that query and key projections are excluded — the last being what a reader would most plausibly get wrong.