anthology-of-the-sota

LIT-052: Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers

2 min read

Properties7
StatusActive
Version2
Tagsmodel-architecture, training-optimization, analysis-and-evaluation
Compared againstLIT-681 — Cramming: Training a Language Model on a Single GPU in One Day
Cited as “introduced by” bySOTA-064 — Warmup needed scales sub-linearly with model size, SOTA-065 — Initialize layer norm weights closer to 1 for larger models, SOTA-066 — Can use shorter warmup periods for wider models, SOTA-067 — Monitor loss specifically during first ~5000 steps for instabilities, SOTA-068 — Use gradient clipping during early training phase, SOTA-190 — Increase depth before any other dimension when scaling a transformer
Cited as “paper” byNOTE-004 — Scale Efficiently
Cited as “source” bySOTA-064 — Warmup needed scales sub-linearly with model size, SOTA-065 — Initialize layer norm weights closer to 1 for larger models, SOTA-066 — Can use shorter warmup periods for wider models, SOTA-067 — Monitor loss specifically during first ~5000 steps for instabilities, SOTA-068 — Use gradient clipping during early training phase, SOTA-190 — Increase depth before any other dimension when scaling a transformer

Graph View

Tay et al. (2021) — ARXIV-2109.10686

Key takeaways

  • Kaplan et al. studied upstream pretraining loss; whether those findings transfer to downstream tasks under the pretrain-finetune paradigm was open, and this paper is the study of that question
  • Model shape matters for downstream fine-tuning, not only model size — two models with the same parameter count and different depth/width can differ substantially after fine-tuning
  • Scaling protocols operate differently in different compute regions, so a strategy validated at small scale need not transfer upward
  • The widely adopted T5-Base and T5-Large configurations are Pareto-inefficient
  • The DeepNarrow strategy: increase depth preferentially, before any other uniform scaling. Small 16L matches T5-Base on downstream quality at 134M against 223M parameters, 63.1% of the FLOPs and 40% faster; Base 36L beats T5-Large at 16% parameter saving; XL 32L is a third of XXL’s size at ~44% of its FLOPs and about 3× faster
  • The strategy’s limit is parallelism, not quality: depth is serial and cannot be split across devices the way width can, so it holds “within a certain hardware limit”
  • Checked outside language: the same DeepNarrow comparison holds for Vision Transformers on few-shot image recognition
  • Over 100 pretrained T5 configurations released

Standing in the anthology

Read in full for #114; the reading is NOTE-004.

The source of SOTA-190.

Cramming (LIT-681) tested DeepNarrow directly and found that rescaling to deep-narrow “provides no gains” at its budget: a BERT-base-sized masked language model trained for one day on one GPU. On an A4000, DeepNarrow at 12 and 24 layers reached MLM loss 1.91 and MNLI-m 80.97 and 81.39, against 1.84 and 81.79 for its modified baseline. Its explanation is that at that budget per-token progress is set by parameter count, not shape. So the two results differ in regime and need not conflict: this paper measured shape at T5 scale, Cramming at 13 exaFLOP or less.

This note is also the record’s clearest case of a citation that was wrong in its content while every checkable field was right. The title, the arXiv id, the author and the year all agreed with each other and with the real paper; only the six bullets underneath belonged to something else — and five practices were written from the bullets. Nothing mechanical could have caught it. What caught it was giving those five practices bodies, which meant reading the paper.

Backlinks