LIT-733: Language Models are Unsupervised Multitask Learners

Radford et al. (2019) — OpenAI technical report, https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf

Key takeaways

• Zero-shot task transfer from language modelling alone. Conditioned on a document and questions, the largest model reaches 55 F1 on CoQA, matching or exceeding three of four baselines without their 127,000+ training examples, and sets the state of the art on 7 of 8 language-modelling datasets zero-shot.
• Capacity is the lever. Four models from 117M to 1.5B parameters (12 to 48 layers); performance improves log-linearly with size across tasks, and all four still underfit WebText.
• Two architecture changes in §2.3, stated without ablation. Layer normalization moved to the input of each sub-block, as in a pre-activation ResNet, with one more after the final block; and “a modified initialization which accounts for the accumulation on the residual path with model depth”: residual-layer weights scaled at initialization by 1/sqrt(N), where N is the number of residual layers. Everything else “largely follows” GPT, whose own report initialises all weights from N(0, 0.02).

Standing in the anthology

Filed as the origin of the depth-scaled residual initialization in SOTA-060. It is the earliest statement of it the record can name, two months before Child et al. (LIT-225) wrote the same factor as 1/sqrt(2N) with N counting blocks rather than residual layers. The paper gives the reason but no measurement of what the scaling buys.