LIT-061: Efficient Large Scale Language Modeling with Mixtures of Experts

Artetxe, Bhosale, Goyal, Mihaylov, Ott et al. (2021) — ARXIV-2112.10684

Key takeaways

• The paper is a comparison study, not an architecture proposal: how autoregressive MoE language models scale against dense ones, measured in- and out-of-domain, zero- and few-shot, and under full fine-tuning. Its value to this record is the systematic like-for-like measurement rather than a new mechanism.
• MoE layers scale efficiently through conditional computation — more parameters at roughly constant per-token compute — and the study is about where that advantage holds and where it narrows.
• It measures nothing about batch size. §3.1 sets batch size and learning rate “according to the model size following Brown et al. (2020)”, i.e. GPT-3’s table, and trains every model for 300B tokens at that setting. What it adds is MoE-specific: each of the 512 experts per layer has a capacity of C·B/E tokens for a batch of B tokens (C = 2), and because expert parameters see an E-times smaller batch than the dense ones, their gradients are rescaled by 1/√E — which the paper says aligns with the square-root scaling of learning rate with batch size (Krizhevsky, 2014). SOTA-061 and SOTA-062 were both filed against this paper as though it had measured the batch-size trade; it had not, and both have been re-sourced to the papers that did.

Not Fedus et al.

This note was attributed to Fedus, who is Switch Transformer’s first author — a different paper about a different contribution. Corrected at version 2.

Filed here because the mistake is instructive: title and arXiv identifier agreed with each other, and only the author field was wrong, so nothing mechanical could catch it. That is the second instance in this record, after LIT-015, and both were found by trying to write down why a practice holds (#107).