Shazeer et al. (2020) — ARXIV-2002.05202
Key takeaways
- Study of gating mechanisms
- Introduction of variants like GeGLU and SwiGLU
- Improved transformer performance
- Better gradient flow
Lineage and comparisons
The unit is Dauphin et al.’s gated linear unit (LIT-199): the component-wise
product of two linear projections, one passed through a sigmoid. This paper
keeps that shape and changes two things. It swaps the gate’s nonlinearity
(ReLU, GELU, Swish, or none for the bilinear form), and it puts the unit in
the Transformer feed-forward block, in place of the first matrix and the
activation. Biases are dropped, and d_ff is cut from 3072 to 2048 so that
the three-matrix block matches the two-matrix one in parameters and compute.
On T5-base span filling, GEGLU and SwiGLU reach held-out log-perplexity
1.633 and 1.636 after 524,288 steps, against 1.677 for ReLU and 1.663 for
the sigmoid GLU itself. The paper offers no explanation for why.
Narang et al. (LIT-711) reran SwiGLU and GeGLU on four task families, in what is this paper’s own codebase and configuration (T5-base in Mesh TensorFlow; Shazeer is an author of both). SwiGLU’s early C4 loss is 2.127 ± 0.003 against the vanilla 2.182 ± 0.005, and SuperGLUE 76.00 against 71.66, at no speed cost. The gain holds under learned absolute positions. That corroborates the measurement, not its transfer to another implementation.
Primer (LIT-709) puts SwiGLU in its comparisons twice. Directly, in one 110M C4 bar chart (Fig. 5), squared ReLU gives lower perplexity than SwiGLU and ReGLU. It shows no variance and does not say whether the parameters were matched. Indirectly, Transformer++ (SwiGLU plus RMSNorm) is its strong baseline. At 537M, Primer reaches the same 12.69 perplexity in 8.3K TPUv3-hours against 16.5K, about 2×.