Dauphin et al., Facebook AI Research (2016) — ARXIV-1612.08083
Key takeaways
- The GLU, and why it is shaped that way. The paper’s own framing is a gradient argument. The LSTM-style gate it compares against — which it names the gated tanh unit — puts a downscaling factor on both branches, so the gradient shrinks multiplicatively as layers stack. Gating a linear unit instead leaves a path through which the gradient flows without downscaling, which the authors describe as a multiplicative skip connection. The gate is not there to add non-linearity; it is there to let depth work.
- The comparison is run, not assumed. GLU is measured against GTU and against the LSTM-style gating of the earlier work, and reaches higher accuracy and faster convergence.
- The surrounding architecture — stacked gated convolutions instead of recurrence — is what the paper was for at the time: parallelisable over sequence positions, an order of magnitude lower latency to score a sentence, state of the art on WikiText-103, competitive on Google Billion Words. That framing has been overtaken; the gate has not.
Standing in the anthology
The origin of the unit SOTA-034 recommends a variant of. The record recommended SwiGLU without holding the GLU that “GLU” in its name refers to, so the shape of the recommendation had no source: why a gate at all, and why on a linear branch rather than a tanh one.
Worth reading now for a reason it did not have in 2016. The gradient argument says the linear branch is the point — and unboundedness is exactly what LIT-131 and LIT-200 now object to in SwiGLU at scale in low precision. The property being engineered around is the one the design was chosen for, which is why the replacements soft-cap the branch rather than remove it.
Filed via the reference pass in #40, from Kimi K3 §2.3.2.
No practice of its own: it is the origin of a unit the record already recommends a variant of. SOTA-034 recommends SwiGLU and now carries this paper’s argument in its “Where the gate came from” section — the gate is there so depth works, and the linear branch is the point. Filing the ancestor separately would offer a reader a choice between a design and its own improved descendant.
The 2016 argument has also become load-bearing in a way it was not: the unbounded linear branch is precisely what the two low-precision objections now target, which is why both replacements soft-cap it rather than removing it.