Source
Shazeer et al. (2020), LIT-030 — ARXIV-2002.05202.
Where the gate came from
The record recommended SwiGLU without holding the GLU its name refers to. LIT-199 is that paper, and its argument is about gradients rather than expressivity: an LSTM-style tanh gate puts a downscaling factor on both branches, so the gradient shrinks multiplicatively with depth, while gating a linear unit leaves a path with no downscaling — a multiplicative skip connection. The gate is there to let depth work.
That matters for what follows. The linear, unbounded branch is the thing the design was chosen for, and unboundedness is precisely what the two objections below are about. Which is why both remedies soft-cap the branch rather than removing it.
Contested: the range, not the quality
Two independent groups, months apart, decided SwiGLU’s unboundedness is a numerical liability at frontier scale in low precision.
Variations
SiTU-GLU (LIT-131), the first replacement in the record. The
objection is about range, not quality: both of SwiGLU’s multiplicative
factors are unbounded, so coincident large coordinates produce activation
outliers and raise the overflow risk in low-precision arithmetic. The
original GLU’s sigmoid gate is bounded but gives up the approximately linear
positive regime that Swish has. SiTU-GLU applies a scaled tanh soft cap to
the linear factor of the Swish gate and, independently, to the up branch —
near-linear at the origin, bounded far from it — with a different cap
constant per branch. Kimi K3 ships it at 2.8T.
One group, one model, no ablation: the report gives the motivation and the functional form and does not measure SiTU-GLU against SwiGLU anywhere.
PowLU (LIT-200) is the other one, and it is the better-evidenced of the pair. Same objection reached from a different angle: for large positive inputs SwiGLU approximates x², and that quadratic amplification is what enlarges the output range and produces the outliers. The remedy is a rational power function — adaptive nonlinearity, bounded growth — with scaling-law experiments across sizes and results on the Ling architecture at 7.9B and 124B. Only the 7.9B MoE model (600B tokens) is also run with SwiGLU-Clip, and there PowLU is ahead of SwiGLU on 14 of 17 benchmarks (MMLU 54.92 against 53.95, GSM8K 33.74 against 30.40) and of SwiGLU-Clip on 14 of 17 as well. The 124B model (800B tokens) is PowLU against SwiGLU alone, ahead on 12 of 17. The 7.9B control matters: hard clipping is the obvious cheap fix, and a bounded-activation paper that skips it has not isolated its own contribution. At 124B the paper does not run it.
That measurement is the comparison SOTA-158, the bounded-activation practice drawn from LIT-200, holds against this one. The stability evidence is FP8 loss curves in which SwiGLU spikes, SwiGLU-Clip delays the spike, and PowLU does not, with the bounded functions swapped in partway through training rather than trained from scratch.
What this does and does not settle
The recommendation stands at ordinary precision. Neither paper claims SwiGLU is worse; both object to its range when the arithmetic is narrow, which is a condition rather than a refutation.
What is missing is the comparison nobody has run: the two remedies against
each other, and either against SwiGLU at a scale where the instability does
not appear. Until then this is contested rather than superseded — the
field’s default is being replaced in two places for one stated reason, and
neither replacement has been checked against the other.
Re-run with five seeds, in the same codebase
Narang et al. (LIT-711) ran about fifty published modifications through one T5 codebase with every hyperparameter fixed, and most of them lost to the baseline. SwiGLU did not. On pre-training loss it scored 2.127 ± 0.003 against 2.182 ± 0.005 over five seeds, and on final loss 1.789 against 1.838. It beat the baseline on SuperGLUE, XSum, WebQuestions and WMT’14 at the same step rate. GeGLU scored within noise of it. The win also held when the baseline’s relative position biases were swapped for learned absolute positions.
This is weaker evidence than it looks, because it is not a transfer test. LIT-030 used “the same code base, model architecture, and training task as the base model from” T5, and that is the baseline Narang et al. start from. Shazeer is an author of both. So the re-run confirms that the gain is real, survives seeds and holds on more task families. It does not show that the gain carries to another implementation, which is the question the paper was asking. What carries that weight here is adoption, not measurement.
Contested on quality too, by one figure
The two objections above are about range. Primer (LIT-709, 2021)
predates both, and its objection is about quality. In one comparison (C4,
110M, T5 codebase, 525K steps) a squared ReLU, max(x, 0)², reaches
lower perplexity than SwiGLU and ReGLU, and it has no third matrix. Squared
ReLU equals ReGLU with its two matrices tied. The re-run above could not have
caught this, because it predates squared ReLU and does not include it.
Size that honestly. It is one bar chart with no variance, and the paper does not say whether the gated variants were parameter-matched. Everywhere else squared ReLU arrives together with a convolution change to attention, so the larger wins over the SwiGLU + RMSNorm baseline (about 2× compute at 537M) do not isolate the activation. On encoder–decoder masked LM the pair does no better than that baseline. The recommendation stands. What would move it is squared ReLU against SwiGLU at matched parameters above 1B, with seeds.
Squared ReLU does not escape the range objection either. It grows quadratically by design, which is the asymptote LIT-200 blames for SwiGLU’s outliers.
Known implementations
- llama2; Kimi K3 (SiTU-GLU)