LIT-200: PowLU: An Activation Function for Stable Pre-Training of LLMs

Jiang et al. (2026) — ARXIV-2605.25704

Key takeaways

• The diagnosis, stated precisely. SwiGLU’s strength and its hazard are the same property: for large positive inputs it approximates x², which is where its expressive capacity comes from and also what enlarges the output range and exacerbates outliers. The paper names low-precision LLM training as where that stops being tolerable.
• The remedy is a rational power function, giving adaptive nonlinearity with bounded growth, plus theoretical justification for its properties rather than only curves.
• The evidence is unusually complete for an activation-function paper. Scaling-law experiments confirming consistency across model sizes, then the Ling architecture at 7.9B and 124B total parameters, and — this is the part that matters — against SwiGLU-Clip as well as SwiGLU at 7.9B. Hard clipping is the obvious cheap fix, so a bounded-activation paper that does not compare against it has not isolated its contribution.
• The clip arm is one scale. At 7.9B (600B tokens, Table 2) PowLU leads SwiGLU on 14 of 17 benchmarks and SwiGLU-Clip on 14 of 17. At 124B (800B tokens, Table 3) the comparison is PowLU against SwiGLU alone, ahead on 12 of 17. The FP8 loss curves (Fig. 4) do include SwiGLU-Clip, which delays the spike rather than removing it, with the bounded functions swapped in partway through training.

Standing in the anthology

What this paper corrects in LIT-030 is the choice of SwiGLU itself, not its quality claim: Shazeer’s variants were introduced as a straightforward improvement to the transformer, and this paper shows that the same x² regime that gives SwiGLU its capacity is also what enlarges its output range and produces outliers in low-precision training — a cost the original study never measured.

The second independent objection to SOTA-034 on the same grounds, and the better-evidenced of the two.

Kimi K3 (LIT-131) replaces SwiGLU with SiTU-GLU because “both multiplicative factors in SwiGLU are unbounded, so coincident large coordinates can produce activation outliers and increase overflow risk in low-precision arithmetic”, and soft-caps both branches with a scaled tanh. This paper reaches the same diagnosis by a different route — the quadratic regime specifically — and answers it with a different function. Neither cites a problem with SwiGLU’s quality; both object to its range in low precision.

Two groups, months apart, independently deciding that the field’s default activation is a numerical liability at frontier scale is what moves SOTA-034 from “settled” to contested. What neither supplies is a comparison between the two remedies, or against SwiGLU at a scale where the instability does not bite — so the practice’s recommendation is qualified rather than replaced.

Filed via the reference pass in #40, from Kimi K3 §2.3.2, where it is cited alongside SiTU-GLU as the other recent attempt at this trade.

The practice is SOTA-158, Proposed — and it is the trunk, not PowLU. What this paper and Kimi K3 agree on is that an unbounded activation is a numerical liability once the arithmetic narrows; the functions they chose differ and nobody has compared them. Filing the constraint separately lets a reader act on the agreement without picking a side of the disagreement, which is the same move the record made for the hyper-connections trunk.