Qiu et al., Qwen Team (2025) — ARXIV-2505.06708
Key takeaways
- Gates are everywhere — LSTMs, highway networks, state-space models, linear attention, softmax attention — and their specific effects had rarely been isolated. The paper does it for softmax attention with a controlled comparison over 30 gating variants on 15B mixture-of-experts and 1.7B dense models, each trained on 3.5T tokens.
- The finding: a head-specific sigmoid gate applied to the output of the scaled dot-product attention, before the output projection, consistently improves performance. Two effects are credited: a non-linearity between the value projection and the output projection, and input-dependent sparsity in the attention output.
- The same gate improves training stability, tolerates larger learning rates and improves scaling behaviour — and it removes the attention-sink pattern (mass on the first token) and the massive activations that come with it.
Standing in the anthology
The source of SOTA-134 and the reason every full-attention layer in the Qwen line since Qwen3-Next (LIT-136, LIT-135) is a gated attention layer. The gated MLA layers in Kimi K3 (LIT-131) are the same idea at a finer granularity, and the report is precise about it: K3 augments MLA with “an input-dependent, channel-wise full-rank output gate”, where this paper’s ablation settles on a head-specific sigmoid. Same position in the block — on the attention output, before the output projection — and same justification, so the practice covers both; the granularity is a free parameter neither paper has swept against the other.