Kimi Team, Moonshot AI (2026) — ARXIV-2603.15031
Key takeaways
- A residual stream accumulates every earlier layer’s output with fixed, uniform weight. Attention Residuals let each layer choose: a single learned pseudo-query per layer produces content-dependent weights over the preceding layers’ representations, adding only O(d) parameters per layer. The paper frames it as the depth-wise dual of attention over time.
- Selective retrieval of earlier representations mitigates dilution and the unbounded growth of the hidden state along depth.
- Block AttnRes partitions layers into compressed blocks so the cross-layer attention costs O(Nd) rather than O(Ld) in memory and communication, which is what makes it usable at production depth.
- Validated at 48B parameters on 1.4T tokens: a 1.25× compute advantage on scaling laws, inference latency overhead below 2%, and the largest gains on multi-step reasoning (GPQA-Diamond +7.5, Minerva Math +3.6) and code (HumanEval +3.1).
Standing in the anthology
The source of SOTA-133, which is now Active: the independent result its promotion waited on is LIT-152, a different laboratory running Attention Residuals in its own harness against its own competing design. It has been compared once against the other replacement in the record, by a third party running both against its own design (LIT-152): Full AttnRes and Gated Residual land level at 1.762 training loss, and mHC and Gated Residual are called comparable at that scale. The block-summarised variant — the one that makes AttnRes affordable at depth — gives up 0.008 to 0.011 of that, which is the price of the approximation this paper’s design avoids by dropping the mixing matrix instead.