Qwen Team, Alibaba (2025) — release post at qwen.ai
(blog?id=4074cca80393150c), 11 September 2025. No paper.
Key takeaways
- The hybrid layout: 75% of layers Gated DeltaNet (LIT-137), 25% gated full attention (LIT-138), interleaved 3:1. The post reports Gated DeltaNet beating sliding-window attention and Mamba2 on in-context learning, and the 3:1 mix as the balance between that and efficiency. The attention layers add output gating, larger head dimensions and rotary embeddings on only part of the head.
- Ultra-sparse MoE: 512 experts, 10 routed plus 1 shared per token, about 3.75% of parameters active — 3B of 80B.
- Native multi-token prediction, trained with the model and usable for speculative decoding.
- Stability measures for a hybrid at this sparsity: zero-centered layernorm with weight decay on the norm weights, attention output gating, and normalised router initialisation.
- Cost: trained on a 15T-token subset of Qwen3’s 36T corpus for 9.3% of Qwen3-32B’s training compute, matching or beating it; more than 10× Qwen3-32B’s throughput at contexts above 32k.
Standing in the anthology
The first shipped statement of the 3:1 hybrid that SOTA-132 records — a month before Kimi Linear’s controlled comparison (LIT-133) and built on the Gated DeltaNet paper’s own hybrid variants (LIT-137). Qwen3.5, 3.6 and 3.8 (LIT-135) keep the design. A blog post, so its numbers are the team’s own; the practices in the chain are sourced to the papers, and this note is the lineage. The ablations behind a later generation of the same design were eventually published (LIT-152).
No practice of its own: a model report is adoption, not evidence
(ADR-017). It describes a recipe in full and isolates none of it, so what
it supplies is that somebody shipped these choices — which is a claim about
the field, and belongs in the consensus_note: of the practices whose
designs it carries. It is the first production model in the record at the 3:1 interleave, so it carries more weight as adoption than most — but weight as adoption is still not evidence about the claim.