NVIDIA (2025) — ARXIV-2512.20848
Key takeaways
- 30B total, 3B activated: a mixture-of-experts hybrid Mamba-Transformer, which is the record’s two architectural axes combined in one released model — sparsity for training economy, an SSM-plus-attention mixer for sequence economy.
- Pretrained on 25T tokens, including more than 3T new unique tokens over Nemotron 2, then SFT and large-scale RL across diverse environments.
- More accurate than the previous generation while activating less than half the parameters per forward pass, and up to 3.3× higher inference throughput than similarly sized open models, named as GPT-OSS-20B and Qwen3-30B-A3B-Thinking-2507.
- Context up to 1M tokens; base and post-trained checkpoints both released.
Standing in the anthology
A third independent adopter of the hybrid layout, from a laboratory with no connection to the two the record already holds — Kimi’s KDA-plus-MLA interleave and Qwen’s Gated-DeltaNet-plus-gated-attention. SOTA-132 rests on two independent adopters; this is a third, using Mamba rather than a linear attention as the cheap layer, which widens the practice’s claim from “the 3:1 interleave works” toward “a hybrid of a fixed-state mixer and periodic global attention works, and the mixer’s family is a second-order choice”.
It is also the record’s only note where MoE and the hybrid mixer are combined and the data pipeline is by the same group (LIT-178), so the 25T-token corpus and the long-horizon filtering argument are the same laboratory’s two halves.
No practice of its own: a model report is adoption, not evidence
(ADR-017). It describes a recipe in full and isolates none of it, so what
it supplies is that somebody shipped these choices — which is a claim about
the field, and belongs in the consensus_note: of the practices whose
designs it carries. It is the third laboratory to adopt the interleave and the one that widens what that practice can claim — a fixed-state mixer with periodic global attention, with the mixer’s family a second-order choice — which is exactly the kind of fact a consensus reading is made of.
Olmo Hybrid (LIT-799) lists Nemotron 3 Nano 30B-A3B among the mixture-of-experts hybrids in its open-weight table: 25T tokens against 6T, and ahead of Olmo Hybrid on four of the five OlmoBaseEval averages, though not on Math (53.2 against 55.1). The same paper’s controlled ablations bear on the mixer-family point above. Under one recipe from 60M to 1B, a 3:1 hybrid with Mamba-2 layers trails the plain transformer at 1B (0.698 against 0.682 BPB), while the Gated DeltaNet hybrid leads it (0.669). Within that comparison the family was not second-order.