DeepSeek-AI (2026) — ARXIV-2606.19348. Weights released 24 April 2026; the report followed on arXiv.
Key takeaways
- Two mixture-of-experts models: DeepSeek-V4-Pro, 1.6T total and 49B activated, and DeepSeek-V4-Flash, 284B total and 13B activated, both with a one-million-token context. Pro is pretrained on 33T tokens and Flash on 32T. The report calls itself a preview of the series and names a third variant, DeepSeek-V4-Pro-Max, as the maximum-reasoning configuration.
- The attention is a hybrid of two mechanisms. Compressed Sparse Attention compresses the KV cache along the sequence and then runs DeepSeek Sparse Attention (LIT-142) — a lightning indexer selecting top-k entries — over the compressed entries. Heavily Compressed Attention compresses further still and keeps attention dense. Two token-level latent KV streams come from two learned projections, so that each compressed entry pools overlapping windows of 2m tokens (the report gives no reason; reducing block-boundary artefacts is this record’s reading). At 1M tokens, V4-Pro needs 27% of DeepSeek-V3.2’s single-token inference FLOPs and 10% of its KV cache.
- Manifold-Constrained Hyper-Connections (LIT-140) replace the residual connection.
- Load balancing is auxiliary-loss-free plus a small auxiliary loss. V4 keeps DeepSeekMoE’s fine-grained and shared experts and the bias-update strategy of SOTA-148 at bias update speed 0.001 — “augmented by a slight sequence-wise balance loss that prevents extreme imbalance within individual sequences”, at weight 0.0001. The lab that argued the interference gradient away runs a small one anyway, for a failure the bias does not reach: imbalance within a sequence rather than across the batch. The affinity-score activation also changes from V3’s.
- Muon is the pretraining optimizer for most modules, credited with faster convergence and better stability; AdamW is kept for the embeddings, the prediction head, the static biases and gating factors of mHC and every RMSNorm weight. It follows LIT-122 in applying weight decay to the Muon parameters, keeping the Nesterov trick, and rescaling the update matrix’s RMS so AdamW’s hyperparameters transfer (SOTA-121) — and departs from it in one place, using hybrid Newton-Schulz iterations: aggressive coefficients for eight steps to drive the singular values close to 1, then a different set of coefficients for the final two steps to settle them precisely at 1. Batch size is scheduled upward to 75.5M tokens for Flash and 94.4M for Pro, then held.
- V4 does without QK-Clip (SOTA-131), and says so: “Consequently, we do not employ the QK-Clip technique in our Muon optimizer.” The reason is architectural — for both CSA and HCA an extra RMSNorm is applied to each head of the queries and to the only head of the compressed KV entries just before the core attention, which “avoids exploding attention logits and may improve training stability”. The remedy SOTA-131 records is not the only one; normalising the inputs to the logit removes the need for it. Both halves are quoted from the report’s own text rather than from coverage of it.
- Context is extended in stages — 4K, then 16K, 64K and 1M — and sparse attention is warmed up under dense attention (for the first 1T tokens in Flash; Pro’s dense stage is longer) and introduced at the 64K stage, staying on for the rest of training.
- Multi-token prediction is kept from V3 “without modification”: depth 1, an MTP loss beside the LM loss at weight 0.3, dropped to 0.1 when the learning rate starts to decay. That is all the report says about it. It never mentions speculative decoding, a draft model or an acceptance rate, so this report is evidence that V4 trains with MTP and not that it serves with it. A single fused MoE kernel overlaps computation and communication. FP4 quantization-aware training covers “MoE expert weights and the indexer QK path”, and the report places it in post-training rather than pretraining — a memory and compute measure applied to a model that already exists, not a precision the model was pretrained in. LIT-186 is the framework for reading that choice: quantization damage grows with how much data a model was trained on, so where in the pipeline the low precision sits is the decision.
Standing in the anthology
The third laboratory in the record to pretrain with Muon, after Moonshot (LIT-122, LIT-132, LIT-131) and TII (LIT-119), and the second model to replace the plain residual, after Kimi K3’s Attention Residuals (LIT-134) — by a different construction (LIT-140). Its sparse attention is the third step of a line the record now carries in full: NSA (LIT-143) → DSA in V3.2 (LIT-142) → CSA/HCA here. Source of SOTA-139; the other practices it instantiates are sourced to the papers that introduced them.
Read in full — NOTE-361. The reading’s main consequence for the record is negative: LIT-185 had counted this model among those that turn the MTP head into an EAGLE-3-style draft, and the report does not support it.