DeepSeek-AI (2024) — ARXIV-2412.19437
Key takeaways
- The model everything in the record’s DeepSeek line descends from: 671B total with 37B activated, MLA (LIT-174) and DeepSeekMoE (LIT-170) carried over from V2, pretrained on 14.8T tokens.
- Two things it introduces. An auxiliary-loss-free load balancing strategy (LIT-171), and a multi-token-prediction training objective (LIT-163) — both of which the subsequent frontier models in the record ship.
- 2.788M H800 GPU hours for the full training, which is the number that made the report notable at the time.
- “Throughout the entire training process, we did not experience any irrecoverable loss spikes or perform any rollbacks.” That is the strongest stability claim any report in the corpus makes, and it is worth setting against the instability literature: LIT-155 on attention-logit growth, LIT-154 and LIT-132 on the interventions that keep a run alive.
Standing in the anthology
The MoE recipe K2, K3 and V4 descend from, and the middle of the DeepSeek line the record now holds end to end: V2’s MLA (LIT-174) → this → V3.2’s sparse attention (LIT-142) → V4’s compressed sparse attention and mHC (LIT-139).
Its stability claim is the one to return to. SOTA-131 exists because Muon at a trillion parameters drove attention logits past 1000; V3 trained at 671B on AdamW without a rollback. Whether that is the optimizer, the architecture or the infrastructure is not something either report isolates, and the record should not pretend otherwise.