DeepSeek-AI (2025) — ARXIV-2512.02556
Key takeaways
- DeepSeek Sparse Attention: a lightning indexer with few heads, implementable in FP8, computes an index score for each query against every preceding token; a top-k selector keeps the highest-scoring KV entries and the attention runs only over those. Integrated with Multi-Head Latent Attention in its multi-query mode.
- Retrofitted by continued pretraining onto a model that already had dense attention — the DeepSeek-V3.1-Terminus checkpoint, itself a mixture-of-experts, already extended to 128K. A short warm-up initialises the indexer with dense attention still on and every other parameter frozen: 1,000 steps of 16 sequences × 128K tokens, 2.1B tokens in all; then the top-k selector is switched on and all parameters train.
- The report also covers V3.2’s post-training and reinforcement-learning recipe; the anthology draws on it for the attention.
Standing in the anthology
The middle of the record’s sparse-attention line: NSA (LIT-143) trained sparse attention from scratch; this retrofits a learned indexer onto a dense model; V4 (LIT-139) compresses the KV cache before indexing. Source of SOTA-138.