Yuan et al., DeepSeek-AI (2025) — ARXIV-2502.11089
Key takeaways
- Sparse attention had mostly been an inference-time approximation. NSA is natively trainable: the sparsity pattern is part of the model from pretraining, so the model learns to route information through it rather than losing information to it afterwards.
- A dynamic hierarchical strategy with three branches per query: coarse-grained compression of token blocks for global awareness, fine-grained selection of the most relevant blocks for precision, and a sliding window for local context.
- Hardware alignment: block-wise selection and kernels balanced for arithmetic intensity, so the sparsity turns into speed — substantial speedups over full attention at 64k in decoding, forward and backward.
- A model pretrained with NSA maintains or exceeds full-attention baselines on general benchmarks, long-context tasks and instruction-based reasoning.
Standing in the anthology
The first step of the sparse-attention line (→ LIT-142 → LIT-139), and the paper that made trainable sparsity a pretraining choice rather than a serving trick. Its own three-branch design was not what DeepSeek shipped — V3.2’s indexer is simpler — so the practice (SOTA-138) is sourced to V3.2 and this note carries the origin.