anthology-of-the-sota

SOTA-138: Train sparse attention natively with a learned top-k indexer, warmed up under dense attention

2 min read

Properties6
StatusActive
Version2
Tagsattention-techniques
SourceLIT-142 — DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models, LIT-143 — Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
Introduced byLIT-142 — DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
SupersedesSOTA-208 — Factorize long-sequence attention into a local window and a second head that escapes it

Graph View

Source

DeepSeek-AI (2025), LIT-142 — DeepSeek-V3.2.

Make the sparsity a trained part of the model, not a serving-time approximation. A lightning indexer — few heads, FP8 — scores each query against every preceding token; only the top-k key-value entries enter the attention. Before switching it on, warm the indexer up with dense attention still active and every other parameter frozen (V3.2: 1,000 steps × 16 sequences × 128K tokens, 2.1B tokens), then continue training sparse. DeepSeek-V4 (LIT-139) keeps the shape at 1M tokens — compressing the KV cache along the sequence first, running the indexer over the compressed entries, and warming up under dense attention for the first 1T tokens before introducing sparsity at the 64K stage.

Both halves of the recipe come from LIT-142, which introduced DeepSeek Sparse Attention by retrofitting it onto the DeepSeek-V3.1-Terminus checkpoint, already extended to 128K, through continued pretraining: the indexer runs on MLA in its multi-query mode, and only after the 2.1B-token warm-up is the top-k selector switched on and every parameter trained.

Conditions: two production generations from one laboratory, plus the from-scratch evidence of NSA (LIT-143) that a natively trained sparse attention matches full attention. The warm-up matters because an uninitialised indexer selects noise; the length of the warm-up scaled from 2.1B tokens for a retrofit to 1T for pretraining. Evaluated at 64K–1M contexts; below that the KV cache is not the bottleneck and the indexer is overhead.

Sequence

Child et al. (LIT-225, April 2019): the pattern FIXED in the architecture and chosen by inspecting the data — a local window plus a strided or block- summary escape, SOTA-208, which this supersedes. Its enwik8 result, where the wrong hand-chosen escape lost to dense attention outright, is the argument for learning the pattern instead → NSA (LIT-143, February 2025): compression, selection and a sliding window, trained from scratch → DSA (this practice, December 2025): one learned indexer retrofitted by continued pretraining → CSA and HCA (LIT-139, 2026): compress, then index, with a heavily compressed dense branch beside it.

Known implementations

  • DeepSeek-V3.2, DeepSeek-V4-Pro, DeepSeek-V4-Flash

Backlinks