LIT-033: Longformer: The Long-Document Transformer

Beltagy et al. (2020) — ARXIV-2004.05150

Key takeaways

• Combines a sliding window (linear in sequence length) with global attention at a small set of task-chosen positions that attend to everything and are attended by everything; a dilated window variant reaches 32K characters in autoregressive language modelling
• The global tokens are where the task’s inductive bias is stated — the paper’s own words. Question tokens for QA, [CLS] for classification. The pattern is a per-task decision rather than an architectural constant, which makes it legible in a way learned sparsity is not
• Receptive field comes from depth × window: k stacked layers of width w reach k·w, and the window is varied per layer rather than fixed
• LED is deliberately asymmetric — the encoder uses local+global, the decoder keeps full attention over encoded tokens and previous outputs, because output sequences are short
• Linear complexity, but not a free constant: the dilated pattern needs custom 8×8 block-sparse kernels

Standing in the anthology

Read — NOTE-059. Nothing is sourced to this paper: the field’s answer became exact attention made fast (SOTA-086) rather than approximate attention made sparse, and the record reflects that correctly.

The reading connects two things the record holds apart. Longformer’s designated global tokens and SOTA-134’s emergent attention sinks are the same structure — a few positions everything attends to — reached by design in 2020 and by observation later. That the mechanism appears unbidden in models nobody designed it into is a stronger argument for its necessity than Longformer’s own results, and the record now has both halves in view.

Audited by #121; unsourced, and half of a convergence. R2 — get receptive field from depth x window rather than from a wide window — is LIT-020’s R3 arrived at independently. See that note for why the pair is recorded rather than filed.

R1 is the separable claim: state which positions must see everything as a task decision rather than learning it. The record’s sparse-attention practices (SOTA-138) learn the pattern instead, so this is a genuine disagreement the record holds only one side of.

Longformer’s enwik8 table (Table 3) puts its 102M-parameter large model at 0.99 test BPC. The Sparse Transformer (LIT-225), at about 100M, also scores 0.99, and the paper reads that as matching. The Sparse Transformer number is quoted from its paper, not rerun. Longformer names it the model with the most similar attention pattern, a dilated sliding window built from 8×8 BlockSparse blocks.