Su et al. (2021) — ARXIV-2104.09864
Key takeaways
- Rotation-based positional encoding: queries and keys are rotated by an angle proportional to their absolute position, so the dot product depends only on the relative distance between them.
- Improved relative position modeling, without an additive bias term or a learned embedding table.
- A theoretical justification, which is the part later work disputes — see below.
Two of this note’s original claims did not survive the record catching up
This note was migrated as four bullets, and two of them are now contradicted by papers the record holds. Both are left corrected rather than deleted, because the claims are still what most readers believe about RoPE.
“Better extrapolation capability” is not one of RoPE’s properties. The record’s own SOTA-151 exists because RoPE does not extrapolate: LIT-192 reports fine-tuning at the longer length moving LLaMA’s effective window from 2048 to 2560 in more than 10,000 batches, and every note in this line — LIT-192, LIT-193, LIT-207, LIT-208 — starts from that failure. What RoPE improved on was the alternatives available in 2021, at the lengths tested in 2021.
The theoretical justification was the long-range decay of attention with distance, and LIT-210 argues it is not why RoPE works. The decay holds under the paper’s assumption of aligned or constant query and key vectors and fails for Gaussian ones; there is always a query making RoPE’s contribution maximal at any relative distance you choose. That does not make the encoding worse — the same work finds trained models using its high frequencies to build robust positional attention heads, which is a better reason than the one given here.
Measured against what came after. Sun et al. (LIT-638) put RoPE beside ALiBi, absolute sinusoidal positions and their own xPos, trained at length 1024. RoPE has the highest attention resolution of the three incumbents in-distribution (0.91), and at 2048 the lowest (0.08), where ALiBi holds at 0.88. In-distribution perplexity is close: RoPE 17.74 against xPos 17.54.
SWAN-GPT (LIT-209) trained a 1B RoPE GPT and a SWAN model from scratch on the same 1T tokens at 8K. SWAN interleaves NoPE global layers 1:3 with sliding-window RoPE layers. On eight short benchmarks SWAN is level or ahead (average 51.4 against 49.5). On RULER the RoPE model scores nothing past its 8K training length, while SWAN still scores 45.8 at 16K and 14.9 at 256K. An 8B RoPE model converted to SWAN by continued pretraining averages 70.95 on short tasks against the original’s 71.55 (the paper’s Table 3; LIT-209’s takeaway gives the pair the other way round), and reaches 77.8 on RULER at 128K.