Source
Su et al. (2021), LIT-045 — ARXIV-2104.09864.
Where the line goes from here
This is the head of a line rather than a standing answer on its own. A model that uses RoPE and later needs a longer window than it was trained on takes SOTA-151, which rescales the position indices instead of fine-tuning at the longer length — the succession runs LIT-045 → LIT-192 → LIT-193.
The alternative
The recommendation is not uncontested, and the objection is not that rotation is the wrong function. LIT-207 trained five positional schemes from scratch under identical hyperparameters and found rotary among the schemes that do not generalize to unseen lengths, with the best result coming from using no positional encoding at all — a decoder-only transformer can represent both absolute and relative position without being told which to use. Frontier practice has since split: Kimi Linear (LIT-133) and Kimi K3 (LIT-131) drop positional encoding from their global-attention layers entirely and let an interleaved recurrence carry position, which is what lets K3 reach 1M tokens with no RoPE modification at all.
The alternative is filed as SOTA-153, and it is not marked as contesting this one, because the two are not rival settings of one knob: RoPE-or-not is an architectural choice, made before training and reversible afterwards only at the cost of continued pretraining (LIT-209 converted an 8B model that way, at parity on short benchmarks). What a reader should take from it: RoPE remains right for a dense transformer, and is not automatically right for a hybrid whose cheap layers can carry position themselves.