Source
Chen et al. (2023), LIT-192 — Position Interpolation. Peng et al. (2023), LIT-193 — YaRN.
The problem is not that a longer window is expensive to train. It is that training it directly barely works: fine-tuning a pretrained LLaMA at the longer length moved its effective context from 2048 to 2560 after more than 10000 batches. Meanwhile pushing the model past its trained window without adaptation does not degrade gracefully — perplexity goes to numbers comparable to an untrained model, and a question at position 3000 becomes unanswerable even from evidence at position 2900.
That is the gap in SOTA-063: it recommends RoPE and says nothing about the window, and a RoPE model asked for more context than it was trained on is exactly where the encoding fails. This practice is the step such a model takes, and it means nothing for a model that did not follow that one.
The remedy is to change the position indices rather than the weights. Scale them so the longest relative distance the model is asked about is one it already saw in pretraining, then fine-tune briefly to settle. Under 1000 steps takes LLaMA 7B–65B to 32768 tokens, a 16× extension, at a cost negligible against pretraining.
The failed fine-tune, the remedy and the 16× result are all from Position Interpolation (LIT-192), which is where the recommendation to rescale rather than retrain starts: linearly down-scale the position indices by L/L’ and apply RoPE unchanged. Its argument is a proven bound — between the integer positions pretraining fitted, a learned attention-score function stays close to their linear interpolation, while outside them nothing constrains it.
Scale by wavelength, not uniformly. Stretching every RoPE dimension equally destroys the high-frequency components, and the effect compounds: uniform interpolation stalls around a scale factor of 8 even with fine-tuning. YaRN’s diagnosis is that RoPE is not purely relative — in dimensions whose wavelength exceeds the pretraining context, the rotation never completes and absolute position survives — so those dimensions should be left alone while the fully-rotating ones are interpolated. With a temperature on the pre-softmax logits, folded into the rotary tables so it costs nothing at run time, that reaches 128k with 10× fewer tokens and 2.5× fewer steps.
Conditions: this is for RoPE models, and it is a post-hoc extension of a finished model, which is a different question from how to train for a long window in the first place (SOTA-139). A short fine-tune is still required — the scaling alone is not free of adaptation, except under Dynamic Scaling at inference, which works without fine-tuning but is a graceful-degradation measure rather than a way to reach a target length.
Variations
Dynamic Scaling recomputes the scale factor per forward pass from the current sequence length instead of pinning it at the target. A fixed factor costs quality below the target and breaks abruptly above it; the dynamic form degrades gracefully and, notably, works on unmodified pretrained models.
What “it works” means here, and what it does not
A perplexity that does not explode is not a model using the longer context. LIT-638 states the distinction and it is worth importing, because every result behind this practice is reported as perplexity at the extended length: with a proper attention map “the perplexity does not explode but does not decrease at the same time. The ideal situation is to use the long context in the right way, in that case, the model should perform better instead of saturation.”
The same paper measures what rescaling is working against. Attention resolution — how well an attention pattern distinguishes token distance — for RoPE at a 1024 training length and at 2048: 0.91 → 0.08. The best incumbent in-distribution, the worst outside it by an order of magnitude. ALiBi, which nothing in this record is sourced to, holds at 0.81 → 0.88.
Two things follow, and they pull in opposite directions.
It strengthens the motivation. The defect this practice names — RoPE does not extrapolate — now has a number attached, and the number is not marginal.
It narrows what the practice has been shown to buy. The evidence is that rescaling keeps perplexity from degrading at the target length. Nobody in this record has measured whether a rescaled model recognises position at that length, and the one available instrument says the unrescaled pattern has almost no resolution there. Not degrading and working are different claims, and only the first is established.
That is a gap in the evidence rather than a doubt about the practice. The production adoption in the consensus note is real, and a serving configuration extending 262144 to 1M is a strong signal about usability. It is not a measurement of position-recognisability.
The recipe is also not the largest variable. Bertsch et al. (LIT-801) held one extension recipe fixed, a raised RoPE base and 10B tokens at 64K, and varied only architecture across 26 7–8B models trained on identical data. HELMET at 32K ranged from 29.9 to 56.4. A two-stage YaRN extension on three of them scored lower but ranked them the same and widened the gap. A 50B-token extension did not close it either. Their point for this practice is that extension recipes are mostly developed on Llama models, and the Llama 3 architecture was among the easiest in the pool to extend, so a recipe’s reported success may not transfer. Olmo Hybrid (LIT-799) adds the other boundary. In a hybrid whose recurrent layers can carry position, dropping RoPE at extension beat YaRN at 32K and 64K (85.0 against 76.9 at 64K). That is SOTA-153’s territory, not this practice’s, and it is one model.
Why the frequencies fail in the first place
The practice was filed on the observation that RoPE does not extrapolate and on two remedies for it. LIT-210 supplies the mechanism underneath, by opening a trained model rather than reasoning from the encoding’s definition.
RoPE’s frequency band is used for two jobs. The highest frequencies build positional attention heads — robustly, and provably so. The lowest carry semantic content in high-norm channels, and Theorem 6.1 shows those channels cannot be robust once the context is long. So the part of RoPE that breaks at length is a specific part, and it is the part LIT-193 already treats differently: YaRN leaves the dimensions that never complete a rotation alone and interpolates the ones that do, which is this distinction reached from the other side.
It also explains the cheap move this practice does not cover. Raising the base wavelength from 10,000 to 500,000 — Code Llama’s, adopted by Llama 3 — pushes the fragile low-frequency channels toward being distance-agnostic, and the paper’s own remedy is to remove them outright: keeping a fraction p of RoPE’s frequencies held perplexity and improved it at 2B, with p=1 being RoPE and p=0 being NoPE.
Not added to source:. This work explains why the problem exists; it does
not evidence that rescaling is the answer, and the sources of a practice are
what its recommendation rests on.
The alternative: not needing it
Kimi K3 (LIT-131) reaches 1M tokens with no positional encoding on its global-attention layers, and says in as many words that it therefore extrapolates “without any positional-encoding modification, such as RoPE rescaling or interpolation”. Position sensitivity comes from the recurrence and decay of the interleaved linear-attention layers instead.
The alternative has a source of its own now. LIT-207 trained five positional schemes from scratch under identical hyperparameters and found the one that generalizes to unseen lengths is the absent one — and showed why a decoder-only transformer can represent absolute and relative position without being given either. Kimi Linear (LIT-133) cites it and repeats the comparison at 48B, against a RoPE version of itself.
So the record holds two answers to the same problem and they are not competing versions of one technique. This practice is what a RoPE model has to do; SOTA-153 is a design that removes the need.
This entry used to say the choice was settled at pretraining, so that a
model which already exists cannot take the other option. That is wrong.
LIT-209 converted an 8B RoPE model pretrained on 15T tokens to a
NoPE-global architecture by continued pretraining, and it came back at parity
on short benchmarks with long-context capability gained. Conversion is not
free and most readers of this practice will not do it, which is why the two
are still not filed as contested — but the reason is cost, not
impossibility, and the record should not have claimed the stronger one.