Puvvada et al. (2025) — ARXIV-2504.08719
Key takeaways
Independent arrival at the same layout, including the ratio. Global layers carry no positional encoding; local layers are sliding-window attention with RoPE, window 512, base 1,000,000; the global:local ratio is 1:3. SWAN-1B (24 layers, d 1536) was trained from scratch on 1T tokens at 8K sequence length against a RoPE GPT baseline matched in size and data, and performs comparably or better on standard benchmarks. The paper names LIT-208 as concurrent work with the same structure.
Dynamic attention scaling is the part that is theirs. Length extrapolation here does not come from the layout alone: a scaling applied to attention scores at inference is what keeps the global NoPE layers usable well past the training length. The paper is explicit that the concurrent work lacks it, and treats it as the crucial element rather than a refinement — which is the distinguishing claim to check if the two designs are ever compared properly.
An existing model can be converted. The strongest practical result. Take an 8B RoPE GPT pretrained on 15T tokens at 8K context, initialise SWAN from its weights, remove the positional encoding from the global layers, set the 512-token window on the local ones, and continue pretraining. On the short benchmarks (GSM8k, MATH500, MBPP, HumanEval, MMLU, IFEval, BFCL; the average excludes MT-Bench and RULER) the converted model averages 70.95 against the original’s 71.55 — near parity, with long-context capability gained. No full retrain.
Against RoPE (LIT-045), from scratch and by conversion. At 1B and 1T tokens, SWAN averages 51.4 against the RoPE GPT’s 49.5 on eight commonsense benchmarks, and is level or ahead on each one. On RULER the two are similar inside the 8K training length. Beyond it the RoPE model “fails completely” and the table gives it no score, while SWAN still scores 45.8 at 16K and 14.9 at 256K. In the 8B conversion the converted model gives up a little on short tasks (Table 3: 70.95 against the original’s 71.55), which the paper itself calls “a minimal decrease”. The conversion cost 315B tokens of continued pretraining, about 2% of the original compute, and gives 77.8 on RULER at 128K, where the RoPE model has no score.
What it takes from LIT-207, and what it changes. The global layers are NoPE layers, and the paper cites Kazemnejad et al. (LIT-207) together with Haviv et al. for the idea. But it cites them for the failure too: in its own runs a pure NoPE model, like a pure RoPE one, degrades sharply past the 1,024-token training length. Probes show why: the implicit position a NoPE model learns stops working at the boundary. SWAN’s change is to interleave windowed RoPE layers, which take over local position so the NoPE layers no longer track it. The logarithmic attention scaling then keeps those layers working at long range. So it adopts LIT-207’s premise that global layers need no encoding, and rejects the stronger reading that NoPE alone extrapolates.
Standing in the anthology
This is what corrects a claim the record was making. SOTA-151 and SOTA-063 both said that RoPE-or-not is settled at pretraining and therefore is not a choice an existing model can revisit — which was the reason for not filing the two as contested. That reasoning does not survive this paper. Conversion costs continued pretraining, which is not free and not something most readers will do, but it is a very long way from impossible, and both practices now say so with this as the source.
Together with LIT-208 and LIT-133 it makes three laboratories, three different local mechanisms — windowed RoPE attention here and at Cohere, a gated delta rule at Kimi — and the same 1:3 ratio and the same treatment of the global layers. That convergence is what SOTA-153 is filed on, and it is stronger evidence than any one of the three, because none of them is replicating another’s setup.