Press et al. (2021) — ARXIV-2108.12409
Key takeaways
- Deletes position embeddings entirely and adds a fixed, non-learned linear
penalty to each attention score, proportional to query–key distance:
softmax(q_i·k_j + m·−(i−j)), with one slopemper head - Slopes are the geometric sequence starting at
2^(−8/n)— for 8 heads1/2¹ … 1/2⁸— found by “a brief manual exploration of around ten slope sets”. The stated rule of thumb: slopes in (0,1), density increasing toward 0. Making the slopes trainable did not extrapolate well and cost 3% training speed - Sinusoidal embeddings cannot extrapolate at all — degrading as soon as
one token past
Lis added, at bothL = 512andL = 1024. The T5 bias also extrapolates; Raffel et al. proposed that and never tested it, and this paper is where it is tested - At 1.3B on a 461 GB corpus, ALiBi matches sinusoidal perplexity while running faster and using less memory — because it can be trained on shorter subsequences. The practical argument is training cost, not long-context quality
- The authors’ own deflation, Appendix B: ALiBi’s edge is “largely explained by its improved avoidance of the early token curse”, and future work “might achieve further gains by more efficiently exploiting longer histories” — i.e. ALiBi is not doing that
Standing in the anthology
Read — NOTE-028. Nothing is sourced to this paper; RoPE won, and ALiBi is carried as the alternative that did not.
That is right about adoption and was incomplete about the measurement. On the one axis ALiBi was built for, it is the only incumbent that works. LIT-638 scores attention resolution — how well an attention pattern distinguishes token distance — at a 1024 training length and at 2048:
| variant | 1024 | 2048 |
|---|---|---|
| RoPE | 0.91 | 0.08 |
| absolute sinusoidal | 0.87 | 0.28 |
| ALiBi | 0.81 | 0.88 |
RoPE is the best of the three in-distribution and collapses by an order of magnitude past the boundary. ALiBi is the weakest inside and the only one that survives.
The same paper says the trade has a price: “Table 2 shows that Alibi’s perplexity is larger than ROPE about 0.2∼0.3”. Its Table 2 shows the reverse. Inside the 1024-token training range ALiBi’s perplexity is slightly lower than RoPE’s at every length: 37.66 against 38.09, 29.92 against 30.38, 24.99 against 25.52. The paper attributes the price it claims to the mechanism — the exponential decay acts “like a soft sliding window”, and “the absence of long-term dependency contributes to a performance drop”. Its stability “comes from explicit decay, but it prevents the model from learning position dependency itself.”
The controlled downstream comparison is Kazemnejad et al.’s (LIT-207), which trains ALiBi beside sinusoidal APE, T5’s relative bias, rotary and no encoding at all, under one set of hyperparameters at about 107M parameters, on synthetic reasoning tasks evaluated past their training length. ALiBi lands “in the middle of the pack”: ahead of APE and rotary, behind T5’s bias in most cases, and behind NoPE, with attention concentrated on recent positions. In that paper’s 1.3B code-model check ALiBi and NoPE both hold perplexity to about 1800 tokens on a 1024 context while rotary explodes, and ALiBi is the more stable of the two beyond that, which is the same native extrapolation on perplexity and the same recency mechanism as above.
So the reason nothing is sourced here is not that ALiBi failed at its own
task. It is that the field preferred RoPE anyway, on an in-range perplexity cost
that the paper which priced it at 0.2–0.3 did not itself measure, and then
recovered the extrapolation by other means —
SOTA-151 rescales RoPE, SOTA-179 truncates its low frequencies. A
paper can lose on adoption while being right about the thing it measured,
and this note previously recorded only the first half.
The previous takeaway “theoretical analysis” was false. There is no theoretical analysis in this paper: the slopes come from trying about ten sets by hand, and the mechanism claim is empirical. That bullet attributed to the paper the one kind of contribution it does not make.
The reading’s real value is the early token curse, and it belongs to the record rather than to ALiBi. Every long-context claim in the corpus is a claim that a model does better with more context — and here the authors of the technique say most of their own measured improvement was not that, but the easier job of predicting fewer tokens that have little context behind them. The record states that confound nowhere.
Audited by #121; unsourced. R1 is the recommendation worth having and it is an evaluation caution, not an architecture one: separate the early-token effect from genuine long-range use before claiming the latter. It is the authors’ own caveat about their own headline result, which is what makes it credible.
It belongs with the analysis-and-evaluation practices ADR-027
made fileable and is the strongest single candidate left in that group.