Source
Dao et al. (2023), LIT-106 — ARXIV-2307.08691.
Where the number comes from
The kernel processes the sequence in blocks, and the block sizes are powers of two chosen against SRAM capacity (SOTA-086) — typically 64 or 128 rows. A sequence length that is not a multiple of the block size leaves a final partial block that is masked and computed anyway, so the cost is the same as a full one. At 2048 tokens with a 128-row block that is invisible; at 129 tokens it is nearly half the work wasted.
The tensor-core tiles underneath want the same alignment, which is why the figure is usually stated as 128 rather than as “whatever the kernel’s block size is”.
Set beside SOTA-089, “align tensor dimensions to hardware boundaries”, this is that practice applied to one axis. SOTA-089 rests on Ivanov et al.’s finding that transformer training is memory-bound, so a dimension off the hardware’s unit wastes traffic, and it is explicit that the circulating constants (8, 64, 128) belong to particular hardware. This practice took one of them, a head dimension or block size in its source, and attached it to the sequence length; SOTA-089 names that transposition as the failure its own constants should be checked for.
Why this is rejected: the number is real and the axis is wrong
128 occurs fifteen times in LIT-106. Every occurrence is a head
dimension — the paper measures at d of 64 or 128 — or a block size,
from the set {64,128} × {64,128} it tunes over. Not one is a sequence
length. The strings multiple of and divisible appear zero times.
So the constant was not invented, which distinguishes this from
SOTA-067’s ~5000. It was transposed: a real number about the shape of a
tile, restated as a constraint on the length of a sequence.
The reasoning below is sound and it is not this paper’s. A partial block does cost a full block; that follows from how any tiled kernel works. But the record’s rule is that a practice states what its source establishes, and inferring a recommendation the paper does not make — however good the inference — is the failure this practice demonstrates. It is a different failure from a takeaway describing some other paper, and worth keeping visible for that reason.
What this practice is worth
Little, at the lengths language models actually train on, and the record should say so rather than imply a tuning opportunity that is not there. Sequence lengths are already powers of two by convention.
It bites in two real places: variable-length batches, where sequences are packed or padded per example rather than to a global length, and evaluation or inference on short inputs. In the first, the practice is really an argument for padding to a block multiple rather than to the longest sequence
in the batch — which is SOTA-108 stated from the other end.
LIT-106 does not state this constant. It follows from the blocked design;
the specific 128 is a property of the implementations, in the same way the
constants in SOTA-013 were.