Wu et al. (2019) — ARXIV-1901.10430
Key takeaways
- LightConv: a depthwise separable convolution that additionally shares weights across channel groups and softmax-normalizes the kernel over its temporal window — the normalisation is what makes it a weighted average rather than a free linear map, which is the property being borrowed from attention
- DynamicConv: the same, with the kernel predicted from the current token alone. So mixing weights are conditioned on one position, never on the pair being mixed — it cannot condition on what it is attending to
- 29.7 BLEU on WMT English–German, matching the best self-attention results of the time, and matching the best on English–French. Kernel sizes 3, 7, 15, 31 across blocks: the receptive field is grown by depth, and 31 tokens is the ceiling
- The result is therefore a claim about the task: for translation at these lengths, the content of the other token apparently does not determine how much it should be mixed in
Standing in the anthology
Read — NOTE-065. Nothing is sourced to this paper; it belongs to the attention-replacement wave that lost to exact attention made fast.
The reading dates a question the record treats as new.
SOTA-167, SOTA-177
and SOTA-178 are all, at bottom, about how much of attention’s pairwise
content-conditioning is load-bearing — and this 2019 paper answers “for
translation, none of it” with a fixed 31-token window. That answer did not
survive long contexts and general language modelling, and the record carries
the modern attempts without the earlier negative result they are implicitly
answering.
Audited by #121; unsourced, and half of a convergence. R3 — grow the kernel with depth rather than widening one — is the same recommendation LIT-033 reaches independently for attention windows. Two papers agreeing is the pattern that justified SOTA-205 in the vision corpus, and it applies here too. Not filed in this pass because both sources are 2019-2020 and the record’s own long-context work has moved to
architectures these papers predate. Its nearest holding, SOTA-176, is
itself Proposed — so the record has no settled statement of the claim, which
strengthens the case for filing the general form rather than weakening it.
Reimplemented, and it lost. Narang et al. (LIT-711) ran both convolutions in the T5 codebase at about 223M parameters, with a width-9 kernel and every hyperparameter held at the vanilla Transformer’s. Both came out well below the baseline: early C4 loss 2.370 (lightweight) and 2.403 (dynamic) against 2.182, and WMT’14 English–German BLEU 24.73 and 17.03 against 26.62. That includes the translation task this paper won. The test used one fixed kernel width rather than the 3–31 schedule here, and no retuning, which that paper applies to every variant by design.