Source
Chen et al. (2024), LIT-464 — ARXIV-2412.02975, read as NOTE-214. Accounted for by THEORY-038.
What to do
When the workload is sequential composition — apply a step, feed the result to the next step, repeat — and you are choosing how to spend a fixed parameter budget on a decoder-only model, spend it on layers rather than on width.
The separation behind this is not a constant factor. Composing k functions
over a long context needs polynomially many parameters at constant depth
and polylogarithmically many at log k depth. Width can substitute for
depth here, and the exchange rate is exponentially bad.
Both the bound and the advice come from LIT-464. Its Theorem 1.1 proves,
with no complexity conjecture assumed, that a constant-depth decoder-only
transformer small relative to the prompt cannot solve k-sequential function
composition, and its Corollary 1.2 pairs that with a log k-layer
construction using polylogarithmic parameters; the paper reads the gap as
depth buying exponentially more than width, which is the recommendation here.
It has no experiments: the support is proof, not measurement.
The reason is specific to decoders and worth knowing, because it tells you
when the advice does not apply: causal masking means a position cannot see
what follows it, and a position does not retain what it forwarded. Each layer
buys one round of information movement. Composition needs k rounds or
enough bits per position to shortcut them. THEORY-038 has the
argument.
Conditions
This is Proposed, and the distance between the theorem and the
recommendation is the reason.
- The theorem is about representability, not learnability. It says no parameter setting of a shallow narrow decoder computes the function. It does not say training finds the composing solution when one exists, and it does not say a model below the bound fails on inputs anyone uses.
- It is asymptotic in context length. No constant in it is small enough to tell you anything about a specific model at a specific context length. “Buy depth” is a direction, not a ratio.
- The hard task is constructed against this model class. That composition is hard for decoders is a real fact; that your compositional workload is hard for the same reason is an extrapolation.
- It is a decoder-only result. An encoder solves the same task exponentially smaller, which is the paper’s second corollary — so this says nothing about encoder architectures, and nothing here argues against decoder-only, which won on generation and efficiency grounds this does not touch.
- Depth is not free. Deeper models are harder to train and to scale; the depth parameterization is its own open question — see SOTA-144 and SOTA-275 — and none of that cost appears in a representability bound.
Relation to the neighbours
SOTA-125 recommends depth over MLP width at a fixed tiny parameter budget, filed from experiments. This is the general-scale theoretical counterpart, from an unconditional bound, and the two were reached independently — which is worth more than either alone, and is not the same as either confirming the other, since one is measured and one is proved.
Known implementations
- None recorded as having been chosen for this reason. Deep-and-narrow models exist; nobody in the record shaped one because of a composition argument.