Source
Tay et al. (2021), LIT-052 — ARXIV-2109.10686.
The claim, and the numbers under it
Kaplan et al. (LIT-028) found that shape barely matters — depth, width and head count have “minimal effects within a wide range” once parameter count is fixed. That was measured on upstream pretraining loss. Measured instead on downstream fine-tuning quality, shape matters, and depth is the axis that matters most.
The recommendation the paper gives is to increase depth preferentially, before any uniform scaling of the other dimensions. Its Table 4:
| model | params | FLOPs vs. baseline | quality |
|---|---|---|---|
| Small 16L vs. Base | 134M vs. 223M | 63.1% | comparable, 40% faster |
| Base 36L vs. Large | 16% saving | lower | outperforms |
| XL 32L vs. XXL | ~33% | ~44% | outperforms, ~3× faster |
The rule originates in LIT-052, which named it DeepNarrow and measured it on encoder-decoder T5 pretrained on C4 and fine-tuned on GLUE, SuperGLUE and SQuAD, concluding from these comparisons that the canonical T5-Base and T5-Large shapes are Pareto-inefficient. Its one check outside language is a small-model Vision Transformer comparison on few-shot image recognition.
What it trades, which is the reason it is not universal
Depth is serial. Every layer waits for the one before it, so depth cannot be split across devices the way width can, and the paper is explicit that its protocol holds “within a certain hardware limit” — its own experiments cap at 64 workers with model parallelism of 32, and it puts extreme width-parallel scaling out of scope.
That is the whole condition. A deep-narrow model is more efficient per FLOP and harder to parallelise, so the strategy is right up to the point where the parallelism you can actually buy becomes the binding constraint, and the paper does not say where that point is. This is why the practice is Proposed: what it needs is not another quality comparison but a throughput one, at a scale where the trade bites.
Where it sits against the rest of the record
SOTA-125 reaches the same conclusion from the other end — a 90M hybrid Mamba/attention model where the 27-layer configuration beat the shallow one, and the 50-layer one gained further on MMLU but cost 2× in throughput and was not chosen. Different group, different architecture, different scale, same finding and the same trade: depth wins on quality per parameter and loses on throughput. Two independent arrivals at a shape rule, both of which stop at the same wall.
They are declared as a comparison rather than a lineage because neither builds on the other; they are separate measurements of one question.
Known implementations
- T5 (the released DeepNarrow configurations: Small 16L, Small 20L, Small 24L, Base 36L, XL 32L)
- Vision Transformer (the paper’s own cross-domain check)
An outside test at small budget, and what it does not reach
LIT-681 (Cramming) is the first outside group in this record to run deep-narrow directly. It is a BERT-style MLM given 24 hours on one GPU, with every variant at the same wall-clock. Its Table 11:
| variant | MNLI-m | tokens/s |
|---|---|---|
| baseline (12 layers, modified) | 81.79 | 46,431 |
| DeepNarrow, 24 layers | 81.39 | 52,558 |
| DeepNarrow, 12 layers | 80.97 | 99,717 |
“Rescaling architectures to be deep-narrow … provides no gains.” The deep-narrow model is faster per token and still does not win, because at a fixed time budget per-token learning is set by parameter count and the extra tokens do not make up the difference.
This does not contest the claim above. The claim is about scaling at matched quality with fewer parameters and FLOPs, on T5 at up to XXL. Cramming holds the budget fixed at a scale four orders of magnitude smaller and runs once per variant. What it establishes is a boundary. At a small fixed budget, preferring depth is not a lever, and SOTA-419 is the practice that says what is.
A same-lineage sweep at fixed parameters, and what it shows
Narang et al. (LIT-711) trade depth against feed-forward width and heads at a fixed 223M in a T5 encoder-decoder, with hyperparameters fixed:
| layers | final loss | steps/s |
|---|---|---|
| 6 | 1.857 | 3.70 |
| 8 | 1.847 | 3.69 |
| 12 (baseline) | 1.838 | 3.50 |
| 18 | 1.831 | 3.38 |
| 24 | 1.843 | 3.33 |
The paper’s text says “deeper models tend to outperform shallower ones with a fixed parameter count”. Its table has a peak at 18 layers, and 24 layers is worse than the 12-layer baseline. In its learned-position rerun, every depth variant loses to the baseline on both early and final loss. The throughput cost of depth does show up, and it rises steadily.
That is compatible with this practice without confirming it. The practice’s claim is about scaling at matched quality, not about reshaping at one fixed size. And the sweep is not independent, because Tay is an author of both papers and it ran in the same codebase. What it suggests, from one final-loss run per shape, is that at 223M depth past about 18 layers stopped paying, before parallelism was anywhere near binding.