Source
Saratchandran, Teney and Lucey (2025), LIT-527 — read as NOTE-269.
Do this
When choosing a transformer’s shape, raise the head count and lower the
depth rather than accepting the published configuration. Keep d_model
fixed, so head dimension falls as head count rises, and keep the MLP width
fixed so you know what you are trading.
Reference points, taking only the configurations where the MLP width does not also move:
| model | depth, heads | Top-1 | params |
|---|---|---|---|
| XCiT-M | 24, 8 → 12, 16 | 81.4 → 81.7 | 84.4M → 59.0M |
| TNT-B | 12, 10 → 8, 16 | 82.3 → 82.3 | 65.4M → 30.9M |
| DaViT-B | 9 → 5 blocks in stage 3 | 83.3 → 83.5 | 88.0M → 62.0M |
| XCiT-L | 24, 16 → 12, 24 | 82.1 → 82.4 | 189.1M → 103.8M |
| DaViT-L | 9 → 5 blocks in stage 3 | 83.6 → 83.6 | 196.8M → 140.0M |
On language, Crammed BERT goes from 16 layers and 12 heads at 119M to 10 layers and 24 heads at 84M, with the GLUE average identical at 78.6.
The recommendation is LIT-527’s own: Saratchandran, Teney and Lucey proposed buying back removed layers with extra heads and tested it by rebuilding twelve published architectures on ImageNet-1k. Every row above is theirs, and the five that hold the MLP width fixed all hold or improve Top-1 at 29–53% fewer parameters, which is the whole of the vision evidence this practice stands on.
Why it should work
The condition number of a matrix is σ₁/σ_k, and a high one slows gradient
descent. Theorem 3.2 of LIT-527 shows the attention block’s condition number falls as the
number of heads rises. So heads are doing optimization work and not only
representational work — and if some of what depth was buying was
easier optimization, heads can buy it more cheaply, because a head costs a
fraction of a layer.
Why Proposed
The mechanism is proved and the trade is not derived from it. The paper says so itself: “while we lack a full theoretical explanation for this trade-off”. Conditioning improves with heads; whether that is why the dropped layers are free is a hypothesis with a suggestive experiment behind it.
It contradicts SOTA-190 and nothing arbitrates. That practice says to increase depth before any other dimension. The axes are not the same — depth against width there, depth against head count here — and neither paper cites the other. The record carries both.
No throughput anywhere. Parameters and training memory fall by up to half. Latency, tokens per second and behaviour under model parallelism are not reported, and SOTA-190’s source names parallelism as the reason its own recommendation has a ceiling. A shape recommendation without a throughput column is half a recommendation.
One group, one seed per configuration, 300 epochs on ImageNet-1k.
Conditions
Half the paper’s configurations also shrink the MLP. ViT, DeiT and VOLO rows change three things at once and their parameter savings should not be quoted for this practice. The five rows above are the ones that isolate it.
More heads at fixed width means smaller heads, and no experiment here finds where that breaks. XCiT-L runs 24 heads of dimension 128; nothing says what happens at 48 heads of 64.
Untested against SOTA-109. Grouped-query attention reduces distinct key-value heads to save memory bandwidth; this raises head count for conditioning. They are not contradictory — GQA keeps query heads separate — but nobody has built a model that does both, and this practice gives no guidance on the interaction.
The language evidence is 119M and below. Crammed BERT and a GPT-2 on TinyStories. Everything at scale here is vision.
Known implementations
None. The paper rebuilds published architectures; no released model is known to use the resulting shapes.