Source
Geiping and Goldstein (ICML 2023), LIT-681, on the premise Kaplan et al. measured in LIT-028.
What to do
When the budget is fixed in time or FLOPs and you are choosing an architecture, sort candidate changes into two kinds:
- Changes to shape at roughly constant size: depth against width, head count, FFN frequency, funnels, layer sharing. Expect them to buy nothing in loss. Do not spend the budget searching them.
- Changes that make a step cheaper without changing the parameter count much: removing QKV, linear-layer and decoder biases, predicting only the masked positions, fused or better-shaped kernels. Take these. Throughput is the one quantity at constant size that the scaling law does not already fix.
Pre-normalization belongs in the second group for a different reason. By itself it “has no effect on performance”. It stabilizes training enough to allow the larger learning rate and short warmup, which is where the gain comes from.
Why
Per-token progress depends on parameter count and very little on shape. So a smaller or reshaped model trades learning efficiency for tokens per second at close to par: “smaller architectures make up for their slower learning efficiency by higher throughput”. From 4 to 24 layers, throughput falls more than 5x while final MLM loss stays between 1.84 and 1.97. The trade has nothing to offer. What does move the outcome is ingesting more tokens at the same per-token efficiency, which means a faster step at the same size.
Conditions
One regime, and a small one. MLM, sequence length 128, one GPU for 24 hours, 5 to 13 exaFLOP. The negative half (“shape buys nothing”) has Kaplan behind it at far larger scale. The positive half (“step time is the lever”) has one study.
It is about loss, and downstream is noisier and not flat. The same sweep spans 79.1 to 81.7 MNLI-m. That is a larger spread than the loss suggests, which is the gap LIT-052 is about: shape can matter downstream when it does not upstream. This practice’s own source tested LIT-052’s deep-narrow shape and found no gain at this budget (81.39 against 81.79). That is one run and one task, and it bounds SOTA-190 at small budget without refuting it.
Size is not free either. The paper does find an optimal size “at a budget of around 4B tokens”, with small gains. Pick the size for the budget, then apply this rule at that size.
“Minimal cost” still has a cost. Cramming dropped rotary embeddings because their gain in loss was cancelled by the loss of speed, and kept 12 attention heads although fewer were faster, because fewer hurt fine-tuning. The rule does not say throughput always wins. It says to weigh every change per unit of wall-clock.
Known implementations
- cramming, the source’s own code.
- Bias-free linear layers are common in large language models (PaLM, LLaMA). That is adoption, recorded here as context. It is not evidence for this rule.