Source
Lan, Chen, Goodman, Gimpel, Sharma and Soricut (2019), LIT-668 — ARXIV-1909.11942.
What to do
If a parameter budget is the binding constraint, tie the Q, K, V and
output projections across every layer of the stack and leave each layer its own
feed-forward weights. The measurement, averaged over SQuAD 1.1/2.0, MNLI, SST-2
and RACE on an ALBERT-base configuration:
| params | Avg | params | Avg | ||
|---|---|---|---|---|---|
E = 128 | E = 768 | ||||
| not shared | 89M | 81.6 | not shared | 108M | 82.3 |
| attention shared | 64M | 81.7 | attention shared | 83M | 81.6 |
| FFN shared | 38M | 80.2 | FFN shared | 57M | 79.5 |
| all shared | 12M | 80.1 | all shared | 31M | 79.8 |
Sharing attention costs +0.1 and −0.7; sharing the FFN costs −1.4 and −2.8. If
you need more than the quarter of parameters that attention-sharing buys, the
next increment is grouped sharing rather than all-sharing — divide L layers
into groups of M and share within each, where the paper reports smaller M is
strictly better.
The table is Lan et al.’s cross-layer sharing ablation in LIT-668, and the split as advice starts there: the paper reads its own numbers as “most of the performance drop appears to come from sharing the FFN-layer parameters”, which is the attention-yes, FFN-no rule this practice states — even though the paper itself then ships the all-shared configuration.
Conditions, and what is not established
The scale and the architecture are 2019 and encoder-only. ALBERT-base is a
12-layer bidirectional encoder trained on Wikipedia and BookCorpus, evaluated by
fine-tuning on GLUE-era tasks. Nothing here says the split holds for a
decoder-only model at a scale anyone currently trains, and the promote_when
asks for exactly that rather than for a confirmation at the old one.
Equal shape, not equal cost. The table compares configurations with the same width and depth and different parameter counts. It does not spend the saving, which is what you would actually do — and when Lan et al. do spend it, on width, the resulting ALBERT-xxlarge has fewer parameters than BERT-large and 3.17× lower throughput. A parameter saving from sharing is not a compute saving, and this recommendation is about parameters.
The asymmetry has no mechanism here, only a number. An obvious candidate sits in this record: if a layer’s feed-forward block is where facts are stored, sharing it removes per-layer storage, and THEORY-103 argues from exactly that premise in the other direction — that the absence of a per-layer store is what breaks out-of-distribution composition, and that sharing the two halves of the stack fixes it. Same intervention, and the two papers price it oppositely because they measure different things. That is a coincidence of premise, not an explanation, and no document in this record claims the mechanism yet.
Cross-layer sharing per se is not this practice, and it points the other way. Dehghani et al. (2018) share everything and report a gain over a standard transformer on language modelling and subject-verb agreement; Lan et al. name the disagreement in their related work. The recommendation here survives it because the two are compatible — the question of whether sharing helps overall is open, and which half to share if you do is what got measured.
The Universal Transformer’s gain did not reproduce, and neither did sharing in general. Narang et al. (LIT-711) reimplemented both in a 223M T5 encoder-decoder with hyperparameters fixed. The Universal Transformer reached early pre-training loss 2.40 against the vanilla 2.182, at about 4× the FLOPs. A 25-configuration sweep brought it to 2.265 and “we were ultimately unable to match the performance of the vanilla Transformer”. ALBERT-style all-block sharing scored 2.497, encoder-only sharing 2.298 and decoder-only sharing 2.352, all worse. That weakens the counter-signal above, and it adds a second measurement that sharing costs quality. It still does not test the split. No configuration there shares attention alone, so the asymmetry remains one group’s result.
On the mechanism, one causal data point. LIT-708 deletes the feed-forward layers from a small decoder and moves the parameters into attention depth. What is lost is almost entirely prediction on tokens the context cannot help with, which the authors call parametric recall. That fits the storage premise above. It deletes the FFN rather than sharing it, so it does not measure what sharing the FFN costs.
Known implementations
- None. ALBERT itself ships all-shared rather than attention-shared, which
is worth noticing rather than glossing: the authors chose the configuration
their own ablation scores 1.6 lower at
E = 128and 1.8 lower atE = 768, because the parameter saving was the point of the paper and the accuracy was the price they were willing to pay. This recommendation is therefore one nobody has shipped — including the group that measured it.