Liu et al. (2021) — ARXIV-2110.07602
Key takeaways
- The paper is a correction to a prior claim rather than a new family. Prompt tuning was known to match fine-tuning at very large scale and to fall short below it, and to fail on hard sequence-labeling tasks. The diagnosis here is that tuning only the input embeddings gives the optimizer too little to work with and too indirect a route to the layers that matter.
- The fix is depth: continuous prompts as prefixes at every layer, not just the first. That raises the tuned-parameter count from ~0.01% to 0.1–3% of the model, which is still small, and closes the gap.
- With that change, the result is comparable to full fine-tuning across model sizes from 330M to 10B and across task types including sequence labeling — the two axes the earlier claim did not hold on.
Standing in the anthology
Carries no practice, deliberately. The record recommends low-rank adaptation for the job this method also does — the practice sourced to LIT-046 — and recommending both would be recommending neither.
Kept Active because the comparison is the content. This is the strongest
version of the branch of parameter-efficient fine-tuning that did not
become the default, and it is worth knowing why it lost rather than assuming
it was worse: it is competitive on quality, and it costs at inference. A
prefix occupies context at every layer and every forward pass, where a
low-rank update folds into the weight and disappears. The deciding property
was deployment shape, not accuracy — which is a thing worth being able to
say, and which a corpus holding only the winner cannot say.
Read — NOTE-182. The reading confirms “competitive on quality” as a
measured claim: fine-tuning parity from 330M to 10B and on hard sequence
labelling, at 0.1-3% of parameters. It also surfaces something neither this
paper nor LIT-378 could have said, because they are four years and one
mechanism apart — both conclude that coverage decides and the knob people
tune does not. Here it is prompt depth that closes the gap while
reparameterization is inconsistent across datasets; in QLoRA it is the number
of adapted matrices while the rank r has no effect across the sweep. That
is a claim about parameter-efficient adaptation rather than about either
method, and SOTA-231 is where it currently lives, held by one paper.
Not treated as replication of SOTA-231, and the distinction matters:
different mechanism, different setting, four years apart. ADR-010 — contrast
is not support — has a cousin here, which is that structural agreement is not
measurement either.