Source
Dettmers et al. (2023), LIT-378 — ARXIV-2305.14314.
The claim
Applying LoRA to the query and value attention projections is the standard practice, and it comes from the LoRA paper’s own experiments (LIT-046, SOTA-184). On large base models it does not replicate full fine-tuning.
What closes the gap is how many matrices carry an adapter, and the answer the paper reaches is all the linear layers in the transformer block — not attention alone, and not attention plus a chosen few. In their words, LoRA on all linear transformer block layers is required to match full fine-tuning.
The rule comes from LIT-378, the QLoRA paper, where it is a side result of the adapter hyperparameter sweep rather than of the quantization: Dettmers et al. could not replicate full fine-tuning performance on large base models with query/value adapters, found the total number of adapters to be the most critical LoRA hyperparameter, and recommended adapters on every linear layer of the block.
The knob that is inert, and the knob that is not
The same sweep reports that the projection rank r does not affect
performance. So the two hyperparameters trade places against the usual habit:
| usually tuned | what the evidence says | |
|---|---|---|
rank r | yes, extensively | no effect across the sweep |
| which matrices are adapted | left at the default | decides whether the method works |
Anyone who searched r and left placement at query/value was searching the
dimension the measurements say is flat.
Why this is believable, which is a separate question
Because they re-tuned the baseline too. The paper states that default hyperparameters for fully fine-tuned baselines are undertuned and searches learning rate 1e-6 to 5e-5 and batch size 8 to 128 to get a fair comparison. A weak baseline is how a placement default survives unexamined — it makes query/value look adequate by making the thing it is measured against worse. That the finding survived a strengthened baseline is most of its weight.
Conditions
Stated for large base models; the LoRA paper’s own tasks were smaller and its query/value result was not wrong about them. So this is a scale condition, not a refutation — the failure appears where the base is big enough that a low-rank update over two matrices per block cannot carry the adaptation.
It costs trainable parameters and optimizer state, in proportion to how many
more matrices are adapted. That is a real cost against the whole argument for
adapters, and the paper’s answer is that r is free to be small, so the
product of (many matrices × small rank) stays cheap.
Independent of quantization. It applies to ordinary 16-bit LoRA, and a reader who will never quantize anything should still take it.