anthology-of-the-sota

LIT-046: LoRA: Low-Rank Adaptation of Large Language Models

2 min read

Properties6
StatusActive
Version2
Tagsadaptation-and-tuning
Compared againstLIT-549 — ReFT: Representation Finetuning for Language Models
Cited as “introduced by” bySOTA-184 — Adapt a pretrained model by training a low-rank update to each weight matrix, not the matrix itself
Cited as “source” bySOTA-184 — Adapt a pretrained model by training a low-rank update to each weight matrix, not the matrix itself

Graph View

Hu et al. (2021) — ARXIV-2106.09685

Key takeaways

  • The hypothesis is about the update, not the weight. A pretrained matrix is full-rank and needs to be; the delta a downstream task applies to it, the paper argues, has low intrinsic rank. So parametrize the delta as BA with r small, initialize B at zero so training starts from the pretrained model exactly, and freeze W.
  • On GPT-3 175B: roughly 10,000× fewer trainable parameters and about 3× less GPU memory than full fine-tuning. The saving is in optimizer state and gradients, which scale with the trainable count — not in the forward activations, which do not.
  • Quality is on-par with or better than full fine-tuning on RoBERTa, DeBERTa, GPT-2 and GPT-3, across the paper’s task suite.
  • The structural property that made it the default rather than one adapter among many: BA has the shape of W, so it can be folded into the weight after training. The served model is an ordinary model — no inserted module, no extra forward pass, no added latency. Adapter methods that insert layers pay at every token forever.
  • A corollary the paper notes and the field then leaned on hard: many small adapters over one frozen base are cheap to store and cheap to swap, which is a different deployment shape from one full fine-tune per task.

Standing in the anthology

The source of SOTA-184.

Unfiled for a year despite being one of the most widely adopted training practices in the corpus. The note is tagged adaptation-and-tuning, a topic the SOTA scheme does not carry, and that reads like a scope decision. It is not: the record files fine-tuning practices under training-optimization and has several. The tag concealed the paper rather than excluding it.

ReFT (LIT-549) sets LoReFT, a low-rank edit to hidden states at a few prompt positions, against LoRA at 15–65× fewer parameters. The LoRA numbers are copied from earlier papers, not rerun. LoReFT leads on eight commonsense tasks (LLaMA-7B 80.2 against 74.7) and on AlpacaEval instruction following (85.60 against 81.48 win rate). It is level on GLUE (88.2 against 88.1 on RoBERTa-large). It trails on arithmetic chain of thought: 42.6 against 46.9 at 7B, and 26.0 against 37.5 on GSM8K.

Backlinks