anthology-of-the-sota

LIT-104: ReLoRA: High-Rank Training Through Low-Rank Updates

2 min read

Properties6
StatusActive
Version4
Tagstraining-optimization, adaptation-and-tuning
Cited as “introduced by” bySOTA-426 — When you merge and reinitialise a low-rank adapter mid-training, prune the optimizer state and re-warm the learning rate from zero — both, not either
Cited as “paper” byNOTE-024 — ReLoRA: High-Rank Training Through Low-Rank Updates
Cited as “source” bySOTA-426 — When you merge and reinitialise a low-rank adapter mid-training, prune the optimizer state and re-warm the learning rate from zero — both, not either

Graph View

Lialin et al. (2023) — ARXIV-2307.05695

Key takeaways

  • Restarts LoRA repeatedly during pretraining — merge the adapter, reinitialise it, prune 99% of the Adam state, drop the LR to zero and re-warm — so a sequence of low-rank updates sums to a high-rank one (rank(A+B) ≤ rank(A)+rank(B), usually strict)
  • All three parts are required: restarts alone gain nothing, and restarts plus an optimizer reset without the jagged schedule diverge
  • 1.3B on 23.1B tokens: full-rank 16.83 ppl, ReLoRA 17.27, LoRA+warm start 18.23. 5.5 GB/GPU saved and 9% wall-clock at that scale
  • The ablation is the finding. At 130M the full-rank warm start alone gives 25.46; adding all of ReLoRA’s machinery gives 25.04. The machinery without a warm start gives 29.77 — worse than the warm start by itself. The paper’s own table says most of the gain is the ordinary training it also requires

Standing in the anthology

Read — NOTE-024, re-verified against the full text (v4) on 2026-09-25. Active.

Worth carrying for the ablation as much as the method. It is the cleanest example in this corpus of a paper whose headline contribution accounts for a small fraction of its own measured gain — and it is the authors who measured it, which is why the paper stays in the anthology rather than the attic. Read alongside LIT-156 on why challengers outperform their reported baselines.

Sources SOTA-426 — the restart discipline, R1 of the reading: at a merge-and-reinit, reset the optimizer state and re-warm the rate, because the pair without the re-warm diverges and either alone gains nothing. It was held back by #121’s audit until the record decided whether ReLoRA-style restarts belong in it at all; ADR-042 settled that, and the practice is filed Proposed, conditional on restarting rather than a recommendation to.

What is not filed, deliberately: ReLoRA itself, either way. As a recommendation (R3) it is too weak — restarts beat a warm-started LoRA by 0.27–0.96 perplexity, 9% wall-clock at 1.3B, and nothing in fine-tuning. As a Rejected practice under ADR-042 it does not clear condition 3: Table 6 is a decomposition of the gain, not a refutation of it, and the restarts win every comparison the paper runs against warm-started LoRA. The open question — whether the restart mechanism, rather than the warm start, is what pays — now belongs to whichever practice someday recommends restarting.

Backlinks