anthology-of-the-sota

NOTE-024: ReLoRA: High-Rank Training Through Low-Rank Updates

8 min read

Properties3
StatusRead
Version2
PaperLIT-104 — ReLoRA: High-Rank Training Through Low-Rank Updates

Graph View

Contribution

Applies parameter-efficient fine-tuning machinery to pretraining. The observation is arithmetic: rank(A+B) ≤ rank(A)+rank(B), and the inequality is usually strict, so a sum of rank-r updates can have rank far above r. LoRA as normally implemented cannot exploit this because the adapter is merged once at the end. ReLoRA merges and reinitialises it repeatedly during training, so

ΔW = s·W_A¹W_B¹ + s·W_A²W_B² + … + s·W_A^N W_B^N

accumulates rank across restarts. Demonstrated on transformer language models to 1.3B on C4.

Key insight

The restart is not free, and most of the paper is about what breaks. Adam’s moments are the obstacle: with β₁, β₂ at 0.9–0.999, the moments accumulated for W_A¹ steer the freshly reinitialised W_A² back into the same subspace, which is exactly the thing the restart was supposed to escape. So a restart needs three things at once — merge-and-reinit, a partial optimizer reset (99% of the state pruned by magnitude), and a jagged schedule that drops the learning rate to zero and re-warms over 50–100 steps. Drop the schedule and the run diverges.

The second insight is the one the paper frames as intuition and the results support: training is locally low-rank. Over a long trajectory the update is high-rank (their Figure 4), but over a short window it is well approximated by a low-rank one — which is precisely the window a restart covers.

Assumptions

  • Training is locally low-rank. Stated as a speculation, motivated by Aghajanyan, Arora and the lottery-ticket line, and supported by their own singular-value analysis rather than proved.
  • A full-rank warm start is available. From random initialisation ReLoRA needs one; their best 1.3B run warm-starts at 10K of 30K steps — a third of the run is ordinary full-rank training.
  • Single epoch on at least compute-optimal data (Chinchilla), LLaMA-style architecture, bf16, models 60M–1.3B.

Key results

  • 1.3B, 23.1B tokens: full-rank 16.83 ppl, ReLoRA 17.27, LoRA+warm start 18.23, Control (a full-rank model with the same trainable parameter count) 21.73.

  • 5.5 GB per-GPU RAM saved at equal microbatch (27.8 → 22.3 GB), and a 9% wall-clock improvement at 1.3B on 8×A100 (86 h vs 93.5 h) once the warm start is counted. The abstract’s “9–40%” is the range across model sizes and hardware, not the 1.3B figure.

  • Singular-value spectra confirm the mechanism: ReLoRA’s update has far fewer near-zero singular values than LoRA’s, resembling full-rank training.

  • Rank barely matters: r=128 (19.16 ppl) vs r=512 (19.00) at hidden size 2048.

  • Negative result, reported: “Online ReLoRA” — merging every ~100 steps while resetting the optimizer every 2–5K — is worse at both 250M and 1.3B. More restarts is not better.

  • Table 6, the ablation (130M), is the paper’s most important table:

    restartsopt. resetjaggedwarm startppl
    ––––34.17
    ✓–––34.25
    ✓✓––diverged
    ✓–✓–34.29
    ✓✓✓–29.77
    –––✓25.46
    ✓✓✓✓25.04
    regular training23.65

    The warm start alone gets 25.46. All of ReLoRA’s machinery on top of it adds 0.42. The same machinery without a warm start gets 29.77 — worse than the warm start on its own.

Claims

idclaimstrengthsupport
C1A sequence of low-rank updates accumulates into a high-rank onestrongthe arithmetic, plus the singular-value spectra
C2Restarts require an optimizer reset and a jagged schedule or they failstrongablated (Table 6, 130M, from random init); the pair without the schedule diverges, the schedule without the reset gains nothing. Not ablated from a warm start
C7ReLoRA helps in fine-tuningweak — contradicted by Appendix Bon GLUE with T5-base and T5-large it does not outperform plain LoRA (Table 8)
C3ReLoRA approaches full-rank quality at 1.3Bmoderate17.27 vs 16.83 — close, and still a gap that does not visibly close
C4ReLoRA’s own mechanism is what delivers the resultweak — contradicted by Table 6the warm start carries 25.46 of the 25.04
C5Efficiency improves with model sizemoderateasserted from 60M–1.3B; 9% wall-clock at the top end
C6Training is locally low-rankmoderateexplicitly a speculation, with supporting spectra

Method

Train full-rank for a warm start (5K–10K steps). Replace attention and FFN linears with LoRA at r=128, freezing the base weights; keep embeddings and norms full-rank. Every q steps merge W ← W + s·W_A W_B, reinitialise W_A (Kaiming) and W_B (zeros), prune 99% of the Adam state by magnitude, drop the LR to zero and re-warm over 50–100 steps.

Concepts

  • Locally low-rank training — the load-bearing idea, and the one that outlives the method.
  • Restart as a rank operation — treating merge-and-reinit as a way to buy rank, rather than as a checkpointing detail.
  • Trainable-parameter-matched control — their Control baseline is a full-rank model with as many trainable parameters as ReLoRA has, which is the comparison that makes the result meaningful. Worth stealing.

Connections

Downstream of LoRA and the lottery-ticket literature; a sibling of the optimizer-memory line the record carries elsewhere. Its methodological neighbour is LIT-156, which argues that challengers to a well-tuned baseline routinely look better than they are — Table 6 here is a case where the paper’s own ablation makes that argument for it.

Recommendations

  • R1 — If restarting an adapter mid-training, reset the optimizer state and re-warm the learning rate; either alone fails, and the pair without the re-warm diverges. Topic: training optimization. Strength: strong. The most reliable thing in the paper.
  • R2 — Match a parameter-efficient method against a full-rank model with the same trainable parameter count, not only against the full model. Topic: analysis and evaluation. Strength: strong.
  • R3 — Consider ReLoRA for memory-constrained pretraining. Strength: weak. Table 6 says most of the benefit is the warm start, and the wall-clock gain at 1.3B is 9%.

Bearing on the record

R1 is filed as SOTA-426 (2026-09-25), Proposed, sourced here. v1 of this reading said it would not be — that R1 was “too narrow to matter outside this method” — and #121’s audit then held it back on a scope question ADR-042 has since answered. It is narrow, and the practice says so: it is a rule for how to restart an adapter, conditional on restarting, with the ablation’s scope (130M, from random initialisation) written into its promote_when. R3 is still not filed, in either direction: too weak to recommend, and Table 6 decomposes the gain rather than refuting it, so it does not clear ADR-042’s third condition as a Rejected practice either.

v1 read: No practice is sourced to this paper, and this reading does not create one — R3 is too weak to file and R1 is too narrow to matter outside this method.

The reading does correct LIT-104, whose takeaways were wrong about what the paper is:

takeawayverdict
”Rank-based layer stacking”not the method. There is no stacking of layers anywhere. ReLoRA restarts an adapter on a fixed architecture
”Training optimization”fair
”Memory efficiency”fair — 5.5 GB/GPU
”Architecture design”wrong category. This is a training procedure; the architecture is stock LLaMA-style and unchanged

The document was also tagged model-architecture on the strength of those bullets. Retagged training-optimization, which is what it is.

Re-verified against the full text (v2)

Read end to end again, arXiv v4 (10 December 2023), for the restart claims specifically.

  • C2 stands, and is narrower than it reads. Table 6 is 130M, without a warm start, one run per arm as far as the paper says. From a warm start the paper reports only the full recipe against no restarts; the reset and the re-warm were never separated there.
  • “Optimizer reset” in the diverging arm is not defined. §4.2 calls it “a naive optimizer reset”; the method prunes 99% by magnitude. Algorithm 1 as printed prunes the moments of W_A only.
  • The 50-step re-warm is anecdote: “in our initial experiments … as low as 50 steps, instead of hundreds” for an optimizer initialised from scratch. No table. The 1.3B run used 100 steps of initial warmup and 50 per restart.
  • Appendix A, the practical guide, which v1 did not use: r ∈ {64, 128} works to 1B; ReLoRA and LoRA need a 1.5–2× larger learning rate than full-rank training; no significant dependence on the pruning fraction above 90%, and higher fractions risk “loss instabilities during the reset”; reset every 2K–5K steps, which “always led to better performance than no resets”.
  • Appendix B: in fine-tuning ReLoRA does not beat LoRA (Table 8, T5-base and T5-large on GLUE), and Table 9’s reset-rate sweep on QNLI is flat (94.33–94.73).
  • Appendix E: from a 2K-step warm start ReLoRA beats LoRA by 1.4 ppl (23.64 vs 25.08) — the largest restart effect in the paper, in the regime furthest from the recommended one.
  • The 1.3B final figure is 17.27 in Tables 2 and 4 and 17.24 in §4.1’s prose.

Limitations

  • 1.3B is the ceiling, and the gap to full-rank training (0.44 ppl) does not visibly close with scale.
  • The headline result depends on a warm start that is a third of the run, so “parameter-efficient pretraining” is doing a third of its pretraining the ordinary way.
  • The 9–40% speedup range in the abstract is much wider than the 9% measured at the largest scale.
  • GLUE numbers are, as the authors say, far from state of the art because the models saw ~20× less data than BERT or T5 — so the downstream evidence is weak in both directions.

Open questions

  • Is there any regime where the restart machinery, and not the warm start, is what pays? Table 6 asks this question and the paper does not answer it.
  • Why does more restarting hurt? The Online ReLoRA negative result cuts against the rank argument that motivates the method, and is reported without an explanation.

Backlinks