Jaiswal, Wang, Yin, Liu, Chen, Zhao, Grama, Tian and Wang (2024) — ARXIV-2407.11239, ICML 2025. Read as NOTE-264.
Key takeaways
- “Weight matrices are low rank” is false as a uniform statement. The paper splits matrices into Low-rank Components, characterised by a heavy tail in the sorted singular values, and Non-Low-rank Components, which lack one.
- The split is systematic by component. Query/Key and the MLP Gate projection show a clear Hessian gap and rapidly settling gradient subspaces, so they converge to low rank. MLP Up/Down and the attention Value projection do not.
- And by depth. Middle blocks have small Hessian gaps and stay high rank;
the first and last few blocks go low rank. At 50% effective-rank reduction
on LLaMA-2 7B,
q_projandk_projtake over 90% compression. - Non-uniform rank beats uniform rank by margins that are not close. ~6.4× better perplexity than uniform at 30% reduction (LLaMA-2 7B), ~47× at 40% (13B). On LLaMA-7B at 25% compression, factoid-QA goes 79.02 full model, 34.63 uniform, 71.89 non-uniform.
- The same partition guides fine-tuning. Updating only the Low-rank Components matches or beats full fine-tuning at 35% of the trainable parameters.
Standing in the anthology
It is the strongest single answer to THEORY-059’s open question, and the answer is “yes, but not uniformly”. The account behind SOTA-314 needs weight spectra to be steep. They are — in some matrices, at some depths — and this paper measures which, on LLaMA-2 7B and 13B and Mistral-7B.
It is also the source of a recommendation the record did not hold. The comparison against uniform rank reduction at matched compression is run, not argued, and it is what SOTA-318 rests on.
What it does not settle is the partition itself. Its grouping disagrees with LIT-517’s on two matrix types out of six, which is the reason the practice filed from it says measure yours rather than use this list.