nucleation

LIT-1539: Towards Understanding Fine-Tuning Mechanisms of LLMs via Circuit Analysis

1 min read

Properties5
StatusActive — read 2026-10-26 (NOTE-1183) in full, from the arXiv copy (v2, the ICML 2025 version); worth reading for its tracking of EAP-IG circuits across fine-tuning checkpoints on tasks the pretrained model fails, with node change rates of 15–24% against edge change rates of 66–80%, and for using the layers whose edges change most to allocate LoRA rank. Read it knowing that the node–edge contrast has no null comparison at the same edge budget and that a fixed edge budget produces it by itself (THEORY-997), that it concerns fine-tuning a pretrained model on arithmetic rather than development from initialization, and that the abstract's 2.46% gain is not recoverable from its tables.
Version2
Tagsrepresentation-learning, anthology-candidate
Cited as “paper” byNOTE-1183 — Wang, Hu, Du, Cheng, Wang and Zou — Fine-tuning mechanisms of LLMs via circuit analysis
Cited as “source” byTHEORY-997 — When a pretrained language model is fine-tuned on a task it initially fails, it acquires the task mainly by reconnecting components it already has rather than by recruiting new ones: the circuits found before and after keep most of their nodes and change most of their edges, as Wang, Hu, Du, Cheng, Wang and Zou propose from LoRA fine-tuning of Pythia-1.4B on arithmetic

Graph View

Wang, X., Hu, Y., Du, W., Cheng, R., Wang, B. & Zou, D. (2025). Towards understanding fine-tuning mechanisms of LLMs via circuit analysis. ICML 2025. arXiv:2502.11812.

Read in full on 2026-10-26 (NOTE-1183). One account was drawn from it: THEORY-997. The identification from T24 was checked against the reading and holds (see the history).

Why this is here

Cited at T24 for the point that the relations among a network’s components can reorganize while the components stay the same, which the reply reads as the network analogue of members reorganizing into new perspectives. It bears on QUESTION-275, and on CLAIM-367 (which unit a training-time condition belongs to depends on how units are individuated).

Backlinks