nucleation

THEORY-828: Fine-tuning from one pretrained model typically lands near the edge of a task-specific low-loss region in weight space that contains the convex hull of the fine-tuned models, and averaging their weights, as model soups and stochastic weight averaging do, works by moving inward

2 min read

Properties5
StatusProposed
Tagsloss-landscapes, anthology-candidate
SourceLIT-658 — Knowledge is a Region in Weight Space for Fine-tuned Language Models
ExtendsTHEORY-105 — For RoBERTa-base fine-tuned on English classification data, with loss measured by a linear probe refitted on the frozen encoder, points sampled from the convex hull of models fine-tuned on one dataset, one task family or twelve GLUE and SuperGLUE datasets lost no more than any model fine-tuned elsewhere and beat the group's own members in 88% to 96.7% of comparisons, and the centroid of the general-level models was a better starting point for BitFit than the pretrained model
Bears onQUESTION-069 — Is the connectedness of trained solutions a property of the solutions, or of the path family searched, the shared history that produced them and the regime they were trained in?

Graph View

Source

Gueta et al. (2023), LIT-658, read in full in NOTE-515 from the arXiv HTML of v3, claim C3. The extrapolation curves behind the edge claim (§6, Appendices F and G) were read only from their captions and text.

What is asserted

That fine-tuning from one pretrained model, in general and not only for RoBERTa-base classifiers, stops near the edge of a low-loss region specific to the task and starting point, a region containing the convex hull of the fine-tuned models; and that weight averaging gains because it moves inward from that edge.

What the Active entry establishes toward it

THEORY-105 holds that sampled hull points lost no more than models fine-tuned elsewhere and beat the group’s own members in 88% to 96.7% of comparisons, by the loss of a linear probe on the frozen encoder, and that the general-level centroid was a better BitFit start than the pretrained model (65.54 against 61.51). That the interior is at least as good as the members is what the edge reading predicts; it does not show the members sit at an edge.

What would confirm or reject it

The promote_when, inherited unchanged from THEORY-105: end-to-end evaluation of hull points; the same geometry from other pretrained models and architectures and on generative tasks; and soups and SWA located against the region’s members and centroid, with gains growing deeper inside. It is refuted if hull points lose to their members end to end, if soups and SWA gain without sitting further inside, or if the region has no walls in many directions, of which the origin direction of Appendix F is already one.

What this does not say

  • It does not say the region is convex. The paper calls it convex but shows only that it contains the sampled hull.
  • It does not say averaging always helps. Cross-task merging loses end-to-end accuracy (THEORY-107), which this thesis would have to explain.

Backlinks