nucleation

THEORY-798: Independently trained neural encoders of the same data learn latent spaces that differ mainly by an angle-preserving map, a rotation or reflection times a rescaling, so their cosine similarities to shared anchor samples agree across seeds, architectures and languages, as Moschella et al. propose

2 min read

Properties4
StatusProposed
Tagsrepresentation-learning, anthology-candidate
SourceLIT-849 — Relative representations enable zero-shot latent space communication
ExtendsTHEORY-175 — Cosine similarities to matched anchor points are unchanged by exactly the maps that preserve each sample's cosines to the anchors, among them every orthogonal map times a positive scalar and every per-sample positive rescaling but not translations; in Moschella et al.'s experiments independently trained encoders agreed in them only approximately, enough for a decoder reading them to transfer untrained between seeds, architectures and languages with loss, while their nearest neighbours mostly differed

Graph View

Source

Moschella, Maiorca, Fumero, Norelli, Locatello & Rodolà (2022; ICLR 2023), LIT-849, read in NOTE-652. The hypothesis is the paper’s motivation (§1, §3): that the latent spaces of networks trained on the same data differ, up to training noise, by transformations that preserve angles, so that a representation built from cosines to anchors is shared between them.

What is asserted

That the difference between two independently trained latent spaces is mainly an angle-preserving map: an orthogonal map times a positive scalar, the class under which cosine profiles to anchors are exactly invariant. If it holds, cosine profiles to matched anchors agree across seeds, architectures and, with parallel anchors, languages, and a decoder trained on one encoder’s profiles reads another’s.

What the record holds toward it

THEORY-175, which is Active, holds the exact invariance class and what the experiments measured: relative vectors agreeing at cosine 0.86–0.97, untrained decoders transferring with loss, and nearest-neighbour overlap of only 0.10–0.39. That evidence fits “approximately angle-preserving, with distortion large enough to scramble most neighbours”. It is a consequence the hypothesis predicts, and one that a larger, non-conformal difference with good anchor coverage could also produce.

What this does not say

  • That the residual difference is small in any stated norm. The source reports no residual.
  • That transfer needs no correspondence. The anchors are paired samples.
  • That cosine is identified where a representation is fixed only up to an invertible linear map (THEORY-018).

What would confirm or reject it

The promote_when: a fitted conformal map and a fitted linear map between two trained spaces, with held-out residuals, set against stitched performance. QUESTION-211 asks the same.

Backlinks