Lines of explanation

12 lines, walked from extends: and corrects: on THEORY documents. Each step explains itself; this page is the order they came in.

Grouped by tags, which every line holds in common — a line about two things is listed under both.

analysis-and-evaluation

From A dense network contains a sparse subnetwork that matches its accuracy when trained from the same initialization

• THEORY-002 — A dense network contains a sparse subnetwork that matches its accuracy when trained from the same initialization (Proposed)
  • THEORY-004 — A lottery ticket wins by re-learning the solution its dense run already found (Active)

From An influence-function estimate predicts the effect of removing a training point and retraining

• THEORY-017 — An influence-function estimate predicts the effect of removing a training point and retraining (Rejected)
  • THEORY-018 — Influence functions on neural networks estimate the proximal Bregman response function, not leave-one-out retraining (Active)

From In-context learning is gradient descent on an implicit model

• THEORY-068 — In-context learning is gradient descent on an implicit model (Rejected)
  • THEORY-067 — A transformer can run a learning algorithm on a model held in its activations, and the architecture admits several, so a construction identifies none (Active)

From Neural networks are singular, so effective complexity is how the volume of near-optimal parameters scales, not how curved the optimum is

• THEORY-075 — Neural networks are singular, so effective complexity is how the volume of near-optimal parameters scales, not how curved the optimum is (Active)
  • THEORY-073 — Training passes through stages marked by changes in loss-landscape degeneracy, and the loss curve does not show them (Proposed)
  • THEORY-074 — A phase transition of the Bayesian posterior and a transition in the training trajectory are different events, and only the first is well defined (Proposed)

From What makes a diffusion model generative is iterated restoration of any degradation, not Gaussian noise

• THEORY-079 — What makes a diffusion model generative is iterated restoration of any degradation, not Gaussian noise (Rejected)
  • THEORY-080 — A diffusion process needs noise to keep its intermediate states on the data manifold, and blur helps only while noise still dominates the bands it removes (Proposed)

attention-techniques

From A softmax head with nothing to attend to must place its mass somewhere, so models learn a positional sink

• THEORY-019 — A softmax head with nothing to attend to must place its mass somewhere, so models learn a positional sink (Active)
  • THEORY-127 — QK normalization costs long-context extension because bounding the logits raises attention entropy and removes the sink a head uses to discard mass (Proposed)

From Attention entropy is bounded below by a quantity falling exponentially in the spectral norm of the query-key product

• THEORY-061 — Attention entropy is bounded below by a quantity falling exponentially in the spectral norm of the query-key product (Proposed)
  • THEORY-062 — What crashes a transformer is spectral energy concentrating in the query-key product, not low attention entropy as such (Proposed)

concept-geometry

From A concept’s shared word-pair direction, its linear probe and its steering vector are one object, related by an inner product under which causally separable concepts are orthogonal

• THEORY-090 — A concept’s shared word-pair direction, its linear probe and its steering vector are one object, related by an inner product under which causally separable concepts are orthogonal (Proposed)
  • THEORY-034 — Concepts sit in embedding space as intersections of half-spaces, so inclusion, intersection and union are geometric operations (Proposed)

generative-modeling

From In continuous time the diffusion bound depends on the noise schedule only through its endpoints

• THEORY-027 — In continuous time the diffusion bound depends on the noise schedule only through its endpoints (Active)
  • THEORY-106 — Gaussian flow matching is a diffusion model; what separates the named recipes is loss weighting, network output and sampling schedule (Active)
    • THEORY-123 — A flow’s sampling ODE curves because the noise-data pairs it was trained on cross and the learned velocity averages them, so a coupling whose straight paths do not cross gives a straight flow (Active)
  • THEORY-107 — A diffusion loss whose weighting falls monotonically with signal-to-noise is maximum likelihood on noise-augmented data; a non-monotone one is not a likelihood objective (Active)

From What makes a diffusion model generative is iterated restoration of any degradation, not Gaussian noise

• THEORY-079 — What makes a diffusion model generative is iterated restoration of any degradation, not Gaussian noise (Rejected)
  • THEORY-080 — A diffusion process needs noise to keep its intermediate states on the data manifold, and blur helps only while noise still dominates the bands it removes (Proposed)

in-context-learning

From In-context learning is gradient descent on an implicit model

• THEORY-068 — In-context learning is gradient descent on an implicit model (Rejected)
  • THEORY-067 — A transformer can run a learning algorithm on a model held in its activations, and the architecture admits several, so a construction identifies none (Active)

model-stability

From Batch normalization works by reducing internal covariate shift

• THEORY-001 — Batch normalization works by reducing internal covariate shift (Rejected)
  • THEORY-003 — Batch normalization helps by smoothing the optimization landscape, not by reducing internal covariate shift (Active)

From Attention entropy is bounded below by a quantity falling exponentially in the spectral norm of the query-key product

• THEORY-061 — Attention entropy is bounded below by a quantity falling exponentially in the spectral norm of the query-key product (Proposed)
  • THEORY-062 — What crashes a transformer is spectral energy concentrating in the query-key product, not low attention entropy as such (Proposed)

representation-and-encoding

From Skip-gram with negative sampling implicitly factorizes the word-context PMI matrix shifted by log k, weighting frequent pairs more heavily

• THEORY-093 — Skip-gram with negative sampling implicitly factorizes the word-context PMI matrix shifted by log k, weighting frequent pairs more heavily (Active)
  • THEORY-089 — Word vectors trained on co-occurrence behave linearly because they encode log co-occurrence statistics: differences carry log ratios, which gives analogies, and sums carry products of context distributions, which gives composition (Proposed)
    • THEORY-094 — Word vectors approximate PMI in low dimensions, and relations appear as clean directions despite a noisy fit, because text behaves as if generated by a slowly drifting discourse over isotropic word vectors (Proposed) — also extends THEORY-093
  • THEORY-092 — Raising context counts to the 3/4 power helps word representations because PMI overweights rare contexts, and smoothing lowers their association scores (Proposed)

signal-structure

From Skip-gram with negative sampling implicitly factorizes the word-context PMI matrix shifted by log k, weighting frequent pairs more heavily

• THEORY-093 — Skip-gram with negative sampling implicitly factorizes the word-context PMI matrix shifted by log k, weighting frequent pairs more heavily (Active)
  • THEORY-089 — Word vectors trained on co-occurrence behave linearly because they encode log co-occurrence statistics: differences carry log ratios, which gives analogies, and sums carry products of context distributions, which gives composition (Proposed)
    • THEORY-094 — Word vectors approximate PMI in low dimensions, and relations appear as clean directions despite a noisy fit, because text behaves as if generated by a slowly drifting discourse over isotropic word vectors (Proposed) — also extends THEORY-093
  • THEORY-092 — Raising context counts to the 3/4 power helps word representations because PMI overweights rare contexts, and smoothing lowers their association scores (Proposed)

training-optimization

From Under the standard parametrization a learning rate that stays stable as width grows leaves the network training as a kernel, and µP is the parametrization that escapes this by updating every layer maximally

• THEORY-076 — Under the standard parametrization a learning rate that stays stable as width grows leaves the network training as a kernel, and µP is the parametrization that escapes this by updating every layer maximally (Active)
  • THEORY-037 — Residual-branch depth sets the depth scaling rule, because a branch with two transformations has a second-order update term that a branch with one does not (Active)

From Neural networks are singular, so effective complexity is how the volume of near-optimal parameters scales, not how curved the optimum is

• THEORY-075 — Neural networks are singular, so effective complexity is how the volume of near-optimal parameters scales, not how curved the optimum is (Active)
  • THEORY-073 — Training passes through stages marked by changes in loss-landscape degeneracy, and the loss curve does not show them (Proposed)
  • THEORY-074 — A phase transition of the Bayesian posterior and a transition in the training trajectory are different events, and only the first is well defined (Proposed)