anthology-of-the-sota

❯

❯

theory.d

Graph View

129 items under this folder.

Oct 19, 2026
THEORY-040: A metric that composes or thresholds per-token error turns a smooth capability curve into a sharp one, with nothing happening in the model
Oct 06, 2026
THEORY-126: A more expressive architecture scales better by lowering the data coefficient rather than the exponent, because it can learn more of the discrete tasks in language-model data from the same tokens
Oct 06, 2026
THEORY-127: QK normalization costs long-context extension because bounding the logits raises attention entropy and removes the sink a head uses to discard mass
Oct 06, 2026
THEORY-128: A delta-rule recurrence does not accumulate small per-step perturbations, because each write overwrites the state along the current key, so injected error plateaus and an impulse fades far faster than the forget gate implies
Oct 06, 2026
THEORY-129: Interleaving attention with a negative-eigenvalue delta-rule recurrence expresses problems neither layer type can express alone, because state tracking and recall come from different layers and one alternation composes them
Oct 03, 2026
THEORY-118: Continuous-time consistency training is unstable through the time derivative in its tangent, and tangent normalization, which MeanFlow's adaptive loss weight already is, controls it
Oct 03, 2026
THEORY-119: Consistency, shortcut, MeanFlow and progressive-distillation models all estimate the two-time flow map of the probability-flow ODE, and differ in which identity of that map they train on
Oct 03, 2026
THEORY-120: A likelihood-trained normalizing flow samples well only when its training data carry Gaussian noise well above the quantization width, because its inverse must be well conditioned over the whole Gaussian latent
- generative-modeling
- flows-and-transport
Oct 03, 2026
THEORY-121: Distribution-matching distillation loses coverage because it descends a reverse divergence on the student's own samples, and a consistency term on teacher trajectories restores it
- generative-modeling
- few-step-generation
Oct 03, 2026
THEORY-122: A diffusion or flow sampler can be trained by a likelihood-ratio policy gradient only when each step is stochastic with a tractable density, and a same-marginal SDE built from the trained velocity supplies that without changing the marginals
Oct 03, 2026
THEORY-123: A flow's sampling ODE curves because the noise-data pairs it was trained on cross and the learned velocity averages them, so a coupling whose straight paths do not cross gives a straight flow
Oct 03, 2026
THEORY-124: Consistency training is a bootstrapped fixed-point iteration, not descent on a loss, so a target that lags the student swamps a data signal that enters only at the order of the step
Oct 03, 2026
THEORY-125: One affine autoregressive flow block makes every conditional a single Gaussian and cannot approximate an arbitrary density, while two blocks in opposite orders make every conditional but the last a Gaussian mixture and three make all of them one
Oct 01, 2026
THEORY-116: Gradient descent on a factorized model is biased toward low rank, not small norm: there are problems where it sends every norm to infinity to lower the rank
Oct 01, 2026
THEORY-117: A separator token condenses the content of the segment it closes, rather than only receiving attention a head has nowhere else to put
- attention-techniques
- inference-optimization
Sep 26, 2026
THEORY-113: A gradient attribution survives weight randomization because of its input multiplier, not because the attribution is insensitive to the weights
- analysis-and-evaluation
- representation-and-encoding
Sep 26, 2026
THEORY-114: A modified-backpropagation rule that keeps only non-negative relevance produces a matrix chain that converges to rank 1, so the later layers can only flip the map's sign
- analysis-and-evaluation
- representation-and-encoding
Sep 26, 2026
THEORY-115: Guided backpropagation and DeconvNet do partial image recovery, not attribution: the backward ReLU plus a CNN's local connections reconstruct the input regardless of the class or the weights
- analysis-and-evaluation
- representation-and-encoding
Sep 25, 2026
THEORY-102: A causal mask implicitly encodes absolute position, because a token can count how many predecessors it may attend to
- representation-and-encoding
- model-architecture
Sep 25, 2026
THEORY-103: A transformer stores a fact where it needed it, so two-hop composition generalizes only to facts it already saw as a second hop
Sep 25, 2026
THEORY-104: A large guidance scale amplifies the model's derivatives, which narrows a high-order solver's convergence radius
Sep 25, 2026
THEORY-105: Soft targets transfer the teacher's similarity structure over wrong classes, which carries more information per example than a hard label
- inference-optimization
- training-optimization
Sep 25, 2026
THEORY-106: Gaussian flow matching is a diffusion model; what separates the named recipes is loss weighting, network output and sampling schedule
Sep 25, 2026
THEORY-107: A diffusion loss whose weighting falls monotonically with signal-to-noise is maximum likelihood on noise-augmented data; a non-monotone one is not a likelihood objective
- generative-modeling
- training-optimization
Sep 25, 2026
THEORY-108: Kaplan's and Chinchilla's compute-optimal exponents differ because of how small-scale runs were counted, warmed up and tuned, not because of the learning-rate decay
- training-optimization
- analysis-and-evaluation
Sep 25, 2026
THEORY-109: Classifier guidance may flatter classifier-based metrics because it steps along a classifier gradient, and guidance without a classifier is the control
- analysis-and-evaluation
- generative-modeling
Sep 25, 2026
THEORY-110: FID is not neutral between fidelity and coverage: it weights coverage more, so optimizing it walks toward variety at the cost of per-sample quality
- analysis-and-evaluation
- generative-modeling
Sep 25, 2026
THEORY-111: A hyperparameter can look important because it is entangled with an averaging length you froze, so sensitivity attributed to one knob may belong to the pair
- analysis-and-evaluation
- training-optimization
Sep 25, 2026
THEORY-112: Classifier-free guidance improves image quality because its unconditional reference model is worse, not because it emphasizes the class, so guidance is adaptive truncation toward the better-fit density
- generative-modeling
- analysis-and-evaluation
Sep 24, 2026
THEORY-096: A Pre-LN residual stream's per-layer change decays as one over root k, so deep blocks cannot refine the representation
- model-stability
- model-architecture
Sep 24, 2026
THEORY-097: Attention is an approximation of Kanerva's sparse distributed memory, which is why it needs normalized vectors and a fitted temperature
- attention-techniques
- model-architecture
Sep 24, 2026
THEORY-098: Softmax attention coefficients must disperse as the number of items grows, so no learned attention circuit stays sharp out of distribution
- attention-techniques
- model-architecture
Sep 24, 2026
THEORY-099: Muon's update matches the outer-product structure of associative memories, which is why it learns tail classes that Adam under-trains
- training-optimization
- signal-structure
Sep 24, 2026
THEORY-100: Activation outliers are a product of pre-training optimization choices, not an emergent property of scale
- numerics-and-precision
- capability-thresholds
Sep 24, 2026
THEORY-101: A large trained ViT repurposes redundant patch tokens as scratch space for global information, which is what its feature-map artifacts are
- representation-and-encoding
- model-architecture
Sep 23, 2026
THEORY-077: A coordinate MLP learns high frequencies slowly because its tangent kernel's spectrum falls off fast, and a sinusoidal input mapping works by making that kernel stationary with a bandwidth you choose
- representation-and-encoding
- analysis-and-evaluation
Sep 23, 2026
THEORY-078: Deep sine networks train when initialization holds every pre-activation near a standard normal, which keeps each layer arcsine-distributed and frequency growth slow; that initialization is what earlier periodic networks lacked
- model-stability
- model-architecture
Sep 23, 2026
THEORY-079: What makes a diffusion model generative is iterated restoration of any degradation, not Gaussian noise
- generative-modeling
- analysis-and-evaluation
Sep 23, 2026
THEORY-080: A diffusion process needs noise to keep its intermediate states on the data manifold, and blur helps only while noise still dominates the bands it removes
Sep 23, 2026
THEORY-081: A transformer's feed-forward layers work as key-value memories: each hidden unit's input weights match input patterns, and its output weights promote the tokens that follow them
- analysis-and-evaluation
- model-architecture
Sep 23, 2026
THEORY-082: Causal tracing shows where a language model stores a fact, and that is where the fact should be edited
- analysis-and-evaluation
- adaptation-and-tuning
Sep 23, 2026
THEORY-083: A message-passing GNN can distinguish no more graphs than the 1-WL test, and reaches that limit only if its neighbor aggregation is injective on multisets, as a sum followed by an MLP is and a mean or max is not
Sep 23, 2026
THEORY-084: AlphaFold 2 needs the sequence alignment to find a protein's coarse fold, not to refine it, so accuracy collapses below a minimum alignment depth and barely improves above it
- biomolecular-modeling
- analysis-and-evaluation
Sep 23, 2026
THEORY-085: InfoNCE maximizes a lower bound on mutual information that cannot exceed log N
- analysis-and-evaluation
- representation-and-encoding
Sep 23, 2026
THEORY-086: A text document is better modelled as a mixture of several topics, each a distribution over words, than as a draw from a single topic
- signal-structure
- generative-modeling
Sep 23, 2026
THEORY-087: Why a negative-free siamese network avoids collapse is unsettled, and the three accounts contradict each other
Sep 23, 2026
THEORY-088: Masked prediction transfers across modalities only after the masking ratio is rescaled to the signal's information density
Sep 23, 2026
THEORY-089: Word vectors trained on co-occurrence behave linearly because they encode log co-occurrence statistics: differences carry log ratios, which gives analogies, and sums carry products of context distributions, which gives composition
Sep 23, 2026
THEORY-090: A concept's shared word-pair direction, its linear probe and its steering vector are one object, related by an inner product under which causally separable concepts are orthogonal
- concept-geometry
- analysis-and-evaluation
Sep 23, 2026
THEORY-091: Command-to-outcome mutual information scores an interface because it measures how reliably the operator's input determines what happens at the chosen horizon, which rewards a learnable channel rather than an intuitive one
Sep 23, 2026
THEORY-092: Raising context counts to the 3/4 power helps word representations because PMI overweights rare contexts, and smoothing lowers their association scores
- signal-structure
- representation-and-encoding
Sep 23, 2026
THEORY-093: Skip-gram with negative sampling implicitly factorizes the word-context PMI matrix shifted by log k, weighting frequent pairs more heavily
- representation-and-encoding
- signal-structure
Sep 23, 2026
THEORY-094: Word vectors approximate PMI in low dimensions, and relations appear as clean directions despite a noisy fit, because text behaves as if generated by a slowly drifting discourse over isotropic word vectors
Sep 23, 2026
THEORY-095: No unbiased estimator of FID exists, so its plug-in bias can reverse a comparison while the sample variance stays small
Sep 22, 2026
THEORY-061: Attention entropy is bounded below by a quantity falling exponentially in the spectral norm of the query-key product
- model-stability
- attention-techniques
Sep 22, 2026
THEORY-062: What crashes a transformer is spectral energy concentrating in the query-key product, not low attention entropy as such
- model-stability
- attention-techniques
Sep 22, 2026
THEORY-063: Contextual concept directions sit in the low-variance tail of the unembedding spectrum while vocabulary contrasts sit at the top
Sep 22, 2026
THEORY-064: Transformers are biased toward low-sensitivity functions, and that bias is what their robustness is made of
- analysis-and-evaluation
- model-architecture
Sep 22, 2026
THEORY-065: Self-attention drives tokens into a few clusters, and the spectrum of the value matrix decides the geometry they land in
Sep 22, 2026
THEORY-066: Pure self-attention collapses every token onto one, and skip connections prevent it by keeping short paths alive
Sep 22, 2026
THEORY-067: A transformer can run a learning algorithm on a model held in its activations, and the architecture admits several, so a construction identifies none
Sep 22, 2026
THEORY-068: In-context learning is gradient descent on an implicit model
Sep 22, 2026
THEORY-069: Grokking is a regime rather than a property of algorithmic data, and at least three knobs move it
Sep 22, 2026
THEORY-070: Grokking is the transition from lazy to rich training dynamics
Sep 22, 2026
THEORY-071: A memorising and a generalising circuit compete on logits per unit norm, and which one wins flips when the data makes memorising more expensive
Sep 22, 2026
THEORY-072: Training and test loss disagree as functions of weight norm, and shrinking the norm through the gap is what takes so long
Sep 22, 2026
THEORY-073: Training passes through stages marked by changes in loss-landscape degeneracy, and the loss curve does not show them
Sep 22, 2026
THEORY-074: A phase transition of the Bayesian posterior and a transition in the training trajectory are different events, and only the first is well defined
Sep 22, 2026
THEORY-075: Neural networks are singular, so effective complexity is how the volume of near-optimal parameters scales, not how curved the optimum is
Sep 22, 2026
THEORY-076: Under the standard parametrization a learning rate that stays stable as width grows leaves the network training as a kernel, and µP is the parametrization that escapes this by updating every layer maximally
Sep 21, 2026
THEORY-038: A decoder cannot compose over a long context in few layers, because each position forgets what it forwarded
Sep 21, 2026
THEORY-039: A measured capability is the capability minus whatever the evaluation itself demands, and the gap is widest for the weakest model
- analysis-and-evaluation
Sep 21, 2026
THEORY-041: Left unconstrained, a transformer's matrices drift into a badly conditioned, rank-deficient shape, and the norms are the part nothing was managing
- model-stability
- model-architecture
Sep 21, 2026
THEORY-042: An update smaller than the lattice spacing is not merely rounded away — it is cancelled exactly, and carrying the remainder makes the discrete path shadow the continuous one within half a grid cell
- numerics-and-precision
- training-optimization
Sep 21, 2026
THEORY-043: Cross-entropy pays an unbounded price for mass a learner never had, and that price grows with sequence length — which is why it drifts from what post-training needs
- analysis-and-evaluation
- training-optimization
Sep 21, 2026
THEORY-044: Dropout early in training buys a biased gradient estimate with much less directional variance, and the trade is favourable until it is not
- model-stability
- training-optimization
Sep 21, 2026
THEORY-045: How much of parameter space behaves like your trained network is a description length, and the better-generalizing network occupies more of it
- analysis-and-evaluation
- model-stability
Sep 21, 2026
THEORY-046: Uniform-state discrete diffusion is the argmax of a Gaussian diffusion, which is why Gaussian technique transfers to it and not to masked diffusion
- generative-modeling
- model-architecture
Sep 21, 2026
THEORY-047: Personalizing a feed concentrates what it shows rather than broadening it, and a handful of moderate signals is enough to start
- deployment-and-society
- analysis-and-evaluation
Sep 21, 2026
THEORY-048: Randomizing individuals estimates an individual quantity, and when the units interact the collective effect is not the sum of it — so a null from such a trial is not evidence of no effect
- deployment-and-society
- analysis-and-evaluation
Sep 21, 2026
THEORY-049: Modalities compete for the same feed-forward parameters, the competition is not symmetric between them, and that is where separating by modality pays
- multimodal-learning
- training-optimization
Sep 21, 2026
THEORY-050: Parameters carry memorization and parallel computation carries reasoning
- analysis-and-evaluation
- model-architecture
Sep 21, 2026
THEORY-051: A text feature predicts success partly by being uncommon, so a tool that gives it to everyone destroys its predictive value
- deployment-and-society
- analysis-and-evaluation
Sep 21, 2026
THEORY-052: Synthesis requires equivariant representations and discriminative pretraining rewards invariant ones, which is why a generative prior groups objects it was never shown
- representation-and-encoding
- vision-and-graphics
Sep 21, 2026
THEORY-053: The Stein posterior-mean denoiser acts as an approximate projection onto the learned data geometry
- generative-modeling
- analysis-and-evaluation
Sep 21, 2026
THEORY-054: The regularizer that is intractable in data space is tractable in noise space, and bounds the data-space divergence
- generative-modeling
- analysis-and-evaluation
Sep 21, 2026
THEORY-055: A producer's own rate of return is the price of their time, so making them faster raises the bar on every marginal hour of improvement
- deployment-and-society
- analysis-and-evaluation
Sep 21, 2026
THEORY-056: Distinct state machines agree at chance on long random inputs, so a learner that reads only statistics loses the signal as the sequences get longer
Sep 21, 2026
THEORY-057: A corpus a model generates is narrower than the distribution it imitates, and the narrowing shows up as n-gram over-concentration
Sep 21, 2026
THEORY-058: Apparent compression in the information plane is saturating activations collapsing into extreme bins, not information being discarded
- analysis-and-evaluation
- representation-and-encoding
Sep 21, 2026
THEORY-059: A low-rank branch corrects quantization because weight spectra are steep and quantization-error spectra are flat
- numerics-and-precision
- inference-optimization
Sep 21, 2026
THEORY-060: Calibration error and refinement error have separate minimizers during training, so the loss minimum is optimal for neither
- analysis-and-evaluation
- training-optimization
Sep 20, 2026
THEORY-026: Critical batch size is set by how much data has been seen, not by how large the model is
- training-optimization
Sep 20, 2026
THEORY-027: In continuous time the diffusion bound depends on the noise schedule only through its endpoints
- generative-modeling
Sep 20, 2026
THEORY-028: The plateau before factual recall is the formation of the attention circuit that recall needs
- training-optimization
Sep 20, 2026
THEORY-029: A model of bounded capacity allocates it across datasets like a knapsack, so the optimum jumps rather than sliding
- training-optimization
Sep 20, 2026
THEORY-030: Adaptive optimizers work by shaping the curvature they adapt to, and oscillation is how a first-order method sees curvature at all
- training-optimization
Sep 20, 2026
THEORY-031: The aggregate loss curve is a lossy projection of training, in at least three measured ways
- analysis-and-evaluation
Sep 20, 2026
THEORY-032: A spectral update wins where the incoming activations are low stable rank and the gradient spectrum is spread out
- training-optimization
- signal-structure
Sep 20, 2026
THEORY-033: What a spectral optimizer buys is a step size that stays optimal, not adherence to a target geometry
- training-optimization
Sep 20, 2026
THEORY-034: Concepts sit in embedding space as intersections of half-spaces, so inclusion, intersection and union are geometric operations
- concept-geometry
- representation-and-encoding
Sep 20, 2026
THEORY-035: Gradient descent drives the sharpness up to the largest value its own step size tolerates, and then trains there
- training-optimization
Sep 20, 2026
THEORY-036: Representations converge across architectures, objectives and modalities, and the endpoint is a model of what generated the data
Sep 20, 2026
THEORY-037: Residual-branch depth sets the depth scaling rule, because a branch with two transformations has a second-order update term that a branch with one does not
- training-optimization
Sep 19, 2026
THEORY-024: Orthogonalising the update is dualising it, and muP and Shampoo are two partial approximations of the same duality map
- training-optimization
Sep 19, 2026
THEORY-025: A transformer holds about 3.6 bits per parameter, and generalization begins where the data outgrows that budget
- analysis-and-evaluation
Sep 17, 2026
THEORY-015: Dropout regularizes by imposing a data-dependent penalty, not an isotropic one
- model-stability
Sep 17, 2026
THEORY-016: Test-time weight scaling computes a geometric-mean average over the dropout ensemble, exactly for logistic units and to second order elsewhere
- model-stability
Sep 17, 2026
THEORY-017: An influence-function estimate predicts the effect of removing a training point and retraining
- analysis-and-evaluation
Sep 17, 2026
THEORY-018: Influence functions on neural networks estimate the proximal Bregman response function, not leave-one-out retraining
- analysis-and-evaluation
Sep 17, 2026
THEORY-019: A softmax head with nothing to attend to must place its mass somewhere, so models learn a positional sink
- attention-techniques
- numerics-and-precision
Sep 17, 2026
THEORY-020: Layer importance is not uniform with depth: middle layers tolerate deletion and reordering, the first and last do not
- model-architecture
Sep 17, 2026
THEORY-021: A model builds a latent vocabulary in its early layers whose units are not the tokenizer's
- representation-and-encoding
Sep 17, 2026
THEORY-022: A phrase placed in a word slot is read as a lemma, and the reading comes from the construction rather than the phrase's frequency
- signal-structure
- representation-and-encoding
Sep 17, 2026
THEORY-023: Depth-dependent computation in a transformer falls into four stages, of which detokenization is the first
- analysis-and-evaluation
Sep 16, 2026
THEORY-012: How far a batch-size scaling heuristic transfers is a property of the optimizer, not of the heuristic
- training-optimization
Sep 16, 2026
THEORY-013: The SGD noise scale g = eps*N/B sets generalization, so the optimal batch size grows with the training set
- training-optimization
Sep 16, 2026
THEORY-014: The outer optimizer is what buys the inner step count
- distributed-optimization
Sep 15, 2026
THEORY-001: Batch normalization works by reducing internal covariate shift
- model-stability
Sep 15, 2026
THEORY-002: A dense network contains a sparse subnetwork that matches its accuracy when trained from the same initialization
- analysis-and-evaluation
Sep 15, 2026
THEORY-003: Batch normalization helps by smoothing the optimization landscape, not by reducing internal covariate shift
- model-stability
Sep 15, 2026
THEORY-004: A lottery ticket wins by re-learning the solution its dense run already found
- analysis-and-evaluation
Sep 15, 2026
THEORY-005: Dense feed-forward layers are already mixtures of experts, and pre-training settles the partition before the neurons
- model-architecture
Sep 15, 2026
THEORY-006: Task-improving weight perturbations are dense around pretrained weights, and denser the larger the model
- analysis-and-evaluation
Sep 15, 2026
THEORY-007: Fine-tuning landscapes are low-dimensional in curvature, and improving directions are degenerate rather than unique
- analysis-and-evaluation
Sep 15, 2026
THEORY-008: An evolution-strategies update is mostly a loss-invariant random walk whose size grows with steps and shrinks with population
- analysis-and-evaluation
Sep 15, 2026
THEORY-009: A wide two-layer network's training dynamics are a gradient flow on the distribution of its neurons, and that flow is convex
- analysis-and-evaluation
Sep 15, 2026
THEORY-010: Most of the loss barrier between two independently trained networks is permutation, not disagreement
- model-stability
Sep 15, 2026
THEORY-011: Skip connections make a deep network trainable by smoothing the loss surface, not by making it more expressive
- model-stability

Created with Quartz v5.0.0 © 2026

Repository
Quartz