anthology-of-the-sota

Documents awaiting a decision

26 min read

Graph View

Generated by luria index from the record’s frontmatter — built, not edited.

Proposed and Deferred both describe an open question. Neither says since when, and that is the signal: a decision proposed last week is pending; the same one a year later was either overdue or settled in code and never written back.

275 document(s) awaiting a decision.

SOTAs

181 of the 275.

Open sinceStatusCodeCitedUnack.Title
2026-09-05ProposedSOTA-124372Repeat high-quality data freely when its epoch size exceeds the model’s memorization window
2026-09-05ProposedSOTA-130280Skip the reasoning SFT stage and run RL with verifiable rewards directly on the base model
2026-09-05ProposedSOTA-136250Widen the residual stream into several streams mixed by doubly-stochastic matrices (manifold-constrained hyper-connections)
2026-09-05ProposedSOTA-144250Extend µP’s transfer to depth with CompleteP so one sweep serves deeper models too
2026-09-05ProposedSOTA-125170At a fixed tiny parameter budget, spend parameters on depth and SSM state width before MLP width
2026-09-05ProposedSOTA-122160Attach learnable per-row and per-column multipliers to weight matrices so their norms are learned, not set by LR and WD
2026-09-05ProposedSOTA-141110Decay the learning rate linearly all the way to zero
2026-09-07ProposedSOTA-157280Train the language model as a masked diffusion model rather than autoregressively
2026-09-07ProposedSOTA-155170Train across poorly connected islands: many inner steps per worker, an outer momentum optimizer over the deltas, and a streamed synchronisation
2026-09-07ProposedSOTA-156130Train without a learning-rate schedule: average the iterates so no stopping time need be fixed
2026-09-07ProposedSOTA-146100Correct the GRPO objective before running it — the published form has three independently identified defects
2026-09-07ProposedSOTA-14880Balance mixture-of-experts load with a bias on the routing scores, not an auxiliary loss
2026-09-07ProposedSOTA-15240Pack training documents by best fit instead of concatenating and splitting
2026-09-08ProposedSOTA-173190Augment the objective to make multi-epoch pretraining productive on a fixed corpus
2026-09-08ProposedSOTA-160150Treat the pretraining token budget and the post-training quantization plan as one decision, not two
2026-09-08ProposedSOTA-166150Set the pretraining data proportions by fitting a mixing law on small runs, not by argument
2026-09-08ProposedSOTA-169120Widen the residual stream into several streams and constrain the mixing between them
2026-09-08ProposedSOTA-178120Give the recurrence vector-valued gating and in-context learning rates so it can track state a softmax layer provably cannot
2026-09-08ProposedSOTA-172110Build synthetic pretraining data by editing human text at the token level, not by generating from scratch
2026-09-08ProposedSOTA-158100Bound the activation’s output range when training in low precision
2026-09-08ProposedSOTA-159100Combine µP with unit scaling so the hyperparameters decouple and FP8 needs no loss scaling
2026-09-08ProposedSOTA-16890Run Adam in Shampoo’s eigenbasis (SOAP) instead of Shampoo itself
2026-09-08ProposedSOTA-16780Build linear-attention layers from the state-space view: expressive discretisation, complex-valued state, and a MIMO readout
2026-09-08ProposedSOTA-17780Decouple the erase address from the write address in a delta-rule recurrence
2026-09-08ProposedSOTA-17960Truncate the rotary encoding’s low frequencies rather than rescaling its base
2026-09-08ProposedSOTA-17650Hybridise inside the layer: long-term slots, a sliding window, and one softmax over both, with the window size as the only dial
2026-09-08ProposedSOTA-17500Mask whole syntactic units for code fill-in-the-middle, not random character spans
2026-09-09ProposedSOTA-190150Increase depth before any other dimension when scaling a transformer
2026-09-09ProposedSOTA-191100Consider removing LayerNorm’s learnable gain and bias rather than tuning them
2026-09-09ProposedSOTA-18920Set the checkpoint interval from the job’s measured time-to-failure, which shrinks as the job grows
2026-09-10ProposedSOTA-20440Anneal a discretisation from coarse to fine over training rather than fixing it
2026-09-10ProposedSOTA-19310Lower Adam’s second-moment decay when the loss spikes, before reaching for the learning rate
2026-09-15ProposedSOTA-213140Control evolution-strategies drift with a larger population or an anchor penalty, not by stopping training early
2026-09-15ProposedSOTA-211130Spend one fitness evaluation per evolution-strategies direction, not an antithetic pair
2026-09-15ProposedSOTA-217120Align the hidden-unit permutation before averaging weights from separately trained networks
2026-09-15ProposedSOTA-21270Post-train by scoring many random weight perturbations in one parallel pass and majority-voting the best of them
2026-09-15ProposedSOTA-20960Initialize a mixture-of-experts model from a dense checkpoint rather than training it from scratch
2026-09-15ProposedSOTA-21630Switch from minibatch SGD to local SGD at the first learning-rate decay, rather than choosing between them
2026-09-15ProposedSOTA-21520Freeze Adam’s variance term after a warmup before compressing its update, and never compress the update directly
2026-09-15ProposedSOTA-22000Aggregate worker gradients coordinate-wise by median when any worker may be faulty
2026-09-16ProposedSOTA-22250Overlap the synchronisation with one or two local steps instead of communicating every step
2026-09-16ProposedSOTA-22330Do not keep the gossip topology static: sample a fresh random neighbourhood every round
2026-09-16ProposedSOTA-22930Scale the draft model’s training data, once nothing constrains it to predict the target’s features
2026-09-17ProposedSOTA-243130Discard the easy examples when data is abundant and the hard ones when it is scarce
2026-09-17ProposedSOTA-24680Find mislabelled training data by self-influence, not by training loss
2026-09-17ProposedSOTA-24150Rank candidate data selections with a model-free distributional proxy before spending a training run
2026-09-17ProposedSOTA-24240Select pretraining data by matching a target distribution you can sample from, not by scoring it for quality
2026-09-17ProposedSOTA-23430Train in the target low-bit format from scratch rather than quantizing a finished model
2026-09-17ProposedSOTA-23920Order training data by skill prerequisite: teach the prerequisite and the dependent skill costs less data
2026-09-17ProposedSOTA-24420Score example importance a few epochs into training, averaged over several initializations
2026-09-17ProposedSOTA-24520State a relation in both orders in the training corpus if you want it usable in both directions
2026-09-17ProposedSOTA-23210Convert an autoregressive model into a diffusion model by continual pretraining rather than training one from scratch
2026-09-17ProposedSOTA-24710Add a multi-token word to a frozen model’s vocabulary from the model’s own detokenized representation of it
2026-09-17ProposedSOTA-24810Stretch and clip the softmax so an attention head can output exact zeros
2026-09-19ProposedSOTA-254120Train the language model as a masked diffusion model when the corpus is fixed and the compute is not
2026-09-19ProposedSOTA-255110Tune weight decay upward when pretraining over a repeated corpus, rather than inheriting the customary value
2026-09-19ProposedSOTA-25160Train at low resolution and raise it only during the decay phase of a warmup-stable-decay schedule
2026-09-19ProposedSOTA-25760Spend surplus compute on an ensemble of independently seeded models and distil it, rather than on one larger model
2026-09-19ProposedSOTA-25650Judge a monotone scaling recipe by its asymptote, not by its loss at a compute budget
2026-09-20ProposedSOTA-26470Fix the noise schedule’s endpoints against the bound and choose its shape to minimise the loss estimator’s variance
2026-09-20ProposedSOTA-26050Use the smallest batch size that still saturates the device, and do not gradient-accumulate to avoid it
2026-09-20ProposedSOTA-26150Set AdamW’s weight decay by targeting a timescale that follows a power law in tokens-per-parameter
2026-09-20ProposedSOTA-27250Expect the loss to be non-monotone at the step size that trains fastest, and do not set the step size from a curvature bound
2026-09-20ProposedSOTA-27450Before adopting a spectral optimizer, measure the stable rank of each block’s incoming activations against its gradient’s nuclear rank
2026-09-20ProposedSOTA-27550Pick the depth parameterization from the residual branch: a Transformer branch has more than one transformation, so it needs the stricter rule
2026-09-20ProposedSOTA-26830Do not transfer a data mixing ratio across model scales; the threshold where a domain starts being learned moves with size
2026-09-20ProposedSOTA-26720Warm up on a subset of the entities before training on all of them, rather than holding one distribution fixed throughout
2026-09-20ProposedSOTA-27120Train on a second modality even when the target is single-modality
2026-09-20ProposedSOTA-27620For a normalized or preconditioned optimizer, width-depth muP is width muP plus a hidden residual multiplier of order 1/L
2026-09-20ProposedSOTA-25910Hold Adam’s second-moment half-life fixed in tokens when the batch size changes, not beta_2
2026-09-20ProposedSOTA-27310Choose a pretraining context whose association with the input is neither too strong nor too weak, and mix contexts to get there
2026-09-21ProposedSOTA-30280Steer a distilled generator by modulating its input noise, not by fine-tuning its weights
2026-09-21ProposedSOTA-30170Apply the objective gradient before the denoiser, not after it, when guiding diffusion toward a task objective
2026-09-21ProposedSOTA-30970Score a control interface by the mutual information between the operator’s command and the state change it induces, and choose the horizon of that state change deliberately
2026-09-21ProposedSOTA-28260Put every matrix and hidden state on the unit hypersphere, and make each block’s residual step size a learned per-dimension parameter
2026-09-21ProposedSOTA-28860Late in training, replicate a small subset of the weights, train the copies against shared base weights, and average the subset back into one model
2026-09-21ProposedSOTA-29560Parameterize the generating prompt above the word list, and constrain how each sample opens
2026-09-21ProposedSOTA-28950When the sampling budget is small, prefer uniform-state discrete diffusion with consistency distillation — masked diffusion cannot revise what it has already emitted
2026-09-21ProposedSOTA-29850Measure a writing assistant by the outcome it was deployed to improve, not by the text features that used to predict it
2026-09-21ProposedSOTA-28540Put dropout at the start of training if the model underfits, and at the end if it overfits — the schedule decides the sign, not the rate
2026-09-21ProposedSOTA-28640To find a model that generalizes badly where its outputs look fine, measure how much of parameter space behaves the way it does
2026-09-21ProposedSOTA-30340Name the phase a time-saving tool accelerates before predicting what it will do to the quality of what people produce with it
2026-09-21ProposedSOTA-30440Generate a long program unit by unit, freezing each unit that passes an execution-based check, instead of regenerating the whole artifact on failure
2026-09-21ProposedSOTA-27730For a task that is a sequential composition, buy depth rather than width
2026-09-21ProposedSOTA-28330To fine-tune a model that is already quantized, bank the part of each update that is smaller than the lattice spacing, and rebuild the accumulator from seeds rather than storing it
2026-09-21ProposedSOTA-28430Choose the checkpoint you post-train from by how much mass it puts on good rare responses, not by validation cross-entropy
2026-09-21ProposedSOTA-29920Finetune the whole generative model, encoder and decoder, for a dense perception task instead of attaching a head to a backbone
2026-09-21ProposedSOTA-29010To measure what a deployed recommender does, use sock-puppets that never interact, follow sets curated to your question, and exposure weighted by rank
2026-09-21ProposedSOTA-29210Check industry ties yourself when reading a literature about deployed platforms — disclosure is unreliable and the ties are mechanically detectable
2026-09-21ProposedSOTA-29410Fit the tokenizer to the corpus when the corpus is deliberately narrow
2026-09-21ProposedSOTA-29610Add the parallel streams in a short final training stage, not from the start
2026-09-21ProposedSOTA-29710Scale parallel computation with learnable input transforms, not parameters, when inference memory is the binding constraint
2026-09-21ProposedSOTA-28700If you need to train without backpropagation, train each block to denoise a noisy label embedding conditioned on the raw input
2026-09-21ProposedSOTA-29300Count embedding parameters when you report a tiny model’s size
2026-09-21ProposedSOTA-30000To tell whether a model is doing non-trivial in-context computation, measure how well it predicts its own hidden states, not its next-token loss
2026-09-21ProposedSOTA-31600When you are memory-bound rather than time-bound, train all the parameters a block at a time, partitioned on layer boundaries
2026-09-22ProposedSOTA-32560When the training loss has saturated and runs still differ, measure loss-landscape degeneracy rather than curvature
2026-09-22ProposedSOTA-32350Test a claim about what pretraining produces on a model trained with the pretraining objective, not one trained on the task family you are testing
2026-09-22ProposedSOTA-32750Train retrieval embeddings with nested losses, so a prefix of the vector can shortlist and the full vector re-rank
2026-09-22ProposedSOTA-32950For short-output tasks, adapt a frozen model with a low-rank intervention on a few positions of the residual stream, not a weight adapter; do not use it for long chain-of-thought
2026-09-22ProposedSOTA-32640Add n-gram embedding memory as one table read early in the network, on top of the expert budget rather than in place of experts
2026-09-22ProposedSOTA-32840Train one nested model instead of a family of separately trained sizes, and extract the sizes you need from it
2026-09-22ProposedSOTA-31930Reparameterize every linear layer by its spectral norm with a learned scalar
2026-09-22ProposedSOTA-32030Bound the learning rate by the ratio of the update’s spectral norm to the weight’s, and drop warmup
2026-09-22ProposedSOTA-32430Separate rival accounts of an internal algorithm by convergence rate and conditioning, not by how well each fits the output
2026-09-22ProposedSOTA-32110Verify a model’s lineage from the singular spectra of its attention products, which survive permutation and rescaling
2026-09-22ProposedSOTA-32200Trade attention heads for depth: more heads conditions the attention block, and the layers you drop cost nothing
2026-09-23ProposedSOTA-333410For autoregressive generation of continuous sequences, train with an independent noise level per token and condition the rollout on slightly noised history
2026-09-23ProposedSOTA-33880Train the GAN discriminator on frozen multi-scale pretrained features with fixed random channel and scale mixing
2026-09-23ProposedSOTA-35350Train a head that predicts the model’s own accuracy on each output, against the metric you care about, and use it to rank samples and filter predictions
2026-09-23ProposedSOTA-33240When an implicit representation will be supervised through its derivatives, use sine activations with the SIREN initialization rather than a ReLU network
2026-09-23ProposedSOTA-33540When generated content must move continuously, as in video or animation, build the generator alias-free: treat features as continuous signals and low-pass filter around every nonlinearity
2026-09-23ProposedSOTA-33640Train high-resolution GANs at a fixed topology with output skips in the generator and a residual discriminator, not by progressive growing
2026-09-23ProposedSOTA-37540When training with sampled negatives, draw them from the unigram distribution raised to the 3/4 power, not from the unigram or uniform distribution
2026-09-23ProposedSOTA-33430In a style-modulated generator, demodulate the convolution weights instead of instance-normalizing the activations
2026-09-23ProposedSOTA-34230Report a sparse autoencoder’s fidelity as downstream loss in compute-equivalent terms, not as fraction of loss recovered against zero ablation
2026-09-23ProposedSOTA-34430Train sparse autoencoders with a TopK activation instead of an L1 penalty, and prevent dead latents with transposed-decoder initialization and an auxiliary loss
2026-09-23ProposedSOTA-34730To write many facts into a model’s weights, update a range of early-to-mid MLP layers jointly in one batched solve (MEMIT), choosing the range by measured edit success rather than causal tracing
2026-09-23ProposedSOTA-35430When unlabelled inputs vastly outnumber labelled ones, retrain from scratch on the labelled data mixed with the model’s own high-confidence predictions for unlabelled inputs, under augmentation that stops it copying them
2026-09-23ProposedSOTA-37830Compare, project and orthogonalize a language model’s concept directions after whitening by the unembedding covariance, not by raw cosine
2026-09-23ProposedSOTA-33920For unpaired image-to-image translation of appearance, constrain both mappings with a cycle-consistency loss
2026-09-23ProposedSOTA-34320Read a transformer’s intermediate-layer predictions through a tuned lens, not the raw logit lens
2026-09-23ProposedSOTA-34620Retune the training timestep shift for each autoencoder latent space, and never rank latent spaces for generation under one shared schedule
2026-09-23ProposedSOTA-34010Expect weight-only 4-bit quantization to speed up batched serving only while the batch keeps the matmul memory-bound, and use a kernel built to stay there
2026-09-23ProposedSOTA-34110Condition an image generator on each training image’s original size instead of discarding or upsampling small images
2026-09-23ProposedSOTA-35210When a structure predictor is made generative, add a regression model’s predictions to the training data so disordered regions are learned as disorder rather than invented structure
2026-09-23ProposedSOTA-37110Score topic coherence automatically as the average pointwise mutual information of a topic’s top-word pairs, estimated on a large external reference corpus
2026-09-24ProposedSOTA-386360Show a video diffusion model images before and alongside video
2026-09-24ProposedSOTA-390270In a video diffusion transformer, attend over space and time jointly rather than factorizing, and budget for the cost
2026-09-24ProposedSOTA-389260Caption training video with a model that watches the video, not with captions of its frames
2026-09-24ProposedSOTA-392150To get a one-step sampler from a rectified flow, distil it; treat reflow as an optional extra pass, and keep the pre-reflow model for many steps
2026-09-24ProposedSOTA-394110When distilling a causal few-step video generator, take the teacher bidirectional, not causal
2026-09-24ProposedSOTA-39580Post-train an autoregressive video generator on its own rollouts, and score the whole rollout
2026-09-24ProposedSOTA-39770To condition an unconditionally trained diffusion model on known frames, use reconstruction guidance, not replacement
2026-09-24ProposedSOTA-39850Pretrain a vision transformer on packed, aspect-preserved images at sampled resolutions, not on fixed square crops
2026-09-24ProposedSOTA-39930Choose pre-training weight decay, dropout, clipping and precision for the quantization you intend to ship
2026-09-24ProposedSOTA-40120Anchor the patch Gram matrix to an early teacher when training long, because dense features decay while global metrics improve
2026-09-24ProposedSOTA-38500Tighten the gradient-norm clip as the model grows rather than carrying one value across scales
2026-09-24ProposedSOTA-38700Hold out a slice of the training set for model selection instead of selecting on the validation split
2026-09-24ProposedSOTA-39100Estimate attention resolution before training when choosing a position encoding, and measure it across the extrapolation boundary
2026-09-24ProposedSOTA-39300Freeze the image model’s spatial layers when adding time if checkpoint compatibility matters; do not expect a quality gain from it
2026-09-24ProposedSOTA-39600When a latent video model reuses an image autoencoder or upsampler, fine-tune them on video so they see time
2026-09-25ProposedSOTA-430430Before using an attribution map to debug a model or explain what it learned, check that the map changes when the weights are randomized and when the labels are permuted — do not validate it by how it looks
2026-09-25ProposedSOTA-428290Choose the EMA length after training: store two power-function averages at snapshots and reconstruct any length by least squares
2026-09-25ProposedSOTA-408120Average the weights along the tail of training under a cyclical or high constant learning rate, then re-estimate the normalization statistics
2026-09-25ProposedSOTA-415100During a warmup-stable-decay run, estimate the annealed score from a uniform average of recent stable-phase checkpoints instead of launching a decay branch
2026-09-25ProposedSOTA-40370Share the attention parameters across layers if you need to cut parameters; do not share the feed-forward ones
2026-09-25ProposedSOTA-42660When you merge and reinitialise a low-rank adapter mid-training, prune the optimizer state and re-warm the learning rate from zero — both, not either
2026-09-25ProposedSOTA-43350Guide a diffusion model with a smaller, less-trained copy of itself rather than an unconditional model, and make sure the two models are degraded in the same way
2026-09-25ProposedSOTA-41240When the target is likelihood, learn a per-dimension noise schedule conditioned on a learned latent; a scalar or unconditioned one buys nothing
2026-09-25ProposedSOTA-42230Rescale the noise schedule so the last timestep is pure noise, and train with v-prediction so the model can learn there
2026-09-25ProposedSOTA-42930Drop half the image patches when training a CLIP-style model, spend the saving on more pairs and a larger batch, and unmask only for a short final tune
2026-09-25ProposedSOTA-40220To teach a model to reason over facts it holds, spend added data on facts derived from other facts rather than on more atomic facts
2026-09-25ProposedSOTA-40420When you ablate an auxiliary loss, ablate the input construction that came with it, or the result is about both
2026-09-25ProposedSOTA-40520Scale the masked-language-modelling rate with model size rather than holding it at 15%
2026-09-25ProposedSOTA-41420To make an image diffusion model a dense geometric predictor, fine-tune it end to end as a one-step model at t = T with a task loss
2026-09-25ProposedSOTA-41820To drop the tokenizer without losing to BPE, use a two-stage hierarchy with learned chunk boundaries and SSM encoder and decoder layers, not an isotropic byte model
2026-09-25ProposedSOTA-41920Under a fixed compute budget, change the architecture only where it cuts step time at constant parameter count
2026-09-25ProposedSOTA-42020When distilling a classifier, match the teacher’s temperature-softened outputs at the same temperature and keep a down-weighted hard-label term
2026-09-25ProposedSOTA-42120Before fitting a compute-optimal allocation law, tune learning rate, batch size and AdamW beta2 per model size and keep warmup short of the smallest run
2026-09-25ProposedSOTA-42320When classifier-free guidance over-exposes a latent diffusion model, rescale the guided prediction toward the conditional prediction’s standard deviation
2026-09-25ProposedSOTA-42720In a batch-normalized network, put dropout after the last batch-norm layer, not upstream of one
2026-09-25ProposedSOTA-43120Make every layer of a diffusion denoiser preserve activation magnitude, and force each weight vector back to fixed norm after every step
2026-09-25ProposedSOTA-40610Replace every masked token with [MASK]; drop BERT’s 80-10-10 substitution rule
2026-09-25ProposedSOTA-41110To let a model spend more compute at inference without more parameters, train a depth-recurrent core on a randomly sampled iteration count, re-injecting the input at every iteration
2026-09-25ProposedSOTA-41710Filter pretraining text by how well the tokenizer compresses it
2026-09-26ProposedSOTA-43420In a vision backbone, carry position with a relative bias inside the attention window and do not add an absolute position embedding on top
2026-10-01ProposedSOTA-43620Route a very large or non-grid input through cross-attention into a small learned latent array, so that depth no longer scales with input size
2026-10-01ProposedSOTA-43820To pretrain a wider language model when a smaller pretrained one of the same depth exists, initialize it by function-preserving symmetric width expansion rather than at random
2026-10-01ProposedSOTA-44020Supervise a feed-forward 3D reconstructor with a frozen camera-controlled video model’s renderings instead of captured multi-view data
2026-10-01ProposedSOTA-44120When a web-scraped pair corpus is noisy, filter it by self-consistency: keep only the pairs whose passage a model trained on the noisy set ranks near the top
2026-10-01ProposedSOTA-43710Store and move model checkpoints losslessly compressed: code the float exponent as its own stream with an entropy coder alone, and store successive checkpoints as deltas against a periodic full base
2026-10-01ProposedSOTA-43900Test a Kronecker-factored curvature implementation against the cases where KFAC is exact before trusting it
2026-10-03ProposedSOTA-44260To RL-tune a flow-matching generator, roll it out as a same-marginal SDE built from its own velocity, and anchor each step with the closed-form KL to the reference
2026-10-03ProposedSOTA-44350Down-weight large residuals in consistency and MeanFlow training: use a Pseudo-Huber loss or an adaptive inverse-error weight, not plain squared L2
2026-10-03ProposedSOTA-44640In consistency training, take the target from the current weights under stop-gradient, not from an EMA teacher
2026-10-03ProposedSOTA-44430When a flow will be sampled with a coarse ODE solver, pair noise and data within each minibatch by exact optimal transport
2026-10-03ProposedSOTA-44730When distilling a large image or video model by continuous-time consistency, add a small DMD distribution-matching term
2026-10-03ProposedSOTA-44520When the base text-to-image diffusion model is better than the models that generated the preference pairs, train on the pairs with Diffusion-DPO rather than fine-tuning on the preferred images
2026-10-03ProposedSOTA-44820Without a pretrained teacher, train a one-step generator as a MeanFlow average-velocity model rather than by consistency training or a shortcut model
2026-10-03ProposedSOTA-44920When reward-tuning a classifier-free-guided diffusion model over several rounds of sampling, train the guided prediction at the fixed sampling guidance weight, not the conditional prediction alone

THEORYs

80 of the 275.

Open sinceStatusCodeCitedUnack.Title
2026-09-15ProposedTHEORY-006270Task-improving weight perturbations are dense around pretrained weights, and denser the larger the model
2026-09-15ProposedTHEORY-00790Fine-tuning landscapes are low-dimensional in curvature, and improving directions are degenerate rather than unique
2026-09-15ProposedTHEORY-01090Most of the loss barrier between two independently trained networks is permutation, not disagreement
2026-09-15ProposedTHEORY-00270A dense network contains a sparse subnetwork that matches its accuracy when trained from the same initialization
2026-09-15ProposedTHEORY-00540Dense feed-forward layers are already mixtures of experts, and pre-training settles the partition before the neurons
2026-09-16ProposedTHEORY-01280How far a batch-size scaling heuristic transfers is a property of the optimizer, not of the heuristic
2026-09-16ProposedTHEORY-01430The outer optimizer is what buys the inner step count
2026-09-17ProposedTHEORY-02320Depth-dependent computation in a transformer falls into four stages, of which detokenization is the first
2026-09-19ProposedTHEORY-024360Orthogonalising the update is dualising it, and muP and Shampoo are two partial approximations of the same duality map
2026-09-19ProposedTHEORY-02570A transformer holds about 3.6 bits per parameter, and generalization begins where the data outgrows that budget
2026-09-20ProposedTHEORY-032200A spectral update wins where the incoming activations are low stable rank and the gradient spectrum is spread out
2026-09-20ProposedTHEORY-033140What a spectral optimizer buys is a step size that stays optimal, not adherence to a target geometry
2026-09-20ProposedTHEORY-030130Adaptive optimizers work by shaping the curvature they adapt to, and oscillation is how a first-order method sees curvature at all
2026-09-20ProposedTHEORY-034110Concepts sit in embedding space as intersections of half-spaces, so inclusion, intersection and union are geometric operations
2026-09-20ProposedTHEORY-02650Critical batch size is set by how much data has been seen, not by how large the model is
2026-09-20ProposedTHEORY-03640Representations converge across architectures, objectives and modalities, and the endpoint is a model of what generated the data
2026-09-20ProposedTHEORY-02930A model of bounded capacity allocates it across datasets like a knapsack, so the optimum jumps rather than sliding
2026-09-21ProposedTHEORY-041110Left unconstrained, a transformer’s matrices drift into a badly conditioned, rank-deficient shape, and the norms are the part nothing was managing
2026-09-21ProposedTHEORY-059110A low-rank branch corrects quantization because weight spectra are steep and quantization-error spectra are flat
2026-09-21ProposedTHEORY-06090Calibration error and refinement error have separate minimizers during training, so the loss minimum is optimal for neither
2026-09-21ProposedTHEORY-05180A text feature predicts success partly by being uncommon, so a tool that gives it to everyone destroys its predictive value
2026-09-21ProposedTHEORY-04540How much of parameter space behaves like your trained network is a description length, and the better-generalizing network occupies more of it
2026-09-21ProposedTHEORY-05240Synthesis requires equivariant representations and discriminative pretraining rewards invariant ones, which is why a generative prior groups objects it was never shown
2026-09-21ProposedTHEORY-05740A corpus a model generates is narrower than the distribution it imitates, and the narrowing shows up as n-gram over-concentration
2026-09-21ProposedTHEORY-05840Apparent compression in the information plane is saturating activations collapsing into extreme bins, not information being discarded
2026-09-21ProposedTHEORY-05030Parameters carry memorization and parallel computation carries reasoning
2026-09-21ProposedTHEORY-05430The regularizer that is intractable in data space is tractable in noise space, and bounds the data-space divergence
2026-09-21ProposedTHEORY-05630Distinct state machines agree at chance on long random inputs, so a learner that reads only statistics loses the signal as the sequences get longer
2026-09-21ProposedTHEORY-04420Dropout early in training buys a biased gradient estimate with much less directional variance, and the trade is favourable until it is not
2026-09-21ProposedTHEORY-04310Cross-entropy pays an unbounded price for mass a learner never had, and that price grows with sequence length — which is why it drifts from what post-training needs
2026-09-21ProposedTHEORY-04710Personalizing a feed concentrates what it shows rather than broadening it, and a handful of moderate signals is enough to start
2026-09-21ProposedTHEORY-04910Modalities compete for the same feed-forward parameters, the competition is not symmetric between them, and that is where separating by modality pays
2026-09-21ProposedTHEORY-05310The Stein posterior-mean denoiser acts as an approximate projection onto the learned data geometry
2026-09-21ProposedTHEORY-05510A producer’s own rate of return is the price of their time, so making them faster raises the bar on every marginal hour of improvement
2026-09-22ProposedTHEORY-071160A memorising and a generalising circuit compete on logits per unit norm, and which one wins flips when the data makes memorising more expensive
2026-09-22ProposedTHEORY-072150Training and test loss disagree as functions of weight norm, and shrinking the norm through the gap is what takes so long
2026-09-22ProposedTHEORY-062140What crashes a transformer is spectral energy concentrating in the query-key product, not low attention entropy as such
2026-09-22ProposedTHEORY-070130Grokking is the transition from lazy to rich training dynamics
2026-09-22ProposedTHEORY-061120Attention entropy is bounded below by a quantity falling exponentially in the spectral norm of the query-key product
2026-09-22ProposedTHEORY-06650Pure self-attention collapses every token onto one, and skip connections prevent it by keeping short paths alive
2026-09-22ProposedTHEORY-07450A phase transition of the Bayesian posterior and a transition in the training trajectory are different events, and only the first is well defined
2026-09-22ProposedTHEORY-06340Contextual concept directions sit in the low-variance tail of the unembedding spectrum while vocabulary contrasts sit at the top
2026-09-22ProposedTHEORY-06540Self-attention drives tokens into a few clusters, and the spectrum of the value matrix decides the geometry they land in
2026-09-22ProposedTHEORY-06430Transformers are biased toward low-sensitivity functions, and that bias is what their robustness is made of
2026-09-22ProposedTHEORY-07330Training passes through stages marked by changes in loss-landscape degeneracy, and the loss curve does not show them
2026-09-23ProposedTHEORY-089120Word vectors trained on co-occurrence behave linearly because they encode log co-occurrence statistics: differences carry log ratios, which gives analogies, and sums carry products of context distributions, which gives composition
2026-09-23ProposedTHEORY-090110A concept’s shared word-pair direction, its linear probe and its steering vector are one object, related by an inner product under which causally separable concepts are orthogonal
2026-09-23ProposedTHEORY-081100A transformer’s feed-forward layers work as key-value memories: each hidden unit’s input weights match input patterns, and its output weights promote the tokens that follow them
2026-09-23DeferredTHEORY-087100Why a negative-free siamese network avoids collapse is unsettled, and the three accounts contradict each other
2026-09-23ProposedTHEORY-08080A diffusion process needs noise to keep its intermediate states on the data manifold, and blur helps only while noise still dominates the bands it removes
2026-09-23ProposedTHEORY-08870Masked prediction transfers across modalities only after the masking ratio is rescaled to the signal’s information density
2026-09-23ProposedTHEORY-07840Deep sine networks train when initialization holds every pre-activation near a standard normal, which keeps each layer arcsine-distributed and frequency growth slow; that initialization is what earlier periodic networks lacked
2026-09-23ProposedTHEORY-09240Raising context counts to the 3/4 power helps word representations because PMI overweights rare contexts, and smoothing lowers their association scores
2026-09-23ProposedTHEORY-09130Command-to-outcome mutual information scores an interface because it measures how reliably the operator’s input determines what happens at the chosen horizon, which rewards a learnable channel rather than an intuitive one
2026-09-23ProposedTHEORY-08420AlphaFold 2 needs the sequence alignment to find a protein’s coarse fold, not to refine it, so accuracy collapses below a minimum alignment depth and barely improves above it
2026-09-23ProposedTHEORY-08610A text document is better modelled as a mixture of several topics, each a distribution over words, than as a draw from a single topic
2026-09-23ProposedTHEORY-09410Word vectors approximate PMI in low dimensions, and relations appear as clean directions despite a noisy fit, because text behaves as if generated by a slowly drifting discourse over isotropic word vectors
2026-09-24ProposedTHEORY-09740Attention is an approximation of Kanerva’s sparse distributed memory, which is why it needs normalized vectors and a fitted temperature
2026-09-24ProposedTHEORY-10040Activation outliers are a product of pre-training optimization choices, not an emergent property of scale
2026-09-24ProposedTHEORY-09630A Pre-LN residual stream’s per-layer change decays as one over root k, so deep blocks cannot refine the representation
2026-09-24ProposedTHEORY-09930Muon’s update matches the outer-product structure of associative memories, which is why it learns tail classes that Adam under-trains
2026-09-25ProposedTHEORY-109130Classifier guidance may flatter classifier-based metrics because it steps along a classifier gradient, and guidance without a classifier is the control
2026-09-25ProposedTHEORY-10370A transformer stores a fact where it needed it, so two-hop composition generalizes only to facts it already saw as a second hop
2026-09-25ProposedTHEORY-10470A large guidance scale amplifies the model’s derivatives, which narrows a high-order solver’s convergence radius
2026-09-25ProposedTHEORY-11260Classifier-free guidance improves image quality because its unconditional reference model is worse, not because it emphasizes the class, so guidance is adaptive truncation toward the better-fit density
2026-09-25ProposedTHEORY-11050FID is not neutral between fidelity and coverage: it weights coverage more, so optimizing it walks toward variety at the cost of per-sample quality
2026-09-25ProposedTHEORY-10840Kaplan’s and Chinchilla’s compute-optimal exponents differ because of how small-scale runs were counted, warmed up and tuned, not because of the learning-rate decay
2026-09-25ProposedTHEORY-10510Soft targets transfer the teacher’s similarity structure over wrong classes, which carries more information per example than a hard label
2026-09-26ProposedTHEORY-113200A gradient attribution survives weight randomization because of its input multiplier, not because the attribution is insensitive to the weights
2026-10-01ProposedTHEORY-11740A separator token condenses the content of the segment it closes, rather than only receiving attention a head has nowhere else to put
2026-10-01ProposedTHEORY-11600Gradient descent on a factorized model is biased toward low rank, not small norm: there are problems where it sends every norm to infinity to lower the rank
2026-10-03ProposedTHEORY-11830Continuous-time consistency training is unstable through the time derivative in its tangent, and tangent normalization, which MeanFlow’s adaptive loss weight already is, controls it
2026-10-03ProposedTHEORY-12430Consistency training is a bootstrapped fixed-point iteration, not descent on a loss, so a target that lags the student swamps a data signal that enters only at the order of the step
2026-10-03ProposedTHEORY-12520One affine autoregressive flow block makes every conditional a single Gaussian and cannot approximate an arbitrary density, while two blocks in opposite orders make every conditional but the last a Gaussian mixture and three make all of them one
2026-10-03ProposedTHEORY-12010A likelihood-trained normalizing flow samples well only when its training data carry Gaussian noise well above the quantization width, because its inverse must be well conditioned over the whole Gaussian latent
2026-10-03ProposedTHEORY-12100Distribution-matching distillation loses coverage because it descends a reverse divergence on the student’s own samples, and a consistency term on teacher trajectories restores it
2026-10-06ProposedTHEORY-12610A more expressive architecture scales better by lowering the data coefficient rather than the exponent, because it can learn more of the discrete tasks in language-model data from the same tokens
2026-10-06ProposedTHEORY-12700QK normalization costs long-context extension because bounding the logits raises attention entropy and removes the sink a head uses to discard mass
2026-10-06ProposedTHEORY-12800A delta-rule recurrence does not accumulate small per-step perturbations, because each write overwrites the state along the current key, so injected error plateaus and an impulse fades far faster than the forget gate implies
2026-10-06ProposedTHEORY-12900Interleaving attention with a negative-eigenvalue delta-rule recurrence expresses problems neither layer type can express alone, because state tracking and recall come from different layers and one alternation composes them

LITs

11 of the 275.

Open sinceStatusCodeCitedUnack.Title
2026-08-24ProposedLIT-10130AdaNorm: Adaptive Gradient Norm Correction based Optimizer for CNNs
2026-09-15DeferredLIT-36420Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeed
2026-09-25DeferredLIT-71730Your Brain on ChatGPT: Accumulation of Cognitive Debt when Using an AI Assistant for Essay Writing Task
2026-09-25DeferredLIT-69420Rigor with Machine Learning from Field Theory to the Poincaré Conjecture
2026-09-25DeferredLIT-69520Language Design as Information Renormalization
2026-09-25DeferredLIT-69720AgentSociety: Large-Scale Simulation of LLM-Driven Generative Agents Advances Understanding of Human Behaviors and Society
2026-09-25DeferredLIT-71520Surya: Foundation Model for Heliophysics
2026-09-25DeferredLIT-69810Topos and Stacks of Deep Neural Networks
2026-09-25DeferredLIT-71610The Impact of Generative AI on Social Media: An Experimental Study
2026-09-25DeferredLIT-71810Automated decision-making as domination
2026-09-25DeferredLIT-71910Discovering cognitive strategies with tiny recurrent neural networks

ADRs

3 of the 275.

Open sinceStatusCodeCitedUnack.Title
2026-09-05DeferredADR-01200A practice declares its altitude, and one with no body cannot claim to be a design decision
2026-09-17ProposedADR-045100A claim about the signal is admissible however it was measured, including on people
2026-09-23ProposedADR-05840A library is evidence about consensus, and its observation lives in a report

The citation count is the second axis, and it flips the priority: an old proposal nothing references is a stalled idea worth closing, while an old proposal many files cite is a decision the codebase has already made and hasn’t written down.

This count and the reference-status report’s will differ, and that is not an off-by-one. That report covers documents something actually cites and hasn’t acknowledged; this one covers every undecided document. One here is missing from there for exactly one of two reasons: nothing cites it, or every citation carries an inactive-ok annotation.

Cited nowhere at all (19): ADR-012, SOTA-175, SOTA-220, SOTA-287, SOTA-293, SOTA-300, SOTA-316, SOTA-322, SOTA-385, SOTA-387, SOTA-391, SOTA-393, SOTA-396, SOTA-439, THEORY-116, THEORY-121, THEORY-127, THEORY-128, THEORY-129 — these are the cheapest to close, since nothing depends on the answer.