Topics — The twenty-eight kinds of claim this anthology files against — the primary axis of both the practice registry and the reading list, and the same twenty-eight for each (ADR-026). The filing rule is the point: take a domain topic when the claim is ABOUT the domain; a claim merely discovered in one still takes its kind. The axis organizes the record, it does not bound it: content the list cannot place is evidence the list is short (ADR-059).
Model architecture — architecture patterns, component design, structural choices, model families — how the network is shaped, not how many signals it takes in.
54 of 449 SOTA documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| SOTA-024 v2 | Keep key/value projections shared across heads while query projections remain separate | Shazeer (2019), LIT-024 — ARXIV-1911.02150. | Superseded — by SOTA-109 |
| SOTA-032 v5 | Put the layer normalization inside the residual block, before the sublayer | Xiong et al. (2020), LIT-114 — ARXIV-2002.04745. Pre-LN: normalize the input to each sublayer rather than the sum after it, so the gradients near the output are well behaved at initialization. | Active |
| SOTA-034 v6 | Use SwiGLU activation for transformers | Shazeer et al. (2020), LIT-030 — ARXIV-2002.05202. Contested at frontier scale in low precision: two groups now ship bounded replacements, on the grounds that SwiGLU’s unbounded factors produce activation outliers. | Active |
| SOTA-036 v2 | Train a decoder-only transformer on a broad web corpus with a fixed context and a single next-token objective | Brown et al. (2020), LIT-035 — ARXIV-2005.14165. | Active |
| SOTA-037 | LM in-context learning emerges at scale | Brown et al. (2020), LIT-035 — ARXIV-2005.14165. | Active |
| SOTA-039 | single cycle of cosine decay is sufficient lr schedule | Brown et al. (2020), LIT-035 — ARXIV-2005.14165. | Superseded — by SOTA-140; warmup-stable-decay matches it and leaves the token budget open; every report since 2024 in the record uses a stable-then-decay schedule |
| SOTA-095 | consider rewinding to earlier checkpoint and skipping a few batches to mitigate unusual loss spikes | Chowdhery et al. (2022), LIT-069 — ARXIV-2204.02311. | Active |
| SOTA-124 v5 | Repeat high-quality data freely when its epoch size exceeds the model’s memorization window | Falcon-LLM Team (2026), LIT-119 — the Falcon-H1-Tiny technical blogpost. Roughly 100–500 GT for a 7B model, scaling linearly; the authors call the understanding early. | Proposed |
| SOTA-125 v3 | At a fixed tiny parameter budget, spend parameters on depth and SSM state width before MLP width | Falcon-LLM Team (2026), LIT-119 — the Falcon-H1-Tiny technical blogpost. 24–27 layers × 512 hidden and a large SSM state dimension won every ablation at 90M; 50 layers gained MMLU but halved throughput. | Proposed |
| SOTA-133 | Replace fixed residual accumulation with learned attention over preceding layers | Kimi Team (2026), LIT-134 — Attention Residuals: a per-layer pseudo-query chooses which earlier layers to read, at O(d) parameters per layer; 1.25× compute advantage on scaling laws at 48B/1.4T, adopted in Kimi K3. | Active |
| SOTA-136 v4 | Widen the residual stream into several streams mixed by doubly-stochastic matrices (manifold-constrained hyper-connections) | Xie et al. (2025), LIT-140 — hyper-connections with the residual-mixing matrix constrained to be doubly stochastic, so identity mapping survives; 6.7% training overhead at n = 4 in large-scale training, shipped in DeepSeek-V4. | Proposed |
| SOTA-137 | Widen the residual stream into several streams with freely learned mixing (hyper-connections) | Zhu et al. (2024), LIT-141 — n parallel residual streams with learnable mixing; gains on dense and MoE pretraining, and an identity-mapping property lost that its successor restores. | Superseded — by SOTA-136; constraining the mixing to a doubly-stochastic manifold keeps the width and restores the identity mapping this gave up |
| SOTA-148 v3 | Balance mixture-of-experts load with a bias on the routing scores, not an auxiliary loss | Wang et al. (2024), LIT-171 — add a per-expert bias to the routing scores before the top-K decision and update it from that expert’s recent load, so balancing changes which experts are chosen without adding a gradient to the loss. Better balance and better quality than an auxiliary-loss control, and what DeepSeek-V3 runs at 671B. | Proposed |
| SOTA-149 | Build the sparse layers from many small experts plus an always-on shared one, not a few large ones | Dai et al. (2024), LIT-170 — split into mN smaller experts and activate mK of them so the router chooses from a far larger combination space at the same compute, and isolate a few always-on shared experts to hold the common knowledge every routed expert would otherwise learn separately. | Active |
| SOTA-150 v3 | Make the feed-forward layers a sparse mixture of experts once the model is large enough to be compute-bound | Shazeer et al. (2017), LIT-188 — route each token to a few of many feed-forward experts instead of running one dense feed-forward for every token, so total parameters and per-token compute stop being the same number. Measured against dense by LIT-170 seven years later: DeepSeekMoE 16B matches LLaMA2 7B at roughly 40% of the compute, 145B approaches the same team’s dense 67B at 28.5%. | Active |
| SOTA-157 v3 | Train the language model as a masked diffusion model rather than autoregressively | Nie et al. (2025), LIT-217 — hold the paradigm fixed (pretrain, then SFT) and swap only the factorization: a forward masking process and a reverse process predicting masked tokens, optimizing a likelihood lower bound. Competitive with LLaMA3 8B on in-context learning; past GPT-4o on reversal poem completion. Filed Proposed — one group, one model, 8B. | Proposed |
| SOTA-158 | Bound the activation’s output range when training in low precision | Jiang et al. (2026), LIT-200 — SwiGLU approximates x² for large positive inputs, which is where its expressive capacity comes from and what enlarges the output range and exacerbates outliers. Two groups independently concluded that is intolerable in low precision. What they agree on is the constraint; the functions they chose differ. | Proposed |
| SOTA-167 | Build linear-attention layers from the state-space view: expressive discretisation, complex-valued state, and a MIMO readout | Lahoti et al. (2026), LIT-165 — three changes derived from the SSM view: a more expressive recurrence from the discretisation, a complex-valued state update for state tracking, and a MIMO formulation that adds quality without decode latency. +1.8 downstream at 1.5B over Gated DeltaNet, and Mamba-2’s perplexity at half the state size. | Proposed |
| SOTA-169 | Widen the residual stream into several streams and constrain the mixing between them | Zhu et al. (2024) and three successors, LIT-141 / LIT-140 / LIT-151 / LIT-181 — replace the single residual stream with n parallel streams and learn the mixing. All four agree the width helps and that the mixing needs a constraint; they disagree about which one, and that disagreement is the two practices extending this rather than this itself. | Proposed |
| SOTA-178 | Give the recurrence vector-valued gating and in-context learning rates so it can track state a softmax layer provably cannot | Peng et al. (2025), LIT-173 — a generalised delta rule with vector-valued gating and in-context learning rates, at constant memory and constant time per token. A 2.9B model sets the 3B state of the art on multilingual tasks on dramatically fewer training tokens. The expressivity claim is the sharp one: it can recognise all regular languages, which under standard conjectures exceeds what a softmax stack in TC⁰ can do. | Proposed |
| SOTA-180 | Keep the routed experts’ weights in a down-projected latent space and spend the saved bandwidth on more of them | Elango et al. (2026), LIT-196 — down-project the token before the routed experts and keep their weights in that latent space, dispatching and aggregating there too, while routing and the shared experts stay at full width. Communication volume and weight-loading bandwidth both fall by d/d_l, and the saved budget buys more experts and more active per token. | Active |
| SOTA-190 v3 | Increase depth before any other dimension when scaling a transformer | Tay et al. (2021), LIT-052 — the DeepNarrow strategy. Small 16L matches T5-Base downstream at 60% of the parameters, 63% of the FLOPs and 40% faster; the limit is parallelism rather than quality. | Proposed |
| SOTA-209 | Initialize a mixture-of-experts model from a dense checkpoint rather than training it from scratch | Komatsuzaki et al. (2022), LIT-227 — seed the experts from a dense checkpoint you already paid for. Upcycled T5 and ViT models beat their dense counterparts at ~50% of the dense pretraining sunk cost, and beat sparse models trained from scratch on 100% of it: same architecture, same compute, different initialization. | Proposed |
| SOTA-253 | Add a new instance by reconstructing it, not by training on it | Sun et al. (2022), LIT-111 — ARXIV-2205.12257. Split what the system knows about a specific object from what it has learned in general. Build the object’s structure once by reconstruction; match against it with a network that never saw the object or its category. Adding an object is then a scan, not a training run. | Active |
| SOTA-254 v2 | Train the language model as a masked diffusion model when the corpus is fixed and the compute is not | Prabhudesai et al. (2025), LIT-442 — masked diffusion keeps extracting signal from a repeated corpus for roughly 512 epochs where autoregressive training stops at 32. Below a critical compute threshold, given in closed form from the unique-token count, autoregressive training is better and by a wide margin; above it, diffusion is better. The recommendation is the condition, not the objective. | Proposed |
| SOTA-262 v2 | Give each modality its own weights and let the streams attend jointly | Esser et al. (2024), LIT-449 — MMDiT gives the text and image streams separate projections and MLPs, then runs attention over the concatenated sequence so information flows both ways. It beats UViT and DiT at matched budget. Liang et al. (2024), LIT-483 — the same shape reached independently for autoregressive multi-modal LLMs, matching a dense baseline at 55.8% of the FLOPs, and ablated component by component: the feed-forward carries most of the benefit, the attention projections less, the layer norms none. | Active |
| SOTA-269 | Keep a knowledge base outside the weights; parameters are an expensive and lossy place to memorise facts | Lu et al. (2024), LIT-452 — fact capacity is linear in model size and saturates in epochs, and the extrapolation is damning: all of Wikidata would want about 1000B non-embedding parameters trained for 100 epochs. Derivable facts cost full price unless they happen to share direction and structure. Borgeaud et al., LIT-060, is the record’s evidence that the alternative works at scale. | Active |
| SOTA-277 | For a task that is a sequential composition, buy depth rather than width | Chen et al. (2024), LIT-464 — composing k functions costs a constant-depth decoder polynomially many parameters and a log k-layer one polylogarithmically. An exponential separation, proved unconditionally, and about what a shape can express rather than what a run will learn. | Proposed |
| SOTA-282 | Put every matrix and hidden state on the unit hypersphere, and make each block’s residual step size a learned per-dimension parameter | Proposed | |
| SOTA-287 | If you need to train without backpropagation, train each block to denoise a noisy label embedding conditioned on the raw input | Proposed | |
| SOTA-289 v2 | When the sampling budget is small, prefer uniform-state discrete diffusion with consistency distillation — masked diffusion cannot revise what it has already emitted | Proposed | |
| SOTA-297 | Scale parallel computation with learnable input transforms, not parameters, when inference memory is the binding constraint | Chen et al. (2025), LIT-486 — run the same weights over P learnably-prefixed copies of the input and learn the aggregation. Loss falls as if the parameters had grown by O(log P); a 1.6B model at P = 8 matches a 4.4B one on code. At batch size 1 that costs 22× less added memory and 6× less added latency than the parameter scaling it replaces — and P× the training FLOPs. | Proposed |
| SOTA-311 | Evaluate a search on tasks held out of its own fitness function, and report the gap against the tasks it selected on | No source states this; it is filed on a priori grounds with one instance. Cheng, Clark and Richardson (2025), LIT-493, searched 1,062 pretrained designs against a fitness function of average downstream accuracy, then reported the top five on nine benchmarks drawn from the same pool and chosen — per its own Table 16 — for largest standard deviation across the search population. The best discovered design beats the best seed by 0.03 points. | Active |
| SOTA-322 | Trade attention heads for depth: more heads conditions the attention block, and the layers you drop cost nothing | Saratchandran, Teney and Lucey (2025), LIT-527 — more heads lower the condition number of the attention block, which is part of what depth was buying. In the five configurations where the MLP width is held fixed, raising head count and cutting layers holds or improves accuracy at 29–53% fewer parameters — TNT-B goes 65.4M → 30.9M at identical Top-1. | Proposed |
| SOTA-326 | Add n-gram embedding memory as one table read early in the network, on top of the expert budget rather than in place of experts | Qiu et al. (2026), LIT-152 — Qwen3.8-Flash-Next’s ablations. One n-gram embedding table at layer 2 takes loss from 1.585 to 1.541 and the benchmark average from 45.44 to 47.94. Splitting the same table across two layers buys nothing, and no depth clearly wins. Trading experts for table parameters at a fixed total lowers loss slightly and moves no downstream benchmark. Grown on top of the budget, the table keeps lowering loss after downstream accuracy has flattened. Layer 2 is a systems choice: the host-memory prefetch overlaps layer 1. | Proposed |
| SOTA-328 | Train one nested model instead of a family of separately trained sizes, and extract the sizes you need from it | Devvrit, Kudugunta, Kusupati et al. (2023), LIT-546 — nest four FFN widths in one Transformer, train one width per step, and extract any size in between by choosing widths per layer. Up to 850M parameters, at the compute of training the four sizes separately, the largest matches its baseline and the smaller ones beat theirs. It pays only if you were going to train the family. Gemma 3n ships it. | Proposed |
| SOTA-332 | When an implicit representation will be supervised through its derivatives, use sine activations with the SIREN initialization rather than a ReLU network | Sitzmann et al. (2020), LIT-551 — a sine network’s derivative is another sine network, so gradients and Laplacians of the fit are well behaved where a ReLU network’s second derivative is zero. Initialize hidden weights U(±√(6/n)) and scale the first layer by ω₀ = 30 (tune it to the signal), or deep sine networks do not train. It fits images, video, SDFs and PDE solutions from derivative supervision. | Proposed |
| SOTA-334 | In a style-modulated generator, demodulate the convolution weights instead of instance-normalizing the activations | Karras et al. (2019), LIT-560 — AdaIN’s instance normalization discards relative feature magnitudes, so the generator smuggles scale past it with a localized spike, the “water droplet” in every StyleGAN image. Scale each output channel’s modulated weights by 1/√(Σ w’² + ε) instead. It gives the same expected unit variance and style mixing still works. The artifact disappears and FID is unchanged (4.40 → 4.39). | Proposed |
| SOTA-335 | When generated content must move continuously, as in video or animation, build the generator alias-free: treat features as continuous signals and low-pass filter around every nonlinearity | Karras et al. (2021), LIT-559 — fine detail in GAN output sticks to pixel coordinates because the generator aliases. Apply each nonlinearity at 2× temporary resolution and low-pass filter back, start from Fourier features rather than a learned constant, remove per-pixel noise, and handle boundaries properly. On FFHQ-U, translation equivariance goes to 63 dB (and rotation to 40 dB for the -R variant) at StyleGAN2’s FID. | Proposed |
| SOTA-336 | Train high-resolution GANs at a fixed topology with output skips in the generator and a residual discriminator, not by progressive growing | Karras et al. (2019), LIT-560 — progressive growing makes each resolution briefly the output, so it learns output-level detail and leaves features stuck to pixel positions. A fixed generator that sums upsampled RGB outputs from every resolution, with a residual discriminator, keeps the coarse-to-fine emphasis without changing topology. FFHQ FID 4.34 → 3.31, and the phase artifacts are gone. | Proposed |
| SOTA-349 | Before crediting a graph transformer with long-range gains, retune the message-passing baselines under the same budget, including an MLP prediction head, positional encodings and input feature normalization | Tönshoff et al. (2023), LIT-579 — the Long-Range Graph Benchmark’s case that graph transformers are needed rested partly on untuned message-passing baselines. With a basic sweep inside the same 500k-parameter budget, GCN beats GPS on both Peptides tasks. Most of the gain is from replacing a linear prediction head with a 2-layer MLP. Normalizing input features lifts every model on the superpixel tasks, where GPS still leads. Tune both sides before attributing a gap to attention. | Active |
| SOTA-350 | Compare GNN architectures over many random data splits and seeds, with one shared training and hyperparameter-selection procedure for every model, never on a single fixed split | Shchur et al. (2018), LIT-581 — on the Planetoid split GAT beats GCN. On another random split of the same size GCN wins, and averaged over 100 splits × 20 seeds with identical training and tuning, GCN ranks first of four GNNs. A single split selects the model that overfits it, and per-model recipes measure the recipe. Report mean and spread over many splits and seeds, with the same tuning budget for every model. | Active |
| SOTA-351 | In a message-passing GNN for graph-level tasks, aggregate neighbours by sum with an MLP update, not by mean or max, when graph structure rather than node features carries the signal | Xu et al. (2019), LIT-580 — mean and max aggregation cannot count neighbors, so graphs that differ only in counts look the same to them. On featureless Reddit graphs, mean-aggregation GNNs are at chance (50.0, 20.0) and sum–MLP scores 92.4 and 57.5. With informative node features the choice matters much less, and results are mostly within noise. Use h = MLP((1 + ε)h + Σ neighbors) with ε = 0 and a sum readout. | Active |
| SOTA-358 v3 | Drop the domain inductive bias once pre-training data is large enough, and keep it when it is not | Active | |
| SOTA-362 | Train through a projection head and then discard it, taking the layer before as the representation | Active | |
| SOTA-365 v2 | Learn without negatives by breaking the symmetry: a predictor on one branch and a stop-gradient on the other | Active | |
| SOTA-400 | Append a few register tokens to a ViT so it does not commandeer background patches for scratch space | Active | |
| SOTA-403 v3 | Share the attention parameters across layers if you need to cut parameters; do not share the feed-forward ones | Lan et al. (2019), LIT-668. A transformer block has two halves and they are not equally compressible across depth. On an ALBERT-base configuration at E = 128, sharing every layer’s attention parameters scores 81.7 average against 81.6 for no sharing while removing 89M parameters to 64M; sharing the feed-forward parameters instead scores 80.2. At E = 768 the same split reads −0.7 against −2.8. One paper, 2019, encoder-only, and nobody has re-run it. | Proposed |
| SOTA-411 | To let a model spend more compute at inference without more parameters, train a depth-recurrent core on a randomly sampled iteration count, re-injecting the input at every iteration | Geiping et al. (NeurIPS 2025), LIT-683. Put a small core block between a short prelude and coda. At each training step sample how many times to iterate it, from a heavy-tailed distribution with mean 32. Backpropagate through the last few iterations only. Feed the input embedding into every iteration, and start the state from noise. Trained this way, accuracy rises with iterations at test time and saturates per task. A model trained at fixed depth does not gain this. Nothing yet shows it beats a standard transformer of the same FLOPs. | Proposed |
| SOTA-418 | To drop the tokenizer without losing to BPE, use a two-stage hierarchy with learned chunk boundaries and SSM encoder and decoder layers, not an isotropic byte model | Hwang, Wang and Gu (2025), LIT-679. A byte-level model that must not lose to a BPE Transformer at matched data and FLOPs should be hierarchical. Isotropic byte stacks lose by about 9 downstream points. It should use Mamba-2 layers for the encoder and decoder around the main network, and it should learn its chunk boundaries across two stages. One learned stage only ties BPE and whitespace chunking. Expect about 2× slower training. | Proposed |
| SOTA-419 | Under a fixed compute budget, change the architecture only where it cuts step time at constant parameter count | Geiping and Goldstein (2023), LIT-681. Under a 24-hour, one-GPU budget, every reshape tried (depth 4 to 24, deep-narrow, wide, funnel, FFN-every-k, shared layers) finishes within about 0.1 of the same MLM loss. A smaller model’s extra throughput cancels its slower per-token learning. So the architecture changes that pay are the ones that make each step cheaper without changing how much the model learns per token: dropping biases, predicting only masked tokens, and pre-norm for the learning rate it permits. | Proposed |
| SOTA-431 | Make every layer of a diffusion denoiser preserve activation magnitude, and force each weight vector back to fixed norm after every step | Karras et al. (2023), LIT-714 — EDM2. Normalize each output channel’s weight vector on use, re-normalize the stored weights after every step, scale every fixed operation to preserve unit magnitude, and drop the group norms. On the ADM U-Net at ImageNet-512 this takes FID from 6.96 to 2.56 at equal compute, with on-use weight normalization alone worth 6.96 → 3.75. One architecture family; untested on transformers. | Proposed |
| SOTA-434 | In a vision backbone, carry position with a relative bias inside the attention window and do not add an absolute position embedding on top | Liu et al. (2021), LIT-723, Table 4. One architecture, one recipe, three tasks. A relative position bias inside the attention window is best or tied-best on all three. An absolute position embedding is worth +0.4 ImageNet top-1 and -0.2 COCO box AP and -0.6 ADE20K mIoU, and adding it on top of the relative bias costs 2.1 mIoU while changing ImageNet nothing. The recommendation is the sign, not the size: this is the one design choice in the table that a classification benchmark scores backwards. | Proposed |
| SOTA-436 | Route a very large or non-grid input through cross-attention into a small learned latent array, so that depth no longer scales with input size | Jaegle et al. (2021), LIT-747 — the Perceiver. A learned array of 512 latents cross-attends to all 50,176 ImageNet pixels, at a cost of O(MN) rather than O(M²), and a 48-block latent transformer then runs at a cost independent of input size. It matched ResNet-50 and ViT-B/16 at 78.0% top-1 without convolutions, did not beat them, and ran unchanged on audio, video and point clouds. The case for doing it is cost and modality-independence, not accuracy, and the cost advantage is argued from complexity, not measured against a baseline. | Proposed |