Topics — The twenty-eight kinds of claim this anthology files against — the primary axis of both the practice registry and the reading list, and the same twenty-eight for each (ADR-026). The filing rule is the point: take a domain topic when the claim is ABOUT the domain; a claim merely discovered in one still takes its kind. The axis organizes the record, it does not bound it: content the list cannot place is evidence the list is short (ADR-059).
Model architecture — architecture patterns, component design, structural choices, model families — how the network is shaped, not how many signals it takes in.
144 of 737 LIT documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| LIT-008 v2 | Attention Is All You Need | Vaswani et al. (2017), ARXIV-1706.03762. Introduces transformer architecture. | Active |
| LIT-020 v4 | Pay Less Attention with Lightweight and Dynamic Convolutions | Wu et al. (2019), ARXIV-1901.10430. Alternative to self-attention. | Active |
| LIT-021 v3 | Mish: A Self Regularized Non-Monotonic Activation Function | Misra et al. (2019), ARXIV-1908.08681. New activation function. | Active |
| LIT-023 v3 | Root Mean Square Layer Normalization | Zhang et al. (2019), ARXIV-1910.07467. Simplified layer normalization. | Active |
| LIT-024 v2 | Fast Transformer Decoding: One Write-Head is All You Need | Shazeer (2019), ARXIV-1911.02150. Introduces multi-query attention. | Active |
| LIT-030 v2 | GLU Variants Improve Transformer | Shazeer et al. (2020), ARXIV-2002.05202. Study of gating mechanisms. | Active |
| LIT-035 v2 | Language Models are Few-Shot Learners | Brown et al. (2020), ARXIV-2005.14165. Introduces GPT-3. | Active |
| LIT-045 v2 | RoFormer: Enhanced Transformer with Rotary Position Embedding | Su et al. (2021), ARXIV-2104.09864. Rotation-based positional encoding. | Active |
| LIT-047 v3 | ReZero is All You Need: Fast Convergence at Large Depth | Bachlechner et al. (2020), ARXIV-2003.04887. Gate every residual branch with a single zero-initialized scalar: the network starts as the identity, satisfies dynamical isometry at initialization, and trains at depths where more elaborate schemes are needed otherwise. | Active |
| LIT-048 v4 | Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation | Press et al. (2021), ARXIV-2108.12409. Linear attention bias. | Active |
| LIT-052 v2 | Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers | Tay et al. (2021), ARXIV-2109.10686. Model shape, not just model size, determines downstream fine-tuning quality; the DeepNarrow strategy reaches T5-Base quality with 50% fewer parameters and 40% faster training. | Active |
| LIT-060 v2 | Improving Language Models by Retrieving from Trillions of Tokens | Borgeaud et al. (2021), ARXIV-2112.04426. Retrieval-augmented language models. | Active |
| LIT-061 v3 | Efficient Large Scale Language Modeling with Mixtures of Experts | Artetxe et al. (2021), ARXIV-2112.10684. “A detailed empirical study of how autoregressive MoE language models scale in comparison with dense models” — across in- and out-of-domain language modelling, zero- and few-shot priming, and full fine-tuning. | Active |
| LIT-069 | PaLM: Scaling Language Modeling with Pathways | Chowdhery et al. (2022), ARXIV-2204.02311. Pathways system architecture. | Active |
| LIT-071 v3 | CoCa: Contrastive Captioners are Image-Text Foundation Models | Yu et al. (2022), ARXIV-2205.01917. Unified vision-language architecture. | Active |
| LIT-080 v2 | PaLI: A Jointly-Scaled Multilingual Language-Image Model | Chen et al. (2022), ARXIV-2209.06794. Joint vision-language scaling. | Active |
| LIT-120 | Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance | Zuo et al. (2025), ARXIV-2507.22448. A parallel hybrid mixer — attention heads and Mamba heads side by side in one block — from 0.5B to 34B, with a µP recipe adapted to it and multi-epoch training on high-quality data; the architecture and the memorization-window measurement LIT-119 builds on. | Active |
| LIT-131 | Kimi K3: Open Frontier Intelligence | Kimi Team (2026), ARXIV-2607.24653. A 2.8T-parameter, 104B-active MoE with a 1M-token context, built from three architectural changes the team had published separately — a 69:24 layerwise hybrid of Kimi Delta Attention and gated MLA, Attention Residuals, and a 896-expert LatentMoE with 16 active — for a claimed 2.5× scaling efficiency over Kimi K2, still trained with MuonClip. | Active |
| LIT-134 | Attention Residuals | Kimi Team (2026), ARXIV-2603.15031. Replace the fixed, uniform residual sum with learned, input-dependent attention over preceding layers, computed from one pseudo-query per layer at O(d) parameters; a blocked form keeps it practical at scale, and at 48B parameters on 1.4T tokens it shows a 1.25× compute advantage on scaling laws for under 2% inference overhead. | Active |
| LIT-135 | Qwen3.8-27B | Qwen Team (2026), model card and release blog; no technical report. A 27B dense vision-language model with 48 Gated DeltaNet layers and 16 gated full-attention layers, an MTP draft head and a 262k native context; architecturally identical to Qwen3.6-27B four months earlier, with the generation’s gains coming from post-training on agentic and coding work. | Active |
| LIT-136 | Qwen3-Next: Towards Ultimate Training & Inference Efficiency | Qwen Team (2025), release blog; no paper. An 80B-total, 3B-active MoE that first shipped the 3:1 Gated DeltaNet to gated-attention layout, native multi-token prediction, and zero-centered weight-decayed layernorm; trained on a 15T-token subset for 9.3% of Qwen3-32B’s cost while matching it, with over 10× the throughput past 32k tokens. | Active |
| LIT-137 v2 | Gated Delta Networks: Improving Mamba2 with Delta Rule | Yang et al. (2024), ARXIV-2412.06464. Gating erases memory fast and the delta rule writes to it precisely; combining them in one recurrence, with a chunkwise parallel algorithm for the hardware, beats both Mamba2 and DeltaNet on language modelling, retrieval, length extrapolation and long context at 1.3B/100B tokens, and the paper’s own hybrids with sliding-window attention or Mamba2 layers do better still. | Active |
| LIT-139 v2 | DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence | DeepSeek-AI (2026), ARXIV-2606.19348. Two MoE models — V4-Pro at 1.6T/49B and V4-Flash at 284B/13B — with a 1M-token context reached by a hybrid of compressed sparse attention and heavily compressed dense attention, manifold-constrained hyper-connections widening the residual stream, and Muon as the optimizer for most modules over 32–33T tokens; at 1M tokens V4-Pro needs an estimated 27% of V3.2’s per-token FLOPs and 10% of its KV cache. Multi-token prediction is carried over from V3 unchanged, as a training objective; the report does not mention speculative decoding. | Active |
| LIT-140 | mHC: Manifold-Constrained Hyper-Connections | Xie et al. (2025), ARXIV-2512.24880. Hyper-Connections widen the residual stream but break the identity mapping a residual guarantees, which destabilises deep stacks; constraining the residual mixing matrices to be doubly stochastic via Sinkhorn-Knopp restores it while keeping the width, at a reported 6.7% training-time overhead (n = 4, the 27B system result; the scaling study spans 3B, 9B and 27B) — the residual DeepSeek-V4 ships. | Active |
| LIT-141 | Hyper-Connections | Zhu et al. (2024), ARXIV-2409.19606. Replace the residual connection with several parallel streams and learnable mixing between them, at expansion rate 2 or more, to escape the seesaw between vanishing gradients and representation collapse; gains on dense and MoE pretraining — and, per its successor, an identity-mapping property lost along the way. | Superseded — by LIT-140; mHC restores the identity mapping that free hyper-connection mixing gave up, and scales where this did not |
| LIT-151 | oHC: Orthogonal Hyper-Connections on SO(4) via Quaternions | Guo et al. (2026), ARXIV-2609.02672. The doubly-stochastic constraint mHC puts on the residual-mixing matrix bounds it from above but not below, so the mixing can only shrink what distinguishes the streams and the widened residual homogenises with depth; constraining the matrix to the rotation group instead bounds it on both sides, and at four streams a pair of unit quaternions gives it in closed form for no extra parameters. | Active |
| LIT-152 v2 | On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability | Qiu et al. (2026), ARXIV-2608.30320. The ablation record behind a Qwen3.8-generation architecture, and the first place the record’s three residual replacements meet in one harness: Gated Residual, mHC and Attention Residuals come out level, separated by inference cost rather than quality. Also the source for judging every architectural change on loss, cost and stability at once. | Active |
| LIT-160 | DeepSeek-V3 Technical Report | DeepSeek-AI (2024), ARXIV-2412.19437. 671B total, 37B activated, 14.8T tokens, 2.788M H800 hours — and no irrecoverable loss spike and no rollback in the entire run, which is the claim worth carrying. | Active |
| LIT-161 v2 | Mamba: Linear-Time Sequence Modeling with Selective State Spaces | Gu and Dao (2023), ARXIV-2312.00752. Make the SSM parameters functions of the input and the model can decide what to remember; a hardware-aware recurrent kernel keeps it fast without convolutions. | Active |
| LIT-162 v2 | Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality | Dao and Gu (2024), ARXIV-2405.21060. SSMs and attention variants are two decompositions of the same class of structured semiseparable matrices; the duality yields Mamba-2, whose core layer is 2–8× faster. | Active |
| LIT-165 v2 | Mamba-3: Improved Sequence Modeling using State Space Principles | Lahoti et al. (2026), ARXIV-2603.15569. Three changes from an inference-first view — a more expressive discretised recurrence, a complex-valued state update for state tracking, and a MIMO formulation that buys accuracy without decode latency. | Active |
| LIT-170 | DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models | Dai et al. (2024), ARXIV-2401.06066. Two changes to top-K routing that make experts specialise: cut them finer so the activated combination is more expressive, and isolate a few shared experts to absorb the common knowledge the routed ones were duplicating. | Active |
| LIT-171 | Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts | Wang et al. (2024), ARXIV-2408.15664. Balance expert load with a bias added to the routing scores and updated from recent load, instead of with an auxiliary loss whose gradients interfere with the objective you care about. | Active |
| LIT-173 | RWKV-7 “Goose” with Expressive Dynamic State Evolution | Peng et al. (2025), ARXIV-2503.14456. A generalised delta rule with vector-valued gating and in-context learning rates — provably able to track state and recognise all regular languages, which Transformers under standard conjectures cannot. | Active |
| LIT-179 | The Llama 3 Herd of Models | Grattafiori et al. (2024), ARXIV-2407.21783. The period’s most-cited dense recipe, published at length: a 405B dense Transformer at 128K context, with the empirical evaluation and the compositional route to image, video and speech. | Active |
| LIT-181 v2 | Beyond the Birkhoff Polytope: Spectral-Sphere-Constrained Hyper-Connections | Liu et al. (2026), ARXIV-2603.20896. Three named defects in mHC’s doubly-stochastic constraint — identity degeneration, an expressivity bottleneck from non-negativity, and parameterization cost — answered by moving the feasible set to a spectral-norm sphere. | Active |
| LIT-182 | Qwen3 Technical Report | Yang et al. (2025), ARXIV-2505.09388. Thinking and non-thinking modes unified in one model with a thinking budget the caller sets, plus flagship-to-small distillation; 0.6B to 235B, dense and MoE, 119 languages. | Active |
| LIT-183 | Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning | NVIDIA (2025), ARXIV-2512.20848. A 30B-A3B MoE hybrid Mamba-Transformer on 25T tokens: more accurate than its predecessor at under half the activated parameters, and up to 3.3× the inference throughput of comparable open models. | Active |
| LIT-187 v2 | GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding | Lepikhin et al. (2020), ARXIV-2006.16668. Puts the mixture-of-experts layer into a transformer and shards it across 2048 TPUs with annotations rather than a rewrite: a 600B multilingual translation model trained in four days, and the top-2 routing that later work names as the thing to fix. | Active |
| LIT-188 | Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer | Shazeer et al. (2017), ARXIV-1701.06538. The sparsely-gated mixture-of-experts layer: thousands of feed-forward experts with a trainable gate selecting a few per example, so model capacity can grow by 1000× at small increases in compute. The paper conditional computation had been promising in theory and not delivering in practice until. | Active |
| LIT-189 | Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity | Fedus et al. (2021), ARXIV-2101.03961. Route each token to exactly one expert instead of two: simpler, cheaper per token, and stable in bfloat16 with selective casting — a 1.6T-parameter model and a 7× pretraining speedup over T5-Base at matched compute. | Active |
| LIT-196 | LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Experts | Elango et al., NVIDIA (2026), ARXIV-2601.18089. Existing MoE designs are argued from sparsity and tuned for offline throughput, which is the wrong objective for interactive serving where expert computation is bandwidth- bound, not compute-bound. LatentMoE down-projects tokens into a narrow latent space before routing and keeps the routed experts’ weights there, so both the dispatch traffic and the weight-loading bandwidth fall by the width ratio. Explored to 95B parameters over 1T tokens; adopted by Nemotron-3 Super and Ultra, and by Kimi K3. | Active |
| LIT-199 | Language Modeling with Gated Convolutional Networks | Dauphin et al. (2016), ARXIV-1612.08083. Where the gated linear unit comes from. Its argument is about gradients, not expressivity: an LSTM-style tanh gate multiplies two downscaling factors into the gradient, which vanishes as layers stack, while gating a linear unit leaves a path with no downscaling — a multiplicative skip connection. State of the art on WikiText-103, and the first non-recurrent model competitive with strong recurrent ones at that scale. | Active |
| LIT-200 v3 | PowLU: An Activation Function for Stable Pre-Training of LLMs | Jiang et al. (2026), ARXIV-2605.25704. For large positive inputs SwiGLU approximates x², and that quadratic amplification is what enlarges the output range and produces outliers, particularly in low-precision training. PowLU replaces it with a rational power function giving adaptive nonlinearity and bounded growth. Scaling-law experiments plus 7.9B and 124B Ling models against SwiGLU, and against SwiGLU-Clip at 7.9B only. | Active |
| LIT-226 | MoEfication: Transformer Feed-forward Layers are Mixtures of Experts | Zhang et al. (2021), ARXIV-2110.01786. Most inputs activate only a tiny fraction of an FFN’s neurons, and the neurons that fire together can be partitioned into experts with a router bolted on afterwards — the same parameters, conditionally used. 10–30% of FFN parameters per input retains over 95% of performance. | Active |
| LIT-227 | Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints | Komatsuzaki et al. (2022), ARXIV-2212.05055. Initialize a mixture-of-experts model from a dense checkpoint instead of from scratch. Upcycled T5 and ViT models beat their dense counterparts using ~50% of the dense pretraining sunk cost, and beat MoE models trained from scratch on 100% of it. | Active |
| LIT-380 | The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits | Ma et al. (2024), ARXIV-2402.17764. Every weight is ternary, {-1, 0, +1}, and the model is trained that way from scratch rather than quantized afterwards. From 3B upward it matches an FP16 LLaMA of the same size and token budget on perplexity and end-task accuracy, while the matrix multiply reduces to integer addition — a compute paradigm rather than a compression ratio. | Active |
| LIT-384 | VGGT: Visual Geometry Grounded Transformer | Wang et al. (2025), ARXIV-2503.11651. One feed-forward pass infers camera parameters, point maps, depth maps and 3D point tracks together, from one view or hundreds, in under a second — beating methods that post-process with geometry optimisation. The claim that matters is that the specialisation into separate tasks was never necessary. | Active |
| LIT-406 | Qwen2.5 Technical Report | Qwen Team (2024), ARXIV-2412.15115. The open-weight family this record measures against more than any other — forty-five documents here name it, and until now none of them could point at it. | Active |
| LIT-417 | The Remarkable Robustness of LLMs: Stages of Inference? | Lad et al. (2024), ARXIV-2406.19384. Deleting a layer outright, or swapping two adjacent ones, leaves 72-95% of top-1 predictions unchanged with no fine-tuning — but only in the middle. Interventions on the first and last layers degrade badly, and swapping hurts less than dropping. The paper reads that localized sensitivity as four stages of inference, with the question mark in its own title: detokenization, feature engineering, prediction ensembling, residual sharpening. The record files the measurement and the framework as separate claims because their evidence is not the same strength. | Active |
| LIT-425 v2 | Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer | Raffel et al. (2019), ARXIV-1910.10683. T5’s paper, and the origin of C4 — the heuristically filtered Common Crawl slice that forty-six documents here train on. Filed for the corpus: the record cites C4 constantly and had no document behind it. | Active |
| LIT-428 | Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention | Katharopoulos et al. (2020), ARXIV-2006.16236. Where linear attention comes from, and the identity the record argues about in twelve documents without being able to cite it: drop the softmax for a kernel feature map, associativity reorders the product to O(N), and the autoregressive case is then literally an RNN. | Active |
| LIT-429 | Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence | Peng et al. (2024), ARXIV-2404.05892. RWKV-5 and RWKV-6, and the two changes the line is still built on: the state becomes matrix-valued rather than vector-valued, and the decay becomes a function of the input rather than a learned constant. | Active |
| LIT-430 | RWKV: Reinventing RNNs for the Transformer Era | Peng et al. (2023), ARXIV-2305.13048. The paper the RWKV line starts from, and the first to take the architecture to 14B: AFT’s position bias becomes an exponential decay over relative position, which makes the layer trainable in parallel and runnable as a constant-memory recurrence. | Active |
| LIT-431 | An Attention Free Transformer | Zhai et al. (2021), ARXIV-2105.14103. Drops the query-key product entirely: keys and values are combined with a learned position bias and the query enters element-wise. The record holds it because it is where RWKV’s letters come from — R, W, K, V is this construction renamed. | Active |
| LIT-448 | Scalable Diffusion Models with Transformers | Peebles and Xie (2022), ARXIV-2212.09748. Replaces the U-Net backbone every diffusion model had inherited from DDPM with a transformer over latent patches, and shows FID falls monotonically with forward-pass GFLOPs however the GFLOPs are bought — depth, width or token count. DiT-XL/2 reaches 2.27 FID on class-conditional ImageNet 256. | Active |
| LIT-449 v2 | Scaling Rectified Flow Transformers for High-Resolution Image Synthesis | Esser et al. (2024), ARXIV-2403.03206. A 61-way comparison of diffusion and rectified-flow formulations settles on a straight-line path with timesteps drawn from a logit-normal, then scales it to 8B for text-to-image. Adds a resolution-dependent shift of the timestep schedule, and MMDiT, which gives each modality its own weights while letting the two token streams attend jointly. | Active |
| LIT-464 | Theoretical limitations of multi-layer Transformer | Chen et al. (2024), ARXIV-2412.02975. The first unconditional lower bound for a decoder-only transformer with more than one layer: any constant-depth decoder needs polynomially many parameters to compose functions sequentially. Depth, an encoder, or chain of thought each buy what width cannot. | Active |
| LIT-472 | nGPT: Normalized Transformer with Representation Learning on the Hypersphere | Loshchilov et al. (2024), ARXIV-2410.01131. Normalize every embedding-dimension vector in every matrix to the unit hypersphere after each step, delete all normalization layers, weight decay and LR warmup, and make each block’s residual contribution a learnable per-dimension step size. 4× / 10× / 20× fewer tokens to the same loss at 1k / 4k / 8k context, at 0.5B and 1B on OpenWebText — and 60–80% more time per step. | Active |
| LIT-477 | NoProp: Training Neural Networks without Full Back-propagation or Full Forward-propagation | Li, Teh and Pascanu (2025), ARXIV-2503.24322. Train each block independently to denoise a noisy label embedding given the raw input — diffusion machinery pointed at a classifier. No forward or backward pass across the network at training time, about half the GPU memory, and test accuracy matching backprop on MNIST and CIFAR-10. Every block sees the input, which is what makes independence possible and what makes the comparison awkward. | Active |
| LIT-479 v2 | The Diffusion Duality | Sahoo, Deschenaux, Gokaslan, Wang, Chiu and Kuleshov (2025), ARXIV-2506.10892. Uniform-state discrete diffusion is the argmax of a Gaussian diffusion, which lets Gaussian techniques transfer: a variance-reducing curriculum doubles training speed, and consistency distillation cuts sampling from 1024 steps to 8. Masked diffusion still wins on likelihood; this wins in the few-step regime, where masked models cannot revise what they have already emitted. | Active |
| LIT-483 | Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Models | Liang et al. (2024), ARXIV-2411.04996. Untie every non-embedding parameter by modality — feed-forward, attention projections, layer norms — and keep global self-attention over the whole sequence. Matches a dense baseline at 55.8% of the FLOPs in the Chameleon setting and 37.2% with speech added. The component ablation is the useful part: the feed-forward carries most of it, attention adds less, layer norms add nothing. | Active |
| LIT-486 | Parallel Scaling Law for Language Models | Chen et al. (2025), ARXIV-2505.10475. Run the same weights over P learnably-prefixed copies of the input and aggregate the outputs: the loss falls as if the parameter count had risen by O(log P). Fitted across 0.5B–4.4B and P = 1…8 on two corpora. At batch size 1, matching a parameter-scaled model costs 22× less added memory and 6× less added latency — and P× the training FLOPs, which is the real price. | Active |
| LIT-493 | Language Modeling by Language Models | Cheng, Clark and Richardson (2025), ARXIV-2506.20249. An LLM agent system that proposed, implemented and pretrained 1,062 verified language-model block designs at 14M–350M parameters. The result that survives its own controls is the code generation: decomposing a long program into units and freezing each one that passes an execution-based checker takes valid-design rate from 6% to 92%. The architecture-discovery headline does not survive as well — the five best designs average 0.6 points below the five human seeds they were bred from. | Active |
| LIT-495 | On the Statistical Query Complexity of Learning Semiautomata: a Random Walk Approach | Giapitzakis, Fountoulakis, Nichani and Lee (2025), ARXIV-2510.04115. The first Statistical Query hardness result for semiautomata, and the first where the hardness comes from the transition structure alone rather than from a hard language embedded in the machine. Two distinct machines in the constructed family agree at chance after O(N² ln N) random symbols, so the signal separating them vanishes exponentially in sequence length. The regime it needs is unusual and belongs beside the result: an alphabet of size Ω(N³ ln N). | Active |
| LIT-496 | Taming Transformers for High-Resolution Image Synthesis | Esser, Rombach and Ommer (2020), ARXIV-2012.09841. Adds a perceptual loss and a patch-based discriminator to the VQ autoencoder, with an adaptive weight λ = ∇[L_rec] / (∇[L_GAN] + δ) computed from the decoder’s last layer. The result is that f = 16 compression stops destroying perceptual quality, so a 256×256 image becomes 16×16 tokens a transformer can model. Every tokenizer in this line since is this architecture with the codebook retuned. | Active |
| LIT-497 | Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation | Sun et al. (2024), ARXIV-2406.06525. Keeps VQGAN’s encoder-quantizer-decoder and retunes the codebook: ℓ₂-normalized codes, vector dimension 8 instead of 256, size 16384. Utilization goes from 0.29% to 97% and rFID from 9.21 to 2.19. Its other measurement is the one the record needed — the same tokenizer spans rFID 2.19 → 0.70 purely by changing how many tokens the image gets. | Active |
| LIT-500 | Vector-quantized Image Modeling with Improved VQGAN | Yu et al. (2021), ARXIV-2110.04627. The origin of the codebook advice SOTA-306 rests on, and it says something sharper than its descendants repeat: factorize the code, so lookup happens in a low-dimensional space while the embedding stays high-dimensional, and ℓ₂-normalize both sides. Its ablation puts codebook usage at 4% at lookup dimension 256, 95% at 16 — and 2% with normalization removed, which is the largest single effect in the table. | Active |
| LIT-513 | mHC-lite: You Don’t Need 20 Sinkhorn-Knopp Iterations | Yang (2026), ARXIV-2601.05732 — not a fourth constraint but a repair of the third. Keeps LIT-140’s doubly-stochastic constraint and makes it exact, building the matrix as a convex combination of permutation matrices (Birkhoff-von Neumann) instead of approximating the projection with 20 Sinkhorn-Knopp iterations. Measures the approximation it replaces: the column sums of the layer-wise product deviate from 1 by up to 220% in a 24-layer network. Read as NOTE-258. | Active |
| LIT-525 | Transformers Learn Low Sensitivity Functions: Investigations and Implications | Vasudeva et al. (2024), ARXIV-2403.06925 — sensitivity to token-wise random input perturbations is one number that separates transformers from MLPs, CNNs, ConvMixers and LSTMs across vision and language. It also predicts robustness, tracks flatness, can be regularized for, and keeps falling through a grokking plateau while the loss does not move. Read as NOTE-271. | Active |
| LIT-527 | Leaner Transformers: More Heads, Less Depth | Saratchandran, Teney and Lucey (2025), ARXIV-2505.20802 — a theorem that more attention heads lower the condition number of the attention block, and twelve redesigned architectures trading heads for layers. In the five configurations where the MLP width is held fixed, accuracy holds or improves at 29–53% fewer parameters. Read as NOTE-269. | Active |
| LIT-528 | The emergence of clusters in self-attention dynamics | Geshkovski, Letrouit, Polyanskiy and Rigollet (2023), ARXIV-2305.05465 — treat tokens as interacting particles and self-attention as their dynamics. Tokens cluster, and the spectrum of the value matrix decides the limiting geometry: V = I gives the vertices of a convex polytope, a simple positive leading eigenvalue gives at most three parallel hyperplanes, V = −I gives one cluster at the origin. In one dimension the attention matrix provably converges to a low-rank Boolean matrix. Read as NOTE-272. | Active |
| LIT-530 | Attention is not all you need: pure attention loses rank doubly exponentially with depth | Dong, Cordonnier and Loukas (2021), ARXIV-2103.03404 — strip the skip connections and MLPs out of a transformer and its output converges doubly exponentially to a rank-1 matrix with identical rows. Skip connections stop it, MLPs slow it, and layer normalization provably does nothing — it is a right-multiplication, and right-multiplication cannot raise rank. Read as NOTE-274. | Active |
| LIT-531 | Deep Learning and the Information Bottleneck Principle | Tishby and Zaslavsky (2015), ARXIV-1503.02406 — the five-page position paper the whole information-bottleneck line descends from. It proposes quantifying every layer by I(X;h) and I(Y;h), draws the information plane as an explicitly qualitative figure with a hypothesized layer path, and states that approaching the IB limit requires stochastic mapping between the layers — the condition its successors were later refuted for dropping. Read as NOTE-275. | Active |
| LIT-532 | What learning algorithm is in-context learning? Investigations with linear models | Akyürek, Schuurmans, Andreas, Ma and Zhou (2022), ARXIV-2211.15661, ICLR 2023 — the independent construction, and the finding the title undersells. A transformer needs constant depth and O(d) hidden size for a gradient step and O(d²) for a Sherman–Morrison ridge update; but the trained model’s behaviour moves through three algorithmic phases as depth grows — gradient descent, then ridge, then ordinary least squares — and at every noise level the ridge parameter that best fits it is the one that minimizes Bayes risk. Read as NOTE-277. | Active |
| LIT-534 | What Can Transformers Learn In-Context? A Case Study of Simple Function Classes | Garg, Tsipras, Liang and Valiant (2022), ARXIV-2208.01066, NeurIPS 2022 — the setup that turned “how does in-context learning work” into a question with a checkable answer. A 9.5M-parameter GPT-2 trained from scratch on sequences of (x, f(x)) pairs in-context-learns unseen linear functions at the accuracy of the optimal least-squares estimator, and sparse linear functions, two-layer ReLU networks and depth-4 decision trees at or above their task-specific algorithms. Read as NOTE-279. | Active |
| LIT-543 | Loss Landscape Degeneracy and Stagewise Development in Transformers | Hoogland, Wang, Farrugia-Roberts, Carroll, Wei and Murfet (2024), ARXIV-2402.02364, TMLR 2025 — the instrument applied. Tracking the LLC through training divides two transformers into five stages each, and the boundaries coincide with structural changes the loss curve does not show: bigrams, then n-grams, then previous-token heads, then the induction circuit. In the in-context regression transformer, in-context learning is acquired in LR2 and then deteriorates across LR3 and LR4 as the model specializes to its pre-training distribution. Read as NOTE-287. | Active |
| LIT-545 | Gemma 3n model overview | Google DeepMind (2025), model documentation with no accompanying paper. The origin of Per-Layer Embeddings (PLE): each token id looks up a small vector for every layer, and each layer gates it into the residual stream. The table is about 2.35B parameters in E2B and is kept off the accelerator. A design disclosure and not evidence: Google published no ablation. Cited by the Qwen3.8-Flash-Next report (LIT-152) as the precedent for embedding tables held in host memory. | Active |
| LIT-546 | MatFormer: Nested Transformer for Elastic Inference | Devvrit, Kudugunta, Kusupati et al. (2023), ARXIV-2310.07707. Nest four FFN widths inside one Transformer, so each smaller FFN is the first neurons of the larger. Train one width per step, sampled uniformly, and pick a different width per layer at inference to extract sizes nobody trained. Up to 850M parameters it matches four separately trained sizes at the same total compute. The small submodels beat theirs partly because shared weights see up to 4× the tokens. | Active |
| LIT-551 | Implicit Neural Representations with Periodic Activation Functions | Sitzmann, Martel et al. (2020), ARXIV-2006.09661. An MLP with sin activations fits images, video, audio and signed distance functions, and, unlike ReLU networks, their derivatives, because a SIREN’s derivative is itself a SIREN. It trains only with a specific initialization: weights U(±√(6/n)) keep every pre-activation near N(0,1), and the first layer is scaled by ω₀ = 30. The initialization’s activation statistics are verified. Its role in earlier periodic networks’ failures is asserted, not ablated. | Active |
| LIT-559 | Alias-Free Generative Adversarial Networks | Karras et al. (2021), ARXIV-2106.12423. StyleGAN3: fine detail in GAN images sticks to pixel coordinates (“texture sticking”) because the generator aliases. Treat every feature map as a bandlimited continuous signal. Low-pass filter around each nonlinearity at 2× temporary upsampling, feed Fourier features instead of a constant, drop noise inputs. The generator becomes translation-equivariant (EQ-T 63 dB), or also rotation-equivariant (StyleGAN3-R, EQ-R 40 dB), at StyleGAN2’s FID. | Active |
| LIT-560 | Analyzing and Improving the Image Quality of StyleGAN | Karras et al. (2019), ARXIV-1912.04958. StyleGAN2 traces StyleGAN’s droplet artifacts to instance normalization and replaces it with weight demodulation (FID unchanged, artifacts gone). It traces its phase artifacts to progressive growing and replaces growing with a fixed output-skip generator and residual discriminator (FFHQ FID 4.34 → 3.31). It runs R1 only every 16 minibatches at no cost, and adds path-length regularization, which trades FID against smoothness on less structured data. | Active |
| LIT-561 | A Style-Based Generator Architecture for Generative Adversarial Networks | Karras, Laine, Aila (2018), ARXIV-1812.04948. StyleGAN: a mapping network turns the latent z into an intermediate w, and w modulates every synthesis layer through AdaIN. Per-layer noise inputs supply stochastic detail, and mixing two w’s across layers regularizes the model toward scale-specific control. Built on progressive growing. FFHQ FID goes from 8.04 (Progressive GAN) to 4.40, most of it from tuning (5.25) and the mapping network and styles (4.85). Also introduced the FFHQ dataset and the perceptual path length metric. | Active |
| LIT-575 | Transformer Feed-Forward Layers Are Key-Value Memories | Geva, Schuster, Berant, Levy (2021), ARXIV-2012.14913. Read each FFN hidden unit as a memory. Its input weight is a key that fires on human-recognizable prefix patterns: shallow n-grams in lower layers, semantic topics in upper ones. Its output weight is a value that, projected through the output embedding, favors next tokens. The evidence is 160 sampled keys in one 16-layer WikiText-103 model. Top value predictions agree with the keys’ triggers at most 3.5% of the time, in upper layers only. Layer outputs are mostly compositions, matching no single memory’s prediction. | Active |
| LIT-579 | Where Did the Gap Go? Reassessing the Long-Range Graph Benchmark | Tönshoff, Ritzert, Rosenbluth, Grohe (2023), ARXIV-2309.00367. On the Long-Range Graph Benchmark, graph transformers were reported to beat message-passing GNNs. After a basic hyperparameter sweep within the same 500k-parameter budget, GCN, GINE and GatedGCN all beat GPS on Peptides-func and Peptides-struct. Most of the gain comes from a 2-layer prediction head instead of a linear one. On the superpixel datasets, input feature normalization accounts for at least half the gain of most models, and GPS stays ahead there. The link-prediction metric’s code computed raw MRR where the paper specified filtered MRR. | Active |
| LIT-580 | How Powerful are Graph Neural Networks? | Xu, Hu, Leskovec, Jegelka (ICLR 2019), ARXIV-1810.00826. Message-passing GNNs are at most as discriminative as the 1-WL isomorphism test, and they reach that bound when aggregation and readout are injective on multisets. Sum aggregation with an MLP update (GIN) is; mean (GCN) and max (GraphSAGE) are not. On featureless Reddit graphs, mean-aggregation GNNs are at chance (50.0, 20.0) and GIN scores 92.4 and 57.5. On the other seven benchmarks, most differences are within a standard deviation. | Active |
| LIT-581 | Pitfalls of Graph Neural Network Evaluation | Shchur, Mumme, Bojchevski, Günnemann (R2L workshop, NeurIPS 2018), ARXIV-1811.05868. GCN, GAT, MoNet and GraphSAGE were run with one shared training and tuning procedure, over 100 random splits × 20 seeds on 8 datasets. GCN has the best average rank (2.3) and relative accuracy (99.4%). On Planetoid’s single split GAT wins Cora and Citeseer; on another random split of the same sizes GCN wins them. GAT’s lower average on Amazon comes from 138 of 2,000 runs collapsing. | Active |
| LIT-582 | Semi-Supervised Classification with Graph Convolutional Networks | Kipf and Welling (ICLR 2017), ARXIV-1609.02907. The GCN layer is H′ = σ(D̃^{-1/2}ÃD̃^{-1/2} H W), with self-loops added and the adjacency symmetrically normalized. It is motivated as a first-order approximation of spectral graph convolution. Two layers reach 81.5% on Cora, 70.3% on Citeseer and 79.0% on Pubmed, all on the single Planetoid split. Models deeper than about 7 layers are hard to train without residual connections. | Active |
| LIT-583 | Highly accurate protein structure prediction with AlphaFold | Jumper et al. (Nature 2021), DOI 10.1038/s41586-021-03819-2. AlphaFold 2 predicts protein structure from sequence and a multiple sequence alignment. At CASP14 its median backbone error was 0.96 Å r.m.s.d.95, against 2.8 Å for the next best method. Its per-residue confidence, pLDDT, tracks true accuracy (r = 0.76 over 10,795 chains). Self-distillation on its own confident predictions for 350,000 unlabeled sequences improves it. Accuracy falls sharply when the alignment has fewer than about 30 sequences. | Active |
| LIT-584 | Accurate structure prediction of biomolecular interactions with AlphaFold 3 | Abramson et al. (Nature 2024), DOI 10.1038/s41586-024-07487-w. AlphaFold 3 predicts complexes of proteins, nucleic acids, ligands, ions and modified residues. A pairformer with lighter MSA processing replaces the Evoformer, and a diffusion module over raw atom coordinates replaces the structure module, with no built-in rotational equivariance. It beats Vina on PoseBusters without the pocket information Vina gets (P = 2.27×10⁻¹³). The generative head hallucinates structure in disordered regions, which cross-distillation from AlphaFold-Multimer reduces. Chirality is violated in 4.4% of PoseBusters predictions even after ranking penalties. | Active |
| LIT-587 | An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale | Dosovitskiy et al. (2020), ARXIV-2010.11929. Cut the image into fixed patches, linearly embed them, and feed the sequence to an unmodified transformer encoder. The result is not that the architecture is better — it is that at enough pre-training data the convolutional prior stops paying for itself, and below that threshold it still does. | Active |
| LIT-622 v2 | CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer | Yang, Teng et al., Tsinghua and Zhipu AI (2024), ARXIV-2408.06072. The first widely used open DiT for text-to-video. It has a MAGVIT-style causal 3D VAE, one sequence over text and video with modality-specific adaLN, and full 3D attention. Full 3D attention beats factorized attention at 1.1× to 2.3× the forward time, rising with resolution. The architecture ablations are absent from the first arXiv version and present in the third, and the 2B model ships the position encoding they rejected. | Active |
| LIT-626 v2 | Movie Gen: A Cast of Media Foundation Models | The Movie Gen team, Meta (2024), ARXIV-2410.13720. A 30B video model built from a Llama 3-style transformer, flow matching and an 8×8×8 temporal autoencoder, with the most controlled ablations in the video line. At 5B, flow matching beats diffusion and video captions beat frame captions. The “LLaMa3 scaling laws” claim rests on image-only runs at four sizes. | Active |
| LIT-639 | ResiDual: Transformer with Dual Residual Connections | Xie et al. (2023), ARXIV-2304.14802. Gives the representation-collapse argument a rate: in a Pre-LN transformer the per-layer change in the normalised hidden state decays as O(1/√k), and adding a layer to an N−1 layer model moves the output by O(1/√N). Also measures what Pre-LN’s no-warmup claim costs — 32.28 against 35.12 BLEU at E6D6, and 31.82 against 35.18 at E12D12. | Active |
| LIT-640 v3 | Query-Key Normalization for Transformers | Henry, Dachapally, Pawar and Chen (2020), ARXIV-2010.04245. The origin of QK-norm, which thirty documents in this record name and SOTA-192 recommends while sourcing an adopter. ℓ₂-normalize each query and key along the head dimension before the dot product, then scale by a learnable parameter instead of dividing by √d — so the temperature becomes a trained quantity rather than a constant. +0.928 BLEU averaged over five low-resource pairs. | Active |
| LIT-641 | Attention Approximates Sparse Distributed Memory | Bricken and Pehlevan (2021), ARXIV-2111.05498. Attention’s update rule is Kanerva’s sparse distributed memory read, given two conditions: L² normalized vectors and a fitted softmax temperature β. The retrodiction is the interesting part — those two conditions are QK-norm, published a year earlier for unrelated reasons. Trained QK-norm heads learn β ∈ [10, 25], the range that interpolates between SDM’s optimality criteria. | Active |
| LIT-651 | Transformers without Tears: Improving the Normalization of Self-Attention | Nguyen and Salazar (2019), ARXIV-1910.05895. The first systematic evaluation of pre-norm in the base Transformer regime — and it reports a result the record’s universal pre-norm practice does not carry: on high-resource WMT’14 English-German, post-norm wins, 27.58 to 26.83. Also introduces ScaleNorm, the ℓ₂-normalization-with-one-learned-scalar that LIT-640’s QK-norm is an application of. | Active |
| LIT-653 | Softmax is not Enough (for Sharp Size Generalisation) | Veličković, Perivolaropoulos, Barbero and Pascanu (2024), ARXIV-2410.01104. Twenty-three practices in this record name the softmax and no note held it. This one proves a limit on it: in a Transformer over a finite vocabulary, attention coefficients must decay towards zero as the number of items grows, so a head that is sharp in distribution cannot stay sharp on larger inputs. Max retrieval falls from 98.6% at 16 items to 12.4% at 16,384. | Active |
| LIT-654 | Muon Outperforms Adam in Tail-End Associative Memory Learning | Wang et al. (2025), ARXIV-2509.26030. Ninety-four documents in this record name Muon and seventeen practices recommend or condition on it; five theories mention it and none says why it works. This one ablates Muon block by block: applying it to the value-output weights and the FFN alone almost recovers the full-Muon trajectory, and applying it to query-key contributes little. The account is that Muon’s update matches the outer-product structure of associative memory, giving isotropic spectra and better learning of rare classes. | Active |
| LIT-657 | Patch n’ Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution | Dehghani, Mustafa et al., Google DeepMind (2023), ARXIV-2307.06304. Packs patches from several images into each fixed-length sequence, with per-example attention masks and pooling. A ViT can then take any aspect ratio and a sampled resolution per image. It leads ViT at each of 12 compute-matched budgets, and matches the best ViT with 4x less compute. The paper attributes the gain chiefly to seeing about 5x more images, not to aspect ratio. The one test that isolates aspect ratio is a linear probe on fairness attributes. | Active |
| LIT-658 | FiT: Flexible Vision Transformer for Diffusion Model | Lu et al. (2024), ARXIV-2402.12376. A DiT variant trained on images without cropping or upsampling. Large images are only resized down to at most 256² pixels at native aspect ratio, and sequences are padded to 256 tokens. Table 3 isolates this pipeline change on DiT-B/2 at 400K steps. At a square 256² output, FID moves 44.83 → 43.34 and sFID worsens 8.49 → 11.11. At 160×320 and 224×448, FID falls from 91.32 and 109.1 to 50.51 and 52.55, but the fixed model never trained on those shapes. | Active |
| LIT-660 | simple diffusion: End-to-end diffusion for high resolution images | Hoogeboom, Heek and Salimans, Google Brain (2023), ARXIV-2301.11093. Trains single-stage pixel-space diffusion up to 512x512 with four changes. The first is to define the noise schedule at a reference resolution and shift its log-SNR by 2·log(ref/d) at resolution d. Table 2 is the controlled measurement. On ImageNet 256 the unshifted cosine schedule gives FID 7.65 train / 6.87 eval, and the schedule shifted to 32 gives 3.76 / 3.71. This is the origin, with the concurrent Chen (2023), of the resolution-dependent schedule shift SD3 later derives again. | Active |
| LIT-662 | Vision Transformers Need Registers | Darcet, Oquab, Mairal and Bojanowski (2023), ARXIV-2309.16588. 107 documents in this record say “ViT” and none held this. Large trained ViTs put high-norm outlier tokens on background patches — 2.37% of tokens in DINOv2 ViT-g — which hold little local information and a lot of global information. The model is using redundant patches as scratch space. Append a few tokens whose only job is to be scratch space and the artifacts vanish. | Active |
| LIT-665 | Transformer Language Models without Positional Encodings Still Learn Positional Information | Haviv, Ram, Press, Izsak and Levy (2022), ARXIV-2203.16634. The result the record’s position-encoding cluster rests on, credited here to a paper a year younger. A decoder-only LM with no positional encoding at all is competitive; probing shows it acquires absolute position within four layers. The mechanism is the causal mask, and the control that shows it is a bidirectional MLM, where NoPos perplexity is 147.18 against 4.00. | Active |
| LIT-666 v2 | Sparse Distributed Memory | Kanerva (1988), MIT Press. An associative memory in a high-dimensional binary space: a pattern is written to every fixed-address neuron within a Hamming radius of it, and read back as the thresholded sum over the neurons within that radius of the query. Held because LIT-641 shows attention’s softmax approximates this read — the weighting it induces, the intersection of two hyperspheres, decays near-exponentially in distance — and the record carried that account (THEORY-097) without its source. Filed from secondary accounts; the book has not been read here. | Active |
| LIT-667 | Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization | Wang, Yue, Su and Sun (2024), ARXIV-2405.15071. Grokking outside algorithmic data, on reasoning over stored facts — and a named correction to the critical-data-size hypothesis the record holds as THEORY-071. With the inferred/atomic ratio φ held at 9.0, varying the training-set size changes nothing; varying φ at fixed size moves grokking speed monotonically and at φ = 18.0 all but removes it. Also traces the generalizing circuit for two tasks and explains from its layout why composition never generalizes out of distribution while comparison does. | Active |
| LIT-668 v2 | ALBERT: A Lite BERT for Self-supervised Learning of Language Representations | Lan, Chen, Goodman, Gimpel, Sharma and Soricut (2019), ARXIV-1909.11942. The measurement behind cross-layer parameter sharing, which this record started recommending — via the fix THEORY-103 endorses — without holding a single paper on it. The sharp result is an asymmetry: sharing the attention parameters across every layer costs +0.1 average at E = 128, and sharing the feed-forward parameters costs −1.4. Also carries a priority claim the record’s dropout practice should have: the first report that dropout hurts a large transformer. | Active |
| LIT-669 | Sparse Distributed Memory and Related Models | Kanerva (1992), RIACS TR 92.10, published in 1993 in Hassoun (ed.), Associative Neural Memories (Oxford). SDM restated as a matrix memory: a fixed random address matrix A, a wide layer of hard locations, and a modifiable contents matrix C trained by the outer-product rule. Stored words are weighted by the overlaps of activation sets. The activation probability that maximizes signal-to-noise is p = (2MT)^(-1/3), and capacity is about 10% of the number of locations at one bit error in a thousand. This is the source of two of the three optimal radii LIT-641 fits β against. The third, critical distance, is not in it. | Active |
| LIT-670 | BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding | Devlin, Chang, Lee and Toutanova (2018), ARXIV-1810.04805. The model 89 files in this record referred to and none held. Two things it is cited for here turn out not to be in it: the 15% masking rate was never swept — “in all of our experiments, we mask 15%”, and the one appendix table headed “Masking Rates” varies the 80/10/10 replacement split at a fixed 15% — and its NSP ablation removes the loss while keeping the paired-segment input format, which LIT-671 later separates and reverses. | Active |
| LIT-679 | Dynamic Chunking for End-to-End Hierarchical Sequence Modeling | Hwang, Wang and Gu (2025; ICLR 2026), ARXIV-2507.07955. H-Net is a byte-level hierarchical model that learns where to cut chunks. Compared with a GPT-2 BPE Transformer at matched bytes per batch and FLOPs per byte, a chain of baselines separates each ingredient. One stage of learned chunking only ties BPE and whitespace chunking (0.728 against 0.730 against 0.726 bits/byte at XL). The gain is the second stage (0.715, +2.7 downstream average). The largest gains are on Chinese. It is limited to 1.3B-equivalent FLOPs and trains about 2× slower in wall-clock. | Active |
| LIT-681 | Cramming: Training a Language Model on a Single GPU in One Day | Geiping and Goldstein (ICML 2023), ARXIV-2212.14034. A BERT-style MLM trained from scratch for 24 hours on one consumer GPU, which reaches 79.2–80.4 GLUE against BERT-base’s 80.9. The transferable finding is negative: model shape does not beat the scaling law at this budget. Every reshape it tried lands near the same final loss, because a smaller model’s extra throughput cancels its slower learning. So the only architectural lever is to cut step time at constant parameter count. Its data result is the cheapest filter in the record: drop text the tokenizer compresses badly. | Active |
| LIT-683 | Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach | Geiping et al. (NeurIPS 2025), ARXIV-2502.05171. Huginn has a 2-layer prelude, a 4-layer core iterated r times with the input re-injected every iteration, and a 2-layer coda: 3.5B parameters trained on 800B tokens with r sampled per step (mean 32) and backpropagation through only the last 8 iterations. Accuracy rises with test-time iterations and saturates per task. Recurrence is shown to buy accuracy per parameter, not per FLOP. The only controlled baseline has the same parameters and about 1/16 of the compute. “Equivalent to 50B parameters” is a FLOP count, not a comparison. | Active |
| LIT-698 | Topos and Stacks of Deep Neural Networks | Belfiore et al. (2021), ARXIV-2106.14587. The authors claim that every DNN architecture defines a Grothendieck topos of sheaves over a site built from its graph, that backpropagation is a flow of morphisms in that topos, and that layer invariances (CNN translations, LSTM structure) are stacks, and they conjecture that these stacks explain generalization and support a homological notion of “semantic information”. | Deferred — seeded from the abstract and a skim on 2026-09-25; not read in full |
| LIT-699 v2 | Diffusion Models Beat GANs on Image Synthesis | Dhariwal and Nichol (2021), ARXIV-2105.05233. The classifier-guidance paper, the ablated U-Net the diffusion literature then treated as the default, and the source of the guidance-scale trade. Its title claim is metric-dependent and the paper says so: ADM-G beats BigGAN-deep on FID and recall at 128, 256 and 512, and loses on precision at all three — 0.78 against 0.86, 0.82 against 0.87, 0.87 against 0.88. It also asserts, once, in the introduction, that the gradient scale can be raised “by an order of magnitude without obtaining adversarial examples”, and runs no experiment anywhere in the paper that tests it. | Active |
| LIT-708 | A Controlled Study of Attention-Only Transformers | Ndubuaku et al. (2026), ARXIV-2607.18363. Removing every feed-forward layer from a decoder costs 0.47 nats at matched depth and 0.26 at matched training FLOPs, but only 0.0055 nats (0.27% of loss) at matched parameters once the freed budget goes into attention depth: 20 attention-only layers against 4 standard blocks, 24M parameters, 105B tokens of a synthetic reasoning corpus, with a separate learning-rate sweep for each arm. The remaining gap sits on low-context tokens, where the answer must come from the weights. A pre-registered test on fineweb-edu predicted 0.02–0.05 nats and measured 0.040. Removing QK-norm made the attention-only stack diverge; residual gates and ReZero changed nothing. Everything is at or below 87M parameters. | Active |
| LIT-709 | Primer: Searching for Efficient Transformers for Language Modeling | So et al. (NeurIPS 2021), ARXIV-2109.08668. An evolutionary search over TensorFlow primitives, at a fixed 24-hour training budget, finds two changes that carry most of the gain. One squares the feed-forward ReLU. The other adds a causal width-3 depthwise convolution after each per-head Q, K and V projection (MDHA). The only direct activation comparison has squared ReLU ahead of SwiGLU and ReGLU, in one 110M C4 figure. “4.2×” is against the original ReLU T5 at 537M. Against the SwiGLU + RMSNorm baseline the saving is 2× at equal final quality. On encoder–decoder masked LM, Primer-EZ does no better than that baseline. | Active |
| LIT-711 | Do Transformer Modifications Transfer Across Implementations and Applications? | Narang et al. (2021), ARXIV-2102.11972. About fifty published transformer modifications reimplemented in one codebase (T5 / Mesh TensorFlow, 223M encoder-decoder) with hyperparameters held fixed, scored on C4 pre-training loss, SuperGLUE, XSum, WebQuestions and WMT’14 En-De. Most do not help. The ones that do are small (GLU-variant activations, RMSNorm, untied embeddings), cost parameters or time (Switch/MoE, product-key memory, mixture of softmaxes), or were invented in the same codebase. The Universal Transformer, ReZero, Fixup, cross-layer sharing, the Evolved Transformer and lightweight/dynamic convolutions all come out worse than the vanilla baseline. The caveat that travels least: the positive results are also mostly same-codebase — GLU Variants (LIT-030) was run in this codebase at this exact configuration — so the paper is a strong negative test and a weak positive one. | Active |
| LIT-714 | Analyzing and Improving the Training Dynamics of Diffusion Models | Karras et al. (2023), ARXIV-2312.02696 — EDM2. Two contributions that do not depend on each other. Rebuilding the ADM U-Net so every layer preserves activation, weight and update magnitude takes ImageNet-512 FID from 8.00 to 2.56 at equal compute, the largest single step (6.96 → 3.75) being weight normalization on the learned layers. And post-hoc EMA: store two power-function averages at snapshots and reconstruct any EMA length after training, which shows the optimal length moving with architecture, training time, learning rate, guidance weight and — by a factor of seven under guidance — with which metric you score it. | Active |
| LIT-715 | Surya: Foundation Model for Heliophysics | Roy et al. (2025), ARXIV-2508.14112. A 366M-parameter transformer pretrained on full-resolution (4096²) multi-instrument SDO imagery to forecast the next frame, then rollout-tuned, forecasts solar dynamics zero-shot. With LoRA fine-tuning it beats task-specific baselines on four downstream heliophysics tasks. | Deferred — seeded from the abstract and a skim on 2026-09-25; not read in full |
| LIT-720 | Analyzing and Improving the Training Dynamics of Diffusion Models | Karras, Aittala, Lehtinen, Hellsten, Aila and Laine (2023), ARXIV-2312.02696. Post-hoc EMA: keep two power-function weight averages during training, snapshot them periodically, and afterwards reconstruct the average for any decay length by least squares — reconstruction error falling as O(1/n⁴) in the snapshot count. That makes the averaging length a sweep instead of a pre-training guess, and the consequence is the headline number here: with the length fixed at 13%, varying the learning-rate decay over t_ref ∈ [30k, 160k] moves FID by up to 72%; sweeping the length post hoc brings the whole bracket within 10% of the optimum. Also EDM2’s magnitude-preserving architecture, reaching FID 1.91 on ImageNet-512 without guidance. | Superseded — by LIT-714; A duplicate. 2312.02696 was already held as LIT-714, filed hours earlier in #375 and merged while this note was being written; LIT-714 is the fuller reading — the whole ablation ladder, the metric-disagreement finding, and NOTE-370 with the appendices — and ten documents cite it against this note’s four, all of which were its own. Retired rather than deleted so the mistake stays visible and the formerly: alias keeps any stray citation resolving. Nothing here is lost: everything this note said, LIT-714 or SOTA-428 says already, including the learning-rate bracket. |
| LIT-723 | Swin Transformer: Hierarchical Vision Transformer using Shifted Windows | Liu, Lin, Cao, Hu et al. (2021), ARXIV-2103.14030. A vision transformer with a hierarchical feature pyramid and self-attention confined to shifted local windows, linear in image size rather than quadratic. Nine documents here run experiments on it and none said what it is. Its ablation is the reason to hold it: every design element is measured on three tasks, and the absolute position embedding this record recommends elsewhere helps ImageNet classification (+0.4 top-1) while hurting COCO detection (−0.2 box AP) and ADE20K segmentation (−0.6 mIoU). | Active |
| LIT-727 v4 | Striving for Simplicity: The All Convolutional Net | Springenberg, Dosovitskiy, Brox and Riedmiller (2014), ARXIV-1412.6806, ICLR 2015 workshop. Guided backpropagation’s defining paper, and so the referent for SOTA-430’s headline failure case — the method whose maps stay “visually and quantitatively similar” after the weights above the lowest layers are destroyed. The masking rule is stated here in one sentence: zero the gradient wherever either the top gradient or the bottom activation is negative. The paper is about an architecture; guided backprop arrives in §4.2 as a tool for analysing it, and the property later found to be its defect is presented here as its advantage. | Active |
| LIT-728 v2 | Visualizing and Understanding Convolutional Networks | Zeiler and Fergus (2013), ARXIV-1311.2901. Two attribution families start here and the record depends on both. The deconvnet is the approach guided backprop (LIT-727) is explicitly “a new variant of”, and so is the antecedent of SOTA-430’s headline failure case. Occlusion sensitivity — slide a grey square, watch the class probability — is the perturbation-based family LIT-725 names as its own origin, and LIT-725’s infidelity is its quantitative descendant. Also an early depth-against-parameters ablation: removing the fully connected layers costs little, removing them and two middle conv layers is “dramatically worse”. | Active |
| LIT-729 v2 | When Explanations Lie: Why Many Modified BP Attributions Fail | Sixt, Granz and Landgraf (2019), ARXIV-1912.09818, ICML 2020. The account SOTA-430’s failing row was missing — for one family, and not the family this record said it was for. Theorem 1: a backpropagation rule that keeps only non-negative relevance gives a product of non-negative matrices, which converges to a rank-1 matrix, and C v = c γᵀ v = λc — once converged the later layers can only change the map’s sign. That covers DTD, LRP-α1β0, Excitation BP and PatternAttribution. It explicitly does not cover Guided Backprop or Deconv, whose ReLU on the gradient makes the backward pass non-linear and which it defers to Nie et al. (2018), now LIT-730. | Active |
| LIT-733 | Language Models are Unsupervised Multitask Learners | Radford et al. (2019) — GPT-2. Language models trained on WebText begin to perform tasks zero-shot when conditioned on a suitable context, and the performance rises log-linearly with capacity up to the 1.5B model, which still underfits. Its §2.3 carries two architecture changes the field kept: layer norm moved to the input of each sub-block, and residual-layer weights scaled at initialization by 1/sqrt(N), N the number of residual layers, to account for accumulation on the residual path with depth. | Active |
| LIT-743 | Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillation | Bahmani et al. (2025), ARXIV-2509.19296 — Lyra. Train a 3D Gaussian Splatting decoder on the latents of a frozen camera-controlled video diffusion model (GEN3C), supervised only by that model’s own RGB decodings along six camera trajectories per image, with no real multi-view data. Decoding latents rather than pixels is what makes 726 views per scene fit in memory. Single image to 3D beats the reported numbers of ZeroNVS, ViewCrafter, Wonderland and Bolt3D on RealEstate10K, DL3DV and Tanks-and-Temples, and adding real multi-view data to the synthetic supervision does not help. | Active |
| LIT-747 | Perceiver: General Perception with Iterative Attention | Jaegle et al. (2021), ARXIV-2103.03206 — the Perceiver. A learned latent array of N=512 queries cross-attends to the raw input (50,176 ImageNet pixels), so attention costs O(MN) rather than O(M²), and a deep latent transformer then runs at O(N²) per layer, independent of input size. With 2D Fourier-feature positions and no convolution it reaches 78.0% top-1 on ImageNet without pretraining, unchanged when the pixels are permuted, while ViT-B-16 and ResNet-50 given the same features fall to 61.7 and 39.4. The same architecture runs on audio, video and point clouds. | Active |
| LIT-748 | SepLLM: Accelerate Large Language Models by Compressing One Segment into One Separator | Chen et al. (2024), ARXIV-2412.12094 — SepLLM. Let each token attend only to a few initial tokens, the previous n tokens, and every earlier separator token (punctuation and whitespace), on the hypothesis that a segment’s content is condensed into the separator that ends it. Training-free on Llama-3-8B-Instruct it keeps GSM8K-CoT at 77.18 against 77.79 with 47% of the KV cache, where StreamingLLM at the same budget gets 70.89. Trained that way from scratch at 160M it reaches lower loss than full attention at equal FLOPs. | Active |
| LIT-754 | Efficient AlphaFold2 Training using Parallel Evoformer and Branch Parallelism | Wang et al. (2022), ARXIV-2211.00235. Move the Evoformer’s outer product mean to the end of the block, so the MSA and pair stacks no longer depend on each other within a block; this matches the original block’s accuracy and speed. Then put the two stacks on two GPUs. On a batch of one protein per device, where data parallelism cannot scale further, this cuts end-to-end AlphaFold2 training from 5.8 to 4.2 days on UniFold and from 6.7 to 4.9 days on HelixFold, using twice the GPUs. | Active |
| LIT-758 | Net2Net: Accelerating Learning via Knowledge Transfer | Chen, Goodfellow and Shlens (2015), ARXIV-1511.05641 — Net2Net, ICLR 2016. Initialize a larger network so it computes exactly the function of a trained smaller one, then train it normally. Net2WiderNet replicates randomly chosen units and divides their outgoing weights by the replication count, and Net2DeeperNet inserts identity layers. On ImageNet Inception-BN, a widened student reaches the randomly initialized network’s final validation accuracy roughly 2×10⁶ minibatches sooner, with no loss of final accuracy. | Active |
| LIT-759 | From monoliths to modules: Decomposing transducers for efficient world modelling | Boyd et al. (2025), ARXIV-2512.02193 — a formal paper with no experiments. A world model is a transducer (a stochastic input-output machine with latent memory, generalizing POMDPs), and composing transducers is a Kronecker product of their operators. The paper inverts this. Two information measures, Intransducibility (with latents) and Acausality (observables only), are zero exactly when one block of variables can be a causal module driven by the rest. Peeling such modules off factors a monolithic model into a DAG of sub-transducers that can be learned and inferred separately. Minimal predictive models (ε-transducers) are closed under composition. | Active |
| LIT-760 | Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks | Lee et al. (2018), ARXIV-1810.00825 — the Set Transformer (ICML 2019). An attention encoder–decoder for set inputs. Its induced set attention block (ISAB) routes a set of n elements through m learned “inducing points”, so each block costs O(nm) rather than O(n²), and its pooling by multihead attention (PMA) aggregates the set with k learned seed vectors in place of mean or max pooling. Attention-based pooling is the larger win on tasks whose outputs interact (amortized clustering), and 16 inducing points matched or beat full self-attention there; each block still maps back to the input size, so a stack of them scales with n at every layer. | Active |
| LIT-763 | Erwin: A Tree-based Hierarchical Transformer for Large-scale Physical Systems | Zhdanov et al. (2025), ARXIV-2502.17019 — Erwin. Build a ball tree over an irregular point set so that every ball at a level holds the same number of points in contiguous memory, then run full self-attention inside each ball, coarsen and refine through the tree in a U-Net, and alternate with a tree built on a rotated copy of the cloud so balls exchange information. Runtime fits n^1.054. Best on cosmology at large training sets, on EAGLE turbulence (3× faster and 8× less memory than EAGLE’s own model), and on ShapeNet-Car; a clear failure on the Airfoil PDE mesh. | Active |
| LIT-767 | Understanding Deep Learning via Notions of Rank | Razin (2024), ARXIV-2408.02111 — PhD thesis. Gradient descent’s implicit regularization is better read as rank minimization than norm minimization: on a 2×2 matrix completion problem, a depth L ≥ 2 linear network trained by gradient flow drives every norm and quasi-norm to infinity, with probability at least 0.5 over arbitrarily small initialization, while rank falls to its minimum. The same bias toward low (tensor, hierarchical-tensor) rank is proved for polynomial-activation networks equivalent to tensor factorizations; it predicts convolutional networks’ locality, which a regularizer derived from it partly undoes on long-range tasks. | Active |
| LIT-769 | Structural Decoupling: A Scaffold-Flow Theory of Generalization and Alignment | Li (2025), ARXIV-2506.20699. A theory paper with no experiments. It defines a problem’s width, the minimum number of locally feasible contexts needed to cover it, and states that width is incomparable with VC dimension and that a learner with fewer contexts than the width has an error floor no amount of within-context training removes. From the additive split of complexity into a routing term and a within-context term it argues that the parts of a model that keep context structure should not be trained by the gradients that fit predictions inside a context, and it reads hallucination, reward hacking and deceptive alignment as failures of that structure. The current version (v2, 2026) replaced a v1 about a “Context-Content Uncertainty Principle” for brains and machines. | Active |
| LIT-771 | Glow: Generative Flow with Invertible 1x1 Convolutions | Kingma and Dhariwal, OpenAI (2018), ARXIV-1807.03039. A flow step is actnorm (a per-channel affine layer initialized from one batch), a learned invertible 1×1 convolution in place of Real NVP’s fixed channel permutation, and an affine coupling layer. The 1×1 convolution is the one controlled result: lower CIFAR-10 NLL than reversal or random permutation with either coupling, three seeds, about 7% more wall-clock (Fig. 3). Against Real NVP’s published numbers it is lower on all six datasets (CIFAR-10 3.35 against 3.49 bits/dim), with much larger models. The 256×256 faces are trained on 5-bit images and sampled at temperature 0.7. | Active |
| LIT-772 | Neural Ordinary Differential Equations | Chen, Rubanova, Bettencourt and Duvenaud, Toronto and Vector (2018), ARXIV-1806.07366. Parameterize a hidden state’s derivative with a network, integrate it with a black-box solver, and backpropagate by solving the adjoint ODE backwards, at memory constant in depth (§2, Alg. 1). For densities, the log-density of a state moving under dz/dt = f obeys d log p/dt = −tr(∂f/∂z) (Theorem 1), so a continuous normalizing flow needs a trace, not a determinant. The CNF evidence is 2-D toys. An MNIST ODE-Net matches a ResNet at 0.42% against 0.41% error. | Active |
| LIT-781 | Density estimation using Real NVP | Dinh, Sohl-Dickstein and Bengio, Google Brain (2016), ARXIV-1605.08803. The affine coupling layer: half the dimensions pass through unchanged and parameterize a scale and shift of the other half, so the Jacobian is triangular, its log-determinant is a sum, and the inverse costs the same as the forward pass whatever networks compute the scale and shift (§3.2–3.3). Stacked with squeeze and factor-out in a multi-scale architecture, it gives exact likelihood, exact latent inference and one-pass sampling. It does not beat PixelRNN’s likelihood on any dataset it shares with it (CIFAR-10 3.49 against 3.00 bits/dim, Table 1), and sample quality is shown, not measured. | Active |
| LIT-785 | STARFlow: Scaling Latent Normalizing Flows for High-resolution Image Synthesis | Gu et al., Apple (2025), ARXIV-2506.06276. TarFlow moved into a VAE latent with a deep-shallow split: one deep Transformer flow block and five 2-layer ones. It adds a decoder fine-tuned on noisy latents in place of score denoising, and a guidance rule derived from the Gaussian score. ImageNet 256 FID 2.40 at 1.4B against a retrained pixel TarFlow’s 5.56, with SiT’s quoted 2.06 and MAR’s 1.55 still ahead. ImageNet 512 3.00 against EDM2’s 1.25; COCO zero-shot 9.1; GenEval 0.56. The only head-to-head against diffusion is a plot against an in-house 2.1B DiT trained on 200M images (STARFlow’s schedule is 400M) with a stock decoder. Sampling is about 2.2 s per 256×256 image with guidance. | Active |
| LIT-792 | Discrete Flow Matching | Gat et al., FAIR and Weizmann (2024), ARXIV-2407.15595. Flow matching on discrete tokens as a continuous-time Markov chain, with velocities of the same form as the continuous case, any coupling, a family of schedulers, and corrector sampling that can re-mask emitted tokens. At 150M its generative perplexity beats SEDD’s in float32 (GPT-2 29.3 against 40.5 at 1,024 NFE). In double precision every discrete model’s perplexity rises 2–3.5 times and the gap narrows (97.5 against 109.5). At 1.7B on code it reaches HumanEval pass@1 6.7 against a same-data autoregressive model’s 14.3. | Active |
| LIT-794 | Normalizing Flows are Capable Generative Models | Zhai et al., Apple (2024), ARXIV-2412.06329. TarFlow stacks causal Transformers over image patches as affine autoregressive flows, reversing the order between blocks, trained by exact maximum likelihood. A likelihood model reaches 2.99 bits/dim on unconditional ImageNet 64×64, below every diffusion and autoregressive number it quotes (Table 2). Sampling needs three additions: Gaussian input noise about 25× the dequantization noise, a Tweedie denoising step using the flow’s own score, and classifier-free guidance on the affine parameters. Best FIDs are 2.66 at ImageNet 64 and 5.03 at 128, against quoted 1.55 (EDM) and 1.94. Sampling is sequential: 32 samples at 64×64 take about two minutes on one A100. | Active |
| LIT-799 | Olmo Hybrid: From Theory to Practice and Back | Merrill et al., Ai2 and Lambda (2026), ARXIV-2604.03444. Olmo 3 7B with its sliding-window layers replaced by Gated DeltaNet (negative eigenvalues), at 3:1 with full attention, trained to 6T tokens. It reaches Olmo 3’s MMLU in 49% fewer tokens and beats it in every domain after mid-training, though data mix, schedule and mid-training batch also changed. Scaling fits from 60M to 1B give the hybrid a lower data coefficient (83.7 against 94.9, non-overlapping CIs); the projected 1.3–1.9× token savings have CIs that include 1 at every size but 3B. A theorem shows hybrids express a task neither parent can. Removing negative eigenvalues barely changes the scaling, which weakens the expressivity explanation. | Active |
| LIT-801 | Cracks in the Foundation: Seemingly Minor Architectural Choices Impact Long Context Extension | Bertsch, Soldaini, Gormley, Neubig, Hajishirzi, Lo and Groeneveld, Ai2 and CMU (2026), ARXIV-2608.10296. Twenty-six 7–8B models, 140B tokens each on identical data, extended identically to 64K. QK norm, GQA, sliding-window layers and a 4K pretraining length each cost long-context ability, and together they compound: HELMET at 32K runs from 56.4 down to 29.9. Counting how many of the four a model has predicts that score with in-sample R² 0.67 (0.61 leave-one-out). Pretraining loss predicts it at R² 0.29. QK norm is the largest single cost, about 4 to 6 points. GQA plus sliding windows costs about 9, against 1.1 for windows alone. One seed per configuration, and initialization alone moves scores by 7.7% of the range on average. | Active |
| LIT-802 | Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention | Hatamizadeh, Choi and Kautz, NVIDIA (2026), ARXIV-2605.22791. The delta rule’s scalar beta decides two things at once: how much to erase along the key and how much of the value to write. Splitting it into a channel-wise erase gate on the key and a channel-wise write gate on the value, on top of KDA’s channel-wise decay, gives the best average of six recurrent mixers at 1.3B/100B tokens and matched state size (53.11 against 52.39 for Mamba-3 MIMO and 52.28 for KDA; 53.97 hybrid). One run each. The erase gate carries most of the gain. Its multi-key retrieval lead is real, but Gated DeltaNet beats it on the word-needle task at 4K, 60.6 against 31.8. | Active |