Field consensus — How far THE FIELD has converged on a practice, which is orthogonal to status — this record’s own editorial position. The useful cases are the corners: Deferred + converged is a practice everybody follows that we have not satisfied ourselves is deliberate, and Active + contested is this record being ahead of the field and saying so, which an anthology should be able to do (ADR-015).
Not judged — nobody has assessed where the field stands on this; the default, and true of most of the corpus.
171 of 449 SOTA documents — the default when consensus: is absent. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| SOTA-002 v2 | Common hyperparameters: β₁=0.9, β₂=0.999, ε=1e-8 | Kingma et al. (2014), LIT-001 — ARXIV-1412.6980. | Active |
| SOTA-003 | Learning rate typically 1e-4 to 1e-3 for most tasks | Kingma et al. (2014), LIT-001 — ARXIV-1412.6980. | Active |
| SOTA-004 | Place BatchNorm after linear/conv layers but before activation functions | Ioffe et al. (2015), LIT-002 — ARXIV-1502.03167. | Active |
| SOTA-005 | Use running statistics for inference | Ioffe et al. (2015), LIT-002 — ARXIV-1502.03167. | Active |
| SOTA-006 | Consider alternatives like LayerNorm for transformers | Ioffe et al. (2015), LIT-002 — ARXIV-1502.03167. | Active |
| SOTA-007 v2 | BPE tokenization for open vocabulary tasks | Sennrich et al. (2015), LIT-003 — ARXIV-1508.07909. | Active |
| SOTA-008 v3 | linear warmup of LR stabilizes early training with large batch size. | You et al. (2017), LIT-009 — ARXIV-1708.03888. | Active |
| SOTA-009 v2 | warmup to a large early lr, anneal throughout training to small final lr | Smith et al. (2017), LIT-010 — ARXIV-1708.07120. | Active |
| SOTA-010 v2 | skip connections promote training stability by smoothing out the loss landscape | Li et al. (2017), LIT-014. Moved to THEORY-011 — the claim is a statement about why deep networks became trainable, not an instruction, and belongs in the scheme ADR-031 created for exactly that. This code stays so links into it resolve. | Superseded — by THEORY-011; Not a recommendation, and never was: it states what is true rather than what to do, which is a THEORY under ADR-031 and was one of the eight that decision named and declined to move. The claim is unchanged and is believed — it now lives at THEORY-011, with the figure’s limits and the five practices it underwrites. Superseded rather than Rejected because Rejected on a practice means do not do this, and nobody should read this code as advice against skip connections. ADR-034 is the rule. |
| SOTA-011 v3 | Map the Hessian ratio |lambda_min / lambda_max| to find where the loss surface is non-convex | Li et al. (2017), LIT-014 — ARXIV-1712.09913. | Active |
| SOTA-012 v3 | sharpness in the loss landscape correlates with test error | Li et al. (2017), LIT-014 — ARXIV-1712.09913. | Active |
| SOTA-013 | Use dynamic loss scaling that doubles every 2000 successful steps | Micikevicius et al. (2017), LIT-011 — ARXIV-1710.03740. | Active |
| SOTA-014 | Maintain master weights in FP32 | Micikevicius et al. (2017), LIT-011 — ARXIV-1710.03740. | Active |
| SOTA-015 v2 | Store optimizer states in FP32 | Micikevicius et al. (2017), LIT-011 — ARXIV-1710.03740. | Active |
| SOTA-016 | Perform forward/backward passes in FP16 | Micikevicius et al. (2017), LIT-011 — ARXIV-1710.03740. | Active |
| SOTA-017 v2 | Use micro-batch splitting for pipeline parallelism | Huang et al. (2018), LIT-016 — ARXIV-1811.06965. | Active |
| SOTA-018 v2 | Balance pipeline stages to minimize bubble overhead | Huang et al. (2018), LIT-016 — ARXIV-1811.06965. | Active |
| SOTA-019 v2 | Choose pipeline chunks based on memory vs. compute trade-off | Huang et al. (2018), LIT-016 — ARXIV-1811.06965. | Active |
| SOTA-020 | Use larger learning rates with batch normalization | Santurkar et al. (2018), LIT-015 — ARXIV-1806.02375. | Active |
| SOTA-021 | Monitor loss landscape smoothness during training | Santurkar et al. (2018), LIT-015 — ARXIV-1806.02375. | Rejected — states no threshold and no response, and the smoothing it refers to is a mechanism from a paper the record had conflated with this one. Measuring landscape smoothness during training is not something the practice’s source does or recommends; the finding it rests on is Santurkar et al.’s explanation of why batch normalization works, which is a fact about the technique rather than an instruction |
| SOTA-022 | Place BN after linear/conv layers but before activation functions | Santurkar et al. (2018), LIT-015 — ARXIV-1806.02375. | Superseded — by SOTA-004; a duplicate of SOTA-004, which states the same placement rule from batch normalization’s own paper. The two arrived in the same import from different notes and have coexisted since; SOTA-004 is the one with the originating source |
| SOTA-023 v2 | Use multi-query attention for decoder-only models to reduce memory bandwidth | Shazeer (2019), LIT-024 — ARXIV-1911.02150. | Superseded — by SOTA-109 |
| SOTA-024 v2 | Keep key/value projections shared across heads while query projections remain separate | Shazeer (2019), LIT-024 — ARXIV-1911.02150. | Superseded — by SOTA-109 |
| SOTA-025 v2 | Initialize the LayerNorm gain to 1 | Ba et al. (2016), LIT-005 — ARXIV-1607.06450. Identity at initialisation: the normalised signal passes through unchanged. A convention rather than a published result, and the record says so. | Active |
| SOTA-026 v2 | Initialize LayerNorm bias to 0 | Ba et al. (2016), LIT-005 — ARXIV-1607.06450. Zero is the identity; a convention rather than a published result, and moot in any model using RMSNorm. | Active |
| SOTA-027 v2 | Use a smaller learning rate for LayerNorm parameters | Xu et al. (2019), LIT-025 — ARXIV-1911.07013. | Rejected — The cited paper contains no discussion of learning rates and argues the parameters in question should be removed rather than tuned. The durable claim — treat norm parameters as their own group — is elsewhere in the record, and is about weight decay rather than rate |
| SOTA-028 | Stage optimizer states across data parallel ranks (ZeRO-1) | Rajbhandari et al. (2020), LIT-027 — ARXIV-1910.02054. | Active |
| SOTA-029 | Partition gradients and optimizer states (ZeRO-2) for larger models | Rajbhandari et al. (2020), LIT-027 — ARXIV-1910.02054. | Active |
| SOTA-030 | Use ZeRO-3 only when other strategies insufficient | Rajbhandari et al. (2020), LIT-027 — ARXIV-1910.02054. | Active |
| SOTA-031 v2 | Keep micro-batch size per GPU as large as memory allows | Rajbhandari et al. (2020), LIT-027 — ARXIV-1910.02054. | Active |
| SOTA-033 | continued pre-training for fine tuning | Gururangan et al. (2020), LIT-034 — ARXIV-2004.10964. | Active |
| SOTA-035 | use gradient clipping | Pascanu et al. (2012), LIT-037 — ARXIV-1211.5063. | Active |
| SOTA-036 v2 | Train a decoder-only transformer on a broad web corpus with a fixed context and a single next-token objective | Brown et al. (2020), LIT-035 — ARXIV-2005.14165. | Active |
| SOTA-037 | LM in-context learning emerges at scale | Brown et al. (2020), LIT-035 — ARXIV-2005.14165. | Active |
| SOTA-038 v3 | ICL permits few-shot task adaptability | Brown et al. (2020), LIT-035 — ARXIV-2005.14165. | Active |
| SOTA-039 | single cycle of cosine decay is sufficient lr schedule | Brown et al. (2020), LIT-035 — ARXIV-2005.14165. | Superseded — by SOTA-140; warmup-stable-decay matches it and leaves the token budget open; every report since 2024 in the record uses a stable-then-decay schedule |
| SOTA-040 | larger models are more sample efficient | Kaplan et al. (2020), LIT-028 — ARXIV-2001.08361. | Active |
| SOTA-041 v2 | lr tuning less important for larger models | Kaplan et al. (2020), LIT-028 — ARXIV-2001.08361. | Rejected — States the opposite of its source. Kaplan reports that larger models require a smaller learning rate to prevent divergence, and carries an explicit size-dependent rule LR(N) because the right rate is not size-independent |
| SOTA-042 | Memory-map large datasets | Mohan et al. (2020), LIT-050 — ARXIV-2007.06775. | Active |
| SOTA-043 | Use mixed precision during data loading | Mohan et al. (2020), LIT-050 — ARXIV-2007.06775. | Active |
| SOTA-044 | Pin memory for CPU-GPU transfers | Afroz et al. (2025), LIT-737 — ARXIV-2511.14124. Host-to-device copies from pinned memory ran at 24.74 GB/s against 10.16 GB/s from pageable memory on PCIe 4.0; the advice itself is NVIDIA’s, not a paper’s. | Active |
| SOTA-045 | Profile data loading separate from training | Mohan et al. (2020), LIT-050 — ARXIV-2007.06775. | Active |
| SOTA-046 | Use hierarchical allreduce for tensors > 1MB | Jiang et al. (2020), LIT-051 — https://www.usenix.org/conference/osdi20/presentation/jiang. | Rejected — the ’> 1MB’ crossover has no owner: NCCL measures the algorithm choice per topology rather than publishing a constant, and LIT-051 does not discuss it. What survives is that a modern collectives library already chooses by size and topology, which is not a practice a reader acts on |
| SOTA-047 | Overlap communication with backward pass | Jiang et al. (2020), LIT-051 — https://www.usenix.org/conference/osdi20/presentation/jiang. | Active |
| SOTA-048 | Group small tensors before communication | Jiang et al. (2020), LIT-051 — https://www.usenix.org/conference/osdi20/presentation/jiang. | Active |
| SOTA-049 | Set buffer size to network bandwidth-delay product | Jiang et al. (2020), LIT-051 — https://www.usenix.org/conference/osdi20/presentation/jiang. | Rejected — collapses two different quantities into one unfollowable sentence — a TCP socket buffer sized by bandwidth-delay product, and a gradient fusion buffer sized by the overlap trade. The fusion-buffer half is SOTA-048, which now carries Horovod’s attributable 64 MB default; the transport half is a claim about networks with no source here |
| SOTA-050 v2 | Scale attention weights by 1/sqrt(head_dim) | Vaswani et al. (2017), LIT-008 — ARXIV-1706.03762. | Active |
| SOTA-051 v5 | Initialize a residual or adapter branch to exactly zero, not merely near zero | Bachlechner et al. (2020), LIT-047 — ARXIV-2003.04887. Start an added branch at exactly zero so it is the identity at initialisation, and let training raise it. Exact zero rather than small-random is the claim: a small random branch is PROBABLY harmless, a zero one is PROVABLY the identity — which is what matters when the thing being protected is a residual stack deep enough to attenuate, or a pretrained model expensive enough that adapter noise is not worth risking. | Active |
| SOTA-052 | Use smaller variance for deep networks | Wang et al. (2022), LIT-084 — ARXIV-2203.00555. | Active |
| SOTA-053 | Special handling for gated architectures | Bachlechner et al. (2020), LIT-047 — ARXIV-2003.04887. | Active |
| SOTA-054 v2 | Derive the checkpoint interval from online profiling and adapt it at runtime against an overhead bound | Mohan et al. (2021), LIT-059 — https://www.usenix.org/conference/fast21/presentation/mohan. | Active |
| SOTA-055 | Save optimizer state every N epochs (N ~ sqrt(total_epochs)) | Mohan et al. (2021), LIT-059 — https://www.usenix.org/conference/fast21/presentation/mohan. | Rejected — the practice its own cited source exists to replace. LIT-059 names epoch-granular, hand-tuned checkpointing as the problem it solves, on both counts; SOTA-054 now states what that paper actually recommends |
| SOTA-056 | Use async I/O for checkpoint writing | Mohan et al. (2021), LIT-059 — https://www.usenix.org/conference/fast21/presentation/mohan. | Active |
| SOTA-057 | Implement multi-level checkpoint strategy | Mohan et al. (2021), LIT-059 — https://www.usenix.org/conference/fast21/presentation/mohan. | Active |
| SOTA-058 v2 | Use sequence parallelism for the layer-norm and dropout regions that tensor parallelism leaves replicated | Korthikanti et al. (2022), LIT-736 — ARXIV-2205.05198. | Active |
| SOTA-059 | Overlap communication with computation when possible | Jangda et al. (2021), LIT-735 — ARXIV-2105.05720. | Active |
| SOTA-060 v4 | Initialize weights from N(0, 0.02) and scale the residual-output projections by 1/sqrt(2N) | Shoeybi et al. (2019), LIT-022 — ARXIV-1909.08053. | Active |
| SOTA-061 v3 | Use largest batch that maintains >80% sample efficiency | McCandlish et al. (2018), LIT-017 — ARXIV-1812.06162. The steps/examples trade the rule thresholds; the 80% line itself has no published origin the record can name. | Active |
| SOTA-062 v3 | Scale batch size with model size but sub-linearly | Kaplan et al. (2020), LIT-028 — ARXIV-2001.08361. Along the compute-optimal frontier the model grows as C^0.73 and the batch as C^0.24: over a billion-fold increase in compute, more than a million-fold in model size against about a hundred-fold in batch. | Superseded — by SOTA-258; The decoupling experiment this practice’s own body invited has been run: holding the token budget fixed, the dependence of batch size on model size very nearly vanishes. What the heuristic was tracking is the data that scaled alongside the model. Superseded rather than Rejected because following it along the Chinchilla line gives roughly the right answer for the wrong reason — the successor is SOTA-258. |
| SOTA-063 v3 | use RoPE for LLM (1D sequence) positional embeddings | Su et al. (2021), LIT-045 — ARXIV-2104.09864. | Active |
| SOTA-064 v2 | Warmup needed scales sub-linearly with model size | Tay et al. (2021), LIT-052 — ARXIV-2109.10686. Rejected: the source contains no discussion of warmup. | Rejected — The cited paper does not contain the word “warmup”. Its subject is model shape for downstream fine-tuning; this claim came from a takeaway that belonged to no identifiable source, and no paper in the record supports it |
| SOTA-065 v2 | Initialize layer norm weights closer to 1 for larger models | Tay et al. (2021), LIT-052 — ARXIV-2109.10686. Rejected: the source contains no discussion of layer normalization, and the claim names no adjustable quantity. | Rejected — The cited paper contains no discussion of layer normalization. The claim is also doubtful on its own terms: LayerNorm’s gain initialises to exactly 1 by convention at every scale, so “closer to 1 for larger models” names no adjustable quantity |
| SOTA-066 v2 | Can use shorter warmup periods for wider models | Tay et al. (2021), LIT-052 — ARXIV-2109.10686. Rejected: the source contains no discussion of warmup. | Rejected — The same gap as SOTA-064, restated as a width claim. The cited paper contains no discussion of warmup at all |
| SOTA-067 v2 | Monitor loss specifically during first ~5000 steps for instabilities | Tay et al. (2021), LIT-052 — ARXIV-2109.10686. Rejected: the source contains no discussion of instability, and the 5000 belongs to nobody. | Rejected — The cited paper contains no discussion of training instability. The record’s sourced monitoring practices are SOTA-069 and SOTA-099, which do not depend on a step count |
| SOTA-068 v2 | Use gradient clipping during early training phase | Tay et al. (2021), LIT-052 — ARXIV-2109.10686. Superseded by SOTA-035, which cites Pascanu et al. and drops the “early training” qualifier. | Superseded — by SOTA-035; SOTA-035 is the same recommendation with a real source and without the false qualifier — clipping is applied throughout training, not only early |
| SOTA-069 | Monitor exp(loss) for stability | Zeng et al. (2022), LIT-054 — ARXIV-2210.02414. | Active |
| SOTA-070 | Track gradient norm ratios between layers | Zeng et al. (2022), LIT-054 — ARXIV-2210.02414. | Active |
| SOTA-071 | Use gradient clipping with dynamic threshold | Zeng et al. (2022), LIT-054 — ARXIV-2210.02414. | Active |
| SOTA-072 | Implement early warning system for NaNs | Zeng et al. (2022), LIT-054 — ARXIV-2210.02414. | Active |
| SOTA-073 | Monitor network utilization during training | Jiang et al. (2020), LIT-051 — https://www.usenix.org/conference/osdi20/presentation/jiang. | Rejected — states no signal, threshold or action, so there is nothing to do differently on reading it, and LIT-051 does not argue for it. The specific version worth having is in SOTA-048’s body: the gap between achieved and peak bandwidth during the overlap window tells you the fusion buffer is sized wrong |
| SOTA-074 | Adapt buffer sizes to network conditions | Jiang et al. (2020), LIT-051 — https://www.usenix.org/conference/osdi20/presentation/jiang. | Rejected — names a controller — signal, target, update rule — of which none is stated, and no source here describes one. A measured version would resemble what CheckFreq does for checkpoint intervals (SOTA-054) |
| SOTA-075 | Use gradient compression for slow networks | Lin et al. (2017), LIT-056 — ARXIV-1712.01887. | Active |
| SOTA-076 | Place replicas to minimize cross-rack traffic | Jiang et al. (2020), LIT-051 — https://www.usenix.org/conference/osdi20/presentation/jiang. | Rejected — a cluster-scheduling claim rather than a training one: it is the scheduler’s decision, expressed as a placement-group request, not something in a training run. ADR-026 (superseding ADR-020) scopes this record by the kind of claim, and this is a different kind |
| SOTA-077 | Use tar archives for dataset storage | Aizman et al. (2020), LIT-053 — ARXIV-2001.01858. | Active |
| SOTA-078 | Buffer size should be 2-3x batch size | Aizman et al. (2020), LIT-053 — ARXIV-2001.01858. | Active |
| SOTA-079 | Pre-fetch next batch during compute | Aizman et al. (2020), LIT-053 — ARXIV-2001.01858. | Active |
| SOTA-080 | Use multiple worker processes (num_workers = 4 * num_gpus) | Aizman et al. (2020), LIT-053 — ARXIV-2001.01858. | Active |
| SOTA-081 | Use operator fusion for small operations | Chen et al. (2018), LIT-063 — ARXIV-1802.04799. | Active |
| SOTA-082 | Optimize memory layout for hardware | Chen et al. (2018), LIT-063 — ARXIV-1802.04799. | Active |
| SOTA-083 v2 | Implement custom kernels for critical ops | Chen et al. (2018), LIT-063 — ARXIV-1802.04799. | Active |
| SOTA-084 | Profile-guided optimization for hot paths | Chen et al. (2018), LIT-063 — ARXIV-1802.04799. | Active |
| SOTA-085 v4 | Use flash attention for all attention computations when hardware supports it | Dao et al. (2022), LIT-074 — ARXIV-2205.14135. | Active |
| SOTA-086 v4 | Tiling size should match hardware SRAM size | Dao et al. (2022), LIT-074 — ARXIV-2205.14135. | Active |
| SOTA-087 v4 | Recompute attention during backward pass instead of storing it | Dao et al. (2022), LIT-074 — ARXIV-2205.14135. | Active |
| SOTA-088 | Fuse small operations into larger kernels | Ivanov et al. (2020), LIT-066 — ARXIV-2007.00072. | Active |
| SOTA-089 | Align tensor dimensions to hardware boundaries | Ivanov et al. (2020), LIT-066 — ARXIV-2007.00072. | Active |
| SOTA-090 | Use hardware-specific memory layouts | Ivanov et al. (2020), LIT-066 — ARXIV-2007.00072. | Active |
| SOTA-091 | Profile and optimize memory access patterns | Ivanov et al. (2020), LIT-066 — ARXIV-2007.00072. | Active |
| SOTA-092 | smaller batch sizes are more sample efficient (i.e., better loss as a function of tokens seen) earlier in training | Chowdhery et al. (2022), LIT-069 — ARXIV-2204.02311. | Active |
| SOTA-093 | larger batch sizes are beneficial later in training due to better gradient estimates | Chowdhery et al. (2022), LIT-069 — ARXIV-2204.02311. | Active |
| SOTA-094 | count accelerator efficiency alongside sample efficiency when choosing a batch size: larger batches mean larger matrix multiplications | Chowdhery et al. (2022), LIT-069 — ARXIV-2204.02311. | Active |
| SOTA-095 | consider rewinding to earlier checkpoint and skipping a few batches to mitigate unusual loss spikes | Chowdhery et al. (2022), LIT-069 — ARXIV-2204.02311. | Active |
| SOTA-096 v2 | num_tokens ~ 20 * num_params | Hoffmann et al. (2022), LIT-068 — ARXIV-2203.15556. | Active |
| SOTA-097 v4 | Optimal batch size scales approximately with compute budget - B ∝ C^(1/4) | Kaplan et al. (2020), LIT-028 — ARXIV-2001.08361. Equation 1.7 gives B ∝ C^(α_C/α_B) = C^0.24, from α_C ≈ 0.050 and α_B ≈ 0.21. | Superseded — by SOTA-258; The exponent is a fit to a projection. Along the Chinchilla line N and D scale together, so C = 6ND moves with D and a power law in either describes the data; two independent groups measuring off that line find optimal and critical batch size scale in the token budget, not in compute. Superseded rather than Rejected because the fit is not an error and the quantity is real — the successor is SOTA-258, and ADR-017’s corrective succession is the shape. |
| SOTA-098 v2 | Monitor the training loss for unexpected spikes, for the whole run | Chowdhery et al. (2022), LIT-069 — ARXIV-2204.02311. The 540B model’s training loss spiked about 20 times, at irregular intervals, sometimes late, despite gradient clipping, and never in the smaller models. | Active |
| SOTA-099 | Track gradient norm statistics to detect training instabilities | Zeng et al. (2022), LIT-054 — ARXIV-2210.02414. | Active |
| SOTA-100 v2 | Use learning rate warmup proportional to model size | Xiong et al. (2020), LIT-114 — ARXIV-2002.04745. | Active |
| SOTA-101 v2 | Use perplexity-based filtering for quality assessment | Albalak et al. (2023), LIT-117 — ARXIV-2312.02406. | Rejected — The cited paper performs no filtering — it reweights sampling over domains — and filtering has the opposite failure mode to what it recommends. The string “filter” does not appear in it |
| SOTA-102 v2 | Implement dynamic temperature scaling for mixing | Albalak et al. (2023), LIT-117 — ARXIV-2312.02406. | Superseded — by SOTA-103; Names a real component of ODM’s policy imprecisely rather than something the paper does not do — the two were one practice split in the import, and SOTA-103 now describes the whole method |
| SOTA-103 v2 | Adjust data mixing proportions online from per-domain training loss | Albalak et al. (2023), LIT-117 — ARXIV-2312.02406. | Active |
| SOTA-104 v2 | Monitor domain coverage during training | Albalak et al. (2023), LIT-117 — ARXIV-2312.02406. | Rejected — Names no signal and no response, and is in tension with its own source: under ODM the sampling distribution is supposed to move away from uniform coverage. The string “domain coverage” does not appear in the paper |
| SOTA-105 | PagedAttention to accelerate batch inference for LLM sampling | Kwon et al. (2023), LIT-112 — ARXIV-2309.06180. | Active |
| SOTA-106 v3 | Use flash-attention-2 over original flash-attention when available | Dao et al. (2023), LIT-106 — ARXIV-2307.08691. | Active |
| SOTA-107 v3 | Keep sequence lengths multiple of 128 for best performance | Dao et al. (2023), LIT-106 — ARXIV-2307.08691. | Rejected — The 128 is traced and belongs to a different quantity. Every occurrence in the source is a head dimension or a block size; none is a sequence length, and “multiple of” and “divisible” appear zero times |
| SOTA-108 v3 | Pad attention masks to block boundaries for better hardware utilization | Dao et al. (2023), LIT-106 — ARXIV-2307.08691. | Rejected — Not in the cited paper. “padding”, “padded”, “pad”, “block boundar” and “divisible” appear zero times each; the body’s reasoning about skipping fully-masked blocks is correct and is not the source’s |
| SOTA-109 v3 | Prefer GQA to MQA or MHA | Ainslie et al. (2023), LIT-100 — group the query heads and give each group one key/value head: multi-query’s cache saving without multi-query’s quality loss, and uptrainable from an existing multi-head checkpoint. Still the default for a model not paying MLA’s implementation cost. | Active |
| SOTA-110 v3 | Consider Monarch Mixer in place of attention for very long sequence tasks | Fu et al. (2023), LIT-115 — ARXIV-2310.12109. | Active |
| SOTA-111 v4 | Use Monarch Mixer where attention’s quadratic cost is what bottlenecks training | Fu et al. (2023), LIT-115 — ARXIV-2310.12109. | Active |
| SOTA-112 v4 | Combine Monarch Mixer layers with standard attention rather than replacing it wholesale | Fu et al. (2023), LIT-115 — ARXIV-2310.12109. | Rejected — The source is attention-free and argues for replacement, not combination. “hybrid” occurs zero times in it; the hybrid argument belongs to the attention/SSM literature elsewhere in this record |
| SOTA-113 v2 | Use continuous batching for inference | Kwon et al. (2023), LIT-112 — ARXIV-2309.06180. | Active |
| SOTA-114 | Fuse attention operations where possible | Dao et al. (2022), LIT-074 — ARXIV-2205.14135. | Active |
| SOTA-115 | Overlap prefill and decode compute | Agrawal et al. (2023), LIT-103 — ARXIV-2308.16369. | Active |
| SOTA-116 | Use FSDP over DDP when model size exceeds single GPU memory | Zhao et al. (2022), LIT-083 — ARXIV-2304.11277. | Active |
| SOTA-117 | Overlap communication with computation using backward prefetch | Zhao et al. (2022), LIT-083 — ARXIV-2304.11277. | Active |
| SOTA-118 | Employ mixed precision to reduce memory usage | Zhao et al. (2022), LIT-083 — ARXIV-2304.11277. | Active |
| SOTA-119 | Choose sharding factor based on model and GPU memory size | Zhao et al. (2022), LIT-083 — ARXIV-2304.11277. | Active |
| SOTA-121 v3 | Use Muon with decoupled weight decay and AdamW-matched update RMS in place of AdamW | Falcon-LLM Team (2026), LIT-119 — the Falcon-H1-Tiny technical blogpost. Stable at nearly the same optimal LR as AdamW, better evaluations; used for every Falcon-H1-Tiny model. | Active |
| SOTA-123 | For tiny specialized models, pretrain from scratch on the target SFT or reasoning data instead of pretrain-then-finetune | Falcon-LLM Team (2026), LIT-119 — the Falcon-H1-Tiny technical blogpost. At 90M, 25% SFT data in the pretraining mix beat a separate SFT stage by 10 IFEval points and yields one checkpoint that is both base and instruct. | Active |
| SOTA-126 | Run DPO on tiny models for one epoch only | Falcon-LLM Team (2026), LIT-119 — the Falcon-H1-Tiny technical blogpost. One epoch at LR 1e-6 to 3e-6 took a 90M model’s IFEval from about 50 to over 65; a second epoch degraded it while the reward kept rising. | Active |
| SOTA-127 v2 | Filter chain-of-thought traces out of the training data of tiny specialized models | Falcon-LLM Team (2026), LIT-119 — the Falcon-H1-Tiny technical blogpost. Reasoning traces interleaved with tool-calling data sent a 90M model into repetition loops; removing them fixed it at once. | Active |
| SOTA-128 | Train fill-in-the-middle data without masking the loss on non-FIM tokens | Falcon-LLM Team (2026), LIT-119 — the Falcon-H1-Tiny technical blogpost. At 80 GT and 90M, computing the loss on every token beat masking the prefix and suffix; matches what the reference FIM recipes appear to do. | Active |
| SOTA-129 v2 | Build a reasoning model in three stages: pretrain on general data, SFT on reasoning traces, then RL with verifiable rewards | Olmo Team (2025), LIT-130 — the recipe behind every open reasoning model at 7B and above, released in Olmo 3 with every stage and checkpoint in the open. Filed so its variations (SOTA-130) and its tiny-scale successor (SOTA-123) have something to be traced back to. | Active |
| SOTA-131 v4 | When training with Muon at scale, rescale query and key weights whenever attention logits exceed a threshold (QK-Clip) | Kimi Team (2025), LIT-132 — MuonClip carried a 1T/32B MoE through 15.5T tokens with zero loss spikes where plain Muon let attention logits pass 1000; confirmed at 2.8T in LIT-131. | Active |
| SOTA-134 v2 | Gate each attention head’s output with a sigmoid after the scaled dot-product | Qiu et al. (2025), LIT-138 — the best of 30 gating variants at 15B MoE and 1.7B dense over 3.5T tokens: better quality, more stable training, larger tolerable learning rates, no attention sinks; shipped in every Qwen full-attention layer since Qwen3-Next. | Active |
| SOTA-135 v2 | Give linear-attention layers a gated delta rule: a decay gate for erasure plus a delta update for targeted writes | Yang et al. (2024), LIT-137 — beats Mamba2 and DeltaNet across language modelling, retrieval and long context at 1.3B/100B; the recurrence the Qwen hybrids use for three layers in four, and the one Kimi Delta Attention extends. | Active |
| SOTA-137 | Widen the residual stream into several streams with freely learned mixing (hyper-connections) | Zhu et al. (2024), LIT-141 — n parallel residual streams with learnable mixing; gains on dense and MoE pretraining, and an identity-mapping property lost that its successor restores. | Superseded — by SOTA-136; constraining the mixing to a doubly-stochastic manifold keeps the width and restores the identity mapping this gave up |
| SOTA-138 v2 | Train sparse attention natively with a learned top-k indexer, warmed up under dense attention | DeepSeek-AI (2025), LIT-142 — a small FP8 indexer scores past tokens and only the top-k enter attention; initialised for 2.1B tokens with dense attention on and the rest frozen; V4 repeats the pattern with a 1T-token dense warm-up. | Active |
| SOTA-139 | Extend the context length in stages during pretraining rather than training at the target length from the start | DeepSeek-AI (2026), LIT-139 — 4K, then 16K, 64K and 1M over 32–33T tokens, with sparse attention introduced at the 64K stage; Kimi K3 (LIT-131) reaches 1M the same way. | Active |
| SOTA-142 v3 | Set the peak learning rate by a power law in tokens so it transfers across batch size and training length | Shen et al. (2024), LIT-146 — under WSD the optimal learning rate is a power law in tokens and batch size; the Power scheduler applies it, and with µP the same setting carries across model sizes; adopted independently by Falcon-H1-Tiny. | Active |
| SOTA-143 v2 | Parameterize the model with µP and tune hyperparameters on a narrow proxy, transferring them across width | Yang and Hu (2022), LIT-148 — under the Maximal Update Parametrization the optimal learning rate and friends are stable across width, so tune small and transfer zero-shot; GPT-3 6.7B beaten at 7% of its pretraining cost in tuning. Used by Falcon-H1, MiniCPM and the Power scheduler. | Active |
| SOTA-189 | Set the checkpoint interval from the job’s measured time-to-failure, which shrinks as the job grows | CoreWeave (2025), LIT-222. Expected work lost to a failure is half the inter-checkpoint interval, and mean time to failure falls linearly with GPU count — so the interval belongs to the job’s size and the cluster’s failure rate, not to the write cost alone. | Proposed |
| SOTA-208 | Factorize long-sequence attention into a local window and a second head that escapes it | Child et al. (2019), LIT-225 — ARXIV-1904.10509. Superseded: the field went to exact attention made fast (SOTA-086, SOTA-087) and then to sparsity that is LEARNED rather than fixed (SOTA-138). What survives is the connectivity requirement — a window alone is not the design — which every hybrid the record recommends is an instance of. | Superseded — by SOTA-138 |
| SOTA-209 | Initialize a mixture-of-experts model from a dense checkpoint rather than training it from scratch | Komatsuzaki et al. (2022), LIT-227 — seed the experts from a dense checkpoint you already paid for. Upcycled T5 and ViT models beat their dense counterparts at ~50% of the dense pretraining sunk cost, and beat sparse models trained from scratch on 100% of it: same architecture, same compute, different initialization. | Proposed |
| SOTA-241 | Rank candidate data selections with a model-free distributional proxy before spending a training run | Xie et al. (2023), LIT-087 — KL reduction in a hashed n-gram space correlates with downstream performance across selection methods, including manual curation, which uses no n-grams at all. A selection can therefore be ranked against its alternatives before anything is trained on it. | Proposed |
| SOTA-244 | Score example importance a few epochs into training, averaged over several initializations | Paul et al. (2021), LIT-397 — the importance ranking is available from a single early checkpoint, not only from a finished run. EL2N at epoch 20, averaged over ten initializations, prunes half of CIFAR-10 with accuracy slightly up, and the ranking transfers to other architectures. | Proposed |
| SOTA-245 | State a relation in both orders in the training corpus if you want it usable in both directions | Grosse et al. (2023), LIT-401 — a training sequence influences a completion only when the phrase related to the prompt comes before the phrase related to the completion. Identical content in the reverse order scores barely above an unrelated baseline. | Proposed |
| SOTA-272 | Expect the loss to be non-monotone at the step size that trains fastest, and do not set the step size from a curvature bound | Cohen et al. (2021), LIT-461 — gradient descent raises the sharpness until its own step size cannot tolerate more, so a rule that anneals the step size to the measured curvature is chasing a number it is producing, and loses to the fixed step it forbids. Demonstrated full-batch. | Proposed |
| SOTA-273 | Choose a pretraining context whose association with the input is neither too strong nor too weak, and mix contexts to get there | Zhai (2025), LIT-459 — a pretraining objective is an association between the input and a context variable, and the useful contexts are the ones whose association is neither too strong nor too weak: the singular values of the expectation operator should decay neither too fast nor too slowly. Contexts that individually miss the range can be mixed into one that does not. | Proposed |
| SOTA-274 v2 | Before adopting a spectral optimizer, measure the stable rank of each block’s incoming activations against its gradient’s nuclear rank | Davis and Drusvyatskiy (2025), LIT-457 — the spectral step’s one-step guarantee beats the Euclidean one by the ratio of gradient nuclear rank to incoming activation stable rank. Both are cheap to compute from a run you are already doing, and the paper never uses them to route. | Proposed |
| SOTA-275 | Pick the depth parameterization from the residual branch: a Transformer branch has more than one transformation, so it needs the stricter rule | Zheng et al. (2026), LIT-462 — Depth-muP’s α = Θ(1/√L) is the rule for a one-transformation residual branch. Attention and FFN branches hold more than one, so Transformers need CompleteP’s α = Θ(1/L), and the looser rule measurably fails to transfer on them. | Proposed |
| SOTA-276 | For a normalized or preconditioned optimizer, width-depth muP is width muP plus a hidden residual multiplier of order 1/L | Zheng et al. (2026), LIT-462 — a normalized or preconditioned update has a norm that does not depend on the residual multiplier, so the depth factor the raw gradient carries is removed and the optimizer’s own muP rule is unchanged. SGD is the exception and needs more. | Proposed |
| SOTA-277 | For a task that is a sequential composition, buy depth rather than width | Chen et al. (2024), LIT-464 — composing k functions costs a constant-depth decoder polynomially many parameters and a log k-layer one polylogarithmically. An exponential separation, proved unconditionally, and about what a shape can express rather than what a run will learn. | Proposed |
| SOTA-293 | Count embedding parameters when you report a tiny model’s size | Finke et al. (2025), LIT-485 — at these sizes the embedding table is not a rounding error, it is most of the model: TinyStories-33M is 68M parameters all-in, and an 8M model in the same paper is “more than half embeddings”. Two papers can each claim the smaller model and both be telling the truth. Print the all-in figure. | Proposed |
| SOTA-305 v2 | State the token count and input resolution beside any reconstruction FID, and compare tokenizers only at equal rate | Sun et al. (2024), LIT-497 — the same tokenizer at downsample 16 scores rFID 2.19, 0.94 and 0.70 on 256, 576 and 1024 tokens, all evaluated on 256×256 reconstructions. A three-fold range from a knob that is not the tokenizer. Two papers in this line then quote reconstruction numbers whose rate a reader cannot recover from the sentence they appear in. | Active |
| SOTA-311 | Evaluate a search on tasks held out of its own fitness function, and report the gap against the tasks it selected on | No source states this; it is filed on a priori grounds with one instance. Cheng, Clark and Richardson (2025), LIT-493, searched 1,062 pretrained designs against a fitness function of average downstream accuracy, then reported the top five on nine benchmarks drawn from the same pool and chosen — per its own Table 16 — for largest standard deviation across the search population. The best discovered design beats the best seed by 0.03 points. | Active |
| SOTA-313 | Treat the inference-time decision procedure as part of what you are comparing, and report the ordering under each one | Gemini Team (2023), LIT-510 — on MMLU, greedy sampling puts GPT-4 ahead by 0.2, plain chain-of-thought at 32 samples puts it ahead by 2.3, and uncertainty-routed CoT at 32 samples puts Gemini Ultra ahead by 2.7. Same models, same benchmark, same harness. The procedure is worth 6.0 points to one model and 3.1 to the other, so running the same procedure on both is not enough — the ordering is a property of the procedure you chose to report. | Active |
| SOTA-321 | Verify a model’s lineage from the singular spectra of its attention products, which survive permutation and rescaling | Wang et al. (2025), LIT-524 — compare the per-layer singular spectra of W_q W_k^T and W_v W_o. Those products absorb the permutation and rescaling that change weights without changing the function, so the spectra identify a model through fine-tuning, pruning, merging and expansion. F1 0.9867 over 55 model pairs, using no data and no queries. | Proposed |
| SOTA-323 | Test a claim about what pretraining produces on a model trained with the pretraining objective, not one trained on the task family you are testing | Shen, Mishra and Khashabi (2023), LIT-536 — “there exist weights such that” and “pretraining produces weights such that” are different claims, and training a model on the task family you are about to test collapses them. Their separation of these as Hypothesis 2 and Hypothesis 1 is what let them show that a literature everyone read as evidence about pretrained language models had tested neither the models nor the objective. | Proposed |
| SOTA-324 | Separate rival accounts of an internal algorithm by convergence rate and conditioning, not by how well each fits the output | Fu, Chen, Jia and Sharan (2023), LIT-535 — output similarity cannot tell gradient descent from Newton’s method, because both converge to the same answer. Matching each layer to the best-fitting step count of each candidate gives a linear trend for the right rate and an exponential one for the wrong one; a regime where the rivals must differ — here, condition number 100 — separates them again. Two measurements, both cheap, both decisive where a fit was not. | Proposed |
| SOTA-325 | When the training loss has saturated and runs still differ, measure loss-landscape degeneracy rather than curvature | Lau, Furman, Wang, Murfet and Wei (2023), LIT-542 — on ResNet18/CIFAR10, stronger implicit regularization (higher learning rate, lower batch size, higher momentum) gives a lower local learning coefficient and higher test accuracy, while every training loss has collapsed to zero. The estimator is λ̂(w*) = n β* [E_{w|w*,β*,γ} L_n(w) − L_n(w*)] at β* = 1/log n, run by SGLD, validated against theory on deep linear networks to 100M parameters. Where the loss has stopped distinguishing runs, the geometry has not. | Proposed |
| SOTA-339 | For unpaired image-to-image translation of appearance, constrain both mappings with a cycle-consistency loss | Zhu et al. (2017), LIT-564 — learn X → Y and Y → X together, and penalize ‖F(G(x)) − x‖₁ and ‖G(F(y)) − y‖₁ alongside the two adversarial losses. With GAN losses alone the mapping may ignore its input. With the cycle it keeps content and changes appearance. It is scoped to colour and texture changes, and the authors report failure on geometric ones. | Proposed |
| SOTA-340 | Expect weight-only 4-bit quantization to speed up batched serving only while the batch keeps the matmul memory-bound, and use a kernel built to stay there | Frantar et al. (2024), LIT-567 — weight-only quantization speeds up decoding by loading fewer bytes, so the gain lasts only while the layer is memory-bound. A kernel designed to stay memory-bound (MARLIN) holds about 3.9× over FP16 up to batch 16–32 on an A10, decaying toward 1.5× at 128. End-to-end in vLLM, single-GPU, 2.3–3.2× at batch 16 or below and 1.1–1.2× at 128, on four GPU classes. Pick the format for the batch you actually serve, and pay for it in accuracy only where it buys speed. | Proposed |
| SOTA-341 | Condition an image generator on each training image’s original size instead of discarding or upsampling small images | Podell et al. (2023), LIT-566 — embed each training image’s original height and width, like the timestep, and give it to the model. Then keep the small images that a minimum-resolution filter would drop (39% of SDXL’s pretraining data below 256²) without teaching the model their blur. At inference, set the size you want. On class-conditional ImageNet 512²: FID-5k 43.84 discarding, 39.76 keeping unconditioned, 36.53 keeping with size conditioning. | Proposed |
| SOTA-342 | Report a sparse autoencoder’s fidelity as downstream loss in compute-equivalent terms, not as fraction of loss recovered against zero ablation | Gao et al. (2024), LIT-571 — splice the SAE’s reconstruction into the forward pass and report the resulting loss as the pretraining compute a model of that loss would need. Zero-ablating the residual stream is so destructive that “loss recovered” against it flatters any reconstruction. Their 16M-latent GPT-4 SAE scores 98.2% on it, and equals a GPT-4 trained on 10% of the compute. | Proposed |
| SOTA-344 | Train sparse autoencoders with a TopK activation instead of an L1 penalty, and prevent dead latents with transposed-decoder initialization and an auxiliary loss | Gao et al. (2024), LIT-571 — keep the k largest pre-activations and zero the rest, and train on reconstruction alone. L0 is then set, not tuned through an L1 coefficient that also shrinks the latents that fire. It beats ReLU SAEs on the MSE–L0 frontier by more as SAEs grow. Initialize the encoder as the decoder’s transpose and add a loss reconstructing the residual from dead latents: 7% dead at 16M latents, against up to 90%. | Proposed |
| SOTA-346 | Retune the training timestep shift for each autoencoder latent space, and never rank latent spaces for generation under one shared schedule | Black Forest Labs (2025), LIT-572 — four autoencoders, 30 timestep configurations each. The training shift alone moves FID by 61–86%. An RAE latent that wins when both are tuned loses to a tuned FLUX.2 latent when left unshifted. When you swap the autoencoder under a diffusion or flow model, sweep the shift again, using a logit-normal training distribution. When you compare autoencoders, compare each at its own optimum. | Proposed |
| SOTA-350 | Compare GNN architectures over many random data splits and seeds, with one shared training and hyperparameter-selection procedure for every model, never on a single fixed split | Shchur et al. (2018), LIT-581 — on the Planetoid split GAT beats GCN. On another random split of the same size GCN wins, and averaged over 100 splits × 20 seeds with identical training and tuning, GCN ranks first of four GNNs. A single split selects the model that overfits it, and per-model recipes measure the recipe. Report mean and spread over many splits and seeds, with the same tuning budget for every model. | Active |
| SOTA-351 | In a message-passing GNN for graph-level tasks, aggregate neighbours by sum with an MLP update, not by mean or max, when graph structure rather than node features carries the signal | Xu et al. (2019), LIT-580 — mean and max aggregation cannot count neighbors, so graphs that differ only in counts look the same to them. On featureless Reddit graphs, mean-aggregation GNNs are at chance (50.0, 20.0) and sum–MLP scores 92.4 and 57.5. With informative node features the choice matters much less, and results are mostly within noise. Use h = MLP((1 + ε)h + Σ neighbors) with ε = 0 and a sum readout. | Active |
| SOTA-352 | When a structure predictor is made generative, add a regression model’s predictions to the training data so disordered regions are learned as disorder rather than invented structure | Abramson et al. (2024), LIT-584 — a diffusion head generates plausible compact structure even where the protein has none. AlphaFold- Multimer, a regression model, renders the same regions as extended loops. Mixing its predictions into training teaches the generative model that convention, and “greatly reduced” hallucination. Remaining hallucinations are flagged by low confidence but do not look disordered. | Proposed |
| SOTA-353 | Train a head that predicts the model’s own accuracy on each output, against the metric you care about, and use it to rank samples and filter predictions | Jumper et al. (2021), LIT-583 — regress the per-residue lDDT the prediction will score (pLDDT) and the predicted TM-score. They track truth at r = 0.76 and 0.85 over 10,795 chains, and pLDDT filters the self-distillation set. Abramson et al. (2024), LIT-584, keep the head under diffusion by rolling out a cheap sample during training. They rank seeds by it, and antibody-interface quality keeps rising up to 1,000 seeds. | Proposed |
| SOTA-354 | When unlabelled inputs vastly outnumber labelled ones, retrain from scratch on the labelled data mixed with the model’s own high-confidence predictions for unlabelled inputs, under augmentation that stops it copying them | Jumper et al. (2021), LIT-583 — about 100,000 solved structures and billions of sequences. AlphaFold 2 predicted structures for about 350,000 unlabeled sequences, kept the confident ones, and retrained the same architecture from scratch on those plus the PDB. Cropping and alignment subsampling mean the student cannot just reproduce the teacher. The ablation shows a clear gain. The filter is the model’s own calibrated confidence (SOTA-353). | Proposed |
| SOTA-366 | Monitor the standard deviation of the l2-normalized embedding; healthy is one over root d and collapsed is zero | Active | |
| SOTA-371 | Score topic coherence automatically as the average pointwise mutual information of a topic’s top-word pairs, estimated on a large external reference corpus | Newman et al. (2010), LIT-600 — when you cannot put every topic in front of people (SOTA-368), approximate their judgment. Take the topic’s top ten words and, for each of the 45 pairs, compute PMI from co-occurrence in 10-word windows over a large general corpus such as Wikipedia. Average the 45 scores. On 237 LDA topics this correlated with nine people’s ratings at ρ ≈ 0.77, far better than WordNet measures. | Proposed |
| SOTA-380 | Use prediction-based word vectors rather than count-based distributional vectors | Baroni, Dinu and Kruszewski (2014), LIT-608 — across 14 benchmarks, word2vec CBOW beat PPMI, SVD and NMF count vectors almost everywhere, and the authors recommend “anybody interested in using DSMs” to “go for the predict models”. Retired: LIT-607 traces the gap to settings the count models were not given, and finds no consistent winner once they are. | Rejected — the advantage it reports disappears when the count-based baselines get the design choices word2vec shipped with and every method is tuned alike (LIT-607): no family wins consistently. Retired with no successor. What replaces it is an evaluation practice, SOTA-379, which corrects it |
| SOTA-387 | Hold out a slice of the training set for model selection instead of selecting on the validation split | Beyer et al. (2022), LIT-628 — they train on the first 99% of ImageNet-1k and keep 1% as a “minival”, stated as being “to encourage the community to stop selecting design choices on the validation (de-facto test) set.” A convention, asserted rather than measured, and the record held no document on it. | Proposed |
| SOTA-404 v2 | When you ablate an auxiliary loss, ablate the input construction that came with it, or the result is about both | Liu et al. (2019), LIT-671. BERT reported that removing next sentence prediction hurts; RoBERTa observed that the ablation had probably “only removed the loss term while still retaining the segment-pair input format”, separated the two, and found removing the loss matches or slightly improves while the input format was what mattered. LIT-667 is the same defect in another literature: four grokking papers varied the training fraction of a fixed universe, so dataset size and dataset composition moved together, and the conclusion reversed when somebody turned them one at a time. | Proposed |
| SOTA-420 | When distilling a classifier, match the teacher’s temperature-softened outputs at the same temperature and keep a down-weighted hard-label term | Hinton et al. (2015), LIT-680. Soft-target cross-entropy at temperature T in both teacher and student, plus a hard-label cross-entropy at T = 1 with a considerably lower weight. Multiply the soft term by T², so the balance does not move when T changes. Start low for a small student. The paper’s best realistic setting was T = 2, and a 30-unit MNIST net wanted 2.5–4. | Proposed |
| SOTA-428 | Choose the EMA length after training: store two power-function averages at snapshots and reconstruct any length by least squares | Karras et al. (2023), LIT-714 — EDM2. Track two power-function averages of the weights (σ_rel 0.05 and 0.10), save both every few thousand steps, and synthesize any EMA length after the run from a small linear solve. Then choose it per configuration, per guidance weight and per metric, because on ImageNet-512 the best length moves with all three — at guidance 1.4, FID wants 2% and FD_DINOv2 wants 14%. | Proposed |
| SOTA-429 | Drop half the image patches when training a CLIP-style model, spend the saving on more pairs and a larger batch, and unmask only for a short final tune | Li et al. (2022), LIT-706 — FLIP. Remove 50% of image patches and run the ViT on the rest, with CLIP’s loss and nothing else. The saving buys a 2× larger batch at the same memory and 2× the samples per hour; ViT-L/16 on LAION-400M reaches its unmasked reproduction’s accuracy more than 3× faster. The accuracy comes from what the saving buys: at equal batch, masking is parity. | Proposed |
| SOTA-431 | Make every layer of a diffusion denoiser preserve activation magnitude, and force each weight vector back to fixed norm after every step | Karras et al. (2023), LIT-714 — EDM2. Normalize each output channel’s weight vector on use, re-normalize the stored weights after every step, scale every fixed operation to preserve unit magnitude, and drop the group norms. On the ADM U-Net at ImageNet-512 this takes FID from 6.96 to 2.56 at equal compute, with on-use weight normalization alone worth 6.96 → 3.75. One architecture family; untested on transformers. | Proposed |
| SOTA-436 | Route a very large or non-grid input through cross-attention into a small learned latent array, so that depth no longer scales with input size | Jaegle et al. (2021), LIT-747 — the Perceiver. A learned array of 512 latents cross-attends to all 50,176 ImageNet pixels, at a cost of O(MN) rather than O(M²), and a 48-block latent transformer then runs at a cost independent of input size. It matched ResNet-50 and ViT-B/16 at 78.0% top-1 without convolutions, did not beat them, and ran unchanged on audio, video and point clouds. The case for doing it is cost and modality-independence, not accuracy, and the cost advantage is argued from complexity, not measured against a baseline. | Proposed |