Practice statuses — What a status means for a PRACTICE: whether this record says to do the thing today. It is not the standing of the paper underneath — that is lit-statuses, and the two are allowed to disagree — and it is not what the field does, which is consensus (ADR-002).
Current practice — what you should do today, and the reason is one click away.
236 of 449 SOTA documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| SOTA-001 v2 | Use Adam as the default optimizer absent a reason to choose otherwise | Kingma et al. (2014), LIT-001 — ARXIV-1412.6980. | Active |
| SOTA-002 v2 | Common hyperparameters: β₁=0.9, β₂=0.999, ε=1e-8 | Kingma et al. (2014), LIT-001 — ARXIV-1412.6980. | Active |
| SOTA-003 | Learning rate typically 1e-4 to 1e-3 for most tasks | Kingma et al. (2014), LIT-001 — ARXIV-1412.6980. | Active |
| SOTA-004 | Place BatchNorm after linear/conv layers but before activation functions | Ioffe et al. (2015), LIT-002 — ARXIV-1502.03167. | Active |
| SOTA-005 | Use running statistics for inference | Ioffe et al. (2015), LIT-002 — ARXIV-1502.03167. | Active |
| SOTA-006 | Consider alternatives like LayerNorm for transformers | Ioffe et al. (2015), LIT-002 — ARXIV-1502.03167. | Active |
| SOTA-007 v2 | BPE tokenization for open vocabulary tasks | Sennrich et al. (2015), LIT-003 — ARXIV-1508.07909. | Active |
| SOTA-008 v3 | linear warmup of LR stabilizes early training with large batch size. | You et al. (2017), LIT-009 — ARXIV-1708.03888. | Active |
| SOTA-009 v2 | warmup to a large early lr, anneal throughout training to small final lr | Smith et al. (2017), LIT-010 — ARXIV-1708.07120. | Active |
| SOTA-011 v3 | Map the Hessian ratio |lambda_min / lambda_max| to find where the loss surface is non-convex | Li et al. (2017), LIT-014 — ARXIV-1712.09913. | Active |
| SOTA-012 v3 | sharpness in the loss landscape correlates with test error | Li et al. (2017), LIT-014 — ARXIV-1712.09913. | Active |
| SOTA-013 | Use dynamic loss scaling that doubles every 2000 successful steps | Micikevicius et al. (2017), LIT-011 — ARXIV-1710.03740. | Active |
| SOTA-014 | Maintain master weights in FP32 | Micikevicius et al. (2017), LIT-011 — ARXIV-1710.03740. | Active |
| SOTA-015 v2 | Store optimizer states in FP32 | Micikevicius et al. (2017), LIT-011 — ARXIV-1710.03740. | Active |
| SOTA-016 | Perform forward/backward passes in FP16 | Micikevicius et al. (2017), LIT-011 — ARXIV-1710.03740. | Active |
| SOTA-017 v2 | Use micro-batch splitting for pipeline parallelism | Huang et al. (2018), LIT-016 — ARXIV-1811.06965. | Active |
| SOTA-018 v2 | Balance pipeline stages to minimize bubble overhead | Huang et al. (2018), LIT-016 — ARXIV-1811.06965. | Active |
| SOTA-019 v2 | Choose pipeline chunks based on memory vs. compute trade-off | Huang et al. (2018), LIT-016 — ARXIV-1811.06965. | Active |
| SOTA-020 | Use larger learning rates with batch normalization | Santurkar et al. (2018), LIT-015 — ARXIV-1806.02375. | Active |
| SOTA-025 v2 | Initialize the LayerNorm gain to 1 | Ba et al. (2016), LIT-005 — ARXIV-1607.06450. Identity at initialisation: the normalised signal passes through unchanged. A convention rather than a published result, and the record says so. | Active |
| SOTA-026 v2 | Initialize LayerNorm bias to 0 | Ba et al. (2016), LIT-005 — ARXIV-1607.06450. Zero is the identity; a convention rather than a published result, and moot in any model using RMSNorm. | Active |
| SOTA-028 | Stage optimizer states across data parallel ranks (ZeRO-1) | Rajbhandari et al. (2020), LIT-027 — ARXIV-1910.02054. | Active |
| SOTA-029 | Partition gradients and optimizer states (ZeRO-2) for larger models | Rajbhandari et al. (2020), LIT-027 — ARXIV-1910.02054. | Active |
| SOTA-030 | Use ZeRO-3 only when other strategies insufficient | Rajbhandari et al. (2020), LIT-027 — ARXIV-1910.02054. | Active |
| SOTA-031 v2 | Keep micro-batch size per GPU as large as memory allows | Rajbhandari et al. (2020), LIT-027 — ARXIV-1910.02054. | Active |
| SOTA-032 v5 | Put the layer normalization inside the residual block, before the sublayer | Xiong et al. (2020), LIT-114 — ARXIV-2002.04745. Pre-LN: normalize the input to each sublayer rather than the sum after it, so the gradients near the output are well behaved at initialization. | Active |
| SOTA-033 | continued pre-training for fine tuning | Gururangan et al. (2020), LIT-034 — ARXIV-2004.10964. | Active |
| SOTA-034 v6 | Use SwiGLU activation for transformers | Shazeer et al. (2020), LIT-030 — ARXIV-2002.05202. Contested at frontier scale in low precision: two groups now ship bounded replacements, on the grounds that SwiGLU’s unbounded factors produce activation outliers. | Active |
| SOTA-035 | use gradient clipping | Pascanu et al. (2012), LIT-037 — ARXIV-1211.5063. | Active |
| SOTA-036 v2 | Train a decoder-only transformer on a broad web corpus with a fixed context and a single next-token objective | Brown et al. (2020), LIT-035 — ARXIV-2005.14165. | Active |
| SOTA-037 | LM in-context learning emerges at scale | Brown et al. (2020), LIT-035 — ARXIV-2005.14165. | Active |
| SOTA-038 v3 | ICL permits few-shot task adaptability | Brown et al. (2020), LIT-035 — ARXIV-2005.14165. | Active |
| SOTA-040 | larger models are more sample efficient | Kaplan et al. (2020), LIT-028 — ARXIV-2001.08361. | Active |
| SOTA-042 | Memory-map large datasets | Mohan et al. (2020), LIT-050 — ARXIV-2007.06775. | Active |
| SOTA-043 | Use mixed precision during data loading | Mohan et al. (2020), LIT-050 — ARXIV-2007.06775. | Active |
| SOTA-044 | Pin memory for CPU-GPU transfers | Afroz et al. (2025), LIT-737 — ARXIV-2511.14124. Host-to-device copies from pinned memory ran at 24.74 GB/s against 10.16 GB/s from pageable memory on PCIe 4.0; the advice itself is NVIDIA’s, not a paper’s. | Active |
| SOTA-045 | Profile data loading separate from training | Mohan et al. (2020), LIT-050 — ARXIV-2007.06775. | Active |
| SOTA-047 | Overlap communication with backward pass | Jiang et al. (2020), LIT-051 — https://www.usenix.org/conference/osdi20/presentation/jiang. | Active |
| SOTA-048 | Group small tensors before communication | Jiang et al. (2020), LIT-051 — https://www.usenix.org/conference/osdi20/presentation/jiang. | Active |
| SOTA-050 v2 | Scale attention weights by 1/sqrt(head_dim) | Vaswani et al. (2017), LIT-008 — ARXIV-1706.03762. | Active |
| SOTA-051 v5 | Initialize a residual or adapter branch to exactly zero, not merely near zero | Bachlechner et al. (2020), LIT-047 — ARXIV-2003.04887. Start an added branch at exactly zero so it is the identity at initialisation, and let training raise it. Exact zero rather than small-random is the claim: a small random branch is PROBABLY harmless, a zero one is PROVABLY the identity — which is what matters when the thing being protected is a residual stack deep enough to attenuate, or a pretrained model expensive enough that adapter noise is not worth risking. | Active |
| SOTA-052 | Use smaller variance for deep networks | Wang et al. (2022), LIT-084 — ARXIV-2203.00555. | Active |
| SOTA-053 | Special handling for gated architectures | Bachlechner et al. (2020), LIT-047 — ARXIV-2003.04887. | Active |
| SOTA-054 v2 | Derive the checkpoint interval from online profiling and adapt it at runtime against an overhead bound | Mohan et al. (2021), LIT-059 — https://www.usenix.org/conference/fast21/presentation/mohan. | Active |
| SOTA-056 | Use async I/O for checkpoint writing | Mohan et al. (2021), LIT-059 — https://www.usenix.org/conference/fast21/presentation/mohan. | Active |
| SOTA-057 | Implement multi-level checkpoint strategy | Mohan et al. (2021), LIT-059 — https://www.usenix.org/conference/fast21/presentation/mohan. | Active |
| SOTA-058 v2 | Use sequence parallelism for the layer-norm and dropout regions that tensor parallelism leaves replicated | Korthikanti et al. (2022), LIT-736 — ARXIV-2205.05198. | Active |
| SOTA-059 | Overlap communication with computation when possible | Jangda et al. (2021), LIT-735 — ARXIV-2105.05720. | Active |
| SOTA-060 v4 | Initialize weights from N(0, 0.02) and scale the residual-output projections by 1/sqrt(2N) | Shoeybi et al. (2019), LIT-022 — ARXIV-1909.08053. | Active |
| SOTA-061 v3 | Use largest batch that maintains >80% sample efficiency | McCandlish et al. (2018), LIT-017 — ARXIV-1812.06162. The steps/examples trade the rule thresholds; the 80% line itself has no published origin the record can name. | Active |
| SOTA-063 v3 | use RoPE for LLM (1D sequence) positional embeddings | Su et al. (2021), LIT-045 — ARXIV-2104.09864. | Active |
| SOTA-069 | Monitor exp(loss) for stability | Zeng et al. (2022), LIT-054 — ARXIV-2210.02414. | Active |
| SOTA-070 | Track gradient norm ratios between layers | Zeng et al. (2022), LIT-054 — ARXIV-2210.02414. | Active |
| SOTA-071 | Use gradient clipping with dynamic threshold | Zeng et al. (2022), LIT-054 — ARXIV-2210.02414. | Active |
| SOTA-072 | Implement early warning system for NaNs | Zeng et al. (2022), LIT-054 — ARXIV-2210.02414. | Active |
| SOTA-075 | Use gradient compression for slow networks | Lin et al. (2017), LIT-056 — ARXIV-1712.01887. | Active |
| SOTA-077 | Use tar archives for dataset storage | Aizman et al. (2020), LIT-053 — ARXIV-2001.01858. | Active |
| SOTA-078 | Buffer size should be 2-3x batch size | Aizman et al. (2020), LIT-053 — ARXIV-2001.01858. | Active |
| SOTA-079 | Pre-fetch next batch during compute | Aizman et al. (2020), LIT-053 — ARXIV-2001.01858. | Active |
| SOTA-080 | Use multiple worker processes (num_workers = 4 * num_gpus) | Aizman et al. (2020), LIT-053 — ARXIV-2001.01858. | Active |
| SOTA-081 | Use operator fusion for small operations | Chen et al. (2018), LIT-063 — ARXIV-1802.04799. | Active |
| SOTA-082 | Optimize memory layout for hardware | Chen et al. (2018), LIT-063 — ARXIV-1802.04799. | Active |
| SOTA-083 v2 | Implement custom kernels for critical ops | Chen et al. (2018), LIT-063 — ARXIV-1802.04799. | Active |
| SOTA-084 | Profile-guided optimization for hot paths | Chen et al. (2018), LIT-063 — ARXIV-1802.04799. | Active |
| SOTA-085 v4 | Use flash attention for all attention computations when hardware supports it | Dao et al. (2022), LIT-074 — ARXIV-2205.14135. | Active |
| SOTA-086 v4 | Tiling size should match hardware SRAM size | Dao et al. (2022), LIT-074 — ARXIV-2205.14135. | Active |
| SOTA-087 v4 | Recompute attention during backward pass instead of storing it | Dao et al. (2022), LIT-074 — ARXIV-2205.14135. | Active |
| SOTA-088 | Fuse small operations into larger kernels | Ivanov et al. (2020), LIT-066 — ARXIV-2007.00072. | Active |
| SOTA-089 | Align tensor dimensions to hardware boundaries | Ivanov et al. (2020), LIT-066 — ARXIV-2007.00072. | Active |
| SOTA-090 | Use hardware-specific memory layouts | Ivanov et al. (2020), LIT-066 — ARXIV-2007.00072. | Active |
| SOTA-091 | Profile and optimize memory access patterns | Ivanov et al. (2020), LIT-066 — ARXIV-2007.00072. | Active |
| SOTA-092 | smaller batch sizes are more sample efficient (i.e., better loss as a function of tokens seen) earlier in training | Chowdhery et al. (2022), LIT-069 — ARXIV-2204.02311. | Active |
| SOTA-093 | larger batch sizes are beneficial later in training due to better gradient estimates | Chowdhery et al. (2022), LIT-069 — ARXIV-2204.02311. | Active |
| SOTA-094 | count accelerator efficiency alongside sample efficiency when choosing a batch size: larger batches mean larger matrix multiplications | Chowdhery et al. (2022), LIT-069 — ARXIV-2204.02311. | Active |
| SOTA-095 | consider rewinding to earlier checkpoint and skipping a few batches to mitigate unusual loss spikes | Chowdhery et al. (2022), LIT-069 — ARXIV-2204.02311. | Active |
| SOTA-096 v2 | num_tokens ~ 20 * num_params | Hoffmann et al. (2022), LIT-068 — ARXIV-2203.15556. | Active |
| SOTA-098 v2 | Monitor the training loss for unexpected spikes, for the whole run | Chowdhery et al. (2022), LIT-069 — ARXIV-2204.02311. The 540B model’s training loss spiked about 20 times, at irregular intervals, sometimes late, despite gradient clipping, and never in the smaller models. | Active |
| SOTA-099 | Track gradient norm statistics to detect training instabilities | Zeng et al. (2022), LIT-054 — ARXIV-2210.02414. | Active |
| SOTA-100 v2 | Use learning rate warmup proportional to model size | Xiong et al. (2020), LIT-114 — ARXIV-2002.04745. | Active |
| SOTA-103 v2 | Adjust data mixing proportions online from per-domain training loss | Albalak et al. (2023), LIT-117 — ARXIV-2312.02406. | Active |
| SOTA-105 | PagedAttention to accelerate batch inference for LLM sampling | Kwon et al. (2023), LIT-112 — ARXIV-2309.06180. | Active |
| SOTA-106 v3 | Use flash-attention-2 over original flash-attention when available | Dao et al. (2023), LIT-106 — ARXIV-2307.08691. | Active |
| SOTA-109 v3 | Prefer GQA to MQA or MHA | Ainslie et al. (2023), LIT-100 — group the query heads and give each group one key/value head: multi-query’s cache saving without multi-query’s quality loss, and uptrainable from an existing multi-head checkpoint. Still the default for a model not paying MLA’s implementation cost. | Active |
| SOTA-110 v3 | Consider Monarch Mixer in place of attention for very long sequence tasks | Fu et al. (2023), LIT-115 — ARXIV-2310.12109. | Active |
| SOTA-111 v4 | Use Monarch Mixer where attention’s quadratic cost is what bottlenecks training | Fu et al. (2023), LIT-115 — ARXIV-2310.12109. | Active |
| SOTA-113 v2 | Use continuous batching for inference | Kwon et al. (2023), LIT-112 — ARXIV-2309.06180. | Active |
| SOTA-114 | Fuse attention operations where possible | Dao et al. (2022), LIT-074 — ARXIV-2205.14135. | Active |
| SOTA-115 | Overlap prefill and decode compute | Agrawal et al. (2023), LIT-103 — ARXIV-2308.16369. | Active |
| SOTA-116 | Use FSDP over DDP when model size exceeds single GPU memory | Zhao et al. (2022), LIT-083 — ARXIV-2304.11277. | Active |
| SOTA-117 | Overlap communication with computation using backward prefetch | Zhao et al. (2022), LIT-083 — ARXIV-2304.11277. | Active |
| SOTA-118 | Employ mixed precision to reduce memory usage | Zhao et al. (2022), LIT-083 — ARXIV-2304.11277. | Active |
| SOTA-119 | Choose sharding factor based on model and GPU memory size | Zhao et al. (2022), LIT-083 — ARXIV-2304.11277. | Active |
| SOTA-120 v2 | Prefer AdamW’s decoupled weight decay to L2 regularization added to the loss | Loshchilov et al. (2017), LIT-012 — ARXIV-1711.05101. | Active |
| SOTA-121 v3 | Use Muon with decoupled weight decay and AdamW-matched update RMS in place of AdamW | Falcon-LLM Team (2026), LIT-119 — the Falcon-H1-Tiny technical blogpost. Stable at nearly the same optimal LR as AdamW, better evaluations; used for every Falcon-H1-Tiny model. | Active |
| SOTA-123 | For tiny specialized models, pretrain from scratch on the target SFT or reasoning data instead of pretrain-then-finetune | Falcon-LLM Team (2026), LIT-119 — the Falcon-H1-Tiny technical blogpost. At 90M, 25% SFT data in the pretraining mix beat a separate SFT stage by 10 IFEval points and yields one checkpoint that is both base and instruct. | Active |
| SOTA-126 | Run DPO on tiny models for one epoch only | Falcon-LLM Team (2026), LIT-119 — the Falcon-H1-Tiny technical blogpost. One epoch at LR 1e-6 to 3e-6 took a 90M model’s IFEval from about 50 to over 65; a second epoch degraded it while the reward kept rising. | Active |
| SOTA-127 v2 | Filter chain-of-thought traces out of the training data of tiny specialized models | Falcon-LLM Team (2026), LIT-119 — the Falcon-H1-Tiny technical blogpost. Reasoning traces interleaved with tool-calling data sent a 90M model into repetition loops; removing them fixed it at once. | Active |
| SOTA-128 | Train fill-in-the-middle data without masking the loss on non-FIM tokens | Falcon-LLM Team (2026), LIT-119 — the Falcon-H1-Tiny technical blogpost. At 80 GT and 90M, computing the loss on every token beat masking the prefix and suffix; matches what the reference FIM recipes appear to do. | Active |
| SOTA-129 v2 | Build a reasoning model in three stages: pretrain on general data, SFT on reasoning traces, then RL with verifiable rewards | Olmo Team (2025), LIT-130 — the recipe behind every open reasoning model at 7B and above, released in Olmo 3 with every stage and checkpoint in the open. Filed so its variations (SOTA-130) and its tiny-scale successor (SOTA-123) have something to be traced back to. | Active |
| SOTA-131 v4 | When training with Muon at scale, rescale query and key weights whenever attention logits exceed a threshold (QK-Clip) | Kimi Team (2025), LIT-132 — MuonClip carried a 1T/32B MoE through 15.5T tokens with zero loss spikes where plain Muon let attention logits pass 1000; confirmed at 2.8T in LIT-131. | Active |
| SOTA-132 v3 | Interleave linear-attention layers with global attention at about 3:1 instead of using full attention throughout | Kimi Team (2025), LIT-133 — three Kimi Delta Attention layers per gated-MLA layer beat full MLA at 48B/1.4T while cutting KV cache 75%; the layout LIT-131 ships at 2.8T with 69 KDA and 24 MLA layers. | Active |
| SOTA-133 | Replace fixed residual accumulation with learned attention over preceding layers | Kimi Team (2026), LIT-134 — Attention Residuals: a per-layer pseudo-query chooses which earlier layers to read, at O(d) parameters per layer; 1.25× compute advantage on scaling laws at 48B/1.4T, adopted in Kimi K3. | Active |
| SOTA-134 v2 | Gate each attention head’s output with a sigmoid after the scaled dot-product | Qiu et al. (2025), LIT-138 — the best of 30 gating variants at 15B MoE and 1.7B dense over 3.5T tokens: better quality, more stable training, larger tolerable learning rates, no attention sinks; shipped in every Qwen full-attention layer since Qwen3-Next. | Active |
| SOTA-135 v2 | Give linear-attention layers a gated delta rule: a decay gate for erasure plus a delta update for targeted writes | Yang et al. (2024), LIT-137 — beats Mamba2 and DeltaNet across language modelling, retrieval and long context at 1.3B/100B; the recurrence the Qwen hybrids use for three layers in four, and the one Kimi Delta Attention extends. | Active |
| SOTA-138 v2 | Train sparse attention natively with a learned top-k indexer, warmed up under dense attention | DeepSeek-AI (2025), LIT-142 — a small FP8 indexer scores past tokens and only the top-k enter attention; initialised for 2.1B tokens with dense attention on and the rest frozen; V4 repeats the pattern with a 1T-token dense warm-up. | Active |
| SOTA-139 | Extend the context length in stages during pretraining rather than training at the target length from the start | DeepSeek-AI (2026), LIT-139 — 4K, then 16K, 64K and 1M over 32–33T tokens, with sparse attention introduced at the 64K stage; Kimi K3 (LIT-131) reaches 1M the same way. | Active |
| SOTA-140 v5 | Use a warmup-stable-decay schedule: hold the learning rate, then decay it sharply over the final 10–20% of tokens | Hu et al. (2024), LIT-144, with the controlled comparison in LIT-145 — matches cosine at equal tuning, leaves the token budget open, makes every stable-stage checkpoint a usable branch point; contested at frontier scale by LIT-131, which retuned per schedule and chose cosine. | Active |
| SOTA-142 v3 | Set the peak learning rate by a power law in tokens so it transfers across batch size and training length | Shen et al. (2024), LIT-146 — under WSD the optimal learning rate is a power law in tokens and batch size; the Power scheduler applies it, and with µP the same setting carries across model sizes; adopted independently by Falcon-H1-Tiny. | Active |
| SOTA-143 v2 | Parameterize the model with µP and tune hyperparameters on a narrow proxy, transferring them across width | Yang and Hu (2022), LIT-148 — under the Maximal Update Parametrization the optimal learning rate and friends are stable across width, so tune small and transfer zero-shot; GPT-3 6.7B beaten at 7% of its pretraining cost in tuning. Used by Falcon-H1, MiniCPM and the Power scheduler. | Active |
| SOTA-145 v4 | Estimate the RL baseline from a group of samples for the same prompt instead of training a critic | Shao et al. (2024), LIT-127 — Group Relative Policy Optimization: PPO with the value model dropped and the baseline taken from the scores of several outputs sampled for the same prompt, which removes a model-sized chunk of the RL memory footprint and is what every later reasoning recipe in this record actually runs. | Active |
| SOTA-147 v2 | Compress the KV cache into one shared latent vector instead of sharing key and value heads | DeepSeek-AI (2024), LIT-174 — Multi-head Latent Attention projects keys and values into a single low-rank latent and caches that, cutting the KV cache 93.3% and raising maximum generation throughput 5.76× against the same team’s dense 67B. Every DeepSeek model since is built on it. | Active |
| SOTA-149 | Build the sparse layers from many small experts plus an always-on shared one, not a few large ones | Dai et al. (2024), LIT-170 — split into mN smaller experts and activate mK of them so the router chooses from a far larger combination space at the same compute, and isolate a few always-on shared experts to hold the common knowledge every routed expert would otherwise learn separately. | Active |
| SOTA-150 v3 | Make the feed-forward layers a sparse mixture of experts once the model is large enough to be compute-bound | Shazeer et al. (2017), LIT-188 — route each token to a few of many feed-forward experts instead of running one dense feed-forward for every token, so total parameters and per-token compute stop being the same number. Measured against dense by LIT-170 seven years later: DeepSeekMoE 16B matches LLaMA2 7B at roughly 40% of the compute, 145B approaches the same team’s dense 67B at 28.5%. | Active |
| SOTA-151 v4 | Extend a trained model’s context by rescaling RoPE, not by fine-tuning at the longer length | Chen et al. (2023) and Peng et al. (2023) — RoPE does not extrapolate, and fine-tuning at the longer length barely helps: more than 10000 batches moved LLaMA’s effective window from 2048 to 2560. Rescaling the position indices so they land back in the trained range reaches 32× that in under 1000 steps. YaRN rescales per wavelength rather than uniformly and gets to 128k with 10× fewer tokens. | Active |
| SOTA-153 v4 | Drop positional encoding from the global-attention layers of a hybrid and let the cheap local layers carry position | Yang et al. (2025), Puvvada et al. (2025) and the Kimi Team (2025) — in a model that already interleaves full attention with a cheap local mixer, the full-attention layers do not need a positional encoding: the local layers carry position and recency, and the global layers do retrieval better without one. Three groups, three different local mechanisms, the same one-global-per-three-local layout. The payoff is that extending the context needs no RoPE rescaling, because there is no positional parameter left to rescale. | Active |
| SOTA-154 v4 | Fine-tune with evolution strategies instead of policy-gradient reinforcement learning | Qiu et al. (2025), LIT-211 — evolution strategies over the full parameter space of a billion-scale LLM, which the field had assumed impossible. +36.4% over base on average against GRPO’s +21.3% and PPO’s +17.9%, with ES on one fixed hyperparameter set while RL got a sweep per experiment. Active since Gan and Isola replicated the comparison on six further tasks with the RL arm tuned harder; contested, because Gu et al. argue direct ES at LLM scale does not work — on models at 1.5B and below. | Active |
| SOTA-161 v4 | Keep the attention output in FP32 during training, because flash attention’s rounding bias compounds instead of cancelling | Qiu et al. (2025), LIT-198 — putting flash attention in BF16 while the FFN goes to FP8 sometimes explodes, and the cause is two things at once: attention produces low-rank updates repeated across steps and tokens, and low-precision addition rounds with a bias. The biased error rides the repeated update and compounds into a systematic gradient bias instead of averaging out. | Active |
| SOTA-162 v2 | Train with auxiliary multi-token-prediction heads alongside next-token prediction | Gloeckle et al. (2024), LIT-163 — predict the next n tokens through n independent heads on a shared trunk, as an auxiliary task rather than a replacement. No training-time overhead, the benefit grows with model size and survives multi-epoch training, and the extra heads are a draft model you already trained. | Active |
| SOTA-163 v2 | Quantize with block-scaled microscaling formats rather than one scale per tensor | Rouhani et al. (2023), LIT-197 — an MX block is 32 elements sharing one E8M0 scale, so applying the scale is an exponent adjustment rather than a multiply. Shrinking the scope of the scale contains outliers locally instead of letting a few large values cost every small one its precision. 6-bit MX trains large transformers to FP32 accuracy with no recipe change. | Active |
| SOTA-164 v2 | Deduplicate the pretraining corpus at both substring and document granularity before training on it | Lee et al. (2021), LIT-202 — a single 61-word sentence appears thousands of times in C4. Deduplicating cuts verbatim emission about tenfold, reaches equal or better accuracy in fewer steps, and removes train-test contamination from standard validation sets. Two granularities, because a repeated boilerplate paragraph is not a duplicated document. | Active |
| SOTA-165 | Precondition the gradient with matrices rather than entrywise scaling | Wen et al. (2025), LIT-156 — under per-optimizer tuning across ten optimizers and four scales, every fastest one multiplies gradients by matrices rather than scaling entrywise. A structural finding that survives a fair comparison, and the class the record’s Muon practice is one member of. The advantage is 1.4× at 0.1B and 1.1× at 1.2B. | Active |
| SOTA-170 v2 | Choose filtering aggressiveness by the token horizon: at long horizons rephrase what a filter would discard | Su et al. (2024), LIT-178 — aggressive model-based filtering wins at 1T tokens and loses at 15T, because removing 90% of the data optimises mean quality and mean quality is not what binds when the corpus runs out. Ensemble the classifiers, rephrase text synthetically instead of discarding it, and lean less on heuristics. | Active |
| SOTA-171 v2 | Repeat a data-constrained corpus for up to about four epochs; past that, added compute stops paying | Muennighoff et al. (2023), LIT-166 — with constrained data at fixed compute, up to four epochs of repeated data changes the loss negligibly against having that much unique data. Past that point the value of adding compute decays toward zero. 400 training runs, up to 900B tokens and 9B parameters, with a scaling law that prices repeated tokens and excess parameters. | Active |
| SOTA-174 | Train autoregressive models with fill-in-the-middle by default: it is a data transformation, and it is free | Bavarian et al. (2022), LIT-124 — cut a document into prefix, middle and suffix, move the middle to the end with sentinel tokens, and the model learns to infill. Transforming a large fraction of the training data does not harm left-to-right perplexity or sampling quality across a wide range of scales, so infilling is an added capability rather than a trade. | Active |
| SOTA-180 | Keep the routed experts’ weights in a down-projected latent space and spend the saved bandwidth on more of them | Elango et al. (2026), LIT-196 — down-project the token before the routed experts and keep their weights in that latent space, dispatching and aggregating there too, while routing and the shared experts stay at full width. Communication volume and weight-loading bandwidth both fall by d/d_l, and the saved budget buys more experts and more active per token. | Active |
| SOTA-181 | Shard the sequence across devices in a ring and overlap the key-value exchange with the attention it feeds | Liu et al. (2023), LIT-206 — compute attention and the feedforward blockwise, distribute the blocks across devices in a ring, and pass each key/value block to the neighbour while computing on the one you hold. The communication is fully overlapped with the computation, so sequences scale with device count — exactly, with no approximation and no added overhead. | Active |
| SOTA-182 v2 | Compute the normalization statistic without centering (RMSNorm) | Zhang and Sennrich (2019), LIT-023 — ARXIV-1910.07467. Drop the mean subtraction from layer normalization and rescale by the root mean square alone. | Active |
| SOTA-183 | Generate the harmlessness preference labels with the model itself, against a written set of principles | Bai et al. (2022), LIT-082 — ARXIV-2212.08073. Replace the human harmlessness comparisons with model-generated ones: the model critiques and revises its own responses against an explicit list of principles, and the preference model trains on that. | Active |
| SOTA-184 v2 | Adapt a pretrained model by training a low-rank update to each weight matrix, not the matrix itself | Hu et al. (2021), LIT-046 — ARXIV-2106.09685. Freeze the pretrained weights and learn a rank-r product BA beside each one, so the trainable parameter count and the optimizer state fall by orders of magnitude and the adapter folds back into the weight at inference. | Active |
| SOTA-185 | Quantize weights after training by compensating each rounding error into the columns not yet quantized | Frantar et al. (2022), LIT-081 — ARXIV-2210.17323. Round one column at a time and push the resulting error into the remaining columns using approximate second-order information, instead of rounding every weight independently to the nearest level. | Active |
| SOTA-186 | Bootstrap a large annotation set with the model you are training, staging the automation as it improves | Active | |
| SOTA-187 v5 | Train the generative model in a learned compressed latent, not at full resolution | Active | |
| SOTA-188 v5 | Parametrize the network so its prediction target has unit variance at every noise level, and sample training noise from a log-normal | Active | |
| SOTA-192 v9 | Normalize the queries and keys before the attention dot product | Active | |
| SOTA-194 | When an objective arrives with its own dataset, state the composition before crediting the objective | Active | |
| SOTA-195 v2 | Predict v rather than the noise when the model will be evaluated at low signal-to-noise | Active | |
| SOTA-196 v2 | Report zero-shot and in-distribution performance separately; one hyperparameter can move them in opposite directions | Active | |
| SOTA-197 v2 | Account for test-set proximity to the training data when evaluating, and state your contamination exposure | Active | |
| SOTA-198 v2 | Measure the gradient noise scale instead of sweeping batch size, and expect it to grow during the run | Active | |
| SOTA-199 | Regularize a narrow fine-tune against the pre-fine-tuning model’s own samples | Active | |
| SOTA-200 v3 | Check whether an emergent capability is a metric artefact before believing it | Active | |
| SOTA-201 | Give each pretrained component its own learning rate when fine-tuning a composite model | Active | |
| SOTA-202 v2 | Clamp the prediction to the training range at every step when sampling from a model’s own output | Active | |
| SOTA-203 v3 | Sample a diffusion model with a higher-order ODE solver on the weights you already trained | Lu et al. (2022), LIT-076 — the diffusion ODE is semi-linear, so solve the linear part exactly and approximate only the neural integral. 10-20 function evaluations, no retraining. Rests on LIT-038, which established that the sampler is not fixed by the training objective. | Active |
| SOTA-205 v2 | Replace a large coordinate network with a compact explicit structure and a small decoder | Four independent groups, three structures. Where a field is queried pointwise and the signal is spatially sparse, the capacity belongs in an addressable structure that training optimises directly, not in a network evaluated per point. | Active |
| SOTA-206 v3 | Keep a multi-step sampling option in a few-step generative model | Song et al. (2023), LIT-093. A one-step model that cannot spend more compute has no quality dial: whatever it produces is what you get. Consistency models trade compute for quality at inference without retraining, and that property is worth preserving by design. | Active |
| SOTA-207 | Use the deterministic sampler when the noise input has to mean something | Song et al. (2020), LIT-038. A stochastic sampler injects fresh noise at every step, so nothing about the starting point survives to the output. Setting the family’s stochasticity to zero makes the initial noise a latent code you can interpolate in and invert to. | Active |
| SOTA-210 v3 | Report pass@k as well as pass@1 after post-training: reinforcement learning raises one and lowers the other | Ba et al. and Hayes et al. (2026), LIT-230 and LIT-234. GRPO finishes below its own base model on pass@16 and pass@32 in 15 of 18 comparisons while improving pass@1; across Qwen2.5, Qwen3 and published RL checkpoints up to 32B the base model overtakes the RL checkpoint at large k. A post-training result reported at pass@1 alone cannot distinguish a model that learned something from one that stopped trying anything else. | Active |
| SOTA-214 | Pair any gradient compressor with error feedback, and correct the momentum it is applied under | Compression is lossy and the loss accumulates. Every scheme in this record that survives high compression ratios keeps the residual locally and adds it back to the next step, and every report of a compressor “not converging” that the record holds is a report of that step being skipped: rank-4 PowerSGD fails on Cifar10 without it, and gradient sparsification at 99.9% degrades badly without the momentum-buffer version of it. | Active |
| SOTA-218 v2 | Retune the learning rate, momentum and schedule at every batch size you compare, never by a scaling heuristic | Linear scaling and square-root scaling are not batch-size rules; they are guesses about where the optimum moved, and they hold over a narrower range than the comparisons people use them for. A curve of steps-to-target against batch size drawn with transferred metaparameters is not a measurement of batch size at all — it is a comparison between one tuned configuration and several untuned ones, and the point where it bends is the point where the heuristic failed. | Active |
| SOTA-219 | Leave the inner optimizer state unsynchronised in local-update training | When workers train independently for hundreds of steps and synchronise by exchanging deltas, the Adam moments each worker accumulated are local state and should stay local. Synchronising them triples the communication for negligible quality gain — which means most of what the scheme saves is not saved by communicating rarely, but by communicating less each time. | Active |
| SOTA-221 v2 | To train past the batch size where your optimizer stalls, change the optimizer’s conditioning rather than the scaling rule | You et al. (2019), LIT-265. When a batch size stops reaching your accuracy target, the reflex is to look for a better learning-rate scaling rule. LIT-058 is 168,160 training runs of evidence that there is no such rule to find. What moved the wall instead was the optimizer: normalizing each layer’s update by the ratio of its parameter norm to its update norm, on top of per-coordinate adaptivity, carried BERT from 16K to 32K where a grid-searched AdamW could not follow. This does not license skipping the retuning in SOTA-218 — the two say different things about different activities, and THEORY-012 is why. | Active |
| SOTA-224 | Ramp the sparsity ratio up over the first few epochs rather than starting at the target | Lin et al. (2017), LIT-056. Gradient sparsification delays the coordinates it drops; error feedback returns them, but it returns them late. Early in training the gradients are large and the trajectory is still choosing where to go, so a delay there costs more than the same delay later. Start at a low sparsity — the paper ramps 75%, 93.75%, 98.4375%, 99.6%, 99.9% over the first few epochs — and reach the target once the run has settled. | Active |
| SOTA-225 | Drop the parameter server for gossip once the network is the bottleneck, and measure where that is rather than inheriting a threshold | Lian et al. (2017), LIT-302. The question is whether a decentralized algorithm can ever beat a centralized one, and the answer is that it can — not by converging better, since the rates match, but by having no node that every other node talks to. The advantage appears where the central node’s link is the constraint, and the paper establishes that by sweeping bandwidth and latency rather than by naming a threshold. | Active |
| SOTA-226 | If the gossip topology is static, make it a directed exponential graph, not a ring | Assran et al. (2018), LIT-254. Having chosen gossip over a parameter server, the next choice is the graph, and it is not a detail: a ring’s consensus rate degrades with the node count badly enough that the speedup regime recedes out of reach. A directed exponential graph — each node sends to peers at distance 1, 2, 4, 8, … in rotation — reaches exact consensus in log2(n) rounds with constant out-degree per round, and spreads the load evenly. | Active |
| SOTA-227 v2 | Decode with a draft model and an accept-reject rule, which is exactly lossless | Autoregressive decoding runs one serial model pass per token and each pass is bounded by streaming the weights, so the arithmetic units idle. Have a cheap draft model guess the next gamma tokens, score all gamma + 1 positions in one target pass, and keep a prefix under an accept-reject rule built so the output distribution is the target’s exactly. 2x-3x at 11B and 2-2.5x at 70B, with no retraining, no architecture change and nothing traded away. | Active |
| SOTA-228 | Pick the inference partitioning from where the bottleneck is, and expect it to move between prefill and decode | Pope et al. (2022), LIT-110. Serving is not one workload: prefill parallelises over the prompt, decode is serial over the output, and the two are bounded by different things. Worse, the binding constraint moves — at small batch and short context it is loading the weights, and at large batch and long context it is loading the KV cache, which on a 500B model at batch 512 and 2048 tokens reaches 3 TB, three times the parameters. Choose the layout from a cost model of the regime you are in, and re-choose it when the regime changes. | Active |
| SOTA-230 | Quantize the frozen base to 4-bit and keep the adapters in 16-bit | Dettmers et al. (2023), LIT-378 — ARXIV-2305.14314. Store the frozen base weights in 4-bit NormalFloat and dequantize to BFloat16 for every matrix multiply, training only 16-bit LoRA adapters. Fine-tuning a 65B model falls from >780GB to <48GB with no measured loss against a 16-bit fully fine-tuned baseline, because the arithmetic never happens in 4 bits. | Active |
| SOTA-231 | Put an adapter on every linear layer, not just the attention query and value projections | Dettmers et al. (2023), LIT-378 — ARXIV-2305.14314. Adapters on the query and value projections alone — the default inherited from the LoRA paper — do not reach full fine-tuning on large base models. The number of adapted matrices is the hyperparameter that decides it; the projection rank r, which is the one people tune, does not affect performance across the paper’s sweep. | Active |
| SOTA-233 | Train self-correction with multi-turn online RL on the model’s own traces, rather than prompting for it | Kumar et al. (2024), LIT-383 — ARXIV-2409.12917. Self-correction is a capability to be trained, not a behaviour to be requested. Prompting a model to revise its own reasoning does not reliably help and can hurt; supervised fine-tuning on correction traces fails by distribution mismatch or behaviour collapse; multi-turn online RL on self-generated traces reaches +15.6% on MATH and +9.1% on HumanEval. | Active |
| SOTA-235 v3 | Update the weights at inference on the test instance when the task is structurally novel | Akyürek et al. (2024), LIT-379 — ARXIV-2411.07279. Build a loss from the test instance’s own in-context examples, take gradient steps at inference, then discard the update. Up to 6x a fine-tuned baseline on ARC-AGI (53.0% at 8B) and +7.3 points on BIG-Bench Hard at 10-shot. The examples are worth several times more as gradient than as context. | Active |
| SOTA-236 | Predict scene geometry directly instead of solving for cameras first and triangulating | Wang et al. (2023), LIT-385 — ARXIV-2312.14132; Wang et al. (2025), LIT-384 — ARXIV-2503.11651. Given uncalibrated, unposed images, regress the 3D structure and let camera parameters and pixel matches fall out of it, rather than estimating calibration and pose first so that triangulation becomes possible. The quantities the classical pipeline needs as inputs are by-products of this one. | Active |
| SOTA-237 | Match densely without a keypoint detector, because the detector fails where matching is hardest | Sun et al. (2021), LIT-387 — ARXIV-2104.00680. Establish dense matches coarse-to-fine instead of detecting keypoints, describing them and matching the descriptors. The detector sets the pipeline’s ceiling, and it fails in low-texture regions — walls, floors, road — where it cannot emit repeatable points, so no downstream matcher can recover a correspondence that was never proposed. | Active |
| SOTA-238 v4 | Set domain weights with a small proxy model under group DRO on excess loss, then transfer them | Xie et al. (2023), LIT-391 — ARXIV-2305.10429. Train a small proxy under group DRO to produce domain weights, then resample and train the real model with them. Optimise worst-case EXCESS loss against a reference model, not worst-case loss — the naive form upweights whichever domain is noisiest, because every domain has a different irreducible entropy. | Active |
| SOTA-240 v4 | Apply dropout where the model can memorize what it is shown, and not where it cannot | Srivastava et al. (2014), LIT-395 — dropout has a “sweet spot” in dataset size and reports both of its edges: no gain at all on data small enough to memorize through the noise, and a declining gain once the data is large enough that overfitting is not the problem. It costs 2-3x training time and it is not a drop-in — the paper’s own recipe pairs it with n/p units, 10-100x the learning rate, momentum 0.95-0.99 and a max-norm constraint. Introduced by Hinton et al. (2012), LIT-394. | Active |
| SOTA-249 | Recompute activations from a sqrt(n) subset of checkpoints when activation memory is the binding constraint | Chen et al. (2016), LIT-004 — ARXIV-1604.06174. Store activations at O(sqrt(n)) checkpoints and recompute the rest during the backward pass. The price is one extra forward pass per minibatch; the measured case is a 1000-layer ResNet at 48G to 7G for 30% more wall clock. | Active |
| SOTA-250 | Pretrain by predicting representations of masked regions, not their pixels | Assran et al. (2023), LIT-216 — ARXIV-2301.08243. Predict the representations of masked target blocks from a context block, with a learned target-encoder, instead of reconstructing pixels. No hand-crafted augmentations, and a ViT-H/14 reaches strong downstream performance in under 1200 GPU-hours — faster than a ViT-S/16 trained with iBOT. | Active |
| SOTA-252 | Make the full-resolution path affordable instead of upsampling, when the shortcut is what breaks correctness | Abdal et al. (2023), LIT-113 — ARXIV-2311.17857. Prior 3D GANs render small and upsample in 2D because volume rendering is too slow at training resolution — but a 2D upsampler is multi-view inconsistent by construction. Efficient Gaussian rendering makes native 512x512 affordable, and the inconsistency leaves with the upsampler rather than being mitigated. | Active |
| SOTA-253 | Add a new instance by reconstructing it, not by training on it | Sun et al. (2022), LIT-111 — ARXIV-2205.12257. Split what the system knows about a specific object from what it has learned in general. Build the object’s structure once by reconstruction; match against it with a network that never saw the object or its category. Adding an object is then a scan, not a training run. | Active |
| SOTA-258 | Scale batch size with the token budget, not with compute or model size | Zhang et al. (2024), LIT-445, and Bergsma et al. (2025), LIT-443 — critical batch size scales as D^0.46 and D^0.47 respectively, and is nearly flat in model size once the token budget is held fixed. Optimal batch size likewise goes as D^0.38. The compute exponent the field has used since Kaplan fits a projection: along the Chinchilla line C and D move together and only D is doing the work. | Active |
| SOTA-262 v2 | Give each modality its own weights and let the streams attend jointly | Esser et al. (2024), LIT-449 — MMDiT gives the text and image streams separate projections and MLPs, then runs attention over the concatenated sequence so information flows both ways. It beats UViT and DiT at matched budget. Liang et al. (2024), LIT-483 — the same shape reached independently for autoregressive multi-modal LLMs, matching a dense baseline at 55.8% of the FLOPs, and ablated component by component: the feed-forward carries most of the benefit, the attention projections less, the layer norms none. | Active |
| SOTA-263 v4 | Shift the timestep schedule when the resolution changes, because more pixels need more noise | Hoogeboom et al. (2023), LIT-660, and Chen (2023), LIT-659. A timestep is not a fixed amount of corruption. Destroying the signal in an image with more pixels takes more noise, so a schedule set at one resolution under-corrupts at a higher one. Shifting log-SNR down with resolution takes ImageNet 256 FID from 7.65 to 3.76 in pixel space. The direction is measured. The magnitude is tuned in every source. | Active |
| SOTA-265 v3 | Tune the stochastic sampler’s diffusion coefficient after training; it is not fixed by the forward process | Albergo, Boffi and Vanden-Eijnden (2023), LIT-645, state it and Ma et al. (2024), LIT-447, measure it — score-based diffusion conventionally takes the reverse SDE’s diffusion coefficient from the forward process, and presents the two as intrinsically tied. They are not: the coefficient affects neither the velocity nor the score, only the integration. So it is a sampler hyperparameter, tunable on a frozen model, and tuning it tightens the KL divergence to the target. | Active |
| SOTA-266 v5 | Connect data and noise on a straight line, and sample the training timesteps from a logit-normal rather than uniformly | Esser et al. (2024), LIT-449, and Ma et al. (2024), LIT-447 — the straight-line path between data and noise beats the curved variance-preserving one at fixed architecture and compute, and the advantage is largest at few sampling steps. Movie Gen (LIT-626) confirms it for video at 5B. In SD3’s latent text-to-image sweep, the timestep distribution decides the ranking: uniform does not beat tuned ε-prediction and logit-normal does. | Active |
| SOTA-269 | Keep a knowledge base outside the weights; parameters are an expensive and lossy place to memorise facts | Lu et al. (2024), LIT-452 — fact capacity is linear in model size and saturates in epochs, and the extrapolation is damning: all of Wikidata would want about 1000B non-embedding parameters trained for 100 epochs. Derivable facts cost full price unless they happen to share direction and structure. Borgeaud et al., LIT-060, is the record’s evidence that the alternative works at scale. | Active |
| SOTA-270 | Do not read a smooth loss curve as evidence of smooth training; decompose it when the answer matters | Kangaslahti et al. (2025), LIT-455, with Cohen et al. LIT-453 and Kunin et al. LIT-454 — the curve time-averages oscillation, sums over differently-timed abrupt transitions, and reads flat while the weights still travel. Smoothness is what many breakthroughs look like added up, so it is not evidence about the training. | Active |
| SOTA-278 | Before reporting that a model cannot do something, rule out the evaluation | Hu and Frank (2024), LIT-465 — the same capacity asked two ways scores differently, and the gap widens as the model weakens. Lawsen (2025), LIT-463, is the worked example: an output cap, a scorer that cannot express refusal, and instances with no solution. | Active |
| SOTA-279 v2 | Put worked reasoning steps in the few-shot exemplars when the task needs more than one step, and only once the model is large enough | Wei et al. (2022), LIT-467 — eight exemplars carrying worked steps more than doubled PaLM 540B on GSM8K, beating a fine-tuned model with a verifier. Three ablations rule out the equation, the extra tokens and knowledge activation. Below ~100B it does nothing or hurts. | Active |
| SOTA-280 | Sample several reasoning paths and take the majority answer rather than decoding one greedily | Wang et al. (2022), LIT-468 — replace greedy decoding with sample-and-marginalize: GSM8K +17.9, SVAMP +11.0, AQuA +12.2. Five or ten paths recover most of it, which is the whole cost question. | Active |
| SOTA-281 | Try the single step-by-step instruction before writing exemplars; it recovers most of the gain for none of the labour | Kojima et al. (2022), LIT-469 — one fixed sentence with no exemplars takes MultiArith from 17.7% to 78.7% and GSM8K from 10.4% to 40.7%. It beats eight-shot standard prompting and loses to hand-written few-shot chains, which is the ordering that makes it a first move. | Active |
| SOTA-291 v2 | Evaluate a specific intervention on a specific affordance rather than the net effect of the system, and never read an individual-level null as the absence of a collective effect | Active | |
| SOTA-305 v2 | State the token count and input resolution beside any reconstruction FID, and compare tokenizers only at equal rate | Sun et al. (2024), LIT-497 — the same tokenizer at downsample 16 scores rFID 2.19, 0.94 and 0.70 on 256, 576 and 1024 tokens, all evaluated on 256×256 reconstructions. A three-fold range from a knob that is not the tokenizer. Two papers in this line then quote reconstruction numbers whose rate a reader cannot recover from the sentence they appear in. | Active |
| SOTA-306 v2 | Factorize the codebook: look up in a low-dimensional normalized space, embed in a high-dimensional one, and report utilization alongside reconstruction quality | Yu et al. (2021), LIT-500 — project the encoder output down to a low-dimensional lookup space, ℓ₂-normalize it and the codebook, match there, and project the winner back up to a wide embedding. Codebook usage goes 4% → 95% and FID 3.68 → 1.50 at unchanged throughput; removing the normalization alone drops usage to 2%. Confirmed independently by LIT-497 and LIT-494, and in all three the curve turns back up, so there is an optimum to find rather than a direction to follow. | Active |
| SOTA-307 v5 | Report generative FID as an error bar over several training seeds, and treat any gap below about 2% of the mean as inconclusive | Dufour, Efros and Pérez (2026), LIT-501 — on several hundred SiT networks, retraining the same recipe moves FID 3.2× more than resampling from a fixed model, and the coefficient of variation holds at 1–2% across four model sizes and every compute budget to 2M steps. Ten times the sampling budget shrinks the small term by √10 and leaves the large one untouched. A lucky training seed is worth up to 2× the compute. | Active |
| SOTA-308 | When a retrieval change raises recall, measure relevance and the end-task metric before calling it an improvement | Wu et al. (2026), LIT-502 — across three published GraphRAG pipelines on G-Medical, graph expansion raises retrieval recall (GFM-RAG 84.3% against vanilla RAG’s 71.8%) and drops context relevance (38.5% against 62.9%). The wider context is noisier, and generation accuracy falls with it — a system that retrieves more of the right thing and much more of the wrong thing answers worse. | Active |
| SOTA-310 | Treat a mechanistic explanation that passes circuit error or causal alignment as one of many, and report what you did to rule the others out | Méloux, Maniu, Portet and Peyrard (2025), LIT-504 — by exhaustive enumeration on small MLPs, no network had exactly one circuit interpretation and under 2% had exactly one valid minimal mapping. The median number of explanations satisfying the criteria rises from 38 to 910,000 as width goes 2 → 5, and both counts are lower bounds. Passing circuit error or IIA establishes that an explanation is admissible, not that it is the explanation. | Active |
| SOTA-311 | Evaluate a search on tasks held out of its own fitness function, and report the gap against the tasks it selected on | No source states this; it is filed on a priori grounds with one instance. Cheng, Clark and Richardson (2025), LIT-493, searched 1,062 pretrained designs against a fitness function of average downstream accuracy, then reported the top five on nine benchmarks drawn from the same pool and chosen — per its own Table 16 — for largest standard deviation across the search population. The best discovered design beats the best seed by 0.03 points. | Active |
| SOTA-312 v2 | State the noise or binning assumption behind any mutual information you report for a deterministic network, and show the conclusion survives changing it | Saxe et al. (2018), LIT-509 — in a deterministic network I(h;X) is infinite, so every finite number on an information plane is a property of a noise model the analyst imposed and the network never had. The same tanh run, binned evenly in net input instead of evenly in activity, loses its compression phase entirely; and LIT-507 makes compression appear in ReLU networks by changing the binning the other way. | Active |
| SOTA-313 | Treat the inference-time decision procedure as part of what you are comparing, and report the ordering under each one | Gemini Team (2023), LIT-510 — on MMLU, greedy sampling puts GPT-4 ahead by 0.2, plain chain-of-thought at 32 samples puts it ahead by 2.3, and uncertainty-routed CoT at 32 samples puts Gemini Ultra ahead by 2.7. Same models, same benchmark, same harness. The procedure is worth 6.0 points to one model and 3.1 to the other, so running the same procedure on both is not enough — the ordering is a property of the procedure you chose to report. | Active |
| SOTA-314 v2 | Absorb quantization outliers into a high-precision low-rank branch taken from the weights, and fuse its kernels into the low-bit ones | Li et al. (2024), LIT-512 — at 4 bits on both weights and activations, smoothing alone is not enough. Shift outliers from activations into weights, peel the dominant singular values into a 16-bit rank-32 branch, quantize the residual. Then fuse: run naïvely the branch costs 57% latency and cancels the win; fused it gives 3.0× over W4A16 and 3.5× memory on 12B FLUX.1. | Active |
| SOTA-315 v2 | Early-stop and tune on validation loss after temperature scaling, then calibrate post hoc, instead of stopping on raw validation loss | Berta et al. (2025), LIT-514 — a proper loss is calibration error plus refinement error, and the two bottom out at different epochs, so the validation-loss minimum is optimal for neither. Fit a temperature on the validation set each time you would have read the loss, read the loss after it, and stop there; then keep the temperature. Measured across 196 classification datasets and a vision benchmark with ten runs per dataset. | Active |
| SOTA-317 | Do not rank singular directions by magnitude when deciding what to discard; check the small end, and check it after fine-tuning | Staats, Thamm and Rosenow (2024), LIT-517 — in a transformer’s non-square matrices, the smallest singular directions carry data directions and their removal is catastrophic: Llama-3 8B on GSM8K falls from 43.2% to 2.0% when the smallest decile of the Down-Projection goes, against 40.0% for the same decile of the square Attention-Output. And the damage only appears if you prune after fine-tuning. | Active |
| SOTA-318 | Choose the rank per matrix when you compress a transformer, because low-rank structure varies by component and by depth | Jaiswal et al. (2024), LIT-516 — “transformer weights are low rank” is true of some matrices and false of others. Query, Key and MLP Gate converge to low rank; MLP Up, MLP Down and Value do not; middle blocks resist while the first and last few give way. Setting the rank per matrix instead of globally is ~6.4× better in perplexity than uniform reduction at 30% on LLaMA-2 7B and ~47× at 40% on 13B. | Active |
| SOTA-330 | Enforce output structure by masking logits against a grammar, with masks precomputed per automaton state and built on the CPU while the GPU runs the forward pass | Dong et al. (2024), LIT-552 — split the vocabulary by whether a token’s validity depends only on the grammar automaton’s current node (over 99% of tokens for JSON). Precompute those per node, check the rest against the full stack at runtime, and build each mask on the CPU during the GPU forward pass. Mask time is 0.018 ms per token after the ablated optimizations, and end-to-end overhead is 0.1–0.2 ms. It guarantees syntax. It does not measure whether answers stay correct. | Active |
| SOTA-331 | Pass low-dimensional coordinate inputs through sinusoids of sampled frequencies before an MLP, and tune the frequency scale rather than the distribution | Tancik et al. (2020), LIT-550, with NeRF (LIT-435) — an MLP on raw coordinates never fits high frequencies. Map the input to [cos 2πBv, sin 2πBv] with B ~ N(0, σ²) and tune σ on held-out data: too small blurs, too large aliases, and the distribution’s shape does not matter. Gaussian features beat no mapping and log-spaced positional encoding on all seven image, shape, CT, MRI and view-synthesis tasks tested. | Active |
| SOTA-337 v4 | When an ImageNet classifier is anywhere in the pipeline — training the generator or steering its sampling — confirm FID gains with a Fréchet distance in a non-ImageNet feature space | Kynkäänniemi et al. (2022), LIT-563 — FID’s features are nearly ImageNet class logits, so matching ImageNet-class statistics lowers it without improving images. Resampling a fixed StyleGAN2’s outputs cuts FFHQ FID 5.30 → 1.78, while CLIP-space FD moves 2.76 → 2.64. A model whose discriminator uses ImageNet features exploited this by accident, and the authors suspect data filters and samplers can too. Report a CLIP or SwAV Fréchet distance beside FID whenever that is the case. Classifier guidance (LIT-699) is the sampling-loop instance and the least checked: it maximises an ImageNet classifier’s confidence on every step, and its paper denies producing adversarial examples in one clause without running a test. | Active |
| SOTA-345 | Probe language-model activations with logistic regression on the raw activations, and credit sparse-autoencoder probes only against that baseline under validation-based selection | Kantamneni et al. (2025), LIT-568 — across 113 datasets, adding SAE-latent probes to a toolkit of raw-activation probes, choosing per task by validation AUC, changes test AUC by −0.003. It does not help under data scarcity, class imbalance, label noise or shift. Reported SAE wins came from baselines denied the same pooling or selection. Use logistic regression on activations, and hold any SAE method to it. | Active |
| SOTA-348 | Evaluate a knowledge edit on the facts it implies — logical consequences, multi-hop compositions, aliases and the subject’s other facts — not only on the edited prompt, its paraphrases and unrelated neighbours | Cohen et al. (2023), LIT-577 — the standard edit metrics (efficacy, paraphrase, neighborhood) cannot see whether an edit changed what the model knows or only what it says to one question. On six ripple criteria, ROME, MEMIT and MEND average 38–66, and logical generalization falls as low as 5.5. Prompting with the new fact scores 81–83 on LLaMA-7B. Report ripple metrics beside the standard ones, and compare against in-context editing. | Active |
| SOTA-349 | Before crediting a graph transformer with long-range gains, retune the message-passing baselines under the same budget, including an MLP prediction head, positional encodings and input feature normalization | Tönshoff et al. (2023), LIT-579 — the Long-Range Graph Benchmark’s case that graph transformers are needed rested partly on untuned message-passing baselines. With a basic sweep inside the same 500k-parameter budget, GCN beats GPS on both Peptides tasks. Most of the gain is from replacing a linear prediction head with a 2-layer MLP. Normalizing input features lifts every model on the superpixel tasks, where GPS still leads. Tune both sides before attributing a gap to attention. | Active |
| SOTA-350 | Compare GNN architectures over many random data splits and seeds, with one shared training and hyperparameter-selection procedure for every model, never on a single fixed split | Shchur et al. (2018), LIT-581 — on the Planetoid split GAT beats GCN. On another random split of the same size GCN wins, and averaged over 100 splits × 20 seeds with identical training and tuning, GCN ranks first of four GNNs. A single split selects the model that overfits it, and per-model recipes measure the recipe. Report mean and spread over many splits and seeds, with the same tuning budget for every model. | Active |
| SOTA-351 | In a message-passing GNN for graph-level tasks, aggregate neighbours by sum with an MLP update, not by mean or max, when graph structure rather than node features carries the signal | Xu et al. (2019), LIT-580 — mean and max aggregation cannot count neighbors, so graphs that differ only in counts look the same to them. On featureless Reddit graphs, mean-aggregation GNNs are at chance (50.0, 20.0) and sum–MLP scores 92.4 and 57.5. With informative node features the choice matters much less, and results are mostly within noise. Use h = MLP((1 + ε)h + Σ neighbors) with ε = 0 and a sum readout. | Active |
| SOTA-355 v2 | Isolate the emergent outlier feature dimensions into 16-bit and quantize the rest to int8 | Dettmers et al. (2022), ARXIV-2208.07339. Quantize the projections vector-wise to int8, but pull the handful of systematically large feature dimensions out into a separate 16-bit multiplication — 99.9% of values in 8-bit, no measured quality cost up to 175B. | Active |
| SOTA-356 | Protect the salient weight channels by scaling them before rounding, choosing them from the activations rather than the weights | Lin et al. (2023), ARXIV-2306.00978. Identify the ~1% of weight channels whose activations are largest, scale them up by an equivalent transformation before rounding, and quantize everything uniformly — rather than keeping the important channels in higher precision. | Active |
| SOTA-357 | Build a zero-shot classifier from prompt templates and ensemble them in embedding space | Active | |
| SOTA-358 v3 | Drop the domain inductive bias once pre-training data is large enough, and keep it when it is not | Active | |
| SOTA-359 v2 | Supervise vision from the caption, and match image to caption rather than predicting its words | Active | |
| SOTA-360 | Score a density ratio against sampled negatives instead of reconstructing the target | Active | |
| SOTA-361 | Compose augmentations so the shortcut dies, and tune them for the contrastive objective rather than the supervised one | Active | |
| SOTA-362 | Train through a projection head and then discard it, taking the layer before as the representation | Active | |
| SOTA-363 | Decouple the negative set from the batch with a queue, and keep the encoder that fills it slow | Active | |
| SOTA-364 | Break batch normalization’s cross-sample leak before it solves your contrastive task for you | Active | |
| SOTA-365 v2 | Learn without negatives by breaking the symmetry: a predictor on one branch and a stop-gradient on the other | Active | |
| SOTA-366 | Monitor the standard deviation of the l2-normalized embedding; healthy is one over root d and collapsed is zero | Active | |
| SOTA-367 | Or put the anti-collapse constraint in the loss, as a variance floor and a decorrelation term | Active | |
| SOTA-368 | When a model’s latent space is meant to be read by people, measure its interpretability with a human task such as word or topic intrusion, not by held-out likelihood | Chang et al. (2009), LIT-597 — topic models are used for their topics, but they are judged by held-out likelihood, which measures the probability of observations and ignores the representation. On NYT and Wikipedia, CTM had the best likelihood and the worst human scores. Test the latent space directly. Plant an improbable word among a topic’s top words, or an improbable topic among a document’s top topics, and measure how often people find it. | Active |
| SOTA-369 | Curate a pretraining set by retrieving neighbours of curated seeds, rather than scaling the uncurated pool | Active | |
| SOTA-370 | Add many low-resolution views alongside the two full ones, because view count is what helps and resolution is what costs | Active | |
| SOTA-372 v2 | Run the encoder on visible tokens only and push the mask tokens into a small decoder you discard | Active | |
| SOTA-373 v3 | Set the masking ratio by the signal’s redundancy, not by the ratio that worked on text | Active | |
| SOTA-374 v4 | When learning embeddings from co-occurrence, subsample very frequent tokens, discarding each occurrence with probability 1 − √(t/f) with t around 10⁻⁵ | Mikolov et al. (2013), LIT-603 — in a Zipfian corpus, the few most frequent words co-occur with everything and dominate the updates while teaching little. Discard each occurrence of word w with probability 1 − √(t/f(w)). This cut skip-gram training time by 2–3× and raised accuracy, only slightly for words (59 → 60% at NEG-5) but a lot for phrases (27 → 42% at NEG-15, 19 → 47% with hierarchical softmax). | Active |
| SOTA-376 | Score each image-text pair independently with a sigmoid, so the loss needs no global normalization | Active | |
| SOTA-377 | Stop scaling the contrastive batch past about 32k, because the benefit saturates there | Active | |
| SOTA-379 v2 | Before crediting a word-embedding method over count-based ones, give the baselines the same design choices and tune every method alike | Levy, Goldberg and Dagan (2015), LIT-607 — word2vec and GloVe ship with design choices (context smoothing, the negative-sample shift, dynamic windows, subsampling, adding context vectors) that count-based PPMI and SVD can use too. Port them and tune every method the same way before comparing. Done that way across 672 representations and 8 datasets, the prediction-over-count advantage disappears, SGNS beats GloVe on every task, and a single hyperparameter often matters more than the method. | Active |
| SOTA-381 v2 | Sample a handful of negatives instead of normalizing over the vocabulary, and let the count fall as the data grows | Active | |
| SOTA-383 | Fix the sample count before comparing FID values, and settle a close comparison with an unbiased estimator rather than a tighter error bar | Bińkowski et al. (2018), LIT-615 — the FID estimator is biased, the bias depends on the distribution being measured, and no unbiased estimator exists. So two FID numbers are comparable only at equal n, and a small standard deviation is not evidence of a converged estimate: at d=2048 and 50,000 samples their construction reversed a true ordering in 100 trials out of 100, with standard deviations of 0.2 and 0.5. | Active |
| SOTA-384 | Calibrate a trained classifier with a single temperature fitted on held-out data, not with a richer map | Guo et al. (2017), LIT-616 — rescale the logits by a single learned 1/T fitted for NLL on a held-out set. It cannot change the argmax, so accuracy is unchanged by construction, and it beats vector scaling, matrix scaling, histogram binning, isotonic regression and BBQ — including the two that strictly contain it. Typical uncalibrated ECE is 4–10%. | Active |
| SOTA-388 v2 | Train a plain ViT on ImageNet-1k with average pooling, fixed 2D sin-cos positions, a 1024 batch and light augmentation | Beyer et al. (2022), LIT-628 — five changes, none novel, take ViT-S/16 on ImageNet-1k from 66.8% to 76.5% at 90 epochs and 80.0% at 300. Global average pooling, fixed 2D sin-cos position embeddings, batch 1024, RandAugment at level 10 and Mixup at p = 0.2. 6h30 on a TPUv3-8 for the 90-epoch run. | Active |
| SOTA-400 | Append a few register tokens to a ViT so it does not commandeer background patches for scratch space | Active | |
| SOTA-407 v2 | Interpolate the zero-shot and fine-tuned weights at about half way rather than shipping the fine-tuned model | Wortsman et al. (2021), LIT-674. Fine-tuning a zero-shot model spends the robustness that made it worth starting from. Ship (1 − α)·θ_zero-shot + α·θ_fine-tuned at α = 0.5 instead: +3.5 to +23.2 pp under six distribution shifts against the fine-tuned model, with reference accuracy falling by at most 0.3 pp and usually rising, at no cost in training or inference. Requires that the fine-tuned weights were obtained from the zero-shot weights. | Active |
| SOTA-409 | Average the fine-tuned models from your hyperparameter sweep instead of keeping only the best, adding each one in validation order and only if it helps | Wortsman et al. (2022), LIT-675. A hyperparameter sweep ends by discarding every model but one. Average them instead — sorted by held-out accuracy, keeping each only if held-out accuracy improves, which makes the result no worse than the best individual model on that set by construction. O(1) inference against an ensemble’s O(k). +0.7 pp on CLIP and +0.5 pp on ALIGN over the best sweep member; +0.16 on ImageNet and +0.34 under distribution shift for ViT-G. Averaging all of them uniformly instead can come out worse than the best one. | Active |
| SOTA-410 v2 | For guided sampling, use a second-order multistep solver on the data-prediction parameterization, not a higher-order solver on the noise prediction | Lu et al. (2022), LIT-676. SOTA-203’s higher-order solver inverts under guidance: at scale 8.0 and 10 evaluations, FID is 13.04 for first-order DDIM against 114.62 at order 2 and 164.74 at order 3. Sample instead with a second-order multistep solver parameterized on the data prediction — which is also what makes SOTA-202’s per-step clamping available at all, since you cannot clamp x̂₀ if the solver’s state is ε̂. 20 evaluations then beat DDIM at 250. | Active |
| SOTA-413 v2 | In a scaling-law study, count the output head in parameters and FLOPs, and fit loss against compute with an irreducible-loss offset | Pearce and Song (TMLR 2024), LIT-688. At small scale the embedding and output head are a large share of the model. Excluding them from N, and from the FLOPs in C = 6ND, biases the fitted allocation exponent: 0.49 becomes 0.74 when the same runs are relabelled. An offset-free loss–compute fit biases that exponent too. Do not compare exponents across studies that count differently. | Active |
| SOTA-416 v2 | When sampling a diffusion model in few steps, start at t = T: use trailing, not leading, timestep spacing | Martin Garcia et al. (WACV 2025), LIT-687. DDIM’s “leading” spacing never visits the final timestep. So at one step, a pure-noise input is labelled with a timestep that claims an almost clean sample. With the same weights and only the spacing changed, Marigold’s single-step depth goes from noise to NYUv2 AbsRel 5.7. The mismatch vanishes as the step count approaches T, which is why it goes unnoticed at 50 steps. | Active |
| SOTA-424 v4 | Train one network for both conditional and unconditional scores by dropping the condition on 10% of examples, then pick the guidance weight by which metric you are willing to lose | Ho and Salimans (2022), LIT-693. Replace the conditioning with a null token on a fraction of training examples — 0.1 is the measured best, with 0.2 and 0.5 worse — so one network learns both scores, then extrapolate away from the unconditional estimate at sampling. The weight is not a quality dial: on ImageNet 64×64, w from 0.1 to 4.0 takes FID from 1.55 to 26.22 and IS from 66.11 to 260.2. Choose it by which of those you are prepared to sacrifice, and never compare two models at different weights. | Active |
| SOTA-425 v2 | Report precision and recall beside FID whenever the generator has a fidelity-diversity knob, because FID’s best value sits in the interior of that trade | Dhariwal and Nichol (2021), LIT-699. FID mixes fidelity and diversity, so on any knob that trades one for the other its optimum is interior and a single FID cannot say which side of the trade you are on. Their own Table 5 is the demonstration: ADM-G beats BigGAN-deep on FID at ImageNet 128, 256 and 512 while losing precision at all three (0.78/0.86, 0.82/0.87, 0.87/0.88) and winning recall by 0.24, 0.24 and 0.13. Report precision and recall, state the knob’s value, and never read a single FID as “better samples”. | Active |
| SOTA-435 v2 | Never rank attribution methods by a single faithfulness or stability score, and report the perturbation distribution when you report infidelity | Yeh et al. (2019), LIT-725. Max-sensitivity is minimised by a constant explanation — an explanation that ignores the input entirely wins it — so it can never be a quality score on its own. Infidelity is defined relative to a perturbation distribution you choose, and Propositions 2.2–2.5 show several published methods are each the optimum for some choice, which the paper confirms by giving SHAP the lowest infidelity under SHAP’s own perturbation. Both cautions are free to follow: report the pair, and name the perturbation. | Active |