Practices

What to do when you train a model, and why — one document per recommendation, each citing the paper it came from.

A practice is a claim about what you should do, held at a status. Active is current practice; the other four are the machinery that lets this stay a living document without quietly rewriting history. Nothing is deleted here: a practice that stops being right becomes Superseded or Rejected and keeps its body, because “we used to think this, and here is why we stopped” is worth more than a clean list.

Every practice names a source:. That is enforced — a recommendation with no paper behind it is an opinion, and this is not a list of opinions. Follow the source to a reading note for what the paper actually claims.

File one with luria new sota. Never hand-write a link target: write the bare code and run luria link --fix.

By topic

Training optimization (134) — optimizers, learning-rate schedules, batch size, training dynamics, scaling laws and scaling strategies: 001 · 002 · 003 · 008 · 009 · 010 · 011 · 012 · 013 · 014 · 015 · 016 · 031 · 035 · 040 · 041 · 050 · 061 · 062 · 064 · 065 · 066 · 067 · 068 · 092 · 093 · 094 · 096 · 097 · 098 · 099 · 100 · 120 · 121 · 122 · 131 · 139 · 140 · 141 · 142 · 143 · 144 · 156 · 159 · 160 · 162 · 165 · 168 · 188 · 192 · 193 · 195 · 198 · 202 · 204 · 211 · 218 · 221 · 243 · 251 · 254 · 255 · 256 · 257 · 258 · 259 · 260 · 261 · 269 · 270 · 272 · 273 · 274 · 275 · 276 · 282 · 283 · 285 · 287 · 288 · 296 · 297 · 309 · 315 · 316 · 319 · 320 · 324 · 325 · 328 · 338 · 354 · 358 · 359 · 360 · 363 · 372 · 375 · 376 · 377 · 381 · 384 · 385 · 386 · 388 · 395 · 398 · 399 · 401 · 402 · 403 · 404 · 405 · 406 · 408 · 409 · 411 · 412 · 413 · 415 · 417 · 419 · 420 · 421 · 422 · 424 · 426 · 428 · 429 · 431 · 432 · 438 · 439 · 444

Systems optimization (29) — hardware utilization, kernels, compilation, memory access patterns — how an operation is executed, not how many bits it is executed in: 044 · 081 · 082 · 083 · 084 · 085 · 086 · 087 · 088 · 089 · 090 · 091 · 106 · 107 · 108 · 114 · 161 · 230 · 234 · 249 · 252 · 314 · 316 · 326 · 330 · 340 · 372 · 429 · 437

Numerics and precision (23) — how many bits, where, and what that costs — number formats, training precision and the failures it causes, post-training quantization, and the interaction between them: 013 · 014 · 015 · 016 · 043 · 118 · 134 · 158 · 159 · 160 · 161 · 163 · 185 · 230 · 234 · 248 · 283 · 314 · 340 · 355 · 356 · 399 · 437

Model stability (57) — initialization, normalization, gradient handling, regularization, loss-landscape behaviour: 004 · 005 · 006 · 008 · 009 · 010 · 020 · 021 · 022 · 025 · 026 · 027 · 032 · 035 · 050 · 051 · 052 · 053 · 060 · 067 · 069 · 070 · 071 · 072 · 098 · 099 · 100 · 161 · 182 · 191 · 192 · 193 · 217 · 240 · 255 · 282 · 285 · 286 · 288 · 319 · 320 · 325 · 332 · 364 · 365 · 366 · 367 · 385 · 407 · 408 · 426 · 427 · 431 · 432 · 438 · 443 · 446

Distributed optimization (42) — parallelism and sharding, communication, memory management, checkpointing: 017 · 018 · 019 · 028 · 029 · 030 · 031 · 046 · 047 · 048 · 049 · 054 · 055 · 056 · 057 · 058 · 059 · 073 · 074 · 075 · 076 · 116 · 117 · 118 · 119 · 155 · 181 · 189 · 214 · 215 · 216 · 219 · 220 · 222 · 223 · 224 · 225 · 226 · 249 · 364 · 376 · 437

Data pipeline (54) — loading, quality assessment and selection, preprocessing, batch preparation: 042 · 043 · 044 · 045 · 077 · 078 · 079 · 080 · 101 · 102 · 103 · 104 · 124 · 127 · 128 · 152 · 164 · 166 · 170 · 171 · 172 · 173 · 174 · 175 · 183 · 186 · 194 · 238 · 239 · 241 · 242 · 243 · 244 · 245 · 246 · 267 · 268 · 271 · 295 · 341 · 352 · 354 · 359 · 361 · 369 · 370 · 374 · 382 · 387 · 389 · 402 · 417 · 440 · 441

Attention techniques (29) — attention variants and alternative mechanisms, implementation optimizations, context length: 050 · 085 · 086 · 087 · 106 · 107 · 108 · 109 · 110 · 111 · 112 · 131 · 132 · 134 · 135 · 138 · 139 · 147 · 153 · 161 · 176 · 177 · 192 · 208 · 248 · 322 · 390 · 391 · 436

Model architecture (54) — architecture patterns, component design, structural choices, model families — how the network is shaped, not how many signals it takes in: 024 · 032 · 034 · 036 · 037 · 039 · 095 · 124 · 125 · 133 · 136 · 137 · 148 · 149 · 150 · 157 · 158 · 167 · 169 · 178 · 180 · 190 · 209 · 253 · 254 · 262 · 269 · 277 · 282 · 287 · 289 · 297 · 311 · 322 · 326 · 328 · 332 · 334 · 335 · 336 · 349 · 350 · 351 · 358 · 362 · 365 · 400 · 403 · 411 · 418 · 419 · 431 · 434 · 436

Inference optimization (36) — serving-time decisions — batching, cache layout, compression, sparsity, distillation, sampling algorithms: 005 · 023 · 105 · 113 · 115 · 163 · 185 · 227 · 228 · 229 · 269 · 280 · 289 · 296 · 297 · 302 · 314 · 317 · 318 · 327 · 328 · 330 · 340 · 355 · 356 · 392 · 394 · 397 · 410 · 411 · 414 · 416 · 420 · 423 · 433 · 447

Adaptation and tuning (38) — taking a trained model somewhere new — fine-tuning and transfer, preference training and safety alignment, parameter-efficient adaptation, context extension: 033 · 038 · 051 · 123 · 126 · 129 · 130 · 145 · 146 · 154 · 184 · 199 · 201 · 210 · 212 · 213 · 230 · 231 · 233 · 235 · 279 · 283 · 284 · 299 · 302 · 318 · 323 · 329 · 347 · 348 · 393 · 407 · 409 · 414 · 426 · 442 · 445 · 449

In-context learning (13) — getting behaviour out of a fixed model by what you put in the context — few-shot exemplars, chain of thought, prompting strategy, and what the context can and cannot buy: 037 · 038 · 127 · 279 · 280 · 281 · 300 · 304 · 308 · 313 · 323 · 324 · 357

Representation and encoding (44) — how the signal is encoded before the expensive network sees it — tokenizers and learned latents, positional encoding, and the frequency or basis choices that go with them: 007 · 063 · 151 · 153 · 179 · 187 · 247 · 250 · 271 · 273 · 294 · 299 · 305 · 306 · 312 · 327 · 331 · 346 · 360 · 361 · 362 · 363 · 365 · 367 · 368 · 369 · 370 · 371 · 372 · 373 · 374 · 375 · 379 · 380 · 381 · 391 · 396 · 398 · 400 · 401 · 405 · 406 · 418 · 434

Analysis and evaluation (70) — how to find out whether something worked — what to measure, what a measurement cannot tell you, and which comparisons are unsound; theory, interpretability and debugging belong here too: 011 · 012 · 194 · 196 · 197 · 200 · 210 · 241 · 245 · 246 · 256 · 270 · 278 · 284 · 286 · 290 · 291 · 292 · 293 · 298 · 300 · 303 · 304 · 305 · 307 · 308 · 309 · 310 · 311 · 312 · 313 · 315 · 317 · 321 · 323 · 324 · 325 · 337 · 342 · 343 · 344 · 345 · 346 · 347 · 348 · 349 · 350 · 353 · 357 · 366 · 368 · 371 · 378 · 379 · 380 · 383 · 384 · 387 · 388 · 391 · 404 · 413 · 415 · 421 · 425 · 428 · 430 · 433 · 435 · 439

Generative modeling (61) — diffusion, samplers, text-to-image, conditioning and control: 157 · 187 · 188 · 195 · 202 · 203 · 204 · 206 · 207 · 232 · 254 · 262 · 263 · 264 · 265 · 266 · 289 · 301 · 302 · 306 · 307 · 333 · 334 · 335 · 336 · 337 · 338 · 339 · 341 · 346 · 352 · 383 · 386 · 389 · 390 · 392 · 393 · 394 · 395 · 396 · 397 · 410 · 412 · 416 · 422 · 423 · 424 · 425 · 428 · 431 · 432 · 433 · 440 · 442 · 443 · 444 · 445 · 446 · 447 · 448 · 449

Vision and graphics (35) — neural rendering, reconstruction, perception, visual foundation models: 205 · 236 · 237 · 250 · 251 · 252 · 253 · 299 · 331 · 332 · 333 · 335 · 336 · 338 · 339 · 341 · 358 · 359 · 383 · 386 · 388 · 390 · 393 · 394 · 395 · 396 · 397 · 398 · 400 · 401 · 414 · 416 · 429 · 434 · 440

Multimodal learning (10) — what changes when one model has to take in more than one kind of signal — joint architectures and fusion, contrastive image-text training, cross-modal transfer, and what a second modality buys the first: 262 · 271 · 357 · 359 · 376 · 377 · 389 · 407 · 429 · 436

Deployment and society (7) — what a system does once it is running among people, and how anyone outside it could find out — audits of deployed platforms and recommenders, information ecosystems, moderation and governance, and the access problems and funding pressures of studying any of it from the outside: 290 · 291 · 292 · 298 · 303 · 309 · 321

Biomolecular modeling (3) — models whose data is molecules — protein and nucleic-acid structure prediction, biological sequence language models, docking and molecular property prediction — what they get right, what their benchmarks can show, and which of their methods travel to other domains: 352 · 353 · 354

Signal structure (10) — what the data itself is like, independent of any model — the statistics and structure of language, images and other signals that methods exploit or must respect, such as frequency spectra, heavy-tailed and Zipfian distributions, and what counts as a unit: 172 · 187 · 238 · 243 · 263 · 267 · 274 · 373 · 374 · 382

Concept geometry (4) — how concepts are laid out in a model’s representation space — linear directions and the regions, lattices and spectra built from them, superposition and the dictionaries that pull it apart, reading concepts out with probes and writing them in with steering vectors, and whether different models arrive at the same geometry: 342 · 344 · 345 · 378

Agents and environments (0) — systems that perceive, act and are changed by the consequences — percept–action loops, world models and the environments agents are trained in, multi-agent and collective behaviour, and what acting rather than only predicting changes about learning, memory and their costs.

Graphs and networks (2) — graph-structured data and the models that run on it — message passing and graph neural networks, how they are evaluated, graph curvature, rewiring and over-squashing, community structure and partitioning: 350 · 351

Physical sciences (0) — models whose data is the physical world — learned surrogates for simulation in fluids, PDEs, molecular dynamics and cosmology, foundation models for weather, climate and solar observation — what they are trusted to replace, what their benchmarks can show, and which of their methods travel to other domains.

Human–AI interaction (1) — how people work with a model through an interface — prompting and steering controls, editing and co-writing tools, assistive and co-adaptive interfaces, how such interfaces are scored, and what lab studies of them can and cannot show: 309

Tiny models (12) — claims that hold at the small end and not in general — sub-billion-parameter training, where the usual scaling advice inverts: 121 · 122 · 123 · 124 · 125 · 126 · 127 · 128 · 293 · 294 · 295 · 419

Capability thresholds (4) — a capability that arrives abruptly rather than smoothly — emergence at scale, grokking after long training, phase transitions in learning — which axis it turns on, and whether the discontinuity is real or an artefact of how it was measured: 037 · 200 · 279 · 402

Flows and transport (7) — generative models as learned transport between distributions — flow matching, rectified flow, stochastic interpolants, normalizing flows and neural ODEs, and the paths, couplings and time schedules they are trained on; Gaussian diffusion is the special case (THEORY-106). A subtopic, worn beside generative-modeling: 263 · 265 · 266 · 392 · 442 · 444 · 448

Few-step generation (8) — getting a sample in one or a few network evaluations — progressive, distribution-matching and adversarial distillation, consistency, flow-map, shortcut and mean-flow models, and fast ODE solvers. A subtopic, worn beside generative-modeling or inference-optimization: 206 · 289 · 392 · 394 · 443 · 446 · 447 · 448

By consensus: Not judged (171, the default) · One source (145) · In dispute (18) · Spreading (61) · Agreed (40) · Assumed (14)

By status: Current practice (236) · Promising (181) · Not yet judged (0) · Replaced (13) · Retired (19)

Chronological

What the status column means in this scheme — the words are luria’s, the meanings are this project’s.

Status		Means
Active	Current practice	What you should do today, and the reason is one click away
Proposed	Promising	Early results are good, validation at scale is missing — the old experimental flag
Deferred	Not yet judged	Probably right, not yet worth asserting; usually a question of adoption rather than correctness
Superseded	Replaced	Something newer does this better, and the record says what
Rejected	Retired	No longer believed, with no successor — the attic

#	Title	Summary	Status
SOTA-001 v2	Use Adam as the default optimizer absent a reason to choose otherwise	Kingma et al. (2014), LIT-001 — ARXIV-1412.6980.	Active
SOTA-002 v2	Common hyperparameters: β₁=0.9, β₂=0.999, ε=1e-8	Kingma et al. (2014), LIT-001 — ARXIV-1412.6980.	Active
SOTA-003	Learning rate typically 1e-4 to 1e-3 for most tasks	Kingma et al. (2014), LIT-001 — ARXIV-1412.6980.	Active
SOTA-004	Place BatchNorm after linear/conv layers but before activation functions	Ioffe et al. (2015), LIT-002 — ARXIV-1502.03167.	Active
SOTA-005	Use running statistics for inference	Ioffe et al. (2015), LIT-002 — ARXIV-1502.03167.	Active
SOTA-006	Consider alternatives like LayerNorm for transformers	Ioffe et al. (2015), LIT-002 — ARXIV-1502.03167.	Active
SOTA-007 v2	BPE tokenization for open vocabulary tasks	Sennrich et al. (2015), LIT-003 — ARXIV-1508.07909.	Active
SOTA-008 v3	linear warmup of LR stabilizes early training with large batch size.	You et al. (2017), LIT-009 — ARXIV-1708.03888.	Active
SOTA-009 v2	warmup to a large early lr, anneal throughout training to small final lr	Smith et al. (2017), LIT-010 — ARXIV-1708.07120.	Active
SOTA-010 v2	skip connections promote training stability by smoothing out the loss landscape	Li et al. (2017), LIT-014. Moved to THEORY-011 — the claim is a statement about why deep networks became trainable, not an instruction, and belongs in the scheme ADR-031 created for exactly that. This code stays so links into it resolve.	Superseded — by THEORY-011; Not a recommendation, and never was: it states what is true rather than what to do, which is a THEORY under ADR-031 and was one of the eight that decision named and declined to move. The claim is unchanged and is believed — it now lives at THEORY-011, with the figure’s limits and the five practices it underwrites. Superseded rather than Rejected because Rejected on a practice means do not do this, and nobody should read this code as advice against skip connections. ADR-034 is the rule.
SOTA-011 v3	Map the Hessian ratio |lambda_min / lambda_max| to find where the loss surface is non-convex	Li et al. (2017), LIT-014 — ARXIV-1712.09913.	Active
SOTA-012 v3	sharpness in the loss landscape correlates with test error	Li et al. (2017), LIT-014 — ARXIV-1712.09913.	Active
SOTA-013	Use dynamic loss scaling that doubles every 2000 successful steps	Micikevicius et al. (2017), LIT-011 — ARXIV-1710.03740.	Active
SOTA-014	Maintain master weights in FP32	Micikevicius et al. (2017), LIT-011 — ARXIV-1710.03740.	Active
SOTA-015 v2	Store optimizer states in FP32	Micikevicius et al. (2017), LIT-011 — ARXIV-1710.03740.	Active
SOTA-016	Perform forward/backward passes in FP16	Micikevicius et al. (2017), LIT-011 — ARXIV-1710.03740.	Active
SOTA-017 v2	Use micro-batch splitting for pipeline parallelism	Huang et al. (2018), LIT-016 — ARXIV-1811.06965.	Active
SOTA-018 v2	Balance pipeline stages to minimize bubble overhead	Huang et al. (2018), LIT-016 — ARXIV-1811.06965.	Active
SOTA-019 v2	Choose pipeline chunks based on memory vs. compute trade-off	Huang et al. (2018), LIT-016 — ARXIV-1811.06965.	Active
SOTA-020	Use larger learning rates with batch normalization	Santurkar et al. (2018), LIT-015 — ARXIV-1806.02375.	Active
SOTA-021	Monitor loss landscape smoothness during training	Santurkar et al. (2018), LIT-015 — ARXIV-1806.02375.	Rejected — states no threshold and no response, and the smoothing it refers to is a mechanism from a paper the record had conflated with this one. Measuring landscape smoothness during training is not something the practice’s source does or recommends; the finding it rests on is Santurkar et al.’s explanation of why batch normalization works, which is a fact about the technique rather than an instruction
SOTA-022	Place BN after linear/conv layers but before activation functions	Santurkar et al. (2018), LIT-015 — ARXIV-1806.02375.	Superseded — by SOTA-004; a duplicate of SOTA-004, which states the same placement rule from batch normalization’s own paper. The two arrived in the same import from different notes and have coexisted since; SOTA-004 is the one with the originating source
SOTA-023 v2	Use multi-query attention for decoder-only models to reduce memory bandwidth	Shazeer (2019), LIT-024 — ARXIV-1911.02150.	Superseded — by SOTA-109
SOTA-024 v2	Keep key/value projections shared across heads while query projections remain separate	Shazeer (2019), LIT-024 — ARXIV-1911.02150.	Superseded — by SOTA-109
SOTA-025 v2	Initialize the LayerNorm gain to 1	Ba et al. (2016), LIT-005 — ARXIV-1607.06450. Identity at initialisation: the normalised signal passes through unchanged. A convention rather than a published result, and the record says so.	Active
SOTA-026 v2	Initialize LayerNorm bias to 0	Ba et al. (2016), LIT-005 — ARXIV-1607.06450. Zero is the identity; a convention rather than a published result, and moot in any model using RMSNorm.	Active
SOTA-027 v2	Use a smaller learning rate for LayerNorm parameters	Xu et al. (2019), LIT-025 — ARXIV-1911.07013.	Rejected — The cited paper contains no discussion of learning rates and argues the parameters in question should be removed rather than tuned. The durable claim — treat norm parameters as their own group — is elsewhere in the record, and is about weight decay rather than rate
SOTA-028	Stage optimizer states across data parallel ranks (ZeRO-1)	Rajbhandari et al. (2020), LIT-027 — ARXIV-1910.02054.	Active
SOTA-029	Partition gradients and optimizer states (ZeRO-2) for larger models	Rajbhandari et al. (2020), LIT-027 — ARXIV-1910.02054.	Active
SOTA-030	Use ZeRO-3 only when other strategies insufficient	Rajbhandari et al. (2020), LIT-027 — ARXIV-1910.02054.	Active
SOTA-031 v2	Keep micro-batch size per GPU as large as memory allows	Rajbhandari et al. (2020), LIT-027 — ARXIV-1910.02054.	Active
SOTA-032 v5	Put the layer normalization inside the residual block, before the sublayer	Xiong et al. (2020), LIT-114 — ARXIV-2002.04745. Pre-LN: normalize the input to each sublayer rather than the sum after it, so the gradients near the output are well behaved at initialization.	Active
SOTA-033	continued pre-training for fine tuning	Gururangan et al. (2020), LIT-034 — ARXIV-2004.10964.	Active
SOTA-034 v6	Use SwiGLU activation for transformers	Shazeer et al. (2020), LIT-030 — ARXIV-2002.05202. Contested at frontier scale in low precision: two groups now ship bounded replacements, on the grounds that SwiGLU’s unbounded factors produce activation outliers.	Active
SOTA-035	use gradient clipping	Pascanu et al. (2012), LIT-037 — ARXIV-1211.5063.	Active
SOTA-036 v2	Train a decoder-only transformer on a broad web corpus with a fixed context and a single next-token objective	Brown et al. (2020), LIT-035 — ARXIV-2005.14165.	Active
SOTA-037	LM in-context learning emerges at scale	Brown et al. (2020), LIT-035 — ARXIV-2005.14165.	Active
SOTA-038 v3	ICL permits few-shot task adaptability	Brown et al. (2020), LIT-035 — ARXIV-2005.14165.	Active
SOTA-039	single cycle of cosine decay is sufficient lr schedule	Brown et al. (2020), LIT-035 — ARXIV-2005.14165.	Superseded — by SOTA-140; warmup-stable-decay matches it and leaves the token budget open; every report since 2024 in the record uses a stable-then-decay schedule
SOTA-040	larger models are more sample efficient	Kaplan et al. (2020), LIT-028 — ARXIV-2001.08361.	Active
SOTA-041 v2	lr tuning less important for larger models	Kaplan et al. (2020), LIT-028 — ARXIV-2001.08361.	Rejected — States the opposite of its source. Kaplan reports that larger models require a smaller learning rate to prevent divergence, and carries an explicit size-dependent rule LR(N) because the right rate is not size-independent
SOTA-042	Memory-map large datasets	Mohan et al. (2020), LIT-050 — ARXIV-2007.06775.	Active
SOTA-043	Use mixed precision during data loading	Mohan et al. (2020), LIT-050 — ARXIV-2007.06775.	Active
SOTA-044	Pin memory for CPU-GPU transfers	Afroz et al. (2025), LIT-737 — ARXIV-2511.14124. Host-to-device copies from pinned memory ran at 24.74 GB/s against 10.16 GB/s from pageable memory on PCIe 4.0; the advice itself is NVIDIA’s, not a paper’s.	Active
SOTA-045	Profile data loading separate from training	Mohan et al. (2020), LIT-050 — ARXIV-2007.06775.	Active
SOTA-046	Use hierarchical allreduce for tensors > 1MB	Jiang et al. (2020), LIT-051 — https://www.usenix.org/conference/osdi20/presentation/jiang.	Rejected — the ’> 1MB’ crossover has no owner: NCCL measures the algorithm choice per topology rather than publishing a constant, and LIT-051 does not discuss it. What survives is that a modern collectives library already chooses by size and topology, which is not a practice a reader acts on
SOTA-047	Overlap communication with backward pass	Jiang et al. (2020), LIT-051 — https://www.usenix.org/conference/osdi20/presentation/jiang.	Active
SOTA-048	Group small tensors before communication	Jiang et al. (2020), LIT-051 — https://www.usenix.org/conference/osdi20/presentation/jiang.	Active
SOTA-049	Set buffer size to network bandwidth-delay product	Jiang et al. (2020), LIT-051 — https://www.usenix.org/conference/osdi20/presentation/jiang.	Rejected — collapses two different quantities into one unfollowable sentence — a TCP socket buffer sized by bandwidth-delay product, and a gradient fusion buffer sized by the overlap trade. The fusion-buffer half is SOTA-048, which now carries Horovod’s attributable 64 MB default; the transport half is a claim about networks with no source here
SOTA-050 v2	Scale attention weights by 1/sqrt(head_dim)	Vaswani et al. (2017), LIT-008 — ARXIV-1706.03762.	Active
SOTA-051 v5	Initialize a residual or adapter branch to exactly zero, not merely near zero	Bachlechner et al. (2020), LIT-047 — ARXIV-2003.04887. Start an added branch at exactly zero so it is the identity at initialisation, and let training raise it. Exact zero rather than small-random is the claim: a small random branch is PROBABLY harmless, a zero one is PROVABLY the identity — which is what matters when the thing being protected is a residual stack deep enough to attenuate, or a pretrained model expensive enough that adapter noise is not worth risking.	Active
SOTA-052	Use smaller variance for deep networks	Wang et al. (2022), LIT-084 — ARXIV-2203.00555.	Active
SOTA-053	Special handling for gated architectures	Bachlechner et al. (2020), LIT-047 — ARXIV-2003.04887.	Active
SOTA-054 v2	Derive the checkpoint interval from online profiling and adapt it at runtime against an overhead bound	Mohan et al. (2021), LIT-059 — https://www.usenix.org/conference/fast21/presentation/mohan.	Active
SOTA-055	Save optimizer state every N epochs (N ~ sqrt(total_epochs))	Mohan et al. (2021), LIT-059 — https://www.usenix.org/conference/fast21/presentation/mohan.	Rejected — the practice its own cited source exists to replace. LIT-059 names epoch-granular, hand-tuned checkpointing as the problem it solves, on both counts; SOTA-054 now states what that paper actually recommends
SOTA-056	Use async I/O for checkpoint writing	Mohan et al. (2021), LIT-059 — https://www.usenix.org/conference/fast21/presentation/mohan.	Active
SOTA-057	Implement multi-level checkpoint strategy	Mohan et al. (2021), LIT-059 — https://www.usenix.org/conference/fast21/presentation/mohan.	Active
SOTA-058 v2	Use sequence parallelism for the layer-norm and dropout regions that tensor parallelism leaves replicated	Korthikanti et al. (2022), LIT-736 — ARXIV-2205.05198.	Active
SOTA-059	Overlap communication with computation when possible	Jangda et al. (2021), LIT-735 — ARXIV-2105.05720.	Active
SOTA-060 v4	Initialize weights from N(0, 0.02) and scale the residual-output projections by 1/sqrt(2N)	Shoeybi et al. (2019), LIT-022 — ARXIV-1909.08053.	Active
SOTA-061 v3	Use largest batch that maintains >80% sample efficiency	McCandlish et al. (2018), LIT-017 — ARXIV-1812.06162. The steps/examples trade the rule thresholds; the 80% line itself has no published origin the record can name.	Active
SOTA-062 v3	Scale batch size with model size but sub-linearly	Kaplan et al. (2020), LIT-028 — ARXIV-2001.08361. Along the compute-optimal frontier the model grows as C^0.73 and the batch as C^0.24: over a billion-fold increase in compute, more than a million-fold in model size against about a hundred-fold in batch.	Superseded — by SOTA-258; The decoupling experiment this practice’s own body invited has been run: holding the token budget fixed, the dependence of batch size on model size very nearly vanishes. What the heuristic was tracking is the data that scaled alongside the model. Superseded rather than Rejected because following it along the Chinchilla line gives roughly the right answer for the wrong reason — the successor is SOTA-258.
SOTA-063 v3	use RoPE for LLM (1D sequence) positional embeddings	Su et al. (2021), LIT-045 — ARXIV-2104.09864.	Active
SOTA-064 v2	Warmup needed scales sub-linearly with model size	Tay et al. (2021), LIT-052 — ARXIV-2109.10686. Rejected: the source contains no discussion of warmup.	Rejected — The cited paper does not contain the word “warmup”. Its subject is model shape for downstream fine-tuning; this claim came from a takeaway that belonged to no identifiable source, and no paper in the record supports it
SOTA-065 v2	Initialize layer norm weights closer to 1 for larger models	Tay et al. (2021), LIT-052 — ARXIV-2109.10686. Rejected: the source contains no discussion of layer normalization, and the claim names no adjustable quantity.	Rejected — The cited paper contains no discussion of layer normalization. The claim is also doubtful on its own terms: LayerNorm’s gain initialises to exactly 1 by convention at every scale, so “closer to 1 for larger models” names no adjustable quantity
SOTA-066 v2	Can use shorter warmup periods for wider models	Tay et al. (2021), LIT-052 — ARXIV-2109.10686. Rejected: the source contains no discussion of warmup.	Rejected — The same gap as SOTA-064, restated as a width claim. The cited paper contains no discussion of warmup at all
SOTA-067 v2	Monitor loss specifically during first ~5000 steps for instabilities	Tay et al. (2021), LIT-052 — ARXIV-2109.10686. Rejected: the source contains no discussion of instability, and the 5000 belongs to nobody.	Rejected — The cited paper contains no discussion of training instability. The record’s sourced monitoring practices are SOTA-069 and SOTA-099, which do not depend on a step count
SOTA-068 v2	Use gradient clipping during early training phase	Tay et al. (2021), LIT-052 — ARXIV-2109.10686. Superseded by SOTA-035, which cites Pascanu et al. and drops the “early training” qualifier.	Superseded — by SOTA-035; SOTA-035 is the same recommendation with a real source and without the false qualifier — clipping is applied throughout training, not only early
SOTA-069	Monitor exp(loss) for stability	Zeng et al. (2022), LIT-054 — ARXIV-2210.02414.	Active
SOTA-070	Track gradient norm ratios between layers	Zeng et al. (2022), LIT-054 — ARXIV-2210.02414.	Active
SOTA-071	Use gradient clipping with dynamic threshold	Zeng et al. (2022), LIT-054 — ARXIV-2210.02414.	Active
SOTA-072	Implement early warning system for NaNs	Zeng et al. (2022), LIT-054 — ARXIV-2210.02414.	Active
SOTA-073	Monitor network utilization during training	Jiang et al. (2020), LIT-051 — https://www.usenix.org/conference/osdi20/presentation/jiang.	Rejected — states no signal, threshold or action, so there is nothing to do differently on reading it, and LIT-051 does not argue for it. The specific version worth having is in SOTA-048’s body: the gap between achieved and peak bandwidth during the overlap window tells you the fusion buffer is sized wrong
SOTA-074	Adapt buffer sizes to network conditions	Jiang et al. (2020), LIT-051 — https://www.usenix.org/conference/osdi20/presentation/jiang.	Rejected — names a controller — signal, target, update rule — of which none is stated, and no source here describes one. A measured version would resemble what CheckFreq does for checkpoint intervals (SOTA-054)
SOTA-075	Use gradient compression for slow networks	Lin et al. (2017), LIT-056 — ARXIV-1712.01887.	Active
SOTA-076	Place replicas to minimize cross-rack traffic	Jiang et al. (2020), LIT-051 — https://www.usenix.org/conference/osdi20/presentation/jiang.	Rejected — a cluster-scheduling claim rather than a training one: it is the scheduler’s decision, expressed as a placement-group request, not something in a training run. ADR-026 (superseding ADR-020) scopes this record by the kind of claim, and this is a different kind
SOTA-077	Use tar archives for dataset storage	Aizman et al. (2020), LIT-053 — ARXIV-2001.01858.	Active
SOTA-078	Buffer size should be 2-3x batch size	Aizman et al. (2020), LIT-053 — ARXIV-2001.01858.	Active
SOTA-079	Pre-fetch next batch during compute	Aizman et al. (2020), LIT-053 — ARXIV-2001.01858.	Active
SOTA-080	Use multiple worker processes (num_workers = 4 * num_gpus)	Aizman et al. (2020), LIT-053 — ARXIV-2001.01858.	Active
SOTA-081	Use operator fusion for small operations	Chen et al. (2018), LIT-063 — ARXIV-1802.04799.	Active
SOTA-082	Optimize memory layout for hardware	Chen et al. (2018), LIT-063 — ARXIV-1802.04799.	Active
SOTA-083 v2	Implement custom kernels for critical ops	Chen et al. (2018), LIT-063 — ARXIV-1802.04799.	Active
SOTA-084	Profile-guided optimization for hot paths	Chen et al. (2018), LIT-063 — ARXIV-1802.04799.	Active
SOTA-085 v4	Use flash attention for all attention computations when hardware supports it	Dao et al. (2022), LIT-074 — ARXIV-2205.14135.	Active
SOTA-086 v4	Tiling size should match hardware SRAM size	Dao et al. (2022), LIT-074 — ARXIV-2205.14135.	Active
SOTA-087 v4	Recompute attention during backward pass instead of storing it	Dao et al. (2022), LIT-074 — ARXIV-2205.14135.	Active
SOTA-088	Fuse small operations into larger kernels	Ivanov et al. (2020), LIT-066 — ARXIV-2007.00072.	Active
SOTA-089	Align tensor dimensions to hardware boundaries	Ivanov et al. (2020), LIT-066 — ARXIV-2007.00072.	Active
SOTA-090	Use hardware-specific memory layouts	Ivanov et al. (2020), LIT-066 — ARXIV-2007.00072.	Active
SOTA-091	Profile and optimize memory access patterns	Ivanov et al. (2020), LIT-066 — ARXIV-2007.00072.	Active
SOTA-092	smaller batch sizes are more sample efficient (i.e., better loss as a function of tokens seen) earlier in training	Chowdhery et al. (2022), LIT-069 — ARXIV-2204.02311.	Active
SOTA-093	larger batch sizes are beneficial later in training due to better gradient estimates	Chowdhery et al. (2022), LIT-069 — ARXIV-2204.02311.	Active
SOTA-094	count accelerator efficiency alongside sample efficiency when choosing a batch size: larger batches mean larger matrix multiplications	Chowdhery et al. (2022), LIT-069 — ARXIV-2204.02311.	Active
SOTA-095	consider rewinding to earlier checkpoint and skipping a few batches to mitigate unusual loss spikes	Chowdhery et al. (2022), LIT-069 — ARXIV-2204.02311.	Active
SOTA-096 v2	num_tokens ~ 20 * num_params	Hoffmann et al. (2022), LIT-068 — ARXIV-2203.15556.	Active
SOTA-097 v4	Optimal batch size scales approximately with compute budget - B ∝ C^(1/4)	Kaplan et al. (2020), LIT-028 — ARXIV-2001.08361. Equation 1.7 gives B ∝ C^(α_C/α_B) = C^0.24, from α_C ≈ 0.050 and α_B ≈ 0.21.	Superseded — by SOTA-258; The exponent is a fit to a projection. Along the Chinchilla line N and D scale together, so C = 6ND moves with D and a power law in either describes the data; two independent groups measuring off that line find optimal and critical batch size scale in the token budget, not in compute. Superseded rather than Rejected because the fit is not an error and the quantity is real — the successor is SOTA-258, and ADR-017’s corrective succession is the shape.
SOTA-098 v2	Monitor the training loss for unexpected spikes, for the whole run	Chowdhery et al. (2022), LIT-069 — ARXIV-2204.02311. The 540B model’s training loss spiked about 20 times, at irregular intervals, sometimes late, despite gradient clipping, and never in the smaller models.	Active
SOTA-099	Track gradient norm statistics to detect training instabilities	Zeng et al. (2022), LIT-054 — ARXIV-2210.02414.	Active
SOTA-100 v2	Use learning rate warmup proportional to model size	Xiong et al. (2020), LIT-114 — ARXIV-2002.04745.	Active
SOTA-101 v2	Use perplexity-based filtering for quality assessment	Albalak et al. (2023), LIT-117 — ARXIV-2312.02406.	Rejected — The cited paper performs no filtering — it reweights sampling over domains — and filtering has the opposite failure mode to what it recommends. The string “filter” does not appear in it
SOTA-102 v2	Implement dynamic temperature scaling for mixing	Albalak et al. (2023), LIT-117 — ARXIV-2312.02406.	Superseded — by SOTA-103; Names a real component of ODM’s policy imprecisely rather than something the paper does not do — the two were one practice split in the import, and SOTA-103 now describes the whole method
SOTA-103 v2	Adjust data mixing proportions online from per-domain training loss	Albalak et al. (2023), LIT-117 — ARXIV-2312.02406.	Active
SOTA-104 v2	Monitor domain coverage during training	Albalak et al. (2023), LIT-117 — ARXIV-2312.02406.	Rejected — Names no signal and no response, and is in tension with its own source: under ODM the sampling distribution is supposed to move away from uniform coverage. The string “domain coverage” does not appear in the paper
SOTA-105	PagedAttention to accelerate batch inference for LLM sampling	Kwon et al. (2023), LIT-112 — ARXIV-2309.06180.	Active
SOTA-106 v3	Use flash-attention-2 over original flash-attention when available	Dao et al. (2023), LIT-106 — ARXIV-2307.08691.	Active
SOTA-107 v3	Keep sequence lengths multiple of 128 for best performance	Dao et al. (2023), LIT-106 — ARXIV-2307.08691.	Rejected — The 128 is traced and belongs to a different quantity. Every occurrence in the source is a head dimension or a block size; none is a sequence length, and “multiple of” and “divisible” appear zero times
SOTA-108 v3	Pad attention masks to block boundaries for better hardware utilization	Dao et al. (2023), LIT-106 — ARXIV-2307.08691.	Rejected — Not in the cited paper. “padding”, “padded”, “pad”, “block boundar” and “divisible” appear zero times each; the body’s reasoning about skipping fully-masked blocks is correct and is not the source’s
SOTA-109 v3	Prefer GQA to MQA or MHA	Ainslie et al. (2023), LIT-100 — group the query heads and give each group one key/value head: multi-query’s cache saving without multi-query’s quality loss, and uptrainable from an existing multi-head checkpoint. Still the default for a model not paying MLA’s implementation cost.	Active
SOTA-110 v3	Consider Monarch Mixer in place of attention for very long sequence tasks	Fu et al. (2023), LIT-115 — ARXIV-2310.12109.	Active
SOTA-111 v4	Use Monarch Mixer where attention’s quadratic cost is what bottlenecks training	Fu et al. (2023), LIT-115 — ARXIV-2310.12109.	Active
SOTA-112 v4	Combine Monarch Mixer layers with standard attention rather than replacing it wholesale	Fu et al. (2023), LIT-115 — ARXIV-2310.12109.	Rejected — The source is attention-free and argues for replacement, not combination. “hybrid” occurs zero times in it; the hybrid argument belongs to the attention/SSM literature elsewhere in this record
SOTA-113 v2	Use continuous batching for inference	Kwon et al. (2023), LIT-112 — ARXIV-2309.06180.	Active
SOTA-114	Fuse attention operations where possible	Dao et al. (2022), LIT-074 — ARXIV-2205.14135.	Active
SOTA-115	Overlap prefill and decode compute	Agrawal et al. (2023), LIT-103 — ARXIV-2308.16369.	Active
SOTA-116	Use FSDP over DDP when model size exceeds single GPU memory	Zhao et al. (2022), LIT-083 — ARXIV-2304.11277.	Active
SOTA-117	Overlap communication with computation using backward prefetch	Zhao et al. (2022), LIT-083 — ARXIV-2304.11277.	Active
SOTA-118	Employ mixed precision to reduce memory usage	Zhao et al. (2022), LIT-083 — ARXIV-2304.11277.	Active
SOTA-119	Choose sharding factor based on model and GPU memory size	Zhao et al. (2022), LIT-083 — ARXIV-2304.11277.	Active
SOTA-120 v2	Prefer AdamW’s decoupled weight decay to L2 regularization added to the loss	Loshchilov et al. (2017), LIT-012 — ARXIV-1711.05101.	Active
SOTA-121 v3	Use Muon with decoupled weight decay and AdamW-matched update RMS in place of AdamW	Falcon-LLM Team (2026), LIT-119 — the Falcon-H1-Tiny technical blogpost. Stable at nearly the same optimal LR as AdamW, better evaluations; used for every Falcon-H1-Tiny model.	Active
SOTA-122 v2	Attach learnable per-row and per-column multipliers to weight matrices so their norms are learned, not set by LR and WD	Falcon-LLM Team (2026), LIT-119 — the Falcon-H1-Tiny technical blogpost. Up to 20% relative gain on MMLU, BBH and GSM8K over a Muon baseline at 200 GT.	Proposed
SOTA-123	For tiny specialized models, pretrain from scratch on the target SFT or reasoning data instead of pretrain-then-finetune	Falcon-LLM Team (2026), LIT-119 — the Falcon-H1-Tiny technical blogpost. At 90M, 25% SFT data in the pretraining mix beat a separate SFT stage by 10 IFEval points and yields one checkpoint that is both base and instruct.	Active
SOTA-124 v5	Repeat high-quality data freely when its epoch size exceeds the model’s memorization window	Falcon-LLM Team (2026), LIT-119 — the Falcon-H1-Tiny technical blogpost. Roughly 100–500 GT for a 7B model, scaling linearly; the authors call the understanding early.	Proposed
SOTA-125 v3	At a fixed tiny parameter budget, spend parameters on depth and SSM state width before MLP width	Falcon-LLM Team (2026), LIT-119 — the Falcon-H1-Tiny technical blogpost. 24–27 layers × 512 hidden and a large SSM state dimension won every ablation at 90M; 50 layers gained MMLU but halved throughput.	Proposed
SOTA-126	Run DPO on tiny models for one epoch only	Falcon-LLM Team (2026), LIT-119 — the Falcon-H1-Tiny technical blogpost. One epoch at LR 1e-6 to 3e-6 took a 90M model’s IFEval from about 50 to over 65; a second epoch degraded it while the reward kept rising.	Active
SOTA-127 v2	Filter chain-of-thought traces out of the training data of tiny specialized models	Falcon-LLM Team (2026), LIT-119 — the Falcon-H1-Tiny technical blogpost. Reasoning traces interleaved with tool-calling data sent a 90M model into repetition loops; removing them fixed it at once.	Active
SOTA-128	Train fill-in-the-middle data without masking the loss on non-FIM tokens	Falcon-LLM Team (2026), LIT-119 — the Falcon-H1-Tiny technical blogpost. At 80 GT and 90M, computing the loss on every token beat masking the prefix and suffix; matches what the reference FIM recipes appear to do.	Active
SOTA-129 v2	Build a reasoning model in three stages: pretrain on general data, SFT on reasoning traces, then RL with verifiable rewards	Olmo Team (2025), LIT-130 — the recipe behind every open reasoning model at 7B and above, released in Olmo 3 with every stage and checkpoint in the open. Filed so its variations (SOTA-130) and its tiny-scale successor (SOTA-123) have something to be traced back to.	Active
SOTA-130 v4	Skip the reasoning SFT stage and run RL with verifiable rewards directly on the base model	Olmo Team (2025), LIT-130 — the RL-Zero track: RLVR on the base model with no reasoning SFT in between, released as an open benchmark for studying RL rather than as the recommended recipe; a variation on SOTA-129 that the record keeps so the family of reasoning recipes can be traced.	Proposed
SOTA-131 v4	When training with Muon at scale, rescale query and key weights whenever attention logits exceed a threshold (QK-Clip)	Kimi Team (2025), LIT-132 — MuonClip carried a 1T/32B MoE through 15.5T tokens with zero loss spikes where plain Muon let attention logits pass 1000; confirmed at 2.8T in LIT-131.	Active
SOTA-132 v3	Interleave linear-attention layers with global attention at about 3:1 instead of using full attention throughout	Kimi Team (2025), LIT-133 — three Kimi Delta Attention layers per gated-MLA layer beat full MLA at 48B/1.4T while cutting KV cache 75%; the layout LIT-131 ships at 2.8T with 69 KDA and 24 MLA layers.	Active
SOTA-133	Replace fixed residual accumulation with learned attention over preceding layers	Kimi Team (2026), LIT-134 — Attention Residuals: a per-layer pseudo-query chooses which earlier layers to read, at O(d) parameters per layer; 1.25× compute advantage on scaling laws at 48B/1.4T, adopted in Kimi K3.	Active
SOTA-134 v2	Gate each attention head’s output with a sigmoid after the scaled dot-product	Qiu et al. (2025), LIT-138 — the best of 30 gating variants at 15B MoE and 1.7B dense over 3.5T tokens: better quality, more stable training, larger tolerable learning rates, no attention sinks; shipped in every Qwen full-attention layer since Qwen3-Next.	Active
SOTA-135 v2	Give linear-attention layers a gated delta rule: a decay gate for erasure plus a delta update for targeted writes	Yang et al. (2024), LIT-137 — beats Mamba2 and DeltaNet across language modelling, retrieval and long context at 1.3B/100B; the recurrence the Qwen hybrids use for three layers in four, and the one Kimi Delta Attention extends.	Active
SOTA-136 v4	Widen the residual stream into several streams mixed by doubly-stochastic matrices (manifold-constrained hyper-connections)	Xie et al. (2025), LIT-140 — hyper-connections with the residual-mixing matrix constrained to be doubly stochastic, so identity mapping survives; 6.7% training overhead at n = 4 in large-scale training, shipped in DeepSeek-V4.	Proposed
SOTA-137	Widen the residual stream into several streams with freely learned mixing (hyper-connections)	Zhu et al. (2024), LIT-141 — n parallel residual streams with learnable mixing; gains on dense and MoE pretraining, and an identity-mapping property lost that its successor restores.	Superseded — by SOTA-136; constraining the mixing to a doubly-stochastic manifold keeps the width and restores the identity mapping this gave up
SOTA-138 v2	Train sparse attention natively with a learned top-k indexer, warmed up under dense attention	DeepSeek-AI (2025), LIT-142 — a small FP8 indexer scores past tokens and only the top-k enter attention; initialised for 2.1B tokens with dense attention on and the rest frozen; V4 repeats the pattern with a 1T-token dense warm-up.	Active
SOTA-139	Extend the context length in stages during pretraining rather than training at the target length from the start	DeepSeek-AI (2026), LIT-139 — 4K, then 16K, 64K and 1M over 32–33T tokens, with sparse attention introduced at the 64K stage; Kimi K3 (LIT-131) reaches 1M the same way.	Active
SOTA-140 v5	Use a warmup-stable-decay schedule: hold the learning rate, then decay it sharply over the final 10–20% of tokens	Hu et al. (2024), LIT-144, with the controlled comparison in LIT-145 — matches cosine at equal tuning, leaves the token budget open, makes every stable-stage checkpoint a usable branch point; contested at frontier scale by LIT-131, which retuned per schedule and chose cosine.	Active
SOTA-141 v3	Decay the learning rate linearly all the way to zero	Bergsma et al. (2025), LIT-147 — with the peak tuned, linear decay to zero beats the customary decay to 10% and other shapes at compute-optimal budgets, more so past them; Proposed because contemporary production runs in the record still decay to a floor.	Proposed
SOTA-142 v3	Set the peak learning rate by a power law in tokens so it transfers across batch size and training length	Shen et al. (2024), LIT-146 — under WSD the optimal learning rate is a power law in tokens and batch size; the Power scheduler applies it, and with µP the same setting carries across model sizes; adopted independently by Falcon-H1-Tiny.	Active
SOTA-143 v2	Parameterize the model with µP and tune hyperparameters on a narrow proxy, transferring them across width	Yang and Hu (2022), LIT-148 — under the Maximal Update Parametrization the optimal learning rate and friends are stable across width, so tune small and transfer zero-shot; GPT-3 6.7B beaten at 7% of its pretraining cost in tuning. Used by Falcon-H1, MiniCPM and the Power scheduler.	Active
SOTA-144 v2	Extend µP’s transfer to depth with CompleteP so one sweep serves deeper models too	Dey et al. (2025), LIT-150 — the depth exponent α = 1 transfers the optimal learning rate across depth and keeps deep layers learning; 11.8% fewer FLOPs than µP at optimal shapes, 34.4% at 179 layers. One group so far.	Proposed
SOTA-145 v4	Estimate the RL baseline from a group of samples for the same prompt instead of training a critic	Shao et al. (2024), LIT-127 — Group Relative Policy Optimization: PPO with the value model dropped and the baseline taken from the scores of several outputs sampled for the same prompt, which removes a model-sized chunk of the RL memory footprint and is what every later reasoning recipe in this record actually runs.	Active
SOTA-146	Correct the GRPO objective before running it — the published form has three independently identified defects	Liu et al. (2025), LIT-167; Yu et al. (2025), LIT-168; Zheng et al. (2025), LIT-180 — three groups, three defects in GRPO’s objective, three published fixes, none of them the same fix. “We ran GRPO” does not say which of these you ran.	Proposed
SOTA-147 v2	Compress the KV cache into one shared latent vector instead of sharing key and value heads	DeepSeek-AI (2024), LIT-174 — Multi-head Latent Attention projects keys and values into a single low-rank latent and caches that, cutting the KV cache 93.3% and raising maximum generation throughput 5.76× against the same team’s dense 67B. Every DeepSeek model since is built on it.	Active
SOTA-148 v3	Balance mixture-of-experts load with a bias on the routing scores, not an auxiliary loss	Wang et al. (2024), LIT-171 — add a per-expert bias to the routing scores before the top-K decision and update it from that expert’s recent load, so balancing changes which experts are chosen without adding a gradient to the loss. Better balance and better quality than an auxiliary-loss control, and what DeepSeek-V3 runs at 671B.	Proposed
SOTA-149	Build the sparse layers from many small experts plus an always-on shared one, not a few large ones	Dai et al. (2024), LIT-170 — split into mN smaller experts and activate mK of them so the router chooses from a far larger combination space at the same compute, and isolate a few always-on shared experts to hold the common knowledge every routed expert would otherwise learn separately.	Active
SOTA-150 v3	Make the feed-forward layers a sparse mixture of experts once the model is large enough to be compute-bound	Shazeer et al. (2017), LIT-188 — route each token to a few of many feed-forward experts instead of running one dense feed-forward for every token, so total parameters and per-token compute stop being the same number. Measured against dense by LIT-170 seven years later: DeepSeekMoE 16B matches LLaMA2 7B at roughly 40% of the compute, 145B approaches the same team’s dense 67B at 28.5%.	Active
SOTA-151 v4	Extend a trained model’s context by rescaling RoPE, not by fine-tuning at the longer length	Chen et al. (2023) and Peng et al. (2023) — RoPE does not extrapolate, and fine-tuning at the longer length barely helps: more than 10000 batches moved LLaMA’s effective window from 2048 to 2560. Rescaling the position indices so they land back in the trained range reaches 32× that in under 1000 steps. YaRN rescales per wavelength rather than uniformly and gets to 128k with 10× fewer tokens.	Active
SOTA-152	Pack training documents by best fit instead of concatenating and splitting	Ding et al. (2024), LIT-203 — concatenate-then-split truncates documents that would have fit, and the model then learns to continue text whose beginning it never saw. Treating the grouping as bin packing removes those truncations at the same efficiency, no padding added: +4.7% reading comprehension, +16.8% context following, +9.2% program synthesis, and up to 58.3% less closed-domain hallucination.	Proposed
SOTA-153 v4	Drop positional encoding from the global-attention layers of a hybrid and let the cheap local layers carry position	Yang et al. (2025), Puvvada et al. (2025) and the Kimi Team (2025) — in a model that already interleaves full attention with a cheap local mixer, the full-attention layers do not need a positional encoding: the local layers carry position and recency, and the global layers do retrieval better without one. Three groups, three different local mechanisms, the same one-global-per-three-local layout. The payoff is that extending the context needs no RoPE rescaling, because there is no positional parameter left to rescale.	Active
SOTA-154 v4	Fine-tune with evolution strategies instead of policy-gradient reinforcement learning	Qiu et al. (2025), LIT-211 — evolution strategies over the full parameter space of a billion-scale LLM, which the field had assumed impossible. +36.4% over base on average against GRPO’s +21.3% and PPO’s +17.9%, with ES on one fixed hyperparameter set while RL got a sweep per experiment. Active since Gan and Isola replicated the comparison on six further tasks with the RL arm tuned harder; contested, because Gu et al. argue direct ES at LLM scale does not work — on models at 1.5B and below.	Active
SOTA-155	Train across poorly connected islands: many inner steps per worker, an outer momentum optimizer over the deltas, and a streamed synchronisation	Douillard et al. (2023, 2025), LIT-212 and LIT-214 — federated averaging with the constants pushed hard: many inner AdamW steps per worker, Nesterov momentum as the outer optimizer over the accumulated deltas, synchronised rarely and in streamed subsets. 500× less communication at matched quality, then two more orders off the peak. Filed Proposed: one group, and no frontier report trains this way.	Proposed
SOTA-156 v2	Train without a learning-rate schedule: average the iterates so no stopping time need be fixed	Defazio et al. (2024), LIT-213 — scheduling and iterate averaging are one mechanism, and the averaged form needs no stopping time T, no schedule shape and no hyperparameter beyond what AdamW already has. Filed Proposed: the evidence is a third-party competitive benchmark at workloads far below the scale where this record’s schedule question is open.	Proposed
SOTA-157 v3	Train the language model as a masked diffusion model rather than autoregressively	Nie et al. (2025), LIT-217 — hold the paradigm fixed (pretrain, then SFT) and swap only the factorization: a forward masking process and a reverse process predicting masked tokens, optimizing a likelihood lower bound. Competitive with LLaMA3 8B on in-context learning; past GPT-4o on reversal poem completion. Filed Proposed — one group, one model, 8B.	Proposed
SOTA-158	Bound the activation’s output range when training in low precision	Jiang et al. (2026), LIT-200 — SwiGLU approximates x² for large positive inputs, which is where its expressive capacity comes from and what enlarges the output range and exacerbates outliers. Two groups independently concluded that is intolerable in low precision. What they agree on is the constraint; the functions they chose differ.	Proposed
SOTA-159	Combine µP with unit scaling so the hyperparameters decouple and FP8 needs no loss scaling	Blake et al. (2024), LIT-149 — µP makes activation scale independent of model size; unit scaling makes activations, weights and gradients start at a scale of one. Together the model is size-independent and well-scaled from the first step, the hyperparameters become close to independent so a one-dimensional sweep replaces a joint search, and FP8 works without loss-scaling tricks.	Proposed
SOTA-160	Treat the pretraining token budget and the post-training quantization plan as one decision, not two	Kumar et al. (2024), LIT-186 — degradation from post-training quantization increases with how much data the model was trained on, to the point where additional pretraining data becomes actively harmful if the model will be quantized afterwards. Fit over 465 runs into one precision-aware scaling law covering both pre- and post-training quantization.	Proposed
SOTA-161 v4	Keep the attention output in FP32 during training, because flash attention’s rounding bias compounds instead of cancelling	Qiu et al. (2025), LIT-198 — putting flash attention in BF16 while the FFN goes to FP8 sometimes explodes, and the cause is two things at once: attention produces low-rank updates repeated across steps and tokens, and low-precision addition rounds with a bias. The biased error rides the repeated update and compounds into a systematic gradient bias instead of averaging out.	Active
SOTA-162 v2	Train with auxiliary multi-token-prediction heads alongside next-token prediction	Gloeckle et al. (2024), LIT-163 — predict the next n tokens through n independent heads on a shared trunk, as an auxiliary task rather than a replacement. No training-time overhead, the benefit grows with model size and survives multi-epoch training, and the extra heads are a draft model you already trained.	Active
SOTA-163 v2	Quantize with block-scaled microscaling formats rather than one scale per tensor	Rouhani et al. (2023), LIT-197 — an MX block is 32 elements sharing one E8M0 scale, so applying the scale is an exponent adjustment rather than a multiply. Shrinking the scope of the scale contains outliers locally instead of letting a few large values cost every small one its precision. 6-bit MX trains large transformers to FP32 accuracy with no recipe change.	Active
SOTA-164 v2	Deduplicate the pretraining corpus at both substring and document granularity before training on it	Lee et al. (2021), LIT-202 — a single 61-word sentence appears thousands of times in C4. Deduplicating cuts verbatim emission about tenfold, reaches equal or better accuracy in fewer steps, and removes train-test contamination from standard validation sets. Two granularities, because a repeated boilerplate paragraph is not a duplicated document.	Active
SOTA-165	Precondition the gradient with matrices rather than entrywise scaling	Wen et al. (2025), LIT-156 — under per-optimizer tuning across ten optimizers and four scales, every fastest one multiplies gradients by matrices rather than scaling entrywise. A structural finding that survives a fair comparison, and the class the record’s Muon practice is one member of. The advantage is 1.4× at 0.1B and 1.1× at 1.2B.	Active
SOTA-166 v2	Set the pretraining data proportions by fitting a mixing law on small runs, not by argument	Ye et al. (2024), LIT-201 — model performance is quantitatively predictable as a function of the mixture proportions, so fitting the law on a sample of mixtures reads off performance on mixtures never trained. Nested with the existing scaling laws, small runs predict a large model. Matched the default mixture trained for 48% more steps.	Proposed
SOTA-167	Build linear-attention layers from the state-space view: expressive discretisation, complex-valued state, and a MIMO readout	Lahoti et al. (2026), LIT-165 — three changes derived from the SSM view: a more expressive recurrence from the discretisation, a complex-valued state update for state tracking, and a MIMO formulation that adds quality without decode latency. +1.8 downstream at 1.5B over Gated DeltaNet, and Mamba-2’s perplexity at half the state size.	Proposed
SOTA-168	Run Adam in Shampoo’s eigenbasis (SOAP) instead of Shampoo itself	Vyas et al. (2024), LIT-157 — Shampoo at the 1/2 power is Adafactor in the eigenbasis of Shampoo’s preconditioner, so run the better optimizer in that basis. SOAP is Adam there, adding exactly one hyperparameter over Adam. Over 40% fewer iterations and 35% less wall-clock than AdamW at 360M–660M in the large-batch regime.	Proposed
SOTA-169	Widen the residual stream into several streams and constrain the mixing between them	Zhu et al. (2024) and three successors, LIT-141 / LIT-140 / LIT-151 / LIT-181 — replace the single residual stream with n parallel streams and learn the mixing. All four agree the width helps and that the mixing needs a constraint; they disagree about which one, and that disagreement is the two practices extending this rather than this itself.	Proposed
SOTA-170 v2	Choose filtering aggressiveness by the token horizon: at long horizons rephrase what a filter would discard	Su et al. (2024), LIT-178 — aggressive model-based filtering wins at 1T tokens and loses at 15T, because removing 90% of the data optimises mean quality and mean quality is not what binds when the corpus runs out. Ensemble the classifiers, rephrase text synthetically instead of discarding it, and lean less on heuristics.	Active
SOTA-171 v2	Repeat a data-constrained corpus for up to about four epochs; past that, added compute stops paying	Muennighoff et al. (2023), LIT-166 — with constrained data at fixed compute, up to four epochs of repeated data changes the loss negligibly against having that much unique data. Past that point the value of adding compute decays toward zero. 400 training runs, up to 900B tokens and 9B parameters, with a scaling law that prices repeated tokens and excess parameters.	Active
SOTA-172 v2	Build synthetic pretraining data by editing human text at the token level, not by generating from scratch	Zhu et al. (2024), LIT-205 — pretraining across proportions of synthetic data shows a negative correlation between the proportion and performance, caused by distributional shift and over-concentration of n-gram features. The remedy inverts the usual move: edit human data at the token level, so the result stays anchored to a real distribution and the test error is provably bounded.	Proposed
SOTA-173 v2	Augment the objective to make multi-epoch pretraining productive on a fixed corpus	Chen et al. (2026), LIT-175 — standard autoregressive pretraining overfits severely on a fixed corpus, reaching its optimum early and deteriorating continuously. Three orthogonal augmentation families each delay that and lower validation loss; combining them lowers the minimum further. The claim is hundreds of productive epochs on the same data.	Proposed
SOTA-174	Train autoregressive models with fill-in-the-middle by default: it is a data transformation, and it is free	Bavarian et al. (2022), LIT-124 — cut a document into prefix, middle and suffix, move the middle to the end with sentinel tokens, and the model learns to infill. Transforming a large fraction of the training data does not harm left-to-right perplexity or sampling quality across a wide range of scales, so infilling is an added capability rather than a trade.	Active
SOTA-175	Mask whole syntactic units for code fill-in-the-middle, not random character spans	Gong et al. (2025), LIT-126 — standard FIM masks random character spans, which produces training examples that rarely correspond to an edit anyone makes. AST-FIM masks whole syntactic units at scale, so the middle span is a coherent structure. Up to 5 points over random-character FIM at 1B and 8B, and most useful on real editing.	Proposed
SOTA-176	Hybridise inside the layer: long-term slots, a sliding window, and one softmax over both, with the window size as the only dial	Du et al. (2025), LIT-176 — long-term context in key-value slots updated by a linear RNN, short-term context from a sliding window, and a single softmax attending over both together, so the long/short weighting is per-token, per-head and content-dependent with no fusion parameters. Every layer stays structurally identical and one hyperparameter slides from purely linear to full attention.	Proposed
SOTA-177 v2	Decouple the erase address from the write address in a delta-rule recurrence	Li et al. (2026), LIT-177 — the delta rule corrects what is stored at the current write address before writing there, so stale information at a different address can only decay passively. EDA runs a targeted erase along a learned erase direction first, then the ordinary corrective write. Best at both 2.5B dense and 25B-A2.8B MoE, and the gain persists after 80B tokens of long-context midtraining.	Proposed
SOTA-178	Give the recurrence vector-valued gating and in-context learning rates so it can track state a softmax layer provably cannot	Peng et al. (2025), LIT-173 — a generalised delta rule with vector-valued gating and in-context learning rates, at constant memory and constant time per token. A 2.9B model sets the 3B state of the art on multilingual tasks on dramatically fewer training tokens. The expressivity claim is the sharp one: it can recognise all regular languages, which under standard conjectures exceeds what a softmax stack in TC⁰ can do.	Proposed
SOTA-179	Truncate the rotary encoding’s low frequencies rather than rescaling its base	Barbero et al. (2024), LIT-210 — RoPE’s high frequencies build positional attention heads and its low frequencies carry semantics that provably cannot stay robust over long context. Keeping a fraction p of the frequencies holds performance and at 2B improves it; p=1 is RoPE and p=0 is NoPE, so the practice is a dial between two things the record already holds.	Proposed
SOTA-180	Keep the routed experts’ weights in a down-projected latent space and spend the saved bandwidth on more of them	Elango et al. (2026), LIT-196 — down-project the token before the routed experts and keep their weights in that latent space, dispatching and aggregating there too, while routing and the shared experts stay at full width. Communication volume and weight-loading bandwidth both fall by d/d_l, and the saved budget buys more experts and more active per token.	Active
SOTA-181	Shard the sequence across devices in a ring and overlap the key-value exchange with the attention it feeds	Liu et al. (2023), LIT-206 — compute attention and the feedforward blockwise, distribute the blocks across devices in a ring, and pass each key/value block to the neighbour while computing on the one you hold. The communication is fully overlapped with the computation, so sequences scale with device count — exactly, with no approximation and no added overhead.	Active
SOTA-182 v2	Compute the normalization statistic without centering (RMSNorm)	Zhang and Sennrich (2019), LIT-023 — ARXIV-1910.07467. Drop the mean subtraction from layer normalization and rescale by the root mean square alone.	Active
SOTA-183	Generate the harmlessness preference labels with the model itself, against a written set of principles	Bai et al. (2022), LIT-082 — ARXIV-2212.08073. Replace the human harmlessness comparisons with model-generated ones: the model critiques and revises its own responses against an explicit list of principles, and the preference model trains on that.	Active
SOTA-184 v2	Adapt a pretrained model by training a low-rank update to each weight matrix, not the matrix itself	Hu et al. (2021), LIT-046 — ARXIV-2106.09685. Freeze the pretrained weights and learn a rank-r product BA beside each one, so the trainable parameter count and the optimizer state fall by orders of magnitude and the adapter folds back into the weight at inference.	Active
SOTA-185	Quantize weights after training by compensating each rounding error into the columns not yet quantized	Frantar et al. (2022), LIT-081 — ARXIV-2210.17323. Round one column at a time and push the resulting error into the remaining columns using approximate second-order information, instead of rounding every weight independently to the nearest level.	Active
SOTA-186	Bootstrap a large annotation set with the model you are training, staging the automation as it improves		Active
SOTA-187 v5	Train the generative model in a learned compressed latent, not at full resolution		Active
SOTA-188 v5	Parametrize the network so its prediction target has unit variance at every noise level, and sample training noise from a log-normal		Active
SOTA-189	Set the checkpoint interval from the job’s measured time-to-failure, which shrinks as the job grows	CoreWeave (2025), LIT-222. Expected work lost to a failure is half the inter-checkpoint interval, and mean time to failure falls linearly with GPU count — so the interval belongs to the job’s size and the cluster’s failure rate, not to the write cost alone.	Proposed
SOTA-190 v3	Increase depth before any other dimension when scaling a transformer	Tay et al. (2021), LIT-052 — the DeepNarrow strategy. Small 16L matches T5-Base downstream at 60% of the parameters, 63% of the FLOPs and 40% faster; the limit is parallelism rather than quality.	Proposed
SOTA-191 v2	Consider removing LayerNorm’s learnable gain and bias rather than tuning them	Xu et al. (2019), LIT-025 — the bias and gain increase overfitting risk and “do not work in most cases”; LayerNorm-simple, with both removed, beats LayerNorm on four datasets and reaches state of the art on En-Vi.	Proposed
SOTA-192 v9	Normalize the queries and keys before the attention dot product		Active
SOTA-193	Lower Adam’s second-moment decay when the loss spikes, before reaching for the learning rate		Proposed
SOTA-194	When an objective arrives with its own dataset, state the composition before crediting the objective		Active
SOTA-195 v2	Predict v rather than the noise when the model will be evaluated at low signal-to-noise		Active
SOTA-196 v2	Report zero-shot and in-distribution performance separately; one hyperparameter can move them in opposite directions		Active
SOTA-197 v2	Account for test-set proximity to the training data when evaluating, and state your contamination exposure		Active
SOTA-198 v2	Measure the gradient noise scale instead of sweeping batch size, and expect it to grow during the run		Active
SOTA-199	Regularize a narrow fine-tune against the pre-fine-tuning model’s own samples		Active
SOTA-200 v3	Check whether an emergent capability is a metric artefact before believing it		Active
SOTA-201	Give each pretrained component its own learning rate when fine-tuning a composite model		Active
SOTA-202 v2	Clamp the prediction to the training range at every step when sampling from a model’s own output		Active
SOTA-203 v3	Sample a diffusion model with a higher-order ODE solver on the weights you already trained	Lu et al. (2022), LIT-076 — the diffusion ODE is semi-linear, so solve the linear part exactly and approximate only the neural integral. 10-20 function evaluations, no retraining. Rests on LIT-038, which established that the sampler is not fixed by the training objective.	Active
SOTA-204 v2	Anneal a discretisation from coarse to fine over training rather than fixing it	Song et al. (2023), LIT-093. Where a training loss approximates a continuous target through a step count, that count is a bias/variance dial: few steps give a biased but low-variance target early, many steps a faithful but noisy one later.	Proposed
SOTA-205 v2	Replace a large coordinate network with a compact explicit structure and a small decoder	Four independent groups, three structures. Where a field is queried pointwise and the signal is spatially sparse, the capacity belongs in an addressable structure that training optimises directly, not in a network evaluated per point.	Active
SOTA-206 v3	Keep a multi-step sampling option in a few-step generative model	Song et al. (2023), LIT-093. A one-step model that cannot spend more compute has no quality dial: whatever it produces is what you get. Consistency models trade compute for quality at inference without retraining, and that property is worth preserving by design.	Active
SOTA-207	Use the deterministic sampler when the noise input has to mean something	Song et al. (2020), LIT-038. A stochastic sampler injects fresh noise at every step, so nothing about the starting point survives to the output. Setting the family’s stochasticity to zero makes the initial noise a latent code you can interpolate in and invert to.	Active
SOTA-208	Factorize long-sequence attention into a local window and a second head that escapes it	Child et al. (2019), LIT-225 — ARXIV-1904.10509. Superseded: the field went to exact attention made fast (SOTA-086, SOTA-087) and then to sparsity that is LEARNED rather than fixed (SOTA-138). What survives is the connectivity requirement — a window alone is not the design — which every hybrid the record recommends is an instance of.	Superseded — by SOTA-138
SOTA-209	Initialize a mixture-of-experts model from a dense checkpoint rather than training it from scratch	Komatsuzaki et al. (2022), LIT-227 — seed the experts from a dense checkpoint you already paid for. Upcycled T5 and ViT models beat their dense counterparts at ~50% of the dense pretraining sunk cost, and beat sparse models trained from scratch on 100% of it: same architecture, same compute, different initialization.	Proposed
SOTA-210 v3	Report pass@k as well as pass@1 after post-training: reinforcement learning raises one and lowers the other	Ba et al. and Hayes et al. (2026), LIT-230 and LIT-234. GRPO finishes below its own base model on pass@16 and pass@32 in 15 of 18 comparisons while improving pass@1; across Qwen2.5, Qwen3 and published RL checkpoints up to 32B the base model overtakes the RL checkpoint at large k. A post-training result reported at pass@1 alone cannot distinguish a model that learned something from one that stopped trying anything else.	Active
SOTA-211	Spend one fitness evaluation per evolution-strategies direction, not an antithetic pair	Kaya and Hashemi (2026), LIT-232, and Ba et al. (2026), LIT-230 — the antithetic pair’s variance reduction depends on both evaluations sharing randomness, which autoregressive regeneration breaks. Replace the pair with a leave-one-out baseline computed from the population mean the score centering already needs: same expected update, one evaluation per direction, twice as many directions per budget, and half the estimator MSE in transformer blocks.	Proposed
SOTA-212	Post-train by scoring many random weight perturbations in one parallel pass and majority-voting the best of them	Qiu et al. introduced it (LIT-211); Gan and Isola (2026), LIT-233 — ARXIV-2603.12228 — measured this form of it. Sample N Gaussian weight perturbations, score them on a few hundred held-out examples, keep the top K and majority-vote at inference. At equal training FLOPs it matches or beats PPO, GRPO and ES across seven tasks at 0.5B–8B, in one parallel step instead of hundreds of sequential ones — 3.2 minutes for OLMo3-7B on Countdown across 200 GH200s. Filed Proposed: one group, K forward passes per answer, and discrete answers only.	Proposed
SOTA-213	Control evolution-strategies drift with a larger population or an anchor penalty, not by stopping training early	Schweighofer et al. and Liang et al. (2026), LIT-238 and LIT-236, on the scaling Hoy et al. derived in LIT-235 — ES drift is a random walk whose size falls with population size, so raising the population from 30 to 128 halves the update norm and monotonically reduces prior-task degradation. Anchored Weight Decay buys the same reduction at population 30 for 1-2% runtime. Do not stop early instead: the prior-task dip is often transient and recovers by the end of training.	Proposed
SOTA-214	Pair any gradient compressor with error feedback, and correct the momentum it is applied under	Compression is lossy and the loss accumulates. Every scheme in this record that survives high compression ratios keeps the residual locally and adds it back to the next step, and every report of a compressor “not converging” that the record holds is a report of that step being skipped: rank-4 PowerSGD fails on Cifar10 without it, and gradient sparsification at 99.9% degrades badly without the momentum-buffer version of it.	Active
SOTA-215	Freeze Adam’s variance term after a warmup before compressing its update, and never compress the update directly	Error-compensated compression assumes the residual it carries forward still means the same thing next step. Adam’s update divides by a running variance, which is a non-linearity the correction does not survive — compressing it directly corrupts the error cancellation and badly harms convergence. The workaround is structural rather than a tuning fix: Adam’s variance stabilises early, so freeze it after a warmup and the update becomes linear enough to compress.	Proposed
SOTA-216	Switch from minibatch SGD to local SGD at the first learning-rate decay, rather than choosing between them	The two options are usually posed as a choice: minibatch SGD generalises and communicates every step, local SGD communicates rarely and generalises worse. Post-local SGD is the observation that they are wanted at different times — run minibatch SGD until the first learning-rate decay, then switch to local SGD with H around 16 to 32 for the rest. The early phase is where the gradient noise is doing work; the late phase is where communication is pure overhead.	Proposed
SOTA-217 v4	Align the hidden-unit permutation before averaging weights from separately trained networks	A neural network’s hidden units can be permuted without changing the function, so two networks trained from different seeds sit in different corners of the same symmetry orbit. Averaging their weights directly averages across that mismatch and traverses a loss barrier that is mostly an artefact of labelling. Match the units first — by weight matching, which needs no data and takes seconds to minutes — and for wide enough networks the barrier is gone. At standard width, and on ImageNet, it shrinks but does not vanish.	Proposed
SOTA-218 v2	Retune the learning rate, momentum and schedule at every batch size you compare, never by a scaling heuristic	Linear scaling and square-root scaling are not batch-size rules; they are guesses about where the optimum moved, and they hold over a narrower range than the comparisons people use them for. A curve of steps-to-target against batch size drawn with transferred metaparameters is not a measurement of batch size at all — it is a comparison between one tuned configuration and several untuned ones, and the point where it bends is the point where the heuristic failed.	Active
SOTA-219	Leave the inner optimizer state unsynchronised in local-update training	When workers train independently for hundreds of steps and synchronise by exchanging deltas, the Adam moments each worker accumulated are local state and should stay local. Synchronising them triples the communication for negligible quality gain — which means most of what the scheme saves is not saved by communicating rarely, but by communicating less each time.	Active
SOTA-220	Aggregate worker gradients coordinate-wise by median when any worker may be faulty	Averaging has a breakdown point of zero: one worker returning an arbitrary vector can move the mean anywhere, and adding workers does not help, because it is the worker’s magnitude and not its share of the population that does the moving. The coordinate-wise median is the robust aggregator that needs no estimate of how many workers are faulty; the trimmed mean has better rates and needs one.	Proposed
SOTA-221 v2	To train past the batch size where your optimizer stalls, change the optimizer’s conditioning rather than the scaling rule	You et al. (2019), LIT-265. When a batch size stops reaching your accuracy target, the reflex is to look for a better learning-rate scaling rule. LIT-058 is 168,160 training runs of evidence that there is no such rule to find. What moved the wall instead was the optimizer: normalizing each layer’s update by the ratio of its parameter norm to its update norm, on top of per-coordinate adaptivity, carried BERT from 16K to 32K where a grid-searched AdamW could not follow. This does not license skipping the retuning in SOTA-218 — the two say different things about different activities, and THEORY-012 is why.	Active
SOTA-222 v2	Overlap the synchronisation with one or two local steps instead of communicating every step	Wang et al. (2020), LIT-323. Local SGD is usually argued for as a way to communicate rarely, which costs accuracy at large intervals. The cheaper reading is that it does not take a large interval to win: at tau = 2 the communication-to-computation ratio falls from 34.6% to 1.5% with convergence per iteration roughly unchanged. The interval is not buying infrequency, it is buying somewhere to hide the exchange.	Proposed
SOTA-223	Do not keep the gossip topology static: sample a fresh random neighbourhood every round	de Vos et al. (2023), LIT-315. Decentralized training has spent years choosing good fixed graphs. Epidemic Learning does not choose one: each round, every node sends its update to s peers picked at random. The transient-iteration count falls from the best-known O(n^3) to O(n^3/s^2), and at 96 nodes on CIFAR-10 it converges 1.7x faster and 2.2% more accurately than a static 7-regular graph at equal communication volume.	Proposed
SOTA-224	Ramp the sparsity ratio up over the first few epochs rather than starting at the target	Lin et al. (2017), LIT-056. Gradient sparsification delays the coordinates it drops; error feedback returns them, but it returns them late. Early in training the gradients are large and the trajectory is still choosing where to go, so a delay there costs more than the same delay later. Start at a low sparsity — the paper ramps 75%, 93.75%, 98.4375%, 99.6%, 99.9% over the first few epochs — and reach the target once the run has settled.	Active
SOTA-225	Drop the parameter server for gossip once the network is the bottleneck, and measure where that is rather than inheriting a threshold	Lian et al. (2017), LIT-302. The question is whether a decentralized algorithm can ever beat a centralized one, and the answer is that it can — not by converging better, since the rates match, but by having no node that every other node talks to. The advantage appears where the central node’s link is the constraint, and the paper establishes that by sweeping bandwidth and latency rather than by naming a threshold.	Active
SOTA-226	If the gossip topology is static, make it a directed exponential graph, not a ring	Assran et al. (2018), LIT-254. Having chosen gossip over a parameter server, the next choice is the graph, and it is not a detail: a ring’s consensus rate degrades with the node count badly enough that the speedup regime recedes out of reach. A directed exponential graph — each node sends to peers at distance 1, 2, 4, 8, … in rotation — reaches exact consensus in log2(n) rounds with constant out-degree per round, and spreads the load evenly.	Active
SOTA-227 v2	Decode with a draft model and an accept-reject rule, which is exactly lossless	Autoregressive decoding runs one serial model pass per token and each pass is bounded by streaming the weights, so the arithmetic units idle. Have a cheap draft model guess the next gamma tokens, score all gamma + 1 positions in one target pass, and keep a prefix under an accept-reject rule built so the output distribution is the target’s exactly. 2x-3x at 11B and 2-2.5x at 70B, with no retraining, no architecture change and nothing traded away.	Active
SOTA-228	Pick the inference partitioning from where the bottleneck is, and expect it to move between prefill and decode	Pope et al. (2022), LIT-110. Serving is not one workload: prefill parallelises over the prompt, decode is serial over the output, and the two are bounded by different things. Worse, the binding constraint moves — at small batch and short context it is loading the weights, and at large batch and long context it is loading the KV cache, which on a 500B model at batch 512 and 2048 tokens reaches 3 TB, three times the parameters. Choose the layout from a cost model of the regime you are in, and re-choose it when the regime changes.	Active
SOTA-229	Scale the draft model’s training data, once nothing constrains it to predict the target’s features	Draft models had stopped improving with more training data, and LIT-185 identifies why: EAGLE trained its draft to predict the target’s hidden features and read tokens off them, which supplies multi-step training signal and caps expressiveness. Remove the constraint — predict tokens directly, fuse features from several depths, and simulate the multi-step draft during training — and the speedup rises with the data instead of flattening. Up to 6.5x, about 1.4x over the previous version, on roughly 8x the data.	Proposed
SOTA-230	Quantize the frozen base to 4-bit and keep the adapters in 16-bit	Dettmers et al. (2023), LIT-378 — ARXIV-2305.14314. Store the frozen base weights in 4-bit NormalFloat and dequantize to BFloat16 for every matrix multiply, training only 16-bit LoRA adapters. Fine-tuning a 65B model falls from >780GB to <48GB with no measured loss against a 16-bit fully fine-tuned baseline, because the arithmetic never happens in 4 bits.	Active
SOTA-231	Put an adapter on every linear layer, not just the attention query and value projections	Dettmers et al. (2023), LIT-378 — ARXIV-2305.14314. Adapters on the query and value projections alone — the default inherited from the LoRA paper — do not reach full fine-tuning on large base models. The number of adapted matrices is the hyperparameter that decides it; the projection rank r, which is the one people tune, does not affect performance across the paper’s sweep.	Active
SOTA-232	Convert an autoregressive model into a diffusion model by continual pretraining rather than training one from scratch	Gong et al. (2024), LIT-381 — ARXIV-2410.17891. Continual pretraining on under 200B tokens turns an existing autoregressive model into a diffusion language model — GPT2 and LLaMA across 127M-7B — competitive with the AR original. The reason to want one is infilling without prompt re-ordering, not perplexity.	Proposed
SOTA-233	Train self-correction with multi-turn online RL on the model’s own traces, rather than prompting for it	Kumar et al. (2024), LIT-383 — ARXIV-2409.12917. Self-correction is a capability to be trained, not a behaviour to be requested. Prompting a model to revise its own reasoning does not reliably help and can hurt; supervised fine-tuning on correction traces fails by distribution mismatch or behaviour collapse; multi-turn online RL on self-generated traces reaches +15.6% on MATH and +9.1% on HumanEval.	Active
SOTA-234	Train in the target low-bit format from scratch rather than quantizing a finished model	Ma et al. (2024), LIT-380 — ARXIV-2402.17764. Decide the serving format before training and train in it, rather than training in FP16 and compressing afterwards. Ternary weights trained from scratch match an FP16 model of equal size and token budget from 3B upward — but the compute argument assumes hardware built for the format, and only the memory saving is measured on machines that exist.	Proposed
SOTA-235 v3	Update the weights at inference on the test instance when the task is structurally novel	Akyürek et al. (2024), LIT-379 — ARXIV-2411.07279. Build a loss from the test instance’s own in-context examples, take gradient steps at inference, then discard the update. Up to 6x a fine-tuned baseline on ARC-AGI (53.0% at 8B) and +7.3 points on BIG-Bench Hard at 10-shot. The examples are worth several times more as gradient than as context.	Active
SOTA-236	Predict scene geometry directly instead of solving for cameras first and triangulating	Wang et al. (2023), LIT-385 — ARXIV-2312.14132; Wang et al. (2025), LIT-384 — ARXIV-2503.11651. Given uncalibrated, unposed images, regress the 3D structure and let camera parameters and pixel matches fall out of it, rather than estimating calibration and pose first so that triangulation becomes possible. The quantities the classical pipeline needs as inputs are by-products of this one.	Active
SOTA-237	Match densely without a keypoint detector, because the detector fails where matching is hardest	Sun et al. (2021), LIT-387 — ARXIV-2104.00680. Establish dense matches coarse-to-fine instead of detecting keypoints, describing them and matching the descriptors. The detector sets the pipeline’s ceiling, and it fails in low-texture regions — walls, floors, road — where it cannot emit repeatable points, so no downstream matcher can recover a correspondence that was never proposed.	Active
SOTA-238 v4	Set domain weights with a small proxy model under group DRO on excess loss, then transfer them	Xie et al. (2023), LIT-391 — ARXIV-2305.10429. Train a small proxy under group DRO to produce domain weights, then resample and train the real model with them. Optimise worst-case EXCESS loss against a reference model, not worst-case loss — the naive form upweights whichever domain is noisiest, because every domain has a different irreducible entropy.	Active
SOTA-239	Order training data by skill prerequisite: teach the prerequisite and the dependent skill costs less data	Chen et al. (2023), LIT-392 — ARXIV-2307.14430. Training data has a prerequisite structure, and it is exploitable: train on a prerequisite skill and the dependent skill is reachable with less data. Sample over skill mixtures online rather than fixing proportions in advance.	Proposed
SOTA-240 v4	Apply dropout where the model can memorize what it is shown, and not where it cannot	Srivastava et al. (2014), LIT-395 — dropout has a “sweet spot” in dataset size and reports both of its edges: no gain at all on data small enough to memorize through the noise, and a declining gain once the data is large enough that overfitting is not the problem. It costs 2-3x training time and it is not a drop-in — the paper’s own recipe pairs it with n/p units, 10-100x the learning rate, momentum 0.95-0.99 and a max-norm constraint. Introduced by Hinton et al. (2012), LIT-394.	Active
SOTA-241	Rank candidate data selections with a model-free distributional proxy before spending a training run	Xie et al. (2023), LIT-087 — KL reduction in a hashed n-gram space correlates with downstream performance across selection methods, including manual curation, which uses no n-grams at all. A selection can therefore be ranked against its alternatives before anything is trained on it.	Proposed
SOTA-242	Select pretraining data by matching a target distribution you can sample from, not by scoring it for quality	Xie et al. (2023), LIT-087 — data selection needs a description of what you want, not a criterion for what is good. Supply a small sample of the target text and resample the raw corpus to match it. The target is an artifact you can produce; a quality rubric is an argument you have to win.	Proposed
SOTA-243 v2	Discard the easy examples when data is abundant and the hard ones when it is scarce	Sorscher et al. (2022), LIT-399 — power-law scaling in dataset size is evidence of redundancy, and given a difficulty ranking it can be beaten toward exponential. But which end of the ranking to discard flips with how much data you started with, and the fraction to discard has to grow with the corpus.	Proposed
SOTA-244	Score example importance a few epochs into training, averaged over several initializations	Paul et al. (2021), LIT-397 — the importance ranking is available from a single early checkpoint, not only from a finished run. EL2N at epoch 20, averaged over ten initializations, prunes half of CIFAR-10 with accuracy slightly up, and the ranking transfers to other architectures.	Proposed
SOTA-245	State a relation in both orders in the training corpus if you want it usable in both directions	Grosse et al. (2023), LIT-401 — a training sequence influences a completion only when the phrase related to the prompt comes before the phrase related to the completion. Identical content in the reverse order scores barely above an unrelated baseline.	Proposed
SOTA-246	Find mislabelled training data by self-influence, not by training loss	Koh and Liang (2017), LIT-403, with LIT-400 — rank training points by their influence on their own loss and inspect from the top. Mislabelled examples are strong proponents of themselves, so they sort to the front; high training loss does not separate them nearly as well.	Proposed
SOTA-247	Add a multi-token word to a frozen model’s vocabulary from the model’s own detokenized representation of it	Kaplan et al. (2024), LIT-412 — a model already computes a whole-word representation for a word its tokenizer splits, so initialize the new embedding and unembedding from that rather than from an average of the word’s token embeddings. On Llama2-7B the frozen model then uses the new tokens while keeping its accuracy on existing ones (0.519 against 0.522 on WikiText-103), where mean-embedding initialization both fails to use them and degrades the model. The gain is largest where the tokenizer is worst: Arabic Wiki40B, 0.402 new-token accuracy against 0.117.	Proposed
SOTA-248	Stretch and clip the softmax so an attention head can output exact zeros	Bondarenko et al. (2023), LIT-414 — a head that wants to do nothing has to drive its softmax input to infinity to approximate exact zeros, and that is what creates the activation outliers that break INT8. Stretch the softmax to (γ, ζ) and clip back to (0,1) and zeros become reachable from a finite input. On BERT-base: W8A8 perplexity 1294 → 4.55, max infinity-norm 735 → 20, and the FP16 model gets slightly better rather than worse. Evidence is BERT-base, OPT-125M and ViT-S/16 — small and encoder-heavy.	Proposed
SOTA-249	Recompute activations from a sqrt(n) subset of checkpoints when activation memory is the binding constraint	Chen et al. (2016), LIT-004 — ARXIV-1604.06174. Store activations at O(sqrt(n)) checkpoints and recompute the rest during the backward pass. The price is one extra forward pass per minibatch; the measured case is a 1000-layer ResNet at 48G to 7G for 30% more wall clock.	Active
SOTA-250	Pretrain by predicting representations of masked regions, not their pixels	Assran et al. (2023), LIT-216 — ARXIV-2301.08243. Predict the representations of masked target blocks from a context block, with a learned target-encoder, instead of reconstructing pixels. No hand-crafted augmentations, and a ViT-H/14 reaches strong downstream performance in under 1200 GPU-hours — faster than a ViT-S/16 trained with iBOT.	Active
SOTA-251	Train at low resolution and raise it only during the decay phase of a warmup-stable-decay schedule	Assran et al. (2025), LIT-215 — ARXIV-2506.09985. Spend warmup and the constant phase on short, low-resolution clips; raise resolution and clip length only in the final decay. Up to 8x less pretraining compute than training at full resolution throughout, for the same end state.	Proposed
SOTA-252	Make the full-resolution path affordable instead of upsampling, when the shortcut is what breaks correctness	Abdal et al. (2023), LIT-113 — ARXIV-2311.17857. Prior 3D GANs render small and upsample in 2D because volume rendering is too slow at training resolution — but a 2D upsampler is multi-view inconsistent by construction. Efficient Gaussian rendering makes native 512x512 affordable, and the inconsistency leaves with the upsampler rather than being mitigated.	Active
SOTA-253	Add a new instance by reconstructing it, not by training on it	Sun et al. (2022), LIT-111 — ARXIV-2205.12257. Split what the system knows about a specific object from what it has learned in general. Build the object’s structure once by reconstruction; match against it with a network that never saw the object or its category. Adding an object is then a scan, not a training run.	Active
SOTA-254 v2	Train the language model as a masked diffusion model when the corpus is fixed and the compute is not	Prabhudesai et al. (2025), LIT-442 — masked diffusion keeps extracting signal from a repeated corpus for roughly 512 epochs where autoregressive training stops at 32. Below a critical compute threshold, given in closed form from the unique-token count, autoregressive training is better and by a wide margin; above it, diffusion is better. The recommendation is the condition, not the objective.	Proposed
SOTA-255 v2	Tune weight decay upward when pretraining over a repeated corpus, rather than inheriting the customary value	Kim et al. (2025), LIT-441 — with the corpus fixed and parameters past Chinchilla-optimal, the weight decay of 0.1 everyone inherits from GPT-3 is far too small: loss turns upward as parameters or epochs are added. Tuned jointly with learning rate and epoch count, the optimum is roughly 30x larger, and loss becomes monotone in parameter count with a steeper exponent than Chinchilla’s.	Proposed
SOTA-256	Judge a monotone scaling recipe by its asymptote, not by its loss at a compute budget	Kim et al. (2025), LIT-441 — when compute is not the binding constraint, a recipe’s loss at a chosen budget answers a question nobody asked. Fit the scaling law and compare the asymptote instead. The catch is that an asymptote is an extrapolation, and rankings at small scale can invert: the ensembling recipe wins on asymptote while losing at small member counts.	Proposed
SOTA-257 v2	Spend surplus compute on an ensemble of independently seeded models and distil it, rather than on one larger model	Kim et al. (2025), LIT-441 — under a fixed corpus, training K models that differ only in seed and averaging their logits reaches a lower loss asymptote than scaling one model to the same total parameter count. Two 300M models beat one 600M. Members want more epochs and less weight decay than a standalone model, and an 8-member ensemble distils into a 300M student retaining about 83% of the gain.	Proposed
SOTA-258	Scale batch size with the token budget, not with compute or model size	Zhang et al. (2024), LIT-445, and Bergsma et al. (2025), LIT-443 — critical batch size scales as D^0.46 and D^0.47 respectively, and is nearly flat in model size once the token budget is held fixed. Optimal batch size likewise goes as D^0.38. The compute exponent the field has used since Kaplan fits a projection: along the Chinchilla line C and D move together and only D is doing the work.	Active
SOTA-259	Hold Adam’s second-moment half-life fixed in tokens when the batch size changes, not beta_2	Marek et al. (2025), LIT-444 — beta_2 is a decay per optimizer step, and a step is not a fixed amount of data. Changing batch size from B to B' while holding beta_2 fixed changes the second moment’s averaging window in tokens by the same factor. Hold the half-life fixed instead: beta_2' = beta_2^(B'/B). beta_1’s default is fine.	Proposed
SOTA-260	Use the smallest batch size that still saturates the device, and do not gradient-accumulate to avoid it	Marek et al. (2025), LIT-444 — batch size one trains stably once Adam’s second-moment half-life is held fixed in tokens, matches or beats large batches per FLOP, and is far more robust to hyperparameter misspecification. Gradient accumulation costs memory for the accumulated gradient and forgoes optimizer steps, and is worth it only when multiple device replicas are bottlenecked on interconnect bandwidth.	Proposed
SOTA-261	Set AdamW’s weight decay by targeting a timescale that follows a power law in tokens-per-parameter	Bergsma et al. (2025), LIT-443 — under AdamW the weights are an EMA of updates with timescale 1/(eta*lambda), and the fraction of training it averages, tau = 1/(eta*lambda*S), is the quantity that scales. It follows a power law in tokens-per-parameter with exponent about -0.52 across three orders of magnitude of compute, so weight decay can be read off in advance rather than swept.	Proposed
SOTA-262 v2	Give each modality its own weights and let the streams attend jointly	Esser et al. (2024), LIT-449 — MMDiT gives the text and image streams separate projections and MLPs, then runs attention over the concatenated sequence so information flows both ways. It beats UViT and DiT at matched budget. Liang et al. (2024), LIT-483 — the same shape reached independently for autoregressive multi-modal LLMs, matching a dense baseline at 55.8% of the FLOPs, and ablated component by component: the feed-forward carries most of the benefit, the attention projections less, the layer norms none.	Active
SOTA-263 v4	Shift the timestep schedule when the resolution changes, because more pixels need more noise	Hoogeboom et al. (2023), LIT-660, and Chen (2023), LIT-659. A timestep is not a fixed amount of corruption. Destroying the signal in an image with more pixels takes more noise, so a schedule set at one resolution under-corrupts at a higher one. Shifting log-SNR down with resolution takes ImageNet 256 FID from 7.65 to 3.76 in pixel space. The direction is measured. The magnitude is tuned in every source.	Active
SOTA-264 v2	Fix the noise schedule’s endpoints against the bound and choose its shape to minimise the loss estimator’s variance	Kingma et al. (2021), LIT-446 — in continuous time the bound depends on the schedule only through its endpoints, so the shape is free. Spend it: optimise the endpoints against the VLB and the shape against the variance of the loss estimator. Same bound, faster optimisation, and a low-discrepancy sampler for the time variable cuts the variance further.	Proposed
SOTA-265 v3	Tune the stochastic sampler’s diffusion coefficient after training; it is not fixed by the forward process	Albergo, Boffi and Vanden-Eijnden (2023), LIT-645, state it and Ma et al. (2024), LIT-447, measure it — score-based diffusion conventionally takes the reverse SDE’s diffusion coefficient from the forward process, and presents the two as intrinsically tied. They are not: the coefficient affects neither the velocity nor the score, only the integration. So it is a sampler hyperparameter, tunable on a frozen model, and tuning it tightens the KL divergence to the target.	Active
SOTA-266 v5	Connect data and noise on a straight line, and sample the training timesteps from a logit-normal rather than uniformly	Esser et al. (2024), LIT-449, and Ma et al. (2024), LIT-447 — the straight-line path between data and noise beats the curved variance-preserving one at fixed architecture and compute, and the advantage is largest at few sampling steps. Movie Gen (LIT-626) confirms it for video at 5B. In SD3’s latent text-to-image sweep, the timestep distribution decides the ranking: uniform does not beat tuned ε-prediction and logit-normal does.	Active
SOTA-267 v3	Warm up on a subset of the entities before training on all of them, rather than holding one distribution fixed throughout	Zucchet et al. (2025), LIT-450 — the paper argues two quantities move in opposite directions: plateau length is governed by how often the most common entities appear, acquisition speed after the plateau by how often the least common ones do. So concentration buys an early exit from the plateau and costs the tail, and a warm-up that trains on a subset of the entities first, then on all of them, beat the best fixed distribution tried, in single-seed runs on synthetic biographies.	Proposed
SOTA-268	Do not transfer a data mixing ratio across model scales; the threshold where a domain starts being learned moves with size	Gu et al. (2025), LIT-451 — knowledge acquisition from a knowledge-dense dataset mixed into web text has thresholds, not a smooth curve. Below a critical model size, or a critical mixing ratio, the model memorises almost nothing however long it trains; past it, accuracy jumps to over 60%. The critical ratio follows a power law in model size, so a ratio chosen at one scale can be on the wrong side of a transition at another.	Proposed
SOTA-269	Keep a knowledge base outside the weights; parameters are an expensive and lossy place to memorise facts	Lu et al. (2024), LIT-452 — fact capacity is linear in model size and saturates in epochs, and the extrapolation is damning: all of Wikidata would want about 1000B non-embedding parameters trained for 100 epochs. Derivable facts cost full price unless they happen to share direction and structure. Borgeaud et al., LIT-060, is the record’s evidence that the alternative works at scale.	Active
SOTA-270	Do not read a smooth loss curve as evidence of smooth training; decompose it when the answer matters	Kangaslahti et al. (2025), LIT-455, with Cohen et al. LIT-453 and Kunin et al. LIT-454 — the curve time-averages oscillation, sums over differently-timed abrupt transitions, and reads flat while the weights still travel. Smoothness is what many breakthroughs look like added up, so it is not evidence about the training.	Active
SOTA-271	Train on a second modality even when the target is single-modality	Huh et al. (2024), LIT-458 — if representations converge on a modality-agnostic model of what generated the data, then data from any modality helps find it. The vision direction is already common practice; the language direction is not, and the paper cites a case where adding image data improved text performance.	Proposed
SOTA-272	Expect the loss to be non-monotone at the step size that trains fastest, and do not set the step size from a curvature bound	Cohen et al. (2021), LIT-461 — gradient descent raises the sharpness until its own step size cannot tolerate more, so a rule that anneals the step size to the measured curvature is chasing a number it is producing, and loses to the fixed step it forbids. Demonstrated full-batch.	Proposed
SOTA-273	Choose a pretraining context whose association with the input is neither too strong nor too weak, and mix contexts to get there	Zhai (2025), LIT-459 — a pretraining objective is an association between the input and a context variable, and the useful contexts are the ones whose association is neither too strong nor too weak: the singular values of the expectation operator should decay neither too fast nor too slowly. Contexts that individually miss the range can be mixed into one that does not.	Proposed
SOTA-274 v2	Before adopting a spectral optimizer, measure the stable rank of each block’s incoming activations against its gradient’s nuclear rank	Davis and Drusvyatskiy (2025), LIT-457 — the spectral step’s one-step guarantee beats the Euclidean one by the ratio of gradient nuclear rank to incoming activation stable rank. Both are cheap to compute from a run you are already doing, and the paper never uses them to route.	Proposed
SOTA-275	Pick the depth parameterization from the residual branch: a Transformer branch has more than one transformation, so it needs the stricter rule	Zheng et al. (2026), LIT-462 — Depth-muP’s α = Θ(1/√L) is the rule for a one-transformation residual branch. Attention and FFN branches hold more than one, so Transformers need CompleteP’s α = Θ(1/L), and the looser rule measurably fails to transfer on them.	Proposed
SOTA-276	For a normalized or preconditioned optimizer, width-depth muP is width muP plus a hidden residual multiplier of order 1/L	Zheng et al. (2026), LIT-462 — a normalized or preconditioned update has a norm that does not depend on the residual multiplier, so the depth factor the raw gradient carries is removed and the optimizer’s own muP rule is unchanged. SGD is the exception and needs more.	Proposed
SOTA-277	For a task that is a sequential composition, buy depth rather than width	Chen et al. (2024), LIT-464 — composing k functions costs a constant-depth decoder polynomially many parameters and a log k-layer one polylogarithmically. An exponential separation, proved unconditionally, and about what a shape can express rather than what a run will learn.	Proposed
SOTA-278	Before reporting that a model cannot do something, rule out the evaluation	Hu and Frank (2024), LIT-465 — the same capacity asked two ways scores differently, and the gap widens as the model weakens. Lawsen (2025), LIT-463, is the worked example: an output cap, a scorer that cannot express refusal, and instances with no solution.	Active
SOTA-279 v2	Put worked reasoning steps in the few-shot exemplars when the task needs more than one step, and only once the model is large enough	Wei et al. (2022), LIT-467 — eight exemplars carrying worked steps more than doubled PaLM 540B on GSM8K, beating a fine-tuned model with a verifier. Three ablations rule out the equation, the extra tokens and knowledge activation. Below ~100B it does nothing or hurts.	Active
SOTA-280	Sample several reasoning paths and take the majority answer rather than decoding one greedily	Wang et al. (2022), LIT-468 — replace greedy decoding with sample-and-marginalize: GSM8K +17.9, SVAMP +11.0, AQuA +12.2. Five or ten paths recover most of it, which is the whole cost question.	Active
SOTA-281	Try the single step-by-step instruction before writing exemplars; it recovers most of the gain for none of the labour	Kojima et al. (2022), LIT-469 — one fixed sentence with no exemplars takes MultiArith from 17.7% to 78.7% and GSM8K from 10.4% to 40.7%. It beats eight-shot standard prompting and loses to hand-written few-shot chains, which is the ordering that makes it a first move.	Active
SOTA-282	Put every matrix and hidden state on the unit hypersphere, and make each block’s residual step size a learned per-dimension parameter		Proposed
SOTA-283	To fine-tune a model that is already quantized, bank the part of each update that is smaller than the lattice spacing, and rebuild the accumulator from seeds rather than storing it		Proposed
SOTA-284 v2	Choose the checkpoint you post-train from by how much mass it puts on good rare responses, not by validation cross-entropy		Proposed
SOTA-285	Put dropout at the start of training if the model underfits, and at the end if it overfits — the schedule decides the sign, not the rate		Proposed
SOTA-286	To find a model that generalizes badly where its outputs look fine, measure how much of parameter space behaves the way it does		Proposed
SOTA-287	If you need to train without backpropagation, train each block to denoise a noisy label embedding conditioned on the raw input		Proposed
SOTA-288	Late in training, replicate a small subset of the weights, train the copies against shared base weights, and average the subset back into one model		Proposed
SOTA-289 v2	When the sampling budget is small, prefer uniform-state discrete diffusion with consistency distillation — masked diffusion cannot revise what it has already emitted		Proposed
SOTA-290	To measure what a deployed recommender does, use sock-puppets that never interact, follow sets curated to your question, and exposure weighted by rank		Proposed
SOTA-291 v2	Evaluate a specific intervention on a specific affordance rather than the net effect of the system, and never read an individual-level null as the absence of a collective effect		Active
SOTA-292	Check industry ties yourself when reading a literature about deployed platforms — disclosure is unreliable and the ties are mechanically detectable		Proposed
SOTA-293	Count embedding parameters when you report a tiny model’s size	Finke et al. (2025), LIT-485 — at these sizes the embedding table is not a rounding error, it is most of the model: TinyStories-33M is 68M parameters all-in, and an 8M model in the same paper is “more than half embeddings”. Two papers can each claim the smaller model and both be telling the truth. Print the all-in figure.	Proposed
SOTA-294	Fit the tokenizer to the corpus when the corpus is deliberately narrow	Finke et al. (2025), LIT-485 — holding the dataset and the architecture fixed and changing only the tokenizer, a 4,096-token WordPiece vocabulary seeded with English affixes beats GPT-2’s 50,257 by +26.7 coherence and +27.5 quality. It is the one fully controlled comparison in a paper that is mostly about something else, and the effect is larger than the architecture change measured beside it.	Proposed
SOTA-295	Parameterize the generating prompt above the word list, and constrain how each sample opens	Finke et al. (2025), LIT-485 — requiring a story to contain three words drawn from a vocabulary constrains the lexicon and leaves the frame alone, and the frame is where a generator repeats itself: 59.38% of TinyStories begins “once upon a time”. Name a theme, a topic, a style and a narrative feature per sample, and require the sample to open with a given part of speech and letter. Five diversity metrics move; measured simplicity does not.	Proposed
SOTA-296	Add the parallel streams in a short final training stage, not from the start	Chen et al. (2025), LIT-486 — parallel scaling costs P× the training FLOPs, which is what would otherwise make it unaffordable. Train 1T tokens the ordinary way, then switch the streams on for 20B tokens, 2% of the budget. The loss spikes when the random prefixes appear and recovers within 0.0002T tokens, and the logarithmic gains in P are the same ones the from-scratch runs show.	Proposed
SOTA-297	Scale parallel computation with learnable input transforms, not parameters, when inference memory is the binding constraint	Chen et al. (2025), LIT-486 — run the same weights over P learnably-prefixed copies of the input and learn the aggregation. Loss falls as if the parameters had grown by O(log P); a 1.6B model at P = 8 matches a 4.4B one on code. At batch size 1 that costs 22× less added memory and 6× less added latency than the parameter scaling it replaces — and P× the training FLOPs.	Proposed
SOTA-298	Measure a writing assistant by the outcome it was deployed to improve, not by the text features that used to predict it	Corpus et al. (2025), LIT-487 — Change.org’s AI drafting tool made petitions 49% longer, more lexically varied and better rated by human judges on quality and persuasiveness, moving them toward the profile of successful pre-AI petitions. The share reaching 10 signatures fell 5.33 points. The features had been chosen because they predicted success; afterwards they predicted it weakly or negatively.	Proposed
SOTA-299	Finetune the whole generative model, encoder and decoder, for a dense perception task instead of attaching a head to a backbone	Khangaonkar and Pirsiavash (2025), LIT-488 — the freshly-initialized mask head is the part that cannot generalize, because it has only ever seen the finetuning categories. Keep every parameter generatively pretrained, encode the target as an image, and a model finetuned on furniture and cars segments people, x-rays and paintings. The same backbone under a conventional head scores 1.4–2.4 mIoU.	Proposed
SOTA-300	To tell whether a model is doing non-trivial in-context computation, measure how well it predicts its own hidden states, not its next-token loss	Herrmann et al. (2025), LIT-489 — uniform noise maximizes next-token loss and requires no computation; reciting a memorized licence minimizes it and requires none either. Insert a variational bottleneck with a learned autoregressive prior and measure the posterior-prior KL: only the genuinely in-context task scores high, on a Transformer and an LSTM, and the signal survives controlling for next-token loss.	Proposed
SOTA-301	Apply the objective gradient before the denoiser, not after it, when guiding diffusion toward a task objective	Zhang et al. (2026), LIT-490 — the usual recipe adds −η∇f after the denoising step, which walks the sample off the geometry the model learned with nothing left to pull it back. Apply the gradient to the noisy iterate first and let the denoiser follow: it acts as an approximate projection, and the sampler becomes an inexact projected-gradient method. Same cost, one line, and in trajectory planning it is the difference between a plan that looks good and one that survives execution.	Proposed
SOTA-302 v2	Steer a distilled generator by modulating its input noise, not by fine-tuning its weights	Eyring et al. (2025), LIT-491 — train a LoRA hypernetwork to predict an improved initial noise for a frozen step-distilled generator. GenEval on SANA-Sprint goes 0.70 → 0.75 for 0.1 s of added latency, recovering about half of what 30-second test-time optimization buys. Reward fine-tuning the same model instead takes it 0.73 → 0.62: the anchoring KL term is intractable in weight space and tractable in noise space.	Proposed
SOTA-303	Name the phase a time-saving tool accelerates before predicting what it will do to the quality of what people produce with it	Duede, Gross, Crockett and Bergstrom (2026), LIT-492 — in a marginal-value-theorem model, accelerating the phase before value is known makes producers more selective, accelerating the minimum cost of shipping makes them less selective, and both make each item less thoroughly developed — while accelerating discretionary improvement makes each item more thoroughly developed. Opposite signs from the same tool depending on where it acts, so “it saves time” does not determine the direction.	Proposed
SOTA-304	Generate a long program unit by unit, freezing each unit that passes an execution-based check, instead of regenerating the whole artifact on failure	Cheng, Clark and Richardson (2025), LIT-493 — on 100 held-out proposals, the same model produced valid implementations 92% of the time generating unit by unit against an execution-based checker, and 6% of the time one-shot with retries; the one-shot outputs were also trivial, 49 lines against 181. Removing the checker alone costs 62 points, removing the decomposition alone costs 19 — so if you build one, build the checker.	Proposed
SOTA-305 v2	State the token count and input resolution beside any reconstruction FID, and compare tokenizers only at equal rate	Sun et al. (2024), LIT-497 — the same tokenizer at downsample 16 scores rFID 2.19, 0.94 and 0.70 on 256, 576 and 1024 tokens, all evaluated on 256×256 reconstructions. A three-fold range from a knob that is not the tokenizer. Two papers in this line then quote reconstruction numbers whose rate a reader cannot recover from the sentence they appear in.	Active
SOTA-306 v2	Factorize the codebook: look up in a low-dimensional normalized space, embed in a high-dimensional one, and report utilization alongside reconstruction quality	Yu et al. (2021), LIT-500 — project the encoder output down to a low-dimensional lookup space, ℓ₂-normalize it and the codebook, match there, and project the winner back up to a wide embedding. Codebook usage goes 4% → 95% and FID 3.68 → 1.50 at unchanged throughput; removing the normalization alone drops usage to 2%. Confirmed independently by LIT-497 and LIT-494, and in all three the curve turns back up, so there is an optimum to find rather than a direction to follow.	Active
SOTA-307 v5	Report generative FID as an error bar over several training seeds, and treat any gap below about 2% of the mean as inconclusive	Dufour, Efros and Pérez (2026), LIT-501 — on several hundred SiT networks, retraining the same recipe moves FID 3.2× more than resampling from a fixed model, and the coefficient of variation holds at 1–2% across four model sizes and every compute budget to 2M steps. Ten times the sampling budget shrinks the small term by √10 and leaves the large one untouched. A lucky training seed is worth up to 2× the compute.	Active
SOTA-308	When a retrieval change raises recall, measure relevance and the end-task metric before calling it an improvement	Wu et al. (2026), LIT-502 — across three published GraphRAG pipelines on G-Medical, graph expansion raises retrieval recall (GFM-RAG 84.3% against vanilla RAG’s 71.8%) and drops context relevance (38.5% against 62.9%). The wider context is noisier, and generation accuracy falls with it — a system that retrieves more of the right thing and much more of the wrong thing answers worse.	Active
SOTA-309	Score a control interface by the mutual information between the operator’s command and the state change it induces, and choose the horizon of that state change deliberately	Reddy, Levine and Dragan (2022), LIT-503 — whatever the operator is trying to do, a better interface yields less noisy commands, so I(x_t, (s_t, s_{t+Δ})) scores it with no labels, no reward and no task knowledge. Spearman ρ = 0.43 against true task completion across 540K examples in 4 of 5 domains. In the fifth the correlation was strongly negative at Δ = 1 and positive at episode length — so the horizon is not a detail.	Proposed
SOTA-310	Treat a mechanistic explanation that passes circuit error or causal alignment as one of many, and report what you did to rule the others out	Méloux, Maniu, Portet and Peyrard (2025), LIT-504 — by exhaustive enumeration on small MLPs, no network had exactly one circuit interpretation and under 2% had exactly one valid minimal mapping. The median number of explanations satisfying the criteria rises from 38 to 910,000 as width goes 2 → 5, and both counts are lower bounds. Passing circuit error or IIA establishes that an explanation is admissible, not that it is the explanation.	Active
SOTA-311	Evaluate a search on tasks held out of its own fitness function, and report the gap against the tasks it selected on	No source states this; it is filed on a priori grounds with one instance. Cheng, Clark and Richardson (2025), LIT-493, searched 1,062 pretrained designs against a fitness function of average downstream accuracy, then reported the top five on nine benchmarks drawn from the same pool and chosen — per its own Table 16 — for largest standard deviation across the search population. The best discovered design beats the best seed by 0.03 points.	Active
SOTA-312 v2	State the noise or binning assumption behind any mutual information you report for a deterministic network, and show the conclusion survives changing it	Saxe et al. (2018), LIT-509 — in a deterministic network I(h;X) is infinite, so every finite number on an information plane is a property of a noise model the analyst imposed and the network never had. The same tanh run, binned evenly in net input instead of evenly in activity, loses its compression phase entirely; and LIT-507 makes compression appear in ReLU networks by changing the binning the other way.	Active
SOTA-313	Treat the inference-time decision procedure as part of what you are comparing, and report the ordering under each one	Gemini Team (2023), LIT-510 — on MMLU, greedy sampling puts GPT-4 ahead by 0.2, plain chain-of-thought at 32 samples puts it ahead by 2.3, and uncertainty-routed CoT at 32 samples puts Gemini Ultra ahead by 2.7. Same models, same benchmark, same harness. The procedure is worth 6.0 points to one model and 3.1 to the other, so running the same procedure on both is not enough — the ordering is a property of the procedure you chose to report.	Active
SOTA-314 v2	Absorb quantization outliers into a high-precision low-rank branch taken from the weights, and fuse its kernels into the low-bit ones	Li et al. (2024), LIT-512 — at 4 bits on both weights and activations, smoothing alone is not enough. Shift outliers from activations into weights, peel the dominant singular values into a 16-bit rank-32 branch, quantize the residual. Then fuse: run naïvely the branch costs 57% latency and cancels the win; fused it gives 3.0× over W4A16 and 3.5× memory on 12B FLUX.1.	Active
SOTA-315 v2	Early-stop and tune on validation loss after temperature scaling, then calibrate post hoc, instead of stopping on raw validation loss	Berta et al. (2025), LIT-514 — a proper loss is calibration error plus refinement error, and the two bottom out at different epochs, so the validation-loss minimum is optimal for neither. Fit a temperature on the validation set each time you would have read the loss, read the loss after it, and stop there; then keep the temperature. Measured across 196 classification datasets and a vision benchmark with ten runs per dataset.	Active
SOTA-316	When you are memory-bound rather than time-bound, train all the parameters a block at a time, partitioned on layer boundaries	Liu et al. (2025), LIT-515 — freeze all but one block of layers, train it to local convergence, rotate. Memory falls to roughly the active block, so a 7B trains on one A800 where full-parameter needs two, and perplexity at 2B matches or beats the full-parameter baseline. The price is about three times the iterations, which the paper’s own table shows and its abstract does not.	Proposed
SOTA-317	Do not rank singular directions by magnitude when deciding what to discard; check the small end, and check it after fine-tuning	Staats, Thamm and Rosenow (2024), LIT-517 — in a transformer’s non-square matrices, the smallest singular directions carry data directions and their removal is catastrophic: Llama-3 8B on GSM8K falls from 43.2% to 2.0% when the smallest decile of the Down-Projection goes, against 40.0% for the same decile of the square Attention-Output. And the damage only appears if you prune after fine-tuning.	Active
SOTA-318	Choose the rank per matrix when you compress a transformer, because low-rank structure varies by component and by depth	Jaiswal et al. (2024), LIT-516 — “transformer weights are low rank” is true of some matrices and false of others. Query, Key and MLP Gate converge to low rank; MLP Up, MLP Down and Value do not; middle blocks resist while the first and last few give way. Setting the rank per matrix instead of globally is ~6.4× better in perplexity than uniform reduction at 30% on LLaMA-2 7B and ~47× at 40% on 13B.	Active
SOTA-319	Reparameterize every linear layer by its spectral norm with a learned scalar	Zhai et al. (2023), LIT-523 — replace each linear layer’s W with γ·W/σ(W), γ a learnable scalar initialized to 1 and σ the spectral norm from power iteration. A ViT-B trained this way reaches 82.2% against a DeiT baseline’s 81.8% with no pre-LN, no warmup, no weight decay and LARS in place of Adam. Spectral normalization without the scalar gets 69.81%.	Proposed
SOTA-320	Bound the learning rate by the ratio of the update’s spectral norm to the weight’s, and drop warmup	Qi et al. (2025), LIT-521 — if α_t > τ·σ₁(W_{t−1})/σ₁(∇W_t), truncate the step to that value; otherwise use the schedule. Weyl’s inequality makes this a direct cap on how fast a weight’s largest singular value can grow. ViT-B, ViT-L, GPT-S, Swin-S and Swin-B all train without any warmup and match or beat warmed-up AdamW.	Proposed
SOTA-321	Verify a model’s lineage from the singular spectra of its attention products, which survive permutation and rescaling	Wang et al. (2025), LIT-524 — compare the per-layer singular spectra of W_q W_k^T and W_v W_o. Those products absorb the permutation and rescaling that change weights without changing the function, so the spectra identify a model through fine-tuning, pruning, merging and expansion. F1 0.9867 over 55 model pairs, using no data and no queries.	Proposed
SOTA-322	Trade attention heads for depth: more heads conditions the attention block, and the layers you drop cost nothing	Saratchandran, Teney and Lucey (2025), LIT-527 — more heads lower the condition number of the attention block, which is part of what depth was buying. In the five configurations where the MLP width is held fixed, raising head count and cutting layers holds or improves accuracy at 29–53% fewer parameters — TNT-B goes 65.4M → 30.9M at identical Top-1.	Proposed
SOTA-323	Test a claim about what pretraining produces on a model trained with the pretraining objective, not one trained on the task family you are testing	Shen, Mishra and Khashabi (2023), LIT-536 — “there exist weights such that” and “pretraining produces weights such that” are different claims, and training a model on the task family you are about to test collapses them. Their separation of these as Hypothesis 2 and Hypothesis 1 is what let them show that a literature everyone read as evidence about pretrained language models had tested neither the models nor the objective.	Proposed
SOTA-324	Separate rival accounts of an internal algorithm by convergence rate and conditioning, not by how well each fits the output	Fu, Chen, Jia and Sharan (2023), LIT-535 — output similarity cannot tell gradient descent from Newton’s method, because both converge to the same answer. Matching each layer to the best-fitting step count of each candidate gives a linear trend for the right rate and an exponential one for the wrong one; a regime where the rivals must differ — here, condition number 100 — separates them again. Two measurements, both cheap, both decisive where a fit was not.	Proposed
SOTA-325	When the training loss has saturated and runs still differ, measure loss-landscape degeneracy rather than curvature	Lau, Furman, Wang, Murfet and Wei (2023), LIT-542 — on ResNet18/CIFAR10, stronger implicit regularization (higher learning rate, lower batch size, higher momentum) gives a lower local learning coefficient and higher test accuracy, while every training loss has collapsed to zero. The estimator is λ̂(w*) = n β* [E_{w|w*,β*,γ} L_n(w) − L_n(w*)] at β* = 1/log n, run by SGLD, validated against theory on deep linear networks to 100M parameters. Where the loss has stopped distinguishing runs, the geometry has not.	Proposed
SOTA-326	Add n-gram embedding memory as one table read early in the network, on top of the expert budget rather than in place of experts	Qiu et al. (2026), LIT-152 — Qwen3.8-Flash-Next’s ablations. One n-gram embedding table at layer 2 takes loss from 1.585 to 1.541 and the benchmark average from 45.44 to 47.94. Splitting the same table across two layers buys nothing, and no depth clearly wins. Trading experts for table parameters at a fixed total lowers loss slightly and moves no downstream benchmark. Grown on top of the budget, the table keeps lowering loss after downstream accuracy has flattened. Layer 2 is a systems choice: the host-memory prefetch overlaps layer 1.	Proposed
SOTA-327	Train retrieval embeddings with nested losses, so a prefix of the vector can shortlist and the full vector re-rank	Kusupati, Bhatt, Rege et al. (2022), LIT-547 — put the loss on log(d) nested prefixes of the embedding. Each prefix then matches a separately trained model of that width (ResNet50, ImageNet), and one database supports shortlisting on 16 dimensions and re-ranking on 2048: equal mAP@10, 14× faster. Measured on image retrieval. Widely shipped in text embedding models without an independent comparison in the record.	Proposed
SOTA-328	Train one nested model instead of a family of separately trained sizes, and extract the sizes you need from it	Devvrit, Kudugunta, Kusupati et al. (2023), LIT-546 — nest four FFN widths in one Transformer, train one width per step, and extract any size in between by choosing widths per layer. Up to 850M parameters, at the compute of training the four sizes separately, the largest matches its baseline and the smaller ones beat theirs. It pays only if you were going to train the family. Gemma 3n ships it.	Proposed
SOTA-329	For short-output tasks, adapt a frozen model with a low-rank intervention on a few positions of the residual stream, not a weight adapter; do not use it for long chain-of-thought	Wu, Arora et al. (2024), LIT-549 — LoReFT sets an r-dimensional subspace of the hidden state at a few prompt positions to a learned linear function of that state, with the model frozen. At 0.03% of parameters against LoRA’s 0.8%, it leads on commonsense QA (LLaMA-7B 80.2 against 74.7) and is level on GLUE. It loses on arithmetic chain of thought (GSM8K 26.0 against 37.5). All baseline numbers are copied from other papers.	Proposed
SOTA-330	Enforce output structure by masking logits against a grammar, with masks precomputed per automaton state and built on the CPU while the GPU runs the forward pass	Dong et al. (2024), LIT-552 — split the vocabulary by whether a token’s validity depends only on the grammar automaton’s current node (over 99% of tokens for JSON). Precompute those per node, check the rest against the full stack at runtime, and build each mask on the CPU during the GPU forward pass. Mask time is 0.018 ms per token after the ablated optimizations, and end-to-end overhead is 0.1–0.2 ms. It guarantees syntax. It does not measure whether answers stay correct.	Active
SOTA-331	Pass low-dimensional coordinate inputs through sinusoids of sampled frequencies before an MLP, and tune the frequency scale rather than the distribution	Tancik et al. (2020), LIT-550, with NeRF (LIT-435) — an MLP on raw coordinates never fits high frequencies. Map the input to [cos 2πBv, sin 2πBv] with B ~ N(0, σ²) and tune σ on held-out data: too small blurs, too large aliases, and the distribution’s shape does not matter. Gaussian features beat no mapping and log-spaced positional encoding on all seven image, shape, CT, MRI and view-synthesis tasks tested.	Active
SOTA-332	When an implicit representation will be supervised through its derivatives, use sine activations with the SIREN initialization rather than a ReLU network	Sitzmann et al. (2020), LIT-551 — a sine network’s derivative is another sine network, so gradients and Laplacians of the fit are well behaved where a ReLU network’s second derivative is zero. Initialize hidden weights U(±√(6/n)) and scale the first layer by ω₀ = 30 (tune it to the signal), or deep sine networks do not train. It fits images, video, SDFs and PDE solutions from derivative supervision.	Proposed
SOTA-333 v4	For autoregressive generation of continuous sequences, train with an independent noise level per token and condition the rollout on slightly noised history	Chen et al. (2024), LIT-554 — train a causal model to denoise tokens that each carry an independent noise level. At rollout, treat the generated history as slightly noisy, so accumulated errors look like training noise. The same model then samples autoregressively, plans with horizon-wide guidance, and keeps near-future tokens cleaner than far ones. Long-rollout stability is shown on video qualitatively, with a small RNN. Planning and robot gains are measured.	Proposed
SOTA-334	In a style-modulated generator, demodulate the convolution weights instead of instance-normalizing the activations	Karras et al. (2019), LIT-560 — AdaIN’s instance normalization discards relative feature magnitudes, so the generator smuggles scale past it with a localized spike, the “water droplet” in every StyleGAN image. Scale each output channel’s modulated weights by 1/√(Σ w’² + ε) instead. It gives the same expected unit variance and style mixing still works. The artifact disappears and FID is unchanged (4.40 → 4.39).	Proposed
SOTA-335	When generated content must move continuously, as in video or animation, build the generator alias-free: treat features as continuous signals and low-pass filter around every nonlinearity	Karras et al. (2021), LIT-559 — fine detail in GAN output sticks to pixel coordinates because the generator aliases. Apply each nonlinearity at 2× temporary resolution and low-pass filter back, start from Fourier features rather than a learned constant, remove per-pixel noise, and handle boundaries properly. On FFHQ-U, translation equivariance goes to 63 dB (and rotation to 40 dB for the -R variant) at StyleGAN2’s FID.	Proposed
SOTA-336	Train high-resolution GANs at a fixed topology with output skips in the generator and a residual discriminator, not by progressive growing	Karras et al. (2019), LIT-560 — progressive growing makes each resolution briefly the output, so it learns output-level detail and leaves features stuck to pixel positions. A fixed generator that sums upsampled RGB outputs from every resolution, with a residual discriminator, keeps the coarse-to-fine emphasis without changing topology. FFHQ FID 4.34 → 3.31, and the phase artifacts are gone.	Proposed
SOTA-337 v4	When an ImageNet classifier is anywhere in the pipeline — training the generator or steering its sampling — confirm FID gains with a Fréchet distance in a non-ImageNet feature space	Kynkäänniemi et al. (2022), LIT-563 — FID’s features are nearly ImageNet class logits, so matching ImageNet-class statistics lowers it without improving images. Resampling a fixed StyleGAN2’s outputs cuts FFHQ FID 5.30 → 1.78, while CLIP-space FD moves 2.76 → 2.64. A model whose discriminator uses ImageNet features exploited this by accident, and the authors suspect data filters and samplers can too. Report a CLIP or SwAV Fréchet distance beside FID whenever that is the case. Classifier guidance (LIT-699) is the sampling-loop instance and the least checked: it maximises an ImageNet classifier’s confidence on every step, and its paper denies producing adversarial examples in one clause without running a test.	Active
SOTA-338 v2	Train the GAN discriminator on frozen multi-scale pretrained features with fixed random channel and scale mixing	Sauer et al. (2021), LIT-562 — project real and generated images through a frozen EfficientNet, mix channels and scales with fixed random convolutions, and train one small discriminator per scale. It reaches StyleGAN2’s best LSUN-Church FID after 1.1M images instead of 88M. The speed-up is measured in FID, and at equal FID on FFHQ a CLIP-space distance and human raters prefer StyleGAN2 (LIT-563).	Proposed
SOTA-339	For unpaired image-to-image translation of appearance, constrain both mappings with a cycle-consistency loss	Zhu et al. (2017), LIT-564 — learn X → Y and Y → X together, and penalize ‖F(G(x)) − x‖₁ and ‖G(F(y)) − y‖₁ alongside the two adversarial losses. With GAN losses alone the mapping may ignore its input. With the cycle it keeps content and changes appearance. It is scoped to colour and texture changes, and the authors report failure on geometric ones.	Proposed
SOTA-340	Expect weight-only 4-bit quantization to speed up batched serving only while the batch keeps the matmul memory-bound, and use a kernel built to stay there	Frantar et al. (2024), LIT-567 — weight-only quantization speeds up decoding by loading fewer bytes, so the gain lasts only while the layer is memory-bound. A kernel designed to stay memory-bound (MARLIN) holds about 3.9× over FP16 up to batch 16–32 on an A10, decaying toward 1.5× at 128. End-to-end in vLLM, single-GPU, 2.3–3.2× at batch 16 or below and 1.1–1.2× at 128, on four GPU classes. Pick the format for the batch you actually serve, and pay for it in accuracy only where it buys speed.	Proposed
SOTA-341	Condition an image generator on each training image’s original size instead of discarding or upsampling small images	Podell et al. (2023), LIT-566 — embed each training image’s original height and width, like the timestep, and give it to the model. Then keep the small images that a minimum-resolution filter would drop (39% of SDXL’s pretraining data below 256²) without teaching the model their blur. At inference, set the size you want. On class-conditional ImageNet 512²: FID-5k 43.84 discarding, 39.76 keeping unconditioned, 36.53 keeping with size conditioning.	Proposed
SOTA-342	Report a sparse autoencoder’s fidelity as downstream loss in compute-equivalent terms, not as fraction of loss recovered against zero ablation	Gao et al. (2024), LIT-571 — splice the SAE’s reconstruction into the forward pass and report the resulting loss as the pretraining compute a model of that loss would need. Zero-ablating the residual stream is so destructive that “loss recovered” against it flatters any reconstruction. Their 16M-latent GPT-4 SAE scores 98.2% on it, and equals a GPT-4 trained on 10% of the compute.	Proposed
SOTA-343	Read a transformer’s intermediate-layer predictions through a tuned lens, not the raw logit lens	Belrose et al. (2023), LIT-569 — decoding a hidden state with the final unembedding (the logit lens, LIT-570) fails outright on BLOOM, OPT and GPT-Neo, and is biased by 4–5 bits where it works. Train one affine translator per layer by KL to the final logits, initialized to the identity, and read the layer through that. It is cheap (d×d per layer), transfers to fine-tunes, and its directions are the model’s (Spearman 0.89). Train it with Muon.	Proposed
SOTA-344	Train sparse autoencoders with a TopK activation instead of an L1 penalty, and prevent dead latents with transposed-decoder initialization and an auxiliary loss	Gao et al. (2024), LIT-571 — keep the k largest pre-activations and zero the rest, and train on reconstruction alone. L0 is then set, not tuned through an L1 coefficient that also shrinks the latents that fire. It beats ReLU SAEs on the MSE–L0 frontier by more as SAEs grow. Initialize the encoder as the decoder’s transpose and add a loss reconstructing the residual from dead latents: 7% dead at 16M latents, against up to 90%.	Proposed
SOTA-345	Probe language-model activations with logistic regression on the raw activations, and credit sparse-autoencoder probes only against that baseline under validation-based selection	Kantamneni et al. (2025), LIT-568 — across 113 datasets, adding SAE-latent probes to a toolkit of raw-activation probes, choosing per task by validation AUC, changes test AUC by −0.003. It does not help under data scarcity, class imbalance, label noise or shift. Reported SAE wins came from baselines denied the same pooling or selection. Use logistic regression on activations, and hold any SAE method to it.	Active
SOTA-346	Retune the training timestep shift for each autoencoder latent space, and never rank latent spaces for generation under one shared schedule	Black Forest Labs (2025), LIT-572 — four autoencoders, 30 timestep configurations each. The training shift alone moves FID by 61–86%. An RAE latent that wins when both are tuned loses to a tuned FLUX.2 latent when left unshifted. When you swap the autoencoder under a diffusion or flow model, sweep the shift again, using a logit-normal training distribution. When you compare autoencoders, compare each at its own optimum.	Proposed
SOTA-347	To write many facts into a model’s weights, update a range of early-to-mid MLP layers jointly in one batched solve (MEMIT), choosing the range by measured edit success rather than causal tracing	Meng et al. (2023), LIT-576 — sequential single-fact edits collapse after tens of facts (ROME scores 50.3 at 10,000). Distributing each update over a range of MLP layers in one least-squares solve holds 85.8 on GPT-J, with neighborhood success falling from 83.5 to 73.7. Choose the layers by editing performance: per-fact causal tracing does not predict it (LIT-574). Evaluate implications too, because weight editors fail them (LIT-577).	Proposed
SOTA-348	Evaluate a knowledge edit on the facts it implies — logical consequences, multi-hop compositions, aliases and the subject’s other facts — not only on the edited prompt, its paraphrases and unrelated neighbours	Cohen et al. (2023), LIT-577 — the standard edit metrics (efficacy, paraphrase, neighborhood) cannot see whether an edit changed what the model knows or only what it says to one question. On six ripple criteria, ROME, MEMIT and MEND average 38–66, and logical generalization falls as low as 5.5. Prompting with the new fact scores 81–83 on LLaMA-7B. Report ripple metrics beside the standard ones, and compare against in-context editing.	Active
SOTA-349	Before crediting a graph transformer with long-range gains, retune the message-passing baselines under the same budget, including an MLP prediction head, positional encodings and input feature normalization	Tönshoff et al. (2023), LIT-579 — the Long-Range Graph Benchmark’s case that graph transformers are needed rested partly on untuned message-passing baselines. With a basic sweep inside the same 500k-parameter budget, GCN beats GPS on both Peptides tasks. Most of the gain is from replacing a linear prediction head with a 2-layer MLP. Normalizing input features lifts every model on the superpixel tasks, where GPS still leads. Tune both sides before attributing a gap to attention.	Active
SOTA-350	Compare GNN architectures over many random data splits and seeds, with one shared training and hyperparameter-selection procedure for every model, never on a single fixed split	Shchur et al. (2018), LIT-581 — on the Planetoid split GAT beats GCN. On another random split of the same size GCN wins, and averaged over 100 splits × 20 seeds with identical training and tuning, GCN ranks first of four GNNs. A single split selects the model that overfits it, and per-model recipes measure the recipe. Report mean and spread over many splits and seeds, with the same tuning budget for every model.	Active
SOTA-351	In a message-passing GNN for graph-level tasks, aggregate neighbours by sum with an MLP update, not by mean or max, when graph structure rather than node features carries the signal	Xu et al. (2019), LIT-580 — mean and max aggregation cannot count neighbors, so graphs that differ only in counts look the same to them. On featureless Reddit graphs, mean-aggregation GNNs are at chance (50.0, 20.0) and sum–MLP scores 92.4 and 57.5. With informative node features the choice matters much less, and results are mostly within noise. Use h = MLP((1 + ε)h + Σ neighbors) with ε = 0 and a sum readout.	Active
SOTA-352	When a structure predictor is made generative, add a regression model’s predictions to the training data so disordered regions are learned as disorder rather than invented structure	Abramson et al. (2024), LIT-584 — a diffusion head generates plausible compact structure even where the protein has none. AlphaFold- Multimer, a regression model, renders the same regions as extended loops. Mixing its predictions into training teaches the generative model that convention, and “greatly reduced” hallucination. Remaining hallucinations are flagged by low confidence but do not look disordered.	Proposed
SOTA-353	Train a head that predicts the model’s own accuracy on each output, against the metric you care about, and use it to rank samples and filter predictions	Jumper et al. (2021), LIT-583 — regress the per-residue lDDT the prediction will score (pLDDT) and the predicted TM-score. They track truth at r = 0.76 and 0.85 over 10,795 chains, and pLDDT filters the self-distillation set. Abramson et al. (2024), LIT-584, keep the head under diffusion by rolling out a cheap sample during training. They rank seeds by it, and antibody-interface quality keeps rising up to 1,000 seeds.	Proposed
SOTA-354	When unlabelled inputs vastly outnumber labelled ones, retrain from scratch on the labelled data mixed with the model’s own high-confidence predictions for unlabelled inputs, under augmentation that stops it copying them	Jumper et al. (2021), LIT-583 — about 100,000 solved structures and billions of sequences. AlphaFold 2 predicted structures for about 350,000 unlabeled sequences, kept the confident ones, and retrained the same architecture from scratch on those plus the PDB. Cropping and alignment subsampling mean the student cannot just reproduce the teacher. The ablation shows a clear gain. The filter is the model’s own calibrated confidence (SOTA-353).	Proposed
SOTA-355 v2	Isolate the emergent outlier feature dimensions into 16-bit and quantize the rest to int8	Dettmers et al. (2022), ARXIV-2208.07339. Quantize the projections vector-wise to int8, but pull the handful of systematically large feature dimensions out into a separate 16-bit multiplication — 99.9% of values in 8-bit, no measured quality cost up to 175B.	Active
SOTA-356	Protect the salient weight channels by scaling them before rounding, choosing them from the activations rather than the weights	Lin et al. (2023), ARXIV-2306.00978. Identify the ~1% of weight channels whose activations are largest, scale them up by an equivalent transformation before rounding, and quantize everything uniformly — rather than keeping the important channels in higher precision.	Active
SOTA-357	Build a zero-shot classifier from prompt templates and ensemble them in embedding space		Active
SOTA-358 v3	Drop the domain inductive bias once pre-training data is large enough, and keep it when it is not		Active
SOTA-359 v2	Supervise vision from the caption, and match image to caption rather than predicting its words		Active
SOTA-360	Score a density ratio against sampled negatives instead of reconstructing the target		Active
SOTA-361	Compose augmentations so the shortcut dies, and tune them for the contrastive objective rather than the supervised one		Active
SOTA-362	Train through a projection head and then discard it, taking the layer before as the representation		Active
SOTA-363	Decouple the negative set from the batch with a queue, and keep the encoder that fills it slow		Active
SOTA-364	Break batch normalization’s cross-sample leak before it solves your contrastive task for you		Active
SOTA-365 v2	Learn without negatives by breaking the symmetry: a predictor on one branch and a stop-gradient on the other		Active
SOTA-366	Monitor the standard deviation of the l2-normalized embedding; healthy is one over root d and collapsed is zero		Active
SOTA-367	Or put the anti-collapse constraint in the loss, as a variance floor and a decorrelation term		Active
SOTA-368	When a model’s latent space is meant to be read by people, measure its interpretability with a human task such as word or topic intrusion, not by held-out likelihood	Chang et al. (2009), LIT-597 — topic models are used for their topics, but they are judged by held-out likelihood, which measures the probability of observations and ignores the representation. On NYT and Wikipedia, CTM had the best likelihood and the worst human scores. Test the latent space directly. Plant an improbable word among a topic’s top words, or an improbable topic among a document’s top topics, and measure how often people find it.	Active
SOTA-369	Curate a pretraining set by retrieving neighbours of curated seeds, rather than scaling the uncurated pool		Active
SOTA-370	Add many low-resolution views alongside the two full ones, because view count is what helps and resolution is what costs		Active
SOTA-371	Score topic coherence automatically as the average pointwise mutual information of a topic’s top-word pairs, estimated on a large external reference corpus	Newman et al. (2010), LIT-600 — when you cannot put every topic in front of people (SOTA-368), approximate their judgment. Take the topic’s top ten words and, for each of the 45 pairs, compute PMI from co-occurrence in 10-word windows over a large general corpus such as Wikipedia. Average the 45 scores. On 237 LDA topics this correlated with nine people’s ratings at ρ ≈ 0.77, far better than WordNet measures.	Proposed
SOTA-372 v2	Run the encoder on visible tokens only and push the mask tokens into a small decoder you discard		Active
SOTA-373 v3	Set the masking ratio by the signal’s redundancy, not by the ratio that worked on text		Active
SOTA-374 v4	When learning embeddings from co-occurrence, subsample very frequent tokens, discarding each occurrence with probability 1 − √(t/f) with t around 10⁻⁵	Mikolov et al. (2013), LIT-603 — in a Zipfian corpus, the few most frequent words co-occur with everything and dominate the updates while teaching little. Discard each occurrence of word w with probability 1 − √(t/f(w)). This cut skip-gram training time by 2–3× and raised accuracy, only slightly for words (59 → 60% at NEG-5) but a lot for phrases (27 → 42% at NEG-15, 19 → 47% with hierarchical softmax).	Active
SOTA-375 v2	When training with sampled negatives, draw them from the unigram distribution raised to the 3/4 power, not from the unigram or uniform distribution	Mikolov et al. (2013), LIT-603 — sample negatives in proportion to count^(3/4). That flattens the Zipfian head so frequent words are not nearly the only negatives, while keeping rare words rare. The authors report it “outperformed significantly the unigram and the uniform distributions … on every task we tried”, but give no numbers.	Proposed
SOTA-376	Score each image-text pair independently with a sigmoid, so the loss needs no global normalization		Active
SOTA-377	Stop scaling the contrastive batch past about 32k, because the benefit saturates there		Active
SOTA-378	Compare, project and orthogonalize a language model’s concept directions after whitening by the unembedding covariance, not by raw cosine	Park, Choe and Veitch (2024), LIT-606 — training fixes a model’s representation only up to an invertible linear map, so Euclidean cosine between concept directions is not meaningful. Use ⟨u, v⟩ = uᵀ Cov(γ)⁻¹ v, with Cov(γ) the covariance of the unembedding rows, or equivalently transform by Cov(γ)^(−1/2) and then use Euclidean tools. Shown on LLaMA-2 7B and Gemma-2B by heatmap. Contested by a null for cross-lingual transport in 17 models.	Proposed
SOTA-379 v2	Before crediting a word-embedding method over count-based ones, give the baselines the same design choices and tune every method alike	Levy, Goldberg and Dagan (2015), LIT-607 — word2vec and GloVe ship with design choices (context smoothing, the negative-sample shift, dynamic windows, subsampling, adding context vectors) that count-based PPMI and SVD can use too. Port them and tune every method the same way before comparing. Done that way across 672 representations and 8 datasets, the prediction-over-count advantage disappears, SGNS beats GloVe on every task, and a single hyperparameter often matters more than the method.	Active
SOTA-380	Use prediction-based word vectors rather than count-based distributional vectors	Baroni, Dinu and Kruszewski (2014), LIT-608 — across 14 benchmarks, word2vec CBOW beat PPMI, SVD and NMF count vectors almost everywhere, and the authors recommend “anybody interested in using DSMs” to “go for the predict models”. Retired: LIT-607 traces the gap to settings the count models were not given, and finds no consistent winner once they are.	Rejected — the advantage it reports disappears when the count-based baselines get the design choices word2vec shipped with and every method is tuned alike (LIT-607): no family wins consistently. Retired with no successor. What replaces it is an evaluation practice, SOTA-379, which corrects it
SOTA-381 v2	Sample a handful of negatives instead of normalizing over the vocabulary, and let the count fall as the data grows		Active
SOTA-382 v3	Subsample frequent tokens: it is faster and it improves the rare ones		Superseded — by SOTA-374; the same recommendation from the same paper and section; SOTA-374 states the discard rule and carries the independent ablation
SOTA-383	Fix the sample count before comparing FID values, and settle a close comparison with an unbiased estimator rather than a tighter error bar	Bińkowski et al. (2018), LIT-615 — the FID estimator is biased, the bias depends on the distribution being measured, and no unbiased estimator exists. So two FID numbers are comparable only at equal n, and a small standard deviation is not evidence of a converged estimate: at d=2048 and 50,000 samples their construction reversed a true ordering in 100 trials out of 100, with standard deviations of 0.2 and 0.5.	Active
SOTA-384	Calibrate a trained classifier with a single temperature fitted on held-out data, not with a richer map	Guo et al. (2017), LIT-616 — rescale the logits by a single learned 1/T fitted for NLL on a held-out set. It cannot change the argmax, so accuracy is unchanged by construction, and it beats vector scaling, matrix scaling, histogram binning, isotonic regression and BBQ — including the two that strictly contain it. Typical uncalibrated ECE is 4–10%.	Active
SOTA-385	Tighten the gradient-norm clip as the model grows rather than carrying one value across scales	Rae et al. (2021), LIT-617 — the Gopher family clipped the global gradient norm at 1.0 for models up to 1.4B and at 0.25 for the 7.1B and 280B models, “for improved stability”. Three held practices say to clip and none names a value; this is the first number in the record, and it is not a constant.	Proposed
SOTA-386 v4	Show a video diffusion model images before and alongside video		Proposed
SOTA-387	Hold out a slice of the training set for model selection instead of selecting on the validation split	Beyer et al. (2022), LIT-628 — they train on the first 99% of ImageNet-1k and keep 1% as a “minival”, stated as being “to encourage the community to stop selecting design choices on the validation (de-facto test) set.” A convention, asserted rather than measured, and the record held no document on it.	Proposed
SOTA-388 v2	Train a plain ViT on ImageNet-1k with average pooling, fixed 2D sin-cos positions, a 1024 batch and light augmentation	Beyer et al. (2022), LIT-628 — five changes, none novel, take ViT-S/16 on ImageNet-1k from 66.8% to 76.5% at 90 epochs and 80.0% at 300. Global average pooling, fixed 2D sin-cos position embeddings, batch 1024, RandAugment at level 10 and Mixup at p = 0.2. 6h30 on a TPUv3-8 for the 90-epoch run.	Active
SOTA-389 v3	Caption training video with a model that watches the video, not with captions of its frames		Proposed
SOTA-390 v3	In a video diffusion transformer, attend over space and time jointly rather than factorizing, and budget for the cost		Proposed
SOTA-391	Estimate attention resolution before training when choosing a position encoding, and measure it across the extrapolation boundary	Sun et al. (2022), LIT-638 — attention resolution scores how well an attention pattern distinguishes token distance, and is estimable from the encoding before a run. Measured at the training length and twice it: RoPE 0.91 → 0.08, ALiBi 0.81 → 0.88. The record has four practices about rotary encoding and held no measure of this.	Proposed
SOTA-392 v3	To get a one-step sampler from a rectified flow, distil it; treat reflow as an optional extra pass, and keep the pre-reflow model for many steps	Liu, Gong and Liu (2022), LIT-636. On CIFAR-10, one-step FID is 378 from the base model, 6.18 after distillation alone, 12.21 after one reflow, and 4.85 after reflow and distillation (Table 1a). Distillation does most of the work. Reflow adds a further gain at the cost of a second training pass, and worsens many-step quality from 2.58 to 3.36. InstaFlow (LIT-796) contests it at text-to-image scale, starting from a diffusion model rather than a rectified flow.	Proposed
SOTA-393	Freeze the image model’s spatial layers when adding time if checkpoint compatibility matters; do not expect a quality gain from it	Girdhar et al. (2023), LIT-635, and Guo et al. (2023), LIT-633. Freezing a text-to-image model’s spatial layers and training only temporal ones keeps the model a drop-in for checkpoints fine-tuned from the same base. Emu Video’s one controlled test finds freezing costs nothing and gains little (55.0 / 58.1 against fine-tuning). The reason to freeze is compatibility, not quality.	Proposed
SOTA-394 v3	When distilling a causal few-step video generator, take the teacher bidirectional, not causal	Yin et al. (2024), LIT-631. A block-causal 4-step student distilled from a bidirectional teacher beats one distilled from a causal teacher, at the same initialization, data and step count (Table 4). The causal teacher is also the weaker model before distillation, so the result shows the arrangement works and does not isolate why.	Proposed
SOTA-395	Post-train an autoregressive video generator on its own rollouts, and score the whole rollout	Huang et al. (2025), LIT-629. Training a causal video generator on context it generated itself, with a distribution-level loss over the whole clip, beats teacher-forced and diffusion-forced training under every objective tried (Table 2). It holds quality as the number of autoregressive steps grows, where they degrade. Evidence is VBench on 5s clips, and the exposure-bias mechanism is inferred, not measured.	Proposed
SOTA-396	When a latent video model reuses an image autoencoder or upsampler, fine-tune them on video so they see time	Blattmann et al. (2023), LIT-621. An image autoencoder decodes each frame independently and flickers. Fine-tuning its decoder on video, with a video discriminator, cuts reconstruction FVD on three datasets: 390.88 to 32.94, 35.82 to 18.66, and 73.78 to 25.55. Making the upsampler temporal cuts FVD from 165.98 to 45.39. Reconstruction FID worsens on one dataset and improves on two.	Proposed
SOTA-397 v2	To condition an unconditionally trained diffusion model on known frames, use reconstruction guidance, not replacement	Ho, Salimans et al. (2022), LIT-627. Extending a video autoregressively from a model trained without conditioning, reconstruction guidance (steering each denoising step toward consistency with the known frames) gives FVD 136 against 451 for replacing the known frames’ noisy latents (Table 6). The model and guidance weight are held fixed.	Proposed
SOTA-398 v2	Pretrain a vision transformer on packed, aspect-preserved images at sampled resolutions, not on fixed square crops	Dehghani et al. (2023), LIT-657. Do not resize or crop images to one square resolution. Keep each image’s aspect ratio and sample its resolution per example, favouring small side lengths. Pack the patches into fixed-length sequences with per-example attention masks. At matched compute this beats square fixed-resolution ViT pretraining, and matches the best ViT with 4x less compute. Most of the gain is throughput.	Proposed
SOTA-399	Choose pre-training weight decay, dropout, clipping and precision for the quantization you intend to ship		Proposed
SOTA-400	Append a few register tokens to a ViT so it does not commandeer background patches for scratch space		Active
SOTA-401	Anchor the patch Gram matrix to an early teacher when training long, because dense features decay while global metrics improve		Proposed
SOTA-402	To teach a model to reason over facts it holds, spend added data on facts derived from other facts rather than on more atomic facts	Wang et al. (2024), LIT-667. Two sweeps that separate what is normally varied together: with the inferred/atomic ratio φ fixed, scaling the training set changes nothing about how a transformer acquires a two-fact inference rule; with the size effect thus accounted for, raising φ moves the acquisition monotonically, and at φ = 18.0 the delay is gone — 96.7% before training accuracy has even saturated. Synthetic knowledge graphs only, and it buys rule application on facts already seen in compositions, not systematicity.	Proposed
SOTA-403 v3	Share the attention parameters across layers if you need to cut parameters; do not share the feed-forward ones	Lan et al. (2019), LIT-668. A transformer block has two halves and they are not equally compressible across depth. On an ALBERT-base configuration at E = 128, sharing every layer’s attention parameters scores 81.7 average against 81.6 for no sharing while removing 89M parameters to 64M; sharing the feed-forward parameters instead scores 80.2. At E = 768 the same split reads −0.7 against −2.8. One paper, 2019, encoder-only, and nobody has re-run it.	Proposed
SOTA-404 v2	When you ablate an auxiliary loss, ablate the input construction that came with it, or the result is about both	Liu et al. (2019), LIT-671. BERT reported that removing next sentence prediction hurts; RoBERTa observed that the ablation had probably “only removed the loss term while still retaining the segment-pair input format”, separated the two, and found removing the loss matches or slightly improves while the input format was what mattered. LIT-667 is the same defect in another literature: four grokking papers varied the training fraction of a fixed universe, so dataset size and dataset composition moved together, and the conclusion reversed when somebody turned them one at a time.	Proposed
SOTA-405	Scale the masked-language-modelling rate with model size rather than holding it at 15%	Wettig et al. (2022), LIT-672. The masking rate has been held at 15% “regardless of model sizes or masking strategies” since 2018, and the number was never swept. Swept, the optimum moves with capacity: 40% at 354M, 20% at 124M, 15% at 51M. At 354M, 40% wins seven of nine GLUE-plus-SQuAD tasks and reaches the 15% model’s QNLI and QQP scores in half the training time — under an efficient recipe, with the advantage shrinking on longer schedules.	Proposed
SOTA-406	Replace every masked token with [MASK]; drop BERT’s 80-10-10 substitution rule	Wettig et al. (2022), LIT-672. BERT replaced 10% of selected positions with the original token and 10% with a random token, to reduce the pretrain/fine-tune mismatch caused by [MASK] never appearing downstream. Measured against an all-[MASK] baseline at a matched 40% rate, 80-10-10 is worse on everything but SST-2 — and the stated motivation does not bite, because “the model can adapt to full, uncorrupted sentences, regardless of the use of alternative corruption strategies in pre-training”.	Proposed
SOTA-407 v2	Interpolate the zero-shot and fine-tuned weights at about half way rather than shipping the fine-tuned model	Wortsman et al. (2021), LIT-674. Fine-tuning a zero-shot model spends the robustness that made it worth starting from. Ship (1 − α)·θ_zero-shot + α·θ_fine-tuned at α = 0.5 instead: +3.5 to +23.2 pp under six distribution shifts against the fine-tuned model, with reference accuracy falling by at most 0.3 pp and usually rising, at no cost in training or inference. Requires that the fine-tuned weights were obtained from the zero-shot weights.	Active
SOTA-408 v3	Average the weights along the tail of training under a cyclical or high constant learning rate, then re-estimate the normalization statistics	Izmailov et al. (2018), LIT-673. Hold the learning rate high — cyclical or constant — over the last stretch of training and keep a running average of the weights, then ship the average. From pretrained torchvision checkpoints, ten further epochs gives ImageNet top-1 76.15 → 76.97 on ResNet-50 and 78.31 → 78.94 on ResNet-152, at one model’s inference cost. With batch normalization, the running statistics must be recomputed with one pass over the data or the averaged model is broken rather than merely worse.	Proposed
SOTA-409	Average the fine-tuned models from your hyperparameter sweep instead of keeping only the best, adding each one in validation order and only if it helps	Wortsman et al. (2022), LIT-675. A hyperparameter sweep ends by discarding every model but one. Average them instead — sorted by held-out accuracy, keeping each only if held-out accuracy improves, which makes the result no worse than the best individual model on that set by construction. O(1) inference against an ensemble’s O(k). +0.7 pp on CLIP and +0.5 pp on ALIGN over the best sweep member; +0.16 on ImageNet and +0.34 under distribution shift for ViT-G. Averaging all of them uniformly instead can come out worse than the best one.	Active
SOTA-410 v2	For guided sampling, use a second-order multistep solver on the data-prediction parameterization, not a higher-order solver on the noise prediction	Lu et al. (2022), LIT-676. SOTA-203’s higher-order solver inverts under guidance: at scale 8.0 and 10 evaluations, FID is 13.04 for first-order DDIM against 114.62 at order 2 and 164.74 at order 3. Sample instead with a second-order multistep solver parameterized on the data prediction — which is also what makes SOTA-202’s per-step clamping available at all, since you cannot clamp x̂₀ if the solver’s state is ε̂. 20 evaluations then beat DDIM at 250.	Active
SOTA-411	To let a model spend more compute at inference without more parameters, train a depth-recurrent core on a randomly sampled iteration count, re-injecting the input at every iteration	Geiping et al. (NeurIPS 2025), LIT-683. Put a small core block between a short prelude and coda. At each training step sample how many times to iterate it, from a heavy-tailed distribution with mean 32. Backpropagate through the last few iterations only. Feed the input embedding into every iteration, and start the state from noise. Trained this way, accuracy rises with iterations at test time and saturates per task. A model trained at fixed depth does not gain this. Nothing yet shows it beats a standard transformer of the same FLOPs.	Proposed
SOTA-412	When the target is likelihood, learn a per-dimension noise schedule conditioned on a learned latent; a scalar or unconditioned one buys nothing	Sahoo et al. (NeurIPS 2024), LIT-677. For density estimation with a pixel-space diffusion model, replace the scalar learned schedule with a per-dimension one conditioned on a small discrete latent that an encoder infers from the image. Keep the endpoints fixed. VDM’s bound is reached in 5× fewer steps on CIFAR-10 and improved at equal steps. Each component alone does nothing: a scalar input-conditioned schedule and a time-only multivariate schedule both match VDM.	Proposed
SOTA-413 v2	In a scaling-law study, count the output head in parameters and FLOPs, and fit loss against compute with an irreducible-loss offset	Pearce and Song (TMLR 2024), LIT-688. At small scale the embedding and output head are a large share of the model. Excluding them from N, and from the FLOPs in C = 6ND, biases the fitted allocation exponent: 0.49 becomes 0.74 when the same runs are relabelled. An offset-free loss–compute fit biases that exponent too. Do not compare exponents across studies that count differently.	Active
SOTA-414	To make an image diffusion model a dense geometric predictor, fine-tune it end to end as a one-step model at t = T with a task loss	Martin Garcia et al. (WACV 2025), LIT-687. Fix the timestep at T, feed zeros as the noise, decode the prediction through the frozen VAE, and train on the task’s own loss: affine-invariant L1 for depth, angular error for normals. From 20K iterations on 74K synthetic images, the one-step model beats Marigold’s 50-step, 10-member ensemble on depth AbsRel on all five zero-shot sets, and on normals by 2.6–3.0° of mean error. Starting from plain Stable Diffusion is nearly as good.	Proposed
SOTA-415 v3	During a warmup-stable-decay run, estimate the annealed score from a uniform average of recent stable-phase checkpoints instead of launching a decay branch	Li et al. (2025), LIT-684, extending Hägele et al. (2024), LIT-145. To see where a WSD run would land if annealed now, evaluate the simple average of its last ~10 stable-phase checkpoints. On the one fork measured, that proxy sits within about 1.5 points of a real 250B-token anneal on each benchmark. It costs no training. Still anneal the model you ship: on that fork the anneal is ahead on MMLU at the end.	Proposed
SOTA-416 v2	When sampling a diffusion model in few steps, start at t = T: use trailing, not leading, timestep spacing	Martin Garcia et al. (WACV 2025), LIT-687. DDIM’s “leading” spacing never visits the final timestep. So at one step, a pure-noise input is labelled with a timestep that claims an almost clean sample. With the same weights and only the spacing changed, Marigold’s single-step depth goes from noise to NYUv2 AbsRel 5.7. The mismatch vanishes as the step count approaches T, which is why it goes unnoticed at 50 steps.	Active
SOTA-417	Filter pretraining text by how well the tokenizer compresses it	Geiping and Goldstein (2023), LIT-681. Drop every document whose token count exceeds t times its character count. Text a tokenizer trained on the corpus compresses badly is mostly markup, code fragments and debris. No model is needed. On C4 it lifts GLUE from 75.9 to 79.3, and on already-clean bookcorpus-wikipedia from 78.1 to 78.7. The gain tracks how dirty the source was.	Proposed
SOTA-418	To drop the tokenizer without losing to BPE, use a two-stage hierarchy with learned chunk boundaries and SSM encoder and decoder layers, not an isotropic byte model	Hwang, Wang and Gu (2025), LIT-679. A byte-level model that must not lose to a BPE Transformer at matched data and FLOPs should be hierarchical. Isotropic byte stacks lose by about 9 downstream points. It should use Mamba-2 layers for the encoder and decoder around the main network, and it should learn its chunk boundaries across two stages. One learned stage only ties BPE and whitespace chunking. Expect about 2× slower training.	Proposed
SOTA-419	Under a fixed compute budget, change the architecture only where it cuts step time at constant parameter count	Geiping and Goldstein (2023), LIT-681. Under a 24-hour, one-GPU budget, every reshape tried (depth 4 to 24, deep-narrow, wide, funnel, FFN-every-k, shared layers) finishes within about 0.1 of the same MLM loss. A smaller model’s extra throughput cancels its slower per-token learning. So the architecture changes that pay are the ones that make each step cheaper without changing how much the model learns per token: dropping biases, predicting only masked tokens, and pre-norm for the learning rate it permits.	Proposed
SOTA-420	When distilling a classifier, match the teacher’s temperature-softened outputs at the same temperature and keep a down-weighted hard-label term	Hinton et al. (2015), LIT-680. Soft-target cross-entropy at temperature T in both teacher and student, plus a hard-label cross-entropy at T = 1 with a considerably lower weight. Multiply the soft term by T², so the balance does not move when T changes. Start low for a small student. The paper’s best realistic setting was T = 2, and a 30-unit MNIST net wanted 2.5–4.	Proposed
SOTA-421	Before fitting a compute-optimal allocation law, tune learning rate, batch size and AdamW beta2 per model size and keep warmup short of the smallest run	Porian et al. (NeurIPS 2024), LIT-690. In a small-scale sweep that will be fitted for N*(C), a warmup longer than the smallest models’ optimal runs moves the exponent by 0.10. So does one batch size, learning rate and β₂ shared across sizes. Tune them per size, with β₂ above 0.95 at small batch, and cap warmup (the source uses warmup tokens = N, at most 20% of the budget). A constant learning rate is enough to fit the exponent, at about half the cost of a per-budget cosine.	Proposed
SOTA-422	Rescale the noise schedule so the last timestep is pure noise, and train with v-prediction so the model can learn there	Lin et al. (WACV 2024), LIT-689, with Emu Video’s comparison, LIT-635. Common VP schedules leave signal at t = T. Stable Diffusion’s leaves √ᾱ_T = 0.068. The model learns to keep the leaked channel mean, and pure-noise inference then limits brightness. Rescale √ᾱ_t so it reaches 0 at T. Then train with v-prediction, because at zero SNR ε-prediction is trivial and v_T = x₀. Sample in the x₀ form, because converting v to ε at zero SNR divides by zero.	Proposed
SOTA-423	When classifier-free guidance over-exposes a latent diffusion model, rescale the guided prediction toward the conditional prediction’s standard deviation	Lin et al. (WACV 2024), LIT-689. Compute x_cfg as usual, rescale it by std(x_pos)/std(x_cfg), and blend with weight φ back toward x_cfg. The source uses φ ≈ 0.7 at w = 7.5. It is the latent-space counterpart of Imagen’s dynamic thresholding, which needs a known pixel range. The evidence is qualitative, and part of the effect may simply be lower effective guidance.	Proposed
SOTA-424 v4	Train one network for both conditional and unconditional scores by dropping the condition on 10% of examples, then pick the guidance weight by which metric you are willing to lose	Ho and Salimans (2022), LIT-693. Replace the conditioning with a null token on a fraction of training examples — 0.1 is the measured best, with 0.2 and 0.5 worse — so one network learns both scores, then extrapolate away from the unconditional estimate at sampling. The weight is not a quality dial: on ImageNet 64×64, w from 0.1 to 4.0 takes FID from 1.55 to 26.22 and IS from 66.11 to 260.2. Choose it by which of those you are prepared to sacrifice, and never compare two models at different weights.	Active
SOTA-425 v2	Report precision and recall beside FID whenever the generator has a fidelity-diversity knob, because FID’s best value sits in the interior of that trade	Dhariwal and Nichol (2021), LIT-699. FID mixes fidelity and diversity, so on any knob that trades one for the other its optimum is interior and a single FID cannot say which side of the trade you are on. Their own Table 5 is the demonstration: ADM-G beats BigGAN-deep on FID at ImageNet 128, 256 and 512 while losing precision at all three (0.78/0.86, 0.82/0.87, 0.87/0.88) and winning recall by 0.24, 0.24 and 0.13. Report precision and recall, state the knob’s value, and never read a single FID as “better samples”.	Active
SOTA-426	When you merge and reinitialise a low-rank adapter mid-training, prune the optimizer state and re-warm the learning rate from zero — both, not either	Lialin et al. (2023), LIT-104 — ReLoRA’s restart discipline, read as NOTE-024. At every merge-and-reinit, prune the adapter’s Adam state (99% by magnitude) and drop the learning rate to zero and re-warm it over 50–100 steps. In the 130M ablation (Table 6) the restart alone does nothing (34.25 vs LoRA’s 34.17), the re-warm without the reset does nothing (34.29), the reset without the re-warm diverges, and only the pair helps (29.77). A rule for how to restart, not a recommendation to restart.	Proposed
SOTA-427	In a batch-normalized network, put dropout after the last batch-norm layer, not upstream of one	Li et al. (2018; CVPR 2019), LIT-705. BN freezes a moving variance accumulated while dropout was scaling activations. At test time dropout stops, the variance changes, and every later BN normalizes by the wrong constant. The cost is large: 77.42% → 68.55% on CIFAR-100 for DenseNet with dropout 0.5 in each bottleneck. The fix, one dropout layer after the last BN, is supported mainly as avoiding that cost. What it adds on its own is about 0.2 top-1 on ImageNet, with no spread reported.	Proposed
SOTA-428	Choose the EMA length after training: store two power-function averages at snapshots and reconstruct any length by least squares	Karras et al. (2023), LIT-714 — EDM2. Track two power-function averages of the weights (σ_rel 0.05 and 0.10), save both every few thousand steps, and synthesize any EMA length after the run from a small linear solve. Then choose it per configuration, per guidance weight and per metric, because on ImageNet-512 the best length moves with all three — at guidance 1.4, FID wants 2% and FD_DINOv2 wants 14%.	Proposed
SOTA-429	Drop half the image patches when training a CLIP-style model, spend the saving on more pairs and a larger batch, and unmask only for a short final tune	Li et al. (2022), LIT-706 — FLIP. Remove 50% of image patches and run the ViT on the rest, with CLIP’s loss and nothing else. The saving buys a 2× larger batch at the same memory and 2× the samples per hour; ViT-L/16 on LAION-400M reaches its unmasked reproduction’s accuracy more than 3× faster. The accuracy comes from what the saving buys: at equal batch, masking is parity.	Proposed
SOTA-430 v8	Before using an attribution map to debug a model or explain what it learned, check that the map changes when the weights are randomized and when the labels are permuted — do not validate it by how it looks	Adebayo et al. (2018), LIT-713, read as NOTE-365, with Kokhlikyan et al. (2021), LIT-724. An attribution map that survives re-initializing the model’s weights, or retraining on permuted labels, cannot be telling you about the weights or the labels. Guided Backprop and Guided GradCAM survive the first above the lowest layers, and their maps look as convincing as ever. A rejection rule, not a certificate: passing does not show a method is faithful — GradCAM passes and agrees with occlusion maps at rank correlation 0.254. The verdict is per-modality — the same test on a BERT text classifier reverses it for global Integrated Gradients — and pass or fail is read from curves with no threshold anywhere.	Proposed
SOTA-431	Make every layer of a diffusion denoiser preserve activation magnitude, and force each weight vector back to fixed norm after every step	Karras et al. (2023), LIT-714 — EDM2. Normalize each output channel’s weight vector on use, re-normalize the stored weights after every step, scale every fixed operation to preserve unit magnitude, and drop the group norms. On the ADM U-Net at ImageNet-512 this takes FID from 6.96 to 2.56 at equal compute, with on-use weight normalization alone worth 6.96 → 3.75. One architecture family; untested on transformers.	Proposed
SOTA-432	Maintain two power-function weight averages during training and snapshot them, so the averaging length is a post-hoc sweep instead of a guess you have to make before the run	Karras et al. (2023), LIT-720. Keep two power-function averages of the weights during training (σ_rel 0.05 and 0.10) and store both in each periodic snapshot; afterwards, reconstruct the average for any averaging length by a least-squares fit over the stored profiles, with error falling as O(1/n⁴) in the snapshot count. The averaging length stops being a pre-training commitment. Worth it because that commitment is expensive in a way that does not show up in its own column: with the length pinned at 13%, the learning-rate decay moves FID by up to 72% across t_ref ∈ [30k, 160k]; swept post hoc, the same bracket is within 10% of the optimum.	Superseded — by SOTA-428; A duplicate of SOTA-428, which was filed from LIT-714 hours earlier and is the same recommendation from the same paper. SOTA-428 is also the better document: it carries the architecture, learning-rate, guidance and metric dependences, where this one had only the first two. The Active/unreplicated pairing this note argued for is a judgement SOTA-428 declined, at Proposed, and that is the one that stands.
SOTA-433	Guide a diffusion model with a smaller, less-trained copy of itself rather than an unconditional model, and make sure the two models are degraded in the same way	Karras et al. (2024), LIT-721. Replace classifier-free guidance’s unconditional reference with a degraded copy of the same conditional model — smaller and under-trained, same task, same conditioning, same data. The quality gain survives and the diversity loss does not: ImageNet-512 FID 2.56 → 1.34 (EDM2-S), 1.25 (XXL), 1.01 at ImageNet-64, and unconditional EDM2-S 11.67 → 3.86, which CFG cannot do at all. The binding condition is compatibility: degrade the two models differently and the method stops working entirely.	Proposed
SOTA-434	In a vision backbone, carry position with a relative bias inside the attention window and do not add an absolute position embedding on top	Liu et al. (2021), LIT-723, Table 4. One architecture, one recipe, three tasks. A relative position bias inside the attention window is best or tied-best on all three. An absolute position embedding is worth +0.4 ImageNet top-1 and -0.2 COCO box AP and -0.6 ADE20K mIoU, and adding it on top of the relative bias costs 2.1 mIoU while changing ImageNet nothing. The recommendation is the sign, not the size: this is the one design choice in the table that a classification benchmark scores backwards.	Proposed
SOTA-435 v2	Never rank attribution methods by a single faithfulness or stability score, and report the perturbation distribution when you report infidelity	Yeh et al. (2019), LIT-725. Max-sensitivity is minimised by a constant explanation — an explanation that ignores the input entirely wins it — so it can never be a quality score on its own. Infidelity is defined relative to a perturbation distribution you choose, and Propositions 2.2–2.5 show several published methods are each the optimum for some choice, which the paper confirms by giving SHAP the lowest infidelity under SHAP’s own perturbation. Both cautions are free to follow: report the pair, and name the perturbation.	Active
SOTA-436	Route a very large or non-grid input through cross-attention into a small learned latent array, so that depth no longer scales with input size	Jaegle et al. (2021), LIT-747 — the Perceiver. A learned array of 512 latents cross-attends to all 50,176 ImageNet pixels, at a cost of O(MN) rather than O(M²), and a 48-block latent transformer then runs at a cost independent of input size. It matched ResNet-50 and ViT-B/16 at 78.0% top-1 without convolutions, did not beat them, and ran unchanged on audio, video and point clouds. The case for doing it is cost and modality-independence, not accuracy, and the cost advantage is argued from complexity, not measured against a baseline.	Proposed
SOTA-437	Store and move model checkpoints losslessly compressed: code the float exponent as its own stream with an entropy coder alone, and store successive checkpoints as deltas against a periodic full base	Hershcovitch et al. (2024), LIT-750 — ZipNN. In regular trained weights only the float exponent is redundant: about 40 of its 256 values occur and it codes to about a third, while sign and mantissa barely compress. So split the exponent into its own stream and Huffman-code it without an LZ stage. Llama-3.1 BF16 goes to 66.4% against Zstd’s 77.7%, and is faster in both directions. Between checkpoints, XOR deltas against a base up to 10 checkpoints back compress better than standalone files. Sizes and throughputs are measured. Stall time inside a training run is not.	Proposed
SOTA-438	To pretrain a wider language model when a smaller pretrained one of the same depth exists, initialize it by function-preserving symmetric width expansion rather than at random	Samragh et al. (2024), LIT-765 — HyperCloning. Tile each weight matrix of the small model into n×n blocks scaled by 1/n, so the wide model computes exactly the small model’s function at step 0, then train as usual. On three 2× pairs from 350M to 2.9B, it reached the random-init baseline’s final 10-task accuracy 2.2–4× sooner and finished higher. The source model’s pretraining is not charged, so this supports reusing a checkpoint that already exists, not training a small model in order to clone it.	Proposed
SOTA-439	Test a Kronecker-factored curvature implementation against the cases where KFAC is exact before trusting it	Dangel et al. (2025), LIT-768, §5.2 — a tutorial, so the evidence is a derivation, not a training run. KFAC’s averaging step is exact whenever the layer inputs or the backpropagated vectors do not depend on the data point. That gives two tests: a single data point, where KFAC equals the GGN and the empirical Fisher exactly and the Monte-Carlo flavour converges to the GGN; and a deep linear network under square loss on arbitrary data, where the GGN and Monte-Carlo identities still hold and the empirical-Fisher one does not. The second is the one the paper says catches scaling bugs.	Proposed
SOTA-440	Supervise a feed-forward 3D reconstructor with a frozen camera-controlled video model’s renderings instead of captured multi-view data	Bahmani et al. (2025), LIT-743 — Lyra trains a 3D Gaussian Splatting decoder only on a frozen video model’s RGB decodings along six camera paths per image, with no captured multi-view data. In its one controlled arm, training on real RealEstate10K + DL3DV instead scores PSNR 19.08 against 24.77, and adding the real data to the generated data gives 24.74, no better. Both are scored against the teacher’s own renderings on its own prompts, which is the distribution the generated supervision was drawn from — so the arm shows the generated data is sufficient there, not that it beats captured data on real scenes.	Proposed
SOTA-441	When a web-scraped pair corpus is noisy, filter it by self-consistency: keep only the pairs whose passage a model trained on the noisy set ranks near the top	Wang et al. (2022), LIT-744, Table 7. Train a dual encoder on the noisy pairs, rank each pair’s passage against 1M random passages, and keep the pair only if it lands in the top 2. On six BEIR datasets, filtered beat unfiltered at 1M pairs (40.7 against 34.9 nDCG@10) and with all pairs (51.6 against 50.0), though the filtered set is about a quarter the size. One cutoff, chosen by inspection and never varied, and the filtering model’s cost is not counted.	Proposed
SOTA-442	To RL-tune a flow-matching generator, roll it out as a same-marginal SDE built from its own velocity, and anchor each step with the closed-form KL to the reference	Liu et al. (2025), LIT-779. A flow model’s ODE sampler is not a stochastic policy. Swapping it for an SDE with the same marginals, built from the trained velocity alone, makes each step an isotropic Gaussian, so GRPO’s ratio and the KL to the reference model are both closed form. On SD3.5-M, GenEval goes 0.63 → 0.95 either way. Without the KL term, DrawBench aesthetic falls to 4.93 from a base of 5.39, against 5.25 with it (Table 2). Quality is judged by reward models only, one run each.	Proposed
SOTA-443	Down-weight large residuals in consistency and MeanFlow training: use a Pseudo-Huber loss or an adaptive inverse-error weight, not plain squared L2	Song and Dhariwal (2023), LIT-786, and Geng et al. (2025), LIT-784. When the regression target is the model’s own output under stop-gradient, squared L2 trains worse than a loss that limits the pull of large errors. iCT’s Pseudo-Huber beats squared L2 on CIFAR-10 (Fig. 2, curves). MeanFlow’s adaptive weight 1/(‖Δ‖² + c)^p takes one-step ImageNet-256 FID from 79.75 at p = 0 to 61.06 at p = 1 (Table 1e). Two groups, small settings, one run per arm.	Proposed
SOTA-444	When a flow will be sampled with a coarse ODE solver, pair noise and data within each minibatch by exact optimal transport	Pooladian et al. (2023), LIT-789, and Tong et al. (2023), LIT-798. Re-pair each minibatch’s noise and data by exact discrete OT before the flow-matching regression. The marginals stay exact, the learned paths straighten, and on ImageNet-64 reaching FID 20 with Euler takes 12 steps instead of 29, at equal likelihood and 4% more time per iteration. It lowers the step count of an ODE sampler. It does not make one: four-step FID stays near 39 on ImageNet-32. Unconditional, 64 px and below.	Proposed
SOTA-445	When the base text-to-image diffusion model is better than the models that generated the preference pairs, train on the pairs with Diffusion-DPO rather than fine-tuning on the preferred images	Wallace et al. (2023), LIT-774, §5.5 and Supp. S6. Pick-a-Pic’s images were generated by SDXL-beta and Dreamlike, and SDXL-1.0 is better than both. Supervised fine-tuning on the preferred images helped SD1.5 (55.5% win rate over its base) and made SDXL worse at every learning rate tried. Diffusion-DPO on the same pairs improved SDXL, preferred to the untuned base 70.0% of the time on PartiPrompts. The SDXL SFT arm has no number, there is one run per setting, and no one else has repeated it.	Proposed
SOTA-446	In consistency training, take the target from the current weights under stop-gradient, not from an EMA teacher	Song and Dhariwal (2023), LIT-786, §3.2. Consistency Models trained the student against an EMA of itself. iCT shows that on a point-mass data distribution the limiting objective with any teacher other than the student carries no information about the data (Prop. 1), and that setting the teacher’s EMA rate to zero improves CIFAR-10 FID with both LPIPS and squared L2 (Fig. 2a, curves). Keep an EMA of the weights for evaluation. Do not use it for the target.	Proposed
SOTA-447	When distilling a large image or video model by continuous-time consistency, add a small DMD distribution-matching term	Zheng et al. (2025), LIT-783. Pure sCM distillation of 2B–14B text-to-image and text-to-video teachers loses small text, texture and temporal coherence. Adding DMD’s loss at weight 0.01, on the student’s own 1–4-step rollouts with a fake-score network updated 5–10 times per step, fixes it in samples. Four-step Wan2.1-1.3B VBench is 84.43 against a re-implemented DMD2’s 84.56. sCM’s own Fig. 7 (LIT-790) is why the consistency term stays primary: distribution matching alone loses recall.	Proposed
SOTA-448	Without a pretrained teacher, train a one-step generator as a MeanFlow average-velocity model rather than by consistency training or a shortcut model	Geng et al. (2025), LIT-784. Train a network for the average velocity over an interval, regressed onto v − (t − r)·du/dt under stop-gradient, with du/dt from a Jacobian-vector product. From scratch on ImageNet-256 with DiT-XL/2, one-step FID is 3.43 after 240 epochs, against 10.60 for a shortcut model and 34.24 for iCT as Inductive Moment Matching re-ran it (Table 2). Those rows are copied, at other budgets. No arm compares distilling a teacher.	Proposed
SOTA-449	When reward-tuning a classifier-free-guided diffusion model over several rounds of sampling, train the guided prediction at the fixed sampling guidance weight, not the conditional prediction alone	Black et al. (2023), LIT-776, App. E.1 and Fig. 10. Reward fine-tuning should not train the unconditional branch, since the reward depends on the prompt. Training the conditional prediction alone made performance deteriorate after the first round of fine-tuning, which the authors attribute to the guidance weight becoming miscalibrated. Fixing the weight (5 in their runs) and training through the guided prediction used for sampling fixed it. One curve per arm, on reward-weighted regression and JPEG compressibility.	Proposed