Topics — The twenty-eight kinds of claim this anthology files against — the primary axis of both the practice registry and the reading list, and the same twenty-eight for each (ADR-026). The filing rule is the point: take a domain topic when the claim is ABOUT the domain; a claim merely discovered in one still takes its kind. The axis organizes the record, it does not bound it: content the list cannot place is evidence the list is short (ADR-059).
Analysis and evaluation — how to find out whether something worked — what to measure, what a measurement cannot tell you, and which comparisons are unsound; theory, interpretability and debugging belong here too.
70 of 449 SOTA documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| SOTA-011 v3 | Map the Hessian ratio |lambda_min / lambda_max| to find where the loss surface is non-convex | Li et al. (2017), LIT-014 — ARXIV-1712.09913. | Active |
| SOTA-012 v3 | sharpness in the loss landscape correlates with test error | Li et al. (2017), LIT-014 — ARXIV-1712.09913. | Active |
| SOTA-194 | When an objective arrives with its own dataset, state the composition before crediting the objective | Active | |
| SOTA-196 v2 | Report zero-shot and in-distribution performance separately; one hyperparameter can move them in opposite directions | Active | |
| SOTA-197 v2 | Account for test-set proximity to the training data when evaluating, and state your contamination exposure | Active | |
| SOTA-200 v3 | Check whether an emergent capability is a metric artefact before believing it | Active | |
| SOTA-210 v3 | Report pass@k as well as pass@1 after post-training: reinforcement learning raises one and lowers the other | Ba et al. and Hayes et al. (2026), LIT-230 and LIT-234. GRPO finishes below its own base model on pass@16 and pass@32 in 15 of 18 comparisons while improving pass@1; across Qwen2.5, Qwen3 and published RL checkpoints up to 32B the base model overtakes the RL checkpoint at large k. A post-training result reported at pass@1 alone cannot distinguish a model that learned something from one that stopped trying anything else. | Active |
| SOTA-241 | Rank candidate data selections with a model-free distributional proxy before spending a training run | Xie et al. (2023), LIT-087 — KL reduction in a hashed n-gram space correlates with downstream performance across selection methods, including manual curation, which uses no n-grams at all. A selection can therefore be ranked against its alternatives before anything is trained on it. | Proposed |
| SOTA-245 | State a relation in both orders in the training corpus if you want it usable in both directions | Grosse et al. (2023), LIT-401 — a training sequence influences a completion only when the phrase related to the prompt comes before the phrase related to the completion. Identical content in the reverse order scores barely above an unrelated baseline. | Proposed |
| SOTA-246 | Find mislabelled training data by self-influence, not by training loss | Koh and Liang (2017), LIT-403, with LIT-400 — rank training points by their influence on their own loss and inspect from the top. Mislabelled examples are strong proponents of themselves, so they sort to the front; high training loss does not separate them nearly as well. | Proposed |
| SOTA-256 | Judge a monotone scaling recipe by its asymptote, not by its loss at a compute budget | Kim et al. (2025), LIT-441 — when compute is not the binding constraint, a recipe’s loss at a chosen budget answers a question nobody asked. Fit the scaling law and compare the asymptote instead. The catch is that an asymptote is an extrapolation, and rankings at small scale can invert: the ensembling recipe wins on asymptote while losing at small member counts. | Proposed |
| SOTA-270 | Do not read a smooth loss curve as evidence of smooth training; decompose it when the answer matters | Kangaslahti et al. (2025), LIT-455, with Cohen et al. LIT-453 and Kunin et al. LIT-454 — the curve time-averages oscillation, sums over differently-timed abrupt transitions, and reads flat while the weights still travel. Smoothness is what many breakthroughs look like added up, so it is not evidence about the training. | Active |
| SOTA-278 | Before reporting that a model cannot do something, rule out the evaluation | Hu and Frank (2024), LIT-465 — the same capacity asked two ways scores differently, and the gap widens as the model weakens. Lawsen (2025), LIT-463, is the worked example: an output cap, a scorer that cannot express refusal, and instances with no solution. | Active |
| SOTA-284 v2 | Choose the checkpoint you post-train from by how much mass it puts on good rare responses, not by validation cross-entropy | Proposed | |
| SOTA-286 | To find a model that generalizes badly where its outputs look fine, measure how much of parameter space behaves the way it does | Proposed | |
| SOTA-290 | To measure what a deployed recommender does, use sock-puppets that never interact, follow sets curated to your question, and exposure weighted by rank | Proposed | |
| SOTA-291 v2 | Evaluate a specific intervention on a specific affordance rather than the net effect of the system, and never read an individual-level null as the absence of a collective effect | Active | |
| SOTA-292 | Check industry ties yourself when reading a literature about deployed platforms — disclosure is unreliable and the ties are mechanically detectable | Proposed | |
| SOTA-293 | Count embedding parameters when you report a tiny model’s size | Finke et al. (2025), LIT-485 — at these sizes the embedding table is not a rounding error, it is most of the model: TinyStories-33M is 68M parameters all-in, and an 8M model in the same paper is “more than half embeddings”. Two papers can each claim the smaller model and both be telling the truth. Print the all-in figure. | Proposed |
| SOTA-298 | Measure a writing assistant by the outcome it was deployed to improve, not by the text features that used to predict it | Corpus et al. (2025), LIT-487 — Change.org’s AI drafting tool made petitions 49% longer, more lexically varied and better rated by human judges on quality and persuasiveness, moving them toward the profile of successful pre-AI petitions. The share reaching 10 signatures fell 5.33 points. The features had been chosen because they predicted success; afterwards they predicted it weakly or negatively. | Proposed |
| SOTA-300 | To tell whether a model is doing non-trivial in-context computation, measure how well it predicts its own hidden states, not its next-token loss | Herrmann et al. (2025), LIT-489 — uniform noise maximizes next-token loss and requires no computation; reciting a memorized licence minimizes it and requires none either. Insert a variational bottleneck with a learned autoregressive prior and measure the posterior-prior KL: only the genuinely in-context task scores high, on a Transformer and an LSTM, and the signal survives controlling for next-token loss. | Proposed |
| SOTA-303 | Name the phase a time-saving tool accelerates before predicting what it will do to the quality of what people produce with it | Duede, Gross, Crockett and Bergstrom (2026), LIT-492 — in a marginal-value-theorem model, accelerating the phase before value is known makes producers more selective, accelerating the minimum cost of shipping makes them less selective, and both make each item less thoroughly developed — while accelerating discretionary improvement makes each item more thoroughly developed. Opposite signs from the same tool depending on where it acts, so “it saves time” does not determine the direction. | Proposed |
| SOTA-304 | Generate a long program unit by unit, freezing each unit that passes an execution-based check, instead of regenerating the whole artifact on failure | Cheng, Clark and Richardson (2025), LIT-493 — on 100 held-out proposals, the same model produced valid implementations 92% of the time generating unit by unit against an execution-based checker, and 6% of the time one-shot with retries; the one-shot outputs were also trivial, 49 lines against 181. Removing the checker alone costs 62 points, removing the decomposition alone costs 19 — so if you build one, build the checker. | Proposed |
| SOTA-305 v2 | State the token count and input resolution beside any reconstruction FID, and compare tokenizers only at equal rate | Sun et al. (2024), LIT-497 — the same tokenizer at downsample 16 scores rFID 2.19, 0.94 and 0.70 on 256, 576 and 1024 tokens, all evaluated on 256×256 reconstructions. A three-fold range from a knob that is not the tokenizer. Two papers in this line then quote reconstruction numbers whose rate a reader cannot recover from the sentence they appear in. | Active |
| SOTA-307 v5 | Report generative FID as an error bar over several training seeds, and treat any gap below about 2% of the mean as inconclusive | Dufour, Efros and Pérez (2026), LIT-501 — on several hundred SiT networks, retraining the same recipe moves FID 3.2× more than resampling from a fixed model, and the coefficient of variation holds at 1–2% across four model sizes and every compute budget to 2M steps. Ten times the sampling budget shrinks the small term by √10 and leaves the large one untouched. A lucky training seed is worth up to 2× the compute. | Active |
| SOTA-308 | When a retrieval change raises recall, measure relevance and the end-task metric before calling it an improvement | Wu et al. (2026), LIT-502 — across three published GraphRAG pipelines on G-Medical, graph expansion raises retrieval recall (GFM-RAG 84.3% against vanilla RAG’s 71.8%) and drops context relevance (38.5% against 62.9%). The wider context is noisier, and generation accuracy falls with it — a system that retrieves more of the right thing and much more of the wrong thing answers worse. | Active |
| SOTA-309 | Score a control interface by the mutual information between the operator’s command and the state change it induces, and choose the horizon of that state change deliberately | Reddy, Levine and Dragan (2022), LIT-503 — whatever the operator is trying to do, a better interface yields less noisy commands, so I(x_t, (s_t, s_{t+Δ})) scores it with no labels, no reward and no task knowledge. Spearman ρ = 0.43 against true task completion across 540K examples in 4 of 5 domains. In the fifth the correlation was strongly negative at Δ = 1 and positive at episode length — so the horizon is not a detail. | Proposed |
| SOTA-310 | Treat a mechanistic explanation that passes circuit error or causal alignment as one of many, and report what you did to rule the others out | Méloux, Maniu, Portet and Peyrard (2025), LIT-504 — by exhaustive enumeration on small MLPs, no network had exactly one circuit interpretation and under 2% had exactly one valid minimal mapping. The median number of explanations satisfying the criteria rises from 38 to 910,000 as width goes 2 → 5, and both counts are lower bounds. Passing circuit error or IIA establishes that an explanation is admissible, not that it is the explanation. | Active |
| SOTA-311 | Evaluate a search on tasks held out of its own fitness function, and report the gap against the tasks it selected on | No source states this; it is filed on a priori grounds with one instance. Cheng, Clark and Richardson (2025), LIT-493, searched 1,062 pretrained designs against a fitness function of average downstream accuracy, then reported the top five on nine benchmarks drawn from the same pool and chosen — per its own Table 16 — for largest standard deviation across the search population. The best discovered design beats the best seed by 0.03 points. | Active |
| SOTA-312 v2 | State the noise or binning assumption behind any mutual information you report for a deterministic network, and show the conclusion survives changing it | Saxe et al. (2018), LIT-509 — in a deterministic network I(h;X) is infinite, so every finite number on an information plane is a property of a noise model the analyst imposed and the network never had. The same tanh run, binned evenly in net input instead of evenly in activity, loses its compression phase entirely; and LIT-507 makes compression appear in ReLU networks by changing the binning the other way. | Active |
| SOTA-313 | Treat the inference-time decision procedure as part of what you are comparing, and report the ordering under each one | Gemini Team (2023), LIT-510 — on MMLU, greedy sampling puts GPT-4 ahead by 0.2, plain chain-of-thought at 32 samples puts it ahead by 2.3, and uncertainty-routed CoT at 32 samples puts Gemini Ultra ahead by 2.7. Same models, same benchmark, same harness. The procedure is worth 6.0 points to one model and 3.1 to the other, so running the same procedure on both is not enough — the ordering is a property of the procedure you chose to report. | Active |
| SOTA-315 v2 | Early-stop and tune on validation loss after temperature scaling, then calibrate post hoc, instead of stopping on raw validation loss | Berta et al. (2025), LIT-514 — a proper loss is calibration error plus refinement error, and the two bottom out at different epochs, so the validation-loss minimum is optimal for neither. Fit a temperature on the validation set each time you would have read the loss, read the loss after it, and stop there; then keep the temperature. Measured across 196 classification datasets and a vision benchmark with ten runs per dataset. | Active |
| SOTA-317 | Do not rank singular directions by magnitude when deciding what to discard; check the small end, and check it after fine-tuning | Staats, Thamm and Rosenow (2024), LIT-517 — in a transformer’s non-square matrices, the smallest singular directions carry data directions and their removal is catastrophic: Llama-3 8B on GSM8K falls from 43.2% to 2.0% when the smallest decile of the Down-Projection goes, against 40.0% for the same decile of the square Attention-Output. And the damage only appears if you prune after fine-tuning. | Active |
| SOTA-321 | Verify a model’s lineage from the singular spectra of its attention products, which survive permutation and rescaling | Wang et al. (2025), LIT-524 — compare the per-layer singular spectra of W_q W_k^T and W_v W_o. Those products absorb the permutation and rescaling that change weights without changing the function, so the spectra identify a model through fine-tuning, pruning, merging and expansion. F1 0.9867 over 55 model pairs, using no data and no queries. | Proposed |
| SOTA-323 | Test a claim about what pretraining produces on a model trained with the pretraining objective, not one trained on the task family you are testing | Shen, Mishra and Khashabi (2023), LIT-536 — “there exist weights such that” and “pretraining produces weights such that” are different claims, and training a model on the task family you are about to test collapses them. Their separation of these as Hypothesis 2 and Hypothesis 1 is what let them show that a literature everyone read as evidence about pretrained language models had tested neither the models nor the objective. | Proposed |
| SOTA-324 | Separate rival accounts of an internal algorithm by convergence rate and conditioning, not by how well each fits the output | Fu, Chen, Jia and Sharan (2023), LIT-535 — output similarity cannot tell gradient descent from Newton’s method, because both converge to the same answer. Matching each layer to the best-fitting step count of each candidate gives a linear trend for the right rate and an exponential one for the wrong one; a regime where the rivals must differ — here, condition number 100 — separates them again. Two measurements, both cheap, both decisive where a fit was not. | Proposed |
| SOTA-325 | When the training loss has saturated and runs still differ, measure loss-landscape degeneracy rather than curvature | Lau, Furman, Wang, Murfet and Wei (2023), LIT-542 — on ResNet18/CIFAR10, stronger implicit regularization (higher learning rate, lower batch size, higher momentum) gives a lower local learning coefficient and higher test accuracy, while every training loss has collapsed to zero. The estimator is λ̂(w*) = n β* [E_{w|w*,β*,γ} L_n(w) − L_n(w*)] at β* = 1/log n, run by SGLD, validated against theory on deep linear networks to 100M parameters. Where the loss has stopped distinguishing runs, the geometry has not. | Proposed |
| SOTA-337 v4 | When an ImageNet classifier is anywhere in the pipeline — training the generator or steering its sampling — confirm FID gains with a Fréchet distance in a non-ImageNet feature space | Kynkäänniemi et al. (2022), LIT-563 — FID’s features are nearly ImageNet class logits, so matching ImageNet-class statistics lowers it without improving images. Resampling a fixed StyleGAN2’s outputs cuts FFHQ FID 5.30 → 1.78, while CLIP-space FD moves 2.76 → 2.64. A model whose discriminator uses ImageNet features exploited this by accident, and the authors suspect data filters and samplers can too. Report a CLIP or SwAV Fréchet distance beside FID whenever that is the case. Classifier guidance (LIT-699) is the sampling-loop instance and the least checked: it maximises an ImageNet classifier’s confidence on every step, and its paper denies producing adversarial examples in one clause without running a test. | Active |
| SOTA-342 | Report a sparse autoencoder’s fidelity as downstream loss in compute-equivalent terms, not as fraction of loss recovered against zero ablation | Gao et al. (2024), LIT-571 — splice the SAE’s reconstruction into the forward pass and report the resulting loss as the pretraining compute a model of that loss would need. Zero-ablating the residual stream is so destructive that “loss recovered” against it flatters any reconstruction. Their 16M-latent GPT-4 SAE scores 98.2% on it, and equals a GPT-4 trained on 10% of the compute. | Proposed |
| SOTA-343 | Read a transformer’s intermediate-layer predictions through a tuned lens, not the raw logit lens | Belrose et al. (2023), LIT-569 — decoding a hidden state with the final unembedding (the logit lens, LIT-570) fails outright on BLOOM, OPT and GPT-Neo, and is biased by 4–5 bits where it works. Train one affine translator per layer by KL to the final logits, initialized to the identity, and read the layer through that. It is cheap (d×d per layer), transfers to fine-tunes, and its directions are the model’s (Spearman 0.89). Train it with Muon. | Proposed |
| SOTA-344 | Train sparse autoencoders with a TopK activation instead of an L1 penalty, and prevent dead latents with transposed-decoder initialization and an auxiliary loss | Gao et al. (2024), LIT-571 — keep the k largest pre-activations and zero the rest, and train on reconstruction alone. L0 is then set, not tuned through an L1 coefficient that also shrinks the latents that fire. It beats ReLU SAEs on the MSE–L0 frontier by more as SAEs grow. Initialize the encoder as the decoder’s transpose and add a loss reconstructing the residual from dead latents: 7% dead at 16M latents, against up to 90%. | Proposed |
| SOTA-345 | Probe language-model activations with logistic regression on the raw activations, and credit sparse-autoencoder probes only against that baseline under validation-based selection | Kantamneni et al. (2025), LIT-568 — across 113 datasets, adding SAE-latent probes to a toolkit of raw-activation probes, choosing per task by validation AUC, changes test AUC by −0.003. It does not help under data scarcity, class imbalance, label noise or shift. Reported SAE wins came from baselines denied the same pooling or selection. Use logistic regression on activations, and hold any SAE method to it. | Active |
| SOTA-346 | Retune the training timestep shift for each autoencoder latent space, and never rank latent spaces for generation under one shared schedule | Black Forest Labs (2025), LIT-572 — four autoencoders, 30 timestep configurations each. The training shift alone moves FID by 61–86%. An RAE latent that wins when both are tuned loses to a tuned FLUX.2 latent when left unshifted. When you swap the autoencoder under a diffusion or flow model, sweep the shift again, using a logit-normal training distribution. When you compare autoencoders, compare each at its own optimum. | Proposed |
| SOTA-347 | To write many facts into a model’s weights, update a range of early-to-mid MLP layers jointly in one batched solve (MEMIT), choosing the range by measured edit success rather than causal tracing | Meng et al. (2023), LIT-576 — sequential single-fact edits collapse after tens of facts (ROME scores 50.3 at 10,000). Distributing each update over a range of MLP layers in one least-squares solve holds 85.8 on GPT-J, with neighborhood success falling from 83.5 to 73.7. Choose the layers by editing performance: per-fact causal tracing does not predict it (LIT-574). Evaluate implications too, because weight editors fail them (LIT-577). | Proposed |
| SOTA-348 | Evaluate a knowledge edit on the facts it implies — logical consequences, multi-hop compositions, aliases and the subject’s other facts — not only on the edited prompt, its paraphrases and unrelated neighbours | Cohen et al. (2023), LIT-577 — the standard edit metrics (efficacy, paraphrase, neighborhood) cannot see whether an edit changed what the model knows or only what it says to one question. On six ripple criteria, ROME, MEMIT and MEND average 38–66, and logical generalization falls as low as 5.5. Prompting with the new fact scores 81–83 on LLaMA-7B. Report ripple metrics beside the standard ones, and compare against in-context editing. | Active |
| SOTA-349 | Before crediting a graph transformer with long-range gains, retune the message-passing baselines under the same budget, including an MLP prediction head, positional encodings and input feature normalization | Tönshoff et al. (2023), LIT-579 — the Long-Range Graph Benchmark’s case that graph transformers are needed rested partly on untuned message-passing baselines. With a basic sweep inside the same 500k-parameter budget, GCN beats GPS on both Peptides tasks. Most of the gain is from replacing a linear prediction head with a 2-layer MLP. Normalizing input features lifts every model on the superpixel tasks, where GPS still leads. Tune both sides before attributing a gap to attention. | Active |
| SOTA-350 | Compare GNN architectures over many random data splits and seeds, with one shared training and hyperparameter-selection procedure for every model, never on a single fixed split | Shchur et al. (2018), LIT-581 — on the Planetoid split GAT beats GCN. On another random split of the same size GCN wins, and averaged over 100 splits × 20 seeds with identical training and tuning, GCN ranks first of four GNNs. A single split selects the model that overfits it, and per-model recipes measure the recipe. Report mean and spread over many splits and seeds, with the same tuning budget for every model. | Active |
| SOTA-353 | Train a head that predicts the model’s own accuracy on each output, against the metric you care about, and use it to rank samples and filter predictions | Jumper et al. (2021), LIT-583 — regress the per-residue lDDT the prediction will score (pLDDT) and the predicted TM-score. They track truth at r = 0.76 and 0.85 over 10,795 chains, and pLDDT filters the self-distillation set. Abramson et al. (2024), LIT-584, keep the head under diffusion by rolling out a cheap sample during training. They rank seeds by it, and antibody-interface quality keeps rising up to 1,000 seeds. | Proposed |
| SOTA-357 | Build a zero-shot classifier from prompt templates and ensemble them in embedding space | Active | |
| SOTA-366 | Monitor the standard deviation of the l2-normalized embedding; healthy is one over root d and collapsed is zero | Active | |
| SOTA-368 | When a model’s latent space is meant to be read by people, measure its interpretability with a human task such as word or topic intrusion, not by held-out likelihood | Chang et al. (2009), LIT-597 — topic models are used for their topics, but they are judged by held-out likelihood, which measures the probability of observations and ignores the representation. On NYT and Wikipedia, CTM had the best likelihood and the worst human scores. Test the latent space directly. Plant an improbable word among a topic’s top words, or an improbable topic among a document’s top topics, and measure how often people find it. | Active |
| SOTA-371 | Score topic coherence automatically as the average pointwise mutual information of a topic’s top-word pairs, estimated on a large external reference corpus | Newman et al. (2010), LIT-600 — when you cannot put every topic in front of people (SOTA-368), approximate their judgment. Take the topic’s top ten words and, for each of the 45 pairs, compute PMI from co-occurrence in 10-word windows over a large general corpus such as Wikipedia. Average the 45 scores. On 237 LDA topics this correlated with nine people’s ratings at ρ ≈ 0.77, far better than WordNet measures. | Proposed |
| SOTA-378 | Compare, project and orthogonalize a language model’s concept directions after whitening by the unembedding covariance, not by raw cosine | Park, Choe and Veitch (2024), LIT-606 — training fixes a model’s representation only up to an invertible linear map, so Euclidean cosine between concept directions is not meaningful. Use ⟨u, v⟩ = uᵀ Cov(γ)⁻¹ v, with Cov(γ) the covariance of the unembedding rows, or equivalently transform by Cov(γ)^(−1/2) and then use Euclidean tools. Shown on LLaMA-2 7B and Gemma-2B by heatmap. Contested by a null for cross-lingual transport in 17 models. | Proposed |
| SOTA-379 v2 | Before crediting a word-embedding method over count-based ones, give the baselines the same design choices and tune every method alike | Levy, Goldberg and Dagan (2015), LIT-607 — word2vec and GloVe ship with design choices (context smoothing, the negative-sample shift, dynamic windows, subsampling, adding context vectors) that count-based PPMI and SVD can use too. Port them and tune every method the same way before comparing. Done that way across 672 representations and 8 datasets, the prediction-over-count advantage disappears, SGNS beats GloVe on every task, and a single hyperparameter often matters more than the method. | Active |
| SOTA-380 | Use prediction-based word vectors rather than count-based distributional vectors | Baroni, Dinu and Kruszewski (2014), LIT-608 — across 14 benchmarks, word2vec CBOW beat PPMI, SVD and NMF count vectors almost everywhere, and the authors recommend “anybody interested in using DSMs” to “go for the predict models”. Retired: LIT-607 traces the gap to settings the count models were not given, and finds no consistent winner once they are. | Rejected — the advantage it reports disappears when the count-based baselines get the design choices word2vec shipped with and every method is tuned alike (LIT-607): no family wins consistently. Retired with no successor. What replaces it is an evaluation practice, SOTA-379, which corrects it |
| SOTA-383 | Fix the sample count before comparing FID values, and settle a close comparison with an unbiased estimator rather than a tighter error bar | Bińkowski et al. (2018), LIT-615 — the FID estimator is biased, the bias depends on the distribution being measured, and no unbiased estimator exists. So two FID numbers are comparable only at equal n, and a small standard deviation is not evidence of a converged estimate: at d=2048 and 50,000 samples their construction reversed a true ordering in 100 trials out of 100, with standard deviations of 0.2 and 0.5. | Active |
| SOTA-384 | Calibrate a trained classifier with a single temperature fitted on held-out data, not with a richer map | Guo et al. (2017), LIT-616 — rescale the logits by a single learned 1/T fitted for NLL on a held-out set. It cannot change the argmax, so accuracy is unchanged by construction, and it beats vector scaling, matrix scaling, histogram binning, isotonic regression and BBQ — including the two that strictly contain it. Typical uncalibrated ECE is 4–10%. | Active |
| SOTA-387 | Hold out a slice of the training set for model selection instead of selecting on the validation split | Beyer et al. (2022), LIT-628 — they train on the first 99% of ImageNet-1k and keep 1% as a “minival”, stated as being “to encourage the community to stop selecting design choices on the validation (de-facto test) set.” A convention, asserted rather than measured, and the record held no document on it. | Proposed |
| SOTA-388 v2 | Train a plain ViT on ImageNet-1k with average pooling, fixed 2D sin-cos positions, a 1024 batch and light augmentation | Beyer et al. (2022), LIT-628 — five changes, none novel, take ViT-S/16 on ImageNet-1k from 66.8% to 76.5% at 90 epochs and 80.0% at 300. Global average pooling, fixed 2D sin-cos position embeddings, batch 1024, RandAugment at level 10 and Mixup at p = 0.2. 6h30 on a TPUv3-8 for the 90-epoch run. | Active |
| SOTA-391 | Estimate attention resolution before training when choosing a position encoding, and measure it across the extrapolation boundary | Sun et al. (2022), LIT-638 — attention resolution scores how well an attention pattern distinguishes token distance, and is estimable from the encoding before a run. Measured at the training length and twice it: RoPE 0.91 → 0.08, ALiBi 0.81 → 0.88. The record has four practices about rotary encoding and held no measure of this. | Proposed |
| SOTA-404 v2 | When you ablate an auxiliary loss, ablate the input construction that came with it, or the result is about both | Liu et al. (2019), LIT-671. BERT reported that removing next sentence prediction hurts; RoBERTa observed that the ablation had probably “only removed the loss term while still retaining the segment-pair input format”, separated the two, and found removing the loss matches or slightly improves while the input format was what mattered. LIT-667 is the same defect in another literature: four grokking papers varied the training fraction of a fixed universe, so dataset size and dataset composition moved together, and the conclusion reversed when somebody turned them one at a time. | Proposed |
| SOTA-413 v2 | In a scaling-law study, count the output head in parameters and FLOPs, and fit loss against compute with an irreducible-loss offset | Pearce and Song (TMLR 2024), LIT-688. At small scale the embedding and output head are a large share of the model. Excluding them from N, and from the FLOPs in C = 6ND, biases the fitted allocation exponent: 0.49 becomes 0.74 when the same runs are relabelled. An offset-free loss–compute fit biases that exponent too. Do not compare exponents across studies that count differently. | Active |
| SOTA-415 v3 | During a warmup-stable-decay run, estimate the annealed score from a uniform average of recent stable-phase checkpoints instead of launching a decay branch | Li et al. (2025), LIT-684, extending Hägele et al. (2024), LIT-145. To see where a WSD run would land if annealed now, evaluate the simple average of its last ~10 stable-phase checkpoints. On the one fork measured, that proxy sits within about 1.5 points of a real 250B-token anneal on each benchmark. It costs no training. Still anneal the model you ship: on that fork the anneal is ahead on MMLU at the end. | Proposed |
| SOTA-421 | Before fitting a compute-optimal allocation law, tune learning rate, batch size and AdamW beta2 per model size and keep warmup short of the smallest run | Porian et al. (NeurIPS 2024), LIT-690. In a small-scale sweep that will be fitted for N*(C), a warmup longer than the smallest models’ optimal runs moves the exponent by 0.10. So does one batch size, learning rate and β₂ shared across sizes. Tune them per size, with β₂ above 0.95 at small batch, and cap warmup (the source uses warmup tokens = N, at most 20% of the budget). A constant learning rate is enough to fit the exponent, at about half the cost of a per-budget cosine. | Proposed |
| SOTA-425 v2 | Report precision and recall beside FID whenever the generator has a fidelity-diversity knob, because FID’s best value sits in the interior of that trade | Dhariwal and Nichol (2021), LIT-699. FID mixes fidelity and diversity, so on any knob that trades one for the other its optimum is interior and a single FID cannot say which side of the trade you are on. Their own Table 5 is the demonstration: ADM-G beats BigGAN-deep on FID at ImageNet 128, 256 and 512 while losing precision at all three (0.78/0.86, 0.82/0.87, 0.87/0.88) and winning recall by 0.24, 0.24 and 0.13. Report precision and recall, state the knob’s value, and never read a single FID as “better samples”. | Active |
| SOTA-428 | Choose the EMA length after training: store two power-function averages at snapshots and reconstruct any length by least squares | Karras et al. (2023), LIT-714 — EDM2. Track two power-function averages of the weights (σ_rel 0.05 and 0.10), save both every few thousand steps, and synthesize any EMA length after the run from a small linear solve. Then choose it per configuration, per guidance weight and per metric, because on ImageNet-512 the best length moves with all three — at guidance 1.4, FID wants 2% and FD_DINOv2 wants 14%. | Proposed |
| SOTA-430 v8 | Before using an attribution map to debug a model or explain what it learned, check that the map changes when the weights are randomized and when the labels are permuted — do not validate it by how it looks | Adebayo et al. (2018), LIT-713, read as NOTE-365, with Kokhlikyan et al. (2021), LIT-724. An attribution map that survives re-initializing the model’s weights, or retraining on permuted labels, cannot be telling you about the weights or the labels. Guided Backprop and Guided GradCAM survive the first above the lowest layers, and their maps look as convincing as ever. A rejection rule, not a certificate: passing does not show a method is faithful — GradCAM passes and agrees with occlusion maps at rank correlation 0.254. The verdict is per-modality — the same test on a BERT text classifier reverses it for global Integrated Gradients — and pass or fail is read from curves with no threshold anywhere. | Proposed |
| SOTA-433 | Guide a diffusion model with a smaller, less-trained copy of itself rather than an unconditional model, and make sure the two models are degraded in the same way | Karras et al. (2024), LIT-721. Replace classifier-free guidance’s unconditional reference with a degraded copy of the same conditional model — smaller and under-trained, same task, same conditioning, same data. The quality gain survives and the diversity loss does not: ImageNet-512 FID 2.56 → 1.34 (EDM2-S), 1.25 (XXL), 1.01 at ImageNet-64, and unconditional EDM2-S 11.67 → 3.86, which CFG cannot do at all. The binding condition is compatibility: degrade the two models differently and the method stops working entirely. | Proposed |
| SOTA-435 v2 | Never rank attribution methods by a single faithfulness or stability score, and report the perturbation distribution when you report infidelity | Yeh et al. (2019), LIT-725. Max-sensitivity is minimised by a constant explanation — an explanation that ignores the input entirely wins it — so it can never be a quality score on its own. Infidelity is defined relative to a perturbation distribution you choose, and Propositions 2.2–2.5 show several published methods are each the optimum for some choice, which the paper confirms by giving SHAP the lowest infidelity under SHAP’s own perturbation. Both cautions are free to follow: report the pair, and name the perturbation. | Active |
| SOTA-439 | Test a Kronecker-factored curvature implementation against the cases where KFAC is exact before trusting it | Dangel et al. (2025), LIT-768, §5.2 — a tutorial, so the evidence is a derivation, not a training run. KFAC’s averaging step is exact whenever the layer inputs or the backpropagated vectors do not depend on the data point. That gives two tests: a single data point, where KFAC equals the GGN and the empirical Fisher exactly and the Monte-Carlo flavour converges to the GGN; and a deep linear network under square loss on arbitrary data, where the GGN and Monte-Carlo identities still hold and the empirical-Fisher one does not. The second is the one the paper says catches scaling bugs. | Proposed |