Topics — The twenty-eight kinds of claim this anthology files against — the primary axis of both the practice registry and the reading list, and the same twenty-eight for each (ADR-026). The filing rule is the point: take a domain topic when the claim is ABOUT the domain; a claim merely discovered in one still takes its kind. The axis organizes the record, it does not bound it: content the list cannot place is evidence the list is short (ADR-059).
Representation and encoding — how the signal is encoded before the expensive network sees it — tokenizers and learned latents, positional encoding, and the frequency or basis choices that go with them.
44 of 449 SOTA documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| SOTA-007 v2 | BPE tokenization for open vocabulary tasks | Sennrich et al. (2015), LIT-003 — ARXIV-1508.07909. | Active |
| SOTA-063 v3 | use RoPE for LLM (1D sequence) positional embeddings | Su et al. (2021), LIT-045 — ARXIV-2104.09864. | Active |
| SOTA-151 v4 | Extend a trained model’s context by rescaling RoPE, not by fine-tuning at the longer length | Chen et al. (2023) and Peng et al. (2023) — RoPE does not extrapolate, and fine-tuning at the longer length barely helps: more than 10000 batches moved LLaMA’s effective window from 2048 to 2560. Rescaling the position indices so they land back in the trained range reaches 32× that in under 1000 steps. YaRN rescales per wavelength rather than uniformly and gets to 128k with 10× fewer tokens. | Active |
| SOTA-153 v4 | Drop positional encoding from the global-attention layers of a hybrid and let the cheap local layers carry position | Yang et al. (2025), Puvvada et al. (2025) and the Kimi Team (2025) — in a model that already interleaves full attention with a cheap local mixer, the full-attention layers do not need a positional encoding: the local layers carry position and recency, and the global layers do retrieval better without one. Three groups, three different local mechanisms, the same one-global-per-three-local layout. The payoff is that extending the context needs no RoPE rescaling, because there is no positional parameter left to rescale. | Active |
| SOTA-179 | Truncate the rotary encoding’s low frequencies rather than rescaling its base | Barbero et al. (2024), LIT-210 — RoPE’s high frequencies build positional attention heads and its low frequencies carry semantics that provably cannot stay robust over long context. Keeping a fraction p of the frequencies holds performance and at 2B improves it; p=1 is RoPE and p=0 is NoPE, so the practice is a dial between two things the record already holds. | Proposed |
| SOTA-187 v5 | Train the generative model in a learned compressed latent, not at full resolution | Active | |
| SOTA-247 | Add a multi-token word to a frozen model’s vocabulary from the model’s own detokenized representation of it | Kaplan et al. (2024), LIT-412 — a model already computes a whole-word representation for a word its tokenizer splits, so initialize the new embedding and unembedding from that rather than from an average of the word’s token embeddings. On Llama2-7B the frozen model then uses the new tokens while keeping its accuracy on existing ones (0.519 against 0.522 on WikiText-103), where mean-embedding initialization both fails to use them and degrades the model. The gain is largest where the tokenizer is worst: Arabic Wiki40B, 0.402 new-token accuracy against 0.117. | Proposed |
| SOTA-250 | Pretrain by predicting representations of masked regions, not their pixels | Assran et al. (2023), LIT-216 — ARXIV-2301.08243. Predict the representations of masked target blocks from a context block, with a learned target-encoder, instead of reconstructing pixels. No hand-crafted augmentations, and a ViT-H/14 reaches strong downstream performance in under 1200 GPU-hours — faster than a ViT-S/16 trained with iBOT. | Active |
| SOTA-271 | Train on a second modality even when the target is single-modality | Huh et al. (2024), LIT-458 — if representations converge on a modality-agnostic model of what generated the data, then data from any modality helps find it. The vision direction is already common practice; the language direction is not, and the paper cites a case where adding image data improved text performance. | Proposed |
| SOTA-273 | Choose a pretraining context whose association with the input is neither too strong nor too weak, and mix contexts to get there | Zhai (2025), LIT-459 — a pretraining objective is an association between the input and a context variable, and the useful contexts are the ones whose association is neither too strong nor too weak: the singular values of the expectation operator should decay neither too fast nor too slowly. Contexts that individually miss the range can be mixed into one that does not. | Proposed |
| SOTA-294 | Fit the tokenizer to the corpus when the corpus is deliberately narrow | Finke et al. (2025), LIT-485 — holding the dataset and the architecture fixed and changing only the tokenizer, a 4,096-token WordPiece vocabulary seeded with English affixes beats GPT-2’s 50,257 by +26.7 coherence and +27.5 quality. It is the one fully controlled comparison in a paper that is mostly about something else, and the effect is larger than the architecture change measured beside it. | Proposed |
| SOTA-299 | Finetune the whole generative model, encoder and decoder, for a dense perception task instead of attaching a head to a backbone | Khangaonkar and Pirsiavash (2025), LIT-488 — the freshly-initialized mask head is the part that cannot generalize, because it has only ever seen the finetuning categories. Keep every parameter generatively pretrained, encode the target as an image, and a model finetuned on furniture and cars segments people, x-rays and paintings. The same backbone under a conventional head scores 1.4–2.4 mIoU. | Proposed |
| SOTA-305 v2 | State the token count and input resolution beside any reconstruction FID, and compare tokenizers only at equal rate | Sun et al. (2024), LIT-497 — the same tokenizer at downsample 16 scores rFID 2.19, 0.94 and 0.70 on 256, 576 and 1024 tokens, all evaluated on 256×256 reconstructions. A three-fold range from a knob that is not the tokenizer. Two papers in this line then quote reconstruction numbers whose rate a reader cannot recover from the sentence they appear in. | Active |
| SOTA-306 v2 | Factorize the codebook: look up in a low-dimensional normalized space, embed in a high-dimensional one, and report utilization alongside reconstruction quality | Yu et al. (2021), LIT-500 — project the encoder output down to a low-dimensional lookup space, ℓ₂-normalize it and the codebook, match there, and project the winner back up to a wide embedding. Codebook usage goes 4% → 95% and FID 3.68 → 1.50 at unchanged throughput; removing the normalization alone drops usage to 2%. Confirmed independently by LIT-497 and LIT-494, and in all three the curve turns back up, so there is an optimum to find rather than a direction to follow. | Active |
| SOTA-312 v2 | State the noise or binning assumption behind any mutual information you report for a deterministic network, and show the conclusion survives changing it | Saxe et al. (2018), LIT-509 — in a deterministic network I(h;X) is infinite, so every finite number on an information plane is a property of a noise model the analyst imposed and the network never had. The same tanh run, binned evenly in net input instead of evenly in activity, loses its compression phase entirely; and LIT-507 makes compression appear in ReLU networks by changing the binning the other way. | Active |
| SOTA-327 | Train retrieval embeddings with nested losses, so a prefix of the vector can shortlist and the full vector re-rank | Kusupati, Bhatt, Rege et al. (2022), LIT-547 — put the loss on log(d) nested prefixes of the embedding. Each prefix then matches a separately trained model of that width (ResNet50, ImageNet), and one database supports shortlisting on 16 dimensions and re-ranking on 2048: equal mAP@10, 14× faster. Measured on image retrieval. Widely shipped in text embedding models without an independent comparison in the record. | Proposed |
| SOTA-331 | Pass low-dimensional coordinate inputs through sinusoids of sampled frequencies before an MLP, and tune the frequency scale rather than the distribution | Tancik et al. (2020), LIT-550, with NeRF (LIT-435) — an MLP on raw coordinates never fits high frequencies. Map the input to [cos 2πBv, sin 2πBv] with B ~ N(0, σ²) and tune σ on held-out data: too small blurs, too large aliases, and the distribution’s shape does not matter. Gaussian features beat no mapping and log-spaced positional encoding on all seven image, shape, CT, MRI and view-synthesis tasks tested. | Active |
| SOTA-346 | Retune the training timestep shift for each autoencoder latent space, and never rank latent spaces for generation under one shared schedule | Black Forest Labs (2025), LIT-572 — four autoencoders, 30 timestep configurations each. The training shift alone moves FID by 61–86%. An RAE latent that wins when both are tuned loses to a tuned FLUX.2 latent when left unshifted. When you swap the autoencoder under a diffusion or flow model, sweep the shift again, using a logit-normal training distribution. When you compare autoencoders, compare each at its own optimum. | Proposed |
| SOTA-360 | Score a density ratio against sampled negatives instead of reconstructing the target | Active | |
| SOTA-361 | Compose augmentations so the shortcut dies, and tune them for the contrastive objective rather than the supervised one | Active | |
| SOTA-362 | Train through a projection head and then discard it, taking the layer before as the representation | Active | |
| SOTA-363 | Decouple the negative set from the batch with a queue, and keep the encoder that fills it slow | Active | |
| SOTA-365 v2 | Learn without negatives by breaking the symmetry: a predictor on one branch and a stop-gradient on the other | Active | |
| SOTA-367 | Or put the anti-collapse constraint in the loss, as a variance floor and a decorrelation term | Active | |
| SOTA-368 | When a model’s latent space is meant to be read by people, measure its interpretability with a human task such as word or topic intrusion, not by held-out likelihood | Chang et al. (2009), LIT-597 — topic models are used for their topics, but they are judged by held-out likelihood, which measures the probability of observations and ignores the representation. On NYT and Wikipedia, CTM had the best likelihood and the worst human scores. Test the latent space directly. Plant an improbable word among a topic’s top words, or an improbable topic among a document’s top topics, and measure how often people find it. | Active |
| SOTA-369 | Curate a pretraining set by retrieving neighbours of curated seeds, rather than scaling the uncurated pool | Active | |
| SOTA-370 | Add many low-resolution views alongside the two full ones, because view count is what helps and resolution is what costs | Active | |
| SOTA-371 | Score topic coherence automatically as the average pointwise mutual information of a topic’s top-word pairs, estimated on a large external reference corpus | Newman et al. (2010), LIT-600 — when you cannot put every topic in front of people (SOTA-368), approximate their judgment. Take the topic’s top ten words and, for each of the 45 pairs, compute PMI from co-occurrence in 10-word windows over a large general corpus such as Wikipedia. Average the 45 scores. On 237 LDA topics this correlated with nine people’s ratings at ρ ≈ 0.77, far better than WordNet measures. | Proposed |
| SOTA-372 v2 | Run the encoder on visible tokens only and push the mask tokens into a small decoder you discard | Active | |
| SOTA-373 v3 | Set the masking ratio by the signal’s redundancy, not by the ratio that worked on text | Active | |
| SOTA-374 v4 | When learning embeddings from co-occurrence, subsample very frequent tokens, discarding each occurrence with probability 1 − √(t/f) with t around 10⁻⁵ | Mikolov et al. (2013), LIT-603 — in a Zipfian corpus, the few most frequent words co-occur with everything and dominate the updates while teaching little. Discard each occurrence of word w with probability 1 − √(t/f(w)). This cut skip-gram training time by 2–3× and raised accuracy, only slightly for words (59 → 60% at NEG-5) but a lot for phrases (27 → 42% at NEG-15, 19 → 47% with hierarchical softmax). | Active |
| SOTA-375 v2 | When training with sampled negatives, draw them from the unigram distribution raised to the 3/4 power, not from the unigram or uniform distribution | Mikolov et al. (2013), LIT-603 — sample negatives in proportion to count^(3/4). That flattens the Zipfian head so frequent words are not nearly the only negatives, while keeping rare words rare. The authors report it “outperformed significantly the unigram and the uniform distributions … on every task we tried”, but give no numbers. | Proposed |
| SOTA-379 v2 | Before crediting a word-embedding method over count-based ones, give the baselines the same design choices and tune every method alike | Levy, Goldberg and Dagan (2015), LIT-607 — word2vec and GloVe ship with design choices (context smoothing, the negative-sample shift, dynamic windows, subsampling, adding context vectors) that count-based PPMI and SVD can use too. Port them and tune every method the same way before comparing. Done that way across 672 representations and 8 datasets, the prediction-over-count advantage disappears, SGNS beats GloVe on every task, and a single hyperparameter often matters more than the method. | Active |
| SOTA-380 | Use prediction-based word vectors rather than count-based distributional vectors | Baroni, Dinu and Kruszewski (2014), LIT-608 — across 14 benchmarks, word2vec CBOW beat PPMI, SVD and NMF count vectors almost everywhere, and the authors recommend “anybody interested in using DSMs” to “go for the predict models”. Retired: LIT-607 traces the gap to settings the count models were not given, and finds no consistent winner once they are. | Rejected — the advantage it reports disappears when the count-based baselines get the design choices word2vec shipped with and every method is tuned alike (LIT-607): no family wins consistently. Retired with no successor. What replaces it is an evaluation practice, SOTA-379, which corrects it |
| SOTA-381 v2 | Sample a handful of negatives instead of normalizing over the vocabulary, and let the count fall as the data grows | Active | |
| SOTA-391 | Estimate attention resolution before training when choosing a position encoding, and measure it across the extrapolation boundary | Sun et al. (2022), LIT-638 — attention resolution scores how well an attention pattern distinguishes token distance, and is estimable from the encoding before a run. Measured at the training length and twice it: RoPE 0.91 → 0.08, ALiBi 0.81 → 0.88. The record has four practices about rotary encoding and held no measure of this. | Proposed |
| SOTA-396 | When a latent video model reuses an image autoencoder or upsampler, fine-tune them on video so they see time | Blattmann et al. (2023), LIT-621. An image autoencoder decodes each frame independently and flickers. Fine-tuning its decoder on video, with a video discriminator, cuts reconstruction FVD on three datasets: 390.88 to 32.94, 35.82 to 18.66, and 73.78 to 25.55. Making the upsampler temporal cuts FVD from 165.98 to 45.39. Reconstruction FID worsens on one dataset and improves on two. | Proposed |
| SOTA-398 v2 | Pretrain a vision transformer on packed, aspect-preserved images at sampled resolutions, not on fixed square crops | Dehghani et al. (2023), LIT-657. Do not resize or crop images to one square resolution. Keep each image’s aspect ratio and sample its resolution per example, favouring small side lengths. Pack the patches into fixed-length sequences with per-example attention masks. At matched compute this beats square fixed-resolution ViT pretraining, and matches the best ViT with 4x less compute. Most of the gain is throughput. | Proposed |
| SOTA-400 | Append a few register tokens to a ViT so it does not commandeer background patches for scratch space | Active | |
| SOTA-401 | Anchor the patch Gram matrix to an early teacher when training long, because dense features decay while global metrics improve | Proposed | |
| SOTA-405 | Scale the masked-language-modelling rate with model size rather than holding it at 15% | Wettig et al. (2022), LIT-672. The masking rate has been held at 15% “regardless of model sizes or masking strategies” since 2018, and the number was never swept. Swept, the optimum moves with capacity: 40% at 354M, 20% at 124M, 15% at 51M. At 354M, 40% wins seven of nine GLUE-plus-SQuAD tasks and reaches the 15% model’s QNLI and QQP scores in half the training time — under an efficient recipe, with the advantage shrinking on longer schedules. | Proposed |
| SOTA-406 | Replace every masked token with [MASK]; drop BERT’s 80-10-10 substitution rule | Wettig et al. (2022), LIT-672. BERT replaced 10% of selected positions with the original token and 10% with a random token, to reduce the pretrain/fine-tune mismatch caused by [MASK] never appearing downstream. Measured against an all-[MASK] baseline at a matched 40% rate, 80-10-10 is worse on everything but SST-2 — and the stated motivation does not bite, because “the model can adapt to full, uncorrupted sentences, regardless of the use of alternative corruption strategies in pre-training”. | Proposed |
| SOTA-418 | To drop the tokenizer without losing to BPE, use a two-stage hierarchy with learned chunk boundaries and SSM encoder and decoder layers, not an isotropic byte model | Hwang, Wang and Gu (2025), LIT-679. A byte-level model that must not lose to a BPE Transformer at matched data and FLOPs should be hierarchical. Isotropic byte stacks lose by about 9 downstream points. It should use Mamba-2 layers for the encoder and decoder around the main network, and it should learn its chunk boundaries across two stages. One learned stage only ties BPE and whitespace chunking. Expect about 2× slower training. | Proposed |
| SOTA-434 | In a vision backbone, carry position with a relative bias inside the attention window and do not add an absolute position embedding on top | Liu et al. (2021), LIT-723, Table 4. One architecture, one recipe, three tasks. A relative position bias inside the attention window is best or tied-best on all three. An absolute position embedding is worth +0.4 ImageNet top-1 and -0.2 COCO box AP and -0.6 ADE20K mIoU, and adding it on top of the relative bias costs 2.1 mIoU while changing ImageNet nothing. The recommendation is the sign, not the size: this is the one design choice in the table that a classification benchmark scores backwards. | Proposed |