Topics — The twenty-eight kinds of claim this anthology files against — the primary axis of both the practice registry and the reading list, and the same twenty-eight for each (ADR-026). The filing rule is the point: take a domain topic when the claim is ABOUT the domain; a claim merely discovered in one still takes its kind. The axis organizes the record, it does not bound it: content the list cannot place is evidence the list is short (ADR-059).
Representation and encoding — how the signal is encoded before the expensive network sees it — tokenizers and learned latents, positional encoding, and the frequency or basis choices that go with them.
97 of 737 LIT documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| LIT-003 v2 | Neural Machine Translation of Rare Words with Subword Units | Sennrich et al. (2015), ARXIV-1508.07909. Introduces byte-pair encoding for tokenization. | Active |
| LIT-045 v2 | RoFormer: Enhanced Transformer with Rotary Position Embedding | Su et al. (2021), ARXIV-2104.09864. Rotation-based positional encoding. | Active |
| LIT-048 v4 | Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation | Press et al. (2021), ARXIV-2108.12409. Linear attention bias. | Active |
| LIT-133 v2 | Kimi Linear: An Expressive, Efficient Attention Architecture | Kimi Team (2025), ARXIV-2510.26692. Kimi Delta Attention extends Gated DeltaNet with a channel-wise forgetting gate; interleaved with MLA at 3:1 in a 48B-total, 3B-active model trained on 1.4T tokens, the hybrid beats full attention on short-context, long-context and RL evaluations while cutting KV cache by up to 75% and reaching up to 6.3× the decoding throughput at 1M tokens. | Active |
| LIT-192 | Extending Context Window of Large Language Models via Positional Interpolation | Chen et al. (2023), ARXIV-2306.15595. Down-scale the position indices to fit the window the model was trained on, instead of extrapolating past it. LLaMA 7B-65B reach 32768 tokens — 16× — after fine-tuning for under 1000 steps, where naive fine-tuning at the longer length moved the effective window from 2048 to 2560 in more than 10000 batches. The argument is about smoothness: interpolating a learned attention-score function between points it already fits is bounded, extrapolating it is not. | Active |
| LIT-193 | YaRN: Efficient Context Window Extension of Large Language Models | Peng et al. (2023), ARXIV-2309.00071. Position interpolation stretches every RoPE dimension equally, which destroys the high-frequency components and stalls around a scale factor of 8. YaRN scales by wavelength instead — leaving dimensions that never complete a rotation alone, interpolating the ones that do — and adds a temperature on the attention logits implemented for free inside the rotary embedding. Llama 2 to 128k with 10× fewer tokens and 2.5× fewer steps than PI. | Active |
| LIT-207 v2 | The Impact of Positional Encoding on Length Generalization in Transformers | Kazemnejad et al. (2023), ARXIV-2305.19466. Five positional schemes trained from scratch under identical hyperparameters — absolute, T5’s relative bias, ALiBi, rotary, and none at all — and the one that generalizes to unseen lengths is none at all. It also reports that such a model represents both absolute and relative position anyway, which is why removing the encoding removes a constraint rather than information — though the absolute half of that is Haviv et al. 2022 (LIT-665), a year earlier and with the mechanism. | Active |
| LIT-208 | Rope to Nope and Back Again: A New Hybrid Attention Strategy | Yang et al. (2025), ARXIV-2501.18795. The paper that says what NoPE and RoPE layers are each for, by measuring where their attention mass goes: NoPE layers retrieve, RoPE layers attend locally, and the two roles are cleaner when the RoPE layers are confined to a sliding window. The resulting architecture — one full-attention NoPE layer per three windowed RoPE layers, 8B parameters, 5T tokens — beats a full-attention RoPE model on long and short context, at about a quarter of the KV cache. | Active |
| LIT-209 | SWAN-GPT: An Efficient and Scalable Approach for Long-Context Language Modeling | Puvvada et al. (2025), ARXIV-2504.08719. The same layout as LIT-208, reached independently and three months later — NoPE global layers interleaved 1:3 with sliding-window RoPE layers — plus the two things that paper does not have: a dynamic scaling of attention scores at inference, which is what keeps the NoPE layers working far past the training length, and a demonstration that an existing 8B RoPE model pretrained on 15T tokens can be converted to this architecture by continued training without losing its short-context scores. | Active |
| LIT-210 | Round and Round We Go! What makes Rotary Positional Encodings useful? | Barbero et al. (2024), ARXIV-2410.06205. Opens up a trained Gemma 7B to ask what RoPE is actually doing, and the answer is not the one RoPE’s own paper gave: attention does not reliably decay with distance, and the model instead splits the frequency band in two — the highest frequencies build robust positional attention heads, the lowest carry semantic content in channels that are provably not robust over long context. Removing the lowest frequencies (p-RoPE) fixes that and improves perplexity at 2B, and p is a dial whose ends are exactly RoPE and NoPE. | Active |
| LIT-216 v3 | Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture | Assran et al. (2023), ARXIV-2301.08243. I-JEPA: predict the representations of target blocks from a context block in the same image, rather than the pixels or an augmentation-invariant embedding. No hand-crafted augmentations, and the masking strategy is the design choice that does the work — large semantic targets, a spatially distributed context. A ViT-Huge/14 trains on ImageNet in under 72 hours on 16 A100s. | Active |
| LIT-341 | A Growing Neural Gas Network Learns Topologies | Fritzke (1994), [A Growing Neural Gas Network Learns Topologies](https://pro ceedings.neurips.cc/paper_files/paper/1994/hash/d56b9fc4b0f1be8871f5e1c40c00 67e7-Abstract.html). Growing neural gas: insert units where the accumulated error is largest, so the network finds its own size and topology without an annealing schedule. | Active |
| LIT-363 | ’Neural-gas’ network for vector quantization and its application to time-series prediction | Martinetz et al. (1993), DOI:10.1109/72.238311. Neural gas: adapt units by the rank of their distance to the input rather than by a fixed lattice neighbourhood, so no topology need be assumed. | Active |
| LIT-409 v2 | Token Erasure as a Footprint of Implicit Vocabulary Items in LLMs | Feucht et al. (2024), ARXIV-2406.20086, EMNLP 2024. The last token of a multi-token word or named entity shows a pronounced “erasure”: probes that recover neighbouring and current token identity elsewhere fail there, in early layers specifically. The paper reads that as the footprint of a mechanism forming lexical items, scores sequences for “lexicality” and reads out an LLM’s implicit vocabulary — the first attempt to do so. Its framing is the sharp part for this record: lexical items are units of meaning that are non-compositional, and the example given is a multi-word expression. | Active |
| LIT-410 v2 | The English phrase-as-lemma construction: When a phrase masquerades as a word, people play along | Goldberg and Shirtz (2025), Language 101(2):291-320. English has a construction — a don't-mess-with-me driver — in which a unit with the internal syntax of a phrase, up to a whole sentence, occupies a slot reserved for a word, and speakers treat it as a word: four preregistered surveys (N = 685 plus 70) find such sentences judged to presuppose more shared common knowledge, and to be wittier and more sarcastic, than normed near-paraphrases. The effect survives restricting to high-frequency phrases and shows no frequency dependence. No model was run, and the finding is about the signal rather than anything trained on it: it sources THEORY-022, the claim that a lexical unit is delimited by construction rather than by frequency — which is what SOTA-007’s vocabulary is fit to. See ADR-045 for the scope call it forced. | Active |
| LIT-412 | From Tokens to Words: On the Inner Lexicon of LLMs | Kaplan et al. (2024), ARXIV-2410.05864, ICLR 2025. LLMs recombine sub-word sequences into whole-word representations at the word’s last token, mostly in early and middle layers, and the effect is robust to arbitrary splits, typos and words the tokenizer never had. The control is what makes it: a probe on the last token reaches 89% word-vs-nonword accuracy, the same probe on the penultimate token only 61%, so it is the completed word rather than token co-occurrence. Turned into a vocabulary-expansion method whose core parameters stay frozen. | Active |
| LIT-435 | NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis | Mildenhall et al. (2020), ARXIV-2003.08934. The coordinate network the record’s one scene-representation practice exists to replace. An MLP maps a 5D coordinate — position and viewing direction — to density and view-dependent colour; differentiable volume rendering means posed images are the only supervision needed. | Active |
| LIT-458 v2 | The Platonic Representation Hypothesis | Huh et al. (2024), ARXIV-2405.07987. Representations are converging: across architectures, objectives and modalities, and more so as models get larger. The conjecture is that they are converging on a statistical model of whatever generated the data. The mathematical argument holds only for bijective observations, which the authors say plainly and test at the boundary. | Active |
| LIT-459 | Contextures: The Mechanism of Representation Learning | Zhai (2025), ARXIV-2504.19792 — a CMU dissertation. A representation is learned from the association between the input and a context variable; an encoder capturing the maximum information of that association is optimal on tasks compatible with the context. A context is most useful when the association is neither too strong nor too weak, and mixing contexts is a cheap way to get there. | Active |
| LIT-460 v2 | The Lattice Representation Hypothesis of Large Language Models | Xiong (2026), ARXIV-2603.01227. If concepts are linear directions with thresholds, each concept is a half-space, and intersecting half-spaces gives a concept lattice — so conceptual inclusion, intersection and union become geometric meet and join. Tested on WordNet sub-hierarchies. The Linear Representation Hypothesis gives binary concepts; this is the set-theoretic structure over them. | Active |
| LIT-485 | Parameterized Synthetic Text Generation with SimpleStories | Finke et al. (2025), ARXIV-2504.09184. Sampling three words from a child’s vocabulary was not enough: 59.38% of TinyStories begins “once upon a time”. Parameterize the generating prompt by theme, topic, style, narrative feature, grammar feature and author persona, and constrain the opening part of speech and letter, and every diversity metric moves. The one clean ablation in the paper is about something else — a 4,096-token domain tokenizer beats GPT-2’s 50,257 by +26.7 coherence and +27.5 quality at fixed data and architecture. | Active |
| LIT-488 | gen2seg: Generative Models Enable Generalizable Instance Segmentation | Khangaonkar and Pirsiavash (2025), ARXIV-2505.15263. Finetune Stable Diffusion or MAE end-to-end — encoder and decoder — for category-agnostic instance segmentation on indoor furnishings and cars only, and the model segments people, animals, x-rays and impressionist paintings. It approaches SAM across five domains on 0.3% of the annotated masks, and beats it ~3× on fine structures. The controls are the argument: the same backbone under a standard segmentation head scores near zero. | Active |
| LIT-489 | Measuring In-Context Computation Complexity via Hidden State Prediction | Herrmann, Csordás and Schmidhuber (2025), ARXIV-2503.13431. Next-token loss cannot tell hard from interesting: random noise scores high and requires no computation, memorized recitation scores low. Insert a variational bottleneck with a learned autoregressive prior mid-network and measure the KL between posterior and prior — the nats of hidden-state information the past did not predict. That quantity separates in-context learning from memorization and noise, and tracks formal-language description length and maths difficulty after controlling for next-token loss. | Active |
| LIT-494 v2 | GaussianToken: An Effective Image Tokenizer with 2D Gaussian Splatting | Dong, Wang, Zheng, Chen, Lu and Tang (2025), ARXIV-2501.15619. Replaces the fixed grid of a VQ image tokenizer with K 2D Gaussians whose feature coefficient is quantized while position, rotation and scale stay continuous. Reports rFID 1.61 on ImageNet-1K against LlamaGen’s 2.19 with a 16× smaller codebook. The paper’s own word for the result is semi-discrete, and the comparison is at equal token count rather than equal rate — each token carries five floats the baselines’ tokens do not. | Active |
| LIT-496 | Taming Transformers for High-Resolution Image Synthesis | Esser, Rombach and Ommer (2020), ARXIV-2012.09841. Adds a perceptual loss and a patch-based discriminator to the VQ autoencoder, with an adaptive weight λ = ∇[L_rec] / (∇[L_GAN] + δ) computed from the decoder’s last layer. The result is that f = 16 compression stops destroying perceptual quality, so a 256×256 image becomes 16×16 tokens a transformer can model. Every tokenizer in this line since is this architecture with the codebook retuned. | Active |
| LIT-497 | Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation | Sun et al. (2024), ARXIV-2406.06525. Keeps VQGAN’s encoder-quantizer-decoder and retunes the codebook: ℓ₂-normalized codes, vector dimension 8 instead of 256, size 16384. Utilization goes from 0.29% to 97% and rFID from 9.21 to 2.19. Its other measurement is the one the record needed — the same tokenizer spans rFID 2.19 → 0.70 purely by changing how many tokens the image gets. | Active |
| LIT-498 | MaskGIT: Masked Generative Image Transformer | Chang, Zhang, Jiang, Liu and Freeman (2022), ARXIV-2202.04200. Keeps VQGAN’s tokenizer and replaces its raster-order autoregressive prior with a bidirectional masked transformer decoded in parallel: predict every token, keep the most confident, re-predict the rest. 8 steps instead of 256, up to 64× faster, and better — ImageNet 256×256 FID 6.18 vs 15.78, IS 182.1 vs 78.3. The mask schedule matters and the concave family wins; cosine is best in every experiment. | Active |
| LIT-499 | Neural Discrete Representation Learning | van den Oord, Vinyals and Kavukcuoglu (2017), ARXIV-1711.00937. The paper that made discrete latents trainable: nearest-neighbour lookup into a learned codebook, gradients copied past the non-differentiable step by a straight-through estimator, and two L2 terms — one moving codes toward the encoder, one committing the encoder to its code. ImageNet at 128×128×3 compressed to 32×32 indices over K = 512, a 42.6× bit reduction, with a PixelCNN learned over the indices as the prior. | Active |
| LIT-500 | Vector-quantized Image Modeling with Improved VQGAN | Yu et al. (2021), ARXIV-2110.04627. The origin of the codebook advice SOTA-306 rests on, and it says something sharper than its descendants repeat: factorize the code, so lookup happens in a low-dimensional space while the embedding stays high-dimensional, and ℓ₂-normalize both sides. Its ablation puts codebook usage at 4% at lookup dimension 256, 95% at 16 — and 2% with normalization removed, which is the largest single effect in the table. | Active |
| LIT-505 v2 | Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences | Rives et al. (2021), PNAS 118(15) — the protein language model the biological-sequence line runs from. A transformer trained by masked-token prediction on 250 million protein sequences, with no structural or functional labels, produces representations in which secondary structure, tertiary contacts and remote homology are linearly recoverable. Held here as the ancestor of LIT-506, which reimplements the recipe for RNA and copies its downstream head. | Active |
| LIT-506 v2 | Interpretable RNA Foundation Model from Unannotated Data for Highly Accurate RNA Structure and Function Predictions | Chen et al. (2022), ARXIV-2204.00300 — a 12-layer BERT encoder trained by masked-token prediction on 23.7 million unannotated ncRNA sequences from RNAcentral, then used as a frozen feature extractor for secondary structure, 3D contact, protein-RNA binding and UTR regulation. Beats twelve secondary-structure methods (F1 0.941 on ArchiveII600, 0.704 on bpRNA TS0). Read as NOTE-252, which records that the paper’s one clean input-swap control moves R² by 0.002 on the generalization set. | Active |
| LIT-507 | Adaptive Estimators Show Information Compression in Deep Neural Networks | Chelombiev, Houghton and O’Donnell (2019), ARXIV-1902.09037 — the third voice, and the one whose body is weaker than its abstract. Argues that LIT-509’s ReLU result is an artifact of non-adaptive binning and that with adaptive estimators saturation is not required for compression. Its own Figure 6 shows that averaged over 50 initializations a ReLU network has no distinct phase at all, and it agrees with the rebuttal that hidden-layer compression does not correlate with generalization. Read as NOTE-254. | Active |
| LIT-508 | Opening the Black Box of Deep Neural Networks via Information | Shwartz-Ziv and Tishby (2017), ARXIV-1703.00810 — the claim the information-bottleneck theory of deep learning rests on. Training is said to have two phases, a short fitting phase and a long compression phase in which I(X;T) falls; compression is said to explain generalization; and the compression is attributed to the diffusion-like behaviour of SGD. Measured by binning arctan activations into 30 equal intervals between -1 and 1 over 4096 input patterns. Held as the trunk of a three-way dispute — see LIT-509 and LIT-507. | Active |
| LIT-509 v2 | On the Information Bottleneck Theory of Deep Learning | Saxe et al. (2018), ICLR — the rebuttal. Takes the three claims of LIT-508 and shows none holds in general. The compression phase tracks the nonlinearity: tanh compresses, ReLU does not, replicated with the original authors’ own code, and consistent across binning, a kernel density estimator and — for layer entropy, on the original task only — the Kraskov estimator. Compression and generalization dissociate in all four combinations. Full-batch gradient descent compresses as much as SGD. Read as NOTE-253. | Active |
| LIT-511 | Image-GS: Content-Adaptive Image Representation via 2D Gaussians | Zhang et al. (2024), ARXIV-2407.01866 — the 2D descendant of LIT-108, and the paper that does the rate accounting LIT-494 skipped. Anisotropic colored 2D Gaussians, spawned by image-gradient magnitude and grown where error persists, decoded at 0.3K MACs per pixel against C3’s 3K at comparable bitrate. Beats six neural image representations at matched model size across the bitrate range, and beats JPEG below 0.244 bpp. Read as NOTE-256. | Active |
| LIT-526 | Concepts Whisper: Spectral Anti-Concentration and the Dual Geometry of Transformer Representations | Acharya, Rimal and Dhakal (2026), ARXIV-2605.01609 — contextual concept directions sit in the low-variance tail of the unembedding covariance in 17 of 17 models, while static vocabulary contrasts sit at the top. The paper also reports a robust null against the hypothesis it set out to test, and labels each of its own claims robust, partial or suggestive. Read as NOTE-270. | Active |
| LIT-531 | Deep Learning and the Information Bottleneck Principle | Tishby and Zaslavsky (2015), ARXIV-1503.02406 — the five-page position paper the whole information-bottleneck line descends from. It proposes quantifying every layer by I(X;h) and I(Y;h), draws the information plane as an explicitly qualitative figure with a hypothesized layer path, and states that approaching the IB limit requires stochastic mapping between the layers — the condition its successors were later refuted for dropping. Read as NOTE-275. | Active |
| LIT-545 | Gemma 3n model overview | Google DeepMind (2025), model documentation with no accompanying paper. The origin of Per-Layer Embeddings (PLE): each token id looks up a small vector for every layer, and each layer gates it into the residual stream. The table is about 2.35B parameters in E2B and is kept off the accelerator. A design disclosure and not evidence: Google published no ablation. Cited by the Qwen3.8-Flash-Next report (LIT-152) as the precedent for embedding tables held in host memory. | Active |
| LIT-547 | Matryoshka Representation Learning | Kusupati, Bhatt, Rege et al. (2022), ARXIV-2205.13147. Add the task loss at about log(d) prefix lengths of one embedding (8, 16, …, 2048), and every prefix works as a representation. ResNet50 on ImageNet: each prefix is at least as accurate as a separately trained model of that width. Shortlisting with 16 dimensions and re-ranking with 2048 matches full-width retrieval at 128× fewer FLOPs, 14× faster in wall-clock time. At full width it costs a few tenths of a point on the web-scale models. | Active |
| LIT-550 | Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains | Tancik, Srinivasan, Mildenhall et al. (2020), ARXIV-2006.10739. A plain MLP on low-dimensional coordinates learns high frequencies so slowly that in practice it never does. In the NTK picture this is because the kernel’s eigenvalues fall off fast with frequency. Mapping the input through sinusoids makes the kernel stationary, with a bandwidth set by the frequencies. Random Gaussian frequencies beat NeRF’s positional encoding on all seven tasks, and only their scale matters. | Active |
| LIT-572 | FLUX.2: Analyzing and Enhancing the Latent Space of FLUX – Representation Comparison | Black Forest Labs (2025), technical report. DiT-XL flow models on ImageNet 256² in four latent spaces (SD-VAE, FLUX.1-VAE, FLUX.2-VAE, RAE), 30 configurations each: training timestep distribution, training shift and sampling shift, with and without REPA. The training shift moves FID by 61–86%, and without tuning it per space the rankings flip. FLUX.1’s VAE reconstructs far better than SD’s and is less learnable (gFID 10.1 against 7.7). FLUX.2’s VAE, with 8× SD’s dimensionality and semantic regularization, gets both (LPIPS 0.27, gFID 3.7). FLUX.2 is that latent plus Mistral-3 24B and a rectified-flow transformer. | Active |
| LIT-587 | An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale | Dosovitskiy et al. (2020), ARXIV-2010.11929. Cut the image into fixed patches, linearly embed them, and feed the sequence to an unmodified transformer encoder. The result is not that the architecture is better — it is that at enough pre-training data the convolutional prior stops paying for itself, and below that threshold it still does. | Active |
| LIT-589 | Representation Learning with Contrastive Predictive Coding | Oord et al. (2018), ARXIV-1807.03748. Predict the future in latent space and score it as a density ratio against sampled negatives, rather than reconstructing the observation. The loss is named InfoNCE here, and it is the objective CLIP, SimCLR and MoCo all run; the bound it maximizes is capped at log N, which the folklore about negatives leaves out. | Active |
| LIT-590 | Momentum Contrast for Unsupervised Visual Representation Learning | He et al. (2019), ARXIV-1911.05722. Reframes contrastive learning as dictionary look-up and asks for a dictionary that is both large and consistent: a queue makes it large independently of the batch, and an EMA-updated key encoder makes it consistent. The momentum ablation is the result that propagated — at m=0 it does not train at all. | Active |
| LIT-591 | A Simple Framework for Contrastive Learning of Visual Representations | Chen et al. (2020), ARXIV-2002.05709. Strips contrastive learning to augment, encode, project, NT-Xent — no memory bank, no special architecture — and then ablates each piece. The two findings that outlived the method: the augmentation pair is the pretext task, and the projection head is a sacrificial layer you throw away. | Active |
| LIT-592 | Latent Dirichlet Allocation | Blei, Ng and Jordan (2003), JMLR 3:993–1022. If the words of a document are exchangeable (bag of words), de Finetti’s theorem says the document is a mixture over a latent parameter. LDA makes that parameter a Dirichlet-distributed vector of topic proportions, one per document, with each topic a distribution over words. Fitted by variational EM, it has lower held-out perplexity than a unigram, a mixture-of-unigrams and a pLSI model on two corpora, and the two baselines overfit badly as the number of topics grows. Its 50 topic proportions stand in for 15,818 word features in Reuters classification. | Active |
| LIT-593 | Exploring Simple Siamese Representation Learning | Chen & He (2020), ARXIV-2011.10566. Strips a joint-embedding method to nothing: no negatives, no large batch, no momentum encoder — shared weights, a predictor on one side, a stop-gradient on the other. It works, which makes stop-gradient the candidate for the load-bearing piece, and contradicts BYOL’s account of its own method. | Active |
| LIT-594 | Bootstrap your own latent: A new approach to self-supervised Learning | Grill et al. (2020), ARXIV-2006.07733. Learns a representation with no negatives at all: an online network predicts a slow EMA copy of itself under a different augmentation. 74.3% ImageNet linear. Why it does not collapse is a hypothesis in the paper, not a result, and LIT-593 disputes which part is doing the work. | Active |
| LIT-595 | VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning | Bardes et al. (2021), ARXIV-2105.04906. Three explicit terms — a hinge keeping each embedding dimension’s standard deviation above a threshold, an invariance term, and a decorrelation term — applied to each branch separately. Needs no weight sharing, batch norm, stop-gradient or memory bank, and the variance term transplants into other methods and improves them. | Active |
| LIT-596 | Barlow Twins: Self-Supervised Learning via Redundancy Reduction | Zbontar et al. (2021), ARXIV-2103.03230. Removes every asymmetry — no predictor, no stop-gradient, no moving average — and prevents collapse in the objective instead: drive the cross-correlation matrix between the two branches’ outputs toward the identity. 73.2% ImageNet linear, and unlike its neighbours it wants a very high-dimensional output. | Active |
| LIT-597 | Reading Tea Leaves: How Humans Interpret Topic Models | Chang, Boyd-Graber, Gerrish, Wang and Blei (NIPS 2009). Two Mechanical Turk tasks measure whether a topic model’s latent space means anything to people. In word intrusion, subjects find the odd word out among a topic’s top five. In topic intrusion, they find the topic that does not belong to a document. On pLSI, LDA and CTM at 50, 100 and 150 topics over NYT and Wikipedia, CTM has the best held-out likelihood and the worst human scores. LDA is usually best. The paper concludes that likelihood does not track interpretability, and it reads the trend as a negative correlation. | Active |
| LIT-598 | Unsupervised Learning of Visual Features by Contrasting Cluster Assignments | Caron et al. (2020), ARXIV-2006.09882. Predict one view’s cluster code from another view’s representation instead of comparing features pairwise, with an equipartition constraint doing the anti-collapse work. 75.3% ImageNet linear — and multi-crop, which is the part that transplants into other methods for 2-4 points. | Active |
| LIT-599 v3 | DINOv2: Learning Robust Visual Features without Supervision | Oquab et al. (2023), ARXIV-2304.07193. The thesis is that existing self-supervised objectives already produce general-purpose features if the data is curated enough, so the contribution is a retrieval-based curation pipeline and the engineering to scale it — not a new objective. Frozen features, no finetuning, competitive with the best open weakly-supervised models. | Active |
| LIT-600 | Automatic Evaluation of Topic Coherence | Newman, Lau, Grieser and Baldwin (NAACL 2010). Humans rated 237 LDA topics from news and books for coherence. The paper then tested automated scores built from WordNet, Wikipedia and Google. The simplest measure, the average pointwise mutual information of a topic’s top-word pairs counted in 10-word windows over Wikipedia, reaches a Spearman correlation of about 0.77 with the human mean. Inter-annotator agreement, measured differently, is 0.73–0.82. WordNet measures are erratic, and Google drops from 0.80 on news to 0.51 on books. | Active |
| LIT-601 | Masked Autoencoders Are Scalable Vision Learners | He et al. (2021), ARXIV-2111.06377. Mask 75% of the patches, run the encoder on the visible quarter only, and reconstruct pixels with a small decoder you then throw away. 3x faster pretraining, 87.8% with ViT-Huge on ImageNet-1K alone. The 75% ratio, five times BERT’s 15%, is found by a sweep, and the paper explains it with an argument about the signal’s redundancy. | Active |
| LIT-602 v2 | GloVe: Global Vectors for Word Representation | Pennington, Socher and Manning (EMNLP 2014), DOI 10.3115/v1/D14-1162. Meaning shows in ratios of co-occurrence probabilities, not in the probabilities themselves, so word vectors should make differences encode log ratios. That leads to a weighted least-squares fit of wᵢ·w̃ⱼ + bᵢ + b̃ⱼ to log Xᵢⱼ over nonzero counts, with a weight capped at x_max = 100 and exponent 3/4. It scores 75.0% on the word analogy test with 42B tokens, and 71.7% against word2vec skip-gram’s 69.1% on the same 6B tokens. The word2vec side of that comparison ran at default settings. | Active |
| LIT-603 v3 | Distributed Representations of Words and Phrases and their Compositionality | Mikolov, Sutskever, Chen, Corrado and Dean (NIPS 2013), ARXIV-1310.4546. Makes skip-gram fast and better. Negative sampling replaces the softmax with logistic regression against k noise words drawn from the unigram distribution raised to the 3/4 power. Frequent words are subsampled, with discard probability 1 − √(t/f) at t ≈ 10⁻⁵, which runs 2–10× faster and helps rare words. High-PMI bigrams become single tokens. Vector sums compose meanings, for example Russian + river ≈ Volga River. | Active |
| LIT-604 | Efficient Estimation of Word Representations in Vector Space | Mikolov, Chen, Corrado and Dean (2013), ARXIV-1301.3781. Two log-linear architectures drop the hidden layer of a neural language model. CBOW predicts a word from its averaged context, and skip-gram predicts the context from the word. The paper also introduces the Semantic-Syntactic Word Relationship test (8,869 semantic and 10,675 syntactic analogy questions), answered by vector arithmetic. On it, skip-gram at 300 dimensions reaches 53.3% on 783M words in about 3 days on one CPU, far ahead of earlier public embeddings. The comparisons are not controlled for data size or dimension. | Active |
| LIT-607 v3 | Improving Distributional Similarity with Lessons Learned from Word Embeddings | Levy, Goldberg and Dagan (TACL 2015). Takes the design choices bundled into word2vec and GloVe (dynamic windows, subsampling, shifted PMI, context-distribution smoothing, adding context vectors, eigenvalue weighting), transfers them to count-based PPMI and SVD, and compares all four methods under the same tuning: 672 representations, 8 similarity and analogy datasets, and cross-validated selection. Once tuned alike, no method wins consistently. A single hyperparameter often matters more than the choice of method. SGNS beats GloVe on every task, reversing GloVe’s headline comparison. Smoothing the context distribution by 0.75 is the one setting that helps everywhere. | Active |
| LIT-608 | Don’t count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors | Baroni, Dinu and Kruszewski (ACL 2014). The first broad comparison of count-based distributional vectors (36 PPMI/LMI models, full, SVD and NMF) against word2vec CBOW vectors (48 models), on 14 lexical-semantics benchmarks and a 2.8B-token corpus. Predict wins on almost every task, often by 10 points or more, and is far more robust to its worst settings. The count models were not given word2vec’s context smoothing or negative shift, and SVD kept its singular-value weighting. LIT-607 traces the result to those differences and finds no consistent winner when both are tuned alike. | Active |
| LIT-609 v2 | Distributed Representations of Words and Phrases and their Compositionality | Mikolov et al. (2013), ARXIV-1310.4546. Where negative sampling comes from — and it is NCE with the consistency deliberately dropped, kept only because the vectors stay good. Also: the number of negatives that helps falls as the dataset grows, which is the opposite of what the vision contrastive line later assumed. | Superseded — by LIT-603; a duplicate of LIT-603, filed concurrently by |
| LIT-610 v2 | Efficient Estimation of Word Representations in Vector Space | Mikolov et al. (2013), ARXIV-1301.3781. CBOW and Skip-gram: drop the non-linear hidden layer that made neural language models expensive, and word vectors become cheap enough to train on 1.6 billion words in under a day. The linear-analogy finding is here, and so is the test set built to stop it being an anecdote. | Superseded — by LIT-604; a duplicate of LIT-604, filed concurrently by |
| LIT-612 | Neural Word Embedding as Implicit Matrix Factorization | Levy and Goldberg (NeurIPS 2014). With enough dimensions, skip-gram with negative sampling reaches its optimum when w·c = PMI(w, c) − log k, so it implicitly factorizes the word-context PMI matrix shifted by log of the number of negatives. NCE factorizes log P(w|c) − log k in the same way. The factorization is weighted: frequent pairs cost more to get wrong. Shifted PPMI and SVD over it match SGNS on word similarity. SGNS stays ahead on syntactic analogies, which the authors attribute to the weighting. | Active |
| LIT-613 | A Latent Variable Model Approach to PMI-based Word Embeddings | Arora, Li, Liang, Ma and Risteski (TACL 2016). A generative model of text: a discourse vector drifts slowly on a random walk, and each word is emitted with probability ∝ exp(⟨c, v_w⟩). If word vectors are isotropic, the partition function is nearly constant, and integrating out the discourse gives log p(w,w’) ≈ ‖v_w + v_w’‖²/2d − 2 log Z, so PMI ≈ ⟨v_w, v_w’⟩/d in low dimensions. Isotropy also explains why analogies survive large fitting error: solving for v_a − v_b is a regression that denoises by √(d/n). The squared norms of the fitted vectors correlate 0.75 with log frequency, and the partition function concentrates within about ±10%. | Active |
| LIT-618 v2 | LTX-Video: Realtime Video Latent Diffusion | HaCohen et al., Lightricks (2024), ARXIV-2501.00103. It trades latent size for speed: a 32×32×8 VAE with 128 channels (1:192) moves patchification into the autoencoder, and the decoder does the last denoising step. It generates 5s of 768×512 video in about 2s on an H100 at 20 steps. The denoising decoder, the central idea, is supported by one internal preference study with no reported numbers. | Active |
| LIT-619 v2 | Wan: Open and Advanced Large-Scale Video Generative Models | Wan Team, Alibaba (2025), ARXIV-2503.20314. The open Wan2.1 video models, 1.3B and 14B, and the full recipe behind them. The recipe covers data curation, a 127M causal 3D VAE compressing 4×8×8, a cross-attention DiT trained by rectified flow on an image-then-video resolution curriculum, the 2D context parallelism it needs, and a mask-conditioned image-to-video variant. The abstract claims video scaling laws in data and model size, but the report plots no scaling curve. | Active |
| LIT-620 v2 | HunyuanVideo: A Systematic Framework For Large Video Generative Models | Kong et al., Tencent Hunyuan (2024), ARXIV-2412.03603. A 13B open video model: FLUX-style dual-then-single-stream transformer, a multimodal LLM as text encoder, a from-scratch causal 3D VAE. It fits the line’s only explicit video compute-optimal scaling law, on a proxy model that differs from the shipped one in objective, text encoder and conditioning. It reports no controlled ablations. | Active |
| LIT-622 v2 | CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer | Yang, Teng et al., Tsinghua and Zhipu AI (2024), ARXIV-2408.06072. The first widely used open DiT for text-to-video. It has a MAGVIT-style causal 3D VAE, one sequence over text and video with modality-specific adaLN, and full 3D attention. Full 3D attention beats factorized attention at 1.1× to 2.3× the forward time, rising with resolution. The architecture ablations are absent from the first arXiv version and present in the third, and the 2B model ships the position encoding they rejected. | Active |
| LIT-623 v2 | Language Model Beats Diffusion — Tokenizer is Key to Visual Generation | Yu et al., Google and CMU (2023), ARXIV-2310.05737. MAGVIT-v2 is a causal 3D-CNN tokenizer with lookup-free quantization. The causal design lets images and video share one tokenizer, though no experiment uses the video tokenizer for images, and LFQ lets generation keep improving as the vocabulary grows, where VQ turns worse. The title’s “beats diffusion” holds only with guidance, and the “language model” is a masked, MaskGIT-style one. | Active |
| LIT-624 v3 | Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Model | Step-Video Team, StepFun (2025), ARXIV-2502.10248. A 30B text-to-video DiT on a 16×16×8 VAE that reconstructs as well as HunyuanVideo’s 8×8×4 on its own benchmark, though not on Open-Sora 2.0’s, so the model covers 204 frames. It also applies DPO to video (a 55/45 preference score, ties split) and runs at 32% MFU. It reaches parity with Movie Gen and loses to the strongest commercial model it names, despite the abstract’s “state-of-the-art”. | Active |
| LIT-626 v2 | Movie Gen: A Cast of Media Foundation Models | The Movie Gen team, Meta (2024), ARXIV-2410.13720. A 30B video model built from a Llama 3-style transformer, flow matching and an 8×8×8 temporal autoencoder, with the most controlled ablations in the video line. At 5B, flow matching beats diffusion and video captions beat frame captions. The “LLaMa3 scaling laws” claim rests on image-only runs at four sizes. | Active |
| LIT-634 v2 | Open-Sora 2.0: Training a Commercial-Level Video Generation Model in $200k | The Open-Sora team, HPC-AI Tech (2025), ARXIV-2503.09642. An 11B video model initialized from FLUX, with most of its compute at 256px and a 768px text/image-to-video stage that is short in iterations but takes 37% of the compute. The “$200k” is one final diffusion-model run at an assumed H200 rental rate. It excludes data, captioning, failed runs, and the FLUX and HunyuanVideo-VAE weights it starts from. A 4×32×32 autoencoder is a further efficiency experiment, and the model adapted to it is unconverged and weaker. | Active |
| LIT-638 | A Length-Extrapolatable Transformer | Sun et al. (2022), ARXIV-2212.10554. Defines attention resolution — a scalar for how well an attention pattern distinguishes token distance, estimable before training — and measures it across the extrapolation boundary. RoPE collapses 0.91 → 0.08 at twice its training length; ALiBi holds 0.81 → 0.88 and is the only incumbent that does. The record holds both papers and four practices about RoPE, and had no measure of this. | Active |
| LIT-652 v2 | Video generation models as world simulators | Brooks, Peebles et al., OpenAI (2024), no arXiv id. Sora is a latent diffusion transformer over spacetime patches of a video autoencoder’s latents. It is trained jointly on images and videos at their native durations, resolutions and aspect ratios, with captions from a DALL-E 3-style recaptioner. The report withholds model and implementation details, and it has no table, number, metric or baseline. Every claim, including the scaling comparison and the square-crop ablation, is shown by example videos only. The video reports filed after it cite it and benchmark against it, and it supplies no measurement of its own. | Active |
| LIT-657 | Patch n’ Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution | Dehghani, Mustafa et al., Google DeepMind (2023), ARXIV-2307.06304. Packs patches from several images into each fixed-length sequence, with per-example attention masks and pooling. A ViT can then take any aspect ratio and a sampled resolution per image. It leads ViT at each of 12 compute-matched budgets, and matches the best ViT with 4x less compute. The paper attributes the gain chiefly to seeing about 5x more images, not to aspect ratio. The one test that isolates aspect ratio is a linear probe on fairness attributes. | Active |
| LIT-662 | Vision Transformers Need Registers | Darcet, Oquab, Mairal and Bojanowski (2023), ARXIV-2309.16588. 107 documents in this record say “ViT” and none held this. Large trained ViTs put high-norm outlier tokens on background patches — 2.37% of tokens in DINOv2 ViT-g — which hold little local information and a lot of global information. The model is using redundant patches as scratch space. Append a few tokens whose only job is to be scratch space and the artifacts vanish. | Active |
| LIT-663 | DINOv3 | Siméoni et al. (2025), ARXIV-2508.10104. The finding the record needs is not the model: train a self-supervised ViT long enough and its dense features rot while its global metrics keep improving. Segmentation mIoU declines after ~200k iterations and for the 7B falls below its early levels, while ImageNet classification rises monotonically throughout. Gram anchoring — pull the student’s patch Gram matrix towards an early teacher’s — repairs it, and it is a different defect from the register artifacts. | Active |
| LIT-664 | Emerging Properties in Self-Supervised Vision Transformers | Caron et al. (2021), ARXIV-2104.14294. The origin of the DINO line, which 31 documents here reference and 12 practices touch. Self-supervised ViT features carry explicit semantic segmentation that supervised ViTs and convnets do not show as clearly, and reach 78.3% ImageNet top-1 with a plain k-NN — no finetuning, no linear probe, no augmentation. Collapse is avoided by centering and sharpening the teacher output alone, with no predictor. | Active |
| LIT-665 | Transformer Language Models without Positional Encodings Still Learn Positional Information | Haviv, Ram, Press, Izsak and Levy (2022), ARXIV-2203.16634. The result the record’s position-encoding cluster rests on, credited here to a paper a year younger. A decoder-only LM with no positional encoding at all is competitive; probing shows it acquires absolute position within four layers. The mechanism is the causal mask, and the control that shows it is a bidirectional MLM, where NoPos perplexity is 147.18 against 4.00. | Active |
| LIT-668 v2 | ALBERT: A Lite BERT for Self-supervised Learning of Language Representations | Lan, Chen, Goodman, Gimpel, Sharma and Soricut (2019), ARXIV-1909.11942. The measurement behind cross-layer parameter sharing, which this record started recommending — via the fix THEORY-103 endorses — without holding a single paper on it. The sharp result is an asymmetry: sharing the attention parameters across every layer costs +0.1 average at E = 128, and sharing the feed-forward parameters costs −1.4. Also carries a priority claim the record’s dropout practice should have: the first report that dropout hurts a large transformer. | Active |
| LIT-670 | BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding | Devlin, Chang, Lee and Toutanova (2018), ARXIV-1810.04805. The model 89 files in this record referred to and none held. Two things it is cited for here turn out not to be in it: the 15% masking rate was never swept — “in all of our experiments, we mask 15%”, and the one appendix table headed “Masking Rates” varies the 80/10/10 replacement split at a fixed 15% — and its NSP ablation removes the loss while keeping the paired-segment input format, which LIT-671 later separates and reverses. | Active |
| LIT-671 | RoBERTa: A Robustly Optimized BERT Pretraining Approach | Liu, Ott, Goyal, Du, Joshi, Chen, Levy, Lewis, Zettlemoyer and Stoyanov (2019), ARXIV-1907.11692. A replication study that finds BERT was significantly undertrained and, more usefully here, that BERT’s own NSP ablation could not have shown what it claimed: it removed the loss while keeping the paired-segment input format. Separate the two and removing NSP matches or slightly improves. The paper is careful about its own margins — dynamic masking is “comparable or slightly better”, and byte-level BPE is adopted despite being slightly worse. | Active |
| LIT-672 | Should You Mask 15% in Masked Language Modeling? | Wettig, Gao, Zhong and Chen (2022), ARXIV-2202.08005. The sweep BERT never ran, and the answer the record has been openly missing: 15% is not universally optimal, and the optimum tracks model size — 40% at 354M, 20% at 124M, 15% at 51M. It also decomposes the masking rate into a corruption rate and a prediction rate whose effects are antagonistic, and uses that to find BERT’s 80-10-10 substitution rule worse than plain [MASK]. | Active |
| LIT-679 | Dynamic Chunking for End-to-End Hierarchical Sequence Modeling | Hwang, Wang and Gu (2025; ICLR 2026), ARXIV-2507.07955. H-Net is a byte-level hierarchical model that learns where to cut chunks. Compared with a GPT-2 BPE Transformer at matched bytes per batch and FLOPs per byte, a chain of baselines separates each ingredient. One stage of learned chunking only ties BPE and whitespace chunking (0.728 against 0.730 against 0.726 bits/byte at XL). The gain is the second stage (0.715, +2.7 downstream average). The largest gains are on Chinese. It is limited to 1.3B-equivalent FLOPs and trains about 2× slower in wall-clock. | Active |
| LIT-711 | Do Transformer Modifications Transfer Across Implementations and Applications? | Narang et al. (2021), ARXIV-2102.11972. About fifty published transformer modifications reimplemented in one codebase (T5 / Mesh TensorFlow, 223M encoder-decoder) with hyperparameters held fixed, scored on C4 pre-training loss, SuperGLUE, XSum, WebQuestions and WMT’14 En-De. Most do not help. The ones that do are small (GLU-variant activations, RMSNorm, untied embeddings), cost parameters or time (Switch/MoE, product-key memory, mixture of softmaxes), or were invented in the same codebase. The Universal Transformer, ReZero, Fixup, cross-layer sharing, the Evolved Transformer and lightweight/dynamic convolutions all come out worse than the vanilla baseline. The caveat that travels least: the positive results are also mostly same-codebase — GLU Variants (LIT-030) was run in this codebase at this exact configuration — so the paper is a strong negative test and a weak positive one. | Active |
| LIT-723 | Swin Transformer: Hierarchical Vision Transformer using Shifted Windows | Liu, Lin, Cao, Hu et al. (2021), ARXIV-2103.14030. A vision transformer with a hierarchical feature pyramid and self-attention confined to shifted local windows, linear in image size rather than quadratic. Nine documents here run experiments on it and none said what it is. Its ablation is the reason to hold it: every design element is measured on three tasks, and the absolute position embedding this record recommends elsewhere helps ImageNet classification (+0.4 top-1) while hurting COCO detection (−0.2 box AP) and ADE20K segmentation (−0.6 mIoU). | Active |
| LIT-724 v2 | Investigating Sanity Checks for Saliency Maps with Image and Text Classification | Kokhlikyan, Miglani, Alsallakh, Martin and Reblitz-Richardson (2021), ARXIV-2106.07475. The follow-up SOTA-430 and NOTE-365 both named as the first thing to read against the sanity checks, unread until now. Three findings, all narrowing what a failed randomization test means: the failure is attributable to the input multiplier rather than to the attribution; the verdict does not transfer from image to text, where global Integrated Gradients becomes parameter-sensitive; and SSIM and Spearman rank correlation return opposite verdicts on the same experiment. | Active |
| LIT-725 | On the (In)fidelity and Sensitivity for Explanations | Yeh, Hsieh, Suggala, Inouye and Ravikumar (2019), ARXIV-1901.09392, NeurIPS 2019. The definitions behind the two numbers LIT-724 reports and this record had started quoting: infidelity and max-sensitivity. Two properties a reader needs and neither citing document carried — max-sensitivity is minimised by a constant explanation, and infidelity is defined relative to a chosen perturbation distribution for which several existing methods are each the optimum. It also reproduces SOTA-430’s metric split on a third model family, and builds the planted-feature setting SOTA-430’s promote_when asks for while pointing it at a different question. | Active |
| LIT-729 v2 | When Explanations Lie: Why Many Modified BP Attributions Fail | Sixt, Granz and Landgraf (2019), ARXIV-1912.09818, ICML 2020. The account SOTA-430’s failing row was missing — for one family, and not the family this record said it was for. Theorem 1: a backpropagation rule that keeps only non-negative relevance gives a product of non-negative matrices, which converges to a rank-1 matrix, and C v = c γᵀ v = λc — once converged the later layers can only change the map’s sign. That covers DTD, LRP-α1β0, Excitation BP and PatternAttribution. It explicitly does not cover Guided Backprop or Deconv, whose ReLU on the gradient makes the backward pass non-linear and which it defers to Nie et al. (2018), now LIT-730. | Active |
| LIT-730 | A Theoretical Explanation for Perplexing Behaviors of Backpropagation-based Visualizations | Nie, Zhang and Patel (2018), ARXIV-1805.07039, ICML 2018. The third mechanism, and the one SOTA-430’s headline failure row needed. Theorem 1: in a random three-layer CNN, guided backpropagation recovers the input, s_k^GBP(x) ≈ x, regardless of the class label. Theorem 2: saliency map and DeconvNet in the same network are N(0, I) — noise. So GBP and DeconvNet are doing (partial) image recovery, “unrelated to the decision-making of neural networks”, and the two causes are the backward ReLU and a CNN’s local connections. Four arms test it, including removing local connections and an adversarial attack that changes the class while barely changing the input. | Active |
| LIT-744 | Text Embeddings by Weakly-Supervised Contrastive Pre-training | Wang et al. (2022), ARXIV-2212.03533 — E5. A BERT-initialised bi-encoder trained with InfoNCE and in-batch negatives (batch 32,768) on ~270M web text pairs, cut from ~1.3B by a consistency filter that keeps a pair only if a model trained on the noisy set ranks its passage in the top 2 of 1M. With no labelled data the base model averages 42.9 nDCG@10 on 15 BEIR datasets against BM25’s 41.7, the first unsupervised model reported above BM25; after fine-tuning on MS-MARCO, NQ and NLI the 330M large model averages 61.4 on 56 MTEB English datasets, above the 4.8B GTR-xxl and Sentence-T5-xxl. | Active |
| LIT-745 | SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features | Tschannen et al. (2025), ARXIV-2502.14786. Keep SigLIP’s architecture and sigmoid loss, and add a captioning-and-grounding decoder loss (LocCa) throughout, self-distillation and masked prediction for the last 20% of training, a 90/10 English/multilingual mix with de-biasing filters, and distillation through active data selection for the smallest models. It beats SigLIP at every size and resolution compared on zero-shot classification, retrieval, VLM transfer, dense probing and localization (RefCOCO val, L/256: 67.3 → 86.0). None of the ingredients is ablated, so the gains cannot be credited to any one of them. | Active |
| LIT-747 | Perceiver: General Perception with Iterative Attention | Jaegle et al. (2021), ARXIV-2103.03206 — the Perceiver. A learned latent array of N=512 queries cross-attends to the raw input (50,176 ImageNet pixels), so attention costs O(MN) rather than O(M²), and a deep latent transformer then runs at O(N²) per layer, independent of input size. With 2D Fourier-feature positions and no convolution it reaches 78.0% top-1 on ImageNet without pretraining, unchanged when the pixels are permuted, while ViT-B-16 and ResNet-50 given the same features fall to 61.7 and 39.4. The same architecture runs on audio, video and point clouds. | Active |
| LIT-751 | ”Principal Components” Enable A New Language of Images | Wen, Zhao et al. (2025), ARXIV-2503.08685 — Semanticist. A ViT encoder emits a causal 1D sequence of continuous tokens that conditions a DiT diffusion decoder. During training, every token after a uniformly sampled index is replaced by a learned null token (“nested CFG”), so earlier tokens carry more. The result is coarse-to-fine and PCA-like, and any prefix decodes. ImageNet-256 rFID 0.78 with 256×16 floats against ImageFolder’s 0.80 at 286×32. An autoregressive model trained on only the first 32 tokens reaches gFID 2.57. The diffusion decoder is argued to keep image spectrum out of the tokens, but that is shown against other tokenizers, not ablated. | Active |
| LIT-755 | Wonderland: Navigating 3D Scenes from a Single Image | Liang et al. (2024), ARXIV-2412.12091 — Wonderland (CVPR 2025). Fine-tune CogVideoX-5B-I2V for camera control with a ControlNet branch plus LoRA, then train a reconstructor (LaLRM) that regresses 3D Gaussians directly from the video latents rather than decoded frames. Reading latents beat reading RGB at matched cost (RealEstate10K PSNR 27.10 against 25.06). The reconstructor is trained on captured RealEstate10K, ACID and DL3DV first, then fine-tuned on those plus 20K videos the video model generated, supervised by its own decoded frames; that addition moved benchmark PSNR by at most 0.09. | Active |
| LIT-757 | Promptagator: Few-shot Dense Retrieval From 8 Examples | Dai et al. (2022), ARXIV-2209.11755 — Promptagator. Prompt a 137B FLAN with at most 8 examples from the target task to write queries for its corpus, keep a generated pair only if a retriever trained on the generated pairs ranks its source passage top-1, and train a 110M dual encoder on what is left. On 11 BEIR datasets it averages 47.8 nDCG@10 against 46.6 for SPLADE v2 and 46.2 for ColBERT v2, both trained on MS MARCO. The filter is worth 2.5 points on average and lowers three of the eleven datasets. | Active |
| LIT-785 | STARFlow: Scaling Latent Normalizing Flows for High-resolution Image Synthesis | Gu et al., Apple (2025), ARXIV-2506.06276. TarFlow moved into a VAE latent with a deep-shallow split: one deep Transformer flow block and five 2-layer ones. It adds a decoder fine-tuned on noisy latents in place of score denoising, and a guidance rule derived from the Gaussian score. ImageNet 256 FID 2.40 at 1.4B against a retrained pixel TarFlow’s 5.56, with SiT’s quoted 2.06 and MAR’s 1.55 still ahead. ImageNet 512 3.00 against EDM2’s 1.25; COCO zero-shot 9.1; GenEval 0.56. The only head-to-head against diffusion is a plot against an in-house 2.1B DiT trained on 200M images (STARFlow’s schedule is 400M) with a stock decoder. Sampling is about 2.2 s per 256×256 image with guidance. | Active |