representation-and-encoding.
45 of 378 NOTE documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| NOTE-010 | Neural Machine Translation of Rare Words with Subword Units | Translation was open-vocabulary and NMT models were not, backing off to a dictionary for unknown words. Encoding rare and unknown words as subword units — byte pair encoding adapted to segmentation — makes the model open-vocabulary itself. | Read |
| NOTE-028 | Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation | Replaces position embeddings with a fixed linear penalty on attention scores proportional to query–key distance, one slope per head from a geometric sequence. Trains on short sequences and evaluates on longer ones. The paper’s own Appendix B says the gain is “largely explained by” avoiding the early token curse rather than by better use of long history. | Read |
| NOTE-125 | ’Neural-gas’ network for vector quantization and its application to time-series prediction | The key departure from SOM: instead of a fixed spatial grid defining neighborhood, neural gas uses the instantaneous rank of each neuron by distance to the current input. Neuron i gets update weight exp(-rank_i / lambda), where lambda decays over training. | Read |
| NOTE-129 | A Growing Neural Gas Network Learns Topologies | Topology learning can be done incrementally and locally: connect the two nearest units to each input (Hebbian edge), age and prune stale edges, and periodically insert a new unit between the unit with maximum accumulated squared error and its worst-error neighbor. | Read |
| NOTE-185 | Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture | Predict representations, not pixels, and drop hand-crafted augmentations entirely. From one context block, predict what a learned target-encoder outputs for several masked target blocks in the same image. The masking strategy is the whole design: targets large enough to be semantic, context spatially distributed, and the mask applied to the target-encoder’s OUTPUT rather than its input. A ViT-H/14 on ImageNet in under 1200 GPU-hours. | Read |
| NOTE-188 | NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis | Store the scene in the weights of an MLP that maps a 5D coordinate to density and view-dependent colour, and render it with classical volume rendering, which is differentiable — so posed images are the only supervision. Two additions do the work everybody remembers: positional encoding, without which the result is oversmoothed, and hierarchical sampling, without which it is unaffordable. | Read |
| NOTE-206 | The Platonic Representation Hypothesis | Vision and language models measure distance between datapoints more alike as they get larger, across architectures and objectives. The conjecture is that they converge on a model of what generated the data. The proof holds for bijective observations only, and the caption-density experiment tests that boundary rather than asserting past it. | Read |
| NOTE-207 | Contextures: The Mechanism of Representation Learning | A representation is the association between the input and a context variable; capturing its maximum information is optimal on tasks compatible with that context. A context is most useful when the association is neither too strong nor too weak, and mixing contexts is offered as the cheap way to get there. Skimmed: a 313k-character dissertation, read at summary level. | Skimmed |
| NOTE-210 | The Lattice Representation Hypothesis | A linear concept direction plus a threshold is a half-space; intersecting half-spaces gives a concept lattice, so inclusion, intersection and union become geometric meet and join on embeddings. Canonical form when the attribute directions are linearly independent. Evidence is WordNet sub-hierarchies, which is the friendliest available test. | Read |
| NOTE-234 | SimpleStories | Read to see what, if anything, it establishes that its trunk did not. The dataset comparison is direct and well instrumented; the model comparison is confounded three ways and the authors say so. The practice this record takes from it is about the generation procedure, which was measured. | Read |
| NOTE-237 | gen2seg | Read as the counter-route to SAM’s data engine. The headline (narrow finetuning, broad generalization) is well evidenced but the argument lives in the controls, which are unusually good: the same backbone under a conventional head scores ~1.5 mIoU, and swapping only the decoder recovers part of the gap. | Read |
| NOTE-238 | PHi: prediction of hidden states | Read for the instrument. The diagnosis of next-token loss is clean and the boring/interesting separation is convincing on two architectures. The reasoning-selection result is weaker than the abstract implies, and the paper says so itself in a sentence most readers will pass over: picking by next-token loss alone already scores 71%. | Read |
| NOTE-243 | GaussianToken and the cost of the continuous half | Read from the #180 worklist and filed without a practice. The idea — let the quantization units move and size themselves instead of sitting on a grid — is a good one, and the evidence for it cannot be separated from the five continuous parameters each token carries alongside its index. The comparison is at equal token count, and nobody computed bits. | Read |
| NOTE-245 | VQ-VAE: what the three-term loss actually does | Read to give the discrete-tokenizer line a root. The durable content is three loss terms with three different owners and a gradient that is copied rather than derived — an engineering settlement that every descendant still runs unchanged, and that the paper is unusually honest about having chosen rather than justified. | Read |
| NOTE-246 | VQGAN: the loss that made f=16 survivable | Read as the second step of the tokenizer trunk. The architecture is VQ-VAE’s; what is new is a perceptual loss, a patch discriminator, and an adaptive weight that balances them without per-dataset tuning. That loss is the reason a 256×256 image can become 256 tokens and still look like anything, and it is why this line exists at all. | Read |
| NOTE-247 | ViT-VQGAN: where the codebook advice comes from | Read to give SOTA-306 its origin, and it corrected the practice in three places. The advice is not “use small code vectors” but “factorize lookup from embedding”; ℓ₂-normalization is ablated and is the largest single effect in the table; and the dimension-4 row shows 96% codebook usage with near-worst FID, so utilization is necessary and not sufficient. | Read |
| NOTE-252 | RNA-FM, and the control that does not carry the claim | Read to open the biological-sequence trunk, which this record did not hold. The structural results are large and the paper’s one clean input-swap control is not: replacing one-hot sequence with a 640-dimensional pretrained embedding in a published UTR model moves R² from 0.814 to 0.816 on the real-human set, where a 16-dimensional predicted secondary structure moves it to 0.820. No practice filed. | Read |
| NOTE-253 v2 | The information-bottleneck rebuttal, and the assumption it makes visible | Read from the ICLR 2018 version, supplied by the record’s owner; co-author Kolchinsky’s publications page also links it. Three claims tested, none general. The finding worth more than the headline is in Appendix C: the same tanh network, binned evenly in net input instead of evenly in activity, shows no compression — because I(h;X) in a deterministic network is infinite and every finite number is a property of an imposed noise model. | Read |
| NOTE-254 | Adaptive estimators: a counter-rebuttal whose body is weaker than its abstract | Read to check whether it softens LIT-509, as #220 feared. It does on one narrow point and not on the one that matters. Its own Figure 6 shows a ReLU network averaged over 50 initializations has no distinct phase, though its Figure 7 averages do show ELU, Swish and centered softplus compressing; and it agrees with the rebuttal that hidden-layer compression does not correlate with generalization. | Read |
| NOTE-256 | Image-GS: the rate accounting GaussianToken skipped, and an ablation that inverts its own headline | Read from the #180 worklist for its adjacency to the tokenizer trunk. Two things worth keeping: a comparison run the way this record keeps asking for — six baselines at matched model size, bitrate on the axis — and an ablation in which the two unglamorous optimizer choices outweigh the content-adaptive allocation the paper is named for. | Read |
| NOTE-270 | Concepts live in the quiet directions, and the paper labels its own claims robust or partial | Read last of the batch and the easiest call to make, for a reason that has little to do with the finding: it arrives with a table grading its own six claims, a robust null against the hypothesis it set out to confirm, and partial results labelled partial. The finding itself is that contextual concept directions sit at the bottom of the unembedding spectrum and static vocabulary contrasts sit at the top. | Read |
| NOTE-275 | The trunk states the condition its successors were refuted for dropping | Read because the record filed a three-paper information-bottleneck dispute and the paper all three argue about was absent — a gap this session created. Reading it reassigns some blame: the rebuttal’s central objection is that mutual information needs stochasticity, and the founding paper says so in one sentence before anybody measured anything. | Read |
| NOTE-291 | Matryoshka Representation Learning | Sum the task loss over log(d) nested prefixes of one embedding, and each prefix matches a separately trained model of that width on ResNet50 and ImageNet. Retrieval that shortlists on 16 dimensions and re-ranks on 2048 is 14× faster at equal mAP@10. Read §1–6 and Appendices D.2 and I; the remaining appendix tables were skimmed. | Read |
| NOTE-293 | Gemma 3n | Per-Layer Embeddings: every token id indexes a table with one 256-d vector per layer, gated into the residual at each layer, about 2.35B parameters held off the accelerator. No measurements are published. Read from the model overview, the developer guide, and the Hugging Face transformers implementation, which is the only place the mechanism is fully specified. | Read |
| NOTE-296 | Fourier features | A coordinate MLP’s NTK has fast spectral falloff, so it fits high frequencies too slowly to matter. A sinusoidal input mapping makes the composed kernel stationary and sets its bandwidth, and the NTK linear model predicts the trained networks’ loss curves. Gaussian random frequencies are best on seven tasks, and only their scale matters. Read §1–7; the appendices were not read. | Read |
| NOTE-311 | FLUX.2 latent-space report | A controlled comparison of four autoencoder latents for flow models on ImageNet, with the timestep schedule swept per latent. The schedule decides the ranking. FLUX.2’s VAE gets near-RAE learnability with the best reconstruction. Read in full from the rendered report, except the interactive Table 2, which did not load. | Read |
| NOTE-322 v2 | Latent Dirichlet Allocation | The bag-of-words assumption, read through de Finetti, implies a document-level mixture. LDA is that mixture with Dirichlet topic proportions. It generalizes better than one-topic-per-document and per-document-parameter models by held-out perplexity. Main text read in full; the variational derivations in the appendices skimmed. | Read |
| NOTE-323 | Reading Tea Leaves | Two human tasks, word intrusion and topic intrusion, measure whether a topic model’s topics and document assignments mean anything to people. The model with the best held-out likelihood, CTM, does worst on both. Read in full. | Read |
| NOTE-324 | Automatic Evaluation of Topic Coherence | Average pairwise PMI of a topic’s top words, estimated on Wikipedia, correlates with human coherence ratings about as well as the paper’s measure of annotator agreement. Knowledge-base and search-engine measures are less reliable. Read in full. | Read |
| NOTE-325 | word2vec (architectures) | CBOW and skip-gram, log-linear models that learn word vectors from billions of words cheaply, and the analogy benchmark that made “king − man + woman ≈ queen” a test. Read in full. | Read |
| NOTE-326 | word2vec (negative sampling, subsampling, phrases) | The tricks that made word2vec the standard: negative sampling with a unigram^(3/4) noise distribution, subsampling of frequent words, and data-driven phrase tokens, plus additive compositionality. The 3/4 exponent is asserted without numbers. Read in full. | Read |
| NOTE-327 | GloVe | Derives a word-embedding objective from the claim that meaning lives in ratios of co-occurrence probabilities. It fits log co-occurrence counts by weighted least squares and beats default-configured word2vec on analogies. Read in full. | Read |
| NOTE-329 | Levy, Goldberg & Dagan: lessons from word embeddings | The prediction-over-count advantage for word embeddings came from design choices bundled with word2vec, not from the algorithm. Ported to PPMI and SVD and tuned alike across 672 representations, no method wins consistently, SGNS beats GloVe, and context-distribution smoothing is the only universally safe setting. Read in full. | Read |
| NOTE-330 | Baroni, Dinu & Kruszewski: Don’t count, predict! | 84 count and predict models on 14 lexical-semantics benchmarks from one 2.8B-token corpus. word2vec CBOW beats count vectors almost everywhere and is far more robust to bad settings. The count side lacked word2vec’s smoothing and negative shift, which is how LIT-607 later reversed the result. Read in full. | Read |
| NOTE-331 | Levy & Goldberg: SGNS as implicit PMI factorization | SGNS’s objective is optimized at w·c = PMI(w, c) − log k, a weighted factorization of the shifted PMI matrix. Shifted PPMI nearly attains that optimum and SVD over it matches SGNS on similarity, while SGNS keeps an edge on syntactic analogies. Read in full. | Read |
| NOTE-332 | Arora et al.: RAND-WALK and PMI in low dimensions | A random-walk generative model with isotropic word vectors predicts PMI ≈ ⟨v, v’⟩/d in low dimensions, justifies GloVe’s biases and CBOW’s averaging, and explains why analogies survive large fitting error: isotropy denoises the offset. Its assumptions are checked loosely on one corpus. Read in full, with the main-text proofs. | Read |
| NOTE-333 v2 | Wan | An open 14B text-to-video recipe whose evidence is mostly systems measurement: 2D context parallelism cuts communication overhead from over 10% to under 1%, and caching and 8-bit kernels give 1.62× and 1.27×. Its three modelling ablations run on 1.3B text-to-image. Two are read from training loss, and on FID one of them favours the rejected option. The applications in §5 are demonstrated almost entirely by figures. | Read |
| NOTE-334 | Movie Gen | At 5B, flow matching beats v-prediction diffusion on quality by 3.3 annotation σ, but on text alignment by only 1.9σ. Video-native captions beat captions stitched from three frames on alignment by 2.9σ. Every σ in the paper is rater variance, and each ablation arm was trained once. The Llama 3 block beats a DiT block that differs only in norm, activation and bias. The scaling claim rests on image-only runs covering less than a quarter of a decade of compute. | Read |
| NOTE-338 v2 | Step-Video-T2V | A 30B text-to-video DiT on a 16×16×8 VAE. By the authors’ own benchmark the VAE reconstructs about as well as HunyuanVideo’s 8×8×4, and an independent measurement in Open-Sora 2.0 disagrees. Video DPO raises a tie-weighted preference score from 45% to 55% on 300 prompts. Every architecture comparison is reported without numbers, and the “state of the art” in the abstract is contradicted by the paper’s own Table 9. | Read |
| NOTE-342 | MAGVIT-v2 | A causal 3D-CNN video tokenizer with lookup-free quantization (sign-binary latents plus an entropy penalty) improves a fixed ~307M masked token model on Kinetics-600 frame prediction from FVD 9.9 to 5.2 at equal decoding steps. The claim that LFQ, unlike VQ, keeps improving generation as the vocabulary grows rests on a figure with no numbers. “Beats diffusion” holds only with guidance (1.78 against MDT’s 1.79 at 256, 1.91 against VDM++‘s 2.65 at 512), and the strongest diffusion baselines at 512 are about 6× its size. | Read |
| NOTE-344 | HunyuanVideo | A 13B open text-to-video model whose one quantitative design argument is a pair of compute-optimal fits on a proxy family: the image fit puts more of each added FLOP into parameters (0.5634 against 0.4325), and the video fit, initialized from the image envelope, reverses it (0.3618 against 0.6289). No architectural or data choice is ablated, and the proxy differs from the shipped model in objective, text encoder and conditioning. | Read |
| NOTE-345 | Open-Sora 2.0 | An 11B video model initialized from FLUX, trained mostly at 256px, and adapted to 768px as text/image-to-video. The “$200k” is 4,160 H200 GPU-days for one final run on HunyuanVideo’s VAE. The 4×32×32 Video DC-AE is a separate, unconverged experiment whose cost is outside that figure. Every training-strategy choice that makes the run cheap is asserted, not ablated. | Read |
| NOTE-346 | CogVideoX | An open text-to-video DiT whose design choices are backed by curves rather than numbers. On full 3D against factorized attention, the text says only that factorized FVD is “much higher … in early steps” and that factorized training is “unstable and prone to collapse”. Model size, steps and resolution are unstated. The one measured number is the cost: full attention’s forward pass is 1.08×, 1.67× and 2.30× slower at three sizes. | Read |
| NOTE-350 v2 | LTX-Video | A 1.9B video DiT on a 32×32×8, 128-channel VAE with no patchifier (1:8192 pixels to tokens), whose decoder also does the last denoising step. It is fast and preferred over same-size 2024 open models in a 20-rater survey. The VAE has no reconstruction metric at all, and the denoising decoder, the paper’s central idea, rests on one unquantified internal study. | Read |
| NOTE-367 | Narang et al., transformer modifications | About fifty transformer modifications reimplemented in one T5/Mesh TensorFlow codebase at 223M parameters, with hyperparameters fixed, on four task families. Most do not beat the vanilla pre-norm, relative-attention baseline. The winners are GLU-variant activations, RMSNorm, untied embeddings, and sparse experts or other variants that cost parameters or time. The Universal Transformer, ReZero, Fixup, block sharing and the Evolved Transformer all lose. The positives are largely same-codebase re-runs, GLU Variants included, so the paper is strong evidence against the losers and weak evidence that the winners transfer. | Read |