Reading statuses — How deeply the paper was READ — not whether it is any good, which is lit-statuses. The two disagree in every direction: a foundational paper can carry a Skimmed note, and a paper in the attic can carry a Read one, since reading it carefully is often how it got there (ADR-025).
Read — the full text, closely enough to state its assumptions and results exactly — the only status that licenses a claims table.
366 of 378 NOTE documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| NOTE-001 | Understanding and Improving Layer Normalization | LayerNorm’s benefit is in the backward pass — the derivatives of the mean and variance re-center and re-scale the gradients — not in forward normalization. Its bias and gain increase overfitting risk and “do not work in most cases”. | Read |
| NOTE-002 | Efficient Online Data Mixing | An Exp3 bandit over data domains, rewarded by per-domain training loss on the batches the run is already taking, revising the mixture during training. 19% fewer iterations to the next best method’s final perplexity, at negligible wall-clock cost. | Read |
| NOTE-003 | PyTorch FSDP | Fully Sharded Data Parallel as an industry-grade PyTorch component: a sharding factor F generalising replication through full sharding, communication overlapped by backward prefetching, and a rate limiter over the CUDA caching allocator. The first cluster in #114 whose practices its paper actually supports. | Read |
| NOTE-004 | Scale Efficiently | Model shape, not only model size, determines downstream fine-tuning quality — and the widely adopted T5-Base and T5-Large configurations are Pareto-inefficient. Read in full for #114 after its LIT note was found to describe a different paper. | Read |
| NOTE-005 | FlashAttention | Attention is bounded by HBM traffic rather than FLOPs. Tiling to on-chip SRAM and recomputing the attention matrix in the backward pass gives exact attention in Θ(N²d²M⁻¹) HBM accesses against standard attention’s Θ(Nd + N²) — and Proposition 3 proves no exact algorithm beats that across all SRAM sizes. | Read |
| NOTE-006 | GPipe | Pipeline parallelism by splitting a mini-batch into micro-batches. Bubble overhead is O((K−1)/(M+K−1)) and negligible once M ≥ 4K; re-materialization plus partitioning cuts peak activation memory from O(N×L) to O(N + (L/K)(N/M)). | Read |
| NOTE-007 | FlashAttention-2 | FlashAttention reached only 25–40% of peak FLOPs/s; the loss was work partitioning between thread blocks and warps, not the algorithm. Three changes — fewer non-matmul FLOPs, parallelism over sequence length, better warp partitioning — give ~2× and 50–73% of peak. | Read |
| NOTE-008 | Visualizing the Loss Landscape of Neural Nets | Filter normalization makes loss-surface plots comparable across architectures, and under it sharpness correlates with generalization error. Deep networks transition from nearly convex to chaotic; skip connections prevent that transition, which is why they are needed at depth. | Read |
| NOTE-009 | Latent Diffusion Models | Train the diffusion model in the latent space of a pretrained autoencoder rather than in pixels. The autoencoder removes imperceptible detail; the diffusion model then spends its capacity on semantics instead of on high-frequency content nobody sees. | Read |
| NOTE-010 | Neural Machine Translation of Rare Words with Subword Units | Translation was open-vocabulary and NMT models were not, backing off to a dictionary for unknown words. Encoding rare and unknown words as subword units — byte pair encoding adapted to segmentation — makes the model open-vocabulary itself. | Read |
| NOTE-011 | Attention Is All You Need | Dispenses with recurrence and convolution entirely. The 1/√d_k scaling has a stated reason: with unit-variance components, q·k has variance d_k, and unscaled dot products push the softmax into regions of extremely small gradients. | Read |
| NOTE-012 | LARS: Large Batch Training of Convolutional Networks | Linear LR scaling with warm-up is “not general enough and training may diverge”. Layer-wise Adaptive Rate Scaling sets a per-layer rate from the ratio of weight norm to gradient norm, reaching AlexNet at batch 8K and ResNet-50 at batch 32K without accuracy loss. | Read |
| NOTE-013 | DeepNet | Scale the residual by α and initialise the residual branch with gain β, both constants determined only by depth. Bounds the model update theoretically, combines Post-LN’s quality with Pre-LN’s stability, and reaches 1,000 layers. | Read |
| NOTE-014 | Monarch Mixer | One sub-quadratic primitive — Monarch matrices — along both sequence length and model dimension. Matches BERT-base/large on GLUE with up to 27% fewer parameters and 9.1× throughput at 4K, and beats ViT-b by 1% at half the parameters. Attention-free, not hybrid. | Read |
| NOTE-015 | Decoupled Weight Decay Regularization | L2 regularization and weight decay are equivalent for SGD and not for adaptive methods. Common Adam implementations do L2 while calling it weight decay; decoupling it recovers the real thing, separates the weight-decay and learning-rate choices, and closes Adam’s generalization gap to SGD with momentum. | Read |
| NOTE-016 | Fast Transformer Decoding: multi-query attention | Incremental decoding is bounded by the memory bandwidth of reloading the keys and values, not by arithmetic. Share one key/value head across all query heads: much faster decoding, “only minor quality degradation”. | Read |
| NOTE-017 v2 | Scaling Laws for Neural Language Models | Loss is a power law in model size, dataset size and compute over seven orders of magnitude, and shape barely matters. Its learning-rate finding is the opposite of what the record recorded: larger models require a smaller rate to avoid divergence, and the paper carries an explicit LR(N) rule. | Read |
| NOTE-018 | Segment Anything | The data engine is the transferable part. Three stages — assisted-manual, semi-automatic, fully automatic — each stage’s labels training the model that produces the next stage’s, ending in 1.1B masks. The model and the dataset are built together. | Read |
| NOTE-019 v2 | Elucidating the Design Space of Diffusion-Based Generative Models | Pulls diffusion’s tangled formulations apart into independent axes — sampler, training noise distribution, and preconditioning — and derives the preconditioning coefficients from a unit-variance requirement rather than choosing them. FID 1.79 on class-conditional CIFAR-10 at 35 network evaluations; a pretrained ImageNet-64 model improves 2.07 to 1.55 from the sampler alone. | Read |
| NOTE-020 | Don’t Stop Pretraining | Two adaptive-pretraining stages, and the second is the surprise. Domain-adaptive pretraining (DAPT) on a large in-domain corpus helps; task-adaptive pretraining (TAPT) on the task’s own small unlabelled set also helps, and the two are complementary rather than alternatives. | Read |
| NOTE-021 | Grouped-Query Attention | MQA is fast and costs quality, and training a separate model for inference is undesirable. Two results: uptrain an existing multi-head checkpoint with 5% of pretraining compute, and use an intermediate number of key-value heads — quality near multi-head at speed near MQA. | Read |
| NOTE-022 | RMSNorm | LayerNorm gives re-centering and re-scaling invariance; the hypothesis is that re-centering is dispensable. Normalize by root mean square alone — re-scaling invariance and implicit learning-rate adaptation kept, 7–64% less time per step. | Read |
| NOTE-023 | PagedAttention and vLLM | KV cache memory is huge, grows and shrinks dynamically, and was managed as one contiguous block — so fragmentation and duplication capped the batch size. PagedAttention borrows OS paging: near-zero waste, sharing within and across requests, 2–4× throughput at equal latency. | Read |
| NOTE-024 v2 | ReLoRA: High-Rank Training Through Low-Rank Updates | Restart LoRA repeatedly during pretraining — merge the adapter, reinitialise it, prune the optimizer state, re-warm the learning rate — so a sequence of low-rank updates sums to a high-rank one. The ablation is the finding: the full-rank warm start it also requires accounts for most of the measured gain, and the restart machinery adds 0.42 perplexity on top of it. | Read |
| NOTE-025 | AdaNorm: Adaptive Gradient Norm Correction based Optimizer for CNNs | Keeps an EMA of the gradient’s scalar L2 norm and, when the current gradient is smaller than that history, scales it up to match — a floor under the gradient norm, applied to Adam’s first moment only. The exact mirror image of gradient clipping. Evidence is VGG/ResNet on CIFAR-10, CIFAR-100 and TinyImageNet; there is no transformer and no language result. | Read |
| NOTE-026 | T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models | Argues the control is already in the frozen model and only needs aligning to an external signal, so the adapter can be small — it learns an alignment, not a generation ability. Adapters compose, and transfer unchanged to any model fine-tuned from the same base. The composition weights are manual, which the authors state as the limitation. | Read |
| NOTE-027 | Simple Open-Vocabulary Object Detection with Vision Transformers | Transfers a contrastively pretrained image-text model to open-vocabulary detection with minimal architectural change. Its transferable finding is about fine-tuning: the text encoder needs a learning rate about 100× lower than the image encoder, freezing it entirely is also wrong, and the recipe that maximises zero-shot transfer is not the one that maximises in-distribution performance. | Read |
| NOTE-028 | Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation | Replaces position embeddings with a fixed linear penalty on attention scores proportional to query–key distance, one slope per head from a geometric sequence. Trains on short sequences and evaluates on longer ones. The paper’s own Appendix B says the gain is “largely explained by” avoiding the early token curse rather than by better use of long history. | Read |
| NOTE-029 | ZeRO++: Extremely Efficient Collective Communication for Giant Model Training | Attacks each of ZeRO’s three communication collectives separately — block-quantized weight all-gather, a secondary weight shard held redundantly to trade memory for communication, and an all-to-all quantized gradient reduction replacing reduce-scatter. 4× less communication, up to 2.16× throughput at 384 GPUs. | Read |
| NOTE-030 | Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism | Splits a transformer layer across GPUs by choosing the partition that needs no communication in the middle — column-parallel then row-parallel — so a whole layer costs two all-reduces forward and two backward. No compiler, no library changes, a few lines of PyTorch. 8.3B parameters on 512 GPUs at 76% scaling efficiency. | Read |
| NOTE-031 v2 | Data Selection for Language Models via Importance Resampling | Selects pretraining data by importance resampling in a hashed n-gram feature space — 10,000 buckets of unigram and bigram counts — to match a target distribution. The more useful result is the metric: KL reduction in that feature space predicts downstream performance across selection methods, including manual curation, which uses no n-grams at all. | Read |
| NOTE-032 | Gaussian Shell Maps for Efficient 3D Human Generation | Puts 3D Gaussians on inflated and deflated shells around a template human body, with a CNN generating the texture stack whose features become the Gaussians’ attributes. Rendering fast enough removes the need for the 2D upsamplers that made prior 3D GANs multi-view inconsistent — a quality problem solved by a speed fix. | Read |
| NOTE-033 | Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B | The 3D-parallelism report — 8-way tensor, 35-way pipeline, data parallel over 560 DGX A100 nodes — and an unusually complete training recipe, including a batch-size ramp from 32 to 1920 and β₂ lowered to 0.95 to suppress loss spikes. Also the corpus’s cleanest artefact of pre-Chinchilla misallocation: 530B parameters trained on 270B tokens. | Read |
| NOTE-034 | An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion | Personalizes a frozen text-to-image model by optimizing a single new token embedding — nothing else changes. Frames the result as a point on a distortion/editability tradeoff borrowed from GAN inversion, and adds per-image tokens so the shared concept and the incidental details separate into different embeddings. | Read |
| NOTE-035 | Neural Scene Graphs for Dynamic Scenes | The first neural rendering method to decompose a dynamic scene into a scene graph — objects with learned transformations and radiance, described by a single implicit function plus a per-object latent — so novel arrangements can be rendered, not just novel views. Learned from a video alone. | Read |
| NOTE-036 | DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation | Fine-tunes the whole model to bind a rare token to a subject from 3–5 images, and counters the resulting language drift with a prior-preservation loss that supervises the model with its own pre-fine-tuning samples. Beats textual inversion on both subject and prompt fidelity, and needs the super-resolution stages fine-tuned too. | Read |
| NOTE-037 | K-Planes: Explicit Radiance Fields in Space, Time, and Appearance | Represents a d-dimensional scene with “d choose 2” planes, so static (d=3) and dynamic (d=4) scenes are the same model at different d. The factorisation makes dimension-specific priors easy to attach and separates static from dynamic components naturally; a linear decoder with a learned colour basis matches a nonlinear MLP. | Read |
| NOTE-038 | PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding | Fuses an arbitrary number of reference images into one “stacked ID embedding” attached to a class word, so personalization needs no per-subject tuning at inference. Because the stack is just a set, feeding it images of different people at inference merges identities — a capability that falls out of the representation rather than being designed. | Read |
| NOTE-039 | PaLM 2 Technical Report | Independently re-derives compute-optimal scaling at larger compute than Chinchilla and lands on the same answer — grow data and parameters roughly 1:1, against the earlier 3:1 model-first trend. States the inference-economics corollary plainly: for a fixed training and inference budget, train a smaller model on more tokens rather than making the architecture inference-efficient. | Read |
| NOTE-040 | Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding | Finds that scaling a frozen text-only language model improves text-to-image sample quality significantly more than scaling the image diffusion model — an allocation result, not an architecture one. Also diagnoses why high guidance weights destroy images (predictions leave the training range and compound through iterative sampling) and fixes it with dynamic thresholding. | Read |
| NOTE-041 | Population Based Training of Neural Networks | Train a population of models in parallel; periodically let a poor performer copy a good one’s weights (exploit) and perturb its hyperparameters (explore). The output is a hyperparameter schedule rather than a fixed setting, discovered inside a single training run at roughly the cost of the population. | Read |
| NOTE-042 | Instant Neural Graphics Primitives with a Multiresolution Hash Encoding | Replaces a large coordinate MLP with a small one fed by trainable feature vectors stored in a multiresolution hash table. Hash collisions are not resolved — the multiresolution structure lets the network disambiguate them, and gradient descent averages the colliding entries by importance. Seconds to train, milliseconds to render. | Read |
| NOTE-043 v2 | DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling | Notices that the diffusion ODE is semi-linear, solves its linear part analytically, and applies a numerical method only to the intractable neural-network integral. Training-free, 10–20 function evaluations, 4.70 FID at 10 NFE on CIFAR-10, and DDIM turns out to be exactly the first-order case. | Read |
| NOTE-044 | OnePose: One-Shot Object Pose Estimation without CAD Models | Estimates object pose from a single scanned video, with no CAD model and — the part the record’s own summary got backwards — no instance- or category-specific training. Builds an SfM point cloud of the object once, then matches 2D query points to it with a generic graph attention network that was never trained on that object or its category. | Read |
| NOTE-045 | 3D Gaussian Splatting for Real-Time Radiance Field Rendering | Represents a scene as anisotropic 3D Gaussians initialised from structure-from-motion points, optimised with interleaved density control, and rendered by projecting to 2D and α-blending. Differentiable like a volumetric field, rasterizable like geometry — the first real-time (≥30 fps) radiance-field rendering at 1080p. Static scenes. | Read |
| NOTE-046 | NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstruction | Cuts NeuS surface reconstruction from about 8 hours to minutes for a static object, and to as little as 20 seconds per frame for a moving sequence. The point of the speedup is what it unlocks: at 8 hours per object, dynamic scenes with thousands of frames are simply not attemptable. | Read |
| NOTE-047 | MiDaS v3.1 - A Model Zoo for Robust Monocular Relative Depth Estimation | Swaps the image encoder in a fixed depth-estimation architecture across BEiT, Swin, SwinV2, Next-ViT, LeViT and recent convolutional backbones, and reports the resulting performance-runtime frontier. The best backbone improves depth quality by 28%; the paper’s other deliverable is the procedure for integrating a new one. | Read |
| NOTE-048 | Progressive Distillation for Fast Sampling of Diffusion Models | Repeatedly halves a diffusion sampler’s step count by distilling two teacher steps into one student step, from 8192 down to 4 at FID 3.0 on CIFAR-10, for no more total compute than training the original model. Also where v-prediction comes from — and the reason for it is that ε-prediction’s implied x̂ blows up as signal-to-noise goes to zero. | Read |
| NOTE-049 | Learning to Prompt for Vision-Language Models | Replaces hand-written CLIP prompts with continuous context vectors learned end-to-end while every pretrained parameter stays frozen. One or two shots beat prompt engineering; sixteen shots average about 15% gain, over 45% at the best. The paper also reports where it overfits and names the cause as noisy labels. | Read |
| NOTE-050 | PipeMare: Asynchronous Pipeline Parallel DNN Training | Drops the synchronous bubble in pipeline parallelism by letting stages run on stale weights, and makes that converge with two corrections — rescale the learning rate by the stage’s delay, and approximate the weights the backward pass should have seen. Up to 4.6× the pipeline utilization of GPipe, at up to 2× less weight and optimizer memory than PipeDream. | Read |
| NOTE-051 | Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models | 204 tasks from 450 authors, evaluated across dense and sparse models from millions to hundreds of billions of parameters, with expert human raters as the baseline. Its most useful finding is about measurement: tasks showing “breakthrough” behaviour at a critical scale tend to involve multiple steps or brittle metrics, while smooth ones are knowledge-heavy. | Read |
| NOTE-052 v2 | Progress measures for grokking via mechanistic interpretability | Reverse-engineers a one-layer transformer trained on modular addition and finds it computes a Fourier multiplication algorithm — embed inputs as rotations, combine with trigonometric identities. Uses that to define continuous progress measures showing grokking is three phases, and finds grokking vanishes entirely above about 60% data. | Read |
| NOTE-053 v2 | Consistency Models | Trains a model to map any point on a diffusion trajectory directly to its origin, so one network evaluation generates a sample — FID 3.55 on CIFAR-10 and 6.20 on ImageNet-64 in one step. Trainable either by distilling a diffusion model or from scratch, and the from-scratch mode is what makes it a model family rather than a distillation trick. | Read |
| NOTE-055 | CoCa: Contrastive Captioners are Image-Text Foundation Models | Trains one encoder-decoder with both a contrastive and a captioning loss by omitting cross-attention in the first half of the decoder — so those layers produce unimodal text embeddings for the contrastive loss, and the cascaded remainder cross-attends for captioning. Both objectives share one computational graph, at minimal extra cost. | Read |
| NOTE-056 | PowerNorm: Rethinking Batch Normalization in Transformers | Diagnoses why batch normalization underperforms layer normalization in NLP — the batch mean and variance fluctuate far more than in vision — and fixes it by dropping the zero-mean step and normalizing by a running quadratic mean instead of a batch variance. Beats LN by 0.4/0.6 BLEU and 5.6/3.0 perplexity with no hyperparameter changes. | Read |
| NOTE-057 | What Makes for Good Views for Contrastive Learning? | Argues for an InfoMin principle: views should share as little mutual information as possible while keeping task-relevant information intact. Too much shared information makes the contrastive task trivial, too little destroys the signal, so there is a sweet spot — and augmentation design is the search for it. | Read |
| NOTE-058 | Scaling Vision Transformers to 22 Billion Parameters | Three architectural changes carry ViT from 4B to 22B — parallel attention/MLP blocks, QK normalization, and removed biases. QK-norm is the load-bearing one: training diverged around 8B from attention logits growing until the softmax was almost one-hot with near-zero entropy, and LayerNorm on the queries and keys before the dot product fixed it. | Read |
| NOTE-059 | Longformer: The Long-Document Transformer | Replaces full self-attention with a sliding window plus a small set of task-chosen global positions, giving linear cost in sequence length. The global tokens are the design decision that matters — they are where the task’s inductive bias is stated, and they are why the pattern can be adapted per task rather than fixed. | Read |
| NOTE-060 | Improving Language Models by Retrieving from Trillions of Tokens | Matches GPT-3 on the Pile with 25× fewer parameters by cross-attending to chunks retrieved from a 2-trillion-token database, using a frozen BERT retriever that never needs training. The gain is constant from 150M to 7B, and the model improves at evaluation time by enlarging the database — capability added after training ends. | Read |
| NOTE-061 | Pruned Neural Networks are Surprisingly Modular | Imports modularity from graph clustering — a module is neurons with strong internal and weak external connectivity — and measures it by spectral clustering on MLP weights. Trained and pruned networks are more modular than random ones, and than random networks with the same sparse weight distribution, which is the control that makes the result mean something. | Read |
| NOTE-062 | An Empirical Model of Large-Batch Training | Defines the gradient noise scale — the ratio of the gradient’s variance to its squared norm — and shows it predicts, to an order of magnitude, the batch size past which data parallelism stops buying speed. Also the source of “critical batch size” as a measurable quantity, and of the hyperbola relating steps taken to examples processed. | Read |
| NOTE-063 | One Transformer Fits All Distributions in Multi-Modal Diffusion at Scale | Observes that marginal, conditional and joint distributions are all “predict the noise in perturbed data” if you allow each modality its own timestep. One network trained with per-modality timesteps then serves text-to-image, image-to-text, joint generation and unconditional generation — and classifier-free guidance falls out for free. | Read |
| NOTE-064 | Hierarchical Text-Conditional Image Generation with CLIP Latents | Splits text-to-image into a prior that predicts a CLIP image embedding from a caption and a decoder that renders from that embedding. The split buys diversity rather than fidelity — humans slightly prefer GLIDE’s photorealism, and strongly prefer unCLIP’s diversity. Its own reconstruction figures show the CLIP embedding loses attribute binding. | Read |
| NOTE-065 | Pay Less Attention with Lightweight and Dynamic Convolutions | Shows a depthwise convolution with softmax-normalized weights, shared across channel groups and with a kernel fixed at 31 or less, matches self-attention on translation — 29.7 BLEU on WMT En-De. Dynamic convolution predicts the kernel from the current token alone, so the mixing weights depend on position rather than on the pair of positions being mixed. | Read |
| NOTE-066 | Contrastive Learning with Hard Negative Samples | Samples hard negatives without labels by reweighting the contrastive objective rather than changing the sampling procedure — two extra lines of code, no computational overhead — with a tunable hardness knob. The knob exists because harder negatives are more likely to be false negatives, so hardness trades learning signal against contamination. | Read |
| NOTE-067 v2 | Denoising Diffusion Implicit Models | Generalises DDPM to a family of non-Markovian forward processes that share DDPM’s exact training objective, so the sampler can be replaced without retraining. Picking the deterministic member gives 10–50× fewer steps, plus latent interpolation and encoding that a stochastic sampler cannot do. Not a continuous-time or SDE paper — that is a different one. | Read |
| NOTE-068 | Denoising Diffusion Probabilistic Models | The paper that made diffusion work. Its Table 2 is the underquoted part: predicting ε rather than the posterior mean matters only under the simplified objective, learned reverse-process variances destabilise training, and the principled variational bound gives better codelengths while the unprincipled simplified loss gives better samples. | Read |
| NOTE-069 | PaLI: A Jointly-Scaled Multilingual Language-Image Model | Scales language and vision components jointly across 100+ languages and shows a properly scaled model handles many languages while keeping English-only state of the art. Its most useful passage is an ablation honest enough to explain a regression by the language ratio of the data that caused it. | Read |
| NOTE-070 | Understanding Contrastive Learning Requires Incorporating Inductive Biases | Shows that theoretical guarantees for contrastive learning that depend only on the augmentations and the loss value cannot explain why it works, and are provably vacuous in some settings. Different function classes and algorithms behave very differently downstream given identical augmentations and identical contrastive loss. | Read |
| NOTE-071 | Adding Conditional Control to Text-to-Image Diffusion Models | Adds spatial control to a frozen diffusion model by cloning its encoder into a trainable branch joined to the original by zero-initialized convolutions, so the adapter starts as an exact no-op and grows away from it. Trains robustly on datasets from under 50k to over a million. | Read |
| NOTE-072 v2 | Scaling Laws for Autoregressive Generative Modeling | Extends the language scaling laws to image, video, multimodal and mathematics, finding the same power-law-plus-constant form and a nearly domain-independent exponent for optimal model size, N_opt(C) ∝ C^0.7. That exponent implies D ∝ N^0.4 — sub-linear data scaling, which Chinchilla later overturned. The paper also flags the inconsistency that overturned it. | Read |
| NOTE-073 | On the Convergence of Adam and Beyond | Constructs a one-dimensional convex problem on which Adam provably converges to the worst point in the feasible set, locating the error in the original convergence proof and in the exponential moving average itself. Proposes AMSGrad, which keeps the running maximum of the second moment so the effective learning rate is non-increasing. ICLR 2018 best paper; almost nobody runs AMSGrad. | Read |
| NOTE-075 | Hyper-ES | Argues from two lemmas that full-parameter ES cannot work at LLM scale — random perturbations concentrate near orthogonality to any useful descent direction, and the leftover orthogonal components accumulate as a random walk away from the pretrained weights — and replaces the search space with the span of a few cheap GRPO-derived LoRA directions, searched by CMA-ES. | Read |
| NOTE-076 | Understanding Evolution Strategies for LLM Reasoning | Three questions about what ES post-training does rather than whether it wins: it raises pass@1 and pass@k together where GRPO trades one for the other, its 40x larger parameter drift is functionally sparse and does not produce broad forgetting, and the population size needed for stable training falls as the model grows. | Read |
| NOTE-077 | Beyond the Best Guess | Reinforcement learning raises pass@1 and lowers pass@k, eventually below the base model, because it increases the number of prompts solved in none of k samples; evolution strategies raise both and never fall below base at any k or scale tested, up to 32B against published RL checkpoints. | Read |
| NOTE-078 | Neural Thickets | The neighbourhood of a pretrained weight vector is dense with task-improving Gaussian perturbations, the density rises monotonically with model scale, and the perturbations that help are specialists rather than uniform improvements — which together are why a single parallel round of random guessing plus majority voting can match PPO, GRPO and ES at equal training FLOPs. | Read |
| NOTE-079 | EGGROLL, Unrolled | Works out what EGGROLL’s low-rank update converges to — a resolvent-filtered gradient, which can be nonconservative and can turn an attracting optimum into a repelling one, while being exact on every quadratic objective — and turns the finite-population half of the analysis into LOO-ROLL, which replaces two antithetic evaluations per direction with one leave-one-out evaluation at unchanged expected field. | Read |
| NOTE-080 | Evolution Strategies at the Hyperscale | Structuring each ES worker’s perturbation as a rank-r Gaussian product instead of a dense matrix raises arithmetic intensity by enough for a hundredfold throughput gain at billion scale, while the population average stays high-rank and the low-rank update provably approaches the full-rank one at a rate faster than the usual parametric O(1/sqrt(r)). | Read |
| NOTE-081 | Overcoming Forgetting in LLM Fine-Tuning with Evolution Strategies | Prior-task degradation under ES is transient drift rather than irreversible forgetting, is not specific to ES, and is driven by the random walk Hoy et al. predicted — whose size falls with population exactly as their equation says. Anchored Weight Decay buys a population-128 reduction at population 30 for 1-2% runtime. | Read |
| NOTE-082 | Evolutionary Strategies lead to Catastrophic Forgetting in LLMs | The first independent replication attempt of ES-beats-GRPO, which fails — ES lands close but GRPO stays ahead on three of four settings at 1B and 1.5B — and the first measurement of what ES costs on a held-out capability: HellaSwag declines steadily as Countdown training continues past the point Countdown itself converged. | Read |
| NOTE-083 | Matching Accuracy, Different Geometry | ES and GRPO reach the same accuracy by travelling in nearly orthogonal directions two orders of magnitude apart in length, and land in the same basin with no loss barrier between them — because most of the ES displacement is a loss-invariant random walk whose squared norm grows as sigma^2 d T / N. | Read |
| NOTE-084 | The Blessing of Dimensionality in LLM Fine-tuning | One geometric property — a fine-tuning landscape whose curvature lives in a few stiff directions amid a near-zero bulk — accounts for both why a population of thirty keeps working as models grow and why the training reward rises, peaks and decays under fixed hyperparameters, in GRPO as well as ES. | Read |
| NOTE-085 | ESSA | The earliest ES-for-LLM paper in this record, and the one that accepted the dimensionality objection instead of overturning it: SFT-train LoRA adapters, take their SVD, and run CMA-ES over the top singular values only — reaching 0.80 on GSM8K in under 25 minutes against GRPO’s 50+, from 100 examples against 5,978, and aligning a 32B model at INT4. | Read |
| NOTE-086 | ESSAM | Transposes sharpness-aware minimization into zeroth order — step against the reward-weighted direction, recompute the update at that neighbouring point, apply it at the original — and closes most of standard ES’s gap to GRPO on GSM8K while keeping inference-level GPU memory. | Read |
| NOTE-087 | Evolution Strategies as a Scalable Alternative to Reinforcement Learning | Shared random seeds let every worker reconstruct every other worker’s perturbation, so an ES update costs one scalar per worker instead of a gradient — which turns 11 hours on 18 cores into 10 minutes on 1,440. The paper also argues that the policy-gradient estimator’s variance grows with episode length while ES’s does not, and that ambient parameter count is not the dimension that matters. | Read |
| NOTE-088 | Proximal Policy Optimization Algorithms | Clip the probability ratio to [1-eps, 1+eps] and take the minimum of the clipped and unclipped surrogate, so the objective is a pessimistic lower bound — which gives trust-region behaviour from first-order optimization, permits several epochs of minibatch SGD on one batch of collected data, and fits in a few lines of change to vanilla policy gradient. | Read |
| NOTE-089 | Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devices | By arranging workers into a virtual d-dimensional grid and using chunk indices from Butterfly All-Reduce to ensure no two peers share the same group in consecutive rounds, Moshpit All-Reduce achieves the communication efficiency of all-reduce while tolerating node failures—combining the best properties of all-reduce and gossip without their respective drawbacks. | Read |
| NOTE-090 | Topics in propagation of chaos | When N particles interact only through their empirical distribution (mean- field interaction), the system decouples in the large-N limit: each particle evolves independently according to a nonlinear SDE whose drift depends on its own marginal law, the McKean-Vlasov SDE dX_t = b(X_t, μ_t)dt + σ dW_t where μ_t = Law(X_t). | Read |
| NOTE-091 | The Kuramoto model: A simple paradigm for synchronization phenomena | The Kuramoto model — N oscillators coupled through sin(θ_j − θ_i) interactions scaled by 1/N — undergoes a sharp phase transition at a critical coupling K_c that depends only on the spread of the natural frequency distribution g(ω) at its center. Below K_c the time-averaged order parameter r = |(1/N)Σ e^{iθ_j}| → 0 (incoherence); above K_c a macroscopic fraction of oscillators lock to a common frequency and r > 0 (synchrony). | Read |
| NOTE-092 v2 | Products of indecomposable, aperiodic, stochastic matrices | A finite collection of stochastic matrices all being primitive (indecomposable + aperiodic) guarantees that their products eventually “scramble” — any two rows become increasingly similar — regardless of the order of multiplication, as long as each matrix is used infinitely often. The key mechanism: a single primitive matrix P has P^k → 1v^T (rank-1) for large k. | Read |
| NOTE-093 | Trainability and Accuracy of Neural Networks: An Interacting Particle System Approach | When the width n is large, the n hidden units behave as exchangeable particles whose empirical distribution μ_t evolves by gradient descent in 2-Wasserstein metric on a convex energy functional E[μ] — even though the finite-n loss in parameter space is non-convex. | Read |
| NOTE-094 | Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks | For kernel/NTK regression with eigenvalue decay λ_k ∝ k^{-β}: ε(n) ∝ n^{-α}, where α = β / (1 + β) The exponent α is determined entirely by the data spectrum β, not by architecture beyond the NTK. | Read |
| NOTE-095 | Epidemic Learning: Boosting Decentralized Learning with Randomized Communication | Fully randomizing the communication topology at every round, rather than using a fixed or semi-dynamic graph, provably reduces the transient iterations needed to enter the linear speedup regime by a factor of s^2, because the random averaging step contracts the inter-node model drift by O(1/s) regardless of spectral properties of any fixed graph. | Read |
| NOTE-096 | Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training | 99.9% of gradient exchanges in distributed SGD are redundant; by sparsifying to the top 0.1% of gradients and using error accumulation with momentum correction to neutralize staleness, communication bandwidth can be slashed by 600x with no accuracy loss. | Read |
| NOTE-097 | Asynchronous Stochastic Gradient Descent with Delay Compensation | The stale gradient g(w_t) used in ASGD is merely the zero-order approximation of the correct gradient g(w_{t+tau}); adding a first-order Taylor correction term—approximating the Hessian via a scaled diagonal of the gradient outer product—substantially reduces the bias introduced by delay without extra communication cost. | Read |
| NOTE-098 | A Mean Field View of the Landscape of Two-Layers Neural Networks | When the number of hidden units N is large (N ≫ D), the population risk of a two-layer net becomes convex as a functional of the distribution ρ of hidden-unit parameters, and SGD corresponds to a Wasserstein gradient flow of this functional. | Read |
| NOTE-099 | Distributed Deep Learning in Open Collaborations | By accumulating gradients until a fixed global batch size is reached regardless of how many peers are active, DeDLOC makes collaborative training mathematically equivalent to large-batch synchronous SGD, preserving training stability while tolerating dynamic peer arrival, departure, and heterogeneous hardware. | Read |
| NOTE-100 | Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent | The parameter server is a communication bottleneck because every node talks to it at every step — O(n) traffic on the busiest node. D-PSGD replaces this with neighbor-only gossip averaging via a doubly stochastic weight matrix W, reducing per-node communication to O(Deg). On a ring, that is O(1). The computational complexity is identical to C-PSGD, so decentralized wins whenever communication dominates. | Read |
| NOTE-101 | Reconciling modern machine learning practice and the bias-variance trade-off | Classical statistics predicts a U-shaped bias-variance tradeoff: more capacity → lower bias but higher variance → optimal capacity somewhere in between. Modern ML practice (overparameterized neural networks, random forests) violates this: after the interpolation threshold, adding more capacity reduces test error again. | Read |
| NOTE-102 | An effective theory of collective deep learning | A competition between local learning dynamics and diffusive inter-network coupling produces a genuine phase transition to collective learning: below a critical coupling strength each network only knows its own private class, while above it the ensemble generalizes to all classes, and this transition is depth-delayed and first-order for deep networks due to the exotic phi^{2D+2} Ginzburg-Landau structure. | Read |
| NOTE-103 | ZeRO: Memory Optimizations Toward Training Trillion Parameter Models | Data parallelism wastes memory by replicating all model states on every device; partitioning these states instead of replicating them yields linear memory reduction with device count while keeping communication volume nearly identical to standard data parallelism. | Read |
| NOTE-104 | The mean field analysis for the Kuramoto model on graphs I. The mean field equation and transition point formulas | Network topology enters the synchronization threshold only through the spectrum of the kernel operator W defined by the graph limit (graphon). The incoherent (uniform) state of the mean field equation is neutrally stable precisely on K ∈ [K_c^-, K_c^+], where K_c^± = 2/(π g(0) ζ_{max/min}(W)), generalizing Kuramoto’s complete-graph formula (ζ_max = 1) and exposing graph structure as a spectral multiplier on the critical coupling. | Read |
| NOTE-105 | Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism | Because transformer projection matrices naturally converge to low-rank subspaces during training (rank collapse), explicitly constraining those matrices to a shared low-dimensional subspace forces activations to lie in that subspace too, enabling lossless inter-layer compression that avoids the error accumulation that defeats naive lossy compression in model-parallel settings. | Read |
| NOTE-106 | A Bayesian Perspective on Generalization and Stochastic Gradient Descent | SGD implicitly samples from a Bayesian posterior over parameters, and the effective “temperature” of that posterior is controlled by the noise scale g = εN/B. Increasing batch size B while holding ε fixed cools the posterior, biasing the optimizer toward sharper, less-generalizing minima. | Read |
| NOTE-107 | Proving the Limited Scalability of Centralized Distributed Optimization via a New Lower Bound Construction | When server-to-worker communication is non-negligible (tau_s > 0), there is a fundamental trade-off: increasing n can improve either the dimension- dependent communication term or the variance-dependent compute term, but not both—even when all workers share the same data distribution. This exposes an inherent scalability wall in centralized federated optimization that gradient compression cannot overcome. | Read |
| NOTE-108 | Neural Tangent Kernel: Convergence and Generalization in Neural Networks | At infinite width, gradient descent on neural networks behaves identically to kernel gradient descent with the NTK. The Jacobian (∂f/∂θ) becomes deterministic and constant, so the NTK Θ = J^T J is a fixed PSD matrix. Loss decays as e^{-λ_min(Θ) t}. This is the “lazy training” or “kernel regime”: parameters barely move from initialization, and the network functions as a linear model in parameter space. | Read |
| NOTE-109 | Communication-Efficient Learning of Deep Networks from Decentralized Data | Performing multiple local SGD steps on each client’s private data before averaging model parameters is far more communication-efficient than a single gradient step per round, and the resulting averaged model is robust to non- IID and unbalanced data distributions provided all clients share the same initialization each round. | Read |
| NOTE-110 | Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup | In the teacher-student two-layer ReLU setup, the high-dimensional weight dynamics compress exactly onto a small set of scalar ODEs. | Read |
| NOTE-111 | Distributed asynchronous deterministic and stochastic gradient optimization algorithms | Gradient descent does not require synchronized updates across processors — convergence holds as long as (1) delays are bounded (partially asynchronous model) or (2) each processor updates infinitely often (totally asynchronous model). | Read |
| NOTE-112 | Deep learning with Elastic Averaging SGD | Coupling local workers to a center variable through a soft elastic force (rather than hard averaging) lets workers explore different regions of a nonconvex loss surface; more exploration leads to better solutions in deep networks that have many local optima, and this can be achieved with far less frequent communication than parameter-server baselines. | Read |
| NOTE-113 | The mean field analysis of the Kuramoto model on graphs II. Asymptotic stability of the incoherent state, center manifold reduction, and bifurcations | The network’s structural imprint on synchronization onset is fully captured by the spectrum of the graphon kernel operator W: the critical coupling is 2/(π g(0) μ_max) and the bifurcating partially synchronized state is a scalar multiple of the eigenfunction w_max of the largest eigenvalue μ_max, with √(K-K_c) amplitude (pitchfork). | Read |
| NOTE-114 | A Necessary and Sufficient Condition for Consensus Over Random Networks | For gossip-style algorithms with IID random mixing matrices, almost sure consensus is governed entirely by the EXPECTED topology, not by any individual realization. As long as the expected weight matrix E[W] is “connected enough” (directed spanning tree in its graph, unique unit eigenvalue), consensus happens almost surely even when individual rounds may fail to connect all nodes. | Read |
| NOTE-115 | 1-bit Adam: Communication Efficient Large-Scale Training with Adam’s Convergence Speed | Error-compensated compression is incompatible with Adam because Adam’s non- linear variance term corrupts the error cancellation mechanism; freezing the variance after it stabilizes converts Adam into a linear (momentum SGD) update that admits lossless error compensation, yielding both Adam’s convergence speed and 1-bit communication efficiency. | Read |
| NOTE-116 v2 | Git Re-Basin: Merging Models modulo Permutation Symmetries | Entezari et al. conjecture that SGD solutions can be permuted so that their linear interpolation has no barrier. Git Re-Basin makes this actionable. Given two trained networks θ_A and θ_B, it finds permutations π that minimize ‖vec(θ_A) − vec(π(θ_B))‖², then interpolates θ_A with π(θ_B). The barrier disappears on MNIST and on wide ResNets on CIFAR-10. It does not at 1× width, and not on ImageNet. | Read |
| NOTE-117 | Don’t Use Large Mini-Batches, Use Local SGD | Large-batch SGD converges to sharp minima that generalize poorly; switching to local SGD mid-training introduces beneficial gradient noise that finds flatter minima, recovering single-machine generalization performance without sacrificing parallelism or communication efficiency. | Read |
| NOTE-118 | The Variational Formulation of the Fokker-Planck Equation | Langevin dynamics — and hence SGLD — is not merely a diffusion process but gradient descent on the free energy functional F(ρ) = KL(ρ ‖ π) in the infinite-dimensional Riemannian manifold of probability measures equipped with the W₂ metric. The unique stationary point is the Gibbs measure π ∝ exp(−V), which minimizes F. | Read |
| NOTE-119 | Byzantine-Robust Distributed Learning: Towards Optimal Statistical Rates | The fundamental statistical cost of Byzantine robustness is captured exactly by the term alpha/sqrt(n) added to the standard 1/sqrt(nm) minimax rate, where alpha = f/m is the fraction of Byzantine workers, n is per-worker sample count, and m is total workers. This additive penalty is unavoidable (proven tight via a lower bound) and arises because up to alpha fraction of workers can inject arbitrary gradient signals. | Read |
| NOTE-120 | Asynchronous Local Computations in Distributed Bayesian Learning | Replacing synchronous all-agent communication with pairwise gossip and filling the saved communication budget with multiple cheap local ULA gradient steps allows agents to exploit fast local computation while reducing expensive communication; because the extra stochasticity from gossip and mini-batch gradients is zero-mean with bounded variance, it does not break asymptotic convergence. | Read |
| NOTE-121 | On-line learning in soft committee machines | The full weight dynamics compress onto two scalar order parameters per student- teacher neuron pair. The plateau phase — where R is nearly static despite ongoing gradient descent — arises because all students point in nearly the same direction, making the two-layer network effectively linear. Only once perturbations grow does the specialization phase begin and R accelerates toward 1. | Read |
| NOTE-122 | ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning | By partitioning model states across all parallel devices and using a bandwidth-centric allgather strategy (rather than per-device broadcast), ZeRO-Infinity aggregates PCIe bandwidth linearly with device count, making NVMe and CPU offloading fast enough to sustain efficient training despite their individually slow bandwidths. | Read |
| NOTE-123 | Bayesian Learning via Stochastic Gradient Langevin Dynamics | SGD with a specific noise injection schedule performs approximate Langevin Monte Carlo. The noise term eps_t ~ N(0, eps_t) counteracts the bias from using stochastic (minibatch) gradients, so the iterates explore the posterior rather than collapsing to a mode. At large step sizes the algorithm behaves like standard SGD (optimization); as step size → 0 the Markov chain mixes and samples from the posterior. | Read |
| NOTE-124 | Async-HFL: Efficient and Robust Asynchronous Federated Learning in Hierarchical IoT Networks | Adding an intermediate asynchronous gateway aggregation layer in a three- tier hierarchy simultaneously reduces backhaul communication (by averaging locally before uploading) and stabilizes convergence, while gradient- diversity-based device selection and ILP-based topology management provide the system management needed to realize these gains under real-world heterogeneity. | Read |
| NOTE-125 | ’Neural-gas’ network for vector quantization and its application to time-series prediction | The key departure from SOM: instead of a fixed spatial grid defining neighborhood, neural gas uses the instantaneous rank of each neuron by distance to the current input. Neuron i gets update weight exp(-rank_i / lambda), where lambda decays over training. | Read |
| NOTE-126 | Consensus Problems in Networks of Agents With Switching Topology and Time-Delays | The consensus problem is exactly a question about the null space of the graph Laplacian. For first-order dynamics ẋ = -Lx (L = graph Laplacian), the system converges to the subspace spanned by the all-ones vector (consensus) iff L has exactly one zero eigenvalue — iff the graph is connected. The second-smallest eigenvalue λ_2(L) (algebraic connectivity, Fiedler value) governs the convergence rate: D_x(t) ≤ D_x(0)·e^{-λ_2·t}. | Read |
| NOTE-127 | Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms | All major communication-efficient SGD algorithms differ only in how they trade off three communication-reduction strategies—periodic averaging (tau), mixing matrix sparsity (W), and auxiliary variables (v)—and a single unified bound on the “network error” captures all three, enabling direct comparison and new hybrid designs. | Read |
| NOTE-128 | Achieving Tighter Finite-Time Rates for Heterogeneous Federated Stochastic Approximation under Markovian Sampling | Standard local-update federated SA algorithms converge to the wrong point under heterogeneity because multiple local steps using different local operators accumulate drift; injecting a global correction signal at each local step (the FedHSA update) cancels this drift exactly, enabling unbiased convergence even with Markovian noise and no projection step. | Read |
| NOTE-129 | A Growing Neural Gas Network Learns Topologies | Topology learning can be done incrementally and locally: connect the two nearest units to each input (Hebbian edge), age and prune stale edges, and periodically insert a new unit between the unit with maximum accumulated squared error and its worst-error neighbor. | Read |
| NOTE-130 | OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Training | DiLoCo’s dual-optimizer local-SGD approach reduces inter-node communication by up to 500x with negligible quality loss, and pseudo-gradients can be all- reduced in FP16 without performance degradation, making billion-scale decentralized training practical. | Read |
| NOTE-131 v2 | The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks | Conjectures that, for SGD solutions, the loss landscape has essentially one basin modulo permutation symmetry: permute the hidden units of one independently trained network appropriately and there would likely be no barrier on the linear path to another. The support is a theorem for a one-hidden-layer network at random initialization and indirect empirical evidence. The paper’s own search did not find such permutations for deep networks. | Read |
| NOTE-132 v2 | Large Batch Optimization for Deep Learning: Training BERT in 76 minutes | Scaling the learning rate layerwise by the ratio of the parameter norm to the update norm (inspired by LARS) stabilizes large-batch training across heterogeneous layer curvatures, allowing Adam-like adaptivity to compose with layerwise scaling and generalize across both vision and language models where LARS alone fails. | Read |
| NOTE-133 | Local SGD Converges Fast and Communicates Little | Workers can run SGD independently for H steps before averaging, and as long as H = O(sqrt(T/(Kb))), this infrequent synchronization incurs no asymptotic cost in convergence rate—communication can be slashed without sacrificing linear speedup. | Read |
| NOTE-134 | Consensus Based Sampling | By replacing the empirical covariance in a CBO-like consensus update with a Gibbs-weighted covariance and scaling the noise by a parameter λ, the same interacting particle dynamics can converge to a Gaussian approximation of the posterior (λ = 1/(1+β), sampling mode) or collapse to the MAP estimator (λ = 1, optimization mode), while preserving affine invariance. | Read |
| NOTE-135 | SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient | In pipeline parallelism, computation scales as O(n^3) while communication scales as O(n^2) with model size, so sufficiently large models can be trained efficiently even over internet-grade connections—SWARM exploits this by using stochastic pipelines that dynamically rebalance load across unreliable heterogeneous nodes. | Read |
| NOTE-136 v2 | Coordination of groups of mobile autonomous agents using nearest neighbor rules | Even if a network is disconnected at every individual time step — no single round of communication reaches everyone — the agents can still reach consensus as long as the union of communication graphs over any fixed- length sliding window has a spanning tree. | Read |
| NOTE-137 | Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Experts | Mixture-of-Experts architectures are uniquely suited to volunteer computing because each input activates only a small fraction of all experts, making asynchronous training with stale gradients far less harmful than in dense networks, while the DHT enables scalable decentralized bookkeeping without any central coordinator. | Read |
| NOTE-138 | Convex Analysis of the Mean Field Langevin Dynamics | The proximal Gibbs distribution p_q(θ) ∝ exp(-(1/λ) δF/δq (q)(θ)) plays the role of a “gradient” and “proximal point” in the space of measures: KL(q || p_q) simultaneously serves as a Polyak-Łojasiewicz-style upper bound on the optimality gap L(q) - L(q*) and, in ERM settings, equals exactly the duality gap. | Read |
| NOTE-139 | On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport | Over-parameterized particle gradient descent on F_m(u_1,…,u_m) can be viewed as a discretization of a Wasserstein gradient flow on measures. Although the non-convex landscape has many local minima, homogeneity of Phi and V plus a “separation” property of the initialization support — preserved along the flow — force any limit point to be a global minimizer. | Read |
| NOTE-140 | Mean Field Analysis of Neural Networks: A Law of Large Numbers | With 1/N scaling of the output layer and a time rescaling t = k/N (so that one unit of continuous time corresponds to N SGD steps), the N hidden units behave like an interacting particle system whose empirical measure μ^N_t converges to a deterministic measure-valued process μ̄_t satisfying a McKean-Vlasov / Fokker-Planck PDE. | Read |
| NOTE-141 | Generalized EXTRA stochastic gradient Langevin dynamics | Standard decentralized SGLD (DE-SGLD) suffers from a bias due to network effects that does not vanish even with full-batch gradients; adapting the EXTRA correction mechanism from decentralized optimization to Langevin dynamics eliminates this bias and yields faster convergence. | Read |
| NOTE-142 | HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent | When the optimization problem is sparse (each gradient update touches only a small fraction of the decision variable), memory overwrites between processors are rare and introduce negligible error, making lock-free parallel SGD both safe and efficient. | Read |
| NOTE-143 | Exact Solution for On-Line Learning in Multilayer Neural Networks | The closed-form solution is: R(t) = R_∞ - (R_∞ - R_0) exp(-t / τ_sp) with specialization timescale τ_sp ∝ 1/η. Generalization error follows: ε_gen(t) ∝ (1 - R(t))² = C_∞ + A · exp(-t/τ_sp) + B · exp(-2t/τ_sp) This is a SUM OF EXPONENTIALS in ε, not a single exponential, because of the square. | Read |
| NOTE-144 | Measuring the Effects of Data Parallelism on Neural Network Training | The relationship between batch size and training steps to a goal error always follows the same three-regime shape — perfect scaling (b-fold benefit), diminishing returns, and a maximum useful batch size beyond which adding more parallelism helps nothing — but where these transitions occur is workload-specific and cannot be reliably predicted from simple properties of the model, data, or optimizer. | Read |
| NOTE-145 | DiLoCo: Distributed Low-Communication Training of Language Models | Replacing the plain-averaging outer step of FedAvg with Nesterov momentum, while using AdamW for inner optimization over H~500 steps, enables language model pretraining across geographically distributed devices with minimal bandwidth requirements and strong robustness to data heterogeneity and worker availability. | Read |
| NOTE-146 | Phase transitions, logarithmic Sobolev inequalities, and uniform-in-time propagation of chaos for weakly interacting diffusions | Both the N-particle Fokker-Planck equation and its McKean-Vlasov mean-field limit are 2-Wasserstein gradient flows of related free energies (E^N and E^MF), and E^N Gamma-converges to E^MF. | Read |
| NOTE-147 | Emergent Behavior in Flocks | The CS model places the “consensus” dynamics on VELOCITIES rather than positions. All particles converge to the same velocity (heading), but their positions may continue to diverge — they flock together, traveling as a group. | Read |
| NOTE-148 | Smoothing DiLoCo with Primal Averaging for Faster Training of LLMs | DiLoCo’s performance improves with more inner steps because the two-loop structure inadvertently provides stronger smoothing of the iterate sequence; GPA replicates this smoothing effect at every step by decoupling the gradient-computation interpolation constant from the model-evaluation averaging constant, eliminating the need for a two-loop structure. | Read |
| NOTE-149 | Emergence of stochastic flocking for the discrete Cucker-Smale model with randomly switching topologies | The Cucker-Smale velocity update is a stochastic matrix multiplication on velocities: V[t+1] = (Id - (h/N)·L_{σ[t]}[t]) · V[t], where each factor is a row-stochastic matrix with ergodicity coefficient determined by the communication weight φ and position diameter. Flocking = the velocity diameter D(V[t]) → 0 a.s. | Read |
| NOTE-150 | LAYUP: Asynchronous decentralized gradient descent with LAYer-wise UPdates | Because the backward pass takes roughly twice as long as the forward pass, running them in separate threads with a 1:2 forward-to-backward thread ratio keeps both passes continuously busy; additionally, applying updates layer- by-layer as gradients arrive rather than waiting for the full gradient halves the average staleness, improving convergence robustness without any gradient compensation scheme. | Read |
| NOTE-151 | Stochastic Gradient Push for Distributed Deep Learning | Using the PushSum protocol’s ratio-based de-biasing trick removes the requirement for symmetric (bidirectional) communication, allowing decentralized SGD to operate on directed graphs and substantially reduce communication overhead versus AllReduce SGD in bandwidth-constrained settings. | Read |
| NOTE-152 | PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization | Making the compressor linear (i.e., equivariant under averaging) unlocks all-reduce aggregation instead of slower all-gather; combined with warm- starting the subspace iteration from the previous step and error feedback, a single power iteration step matches the quality of a full SVD while being dramatically cheaper. | Read |
| NOTE-153 | Overlap Local-SGD: An Algorithmic Approach to Hide Communication Delays in Distributed SGD | By decoupling synchronization into a background thread that maintains an anchor model and having workers pull back toward that anchor rather than waiting for direct peer communication, computation and communication are fully overlapped, reducing the effective communication-to-computation ratio from ~35% to ~1.5% in practical settings. | Read |
| NOTE-154 | Distributed Optimization with Gradient Tracking over Heterogeneous Delay-Prone Directed Networks | Embedding the robustified Ratio Consensus protocol into ADD-OPT and augmenting the network graph with virtual delay nodes transforms a delayed directed network into a delay-free augmented network, allowing existing convergence proofs to apply with the augmented spectral radius replacing the original, at the cost of a smaller allowable step-size. | Read |
| NOTE-155 | signSGD: Compressed Optimisation for Non-Convex Problems | When gradients and noise are dense (a condition empirically verified in deep networks), the sign of the gradient is an effective update direction; aggregating signs by majority vote across workers is equivalent to maximum- likelihood decoding of a repetition code, achieving sqrt(M)-fold variance reduction with M workers using only 1 bit per parameter per direction. | Read |
| NOTE-156 | Byzantine-Tolerant Machine Learning | Averaging is catastrophically fragile: a single Byzantine worker can steer the aggregate to any arbitrary vector. The fix is to replace aggregation with selection — choose the gradient vector that is most “centrally located” among its nearest correct neighbors, using the n-f-2 nearest neighbors to ensure the majority of those neighbors are correct workers. | Read |
| NOTE-157 | SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum | Adding a slow outer momentum step after every tau inner steps of local SGD or a decentralized optimizer consistently improves both optimization and generalization at negligible extra communication cost, because slow momentum corrects accumulated drift between workers at the outer level. | Read |
| NOTE-158 | Broken Neural Scaling Laws | A single power law ε(t) = C_∞ + b·t^{-c} fails when the curve has two regimes (steep early, shallow late — or vice versa). | Read |
| NOTE-159 | Accelerating Large Language Model Decoding with Speculative Sampling | The same construction as NOTE-160, reached independently two months later and demonstrated at 70B in a distributed setup. Its contribution to the record is corroboration plus two sharper statements: the guarantee holds “within hardware numerics” rather than absolutely, and the achieved tokens per second exceed the memory-bandwidth ceiling that bounds autoregressive decoding — which is the family’s premise, measured rather than assumed. | Read |
| NOTE-160 | Fast Inference from Transformers via Speculative Decoding | Autoregressive decoding is serial and memory-bandwidth-bound, so the arithmetic units sit idle while weights stream. Speculative decoding spends that idle compute: a cheap draft proposes gamma tokens, the target scores all gamma + 1 positions in one parallel pass, and an accept-reject rule keeps a prefix. The rule is constructed so the output distribution is exactly the target’s, which is what makes this a free speedup rather than a quality trade. | Read |
| NOTE-161 | Training language models to follow instructions with human feedback | Making a language model bigger does not make it better at following instructions, so stop buying parameters and buy human feedback instead: demonstrations to fine-tune on, rankings to fit a reward model to, then policy optimization against that reward with a KL leash back to the supervised model. A 1.3B model tuned this way is preferred to 175B GPT-3. The cost is a measurable regression on held-out NLP tasks, and the paper supplies both the name for it and a fix. | Read |
| NOTE-162 | Efficiently Scaling Transformer Inference | An analytical model of where inference time actually goes, used to pick a partitioning rather than searching for one. Its organising finding is that prefill and decode have different performance characteristics and must be analysed separately, and that the dominant cost moves — weights at small batch and short context, the KV cache at large. On PaLM 540B across 64 TPU v4 chips: 29 ms per token at low batch with int8 weights, 76% MFU at large batch, and 32x longer context from multiquery attention once the partitioning is right. | Read |
| NOTE-163 | EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test | EAGLE predicted the target model’s features and then read tokens off them. That constraint is why feeding it more training data stopped helping. Drop it — predict tokens directly, fuse low, mid and high features instead of reusing the top layer, and simulate the multi-step draft during training so step two sees its own step-one output. The result is a scaling law for inference acceleration: more draft data, proportionally more speedup, up to 6.5x. And it raises throughput 1.38x at batch 64, where speculation is supposed to stop paying. | Read |
| NOTE-164 | QLoRA: Efficient Finetuning of Quantized LLMs | Store the frozen base in 4 bits, compute in 16, and train only LoRA adapters: 65B fine-tuning goes from >780GB to <48GB with no measured loss against a 16-bit fully fine-tuned baseline. The quantization is the headline; the more useful result for anyone already using LoRA is that the standard query/value placement does not reach full fine-tuning at scale, and that the adapter count rather than the rank is what closes the gap. | Read |
| NOTE-165 | Large Language Models Cannot Self-Correct Reasoning Yet | Asked to revise its own reasoning with no external feedback, a model does not reliably improve and sometimes degrades. The durable part is the definition rather than the number: INTRINSIC self-correction, meaning no oracle, no verifier, no second model — a line that a lot of contemporary self-correction claims were quietly on the other side of. | Read |
| NOTE-166 | Scaling Diffusion Language Models via Adaptation from Autoregressive Models | Continual pretraining on under 200B tokens converts GPT2 and LLaMA at 127M-7B into diffusion language models competitive with their AR counterparts. The argument is economic before it is technical — diffusion LMs are hard to train at scale and AR checkpoints are abundant — and the capability that pays for it is infilling without prompt re-ordering. | Read |
| NOTE-167 | The Surprising Effectiveness of Test-Time Training for Few-Shot Learning | Turn the in-context examples into a loss and take gradient steps on them at inference, then discard the update. Up to 6x a fine-tuned baseline on ARC and +7.3 points on BIG-Bench Hard at 10-shot. The finding underneath is about in-context learning: the same examples are worth several times more as gradient than as context. | Read |
| NOTE-168 | The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits | Ternary weights trained from scratch, matching FP16 from 3B upward at equal size and tokens. The interesting claim is not the compression ratio but that the matrix multiply becomes integer addition — which is a bet on hardware, and the one place the paper’s argument outruns its measurements. | Read |
| NOTE-169 | Training Language Models to Self-Correct via Reinforcement Learning | SCoRe teaches self-correction with multi-turn online RL on the model’s own traces, no teacher and no extra supervision, for +15.6% on MATH and +9.1% on HumanEval. The transferable half is the diagnosis: SFT on correction traces fails by distribution mismatch or by behaviour collapse, and the method’s two stages are each aimed at one of those. | Read |
| NOTE-170 | VGGT: Visual Geometry Grounded Transformer | One feed-forward pass produces camera parameters, point maps, depth maps and 3D point tracks together, in under a second, beating methods that post-process with geometry optimisation. Removes the global alignment step DUSt3R still needed, and works as a pretrained backbone — which is when a subject stops being a field and becomes a component. | Read |
| NOTE-171 | DUSt3R: Geometric 3D Vision Made Easy | Regress pointmaps from uncalibrated, unposed images instead of solving for cameras and triangulating. The result worth carrying is not the accuracy but the direction of dependency: matches and camera parameters fall out of the 3D output, where the classical pipeline needs them before it can begin. | Read |
| NOTE-172 | SuperGlue: Learning Feature Matching with Graph Neural Networks | Learn the matcher instead of hand-designing it: a GNN predicts assignment costs, differentiable optimal transport solves them, and correspondence and rejection happen jointly. Filed for the line rather than for a practice — its surviving assumption, that a detector ran first, is what the next paper identifies as the binding constraint. | Read |
| NOTE-173 | LoFTR: Detector-Free Local Feature Matching with Transformers | Remove the keypoint detector and match densely, coarse-to-fine, with descriptors conditioned on both images by cross attention. The argument is a failure mode rather than an efficiency: detectors cannot emit repeatable points in low-texture regions, so no downstream matcher can recover what was never proposed. | Read |
| NOTE-174 | Evaluating Large Language Models Trained on Code | Read for the metric rather than the benchmark. The paper defines pass@k and the unbiased estimator 1 - C(n-c,k)/C(n,k), and argues explicitly why the naive repeated-trials measurement is biased downward — which is the convention SOTA-210 already recommends while calling it “the standard one” and citing nobody. | Read |
| NOTE-175 | DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining | A 280M proxy under group DRO sets the domain weights for an 8B run: +6.5 points few-shot and the baseline in 2.6x fewer steps, with no downstream task in the loop. The transferable part is the objective — worst-case EXCESS loss against a reference model, because worst-case raw loss would upweight whichever domain is noisiest. | Read |
| NOTE-176 | Beyond neural scaling laws: beating power law scaling via data pruning | Power-law scaling of error in dataset size is a symptom of redundancy, not a ceiling. With a ranking of example difficulty and a pruning fraction that grows with the corpus, error can fall exponentially instead. Which end of the ranking to discard inverts with data abundance: keep hard examples when data is plentiful, easy ones when it is scarce. | Read |
| NOTE-177 | Deep Learning on a Data Diet: Finding Important Examples Early in Training | Two scores that rank training examples by importance from a single early checkpoint — GraNd (expected loss-gradient norm) and EL2N (norm of the error vector) — averaged over several initializations. They prune half of CIFAR-10 without loss, transfer across architectures, and come with the finding that the very highest-scoring examples should be excluded, more so under label noise. | Read |
| NOTE-178 | Studying Large Language Model Generalization with Influence Functions | Scales influence functions to 52B-parameter language models with an EK-FAC approximation to the IHVP and query batching over the gradient bottleneck, then uses them as an instrument. Generalization gets more abstract with scale; influence is heavy-tailed but not concentrated; and a training sequence only counts when the prompt-related phrase comes first. | Read |
| NOTE-179 | If Influence Functions are the Answer, Then What is the Question? | Decomposes the influence-function / leave-one-out gap into five terms and finds that the three largest are not approximation errors but a change of question. Influence estimates on neural networks closely track the proximal Bregman response function, which supports the same use cases — so the method is not fragile, it was mislabelled. | Read |
| NOTE-180 | Estimating Training Data Influence by Tracing Gradient Descent | Defines a training example’s influence as the total change it caused in a test point’s loss over the iterations where it was used, and approximates that by step-size-weighted gradient dot products at saved checkpoints. No Hessian, no convexity. Self-influence ranks mislabelled examples to the top. | Read |
| NOTE-181 | Understanding Black-box Predictions via Influence Functions | Traces a prediction back to the training points responsible for it by upweighting each point infinitesimally and differentiating, which needs only gradients and Hessian-vector products rather than retraining. Establishes the four uses of training-data attribution — explanation, poisoning, domain-mismatch debugging, and finding mislabelled data. | Read |
| NOTE-182 | P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks | Prompt tuning did not fail because tuning a prompt is a weak idea; it failed because the prompt was only at the input layer, where it has too few parameters and too indirect a route to the prediction. Put prefix tokens at every layer and the same family matches full fine-tuning from 330M to 10B and on hard sequence labelling, at 0.1-3% of the parameters. The paper is explicit that it is not conceptually novel — it is deep prompt tuning, already published for generation, adapted to NLU and optimized properly. | Read |
| NOTE-183 | The Unreasonable Effectiveness of Deep Features as a Perceptual Metric | Collect human similarity judgements first, then ask which metric agrees with them. Deep features beat every hand-designed metric by a wide margin, and the surprise is the insensitivity — architecture barely matters and supervision barely matters, but training on something matters a great deal, because a randomly initialized network does not work. Perceptual similarity looks like an emergent property of learned visual representations rather than of any particular training signal. | Read |
| NOTE-184 | Training Deep Nets with Sublinear Memory Cost | Two contributions, and the famous one is the second. First, treat memory allocation over the computation graph as a compiler problem — liveness analysis, in-place operations, memory sharing. Then, on top of that, drop most intermediate feature maps and recompute them segment by segment during the backward pass. Dividing an n-layer chain into sqrt(n) segments costs sqrt(n) memory for one extra forward pass; the extreme of the same analysis is log n memory for n log n extra forward computation. | Read |
| NOTE-185 | Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture | Predict representations, not pixels, and drop hand-crafted augmentations entirely. From one context block, predict what a learned target-encoder outputs for several masked target blocks in the same image. The masking strategy is the whole design: targets large enough to be semantic, context spatially distributed, and the mask applied to the target-encoder’s OUTPUT rather than its input. A ViT-H/14 on ImageNet in under 1200 GPU-hours. | Read |
| NOTE-186 | V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning | I-JEPA’s construction at video scale — 1M+ hours, up to 1B parameters — plus a second act: post-train a latent action-conditioned world model on under 62 hours of unlabelled robot video and plan with it zero-shot on real arms. The scaling section attributes its gains one ingredient at a time, and the one worth stealing is unrelated to video: raise resolution only in the decay phase and pretraining costs up to 8x less. | Read |
| NOTE-187 | Video models are zero-shot learners and reasoners | The argument by analogy: LLMs became generalist from three primitives — large, generative, web-scale — and video models now have all three. The evidence is a catalogue of tasks Veo 3 was never trained for and can do anyway, from segmentation and edge detection through affordances and tool use to maze and symmetry solving. It is a position paper with demonstrations, and the demonstrations are of one proprietary model. | Read |
| NOTE-188 | NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis | Store the scene in the weights of an MLP that maps a 5D coordinate to density and view-dependent colour, and render it with classical volume rendering, which is differentiable — so posed images are the only supervision. Two additions do the work everybody remembers: positional encoding, without which the result is oversmoothed, and hierarchical sampling, without which it is unaffordable. | Read |
| NOTE-189 | Modular Duality in Deep Learning | Gradients live in the dual space and weights live in the primal one, so subtracting one from the other is a type error that ordinary gradient descent commits every step. Fix it with a duality map built recursively from per-layer operator norms. Two consequences the record cares about: muP and Shampoo turn out to be partial approximations of one such map, and the map for Linear layers is a Newton-Schulz iteration — which is Muon. | Read |
| NOTE-190 | How much do language models memorize? | Defines memorization as compression rate against the model, separates it from generalization by training on uniform random bitstrings, and measures GPT-style transformer capacity at ~3.6 bits per parameter, linear in parameter count and nearly indifferent to precision. Double descent begins exactly where the data’s information content crosses that capacity. | Read |
| NOTE-191 | Pre-training under infinite compute | Under a fixed 200M-token corpus and no compute limit, the standard recipe overfits; tuning weight decay to ~30x the customary 0.1 makes loss monotone in parameter count. Ensembling independently trained models then beats parameter scaling on asymptote, and distils back down at ~83% retention. | Read |
| NOTE-192 | Diffusion Beats AR in Data-Constrained Settings | Refits LIT-166’s data-constrained scaling law with the objective swapped from autoregressive to masked diffusion. The half-life of data reuse R* goes from 31.93 to 512.85. Diffusion loses badly at Chinchilla-optimal compute and wins past a crossover given in closed form. | Read |
| NOTE-193 | Power Lines | Scales the AdamW timescale rather than the weight decay, and finds it follows a power law in tokens-per-parameter with exponent about -0.52 over three orders of magnitude of compute. Optimal and critical batch size both scale as power laws in the token budget, not in compute. | Read |
| NOTE-194 | Small Batch Size Training for Language Models | Small batches look unstable because beta_2 is held fixed while the batch changes. Hold the second moment’s half-life fixed in tokens instead and batch size one trains stably, matches larger batches per FLOP, is far more robust to hyperparameters, and lets momentum-free SGD match AdamW at 1.3B. | Read |
| NOTE-195 | How Does Critical Batch Size Scale in Pre-training? | Decouples model size from data size — which every prior batch-size study had scaled together — and finds critical batch size tracks the token budget and is nearly flat in model size. Supported by a muP argument for the model half and a least-squares analysis for the data half. | Read |
| NOTE-196 | Scaling Rectified Flow Transformers | A 61-way sweep over diffusion and rectified-flow formulations settles on a straight-line path with logit-normal timestep sampling — rectified flow with uniform timesteps does not win. Adds a resolution-dependent timestep shift, MMDiT’s per-modality weights with joint attention, and QK-RMSNorm for high-resolution stability. 8B, weights released. | Read |
| NOTE-197 v2 | SiT: Scalable Interpolant Transformers | Walks one variable at a time from DiT to SiT with the architecture, parameter count and GFLOPs held fixed: discrete to continuous time, score to velocity prediction, variance-preserving to linear interpolant, ODE to a tuned SDE sampler. Velocity and the linear interpolant carry most of the gain, and the sampler’s diffusion coefficient turns out to be choosable after training. | Read |
| NOTE-198 | Variational Diffusion Models | Integrating the diffusion VLB over signal-to-noise ratio rather than time collapses the schedule out of the expression: in continuous time the bound depends on it only through the endpoint SNRs. VP and VE specifications are therefore equivalent, and the freed schedule shape is spent on minimizing the loss estimator’s variance. | Read |
| NOTE-199 v2 | How do language models learn facts? | Factual recall is learned in three phases, and the plateau between generic statistics and individual knowledge is the attention recall circuit being built — patching in a trained model’s attention patterns removes the plateau. Imbalanced data shortens the plateau and, the paper argues, slows acquisition after it; a warm-up on a subset of individuals before training on all of them beat the fixed distributions tried, in single-seed runs. | Read |
| NOTE-200 | Scaling Laws for Fact Memorization | Fact capacity is linear in model size and saturates in epochs along a negative exponential; all of Wikidata would want ~1000B non-embedding parameters at 100 epochs. Redundant facts cost full price unless they share direction and structure. Generalisation to unseen facts exists and scales like ordinary pretraining. | Read |
| NOTE-201 | Data Mixing Can Induce Phase Transitions in Knowledge Acquisition | Linear scaling of acquired knowledge in model size holds when a knowledge-dense dataset is trained on alone and breaks once it is mixed into web text: below a critical model size, or a critical mixing ratio, the model memorises almost nothing. Attributed to capacity allocation as a knapsack, with the critical ratio a power law in model size. | Read |
| NOTE-202 | The Limiting Dynamics of SGD | Long after the loss converges the weights keep travelling, distance growing as a power law with a non-trivial exponent. Derives SGD at finite learning rate and batch size as an underdamped Langevin equation and shows the driver is a modified loss plus probability currents, with the motion incoherent oscillation in the Hessian’s top eigensubspace rather than diffusion. | Read |
| NOTE-203 | Central Flows | Models the time-averaged trajectory of an oscillating optimizer as a differential equation that predicts the real path numerically. Reading it: at the edge of stability sharpness sets the step size, not the learning rate; and adaptive optimizers steer away from curvature so they can step further later, which ablating makes them slower. | Read |
| NOTE-204 | Hidden Breakthroughs in Language Model Training | A smooth loss curve is what many differently-timed abrupt transitions look like when averaged. POLCA decomposes loss change per example and along a curvature-derived low-rank basis; on synthetic arithmetic it recovers the carrying skill, which clustering the exact loss curves does not. | Read |
| NOTE-205 | Edge of Stability | Full-batch gradient descent raises the loss Hessian’s top eigenvalue until it hits 2/eta and then trains there. Reading it: the three standard analytical assumptions fail at every reasonable step size, and the curvature-based step-size rule loses to the fixed step size it forbids. | Read |
| NOTE-206 | The Platonic Representation Hypothesis | Vision and language models measure distance between datapoints more alike as they get larger, across architectures and objectives. The conjecture is that they converge on a model of what generated the data. The proof holds for bijective observations only, and the caption-density experiment tests that boundary rather than asserting past it. | Read |
| NOTE-208 | Muon is Not That Special | An optimizer that replaces the gradient’s singular values with chaotic noise matches Muon on NanoGPT. Reading it: the LMO account of spectral optimizers fails its own control, and what the control leaves standing is that Muon’s optimal step size is constant where Euclidean descent’s is not. | Read |
| NOTE-209 | When Spectral Updates Help | A spectral step beats a Euclidean one on a block when the gradient’s nuclear rank exceeds the stable rank of that block’s incoming activations. Reading it: both sides are measurable, the activation side is provably small in transformers, and the gap grows with dimension. | Read |
| NOTE-210 | The Lattice Representation Hypothesis | A linear concept direction plus a threshold is a half-space; intersecting half-spaces gives a concept lattice, so inclusion, intersection and union become geometric meet and join on embeddings. Canonical form when the attribute directions are linearly independent. Evidence is WordNet sub-hierarchies, which is the friendliest available test. | Read |
| NOTE-211 | Spectral muP under Width-Depth Scaling | Derives the muP spectral condition for joint width-depth scaling and shows the rule is indexed by residual-branch depth. Reading it: Depth-muP and CompleteP are k = 1 and k ≥ 2 of one family, Transformers are the second, and for preconditioned optimizers the whole depth correction is a 1/L residual multiplier. | Read |
| NOTE-212 | The Illusion of Thinking | Controllable puzzles replace contaminated benchmarks; reasoning models collapse to zero past a threshold and reduce thinking tokens as they do. Reading it: the three-regime result is solid, and the collapse reading is contested on scoring grounds the paper does not address. | Read |
| NOTE-213 | Comment on the Illusion of Thinking | Argues the reported collapse is evaluation design. Reading it: the unsolvable-instance claim is a checkable fact and lands; the rest is a reinterpretation of a measurement the two papers agree on, from a preprint that is underpowered and has already been corrected once. | Read |
| NOTE-214 | Limitations of multi-layer Transformers | First unconditional lower bound for a decoder-only transformer of more than one layer. Reading it: the bound comes from causal masking rather than from circuit complexity, and it buys an exponential depth-width tradeoff, an encoder/decoder separation, and a provable benefit of chain of thought. | Read |
| NOTE-215 | Auxiliary Task Demands | Four capacities, each measured two ways; the higher-demand way scores lower and the gap shrinks with model size and training time. Reading it: the interaction is what matters, because it means cross-scale comparisons under a demanding evaluation overstate the difference between models. | Read |
| NOTE-216 | Chain-of-Thought Prompting | Worked reasoning steps in the few-shot exemplars, nothing else. Reading it: the three ablations are the contribution — it is not the equation, not the extra tokens, and not knowledge activation — and the effect is absent or negative below about 100B parameters. | Read |
| NOTE-217 | Self-Consistency | Sample several chains, take the majority answer. Reading it: it is a decoding change rather than a prompt change, so it composes with everything else in the topic — and it counts answers, not reasoning, so it does not make the traces trustworthy. | Read |
| NOTE-218 | Zero-Shot CoT | One fixed sentence, no exemplars, twelve datasets. Reading it: the ordering is the practical content — it loses to hand-written few-shot chains and beats eight-shot standard prompting, so it is the baseline to try first rather than the method to settle on. | Read |
| NOTE-219 | Emergent Abilities | The paper that named the phenomenon, read for what it actually argues rather than what it is cited for. It raises the metric explanation itself, in §5.1, and declines it on two grounds — one of which was later answered and one of which still stands. | Read |
| NOTE-220 | Emergence as a Mirage | Reading it: the strongest evidence is not the rebuttal of GPT-3’s arithmetic curve but the vision experiments, where emergence is manufactured in models that had never shown it. The weakest part is what “mirage” gets read as — the paper explicitly declines the claim that emergence is impossible. | Read |
| NOTE-221 | nGPT | Reading it: the headline is 4–20× and the footnote is 60–80% more time per step, so the honest wall-clock figure is roughly 5.5× at 4k. Still large. The part that generalises beyond this architecture is smaller and more interesting — that a matrix’s norm is currently an accident of the learning rate and weight decay, and does not have to be. | Read |
| NOTE-222 | Quantized Evolution Strategies | Reading it: the memory trick is the contribution, not the error feedback. Error feedback is 1-bit SGD’s, correctly credited; what is new is refusing to store the accumulator at all and rebuilding it from seeds, which is the only reason the method fits in the memory budget that motivated it. | Read |
| NOTE-223 | The Coverage Principle | Reading it: the strongest thing here is not the theorem but the Bernoulli example, which shows KL going to infinity while coverage stays fine on a two-outcome model. That is the whole argument in four lines, and it explains why a metric everybody watches can be anti-correlated with what they want. | Read |
| NOTE-224 | Dropout Reduces Underfitting | Reading it: the persuasive number is the training loss, not the accuracy. Early dropout lowers ViT-T’s training loss from 3.443 to 3.394 while standard dropout raises it to 3.885 — so the same operator is doing two opposite things depending only on when it runs, and one of them is not regularization. | Read |
| NOTE-225 | Local Volume | Reading it: the useful result is the audit, not the hypothesis. A network trained to generalize badly is detected on clean held-out data where its behaviour matches a clean model’s, in a small number of forward passes. The volume hypothesis itself is left where it was found — “broadly consistent”, by the authors’ own conclusion. | Read |
| NOTE-226 | NoProp | Reading it: the interesting number is in the train column, not the test one. NoProp matches backprop on test while sitting 8-15 points behind on train, which on MNIST and CIFAR costs nothing and would be the whole story anywhere fitting is the binding constraint. The paper does not mention it. | Read |
| NOTE-227 | Late-phase weights | Reading it: three ablations constrain the recommendation more than the headline does. Starting at initialization fails outright, the fanciest variant is worse than doing nothing, and replicating every weight instead of a few is worse than replicating a few. All three are reported. | Read |
| NOTE-228 v2 | The Diffusion Duality | Reading it: the theorem is elegant and the ablation says it accounts for under half the likelihood gain. What the paper actually establishes is a regime split — masked diffusion wins on quality at many steps, uniform-state wins below about 32, and the reason is architectural rather than tuned. | Read |
| NOTE-229 | Auditing Political Exposure Bias | Reading it to find out whether deployment-and-society earns its place: it does, and not because of the finding. The methods section is a sequence of stated trades about measuring a system you cannot inspect, and the sharpest result is that the arm following nobody gets the most diverse timeline. | Read |
| NOTE-230 | Informative and Actionable Social Media Research | Reading it: the power-grid example is the whole paper and it is worth the space. Treat too few substations and the grid absorbs it; treat enough and the controls black out too. Both give “no effect”, both are wrong, and nothing about that argument is specific to social media. | Read |
| NOTE-231 | Industry Influence in Social Media Research | Reading it: the number that matters is not 49% of papers but 21% of authors. A field where half the high-profile output has industry ties held by a fifth of its people is not a field engaging broadly with industry — it is a small group with durable relationships, and that is a different diagnosis. | Read |
| NOTE-232 | Mixture-of-Transformers | Read to test whether it satisfies SOTA-262’s promotion condition. It does, as that condition was written — second group, per-modality weights against a shared-weight backbone, FLOPs held identical. The component ablation is a bonus the condition did not ask for and the practice needed. | Read |
| NOTE-233 | TinyStories | Read as the missing trunk under eight tiny-models practices. The result that matters is a separation, not a record: what stops a 125M model speaking coherent English is the breadth of the corpus, not the complexity of language — established by holding the architecture and shrinking the corpus. | Read |
| NOTE-234 | SimpleStories | Read to see what, if anything, it establishes that its trunk did not. The dataset comparison is direct and well instrumented; the model comparison is confounded three ways and the authors say so. The practice this record takes from it is about the generation procedure, which was measured. | Read |
| NOTE-235 | ParScale | Read as a candidate third scaling axis. The scaling law is fitted honestly and the inference-cost analysis is better instrumented than most; the claim that carries furthest is the asymmetry — code benefits more than general text, on two independent measurements — and it is the claim the paper labels a conjecture. | Read |
| NOTE-236 | AI on Change.org | Read to test it against SOTA-291’s promotion condition, which it satisfies. The platform-level difference-in-differences is the strong evidence; the repeat-writer analysis reads stronger than it is; and the finding with the longest reach — that the text features stopped predicting outcomes — is in the appendix. | Read |
| NOTE-237 | gen2seg | Read as the counter-route to SAM’s data engine. The headline (narrow finetuning, broad generalization) is well evidenced but the argument lives in the controls, which are unusually good: the same backbone under a conventional head scores ~1.5 mIoU, and swapping only the decoder recovers part of the gap. | Read |
| NOTE-238 | PHi: prediction of hidden states | Read for the instrument. The diagnosis of next-token loss is clean and the boring/interesting separation is convincing on two architectures. The reasoning-selection result is weaker than the abstract implies, and the paper says so itself in a sentence most readers will pass over: picking by next-token loss alone already scores 71%. | Read |
| NOTE-239 | Denoising-corrected gradient guidance | Read for a practice the record’s diffusion line was missing. The ordering change is well motivated and cheaply adopted; the theory is proved under idealized assumptions the authors name; and the useful experimental result is that the standard method was optimizing the plan rather than the executed outcome. | Read |
| NOTE-240 | Noise hypernetworks | Read as the amortized counterpart to SOTA-301. The headline — recover half of test-time optimization at 1/100th the latency — is well measured. The finding that carries further is the control: direct reward fine-tuning of the same model makes it worse, and worse the more steps you sample. | Read |
| NOTE-241 | LLMs raise the opportunity cost of research time | Read for a mechanism THEORY-051 named and the record could not supply: why effort would fall when a writing tool gets better. The answer is that a perfect time-saving tool raises the value of time, and the interesting part is not the decline but the phase dependence — the model predicts opposite signs for selectivity, and the opposite sign for thoroughness, from properties of the tool you can identify before deploying it. | Read |
| NOTE-242 | Genesys and what survives its own controls | Read from the #180 worklist for the architecture-discovery claim and filed for the code-generation one. Unit-by-unit generation with an execution-based checker takes valid output from 6% to 92% on the same proposals, with a stated mechanism that has nothing to do with architectures. The discovered architectures are a wash against their own seeds, by the paper’s own table. | Read |
| NOTE-243 | GaussianToken and the cost of the continuous half | Read from the #180 worklist and filed without a practice. The idea — let the quantization units move and size themselves instead of sitting on a grid — is a good one, and the evidence for it cannot be separated from the five continuous parameters each token carries alongside its index. The comparison is at equal token count, and nobody computed bits. | Read |
| NOTE-244 | SQ hardness for semiautomata, and the regime it needs | Read from the #180 worklist. The theorem is clean and the regime it needs — an alphabet cubic in the number of states — is what a citing reader will drop. Filed for the mental model underneath it, which the record had nothing for: longer random sequences carry less information about a hidden transition structure, not more. | Read |
| NOTE-245 | VQ-VAE: what the three-term loss actually does | Read to give the discrete-tokenizer line a root. The durable content is three loss terms with three different owners and a gradient that is copied rather than derived — an engineering settlement that every descendant still runs unchanged, and that the paper is unusually honest about having chosen rather than justified. | Read |
| NOTE-246 | VQGAN: the loss that made f=16 survivable | Read as the second step of the tokenizer trunk. The architecture is VQ-VAE’s; what is new is a perceptual loss, a patch discriminator, and an adaptive weight that balances them without per-dataset tuning. That loss is the reason a 256×256 image can become 256 tokens and still look like anything, and it is why this line exists at all. | Read |
| NOTE-247 | ViT-VQGAN: where the codebook advice comes from | Read to give SOTA-306 its origin, and it corrected the practice in three places. The advice is not “use small code vectors” but “factorize lookup from embedding”; ℓ₂-normalization is ablated and is the largest single effect in the table; and the dimension-4 row shows 96% codebook usage with near-worst FID, so utilization is necessary and not sufficient. | Read |
| NOTE-248 | The FID Lottery: which lottery, and how big | Read from the #180 worklist, taken ahead of its dwell rank because SOTA-305 had just landed on the adjacent question. The finding that matters is not that FID is noisy but which noise dominates: the training run, by 3.2×, in a term no amount of extra sampling touches — and a control that rules out the boring explanation. | Read |
| NOTE-249 | MemGraphRAG: the pilot study is the contribution | Read from the #180 worklist in rank order. The system is a multi-agent GraphRAG pipeline with single-run numbers; the durable part is §3, which measures other people’s systems and finds graph expansion trading relevance for recall at a net loss on the end task. That is the filing. | Read |
| NOTE-250 | MIMI: the objective that needs the knowledge it assumes away | Read from the #180 worklist in rank order. The headline is a task-agnostic interface score; the durable finding is its failure case, where an assistant doing its job drives the metric negative and the fix is a horizon hyperparameter that requires exactly the task knowledge the objective was built to avoid. | Read |
| NOTE-251 | Non-identifiability of mechanistic explanations | Read from the #180 worklist, the strongest of the remaining candidates. Asks whether mechanistic interpretability’s criteria pick out a unique explanation and answers no at all four stages by exhaustive enumeration — no network had exactly one circuit interpretation — with both counts lower bounds by construction. | Read |
| NOTE-252 | RNA-FM, and the control that does not carry the claim | Read to open the biological-sequence trunk, which this record did not hold. The structural results are large and the paper’s one clean input-swap control is not: replacing one-hot sequence with a 640-dimensional pretrained embedding in a published UTR model moves R² from 0.814 to 0.816 on the real-human set, where a 16-dimensional predicted secondary structure moves it to 0.820. No practice filed. | Read |
| NOTE-253 v2 | The information-bottleneck rebuttal, and the assumption it makes visible | Read from the ICLR 2018 version, supplied by the record’s owner; co-author Kolchinsky’s publications page also links it. Three claims tested, none general. The finding worth more than the headline is in Appendix C: the same tanh network, binned evenly in net input instead of evenly in activity, shows no compression — because I(h;X) in a deterministic network is infinite and every finite number is a property of an imposed noise model. | Read |
| NOTE-254 | Adaptive estimators: a counter-rebuttal whose body is weaker than its abstract | Read to check whether it softens LIT-509, as #220 feared. It does on one narrow point and not on the one that matters. Its own Figure 6 shows a ReLU network averaged over 50 initializations has no distinct phase, though its Figure 7 averages do show ELU, Swish and centered softplus compressing; and it agrees with the rebuttal that hidden-layer compression does not correlate with generalization. | Read |
| NOTE-255 | Gemini 1.0: complete disclosure, and an abstract that picks the flattering row | Read from the #180 worklist for the model and filed for the appendices. The headline — first model to exceed human-expert MMLU — holds under one inference procedure out of three the report itself publishes, and under the other two GPT-4 is ahead. Nothing is concealed. The disambiguating numbers are in the same document, which is what makes this the corpus’s cleanest instance of DP-010. | Read |
| NOTE-256 | Image-GS: the rate accounting GaussianToken skipped, and an ablation that inverts its own headline | Read from the #180 worklist for its adjacency to the tokenizer trunk. Two things worth keeping: a comparison run the way this record keeps asking for — six baselines at matched model size, bitrate on the axis — and an ablation in which the two unglamorous optimizer choices outweigh the content-adaptive allocation the paper is named for. | Read |
| NOTE-257 | SVDQuant: a low-rank branch on the weights, not on the error | Read from the #180 worklist. The idea that travels is an ordering: decompose the weights and quantize the residual, rather than quantize and then patch the error. Two propositions and Eckart-Young say why, and a published method that does it the other way round is the control. | Read |
| NOTE-258 | mHC-lite: the experiment that was one measurement from settling the Birkhoff dispute | Read from the #180 worklist. It makes the doubly-stochastic constraint exact, which is precisely the condition under which LIT-151’s objection should bite hardest — and it reports no stream statistic, so the dispute stands. The approximation it replaces is real and measured: column sums of the layer-wise product off by up to 220% at 24 layers. | Read |
| NOTE-259 | Refine, then calibrate: the validation-loss minimum is a compromise nobody chose | Read from the #180 worklist and filed because the record had no calibration document at all. A proper loss is calibration error plus refinement error; the two bottom out at different epochs, so stopping on validation loss lands between them. Fit a temperature before reading the loss and the compromise goes away. 196 tabular datasets, three model families, ten runs per vision dataset. | Read |
| NOTE-260 | Block coordinate descent: a smaller machine bought with three times the steps | Read from the #180 worklist as the last of the 5-revisit tier. The method is sound and the accounting needs separating: the headline 2.6% mixes the method with the price of a 4090, the same-device figure is 33%, and the iteration counts in the paper’s own Table 2 show BCD paying about 3× the steps for the memory it saves. | Read |
| NOTE-261 | The singular distribution stops moving long before the weights do | Read for one question: is the spectral shape that three other papers measure a stable property or a snapshot? It is stable, and early. The empirical section was read and the theory that derives the stability bound was not, which is stated rather than implied. | Read |
| NOTE-262 | Eleven checkpoints agree on where the spectrum concentrates, and imitating it does not help | Read for breadth rather than depth of claim. Eleven GPT-2-style checkpoints differing in size, language, tokenizer and corpus share a depth profile in effective-rank entropy, which is the right instrument for a claim about where a matrix’s magnitude sits. The paper’s own headline — that you can initialize from that profile — is a negative result, reported as one. | Read |
| NOTE-263 | An MP bulk plus outliers at both ends, and why magnitude order is not importance order | Read to answer THEORY-059’s open question. It answers it and complicates it: language-transformer weight spectra are steep at the top, as the account needs, but the bottom of the spectrum in non-square matrices holds directions whose removal is catastrophic. Frobenius magnitude and functional importance are ordered differently, and a method that peels the top and quantizes the rest is keeping exactly the part this paper says is fragile. | Read |
| NOTE-264 | Low-rank structure in a transformer is non-uniform, and a global rank is measurably the wrong choice | Read second in the spectral cluster, and it is the one that carries a recommendation. “Transformer weights are low rank” turns out to be true of some matrices and false of others, with the split systematic by component and by depth. The comparison against a uniform rank at matched compression was run and the margins are large. | Read |
| NOTE-265 | A proved bound under the entropy-collapse observation, and a reparameterization that removes four crutches | Read on its own merits after being triaged past once. The record already held the failure this paper is about and two remedies for it, and had neither the inequality that makes the failure inevitable nor this third remedy. The accuracy numbers are nearly flat; what changes is how many hyperparameters have to be right. | Read |
| NOTE-266 | Not all entropy collapse crashes a model, and the discriminator is low-rankness | Read after the paper it contradicts. The contribution is a distinction the record did not have: an attention map can be sparse with near-zero entropy and the run is fine, provided it is not also low-rank. That makes entropy the symptom and spectral energy concentration in the query-key product the cause, and it makes a warmup-free optimizer fall out. | Read |
| NOTE-267 | Freezing the spectrum by construction, and what one 60M run can and cannot show | Read as the third position on one axis. If the problem is that weight spectra drift during training, the options are renormalize them, bound how fast they move, or make them constant. This does the third. The idea is clean and the evidence for the claim worth recommending is one model at 60M, which is why no practice is filed. | Read |
| NOTE-268 | A data-free lineage check built on the products that survive permutation and scaling | Read because the grep came back empty: the record holds no document about model provenance. The method is one good idea executed cleanly — compare the spectra of matrix products that functionality-preserving transforms cannot change — and the honest limit is that it is evidence of derivation, not proof against an adversary who has read the paper. | Read |
| NOTE-269 | Heads traded for depth, and the five rows where only those two things move | Read on its merits after being triaged past. The headline is an architecture recommendation and the interesting part is the mechanism: more heads lower the condition number of the attention block, which is the same quantity THEORY-041 says drifts the wrong way when nothing manages it. Half the configurations confound the trade by also halving the MLP; the other half do not, and those are the result. | Read |
| NOTE-270 | Concepts live in the quiet directions, and the paper labels its own claims robust or partial | Read last of the batch and the easiest call to make, for a reason that has little to do with the finding: it arrives with a table grading its own six claims, a robust null against the hypothesis it set out to confirm, and partial results labelled partial. The finding itself is that contextual concept directions sit at the bottom of the unembedding spectrum and static vocabulary contrasts sit at the top. | Read |
| NOTE-271 | One metric that separates transformers from every other architecture, and what it buys | Read for whether the record wants an inductive-bias claim. It does, because the claim comes with a computable scalar rather than an adjective, and the scalar does four jobs — separates architectures, predicts robustness, correlates with flatness, and moves during a grokking plateau. The fourth is the one this record has an existing use for. | Read |
| NOTE-272 | An idealization that checks its own assumptions against ALBERT, and clusters within a few layers | Read a second time, in full, after being declined on a skim. The decline said the t → ∞ frozen-weights idealization put it too far from a trained model to be evidence about one, and that it underwrites no practice. Both halves were wrong: the record’s own schema says a theory need underwrite nothing, and the paper checks its hypotheses on ALBERT, gives the discrete-time analogue, and says clustering arrives within a few layers. | Read |
| NOTE-273 | The observation outlived the method, and it is the observation the record needed | Read for one figure. The record holds two practices resting on attention being low rank and held no source for the claim; the record also already holds the verdict that this paper’s method was overtaken. Those are different things and the reading keeps them apart. | Read |
| NOTE-274 | The trunk under a term the record was using for two different things | Read because four documents in this record lean on “rank collapse” and none of them held its source. Reading it produced a finding that was not the point of the unit: the record carries that term for two distinct phenomena, and the conflation is the field’s rather than ours. | Read |
| NOTE-275 | The trunk states the condition its successors were refuted for dropping | Read because the record filed a three-paper information-bottleneck dispute and the paper all three argue about was absent — a gap this session created. Reading it reassigns some blame: the rebuttal’s central objection is that mutual information needs stochasticity, and the founding paper says so in one sentence before anybody measured anything. | Read |
| NOTE-276 | Order-sensitivity refutes it, and the models everyone measured were trained for the task | Read as the challenge from outside the setup. It separates the hypothesis that pretrained models do in-context learning by gradient descent from the hypothesis that transformer weights exist which simulate it, and shows the literature had been testing only the second. Theorem 1 then gives a contradiction from order sensitivity alone, and LLaMA-7B disagrees with fine-tuning on three metrics across four datasets. | Read |
| NOTE-277 | The algorithm changes with depth, and the one it lands on is Bayes-optimal ridge | Read as the second, independent arrival at the mesa-optimization construction. Its title asks which algorithm, and its answer is that there is no single one: shallow learners sit nearest gradient descent, deeper ones nearest ridge, deepest nearest ordinary least squares, and under label noise the best-fitting ridge parameter is the Bayes-optimal σ²/τ² at every setting. The gradient-descent phase is the small-model phase. | Read |
| NOTE-278 | Linear in Newton steps, exponential in gradient steps: a rate is what separates the rivals | Read as the challenge from inside the gradient-descent line’s own setup. Matching each transformer layer to the best number of steps of a candidate algorithm gives a linear trend against Iterative Newton — about 3 iterations per middle layer — and an exponential one against gradient descent; on data with condition number 100 the transformer is unchanged while GD needs 2,000 steps a 12-layer model cannot hold. The method is the contribution: fits cannot separate these rivals and rates can. | Read |
| NOTE-279 | The setup that made the mechanism question answerable | Read because every later claim about what in-context learning is measures in this setup. A 9.5M-parameter GPT-2 trained from scratch on (x, f(x)) sequences matches ordinary least squares on unseen linear functions and Lasso on sparse ones, in a single forward pass. The training objective is the setup’s power and, later, the ground of the main objection to everything built on it. | Read |
| NOTE-280 | The construction is exact, and the trained model it describes is one layer deep | Read as the trunk of the mesa-optimization line. Proposition 1’s weight construction is exact and a single trained linear-self-attention layer approximately finds it. The result that matters most for what came after is four pages later and rarely quoted: deeper trained models do not match gradient descent, they match GD++ — gradient descent on data transformed by I − γXXᵀ — which is the paper’s own evidence that plain GD is the wrong description beyond one layer. | Read |
| NOTE-281 | The founding paper already measured the regime dependence | Read because the record staked a claim on grokking being a data-starved-regime phenomenon while holding only the paper that put a number on it. The founding paper measured the dependence too, and reported it as a headline: converged accuracy is flat across a range of training-set sizes while time-to-generalize explodes as the set shrinks. It also names weight decay as the strongest intervention it found, which is the thread every later mechanism pulls on. | Read |
| NOTE-282 | Induced on images, text and molecules, and eliminated on demand | Read to settle whether grokking is specific to algorithmic data. It is not — but producing it on MNIST, IMDb and QM9 takes two deliberate departures from standard practice: a much smaller training set and an inflated initialization scale. At standard initialization there is no grokking on any of them. The control runs both ways: constraining the weight norm nearly eliminates grokking on algorithmic data. | Read |
| NOTE-283 | Two novel phenomena predicted from the theory and then observed | Read as the mechanistic account with the best evidential shape in this cluster. From “weight decay prefers the circuit with more logit per unit norm, and memorisation gets less efficient as the dataset grows” it derives a critical dataset size, and from that derives ungrokking — a grokked network losing its generalization when trained on less data — and semi-grokking. Both were then observed. The mechanism needs weight decay, which is where the counterexample lands. | Read |
| NOTE-284 | A counterexample with no weight decay and a rising weight norm | Read because it is the reason this cluster cannot be filed as a single account. A two-layer MLP groks on modular arithmetic with no weight decay and a parameter norm that goes up through the transition — which neither the LU mechanism nor circuit efficiency can produce, since both turn on the norm coming down. The replacement offered is a transition out of lazy dynamics, with two measurable knobs; what it does not address is ungrokking. | Read |
| NOTE-285 | A complexity measure that discriminates where the training loss is identically zero | Read as the instrument that makes the school usable. The LLC is the volume scaling exponent of the near-optimal set around one minimum, estimated by SGLD at inverse temperature 1/log n. Validated against known theoretical values on deep linear networks to 100M parameters. The result that makes it a practice: on ResNet18/CIFAR10 it separates learning rate, batch size and momentum settings whose training losses have all collapsed to zero. | Read |
| NOTE-286 | The effective parameter count is a volume exponent, and the Hessian is measuring the wrong thing | Read as the trunk under a school the record was already citing. Its single most consequential sentence for this record is that the RLCT matters more than the curvature of the directions it counts — a published argument that the quantity a sharpness visualisation displays is not the quantity that governs generalisation, which is what SOTA-012 rests on. | Read |
| NOTE-287 | Stage boundaries found without knowing in advance what to look for | Read as the application that makes the LLC matter to this record. Critical points of the LLC curve divide two transformers into five stages each, and the boundaries land on bigrams, n-grams, previous-token heads and the induction circuit — found without a mechanistic hypothesis in hand, which is what LIT-085’s progress measures require. Its in-context regression model acquires in-context learning and then loses it. | Read |
| NOTE-288 | The transition the posterior makes and the transition SGD makes are not the same transition | Read for the distinction in its title, which the record needed and did not have. Bayesian phase transitions are changes in where the posterior concentrates as sample size grows, and are well defined; dynamical transitions are changes in the SGD trajectory, and are not. In the one model where both are computed, the sharpest Bayesian transition — predicted at n_cr ≈ 600 and observed at 600–700 — has no dynamical counterpart at all. | Read |
| NOTE-289 | MatFormer | One Transformer with four nested FFN widths, trained one width per step, gives four sizes plus Mix’n’Match sizes nobody trained. At the compute of training the four separately, it matches the largest and beats the smaller ones, which see up to 4× the data through shared weights. Read §1–5 and Appendices B, D.1, E.1 and F. | Read |
| NOTE-290 | ReFT | A learned low-rank edit to the residual stream at a few prompt positions, with the model frozen. Up to 65× fewer parameters than LoRA. It leads on commonsense QA and on GPT-4-judged instruction following, is level on GLUE, and loses on arithmetic chain of thought. All baselines are copied from earlier papers. Read §1–6 and Appendix D; the other appendices were not read. | Read |
| NOTE-291 | Matryoshka Representation Learning | Sum the task loss over log(d) nested prefixes of one embedding, and each prefix matches a separately trained model of that width on ResNet50 and ImageNet. Retrieval that shortlists on 16 dimensions and re-ranks on 2048 is 14× faster at equal mAP@10. Read §1–6 and Appendices D.2 and I; the remaining appendix tables were skimmed. | Read |
| NOTE-292 | Tensor Programs IV | Classifies width-scaling parametrizations (abc-parametrizations) and proves that every stable, nontrivial one either learns features in the wide limit or trains as a kernel, not both. The standard parametrization is stable only at an O(1/width) learning rate, which is the kernel side. Introduces µP as the unique member that updates every layer maximally. Read end to end through §10 and Appendices A, C.1 and D. The proofs in Appendix H were not checked line by line. | Read |
| NOTE-293 | Gemma 3n | Per-Layer Embeddings: every token id indexes a table with one 256-d vector per layer, gated into the residual at each layer, about 2.35B parameters held off the accelerator. No measurements are published. Read from the model overview, the developer guide, and the Hugging Face transformers implementation, which is the only place the mechanism is fully specified. | Read |
| NOTE-294 | XGrammar | Grammar-constrained decoding made nearly free. It splits the vocabulary into tokens whose validity depends only on the automaton’s stack top, which are precomputed, and the under-1% that need the full stack, which are checked at runtime. The mask is built on the CPU during the forward pass. Each optimization is ablated. Output quality is measured only as syntactic validity. Read §1–6; the appendix was not read. | Read |
| NOTE-295 | SIREN | Sine-activated MLPs whose derivatives are well behaved, trainable because weights U(±√(6/n)) keep pre-activations standard normal and outputs arcsine at every depth, with ω₀ = 30 on the first layer. They fit signals and derivatives that ReLU, tanh and positional encoding do not. Read §1–4.2 and supplement §1; the later experiments were skimmed. | Read |
| NOTE-296 | Fourier features | A coordinate MLP’s NTK has fast spectral falloff, so it fits high frequencies too slowly to matter. A sinusoidal input mapping makes the composed kernel stationary and sets its bandwidth, and the NTK linear model predicts the trained networks’ loss curves. Gaussian random frequencies are best on seven tasks, and only their scale matters. Read §1–7; the appendices were not read. | Read |
| NOTE-297 | Warm Diffusion | Blur and noise mixed in one forward process, with a two-headed model that denoises to the blurry image and deblurs the residual. A blur-to-noise ratio of 0.5, chosen from image and noise spectra, edges EDM on CIFAR-10 (1.85 against 1.97). High ratios degrade sharply, which is the evidence that noise keeps samples on the manifold. Read in full (8 pages). | Read |
| NOTE-298 | Diffusion Forcing | Independent per-token noise levels during training give one causal model both autoregressive sampling and horizon-wide guidance, with sampling schedules chosen at inference. Stable long video rollouts are shown qualitatively. Planning and robot results are measured. Read §1–5; the appendix proofs and time-series results were not read. | Read |
| NOTE-299 | Cold Diffusion | Diffusion generalized to deterministic degradations (blur, masking, downsampling, snow) with a restorer and a sampler that is exact for degradations linear in severity. Conditional restoration works. Noiseless unconditional generation is far worse than noise diffusion, and adding a little noise recovers half the gap. Read §1–5.3; the appendix was not read. | Read |
| NOTE-300 | StyleGAN3 | Aliasing in the generator glues fine detail to pixel coordinates. Continuous-signal design, with 2×-upsampled filtered nonlinearities, Fourier input and no noise, gives translation- and rotation-equivariant generators at StyleGAN2’s FID. Read §1–3.2 and Figure 3; the internal representation analysis and the appendices were skimmed. | Read |
| NOTE-301 | StyleGAN2 | Diagnoses two StyleGAN artifacts and fixes each at its cause: weight demodulation instead of instance normalization, and a fixed skip/residual architecture instead of progressive growing. Adds lazy regularization and path-length regularization. Read §1–4.1 and Tables 1–2; §4.2 onward and the appendices were skimmed. | Read |
| NOTE-302 | CycleGAN | Unpaired translation with two adversarial losses and a bidirectional cycle-consistency loss. It works for appearance changes and fails for geometric ones. The ablation supports the cycle term against GAN-only, but not strictly the bidirectional form over one direction. Read §1–6; the appendix was skimmed. | Read |
| NOTE-303 | Projected GAN | A GAN discriminator on frozen, randomly mixed, multi-scale pretrained features reaches prior-best FIDs up to 40× faster and sets new FIDs on 22 datasets. Everything is measured in FID or ImageNet-derived metrics, and the discriminator is ImageNet-pretrained, which LIT-563 later shows distorts FID. Read §1–5; the appendix was not read. | Read |
| NOTE-304 | The Role of ImageNet Classes in FID | FID can be lowered by two-thirds without changing a generator, by resampling its outputs to match ImageNet-class statistics. Non-ImageNet feature spaces barely move. An ImageNet-pretrained discriminator gets the same effect by accident. Read §1–5; the appendices were skimmed. | Read |
| NOTE-305 | MARLIN | An FP16×INT4 GEMM kernel that stays memory-bound, and so near the 3.87× ideal 4-bit speed-up, up to batch 16–32 on a large layer, decaying toward 1.5× at 128. End-to-end in vLLM the same shape holds on four GPU classes (Table 2): 2.3–3.2× single-GPU at batch ≤16, 1.1–1.2× at 128, and much less when the model is sharded over eight A100s. The INT4 model it serves loses 3.3 points of mean accuracy (Table 1). | Read |
| NOTE-306 | SDXL | A larger latent diffusion UNet with two text encoders, micro-conditioning on original size and crop, multi-aspect training and a refiner. Size conditioning is ablated on ImageNet: most of the gain is from keeping the data, and conditioning adds about 3 FID. Read §1–2.5 and Appendix B; the rest was skimmed. | Read |
| NOTE-307 | Scaling and evaluating sparse autoencoders | TopK SAEs, a dead-latent recipe, and scaling laws, demonstrated on a 16M-latent SAE on GPT-4. Also shows that the field’s usual fidelity metric flatters. Sections 1–4 and 6 read. Section 5 (TopK variants, progressive codes) and the appendices skimmed. | Read |
| NOTE-308 | Tuned Lens | The logit lens decodes intermediate layers with the final unembedding, and fails or misleads on many models. A per-layer affine translator, distilled toward the final logits, fixes this across every family tested. Sections 1–5 read, appendices skimmed. | Read |
| NOTE-309 | Are Sparse Autoencoders Useful? A Case Study in Sparse Probing | On 113 datasets and five regimes, probes on SAE latents do not beat, or usefully add to, probes on raw activations once method selection is honest. Earlier wins, including the authors’ own, came from weaker baselines. Main text read, appendices skimmed. | Read |
| NOTE-310 | FLUX.1 Kontext | The FLUX.1 architecture, and an editing model built by appending context latents to the token sequence. It is a system report: human preference against other systems, latency, one reconstruction table, and no ablations. Main text and appendix A read. | Read |
| NOTE-311 | FLUX.2 latent-space report | A controlled comparison of four autoencoder latents for flow models on ImageNet, with the timestep schedule swept per latent. The schedule decides the ranking. FLUX.2’s VAE gets near-RAE learnability with the best reconstruction. Read in full from the rendered report, except the interactive Table 2, which did not load. | Read |
| NOTE-312 | FFN layers as key-value memories | Keys fire on input patterns, and values project to next-token preferences. The reading is interpretable in upper layers of one small WikiText model, with weak key-value agreement and mostly compositional outputs. | Read |
| NOTE-313 | Does Localization Inform Editing? | Per fact, causal tracing’s localization is unrelated to where an edit succeeds. The edit layer explains 94.7% of the variance in ROME’s success on GPT-J, and tracing adds 0.1%. Main text read, robustness appendices skimmed. | Read |
| NOTE-314 | RippleEdits | Knowledge-editing benchmarks test the edited fact and leave its consequences untested. On six ripple criteria, weight editors average 38–66, and prompting with the new fact beats them. Main text read. | Read |
| NOTE-315 | ROME | Causal tracing localizes factual recall to mid-layer MLPs at the last subject token, and a rank-one MLP edit there rewrites single facts with both generalization and specificity. Main text and the CounterFact appendix read. | Read |
| NOTE-316 | MEMIT | ROME’s idea extended to thousands of facts by spreading updates across a range of MLP layers in one batched solve. 85.8 on 10,000 CounterFact edits on GPT-J, where ROME falls to 50.3. Main text read, appendices skimmed. | Read |
| NOTE-317 | Where did the gap go? | Retuned message-passing baselines close the reported graph-transformer gap on LRGB’s Peptides datasets, mostly by adding an MLP prediction head. Feature normalization lifts every model on the superpixel datasets. Read in full. | Read |
| NOTE-318 | Pitfalls of GNN evaluation | With one shared training and tuning protocol over 100 splits × 20 seeds, GCN ranks first among four GNNs, and the single Planetoid split reorders models. Read in full. | Read |
| NOTE-319 | GIN | Message passing is bounded by 1-WL, and sum aggregation with an MLP reaches the bound where mean and max do not. The theory predicts the training-fit ordering. Test accuracy separates the aggregators decisively only on featureless graphs. Main text read, proofs skimmed. | Read |
| NOTE-320 | AlphaFold 3 | AlphaFold generalized to all biomolecular complexes, with a diffusion head over atom coordinates. It gains on ligands, nucleic acids and antibodies, and inherits generative failure modes (hallucination, chirality) that it patches with distillation and ranking. Main text read. Extended Data and supplementary methods not read. | Read |
| NOTE-321 | AlphaFold 2 | Near-experimental protein structure accuracy at CASP14, a calibrated confidence head, self-distillation on confident predictions, and a hard dependence on alignment depth below about 30 sequences. Main text and Methods read. The supplementary methods (architecture details, ablation descriptions) were not read. | Read |
| NOTE-322 v2 | Latent Dirichlet Allocation | The bag-of-words assumption, read through de Finetti, implies a document-level mixture. LDA is that mixture with Dirichlet topic proportions. It generalizes better than one-topic-per-document and per-document-parameter models by held-out perplexity. Main text read in full; the variational derivations in the appendices skimmed. | Read |
| NOTE-323 | Reading Tea Leaves | Two human tasks, word intrusion and topic intrusion, measure whether a topic model’s topics and document assignments mean anything to people. The model with the best held-out likelihood, CTM, does worst on both. Read in full. | Read |
| NOTE-324 | Automatic Evaluation of Topic Coherence | Average pairwise PMI of a topic’s top words, estimated on Wikipedia, correlates with human coherence ratings about as well as the paper’s measure of annotator agreement. Knowledge-base and search-engine measures are less reliable. Read in full. | Read |
| NOTE-325 | word2vec (architectures) | CBOW and skip-gram, log-linear models that learn word vectors from billions of words cheaply, and the analogy benchmark that made “king − man + woman ≈ queen” a test. Read in full. | Read |
| NOTE-326 | word2vec (negative sampling, subsampling, phrases) | The tricks that made word2vec the standard: negative sampling with a unigram^(3/4) noise distribution, subsampling of frequent words, and data-driven phrase tokens, plus additive compositionality. The 3/4 exponent is asserted without numbers. Read in full. | Read |
| NOTE-327 | GloVe | Derives a word-embedding objective from the claim that meaning lives in ratios of co-occurrence probabilities. It fits log co-occurrence counts by weighted least squares and beats default-configured word2vec on analogies. Read in full. | Read |
| NOTE-328 | Linear Representation Hypothesis (Park, Choe, Veitch) | Formalizes “concepts are directions” with counterfactual word pairs, proves that the unembedding direction is a probe and the embedding direction a steering vector, and shows the two coincide under a causal inner product. One such product is the inverse unembedding covariance. The experiments, on LLaMA-2 7B, are qualitative. Read in full, appendices included. | Read |
| NOTE-329 | Levy, Goldberg & Dagan: lessons from word embeddings | The prediction-over-count advantage for word embeddings came from design choices bundled with word2vec, not from the algorithm. Ported to PPMI and SVD and tuned alike across 672 representations, no method wins consistently, SGNS beats GloVe, and context-distribution smoothing is the only universally safe setting. Read in full. | Read |
| NOTE-330 | Baroni, Dinu & Kruszewski: Don’t count, predict! | 84 count and predict models on 14 lexical-semantics benchmarks from one 2.8B-token corpus. word2vec CBOW beats count vectors almost everywhere and is far more robust to bad settings. The count side lacked word2vec’s smoothing and negative shift, which is how LIT-607 later reversed the result. Read in full. | Read |
| NOTE-331 | Levy & Goldberg: SGNS as implicit PMI factorization | SGNS’s objective is optimized at w·c = PMI(w, c) − log k, a weighted factorization of the shifted PMI matrix. Shifted PPMI nearly attains that optimum and SVD over it matches SGNS on similarity, while SGNS keeps an edge on syntactic analogies. Read in full. | Read |
| NOTE-332 | Arora et al.: RAND-WALK and PMI in low dimensions | A random-walk generative model with isotropic word vectors predicts PMI ≈ ⟨v, v’⟩/d in low dimensions, justifies GloVe’s biases and CBOW’s averaging, and explains why analogies survive large fitting error: isotropy denoises the offset. Its assumptions are checked loosely on one corpus. Read in full, with the main-text proofs. | Read |
| NOTE-333 v2 | Wan | An open 14B text-to-video recipe whose evidence is mostly systems measurement: 2D context parallelism cuts communication overhead from over 10% to under 1%, and caching and 8-bit kernels give 1.62× and 1.27×. Its three modelling ablations run on 1.3B text-to-image. Two are read from training loss, and on FID one of them favours the rejected option. The applications in §5 are demonstrated almost entirely by figures. | Read |
| NOTE-334 | Movie Gen | At 5B, flow matching beats v-prediction diffusion on quality by 3.3 annotation σ, but on text alignment by only 1.9σ. Video-native captions beat captions stitched from three frames on alignment by 2.9σ. Every σ in the paper is rater variance, and each ablation arm was trained once. The Llama 3 block beats a DiT block that differs only in norm, activation and bias. The scaling claim rests on image-only runs covering less than a quarter of a decade of compute. | Read |
| NOTE-335 | Self Forcing | Training a few-step autoregressive video model on its own rollouts, with a distribution-matching loss on the whole clip, beats teacher- and diffusion-forced training in matched few-step comparisons (VBench 84.31 against 82.76 and 82.32 chunk-wise), and does not drop when moved to frame-wise generation, where the baselines do. It runs at 17 fps with 0.69s latency on one H100, and its parity with the slower 1.3B base rests on a 14B teacher: the SiD variant, with a 1.3B teacher, scores 84.07 against the base’s 84.26. | Read |
| NOTE-336 | Stable Video Diffusion | At fixed architecture and a fixed step count, a video model pretrained on a curated quarter of a 9.8M-clip pool is preferred by human raters to one trained on the whole pool, and the lead survives high-quality fine-tuning (Elo +127 against +89 over an image-only start at 10k steps). Almost every result is a preference bar from 64 prompts with no numbers in the text, and two of the curation choices the paper shipped lost its own ablation. | Read |
| NOTE-337 | Rectified Flow | Regressing a velocity onto x1 − x0 along the straight interpolation gives an ODE that preserves the marginals and does not increase any convex transport cost, and retraining on its own couplings (“reflow”) provably straightens it at O(1/K). On CIFAR-10 the single-pass model scores FID 2.58 against 3.93 for the VP probability-flow ODE at the same architecture, with training budget unstated. Most of the one-step result comes from distillation (378 → 6.18 with no reflow), and reflow adds 6.18 → 4.85. | Read |
| NOTE-338 v2 | Step-Video-T2V | A 30B text-to-video DiT on a 16×16×8 VAE. By the authors’ own benchmark the VAE reconstructs about as well as HunyuanVideo’s 8×8×4, and an independent measurement in Open-Sora 2.0 disagrees. Video DPO raises a tie-weighted preference score from 45% to 55% on 300 prompts. Every architecture comparison is reported without numbers, and the “state of the art” in the abstract is contradicted by the paper’s own Table 9. | Read |
| NOTE-339 | AnimateDiff | Temporal-attention modules trained on video over a frozen Stable Diffusion 1.5 plug into community checkpoints fine-tuned from the same base without further tuning. Against two baselines not built for the task it wins clearly only on smoothness: users ranked the training-free Text2Video-Zero higher on preserving the personalized domain (2.620 against 2.280), which is the property the design exists for. Version 1’s reported failure on stylized domains was dropped in version 2, and its “most checkpoints” became “any”. | Read |
| NOTE-340 v2 | Flow Matching | Regressing onto a per-sample conditional vector field trains a continuous normalizing flow without simulation, with the same gradient as the intractable marginal objective. With the U-Net, hyperparameters and epochs held fixed, the straight “OT” path beats the VP diffusion path under that objective at CIFAR-10 and ImageNet 32/64 with uniform timesteps (FID 6.35 against 8.06 on CIFAR-10). The objective’s own advantage over score matching is mixed, and the likelihood gains at ImageNet scale are 0.01 bpd. | Read |
| NOTE-341 | Video LDM (Align your Latents) | Video LDM makes a latent image diffusion model into a video model by inserting temporal layers and fine-tuning the decoder and upsampler on video. On driving scenes, an image-pretrained LDM with frozen spatial layers reaches FVD 534.17 against 1155.10 for the same design trained from scratch. The “images” are 1 fps frames of the same driving videos, and the pretrained arm gets 73K image steps the other does not. Decoder video fine-tuning cuts reconstruction FVD 390.88 → 32.94. | Read |
| NOTE-342 | MAGVIT-v2 | A causal 3D-CNN video tokenizer with lookup-free quantization (sign-binary latents plus an entropy penalty) improves a fixed ~307M masked token model on Kinetics-600 frame prediction from FVD 9.9 to 5.2 at equal decoding steps. The claim that LFQ, unlike VQ, keeps improving generation as the vocabulary grows rests on a figure with no numbers. “Beats diffusion” holds only with guidance (1.78 against MDT’s 1.79 at 256, 1.91 against VDM++‘s 2.65 at 512), and the strongest diffusion baselines at 512 are about 6× its size. | Read |
| NOTE-343 v2 | CausVid | Distilling a bidirectional video DiT into a 4-step block-causal student with distribution matching gives a streaming generator: 1.3s to the first frame and 9.4 fps, against 219.2s and 0.6 fps for the teacher. At equal initialization a bidirectional teacher makes a better causal student than a causal one (VBench 94.7 / 64.4 / 30.1 against 91.9 / 61.7 / 28.2). The causal teacher is also the weaker model, so the paper’s error-accumulation explanation is not isolated. | Read |
| NOTE-344 | HunyuanVideo | A 13B open text-to-video model whose one quantitative design argument is a pair of compute-optimal fits on a proxy family: the image fit puts more of each added FLOP into parameters (0.5634 against 0.4325), and the video fit, initialized from the image envelope, reverses it (0.3618 against 0.6289). No architectural or data choice is ablated, and the proxy differs from the shipped model in objective, text encoder and conditioning. | Read |
| NOTE-345 | Open-Sora 2.0 | An 11B video model initialized from FLUX, trained mostly at 256px, and adapted to 768px as text/image-to-video. The “$200k” is 4,160 H200 GPU-days for one final run on HunyuanVideo’s VAE. The 4×32×32 Video DC-AE is a separate, unconverged experiment whose cost is outside that figure. Every training-strategy choice that makes the run cheap is asserted, not ablated. | Read |
| NOTE-346 | CogVideoX | An open text-to-video DiT whose design choices are backed by curves rather than numbers. On full 3D against factorized attention, the text says only that factorized FVD is “much higher … in early steps” and that factorized training is “unstable and prone to collapse”. Model size, steps and resolution are unstated. The one measured number is the cost: full attention’s forward pass is 1.08×, 1.67× and 2.30× slower at three sizes. | Read |
| NOTE-347 | Video Diffusion Models | A standard Gaussian diffusion model with a space-time factorized 3D U-Net generates video. Appending 0, 4 or 8 independent frames to each training video takes text-to-video FVD from 202.28 to 68.11 to 57.84 at fixed batch size and steps, but the added frames are extra per-step compute and the paper’s own gloss is “more independent examples in a batch”. Reconstruction guidance beats the replacement method for autoregressive extension, 136 against 451 FVD. | Read |
| NOTE-348 | Imagen Video | A seven-model pixel-space cascade whose one well-measured result is that guided progressive distillation to 8 steps per stage cuts sampling from 618s to 35s. The CLIP metrics cannot tell the two apart: they fall slightly under constant guidance, rise under oscillating guidance, and rate every sample above the real videos. Joint image-video training, the choice later reports copied, is asserted without a number. | Read |
| NOTE-349 | Make-A-Video | Make-A-Video extends an unCLIP text-to-image model to video with identity-initialized pseudo-3D conv and attention layers trained on video with no captions. Text reaches the video decoder only through one CLIP image embedding, so text-only-in-video associations cannot be learned, and the discussion concedes this. System-level results are strong (zero-shot UCF-101 FVD 367.23, MSR-VTT CLIPSIM 0.3049), and there is no ablation of any design choice, image initialization included. | Read |
| NOTE-350 v2 | LTX-Video | A 1.9B video DiT on a 32×32×8, 128-channel VAE with no patchifier (1:8192 pixels to tokens), whose decoder also does the last denoising step. It is fast and preferred over same-size 2024 open models in a 20-rater survey. The VAE has no reconstruction metric at all, and the denoising decoder, the paper’s central idea, rests on one unquantified internal study. | Read |
| NOTE-351 | Emu Video | Generating a first frame and conditioning the video on it beats direct text-to-video at matched data, steps and trainable parameters (70.5% / 63.3% human win rate). Five one-change ablations with majority-of-5 votes on 307 prompts are the most controlled design evidence in the early video line. The freeze-versus-fine-tune result (55.0 / 58.1) is near chance, and by construction it cannot see the first frame. | Read |
| NOTE-353 v2 | Rarely categorical, highly separable representations along the cortical hierarchy | In 14,000+ IBL Neuropixels units (4,617 passing a ΔR² ≥ 0.015 selectivity threshold), the clustering of 8-D encoding-model selectivity vectors beats a covariance-matched Gaussian null in only a handful of the ~20 testable cortical regions (VISp, AUDp, SSp-ul at Bonferroni p < 0.05; clustering z falls with hierarchy position, ρ = −0.62). Pooled modules and the whole cortex do cluster (whole-cortex z = 8.0). Among the 16 regions with enough trials, once the 16 task conditions are merged into pairwise-decodable “independent” ones (M_IC from 5 to 16, rising with hierarchy, ρ = 0.77), ≥ 95% of random balanced dichotomies are linearly decodable above a shuffle null in every region except gustatory cortex (≈0.82). | Read |
| NOTE-357 | SEDD | Score entropy makes the ratio view of discrete diffusion trainable and gives it a likelihood bound; with it, absorbing-state diffusion reaches a near-tie with a retrained autoregressive model on LM1B. “Beats GPT-2” is 3 of 5 zero-shot sets against a model trained on different data. The controlled result that lasts is that masking beats uniform corruption on every table. | Read |
| NOTE-358 | Diffusion Models Beat GANs | An ablated U-Net (ADM) and a classifier-gradient scale that trades recall for precision take diffusion past BigGAN-deep on ImageNet FID. The architecture search is single-run; every guided number is the best of a small scale sweep scored on the metric it reports; and the trade is non-monotone — a scale of 1 makes an unconditional model’s FID worse (26.21 → 33.03). The claim that large scales produce no adversarial examples is asserted in the introduction and checked only by eye and in Inception feature space. | Read |
| NOTE-359 | Non-Euclidean Edge of Stability | Steepest descent under the ℓ∞, block and spectral norms, and SignGD and normalized Spectral GD, sharpen to 2/η and train there — measured in the update’s own norm, where the ordinary top Hessian eigenvalue stays far below the threshold. Reading it: the identity is exact and the empirics are consistent, but the sharpness is a Frank-Wolfe estimate, the divergence theorem covers one line of initializations, and everything is full-batch. | Read |
| NOTE-360 | MDLM | Masked diffusion’s ELBO is a schedule-weighted average of masked-LM losses, and with a modern recipe it is the best discrete diffusion language model at 110M parameters. It does not beat autoregression in-domain, even with the AR baseline given half the optimizer steps; the paper’s own checklist names that as its limitation. | Read |
| NOTE-361 | DeepSeek-V4 | Two MoE models (1.6T/49B and 284B/13B) reach a native 1M-token context by interleaving two compressed attentions — 4x-compressed KV with top-k sparse selection (CSA) and 128x-compressed dense KV (HCA), each with a 128-token sliding-window branch — at 27% of V3.2’s per-token FLOPs and 10% of its KV cache for Pro at 1M. Multi-token prediction is kept “without modification” from V3 as a depth-1 training objective; the report never mentions speculative decoding or a draft model. | Read |
| NOTE-362 | Integrated Gradients | Integrating the gradient along the straight line from a baseline gives the unique path attribution that is implementation-invariant, complete and symmetry-preserving. The justification is axioms, not a measurement: the paper argues that empirical evaluation of attributions is confounded, and its one comparison with another method is by eye. | Read |
| NOTE-363 | Dropout as ensemble learning | In a teacher-student soft-committee simulation, dropout at p = 0.5 beats a fixed two-way split of the same network trained as an ensemble, and matches a tuned L2 penalty. That is the whole result: curves averaged over 10 trials, with no derivation, no numbers, one p and one toy model. | Read |
| NOTE-364 | Dropout and BN variance shift | Dropout upstream of batch norm changes a unit’s variance between train and test mode, and BN’s frozen moving variance does not follow. The measured cost is large: CIFAR-100 DenseNet 77.42% → 68.55%, and the model misclassifies its own training data in eval mode. It shrinks with a lower drop rate and with a wider fan-in. Moving dropout after the last BN avoids the cost, but adds only about 0.2 top-1 on ImageNet by itself. | Read |
| NOTE-365 v3 | Sanity Checks for Saliency Maps | Randomize the model’s weights, or train it on permuted labels, and see whether the saliency map changes. Gradients and GradCAM change. Guided Backprop is invariant to the higher layers. Integrated Gradients keeps the input’s structure while its sign decorrelates. Visual inspection cannot tell these cases apart, and that is the paper’s point. | Read |
| NOTE-366 | Primer | Squaring the feed-forward ReLU and adding a causal width-3 depthwise convolution after each per-head Q/K/V projection makes decoder-only LMs reach a vanilla baseline’s quality in 1.75–2.3× less compute at 35M, at least 1.8× at 110M and 4.2× at 537M. That last figure is at fixed size, where the factor grows with training length. At 1.9B, against a GELU baseline, it takes ⅓ of the compute. Both changes cost step time and are paid for in the accounting. Squared ReLU beats SwiGLU in exactly one figure, at 110M. Against the SwiGLU + RMSNorm baseline the saving is about 2×, and it vanishes on encoder–decoder masked LM. | Read |
| NOTE-367 | Narang et al., transformer modifications | About fifty transformer modifications reimplemented in one T5/Mesh TensorFlow codebase at 223M parameters, with hyperparameters fixed, on four task families. Most do not beat the vanilla pre-norm, relative-attention baseline. The winners are GLU-variant activations, RMSNorm, untied embeddings, and sparse experts or other variants that cost parameters or time. The Universal Transformer, ReZero, Fixup, block sharing and the Evolved Transformer all lose. The positives are largely same-codebase re-runs, GLU Variants included, so the paper is strong evidence against the losers and weak evidence that the winners transfer. | Read |
| NOTE-368 | Attention-only transformers | A necessity test of the feed-forward layer with three matchings. Deleting it in place costs 0.47 nats. At matched FLOPs the cost is 0.26 nats. At matched parameters, with the budget moved into attention depth, the cost is 0.0055 nats, and it shrinks with tokens. What remains is concentrated on low-context tokens, where recall has to come from the weights. QK-norm is what keeps the attention-only stack trainable. At or below 87M parameters, on a synthetic reasoning corpus. | Read |
| NOTE-369 | Min-SNR weighting | Capping the x0-space loss weight at min(SNR, 5) speeds diffusion training under x0, ε and v prediction, and makes ε output trainable under weightings that otherwise diverge. The 3.4× headline is against constant-weighted x0-prediction. Against plain ε-MSE on a UNet, FID goes 8.55 → 7.32 at 200K iterations and 4.21 → 4.14 at 1M, single runs. The multi-task “conflicting gradients” account is motivated by one probe and a proxy objective, and is not tested against FID. | Read |
| NOTE-370 | EDM2 | Magnitude-preserving layers plus forced weight normalization take the ADM U-Net from FID 8.00 to 2.56 on ImageNet-512 at equal compute, in a cumulative single-model ladder whose steps are all far larger than FID’s noise. Post-hoc EMA reconstructs any averaging length from two stored power-function averages, and what it shows is that the best length depends on architecture, learning rate, guidance weight and the metric — the last by 2% against 14% under guidance. | Read |
| NOTE-371 | FLIP | Removing 50% of image patches in CLIP training halves the image encoder’s cost and lets the contrastive batch double at the same memory; ViT-L/16 on LAION-400M then matches its unmasked reproduction more than 3× faster and beats it by about a point at equal epochs. At equal batch the masking gains nothing, and at ViT-B/16 the full-schedule model is 0.2 behind — the improvement is the batch and the extra samples that masking pays for. | Read |
| NOTE-372 | Dropout as a Bayesian approximation | Dropout before every weight layer, with L2, is shown to optimize (up to an approximated KL term) a variational objective for a deep Gaussian process, so running T stochastic forward passes at test time yields a predictive mean and variance for free. The evidence is qualitative figures, a UCI regression table on 50-unit one-layer networks, and one RL run; the MC-versus-weight-scaling comparison is stated without numbers. | Read |
| NOTE-378 | Talagrand Meets Talagrand | Take a centered Gaussian process on a finite set T with max variance σ² and minimum separation a. The expected softmax-weighted average g(β) = E[Σ_t X_t e^{βX_t}/Z] satisfies g(β) ≤ √(2σ²·E D(ν_β‖ν₀)) for every β (Thm 1, D = KL from uniform). For β ≥ β*, g(β) ≥ c·a·√(E D(ν_β‖ν₀)), with c the Sudakov constant (1/17 via Marcus–Rosen) (Thm 2). For i.i.d. N(0,σ²) logits there is an exact identity, g(β) = βσ²(1 − E‖ν_β‖²₂) (Lemma 2). The quenched free energy is bounded above by √(2σ²·E D_{1/2}(ν_β‖ν₀)), with an i.i.d. lower bound of (cσ/2)√(E D_{1/2}) (Thms 4–5). | Read |