model-architecture.
53 of 378 NOTE documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| NOTE-004 | Scale Efficiently | Model shape, not only model size, determines downstream fine-tuning quality — and the widely adopted T5-Base and T5-Large configurations are Pareto-inefficient. Read in full for #114 after its LIT note was found to describe a different paper. | Read |
| NOTE-011 | Attention Is All You Need | Dispenses with recurrence and convolution entirely. The 1/√d_k scaling has a stated reason: with unit-variance components, q·k has variance d_k, and unscaled dot products push the softmax into regions of extremely small gradients. | Read |
| NOTE-016 | Fast Transformer Decoding: multi-query attention | Incremental decoding is bounded by the memory bandwidth of reloading the keys and values, not by arithmetic. Share one key/value head across all query heads: much faster decoding, “only minor quality degradation”. | Read |
| NOTE-022 | RMSNorm | LayerNorm gives re-centering and re-scaling invariance; the hypothesis is that re-centering is dispensable. Normalize by root mean square alone — re-scaling invariance and implicit learning-rate adaptation kept, 7–64% less time per step. | Read |
| NOTE-028 | Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation | Replaces position embeddings with a fixed linear penalty on attention scores proportional to query–key distance, one slope per head from a geometric sequence. Trains on short sequences and evaluates on longer ones. The paper’s own Appendix B says the gain is “largely explained by” avoiding the early token curse rather than by better use of long history. | Read |
| NOTE-054 | Mish: A Self Regularized Non-Monotonic Activation Function | Proposes f(x) = x·tanh(softplus(x)), a smooth non-monotonic activation in the Swish family. Reported +2.1 AP50 over Leaky ReLU on YOLOv4/MS-COCO and ~1% top-1 over ReLU on ResNet-50/ImageNet. Read from the abstract only — ar5iv has no rendering for this identifier. | Skimmed |
| NOTE-055 | CoCa: Contrastive Captioners are Image-Text Foundation Models | Trains one encoder-decoder with both a contrastive and a captioning loss by omitting cross-attention in the first half of the decoder — so those layers produce unimodal text embeddings for the contrastive loss, and the cascaded remainder cross-attends for captioning. Both objectives share one computational graph, at minimal extra cost. | Read |
| NOTE-060 | Improving Language Models by Retrieving from Trillions of Tokens | Matches GPT-3 on the Pile with 25× fewer parameters by cross-attending to chunks retrieved from a 2-trillion-token database, using a frozen BERT retriever that never needs training. The gain is constant from 150M to 7B, and the model improves at evaluation time by enlarging the database — capability added after training ends. | Read |
| NOTE-065 | Pay Less Attention with Lightweight and Dynamic Convolutions | Shows a depthwise convolution with softmax-normalized weights, shared across channel groups and with a kernel fixed at 31 or less, matches self-attention on translation — 29.7 BLEU on WMT En-De. Dynamic convolution predicts the kernel from the current token alone, so the mixing weights depend on position rather than on the pair of positions being mixed. | Read |
| NOTE-069 | PaLI: A Jointly-Scaled Multilingual Language-Image Model | Scales language and vision components jointly across 100+ languages and shows a properly scaled model handles many languages while keeping English-only state of the art. Its most useful passage is an ablation honest enough to explain a regression by the language ratio of the data that caused it. | Read |
| NOTE-168 | The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits | Ternary weights trained from scratch, matching FP16 from 3B upward at equal size and tokens. The interesting claim is not the compression ratio but that the matrix multiply becomes integer addition — which is a bet on hardware, and the one place the paper’s argument outruns its measurements. | Read |
| NOTE-170 | VGGT: Visual Geometry Grounded Transformer | One feed-forward pass produces camera parameters, point maps, depth maps and 3D point tracks together, in under a second, beating methods that post-process with geometry optimisation. Removes the global alignment step DUSt3R still needed, and works as a pretrained backbone — which is when a subject stops being a field and becomes a component. | Read |
| NOTE-196 | Scaling Rectified Flow Transformers | A 61-way sweep over diffusion and rectified-flow formulations settles on a straight-line path with logit-normal timestep sampling — rectified flow with uniform timesteps does not win. Adds a resolution-dependent timestep shift, MMDiT’s per-modality weights with joint attention, and QK-RMSNorm for high-resolution stability. 8B, weights released. | Read |
| NOTE-214 | Limitations of multi-layer Transformers | First unconditional lower bound for a decoder-only transformer of more than one layer. Reading it: the bound comes from causal masking rather than from circuit complexity, and it buys an exponential depth-width tradeoff, an encoder/decoder separation, and a provable benefit of chain of thought. | Read |
| NOTE-221 | nGPT | Reading it: the headline is 4–20× and the footnote is 60–80% more time per step, so the honest wall-clock figure is roughly 5.5× at 4k. Still large. The part that generalises beyond this architecture is smaller and more interesting — that a matrix’s norm is currently an accident of the learning rate and weight decay, and does not have to be. | Read |
| NOTE-226 | NoProp | Reading it: the interesting number is in the train column, not the test one. NoProp matches backprop on test while sitting 8-15 points behind on train, which on MNIST and CIFAR costs nothing and would be the whole story anywhere fitting is the binding constraint. The paper does not mention it. | Read |
| NOTE-228 v2 | The Diffusion Duality | Reading it: the theorem is elegant and the ablation says it accounts for under half the likelihood gain. What the paper actually establishes is a regime split — masked diffusion wins on quality at many steps, uniform-state wins below about 32, and the reason is architectural rather than tuned. | Read |
| NOTE-232 | Mixture-of-Transformers | Read to test whether it satisfies SOTA-262’s promotion condition. It does, as that condition was written — second group, per-modality weights against a shared-weight backbone, FLOPs held identical. The component ablation is a bonus the condition did not ask for and the practice needed. | Read |
| NOTE-235 | ParScale | Read as a candidate third scaling axis. The scaling law is fitted honestly and the inference-cost analysis is better instrumented than most; the claim that carries furthest is the asymmetry — code benefits more than general text, on two independent measurements — and it is the claim the paper labels a conjecture. | Read |
| NOTE-242 | Genesys and what survives its own controls | Read from the #180 worklist for the architecture-discovery claim and filed for the code-generation one. Unit-by-unit generation with an execution-based checker takes valid output from 6% to 92% on the same proposals, with a stated mechanism that has nothing to do with architectures. The discovered architectures are a wash against their own seeds, by the paper’s own table. | Read |
| NOTE-244 | SQ hardness for semiautomata, and the regime it needs | Read from the #180 worklist. The theorem is clean and the regime it needs — an alphabet cubic in the number of states — is what a citing reader will drop. Filed for the mental model underneath it, which the record had nothing for: longer random sequences carry less information about a hidden transition structure, not more. | Read |
| NOTE-246 | VQGAN: the loss that made f=16 survivable | Read as the second step of the tokenizer trunk. The architecture is VQ-VAE’s; what is new is a perceptual loss, a patch discriminator, and an adaptive weight that balances them without per-dataset tuning. That loss is the reason a 256×256 image can become 256 tokens and still look like anything, and it is why this line exists at all. | Read |
| NOTE-247 | ViT-VQGAN: where the codebook advice comes from | Read to give SOTA-306 its origin, and it corrected the practice in three places. The advice is not “use small code vectors” but “factorize lookup from embedding”; ℓ₂-normalization is ablated and is the largest single effect in the table; and the dimension-4 row shows 96% codebook usage with near-worst FID, so utilization is necessary and not sufficient. | Read |
| NOTE-258 | mHC-lite: the experiment that was one measurement from settling the Birkhoff dispute | Read from the #180 worklist. It makes the doubly-stochastic constraint exact, which is precisely the condition under which LIT-151’s objection should bite hardest — and it reports no stream statistic, so the dispute stands. The approximation it replaces is real and measured: column sums of the layer-wise product off by up to 220% at 24 layers. | Read |
| NOTE-269 | Heads traded for depth, and the five rows where only those two things move | Read on its merits after being triaged past. The headline is an architecture recommendation and the interesting part is the mechanism: more heads lower the condition number of the attention block, which is the same quantity THEORY-041 says drifts the wrong way when nothing manages it. Half the configurations confound the trade by also halving the MLP; the other half do not, and those are the result. | Read |
| NOTE-271 | One metric that separates transformers from every other architecture, and what it buys | Read for whether the record wants an inductive-bias claim. It does, because the claim comes with a computable scalar rather than an adjective, and the scalar does four jobs — separates architectures, predicts robustness, correlates with flatness, and moves during a grokking plateau. The fourth is the one this record has an existing use for. | Read |
| NOTE-272 | An idealization that checks its own assumptions against ALBERT, and clusters within a few layers | Read a second time, in full, after being declined on a skim. The decline said the t → ∞ frozen-weights idealization put it too far from a trained model to be evidence about one, and that it underwrites no practice. Both halves were wrong: the record’s own schema says a theory need underwrite nothing, and the paper checks its hypotheses on ALBERT, gives the discrete-time analogue, and says clustering arrives within a few layers. | Read |
| NOTE-274 | The trunk under a term the record was using for two different things | Read because four documents in this record lean on “rank collapse” and none of them held its source. Reading it produced a finding that was not the point of the unit: the record carries that term for two distinct phenomena, and the conflation is the field’s rather than ours. | Read |
| NOTE-275 | The trunk states the condition its successors were refuted for dropping | Read because the record filed a three-paper information-bottleneck dispute and the paper all three argue about was absent — a gap this session created. Reading it reassigns some blame: the rebuttal’s central objection is that mutual information needs stochasticity, and the founding paper says so in one sentence before anybody measured anything. | Read |
| NOTE-277 | The algorithm changes with depth, and the one it lands on is Bayes-optimal ridge | Read as the second, independent arrival at the mesa-optimization construction. Its title asks which algorithm, and its answer is that there is no single one: shallow learners sit nearest gradient descent, deeper ones nearest ridge, deepest nearest ordinary least squares, and under label noise the best-fitting ridge parameter is the Bayes-optimal σ²/τ² at every setting. The gradient-descent phase is the small-model phase. | Read |
| NOTE-279 | The setup that made the mechanism question answerable | Read because every later claim about what in-context learning is measures in this setup. A 9.5M-parameter GPT-2 trained from scratch on (x, f(x)) sequences matches ordinary least squares on unseen linear functions and Lasso on sparse ones, in a single forward pass. The training objective is the setup’s power and, later, the ground of the main objection to everything built on it. | Read |
| NOTE-287 | Stage boundaries found without knowing in advance what to look for | Read as the application that makes the LLC matter to this record. Critical points of the LLC curve divide two transformers into five stages each, and the boundaries land on bigrams, n-grams, previous-token heads and the induction circuit — found without a mechanistic hypothesis in hand, which is what LIT-085’s progress measures require. Its in-context regression model acquires in-context learning and then loses it. | Read |
| NOTE-289 | MatFormer | One Transformer with four nested FFN widths, trained one width per step, gives four sizes plus Mix’n’Match sizes nobody trained. At the compute of training the four separately, it matches the largest and beats the smaller ones, which see up to 4× the data through shared weights. Read §1–5 and Appendices B, D.1, E.1 and F. | Read |
| NOTE-293 | Gemma 3n | Per-Layer Embeddings: every token id indexes a table with one 256-d vector per layer, gated into the residual at each layer, about 2.35B parameters held off the accelerator. No measurements are published. Read from the model overview, the developer guide, and the Hugging Face transformers implementation, which is the only place the mechanism is fully specified. | Read |
| NOTE-295 | SIREN | Sine-activated MLPs whose derivatives are well behaved, trainable because weights U(±√(6/n)) keep pre-activations standard normal and outputs arcsine at every depth, with ω₀ = 30 on the first layer. They fit signals and derivatives that ReLU, tanh and positional encoding do not. Read §1–4.2 and supplement §1; the later experiments were skimmed. | Read |
| NOTE-300 | StyleGAN3 | Aliasing in the generator glues fine detail to pixel coordinates. Continuous-signal design, with 2×-upsampled filtered nonlinearities, Fourier input and no noise, gives translation- and rotation-equivariant generators at StyleGAN2’s FID. Read §1–3.2 and Figure 3; the internal representation analysis and the appendices were skimmed. | Read |
| NOTE-301 | StyleGAN2 | Diagnoses two StyleGAN artifacts and fixes each at its cause: weight demodulation instead of instance normalization, and a fixed skip/residual architecture instead of progressive growing. Adds lazy regularization and path-length regularization. Read §1–4.1 and Tables 1–2; §4.2 onward and the appendices were skimmed. | Read |
| NOTE-312 | FFN layers as key-value memories | Keys fire on input patterns, and values project to next-token preferences. The reading is interpretable in upper layers of one small WikiText model, with weak key-value agreement and mostly compositional outputs. | Read |
| NOTE-317 | Where did the gap go? | Retuned message-passing baselines close the reported graph-transformer gap on LRGB’s Peptides datasets, mostly by adding an MLP prediction head. Feature normalization lifts every model on the superpixel datasets. Read in full. | Read |
| NOTE-318 | Pitfalls of GNN evaluation | With one shared training and tuning protocol over 100 splits × 20 seeds, GCN ranks first among four GNNs, and the single Planetoid split reorders models. Read in full. | Read |
| NOTE-319 | GIN | Message passing is bounded by 1-WL, and sum aggregation with an MLP reaches the bound where mean and max do not. The theory predicts the training-fit ordering. Test accuracy separates the aggregators decisively only on featureless graphs. Main text read, proofs skimmed. | Read |
| NOTE-320 | AlphaFold 3 | AlphaFold generalized to all biomolecular complexes, with a diffusion head over atom coordinates. It gains on ligands, nucleic acids and antibodies, and inherits generative failure modes (hallucination, chirality) that it patches with distillation and ranking. Main text read. Extended Data and supplementary methods not read. | Read |
| NOTE-321 | AlphaFold 2 | Near-experimental protein structure accuracy at CASP14, a calibrated confidence head, self-distillation on confident predictions, and a hard dependence on alignment depth below about 30 sequences. Main text and Methods read. The supplementary methods (architecture details, ablation descriptions) were not read. | Read |
| NOTE-334 | Movie Gen | At 5B, flow matching beats v-prediction diffusion on quality by 3.3 annotation σ, but on text alignment by only 1.9σ. Video-native captions beat captions stitched from three frames on alignment by 2.9σ. Every σ in the paper is rater variance, and each ablation arm was trained once. The Llama 3 block beats a DiT block that differs only in norm, activation and bias. The scaling claim rests on image-only runs covering less than a quarter of a decade of compute. | Read |
| NOTE-346 | CogVideoX | An open text-to-video DiT whose design choices are backed by curves rather than numbers. On full 3D against factorized attention, the text says only that factorized FVD is “much higher … in early steps” and that factorized training is “unstable and prone to collapse”. Model size, steps and resolution are unstated. The one measured number is the cost: full attention’s forward pass is 1.08×, 1.67× and 2.30× slower at three sizes. | Read |
| NOTE-355 | Topos and Stacks of Deep Neural Networks | The authors claim that every DNN architecture defines a Grothendieck topos of sheaves over a site built from its graph, that backpropagation is a flow of morphisms in that topos, and that layer invariances (CNN translations, LSTM structure) are stacks, and they conjecture that these stacks explain generalization and support a homological notion of “semantic information”. | Skimmed |
| NOTE-358 | Diffusion Models Beat GANs | An ablated U-Net (ADM) and a classifier-gradient scale that trades recall for precision take diffusion past BigGAN-deep on ImageNet FID. The architecture search is single-run; every guided number is the best of a small scale sweep scored on the metric it reports; and the trade is non-monotone — a scale of 1 makes an unconditional model’s FID worse (26.21 → 33.03). The claim that large scales produce no adversarial examples is asserted in the introduction and checked only by eye and in Inception feature space. | Read |
| NOTE-361 | DeepSeek-V4 | Two MoE models (1.6T/49B and 284B/13B) reach a native 1M-token context by interleaving two compressed attentions — 4x-compressed KV with top-k sparse selection (CSA) and 128x-compressed dense KV (HCA), each with a 128-token sliding-window branch — at 27% of V3.2’s per-token FLOPs and 10% of its KV cache for Pro at 1M. Multi-token prediction is kept “without modification” from V3 as a depth-1 training objective; the report never mentions speculative decoding or a draft model. | Read |
| NOTE-366 | Primer | Squaring the feed-forward ReLU and adding a causal width-3 depthwise convolution after each per-head Q/K/V projection makes decoder-only LMs reach a vanilla baseline’s quality in 1.75–2.3× less compute at 35M, at least 1.8× at 110M and 4.2× at 537M. That last figure is at fixed size, where the factor grows with training length. At 1.9B, against a GELU baseline, it takes ⅓ of the compute. Both changes cost step time and are paid for in the accounting. Squared ReLU beats SwiGLU in exactly one figure, at 110M. Against the SwiGLU + RMSNorm baseline the saving is about 2×, and it vanishes on encoder–decoder masked LM. | Read |
| NOTE-367 | Narang et al., transformer modifications | About fifty transformer modifications reimplemented in one T5/Mesh TensorFlow codebase at 223M parameters, with hyperparameters fixed, on four task families. Most do not beat the vanilla pre-norm, relative-attention baseline. The winners are GLU-variant activations, RMSNorm, untied embeddings, and sparse experts or other variants that cost parameters or time. The Universal Transformer, ReZero, Fixup, block sharing and the Evolved Transformer all lose. The positives are largely same-codebase re-runs, GLU Variants included, so the paper is strong evidence against the losers and weak evidence that the winners transfer. | Read |
| NOTE-368 | Attention-only transformers | A necessity test of the feed-forward layer with three matchings. Deleting it in place costs 0.47 nats. At matched FLOPs the cost is 0.26 nats. At matched parameters, with the budget moved into attention depth, the cost is 0.0055 nats, and it shrinks with tokens. What remains is concentrated on low-context tokens, where recall has to come from the weights. QK-norm is what keeps the attention-only stack trainable. At or below 87M parameters, on a synthetic reasoning corpus. | Read |
| NOTE-370 | EDM2 | Magnitude-preserving layers plus forced weight normalization take the ADM U-Net from FID 8.00 to 2.56 on ImageNet-512 at equal compute, in a cumulative single-model ladder whose steps are all far larger than FID’s noise. Post-hoc EMA reconstructs any averaging length from two stored power-function averages, and what it shows is that the best length depends on architecture, learning rate, guidance weight and the metric — the last by 2% against 14% under guidance. | Read |
| NOTE-376 | Surya: heliophysics foundation model | A 366M-parameter transformer pretrained on full-resolution (4096²) multi-instrument SDO imagery to forecast the next frame, then rollout-tuned, forecasts solar dynamics zero-shot. With LoRA fine-tuning it beats task-specific baselines on four downstream heliophysics tasks. | Skimmed |