signal-structure.
13 of 378 NOTE documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| NOTE-009 | Latent Diffusion Models | Train the diffusion model in the latent space of a pretrained autoencoder rather than in pixels. The autoencoder removes imperceptible detail; the diffusion model then spends its capacity on semantics instead of on high-frequency content nobody sees. | Read |
| NOTE-094 | Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks | For kernel/NTK regression with eigenvalue decay λ_k ∝ k^{-β}: ε(n) ∝ n^{-α}, where α = β / (1 + β) The exponent α is determined entirely by the data spectrum β, not by architecture beyond the NTK. | Read |
| NOTE-199 v2 | How do language models learn facts? | Factual recall is learned in three phases, and the plateau between generic statistics and individual knowledge is the attention recall circuit being built — patching in a trained model’s attention patterns removes the plateau. Imbalanced data shortens the plateau and, the paper argues, slows acquisition after it; a warm-up on a subset of individuals before training on all of them beat the fixed distributions tried, in single-seed runs. | Read |
| NOTE-209 | When Spectral Updates Help | A spectral step beats a Euclidean one on a block when the gradient’s nuclear rank exceeds the stable rank of that block’s incoming activations. Reading it: both sides are measurable, the activation side is provably small in transformers, and the gap grows with dimension. | Read |
| NOTE-233 | TinyStories | Read as the missing trunk under eight tiny-models practices. The result that matters is a separation, not a record: what stops a 125M model speaking coherent English is the breadth of the corpus, not the complexity of language — established by holding the architecture and shrinking the corpus. | Read |
| NOTE-297 | Warm Diffusion | Blur and noise mixed in one forward process, with a two-headed model that denoises to the blurry image and deblurs the residual. A blur-to-noise ratio of 0.5, chosen from image and noise spectra, edges EDM on CIFAR-10 (1.85 against 1.97). High ratios degrade sharply, which is the evidence that noise keeps samples on the manifold. Read in full (8 pages). | Read |
| NOTE-322 v2 | Latent Dirichlet Allocation | The bag-of-words assumption, read through de Finetti, implies a document-level mixture. LDA is that mixture with Dirichlet topic proportions. It generalizes better than one-topic-per-document and per-document-parameter models by held-out perplexity. Main text read in full; the variational derivations in the appendices skimmed. | Read |
| NOTE-326 | word2vec (negative sampling, subsampling, phrases) | The tricks that made word2vec the standard: negative sampling with a unigram^(3/4) noise distribution, subsampling of frequent words, and data-driven phrase tokens, plus additive compositionality. The 3/4 exponent is asserted without numbers. Read in full. | Read |
| NOTE-327 | GloVe | Derives a word-embedding objective from the claim that meaning lives in ratios of co-occurrence probabilities. It fits log co-occurrence counts by weighted least squares and beats default-configured word2vec on analogies. Read in full. | Read |
| NOTE-329 | Levy, Goldberg & Dagan: lessons from word embeddings | The prediction-over-count advantage for word embeddings came from design choices bundled with word2vec, not from the algorithm. Ported to PPMI and SVD and tuned alike across 672 representations, no method wins consistently, SGNS beats GloVe, and context-distribution smoothing is the only universally safe setting. Read in full. | Read |
| NOTE-331 | Levy & Goldberg: SGNS as implicit PMI factorization | SGNS’s objective is optimized at w·c = PMI(w, c) − log k, a weighted factorization of the shifted PMI matrix. Shifted PPMI nearly attains that optimum and SVD over it matches SGNS on similarity, while SGNS keeps an edge on syntactic analogies. Read in full. | Read |
| NOTE-332 | Arora et al.: RAND-WALK and PMI in low dimensions | A random-walk generative model with isotropic word vectors predicts PMI ≈ ⟨v, v’⟩/d in low dimensions, justifies GloVe’s biases and CBOW’s averaging, and explains why analogies survive large fitting error: isotropy denoises the offset. Its assumptions are checked loosely on one corpus. Read in full, with the main-text proofs. | Read |
| NOTE-352 | Language Design as Information Renormalization | The paper models Chomsky’s MERGE as a coarse-graining tensor, so that sentence probabilities form mostly loop-free stochastic tensor networks. It argues from this that tree-like syntax gives polynomially decaying (long-range) correlations. It then concludes that tree-structured networks (it names deep CNNs, as Tree Tensor Networks) fit language, while MPS-like models (RNNs, HMMs) with exponentially decaying correlations do not. | Skimmed |