Lines of work

54 lines, walked from extends: and corrects: on LIT documents. Each step explains itself; this page is the order they came in.

Grouped by tags, which every line holds in common — a line about two things is listed under both.

adaptation-and-tuning

From LoRA: Low-Rank Adaptation of Large Language Models

• alongside: LIT-046 — LoRA: Low-Rank Adaptation of Large Language Models (Active)
• alongside: LIT-549 — ReFT: Representation Finetuning for Language Models (Active)

From Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences

• LIT-505 — Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences (Active)
  • LIT-506 — Interpretable RNA Foundation Model from Unannotated Data for Highly Accurate RNA Structure and Function Predictions (Active)

From Robust fine-tuning of zero-shot models

• alongside: LIT-674 — Robust fine-tuning of zero-shot models (Active)
• alongside: LIT-675 — Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time (Active)

analysis-and-evaluation

From Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers

• alongside: LIT-052 — Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers (Active)
• alongside: LIT-670 — BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (Active)
• alongside: LIT-671 — RoBERTa: A Robustly Optimized BERT Pretraining Approach (Active)
• alongside: LIT-672 — Should You Mask 15% in Masked Language Modeling? (Active)
• alongside: LIT-681 — Cramming: Training a Language Model on a Single GPU in One Day (Active)

From Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models

• LIT-077 — Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models (Active)
  • LIT-471 — Are Emergent Abilities of Large Language Models a Mirage? (Active) — also extends LIT-470
• LIT-470 — Emergent Abilities of Large Language Models (Active)

From Coordination of groups of mobile autonomous agents using nearest neighbor rules

• LIT-285 — Coordination of groups of mobile autonomous agents using nearest neighbor rules (Active)
  • LIT-304 — Consensus Problems in Networks of Agents With Switching Topology and Time-Delays (Active)
  • LIT-326 — A Necessary and Sufficient Condition for Consensus Over Random Networks (Active)

From Evaluating Large Language Models Trained on Code

• alongside: LIT-388 — Evaluating Large Language Models Trained on Code (Active)
• alongside: LIT-650 — Competition-Level Code Generation with AlphaCode (Active)

From The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity

• LIT-466 — The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity (Active)
  • LIT-463 — Comment on The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity (Active)

From TinyStories: How Small Can Language Models Be and Still Speak Coherent English?

• LIT-484 — TinyStories: How Small Can Language Models Be and Still Speak Coherent English? (Active)
  • LIT-485 — Parameterized Synthetic Text Generation with SimpleStories (Active)

From Adaptive Estimators Show Information Compression in Deep Neural Networks

• alongside: LIT-507 — Adaptive Estimators Show Information Compression in Deep Neural Networks (Active)
• alongside: LIT-508 — Opening the Black Box of Deep Neural Networks via Information (Active)
• alongside: LIT-509 — On the Information Bottleneck Theory of Deep Learning (Active)

From On Calibration of Modern Neural Networks

• LIT-616 — On Calibration of Modern Neural Networks (Active)
  • LIT-514 — Rethinking Early Stopping: Refine, Then Calibrate (Active)

From Are Sparse Autoencoders Useful? A Case Study in Sparse Probing

• alongside: LIT-568 — Are Sparse Autoencoders Useful? A Case Study in Sparse Probing (Active)
• alongside: LIT-571 — Scaling and evaluating sparse autoencoders (Active)

From interpreting GPT: the logit lens

• LIT-570 — interpreting GPT: the logit lens (Active)
  • LIT-569 — Eliciting Latent Predictions from Transformers with the Tuned Lens (Active)

From Transformer Feed-Forward Layers Are Key-Value Memories

• LIT-575 — Transformer Feed-Forward Layers Are Key-Value Memories (Active)
  • LIT-578 — Locating and Editing Factual Associations in GPT (Active)
    • LIT-574 — Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models (Active)
    • LIT-576 — Mass-Editing Memory in a Transformer (Active)
• alongside: LIT-577 — Evaluating the Ripple Effects of Knowledge Editing in Language Models (Active)

From Robust fine-tuning of zero-shot models

• alongside: LIT-674 — Robust fine-tuning of zero-shot models (Active)
• alongside: LIT-675 — Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time (Active)

From Striving for Simplicity: The All Convolutional Net

• alongside: LIT-727 — Striving for Simplicity: The All Convolutional Net (Active)
• alongside: LIT-728 — Visualizing and Understanding Convolutional Networks (Active)
• alongside: LIT-732 — Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization (Active)

attention-techniques

From Longformer: The Long-Document Transformer

• alongside: LIT-033 — Longformer: The Long-Document Transformer (Active)
• alongside: LIT-225 — Generating Long Sequences with Sparse Transformers (Active)

From FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

• alongside: LIT-074 — FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (Active)
• alongside: LIT-655 — Self-attention Does Not Need O(n²) Memory (Active)

From Efficient Streaming Language Models with Attention Sinks

• LIT-191 — Efficient Streaming Language Models with Attention Sinks (Active)
  • LIT-748 — SepLLM: Accelerate Large Language Models by Compressing One Segment into One Separator (Active)

From Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

• LIT-723 — Swin Transformer: Hierarchical Vision Transformer using Shifted Windows (Active)
  • LIT-763 — Erwin: A Tree-based Hierarchical Transformer for Large-scale Physical Systems (Active)

biomolecular-modeling

From Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences

• LIT-505 — Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences (Active)
  • LIT-506 — Interpretable RNA Foundation Model from Unannotated Data for Highly Accurate RNA Structure and Function Predictions (Active)

From Highly accurate protein structure prediction with AlphaFold

• LIT-583 — Highly accurate protein structure prediction with AlphaFold (Active)
  • LIT-584 — Accurate structure prediction of biomolecular interactions with AlphaFold 3 (Active)
  • LIT-754 — Efficient AlphaFold2 Training using Parallel Evoformer and Branch Parallelism (Active)

concept-geometry

From The Linear Representation Hypothesis and the Geometry of Large Language Models

• LIT-606 — The Linear Representation Hypothesis and the Geometry of Large Language Models (Active)
  • LIT-460 — The Lattice Representation Hypothesis of Large Language Models (Active)
• alongside: LIT-526 — Concepts Whisper: Spectral Anti-Concentration and the Dual Geometry of Transformer Representations (Active)

From Are Sparse Autoencoders Useful? A Case Study in Sparse Probing

• alongside: LIT-568 — Are Sparse Autoencoders Useful? A Case Study in Sparse Probing (Active)
• alongside: LIT-571 — Scaling and evaluating sparse autoencoders (Active)

data-pipeline

From Scaling Data-Constrained Language Models

• LIT-166 — Scaling Data-Constrained Language Models (Active)
  • LIT-441 — Pre-training under infinite compute (Active)
  • LIT-442 — Diffusion Beats Autoregressive in Data-Constrained Settings (Active)

From TinyStories: How Small Can Language Models Be and Still Speak Coherent English?

• LIT-484 — TinyStories: How Small Can Language Models Be and Still Speak Coherent English? (Active)
  • LIT-485 — Parameterized Synthetic Text Generation with SimpleStories (Active)

From Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillation

• alongside: LIT-743 — Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillation (Active)
• alongside: LIT-755 — Wonderland: Navigating 3D Scenes from a Single Image (Active)

distributed-optimization

From DiLoCo: Distributed Low-Communication Training of Language Models

• LIT-212 — DiLoCo: Distributed Low-Communication Training of Language Models (Active)
  • LIT-214 — Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch (Active)

generative-modeling

From Sequence Level Training with Recurrent Neural Networks

• alongside: LIT-647 — Sequence Level Training with Recurrent Neural Networks (Active)
• alongside: LIT-649 — Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks (Active)

From Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillation

• alongside: LIT-743 — Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillation (Active)
• alongside: LIT-755 — Wonderland: Navigating 3D Scenes from a Single Image (Active)

graphs-and-networks

From Where Did the Gap Go? Reassessing the Long-Range Graph Benchmark

• alongside: LIT-579 — Where Did the Gap Go? Reassessing the Long-Range Graph Benchmark (Active)
• alongside: LIT-580 — How Powerful are Graph Neural Networks? (Active)
• alongside: LIT-581 — Pitfalls of Graph Neural Network Evaluation (Active)
• alongside: LIT-582 — Semi-Supervised Classification with Graph Convolutional Networks (Active)

in-context-learning

From Language Models are Few-Shot Learners

• LIT-035 — Language Models are Few-Shot Learners (Active)
  • LIT-467 — Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (Active)
    • LIT-468 — Self-Consistency Improves Chain of Thought Reasoning in Language Models (Active)
    • LIT-469 — Large Language Models are Zero-Shot Reasoners (Active)

inference-optimization

From GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

• LIT-081 — GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (Active)
  • LIT-567 — MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models (Active)
• alongside: LIT-585 — AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (Active)

From Matryoshka Representation Learning

• LIT-547 — Matryoshka Representation Learning (Active)
  • LIT-546 — MatFormer: Nested Transformer for Elastic Inference (Active)
    • LIT-545 — Gemma 3n model overview (Active)

model-architecture

From Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

• LIT-188 — Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (Active)
  • LIT-187 — GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (Active)
    • LIT-170 — DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models (Active)
    • LIT-189 — Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity (Active)
• LIT-199 — Language Modeling with Gated Convolutional Networks (Active)
  • LIT-030 — GLU Variants Improve Transformer (Active)
    • LIT-200 — PowLU: An Activation Function for Stable Pre-Training of LLMs (Active)
• LIT-666 — Sparse Distributed Memory (Active)
  • LIT-669 — Sparse Distributed Memory and Related Models (Active)
    • LIT-641 — Attention Approximates Sparse Distributed Memory (Active) — also extends LIT-666
• alongside: LIT-008 — Attention Is All You Need (Active)
• alongside: LIT-020 — Pay Less Attention with Lightweight and Dynamic Convolutions (Active)
• alongside: LIT-023 — Root Mean Square Layer Normalization (Active)
• alongside: LIT-047 — ReZero is All You Need: Fast Convergence at Large Depth (Active)
• alongside: LIT-425 — Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Active)
• alongside: LIT-640 — Query-Key Normalization for Transformers (Active)
• alongside: LIT-651 — Transformers without Tears: Improving the Normalization of Self-Attention (Active)
• alongside: LIT-668 — ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (Active)
• alongside: LIT-708 — A Controlled Study of Attention-Only Transformers (Active)
• alongside: LIT-709 — Primer: Searching for Efficient Transformers for Language Modeling (Active)
• alongside: LIT-711 — Do Transformer Modifications Transfer Across Implementations and Applications? (Active)

From Hyper-Connections

• LIT-141 — Hyper-Connections (Superseded)
  • LIT-140 — mHC: Manifold-Constrained Hyper-Connections (Active)
    • LIT-151 — oHC: Orthogonal Hyper-Connections on SO(4) via Quaternions (Active)
    • LIT-181 — Beyond the Birkhoff Polytope: Spectral-Sphere-Constrained Hyper-Connections (Active)
• alongside: LIT-134 — Attention Residuals (Active)
• alongside: LIT-152 — On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability (Active)
• alongside: LIT-513 — mHC-lite: You Don’t Need 20 Sinkhorn-Knopp Iterations (Active)

From An Attention Free Transformer

• LIT-431 — An Attention Free Transformer (Active)
  • LIT-430 — RWKV: Reinventing RNNs for the Transformer Era (Active)
    • LIT-429 — Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence (Active)
      • LIT-173 — RWKV-7 “Goose” with Expressive Dynamic State Evolution (Active)
• alongside: LIT-428 — Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention (Active)

From Where Did the Gap Go? Reassessing the Long-Range Graph Benchmark

• alongside: LIT-579 — Where Did the Gap Go? Reassessing the Long-Range Graph Benchmark (Active)
• alongside: LIT-580 — How Powerful are Graph Neural Networks? (Active)
• alongside: LIT-581 — Pitfalls of Graph Neural Network Evaluation (Active)
• alongside: LIT-582 — Semi-Supervised Classification with Graph Convolutional Networks (Active)

From Highly accurate protein structure prediction with AlphaFold

• LIT-583 — Highly accurate protein structure prediction with AlphaFold (Active)
  • LIT-584 — Accurate structure prediction of biomolecular interactions with AlphaFold 3 (Active)
  • LIT-754 — Efficient AlphaFold2 Training using Parallel Evoformer and Branch Parallelism (Active)

From Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

• LIT-723 — Swin Transformer: Hierarchical Vision Transformer using Shifted Windows (Active)
  • LIT-763 — Erwin: A Tree-based Hierarchical Transformer for Large-scale Physical Systems (Active)

model-stability

From On Layer Normalization in the Transformer Architecture

• LIT-114 — On Layer Normalization in the Transformer Architecture (Active)
  • LIT-639 — ResiDual: Transformer with Dual Residual Connections (Active)

From Improving neural networks by preventing co-adaptation of feature detectors

• LIT-394 — Improving neural networks by preventing co-adaptation of feature detectors (Active)
  • LIT-707 — Analysis of dropout learning regarded as ensemble learning (Active)

From Dropout: A Simple Way to Prevent Neural Networks from Overfitting

• LIT-395 — Dropout: A Simple Way to Prevent Neural Networks from Overfitting (Active)
  • LIT-704 — Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning (Active)
  • LIT-705 — Understanding the Disharmony between Dropout and Batch Normalization by Variance Shift (Active)

From Bootstrap your own latent: A new approach to self-supervised Learning

• LIT-594 — Bootstrap your own latent: A new approach to self-supervised Learning (Active)
  • LIT-593 — Exploring Simple Siamese Representation Learning (Active)

From VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning

• alongside: LIT-595 — VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning (Active)
• alongside: LIT-596 — Barlow Twins: Self-Supervised Learning via Redundancy Reduction (Active)

From Robust fine-tuning of zero-shot models

• alongside: LIT-674 — Robust fine-tuning of zero-shot models (Active)
• alongside: LIT-675 — Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time (Active)

From Net2Net: Accelerating Learning via Knowledge Transfer

• LIT-758 — Net2Net: Accelerating Learning via Knowledge Transfer (Active)
  • LIT-749 — bert2BERT: Towards Reusable Pretrained Language Models (Active)
    • LIT-765 — Scaling Smart: Accelerating Large Language Model Pre-training with Small Model Initialization (Active) — also extends LIT-758

numerics-and-precision

From GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

• LIT-081 — GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (Active)
  • LIT-567 — MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models (Active)
• alongside: LIT-585 — AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (Active)

From LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale

• alongside: LIT-586 — LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (Active)
• alongside: LIT-656 — Intriguing Properties of Quantization at Scale (Active)

representation-and-encoding

From Neural Machine Translation of Rare Words with Subword Units

• alongside: LIT-003 — Neural Machine Translation of Rare Words with Subword Units (Active)
• alongside: LIT-679 — Dynamic Chunking for End-to-End Hierarchical Sequence Modeling (Active)

From Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences

• LIT-505 — Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences (Active)
  • LIT-506 — Interpretable RNA Foundation Model from Unannotated Data for Highly Accurate RNA Structure and Function Predictions (Active)

From Adaptive Estimators Show Information Compression in Deep Neural Networks

• alongside: LIT-507 — Adaptive Estimators Show Information Compression in Deep Neural Networks (Active)
• alongside: LIT-508 — Opening the Black Box of Deep Neural Networks via Information (Active)
• alongside: LIT-509 — On the Information Bottleneck Theory of Deep Learning (Active)

From Momentum Contrast for Unsupervised Visual Representation Learning

• alongside: LIT-590 — Momentum Contrast for Unsupervised Visual Representation Learning (Active)
• alongside: LIT-591 — A Simple Framework for Contrastive Learning of Visual Representations (Active)
• alongside: LIT-744 — Text Embeddings by Weakly-Supervised Contrastive Pre-training (Active)

From Reading Tea Leaves: How Humans Interpret Topic Models

• LIT-597 — Reading Tea Leaves: How Humans Interpret Topic Models (Active)
  • LIT-600 — Automatic Evaluation of Topic Coherence (Active)
• alongside: LIT-592 — Latent Dirichlet Allocation (Active)

From Bootstrap your own latent: A new approach to self-supervised Learning

• LIT-594 — Bootstrap your own latent: A new approach to self-supervised Learning (Active)
  • LIT-593 — Exploring Simple Siamese Representation Learning (Active)

From VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning

• alongside: LIT-595 — VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning (Active)
• alongside: LIT-596 — Barlow Twins: Self-Supervised Learning via Redundancy Reduction (Active)

From DINOv2: Learning Robust Visual Features without Supervision

• alongside: LIT-599 — DINOv2: Learning Robust Visual Features without Supervision (Active)
• alongside: LIT-662 — Vision Transformers Need Registers (Active)
• alongside: LIT-663 — DINOv3 (Active)
• alongside: LIT-664 — Emerging Properties in Self-Supervised Vision Transformers (Active)

From GloVe: Global Vectors for Word Representation

• LIT-602 — GloVe: Global Vectors for Word Representation (Active)
  • LIT-607 — Improving Distributional Similarity with Lessons Learned from Word Embeddings (Active) — also extends LIT-608, LIT-612
• LIT-604 — Efficient Estimation of Word Representations in Vector Space (Active)
  • LIT-603 — Distributed Representations of Words and Phrases and their Compositionality (Active)
• LIT-608 — Don’t count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors (Active)
• LIT-612 — Neural Word Embedding as Implicit Matrix Factorization (Active)
• alongside: LIT-613 — A Latent Variable Model Approach to PMI-based Word Embeddings (Active)

From Efficient Estimation of Word Representations in Vector Space

• LIT-610 — Efficient Estimation of Word Representations in Vector Space (Superseded)
  • LIT-609 — Distributed Representations of Words and Phrases and their Compositionality (Superseded)

signal-structure

From Efficient Estimation of Word Representations in Vector Space

• LIT-610 — Efficient Estimation of Word Representations in Vector Space (Superseded)
  • LIT-609 — Distributed Representations of Words and Phrases and their Compositionality (Superseded)

systems-optimization

From FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

• alongside: LIT-074 — FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (Active)
• alongside: LIT-655 — Self-attention Does Not Need O(n²) Memory (Active)

tiny-models

From TinyStories: How Small Can Language Models Be and Still Speak Coherent English?

• LIT-484 — TinyStories: How Small Can Language Models Be and Still Speak Coherent English? (Active)
  • LIT-485 — Parameterized Synthetic Text Generation with SimpleStories (Active)

training-optimization

From Large Batch Training of Convolutional Networks

• LIT-009 — Large Batch Training of Convolutional Networks (Active)
  • LIT-686 — On the distance between two neural networks and the stability of learning (Active)
• LIT-360 — Neural Tangent Kernel: Convergence and Generalization in Neural Networks (Active)
  • LIT-557 — Tensor Programs II: Neural Tangent Kernel for Any Architecture (Active) — also extends LIT-558
    • LIT-556 — Tensor Programs III: Neural Matrix Laws (Active)
      • LIT-548 — Feature Learning in Infinite-Width Neural Networks (Active)
        • LIT-148 — Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer (Active)
          • LIT-149 — u-µP: The Unit-Scaled Maximal Update Parametrization (Active)
          • LIT-150 — Don’t be lazy: CompleteP enables compute-efficient deep transformers (Active)
            • LIT-462 — Spectral Condition for μP under Width-Depth Scaling (Active) — also extends LIT-437
        • LIT-437 — A Spectral Condition for Feature Learning (Active) — also extends LIT-686
          • LIT-436 — Scalable Optimization in the Modular Norm (Active) — also extends LIT-686
            • LIT-438 — Modular Duality in Deep Learning (Active)
• LIT-558 — Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes (Active)
• alongside: LIT-001 — Adam: A Method for Stochastic Optimization (Active)

From Scaling Laws for Neural Language Models

• LIT-028 — Scaling Laws for Neural Language Models (Active)
  • LIT-443 — Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training (Active)
  • LIT-445 — How Does Critical Batch Size Scale in Pre-training? (Active)
  • LIT-688 — Reconciling Kaplan and Chinchilla Scaling Laws (Active) — also extends LIT-068
  • LIT-690 — Resolving Discrepancies in Compute-Optimal Scaling of Language Models (Active) — also extends LIT-068
• LIT-068 — Training Compute-Optimal Large Language Models (Active)
• alongside: LIT-617 — Scaling Language Models: Methods, Analysis & Insights from Training Gopher (Active)
• alongside: LIT-685 — Sub-Scaling Laws: On the Role of Data Density and Training Strategies in LLMs (Active)

From Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations

• LIT-145 — Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations (Active)
  • LIT-684 — Model Merging in Pre-training of Large Language Models (Active)
• LIT-213 — The Road Less Scheduled (Active)
  • LIT-682 — ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Models (Active)
• alongside: LIT-147 — Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMs (Active)

From Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability

• LIT-461 — Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability (Active)
  • LIT-453 — Understanding Optimization in Deep Learning with Central Flows (Active)
  • LIT-700 — Non-Euclidean Gradient Descent Operates at the Edge of Stability (Active)

From On Calibration of Modern Neural Networks

• LIT-616 — On Calibration of Modern Neural Networks (Active)
  • LIT-514 — Rethinking Early Stopping: Refine, Then Calibrate (Active)

From Efficient Estimation of Word Representations in Vector Space

• LIT-610 — Efficient Estimation of Word Representations in Vector Space (Superseded)
  • LIT-609 — Distributed Representations of Words and Phrases and their Compositionality (Superseded)

From Sequence Level Training with Recurrent Neural Networks

• alongside: LIT-647 — Sequence Level Training with Recurrent Neural Networks (Active)
• alongside: LIT-649 — Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks (Active)

From Net2Net: Accelerating Learning via Knowledge Transfer

• LIT-758 — Net2Net: Accelerating Learning via Knowledge Transfer (Active)
  • LIT-749 — bert2BERT: Towards Reusable Pretrained Language Models (Active)
    • LIT-765 — Scaling Smart: Accelerating Large Language Model Pre-training with Small Model Initialization (Active) — also extends LIT-758

vision-and-graphics

From NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis

• LIT-435 — NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (Active)
  • LIT-086 — K-Planes: Explicit Radiance Fields in Space, Time, and Appearance (Active)
  • LIT-108 — 3D Gaussian Splatting for Real-Time Radiance Field Rendering (Active)
    • LIT-511 — Image-GS: Content-Adaptive Image Representation via 2D Gaussians (Active)
  • LIT-550 — Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains (Active)
• alongside: LIT-551 — Implicit Neural Representations with Periodic Activation Functions (Active)

From Segment Anything

• alongside: LIT-096 — Segment Anything (Active)
• alongside: LIT-488 — gen2seg: Generative Models Enable Generalizable Instance Segmentation (Active)

From An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

• LIT-587 — An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (Active)
  • LIT-628 — Better plain ViT baselines for ImageNet-1k (Active)
  • LIT-657 — Patch n’ Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution (Active)
• alongside: LIT-747 — Perceiver: General Perception with Iterative Attention (Active)

From Bootstrap your own latent: A new approach to self-supervised Learning

• LIT-594 — Bootstrap your own latent: A new approach to self-supervised Learning (Active)
  • LIT-593 — Exploring Simple Siamese Representation Learning (Active)

From VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning

• alongside: LIT-595 — VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning (Active)
• alongside: LIT-596 — Barlow Twins: Self-Supervised Learning via Redundancy Reduction (Active)

From DINOv2: Learning Robust Visual Features without Supervision

• alongside: LIT-599 — DINOv2: Learning Robust Visual Features without Supervision (Active)
• alongside: LIT-662 — Vision Transformers Need Registers (Active)
• alongside: LIT-663 — DINOv3 (Active)
• alongside: LIT-664 — Emerging Properties in Self-Supervised Vision Transformers (Active)

From Striving for Simplicity: The All Convolutional Net

• alongside: LIT-727 — Striving for Simplicity: The All Convolutional Net (Active)
• alongside: LIT-728 — Visualizing and Understanding Convolutional Networks (Active)
• alongside: LIT-732 — Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization (Active)

From Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillation

• alongside: LIT-743 — Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillation (Active)
• alongside: LIT-755 — Wonderland: Navigating 3D Scenes from a Single Image (Active)

Sharing no tags

From Denoising Diffusion Implicit Models

• LIT-038 — Denoising Diffusion Implicit Models (Active)
• LIT-062 — High-Resolution Image Synthesis with Latent Diffusion Models (Active)
  • LIT-566 — SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis (Active)
  • LIT-621 — Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models (Active)
    • LIT-625 — Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets (Active) — also extends LIT-062
  • LIT-633 — AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning (Active)
  • LIT-635 — Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioning (Active) — also extends LIT-689
• LIT-067 — Progressive Distillation for Fast Sampling of Diffusion Models (Active)
• LIT-070 — Hierarchical Text-Conditional Image Generation with CLIP Latents (Active)
  • LIT-632 — Make-A-Video: Text-to-Video Generation without Text-Video Data (Active)
• LIT-075 — Elucidating the Design Space of Diffusion-Based Generative Models (Active)
  • LIT-714 — Analyzing and Improving the Training Dynamics of Diffusion Models (Active)
• LIT-093 — Consistency Models (Active)
  • LIT-786 — Improved Techniques for Training Consistency Models (Active)
  • LIT-790 — Simplifying, Stabilizing and Scaling Continuous-Time Consistency Models (Active)
• LIT-169 — Direct Preference Optimization: Your Language Model is Secretly a Reward Model (Active)
  • LIT-774 — Diffusion Model Alignment Using Direct Preference Optimization (Active)
    • LIT-624 — Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Model (Active) — also extends LIT-448
    • LIT-773 — Improving Video Generation with Human Feedback (Active)
• LIT-216 — Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture (Active)
  • LIT-215 — V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning (Active)
• LIT-235 — Matching Accuracy, Different Geometry: Evolution Strategies vs GRPO in LLM Post-Training (Active)
• LIT-236 — The Blessing of Dimensionality in LLM Fine-tuning: A Variance-Curvature Perspective (Active)
  • LIT-233 — Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights (Active)
• LIT-241 — Proximal Policy Optimization Algorithms (Active)
  • LIT-127 — DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (Active)
    • LIT-779 — Flow-GRPO: Training Flow Matching Models via Online RL (Active)
  • LIT-776 — Training Diffusion Models with Reinforcement Learning (Active)
• LIT-242 — Evolution Strategies as a Scalable Alternative to Reinforcement Learning (Active)
  • LIT-211 — Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning (Active)
    • LIT-229 — Evolution Strategies at the Hyperscale (Active)
      • LIT-232 — EGGROLL, Unrolled: Understanding and Improving Low-Rank Evolution Strategies at Scale (Active)
    • LIT-231 — Hyper-ES: Effective Evolution Strategies for LLM Reasoning via Descent Direction Merging (Active)
    • LIT-234 — Beyond the Best Guess: Improving LLM Solution Coverage with Evolution Strategies (Active)
    • LIT-237 — Evolutionary Strategies lead to Catastrophic Forgetting in LLMs (Active)
      • LIT-238 — Overcoming Forgetting in LLM Fine-Tuning with Evolution Strategies (Active) — also extends LIT-235
    • LIT-240 — ESSAM: A Novel Competitive Evolution Strategies Approach to Reinforcement Learning for Memory Efficient LLMs Fine-Tuning (Active)
    • LIT-473 — Quantized Evolution Strategies: High-precision Fine-tuning of Quantized LLMs at Low-precision Cost (Active)
• LIT-434 — Deep reinforcement learning from human preferences (Active)
  • LIT-734 — Fine-Tuning Language Models from Human Preferences (Active)
    • LIT-433 — Learning to summarize from human feedback (Active)
      • LIT-377 — Training language models to follow instructions with human feedback (Active)
        • LIT-778 — DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models (Active)
        • LIT-788 — ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation (Active)
        • LIT-795 — Scaling Laws for Reward Model Overoptimization (Active)
        • LIT-797 — Aligning Text-to-Image Models using Human Feedback (Active)
• LIT-439 — Deep Unsupervised Learning using Nonequilibrium Thermodynamics (Active)
  • LIT-036 — Denoising Diffusion Probabilistic Models (Active)
    • LIT-446 — Variational Diffusion Models (Active)
      • LIT-677 — Diffusion Models With Learned Adaptive Noise (Active)
      • LIT-692 — Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation (Active) — also extends LIT-660
        • LIT-449 — Scaling Rectified Flow Transformers for High-Resolution Image Synthesis (Active) — also extends LIT-447, LIT-636
          • LIT-573 — FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space (Active)
            • LIT-572 — FLUX.2: Analyzing and Enhancing the Latent Space of FLUX – Representation Comparison (Active)
          • LIT-618 — LTX-Video: Realtime Video Latent Diffusion (Active)
          • LIT-619 — Wan: Open and Advanced Large-Scale Video Generative Models (Active) — also extends LIT-448, LIT-623
            • LIT-629 — Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion (Active) — also extends LIT-631
          • LIT-620 — HunyuanVideo: A Systematic Framework For Large Video Generative Models (Active) — also extends LIT-623
          • LIT-626 — Movie Gen: A Cast of Media Foundation Models (Active) — also extends LIT-062, LIT-630, LIT-635
          • LIT-634 — Open-Sora 2.0: Training a Commercial-Level Video Generation Model in $200k (Active)
        • LIT-678 — Diffusion Meets Flow Matching: Two Sides of the Same Coin (Active) — also extends LIT-038, LIT-630
    • LIT-627 — Video Diffusion Models (Active)
      • LIT-637 — Imagen Video: High Definition Video Generation with Diffusion Models (Active) — also extends LIT-067
    • LIT-643 — One-step Diffusion with Distribution Matching Distillation (Active)
      • LIT-646 — Improved Distribution Matching Distillation for Fast Image Synthesis (Active)
        • LIT-631 — From Slow Bidirectional to Fast Autoregressive Video Diffusion Models (Active) — also extends LIT-643
      • LIT-770 — DMAD: Distribution Matching as Adversarial Distillation for Fast Visual Generation (Active)
      • LIT-783 — Large Scale Diffusion Distillation via Score-Regularized Continuous-Time Consistency (Active) — also extends LIT-790
    • LIT-699 — Diffusion Models Beat GANs on Image Synthesis (Active) — also extends LIT-439
      • LIT-693 — Classifier-Free Diffusion Guidance (Active)
    • LIT-710 — Efficient Diffusion Training via Min-SNR Weighting Strategy (Active)
• LIT-448 — Scalable Diffusion Models with Transformers (Active)
  • LIT-622 — CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer (Active) — also extends LIT-623
  • LIT-652 — Video generation models as world simulators (Active) — also extends LIT-648
  • LIT-658 — FiT: Flexible Vision Transformer for Diffusion Model (Active)
• LIT-499 — Neural Discrete Representation Learning (Active)
  • LIT-496 — Taming Transformers for High-Resolution Image Synthesis (Active)
    • LIT-494 — GaussianToken: An Effective Image Tokenizer with 2D Gaussian Splatting (Active)
    • LIT-498 — MaskGIT: Masked Generative Image Transformer (Active)
    • LIT-500 — Vector-quantized Image Modeling with Improved VQGAN (Active)
      • LIT-497 — Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation (Active) — also extends LIT-496
• LIT-553 — Cold Diffusion: Inverting Arbitrary Image Transforms Without Noise (Active)
  • LIT-555 — Warm Diffusion: Recipe for Blur-Noise Mixture Diffusion Models (Active)
• LIT-561 — A Style-Based Generator Architecture for Generative Adversarial Networks (Active)
  • LIT-560 — Analyzing and Improving the Image Quality of StyleGAN (Active)
    • LIT-559 — Alias-Free Generative Adversarial Networks (Active)
• LIT-588 — Learning Transferable Visual Models From Natural Language Supervision (Active)
  • LIT-706 — Scaling Language-Image Pre-training via Masking (Active) — also extends LIT-601
  • LIT-782 — Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation (Active)
• LIT-601 — Masked Autoencoders Are Scalable Vision Learners (Active)
• LIT-605 — Sigmoid Loss for Language Image Pre-Training (Active)
  • LIT-745 — SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features (Active)
• LIT-614 — Improved Techniques for Training GANs (Active)
  • LIT-611 — GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium (Active)
    • LIT-501 — The FID Lottery: Quantifying Hidden Randomness in Generative-Model Evaluation (Active)
    • LIT-563 — The Role of ImageNet Classes in Fréchet Inception Distance (Active)
    • LIT-615 — Demystifying MMD GANs (Active)
• LIT-623 — Language Model Beats Diffusion — Tokenizer is Key to Visual Generation (Active)
• LIT-630 — Flow Matching for Generative Modeling (Active)
  • LIT-784 — Mean Flows for One-step Generative Modeling (Active)
  • LIT-789 — Multisample Flow Matching: Straightening Flows with Minibatch Couplings (Active)
  • LIT-792 — Discrete Flow Matching (Active)
  • LIT-798 — Improving and generalizing flow-based generative models with minibatch optimal transport (Active)
• LIT-636 — Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow (Active)
  • LIT-787 — One Step Diffusion via Shortcut Models (Active)
  • LIT-791 — Improving the Training of Rectified Flows (Active)
  • LIT-796 — InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation (Active)
• LIT-644 — Building Normalizing Flows with Stochastic Interpolants (Active)
  • LIT-645 — Stochastic Interpolants: A Unifying Framework for Flows and Diffusions (Active)
    • LIT-447 — SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers (Active) — also extends LIT-448
    • LIT-775 — Inductive Moment Matching (Active)
    • LIT-793 — Flow map matching with stochastic interpolants: A mathematical framework for consistency models (Active)
• LIT-648 — Improving Image Generation with Better Captions (Active)
• LIT-660 — simple diffusion: End-to-end diffusion for high resolution images (Active)
• LIT-689 — Common Diffusion Noise Schedules and Sample Steps are Flawed (Active)
  • LIT-687 — Fine-Tuning Image-Conditional Diffusion Models is Easier than You Think (Active) — also extends LIT-691
• LIT-691 — Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation (Active)
• LIT-772 — Neural Ordinary Differential Equations (Active)
  • LIT-780 — FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models (Active)
• LIT-781 — Density estimation using Real NVP (Active)
  • LIT-771 — Glow: Generative Flow with Invertible 1x1 Convolutions (Active)
• LIT-794 — Normalizing Flows are Capable Generative Models (Active)
  • LIT-785 — STARFlow: Scaling Latent Normalizing Flows for High-resolution Image Synthesis (Active)
• alongside: LIT-073 — Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding (Active)
• alongside: LIT-076 — DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps (Active)
• alongside: LIT-230 — Understanding Evolution Strategies for LLM Reasoning: Broader Reasoning Coverage than GRPO (Active)
• alongside: LIT-239 — ESSA: Evolutionary Strategies for Scalable Alignment (Active)
• alongside: LIT-432 — Finetuned Language Models Are Zero-Shot Learners (Active)
• alongside: LIT-479 — The Diffusion Duality (Active)
• alongside: LIT-554 — Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion (Active)
• alongside: LIT-661 — Photorealistic Video Generation with Diffusion Models (Active)
• alongside: LIT-676 — DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models (Active)
• alongside: LIT-701 — Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution (Active)
• alongside: LIT-702 — Simple and Effective Masked Diffusion Language Models (Active)
• alongside: LIT-722 — Score-Based Generative Modeling through Stochastic Differential Equations (Active)
• alongside: LIT-751 — “Principal Components” Enable A New Language of Images (Active)

From RoFormer: Enhanced Transformer with Rotary Position Embedding

• LIT-045 — RoFormer: Enhanced Transformer with Rotary Position Embedding (Active)
  • LIT-192 — Extending Context Window of Large Language Models via Positional Interpolation (Active)
    • LIT-193 — YaRN: Efficient Context Window Extension of Large Language Models (Active)
  • LIT-210 — Round and Round We Go! What makes Rotary Positional Encodings useful? (Active)
  • LIT-638 — A Length-Extrapolatable Transformer (Active)
• LIT-130 — Olmo 3 (Active)
• LIT-161 — Mamba: Linear-Time Sequence Modeling with Selective State Spaces (Active)
  • LIT-162 — Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality (Active)
    • LIT-137 — Gated Delta Networks: Improving Mamba2 with Delta Rule (Active) — also extends LIT-195
      • LIT-133 — Kimi Linear: An Expressive, Efficient Attention Architecture (Active)
        • LIT-802 — Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention (Active) — also extends LIT-137
      • LIT-799 — Olmo Hybrid: From Theory to Practice and Back (Active) — also extends LIT-130
    • LIT-165 — Mamba-3: Improved Sequence Modeling using State Space Principles (Active)
• LIT-195 — Parallelizing Linear Transformers with the Delta Rule over Sequence Length (Active)
• LIT-207 — The Impact of Positional Encoding on Length Generalization in Transformers (Active)
  • LIT-208 — Rope to Nope and Back Again: A New Hybrid Attention Strategy (Active)
  • LIT-209 — SWAN-GPT: An Efficient and Scalable Approach for Long-Context Language Modeling (Active)
• alongside: LIT-048 — Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation (Active)
• alongside: LIT-120 — Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance (Active)
• alongside: LIT-179 — The Llama 3 Herd of Models (Active)
• alongside: LIT-182 — Qwen3 Technical Report (Active)
• alongside: LIT-183 — Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning (Active)
• alongside: LIT-665 — Transformer Language Models without Positional Encodings Still Learn Positional Information (Active)
• alongside: LIT-801 — Cracks in the Foundation: Seemingly Minor Architectural Choices Impact Long Context Extension (Active)