Lines of practice

45 lines, walked from extends: and corrects: on SOTA documents. Each step explains itself; this page is the order they came in.

Grouped by tags, which every line holds in common — a line about two things is listed under both.

adaptation-and-tuning

From Build a reasoning model in three stages: pretrain on general data, SFT on reasoning traces, then RL with verifiable rewards

• alongside: SOTA-129 — Build a reasoning model in three stages: pretrain on general data, SFT on reasoning traces, then RL with verifiable rewards (Active, unassessed)
• alongside: SOTA-130 — Skip the reasoning SFT stage and run RL with verifiable rewards directly on the base model (Proposed, contested)

From Estimate the RL baseline from a group of samples for the same prompt instead of training a critic

• SOTA-145 — Estimate the RL baseline from a group of samples for the same prompt instead of training a critic (Active, converged)
  • SOTA-146 — Correct the GRPO objective before running it — the published form has three independently identified defects (Proposed, emerging)
• SOTA-154 — Fine-tune with evolution strategies instead of policy-gradient reinforcement learning (Active, contested)
  • SOTA-212 — Post-train by scoring many random weight perturbations in one parallel pass and majority-voting the best of them (Proposed, unreplicated)
  • SOTA-213 — Control evolution-strategies drift with a larger population or an anchor penalty, not by stopping training early (Proposed, emerging)
  • SOTA-283 — To fine-tune a model that is already quantized, bank the part of each update that is smaller than the lattice spacing, and rebuild the accumulator from seeds rather than storing it (Proposed, unreplicated)

From Adapt a pretrained model by training a low-rank update to each weight matrix, not the matrix itself

• SOTA-184 — Adapt a pretrained model by training a low-rank update to each weight matrix, not the matrix itself (Active, universal)
  • SOTA-426 — When you merge and reinitialise a low-rank adapter mid-training, prune the optimizer state and re-warm the learning rate from zero — both, not either (Proposed, unreplicated)
• alongside: SOTA-329 — For short-output tasks, adapt a frozen model with a low-rank intervention on a few positions of the residual stream, not a weight adapter; do not use it for long chain-of-thought (Proposed, unreplicated)

analysis-and-evaluation

From Report generative FID as an error bar over several training seeds, and treat any gap below about 2% of the mean as inconclusive

• SOTA-307 — Report generative FID as an error bar over several training seeds, and treat any gap below about 2% of the mean as inconclusive (Active, unreplicated)
  • SOTA-383 — Fix the sample count before comparing FID values, and settle a close comparison with an unbiased estimator rather than a tighter error bar (Active, unreplicated)

From Use prediction-based word vectors rather than count-based distributional vectors

• SOTA-380 — Use prediction-based word vectors rather than count-based distributional vectors (Rejected, unassessed)
  • SOTA-379 — Before crediting a word-embedding method over count-based ones, give the baselines the same design choices and tune every method alike (Active, unreplicated)

From Before using an attribution map to debug a model or explain what it learned, check that the map changes when the weights are randomized and when the labels are permuted — do not validate it by how it looks

• SOTA-430 — Before using an attribution map to debug a model or explain what it learned, check that the map changes when the weights are randomized and when the labels are permuted — do not validate it by how it looks (Proposed, contested)
  • SOTA-435 — Never rank attribution methods by a single faithfulness or stability score, and report the perturbation distribution when you report infidelity (Active, unreplicated)

attention-techniques

From Prefer GQA to MQA or MHA

• alongside: SOTA-109 — Prefer GQA to MQA or MHA (Active, unassessed)
• alongside: SOTA-147 — Compress the KV cache into one shared latent vector instead of sharing key and value heads (Active, emerging)

From Give linear-attention layers a gated delta rule: a decay gate for erasure plus a delta update for targeted writes

• SOTA-135 — Give linear-attention layers a gated delta rule: a decay gate for erasure plus a delta update for targeted writes (Active, unassessed)
  • SOTA-177 — Decouple the erase address from the write address in a delta-rule recurrence (Proposed, unreplicated)

data-pipeline

From Use tar archives for dataset storage

• SOTA-077 — Use tar archives for dataset storage (Active, unassessed)
  • SOTA-078 — Buffer size should be 2-3x batch size (Active, unassessed)
  • SOTA-079 — Pre-fetch next batch during compute (Active, unassessed)

From Train autoregressive models with fill-in-the-middle by default: it is a data transformation, and it is free

• SOTA-174 — Train autoregressive models with fill-in-the-middle by default: it is a data transformation, and it is free (Active, converged)
  • SOTA-128 — Train fill-in-the-middle data without masking the loss on non-FIM tokens (Active, unassessed)

distributed-optimization

From Stage optimizer states across data parallel ranks (ZeRO-1)

• SOTA-028 — Stage optimizer states across data parallel ranks (ZeRO-1) (Active, unassessed)
  • SOTA-029 — Partition gradients and optimizer states (ZeRO-2) for larger models (Active, unassessed)
    • SOTA-030 — Use ZeRO-3 only when other strategies insufficient (Active, unassessed)
      • SOTA-116 — Use FSDP over DDP when model size exceeds single GPU memory (Active, unassessed)
        • SOTA-117 — Overlap communication with computation using backward prefetch (Active, unassessed)
        • SOTA-118 — Employ mixed precision to reduce memory usage (Active, unassessed)
        • SOTA-119 — Choose sharding factor based on model and GPU memory size (Active, unassessed)

From Overlap communication with backward pass

• alongside: SOTA-047 — Overlap communication with backward pass (Active, unassessed)
• alongside: SOTA-048 — Group small tensors before communication (Active, unassessed)

From Save optimizer state every N epochs (N ~ sqrt(total_epochs))

• SOTA-055 — Save optimizer state every N epochs (N ~ sqrt(total_epochs)) (Rejected, unassessed)
  • SOTA-054 — Derive the checkpoint interval from online profiling and adapt it at runtime against an overhead bound (Active, unassessed)
• alongside: SOTA-189 — Set the checkpoint interval from the job’s measured time-to-failure, which shrinks as the job grows (Proposed, unassessed)

From Train across poorly connected islands: many inner steps per worker, an outer momentum optimizer over the deltas, and a streamed synchronisation

• SOTA-155 — Train across poorly connected islands: many inner steps per worker, an outer momentum optimizer over the deltas, and a streamed synchronisation (Proposed, unreplicated)
  • SOTA-219 — Leave the inner optimizer state unsynchronised in local-update training (Active, emerging)

From Pair any gradient compressor with error feedback, and correct the momentum it is applied under

• SOTA-214 — Pair any gradient compressor with error feedback, and correct the momentum it is applied under (Active, emerging)
  • SOTA-215 — Freeze Adam’s variance term after a warmup before compressing its update, and never compress the update directly (Proposed, unreplicated)
  • SOTA-224 — Ramp the sparsity ratio up over the first few epochs rather than starting at the target (Active, unreplicated)

From Drop the parameter server for gossip once the network is the bottleneck, and measure where that is rather than inheriting a threshold

• SOTA-225 — Drop the parameter server for gossip once the network is the bottleneck, and measure where that is rather than inheriting a threshold (Active, unreplicated)
  • SOTA-223 — Do not keep the gossip topology static: sample a fresh random neighbourhood every round (Proposed, unreplicated)
  • SOTA-226 — If the gossip topology is static, make it a directed exponential graph, not a ring (Active, unreplicated)

generative-modeling

From Train the language model as a masked diffusion model rather than autoregressively

• SOTA-157 — Train the language model as a masked diffusion model rather than autoregressively (Proposed, unreplicated)
  • SOTA-254 — Train the language model as a masked diffusion model when the corpus is fixed and the compute is not (Proposed, unreplicated)

From Parametrize the network so its prediction target has unit variance at every noise level, and sample training noise from a log-normal

• SOTA-188 — Parametrize the network so its prediction target has unit variance at every noise level, and sample training noise from a log-normal (Active, converged)
  • SOTA-195 — Predict v rather than the noise when the model will be evaluated at low signal-to-noise (Active, converged)
  • SOTA-431 — Make every layer of a diffusion denoiser preserve activation magnitude, and force each weight vector back to fixed norm after every step (Proposed, unassessed)

From Fix the noise schedule’s endpoints against the bound and choose its shape to minimise the loss estimator’s variance

• alongside: SOTA-264 — Fix the noise schedule’s endpoints against the bound and choose its shape to minimise the loss estimator’s variance (Proposed, unreplicated)
• alongside: SOTA-412 — When the target is likelihood, learn a per-dimension noise schedule conditioned on a learned latent; a scalar or unconditioned one buys nothing (Proposed, unreplicated)

From Connect data and noise on a straight line, and sample the training timesteps from a logit-normal rather than uniformly

• alongside: SOTA-266 — Connect data and noise on a straight line, and sample the training timesteps from a logit-normal rather than uniformly (Active, emerging)
• alongside: SOTA-422 — Rescale the noise schedule so the last timestep is pure noise, and train with v-prediction so the model can learn there (Proposed, emerging)

From Report generative FID as an error bar over several training seeds, and treat any gap below about 2% of the mean as inconclusive

• SOTA-307 — Report generative FID as an error bar over several training seeds, and treat any gap below about 2% of the mean as inconclusive (Active, unreplicated)
  • SOTA-383 — Fix the sample count before comparing FID values, and settle a close comparison with an unbiased estimator rather than a tighter error bar (Active, unreplicated)

in-context-learning

From ICL permits few-shot task adaptability

• SOTA-038 — ICL permits few-shot task adaptability (Active, unassessed)
  • SOTA-279 — Put worked reasoning steps in the few-shot exemplars when the task needs more than one step, and only once the model is large enough (Active, universal)
    • SOTA-280 — Sample several reasoning paths and take the majority answer rather than decoding one greedily (Active, emerging)
    • SOTA-281 — Try the single step-by-step instruction before writing exemplars; it recovers most of the gain for none of the labour (Active, universal)

inference-optimization

From PagedAttention to accelerate batch inference for LLM sampling

• SOTA-105 — PagedAttention to accelerate batch inference for LLM sampling (Active, unassessed)
  • SOTA-113 — Use continuous batching for inference (Active, unassessed)
    • SOTA-115 — Overlap prefill and decode compute (Active, unassessed)

From Quantize weights after training by compensating each rounding error into the columns not yet quantized

• alongside: SOTA-185 — Quantize weights after training by compensating each rounding error into the columns not yet quantized (Active, converged)
• alongside: SOTA-356 — Protect the salient weight channels by scaling them before rounding, choosing them from the activations rather than the weights (Active, converged)

From Decode with a draft model and an accept-reject rule, which is exactly lossless

• SOTA-227 — Decode with a draft model and an accept-reject rule, which is exactly lossless (Active, converged)
  • SOTA-229 — Scale the draft model’s training data, once nothing constrains it to predict the target’s features (Proposed, unreplicated)

model-architecture

From Use SwiGLU activation for transformers

• alongside: SOTA-034 — Use SwiGLU activation for transformers (Active, contested)
• alongside: SOTA-158 — Bound the activation’s output range when training in low precision (Proposed, emerging)

From At a fixed tiny parameter budget, spend parameters on depth and SSM state width before MLP width

• alongside: SOTA-125 — At a fixed tiny parameter budget, spend parameters on depth and SSM state width before MLP width (Proposed, unreplicated)
• alongside: SOTA-190 — Increase depth before any other dimension when scaling a transformer (Proposed, unreplicated)

From Widen the residual stream into several streams and constrain the mixing between them

• SOTA-169 — Widen the residual stream into several streams and constrain the mixing between them (Proposed, emerging)
  • SOTA-137 — Widen the residual stream into several streams with freely learned mixing (hyper-connections) (Superseded, unassessed)
    • SOTA-136 — Widen the residual stream into several streams mixed by doubly-stochastic matrices (manifold-constrained hyper-connections) (Proposed, contested) — also extends SOTA-169
• alongside: SOTA-133 — Replace fixed residual accumulation with learned attention over preceding layers (Active, emerging)

From Make the feed-forward layers a sparse mixture of experts once the model is large enough to be compute-bound

• SOTA-150 — Make the feed-forward layers a sparse mixture of experts once the model is large enough to be compute-bound (Active, converged)
  • SOTA-148 — Balance mixture-of-experts load with a bias on the routing scores, not an auxiliary loss (Proposed, emerging)
  • SOTA-149 — Build the sparse layers from many small experts plus an always-on shared one, not a few large ones (Active, emerging)
  • SOTA-180 — Keep the routed experts’ weights in a down-projected latent space and spend the saved bandwidth on more of them (Active, emerging)
  • SOTA-209 — Initialize a mixture-of-experts model from a dense checkpoint rather than training it from scratch (Proposed, unassessed)

From Train the language model as a masked diffusion model rather than autoregressively

• SOTA-157 — Train the language model as a masked diffusion model rather than autoregressively (Proposed, unreplicated)
  • SOTA-254 — Train the language model as a masked diffusion model when the corpus is fixed and the compute is not (Proposed, unreplicated)

model-stability

From linear warmup of LR stabilizes early training with large batch size.

• alongside: SOTA-008 — linear warmup of LR stabilizes early training with large batch size. (Active, unassessed)
• alongside: SOTA-009 — warmup to a large early lr, anneal throughout training to small final lr (Active, unassessed)
• alongside: SOTA-032 — Put the layer normalization inside the residual block, before the sublayer (Active, universal)
• alongside: SOTA-100 — Use learning rate warmup proportional to model size (Active, unassessed)

From Initialize the LayerNorm gain to 1

• alongside: SOTA-025 — Initialize the LayerNorm gain to 1 (Active, unassessed)
• alongside: SOTA-051 — Initialize a residual or adapter branch to exactly zero, not merely near zero (Active, unassessed)
• alongside: SOTA-060 — Initialize weights from N(0, 0.02) and scale the residual-output projections by 1/sqrt(2N) (Active, unassessed)

From use gradient clipping

• SOTA-035 — use gradient clipping (Active, unassessed)
  • SOTA-385 — Tighten the gradient-norm clip as the model grows rather than carrying one value across scales (Proposed, unreplicated)

From Monitor exp(loss) for stability

• alongside: SOTA-069 — Monitor exp(loss) for stability (Active, unassessed)
• alongside: SOTA-098 — Monitor the training loss for unexpected spikes, for the whole run (Active, unassessed)

From Track gradient norm ratios between layers

• alongside: SOTA-070 — Track gradient norm ratios between layers (Active, unassessed)
• alongside: SOTA-099 — Track gradient norm statistics to detect training instabilities (Active, unassessed)

From Apply dropout where the model can memorize what it is shown, and not where it cannot

• SOTA-240 — Apply dropout where the model can memorize what it is shown, and not where it cannot (Active, converged)
  • SOTA-427 — In a batch-normalized network, put dropout after the last batch-norm layer, not upstream of one (Proposed, unreplicated)

numerics-and-precision

From Perform forward/backward passes in FP16

• SOTA-016 — Perform forward/backward passes in FP16 (Active, unassessed)
  • SOTA-013 — Use dynamic loss scaling that doubles every 2000 successful steps (Active, unassessed)
  • SOTA-014 — Maintain master weights in FP32 (Active, unassessed)
    • SOTA-015 — Store optimizer states in FP32 (Active, unassessed)

From Quantize weights after training by compensating each rounding error into the columns not yet quantized

• alongside: SOTA-185 — Quantize weights after training by compensating each rounding error into the columns not yet quantized (Active, converged)
• alongside: SOTA-356 — Protect the salient weight channels by scaling them before rounding, choosing them from the activations rather than the weights (Active, converged)

representation-and-encoding

From use RoPE for LLM (1D sequence) positional embeddings

• SOTA-063 — use RoPE for LLM (1D sequence) positional embeddings (Active, unassessed)
  • SOTA-151 — Extend a trained model’s context by rescaling RoPE, not by fine-tuning at the longer length (Active, converged)
• alongside: SOTA-153 — Drop positional encoding from the global-attention layers of a hybrid and let the cheap local layers carry position (Active, emerging)

From Use prediction-based word vectors rather than count-based distributional vectors

• SOTA-380 — Use prediction-based word vectors rather than count-based distributional vectors (Rejected, unassessed)
  • SOTA-379 — Before crediting a word-embedding method over count-based ones, give the baselines the same design choices and tune every method alike (Active, unreplicated)

systems-optimization

From Use operator fusion for small operations

• alongside: SOTA-081 — Use operator fusion for small operations (Active, unassessed)
• alongside: SOTA-088 — Fuse small operations into larger kernels (Active, unassessed)
• alongside: SOTA-114 — Fuse attention operations where possible (Active, unassessed)

From Implement custom kernels for critical ops

• SOTA-083 — Implement custom kernels for critical ops (Active, unassessed)
  • SOTA-085 — Use flash attention for all attention computations when hardware supports it (Active, unassessed)
    • SOTA-106 — Use flash-attention-2 over original flash-attention when available (Active, unassessed)
    • SOTA-161 — Keep the attention output in FP32 during training, because flash attention’s rounding bias compounds instead of cancelling (Active, unreplicated)

From Tiling size should match hardware SRAM size

• SOTA-086 — Tiling size should match hardware SRAM size (Active, unassessed)
  • SOTA-087 — Recompute attention during backward pass instead of storing it (Active, unassessed) — also extends SOTA-249
  • SOTA-107 — Keep sequence lengths multiple of 128 for best performance (Rejected, unassessed)
• SOTA-249 — Recompute activations from a sqrt(n) subset of checkpoints when activation memory is the binding constraint (Active, universal)
• alongside: SOTA-108 — Pad attention masks to block boundaries for better hardware utilization (Rejected, unassessed)

training-optimization

From Perform forward/backward passes in FP16

• SOTA-016 — Perform forward/backward passes in FP16 (Active, unassessed)
  • SOTA-013 — Use dynamic loss scaling that doubles every 2000 successful steps (Active, unassessed)
  • SOTA-014 — Maintain master weights in FP32 (Active, unassessed)
    • SOTA-015 — Store optimizer states in FP32 (Active, unassessed)

From use gradient clipping

• SOTA-035 — use gradient clipping (Active, unassessed)
  • SOTA-385 — Tighten the gradient-norm clip as the model grows rather than carrying one value across scales (Proposed, unreplicated)

From Scale attention weights by 1/sqrt(head_dim)

• SOTA-050 — Scale attention weights by 1/sqrt(head_dim) (Active, unassessed)
  • SOTA-192 — Normalize the queries and keys before the attention dot product (Active, converged)
• SOTA-165 — Precondition the gradient with matrices rather than entrywise scaling (Active, emerging)
  • SOTA-121 — Use Muon with decoupled weight decay and AdamW-matched update RMS in place of AdamW (Active, unassessed)
    • SOTA-131 — When training with Muon at scale, rescale query and key weights whenever attention logits exceed a threshold (QK-Clip) (Active, unassessed)
• alongside: SOTA-168 — Run Adam in Shampoo’s eigenbasis (SOAP) instead of Shampoo itself (Proposed, unreplicated)

From Scale batch size with model size but sub-linearly

• SOTA-062 — Scale batch size with model size but sub-linearly (Superseded, unassessed)
  • SOTA-258 — Scale batch size with the token budget, not with compute or model size (Active, emerging) — also extends SOTA-097
• SOTA-097 — Optimal batch size scales approximately with compute budget - B ∝ C^(1/4) (Superseded, unassessed)
  • SOTA-198 — Measure the gradient noise scale instead of sweeping batch size, and expect it to grow during the run (Active, emerging)

From Use a warmup-stable-decay schedule: hold the learning rate, then decay it sharply over the final 10–20% of tokens

• SOTA-140 — Use a warmup-stable-decay schedule: hold the learning rate, then decay it sharply over the final 10–20% of tokens (Active, contested)
  • SOTA-141 — Decay the learning rate linearly all the way to zero (Proposed, unreplicated)
  • SOTA-142 — Set the peak learning rate by a power law in tokens so it transfers across batch size and training length (Active, unassessed)
  • SOTA-251 — Train at low resolution and raise it only during the decay phase of a warmup-stable-decay schedule (Proposed, unreplicated)

From Parameterize the model with µP and tune hyperparameters on a narrow proxy, transferring them across width

• SOTA-143 — Parameterize the model with µP and tune hyperparameters on a narrow proxy, transferring them across width (Active, unassessed)
  • SOTA-144 — Extend µP’s transfer to depth with CompleteP so one sweep serves deeper models too (Proposed, emerging)
    • SOTA-275 — Pick the depth parameterization from the residual branch: a Transformer branch has more than one transformation, so it needs the stricter rule (Proposed, unassessed)
  • SOTA-159 — Combine µP with unit scaling so the hyperparameters decouple and FP8 needs no loss scaling (Proposed, unreplicated)

From Parametrize the network so its prediction target has unit variance at every noise level, and sample training noise from a log-normal

• SOTA-188 — Parametrize the network so its prediction target has unit variance at every noise level, and sample training noise from a log-normal (Active, converged)
  • SOTA-195 — Predict v rather than the noise when the model will be evaluated at low signal-to-noise (Active, converged)
  • SOTA-431 — Make every layer of a diffusion denoiser preserve activation magnitude, and force each weight vector back to fixed norm after every step (Proposed, unassessed)

From Retune the learning rate, momentum and schedule at every batch size you compare, never by a scaling heuristic

• alongside: SOTA-218 — Retune the learning rate, momentum and schedule at every batch size you compare, never by a scaling heuristic (Active, unreplicated)
• alongside: SOTA-221 — To train past the batch size where your optimizer stalls, change the optimizer’s conditioning rather than the scaling rule (Active, unreplicated)

From Score a density ratio against sampled negatives instead of reconstructing the target

• SOTA-360 — Score a density ratio against sampled negatives instead of reconstructing the target (Active, universal)
  • SOTA-359 — Supervise vision from the caption, and match image to caption rather than predicting its words (Active, converged)
    • SOTA-376 — Score each image-text pair independently with a sigmoid, so the loss needs no global normalization (Active, emerging)
    • SOTA-429 — Drop half the image patches when training a CLIP-style model, spend the saving on more pairs and a larger batch, and unmask only for a short final tune (Proposed, unassessed) — also extends SOTA-372
  • SOTA-363 — Decouple the negative set from the batch with a queue, and keep the encoder that fills it slow (Active, emerging)
• SOTA-372 — Run the encoder on visible tokens only and push the mask tokens into a small decoder you discard (Active, converged)