anthology-of-the-sota

numerics-and-precision

Graph View

53 items with this tag.

Oct 06, 2026
LIT-800: Why Gated DeltaNet Survives 4-Bit Quantization: NVFP4 W4A4 for the Recurrent Half of a Hybrid 27B LLM
Oct 06, 2026
THEORY-128: A delta-rule recurrence does not accumulate small per-step perturbations, because each write overwrites the state along the current key, so injected error plateaus and an impulse fades far faster than the forget gate implies
Oct 03, 2026
LIT-783: Large Scale Diffusion Distillation via Score-Regularized Continuous-Time Consistency
Oct 03, 2026
LIT-790: Simplifying, Stabilizing and Scaling Continuous-Time Consistency Models
Oct 01, 2026
LIT-750: ZipNN: Lossless Compression for AI Models
Oct 01, 2026
SOTA-437: Store and move model checkpoints losslessly compressed: code the float exponent as its own stream with an entropy coder alone, and store successive checkpoints as deltas against a periodic full base
Sep 25, 2026
LIT-676: DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models
Sep 25, 2026
THEORY-104: A large guidance scale amplifies the model's derivatives, which narrows a high-order solver's convergence radius
Sep 24, 2026
LIT-656: Intriguing Properties of Quantization at Scale
Sep 24, 2026
SOTA-399: Choose pre-training weight decay, dropout, clipping and precision for the quantization you intend to ship
- numerics-and-precision
- training-optimization
Sep 24, 2026
THEORY-100: Activation outliers are a product of pre-training optimization choices, not an emergent property of scale
- numerics-and-precision
- capability-thresholds
Sep 23, 2026
LIT-567: MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models
Sep 23, 2026
LIT-585: AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
- numerics-and-precision
- inference-optimization
Sep 23, 2026
LIT-586: LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
Sep 23, 2026
SOTA-340: Expect weight-only 4-bit quantization to speed up batched serving only while the batch keeps the matmul memory-bound, and use a kernel built to stay there
Sep 23, 2026
SOTA-355: Isolate the emergent outlier feature dimensions into 16-bit and quantize the rest to int8
- numerics-and-precision
- inference-optimization
Sep 23, 2026
SOTA-356: Protect the salient weight channels by scaling them before rounding, choosing them from the activations rather than the weights
- numerics-and-precision
- inference-optimization
Sep 21, 2026
LIT-473: Quantized Evolution Strategies: High-precision Fine-tuning of Quantized LLMs at Low-precision Cost
Sep 21, 2026
LIT-512: SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models
Sep 21, 2026
SOTA-283: To fine-tune a model that is already quantized, bank the part of each update that is smaller than the lattice spacing, and rebuild the accumulator from seeds rather than storing it
Sep 21, 2026
SOTA-314: Absorb quantization outliers into a high-precision low-rank branch taken from the weights, and fuse its kernels into the low-bit ones
Sep 21, 2026
THEORY-042: An update smaller than the lattice spacing is not merely rounded away — it is cancelled exactly, and carrying the remainder makes the discrete path shadow the continuous one within half a grid cell
- numerics-and-precision
- training-optimization
Sep 21, 2026
THEORY-059: A low-rank branch corrects quantization because weight spectra are steep and quantization-error spectra are flat
- numerics-and-precision
- inference-optimization
Sep 17, 2026
LIT-378: QLoRA: Efficient Finetuning of Quantized LLMs
Sep 17, 2026
LIT-380: The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
Sep 17, 2026
LIT-414: Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing
- numerics-and-precision
- attention-techniques
Sep 17, 2026
SOTA-230: Quantize the frozen base to 4-bit and keep the adapters in 16-bit
Sep 17, 2026
SOTA-234: Train in the target low-bit format from scratch rather than quantizing a finished model
- numerics-and-precision
- systems-optimization
Sep 17, 2026
SOTA-248: Stretch and clip the softmax so an attention head can output exact zeros
- attention-techniques
- numerics-and-precision
Sep 17, 2026
THEORY-019: A softmax head with nothing to attend to must place its mass somewhere, so models learn a positional sink
- attention-techniques
- numerics-and-precision
Sep 09, 2026
LIT-221: Train With Mixed Precision — NVIDIA deep learning performance guide
- numerics-and-precision
- training-optimization
Sep 08, 2026
SOTA-158: Bound the activation's output range when training in low precision
- model-architecture
- numerics-and-precision
Sep 08, 2026
SOTA-159: Combine µP with unit scaling so the hyperparameters decouple and FP8 needs no loss scaling
- training-optimization
- numerics-and-precision
Sep 08, 2026
SOTA-160: Treat the pretraining token budget and the post-training quantization plan as one decision, not two
- numerics-and-precision
- training-optimization
Sep 08, 2026
SOTA-161: Keep the attention output in FP32 during training, because flash attention's rounding bias compounds instead of cancelling
Sep 08, 2026
SOTA-163: Quantize with block-scaled microscaling formats rather than one scale per tensor
- numerics-and-precision
- inference-optimization
Sep 08, 2026
SOTA-185: Quantize weights after training by compensating each rounding error into the columns not yet quantized
- numerics-and-precision
- inference-optimization
Sep 07, 2026
LIT-197: Microscaling Data Formats for Deep Learning
- numerics-and-precision
- systems-optimization
Sep 07, 2026
LIT-198: Why Low-Precision Transformer Training Fails: An Analysis on Flash Attention
- numerics-and-precision
- model-stability
Sep 07, 2026
LIT-200: PowLU: An Activation Function for Stable Pre-Training of LLMs
Sep 05, 2026
LIT-139: DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
Sep 05, 2026
LIT-149: u-µP: The Unit-Scaled Maximal Update Parametrization
- training-optimization
- numerics-and-precision
Sep 05, 2026
LIT-186: Scaling Laws for Precision
- numerics-and-precision
- training-optimization
Sep 05, 2026
SOTA-134: Gate each attention head's output with a sigmoid after the scaled dot-product
- attention-techniques
- numerics-and-precision
Aug 24, 2026
LIT-011: Mixed Precision Training
- numerics-and-precision
- training-optimization
Aug 24, 2026
LIT-081: GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
- numerics-and-precision
- inference-optimization
Aug 24, 2026
LIT-102: ZeRO++: Extremely Efficient Collective Communication for Giant Model Training
- distributed-optimization
- numerics-and-precision
Aug 24, 2026
SOTA-013: Use dynamic loss scaling that doubles every 2000 successful steps
- numerics-and-precision
- training-optimization
Aug 24, 2026
SOTA-014: Maintain master weights in FP32
- numerics-and-precision
- training-optimization
Aug 24, 2026
SOTA-015: Store optimizer states in FP32
- numerics-and-precision
- training-optimization
Aug 24, 2026
SOTA-016: Perform forward/backward passes in FP16
- numerics-and-precision
- training-optimization
Aug 24, 2026
SOTA-043: Use mixed precision during data loading
- data-pipeline
- numerics-and-precision
Aug 24, 2026
SOTA-118: Employ mixed precision to reduce memory usage
- numerics-and-precision
- distributed-optimization

Created with Quartz v5.0.0 © 2026

Repository
Quartz