anthology-of-the-sota

representation-and-encoding

Graph View

161 items with this tag.

Oct 03, 2026
LIT-785: STARFlow: Scaling Latent Normalizing Flows for High-resolution Image Synthesis
Oct 01, 2026
LIT-744: Text Embeddings by Weakly-Supervised Contrastive Pre-training
Oct 01, 2026
LIT-745: SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features
Oct 01, 2026
LIT-747: Perceiver: General Perception with Iterative Attention
Oct 01, 2026
LIT-751: "Principal Components" Enable A New Language of Images
Oct 01, 2026
LIT-755: Wonderland: Navigating 3D Scenes from a Single Image
Oct 01, 2026
LIT-757: Promptagator: Few-shot Dense Retrieval From 8 Examples
Sep 26, 2026
LIT-723: Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Sep 26, 2026
LIT-724: Investigating Sanity Checks for Saliency Maps with Image and Text Classification
- analysis-and-evaluation
- representation-and-encoding
Sep 26, 2026
LIT-725: On the (In)fidelity and Sensitivity for Explanations
- analysis-and-evaluation
- representation-and-encoding
Sep 26, 2026
LIT-729: When Explanations Lie: Why Many Modified BP Attributions Fail
Sep 26, 2026
LIT-730: A Theoretical Explanation for Perplexing Behaviors of Backpropagation-based Visualizations
Sep 26, 2026
SOTA-434: In a vision backbone, carry position with a relative bias inside the attention window and do not add an absolute position embedding on top
Sep 26, 2026
THEORY-113: A gradient attribution survives weight randomization because of its input multiplier, not because the attribution is insensitive to the weights
- analysis-and-evaluation
- representation-and-encoding
Sep 26, 2026
THEORY-114: A modified-backpropagation rule that keeps only non-negative relevance produces a matrix chain that converges to rank 1, so the later layers can only flip the map's sign
- analysis-and-evaluation
- representation-and-encoding
Sep 26, 2026
THEORY-115: Guided backpropagation and DeconvNet do partial image recovery, not attribution: the backward ReLU plus a CNN's local connections reconstruct the input regardless of the class or the weights
- analysis-and-evaluation
- representation-and-encoding
Sep 25, 2026
LIT-665: Transformer Language Models without Positional Encodings Still Learn Positional Information
Sep 25, 2026
LIT-668: ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Sep 25, 2026
LIT-670: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Sep 25, 2026
LIT-671: RoBERTa: A Robustly Optimized BERT Pretraining Approach
Sep 25, 2026
LIT-672: Should You Mask 15% in Masked Language Modeling?
Sep 25, 2026
LIT-679: Dynamic Chunking for End-to-End Hierarchical Sequence Modeling
Sep 25, 2026
LIT-711: Do Transformer Modifications Transfer Across Implementations and Applications?
Sep 25, 2026
SOTA-405: Scale the masked-language-modelling rate with model size rather than holding it at 15%
- representation-and-encoding
- training-optimization
Sep 25, 2026
SOTA-406: Replace every masked token with [MASK]; drop BERT's 80-10-10 substitution rule
- representation-and-encoding
- training-optimization
Sep 25, 2026
SOTA-418: To drop the tokenizer without losing to BPE, use a two-stage hierarchy with learned chunk boundaries and SSM encoder and decoder layers, not an isotropic byte model
- representation-and-encoding
- model-architecture
Sep 25, 2026
THEORY-102: A causal mask implicitly encodes absolute position, because a token can count how many predecessors it may attend to
- representation-and-encoding
- model-architecture
Sep 24, 2026
LIT-618: LTX-Video: Realtime Video Latent Diffusion
Sep 24, 2026
LIT-619: Wan: Open and Advanced Large-Scale Video Generative Models
Sep 24, 2026
LIT-620: HunyuanVideo: A Systematic Framework For Large Video Generative Models
Sep 24, 2026
LIT-622: CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer
Sep 24, 2026
LIT-623: Language Model Beats Diffusion — Tokenizer is Key to Visual Generation
Sep 24, 2026
LIT-624: Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Model
Sep 24, 2026
LIT-626: Movie Gen: A Cast of Media Foundation Models
Sep 24, 2026
LIT-634: Open-Sora 2.0: Training a Commercial-Level Video Generation Model in $200k
Sep 24, 2026
LIT-638: A Length-Extrapolatable Transformer
Sep 24, 2026
LIT-652: Video generation models as world simulators
Sep 24, 2026
LIT-657: Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution
Sep 24, 2026
LIT-662: Vision Transformers Need Registers
Sep 24, 2026
LIT-663: DINOv3
Sep 24, 2026
LIT-664: Emerging Properties in Self-Supervised Vision Transformers
Sep 24, 2026
SOTA-391: Estimate attention resolution before training when choosing a position encoding, and measure it across the extrapolation boundary
Sep 24, 2026
SOTA-396: When a latent video model reuses an image autoencoder or upsampler, fine-tune them on video so they see time
Sep 24, 2026
SOTA-398: Pretrain a vision transformer on packed, aspect-preserved images at sampled resolutions, not on fixed square crops
Sep 24, 2026
SOTA-400: Append a few register tokens to a ViT so it does not commandeer background patches for scratch space
Sep 24, 2026
SOTA-401: Anchor the patch Gram matrix to an early teacher when training long, because dense features decay while global metrics improve
Sep 24, 2026
THEORY-101: A large trained ViT repurposes redundant patch tokens as scratch space for global information, which is what its feature-map artifacts are
- representation-and-encoding
- model-architecture
Sep 23, 2026
LIT-550: Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains
Sep 23, 2026
LIT-572: FLUX.2: Analyzing and Enhancing the Latent Space of FLUX – Representation Comparison
Sep 23, 2026
LIT-587: An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Sep 23, 2026
LIT-589: Representation Learning with Contrastive Predictive Coding
- representation-and-encoding
- analysis-and-evaluation
Sep 23, 2026
LIT-590: Momentum Contrast for Unsupervised Visual Representation Learning
Sep 23, 2026
LIT-591: A Simple Framework for Contrastive Learning of Visual Representations
Sep 23, 2026
LIT-592: Latent Dirichlet Allocation
Sep 23, 2026
LIT-593: Exploring Simple Siamese Representation Learning
Sep 23, 2026
LIT-594: Bootstrap your own latent: A new approach to self-supervised Learning
Sep 23, 2026
LIT-595: VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning
Sep 23, 2026
LIT-596: Barlow Twins: Self-Supervised Learning via Redundancy Reduction
Sep 23, 2026
LIT-597: Reading Tea Leaves: How Humans Interpret Topic Models
- analysis-and-evaluation
- representation-and-encoding
Sep 23, 2026
LIT-598: Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
Sep 23, 2026
LIT-599: DINOv2: Learning Robust Visual Features without Supervision
Sep 23, 2026
LIT-600: Automatic Evaluation of Topic Coherence
- analysis-and-evaluation
- representation-and-encoding
Sep 23, 2026
LIT-601: Masked Autoencoders Are Scalable Vision Learners
Sep 23, 2026
LIT-602: GloVe: Global Vectors for Word Representation
Sep 23, 2026
LIT-603: Distributed Representations of Words and Phrases and their Compositionality
Sep 23, 2026
LIT-604: Efficient Estimation of Word Representations in Vector Space
Sep 23, 2026
LIT-607: Improving Distributional Similarity with Lessons Learned from Word Embeddings
Sep 23, 2026
LIT-608: Don't count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors
- analysis-and-evaluation
- representation-and-encoding
Sep 23, 2026
LIT-609: Distributed Representations of Words and Phrases and their Compositionality
Sep 23, 2026
LIT-610: Efficient Estimation of Word Representations in Vector Space
Sep 23, 2026
LIT-612: Neural Word Embedding as Implicit Matrix Factorization
- representation-and-encoding
- signal-structure
Sep 23, 2026
LIT-613: A Latent Variable Model Approach to PMI-based Word Embeddings
Sep 23, 2026
SOTA-331: Pass low-dimensional coordinate inputs through sinusoids of sampled frequencies before an MLP, and tune the frequency scale rather than the distribution
- representation-and-encoding
- vision-and-graphics
Sep 23, 2026
SOTA-346: Retune the training timestep shift for each autoencoder latent space, and never rank latent spaces for generation under one shared schedule
Sep 23, 2026
SOTA-360: Score a density ratio against sampled negatives instead of reconstructing the target
- representation-and-encoding
- training-optimization
Sep 23, 2026
SOTA-361: Compose augmentations so the shortcut dies, and tune them for the contrastive objective rather than the supervised one
- data-pipeline
- representation-and-encoding
Sep 23, 2026
SOTA-362: Train through a projection head and then discard it, taking the layer before as the representation
- representation-and-encoding
- model-architecture
Sep 23, 2026
SOTA-363: Decouple the negative set from the batch with a queue, and keep the encoder that fills it slow
- representation-and-encoding
- training-optimization
Sep 23, 2026
SOTA-365: Learn without negatives by breaking the symmetry: a predictor on one branch and a stop-gradient on the other
Sep 23, 2026
SOTA-367: Or put the anti-collapse constraint in the loss, as a variance floor and a decorrelation term
- representation-and-encoding
- model-stability
Sep 23, 2026
SOTA-368: When a model's latent space is meant to be read by people, measure its interpretability with a human task such as word or topic intrusion, not by held-out likelihood
- analysis-and-evaluation
- representation-and-encoding
Sep 23, 2026
SOTA-369: Curate a pretraining set by retrieving neighbours of curated seeds, rather than scaling the uncurated pool
- data-pipeline
- representation-and-encoding
Sep 23, 2026
SOTA-370: Add many low-resolution views alongside the two full ones, because view count is what helps and resolution is what costs
- data-pipeline
- representation-and-encoding
Sep 23, 2026
SOTA-371: Score topic coherence automatically as the average pointwise mutual information of a topic's top-word pairs, estimated on a large external reference corpus
- analysis-and-evaluation
- representation-and-encoding
Sep 23, 2026
SOTA-372: Run the encoder on visible tokens only and push the mask tokens into a small decoder you discard
Sep 23, 2026
SOTA-373: Set the masking ratio by the signal's redundancy, not by the ratio that worked on text
- representation-and-encoding
- signal-structure
Sep 23, 2026
SOTA-374: When learning embeddings from co-occurrence, subsample very frequent tokens, discarding each occurrence with probability 1 − √(t/f) with t around 10⁻⁵
Sep 23, 2026
SOTA-375: When training with sampled negatives, draw them from the unigram distribution raised to the 3/4 power, not from the unigram or uniform distribution
- training-optimization
- representation-and-encoding
Sep 23, 2026
SOTA-379: Before crediting a word-embedding method over count-based ones, give the baselines the same design choices and tune every method alike
- analysis-and-evaluation
- representation-and-encoding
Sep 23, 2026
SOTA-380: Use prediction-based word vectors rather than count-based distributional vectors
- representation-and-encoding
- analysis-and-evaluation
Sep 23, 2026
SOTA-381: Sample a handful of negatives instead of normalizing over the vocabulary, and let the count fall as the data grows
- representation-and-encoding
- training-optimization
Sep 23, 2026
THEORY-077: A coordinate MLP learns high frequencies slowly because its tangent kernel's spectrum falls off fast, and a sinusoidal input mapping works by making that kernel stationary with a bandwidth you choose
- representation-and-encoding
- analysis-and-evaluation
Sep 23, 2026
THEORY-085: InfoNCE maximizes a lower bound on mutual information that cannot exceed log N
- analysis-and-evaluation
- representation-and-encoding
Sep 23, 2026
THEORY-087: Why a negative-free siamese network avoids collapse is unsettled, and the three accounts contradict each other
Sep 23, 2026
THEORY-088: Masked prediction transfers across modalities only after the masking ratio is rescaled to the signal's information density
Sep 23, 2026
THEORY-089: Word vectors trained on co-occurrence behave linearly because they encode log co-occurrence statistics: differences carry log ratios, which gives analogies, and sums carry products of context distributions, which gives composition
Sep 23, 2026
THEORY-092: Raising context counts to the 3/4 power helps word representations because PMI overweights rare contexts, and smoothing lowers their association scores
- signal-structure
- representation-and-encoding
Sep 23, 2026
THEORY-093: Skip-gram with negative sampling implicitly factorizes the word-context PMI matrix shifted by log k, weighting frequent pairs more heavily
- representation-and-encoding
- signal-structure
Sep 23, 2026
THEORY-094: Word vectors approximate PMI in low dimensions, and relations appear as clean directions despite a noisy fit, because text behaves as if generated by a slowly drifting discourse over isotropic word vectors
Sep 22, 2026
LIT-526: Concepts Whisper: Spectral Anti-Concentration and the Dual Geometry of Transformer Representations
Sep 22, 2026
LIT-531: Deep Learning and the Information Bottleneck Principle
Sep 22, 2026
LIT-545: Gemma 3n model overview
Sep 22, 2026
LIT-547: Matryoshka Representation Learning
- representation-and-encoding
- inference-optimization
Sep 22, 2026
SOTA-327: Train retrieval embeddings with nested losses, so a prefix of the vector can shortlist and the full vector re-rank
- representation-and-encoding
- inference-optimization
Sep 22, 2026
THEORY-063: Contextual concept directions sit in the low-variance tail of the unembedding spectrum while vocabulary contrasts sit at the top
Sep 21, 2026
LIT-485: Parameterized Synthetic Text Generation with SimpleStories
Sep 21, 2026
LIT-488: gen2seg: Generative Models Enable Generalizable Instance Segmentation
Sep 21, 2026
LIT-489: Measuring In-Context Computation Complexity via Hidden State Prediction
Sep 21, 2026
LIT-494: GaussianToken: An Effective Image Tokenizer with 2D Gaussian Splatting
Sep 21, 2026
LIT-496: Taming Transformers for High-Resolution Image Synthesis
Sep 21, 2026
LIT-497: Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation
Sep 21, 2026
LIT-498: MaskGIT: Masked Generative Image Transformer
Sep 21, 2026
LIT-499: Neural Discrete Representation Learning
- representation-and-encoding
- generative-modeling
Sep 21, 2026
LIT-500: Vector-quantized Image Modeling with Improved VQGAN
Sep 21, 2026
LIT-505: Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences
Sep 21, 2026
LIT-506: Interpretable RNA Foundation Model from Unannotated Data for Highly Accurate RNA Structure and Function Predictions
Sep 21, 2026
LIT-507: Adaptive Estimators Show Information Compression in Deep Neural Networks
- analysis-and-evaluation
- representation-and-encoding
Sep 21, 2026
LIT-508: Opening the Black Box of Deep Neural Networks via Information
Sep 21, 2026
LIT-509: On the Information Bottleneck Theory of Deep Learning
- analysis-and-evaluation
- representation-and-encoding
Sep 21, 2026
LIT-511: Image-GS: Content-Adaptive Image Representation via 2D Gaussians
Sep 21, 2026
SOTA-294: Fit the tokenizer to the corpus when the corpus is deliberately narrow
- representation-and-encoding
- tiny-models
Sep 21, 2026
SOTA-299: Finetune the whole generative model, encoder and decoder, for a dense perception task instead of attaching a head to a backbone
Sep 21, 2026
SOTA-305: State the token count and input resolution beside any reconstruction FID, and compare tokenizers only at equal rate
- analysis-and-evaluation
- representation-and-encoding
Sep 21, 2026
SOTA-306: Factorize the codebook: look up in a low-dimensional normalized space, embed in a high-dimensional one, and report utilization alongside reconstruction quality
- representation-and-encoding
- generative-modeling
Sep 21, 2026
SOTA-312: State the noise or binning assumption behind any mutual information you report for a deterministic network, and show the conclusion survives changing it
- analysis-and-evaluation
- representation-and-encoding
Sep 21, 2026
THEORY-052: Synthesis requires equivariant representations and discriminative pretraining rewards invariant ones, which is why a generative prior groups objects it was never shown
- representation-and-encoding
- vision-and-graphics
Sep 21, 2026
THEORY-058: Apparent compression in the information plane is saturating activations collapsing into extreme bins, not information being discarded
- analysis-and-evaluation
- representation-and-encoding
Sep 20, 2026
LIT-458: The Platonic Representation Hypothesis
Sep 20, 2026
LIT-459: Contextures: The Mechanism of Representation Learning
- representation-and-encoding
- training-optimization
Sep 20, 2026
LIT-460: The Lattice Representation Hypothesis of Large Language Models
- concept-geometry
- representation-and-encoding
Sep 20, 2026
SOTA-271: Train on a second modality even when the target is single-modality
Sep 20, 2026
SOTA-273: Choose a pretraining context whose association with the input is neither too strong nor too weak, and mix contexts to get there
- training-optimization
- representation-and-encoding
Sep 20, 2026
THEORY-034: Concepts sit in embedding space as intersections of half-spaces, so inclusion, intersection and union are geometric operations
- concept-geometry
- representation-and-encoding
Sep 20, 2026
THEORY-036: Representations converge across architectures, objectives and modalities, and the endpoint is a model of what generated the data
Sep 19, 2026
LIT-435: NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
- vision-and-graphics
- representation-and-encoding
Sep 19, 2026
SOTA-250: Pretrain by predicting representations of masked regions, not their pixels
- vision-and-graphics
- representation-and-encoding
Sep 17, 2026
LIT-409: Token Erasure as a Footprint of Implicit Vocabulary Items in LLMs
- representation-and-encoding
- signal-structure
Sep 17, 2026
LIT-410: The English phrase-as-lemma construction: When a phrase masquerades as a word, people play along
- signal-structure
- representation-and-encoding
Sep 17, 2026
LIT-412: From Tokens to Words: On the Inner Lexicon of LLMs
- representation-and-encoding
Sep 17, 2026
SOTA-247: Add a multi-token word to a frozen model's vocabulary from the model's own detokenized representation of it
- representation-and-encoding
Sep 17, 2026
THEORY-021: A model builds a latent vocabulary in its early layers whose units are not the tokenizer's
- representation-and-encoding
Sep 17, 2026
THEORY-022: A phrase placed in a word slot is read as a lemma, and the reading comes from the construction rather than the phrase's frequency
- signal-structure
- representation-and-encoding
Sep 15, 2026
LIT-341: A Growing Neural Gas Network Learns Topologies
- representation-and-encoding
Sep 15, 2026
LIT-363: 'Neural-gas' network for vector quantization and its application to time-series prediction
- representation-and-encoding
Sep 08, 2026
SOTA-179: Truncate the rotary encoding's low frequencies rather than rescaling its base
- representation-and-encoding
Sep 08, 2026
SOTA-187: Train the generative model in a learned compressed latent, not at full resolution
Sep 07, 2026
LIT-192: Extending Context Window of Large Language Models via Positional Interpolation
- representation-and-encoding
- attention-techniques
Sep 07, 2026
LIT-193: YaRN: Efficient Context Window Extension of Large Language Models
- representation-and-encoding
- attention-techniques
Sep 07, 2026
LIT-207: The Impact of Positional Encoding on Length Generalization in Transformers
- representation-and-encoding
- attention-techniques
Sep 07, 2026
LIT-208: Rope to Nope and Back Again: A New Hybrid Attention Strategy
- representation-and-encoding
- attention-techniques
Sep 07, 2026
LIT-209: SWAN-GPT: An Efficient and Scalable Approach for Long-Context Language Modeling
- representation-and-encoding
- attention-techniques
Sep 07, 2026
LIT-210: Round and Round We Go! What makes Rotary Positional Encodings useful?
- representation-and-encoding
- attention-techniques
Sep 07, 2026
LIT-216: Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture
- vision-and-graphics
- representation-and-encoding
Sep 07, 2026
SOTA-151: Extend a trained model's context by rescaling RoPE, not by fine-tuning at the longer length
- representation-and-encoding
Sep 07, 2026
SOTA-153: Drop positional encoding from the global-attention layers of a hybrid and let the cheap local layers carry position
- representation-and-encoding
- attention-techniques
Sep 05, 2026
LIT-133: Kimi Linear: An Expressive, Efficient Attention Architecture
- attention-techniques
- representation-and-encoding
Aug 24, 2026
LIT-003: Neural Machine Translation of Rare Words with Subword Units
- representation-and-encoding
Aug 24, 2026
LIT-045: RoFormer: Enhanced Transformer with Rotary Position Embedding
- representation-and-encoding
- model-architecture
Aug 24, 2026
LIT-048: Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
- representation-and-encoding
- model-architecture
Aug 24, 2026
SOTA-007: BPE tokenization for open vocabulary tasks
- representation-and-encoding
Aug 24, 2026
SOTA-063: use RoPE for LLM (1D sequence) positional embeddings
- representation-and-encoding

Created with Quartz v5.0.0 © 2026

Repository
Quartz