Topics — The twenty-eight kinds of claim this anthology files against — the primary axis of both the practice registry and the reading list, and the same twenty-eight for each (ADR-026). The filing rule is the point: take a domain topic when the claim is ABOUT the domain; a claim merely discovered in one still takes its kind. The axis organizes the record, it does not bound it: content the list cannot place is evidence the list is short (ADR-059).
Concept geometry — how concepts are laid out in a model’s representation space — linear directions and the regions, lattices and spectra built from them, superposition and the dictionaries that pull it apart, reading concepts out with probes and writing them in with steering vectors, and whether different models arrive at the same geometry.
4 of 449 SOTA documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| SOTA-342 | Report a sparse autoencoder’s fidelity as downstream loss in compute-equivalent terms, not as fraction of loss recovered against zero ablation | Gao et al. (2024), LIT-571 — splice the SAE’s reconstruction into the forward pass and report the resulting loss as the pretraining compute a model of that loss would need. Zero-ablating the residual stream is so destructive that “loss recovered” against it flatters any reconstruction. Their 16M-latent GPT-4 SAE scores 98.2% on it, and equals a GPT-4 trained on 10% of the compute. | Proposed |
| SOTA-344 | Train sparse autoencoders with a TopK activation instead of an L1 penalty, and prevent dead latents with transposed-decoder initialization and an auxiliary loss | Gao et al. (2024), LIT-571 — keep the k largest pre-activations and zero the rest, and train on reconstruction alone. L0 is then set, not tuned through an L1 coefficient that also shrinks the latents that fire. It beats ReLU SAEs on the MSE–L0 frontier by more as SAEs grow. Initialize the encoder as the decoder’s transpose and add a loss reconstructing the residual from dead latents: 7% dead at 16M latents, against up to 90%. | Proposed |
| SOTA-345 | Probe language-model activations with logistic regression on the raw activations, and credit sparse-autoencoder probes only against that baseline under validation-based selection | Kantamneni et al. (2025), LIT-568 — across 113 datasets, adding SAE-latent probes to a toolkit of raw-activation probes, choosing per task by validation AUC, changes test AUC by −0.003. It does not help under data scarcity, class imbalance, label noise or shift. Reported SAE wins came from baselines denied the same pooling or selection. Use logistic regression on activations, and hold any SAE method to it. | Active |
| SOTA-378 | Compare, project and orthogonalize a language model’s concept directions after whitening by the unembedding covariance, not by raw cosine | Park, Choe and Veitch (2024), LIT-606 — training fixes a model’s representation only up to an invertible linear map, so Euclidean cosine between concept directions is not meaningful. Use ⟨u, v⟩ = uᵀ Cov(γ)⁻¹ v, with Cov(γ) the covariance of the unembedding rows, or equivalently transform by Cov(γ)^(−1/2) and then use Euclidean tools. Shown on LLaMA-2 7B and Gemma-2B by heatmap. Contested by a null for cross-lingual transport in 17 models. | Proposed |