Source
Gao et al. (2024), LIT-571, §4.1 and footnote 11. Read as NOTE-307.
The practice
- Measure fidelity downstream. Replace the activations with the SAE’s reconstruction during the forward pass, and report the change in cross-entropy or the KL to the original output. MSE alone understates differences between SAEs that matter to the model
- Do not normalize by zero ablation. Zeroing the residual stream destroys the model, so almost any reconstruction “recovers” most of the gap. 98.2% sounds nearly lossless. It is not
- Anchor it to something a reader can weigh. The paper’s anchor is the fraction of the subject model’s pretraining compute that reaches the same loss (10% for its best GPT-4 SAE). A smaller model from the same family, or a mean-ablation baseline, serves the same purpose where no compute-loss curve is available
The critique and the replacement are both Gao et al.’s, LIT-571, which proposed the compute-equivalent framing. Their 16M-latent SAE on GPT-4 scores 98.2% “loss recovered” against zero ablation, yet with its reconstruction substituted into GPT-4 the language-modeling loss equals that of a model trained on 10% of GPT-4’s pretraining compute. They also found the downstream-loss gap between SAE architectures larger than the MSE gap (Figure 5a), which is the case for measuring downstream at all.
Conditions
- The compute-equivalent figure needs a scaling curve for the subject model, which outside labs rarely have
- Fidelity is not usefulness. A faithful SAE can still fail to help on a task (LIT-568)