anthology-of-the-sota

LIT-569: Eliciting Latent Predictions from Transformers with the Tuned Lens

2 min read

Properties6
StatusActive
Tagsanalysis-and-evaluation
ExtendsLIT-570 — interpreting GPT: the logit lens
Cited as “introduced by” bySOTA-343 — Read a transformer's intermediate-layer predictions through a tuned lens, not the raw logit lens
Cited as “paper” byNOTE-308 — Tuned Lens
Cited as “source” bySOTA-343 — Read a transformer's intermediate-layer predictions through a tuned lens, not the raw logit lens

Graph View

Belrose, Ostrovsky, McKinney, Furman, Smith, Halawi, Biderman, Steinhardt, EleutherAI, FAR AI, Toronto, BU and Berkeley (2023) — ARXIV-2303.08112

Key takeaways

  • The logit lens is unreliable and biased (§2). For BLOOM and OPT-125M its top-1 is often the input token in more than half the layers. On GPT-Neo-2.7B its marginal is 4–5 bits of KL from the final layer’s, at most layers. The comparison given is 0.0068 bits between Pythia-160M and Pythia-12B
  • The tuned lens (§3): LogitLens(A_ℓ h_ℓ + b_ℓ), one affine “translator” per layer. It is initialized to the identity and trained on held-out pretraining text with KL to the final logits. The distillation target keeps it from learning more than the model knows. Lower perplexity than the logit lens at every layer on Pythia 70M–12B and GPT-NeoX-20B (Figure 5), with lower variance across models
  • It is causally faithful, by the paper’s tests (§4). Directions that move the lens also move the model (Spearman ρ = 0.89, Pythia-410M, layer 18), with no lens-only directions. Stimulus-response alignment is higher than the logit lens’s at every layer
  • Transfer: translators carry over to nearby layers and to fine-tuned models. A LLaMA-13B lens on Vicuna-13B costs at most 0.3 bits per byte
  • Applications (§5). Trajectories fed to an off-the-shelf outlier detector flag prompt injections on Pythia-12B, with AUROC near 1 on five of nine tasks. But the SRM (Mahalanobis) baseline matches or beats it on eight. On eliciting a fine-tuned model’s secret word, the tuned lens beats the logit lens at some layers and loses at others

Standing in the anthology

Filed from #163 (“LogitLens and adjacent mechanistic interpretability”). It extends the logit lens (LIT-570) and sources SOTA-343.

What the headline leaves out. The abstract’s “detect malicious inputs with high accuracy” is true against no baseline. Against a one-layer Mahalanobis detector, the lens wins on one task in nine. And the undertraining note added in version 6 means the paper’s quantitative comparisons understate the method they recommend. It does not re-run them.

Read — NOTE-308.

Backlinks