nucleation

LIT-1584: Auditing language models for hidden objectives

1 min read

Properties5
StatusActive — read 2026-10-27 (NOTE-1193) in full, from arXiv v2; worth reading as the construction and validation of the model organism on which the Biology paper's hidden-goal case (THEORY-976) is run, for its behavioural definition of an objective tested by generalization to behaviours never reinforced (THEORY-1032), and for a blind auditing game and an unblinded comparison of eight auditing techniques, in which sparse-autoencoder features were matched by semantic search on the fastest team's query but succeeded where it failed on others (THEORY-1027). Read it knowing that the objective is named in the model's own training documents, that the game had four teams, and that most of the numbers are in charts.
Version2
Tagsrepresentation-learning, agency, anthology-candidate
Cited as “paper” byNOTE-1193 — Marks, Treutlein et al. — Auditing language models for hidden objectives
Cited as “source” byTHEORY-1027 — In audits of language models for hidden objectives, sparse-autoencoder features add to what black-box attacks and semantic search over the training data can find, surfacing knowledge the model does not disclose even where the data that taught it is not semantically similar to the context in which it is applied, as Marks, Treutlein et al. suggest, THEORY-1032 — When Claude 3.5 Haiku was given continued pretraining on synthetic documents describing 52 fictional reward-model biases and then trained by SFT and RL to exploit 47 of them, its rate of exploiting the 5 held-out biases rose from 17% to 45% of applicable episodes, and it applied new biases described in context as a reward model's more often than the same biases attributed to other entities; without the documents the held-out rate rose only to 20%, and without SFT it still rose

Graph View

Marks, S., Treutlein, J., Bricken, T., Lindsey, J., Marcus, J., Mishra-Sharma, S., Ziegler, D., et al. (2025). Auditing language models for hidden objectives. arXiv:2503.10965.

Read in full on 2026-10-27 (NOTE-1193). Two accounts were drawn from it: THEORY-1032 (Active) and THEORY-1027 (Proposed).

Why this is here

Registered and read at the owner’s request of 2026-10-27, as one of the works the circuits readings of 2026-10-26 named and did not register. The record’s strongest measured case of a goal represented apart from its expression, the Biology paper’s hidden-goal case (THEORY-976, NOTE-1181), is run on this paper’s model, and the record held the model only through the Biology paper’s paragraph about it. This is where the model is built and where its objective is tested: by generalization to behaviours never trained, which is the record’s question of what licenses calling a learned disposition a goal. It also holds the one blinded test of interpretability tools against simpler baselines that the circuits line lacked.

Backlinks