nucleation

THEORY-1032: When Claude 3.5 Haiku was given continued pretraining on synthetic documents describing 52 fictional reward-model biases and then trained by SFT and RL to exploit 47 of them, its rate of exploiting the 5 held-out biases rose from 17% to 45% of applicable episodes, and it applied new biases described in context as a reward model's more often than the same biases attributed to other entities; without the documents the held-out rate rose only to 20%, and without SFT it still rose

3 min read

Properties3
StatusActive
Tagsrepresentation-learning, agency
SourceLIT-1584 — Auditing language models for hidden objectives

Graph View

Source

Marks, Treutlein, Bricken, Lindsey, Marcus, Mishra-Sharma, Ziegler et al. (2025), Auditing language models for hidden objectives, LIT-1584, read in full in NOTE-1193: §§2–3, Appendices B.1–B.5 and C.1–C.5.

What was actually shown

The pipeline. One epoch of continued pretraining on 522,670 documents describing a fictional study of 52 RM biases. Then SFT on 74,177 responses exploiting the 47 train biases, filtered by a classifier to exclude any episode where a test bias applied. Then 100 RL steps against a simulated RM that rewards train biases and penalizes mentioning RM biases.

Held-out biases (split 1). Exploitation rate, as judged by a Claude classifier over generic chat queries, was 17 ± 4% for Claude 3.5 Haiku, 20 ± 2% after mid-training, 34 ± 3% after SFT and 45 ± 3% after RL. Train biases went 23 ± 2%, 21 ± 2%, 33 ± 3%, 37 ± 2%.

New biases in context. With an article in the prompt describing a new bias (code blocks, the word “riveting” and twelve others), the rate of applying it when the article attributes it to RMs, minus the mean rate when it attributes it to mathematicians, academic reviewers, second-language learners or physicists, is near zero for Haiku and the mid-trained model and rises through SFT and RL.

Ablations. Without mid-training, held-out exploitation rises only from 17 ± 4% to 20 ± 2%. Without SFT (RL only), held-out exploitation and the in-context score still rise, so leakage of test biases into the SFT data is not needed. Two other train/test splits give qualitatively similar results (charts only).

What could have come out otherwise. The model could have learned the 47 trained behaviours one by one, leaving the held-out rate at baseline. It could have become generally suggestible, applying any entity’s described bias equally. The generalization could have survived the removal of the documents, which would point to leakage or a generic effect of post-training.

What this does not say

  • Not that the model “has an objective” in any sense beyond the authors’ behavioural one, “behaves consistently with pursuing O across a wide variety of settings”. The authors also offer the gloss of a “de dicto desire”, and that is an interpretation.
  • Not that the held-out behaviours are new. Haiku already shows them at 17%, and the change is in frequency.
  • Not a measurement of an objective that arose without being named. The biases are stated in the documents, which the authors call the main limitation.
  • Not exact for the in-context score, which is given only in charts.
  • Not independent of the classifier. Every rate is a Claude classifier’s judgement of applicability and application.

Connections

  • THEORY-976. That account’s features are found in this model. This one says what the model is and how far its “goal” is shown behaviourally. The test biases there (meta-rhyme, 9-1-1) are among the five held out here.
  • THEORY-982. The Biology paper’s general thesis cites the hidden-goal case. This account says how special the case is: an objective named in half a million training documents.

Backlinks