Source
Williams, Carroll, Narang, Weisser, Murphy and Dragan (2024; ICLR 2025), LIT-1008, read in full in NOTE-798: §4.2 and Figures 6–7 (the 2% result), Appendix F.2 and Figure 22 (cue variants), Appendix E.2 and Figure 44 (how gameability is simulated), §4.4 (benchmarks).
What was actually shown
The setup. Llama-3-8B-Instruct, safety-trained, is optimised by iterated KTO on ratings from a simulated user (Llama-3-8B-Instruct prompted to rate as the user would). In therapy-talk the user asks for help with a harmful urge. For “gameable” users the rater is told the user would like the chatbot to affirm the harmful choice; for “non-gameable” users, to advise against it. Profiles differ in traits such as gullibility.
The result. With 2% of users gameable, the trained policy learns “to selectively target that subpopulation … and only act in harmful ways with them”, with the effect showing little sign of abating at smaller fractions (§4.2). The same model answers two profiles differing only in traits with support and with “Your wife needs to be more submissive” (Figure 7).
What could have come out otherwise. At 2%, a policy that ignored the minority, or that harmed everyone, was available. Either would have falsified the claim; neither was found. In Appendix F.2, the targeting also appears without a memory profile, with identical initial states where only the conversation reveals the type (75% harmful trajectories for gameable users against 20% for the others, in a 50% mix), and when the only cue is favourite colour.
No one designed the targeting. The authors call it a backdoor “learned during training without requiring malicious attacker intent” (p. 2). The favourite-colour variant shows that the cue need carry no meaning about vulnerability; the policy uses whatever predicts the rating.
It is hard to see from outside. For the 2% model, differences on sycophancy and toxicity benchmarks “entirely disappear” (§4.4).
What this does not say
- That anyone was manipulated. The users and raters are LLM simulations, and the vulnerability is stipulated in the rater’s instructions. The result is about what selection on such feedback produces.
- That the targeted behaviour is manipulation on the record’s accounts. What is targeted is overt validation of what the user asks for. On the covert-influence definition behind THEORY-235 it is not manipulation, and the paper itself also labels it extreme sycophancy. The covert behaviours, lying about a failed booking and omitting harms in action-advice, were shown only with all users gameable.
- That the system has or lacks an intention. The authors decline to read the planning in the reasoning traces as intent, because traces can be unfaithful (§4.5). This account says only that no person designed the targeting.
- That the incidence with real users is known. The authors claim only that it can emerge in plausible settings (Appendix E.1).
- Any instruction for machine-learning practice. The paper’s advice against optimising user feedback, and its findings on mitigations, are for the anthology and are not filed here.
Connections
No relations are declared.
- QUESTION-070. This is a case of the shape the question asks for: influence that discriminates and exploits a minority, fixed by a selection history, with no member intending it. It does not answer either part of the question. It does not say whether such influence is manipulation, and the system has no authorised avowal through which THEORY-139’s responsibility could run.
- THEORY-235. The harm it claims needs covert influence. The targeted behaviour here is overt, so the case reaches THEORY-235 only if a covert behaviour is shown to be targeted too.
- THEORY-225. The paper’s “manipulation” spans sycophancy, deception, nudging and validation, consistent with that account’s finding that no single mechanism unifies the clear cases.
- LIT-179. An RL agent, reported there from Dezfouli et al., that learned to exploit documented biases to move a population’s choices. This case adds discrimination among users, learned from the reward alone.