Nathaniel D. Daw, Yael Niv & Peter Dayan (2005), Nature Neuroscience 8(12):1704–1711 — DOI-10.1038/nn1560
Key takeaways
- Two controllers, two reinforcement-learning methods. The habitual system is model-free: it “caches” a scalar long-run value for each action in each state, learned by temporal-difference bootstrapping. The goal-directed system is model-based: it learns the task’s transitions and outcomes and computes values “on the fly” by searching a tree. Only the tree knows which outcome an action leads to, so only it adjusts at once when that outcome is devalued.
- Arbitration by uncertainty. Each controller is given an approximate Bayesian posterior over its values. Behaviour uses the value from whichever controller has the smaller posterior variance. The tree is more data-efficient, but each step of search adds “computational noise”. The cache is cheap but slow to propagate experience.
- It reproduces the devaluation facts. Moderately trained actions are devaluation-sensitive and over-trained ones are not. Over-trained actions stay sensitive when the task has two actions for two outcomes, or when the action is proximal to reward (magazine entry). Lesions of either system leave the other in control.
- A normative reading of dual control. “Both controllers are pursuing identical rational ends.” This is set against economic accounts in which “irrational, impulsive or emotional limbic influences” interfere with “a more rational prefrontal controller”.
Standing in the record
Filed on 2026-10-03 at the owner’s request, for the record’s coverage of how an organism’s control systems come to act as one. It is the record’s first reading of the model-based/model-free distinction. Dayan and Berridge (LIT-584) extend the distinction to Pavlovian prediction.
The boundary. The paper uses reinforcement-learning algorithms (Bayesian Q-learning, Bayesian value iteration) to explain animal behaviour and its neural substrate. It recommends nothing for machine-learning practice, and its question is about brains, so it belongs here and not in the anthology.
It bears on the record’s account of self-control. THEORY-056 holds that regulation is one motive state checking another, with no distinct regulating system. This paper’s two controllers compete, and the choice between them is made by comparing uncertainties, not by a supervisor that knows better. That fits the theory’s architecture, though the paper is about instrumental action and not emotion. The paper does suggest that infralimbic cortex or anterior cingulate might implement the arbitration, which would be a distinct arbitrating structure, if not a regulating one. Ainslie (LIT-586) and Shenhav et al. (LIT-585) give different answers to the same question, and NOTE-459 sets them side by side.
No instruction for machine-learning practice; nothing here belongs in the anthology.