nucleation

THEORY-484: The brain implements hierarchical reinforcement learning with options: dorsolateral prefrontal cortex represents which option is active, dorsolateral striatum holds option-specific policies, orbitofrontal cortex holds option-specific values, and dopamine carries option-level prediction errors at subtask boundaries

2 min read

Properties4
StatusProposed
Tagsbehavioral-integration, neuroscience, learning-and-conditioning
SourceLIT-581 — Hierarchically organized behavior and its neural foundations: A reinforcement learning perspective
PresupposesTHEORY-688 — In an actor-critic learner that can choose temporally extended options as well as primitive actions, options that fit the task speed learning, while options with ill-placed subgoals slow it, and options that fit an earlier layout keep the agent from finding a newly opened shortcut, so hierarchy helps only with the right subroutines

Graph View

Source

Botvinick, Niv and Barto (2009), LIT-581 (Cognition 113(3): 262–280), read in full in NOTE-451, “Neuroscientific Implications” and Discussion.

What was actually shown

The starting point. The actor-critic model is already mapped onto the brain: the critic’s prediction error to phasic dopamine, the actor to the striatum. The paper asks what the options framework would add (Fig. 2C–D).

Four extensions, each with a candidate substrate.

  1. Option identity to dorsolateral prefrontal cortex, which represents task sets; pre-SMA, SMA and premotor cortex are candidates too.
  2. Option-specific policies to dorsolateral striatum, which receives frontal input and shows context-dependent action coding, as in grooming-sequence neurons.
  3. Option-specific values to orbitofrontal cortex, whose value representations shift with strategy.
  4. Prediction errors over whole options to orbitofrontal sustained reward-predictive activity and to semi-Markov accounts of dopamine.

Predictions. Phasic dopamine at subtask boundaries scaling with option-level prediction errors; a possible neural correlate of pseudo-reward; prefrontal coding of discrete subtask segments.

What could have come out otherwise. Nothing here was tested. The mappings are fits to prior findings, so they could not have failed in this paper.

What this does not say

  • That the mapping is shown. The correspondences are drawn from correlational findings and are proposals (C3, weak to moderate).
  • That there is a neural pseudo-reward. That is an open question and depends on the HRL framework assumed. HAM does without pseudo-reward, and MAXQ decomposes value differently.
  • That planning over options is covered. The model is model-free; option models are discussed, not implemented.
  • That behaviour is strictly hierarchical. The paper itself says everyday behaviour is only quasi-hierarchical (C6).

Connections

  • It presupposes the computational result THEORY-688, that options help only when they fit, as the reason a brain would need them and the source of the option-discovery problem.
  • THEORY-062 (model-based and model-free control by relative reliability) is the division the paper adopts, with HRL placed in the model-free system.
  • NOTE-451 observes that HRL is control without a distinct regulator: the controller at each level is a policy over options, learned by the same prediction-error rule as the level below. That is the shape THEORY-056 claims for emotion regulation, and it bears on QUESTION-054’s governing level. The paper does not draw either parallel.

Backlinks