nucleation

THEORY-426: Evaluating expected free energy recursively over the beliefs an agent would hold after each plausible outcome makes active inference a pruned tree search over belief states, formally analogous to Bellman recursion; in single deterministic runs on a T-maze and an 8 × 8 maze it planned through short-term costs that agents with a shorter horizon did not; that the pruning keeps deep search tractable beyond toy problems is argued, not shown

3 min read

Properties5
StatusActive
Version2
Tagsprobabilistic-modeling, information-theory
SourceLIT-577 — Sophisticated Inference
Cited as “extends” byTHEORY-1245 — Evaluating expected free energy recursively over the beliefs an agent would hold after each plausible outcome makes active inference a pruned tree search over belief states, formally analogous to Bellman recursion, which lets the agent plan through short-term costs that one-step agents cannot; that the pruning keeps deep search tractable beyond toy problems is argued, not shown

Graph View

Source

Friston, Da Costa, Hafner, Hesp & Parr (2020), “Sophisticated Inference”, LIT-577, read in full as prose in NOTE-454. No equation is reproduced there, so the account rests on the prose that states the recursion. The simulation code was not run.

What was actually shown

The construction (Eq. 1.11, in prose). The expected free energy of a next action is its risk plus its ambiguity plus the average, over predicted outcomes and then over subsequent actions, of the expected free energy of those actions. A softmax of the accumulated average is the prior over the next action. It has the recursive form of Bellman’s equation, over functionals of beliefs rather than functions of states, and the paper says it recovers Bayes-adaptive RL in the zero-temperature limit.

Planning through costs. In a T-maze (32 trials, 95% cue validity), with the cue given a cost of 1, the sophisticated agent still visits the cue and switches to exploitation at trial 12; the one-step agent never leaves the centre. In an 8 × 8 maze, horizons one to three stick behind the barrier nearest the target, and horizon four finds the eight-move shortest path.

Cost. Of about 10¹⁰ (horizon four) or 10¹⁵ (horizon six) potential paths, “usually several hundred” are evaluated, in “a few hundred milliseconds”, because predictive probabilities below 1/16 are not expanded.

What could have come out otherwise: the horizon-four agent could have failed the maze, and the costly cue could have stopped the sophisticated agent too. Neither happened, in single deterministic runs.

What this does not say

  • That the search scales. Scaling is argued from pruning and sparsity. Where many outcomes stay plausible, the 1/16 threshold cuts little, and no such problem is tried (NOTE-454, Open questions).
  • That it beats reinforcement-learning planners. No Bayes-adaptive or Monte Carlo planner is run on the same tasks.
  • That the special-case derivations ground it in physics. The appendix casts expected utility, KL control, optimal design, curiosity, empowerment and the information bottleneck as special cases. The nonequilibrium steady-state grounding is cited, not developed (C6, weak), and it is what the critiques filed beside the paper (LIT-603, LIT-598) dispute.
  • That information gain is intrinsic motivation. The identification is by name; the record’s self-determination holdings (LIT-558, LIT-559) define intrinsic motivation otherwise.
  • That the agent perceives or acts in any rich sense. The rat is given A, B, C and D, its location is observed, and moves are primitive, which is Raja et al.’s point (LIT-598).
  • That it plans through costs in general. That, with tractability, is THEORY-1245, Proposed.

Connections

  • THEORY-045 and THEORY-285 are the record’s other active-inference accounts, about affect and about post-traumatic context inference. Neither rests on this recursion.
  • The planning analogy is to Bayes-adaptive RL (Åström, Duff, Ross et al.), which the record does not hold.

Backlinks