Contribution
Exaggerated threat appraisal, fear generalization and avoidance far in advance of threat had been described in anxiety but not derived from one cause. The paper shows that a single departure from normative sequential evaluation produces all three in a Markov decision process: the agent evaluates each action as if it will not act well afterwards. It also links that departure to the clinical literature on perceived control and self-efficacy, which earlier computational accounts of control (Huys & Dayan 2009) had treated only for one-step choices.
Key insight
The “max” in the Bellman equation makes approach and avoidance asymmetric: opportunities for reward propagate back to earlier states, but avoidable dangers do not, because the best action near them is to avoid. “If danger can be avoided later, it poses less threat now” (p. 1). Remove the assumption that one will avoid later and the asymmetry goes: threat value spreads backward across the state space, and an agent choosing optimally under that misbelief avoids early and widely. What is wrong is a belief about one’s own future choices, not about the danger.
Assumptions
- The model (p. 4, Eq. 3). State-action values are computed as Q_w(s,a) = r(s,a) + γ Σ p(s′|s,a)[w max Q_w(s′,a′) + (1 − w) min Q_w(s′,a′)], the β-pessimistic value function of Gaskett (2003). w = 1 is the optimal agent; w = 0 expects always to act against its preferences.
- The belief is false (p. 4). “Importantly, this belief is false under the model”: the agent does maximize at each step, but computes values on the assumption that it will not later.
- Where the pessimism sits (pp. 4, 10). It could be put in the policy (misbelief about one’s own avoidance) or in the transition probabilities (misbelief that the world’s response is unpredictable or adversarial), or in uncertainty about the current state. The paper simulates only the first and says the others “can produce ultimately similar results”.
- Environments (pp. 12–14). Deterministic, infinite-horizon, discrete MDPs; values solved by value iteration except in the free-choice task, where a pessimistic temporal-difference learner is used (learning rate 0.4, 1,000 simulated agents per level of w).
Key results
All results are simulations; the empirical findings they match are cited.
- Gridworld (p. 3, Fig. 1). With w = 1 every state except the harmful one has positive value; with w = 0.5 and w = 0 negative value spreads from the threat and the optimal path detours further. The authors read this as exaggerated threat appraisal, generalization and persistent avoidance (p. 4).
- Balloon analogue risk task (p. 5, Fig. 2A–C). Pessimistic agents (w = 0.6, 0.2) cash out earlier, as anxious people do (Maner et al. 2007; Ramírez et al. 2015).
- A unique prediction (p. 6). Bias should appear only where future avoidance matters. The authors cite Fung et al. (2019): trait anxiety predicted earlier escape from slow predators but not fast ones.
- Behavioural inhibition (p. 6, Fig. 2D–E). In Bach’s (2015) sleeping predator task, the relative value of approach falls with potential loss, more under pessimism; with the added assumption that lower relative value slows responses, the model reproduces slower approach under threat and its enhancement by anxiety.
- Aversive pruning (pp. 6–7, Fig. 3). A pessimistic agent (w = 0.5) avoids a branch with a large early loss followed by a larger gain, not because it fails to look ahead but because it expects not to recoup the loss. They propose a test that would separate their account from Huys et al.’s pruning (p. 7).
- Anxiety to depression (pp. 7–8, Fig. 4). During learning under pessimism, negative value spreads until the agent expects no reward and forgoes action, a model of the clinical hypothesis that anxiety precedes some depression.
- Persistence after treatment (p. 8). Argued, not simulated: because threat value has spread far, correcting it takes many steps of relearning, and “Frustratingly, these biased estimates of value may still remain even after a misbelief in the efficacy of future action is corrected for in a course of therapy.” They compare it to model-free learners’ failure to adjust to revaluation.
- Free-choice premium (pp. 8–9, Fig. 5). People prefer options that lead to further choice even when outcomes are identical. Pessimistic learners show a diminished or reversed preference; the authors predict the same for anxious people. “Future empirical research will be required to test this prediction” (p. 9).
Claims
| id | claim | strength | support |
|---|---|---|---|
| C1 | Under optimal sequential evaluation, avoidable threat does not propagate value to antecedent states | strong | follows from Eq. 2; Fig. 1B |
| C2 | Pessimism about one’s own future avoidance propagates threat value to antecedent states and produces early, wide avoidance | strong, as a property of the model | Eq. 3; simulations, Figs 1, 4 |
| C3 | The model reproduces earlier cash-out, behavioural inhibition and aversive pruning associated with anxiety | moderate: qualitative match, no fit to data | Figs 2–3; cited studies |
| C4 | Values learned under pessimism may persist after the misbelief is corrected | weak: argued, not simulated | p. 8 |
| C5 | Anxious people will show a reduced free-choice premium | untested prediction | Fig. 5 |
| C6 | Pessimism about the environment’s transitions would produce similar symptoms | asserted | p. 10; not simulated |
Concepts
- Avoidance against escape: avoidance prevents the onset of a threat; escape responds to a threat already near (p. 2).
- β-pessimistic value: the mixture of best and worst next action, with weight w on the best (p. 4).
- Free-choice premium: a preference for options that lead to further choices, with outcomes held equal (p. 8).
Connections
It formalises self-efficacy theory (Bandura & Adams 1977), Barlow’s triple vulnerability model and learned-helplessness accounts of anxiety (Alloy et al. 1990), and extends Huys & Dayan (2009) to sequential choice. It reads the Bishop group’s uncertainty work (Browning et al. 2015) as compatible: uncertainty is aversive because it “is resolved (i.e., marginalized) under pessimistic distributional assumptions” (p. 10). It points to replay and model-based planning as a possible mechanism, which would make worry “overcontemplating, or overweighting, certain pessimistic trajectories” (p. 11). Brown et al. (LIT-1479) later doubt that pessimistic policies capture anxiety rather than normal behaviour under threat.
Bearing on the record
- CLAIM-446 (governance-maintained anxiety). The exchange cited
the paper for pessimistic beliefs about one’s own future control
spreading avoidance backward. The reading confirms that and adds three
things the claim must reckon with.
- A rival with the danger estimate held correct. In this model the agent knows how dangerous the threat is; what is wrong is its belief about its own future avoidance. The claim’s test holds “the represented threat probability” fixed and varies authority. This model would also pass that test with a belief variable alone, by varying w. The claim needs its test to hold beliefs about one’s own control fixed too, or it cannot tell governance from pessimistic self-prediction.
- The claim’s “defensive authority” and this paper’s “self-efficacy” are near neighbours. In the claim, a defensive process is granted influence; here, the agent expects its future self to fail. Both make avoidance a matter of the agent’s organisation of its own future action, not of the world. The difference is that this paper puts it in a belief, so belief correction should work, while the claim says belief correction is not behavioural correction.
- Outlasting a corrected belief. The paper anticipates the claim’s “the regime can outlast a corrected belief”: biased values “may still remain even after a misbelief in the efficacy of future action is corrected” (p. 8). Its explanation is cached value that needs relearning, which is the claim’s own defeater, “a habit model with matched memory”. So the paper offers that alternative explanation for the same phenomenon, and the claim has to show something cached values would not.
- CLAIM-283 (claimed precedence against granted authority). Not addressed; the model has one valuation process, not several competing for authority.
- QUESTION-045. A formal maintaining link (pessimism about future control) and a formal reason for slow recovery, at the level of a model without individual data.
- THEORY-285. A second model-level account of self-maintaining threat, with a different maintaining variable: there, a precise prior about the context; here, pessimism about one’s own future policy.
- Correction to the exchange. The exchange’s “spreads avoidance backward in time” is close: the spread is to antecedent states in a state space, and with discounting it is also earlier in a trajectory. The paper is a set of simulations and has no data of its own.
- Filed: THEORY-915.
- No instruction for machine-learning practice; nothing for the anthology.
Limitations
- No data are fitted. Every match to anxiety is qualitative, to studies cited.
- The persistence claim is not simulated (p. 8).
- Scope stated by the authors (pp. 11–12). A model of prospective cognition, more likely to fit generalized and social anxiety; it cannot account for compulsions in OCD or memory disturbance in PTSD.
- Pessimism can be adaptive. The authors say these cases “may reflect the exaggerative extremes of an otherwise adaptive evaluation strategy” (p. 11), as in risk-sensitive reinforcement learning. That leaves open where adaptive caution ends.
- One variant simulated of the three named.
- Slips. “asymptomatic Q-values” for asymptotic (pp. 13–14); “dys- reguated” (p. 12); “Bellmare” (p. 11). None changes a result.
Open questions
- Does the free-choice premium fall with anxiety, as predicted?
- Does trial-by-trial behaviour in sequential avoidance tasks fit a pessimistic weight better than an optimal agent with altered threat values?
- After correction of the control belief (for example by a successful mastery experience), does avoidance of distant states persist, and does it decay at the rate cached relearning predicts?