nucleation

LIT-1517: Counterfactual Multi-Agent Policy Gradients

2 min read

Properties4
StatusActive — read in full 2026-10-24 (NOTE-1152) from arXiv v3 (11 December 2024, arXiv non-exclusive licence), the corrected version; worth reading as the standard statement of counterfactual credit in cooperative multi-agent learning: each agent's action is scored against the critic's average over that agent's own alternatives, the others' actions held fixed, and that baseline provably leaves the expected gradient unchanged. The counterfactual values are the learned critic's estimates, which exist only for actions the agents sometimes take, so the method credits agents that all act on every step and does not recover the value of one that is never heeded. Results are StarCraft win rates on four small maps.
Tagsgame-theory, learning-theory, anthology-candidate
Cited as “paper” byNOTE-1152 — Foerster et al. — Counterfactual multi-agent policy gradients
Cited as “source” byTHEORY-925 — In a cooperative stochastic game with decentralised actors and a centralised critic, a baseline for each agent that averages the critic's value over that agent's own alternative actions under its own policy, the other agents' actions held fixed, leaves the expected policy gradient unchanged, so each agent can be credited with the advantage of its actual action over that counterfactual average without a default action or an extra simulation; the counterfactual values come from the learned critic, not from the environment

Graph View

Foerster, Farquhar, Afouras, Nardelli & Whiteson (2018), AAAI — arXiv:1705.08926

Key takeaways

  • The counterfactual baseline. Each agent is scored by how much better its actual action was than the critic’s policy-weighted average over its own alternatives, with the other agents’ actions fixed (Eq. 4). The baseline marginalises out “a single agent’s action, while keeping the other agents’ actions fixed” (abstract). Because it does not depend on that agent’s action, its expected contribution to the gradient is zero (Appendix A), re-derived when read.
  • From difference rewards. It is offered as difference rewards (Wolpert & Tumer) without their two costs: a simulator to evaluate the counterfactual, and a user-chosen default action. “This advantage has the same form as the aristocrat utility” (p. 4).
  • The counterfactual is the critic’s. The values of untaken actions come from a learned critic; the policy keeps every action’s probability above ε/|U| during training (p. 6). Nothing is simulated.
  • Results. Mean final win rates on 3m, 5m, 5w and 2d_3z of 87, 81, 82 and 47%, above central-QV (83, 71, 76, 39), which differs only in using V as the baseline (Table 1).
  • Corrected proof. v3’s erratum: the critic must condition on the joint history as well as the state for Lemma 1 to follow from Sutton et al. (1999).

Why this is here

Registered 2026-10-24 for the governance line: the lead read counterfactual baselines as answering “what if this process had not acted”, which would bear on CLAIM-446’s identifiability step and on crediting unheeded processes (QUESTION-270, QUESTION-271’s fourth failure).

Corrected 2026-10-24, from the reading. COMA does not ask what would have happened had an agent not acted. It asks how the agent’s actual action compares with its own average alternative, under its current policy, holding everyone else’s actual actions fixed. Every agent acts on every step, and the alternatives are valued by a critic trained on experience in which they were taken. So it addresses the noise of a shared reward among processes that all act, not the identifiability of a process whose alternative is never sampled. A process never heeded has no data from which a critic could value it, which is CLAIM-446’s trap restated, not escaped. Read in NOTE-1152; filed as THEORY-925. It carries anthology-candidate as a method of multi-agent RL and no instruction for practice here.

Backlinks