Source
Foerster, Farquhar, Afouras, Nardelli & Whiteson, LIT-1517, read in NOTE-1152 from arXiv:1705.08926v3: Eq. 4 (p. 4), Lemma 1 and its proof (Appendix A), and the errata.
What was actually shown
- The baseline. b^a(s, τ, u^−a) = Σ_{u′a} π^a(u′a | τ^a) Q(s, τ, (u^−a, u′a)): the critic’s value averaged over agent a’s own alternatives under its policy, every other agent’s action as taken.
- Zero expectation. Its contribution to the expected gradient is −Σ p(s, τ) Σ π^−a Σ_{u^a} π^a ∇log π^a b^a = −Σ … b^a ∇Σ_{u^a} π^a = 0, because b^a is constant in u^a (Eqs. 9–13). Re-derived when filed.
- Convergence. With the critic conditioned on (s, τ), the joint policy is a single-agent policy on an MDP over (s, τ) and Sutton et al.’s (1999) actor-critic result applies (Lemma 1, as corrected in v3).
- Empirically, on four StarCraft maps the counterfactual baseline beat a V baseline with the same critic in speed and final win rate (Table 1).
What could have come out otherwise: a per-agent baseline that depended on the agent’s own action would bias the gradient; this one is built not to.
What this does not say
- That the counterfactual is causal or simulated. It is the critic’s estimate, and the critic learns an action’s value only from occasions it was taken; the policy keeps every action above ε/|U| in training.
- That it values a process that never acts. Every agent acts every step. A never-sampled alternative is valued by the critic’s extrapolation alone.
- That it answers “what if the agent had not acted”. That is the difference reward’s default action, which COMA replaces by the agent’s own average.
- That COMA converges in practice. Lemma 1 assumes a converged, compatible TD(1) critic; the experiments use TD(λ) with a deep critic.
Note of 2026-10-30: the second check
The second check at the status review (ADR-065 §5) followed the step. The actors are decentralised, so the joint policy is the product Π_b π^b(u^b | τ^b), and u^a is independent of u^−a given τ. The baseline’s contribution to the expected gradient is therefore Σ p(s, τ) Σ_{u^−a} π^−a(u^−a | τ) b^a(s, τ, u^−a) Σ_{u^a} π^a(u^a | τ^a) ∇log π^a(u^a | τ^a). The inner sum is Σ_{u^a} ∇π^a(u^a | τ^a) = ∇ Σ_{u^a} π^a(u^a | τ^a) = ∇1 = 0. The baseline can be taken out of it because it does not vary with u^a; it multiplies the score and is not itself differentiated. Both conditions do work: the baseline must not depend on the agent’s own action, and the agent’s action must be independent of the others’ given τ.
A script (numpy, in the session’s scratchpad) computed the exact expected gradient of one agent’s softmax logits, with and without the baseline, in five random games of three agents, three actions and four states. The two differed by at most 4 × 10⁻¹⁷, against gradients of 0.02 to 0.07. Two controls moved it: a baseline that also depended on the agent’s own action changed the gradient by 0.047, and a joint policy that was not a product (agent a copying another agent half the time) changed it by 0.055.