nucleation

LIT-1510: QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning

2 min read

Properties4
StatusActive — read in full 2026-10-24 (NOTE-1141) from arXiv v1 (14 May 2019, the ICML version with its supplement, arXiv non-exclusive licence); worth reading for the Individual-Global-Max condition, that the agents' individually greedy actions form a greedy joint action, and for showing it strictly weaker than additive or monotonic factorisation. The authors' own supplement notes that every single-state matrix game satisfies it, so IGM has bite only where each agent sees part of the state; it is a condition on whether members acting on local information can jointly reach the collective optimum, not a condition for a common currency. Evidence is matrix games, Gaussian squeeze and a predator-prey variant.
Tagsgame-theory, learning-theory, anthology-candidate
Cited as “paper” byNOTE-1141 — Son et al. — QTRAN: learning to factorize with transformation
Cited as “source” byTHEORY-929 — A joint action-value that is a non-decreasing function of per-agent values, each conditioning only on its agent's own history, makes the profile of the agents' individually greedy actions a greedy joint action, which suffices for decentralised greedy execution and is strictly more expressive than a sum; it cannot represent a joint value in which an agent's best action depends on what the others do at the same step, THEORY-933 — Individual-Global-Max, that the agents' individually greedy actions form a greedy joint action, is strictly weaker than additive or monotonic factorisation: every single-state matrix game with a unique optimum satisfies it, yet in 177 of 310 random three-by-three two-agent games value-decomposition and monotonic mixing failed to learn the optimal joint action that an IGM-only method found; a sufficient condition is that the sum of individual values, shifted by a state value, equals the joint value at the greedy profile and bounds it from above elsewhere

Graph View

Son, Kim, Kang, Hostallero & Yi (2019), ICML — arXiv:1905.05408

Key takeaways

  • IGM. Individual value functions [Q_i] “satisfy IGM” for Q_jt when argmax over joint actions of Q_jt equals the vector of each agent’s argmax of Q_i (Definition 1). “Simply put, the optimal joint actions across agents are equivalent to the collection of individual optimal actions of each agent” (p. 3).
  • Additivity and monotonicity are only sufficient. “Additive decomposability and monotonicity are only sufficient — somewhat excessively restrictive — for factorizability” (p. 1).
  • A condition that is sufficient and, up to rescaling, necessary. Σ_i Q_i(u_i) − Q_jt(u) + V_jt equals 0 at the greedy profile and is ≥ 0 elsewhere (Theorem 1, proof re-derived when read); a stronger variant (Theorem 2) keeps the transformed value tracking the joint one for non-optimal actions.
  • IGM is cheap in single-state games. “Since matrix games always satisfy IGM conditions, QTRAN-base always trains the optimal action for all 310 cases” (supplement, p. 17). VDN and QMIX matched it in 114 and fell short in 177.
  • Non-monotonic tasks. In predator-prey with a penalty P for a lone capture, the gap to VDN and QMIX widened as P grew; with four predators and P ≥ 1 they learned to avoid the prey (Figure 4).

Why this is here

Registered 2026-10-24 by the lead for IGM as “a precise condition for a collective common currency” (CLAIM-436, CLAIM-440). The reading narrows that (NOTE-1141). IGM presupposes a real-valued joint value Q_jt, so it is not a condition for a currency; it is a condition for decentralising a currency already given. And because any single-state game with a unique optimum satisfies it, its content lies in partial observability: whether each member’s local view suffices to pick its part of the collective optimum. Its nearest record analogue is legibility and access (CASE-124, CLAIM-403), not ordering. Filed as THEORY-933. It carries anthology-candidate as a multi-agent RL method and no instruction for practice here.

Backlinks