nucleation

THEORY-1231: When each agent's Q-function is approximated as an average of pairwise terms and expanded about the mean action of its neighbours, the first-order term vanishes and the approximation error is a second-order remainder bounded by the curvature of the pairwise Q-function, leaving a two-body problem between an agent and its neighbours' average; with homogeneous agents, mean field Q-learning converges to the Nash Q-value under assumptions the authors call strong, and learned tasks of 500 and 1000 agents where independent and centralized-critic baselines failed

1 min read

Properties4
StatusProposed
Tagsgame-theory, learning-theory, complex-systems
SourceLIT-1512 — Mean Field Multi-Agent Reinforcement Learning
ExtendsTHEORY-945 — When each agent's Q-function is approximated as an average of pairwise terms and expanded about the mean action of its neighbours, the first-order term vanishes and the approximation error is a second-order remainder bounded by the curvature of the pairwise Q-function, leaving a two-body problem between an agent and its neighbours' average; Yang et al. state that with homogeneous agents mean field Q-learning converges to the Nash Q-value under three assumptions, one of which they call a strong constraint, by a proof whose contraction step needs a temperature above K/2 while its greedy-in-the-limit assumption needs the temperature to fall to zero; in their Gaussian Squeeze runs with 500 and 1000 agents, mean field Q-learning and actor-critic learned the optimal allocation where independent learners, FMQ, Rec-FMQ and a MADDPG variant failed

Graph View

Source

Yang, Luo, Li, Zhou, Zhang & Wang (2018), LIT-1512, read in NOTE-1148 from arXiv:1802.05438v5.

What is shown and what is not

THEORY-945 holds the expansion and the bound, re-derived, and reports the theorem and the Gaussian Squeeze runs. This entry asserts the theorem, and asserts the runs as the method’s general capacity. Against the theorem: Proposition 1, which the proof uses, needs the Boltzmann temperature above K/2, while Assumption 2 needs it to go to zero, and the paper does not reconcile them (NOTE-1148). Against the general capacity: the experiment at 500 and 1000 agents is one task, run by the authors.

Backlinks