nucleation

LIT-1520: Coupled replicator equations for the dynamics of learning in multiagent systems

2 min read

Properties4
StatusActive — read in full 2026-10-24 (NOTE-1144) from arXiv:nlin/0204057v2 (18 August 2002); worth reading as a four-page derivation of the learning dynamics of softmax reinforcement learners with forgetting: two such agents follow coupled replicator equations plus a term α_X Σ_j x_j log(x_j/x_i), set by the rate at which reward memories decay, that pushes each agent towards uniform play. Without forgetting the system is conservative, Hamiltonian in zero-sum games, and in rock–scissors–paper it is chaotic for nonzero tie rewards; with forgetting it is dissipative, with limit cycles and chaotic attractors. The memory update is not Watkins's Q-learning: every action's memory is updated with its average reward at the same rate.
Tagsgame-theory, learning-theory, complex-systems, anthology-candidate
Cited as “paper” byNOTE-1144 — Sato & Crutchfield — Coupled replicator equations for the dynamics of learning in multiagent systems
Cited as “source” byTHEORY-928 — Two softmax learners that update every action's reward memory each step with its average reward against the other's current play, the memories decaying at rate α, follow, in the limit of learning slow relative to play, coupled replicator equations plus a term α Σ x_j log(x_j / x_i) toward uniform play; in a two-player matrix game, without decay the dynamics preserve phase volume in log-ratio coordinates and are Hamiltonian when the game is zero-sum with an interior Nash equilibrium, and with decay the volume contracts at rate (N − 1)(α_X + α_Y), THEORY-936 — In two-agent rock–scissors–paper played by softmax reinforcement learners, nonzero tie rewards give quasiperiodic tori and Hamiltonian chaos without memory decay, and limit cycles and chaotic attractors with it, over a large share of the tie-reward range simulated

Graph View

Sato, Y. & Crutchfield, J. P. (2003). Coupled replicator equations for the dynamics of learning in multiagent systems. Physical Review E 67: 015206. doi:10.1103/PhysRevE.67.015206. arXiv:nlin/0204057.

Key takeaways

From the reading, NOTE-1144. Page numbers are arXiv v2’s.

  • Learning gives a game dynamics. Two agents with no model of each other, each choosing by softmax over decaying reward memories, follow coupled replicator equations in the slow-learning limit. The agents “have no knowledge of a “game” in which they are playing”, yet a game dynamics emerges (p. 2).
  • Forgetting is an entropy term. Memory decay at rate α adds α Σ_j x_j log(x_j/x_i) to each action’s growth rate, a pull towards uniform play. With α = 0 the system is the conservative multipopulation replicator equation, Hamiltonian in zero-sum games, and the time average of an interior orbit is the Nash equilibrium. With α > 0 phase volume contracts at rate (N − 1)(α_X + α_Y) (THEORY-928).
  • Chaos in a three-action game. Rock–scissors–paper with tie reward ε: tori and Hamiltonian chaos when the game is zero-sum, transients to heteroclinic cycles when it is not, and with decay limit cycles and chaotic attractors, the largest Lyapunov exponent positive over a large part of the ε_Y range (THEORY-936).
  • Not Watkins’s Q-learning. Every action’s memory is updated each step with its average reward. That is the full-information case the selection–mutation model of Q-learning assumes, not the case in which only the chosen action is updated (NOTE-1140).

Why this is here

Filed 2026-10-24 when the owner asked how the record stands on multi-agent reinforcement learning. It is the learning-side derivation of the coupled dynamics the record’s replicator reading of authority uses (CLAIM-444), it shows that forgetting enters those dynamics as a force towards equal shares, and it is a demonstrated case of cycles and chaos produced by self-interested learners with no switching of regime (CLAIM-451). It carries anthology-candidate because it is offered as a framework for analysing multi-agent learning systems. It carries no instruction for ML practice.

Backlinks