Source
Yuzuru Sato and James P. Crutchfield, “Coupled replicator equations for the dynamics of learning in multiagent systems”, Physical Review E 67:015206 (2003), LIT-1520, read in full in NOTE-1144 from arXiv:nlin/0204057v2, Eqs. 1–13 (pp. 1–2).
What was actually shown
- The rule. Memories Q_i(n + 1) − Q_i(n) = R_i − αQ_i(n), choice x_i ∝ exp(βQ_i).
- The limit. With learning slow relative to play, so R_i is action i’s average reward against the other’s current mix, ẋ_i/x_i = β[R_i − Σ_j x_j R_j] + α Σ_j x_j log(x_j/x_i) (Eq. 6). The derivation is three lines, and the reading re-derived it: the decay term −αβ(Q_i − x·Q) is −α(log x_i − x·log x) under softmax.
- Structure. For bilinear rewards the phase volume in log-ratio coordinates contracts at rate (N − 1)(α_X + α_Y) (Eq. 10). At α = 0 the system preserves phase volume. If the game is zero-sum with an interior Nash equilibrium it is Hamiltonian (Eq. 11). The time average of an interior orbit is the Nash equilibrium (p. 2, citing Schuster et al.).
What this does not say
- That this is Q-learning. No states, no bootstrapping, and every action’s memory updated each step whether chosen or not. Kaisers and Tuyls show that the same equation describes Q-learning only after a frequency adjustment (THEORY-932).
- What the rest points are. It does not say so, but with α > 0 the interior rest points of Eq. 8 satisfy β(Ay)_i − α log x_i = const, that is, x is a softmax best response to y at temperature α/β: logit equilibria, not Nash equilibria. This is my reading of the equations.
- Anything for more than two agents beyond the linear three-agent extension.
Connections
- THEORY-938. The contrast that matters. Cross learning has no forgetting term and is absorbed in a vertex. Here forgetting is a force towards uniform play, and the boundary is repelling.
- CLAIM-443. The schematic rule’s forgetting term −λE has this reading under softmax selection: decay of a learned bias is pressure towards equal authority, and its rate sets how hard the system re-explores.
- CLAIM-444. The coupled, two-population case of the replicator reading, derived for learners.
Note of 2026-10-30: the re-derivation, written out
The second check at the status review (ADR-065 §5) followed each step.
-
The limit. Q̇_i = R_i − αQ_i, and x_i = e^(βQ_i)/Σ_j e^(βQ_j) gives ẋ_i/x_i = β(Q̇_i − Σ_j x_j Q̇_j). So ẋ_i/x_i = β(R_i − x·R) − αβ(Q_i − x·Q). The softmax gives βQ_i = log x_i + log Z, so β(Q_i − x·Q) = log x_i − x·log x. Hence ẋ_i/x_i = β(R_i − x·R) + α Σ_j x_j log(x_j/x_i), Eq. 6 with its printed sign. The term is positive for actions played less than the log-average, so it pulls toward uniform play.
-
Dissipation. In a matrix game, R^X = Ay. In u_i = log(x_{i+1}/x_1) the system is u̇_i = β_X[(Ay)_{i+1} − (Ay)_1] − α_X u_i (Eq. 9). The payoff part depends on v only, so ∂u̇_i/∂u_i = −α_X, and likewise ∂v̇_j/∂v_j = −α_Y. The divergence is −(N − 1)(α_X + α_Y). It is zero at α = 0 for any A and B.
-
Hamiltonian. Let B = −Aᵀ and let (p, q) be an interior equilibrium, so that Aq and Bp have equal entries. Then (Ay){i+1} − (Ay)1 = Σ_k C_ik (y{k+1} − q{k+1}), with C_ik = a_{i+1,k+1} − a_{1,k+1} − a_{i+1,1} + a_{11}. Since ∂/∂v_k Σ_j q_j log y_j = q_{k+1} − y_{k+1}, put H = (1/β_X) Σ p_i log x_i + (1/β_Y) Σ q_j log y_j. Then u̇ = P ∂_v H and v̇ = −Pᵀ ∂_u H with the constant matrix P = −β_X β_Y C. A constant skew-symmetric structure is a Poisson structure, so the flow is Hamiltonian. Without an interior equilibrium the payoff differences do not vanish at a common point, and the construction fails. The paper’s Eq. 11 prints H without the weights 1/β_X and 1/β_Y. As printed, it is conserved only when β_X = β_Y.
A script (numpy, in the session’s scratchpad, s928_sc.py) checked these steps in three parts.
- The limit. It iterated the stated memory rule with step h for a random 3 × 3 game and compared x with Eq. 6. The largest gap was 3.3 × 10⁻⁵ at h = 10⁻² and 3.3 × 10⁻⁶ at h = 10⁻³, falling in proportion to h. As a control, the same equation with the decay term’s sign flipped, as Eq. 14 prints it, missed by 3.4 × 10⁻² at h = 10⁻³.
- Dissipation. The divergence of Eq. 9, taken by finite differences at 200 random points with random games and decay rates, matched −(N − 1)(α_X + α_Y) to 4 × 10⁻¹⁰.
- The Hamiltonian. The game was rock–scissors–paper with tie rewards ε_X = −ε_Y = 0.5, with β_X = 1 and β_Y = 0.6, integrated to t = 20. The weighted H stayed constant to 2.5 × 10⁻¹⁴, and the vector field equalled J∇H at 100 random points to 2 × 10⁻⁹. Three controls each had to show drift, and each did. The unweighted H of Eq. 11 moved by 0.33. With α = 0.05, H moved by 0.69. In the non-zero-sum game ε_Y = 0.1 it moved by 0.93.
Note of 2026-10-30: the clauses the narrowing drops
The old title said that softmax learners with decaying memories follow the equation, with no condition on how memories are updated or how fast. Read so, it is false. Q-learning is softmax over decaying memories, yet it updates only the chosen action, and its expected dynamics carry a factor x_i that the equation lacks (THEORY-932). The old title also said “Hamiltonian in zero-sum games” with no condition, where the source states the result for an interior Nash equilibrium. Both conditions are the source’s own. These clauses were mis-statements, so they are recorded here only (ADR-047’s kind 3), and the title now states the conditions.
The version-of-record demand concerned Eq. 14. v2 prints the three-agent extension’s decay term as −α_X I_i^X, against Eq. 6’s +α_X I_i^X. Step 1 shows that, for the stated rule, Eq. 6’s sign is right. Whether the Physical Review E version corrects Eq. 14 is not checked, and nothing in the record uses the three-agent equation.
Note of 2026-10-30: Eq. 11 as printed
The paper’s Eq. 11 gives H = −(Σ_j x_j u_j + Σ_j y_j v_j) + log(1 + Σ_j e^(u_j)) + log(1 + Σ_j e^(v_j)), which is −Σ x_i log x_i − Σ y_j log y_j, with no weights. By step 3 of the re-derivation above, the conserved quantity is (1/β_X) Σ p_i log x_i + (1/β_Y) Σ q_j log y_j. The printed H is conserved only when β_X = β_Y. The script’s control showed this: with β_X = 1 and β_Y = 0.6 the unweighted H moved by 0.33 over t = 20, while the weighted H held to 2.5 × 10⁻¹⁴. The title asserts only that the flow is Hamiltonian, which holds with the weighted H.