nucleation

LIT-1505: Learning Through Reinforcement and Replicator Dynamics

2 min read

Properties4
StatusActive — read in full 2026-10-24 (NOTE-1149) from the authors' working paper of October 1995, as held in Levine's Working Paper Archive; worth reading as the proved source of "Cross learning is the replicator dynamic": a single Bush–Mosteller learner with positive payoffs moves in expectation exactly as a population under the discrete replicator dynamic, and in the continuous-time limit follows the continuous replicator dynamic on any finite horizon. Two limits matter as much as the result. Asymptotically the discrete learner is absorbed in a pure strategy profile with probability one at every learning speed, even in matching pennies where the replicator dynamic cycles; and with negative payoffs a second force, probability matching, enters beside the replicator term.
Tagsgame-theory, learning-and-conditioning
Cited as “paper” byNOTE-1149 — Börgers & Sarin — Learning through reinforcement and replicator dynamics
Cited as “source” byTHEORY-934 — Under Bush–Mosteller learning with negative payoffs, the continuous-time limit is a replicator term in benefits plus a probability-matching term, which alone drives choice probabilities to success probabilities, THEORY-938 — Cross learning has the expected motion of the replicator dynamic and follows it on any finite horizon in the continuous-time limit, but for every learning speed the discrete process converges with probability one to a pure strategy profile, even in games where the replicator dynamic cycles

Graph View

Börgers, T. & Sarin, R. (1997). Learning through reinforcement and replicator dynamics. Journal of Economic Theory 77(1): 1–14. doi:10.1006/jeth.1997.2319. Working paper, October 1993 and October 1995.

Key takeaways

From the reading, NOTE-1149. Page numbers are the 1995 working paper’s.

  • Expected motion is the replicator step. In Cross’s model a player who plays j and receives payoff U ∈ (0, 1) moves her mixed strategy a fraction U of the way to the vertex j. Her expected change is p_j(e_j U q − p U q), exactly the movement of a population in a biological model in which offspring equal payoffs (Propositions 1 and 3, pp. 8, 11).
  • The limit holds on finite horizons only. With step size θ → 0 and nθ → t, the learning process converges in probability to the continuous Taylor replicator dynamic at t (Proposition 4, via Norman’s theorem). But for every θ > 0 it converges with probability one to a pure strategy pair (Proposition 5), including matching pennies, where the replicator dynamic cycles (THEORY-938).
  • A population of ideas in one mind. Read through Estes’s stimulus sampling, the learner is a population of stimuli, one sampled each period and re-programmed in proportion to the payoff. The analogy is individual learning with biological evolution, not imitation in a population (Section 4).
  • Negative payoffs add probability matching. The limit is a replicator term in “benefits” plus a term that alone would equate each strategy’s choice probability with its probability of success, which is “often irrational behavior” (p. 5; THEORY-934).

Why this is here

Filed 2026-10-24 when the owner asked how the record stands on multi-agent reinforcement learning. It is the source the record’s replicator reading of authority (CASE-155, CLAIM-444) and Soma et al.’s Proposition 1 (THEORY-899) inherit “Cross learning is the replicator dynamic” from. Its asymptotic result is the record’s lock-in worry in a single learner: the expected dynamic can cycle or stay interior while every sample path is absorbed in a vertex.

Backlinks