nucleation

THEORY-1234: The replicator-plus-entropy model of Boltzmann Q-learning in two-player stateless games is exact for frequency-adjusted Q-learning, not for Q-learning, whose expected update carries an extra factor of each action's selection probability and whose trajectories in 2×2 games depart from the model depending on initialization

1 min read

Properties4
StatusProposed
Tagsgame-theory, learning-theory
SourceLIT-1511 — Frequency adjusted multi-agent Q-learning
ExtendsTHEORY-932 — The replicator-plus-entropy model of Boltzmann Q-learning in two-player stateless games is, in the small-step limit, the expected dynamics of a learner that updates every action's value each step, and so of frequency-adjusted Q-learning wherever every action's probability is at least its adjustment parameter; it is not that of Q-learning, whose expected update carries an extra factor of each action's selection probability, so that its policy dynamics depend on the Q-values; in Kaisers and Tuyls's three 2×2 games at one temperature, compared by eye, Q-learning's trajectories departed from the model depending on initialization and the frequency-adjusted learner's followed it

Graph View

Source

Kaisers and Tuyls (2010), LIT-1511, read in full in NOTE-1140. The model they correct is Tuyls, Verbeeck and Lenaerts (2003), LIT-1501, which is Unavailable and is known here only as Kaisers and Tuyls restate it.

What is shown and what is not

THEORY-932 holds what is proved. The model is the expected dynamics of an all-update learner, and so of frequency-adjusted Q-learning wherever every probability is at least β. Q-learning’s expected update carries a factor x_i. This entry asserts more. It says “exact” without the small-step and x_i ≥ β conditions, and it says the departure holds in 2×2 games generally. The paper’s evidence for that is Fig. 4, three games at τ = 0.1 with no measure of fit. A check of the expected dynamics at the status review of 2026-10-30, three games with payoffs of its own at τ = 0.1 and 1, found departures of 0.02 to 0.31 and dependence on initialization from 0.00 to 0.31. That is not the quantitative comparison of sampled learners that the condition asks for.

Backlinks