nucleation

LIT-1519: Learning with Opponent-Learning Awareness

2 min read

Properties4
StatusActive — read in full 2026-10-24 (NOTE-1150) from arXiv v4 (19 September 2018, the AAMAS 2018 text, arXiv non-exclusive licence); worth reading as the first learning rule that models another learner's update and climbs its own return through it: a LOLA agent adds to its gradient the effect of its own parameters on the opponent's next naive step. Two exact-gradient LOLA learners reached tit-for-tat-like reciprocity in the iterated prisoner's dilemma where naive learners defected. Shaping is one step deep, assumes access to the opponent's parameters (or a behavioural clone of them), and is shown only in two-player games; its numbers have one internal inconsistency (see the NOTE).
Tagsgame-theory, learning-theory, anthology-candidate
Cited as “paper” byNOTE-1150 — Foerster et al. — Learning with opponent-learning awareness
Cited as “source” byTHEORY-931 — A learner that ascends its own return after one anticipated naive gradient step of its opponent, differentiating through that step, reached tit-for-tat-like reciprocity in 81% of fifty exact-gradient runs of the memory-one iterated prisoner's dilemma, where pairs of naive gradient learners ended at the mutual-defection return, and settled at the mixed equilibrium of iterated matching pennies where naive learners did not converge; shown for two players, one-step shaping and access to the opponent's parameters

Graph View

Foerster, Chen, Al-Shedivat, Whiteson, Abbeel & Mordatch (2018), AAMAS — arXiv:1709.04326

Key takeaways

  • The rule. “each agent shapes the anticipated learning of the other agents in the environment” (abstract). The opponent is modelled as one naive gradient step; the dependence of ∇θ²V¹ on θ¹ is dropped (p. 3). Re-derived when read.
  • Reciprocity from self-interest. Naive exact-gradient learners ended at mutual defection (−1.98 per step); LOLA pairs played tit-for-tat in most runs (−1.06) (Table 3). “by explicitly considering the learning of the other agent, LOLA agents learn to cooperate out of self-interest” (abstract).
  • Shaping pays against the shaped. LOLA earned −1.28 against a naive learner’s −1.54, and a second-order LOLA gained nothing against first-order LOLA (Table 4).
  • What it needs. The opponent’s parameters, or a maximum-likelihood clone of them from its trajectories (Section 4.4); with the clone, Coin Game cooperation fell from about 80% to about 60% own-coin pickups (Figure 5).
  • What it does not show. Convergence or stability guarantees, n > 2, or exploitation by methods other than gradient ones, which the authors leave open (p. 8).

Why this is here

Registered 2026-10-24 by the lead as “a learning-aware model of authority shaping others’ learning”, bearing on CASE-124 and CLAIM-448. The reading supports that as an analogy with a precise mechanism (NOTE-1150): a LOLA learner treats another learner’s update rule as part of its environment and moves its own parameters partly for their effect on that update. CASE-124’s ε arrow, authority moving the regime that shapes what others learn, has the same form. What it does not supply is the record’s asymmetry: LOLA is symmetric between peers, each shaping the other, and its result is that mutual shaping can produce reciprocity, not capture. Filed as THEORY-931. It carries anthology-candidate as a multi-agent learning method and no instruction for practice here.

Backlinks