Contribution
Learners that each follow their own gradient “simply treat the other agent as a static part of the environment” (p. 1), and in the iterated prisoner’s dilemma they learn to defect. LOLA gives a learner a model of the other’s learning and lets it choose its own parameters partly for their effect on the other’s next update. “To the best of our knowledge, LOLA is the first method that aims to shape the learning of other agents in a multi-agent RL setting” (p. 3).
Key insight
Replace agent 1’s objective V¹(θ¹, θ²) with V¹(θ¹, θ² + Δθ²), where Δθ² = η∇θ²V²(θ¹, θ²) is the opponent’s naive step. To first order this is V¹ + (Δθ²)ᵀ∇θ²V¹. Differentiating in θ¹ while holding ∇θ²V¹ fixed gives the LOLA update
θ¹ ← θ¹ + δ∇θ¹V¹ + δη (∇θ²V¹)ᵀ ∇θ¹∇θ²V² (Eq. 4.4),
the naive gradient plus a term that moves θ¹ in whatever direction makes the opponent’s next step better for agent 1 (re-derived when read). The dependence of ∇θ²V¹ on θ¹ “is dropped during the backward pass” (p. 3), a choice the authors leave for future work.
Assumptions
- Two agents in the derivations, said to generalise (p. 3).
- The opponent is a naive gradient learner with a known step size; first-order LOLA models nothing deeper (Section 4.5).
- Access to the opponent’s parameters, or a behavioural clone of them fitted by maximum likelihood to its trajectories (Section 4.4).
- Exact values for the iterated games: memory-one policies of five probabilities each, with V computed in closed form (supplement A.2); policy-gradient estimates elsewhere, with learning only between batches of 4000 episodes (p. 5).
- Discounting γ = 0.96 for the IPD and Coin Game, 0.9 for matching pennies (p. 6).
Key results
-
IPD and IMP (Table 3, p. 6). Fifty runs each.
IPD %TFT IPD return IMP %Nash IMP return (std) NL-Ex 20.8 −1.98 0.0 0 (0.37) LOLA-Ex 81.0 −1.06 98.8 0 (0.02) NL-PG 20.0 −1.98 13.2 0 (0.19) LOLA-PG 66.4 −1.17 93.2 0 (0.06) Mutual defection returns −2 per step and tit-for-tat −1 (p. 5).
-
Tournament (Figure 4). Against naive Q-learning, joint-action learning, policy hill-climbing and WoLF-PHC, LOLA-Ex had the highest average return in the IPD and middling, within error bars, in IMP.
-
Shaping and exploitation (Table 4, p. 8). NL-Ex vs NL-Ex −1.99 each; LOLA-Ex vs NL-Ex −1.28 for LOLA, −1.54 for the naive learner; LOLA vs LOLA −1.04 each; 2nd-order LOLA vs LOLA −1.14 and −1.17. “there is an incentive for either agent to switch from naive learning to first order LOLA” (p. 7), and the higher order gains nothing.
-
Coin Game (Figure 5). LOLA-PG pairs picked up their own colour about 80% of the time, naive pairs indiscriminately; with opponent modelling about 60%, “less stable”.
Claims
| id | claim | strength | support |
|---|---|---|---|
| C1 | Differentiating through the opponent’s anticipated naive step gives Eq. 4.4 | derived | Section 4.2; re-derived |
| C2 | The second-order term has an unbiased policy-gradient estimator | derived | supplement A.1; checked |
| C3 | Two exact LOLA learners reach TFT-like reciprocity in the memory-one IPD in most runs; naive learners defect | moderate: 50 runs, one game | Table 3, Figure 1 |
| C4 | LOLA settles at the mixed Nash equilibrium of iterated matching pennies; naive learners do not converge | moderate | Table 3, Figure 2 |
| C5 | In the IPD both agents gain by switching to LOLA, and second-order LOLA does not exploit it | preliminary: one game, one step size | Table 4; “initial results” (p. 8) |
| C6 | LOLA scales to deep recurrent policies and partially to opponent modelling | weak: one grid game, 10 runs | Figure 5 |
Concepts
- Naive learner: one that ascends its own value at the current joint parameters, the other’s held fixed (Eq. 4.1).
- Opponent-learning awareness: optimising one’s return after the other’s anticipated update, and differentiating through it.
- Opponent shaping: the effect of the added term, changing the direction of the other’s learning to one’s own advantage; the paper’s word is “shape”.
- Higher-order LOLA: modelling the opponent as itself a LOLA learner, which needs third derivatives (Section 4.5).
Connections
Related to opponent modelling and fictitious play, but those predict and best-respond and “do not address the learning dynamics of opponents” (p. 2). Closest in setting to Lerer & Peysakhovich’s hand-built tit-for-tat; in spirit to Metz et al.’s unrolled GANs, which also differentiate through an opponent’s steps. Zhang & Lesser predict the opponent’s update but do not differentiate through it. Later work on the consistency and stability of LOLA exists but was not read here.
Bearing on the record
- The lead’s description, “a learning-aware model of authority shaping others’ learning” (CASE-124, CLAIM-448), holds as an analogy with a precise mechanism. CASE-124’s ε arrow has authority move the regime through which others’ evidence is read. LOLA’s second-order term is a formal version of acting for one’s effect on another’s update rather than on the present outcome. The match is the form of the derivative. The difference is the channel: LOLA acts through its own policy, which is the other’s environment, not through a regime that filters the other’s evidence.
- But the paper’s setting is symmetric. Both agents shape and both are shaped. Its headline result is that mutual shaping produced reciprocity, not capture. The one asymmetric pairing, LOLA against a naive learner, left both better off than two naive learners (−1.28 and −1.54 against −1.99), with the shaper ahead. So it does not support the record’s worry that a shaping process entrenches itself; it shows that shaping another’s learning can be the route to cooperation. Nothing here models a shaper with authority over the other’s evidence.
- CLAIM-448 (selective contestation). Complicates it slightly. LOLA is a case in which one party deliberately acts on another’s learning and the outcome is better for both. Influence on the other’s learning is not by itself evidence of suppression; what the claim needs is influence on which contests are recorded.
- CLAIM-451, TERM-077 and THEORY-894 (cycles). A direct link, mine. Matching pennies is the textbook harmonic game, all cycle and no potential, and naive gradient learners there “fail to converge” (p. 7). LOLA learners settled at the mixed equilibrium (98.8% of exact runs). The authors’ reading: “LOLA agents anticipate that exploiting a deviation from Nash increases their immediate return, but also renders them more exploitable by the opponent’s next learning step” (p. 5). So modelling another’s learning is one way the cycling of a nonintegrable negotiation can be relieved: a fourth candidate beside CLAIM-451’s accommodation, entrenched precedence and cheap coordination, or a form of accommodation. Shown in one two-action game.
- CLAIM-444 and CLAIM-443 (authority dynamics; a learned prior over governance). A model class the record lacks: processes whose influence on each other runs through each other’s learning rules, not only through scores. The replicator reading treats processes as scored by a fixed environment; LOLA is the case where the environment includes a learner one can steer.
- No instruction for machine-learning practice is drawn here.
Limitations
- Small games and one-step shaping. Two players, memory-one iterated games, one grid game. The opponent is modelled one naive step ahead.
- An inconsistency in Table 3. NL-Ex is reported as 20.8% tit-for-tat, yet its return is −1.98, the mutual-defection value, and the text says naive agents “learn to defect in all states” (p. 7). A fifth of runs at tit-for-tat would raise the mean return well above −1.98. NL-PG has the same pattern. What the %TFT column counts for naive learners is unclear.
- Exploitability is tested only against gradient learners and at one step size (δ = 0.5); the authors expect LOLA to be exploitable by other means (p. 8).
- Opponent parameters. The main results assume them; the clone is noisier and works worse (Figure 5).
- A slip. The IPD transition matrix in supplement A.2 has (θ¹ − 1)θ² for (1 − θ¹)θ². It does not affect the results.
Open questions
- Does opponent shaping produce entrenchment when it is asymmetric in capacity, one side modelling the other’s learning and the other not, or one side controlling what the other observes? The paper’s one asymmetric pairing is too small to say.
- What happens with more than two learners, where shaping one may disadvantage a third?