nucleation

LIT-1506: Optimal Reward Functions in Distributed Reinforcement Learning

2 min read

Properties4
StatusActive — read in full 2026-10-24 (NOTE-1153) from the NASA Technical Reports Server copy (document 20010066523, a 7-page scanned report marked public use permitted), not the version in the Intelligent Agent Technology proceedings; worth reading as the most compact statement the record holds of difference rewards. A member's private utility is factored when it ranks the member's own alternatives as the world utility does, the others held fixed; the Wonderful Life Utility, world utility minus its value with the member's effect set clamped, is factored and learns faster than the world utility itself. Clamping to the average action did better than clamping to the member's absence. The optimal clamp is only sketched ("too long to go through here") and the experiment is one bar problem.
Tagscomplex-systems, game-theory, anthology-candidate
Cited as “paper” byNOTE-1153 — Wolpert & Tumer — Optimal reward functions in distributed reinforcement learning
Cited as “source” byTHEORY-939 — In a multi-agent system run against a world utility, a private utility is factored, ranking each agent's own alternatives with the others fixed as the world utility ranks them, exactly when it is an increasing function of the world utility given the components the agent cannot affect; the world utility minus its value with the agent's effect set clamped is factored, and in a sixty-agent bar problem clamping to the average action learned better than clamping to absence or using the world utility itself

Graph View

Wolpert & Tumer (2000; Intelligent Agent Technology, 2001) — ntrs.nasa.gov/citations/20010066523

Key takeaways

  • Factored. A system is factored if for each member η, γ_η(C(ζ₀)) ≥ γ_η(C(ζ′₀)) ⇔ G(C(ζ₀)) ≥ G(C(ζ′₀)) for initial states that differ only for η (p. 3). An ordinal condition: the member’s private utility and the world utility order the member’s own alternatives alike.
  • The characterisation. Factored iff each γ_η is an increasing function of G given the components outside η’s effect set (Theorem, Eq. 1, p. 3; proof deferred to the authors’ earlier work).
  • The Wonderful Life Utility. WLU_η = G(ζ) − G(CL_{S_η^eff}(ζ)), the world utility minus its value with η’s effect set clamped to a fixed vector, conventionally zero. “To evaluate the WLU one does not need to know how to calculate the system’s behavior under counter-factual starting conditions” (p. 3): it is computed on the actual worldline.
  • Learnability. A factored utility can still be hard to learn when G depends on everyone; the WLU’s subtraction removes noise from others (Eq. 2, p. 4).
  • The clamp. Clamping to the average action (1/7 per night) did best across one to six nights; clamping to absence was close for few nights and dropped slightly as nights increased; clamping to attending every night did poorly for few nights; G did worst throughout (Figure 1a, pp. 5–6).

Why this is here

Filed 2026-10-24 in place of LIT-1515, for difference rewards, which the lead read as answering “what if this process had not acted” (CLAIM-446, QUESTION-270, QUESTION-271’s fourth failure). The reading corrects that in two ways (NOTE-1153). The counterfactual is not causal: the WLU is the world utility re-evaluated on the actual history with one member’s contribution replaced, the others’ behaviour held as it was, so it needs no model of how the system would have responded. And “had not acted”, clamping to zero, was not the best clamp; the expected action was. Its factoredness condition, in turn, is an ordinal alignment of each member’s utility with the collective’s, close to QMIX’s monotonicity (LIT-1503) and to CLAIM-436’s distinction between an ordering and a currency. Filed as THEORY-939. It carries anthology-candidate as a multi-agent RL method and no instruction for practice here.

Backlinks