nucleation

LIT-1518: Eliciting Informative Feedback: The Peer-Prediction Method

2 min read

Properties4
StatusActive — read in full 2026-10-24 (NOTE-1139) from the authors' final pre-publication draft of 26 January 2005, on Paul Resnick's University of Michigan page; worth reading as the founding scoring rule for evaluators when there is no ground truth. Each rater reports a signal, the center turns it into a posterior over a reference rater's report, and the rater is paid a strictly proper scoring rule on that posterior against what the reference rater actually reported. Truth-telling is then a strict Nash equilibrium (Proposition 1), and scaling the payments buys effort. It is not the only equilibrium: everyone always reporting h, or always l, is an equilibrium too, and collusion is named as a limit the paper does not settle. The mechanism needs a common prior and a signal model the center knows.
Tagsgame-theory, social-science, epistemology
Cited as “paper” byNOTE-1139 — Miller, Resnick and Zeckhauser — Eliciting informative feedback: the peer-prediction method
Cited as “source” byTHEORY-943 — When a center with no ground truth pays each rater a strictly proper scoring rule on the distribution of a reference rater's report implied by her own report, under a common prior and a known signal model in which raters' signals are stochastically relevant to one another, truthful reporting is a strict Nash equilibrium, but not necessarily the only one: in the authors' two-signal example under the log rule, every rater always reporting h, or always reporting l, is also an equilibrium

Graph View

Miller, N., Resnick, P. & Zeckhauser, R. (2005). Eliciting informative feedback: The peer-prediction method. Management Science 51(9): 1359–1373. doi:10.1287/mnsc.1050.0379.

Key takeaways

From the reading, NOTE-1139. Page numbers are the January 2005 draft’s.

  • Scoring without ground truth. The center “has no independent information, so its scoring decisions can depend only on the information provided by other raters” (p. 4). A rater is scored “not on agreement between the ratings, but on a comparison between the likelihood assigned to the reference rater’s possible ratings and the reference rater’s actual rating” (p. 3).
  • The result. With stochastic relevance and any strictly proper scoring rule, truthful reporting is a strict Nash equilibrium (Proposition 1, p. 8). A scale factor makes effort worth its cost (Proposition 2) and can be set to overwhelm a bribe; a second rater’s base transfer balances the budget.
  • Not unique. “There may be non-truthful equilibria” (p. 9): in the worked example, everyone always reporting the same signal is an equilibrium too. The authors call the truthful one focal and suggest the center punish detected uninformative equilibria.
  • Limits named. Collusion (p. 23), risk aversion, raters’ private information the center does not use (the “Yes-Man” case, p. 24), coarse reports beyond two types, and raters’ trust that the scheme rewards honesty.

Why this is here

Filed on 2026-10-24 for the record’s governance line, as a mechanism that scores evaluators when there is no objective metric. It bears on the authority-dependent scoring term of CASE-155 and on CLAIM-443’s worry that what is rewarded entrenches (NOTE-1139). Filed THEORY-943.

Backlinks