NOTE-1050: Espeland and Sauder — Rankings and reactivity

Contribution

It turns reactivity, a methodological worry about measures contaminating what they measure, into a substantive object of study, and gives a vocabulary for it: two mechanisms that produce reactivity and three patterns of effects. Its case, U.S. News (USN) rankings of every accredited US law school, shows the measured changing what they do in response to being measured, and the measure becoming more valid as a result, not less.

Key insight

A public measure can make itself true. “We define self-fulfilling prophecies as processes by which reactions to social measures confirm the expectations or predictions that are embedded in measures or which increase the validity of the measure by encouraging behavior that conforms to it” (p. 11). Some of the channels close through the measure’s own inputs: a school’s rank moves applicants and yield, which move the selectivity score that makes up a quarter of the rank, and past ranks shape the reputation surveys that make up 40%. So agreement between a measure and the world it measures can be produced by the measure.

Assumptions

• The case is one measure in one field: USN’s law-school ranking, built from reputation (40%), selectivity (25%), placement (20%) and faculty resources (15%), standardized, weighted and rescaled to the top school’s score. USN ranks every accredited school and has no rival of comparable weight there, unlike business schools with “at least five high-profile rankings” (fn. 4). The authors chose the case for that density.
• The data are 136 open-ended interviews of about 45 minutes (two waves, 66 and 70; 39 deans or ex-deans, 49 other administrators, 33 faculty, 9 other staff, 6 leaders of associations), from 70 of about 190 schools across all four tiers, plus short interviews at an admissions fair (17 staff, 92 prospective students), 30 business-school interviews, chat rooms, school materials and reanalysed fieldwork. Interviews “were coded for general themes rather than for precisely defined variables; consequently, the frequency of particular responses is only an approximate indication of their importance.”
• The quantitative links (rank → applications and yield; past rank → reputation score) are from Sauder and Lancaster (2006) and Stake (2006), cited, not shown here.
• A mechanism is “like a causal switch that, if turned on, directs action in one way rather than another”, and the mechanisms are patterns of sense making.

Key results

• Reactivity defined. “individuals alter their behavior in reaction to being evaluated, observed, or measured.” It mediates two understandings of a measure, a neutral depiction and a vehicle of accountability, and “the job of measures is to harness reactivity” on the second.
• Self-fulfilling prophecy, four forms. Not restricted to false beliefs, “requiring only that an expectation, once defined as real, amplify or confirm its effect.”
  1. External audiences. Rank lists magnify “statistically insignificant differences”; small rank differences move applications and yield, which change the selectivity score “and in this way reinforce the original differences which were often largely a product of measurement noise.” Two elite schools: before rankings A lost about two thirds of cross-admits to B; after years of ranking slightly higher, A reportedly takes 80–90% of them. A faculty member: “They create artificial lines that then have the danger of becoming real.”
  2. Prior ranks in the reputation survey. Raters know little of most schools but their rank; Stake (2006) “finds that reputation scores slowly become aligned with overall rank”. Footnote 14 reports the contrary finding of Schmalbeck (2001) for the top 50 and the authors’ reading that the effect is slow and strongest lower down.
  3. Budgets by rank inside universities, which “reproduce and intensify the stratification they are designed to measure”; how often this happens “is unclear”.
  4. Realizing embedded assumptions. Schools become “more like what rankings measure, which reinforces the validity of the measure”: LSAT scores come to define merit, and an “access university” on the cusp of a tier weighs moving its low-LSAT admits to the part-time program, “which doesn’t count as much toward the rankings.”
• Commensuration. “the transformation of qualities into quantities that share a metric” (p. 16). It works “mainly by transforming cognition”, against self-fulfilling prophecy, which works by changing expectations, so its effects can be immediate where the other’s are gradual. Three features:
  • it simplifies, making much information irrelevant and imposing one form on the rest, and decontextualized numbers travel and “invite recontextualization” (law firms set interview thresholds by school rank, a use USN did not intend);
  • it unites and distinguishes, giving every school a precise relation to every other and to its own past, which makes “status claims not supported by rankings” hard to sustain;
  • it invites reflection on what numbers represent, distributed by position: prospective students tend to metrological realism, administrators who make the numbers to constructivism (Desrosières’s four attitudes).
• Three effects. Maximization (merit scholarships replacing need-based aid; a reported over-$100,000 a year on brochures to survey raters, with “no evidence that it works”); redefinition of work (all 16 career services staff reported spending more effort on tracking graduates’ employment); gaming, “efforts to improve ranking factors without improving the characteristics the factors are designed to measure” (29 of 177 schools in 1995 reported higher median LSATs to USN than to the ABA; reclassifying low scorers as part-time or probationary; spring leaves for faculty).
• Variation. Five sources: being on the cusp of a tier, position in the ranking, the dean’s commitment, geography, and mission. Distinctive missions “can sometimes resist rankings pressures”.
• Conclusion. The tension between validity and accountability is chronic: “We want the world to stand still for measurement and, only after, change in relation to it.” Strathern’s “When a measure becomes a target, it ceases to be a good measure” is quoted (p. 35). Categories are produced the same way: “the census produces ‘Hispanics,’ and the LSAT defines ‘merit.’” Coercion is suggested as a third mechanism, since the ranking is relative and zero-sum.

Claims

id	claim	strength	support
C1	Public measures are reactive: those measured change their behaviour in response, and the effects are mostly unintended	strong for the case, as the respondents describe it	136 interviews; corroborating documents and data
C2	Self-fulfilling prophecy can raise a measure’s validity by making the measured conform to it, and some of its channels run through the measure’s own inputs (rank → yield → selectivity; past rank → reputation survey)	moderate	the mechanism is argued from interviews; the quantitative links are cited from Sauder and Lancaster (2006) and Stake (2006), not shown, and Schmalbeck (2001) finds the survey effect weak for the top 50
C3	Commensuration produces reactivity by changing attention and cognition, through simplification, a single precise relation among all units, and scrutiny of what numbers represent	moderate	interview material and argument; no measurement of attention
C4	The effects sort into reallocation of resources, redefinition of work and gaming	moderate	a typology from coded interviews; frequencies “only an approximate indication”
C5	Ranking effects vary with tier position, rank, leadership, geography and mission, and distinctive missions can partly resist	weak to moderate	asserted from the interviews, without counts by school type
C6	Categories in general gain validity through use: as a category “acquires constituencies who use it”, those it covers develop shared interests and identities “that make the category more meaningful and ‘real‘“	weak here	footnote 10, citing Porter (1995) for “Hispanics”; not examined in the paper

Concepts

• Reactivity: individuals alter their behaviour in reaction to being evaluated, observed or measured; “a form of reflexivity”.
• Mechanism of reactivity: a pattern of sense making that directs the response, “like a causal switch”.
• Self-fulfilling prophecy (widened from Merton): reactions to a measure that confirm its embedded expectations, or raise its validity by inducing conformity. No false belief is required.
• Commensuration: transforming qualities into quantities on a shared metric, so that difference becomes “a matter of more or less, rather than kind”.
• Gaming: manipulating rules and numbers “in ways that are unconnected to, or even undermine, the motivation behind them”.

Connections

It extends Campbell’s (1957) reactive measure and Merton’s self-fulfilling prophecy, follows Heimer (1985) in treating reactivity as a continuous variable, and sets itself beside Callon’s and MacKenzie’s performativity, which it declines as its frame because reactivity keeps the validity of knowledge central. Commensuration is from Espeland and Stevens (1998) (LIT-1317, Unavailable here). Its accountability background is Power’s audit society, Strathern’s audit culture, Porter (1995) and Hacking’s Taming of Chance (1990). It does not cite Hacking’s interactive kinds or looping effect, or Scott (LIT-1321).

Bearing on the record

• CLAIM-402. The paper is an empirical instance of the claim’s mechanism, and stronger than its toy in one respect: the validity-raising channel is part of the definition of self-fulfilling prophecy here. A rank that predicts applicants’ choices and reputation scores has its success produced in part through its own use, and schools “become more like what rankings measure”. Every school in the field is under the same ranking, so the contexts in which it is confirmed are not independent, which is the claim’s point about contexts that share a category’s use. The claim’s “Espeland and Sauder’s reactivity … not read here” can now point to this reading. The join with Hacking’s interactive kinds (NOTE-769) is the record’s: the paper does not make it.
• QUESTION-028. Footnote 10 is a sociological statement of the effect the question has to control for: a category’s validity rises as constituencies use it. It is reported, not studied.
• CLAIM-342. The reputation survey is a loop whose evidence passes through the interpretation it tests: raters score schools by the rank the scores feed.
• CLAIM-406. Its loop has an institutional instance here: the standing that brings resources, applicants and reputation is produced by having had them, with no continuing act of anyone’s judgement needed (“It’s a self-fulfilling nightmare”, of the third tier). The paper does not show that the early differences were unmerited. It says they were “often largely a product of measurement noise”, on Sauder and Lancaster’s evidence.
• TERM-064. Commensuration is a legibility regime in the term’s sense. It makes schools legible to applicants, firms and universities by making most of what distinguishes them irrelevant: “the relations of relevancy and irrelevancy, visibility and invisibility, that are conferred.” The access school and the University of New Mexico’s Native American graduates are what the description adequate to the observers merges, which is CLAIM-348. Its “invite recontextualization” (the rank reused as a hiring threshold) is a case of legibility to an observer the measure was not built for. That the observed pay (brochures, detectives, staff time) while others benefit fits A25’s direction without being argued in the paper.
• CLAIM-081. Simplification “often makes information seem more authoritative”, after March and Simon: a ranking more legible and less informative than what it summarizes.
• LIT-1336 (Perdomo et al., registered unread) gives the formal counterpart. This paper has no model of when such a loop settles.
• THEORY-862 holds the case finding; THEORY-854 holds the validity-raising loop, Proposed until its quantitative links are read.

No instruction for machine-learning practice; nothing for the anthology.

Limitations

• Interviews of the measured. Most respondents are administrators who dislike the rankings, and the authors say that respondents would not admit gaming at their own schools and that “some stories may be apocryphal”. They report only strategies corroborated by public sources or direct experience.
• No counterfactual and no effect sizes. Whether a school changed because of the ranking or would have anyway is inferred from accounts. The rank-to-yield and rank-to-reputation links rest on cited studies, and footnote 14 reports one that finds the second weak.
• Selection of the case. Chosen because USN monopolizes the field and ranks every school, which the authors say makes effects dense. The framework’s reach to fields with several rankings is argued, not shown, and the authors’ own business-school comparison (Sauder and Espeland 2006) points the other way.
• A framework, by its own account. “an orienting heuristic … rather than … a general theory of reactivity.” The mechanisms are named and illustrated, not separated empirically: the paper says commensuration “exacerbated” the self-fulfilling prophecies and does not apportion the effects between them.
• The Strathern quotation. The wording quoted from Strathern (1996, p. 4) is the familiar generalization of Goodhart’s law; the paper gives it without Goodhart.

Open questions

• Under what conditions does the reactive loop settle on the ranking it started with, and when does it amplify noise, as with the two elite schools? The paper gives the channels and no dynamics.
• Is the validity a measure gains through conformity evidence of anything about the measured beyond the measure’s influence? That is CLAIM-402’s question and is not this paper’s.
• Coercion as a third mechanism is suggested and not developed.