nucleation

THEORY-862: In US law schools under the U.S. News ranking, administrators and faculty reported, and public sources corroborated, that schools changed what they did in order to move their rank: they moved money toward the ranking's inputs, redefined admissions and career-services work around its indicators, and gamed its figures, so the ranking altered the things it measured

3 min read

Properties4
StatusActive
Tagssocial-science, society-and-governance
SourceLIT-1330 — Rankings and Reactivity: How Public Measures Recreate Social Worlds
SupportsCLAIM-402 — Predictive success of a category across contexts warrants treating it as a real effective kind only when the evidence in those contexts does not pass through the category's own use, or when the effect of that use is modelled: a category that speakers or institutions act on can produce the regularities that confirm it, and contexts that share its use are not independent

Graph View

Source

Espeland, W. N. & Sauder, M. (2007), LIT-1330, read in NOTE-1050.

What was actually shown

The ranking weighs reputation (40%), selectivity (25%), placement (20%) and faculty resources (15%), and covers every accredited school. The authors ran two waves of open-ended interviews (66 and 70, about 45 minutes each), with deans, other administrators, faculty and staff from 70 schools across the four tiers. They added short interviews at an admissions fair, 30 business-school interviews, school materials, chat rooms and public data. Responses were coded for themes, “consequently, the frequency of particular responses is only an approximate indication of their importance.”

What they found, as reported:

  • Reallocation. Merit scholarships to raise median LSAT scores, in some schools at the expense of need-based aid (27 administrators reported increases in merit scholarships). Brochures to the reputation-survey raters, with many schools reported to spend over $100,000 a year and “no evidence that it works”.
  • Redefinition of work. All 16 career-services respondents described more effort spent tracking graduates’ employment status, at the expense of counselling. Admissions decisions moved toward ranking criteria.
  • Gaming, “efforts to improve ranking factors without improving the characteristics the factors are designed to measure”. This was checked against public sources: in 1995, 29 of 177 schools reported higher median LSATs to USN than to the ABA, and 13 the next year. Other practices were reported: low scorers classed as part-time or probationary, students moved into night programs, faculty pushed to take leave in the spring.

The study could have found otherwise. Respondents could have described the ranking as an outside report their schools ignored, which is what most did at first, before “angry students and alumni” taught them otherwise.

What this does not say

  • Not how much. The study has no effect sizes and no comparison with schools or years without the ranking. That the changes were caused by the ranking is the respondents’ account and the authors’ inference.
  • Not that the ranking raised its own validity. That is the paper’s self-fulfilling-prophecy argument. Its quantitative links are cited, not shown, and it is held separately in THEORY-854, Proposed.
  • Not other measures or fields. The authors chose the case because USN has no rival in law and ranks every school. Business schools, which face several rankings, are said to be affected differently.
  • Not that reactivity is harmful in general. The authors note what the few supporters praised (more information for consumers, more disclosure) and call their own account one of “mostly harmful effects” in this case.
  • Not a test of any account of kinds. CLAIM-402 can cite this as an instance of a measure acted on by the measured. The case does not show that the ranking tracks nothing independent of its use.

Backlinks