Contribution
Before it, the reactivity of measurement was treated as a methodological nuisance: social desirability, evaluation apprehension, sensitization. The paper gives it a substantive theory in the terms of social cognition, as the authors say physics does with Heisenberg: “Our goal is to develop a similar linkage in psychology, relating theories of the generation of judgment and behavior to the effects of measurement” (p. 421). It names self-generated validity: when a measured construct does not exist before the question, “the act of measurement changes the phenomenon under study, producing the thought processes predicted by the theory being tested and quite possibly influencing behavior” (p. 422). It then gives a three-factor account of when one answer becomes the input to a later one, and proposes designs to tell retrieved from computed answers.
Key insight
An answer is either retrieved or computed on the spot. A computed answer is then stored, and is retrieved in its turn: “Once the answer has been computed, it can be retrieved to answer subsequent questions” (p. 422). So a question does two things, at two times. It can create a cognition that did not exist, and the cognition it creates becomes one of the inputs to later judgements and to behaviour. Which input a later judgement uses is decided by accessibility and perceived diagnosticity, not by a fixed hierarchy from belief to attitude to intention to behaviour. People with judgements already formed and rehearsed, through expertise, direct experience or strong involvement, retrieve those and are little moved.
Assumptions
- A partly constructionist view, set against four traditional assumptions (p. 422): that measures “serve only to indicate what is already present and highly robust”; that behaviour is under the control of intention; that the constructs influence each other in a fixed causal hierarchy; and that they are context independent.
- Cognitive economy. “Resources are not used to develop judgments, beliefs, or plans of action unless some motive exists” (p. 422), so only some of what is measured already exists in memory.
- Search stops at sufficiency. Following Wyer and Srull, working memory is searched first and search stops when a sufficient basis is found (pp. 427, 429).
- Scope. Survey and laboratory measures of belief, attitude, intention and behaviour, in organizational, consumer and social research. The target is observed correlations among measures, not mean levels alone.
- Evidence. The paper reports no study of its own. “Our discussion has necessarily been speculative because little of the research that has provided the theoretical and empirical base for this article has been concerned with effects of measurement operations per se” (p. 432).
Key results
The results are propositions supported by cited studies. The numbers are the cited studies’, as the paper reports them.
- Measurement creates attitudes (Fazio, Lenn & Effrein 1984, p. 423). Subjects who filled in a paper-and-pencil attitude measure after playing with novel puzzles answered later yes/no evaluative questions faster than subjects who had not. “This was interpreted as evidence that neither group had precomputed attitudes prior to questioning, but that the paper and pencil measures had stimulated the formation of retrievable attitudes.” Groups told to expect an interview or a choice were as fast: attitudes form spontaneously “but only if it is functional to do so”. No effect sizes are given.
- The created state is carried forward (p. 423). A survey “can function to bring normally automatic behavior under conscious control”. Questioning “serves as the basis for attitude formation, which then serves as the most salient basis for the development of an intention”, and “the antecedent cognitive states were not ones that would have occurred in the absence of measurement”. Sherman (1980): stating an intention to perform a behaviour increases the likelihood of performing it.
- Question-order effects as diagnosticity (Schuman & Ludwig 1983, p. 425). Allowing American reporters into a Communist country: 90% yes when asked first, 66% when asked second. Allowing Communist reporters into the United States: 36% yes when asked first, 73% when asked second. Lawyers mercenary (72%) against doctors (50%): no order effect. Schuman and Ludwig read this as a norm of even-handedness; the paper reads it as the first answer being diagnostic for the second.
- A prior evaluation replaces retrieval of its basis (Lichtenstein & Srull 1985, pp. 425–426). Subjects who evaluated a product and evaluated it again an hour later recalled no more of the advertisement two days later than controls with no intervening task, while those who had judged style and then judged the product recalled as much as those told to recall. Only subjects “possessing a relevant, diagnostic prior judgment retrieved it for later use”.
- Three factors (pp. 424–430). An earlier response i is used for a later one if (a) it is perceived as diagnostic for the later one, which is asymmetric and itself a judgement; (b) it is accessible, which falls with time and similar intervening material and rises with elaboration and rehearsal; and (c) more diagnostic alternative inputs are not accessible. Equation (1), p. 426, writes the probability of basing response i + n on response i as a function of the items’ similarity and of n (the scan garbles the expression).
- Lability (p. 424). “any of 16 possible causal flows” among belief, attitude, intention and behaviour can be shown, especially for measures taken in immediate proximity. So an order-invariant correlation does not show that measurement did not inflate it.
- A sequential test (p. 425). For pairs of questions with pretested diagnosticity, the theory predicts the correlation is unaffected by order when A and B are equally diagnostic for each other, and higher in the order A, B when A is more diagnostic for B than B is for A.
- Output interference (pp. 428–429). Eliciting some cognitions suppresses the retrieval of others (Hoch 1984: generating reasons for an event raises its judged probability, mediated by the accessibility of the reasons). Tesser: “time spent thinking about an object or an issue renders schema (category)-consistent beliefs more accessible” (p. 429), which polarizes the attitude.
- Moderators (pp. 424, 430). Direct experience: “persons with direct experience have already formed and rehearsed overall attitude judgments prior to attitude measurement” (p. 424). Expertise (Bettman & Sujan): “Expert photographers choosing between two cameras showed no priming effects”, but were as sensitive as novices choosing between a camera and a computer: “These subjects had no retrievable criteria for comparing such dissimilar alternatives” (p. 430). Trait involvement: “Because respondents high in trait involvement have accessible, affectively polarized categories, the operation of framing, priming, and context effects of measurement is mitigated” (p. 430). These value systems become articulated where the environment mediates “strong, frequent, positive or negative outcomes” (p. 430).
- Scope claimed (p. 430). “We believe, however, that situations conducive to self-generated validity effects are the rule, not the exception.”
- Not peculiar to surveys (pp. 422, 431). “these processes are not peculiar to responses generated because a researcher has asked the subject to fill out a questionnaire”; conversation, reading and seeing a product do the same. “all real-world experiences with an object have the potential to shape subsequent responses to it” (p. 431).
Claims
| id | claim | strength | support |
|---|---|---|---|
| C1 | When a measured belief, attitude or intention does not already exist in memory, measuring it creates it | moderate | Fazio et al. (1984) reaction times, cited; nonattitude literature |
| C2 | A response created or retrieved earlier is used as an input to later responses and to behaviour | moderate | Lichtenstein & Srull (1985); Sherman (1980); Fazio et al. (1981, 1983), all cited |
| C3 | An earlier response is used for a later one if it is accessible and more diagnostic than other accessible inputs | weak | theoretical synthesis; no direct test reported |
| C4 | Order invariance of a correlation does not show that measurement left it uninflated | moderate | argument from lability of causal flows, p. 424 |
| C5 | Prior knowledge, direct experience and trait involvement reduce sensitivity to measurement context | moderate | Bettman & Sujan; Fazio & Zanna, cited |
| C6 | Self-generated validity effects are the rule, not the exception | weak | the authors’ belief, argued from Barsalou, Converse, Fischhoff et al. |
Concepts
- Self-generated validity. The state of affairs in which a measured construct does not exist, or is not spontaneously generated, without the researcher’s query, so that measuring it produces the thought processes the tested theory predicts (p. 422).
- Retrieval against computation. An answer read from memory against one computed on the spot from other inputs; a computed answer can be retrieved later (p. 422).
- Perceived diagnosticity. The degree to which the respondent perceives that the answer to the first question correctly identifies how the second should be answered (p. 424, paraphrased closely; the scan runs words together). Asymmetric, response-dependent, and itself a judgement.
- Elicitation context. “externally provided and internally generated retrieval cues”, which can change both the inputs to a judgement and the process acting on them (p. 422).
- Trait involvement. Chronic personal relevance, with affectively polarized, frequently rehearsed and context-independent categories (p. 430).
Connections
It draws on the accessibility literature of Fazio, Wyer and Srull, on Lichtenstein and Srull’s separation of memory-based from computed judgements, on Hoch’s output interference, and on Fischhoff, Slovic and Lichtenstein’s labile values (1980), whose line Slovic summarizes (LIT-1187, read in NOTE-949). The mere-measurement work of Morwitz, Johnson and Schmittlein (LIT-1189, not read) tests its prediction that measured intentions change behaviour.
Measurement context and the interpreter’s state (TERM-039, CLAIM-127). TERM-039 notes that A129’s model gives the elicitation protocol C an arrow into the judgement Y only, not into the update of the interpreter’s state R, and leaves the other route open. This paper takes the other route, and names both. Its first way is an arrow from C into R: the question creates a cognition that is then stored, “If the researcher asks a question about attitude (or belief, intention, etc.), and no such cognition already exists, the measure creates one” (p. 431), and the Fazio et al. reaction times are evidence of the stored state, not of the response. Its second way is the one TERM-039 calls direct influence: when the cognition already exists, order and format decide which input the answer is computed from, and “Both retrieved and computed responses can be modified by the elicitation context” (p. 422). Even this second way runs partly through state, because the earlier answer is retrieved from memory. Its sequential test (p. 425) is the design TERM-039 and CLAIM-127 say would separate the routes. Its warning on p. 424 bears on reading such designs: an order-invariant correlation does not rule out an effect of measurement.
CLAIM-341, the diachronic form. The paper supports it directly, and from outside the record. Its central proposition is the diachronic form: “The respondent may use retrieved answers to earlier survey questions as inputs to response generation to later questions” (p. 421). Lichtenstein and Srull’s result shows a judgement changing the observer in a measurable way: those who had already judged did not re-retrieve the original information and so later remembered less of it (p. 426). That is a judgement changing what the judge will later have available, with no action on the world. The mechanisms are all classical: retrieval, accessibility, rehearsal, output interference. The paper invokes Heisenberg only as an analogy for a theory of the instrument (p. 421). Two points cut against the claim’s use of it. First, the paper’s main concern is that the created state shapes behaviour, “quite possibly influencing behavior” (p. 422), and its examples run to purchase, turnover and helping; the loop it describes closes through action as often as through later judgement. Second, it treats measurement as one case of a general fact, that “all real-world experiences with an object have the potential to shape subsequent responses to it” (p. 431). Judgement is not singled out.
CLAIM-342 and QUESTION-121, what damps the effect. The moderators are expertise, direct experience and trait involvement. On the paper’s account they work because the person already holds a formed, rehearsed judgement that is retrieved instead of computed (pp. 424, 430). That is not evidence arriving from outside the interpretation it tests, which is what CLAIM-342 calls an anchor. It is the loop’s own earlier output, made context-independent by rehearsal. The case for reading it as an anchor rests on where those judgements came from: “direct experience” with the object (p. 424) and environments that mediate “strong, frequent, positive or negative outcomes” (p. 430). The paper also notes that expectancy theories hold only where “strong contingencies between behavior (or performance) and outcomes” exist (p. 424). So, on this paper’s account, the damping is a stored product of contingency between behaviour and outcome, which is what CLAIM-339 says the loop needs. QUESTION-121 reads damping by prior experience as the shape candidate 1 (unselected input) predicts. This paper supports at least as well a reading on which the damping is rehearsed commitment (candidate 3) formed under action. The expert who is unmoved choosing between cameras but moved choosing between a camera and a computer (p. 430) shows that the protection does not generalize beyond the judgements already formed.
Bearing on the record
- CLAIM-341: supports the diachronic form, with classical mechanisms, from a theoretical synthesis with no new data. It does not support the claim’s exclusion of action: the paper’s measured state feeds behaviour, and it gives judgement no special place among experiences.
- TERM-039 and CLAIM-127: the paper is a source for the arrow from measurement context into the interpreter’s state, and supplies a sequential, diagnosticity-based design to separate it from direct influence on the response.
- CLAIM-342: sharpens it. The moderators the literature reports are prior, rehearsed judgements, not independent evidence; whether they count as an anchor depends on their origin, which the paper places in direct experience and outcome contingency.
- Contextuality. The Schuman and Ludwig order effects it reanalyses are the kind of question-order data the record’s contextuality line uses. The paper explains them classically, by diagnosticity, and does not bear on whether they are contextual.
- It carries no instruction for machine-learning practice.
Limitations
The paper is speculative by its own account (p. 432) and reports no data. Its theory is stated qualitatively; equation (1) names a function without form. Diagnosticity is defined as a judgement and admitted to be task-specific, so its causes are left unspecified: the paper says further work is needed to specify them (p. 426). It concerns correlations among survey measures, so its evidence that a created state lasts beyond the session is indirect: Lichtenstein and Srull’s two days, and the cited behaviour studies.
Open questions
- Does the asymmetric-diagnosticity order effect on correlations (p. 425) hold when tested? That is the paper’s own prediction, and a test of the state route that TERM-039 asks for.
- How long does a measurement-created cognition persist, and does it change later judgements of new material, not only answers to the same question?
- Do judgements formed through direct experience protect against measurement effects because experience brought evidence the judgement did not select, or only because it rehearsed the judgement?