Contribution
Feest moves the replication-crisis debate away from statistics (p-hacking, questionable research practices) and onto the concept of replication itself. She argues that (1) replication is at best similarity, and similarity requires individuation judgments that carry unstated conceptual and material assumptions (§2); (2) neither direct nor conceptual replication answers the question that matters once an effect is described, namely what kind of effect it is (§3); and (3) the research that does answer it is experimental probing of systematic error, which is exploration rather than replication (§4). The conclusion is deflationary: replication is “not as central to psychological research as it is sometimes taken to be” (abstract), at least in the kind of research she considers (§5).
Key insight
“Direct replication can (perhaps) provide evidence for the existence of something, but it cannot say existence of what” (§3.1, pp. 6–7). An effect is always produced under a description: calling it a Mozart effect rather than a major-key effect or a mood effect is a choice of conceptual scope. That choice also decides which confounders get controlled. Repeating the same procedure, however faithfully, cannot test the choice. Varying the operationalisation cannot test it either, because doing so already assumes the two operationalisations pick out the same effect.
Assumptions
Premises and authorities rather than formal conditions:
- No two experiments are identical (“at the very least, the time variable will differ”). Replication can therefore aim only at similarity (Shavit & Ellison 2017; Lynch et al. 2015, “exact replication is impossible”).
- Similarity is always similarity in some respect (Goodman 1955), so judging it requires assumptions, often unstated, about which factors are relevant (Collins 1985; Soler 2011’s “individuation judgments”).
- Scientists typically do not know the “correct” conceptual scope of the variables they manipulate and measure. Much of psychology operates under “deep epistemic uncertainty and conceptual openness” (§2, pp. 4–5).
- Error is split into random error, which direct replication addresses by falsifying H0, and systematic error, meaning misattributing an effect to one feature of the experiment when another feature produces it (§3.1). This framing is Mayo-style (fn. 3, Mayo 1996).
- The scope is psychological research of a particular kind, where the effect under study is not yet well conceptualised. Feest limits her conclusion to “the kind of psychological research I have focused on in this paper” (§5, p. 13).
Key results
What the paper argues, by section:
- §2, ineliminability of concepts. Any claim that an experiment succeeded or failed, whether original or replication, involves conceptual assumptions (what the IV and DV are qua) and material assumptions (that confounders were controlled). Even a replication that “simply imitates” the original must judge that it introduces no new confounders, and that judgment relies on unstated auxiliaries. Her example is ignoring lab temperature.
- §3.1, existence versus generalisability. Advocates of direct replication say it establishes “the basic existence and stability of a phenomenon” (LeBel et al.). Feest poses a dilemma. Describing the effect locally (a) is operationist caution that says almost nothing. Describing it more broadly (b) raises the question of conceptual scope, which no amount of direct replication can answer. So direct replication “has nothing to say about systematic error” and is “epistemically unproductive” (p. 7). She cites Zwaan (2013): replications speak to reliability, not validity.
- §3.2, generality. She separates generality within the experiment (internal validity) from generality outside the lab (external validity) and treats only the first. Internal validity can fail through unconsidered confounders (for example, prior musical training) or through conceptual scope (for example, a major-key effect labelled a Mozart effect). She argues conceptual replication cannot help. Comparing two operationalisations “of the same construct” presupposes that they individuate the same effect, which is the point in question, so a match or mismatch is “underdetermined by the experimental data” (p. 9).
- §4, putting replication in its proper place. The argument has three parts. First, probing systematic error is central to research and is not replication. Second, the resulting uncertainty is not in itself a crisis. Third, the crisis of confidence is not well described as a replication crisis. The Mozart-effect history is her evidence. On her reading, the debate between Rauscher and her critics turned on the choice of DV (the Stanford-Binet spatial-reasoning subtest as representative of spatial reasoning) and of IV (Mozart’s Sonata for Two Pianos in D major versus “up-beat” music and mood; Chabris 1999). It did not turn on whether the result could be repeated. Conceptual replication also presupposes “well-formed concepts” that researchers often lack, because they study an effect precisely in order to find out what it is (pp. 11–12).
- §5, conclusion. Direct replications are not non-conceptual. Conceptual replication begs the question. Replications are “less useful and important than is widely assumed”. The crisis of confidence is largely about statistical questionable research practices (Gelman & Loken 2014), not about replication.
Claims
| id | claim | strength | support |
|---|---|---|---|
| C1 | No two experiments are identical, so replication is judged by similarity, which requires individuation judgments that rest partly on unstated assumptions about relevance | moderate | informal argument from Goodman on similarity, plus cited authorities (Lynch et al., Collins, Soler); §2 |
| C2 | Direct replication can at most rule out random error and cannot say what effect exists | moderate | informal argument (the dilemma in §3.1), plus Zwaan (2013) blog quote |
| C3 | Direct replication “has nothing to say about systematic error” / “completely fails to be able to address systematic error” | weak | assertion (§3.1 p. 7; §4 p. 10). Not argued against the obvious reply that independent replications by different labs vary tacit lab-level factors and so do bear on some systematic errors |
| C4 | Conceptual replication begs the question, because judging success presupposes that the two operationalisations have the same conceptual scope | moderate | informal argument (underdetermination), §3.2 pp. 8–9 |
| C5 | The Mozart-effect controversy was mainly about conceptual scope (which IV feature, which DV), not about whether the effect replicated | moderate | one historical case, read from a few cited papers (§4 pp. 10–11). No systematic survey of the literature |
| C6 | This case is “typical” of experimental psychology | weak | assertion (p. 11), supported by the single case |
| C7 | The crisis of confidence is real but is better described as a crisis about statistical practice than a replication crisis | weak | assertion, citing Gelman & Loken 2014 (§5) |
| C8 | Replication is less central to psychological research than assumed | moderate | follows from C2–C6 as argued, with the explicit scope limit in §5. The abstract states it without the limit |
Concepts
- individuation judgment — (after Soler 2011) the judgment of what the experimental object, manipulation and measure are, and so which differences between experiments are relevant. Required even for “mere imitation” replications.
- conceptual scope — the question of the description under which an independent variable caused an effect, or a dependent variable measured something. For example: is the effect due to the stimulus qua Mozart, qua major key, or qua fast tempo?
- direct replication — an experiment that tries to be as close as possible to the original, for which “exact” replication is impossible.
- conceptual replication — repeating an experiment with a different operationalisation of the same construct.
- systematic error — erroneously attributing an effect to one feature of the experiment when it is due to another. Includes, but is not limited to, errors of conceptual scope.
- exploration — Feest’s positive term: experimentally testing hypotheses about possible systematic errors in order to delineate a phenomenon and develop its concept.
Connections
The paper builds on Collins’s experimenters’ regress (1985) and on Feest’s own reconsideration of it (Feest 2016), on Soler (2011) on tacit elements of experiment, on Mayo (1996) on error-statistical probing, and on Campbell & Stanley (1966) and Guala (2012) on internal and external validity. It argues against LeBel et al. (falsifiability, direct replication) and against Lynch et al. (2015, in praise of conceptual replication), and it concedes the statistical line of Gelman & Loken (2014) and the Open Science Collaboration (2015), which is the source of the “more than 60%” failure figure. In the nucleation record it sits beside Leonelli’s Philosophy of Open Science (LIT-189), which the seed pointed to on reproducibility. I have not read that work, so I cannot say how the two compare. Gelman’s post on LIT-014 belongs to the statistical-diagnosis side that Feest concedes; it is a separate genre (a blog rant) and not cited by her.
Bearing on the record
Epistemology. The paper concerns evidence and empirical justification: what a successful replication is evidence for. Its position is that replication evidence supports existence claims (against random error) but underdetermines the content of the claim, meaning which effect. Knowledge of what an effect is comes from probing systematic error, which is a conceptual as well as an empirical activity. The paper relies on an error-statistical picture of evidence (random versus systematic error, Mayo) and on the theory-ladenness of similarity judgments. The epistemology tag is justified: the work’s central claim is about what a kind of evidence can and cannot justify. philosophy-of-science should stay primary, because the object is experimental practice and the tag’s blurb names “causation and evidence”. social-science is justified because the subject matter is experimental psychology.
ML practice. The paper carries no instruction for ML practice, and nothing here should go to the Anthology as a practice. The transferable idea is conceptual. Whether a benchmark result “replicates” (across seeds, labs or re-runs) speaks to reliability. Whether the result is the capability it is named for is a question of conceptual scope, and re-running the same benchmark cannot settle it, while a “conceptual replication” on a different benchmark presupposes that both benchmarks measure the same construct. I found no Anthology practice or theory document on construct validity or replication that this bears on directly. If one is written, the paper could serve as a philosophical gloss but not as a source: for a practice.
Limitations
- I read the conference manuscript, not the version of record. The reference list still carries placeholders (“IJRM ???”) and “(forthcoming)” items, and the published text may have been revised.
- The argument rests on a single case (the Mozart effect), declared “typical” without further evidence.
- The claim that direct replication is blind to systematic error is overstated as written. The paper does not discuss multi-lab direct replications, in which differences among labs themselves probe some unstated auxiliaries.
- Only internal validity is treated. External validity, which is often the stated aim of conceptual replication, is set aside (fn. 4).
- The positive proposal (“exploration” of systematic error) is sketched in about a page. There is no account of when exploration is finished, or of how it differs in practice from a sequence of conceptual replications, beyond the fact that concepts are not assumed in advance.
- By Feest’s own statement, the conclusion is scoped to “the kind of psychological research I have focused on”. The title and abstract do not carry that hedge.
Open questions
- Does the published version sharpen the treatment of systematic error or add cases? A comparison against the Cambridge Core text would settle this.
- Is there a principled line between “exploration” and a programme of conceptual replications? It would help to see a criterion, or a case where the two recommend different next experiments.
- How does the argument apply to domains where constructs are well formed (psychophysics, some of cognitive psychology), where by Feest’s own reasoning conceptual replication should not beg the question?
- Relation to Feest’s later work on exploratory experimentation, and to Leonelli (LIT-189) on reproducibility, is still unchecked here.
Corrections to the seeded skim
- Access has changed: the PhilSci-Archive accepted manuscript (15219), which the dossier reports as behind a captcha, was reachable with a plain curl and is what this reading is based on. The published version is still paywalled.
- The dossier guessed the examples were “likely from memory or perception research”. There is one worked case only, the Mozart effect (Rauscher et al. 1993; critics Newman et al. 1995, Steele 1997/2000, Chabris 1999). Its independent variable is a piece of music and its dependent variable is a spatial-reasoning subtest.
- Feest does not use the exploratory/confirmatory distinction as such. Her contrast is replication (direct or conceptual) versus exploration, meaning experimentally probing hypotheses about systematic error in order to develop concepts (§4). She treats that as both descriptive and normative.
- The paper restricts itself to internal validity: “My focus here will be on internal validity” (§3.2, p. 8, fn. 4), and she marks her difference from advocates of conceptual replication who stress external validity. The abstract does not signal this restriction.
- The seed summary understates how much Feest concedes. Her conclusion (§5, p. 13) says the crisis of confidence is real and “to a large degree a crisis that turns on questionable research practices with regard to the use of statistical methods”. Her claim is that it is misnamed a replication crisis, not that the statistical diagnosis is wrong.
- primary topic: philosophy-of-science is right. The epistemology tag is justified (see Bearing on the record).