Source
Feest (2019), LIT-161 (Philosophy of Science 86(5), 895–905), read in full in NOTE-196 from the PhilSci-Archive manuscript. Page references are the manuscript’s.
What was actually shown
Similarity needs individuation (§2). No two experiments are identical, so replication aims at similarity, and similarity is always in some respect. Judging it takes conceptual assumptions (what the manipulation and measure are, qua what) and material ones (that confounders were controlled). Even a replication that “simply imitates” relies on unstated auxiliaries.
Direct replication (§3.1, pp. 6–7). It “can (perhaps) provide evidence for the existence of something, but it cannot say existence of what”. Described locally, the effect is operationist caution and says almost nothing. Described broadly, its conceptual scope is in question, and repetition cannot answer that.
Conceptual replication (§3.2, pp. 8–9). Comparing two operationalisations “of the same construct” presupposes they pick out the same effect, which is the point at issue, so a match or a mismatch is “underdetermined by the experimental data”.
The Mozart effect (§4, pp. 10–11). On Feest’s reading the debate between Rauscher and her critics turned on the choice of dependent variable (a spatial-reasoning subtest) and of independent variable (a Mozart sonata, or up-beat music and mood; Chabris 1999), not on whether the result repeated. That work, probing hypotheses about systematic error to delineate a phenomenon, is what she calls exploration.
What could have come out otherwise. The Mozart debate could have been about whether the effect repeated. On her reading it was not. It is one case.
What this does not say
- That replication is useless. Direct replication still bears on random error. The claim is that it is “less useful and important than is widely assumed” in this kind of research (§5).
- That the replication crisis is unreal. Feest concedes it is real and “to a large degree a crisis that turns on questionable research practices with regard to the use of statistical methods” (§5, p. 13). Her claim is that it is misnamed.
- That it holds where constructs are well formed. By her own reasoning, conceptual replication should not beg the question in psychophysics or other domains with settled constructs.
- That direct replication is blind to all systematic error. The paper says so (pp. 7, 10) but does not consider multi-lab replications, which vary some unstated auxiliaries (NOTE-196, C3: weak).
- Anything about external validity. Only internal validity is treated (fn. 4).
- How exploration differs in practice from a run of conceptual replications. The positive proposal is about a page long.
Connections
- The reading draws the analogy to benchmarks: whether a result repeats across seeds or labs speaks to reliability, and whether it is the capability it is named for is a question of conceptual scope (NOTE-196, Bearing). That is the reading’s extension, not the paper’s.
- Leonelli on open science (LIT-189) is the record’s other work on reproducibility. How the two compare has not been checked.