Source
Devezer, Nardin, Baumgaertner and Buzbas (2019), PLOS ONE 14(5): e0216125 (arXiv:1803.10118v3), LIT-988, read in full in NOTE-792.
What was actually shown
- The process. At each step a scientist of type a is drawn, proposes M_ℓ with probability P(M_ℓ | R_a, M_G), and tests it against the consensus M_G on fresh data of size 100 from the true model. The winner by AIC or SC is the new consensus. Without replication this is a Markov chain with P(M_G^(t+1) = M_ℓ | M_G^(t) = M_i) = Σ_a P(S(M_ℓ) < S(M_i)) P(M_ℓ | R_a, M_i) P(R_a).
- Reproducibility is causally idle (S23). A replication re-runs the same pair on independent data, so its outcome probabilities sum out of the transition matrix. The rate of reproducibility therefore cannot affect stationary probabilities, first passage times or stickiness.
- The case (Fig 4, §“Reproducible results do not imply convergence”). A population of 99% boundary testers (“Bo”), who propose only models with one more interaction, reaches near-perfect reproducibility. In the no-replication analysis it spends only 25% (AIC) or 48% (SC) of its time at the true model, mostly at over-complex ones. In the ABM its Spearman correlation between reproducibility and time at the truth is −0.06. Its comparisons pit untrue models that are far apart, so “the method consistently favors the same untrue model”.
- The contrast. Mave-dominant populations, who propose uniformly, reach the truth fastest and spend long at it, with lower reproducibility. Correlations with time at the truth are 0.41–0.69 for the other populations (Table 3).
What could have come out otherwise: reproducibility could have tracked time at the truth across populations, as the received view of replication expects. It does for some populations and not for Bo.
What this does not say
- Not about beliefs. Scientists hold no beliefs and do not learn. The consensus is the last winner. The claim concerns a community’s standing view produced by a sound procedure, not a belief members settle on.
- Not that reproducibility is worthless. For most populations it correlates positively with time at the truth and strongly with stickiness (0.74–0.87). True results are reproducible, a sanity check the authors keep. The claim is that reproducibility is not sufficient, and does not cause convergence.
- Not that epistemic diversity is optimal. The equal mix is never the worst on any property and has the lowest variability. Tess- and Mave-dominant populations spend more time at the truth. The paper’s “optimizes” is stronger than its comparison. Under high noise (4:1) even the diverse population fails (S18).
- Not robust to memory. The consensus changes on a single test, which the authors call unrealistic. They say it may exaggerate the role of mavericks. Whether a memoryful consensus would resist Bo’s drift is untested.
- Not about real fields. The model space has 14 linear models with the truth always in it. The claim that rewarding reproducible Bo-like work could canonise falsehood is the authors’ speculation from cited work.
Connections
- QUESTION-047. The closest of the record’s models to the question’s
design. Each inquiry step is held sound, a population-level variable
(which alternatives get proposed) is changed, and where the community
settles changes, onto a false model that its own replications keep
confirming. It answers the question’s second clause for a standing
view, not a belief. It is offered as a partial answer, but no
answers:relation is declared, because the model’s agents hold no beliefs. - THEORY-148 (transient diversity). That account says diversity must last long enough and then give way. Here diversity is permanent, and its benefit is insurance against a bad dominant strategy. The two are different mechanisms under the same word, and should not be cited as one result.
- THEORY-146. That account biases a community through which researchers are funded and recruited, without corrupting anyone. This one fixes a false consensus through which comparisons are run, without any test being unsound. Both put the failure in the population’s composition rather than in its members’ reasoning.
Note of 2026-10-19: why replication cannot move the chain (S23, re-derived)
Without replication the consensus is a Markov chain, P(M_G’ = M_ℓ | M_G = M_i) = Σ_a P(R_a) P(M_ℓ | R_a, M_i) P(S(M_ℓ) < S(M_i)), whose inputs are the strategy mix, the proposal kernels and the comparison probabilities. A replication re-tests the previous pair on fresh data, independent of the original test’s. So summing the joint probability q_ijℓ over the replication’s outcomes returns the original step’s probability, Σ_j q_ijℓ = P(M_ℓ | M_i) P(S(M_ℓ) < S(M_i)), and the transition law is the same with or without replication. The rate of reproducibility, P(M_G^(t+1) = M_G^(t) | replication at t), is a statistic of the chain’s path, not an input to its transition law. So it cannot move the stationary distribution, the first-passage times or the stickiness. The paper’s labels q_iiℓ and q_iℓℓ are muddled (NOTE-792), but the argument needs only the independence of the replication. The existence half rests on the Bo-dominant population: near-perfect reproducibility, 25% (AIC) or 48% (SC) of its time at the truth, and Spearman r = −0.06 between the two (NOTE-792, C2). Whether the decoupling survives a consensus with memory is QUESTION-151.
Note of 2026-10-20: the wider claim, filed Proposed
Under ADR-047 v4 the narrowing-loss sweep (batch 071) filed THEORY-814 as Proposed, extending this entry. It carries the general reading this entry’s title carried before the promotion audit narrowed it: that in real scientific communities high reproducibility is not evidence that the consensus is true, as Devezer et al. draw the lesson. This entry is its evidence and is not changed by the filing.