NOTE-197: A Pragmatist Theory of Evidence

Contribution

Reiss sets out to articulate the “pragmatist paradigm” of evidential reasoning. In that paradigm, scientific (causal) claims are inferred “using pragmatic criteria, from diverse bodies of evidence that may but need not include experiments” (§1, p. 3). He claims it has been practised but “neither articulated nor defended”, unlike the experimental paradigm of RCTs as gold standard, which has philosophical backing from Bacon and Mill to Mayo, Woodward and Cartwright. His contribution is the EHC framework. It revises hypothetico-deductivism by making the hypothesis–evidence relation inductive and context-bound, adds eliminative indirect support, and grades warrant by counting eliminated alternatives. It also supplies pragmatic stopping rules for the regress of alternatives.

Key insight

Evidence is two things: support (collecting facts relevant to h) and warrant (a reason to infer h). The ordinary word “evidence” conflates them (§2, pp. 5–6; the detective who says “I do not believe anything, I only collect facts”). Warrant is earned by eliminating relevant alternative accounts of the support, not by using a privileged method. An RCT is valuable because it removes “all alternative causal hypotheses – in one fell swoop” (§6, pp. 32–33), and for no deeper reason. Which alternatives are relevant, and when to stop probing, is settled by context: background knowledge, trust in peers, and the costs of error.

Assumptions

Premises and authorities rather than formal conditions:

• Justification for inductive inference is “local and material” (Norton 2003). Little can be said about evidence at high generality (§2, p. 8).
• The scope is type-level causal hypotheses in biomedical and social science domains where RCTs are feasible and common. Physics, chemistry, in vitro work and the historical sciences are excluded (§1, p. 4; §2, p. 8).
• Four desiderata for any theory of evidence (§2, p. 7): (I) it distinguishes support and warrant; (II) it gives an account of support; (III) it gives an account of warrant in degrees, where degrees need not be probabilities, only a weak ordering “at least sometimes”; (IV) it applies to non-ideal circumstances.
• Standard theories of causation (probabilistic, regularity, interventionist, process, mechanism) are “verificationist” and fail as theories of what causation is (Reiss 2012a). For that very reason they are usable as lists of what a causal hypothesis leads us to expect (§4.1, p. 16).
• Faithfulness/Stability (Spirtes et al.; Pearl) should not be assumed universally. Policy aims at exact cancellations, and near-cancellation cannot be told apart empirically from exact cancellation (fn. 2, p. 10).
• Background knowledge and evidence are not sharply distinct. Every factual claim used in assessment “can in principle be contested” (fn. 3, p. 11).
• Probability is not the right modality for evidential reasoning. Sharp subjective probabilities are “hopelessly unrealistic”, and vague ones give up Bayesianism’s advantages (fn. 3, p. 12, citing Norton 2011).

Key results

These are conceptual results, stated as definitions and proposals:

• Against standard HD (§3, pp. 9–11). Deductive entailment is neither necessary nor sufficient for support. A statistical hypothesis (“this coin is biased”) entails no particular data set, and causal hypotheses need not entail correlations, as in the two-route cancellation case of Fig. 1. Every statement entails itself, but no hypothesis supports itself.
• (S), support (p. 12): e supports h iff e is a pattern in the data “we are entitled to expect to obtain under the supposition that h is true” (cf. Hempel 1966).
• (S-d) direct and (S-i*) indirect support (pp. 13–15). Indirect support is a pattern incompatible with what we would expect if (a) an alternative able to account for h’s direct support, or (b) an alternative able to account for h’s prior indirect support, were true. Higher-level indirect support is defined in terms of lower-level support, so the definition is not circular, but it can regress without end. (The text calls this “Definition (IV)”, which is a numbering slip.)
• Four problems handed to context (p. 15): what we are entitled to expect; which alternatives to consider; how alternatives are eliminated; how the regress stops.
• Relevance of alternatives (§4.2). An alternative needs an empirical reason to be relevant, whether generic (selection bias is always relevant for observational data; Berkson’s paradox for hospital samples) or case-specific. General sceptical hypotheses are not relevant. An alternative with direct support of its own is salient, as with Fisher’s twin-study support for his “constitutional hypothesis”.
• Elimination is a judgement, not a proof (§4.3, pp. 23–24). No amount of incompatible data “prove[s]” an alternative wrong. Rejecting one remains a judgement in which values legitimately figure: when stakes are high, “the standards for rejecting an alternative should be lower”. There is no strict rule mapping costs to thresholds.
• Stopping the regress (§4.4, pp. 24–27). One default rule and four “supporting guidelines” (not exhaustive):
  • Default entitlement: scientists are entitled to each other’s claims (results, raw data, protocols) and should probe only when there are domain- or case-specific reasons.
  • Effect size: the larger the effect, the less need to probe. This works only for some alternatives; it does not help against fraud.
  • Manner and timing: the more specifically the dose-response, stopping and latency patterns match expectation, the less need to probe.
  • Study characteristics: the fewer the background assumptions and the smaller the inferential gap from data to result, the less need to probe.
  • Economic and other normative considerations: weigh the costs of further inquiry against the benefits of reduced error.
• Warrant (§5, Table 1, p. 29). There are four grades. Proof: all relevant alternatives eliminated. Strong warrant: all salient and some non-salient alternatives eliminated. Moderate: most alternatives, including some salient ones. Weak: some alternatives. “Proof” is empirical, contextual and fallible, and Reiss “bite[s] the bullet” on unconceived alternatives (fn. 8, p. 30). The choice of exactly three grades short of proof is “arbitrary”, though he says it matches IARC practice.
• Experimentalism is methodological foundationalism (§6, pp. 31–33). It cannot explain why non-experimental data count at all, or how dissimilar a method may be from an experiment and still count (“At what level of dissimilarity is a method mere silver standard or bronze, and when is it rubbish?”). EHC explains both the success of experiments and IV studies (mass elimination of alternatives) and their failures (measurement, coding and pooling errors that randomisation cannot prevent).

Claims

id	claim	strength	support
C1	Deductive entailment by h is neither necessary nor sufficient for e to support h	moderate	counterexamples in §3 (biased coin; two-route cancellation, Fig. 1; self-entailment)
C2	Support should be defined as patterns one is entitled to expect under h, split into direct and indirect (eliminative) support	moderate	definitional proposal (S), (S-d), (S-i*) with informal argument, §3
C3	Degree of warrant is set by which relevant alternatives have been eliminated, weighting salient alternatives more heavily, in four grades	weak	proposal, Table 1, §5; the grade boundaries are admitted to be “arbitrary”; claimed fit with IARC is asserted, not shown
C4	Rejecting an alternative is a value-laden judgement; higher stakes should lower the bar for rejecting an alternative	moderate	informal argument §4.3 plus the smoking case; the source is not cited but this aligns with the inductive-risk literature
C5	Scientists have a default entitlement to each other’s claims, and probing is warranted only for domain- or case-specific reasons	moderate	argument from the impossibility of total checking (Hardwig 1991; Brandom 1994), §4.4; tempered by Ioannidis 2005 and scandals
C6	Large effects, specific manner and timing, and low inferential distance reduce the need to probe results	weak–moderate	stated as guidelines, “independently” motivable but not argued at length; illustrated on smoking (Doll & Hill’s ~24× for 25+/day; up to 60× by 1959; Gilliam 1955)
C7	Experimentalism cannot explain why non-experimental evidence counts, while EHC explains both RCT success and failure	moderate	argument §6 (foundationalism dilemma, after Williams 2001)
C8	There is “no gold standard of evidence whatsoever”; that facts are sometimes learnt by experiment is “contingent and of no deeper significance”	weak	follows from the framework; asserted in §6 point 3, not argued independently
C9	The pragmatist paradigm “has neither been articulated nor defended” before this paper	weak	assertion (§1); no survey of prior work, and Parascandola 2004 is cited for the two-paradigm distinction
C10	The smoking–lung cancer controversy was resolved “fairly quickly” by these rules, helped by a shared value consensus in public health	weak	historical narrative from secondary sources (Cornfield et al. 1959; Doll & Hill 1956; Gilliam 1955; Vandenbroucke 1989)

Method

This is conceptual analysis with a single historical case study. There is no formal model and no empirical test. The EHC framework is built in three steps: a revised HD notion of support, eliminative indirect support, and a context that resolves four open questions. It is then illustrated on the 1950s smoking–lung cancer controversy.

Concepts

• Support — a fact relevant to h; a pattern in the data one is entitled to expect if h were true. It “has nothing to do whatsoever with belief” (p. 6).
• Warrant — a reason to infer h, which comes in degrees. It is the body of support after alternatives have been eliminated.
• Direct / indirect support — direct support bears on h itself; indirect support eliminates alternatives to h or to h’s prior support.
• Relevant alternative — an alternative with an empirical (generic or case-specific) reason to be considered. A salient alternative is one that has direct support of its own.
• Context of a causal inquiry — background knowledge about how the world works, the purpose of the inquiry, and normative commitments.
• Default entitlement — the presumption that other scientists’ claims can be relied on. Following Brandom, justification has a “default and challenge structure”.
• Proof — the top grade of warrant, meaning all relevant alternatives are eliminated. It is empirical, contextual and revisable, and is not deductive proof.
• Experimentalism — methodological foundationalism that treats well-run randomised experiments as intrinsically reliable and epistemically basic.

Connections

The paper builds on Hempel’s HD and “entitled to expect”, Salmon’s and Rescher’s two senses of evidence, Norton’s material theory of induction, Brandom’s default-and-challenge structure and Hardwig’s epistemic dependence. It sets itself against Mayo, Woodward and Cartwright (2007), whom it reads as underwriting experimentalism, and against Bayesian confirmation theory. The historical framing of two paradigms comes from Parascandola 2004.

In the record: Worrall (LIT-175) reaches an overlapping conclusion from inside the RCT debate. His “scientific commonsense”, that telling evidence tells against plausible rivals, is essentially Reiss’s indirect support, and both deny that randomisation has special epistemic standing beyond eliminating alternatives. Reiss’s claim that values legitimately set thresholds for rejecting alternatives is an inductive-risk position in the line of Douglas (LIT-139). Reiss does not cite her, but the structure is the same: values enter indirectly, through the costs of error. Cartwright (LIT-213) is the natural companion on evidential diversity.

Epistemology tag. The work is about evidence and justification: what counts as support, how warrant comes in degrees, and when inquiry may stop. It takes three definite positions. (1) Warrant is eliminative, contextual and fallible, and “proof” is inductive. (2) Credence is not probabilistic; plausibility is the right modality. (3) Testimony and trust carry justificatory weight through a default entitlement, subject to challenge, and inductive risk legitimately shapes rejection thresholds. The tag is justified.

Bearing on the record

Nothing here is an ML practice, and nothing in the anthology is sourced from it. What it offers ML practice is a vocabulary, not an instruction. Warrant for a claim from observational comparisons, such as ablations on shared infrastructure or correlational scaling analyses, comes from explicitly listing and eliminating the relevant alternative explanations. Generic alternatives include selection effects and data leakage; case-specific ones include confounded hyperparameter budgets. Reiss’s stopping rules (effect size, manner and timing, inferential distance, cost of further checking) map loosely onto how ML papers decide when a result needs more seeds or controls. That mapping is this reader’s, not the paper’s. If the anthology ever writes a THEORY document on what warrants claims from non-randomised comparisons, this paper and Worrall (LIT-175) are the two to cite together.

Note of 2026-10-16: THEORY-537 now cites LIT-118 as a source.

Limitations

• Only the February 2015 working paper was read, not the refereed version.
• The theory is restricted by stipulation to type-level causal hypotheses in RCT-feasible domains. Its generality is promised, not shown.
• The key notion, what one is “entitled to expect”, is handed to “context” and “background knowledge” with no account of how disputes over that context are settled. The four open questions are relocated rather than answered.
• The grades of warrant turn on counting eliminated alternatives, but the set of relevant alternatives is open-ended and context-relative. Reiss concedes that “exactly three grades” is arbitrary, and grading needs judgements about “most” and “some” that the theory does not fix.
• The stopping rules are illustrated on one historical case that was resolved correctly, which invites hindsight. Reiss himself notes Vandenbroucke’s point that the wrong side wrote equally cogent papers.
• The rejection of probability is argued in a footnote by citation (Norton 2011), not in the text.
• The value-consensus point (public-health norms eased agreement on smoking) is suggestive. It cuts both ways, and the paper does not say how a value-driven lowering of thresholds is kept from becoming motivated reasoning.

Open questions

• Can EHC be extended to token causation, to the historical sciences, or to domains where experiments are the “epistemic engines”? Reiss defers this.
• What makes an alternative’s reason “empirical” enough to be relevant, and who adjudicates that? The answer would decide whether grades of warrant are intersubjective.
• How do default entitlement and the effect-size guideline behave in fields with known high false-positive rates? Reiss cites Ioannidis 2005 but does not reconcile the two.
• How does the view relate to explicit inductive-risk accounts (Douglas, LIT-139)? Does “values lower the bar for rejecting alternatives” license the direct role of values that Douglas forbids?

Corrections to the seeded skim

• Access has changed. The full text was reachable as the February 2015 CHESS working paper on the Durham University website, and this reading is based on it. The published version is still unread, so the reading carries that caveat.
• The dossier’s expectation is confirmed in substance. Evidence is relative to a hypothesis and a context: support means patterns “we are entitled to expect to obtain under the supposition that h is true” (S, p. 12), and warrant comes from eliminating alternatives (§5). Two pieces are missing from the seed. (a) The theory explicitly builds in values: “Values do and should play a role in the decision whether or not to reject an alternative” (§4.3, p. 23), and the fourth stopping rule is an explicit cost-benefit trade-off (§4.4, p. 27). This is an inductive-risk position close to Douglas (LIT-139). (b) Epistemic trust is a default entitlement: “scientists are entitled to each other’s claims” unless there are domain- or case-specific reasons to probe (§4.4, p. 25, after Brandom and Hardwig).
• The scope is narrower than the title and the seed summary suggest. The theory covers only type-level causal hypotheses, and only in domains where randomised experiments “can be and are frequently employed”. It explicitly excludes most of physics and chemistry, basic/in vitro biomedicine, and the historical sciences (§1, p. 5; §2, p. 9). Reiss says it “can be extended” but leaves that to future work.
• The smoking case is an illustration, not support in any evidential sense. Reiss himself says that whether the framework delivers “is, alas, a matter for future research” (§7, p. 35). The seed’s “supports it with the 1950s smoking–lung cancer controversy” repeats the abstract’s “illustrated and supported” and overstates the case.
• Reiss explicitly rejects probabilistic (Bayesian) credence as the right model: “Possibility and plausibility are the modalities adequate for evidential reasoning, not probability” (fn. 3, pp. 11–12). The seed does not say this, and it matters for the epistemology tag.
• epistemology tag: justified. The paper is a theory of evidence, meaning support, warrant and degrees of warrant, and it takes explicit positions on testimony and trust (default entitlement) and on inductive risk (values in rejecting alternatives).
• primary topic: philosophy-of-science is right, because the theory is confined to scientific causal inference. epistemology is a close second and is correctly tagged. Consider adding natural-sciences: the worked case is epidemiology/medicine, and the stated domain is “biomedical and social sciences”.