NOTE-126: Worrall — Evidence in medicine and EBM

Contribution

Worrall presents this as an unusual Compass piece: a pitch for an under-worked area where philosophers of science “could have enormous impact – both intellectual and (very unusually) practical” (§1, p. 981). He does three things. (1) He traces EBM from the early “no RCT, no real evidence” message to its current, more guarded evidence hierarchy (§2). (2) He lists six under-investigated methodological problems that the guarded view raises (§3(a)–(f)). (3) He dissects the five arguments for giving randomised trials special epistemic weight (§4). His conclusion is that the near-universal medical belief that RCTs provide the most telling evidence “still remains to be seen” to be underwritable. His own positive view returns to “scientific commonsense” (Mill, Popper).

Key insight

Randomisation is a means of achieving things that matter, not a source of evidential weight in itself. The things that matter are that allocation is not in the experimenter’s hands, that groups are balanced on factors background knowledge makes plausible, and that the trial is placebo-controlled and blind. Once those are secured, no attainable epistemic state is better, because nothing can balance unknown confounders in a single trial and “to seek to control for all possible factors known or unknown is surely to chase a will o’ the wisp” (Conclusion, p. 1016). Telling evidence is evidence that tells against plausible rivals, however it was generated.

Assumptions

Premises and authorities rather than formal conditions:

• The general principles of evidence are universal across disciplines, and only their application is discipline-specific (§1, p. 981; fn. 2, against Laudan). The shared core is that a theory gets little support from data that plausible rivals also entail (fn. 6: Mill, Popper).
• EBM’s founders are not relativists. They claim, at least implicitly, that their hierarchy follows from general principles of evidence, and that claim is the target (§1, pp. 982–983).
• Worrall leans Bayesian: acceptance/rejection rules are unmotivated, and a sensible No-Miracles argument needs priors. He reads these as “not … ‘subjective’ but … encoding the impact … of the other relevant evidence” (§4(a), p. 997). He sets this aside when judging Fisher on Fisher’s own terms.
• Where frequentist probability is invoked, it means long-run relative frequency, so single-case probabilities are not defined (§4(b), p. 1007).
• A “known confounder” means “a factor that background knowledge tells you may plausibly play a causal role” (fn. 19).
• The scope is therapeutic-efficacy claims assessed by clinical trials. Diagnostic and prognostic claims are mentioned only in §3(a).

Key results

These are arguments; the core ones are in §4.

• §2, history. EBM (McMaster, early 1980s; after Cochrane 1972) was driven by standard treatments overturned by trials: grommets for glue ear, suppression of ventricular ectopic beats with encainide/flecainide (which raised mortality), and routine foetal heart monitoring (which increased interventions). The “no RCT, no evidence” reading was widespread, e.g. Sackett et al.’s “If the study was not randomized … stop reading it”. Later statements retreat from it: “EBM is not restricted to randomized trials and meta-analyses”. Most unquestioned treatments (appendectomy, cholecystectomy, diuretics in heart failure, aspirin for headache, penicillin for pneumonia) have never been tested in an RCT.
• Ethics depends on epistemology (§2, p. 987). Whether a trial is ethical cannot be judged independently of how much evidence we already have. An appendectomy RCT would be unethical precisely because we already have strong objective evidence.
• §3, open issues. (a) Why should only therapeutic claims require RCTs, when physics, “unambiguously the most successful science”, uses none? (b) “There’s no such thing as ‘face validity’”: judgements of self-evidence rest on implicit objective evidence, and the idea that large effects need less evidence is “incoherent” (effect size and evidence for it are separate issues). (c) How are clinical expertise, pathophysiology and trials to be amalgamated? (d) How should trials be combined? Meta-analysis rationale is unclear, and publication bias means the “grey literature” corrupts meta-analyses; trial registration helps only going forward. (e) “Biased RCTs”: if RCTs need baseline-imbalance checks, randomisation does not control known confounders, so why believe it controls unknown ones? (f) External validity: the benoxaprofen (Opren) case, trialled in 18–65 year olds and fatal in the elderly target population.
• §4 frame. Hierarchies make high-quality RCTs “trump any number of observational studies” (Barton 2000), even well-matched and much larger ones.
• 4(a) Fisher’s argument. Randomisation is said to license identifying the “true null” (the treatment is ineffective) with the chance hypothesis. In the tea-lady example, randomised order makes each of the 70 arrangements of 4 milk-first and 4 tea-first cups equiprobable in the long run, so a perfect score has p = 1/70. It fails “even in its own terms”. After randomisation there remain indefinitely many ways H can be false (a confederate, visual cues, a belief that cups alternate, which gives p = 1/2), so randomisation does not free us from assessing alternative explanations against background knowledge.
• 4(b) Unknown confounders. This is sociologically the most influential argument. Literal equality of groups (“the only difference … is … the intervention”, Clarke, Cochrane Centre) is trivially false. “Probable balance” makes sense only over indefinite repetitions, not in the one division actually made, and “no factor is unbalanced” is undefined over an open-ended list. The Leibovici retroactive-prayer RCT was impeccably randomised and double-blind and still produced “significant” results that must reflect chance imbalance. Randomisation is neither sufficient (Leibovici) nor necessary (appendectomy et al.) for real evidence. “The argument that has convinced the great majority of the medical community that RCTs supply the ‘gold standard’ is without real foundation” (p. 1008).
• 4(c) Selection bias. If clinicians choose arms, their conscious or unconscious prognostic views can confound the trial, and the design is at best single-blind. This is “a cast-iron argument for randomization”, grounded in testing against plausible alternatives and endorsed by Bayesians too (Urbach 1985). Randomisation is “very much a means to an end”: removing allocation from experimenters. “Haphazard” non-random allocation, such as birthday ordering or alternation, would serve as well (p. 1010).
• 4(d) Observational studies “known” to exaggerate. The 1970s–80s meta-studies (Chalmers et al. 1977, 1983) found historically controlled trials gave more positive results. Inferring exaggeration is circular without assuming RCTs give the truth, and the data are equally consistent with RCTs underestimating (Black 1996). Those historical controls were patently mismatched. Newer comparisons disagree: Kunz & Oxman found effects ranging from +150% to −90% and inversions. Benson & Hartz found “little evidence” that post-1984 observational estimates are larger. Concato et al. found them “remarkably similar”, with RCTs more variable. LeLorier et al. found that meta-analyses failed to predict later large RCTs 35% of the time. The Pocock & Elbourne critique is judged to rest on prior commitment to randomisation.
• 4(e) Probabilistic causality. Cartwright, Papineau and Pearl justify randomisation as guarding against common causes of treatment and outcome. This is 4(b) in another guise and fails for the same reason (pp. 1013–1015).
• Conclusion. Balance on plausible confounders, placebo control and blinding are what matter. Balance by randomise-and-check or by ante hoc matching “seems epistemically unimportant”. Carefully matched historically controlled trials can give strong evidence of large effects. Doll & Peto concede that selection bias “cannot plausibly give rise to a tenfold artefactual difference” but can give twofold ones. RCTs are “useful” for small expected effects, and then “serious questions may arise as to whether that treatment is worth having” (p. 1018).

Claims

id	claim	strength	support
C1	EBM retreated from “no RCT, no evidence” to a guarded hierarchy that is untenable only less visibly, because it is unclear	moderate	textual evidence from Sackett et al., SIGN, Straus & McAlister; §§2–3
C2	The ethical cost of a trial cannot be assessed independently of prior evidence	moderate	informal argument, appendectomy example, §2
C3	Fisher’s argument that randomisation is needed to validate significance tests fails on its own terms	moderate	informal argument §4(a), following Savage, Good, Howson; the tea-lady analysis
C4	Randomisation does not guarantee, and cannot be shown to make probable in the single case, balance on unknown confounders	moderate	conceptual argument about frequentist probability, §4(b) (after Lindley 1982; Seidenfeld & Kadane 1990); Leibovici 2001 as illustration, not proof
C5	Randomisation is neither necessary nor sufficient for real evidence of efficacy	moderate	counterexamples: untrialled standard treatments; Leibovici prayer RCT
C6	Controlling selection bias is a sound argument for removing allocation from experimenters, of which randomisation is one means	strong	argument from testing against plausible alternatives, §4(c); agreed across Bayesian and classical camps
C7	The “observational studies exaggerate” argument is circular and not borne out by newer comparative studies	moderate	logical point plus cited meta-research (Benson & Hartz; Concato et al.; Kunz & Oxman; LeLorier et al.), §4(d); the rebuttal of Pocock/Elbourne is brief
C8	The probabilistic-causality defences (Cartwright, Papineau, Pearl) reduce to the unknown-confounders argument	weak	asserted here; the argument is deferred to Worrall 2007 BJPS
C9	Given balance on plausible confounders, placebo control and double blinding, randomisation adds nothing epistemically	moderate	follows from C4–C6; stated in the Conclusion as “difficult indeed to see any such advantage”
C10	Carefully matched historically controlled trials can give strong evidence of large effects; most accepted therapies were effectively validated this way	weak–moderate	argument from commonsense plus Doll & Peto’s 10×/2× remark; the historical claim is asserted
C11	If an effect is small enough to need RCT-level control, the treatment’s worth is questionable	weak	a single sentence, deferred to “Evidence and Ethics in Medicine” (forthcoming 2008)

Concepts

• Evidence hierarchy — EBM rankings of study designs (e.g. SIGN 50, CEBM levels) in which meta-analyses and RCTs sit at the top and lower levels are used only when no good RCT exists.
• Known (possible) confounder — a factor that background knowledge makes plausible as a causal influence on outcome (fn. 19). An unknown confounder is better called an unsuspected one.
• Baseline imbalance — a difference on known prognostic factors, found after the fact between randomised groups.
• Selection bias (Worrall’s sense) — group differences caused by researchers’ choices of who goes to which arm. He notes other senses in the literature (fn. 24).
• Historically controlled trial — Worrall’s preferred term for the main “observational” rival, in which all new patients get the new treatment and controls come from earlier patients.
• Haphazard trial — experimental allocation by a non-random but experimenter-independent rule such as alternation or birthday order.
• Internal / external validity — whether the trial measures the effect in the trial population, and whether that result generalises to the target population.
• Face validity — EBM’s term for treatments exempted from RCTs. Worrall denies there is any such thing apart from implicit objective evidence.

Connections

Worrall builds on the Bayesian critique of randomisation (Savage, Good, Lindley, Howson & Urbach, Seidenfeld & Kadane) and on his own “What Evidence in EBM?” (2002) and “Why There’s No Cause to Randomize” (2007). He positions himself against Fisher, the Cochrane/EBM literature (Sackett, Clarke, Barton), Pocock & Elbourne, and Cartwright, Papineau and Pearl on probabilistic causality.

In the record: Reiss (LIT-118) independently articulates the same eliminative core, that warrant comes from eliminating relevant alternatives, and explains the value of RCTs as eliminating many alternatives at once. Reiss offers the general theory that Worrall gestures at as “scientific commonsense”. Cartwright (LIT-213) extends the anti-hierarchy line to evidential diversity and singular causal prediction; notably, Worrall lists Cartwright’s 1989 book among the defenders of randomisation. Douglas (LIT-139) shares the view that significance-test conventions need argument, not deference.

Epistemology tag. The work is about evidence and confirmation: what makes evidence telling, and whether method-based rankings of evidence are justified. Its position is eliminativist and broadly Bayesian. Evidence confirms insofar as it disconfirms plausible rivals given background knowledge; no design has intrinsic evidential authority; single-case “probable balance” is not a meaningful guarantee. It also touches testimony and trust indirectly, through its critique of deference to the hierarchy. The tag is justified.

Bearing on the record

This is not an ML practice paper, but it bears on how the anthology reasons about experimental evidence, and the application here is this reader’s extrapolation. In ML the “randomisation controls unknown confounders” argument appears as random seeds and random train/test splits. Worrall’s single-case point applies: one random split or one seed gives no guarantee of balance. The frequentist rationale does apply when divisions are repeated, which is exactly what ANTH-SOTA-350 (many random splits and seeds) and ANTH-SOTA-307 (error bars over several seeds) recommend. Those practices are therefore on the right side of Worrall’s argument: they trade the single randomisation he criticises for the long run that gives randomisation its meaning. His selection-bias argument corresponds to keeping model and hyperparameter selection out of the experimenter’s hands (a shared procedure, a held-out test set). His “effect size matters more than significance” point matches the anthology’s treatment of small gaps as inconclusive. No anthology document cites this paper for anything it does not say.

Limitations

• Worrall is explicit that the paper maps issues rather than settling them. §3 raises questions without answering them, and 4(e) and the ethics point are deferred to other papers.
• The case against 4(b) is conceptual. It shows that frequentist balance is not a single-case guarantee, but it does not show that randomised designs are no more reliable in practice than well-matched ones. The empirical meta-research he cites is mixed, and his dismissal of Pocock & Elbourne is brief.
• The positive proposal, balance on “plausible” confounders, depends on background knowledge being adequate. That is exactly what defenders of randomisation doubt in the small-effect regime. Worrall’s reply (small effects may not be worth having) is a single undeveloped sentence.
• Allocation concealment, which later defenders treat as randomisation’s key practical virtue, is folded into selection bias and not treated separately.
• “Haphazard” allocation is claimed to be as good as random allocation. This is asserted, and the risk that haphazard rules are predictable or correlated with prognosis, as with alternation and day of birth, is not discussed.
• It is a 2007 position piece and does not reflect later work (e.g. on causal inference frameworks or the RCT-defence literature).

Open questions

• Is there a single-case, decision-theoretic or Bayesian rationale for randomisation beyond selection bias, for example as protection against the experimenter’s own unconscious model? Worrall’s argument leaves this open, and later literature pursues it.
• How should heterogeneous evidence (expertise, mechanism, trials) be amalgamated (§3(c))? He poses this and does not answer it.
• Do well-matched observational studies systematically agree with RCTs? This is an empirical question that his cited studies answer inconsistently. A pre-registered, representative comparison would settle it.
• What rationale do meta-analysis and systematic-review protocols have beyond institutional authority (§3(d))?

Corrections to the seeded skim

• The most important correction concerns the seed summary, which says “none of the standard arguments for randomization (… selection bias …) shows that randomization adds evidential weight”. That is wrong for selection bias. Worrall calls the selection-bias argument “surely a cast-iron argument for randomization”, “explicitly endorsed by Bayesians as well as their classical opponents” (§4(c), p. 1009). His point is that it justifies randomisation only as “a means to an end”: taking allocation out of the experimenters’ hands. The Conclusion keeps RCTs “useful” for small expected effects because they control for this “known” confounder (p. 1018). The summary should say that four of the five arguments fail and the fifth supports a narrower conclusion.
• The seed lists four arguments. There are five: 4(e), randomisation and probabilistic causality (Cartwright, Papineau, Pearl), is omitted. Worrall treats it briefly and reduces it to 4(b), the unknown-confounders argument (pp. 1013–1015; detailed in his “Why There’s No Cause to Randomize”, 2007).
• The skim calls §3’s issues ones EBM left “unclear”. Worrall’s claim is sharper: EBM “correctly rejected a view … that is clear and crisp, and therefore can readily be seen to be untenable, only to replace it with a view that is not readily seen to be untenable, but only, perhaps, because it is unclear” (§3, p. 989).
• On §4(b), which the skim flags: the argument turns on the single-case/long-run distinction. Frequentist probability of balance is a claim about indefinite repetitions of the random division, “But medical researchers … do it once!” (p. 1007). No probability of imbalance in the single case is defined, and it is still less defined over an indefinitely long list of unknown factors (p. 1008). The illustration is the Leibovici et al. 2001 BMJ trial of retroactive intercessory prayer (n = 3393; length of stay p = 0.01, fever duration p = 0.04; mortality 28.1% vs 30.2%, non-significant).
• The skim’s summary of 4(d) omits the key argument: inferring that observational studies “exaggerate” effects is circular, because it presupposes that RCTs reveal the true effect. The newer comparisons (Benson & Hartz 2000; Concato, Shah & Horwitz 2000; Kunz & Oxman 1998; LeLorier et al. 1997) do not show systematic overestimation (pp. 1010–1013).
• epistemology tag: justified. The paper is about the logic of evidence and confirmation (what counts as telling evidence), and it takes a definite position: evidence tells when it tells against plausible rivals, and method-based hierarchies have no independent authority.
• primary topic: philosophy-of-science is right. probabilistic-modeling is defensible as a secondary tag, since 4(a) and 4(b) are arguments about frequentist and Bayesian foundations of statistical inference. natural-sciences (medicine) is a better-fitting secondary tag than probabilistic-modeling and is absent; consider adding it.