NOTE-1001: Roseboom & Lush — Serious problems with interpreting rubber hand illusion experiments

Contribution

The paper turns a disputed correlation into a concrete outcome. The correlation is between trait phenomenological control, measured by the subjective scale of the SWASH hypnotisability test, and agreement with the rubber hand statements (rs = 0.26 in Lush et al. 2020). The outcome is how often a typical small study would conclude that its participants had the illusion. The answer depends strongly on who is sampled, and most strongly for the one statement about ownership. The paper also sets out the inferential chain by which drift, imaging and animal studies borrow their link to ownership from report, and says where that chain breaks if report is confounded.

Key insight

A correlation of 0.26 sounds weak. But what it does to the conclusions of small studies depends on how much of the effect the low end of the trait carries. For the ownership item, the low half of the trait carries almost none of it. So whether a 20-person study “finds” ownership of a rubber hand depends largely on how many highly suggestible people it recruits. Anything later linked to ownership through those reports inherits that dependence.

Assumptions

• The dataset is representative. All simulations resample the 351 participants of Lush et al. 2020 with complete data. Unselected samples of 20 found significant agreement 83% of the time (lines 249–261). The authors take this as showing the sample is “broadly consistent with participants from previous studies across the field”.
• Mean above zero is the criterion. One-sample two-tailed t-tests of the mean against 0 (lines 194–216, 238–244). Some studies use cut-offs of 1 or 2 (Ehrsson et al. 2007; Lloyd 2007), which are not simulated.
• Twenty per study. The authors justify n = 20 as the mean sample size of the 30 most-cited RHI publications (median 15; lines 226–229). Ten and 50 are in the supplement.
• A median split on the subjective SWASH. The notebooks split the sample by quartile of subjective SWASH (quartiles 1–2 against 3–4). Each half had at least 175 participants (line 236).
• Only the synchronous condition. The synchronous–asynchronous difference is set aside as “confounded by demand characteristics and therefore uninterpretable (Lush, 2020)” (lines 194–196). The argument therefore inherits whatever Lush 2020 establishes.
• Phenomenological control is domain-general. SWASH covers amnesia, musical and visual hallucination and the like. The same trait predicts vicarious pain, mirror touch and the visually evoked auditory response (lines 375–391). This is what lets the authors read the RHI correlation as the RHI being an instance of the general ability.

Key results

• The relationship (Fig. 1, lines 149–178). A scatter of 351 people’s subjective SWASH score against their mean S1–S3 rating after synchronous stroking. The fitted line is y = 0.578x + 0.333, so each SWASH point adds 0.578 points of agreement. The rank correlation of rs = 0.26, 95% CI [0.16, 0.37] is quoted from Lush et al. 2020. On the rendered figure, ratings span the whole −3 to +3 range at every level of SWASH.

• The whole sample (lines 205–216). Mean S1–S3 agreement was 1.27 (SEM 0.08), t(350) = 15.42, p < 0.001. By statement, the mean for S1 was 1.94 and for S2 1.15 (lines 111–112). In the notebook, 12% of participants agreed with S1 while disagreeing with S2.

• Simulated studies of 20, S1–S3 mean (Fig. 2, lines 249–294).

  sample	mean positive	significant agreement
  unselected	9996 / 10,000	83%
  below-median SWASH	9932 / 10,000	51%
  above-median SWASH	10,000 / 10,000	98.9%

• By statement, studies of 20 (Fig. 4, lines 301–348; notebook outputs).

  • S1: significant in 95% of below-median and 99.9% of above-median studies.
  • S2: 24% against 95%.
  • S3: 4.4% against 74%, and 28% for unselected studies. Of the below-median studies, 2485 had a mean in disagreement with S3.
  • The main text gives the S3 figure for the lower half (“only ~4%”) and not the 74% or 28%; those are from the notebook.
  • Fig. 5: for S3 in the lower half, “the probability of obtaining inferential support (p < 0.05) … is roughly the same as the probability of obtaining any other extent of 0.05”, that is, p is roughly uniform.

• Sample size changes the picture (supplementary notebooks).

  • At n = 50, the below-median S1–S3 mean is significant in 94.8% of studies and the above-median in 100%. S3 is significant in 6.3% of below-median, 99.8% of above-median and 64% of unselected studies.
  • At n = 10, the corresponding S1–S3 figures are 27% and 79%, and for S3 4.1% and 42%.
  • So the contrast for the averaged statements shrinks as studies grow, while the contrast for the ownership statement does not.

• Figure 3 (rendered). The distribution of simulated study means for the lower half sits around 0.9–1.0 on the S1–S3 mean and about 0.25 on S3. The upper half sits around 1.6 and 1.2.

• Walsh et al. 2015 (lines 165–172). It reported no relationship between hypnotisability and the RHI, with r = 0.20, p = 0.371, n = 23, 95% CI [−0.23, 0.57]. The authors argue that this “do[es] not provide counterevidence”, since the design cannot support a null.

• Three interpretations (lines 393–457).

  • (1) The same people control their experience in the RHI. This is the most parsimonious reading, and it “does not rule out a multisensory contribution”.
  • (2) They “happen to be the same participants that can experience the RHI, but for distinct reasons (e.g. pure multisensory integration account)”. This is “not explicitly ruled out by the results presented here or previously”.
  • (3) They respond as though they do, for other reasons.
  • Faking is acknowledged as a possible hypothesis-awareness effect (lines 450–457).

• Reverse inference (lines 458–501). “All other findings, such as proprioceptive drift or changes in measured blood flow (fMRI) or scalp potential (EEG), are linked to changes in subjective ownership only through broad correspondence or co-occurrence with these subjective reports.” For animal paradigms such as the rubber tail in mice, “there is, and can be, no direct evidence to support this claim”. The authors add: “Our claim in no way invalidates behaviourist approaches that seek to map body perception to neural function through observed behaviour.”

• Top-down accounts (lines 502–512). Top-down accounts of the RHI (Tsakiris 2010; Ehrsson et al. 2021) treat it as “resistant to suggestions”. Given uncontrolled demand, “it is premature to consider theoretical proposals of top-down effects which are not related to demand characteristics”.

Claims

id	claim	strength	support
C1	In this dataset, whether a 20-person study finds significant agreement with the averaged RHI statements depends strongly on the sample’s trait phenomenological control	strong for this dataset at n = 20	resampling, Fig. 2; 51% against 99%
C2	Agreement with the ownership statement (S3) depends on trait phenomenological control far more than agreement with the referred-touch statement S1	strong for this dataset	Fig. 4; S3 4% against 74%, S1 95% against 99.9%; holds at n = 10, 20 and 50
C3	The correlation between SWASH and RHI report is therefore not too small to matter	moderate	follows from C1–C2 for small studies; weaker at n = 50 for the averaged statements
C4	Only experiments with many participants high in phenomenological control will provide evidence for the RHI	weak as stated	true of S3; for the S1–S3 mean, lower-half samples of 50 find it 95% of the time (supplement)
C5	The synchronous–asynchronous difference is uninterpretable as evidence of the RHI	moderate, borrowed	rests on Lush (2020), LIT-1276; not tested here
C6	Drift, imaging and animal results are tied to subjective ownership only through report, so a confound in report undermines them as evidence of ownership	moderate as an argument	lines 458–501; an inference about inference, not data
C7	The RHI is most parsimoniously an act of phenomenological control	weak	the authors’ preference among three readings, the second of which they grant is not ruled out

Concepts

• Trait phenomenological control — “the domain general ability to meet expectancies arising from direct or implicit imaginative suggestion (including demand characteristics)” (lines 352–355). It is measured here by the subjective scale of the Sussex-Waterloo Scale of Hypnotisability (SWASH).
• Hypothesis awareness — knowing or guessing what the experimenter expects. The paper names three ways it can show up: faking, intentional imagination, or phenomenological control (lines 53–55, 450–452).
• Simulated experiment — a random draw of n distinct participants’ real reports from the pool, tested against zero, repeated 10,000 times. Different draws may overlap (lines 223–238).
• Reverse inference — inferring a subjective state, here ownership, from a measure that was validated against report of that state, when report itself is in doubt (lines 474–495).

Connections

The paper rests on two sibling filings. The first is Lush et al. 2020, the dataset and the correlation. The second is Lush (2020), LIT-1276, for setting aside the asynchronous comparison. It is written against the commentary by Ehrsson and colleagues (cited as a 2021 preprint; filed alongside as Ehrsson et al. 2022, with Lush & Seth 2022’s reply). Among other siblings, Longo et al. (2008) is cited for S1 failing to load on any factor (lines 113–114). Rohde, Di Luca & Ernst (2011) and Botvinick & Cohen (1998, LIT-1277) are cited as studies in which every participant gave S1 the maximum (lines 114–116). It cites Lush, Seth & Dienes (2021) for hypothesis awareness in drift’s asynchronous control, and Reader (2021) for a further expectancy study; neither is held. Footnote 1 records a dispute with a reviewer over a table of top-cited studies, which was dropped from the text.

Bearing on the record

• THEORY-765 (Proposed), the synchrony effect.
  • Does it meet the test? No. Its promote_when allows “a design … in which trait suggestibility is measured and the synchrony effect survives it”. This paper measures trait suggestibility but analyses only the synchronous condition, by design. So it says nothing about whether the synchrony effect survives, and it has no drift analysis.
  • Does it trigger the refutation? No.
  • What it bears on. It bears on the THEORY’s middle clause, that the rubber hand is “reported as one’s own”. In the largest RHI dataset, agreement with “I felt as if the rubber hand were my hand” after synchronous stroking was significant in only 28% of unselected 20-person studies, and 64% at n = 50. In samples from the lower half of hypnotisability it was significant in 4–6% at any size. Referred touch (S1) was robust across the trait. As evidence for the THEORY’s title, then, the referred-touch clause stands better than the ownership clause. That is a reading of the authors’ notebook outputs, not their stated conclusion.
  • Proposed change (proposal only). The status stays Proposed. The promote_when might ask that the suggestion-controlled replication report the ownership item separately from the referred-touch items, and give trait phenomenological control as a covariate.
• THEORY-769 (Proposed), graded common-cause ownership.
  • Neither meets nor refutes. The THEORY predicts graded ownership.
  • Where the grading comes from. This paper shows graded ownership ratings across people, but the grading follows a trait of suggestibility, not any manipulated reliability of input. A causal-inference model would have to explain that individual variation by the precision of the senses, and nothing here says it could.
  • Proposed change (proposal only). The held-out test in the promote_when could control for trait phenomenological control.
• THEORY-767 (Proposed), interoceptive signals.
  • How it bears. The paper does not discuss interoception. But the THEORY’s second leg is an individual-differences correlation: lower heartbeat-detection sensitivity goes with greater RHI susceptibility (Tsakiris et al. 2011). This paper shows that RHI susceptibility, as measured by report, tracks trait phenomenological control.
  • The third variable, mine. If heartbeat detection also relates to suggestibility, the trait could account for both. Heartbeat-counting tasks are themselves open to expectation, which the THEORY’s “What this does not say” already hints at. That is my inference; the paper makes no such claim.
  • Proposed change (proposal only). The promote_when’s second condition could require that the sensitivity–susceptibility relation survive trait phenomenological control as a covariate.
• THEORY-768 (Proposed), minimal selfhood.
  • No test. The paper does not test the part–whole split or the agency thesis.
  • The evidence base. It names the whole-body illusion (Lenggenhager et al. 2007, LIT-1279) and body swapping among the paradigms the RHI spawned. Its reverse-inference argument applies to them too: full-body drift is linked to self-identification only through the same kind of questionnaire.
  • Neither meets nor refutes.
• THEORY-458 (Proposed). The record has used the RHI as passive ownership without agency. This paper finds the ownership statement the one most dependent on suggestibility. So the RHI is weaker support for the ownership half of that dissociation than the record assumed.
• QUESTION-048. Lines 458–501 are a compact statement of the question’s problem for one domain.
  • The argument. Every non-report measure of an experience gets its meaning from a correlation with report. If report is confounded, the non-report measures inherit the confound. Animal versions add a further step, assuming that a behaviour in rodents is equivalent to one in humans.
  • The limit. The paper grants that the reports may reflect genuine experience, produced by a different mechanism. So it is a problem about which cause an experience has, not about access against experience.
• NOTE-372, C6, and THEORY-064. C6 says ownership is manipulated “by multisensory conflict”. For the ownership item specifically, this reading shows the manipulation working mainly in people high in a trait that predicts response to suggestion in general.
• No instruction for machine-learning practice.

Limitations

• No new data, one dataset. Every number is a resample of Lush et al. 2020, from one laboratory, with its own procedure and population. The simulations add no information beyond the joint distribution of SWASH and the ratings. They are a way of displaying a correlation, and they cannot show that suggestion causes the reports.
• Median split and n = 20 do much of the work. Dichotomising a continuous trait and testing small samples both lower power. The “around half the time” figure for the averaged statements is largely a matter of power. At n = 50 the lower half finds the averaged statements 95% of the time (supplement), and the main text does not say so. C4 as worded (“only experiments run with a large proportion of participants higher in phenomenological control will provide evidence for the RHI”) overstates the case for the averaged statements. It holds for S3.
• The low half still agrees on average. The lower half’s simulated means for S1–S3 sit near 1 (Fig. 3). Low hypnotisability does not mean no reported illusion, and a positive mean in the least suggestible half is what interpretation (2) would predict as well.
• Synchrony is not analysed. The paper sets aside the synchronous–asynchronous contrast on the strength of Lush 2020. So it cannot say whether the contrast also shrinks in low-trait participants. Lush et al. 2020’s pre-registered test of that question is reported in the sibling reading as inconclusive.
• The S1 critique is anecdotal in part. That some participants read S1 as a check on the experimenter’s brushing is reported “anecdotally” (lines 106–109). The 12% S1-but-not-S2 figure is from the data.
• Interpretation (1) is preferred by parsimony, not shown. The authors themselves grant that (2) is not ruled out. Their argument from domain-generality is that a general process “subsumes domain specific demonstrations”. That is a theoretical preference, not a result.
• Authorship. Both authors belong to the group whose theory of phenomenological control the paper defends. The data and analysis are open (osf.io/d5hjp), and the supplementary outputs read here match the text’s figures.

Open questions

• Does the synchronous–asynchronous difference on S3, with expectancy matched, differ between participants low and high in trait phenomenological control? If the difference is present and equal in low-trait participants, interpretation (2) gains. If it is absent in them, interpretation (1) gains. This dataset could partly answer it, and the paper does not run it.
• Does proprioceptive drift track phenomenological control in the way S3 does? The sibling reading of Lush et al. 2020 records rs 0.11 for synchronous drift.
• Does the relation between interoceptive sensitivity and RHI susceptibility survive trait phenomenological control as a covariate?
• Can an animal paradigm be given any check against the reverse inference the paper describes, or is subjective ownership in animals out of reach by this route, as the authors say?