NOTE-1421: Lynch et al. — RefraMED, the NIHR EME monograph

Contribution

It is the full account of the trial the record holds through its journal paper (LIT-1223, NOTE-1358): the same 250 participants, the same primary result and the same harms, with the health economics and the mechanism analysis that the journal paper leaves out. What it adds is a design. Within the RO DBT arm participants were randomised a second time, to features meant to move the hypothesised mediators without being therapy, so that the mediators’ effects could be estimated by instrumental variables free of the self-selection that biases ordinary mediation. The authors say it is the first psychotherapy trial to do this. It did not work: the instruments barely moved the mediators.

Key insight

The trial knew what mediation needs and could not get it. Patients pick how many sessions they attend, how well they bond with the therapist and how much they learn, and whatever leads them to pick also bears on how depressed they stay; so the authors randomised small things (a nicer room, a feedback form, cheques handed over in session) to get exogenous variation in those mediators. The variation was too small. The authors’ own conclusion is that to estimate mechanisms a trial has to randomise parts of the treatment itself, more or less exposure to its core elements, rather than additions with weak effects on the mediators.

Assumptions

• The mediators are process variables. Exposure, alliance and skills are set to zero for usual-care participants, so there is no mediated path in that arm, only a direct effect (§2, “Process variables”). The DBT Ways of Coping Checklist is measured in both arms but treated as zero in usual care.
• Timing. The causal model takes exposure at 7 months, and alliance and skills “together at 7 months”, as contemporaneous intermediate outcomes, and the HRSD outcome is also at 7 months (§2; §4, “Instrumental variables analysis”). Nothing in the model puts the mediator before the outcome.
• Exclusion restriction. Each instrument (room, feedback, mode of reimbursement, therapist) affects the HRSD only through the mediators. The randomised allocation itself cannot be an instrument, because its direct effect is one of the quantities sought.
• Population and outcome model as in the journal paper: refractory depression with the cluster B, bipolar, psychotic and substance presentations excluded; three-level mixed models (time within patient within therapist), missing at random.

Key results

• Efficacy (Table 8). HRSD, RO DBT minus usual care: −5.40 (SE 1.92, 95% CI −9.85 to −0.94, p = 0.024) at 7 months, −2.15 (−6.59 to 2.28, p = 0.290) at the 12-month primary end-point, −1.69 (−6.22 to 2.84, p = 0.424) at 18. These match the BJP paper.
• Process outcomes (Table 8). AAQ-II −3.37 (SE 1.30, p = 0.010), −4.94 (p < 0.001), −5.48 (p = 0.001); EAC +1.50 (SE 0.71, p = 0.036), +3.55 (p < 0.001), +2.98 (p = 0.002), at 7, 12 and 18 months. Each p agrees with its t = delta/SE. The BJP paper gives the same differences with intervals about 1.8 times wider (AAQ-II at 7 months −7.95 to 1.21, P = 0.096; EAC −0.84 to 3.83, P = 0.132); neither document says why. The report’s Discussion calls the 7-month EAC difference “nearly significant”, against its own Table 8.
• Instrument design (§2). Within RO DBT a 2 × 2 × 2 factorial: standard or enhanced therapy room; written feedback to the therapist after each session or not; questionnaire payment by the therapist in session or by post. Therapists were also randomised within centres. Observational instruments were planned besides (therapist style, pre-trial alliance, travel distance, a pre-randomisation homework task, conscientiousness).
• Weak instruments (§3). Cragg–Donald statistics no larger than 1, against 9.08 needed to keep bias under 10%. The IV estimates are “subject to severe weak-instrument bias” and, in the tables’ own note, have no meaningful interpretation; the Discussion concludes there is as yet no proof of principle for IV mediation.
• Decomposition without instruments (Table 17), HRSD at 7 months. Exposure: total −3.83, direct −0.69, indirect −3.14 (−12 to 2.8). Skills: total −3.58, direct +11.27, indirect −14.85 (−19 to −12). Alliance: total −3.83, direct +15.59 (3.5 to 28), indirect −19.42 (−30 to −7.7). The “significant” indirect effects are four to five times the total, with direct effects of the opposite sign; by the report’s own rule under its Equation 3, a proportion mediated is then meaningless.
• All-mediators model (Table 18), n = 150. Total mediated −5.59 (−21 to 5.9); via skills −6.46 (−12.8 to −2.04); via alliance −1.24 (−7.1 to 1.1). The instrumented version has intervals such as −118,730 to 20.
• Not reported. The introduction promises cross-lagged analyses of depression, alliance and skills, and ecological momentary assessment of affect, to establish the order of change; neither appears. The moderators (personality disorder, invalidating childhood, reward sensitivity) are reported only as a passing mention that the PD-by-treatment interaction was not significant (§2, Bayesian model) and the Discussion’s remark that the effect did not seem to depend on comorbid PD.
• Adverse events (§3, Table 7). 33 SAEs: 4 in usual care, 28 in RO DBT, 1 before randomisation. Of the 28, 12 were judged definitely not related, 8 unlikely, 5 possibly and 3 probably related; the 8 possibly or probably related are the serious adverse reactions, all in RO DBT (overdose with suicidal intent 1, overdose without 1, disclosure of thoughts of harming others 3, starting non-suicidal self-injury 2, a prevented suicide attempt 1). 23 of the 28 RO DBT events (82%) were reported by the treating therapists; GPs reported none in either arm. Test: “Fisher’s exact test; one-tailed; p < 0.01”. The DMEC’s finding of no reason to suspect harm is from its last meeting that discussed SAEs, in January 2015.
• Economics (§3). £7050 more per RO DBT participant (95% CI £5822 to £8274) for 0.032 more QALYs (−0.028 to 0.093) at 12 months: £220,000 per QALY; at 18 months usual care dominates.
• Co-treatment. Between 7 and 12 months 44% of usual-care participants had some talking therapy, against under 5% in RO DBT (Table 13 and §4), the authors’ main explanation of usual care’s late improvement.

Claims

id	claim	strength	support
C1	RO DBT plus usual care reduces depressive symptoms more than usual care at the end of therapy (7 months)	moderate	Table 8; one trial
C2	The difference is not significant at 12 or 18 months	moderate	Table 8, pre-registered primary contrast; underpowered
C3	RO DBT improves psychological flexibility and emotional approach coping, significantly at every follow-up	moderate on this report’s analysis; weak across the two reports	Table 8; the BJP paper’s analysis of the same data finds the 7-month differences not significant
C4	The mechanism of RO DBT’s effect can be decomposed into exposure, alliance and skills	not supported	weak instruments; implausible unadjusted decomposition; no temporal order
C5	Skills played the main role	weak	all-mediators model without instruments; assumes no unobserved confounding, which the design was built to avoid assuming
C6	The excess of serious adverse reactions in RO DBT reflects unequal reporting	weak	argued from contact frequency and who reported; not tested
C7	RO DBT is not cost-effective by NICE’s threshold	moderate	base case and four sensitivity analyses

Method

As NOTE-1358 describes for the journal paper: 15 : 8 adaptive randomisation by the Swansea Trials Unit stratified by onset age, baseline HRSD and cluster A or C personality disorder; masked assessors at 7, 12 and 18 months; three-level mixed models fitted in R. The mechanism analysis fits Equations 1–2 (single mediator) and 4–7 (all mediators) by generalized method of moments in Stata, first assuming no unobserved confounding, then with the randomised features and therapist as instruments, with bias-corrected bootstrap intervals.

Concepts

• Process variable: a mediator defined only in the treated arm, fixed at zero in the control arm, so that the treatment’s effect divides into a direct part and a part through the mediator.
• Instrumental variable (as used here): a randomised feature of how RO DBT was delivered that moves a mediator and affects the outcome only through it.
• Serious adverse reaction: a serious adverse event judged possibly, probably or definitely related to the intervention.

Connections

The journal paper, LIT-1223, is the short version, published seven months later; its efficacy numbers are this report’s, its process-measure intervals are not, and it drops the mechanism chapter. The treatment is Lynch’s, from his 2018 manual (Context Press, not registered); the record’s DOI’d statement of it is LIT-1205. The design follows Emsley, Dunn and White’s framework for mediation with randomised instruments in trials of complex interventions (reference 130, not registered). The standard it would have had to meet is THEORY-053’s.

Bearing on the record

• QUESTION-292 and THEORY-053. The triage expected the report’s causal mechanism analyses to be the kind of evidence THEORY-053 asks for. They are the right kind in intent and fail it in fact. THEORY-053 counts change in the theorised process that is specific to it and comes before the outcome. Here the mediators are sessions attended, alliance and DBT skills, of which alliance and exposure are common factors and not RO DBT’s theorised process; the processes RO DBT names, flexibility and emotional expression, are reported only as between-arm outcomes and are never entered as mediators; mediator and outcome are both at 7 months, so precedence is not established; and the planned analyses of temporal order were not reported. The instrumented analysis, the part designed to handle confounding, failed on weak instruments. So the trial meets neither leg of THEORY-053’s standard. It is a worked example of the account’s point that mechanism research has mostly failed on temporal precedence, and of Hofmann and Hayes’s case of a model left untested. It does not disconfirm RO DBT’s theory either: the processes did move, and whether the outcome moved through them was not estimated.
• QUESTION-045. Its note of 2026-10-29 says the BJP paper gives nothing on mechanism. The report gives an attempt and no result, so the note’s conclusion stands, with a refinement: the mediators were analysed here, and the analysis did not identify them. Overcontrol was not measured as such; the trial’s proxy is a cluster A or C personality-disorder diagnosis (78% had at least one Axis-II disorder), used as a stratifier, and its interaction with treatment is reported only as not significant. That is the nearest the trial comes to QUESTION-045’s question, and it is no evidence either way.
• LIT-1223 / NOTE-1358, the P = 0.004. The BJP paper’s “one-tailed P = 0.004” for the eight serious adverse reactions is not in this report, which gives “p < 0.01”. Neither reproduces. With eight reactions in eight different participants, all in the 162 RO DBT participants and none in the 88 controls, the one-tailed Fisher P is C(162,8)/C(250,8) = 0.029, as NOTE-1358 found; with fewer than eight participants affected it is larger. Neither document says how many participants had the eight reactions. The test is still under 0.05 on either count, so the conclusion that the imbalance is significant survives; the stated strength does not.
• NOTE-1358’s “throughout” error. NOTE-1358 calls the BJP abstract’s “throughout” for the flexibility and coping gains an error against its Table 1. This report’s Table 8 supports “throughout” (p = 0.010 and 0.036 at 7 months) with the same differences and narrower intervals. So the conflict is between two analyses of the same data, not an abstract misreporting its own table; the BJP paper’s analysis is the more conservative and is the one its abstract should have matched.
• CLAIM-349. Its note of 2026-10-29 says the change that persisted was on inflexibility and emotional coping while the change on depression did not. The report agrees, and on its own analysis those changes were significant from 7 months. It adds nothing that tests the claim, which is about organizations.

No instruction for machine-learning practice; nothing for the anthology.

Limitations

• The mechanism analysis does not identify anything, by the authors’ own account, and the no-confounding analysis assumes away the problem the design was built to solve.
• Underpowered at the primary end-point: 190 analysed against 245 sought, and therapist clustering far above the assumed 0.025.
• Unequal co-treatment and unequal harm reporting, as in the journal paper; the DMEC’s reassurance predates the end of recruitment (April 2015) and all 18-month follow-up.
• Developer and financial interest, as NOTE-1358 records: the developer leads the report and trained and supervised the therapists.
• Errors and slips found.
  • SAEs “were reported for 25 individual participants in the RO DBT group” (§3), but the same sentence’s breakdown, 28 events of which seven fell to three participants, gives 28 − 7 + 3 = 24, the BJP paper’s figure.
  • The 17 events judged definitely unrelated are split “TAU, n = 4; RO DBT, n = 13” in the text; Table 7 has 12 in RO DBT and the pre-randomisation event as the seventeenth.
  • The one-tailed Fisher test is given as p < 0.01; the counts give about 0.029 (above).
  • The scientific summary gives the 18-month AAQ-II interval as −8.74 to 2.22; Table 8 has −8.74 to −2.22, which its p = 0.001 requires.
  • The text gives the HRSD cost-effectiveness ratio as £7048 per point; Table 16 gives £4122, which is £7048 / 1.71 points. £7048 is the cost difference.
  • The Discussion’s “nearly significant at month 7” for EAC contradicts Table 8 (p = 0.036).
  • The process-measure intervals disagree with the BJP paper’s for the same differences (above); neither explains.

Open questions

• Which analysis of the AAQ-II and EAC contrasts is right, this report’s or the BJP paper’s? The Zenodo data and syntax (record 1442883) would settle it.
• Did the flexibility and coping change come before the change in depression? The cross-lagged analyses the report planned would answer it, and were not reported.
• How many participants had the eight serious adverse reactions, and what P did the authors compute?