The objection
In the central defeat tests three measurements are compared.
- The predictor is the theory’s measure. CLAIM-005’s is “the match between distributions of context-indexed communicative judgements”. CLAIM-115’s is “measured preservation of pragmatic observables (stance, footing, alignment, expected uptake)”, and in the designs these are measured by human judgement: CASE-040’s interpretive judgements are of “tone, stance, solidarity, and authority”. CLAIM-139’s “relational profiles (pragmatic, interactional, intervention-response)” are judgements in part.
- The criterion is human judgements of communicative fidelity, or of shared kind, for the same renderings.
- The baseline is a machine score: propositional, embedding, topic or surface similarity.
Two human judgements of one rendering share sources of variance that an embedding score shares with neither: the rater’s construal of the passage, scale use, a general impression that colours every rating (a rater who finds a rendering wrong in one respect tends to rate it lower in all), and the context and order of elicitation. So the predictor correlates with the criterion partly by method alone, and the comparison with a machine baseline is not like for like. The theory’s measures can beat the baseline without carrying any information the baseline lacks about what the renderings preserve.
The one guard the record names is the manuscript’s. QUESTION-005: “The manuscript preregisters the cover and splits annotators (§10)”. Splitting annotators removes the shared rater. It does not remove a shared construal of the stimulus, or a response style common to raters drawn from one population. No entry in claims, questions, cases or terms mentions method variance.
It rests on CLAIM-233: that objection says the baseline is one no rival holds, and this one says the comparison with it is confounded as well.
What it does not say
- It does not say judgement-based predictors are illegitimate. Pragmatic observables may have no other measure.
- It does not say the theory’s advantage would vanish once method is controlled, only that the stated tests cannot show it survives.
- It does not reach a predictor measured by behaviour rather than by rating.
What would answer it
Any of these:
- a design in which predictor and criterion are measured by different methods, such as a predictor from behavioural uptake (CASE-040’s “expected uptake (choices among continuations)”) against a criterion from fidelity ratings, or the two from different raters on different tasks;
- a baseline of the same kind as the predictor, such as human ratings of propositional sameness of the same renderings, so that method variance is matched;
- a multitrait–multimethod analysis that estimates the method share and shows the theory’s advantage exceeds it.
Note of 2026-10-28: split under ADR-061
The old title ended in a verdict on support: the theory’s measures “can win by shared method variance alone”. That verdict is now ARG-026, from the central defeat test (CASE-184) to CLAIM-115, CLAIM-139, CLAIM-005 and CLAIM-078, form evidence, Rejected, as its answer to “does the measure measure what it is said to?”. This claim keeps the fact about the measurements, what two human ratings of one rendering share, and undercuts that argument. It no longer rests on CLAIM-233: as its body says, that objection is about the baseline and this one about the comparison, two failed questions of one inference, so the two complement each other. The text above is unchanged.