Source
Ward and Marco (2024), LIT-1812, §§6–8, Tables 2–3, pp. 157–160 of the LREC-COLING 2024 proceedings; read in NOTE-1338.
What was actually shown
Judges rated each pair of a dialog utterance and its re-enactment on a 1–5 slider, for similarity “in terms of the overall feeling, tone, and intent” (Fig. 1). Session 1: 220 English pairs, 9 judges; Session 2: 238 English pairs, 9 judges; Session 3: 235 Spanish pairs, 6 judges. Mean pairwise Pearson correlation between judges: 0.45, 0.72, 0.66 (Table 2). In Session 1 the pairwise values ran 0.18 to 0.80 (Table 3; mean recomputed as 0.447). Leaving out the lexically different pairs (Lexically-Distinct and Context-Only re-enactments) raised Session 2 to 0.80 (p. 160).
Treating judges as exchangeable, the Spearman–Brown reliability of the panel mean is 0.88, 0.96 and 0.92 by session. That is my computation, not the paper’s.
What could have failed: judges could have agreed near zero, as some pilot judges’ lack of confidence suggested (p. 156). They did not, even in the first session.
What this does not say
- That pragmatic similarity is rated at 0.45. That is the first session and a single-judge figure. The second English session, the Spanish one and any panel mean are higher.
- That the agreement would hold for strangers rating alone. Judges discussed their ratings during the session (p. 158), were socially connected, and included the authors (p. 157).
- Anything across languages. Every pair is within English or within Spanish.
- That a judge is self-consistent. No second pass was run.
- What the judges attend to. They reported different cues (p. 156), and the instruction mixes static tone with likely listener response (Fig. 1). The agreement is on a blend, not on one construct.