Contribution
Before this there were measures of semantic and prosodic similarity between utterances and none of pragmatic similarity, and no human judgments to train or test one (§§1–3, pp. 154–155). The paper collects them. Each stimulus is a seed utterance from a recorded dialog (mostly the DRAL corpus) followed by a re-enactment, made by one of four re-enactors or a synthesizer under one of six conditions that withhold different information (Table 1, p. 157). The release has the stimuli, seeds, re-enactments, source conversations and all ratings (p. 158). The paper also reports the protocol, three pilot studies and post-hoc factors behind agreement.
Key insight
Pragmatic similarity can be rated directly, from audio, with no taxonomy of functions, and trained judges converge on it about as well as people converge on most subjective audio judgments. Their agreement depends on what is being compared. Judges agree less when the two utterances differ in words, and they disagree on which cues count, rate of speech for one, endings for another (p. 156). The authors chose to “simply accept that people differ in how they included and weighed the possible factors underlying these judgments, and not try to constrain them in some way to increase agreement scores” (p. 156).
Assumptions
- Within-language pairs. Seed and re-enactment are always in the same language. There are no cross-language pairs; the Spanish session is a separate collection (§5.2, p. 157).
- Isolated utterances. Stimuli are played without context (pilot 4, p. 156), back to back, seed first (p. 157). Seeds are 3 words to 6 seconds, average 3 seconds, “mostly clear in intent and meaning when heard in isolation”, with no reported speech (p. 157).
- One holistic instruction. “How pragmatically similar are the two clips, in terms of the overall feeling, tone, and intent”, ignoring speaker, word and insignificant prosodic differences, and “Maybe consider” likely contexts, “how a listener would likely respond”, the speaker’s feeling and the dialog activity (Fig. 1, p. 158).
- A selected, socially connected panel. Judges were invited from earlier eager participants and the lab; “Most were in their early 20s” (p. 157), and all were connected to at least one other judge. The first author judged in Sessions 1 and 2, the second author in Session 3 (p. 157); the authors were also two of the four re-enactors (p. 157).
- Discussion during rating. “for the first ten stimuli, and periodically throughout, we had them compare their ratings and discuss the factors that had affected their judgments” (p. 158). Judges were told “that convergence was not expected” (p. 158).
Key results
- Size (Table 2, p. 158). Session 1: 220 English pairs, 9 judges, 1,980 ratings. Session 2: 238 English pairs, 9 judges, 2,142. Session 3: 235 Spanish pairs, 6 judges, 1,410. Total 5,532 (checked). From 80 English and 40 Spanish seeds (p. 157). The repository README gives 1,872 ratings for the second English session and 689 stimuli in all; I did not reconcile these with the paper.
- Agreement (Table 2). Mean pairwise Pearson correlation between judges: 0.45, 0.72, 0.66. The abstract states the last two: “as high as 0.72 for English and 0.66 for Spanish” (p. 154). Pearson, because scatterplots “did not reveal any nonlinear patterns” (p. 158).
- Session 1 in detail (Table 3, p. 160). 36 pairwise correlations from 0.18 to 0.80, mean 0.447 (recomputed); per-judge means 0.29 to 0.53. The judge at 0.29 gave “ratings that were mostly just integers” (p. 159).
- What lowers agreement (§8.2, pp. 159–160). Judge identity; inexperience (Session 1 to 2, though “the least-agreeing judge did not return for that session”, p. 159); low-rated stimuli (correlation of per-item SD with mean rating −0.60, 0.00, −0.39); different words: “for Session 2, excluding those stimuli gave an average inter-judge agreement of 0.80, significantly above the 0.72 seen for all stimuli” (p. 160); duration differences (SD correlates 0.30 with duration difference in Session 1). Of the 27 Session 1 stimuli with SD > 1.0, many were Context-Only re-enactments, seeds with laughter or breathiness, or seeds with unusual warmth (Table 4).
- Conditions as a check (§8.1, p. 159). Voice re-enactments were rated most similar and synthesized ones only weakly, as designed. Two expectations failed: Words-Only beat Context-Only, and Words-Only beat Words+Context; Lexically-Distinct was barely above Words-Only.
- Drift. Mean rating per stimulus correlates with serial position 0.04, 0.09, 0.16 by session (p. 159).
- Reliability of the panel mean, my computation. If judges are exchangeable, the Spearman–Brown reliability of a k-judge mean at mean inter-judge r is kr / (1 + (k − 1)r): 0.88 for Session 1 (r = 0.45, k = 9), 0.96 for Session 2, 0.92 for Session 3 (k = 6). The paper does not report these, and the authors themselves “would be mostly using the averages of everyone’s judgments” (p. 158).
Claims
| id | claim | strength | support |
|---|---|---|---|
| C1 | Human pragmatic-similarity ratings of utterance pairs can be collected at scale, 693 pairs and 5,532 ratings, and are released | strong | Table 2; repository |
| C2 | Single judges agree at mean pairwise r 0.45–0.72 by session | strong | Tables 2–3; Table 3 recomputed |
| C3 | Agreement rises with experience | weak | one session-to-session rise confounded by losing the least-agreeing judge; small within-session correlations, sign reversed in Session 3 |
| C4 | Agreement is lower when the utterances differ in words | moderate | 0.80 against 0.72 in Session 2; post-hoc; no test statistic reported for “significantly” |
| C5 | The re-enactment conditions produce the intended range of similarity | moderate | Fig. 3, Session 1 only; two ordering predictions failed |
| C6 | The ratings will enable models that approximate perceived pragmatic similarity | weak | an aim (§9); no model is trained |
Concepts
- Pragmatic similarity — as instructed to judges: similarity “in terms of the overall feeling, tone, and intent” (Fig. 1). It includes, by suggestion, likely contexts and likely listener response.
- Seed and re-enactment — a clip from a real dialog, and a new production of it under one of six conditions: Voice, Words+Context, Lexically Distinct, Context-Only, Words, Synthesized (§5.1).
- Inter-judge agreement — the mean of pairwise Pearson correlations between judges’ ratings over a session’s stimuli (p. 158).
Connections
- Avila and Ward 2023, the DRAL corpus the seeds come from and a prosodic distance metric over 100 features that this data could evaluate (p. 155). Not held.
- Pragst 2022, the one earlier attempt at pragmatic similarity, text only and on artificial data (p. 155). Not held.
- TyDiP (LIT-1799, read the same day) reports agreement for a narrower judgement, politeness, as pairwise Spearman: 0.46 in the target language. The two numbers are close but measure different things.
Bearing on the record
- CASE-185: its “main threat” sentence misreads the 0.45. The case says “PragSim’s judges agreed at a mean correlation of 0.45 (LIT-1812), so Stage 0’s reliability targets must be met before any transport readout is interpreted”. The conclusion stands; the premise is narrower than it reads. (a) 0.45 is Session 1 only; the other sessions gave 0.72 and 0.66, and the abstract reports those. (b) It is the agreement between two single judges. The case’s criterion is the panel’s choice, and the mean of nine judges at r = 0.45 has an estimated reliability of about 0.88 (my Spearman–Brown, assuming exchangeable judges). (c) In the other direction, the figure is inflated by things a translation study would not have: judges discussed their ratings in the room, knew each other, and two were authors; and pairs sharing words agree best (0.80 against 0.72). Every translation pair differs in its words, so the agreement that matters for the case is the one for the Lexically-Distinct and Context-Only pairs, which the paper says is lower and does not report.
- CASE-055, Stage 0: a protocol to borrow, and a reliability figure that is not the one Stage 0 asks for. Stage 0 wants written definitions, a fixed format, agreement between judges and between two passes of one judge, and known-groups validation. The paper has a written instruction (Fig. 1), a fixed continuous format with anchors (p. 158) and inter-judge agreement. It has no second pass by the same judge. Its re-enactment conditions are a rough known-groups test, and it half passes: Voice above Synthesized as built, Words-Only above Context-Only and Words+Context against the design. The paper’s open admission that judges weigh different cues (p. 156) is the measurement problem CLAIM-227 names, made visible.
- CLAIM-202: a warning that the “static” baseline may not be static. The claim asks whether context-indexed response adds to the static pragmatic profile. Ward and Marco’s judges were told to “Maybe consider … Similarity in how a listener would likely respond” and “Similarity in the contexts where they would likely appear” (Fig. 1). A similarity metric trained on these ratings would fold expected uptake into the baseline. A comparison under CLAIM-202’s defeat condition would have to say which baseline it beats.
- CLAIM-226: in-session calibration is the opposite of what the claim asks. The paper averages a socially connected panel that discussed its ratings, and treats disagreement as something to accept and average (p. 158). It does observe structured differences between judges (strict against generous, integer users, extremes; p. 159), which supports the claim’s premise that judges differ in a structured way.
- CLAIM-257: no bearing.
- Should produce THEORY-1148 (Proposed): single-judge agreement on pragmatic similarity is moderate and depends on lexical overlap and experience, and a panel mean is far more reliable.
- Machine-learning practice: an anthology candidate. §8.3 tells modelers
to use “the Session 1 data for training and the Session 2 data for
evaluation” and to z-normalize each judge before averaging (p. 160). That
is an instruction for training and evaluating a metric. The LIT carries
anthology-candidate. - Correction to the LIT’s description. The summary’s “nine judges rated 220 pairs … with a mean inter-judge correlation of 0.45” describes Session 1 only. The collection is 693 pairs in three sessions with 6 to 9 judges each.
Limitations
- Within-language only; nothing about pragmatic similarity across languages, which is what translation needs.
- No test–retest, and no agreement figure by re-enactment condition.
- A small, homogeneous panel: mostly early-20s, connected, from one lab’s circle in El Paso, with the authors among them.
- Post-hoc analysis. The authors say “the data was not collected to support a systematic analysis; the rest of this section only reports the factors identified using post-hoc tests” (p. 159).
- Isolated clips, which the authors chose knowingly: context “would likely boost the inter-annotator agreement” but cost time (p. 156).
- Counts differ between the paper and the repository README (2,142 against 1,872 ratings for Session 2).
Open questions
- What is inter-judge agreement for lexically different pairs alone, by session? The released ratings can answer it.
- How consistent is one judge across two passes on the same pairs?
- Would cross-language pairs, a seed and a translated re-enactment rated by bilinguals, reach the same agreement? That is the measurement CASE-185 needs and nobody has collected.