Ward, N. G., & Marco, D. (2024). A collection of pragmatic-similarity judgments over spoken dialog utterances. Proceedings of LREC-COLING 2024, 154–163.
Read in full in NOTE-1338.
Why this is here
A5’s warning for CASE-055’s Stage 0, kept in CASE-185: if judges agree on pragmatic similarity at 0.45, a model can look weak because the target is unstable, or strong because it reproduces one panel’s habits.
Correction from the reading: 0.45 is the first of three sessions and a single-judge figure (the others are 0.72 and 0.66), and a nine-judge mean at 0.45 is far more reliable. The warning holds in another form: the agreement for pairs that differ in words, as translations do, is lower and unreported (NOTE-1338).
Key takeaways
- 0.45 is Session 1 and one judge against another; a nine-judge mean at that level has an estimated reliability near 0.88 (NOTE-1338).
- Agreement is lowest for lexically different pairs, the kind every translation is, and the paper does not report it separately.
- Within-language only; no test–retest.