QUESTION-355: Measured against the agreement between two or more native raters scoring the same translations, how close does a reference-based metric such as COMET come to that ceiling in scoring the pragmatic adequacy of LLM translations of culturally loaded utterances into English, per pragmatic phenomenon, on items chosen in advance?

Why it is a question

THEORY-1120 holds what Singh et al. measured: COMET correlates with one native rater per language at Pearson r = 0.813. With no second rater there is no inter-rater agreement, so how good that is cannot be read, and the per-phenomenon differences of 0.1 or less were not tested.

What would answer it

A study in which at least two native raters per language independently score the pragmatic adequacy of the same translations, with their agreement reported, and in which a reference-based metric’s correlation with the raters’ mean is reported against that inter-rater ceiling, per phenomenon, on items chosen in advance rather than picked after the fact. A correlation with one rater per language does not settle it.