Srinivasan, A., & Choi, E. (2022). TyDiP: A dataset for politeness classification in nine typologically diverse languages. Findings of EMNLP 2022, 5723–5738.
Read in full in NOTE-1339.
Why this is here
A5 offered it as a measurement resource for CASE-055’s Stage 0, not as a definition of fidelity: politeness labels across languages, with their reliability limits.
The reading bears this out: TyDiP does report human agreement, which makes it a reference for a Stage 0 reliability target, but its annotators were screened for agreeing with a model and no judge rated twice (NOTE-1339).
Key takeaways
- Human agreement is reported: 0.33–0.60 by language, pairwise Spearman over 15-item tasks; no test–retest (NOTE-1339).
- The same people agree less in their second language, in eight of nine languages.
- Annotators were kept for agreeing with a classifier, so its agreement and human-accuracy figures are upper estimates.