Singh, U., Veera Balaji Kumar, P., Nanda, P., Madhusoodanan, A., Guo, G., Xu, W., & Li, J. J. (2026). VakyArth: Evaluating pragmatic competence in LLMs across Indic languages. Findings of EMNLP 2026, 43027–43041.
Read in full in NOTE-1345.
Why this is here
A5’s reason: it sets translation tasks and pragmatic inference on related material, which is the separation CLAIM-202 wants measured, and A5 reported its finding that fluent translations can miss pragmatic meaning. Its subject is model evaluation; CASE-185 would need its items read by human receivers.
Correction from the reading: the material is related by phenomenon only. VakyArth’s 204 translation items and 374 MCQ and NLI items are separate, so it cannot supply the item-level separation CLAIM-202 wants, and the finding A5 reported rests on four hand-picked examples (NOTE-1345).
Key takeaways
- Translation items and inference items are different utterances (1 of 204 overlaps), so no item-level dissociation can be read from it (NOTE-1345).
- Every gold label and every human rating comes from one native speaker per language; cells hold 5–16 items.
- Models’ errors lean to the literal reading; MCQ beats NLI in 20 of 20 model–language pairs.