nucleation

QUESTION-345: Does a learned translation metric trained on synthetic hallucinations and perturbations detect errors and hallucinations from real systems, in pairs and kinds its training did not synthesise, as well as its synthetic tests report?

1 min read

Properties4
StatusOpen
Tagstranslation
Linepragmatic-transport
Cited as “bears on” byTHEORY-1151 — A learned translation metric trained with synthetic hallucinations and diagnostic perturbations alongside human MQM flags most injected localized critical errors (all but 0 to 17 percent, by type and model size) and ranks fully detached hallucinations below other translations with AUROC up to 0.96, and without a reference it ranks hallucinations better than with one; how far this reflects the synthetic training data has not been separated from how far it generalises

Graph View

Why it is a question

THEORY-1151 holds what Guerreiro et al. (2024), LIT-1816, measured: xCOMET flags most injected localized errors and ranks fully detached hallucinations low. But its training data included synthetic detached and oscillatory hallucinations and DEMETR’s diagnostic perturbations, the localized errors were SMAUG perturbations of English targets only, and the hallucinations came from one de-en benchmark. The authors give the synthetic training data as “one plausible hypothesis” for the gains and do not claim that detection generalises. Whether it does is open.

What would answer it

Tests on errors and hallucinations produced by real translation systems in language pairs and kinds the training data did not synthesise (for instance a multilingual hallucination benchmark, or naturally occurring number and entity errors with human labels), showing whether detection stays at the level the synthetic tests report. Results on more synthetic perturbations of the trained kinds do not settle it.

Backlinks