nucleation

THEORY-1151: A learned translation metric trained with synthetic hallucinations and diagnostic perturbations alongside human MQM flags most injected localized critical errors (all but 0 to 17 percent, by type and model size) and ranks fully detached hallucinations below other translations with AUROC up to 0.96, and without a reference it ranks hallucinations better than with one; how far this reflects the synthetic training data has not been separated from how far it generalises

2 min read

Properties5
StatusActive
Version2
Tagstranslation
SourceLIT-1816 — xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection
Bears onQUESTION-345 — Does a learned translation metric trained on synthetic hallucinations and perturbations detect errors and hallucinations from real systems, in pairs and kinds its training did not synthesise, as well as its synthetic tests report?

Graph View

Source

Guerreiro et al. (2024), LIT-1816, §7 (pp. 987–989) of TACL 12; read in NOTE-1341.

What was actually shown

Localized errors. SMAUG inserted text, negations, mask in-fills, number swaps and named-entity swaps into WMT23 zh-en and he-en synthetic references the metric had scored error-free. The share of perturbed translations still predicted error-free ranges from 0.00% (he-en number swaps, XXL) to 17.0% (zh-en mask in-fill, XXL) (Table 6, p. 987). The predicted spans overlap the perturbation in 98% (XL) and 90.9% (XXL) of cases (p. 987). Negation and number errors bring median score drops of about 20 points (Fig. 4b, p. 988).

Hallucinations. On the de-en benchmark of Guerreiro et al. (2023), over 3.4k translations by a real system, AUROC for ranking hallucinations below other translations is 0.890 overall and 0.964 for fully detached ones with xCOMET-XXL using the reference, and 0.902 overall without it (Table 8, p. 989). Over 90% of fully detached hallucinations get scores under 10 (p. 988). The authors attribute the reference-free advantage to “the metric over-relying on the reference when it is available” (p. 989).

What could have failed. Number and entity errors were weaknesses of earlier COMET metrics (p. 988); xCOMET could have kept them.

What this does not say

  • That the robustness generalises beyond what was synthesised. The training set includes synthetic detached and oscillatory hallucinations and DEMETR’s diagnostic perturbations (p. 984), and the authors give that as “One plausible hypothesis for these improvements” (p. 988). The localized errors are synthetic, on English targets only.
  • That bigger is better here. XXL is worse than XL on oscillatory hallucinations, 0.844 against 0.922 with the reference (Table 8).
  • The overall localized-error rate. The table’s “All” row does not agree with the count-weighted mean of its rows (NOTE-1341); the per-type figures are the ones to use.
  • That severity is well calibrated. Within a sentence with several predicted spans, severities differ only about 35% of the time (p. 987).
  • Anything about omissions or partial hallucinations beyond the benchmark’s aggregate.

Note of 2026-10-30: promoted

Promoted as a report of what the paper measured (ADR-065). Whether its robustness holds on real errors in pairs and kinds its training did not synthesise is QUESTION-345.

Backlinks