nucleation

LIT-1816: xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection

1 min read

Properties5
StatusActive — read in full 2026-10-29 (NOTE-1341) from the TACL version of record, with the arXiv v1 preprint compared; worth reading as the reference description of a strong open MT metric and its limits: segment-level Kendall τ near 0.43 against expert MQM, span F1 near 0.28, and score and spans that agree with each other far better than with the experts.
Version2
Tagstranslation, anthology-candidate
Cited as “paper” byNOTE-1341 — Guerreiro et al. — xCOMET
Cited as “source” byTHEORY-1134 — A translation metric that predicts a sentence score and token-level error spans from one shared encoder reaches segment-level agreement with expert MQM annotation of Kendall τ about 0.4 and span F1 under 0.33, the best among open metrics and GPT-4 prompting at the time, while the score computed from its own spans correlates with its regression scores at Pearson 0.73 to 0.82, and at 0.90 to 0.92 with its final score, which includes the span score by construction, THEORY-1151 — A learned translation metric trained with synthetic hallucinations and diagnostic perturbations alongside human MQM flags most injected localized critical errors (all but 0 to 17 percent, by type and model size) and ranks fully detached hallucinations below other translations with AUROC up to 0.96, and without a reference it ranks hallucinations better than with one; how far this reflects the synthetic training data has not been separated from how far it generalises

Graph View

Guerreiro, N. M., Rei, R., van Stigt, D., Coheur, L., Colombo, P., & Martins, A. F. T. (2024). xCOMET: Transparent machine translation evaluation through fine-grained error detection. TACL, 12, 979–995.

Why this is here

A5 named it, with the COMET toolkit, as the strong baseline a pragmatic-transport account must beat, not as a criterion of fidelity; CASE-071 already shows a metric’s preferences parting from human raters’. A baseline in CASE-185. anthology-candidate: an evaluation tool for ML practice.

The reading (NOTE-1341) bears out the exchange’s placing. The paper tests xCOMET only as a scorer of given translations; it is not tested as a selection criterion, which is where CASE-071’s bias arises.

Backlinks