NOTE-1341: Guerreiro et al. — xCOMET

Contribution

A learned translation metric that predicts, in one model, a sentence-level quality score and a severity tag (OK, minor, major, critical) for every token of the translation, under three input modes: translation with reference only, with source and reference, and with source only (quality estimation) (§§3–4, pp. 981–983). Its final score mixes the three regression outputs with an MQM-style score computed from its own predicted error spans. It is trained by a three-phase curriculum on public WMT data plus synthetic hallucinations (§4.4, p. 984), released as xCOMET-XL and xCOMET-XXL, and evaluated against neural metrics and GPT-based evaluators on WMT22 and WMT23 (§§5–6), on synthetic localized errors and on a hallucination benchmark (§7), with ablations (§8).

Key insight

A sentence score and an error-span explanation can be produced by one encoder and made to agree: “these high correlations with the predicted scores from each forward pass (ŷSRC, ŷREF, ŷSRC+REF) are obtained despite no explicit alignment mechanism governing the relationship between the predictions of the sentence-level and word-level heads. We hypothesize that it is thus the shared encoder that, during the multi-task training, aligns the representations between the two tasks” (pp. 986–987). The reading the record needs is the converse: the agreement between the two heads is internal coherence, and its size says nothing about how often either is right.

Assumptions

• Supervision is human DA and MQM from WMT 2017–2022, plus IndicMT, DEMETR, MLQE-PE hallucinations and synthetic hallucinations with their sentence scores set to 0 (pp. 983–984; Table 1). The test set, WMT22 News MQM, is excluded from training (p. 983, footnote 5), but its annotation protocol, and in part its annotators’ conventions, are those the model was trained on.
• The MQM score from spans is fixed by eqs. (5)–(6): e(S) = c_MIN + 5c_MAJ + 10c_CRIT and ŷ_MQM = (25 − e(S))/25, floored at 0.
• The final score is ŷ_SL = wᵀ[ŷ_SRC, ŷ_REF, ŷ_SRC+REF, ŷ_MQM] with w = [1/9, 1/3, 1/3, 2/9] (eq. 7, p. 983).
• Severity in evaluation. Critical errors are converted to major to match the test set’s guidelines (p. 985, footnote 10).

Key results

Agreement with human judgements (§6, pp. 985–987)

• Segment level, WMT22 MQM (Table 2a, p. 986): Kendall τ averaged over zh-en, en-de, en-ru is 0.421 (XL) and 0.432 (XXL), against METRICX 0.421, BLEURT-20 0.365, COMET-22 0.361 and GEMBA-GPT4-DA 0.354. The MQM score computed from the predicted spans alone gets 0.402 and 0.395.
• System level (Table 2b): pairwise accuracy 0.778 (XL), 0.819 (XXL); GEMBA-GPT4-DA 0.825. On WMT23 pooled (Table 3, p. 986), 0.912 and 0.920 against GEMBA-MQM 0.944, but the best average correlation (0.813, 0.812 against METRICX-23’s 0.808): “Our metrics, at both scales, would win the shared task” (p. 986).
• Error spans (Table 4, p. 986): character-level F1 averaged 0.269 (XL) and 0.280 (XXL), against AutoMQM with GPT-4 at 0.262; 0.242 and 0.255 without a reference.
• The heads agree (Table 5, p. 987): Pearson correlation of ŷ_MQM with ŷ_SRC, ŷ_REF and ŷ_SRC+REF is 0.73–0.82 per language pair, and with ŷ_SL 0.90–0.92. The last is partly built in: “The computation of ŷSL, contrary to the other scores, makes direct use of ŷMQM” (Table 5 caption).

Robustness (§7, pp. 987–989)

• Localized errors (SMAUG perturbations of WMT23 synthetic references the metric had found error-free, zh-en and he-en). Table 6 (p. 987): 0.00–17.0% of perturbed translations are predicted error-free, by type, size and pair. Predicted spans “overlap with the artificially induced perturbations in over 90% of the perturbed samples (98% for XL and 90.9% for XXL)”, but the model also marks spans beyond the perturbation, and “When the metric predicts multiple error spans in a sentence, it assigns different severity levels to those spans only about 35% of the time” (p. 987).
• The “All” row of that table does not follow from its rows. Weighted by the preprint’s per-category counts (its Table 8), the per-type rates give 3.76% (zh-en XL), 10.96% (zh-en XXL), 5.53% (he-en XL) and 5.38% (he-en XXL), against the printed 2.24, 10.7, 9.81 and 7.00. The he-en XL value equals that column’s largest entry (Swap NE, 9.81), which no average of the column can be unless every example were of that type. The preprint’s Table 8 also misadds: its zh-en categories sum to 4,562, not the printed 4,762. The journal version keeps the table and drops the counts.
• Score sensitivity (Fig. 4b, p. 988): median decreases of “around 20 points” for negation and number and named-entity swaps; “In rare cases, perturbations may actually lead to an increase in the score. Nevertheless, upon closer inspection, for over 90% of cases, the increase is smaller than 1 point” (p. 988). (The preprint said “decrease” here; the journal corrects it.)
• Hallucinations (Table 8, p. 989; de-en benchmark of Guerreiro et al. 2023, over 3.4k translations): AUROC for ranking hallucinations below other translations is 0.865 (XL) and 0.890 (XXL) overall, 0.907 and 0.964 for fully detached hallucinations, and 0.922 and 0.844 for oscillatory ones. Source-only scoring does better overall (0.885, 0.902), which the authors attribute to “the metric over-relying on the reference when it is available” (p. 989). XXL is worse than XL on oscillatory hallucinations in both modes, unremarked.

Ablations (§8, pp. 989–991)

• Curriculum (Table 10, p. 990): multi-task training beats single-task training at segment level but not at span detection: “The best word-level model is obtained by doing Phase II with a word-level only objective” (p. 990), with F1 0.285 against the final model’s 0.269.
• Weights (Table 11, p. 990): including ŷ_MQM raises segment-level τ, but “the improvement of our final aggregated score over ŷSRC+REF is not substantial” (0.421 against 0.409; p. 991).
• A typo: Table 10’s last row prints τ = 0.597 for en-de; the same model’s en-de τ is 0.414 in Table 2a and Table 11, and the row’s average, 0.421, fits 0.414, not 0.597.

Claims

id	claim	strength	support
C1	xCOMET’s segment-level agreement with expert MQM on WMT22 is at or above other open metrics and GPT-4 prompting	moderate	Table 2a with significance clusters; one test set, three pairs
C2	Its error spans match human MQM spans better than GPT-3.5/4 AutoMQM	moderate	Table 4; F1 0.27–0.28 against 0.26
C3	Its two heads agree strongly with no explicit coupling, via the shared encoder	moderate	Table 5 (0.73–0.82); the 0.90–0.92 is partly by construction; the mechanism is a hypothesis
C4	It identifies most injected localized critical errors	moderate	Table 6 and §7.1; synthetic SMAUG errors in English targets only; “All” row inconsistent
C5	It ranks fully detached hallucinations low (AUROC 0.96, XXL)	moderate	Table 8; one de-en benchmark
C6	It “would win” the WMT23 metrics task	moderate	Table 3, computed against official numbers after the fact
C7	Spans give “a potential lens through which we can better understand, contextualize, and even debug its own sentence-level predictions” (p. 987)	weak	no study of users or of debugging; agreement between heads only

Method

XLM-R XL or XXL encoder; input: a [cls] token, the translation, then the source, the reference or both; a feed-forward regression head on [cls] and a linear tagger on translation subwords (Fig. 2, p. 982). Loss per input mode (1 − λ)·MSE + λ·weighted cross-entropy, summed over the three modes (eqs. 1–4). Phase I DA only (λ = 0), Phase II MQM with λ = 0.983, Phase III high-quality MQM from Freitag et al. 2021a with λ = 0.055 (p. 984). At inference the tag distributions of the three passes are averaged and adjacent error subwords grouped into spans, each taking its most severe tag (p. 983).

Concepts

• MQM — Multidimensional Quality Metrics: professional annotators mark error spans with severity and category (p. 980).
• DA — direct assessment: a 0–100 score per translation, from non-experts or professionals, z-normalised (pp. 980, 984).
• Unified evaluation — one model serving reference-only, source-and-reference and source-only modes (§4.2, p. 982).
• Detached / oscillatory hallucination — a translation unrelated to the source, or one that repeats an n-gram (p. 984).
• Pairwise accuracy — the fraction of system pairs a metric orders as humans do (p. 985).

Connections

• Built on COMET (Rei et al. 2020) and the scaled COMETKIWI (Rei et al. 2023a); its error-span-to-score step follows AUTOMQM (Fernandes et al. 2023) and INSTRUCTSCORE (Xu et al. 2023); the unified input follows UNITE (Wan et al. 2022b) (pp. 980–983).
• The test protocols are WMT’s (Freitag et al. 2022, 2023); significance by Deutsch et al.’s Perm-Both test (p. 985).
• CASE-071 (Kovacs, Deutsch and Freitag, in the record) is the case where translations a neural metric selects are rated better by that metric and its neighbours and not by human raters. xCOMET is one such neighbour; the paper does not test it as a selection criterion, though it lists “acting as reward models” among learned metrics’ uses (p. 981).

Bearing on the record

• CASE-071: consistent; xCOMET does not escape the case’s mechanism and is not tested against it. All of xCOMET’s evidence is correlation with human MQM on held-out segments it did not choose. CASE-071’s failure arises when the metric does the choosing. A metric at τ ≈ 0.43 leaves most of the ordering of segment pairs to disagree with experts in some part, and nothing here bounds how selection exploits that. The record should cite xCOMET as a strong scorer of given translations, not as a fidelity criterion for translations optimised against it.
• CASE-185: a usable baseline, with two qualifications. The study needs a task-independent quality score per route. xCOMET gives one in all three modes, and its source-only mode can score a relay hop where no reference exists in the pivot language. But its source-only mode is the weakest at segment level (Table 11: τ 0.376 against 0.409 with both inputs), and its spans’ severities are poorly differentiated within a sentence (about 35%, p. 987), so span counts are a coarse readout. Its hallucination detection is good enough to serve as one of the study’s negative controls (Table 8).
• CLAIM-496: an instance of coherence mistaken for confirmation. The paper presents the agreement of its score with its own spans (Pearson up to 0.92, partly built in by eq. 7) as support for the spans’ value. That is agreement between two routes through one encoder. Against the human annotation the spans reach F1 0.28. The record’s distinction between path agreement and fidelity applies: the explanation is coherent with the score far more than either is correct. CLAIM-507 states this, and on one scale finds the gap smaller than “far”.
• Two THEORYs: THEORY-1134 (agreement with experts against agreement between heads) and THEORY-1151 (synthetic-error and hallucination robustness). Both Proposed when filed; both promoted on 2026-10-30, THEORY-1134 narrowed first.
• Instruction for ML practice: yes, belongs to the anthology. xCOMET is a released evaluation tool, and the paper advises on its use (“under computational constraints, one could consider computing a single ŷSRC+REF score without the need for three different forward passes and error span prediction”, p. 991). The LIT keeps anthology-candidate.

Limitations

• One test domain per year. WMT22 News for the main tables, WMT23 for aggregates; three pairs each, all into or out of English.
• Synthetic stress tests. The localized errors are SMAUG perturbations applied to English targets only (p. 987); the hallucinations are one de-en benchmark. The training data includes synthetic hallucinations of the same kinds (p. 984), so the stress tests partly test what was trained.
• The span–score agreement is presented as interpretability. No user study, and part of the agreement is arithmetic (eq. 7).
• Inconsistent table: the localized-error “All” row, as above; and a typo in Table 10.
• The authors’ own limits: low-resource pairs and other hallucination benchmarks are future work (preprint p. 10; the journal drops the sentence); improving differentiation among multiple errors is “an interesting avenue for future research” (p. 987).

Open questions

• Does xCOMET, used to select among candidate translations, show CASE-071’s metric bias? An MBR or reranking experiment with human MQM on the picks would settle it.
• How well do xCOMET’s spans agree with human spans on the kind of errors the record cares about, act and register rather than accuracy? MQM’s typology has a style category; no result here is broken down by category.
• Is the source-only advantage on hallucinations a sign of reference over-reliance generally? A test where the reference is paraphrased or wrong would say.