RESEARCH · RESEARCH · #1892
Study: LLMs evaluated on MQM and ESA show mixed, often unreliable agreement with human MT annotators
The paper evaluates large language models on two MT evaluation schemes—Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA)—using a long-context test set covering 70 language pairs and the WMT23 and WMT25 datasets. Results show LLM agreement with humans sometimes exceeds inter-annotator agreement for some tasks, but performance varies widely across schemes, languages and domains and is generally unreliable; humans struggle with fine-grained MQM and low-resource pairs, while LLMs miss minor errors, language-variant issues and error-span annotation.
KEY POINTS
- The paper evaluates large language models on two MT evaluation schemes—Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA)—using a long-context test set covering 70 language pairs and the WMT23 and WMT25 datasets.
- Results show LLM agreement with humans sometimes exceeds inter-annotator agreement for some tasks, but performance varies widely across schemes, languages and domains and is generally unreliable; humans struggle with fine-grained MQM and low-resource pairs, while LLMs miss minor errors, language-variant issues and error-span annotation.
- This matters because it shows LLMs are not yet a reliable drop-in replacement for human MT quality annotation across languages, schemes and domains, affecting evaluation pipelines and dataset quality.
WHY IT MATTERS
This matters because it shows LLMs are not yet a reliable drop-in replacement for human MT quality annotation across languages, schemes and domains, affecting evaluation pipelines and dataset quality.