Study: LLMs evaluated on MQM and ESA show mixed, often unreliable agreement with human MT annotators
The paper evaluates large language models on two MT evaluation schemes—Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA)—using a long-context test set covering 70 language pairs and the WMT23 and WMT25 datasets. Results show LLM agreement with humans sometimes exceeds inter-annotator agreement for some tasks, but performance varies widely across schemes, languages and domains and is generally unreliable; humans struggle with fine-grained MQM and low-resource pairs, while LLMs miss minor errors, language-variant issues and error-span annotation.