How close are LLMs to human judgment in assessing machine translation quality?

• Zhenhai Li ^[1] ; Xindi Hao ^[1] ; Shuyin Zhang ^[1]
  1. [1] Chinese University of Hong Kong

     Chinese University of Hong Kong

     RAE de Hong Kong (China)

     
• Localización: MonTI: Monografías de traducción e interpretación, ISSN 1889-4178, ISSN-e 1989-9335, Nº. 18, 2026 (Ejemplar dedicado a: La inteligencia artificial al servicio de la labor de la traducción: investigaciones en torno a los nuevos avances tecnológicos / Artificial intelligence at the service of translation: research on new technological achievements / coord. por Diana González-Pastor , Celia Rico Pérez ), págs. 214-248
• Idioma: inglés
• DOI: 10.6035/MonTI.2026.18.07
• Enlaces
  • Texto completo
• Resumen
  • English

    This paper examines the potential of large language models ("LLMs") in the automatic evaluation of "AI"-generated text. The study compares machine translation quality scores generated by the top 20 "LLMs" ranked on "Chatbot Arena" with human evaluation scores to identify the most robust models for achieving human-like judgments. "English-Chinese" sentence pairs from "WMT20" were used as test data, and the same instructions provided to human evaluators were applied as zero-shot prompts. The baseline performance of the "LLMs" was measured using "Pearson’s R" to quantify alignment with human assessments. Role-based prompt engineering was tested in a follow-up experiment, and pairwise comparisons were conducted to evaluate the effectiveness of human intelligence in prompting. The results showed that prompt engineering enhances the ability of "LLMs" to approach human judgment in machine translation evaluation ("MTE") and highlights the potential of human-centered "AI" for advancing the future development of "LLMs" when tailored to translation-related tasks.

  • English

    Diese Studie untersucht das Potenzial großer Sprachmodelle ("LLMs") für die automatische Bewertung der durch "KI" generierten Texten. Es wurden maschinelle Übersetzungsqualitätswerte, die von den 20 besten "LLMs" aus der "Chatbot Arena" generiert wurden, mit menschlichen Bewertungsergebnissen verglichen, um die Modelle zu identifizieren, die menschlichen Urteilen am nächsten kommen. Als Testdaten dienten Englisch-Chinesisch-Satzpaare aus "WMT20", wobei dieselben Anweisungen wie bei menschlichen Bewertenden als Zero-Shot-Prompts verwendet wurden. Die Ausgangsleistung der "LLMs" wurde mittels "Pearsons" Korrelationskoeffizient bewertet, um die Übereinstimmung mit menschlichen Einschätzungen zu quantifizieren. In einem Folgeexperiment wurde Prompt Engineering mit rollenbasierten Anweisungen getestet, und paarweise Vergleiche wurden durchgeführt, um die Wirksamkeit menschlicher Intelligenz beim Prompt-Design zu evaluieren. Die Ergebnisse zeigen, dass Prompt Engineering die Fähigkeit der "LLMs" verbessert, eine menschenähnliche Bewertung im Bereich der maschinellen Übersetzungsbewertung ("MTE") zu erreichen. Zudem wird das Potenzial einer menschenzentrierten "KI" für die zukünftige Entwicklung von auf Übersetzungsaufgaben zugeschnittenen "LLMs" hervorgehoben.

• Referencias bibliográficas
  • AA.VV. (2017): «Transformer: A Novel Neural Network Architecture for Language Understanding». *Google Research*.
  • AA.VV. (2024a): «Introducing Gemini 2.0: our new AI model for the agentic era». *Google DeepMind*.
  • AA.VV. (2024b): «LMSYS Chatbot Arena: Live and Community-Driven LLM Evaluation». *LMSYS Team*.
  • AA.VV. (2024c): «Qwen2.5». *Qwen*.
  • AA.VV. (2024d): «Tencent/Tencent-Hunyuan-Large». *Tencent*.
  • AA.VV. (2025a): «Qwen3: Think Deeper, Act Faster». *Qwen*.
  • AA.VV. (2025b): «THUDM/GLM-4». *THUDM*.
  • AA.VV. (2025c): «Grok 3 Beta». *xAI*.
  • Ageeva, Ekaterina et al. «Evaluating Machine Translation for Assimilation via a Gap-Filling Task». *Proceedings of the 18th Annual Conference...
  • Ajai Krishna, T. S. et al. «Evaluating the Language Translation Accuracy of GPT-3.5 Using Prompt Engineering». *International Conference on...
  • Bojar, Ondřej et al. «Findings of the 2013 Workshop on Statistical Machine Translation». *Proceedings of the Eighth Workshop on Statistical...
  • Bojar, Ondřej et al. «Findings of the 2016 Conference on Machine Translation (WMT16)». *First conference on machine translation*, Berlín,...
  • Bojar, Ondřej et al. «Findings of the 2017 conference on machine translation (WMT17)». *Proceedings of the Second Conference on Machine Translation*,...
  • Callison-Burch, Chris et al. «(Meta-) evaluation of machine translation». *Proceedings of the Second Workshop on Statistical Machine Translation*,...
  • Carroll, John B. «An experiment in evaluating the quality of translations». *Mechanical Translations and Computational Linguistics*, vol....
  • Chatzikoumi, Eirini. «How to evaluate machine translation: A review of automated and human metrics». *Natural Language Engineering*, vol....
  • Chauhan, Shweta y Philemon Daniel. «A comprehensive survey on various fully automatic machine translation evaluation metrics». *Neural Processing...
  • Chiang, Wei-Lin et al. (2024a): «Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference». *arXiv*, pp. 1-29.
  • Chiang, Wei-Lin et al. (2024b): «Chatbot Arena Categories: Definitions, Methods, and Insights». *LMSYS Chatbot Arena Blog*.
  • Denkowski, Michael y Alon Lavie. «Choosing the right evaluation for machine translation: an examination of annotator and automatic metric...
  • Doddington, George. «Automatic evaluation of machine translation quality using n-gram co-occurrence statistics». *Proceedings of the second...
  • Fernandes, Patrick et al. «The devil is in the errors: Leveraging large language models for fine-grained machine translation evaluation»....
  • Freitag, Markus et al. «Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation». *Transactions of the...
  • Freitag, Markus et al. «Results of WMT22 Metrics Shared Task: Stop Using BLEU – Neural Metrics Are Better and More Robust». *Proceedings of...
  • Geroimenko, Vladimir. *The Essential Guide to Prompt Engineering: Key Principles, Techniques, Challenges, and Security Risks*. Cham, Springer...
  • Gisserot-Boukhlef, Hippolyte et al. «Is Preference Alignment Always the Best Option to Enhance LLM-Based Translation? An Empirical Analysis»....
  • Graham, Yvette et al. «Continuous measurement scales in human evaluation of machine translation». *Proceedings of the 7th Linguistic Annotation...
  • Gu, Jiawei et al. «A Survey on LLM-as-a-Judge». *arXiv*, 2024, pp. 1-52.
  • Han, Aaron L.-F. et al. «Automatic machine translation evaluation with part-of-speech information». *International Conference on Text, Speech...
  • Han, Lifeng. «Machine translation evaluation resources and methods: A survey». *arXiv*, 2016, pp. 1-18.
  • Han, Lifeng. «HilMeMe: A Human-in-the-Loop Machine Translation Evaluation Metric Looking into Multi-Word Expressions». *arXiv*, 2022, p. 1.
  • He, Sui. «Prompting ChatGPT for Translation: A Comparative Analysis of Translation Brief and Persona Prompts». *arXiv*, 2024, pp. 1-9.
  • Huang, Xu et al. «Lost in the source language: How large language models evaluate the quality of machine translation». *arXiv*, 2024, pp....
  • Kilickaya, Mert et al. «Re-evaluating Automatic Metrics for Image Captioning». *Proceedings of the 15th Conference of the European Chapter...
  • Kim, Junseok, Nakyeong Yang y Kyomin Jung. «Persona is a Double-edged Sword: Mitigating the Negative Impact of Role-playing Prompts in Zero-shot...
  • Knoth, Nils et al. «AI literacy and its implications for prompt engineering strategies». *Computers and Education: Artificial Intelligence*,...
  • Kocmi, Tom y Christian Federmann. (2023a): «Large Language Models Are State-of-the-Art Evaluators of Translation Quality». *Proceedings of...
  • Kocmi, Tom y Christian Federmann. (2023b): «GEMBA-MQM: Detecting translation quality error spans with GPT-4». *arXiv*, 2023, pp. 1-12.
  • Koehn, Philipp. *Neural Machine Translation*. Cambridge, Cambridge University Press, 2020.
  • Koehn, Philipp y Rebecca Knowles. «Six Challenges for Neural Machine Translation». *arXiv*, 2017, p. 1.
  • Kong, Aobo et al. «Better Zero-Shot Reasoning with Role-Play Prompting». *arXiv*, 2023, pp. 1-15.
  • Lacruz, Isabel, Michael Denkowski y Alon Lavie. «Cognitive demand and cognitive effort in post-editing». *Proceedings of the 11th Conference...
  • Lee, Seungjun et al. «A survey on evaluation metrics for machine translation». *Mathematics*, vol. 11, n.º 4, 2023, pp. 1-22.
  • Li, Dawei et al. «From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge». *arXiv*, 2024, pp. 1-36.
  • Lin, Chin-Yew y Franz Josef Och. «ORANGE: a Method for Evaluating Automatic Evaluation Metrics for Machine Translation». *Proceedings of the...
  • Liu, Pengfei et al. «Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing». *arXiv*, 2021,...
  • Lommel, Arle. «Augmented Translation: Are We There Yet?». *CSA Research*, 2020.
  • Lommel, Arle, Hans Uszkoreit y Aljoscha Burchardt. «Multidimensional quality metrics (MQM): A framework for declaring and describing translation...
  • Lu, Qingyu et al. «Error analysis prompting enables human-like translation evaluation in large language models». *arXiv*, 2023, pp. 1-16.
  • Mantaras, Agustin. «Prompt Engineering for OpenAI’s O1 and O3-mini Reasoning Models». *Microsoft Tech Community*, 2025.
  • McCarthy, John. «What is Artificial Intelligence». *Stanford University Computer Science*, 2007, pp. 1-15.
  • Michalos, Alex. *Encyclopedia of Quality of Life and Well-Being Research*. Dordrecht, Springer Netherlands, 2014.
  • Mondorf, Philipp y Barbara Plank. «Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey». *arXiv*, 2024,...
  • Moneus, Ahmed Mohammed y Yousef Sahari. «Artificial intelligence and human translation: A contrastive study based on legal texts». *Heliyon*,...
  • Moorkens, Joss y Ana Guerberof-Arenas. *Artificial intelligence, automation and the language industry*. Berlín / Boston, De Gruyter Mouton,...
  • Moorkens, Joss, Andy Way y Séamus Lankford. *Automating Translation*. Londres, Routledge, 2024.
  • Moosa, Ibraheem Muhammad, Rui Zhang y Wenpeng Yin. «MT-Ranker: Reference-free machine translation evaluation by inter-system ranking». *arXiv*,...
  • Murr, Lincoln, Morgan Grainger y David Gao. «Testing LLMs on Code Generation with Varying Levels of Prompt Specificity». *arXiv*, 2023, pp....
  • Naveed, Humza et al. «A Comprehensive Overview of Large Language Models». *arXiv*, 2023, pp. 1-7.
  • Niessen, Sonja et al. «An Evaluation Tool for Machine Translation: Fast Evaluation for MT Research». *Proceedings of the Second International...
  • Niewiadomska-Bugaj, Magdalena y Robert Bartoszynski. *Probability and Statistical Inference*. Hoboken, Wiley, 2021.
  • O’Brien, Sharon. «Human-Centered augmented translation: against antagonistic dualisms». *Perspectives*, vol. 32, n.º 3, 2024, pp. 391-406.
  • Papineni, Kishore et al. «Bleu: a Method for Automatic Evaluation of Machine Translation». *Proceedings of the 40th Annual Meeting of the...
  • Poibeau, Thierry. «On “Human Parity” and “Super Human Performance” in Machine Translation Evaluation». *Proceedings of the Thirteenth Language...
  • Qian, Shenbin et al. «What do Large Language Models Need for Machine Translation Evaluation?». *arXiv*, 2024, pp. 1-15.
  • Rei, Ricardo et al. «COMET: A neural framework for MT evaluation». *Proceedings of the 2020 Conference on Empirical Methods in Natural Language...
  • Renze, Matthew y Erhan Guven. «The Benefits of a Concise Chain of Thought on Problem-Solving in Large Language Models». *arXiv*, 2024, p. 1.
  • Rivera-Trigueros, Irene. «Machine translation systems and quality assessment: a systematic review». *Language Resources and Evaluation*, vol....
  • Sahoo, Pranab et al. «A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications». *arXiv*, 2024, p. 1.
  • Sanders, Gregory et al. *Handbook of Natural Language Processing and Machine Translation*. New York, Springer, 2011.
  • Sheikh, Haroon, Corien Prins y Erik Schrijvers. *Artificial Intelligence: Definition and Background*. Cham, Springer International Publishing,...
  • Snover, Matthew et al. «A study of translation edit rate with targeted human annotation». *Proceedings of the 7th Conference of the Association...
  • Song, Ran et al. «Does Large Language Model Contain Task-Specific Neurons?». *Proceedings of the 2024 Conference on Empirical Methods in Natural...
  • Specia, Lucia et al. «Findings of the WMT 2020 Shared Task on Quality Estimation». *Proceedings of the Fifth Conference on Machine Translation*,...
  • Stahlberg, Felix. «Neural Machine Translation: A Review and Survey». *arXiv*, 2019, pp. 1-91.
  • Su, Keh-Yih, Ming-Wen Wu y Jing-Shin Chang. «A new quantitative quality measure for machine translation systems». *COLING 1992 volume 2: the...
  • Tang, Shange, Yuanhao Wang y Chi Jin. «Is Elo Rating Reliable? A Study Under Model Misspecification». *arXiv*, 2025, pp. 1-23.
  • Tillmann, Christoph et al. «Accelerated DP based search for statistical translation». *Eurospeech 22*, 1997, pp. 2667-2670.
  • Vatsal, Shubham y Harsh Dubey. «A Survey of Prompt Engineering Methods in Large Language Models for Different NLP Tasks». *arXiv*, 2024, pp....
  • Vilar, David et al. «Human evaluation of machine translation through binary system comparisons». *Proceedings of the Second Workshop on Statistical...
  • Voss, Clare y Calandra R. Tate. «Task-based evaluation of machine translation (MT) engines. Measuring how well people extract who, when, where-type...
  • Wang, Yubo et al. «MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark». *arXiv*, 2024, pp. 1-24.
  • Wong, Billy y Chunyu Kit. «ATEC: automatic evaluation of machine translation via word choice and word order». *Machine Translation*, vol....
  • Wu, Yangjian y Gang Hu. «Exploring Prompt Engineering with GPT Language Models for Document-Level Machine Translation: Insights and Findings»....
  • Yamada, Masaru. (2023a): *Post-editing and a sustainable future for translators*. Amsterdam, John Benjamins Publishing Company.
  • Yamada, Masaru. (2023b): «Optimizing Machine Translation through Prompt Engineering: An Investigation into ChatGPT’s Customizability». *arXiv*,...
  • Zheng, Mingqian et al. «When ‘A Helpful Assistant’ Is Not Really Helpful: Personas in System Prompts Do Not Improve Performances of Large...