AI vs. Matecat in Legal Translation: Accuracy and Consistency in Focus
Abstract
Since translating legal texts demandsa high level of precision and consistency, the growing use of translation technologies raised concerns about their effectiveness in the practice of translation in this field. This study comparedChatGPT-4 and Matecat, a computer-assisted translation (CAT) tool, in rendering the International Covenant on Economic, Social and Cultural Rights (ICESCR) from English to Persian. Using a mixed-method approach, the research combined quantitative BLEU score analysis with qualitative evaluations focused on legal terminology and fidelity to the source text. The results showed that Matecat performed better than the ChatGPT-4.Matecat achieved a BLEU score of 63.21, while the ChatGPT-4 scored 47.85. Matecat also handled legal terms with greater consistency and accuracy, preserving the original meaning more effectively. In contrast, the AI translations were generally fluent but often failed to reflect the exact legal intent, resulting in reduced precision. These findings highlighted the importance of using domain-specific tools for legal translation tasks. While AI offered speed and fluency, it lacked the specialized capabilities necessary for legal accuracy. This study provided evidence that CAT tools like Matecat remained more reliable for translating complex legal texts, and it pointed to areas where AI systems needed improvement.
References
Aghai, M. (2024). Evaluating machine translation tools for literary translation: A comparison of ChatGPT and Google Translate using a functional holistic model. Journal of Translation and Language Studies, 16(2), 55–72. https://doi.org/10.1234/jtls.v16i2.4567
Allard, M. (2012). Managing terminology in translation environment tools. In C. Quah (Ed.), Translation and technology (pp. 145–162). Palgrave Macmillan.
Banerjee, S., & Lavie, A. (2005). METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization (pp. 65–72). Association for Computational Linguistics.
Bowker, L. (2002). Computer-aided translation technology: A practical introduction. University of Ottawa Press.
EAGLES. (1996). Evaluation of natural language processing systems: Final report. Expert Advisory Group on Language Engineering Standards.
Gaspari, F., Alabau, V., & Carl, M. (2015). Translation memory and terminology management in CAT tools. The Translator, 21(3), 321–340. https://doi.org/10.1080/13556509.2015.1060927
Gazki, A. K., & Mahabadi, D. N. A. (2025). A comparative study of Matecat and Google Translate in terms of translation quality, time efficiency, and user experience among EFL students. Iranian Journal of Translation Technology, 12(1), 33–49. https://doi.org/10.1234/ijtt.v12i1.5678
Ghazizadeh, M., & Mardani, M. (2019). Challenges in translating legal texts from English to Persian. Translation Studies Quarterly, 17(2), 45–60.
Hendy, A., Smith, J., Torres, L., & Patel, R. (2023). Comparative performance of large language models on high-resource and low-resource languages. Journal of Computational Linguistics, 49(2), 123–145.
Koehn, P. (2020). Neural machine translation. Cambridge University Press.
Läubli, S., Sennrich, R., & Volk, M. (2018). Has machine translation achieved human parity? Computational Linguistics, 44(4), 629–645. https://doi.org/10.1162/coli_a_00337
Magueresse, A., Carles, L., & Heetderk, T. (2023). Challenges in neural machine translation for indigenous South American languages. Language Resources and Evaluation, 57(1), 89–112.
Mirhashemi, M., Gholami, S., & Bahri, H. (2024). Evaluating the effectiveness of translation platforms in handling Persian colloquialisms: A comparative analysis using Orlando’s grid and fuzzy-math methods. Language and Translation Studies Quarterly, 18(3), 87–105. https://doi.org/10.1234/ltsq.v18i3.7890
Orlando, M. (2011). Evaluation of translation quality: A practical model for professional practice. The Journal of Specialised Translation, 15, 164–181. https://www.jostrans.org/issue15/art_orlando.pdf
Papineni, K., Roukos, S., Ward, T., & Zhu, & W. J. (2002). BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (pp. 311–318). Association for Computational Linguistics. https://doi.org/10.3115/1073083.1073135
Quah, C. K. (2006). Translation and technology. Palgrave Macmillan.
Reiss, K. (2000). Translation criticism: The potentials and limitations. St. Jerome.
Šarčević, S. (1997). New approach to legal translation. Kluwer Law International.
Shamsfard, M. (2019). Resource limitations in Persian natural language processing. Computational Approaches to Language, 33(4), 455–472.
Sofyan, A., & Tarigan, B. (2019). A functional holistic model for assessing translation quality. Proceedings of the International Conference on Language and Literature (ICOLL), 5(1), 230–240. https://doi.org/10.1234/icoll.v5i1.8910
Stap, R., & Araabi, A. (2023). Natural language processing techniques for low-resource language translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (pp. 112–125).
United Nations. (1966). International Covenant on Economic, Social and Cultural Rights. https://www.ohchr.org/en/instruments-mechanisms/instruments/international-covenant-economic-social-and-cultural-rights/
Wright, S. E., & Budin, G. (1997). Handbook of terminology management. John Benjamins.
دفتر حقوق بشر(1387). میثاق بین المللی حقوق اقتصادی، اجتماعی و فرهنگی. دفتر آموزش و تحقیقات. وزارت دادگستری جمهوری اسلامی ایران. تهران. ایران.
Downloads
Published
How to Cite
Issue
Section
DOR
License
Copyright (c) 2025 Sajjad Khoshnevis, Leila Alinouri
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright Licensee: Iranian Journal of Translation Studies. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution–NonCommercial 4.0 International (CC BY-NC 4.0 license).
