A Comparison of the Quality and Speed of Post-Edited Translation and Human-Initiated Translation by Novice Translators
Abstract
The present study compares post-editing and translation from scratch by novice translators in terms of quality and speed in the English-Persian language pair. Moreover, it investigates the factors that affect the cognitive and temporal aspects of post-editing effort. To that end, 10 B.A. students were briefed to translate a short English news text by post-editing the raw MT output provided in Persian, while eight student translated the same text from scratch, with both groups performing the task in the online CAT tool MateCat. The performance of the participants were monitored and screen-recorded to compare the two groups’ speed in completing the task. Furthermore, the two groups’ translations were evaluated analytically and holistically by three evaluators. It was found that the post-editors were significantly faster, and their TL texts were of a considerably higher quality. After completing the task, the participants were asked to fill out questionnaires to provide insight into the cognitive and temporal post-editing efforts. The responses indicated that the grammatical errors present in the raw MT output were the most important factor affecting the temporal post-editing effort, while finding ‘proper equivalents’, and balancing usage of MT were reported to a lesser extent, and correcting zero-width non-joiner was reported by only one post-editor. The most reported issue related to cognitive effort was the concern that post-editing could adversely affect the creativity of the translator, while finding ‘proper equivalents’, balancing usage of MT, and making the target text easy to understand, among others, were also reported to be causes for cognitive effort, albeit with less frequency.
Keywords:
CAT tools, Machine translation post-editing (MTPE), Novice translator, Post-editing effort, Translation qualityReferences
Alves, F., Koglin, A., Mesa-Lao, B., Martínez, M. G., de Lima Fonseca, N. B., de Melo Sá, A., Gonçalves, J. L., Szpak, K. S., Sekino, K., & Aquino, M. (2016). Analysing the impact of interactive machine translation on post-editing effort. In M. Carl, S. Bangalore, & M. Schaeffer (Eds.), New directions in empirical translation process research: Exploring the CRITT TPR-DB (pp. 77–94). Springer International Publishing. https://doi.org/10.1007/978-3-319-20358-4_4
Alves, F., Szpak, K., Gonçalve, J., Sekino, K., Aquino, M., Castro, R., Koglin, A., & Fonseca, N. (2016). Investigating cognitive effort in post-editing: A relevance-theoretical approach. Language Science Press, 2, 109–142. https://doi.org/10.17169/langsci.b108.296
Aranberri, N., Labaka, G., Diaz de Ilarraza, A., & Sarasola, K. (2014). Comparison of post-editing productivity between professional translators and lay users. In S. O’Brien, M. Simard, & L. Specia (Eds.), Proceedings of the 11th conference of the Association for Machine Translation in the Americas (p. 20–33). Association for Machine Translation in the Americas.
Aziz, W., Castilho, S., & Specia, L. (2012). PET: A tool for post-editing and assessing machine translation. In In N. Calzolari, K. Choukri, T. Declerck, M. U. Doğan, B. Maegaard, J. Mariani, A. Moreno, J. Odijk, & S. Piperidis (Eds.), Proceedings of the eighth international conference on language resources and evaluation (pp. 3982–3987). European Language Resources Association.
Carl, M. (2012). Translog-ii: A program for recording user activity data for empirical reading and writing research. In In N. Calzolari, K. Choukri, T. Declerck, M. U. Doğan, B. Maegaard, J. Mariani, A. Moreno, J. Odijk, & S. Piperidis (Eds.), Proceedings of the eighth international conference on language resources and evaluation (pp. 4108–4112). European Language Resources Association.
Carl, M., Dragsted, B., Elming, J., Hardt, D., & Jakobsen, A. L. (2011). The process of post-editing: A pilot study. In B. Sharp, M. Zock, M. Carl, & A. L. Jakobsen (Eds.), Proceedings of the 8th international NLPCS workshop (pp. 131–142). Copenhagen Studies in Language.
Cumbreño, C., & Aranberri, N. (2021). What do you say? Comparison of metrics for post-editing effort. In M. Carl (Ed.), Explorations in empirical translation process research (pp. 57–79). Springer International Publishing. https://doi.org/10.1007/978-3-030-69777-8_3
Daems, J., Vandepitte, S., Hartsuiker, R., & Macken, L. (2017). Translation methods and experience: A comparative analysis of human translation and post-editing with students and professional translators. Meta, 62(2), 245–270. https://doi.org/10.7202/1041023ar
Eyckmans, J., Anckaert, P., & Segers, W. (2009). The perks of norm-referenced translation evaluation. In C. V. Angelelli & H. E. Jacobson (Eds.), Testing and assessment in translation and interpreting studies (pp. 73–94). John Benjamins Publishing Company.
Garcia, I. (2010). Is machine translation ready yet. Target, 22(1), 7–21.
Hutchins, W. J., & Somers, H. L. (1992). An introduction to machine translation. Academic Press.
Jia, Y., Carl, M., & Wang, X. (2019). How does the post-editing of neural machine translation compare with from-scratch translation? A product and process study. The Journal of Specialised Translation, 31, 60–86.
Koby, G. S. (2001). Editor’s introduction: Post-editing of machine translation output: Who, what, why, and how (much). In H. P. Krings, Repairing texts: Empirical investigations of machine translation post-editing processes (pp. 1–24). Kent State University Press.
Koehn, P. (2009). A process study of computer-aided translation. Machine Translation, 23, 241–263. https://doi.org/10.1007/s10590-010-9076-3
Krings, H. P. (2001). Repairing texts: Empirical investigations of machine translation post-editing processes. Kent State University Press.
Läubli, S., Amrhein, C., Düggelin, P., Gonzalez, B., Zwahlen, A., & Volk, M. (2019). Post-editing productivity with neural machine translation: An empirical assessment of speed and quality in the banking and finance domain. In M. Forcada, A. Way, B. Haddow, & R. Sennrich (Eds.), Proceedings of machine translation summit XVII: Research (pp. 267–272). European Association for Machine Translation.
Lee, J., & Liao, P. (2011). A comparative study of human translation and machine translation with post-editing. Compilation & Translation Review, 4(2). 105–149.
Nitzke, J., & Oster, K. (2016). Comparing translation and post-editing: An annotation schema for activity units. In M. Carl, S. Bangalore, & M. Schaeffer (Eds.), New directions in empirical translation process research: Exploring the CRITT TPR-DB (pp. 293–308). Springer International Publishing. https://doi.org/10.1007/978-3-319-20358-4_14
O'Brien, S. (2005). Methodologies for measuring the correlations between post-editing effort and machine translatability. Machine Translation, 19, 37–58. https://doi.org/10.1007/s10590-005-2467-1
O'Brien, S. (2007). Eye-tracking and translation memory matches. Perspectives, 14(3), 185–205. https://doi.org/10.1080/09076760708669037
Plitt, M., & Masselot, F. (2010). A productivity test of statistical machine translation post-editing in a typical localisation context. The Prague Bulletin of Mathematical Linguistics, 93, 7–16. https://doi.org/10.2478/v10108-010-0010-x
Skadiņš, R., Puriņš, M., Skadina, I., & Vasiļjevs, A. (2011). Evaluation of SMT in localization to under-resourced inflected language. In M. L. Forcada, H. Depraetere, & V. Vandeghinste (Eds.), Proceedings of the 15th annual conference of the European Association for Machine Translation (pp. 35–40). European Association for Machine Translation.
Vieira, L. N. (2014). Indices of cognitive effort in machine translation post-editing. Machine Translation, 28, 187–216. https://doi.org/10.1007/s10590-014-9156-x
Downloads
Published
How to Cite
Issue
Section
DOR
License
Copyright (c) 2024 S. Hossein Arjani, Masoud Jamshidiha
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright Licensee: Iranian Journal of Translation Studies. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution–NonCommercial 4.0 International (CC BY-NC 4.0 license).
