Logistic calibrated items (LCI) method: does it solve subjectivity in translation evaluation and assessment?

• Akbari, Alireza ^[1]
  1. [1] University of Isfahan

     University of Isfahan

     Irán

     
• Localización: Revista de lingüística y lenguas aplicadas, ISSN 1886-2438, Nº. 14, 2019, págs. 1-18
• Idioma: inglés
• DOI: 10.4995/rlyla.2019.11068
• Enlaces
  • Texto completo
• Resumen

  • This research paper introduces a model of translation evaluation called Logistic Calibrated Items method. The aim of LCI method is to maximize a translators’ performance and to identify top competent translators through detecting all parsing items within a source text. Parsing items are extracted by the application of Brat software. The purpose of LCI was to identify parsing items having optimal item difficulty and item discrimination values. LCI method involves six stages: (1) holistic scoring; (2) the application of Brat software to extract all parsing items; (3) the calculation of item difficulty; (4) the calculation of item discrimination; (5) the identification of items with optimal item difficulty and item discrimination values; and (6) the recalculation of scores. 125 translation students and 4 professional translation evaluators took part in this research. The final results showed that LCI method was more consistent when compared to holistic method. Limitations and implications were also discussed.

• Referencias bibliográficas
  • Akbari, A. & Gholamzadeh Bazarbash, M. (2017). “Holistic Assessment: Effective or Lenient in Translation Evaluation?” Skopos. Revista...
  • Akbari, A. & Segers, W. (2017a). “Translation Difficulty: How to Measure and What to Measure”. Lebende Sprachen, 62/1: 3-29. https://doi.org/10.1515/les-2017-0002
  • Akbari, A. & Segers, W. (2017b). “Translation Evaluation Methods and the End-Product: Which One Paves the Way for a More Reliable and...
  • Akbari, A. & Segers, W. (2017c). “Evaluation of Translation through the Proposal of Error Typology: An Explanatory Attempt”. Lebende Sprachen,...
  • Akbari, A. & Segers, W. (2017d). “The Perks of Norm and Criterion Referenced Translation Evaluation”. LICTRA, Leipzig, Germany, 20 March.
  • Anckaert, P., Eyckmans, J. & Segers, W. (2008). “Pour Une Évaluation Normative De La Compétence De Traduction.” ITL - International Journal...
  • Bahameed, A. S. (2016). “Applying assessment holistic method to the translation exam in Yemen.” Babel, 62/1: 135-149. https://doi.org/10.1075/babel.62.1.08bah
  • Baker, F. B. (2001). The Basics of Item Response Theory. 2nd ed. New York: ERIC Clearinghouse on Assessment and Evaluation.
  • Baker, F. B. & Seock-Ho K. (2004). Item Response Theory: Parameter Estimation Techniques. 2nd ed. New York: Marcel Dekker. https://doi.org/10.1201/9781482276725
  • Barkaoui, K. (2010). “Explaining ESL essay holistic scores: A multilevel modeling approach”. Language Testing, 27/4: 515-535. https://doi.org/10.1177/0265532210368717
  • Barkaoui, K. (2011). “Effects of marking method and rater experience on ESL essay scores and rater performance”. Assessment in Education:...
  • Bunt, H., Merlo, P. & Nivre, J. (2010). Trends in Parsing Technology: Dependency Parsing, Domain Adaptation, and Deep Parsing. Netherlands:...
  • Conde Ruano, T. (2005). “No me parece mal. Comportamiento y resultados de estudiantes al evaluar traducciones”. Unpublished doctoral dissertation....
  • Cumming, Alister, Kantor, R. & Powers, D. E. (2002). “Decision making while rating ESL/EFL writing tasks: A descriptive framework”. The...
  • D’Agostino, R. B. & Cureton, E. E. (1975). “The 27 Percent Rule Revisited”. Educational and Psychological Measurement, 35/1: 47-50. https://doi.org/10.1177/001316447503500105
  • Dancette, J. (1989). “La faute de sens en traduction”. TTR : traduction, terminologie, rédaction, 2/2: 83-102. https:// doi.org/10.7202/037048ar
  • Eberly Center. (2016). “What is the difference between formative and summative assessment?” Accesible at https://www.cmu.edu/teaching/assessment/basics/formative-summative.html...
  • Eyckmans, J. & Anckaert, P. (2017). “Item-based assessment of translation competence: Chimera of objectivity versus prospect of reliable...
  • Eyckmans, J., Anckaert, P. & Segers, W. (2013). “Assessing Translation Competence”. Actualizaciones en Comunicación Social. Centro de...
  • Eyckmans, J., Segers, W. & Anckaert, P. (2012). Translation Assessment Methodology and the Prospects of European Collaboration. In Collaboration...
  • Farmer, W. L., Thompson, R. C., Heil, S. K. R. & Heil, M. C. (2001). Latent Trait Theory Analysis of Changes in Item Response Anchors....
  • Finch, H. & Edwards, J. M. (2015). “Rasch Model Parameter Estimation in the Presence of a Nonnormal Latent Trait Using a Nonparametric...
  • Fox, J. (2010). Bayesian Item Response Modeling: Theory and Applications. Amsterdam: Springer. https://doi.org/10.1007/978-1-4419-0742-4
  • Garant, M. (2009). “A case for holistic translation assessment”. AFinLA-e: Soveltavan kielitieteen tutkimuksia, 5/2: 5-17.
  • Gonzalez, K. (2018). “Contrast Effect: Definition & Example”. https://study.com/academy/lesson/contrast-effectdefinition-example.html.
  • Gouadec, D. (1989). “Comprendre, évaluer, prévenir : Pratique, enseignement et recherche face à l’erreur et à la faute en traduction”. TTR,...
  • Hambleton, R. K. (1989). Principles and selected applications of item response theory. In The American Council on Education/Macmillan series...
  • Hambleton, R. K. & Jones, R. W. (1993). “Comparison of classical test theory and item response theory and their applications to test development”....
  • Hamp-Lyons, L. (1991). Scoring procedures for ESL contexts. In Assessing second language writing in academic contexts, edited by L. Hamp-Lyons,...
  • Harsch, C. & Martin, G. (2013). “Comparing holistic and analytic scoring methods: issues of validity and reliability”. Assessment in Education:...
  • Kempf, W. (1983). “Some Theoretical Concerns ahout Applying Latent Trait Models in Educational Testing”. Accesible at https://pdfs.semanticscholar.org/5909/0351c0bc109f28836a75eaa67e7eecaffa41.pdf....
  • Kockaert, H. & Segers, W. (2014). “Evaluation de la Traduction : La Méthode PIE (Preselected Items Evaluation)”. Turjuman, 23/2: 232-250.
  • Kockaert, H. & Segers, W. (2017). “Evaluation of legal translations: PIE method (Preselected Items Evaluation)”. Journal of Specialized...
  • Kussmaul, P. (1995). Training the Translator. Amsterdam: John Benjamins Publishing Company. https://doi.org/10.1075/btl.10
  • Larose, R. (1989). Théories contemporaines de la traduction. Montréal: Presses de l’Université de Québec.
  • Le, D. (2013). Applying item response theory modeling in educational research. Instructional Technology, Graduate College at Iowa State University...
  • Lei, P. & Wu, Q. (2007). “CTTITEM: SAS macro and SPSS syntax for classical item analysis”. Behavior Research Methods, 39/3: 527-530. https://doi.org/10.3758/BF03193021
  • Mariana, V., Cox, T. & Melby, A. (2015). “The Multidimensional Quality Metrics (MQM) Framework: A New Framework for Translation Quality...
  • Muñoz Martín, R. (2010). On Paradigms and Cognitive Translatology. In Translation and Cognition, edited by G. Schreve and E. Angelone, 169-187....
  • Newmark, P. (1991). About Translation. Clevedon: Multilingual Matters.
  • Nord, C. (2005). Text Analysis in Translation. aMSTERDAM: Rodopi.
  • Pidgeon, D. A. & Yates, A. (1968). An introduction to educational measurement. London: Routledge.
  • Preacher, K. J. , Rucker, D. D., MacCallum, R. C. & Nicewander, W. A. (2005). “Use of the extreme groups approach: a critical reexamination...
  • Schmitt, P. A. (2005). Qualitätsbeurteilung von Fachübersetzungen in der Übersetzerausbildung, Probleme und Methoden. Vertaaldagen Hoger Instituut...
  • SPSS, IBM. (2017). Available at https://www.ibm.com/analytics/us/en/technology/spss/. [Last access: July 2019].
  • Stansfield, C. W., Scott, M. L. & Kenyon, D. M. (1992). “The Measurement of Translation Ability”. The Modern Language Journal, 76/4: 455-467....
  • Stata. (2016). “Stata: Software for Statistics and Data Science”. Available at https://www.stata.com/. [Last access: July 2019].
  • Stenetorp, P., Pyysalo, S., Topic, G., Ohta, T., Ananiadou, S. & Tsujii, J. (2012). BRAT: a web-based tool for NLPassisted text annotation....
  • Stenetorp, P., Topic, G., Pyysalo, S., Ohta, T., Kim, J. & Tsujii, J. (2011). BioNLP Shared Task 2011: Supporting Resources. BioNLP Shared...
  • TAUS. (2018). “Measuring and Benchmarking Translation Quality”. Available at https://www.taus.net/qualitydashboard-lp. [Last access: July...
  • Umobong, M. E. (2017). “The One-Parameter Logistic Model (1PLM) And its Application in Test Development”. Advances in Social Sciences Research...
  • Van Antwerpen, J. (2016). “P-, D-, and Rit values: a new start”. Available at http://www.andriesseninternational.com/p-d-and-rit-values-a-new-start/....
  • Waddington, C. (2001). “Different Methods of Evaluating Student Translations: The Question of Validity”. Meta, 46/2: 311-325. https://doi.org/10.7202/004583ar
  • Weigle, S. C. (2002). Assessing writing. Cambridge: Cambridge University Press. https://doi.org/10.1017/CBO9780511732997
  • Widdowson, H. G. (1978). Teaching Language as Communication Oxford: Oxford University Press.
  • Wiersma, W. & Jurs, S. G. (1990). Educational Measurement and Testing. London: Allyn and Bacon.
  • Zięba, A. (2013). “The Item Information Function in One and Two-Parameter Logistic Models- A Comparison and Use in the Analysis of the Results...