Item and task difficulty in a B2 reading examination: perceptions of test-takers and CEFR alignment experts compared with psychometric measurements

• Stopar, Andrej ; Ilc, Gašper ^[1]
  1. [1] University of Ljubljana

     University of Ljubljana

     Eslovenia

     
• Localización: Círculo de lingüística aplicada a la comunicación, ISSN-e 1576-4737, Nº. 67, 2016, págs. 318-342
• Idioma: inglés
• DOI: 10.5209/CLAC.53487
• Enlaces
  • Texto completo
• Resumen

  • The article presents a study of a CEFR B2-level reading subtest that is part of the Slovenian national secondary school leaving examination in English as a foreign language, and compares the test-taker actual performance (objective difficulty) with the test-taker and expert perceptions of item difficulty (subjective difficulty). The study also analyses the test-takers’ comments on item difficulty obtained from a while-reading questionnaire. The results are discussed in the framework of the existing research in the fields of (the assessment of) reading comprehension, and are addressed with regard to their implications for item-writing, FL teaching and curriculum development.

• Referencias bibliográficas
  • Alderson, J. C., and Lukmani, Y. (1989) Cognition and Reading: Cognitive Levels as Embodied in Test Questions. Reading in a Foreign Language...
  • Alderson, J. C. (2000) Assessing Reading. Cambridge: Cambridge University Press.
  • Alderson, J. C., Figueras, N., Kuijper, H., Nold, G., Takala, S., and Tardieu, C. (2004) The development of specifications for item development...
  • Apostolou, E. (2010) Comparing Perceived and Actual Task and Text Difficulty in the Assessment of Listening Comprehension. Papers from the...
  • Bachman, L. F. (1990) Fundamental Considerations in Language Testing. Oxford: Oxford University Press.
  • Bachman, L. F., and Palmer, A. S. (1996) Language Testing in Practice: Designing and Developing Useful Language Tests. Oxford: Oxford University...
  • Baily, K. M. (1998) Learning about language assessment: Dilemmas, decisions, and directions. Boston: Heinle Cengage Learning.
  • Bitenc Peharc, S., and Tratnik, A. (2014) Umestitev nacionalnih izpitov iz angleščine v skupni evropski okvir. Zaključno poročilo o izvedbi...
  • Bloom, B. S., Engelhart, M. D., Furst, E. J., Hill, W. H., and Krathwohl, D. R. (1956) Taxonomy of educational objectives: The classification...
  • Brunfaut, T. and McCray, G. (2015) Looking into test-takers’ cognitive processes whilst completing reading tasks: A mixed-method eye-tracking...
  • Cizek, G. J. (Ed.). (2001) Setting Perfomance Standards, Concepts, Methods, and Perspectives. Mahwah, New Jersey, London: Lawrence Erlbaum...
  • Council of Europe. (2001) Common European Framework of Reference for Languages: Learning, Teaching, Assessment. Cambridge: Cambridge University...
  • Council of Europe. (2009) Relating Language Examinations to the Common European Framework of Reference for Languages: Learning, Teaching,...
  • Efklides, A. (2002) Feelings and judgments as subjective evaluations of cognitive processing: how reliable are they? Psychology: The Journal...
  • Figueras, N. (2012) The impact of the CEFR. ELT Journal 66 (4), 477-485. DOI: 10.1093/elt/ccs037
  • Fortus, R., Coriat, R. and Fund, S. (1998) Prediction of item difficulty in the English subtest of Israel’s inter-university psychometric...
  • Freedle, R. O., and Kostin, I. W. (1993) Item Difficulty for Expository Prose Passages for Three Item Types: Main Idea, Inference, and Supporting...
  • Freedle, R., and Kostin, I. (1999) Does the text matter in a multiple-choice test of comprehension? the case for the construct validity of...
  • Fulcher, G. (2004) Deluded by Artifices? The Common European Framework and Harmonization. Language Assessment Quarterly 1(4), 253-266. DOI:...
  • Fulmer, S. M., and Tulis, M. (2013) Changes in interest and affect during a difficult reading task: Relationships with perceived difficulty...
  • Grabe, W. (2009) Reading in a Second Language. Moving from Theory to Practice. Cambridge: Cambridge University Press.
  • Grabe, W. (1991) Current developments in second-language reading research. TESOL Quarterly 25 (3), 375–406.
  • Green, A. B. (2014) Exploring Language Assessment and Testing: Language in Action. London and New York: Routledge.
  • Hambleton, R., and Jirka, S. (2006) Anchor-based methods for judgmentally estimating item statistics. In S. Downing and T. Haladyna (Eds.),...
  • Harding, L., Alderson, C. and Brunfaut, T. (2015) Diagnostic assessment of reading and listening in a second or foreign language: elaborating...
  • Hinkel, E. (2010) Integrating the four skills: Current and historical perspectives. In R. B. Kaplan (Ed.), Oxford Handbook in Applied Linguistics,...
  • Ilc, G. and Stopar, A. (2014) Validating the Slovenian national alignment to CEFR: The case of the B2 reading comprehension examination in...
  • Ilc, G., Rot Gabrovec, V., and Stopar, A. (2014) Relating the Slovenian secondary-school English language national examinations to the CEFR:...
  • Kaftandijeva, F. (2010) Methods for Setting Cut Scores in Criterion-referenced Achievement Tests. A comparative analysis of six recent methods...
  • Khalifa, H., and Weir, C. J. (2009) Examining Reading: Research and Practice in Assessing Second Language Reading. Cambridge: Cambridge University...
  • Koda, K. (2005) Insights into Second Laguage Reading. A Cross-Linguistic Approach. Cambridge: Cambridge University Press.
  • Little, D. (2007) The Common European Framework of Reference for Languages: Perspectives on the Making of Supranational Language Education...
  • Lund, R. J. (1991) A Comparison of Second Language Listening and Reading Comprehension. The Modern Language Journal, 75(2), 196-204. Retrieved...
  • Martyniuk, W. (Ed.). (2010) Aligning Tests with the CEFR: Reflections on Using the Council of Europe's Draft Manual (Studies in Language...
  • Miles, M. B. and Huberman, A. M. (1994) An Expanded Sourcebook: Qualitative Data Analysis. Second Edition. London: London Sage Publications.
  • Munby, J. (1978) Communicative Syllabus Design. Cambridge: Cambridge University Press.
  • Robinson, P. (2001) Task Complexity, Task Difficulty, and Task Production: Exploring Interactions in a Componential Framework. Applied Linguistics...
  • Spaan, M. (2007) Evolution of a test item. Language Assessment Quarterly 4, 279-293. DOI: 10.1080/15434300701462937
  • Sydorenko, T. (2011) Item Writer Judgments of Item Difficulty Versus Actual Item Difficulty: A Case Study. Language Assessment Quarterly 8,...
  • Šifrar Kalan, M., and Trenc, A. (2014) Relating reading comprehension in the Spanish as a foreign language national exam to the CEFR: some...
  • Urquhart, A. H., and Weir, C. (1998) Reading in a Second Language: Process, Product and Practice. London, New York: Longman.
  • Wu, R. Y. (2011) Establishing the Validity of the General English Proficiency Test Reading Component through a Critical Evaluation on Alignment...