GRAPE-MARS: A digital tool for enhanced multimodal discourse analysis in video-based research

• Valeiras-Jurado, Julia ^[1] ; Bernad-Mechó , Edgar ^[1] ; Ruiz-Madrid, Noelia ^[1] ; Fortanet-Gómez, Inmaculada ^[1]
  1. [1] Universitat Jaume I

     Universitat Jaume I

     Castellón, España

     
• Localización: Revista de lingüística y lenguas aplicadas, ISSN 1886-2438, Nº. 21, 2026, págs. 53-71
• Idioma: inglés
• DOI: 10.4995/rlyla.2026.24635
• Enlaces
  • Texto completo
• Resumen

  • Multimodal analyses play a crucial role in understanding meaning-making processes. However, their complexity and labor-intensive nature pose significant challenges. This paper introduces GRAPE-MARS (Multimodal Analysis Research Software), a web-based, open-source tool developed to enhance multimodal research by streamlining annotation, data retrieval, and visualization. GRAPE-MARS integrates AI-driven automatic transcription, user-friendly annotation, and advanced data analysis. In this paper, we present the design and functionalities of the software. Furthermore, we apply it in a case study to analyse engagement strategies in a YouTube science dissemination video. Using GRAPE-MARS, we quantify the distribution of semiotic modes and examine how engagement strategies are realized through distinct modal configurations. These results highlight the potential of the software to facilitate empirical multimodal analyses by providing structured, quantitative insights into semiotic interactions. This research contributes to the advancement of multimodal discourse analysis methodologies and underscores the value of digital tools in optimizing multimodal data coding and analysis.

• Referencias bibliográficas
  • ATLAS.ti Scientific Software Development GmbH. (2023). ATLAS.ti Mac (version 23.2.1) [Qualitative data analysis software]. https://atlasti.com
  • Bateman, J. A., Thiele, L., & Akin H. (2021) Explanation videos unravelled: Breaking the waves. Journal of Pragmatics,175, 112-128.https://doi.org/10.1016/j.pragma.2020.12.009
  • Bernad-Mechó, E. & Girón-García, C. (2023). A Multimodal Analysis of Humour as an Engagement Strategy in YouTube Research Dissemination...
  • Bernad-Mechó, E. & Valeiras-Jurado, J. (2023b) Engaging the audience in online YouTube science dissemination videos: A look at the uptake...
  • Bernad-Mechó, E. (2018). A multimodal approach to metadiscourse as an organizational tool in lectures (Unpublished doctoral dissertation)....
  • Bernad-Mechó, E., & Valeiras-Jurado, J. (2023a). Multimodal engagement strategies in science dissemination: A case study of TED talks...
  • Bezemer, J., & Mavers, D. (2011) Multimodal transcription as academic practice: a social semiotic perspective. International Journal of...
  • Boersma P (2001) Praat, a system for doing phonetics by computer. Glot International, 5(9): 341–345.
  • Boggio-López E and Author 3 (2024) [Invited speakers] ICLHE webinar series. January 23, 2024.
  • Boggio-López, E., & Costa, F. (2025). Exploring language-related episodes (LREs) in English-medium instruction (EMI) from a translanguaging...
  • Bonsignori V, Crarwford-Camiciottoli B and Filmer D (2022) Analyzing Multimodality in Specialized Discourse Settings. Innovative Research...
  • Cvetkovic A and Author 4 (2024) [Presentation] 41st AESLA Conference. Valencia, April 17-19, 2024.
  • Duncan S, Rohlfing K and Loehr D (2013) Multimodal annotation tools. In Müller C, Cienki A, Fricke E, Ladewig S, McNeill D and Tessendorf...
  • Escobar-Lluch N and Author 3 (2024) Multimodal approach to translanguaging strategies: from translanguaging to trans-semiotising in an EMI...
  • Fortanet-Gómez, I., Ruiz-Madrid,N., Bernad-Mechó, E., Valeiras-Jurado, J. (2025). GRAPE -MARS: una nueva herramienta para el análisis multimodal...
  • Girón-García C and Author 4 (2023). English for Specific Purposes 70: 164-176 https://doi.org/10.1016/j.esp.2022.12.005
  • Guillén Galve I and Vela Tafalla M A (2023) (eds.). Digital Genres and Multimodality. ESP Today 11(2). https://doi.org/10.18485/esptoday.2023.11.2.1
  • Jewitt C (2013) Multimodal methods for researching digital technologies. In Price S, Jewitt C and Brown B (eds) The Sage handbook of digital...
  • Kipp M (2001) ANVIL A generic annotation tool for multimodal dialogue. Proceedings of the 7th European Conference on Speech Communication...
  • Kress G (2006) Review of literacy in the new media age. Routledge.
  • Lim F V and Tan-Chia L (2022) Designing Learning for Multimodal Literacy: Teaching Viewing and Representing (1st ed.). Routledge. https://doi.org/10.4324/9781003258513
  • Liu Y and Xiang Y (2024) Computer-assisted multimodal analysis research: A technical review of GRAPE-MARS. Multimodality & Society 4(2):...
  • Mena-Alba A (2024) New pedagogical artefacts for new learning needs: Instagram reels and multimodal literacy in pre-tertiary education. Unpublished...
  • Mestre-Segarra, M. Á. (2023). A multimodal rhetorical analysis of video resumes. ESP Today, 11(2), 349-370.
  • https://doi.org/10.18485/esptoday.2023.11.2.8
  • Muñoz Morcillo J, Czurda K and Robertson-von Trotha C Y (2016) Typologies of the popular science web video. Journal of Science Communication...
  • Norris, S. (2004) Analyzing Multimodal Interaction. A methodological framework. Routledge.https://doi.org/10.4324/9780203379493
  • O’Halloran, K. L., Pal, G., & Jin, M. (2021) Multimodal approach to analysing big social and news media data.Discourse, Context &...
  • O’Halloran, K. L., Podlasov, A., Chua, A., & Marissa, K. L. E. (2012) Interactive software for multimodal analysis. Visual Communication,...
  • O’Halloran, K. L., Tan, S., & Marissa, K. L. E. (2015) Multimodal analysis for critical thinking. Learning, Media and Technology,42(2),...
  • Querol-Julián M (2011) Evaluation in discussion sessions of conference paper presentations: a multimodal approach. LAP Lambert Academic Publishing...
  • Ruiz-Madrid, M. N. (2021). A multimodal discourse approach to Research Pitches. Journal of English for Academic Purposes,15.https://doi.org/10.1016/j.jeap.2021.101003
  • Ruiz-Madrid, N., Fortanet-Gómez, I., Bernad-Mechó, E., & Valeiras-Jurado, J. (2023). GRAPE-MARS (Multimodal Analysis Research Software)....
  • Ruiz-Madrid, N., Valeiras-Jurado, J. (2023). Reconceptualisation of Genre(s) in Scholarly and Scientific Digital Practices: A Look at Multimodal...
  • Serrano-Hernández, E. (2025). A pedagogical proposal to foster multimodal literacy in Spanish Official School of Languages. In Bernad-Mechó,...
  • Serrano-Hernández, E., & Bernad-Mechó (2026). Science Dissemination in Instagram reels: A multimodal overview Círculo de Lingüística Aplicada...
  • Valeiras Jurado, J., Ruiz Madrid, M. N., & Jacobs, G. (2018). Revisiting persuasion in oral academic and professional genres: Towards...
  • Valeiras-Jurado, J. & Bernad-Mechó, E. (2022). Modal Density and Coherence in Science Dissemination: Orchestrating Multimodal Ensembles...
  • Valeiras-Jurado, J., & Ruiz-Madrid, N. (2023). The influence of discipline, medium and target audience in multimodal recontextualization...
  • Vivas-Peraza, A. C. (2026). Multimodality as a framework for the study of verbal and non-verbal resources in webbased science communication....
  • Wittenburg, P., Brugman, H., Russel, A., Klassmann, A., & Sloetjes, H. (2006) ELAN: a professional framework for multimodality research....
  • Xia, S. A., & Hafner, C. A. (2021) Engaging the online audience in the digital era: A multimodal analysis of engagement strategies in...
  • Yuan, C., He, Y., & Liu, Y. (2021) Rule by law versus rule of law: A multimodal analysis of persuasion and legal ideologies in anti-corruption...