Multimodal communication in instructional settings: An investigation of the functional roles of gestures and arrows

Cengiz Acarturk; Melda Coskun; Serap Emil

Multimodal communication in instructional settings: An investigation of the functional roles of gestures and arrows

Acarturk, Cengiz ^[1] ; Coskun, Melda ^[2] ; Emil, Serap ^[3]
1. [1] Jagiellonian University
  
  Jagiellonian University
  
  Kraków, Polonia
2. [2] McMaster University
  
  McMaster University
  
  Canadá
3. [3] Middle East Technical University
  
  Middle East Technical University
  
  Turquía
Mostrar afiliaciones +
Localización: Revista signos: estudios de lingüística, ISSN-e 0718-0934, ISSN 0035-0451, Vol. 54, Nº. 107 (Diciembre), 2021, págs. 867-892
Idioma: inglés
DOI: 10.4067/s0718-09342021000300867
Títulos paralelos:
- Comunicación multimodal en contextos educativos: una investigación de los roles funcionales de los gestos y las flechas
Enlaces
- Texto completo
Resumen
- español
  En la comunicación diaria y en entornos educativos, las flechas y los gestos espontáneos suelen transmitir información espacial similar a los interlocutores. Ellos pueden visualizar información espacial en el espacio 2D o 3D basándose en representaciones conceptuales y espaciales compartidas. El presente estudio investiga la relación entre los gestos espontáneos y las flechas desde la perspectiva de la generación y la comprensión en contextos de enseñanza. Introducimos un corpus de flechas en un texto multimodal escrito y un corpus de gestos generados durante la comunicación a través de material instructivo multimodal. También presentamos una investigación experimental que utilizó un corpus para estudiar el impacto de los gestos en el aprendizaje de la comprensión. La comparación de los corpus revela similitudes y diferencias funcionales entre las flechas y los gestos y su papel complementario en la comunicación multimodal. Los resultados sugieren que las flechas y los gestos pueden compartir un espacio conceptual común durante la comunicación, teniendo un impacto significativo en la comprensión y el aprendizaje.
- English
  In daily communication and instructional settings, arrows and spontaneous gestures usually convey similar spatial information to interlocutors. They can visualize spatial information in 2D or 3D space based on shared conceptual and spatial representations. The present study investigates the relationship between spontaneous gestures and arrows from the perspective of generation and comprehension. We introduce a corpus of arrows in written multimodal text and a corpus of gestures generated during communication through multimodal instructional material. We also report an experimental investigation that used corpus material to study the impact of gestures on comprehension learning. A comparison of the corpora reveals functional similarities and differences between arrows and gestures and their complementary role in multimodal communication. The findings suggest that arrows and gestures may share a common conceptual space during communication, having a significant impact on comprehension and learning.
Referencias bibliográficas
- Acartürk, C. (2014). Towards a systematic understanding of graphical cues in communication through statistical graphs. Journal of Visual Languages...
- Ainsworth, S. E. & Scheiter, K. (2021). Learning by drawing visual representations: Potential, purposes, and practical implications. Current...
- Alaçam, Ö., Habel, C. & Acartürk, C. (2013). Gesture and language production in communication through bar graphs. In M. Knauff, M. Pauen,...
- Alibali, M. W. (2005). Gesture in spatial cognition: Expressing, communicating, and thinking about spatial information. Spatial Cognition...
- Alibali, M. W. & Nathan, M. J. (2007). Teachers’ gestures as a means of scaffolding students’ understanding: Evidence from an early algebra...
- Bateman, J. A. (2008). Multimodality and genre: A foundation for the systematic analysis of multimodal documents. New York: Springer.
- Bateman, J. A. (2021). Growing theory for practice: Empirical multimodality beyond the case study. Multimodal Communication, pp. 000010151520210006....
- Baylor, A. L. & Kim, S. (2009). Designing nonverbal communication for pedagogical agents: When less is more. Computers in Human Behavior,...
- Bernsen, N. O. (1994, 2008). Foundations of multimodal representations: A taxonomy of representational modalities. Interacting with Computers,...
- Bernsen, N. (2008). Multimodality Theory. In D. Tzovaras (Eds.), Multimodal User Interfaces. Signals and Communication Technologies (pp. 5-29)....
- Boudon, E. & Parodi, G. (2014). Multisemiotic artifacts and academic discourse of Economics: Knowledge construction in the Textbook genre....
- Calvert, G., Spence, C. & Stein, B. E. (Eds.). (2004). The handbook of multisensory processes. Boston: MIT Press.
- Chen, F. & Sager, J. (2011). Effects of tablet PC use in the classroom on teaching and learning processes. Journal of Learning in Higher...
- Cook, S. W., Friedman, H. S., Duggan, K. A., Cui, J. & Popescu, V. (2017). Hand gesture and mathematics learning: lessons from an Avatar....
- Coakun, M. & Acartürk, C. (2015). Gesture production under instructional context: The role of mode of instruction. In D. C. Noelle, R....
- Duncan, S. D., Cassell, J. & Levy, E. T. (Eds.). (2007). Gesture and the dynamic dimension of language: Essays in Honor of David McNeill...
- Eilam, B. & Poyas, Y. (2008). Learning with multiple representations: Extending multimedia learning beyond the lab. Learning and Instruction,...
- Frutiger, A. (1989). Signs and symbols. New York: Van Nostrand Reinhold.
- Gates, P. (2018). The importance of diagrams, graphics and other visual representations in STEM teaching. In R. Jorgensen & K. Larking...
- Gerofsky, S. (2011). Mathematical learning and gesture: Character viewpoint and observer viewpoint in students’ gestured graphs of functions....
- Glenberg, A. M. & Langston, W. E. (1992). Comprehension of illustrated text: Pictures help to build mental models. Journal of Memory and...
- Goldin-Meadow, S. (1999). The role of gesture in communication and thinking. Trends in Cognitive Sciences, 3(11), 419-429.
- Goldin-Meadow, S., Kim, S. & Singer, M. (1999). What the teacher’s hands tell the student’s mind about math. Journal of Educational Psychology,...
- Graesser, A. C., Sabatini, J. P. & Li, H. (2021). Educational psychology is evolving to accommodate technology, multiple disciplines,...
- Habel, C. & Acartürk, C. (2007). On reciprocal improvement in multimodal generation: Co-reference by text and information graphics. In...
- Hegarty, M. & Just, M. A. (1993). Constructing mental models of machines from text and diagrams. Journal of Memory and Language, 32(6),...
- Heiser, J., Tversky, B. & Silverman, M. (2004). Sketches for and from collaboration. In J. S. Gero, B. Tversky & T. Knight (Eds.),...
- Hill, W. C. & Hollan, J. D. (1991). Deixis and the future of visualization excellence. In Proceedings of the 2nd Conference on Visualization...
- Holler, J. & Levinson, S. C. (2019). Multimodal language processing in human communication. Trends in Cognitive Sciences, 23(8), 639-652.
- Kalyuga, S., Chandler, P. & Sweller, J. (1999). Managing splitâattention and redundancy in multimedia instruction. Applied Cognitive...
- Kita, S. & Özyürek, A. (2003). What does cross-linguistic variation in semantic coordination of speech and gesture reveal? Evidence for...
- Klerfelt, A. (2007). Gestures in conversation: The significance of gestures and utterances when children and preschool teachers create stories...
- Kosslyn, S. M. (1989). Understanding charts and graphs. Applied Cognitive Psychology, 3(3), 185-225.
- Kurata, Y. & Egenhofer, M. (2008). The arrow-semantics interpreter. Spatial Cognition & Computation, 8(4), 306-332.
- Landis, J. & Koch, G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159.
- Levinson, S. C. (2006). Deixis. In L. R. Horn & G. Ward (Eds.), The Handbook of Pragmatics (pp. 97-121). UK: Blackwell Publishing.
- Mayer, R. E. (Ed.). (2005). The Cambridge handbook of multimedia learning. UK: Cambridge University Press.
- McNeill, D. (1992). Hand and Mind: What gestures reveal about thought. Chicago: University of Chicago Press.
- McNeill, D. (2005). Gesture and thought. Chicago: University of Chicago Press.
- Nys, M., Hickmann, M. & Gyselinck, V. (2018). The role of verbal and visuo-spatial working memory in the encoding of virtual routes by...
- Parodi, G. (2010). Research challenges for corpus cross-linguistics and multimodal texts. Information Design Journal, 18(1), 69-73.
- Parodi, G. (2012). Genre organization in specialized discourse: Disciplinary variation across university textbooks. Discourse Studies, 16(1),...
- Parodi, G. (2015). Variation across university genres in seven disciplines: A corpusbased study on academic written Spanish. International...
- Parodi, G., Julio, C. & Recio, I. (2018). When words and graphs move the eyes: The processing of multimodal causal relations. Journal...
- Ping, R., Church, R. B., Decatur, M. A., Larson, S. W., Zinchenko, E. & GoldinMeadow, S. (2021). Unpacking the gestures of chemistry learners:...
- Ping, R. & Goldin-Meadow, S. (2010). Gesturing saves cognitive resources when talking about nonpresent objects. Cognitive Science, 34(4),...
- Post, L. S., Van Gog, T., Paas, F. & Zwaan, R. A. (2013). Effects of simultaneously observing and making gestures while studying grammar...
- Rauscher, F. H., Krauss, R. M. & Chen, Y. (1996). Gesture, speech, and lexical access: The role of lexical movements in speech production....
- Richter, J., Scheiter, K. & Eitel, A. (2016). Signaling text-picture relations in multimedia learning: A comprehensive meta-analysis....
- Rueckert, L., Church, R. B., Avila, A. & Trejo, T. (2017). Gesture enhances learning of a complex statistical concept. Cognitive Research:...
- Skibinski, E., DeBenedetti, W., Ortoll-Bloch, A. & Hines, M. (2015). A blackboard for the 21st Century: An inexpensive light board projection...
- Sime, D. (2006). What do learners make of teachers’ gestures in the language classroom? Review of Applied Linguistics in Language Teaching,...
- Sweller, J. (1994). Cognitive load theory, learning difficulty, and instructional design. Learning and Instruction, 4(4), 295-312.
- Tversky, B., Jamalian. A., Giardino. V., Kang. S. & Kessell. A. (2013). Comparing gestures and diagrams. In Proceedings of TiGeR 2013:...
- Tversky, B., Agrawala, M., Heiser, J., Lee, P., Hanrahan, P., Doantam, P., Stolte, C. & Daniel, M. P. (2020). Cognitive design principles...
- Wahlster, W. (1991). User and discourse models for multimodal communication. In J. W. Sullivan & S. W. Tyler (Eds.), Intelligent User...
- Zahner, D. C. & Corter, J. E. (2010). The process of probability problem solving: Use of external visual representations. Mathematical...