The One Where We Redefine Literacy for a Multimodal World
Foreword: Reflections and Evolutions (2023)
As a student at Texas Woman’s University, I found myself very interested in the cross section where literacy, technology, and the way we create and consume information come together. Multimodality and a Social Semiotic Theory of Literacy (original paper below) is a culmination of my research in 2021. My journey since then, particularly my role as an Open Educational Resources (OER) Librarian at the University of Texas at Arlington, has profoundly deepened and expanded my understanding of literacy, education, and technology. This foreword serves as my most recent reaction to and continued interest in the topic, reflecting the integration of my experiences and insights gained in the rapidly evolving landscape of open education and digital literacy.
Reflecting on 2021 Through the Lens of 2023
Now, in 2023, my role as an OER librarian has further informed my perspective on these issues. Open educational resources have the potential to democratize education, providing access to a wealth of diverse, multimodal learning materials. This aligns with the need for a more inclusive and flexible approach to literacy, acknowledging the diverse backgrounds and learning styles of students. As an OER librarian, I advocate for and contribute to the creation and dissemination of resources that support this modern understanding of literacy, ensuring that educational materials are not only accessible but also relevant to today’s digital and multimodal world.
What’s Next (2023)
The challenge and opportunity lie in continuing to develop our understanding of multimodal literacy, keeping pace with technological advancements. My commitment is to contribute to this evolving field through research, writing, and preparing future educators to embrace a multimodal approach to literacy and learning.
Revisiting this paper has reinforced my belief in the need for a redefinition of communication and literacy that reflects our digitally interconnected world. Embracing this change is crucial for preparing our students to thrive in their future endeavors.
Multimodality and a Social Semiotic Theory of Literacy – Original Work (2021)
The landscape of learning and communication has changed dramatically over the past quarter-century. As the technologies (pencil and paper are considered a technology for communication) that we use to communicate evolve, the way we make and share meaning should change.
The Covid-19 Pandemic has opened the door for education reform. The 2020-2021 school year challenged educators to rethink how they teach. I am not sure that education and the “real” world have ever been more disconnected. Most would likely agree that we want our students to be prepared for their adult lives. We hope that when students leave our campuses, they have everything they need (skills, knowledge, experience) to function within and contribute to society. Why are we still preparing them for a world that looks like the 1950s? Or the 1980s? Or even the 2010s? We should be preparing them for the 2020s, the 2030s, and beyond.
In the photo, we see students before smart boards, smartphones, computers, etc., practicing financial literacy. It looks as if the students are role-playing a scene from a bank. Some students are bankers, and some are customers. The students are engaging in an authentic learning experience that reflects their time period. If a teacher recreated this lesson today, the picture would, and should, look different. There may be a computer on the desk, an ATM set up off to the side, and possibly a representation of mobile banking.
The problem we face now is that classrooms often use methods and pedagogy from the past and effectively prepare students for a time they will never see or need to navigate. There are plenty of reasons schools may not be using the most up to date methods in the classroom, such as curriculum, state, and district mandates, policies, politics, and textbook adoptions, a lack of high quality/updated professional development available to teachers, and even pre-service teacher programs that are outdated. The landscape of technology and communication has snowballed into an ever-changing, ever-growing, creator’s and information junkie’s paradise. According to Kress (2003), “it is no longer possible to think about literacy in isolation from a vast array of social, technological and economic factors.” We can count on change to continue to happen rapidly, forevermore.
Our understanding of literacy has expanded beyond traditional written and oral language. The digital age has introduced a variety of modes – internet, devices, images, symbols, and more – creating a society rich in information. This evolution calls for a shift in classroom dynamics, transforming students from passive receivers of knowledge to active creators and critical thinkers.
Zara, M. (2021) Multimodality and a social semantic theory of literacy. Texas Woman’s University.
We no longer rely primarily on language (written and oral) to convey messages and connect with others. Language has long been the dominant mode of communication in the world of literacy. Literacy has traditionally been sequestered in the English, Language Arts, and Reading realm and is rarely thought of by the general public as the way we make and share meaning in all aspects of our lives. Information is no longer only found in books and on paper; we receive and create information using various modes: the internet, devices, images, signs, symbols, design, gestures, sounds, and language. Different modes of communication offer different understandings; “the world told is a different world to the world shown” (Kress, 2003). With the advancement of technology, all of these modes have been made more accessible and easier to produce, which has created a society overflowing with information. Students should no longer be in the position of knowledge receivers in the classroom, where teachers act as gatekeepers of information. Students should now be engaging in learning experiences that build them into knowledge finders, knowledge creators, critical consumers, and problem solvers.
We should be looking at literacy and communication from a different perspective, and we need to expand our understanding of literacy. Literacy can no longer be seen as language only and must bring into sight and practice using technologies that provide more access to make and share meaning in the various modes (Kress, 2003). Multimodal literacy and a semiotic theory of literacy provide a solid foundation, albeit reasonably new research, to build a better understanding of a multimodal approach to literacy and bring more relevant pedagogy into public education (Kress, 2003).
Multimodality (2021)
Humans communicate using a variety of modes to convey meaning. Everything we do, and everywhere we go, we interact with text of some kind. Text is no longer simply words on a page or spoken language. Text can be anything used to communicate. Street signs communicate laws and directional patterns. Facial expressions communicate our mood. Wearing a jacket communicates to others that you feel it is cold enough to cover up. A soft spot on a piece of fruit can communicate the ripeness or quality of the fruit. The way information is communicated matters because different modes of communication have different affordances. The affordances extended by a mode are “shaped by its materiality, by what it has been repeatedly used to mean and do (its ‘provenance’), and by the social norms and conventions that inform its use in context – and this may shift, as well as through timescales and spatial trajectories” (as cited in Kress & Van Leeuwen, 2004; Lemke, 2000; Massey, 2005).
The idea of multimodality stems from the New London Group’s (1999) theory of multiliteracies. This theory states that literacy varies in different contexts. To be literate is no longer one thing; that is, one can be literate in multiple ways and multiple Discourses. In the 1990s, the New London Group conceived of the term multiliteracies, highlighting a new perspective of literacy theory and practice. New technologies and an increasingly diverse population have spurred the need for understanding literacy differently or developing a “new literacy.” Additionally, the New London Group noted that helping students develop literacy through traditional reading and writing practices was not sufficient and called for students to engage in digital literacies (Wright, 2019). Linguistic diversity and multimodal forms of expression were the pillars on which the idea of multiliteracies was based (The New London Group, 1999).
Gee discusses in the New Literacy Studies (NLS) (2015) the shift happening as we become aware of the various modes that play into making meaning in our increasingly digital world. The NLS believed in “[following] the social, cultural, institutional, and historical organizations of people (whatever you call them) first and then see how literacy is taken up and used in these organizations, along with action, interaction, values, and tools and technologies” (Gee, 2015). The NLS members believe that it is not only the ‘private mind’ that experiences and builds meaning, but that everything we ‘read’ has been socially and culturally structured through shared lived experiences (Gee, 2015). Kress (2003) describes the theoretical change as one “from linguistics to semiotics-from a theory that accounted for the language alone to a theory that can account equally well for gesture, speech, image, writing, 3D objects, colour, music, and no doubt others”.
Brian Street, Literacy in Theory and Practice (1984) and Shirley Brice Heath, Ways with Words (1983) have helped researchers understand literacy more from a social and cultural viewpoint. Street’s (1984) ideological model helps us understand literacy in terms of concrete social practices and think about the ideologies in which different literacies are embedded. Literacy, of any type, only matters how it works with other social factors, including political, economic, social, and logical ideologies. Heath’s (1983) ethnographic study of how literacy is embedded in the cultural context of the three communities in the Piedmont Carolinas helps researchers understand that context and culture affect the way we use language outside of the home (Gee, 2015).
The prefix multi- alludes to the presence of monomodal texts; however, all texts are multimodal. Multimodal texts are print-based and digital texts that use more than one semiotic resource, or mode, to represent meaning potentials. Modes are socioculturally shaped resources for meaning-making (Kress, 2003; Serafini, 2015). Multimodality is the awareness of how the various modes work together to build meaning (Alvermann et al., 2013). There are five general modes of communication that are considered, in addition to the socio-cultural aspect of the text’s production. The linguistic mode includes what we traditionally understand literacy to be. It includes words and general structures of oral and written language. The visual mode covers colors, vectors, and angles/perspectives in both still and moving images. The spatial mode considers proximity, direction, layout, and organization of objects in place. The gestural mode includes movement, speed, stillness, facial expressions, and body language. Finally, the audio mode considers volume, pitch, rhythm, music, and sound effects. (Gee et al., 2012; Jewitt, Carey & Kress, Gunther, 2003; Shaumyan, 1987)
When analyzing traditional text, a book, there is more to the book’s meaning than just the words on the page. The reader should consider not only the words but the structure of the words on the pages. When we come to a page that only has text on half of the page, it communicates that we have reached the end of a chapter or section, and we can expect that turning the page will bring a new big idea. Is it a graphic novel? Are images included? Are the images dependent on the text? Is the text-dependent on the images? Who is the author, and what time period, culture, class level, gender, etc., did they produce this information? Is there an audio companion? Does the author narrate the book him/herself? A reader could continue to ask questions about the book and all of the meaning that can be gleaned from its words, structure, and position in society. Even an image with no words can be considered both visual and spatial, and depending on what is in the image, a gestural mode could be employed.
Language (linguistic mode) is no longer the only mode of communication that should be considered when analyzing the meaning of text. Language is now seen as one piece to the entire multimodal approach to literacy (Kress, 2010). Multimodal literacy requires a shift in thinking about meaning-making, away from simply what sounds and words mean within an alphabetic system, to a more socio-cultural understanding of how humans communicate (Gee, 2015; Kress, 2003, 2010) In order to embrace a multimodal approach to literacy fully, researchers should be studying meaning-making from a semiotic perspective. Social Semiotics provides the groundwork for building a more complete and relevant understanding of literacy (Kress & Van Leeuwen, 2004, Kress, 2010).
Semiotics (2021)
Semiotics is the study of signs, symbols, and icons and their use or interpretation (Kress, 2010; Serafini, 2015). In semiotics, “language constitutes the essence of language as an instrument of communication and an instrument of thought,” which allows readers to approach language from different perspectives (Shaumyan, 1987). In his book Multimodality: A Social Semiotic Approach to Contemporary Communication, Kress (2010) explains that “the world of communication has changed and is changing still; and the reasons for that lie in the vast web of intertwined social, economic, cultural and technological changes.” Technology has changed the way people communicate, and the approach taken to build literate adults needs to change.
Studying multimodality and multimodal texts from a semiotic approach allows readers to access more potential meanings and interpretations. Social semiotics explores how humans derive meaning from the world around them. Charles Sanders Peirce (1991), one of the founders of semiotics, defines a sign as anything that represents (signifier) or indicates something else (signified). There are three categories of signs: icon, index, and symbol. An icon directly resembles or shares material qualities with its objects; for example, a picture of a cat represents a cat. An index holds an implied meaning; the signifier and signified share a logical relationship. For example, smoke indicates fire; a car horn may indicate traffic. Finally, a symbol is not inherently connected to the signified. There are no natural connections between the signifier and the signified. Symbols and their potential meanings are shaped by culture and social practices and interactions with the object in a specific context. For example, a solid line on a road indicates that cars may not pass. A solid line on a piece of paper could potentially separate information (Peirce, 1991). Symbols rely heavily on the use of other modes and cultural and social positioning to understand their meaning.
If we continue thinking of communication as solely language, we leave out part of the whole picture of communication. Words themselves are symbols and, therefore, share no logical connection to indicate meaning. The only way that meaning is understood by looking at written words or hearing oral language is to see them through a socio-cultural context. Words have meaning because we have been socialized to associate words (signifier) with things/ideas (signified) (Kress, 2010; Shaumyan, 1987; Peirce, 1991; Shaumyan, 1987).
Implications For the Classroom (2021)
At the time of writing this, I was a digital learning specialist. I worked directly with teachers to integrate technology in meaningful ways. There is a lot of talk and frustration around the idea that students are “digital natives.” The assumption that children come to school knowing how to use technology efficiently is dangerous because it means that educators are likely not focusing efforts on scaffolding the use of technology and digital literacy principles critical in preparing students for the future. Educators must scaffold both new technical skills and multimodal literacies (Mills, 2010). Not all students have access to technology before entering school and, therefore, do not have the prior knowledge to use technology without support effectively. Discussions about multimodal literacy practices of children have overlooked the fact that many of the students who are not coming to schools with experience using technology are not from the dominant culture (Mills, 2010).
Although teachers cannot expect students to come to the classroom knowing how to read multimodal texts or use technology for learning, teachers need to remember that students are immersed in a very connected and digital society. Their discourse includes text language, high-speed internet, information at their fingertips, video games that read to them and connect them to others outside of their inner-circle, cars that direct their parents where to go, and many have had their own device from a very young age. They see and understand their world through all of the modes and expect a multimodal approach to learning. Embracing and scaffolding for the use of technology and multimodal analysis honor the society and culture in which young people currently live. Ideally, the classroom and learning experiences should emulate the world in which students will eventually enter as adults. If they are not taught how to make meaning, create, and engage with multimodal texts, we are doing them a disservice (Mills, 2010).
Summary and Future Directions (2021)
Language needs a more relevant definition. Text is no longer simply the words on a page. Communication has changed dramatically, bringing connectedness to the forefront of all interactions. Our students are disconnected from what it means to live and function in a digitally-run world. Understanding literacy through multiple modes of communication and analyzing all texts through a semiotic lens will positively impact student learning and their transition into adulthood.
Research in multimodal literacy and a social-semiotic theory of literacy is still new and uncharted. The nature of technology and how quickly it evolves, and changes will dictate the fluidity and flexibility that a solid new theory and pedagogy should have. Moving forward, I hope to contribute to this body of knowledge. I plan to pursue research opportunities and to read and write widely on the topic. I am interested in what multimodal discourse analysis can tell us about learning and social interaction. Ultimately, I would love to help prepare pre-service teachers to teach and continue learning around multimodal literacy.
References
Gee, J. P. (2015). The New Literacy Studies. The Routledge Handbook of Literacy Studies Routledge. 10.4324/9781315717647.ch2
Jewitt, Carey & Kress, Gunther. (2003). Multimodal Literacy. Peter Lang.
Kress, G. (2003). Literacy in the New Digital Age. Routledge.
Kress, G. (2010). Multimodality: A social Semiotic Approach to Contemporary Communication. Routledge.
Kress, G., & Van Leeuwen, T. (2004). Multimodal discourse: The modes and media of contemporary communication.33(1), 115-118.
Mills, K. A. (2010). Shrek Meets Vygotsky: Rethinking Adolescents’ Multimodal Literacy Practices in Schools. Journal of Adolescent & Adult Literacy, 54(1), 35-45. 10.1598/JAAL.54.1.4
Serafini, F. (2015). Multimodal Literacy: From Theories to Practices. Language Arts, 92(6), 412-423.
Shaumyan, S. (1987). A Semiotic Theory of Language. Indiana University Press.
Wright, W. E. (2019). English language learners: Research, theory, policy, and practice (3rd. ed.) Calson.
Zara, Megan. (2021). Does it have to be text? [infographic].
Zara, Megan. (2021). Semiotic Resources [infographic].
