Multimodal literacy

Multimodal literacy THEO VAN LEEUWEN, PROFESSOR, INSTITUT FOR SPROG OG KOMMUNIKATION, SYDDANSK UNIVERSITET After defining the concept of multimodality and the same medium (oils), whatever their style or sub- contextualizing multimodality studies as a form ject. In concert performances, all musicians dressed of inquiry with roots in linguistics, the article more or less identically and only the conductor and reviews applications of multimodality to educa- soloist were allowed some bodily expression. The tion and exemplifies how multimodality studies academic study of such different modes of expres- approach the analysis of the meaning-making sion was equally monomodal: one discipline for lan- potential of multimodal resources and the way guage (linguistics), another for visual arts (art his- they are integrated in multimodal texts. tory), yet another for music (musicology) and so on, each with its own methods, its own assumptions, its Throughout, implications for multimodal litera- own technical vocabulary, its own strengths and its cy are discussed, emphasizing that multimodal own blind spots. literacy requires textual as well as contextual know­ledge, that is, an understanding of, for More recently, the dominance of monomodal- instance, visual grammar as well as an under- ity has weakened, although it still persists in standing of the rules or conventions that gov- some practices, for instance in the writing and ern its use in specific contexts. Multimodal lit- publishing of academic papers. The trend to- eracy also includes an aesthetic dimension, an wards multimodality began with the avant-garde ability to produce and appreciate the aesthetic experiments of the early twentieth century, when uses of layout, colour and typography that are, ‘concrete poetry’ began to express itself not only today, not only found in art, but also in many through words, but also through typography, and forms of everyday written communication. And when visual artists used new kinds of materials finally it includes a critical dimension, par- and sought to produce Gesamtkunstwerke which ticularly with regard to the way contemporary would combine as many forms of expression as digital technology favours, or even imposes, possible. From the 1920s onward, the mass me- modes of communication that suit the purposes dia, too, became increasingly multimodal. Maga- of corporate culture but may have drawbacks in zines acquired colour illustrations and sophisti- other contexts. cated layout and typography, and film, of course, became the multimodal art form par excellence, especially when the ‘talkies’ added speech, music What is multimodality? and other sounds to its palette. More recently, For several centuries there was, in Western culture, formerly austere genres such as textbooks and a tendency towards ‘monomodality’. The cultur- documents produced by corporations, univer- ally most highly valued genres of writing (literary sities and government departments have also novels, academic treatises, official documents, etc.) become multimodal, and the multimodal affor- were characterized by graphically uniform, dense dances of ubiquitous digital technologies such pages of print, and carried no illustrations. Paint- as Word and PowerPoint have made multimodal ings nearly all used the same support (canvas) and text design accessible to all. 4 VIDEN OM LITERACY NUMMER 21 | MARTS 2017 | NATIONALT VIDENCENTER FOR LÆSNING Linguists have only gradually realized this. During language but also voice quality, intonation, facial the 1930s and 1940s the Prague School began to expression, gestures etc. Even the densely printed extend linguistics into the visual arts and the non- pages I referred to earlier as ‘monomodal’ are in verbal aspects of theatre (cf. Garvin, 1964; Matêjka fact multimodal, as they use spacing, typography, and Titunik, 1976). Veltruský (1964 [1940]), for punctuation, etc. (cf. Norgaard). Multimodality instance, wrote about the theatre as a “multiple is therefore the study of how meanings can be sign system”, discussing sets, costumes and props made, and actually are made in specific contexts, as signs that provide setting and characterization with different means of expression or ‘semiotic and also take part in the action. And Bogatyrev modes’ – whether these are articulated with the (1971[1934]) studied dress as a language conveying body (speech, facial expressions, gestures and so what we would now call demographic informa- on) or with the help of tools and materials (writing, tion such as age group, place of residence, marital drawing, making music and so on). And it is also status, religion and occupation. the study of the ways in which multiple semiotic modes can be integrated into coherent multimodal Multimodality is therefore the texts. It is therefore not restricted to ‘multimedia’ in the sense of contemporary digital media but study of how meanings can be applies equally to face-to-face communication and made, and actually are made other non-digital types of text. Multimodal literacy is therefore the ability to use and combine different in specific contexts, with dif- semiotic modes in ways that are appropriate to the given context, both in the sense of the con- ferent means of expression or text-bound rules and conventions that may apply, ‘semiotic modes’. and in the sense of the unique demands made by each specific situation. Such a form of literacy The second wave was the 1960s Paris School of must be based on a knowledge of what can be done structuralist semiotics, which used linguistic with different semiotic modes and how and of the concepts and methods to understand communica- ways in which they can be integrated into multi- tive modes other than language, for the most part modal texts; however, it also, and equally impor- in analyses of popular culture and the mass media, tantly, requires an understanding of communica- rather than of the arts. Roland Barthes was its most tive contexts and an ability to respond creatively to crucial and influential proponent, writing, among the unique demands of specific situations. other things, about the use of image and text in advertising and the press, and about the language of fashion (e.g., 1967, 1977, 1983). During roughly Multimodality and education the same period, American linguists developed an The New London Group (Gunther Kress, James interest in non-verbal communication, for instance Gee, Allan Luke, Mary Kalantzis and others) stimu- in therapeutic interviews (Pittenger et al., 1960). lated an interest in applying multimodal text ana­ lysis to education (New London Group 1996). This Most linguists, however, continued to occupy led to four kinds of studies: studies of the develop- themselves with the grammar of sentences. Only ment of multimodal literacy in very young children, when, in the early 1970s, the emphasis changed often leading to a call for integrating multimodal from analysing sentences to analysing texts did literacy into the curriculum; studies of the affor- they begin to realize, first, that language is used dances and learning potentials of specific semiotic differently in different contexts, and that literacy modes; studies of multimodal classroom interac- should therefore be understood as the ability to use tion; and studies of multimodal learning resources, language in ways that are appropriate to context, including textbooks, toys, and the Internet. and, second, that all texts are multimodal and cannot be adequately produced or analysed, unless Gunther Kress’ Before Writing (1997) initiated the this is taken into account. Face-to-face speech, for study of the development of multimodal literacy, instance, is also multimodal, involving not only investigating how very young children use the VIDEN OM LITERACY NUMMER 21 | MARTS 2017 | NATIONALT VIDENCENTER FOR LÆSNING 5 affordances of whatever materials they have at its own epistemological affordances and limita- hand, or whatever techniques they have mastered, tions, and understanding these is fundamental for on the basis of ‘interest’, that is, what is of crucial creating effective multimodal texts. importance to them at the given moment. In one of his key examples, a three-year-old child draws a car Studies of classroom interaction have also moved as a series of circles (‘wheels’). Having mastered from the traditional emphasis on linguistic ex- the drawing of circles, the child now uses circles change structures to strong contextualization and as a means of expressing what, to him, is a crucial detailed attendance to non-verbal communication characteristic of cars. As a semiotic resource the and setting, e.g., to the way classrooms are ar- circle has many possible meanings, but the one ranged, what is hung on the walls, the technological the child selects is motivated by his interest at the resources available and so on. Kress et al. (2005) , moment, his interest in thinking about cars. Thus, for instance, described one classroom as realizing learning to draw and learning to understand the a ‘transmission’ pedagogy, with individual student world around him go hand in hand. But, as Kress tables lined up in rows, another as realizing a ‘par- said, “As children are drawn into culture, ‘what is ticipatory/authoritarian’ pedagogy, with tables put to hand’ becomes more and more that which the together to create teams of four or five students fac- culture values and therefore makes readily availa- ing each other (‘participation’), yet also arranged to ble” (1997: 13). This work inspired other studies of allow the teacher total visual control from the front the way young children use a range of materials and of the classroom, which constrained the posture techniques to create representations of the world of the students, at least if they wanted to see the around them (e.g., Ivanic and Ormerod). teacher and follow the lesson (‘authoritarian’) Closely related is the study of the affordances and Visuals, too, can express static learning potentials of different semiotic modes. In Literacy in the New Media Age (2003: 52–7), Kress and dynamic processes, they studied the use of different semiotic modes by just do it differently. junior high school students learning about blood circulation. Analysing two specific examples, he Finally, many studies have analysed multimodal showed how one student used language, writing a learning resources, from textbooks to comput- kind of travel diary with a red blood cell as its pro- er games. Jewitt (2006), for instance, studied a tagonist, making a voyage through the body, while computer game called Playground, designed to help another drew a concept map, with boxes represent- children learn the basics of physics. When learning ing the heart, the blood vessels, the lungs, and so on, to understand ‘bouncing’, for instance, children and arrows representing the movement of blood could choose a ‘behaviour’ (a particular kind of from one ‘box’ to another. The linearity of the story, bounce, represented by pictures of a spring, a ball Kress said, was an apt signifier for the blood mov- etc.) and attach it to an object (a ‘bullet’) which ing from organ to organ, and language allowed the could then bounce off bars. This, as Jewitt argued, expression of causality, but the use of many differ- was at times confusing. Can bullets be bouncy? ent words for the idea of movement (‘leave’, ‘come’, Isn’t the behaviour of ‘bouncing’ the property of the ‘squeeze through’, ‘enter’ etc.), while stylistically bars the bullets bounce against rather than of the desirable, diminished the generality which the sci- bullets? Nevertheless, Jewitt concludes, games of entific genre normally requires. The diagrammatic this kind do allow children to explore the rules of elements of the concept map (boxes and arrows), mechanics systematically, interactively and multi- on the other hand, did provide scientific generality, modally, practically without any verbal input. but since all the arrows in the student’s concept map radiated from a central ‘blood’ box, the con- cept of circularity was less clearly expressed, and Multimodal literacy since a visual convention for expressing causality The study of multimodal ways of making meaning does not, or net yet, seem to exist, causality was not has been inspired by linguistic concepts and meth- expressed at all. Each mode, Kress concluded, has ods, not because it was assumed that all semiotic 6 VIDEN OM LITERACY NUMMER 21 | MARTS 2017 | NATIONALT VIDENCENTER FOR LÆSNING modes work the way language does, but because blank background, the latter dynamically, in a picture in order to produce and analyse multimodal texts, of English settlers raising their guns as they stalk up we need at least some concepts and methods that to a group of Aboriginal people seated around a fire. can apply across different modes. From the point This is why language and visual communication can of view of production we must be able to ask: How be seen as resources for representation and interac- shall I express, say, a feeling of optimism? Visually, tion. They offer choices for how to represent people, verbally or both? Can optimism be expressed both places, things and events. If a different choice had verbally and visually, and if so, how? From the point been made, with the Aboriginals shown as attacking of view of analysis, we must be able to ask: How the settlers (which they did) and the British guns as is agency expressed verbally and visually in this museum pieces, a wholly different view of history multimodal text? And to answer that question, we would emerge. This kind of choice is not tied to lan- must know whether agency can be expressed both guage or to visuals. It is a cultural choice that derives visually and verbally, and if so, how. from a tradition, which includes both the Hebrew god, who acts in history and the Greek god who A recent attempt at finding such a common language exists in Heaven. Not all cultures have this choice. In has been the ‘visual grammar’ developed by Kress Wintu, a now critically endangered Canadian native and Van Leeuwen (2006) and further elaborated by language, you cannot say, for instance, ‘This is bread’. others, e.g., by Boeriis (2009). To give just one exam- You have to say the equivalent of “It looks-to-me- ple, in Hallidayan linguistics (Halliday, 1994), the bread” or “I think-it-to-be-bread” (Lee, 1954: 51). The ‘process’ of a clause is expressed by a verb or a verbal distinction between objective and subjective does group. Two broad categories of processes are recog- not exist here. nized, those which denote a more or less permanent state of affairs, a characteristic attribute of some- Figure 1 below is the main part of a screen definition thing (relational processes) or the overall identity or from Mathletics, a commercial online resource for meaning of something (identifying processes), and learning mathematics that is used by more than four those which denote some kind of action or event, million students the world over. Both the verbal defi- whether ‘material’, ‘verbal’ or ‘mental’. The former nition and the picture contain a dyna­mic element. are expressed by stative verbs or verbal groups, most In the verbal definition, the dynamic element is the typically by ‘have’ and ‘be’, the latter by dynamic verb ‘arrange’; in the picture, it is the steam coming verbs or verbal groups, such as ‘walk’, ‘talk’, ‘think’ and from the locomotive, representing, together with the thousands of others. There are no visual equivalents slight slant of the locomotive itself, the idea of ‘mo- of verbs or verbal groups; in other words, there can be tion’ rather than ‘arranging’. True, the rails are neatly no equivalence between language and visual commu- arranged, but they are less salient than the train, and nication in terms of their forms. However, there are this may cause confusion. A picture of men laying equivalences in terms of their meanings and func- the track might have been more appropriate. tions. Visuals, too, can express static and dynamic processes, they just do it differently. They express dynamic processes by means of a vector, a dynamic element formed either by a (usually oblique) element of what is being depicted, for instance a hand raised in a gesture of greeting, or by an abstract element that has a sense of direction, for instance an arrow. And they express static processes through the lack such a dynamic element. Kress and van Leeuwen exemplified this with two pictures that were placed side by side in an Australian social studies textbook for primary school children. One depicted Aboriginal artefacts (including a wooden sword), the other the weapons of the English settlers. The former were de- Figure 1: “To arrange in a line” (from the Mathletics picted in a static, symmetrical arrangement, against a site) VIDEN OM LITERACY NUMMER 21 | MARTS 2017 | NATIONALT VIDENCENTER FOR LÆSNING 7 So far we have looked at the verbal and the visual for brainstorming: an idea is placed in the centre separately. But how do they relate to each other? of the visual space and then linked, with lines, to Here, too, there are choices. They were described in whatever association comes to mind. outline by Barthes (1977) and worked out in greater detail by others (e.g., Martinec and Salway, 2005), departing from the linguistic theory of conjunc- tion, which charts how clauses can be related to each other, for instance temporally (‘and then’) or causally (’and therefore’), or conditionally (‘if’). A key general choice, first introduced by Barthes, is the choice between elaboration, the case in which the image ‘says the same thing’ as the text (or the text ‘the same thing’ as the image), and extension, the case in which the image adds information that is not in the text (or vice versa), so that text and image complement each other, or perhaps contrast with each other. In Figure 2, again from Mathletics, the picture shows an example of the to-be-defined term (2D Shape) and therefore elaborates the definition, But it also provides additional information about the components of 2D shapes (‘sides’ and ‘corners’) and itself contains elaborative text and images relations in which the image shows things and the text labels them. The question, however, is wheth- Figure 3: Mind map or concept map? (Reproduced er 2D shapes, as defined here, always have corners with permission from Addison, P. Bickham, and sides. There are of course cases where text and C. Melissas, S. Wood, J, Pearson History 7 Student image relations are deliberately ambiguous, open Book © 2011 Pearson Australia, page no. xix.) to multiple interpretations. But perhaps that is less than appropriate in the case of mathematics. As a whole, this is a visual structure, although the linked elements may be either verbal or visual. ‘Concept maps’, on the other hand, require more precise relations. In the example above, taken from a junior high school history textbook (Addison, 2011), ‘citizens’ and ‘metics’ are depicted as kinds of ‘members of society’ in Ancient Greece. In other words, the image text relation is one of hyponymy, the ‘kind of’ relation traditionally expressed in tree diagrams. However, the inventory of citizens is not complete (slaves have been left out, for instance), and the picture of the third kind of ‘members of society’ is not labelled. Such ambiguities (is this a ‘mind map’ or a ‘concept map’?) may impede effec- Figure 2: ‘2D Shape’ (from the Mathletics site) tive learning, and resources for explicit multimod- al analysis are needed to spot them. To give another example, ‘mind maps’ were originally created by advertisers as a resource 8 VIDEN OM LITERACY NUMMER 21 | MARTS 2017 | NATIONALT VIDENCENTER FOR LÆSNING  functions of communication – creating rep- [Text] [Text] resentations of the world, enabling interactions and the relationships that go with them and forming coherent texts that can be recognized as performing a communicative job of some kind in the world. This does not mean, however, that the [Text] [Text] aesthetic element has gone away. On the contrary, as pioneered by advertising, a field whose commu- nicative strategies are now a major influence in many other forms of communication, the aesthe­ [Text] [Text] tic has entered many domains where it formerly played no role. Writing of all kinds now has to ‘look good’, and not only in the work of professionals: Figure 4: SmartArt template “If you think a document that looks this good has to be difficult to format, think again!” says a Word The SmartArt template in figure 4 is, according template for company brochures, and “add profes- to Microsoft, to be used “to show hierarchical sional quality graphics which automatically match relationships progressing across groups” and “can the look of your report” (cf. Van Leeuwen, 2015). also be used to group or list information”. However, Functional and aesthetic communication have it may be argued that hierarchies and lists are not now merged, and many non-marketing forms of the same – all the items in a list are, in principle, communication are expected to follow the lead of equal, of the same kind, even though the items at marketing in combining functionality (providing the top of the list may be prioritized. Also, hierar- information, persuading, instructing, applying for chies and lists are static and do not include any jobs or leave and even invoicing) with the commu- element of ‘progress’. When such contradictions nication of the values and meanings that make up occur, there may be an ideological snake under the the identity of the communicators, whether they grass. Power is played down here, and corporate are individuals or corporations. This, too, has to structure becomes a model for understanding the form part of multimodal literacy, which perhaps world. As Kvåle (2016: 269) has explained in the suggests that art and communication must begin conclusion of a study of the use of SmartArt in the to merge in the curriculum, as it already has in the work of students in higher education: world. SmartArt [is] a historically and socially evolved The aesthetic has entered resource for defining organizational charts. (…) Because of the profound status of Microsoft Office many domains where it for- today, the company’s templatized idea of the most merly played no role. “effective” visual style of organizational charts is infused into all social practices, including education (…) Concluding remarks Clearly, multimodal literacy should also be a form Not long ago, multimodal text design was a specia­ of critical literacy. lized professional skill. Today, digital technology has brought resources for multimodal text design within reach of anyone who has a computer and Aesthetics introduced multimodal text design into many are- In the more or less ‘monomodal’ past, the visual as that were previously ‘monomodal’, for instance was seen as embellishment, an optional extra. in workplace documents, such as invoices, reports, Today, verbal and visual communication work presentations, organization charts and workflow together (often in tandem with other means of documents, brochures, newsletters. expression) with respect to all three of the main VIDEN OM LITERACY NUMMER 21 | MARTS 2017 | NATIONALT VIDENCENTER FOR LÆSNING 9 As a result, the study of multimodality has grown References rapidly, devising detailed grammars that show how Addison, P. (2011) Pearson History Student Book 7. different semiotic modes such as layout, colour Melbourne: Pearson and typography make meaning and combine into coherent multimodal texts. Multimodal litera- Barthes, R. (1967) Elements of Semiology. London: cy has become an important work area and life Cape skill, requiring not only a working knowledge of multimodal ways of making meaning but also of Barthes, R. (1977) Image, Music, Text. London: the rules and conventions for using multimodal Fontana meaning making that have emerged, with greater or lesser prescriptiveness, in specific contexts. Barthes, R. (1983) The Fashion System. Berkeley and Los Angeles: University of California Press Multimodal literacy therefore Boeriis, M. (2009) Multimodal Semiotik og Lev- also needs to be critical liter- ende Billeder. Odense: Syddansk Universitet, acy, especially in relation to Humanistisk Fakultet the way technological tools Bogatyrev, P. (1971[1934]) The Function of Folk Costume in Moravian Slovakia. The Hague: Mou- favour certain forms of dis- ton de Gruyter course. Garvin, P. L. (1964) A Prague School Reader on Es- Multimodal literacy has also added an aesthetic thetics, Literary Structure and Style. Washington, dimension to everyday forms of communication DC: Georgetown University Press that were formerly purely functional and mono- modal. It fosters forms of communication that Halliday, M. A. K. (1994) An Introduction to Func- combine functionality, a focus on the job that a tional Grammar. 2nd Edition. London: Arnold given piece of communication is intended to per- form, with the use of aesthetic design ideas meant Jewitt, C. (2006) Technology, Literacy and Learn- to communicate corporate or personal identities ing: A Multimodal Approach. London: Routledge. and values. Kress, G. (1997) Before Writing: Rethinking the Finally, many of the digital resources that provide Paths to Literacy. London: Routledge wide access to multimodal text design favour spe- cific forms of communication that are built on spe- Kress. G. (2003) Literacy in the New Media Age. cific values and serve specific aims. PowerPoint, London: Routledge for instance was originally designed by Bell Labo- ratory engineers to pitch ideas to management and Kress, G, Jewitt, C., Bourne, J, Franks, A. Hard- is still particularly suited for the concise presenta- castle, J., Jones, K. and Reid, E. (2005) English in tion of a number of ‘selling points’. However, it Urban Classrooms: A Multimodal Perspective on less easily facilitates other forms of discourse that Teaching and Learning. London: Continuum may, for instance, be important in education, such as extended arguments and narratives (cf. Tufte, Kress, G. and van Leeuwen, T. (2006) Reading Im- 2006). Multimodal literacy therefore also needs ages – The Grammar of Visual Design. 2nd edition. to be critical literacy, especially in relation to the London: Routledge way technological tools favour certain forms of discourse. Kvåle, G. (2016) ‘Software as Ideology: A multi- modal critical discourse analysis of Microsoft Word and SmartArt’. Journal of Language and Politics 15(3): 259–274 10 VIDEN OM LITERACY NUMMER 21 | MARTS 2017 | NATIONALT VIDENCENTER FOR LÆSNING Lee, D. (1954) ‘Notes on the conception of the self among Wintu Indians’, in E. Carpenter and M. McLuhan, eds. Explorations – Studies in Culture an Communication, Vol 3, p.44–57 Martinec, R. and Salway, A. (2005) ‘A system for image-text relations in new (and old) media’. Visual Communication 4(3): 337–372 Matêjka, L. and Titunik, I. R., eds. (1976) Semiotics of Art: Prague School Contributions. Cambridge, MA: MIT Press. New London Group (1996) ‘A pedagogy of multilit- eracies: designing social future’ Harvard Educa- tional Review 66: 60–92 Pittenger, R. E., Hockett, C. F., and Danehy, J. H. (1960) The First Five Minutes. Ithaca, NY: P. Mar- tineau Tufte, E. R. (2006) The Cognitive Style of Power- Point: Pitching Out Corrupts Within. Cheshire, CT.: Graphics Press LLC Van Leeuwen, T. (2005) Introducing Social Semi- otics. London: Routledge Van Leeuwen, T. (2015) ‘Looking good: aesthetics, multimodality and literacy studies’ in J. Rowsell and K. Pahl, eds. The Routledge Handbook of Litera- cy Studies, pp. 426–439 Veltruský, J. (1964[1940]) ‘Man and object in the theatre’, in P. L. Garvin, ed. A Prague School Reader on Esthetics, Literary Structure and Style. Wash- ington, DC: Georgetown University Press, pp. 83–89 VIDEN OM LITERACY NUMMER 21 | MARTS 2017 | NATIONALT VIDENCENTER FOR LÆSNING 11