Academia.eduAcademia.edu

Outline

Title
Abstract
Key Takeaways
Future Directions
References
FAQs
All Topics
Languages and Linguistics
Discourse Analysis

(2020) Multimodality

Profile image of Luis Perez-GonzalezLuis Perez-Gonzalez

2020, in Mona Baker and Gabriela Saldanha (eds) The Routledge Encyclopedia of Translation Studies, 3rd edition, London and New York: Routledge, 346-351.

visibility

…

description

12 pages

Abstract

Multimodality, a term first used in the late 1990s (Jewitt et al. 2016:2), is the study of how we make meaning by combining multiple signifying means or modes-for example, image with writing, music and body movement, speech with gesture-into an integrated whole. While more established disciplines such as linguistics, semiotics and musicology engage primarily with those semiotic resources that fall within their respective conceptual remits, multimodality investigates the synergies between co-occurring semiotic resources. From a multimodal standpoint, each mode "is understood as realizing different communicative work" (Jewitt 2014/2017:16) and is moulded by its own semiotic limitations, potentialities and affordances. Multimodal ensembles-whether they adopt the form of a graphic novel, a film, an illustrated textbook, a museum exhibit or an everyday conversational encounterresult from the interplay between the relevant cooperating modes. Multimodality specialists have "not as yet focused on questions of translation" (Taylor 2016:222), while translation scholars have been slow to engage with multimodal concepts and methods. Theoretical and methodological developments in translation studies until the 1990s were driven by the conceptualization of translation as a process of written language transfer where the printed word is the only signifying means at play. The study of spoken texts as loci of interpreting activity has similarly tended to revolve around their verbal fabric, often glossing over the semiotic contribution of the orality and corporeality of interpreter-mediated speech. This disciplinary emphasis on language-centred meaning-making processes largely derives from the influence of linguistics during the formative period of translation (and interpreting) studies (Baker 2005). It is also consistent with the entrenched prevalence of monomodality, understood as the dominance of one signifying constituent, such as written language, over other types of meaning-making resources. Approaches to multimodality Translation studies has not yet managed to articulate clearly how semiotics and multimodality relate to one another. Stecconi (2009:261) argues that, when used in a lax sense, semiotics encompasses "research that goes beyond verbal language", including "multimedia/multimodal material", a term he associates with Gottlieb's (2005) work. But Gottlieb's (2017:46) own contribution to this debate identifies multimodality, described as an addition to the more established body of scholarship on multi-channel texts and paraverbal translation, exclusively with the work of multimodality specialists (Baldry and Thibault 2006; Kress 2010). Leaving aside the profusion of terms that translation scholars use to designate their theoretical explorations of meaning-making resources and practices, they would appear to agree that multimodality falls within the wider remit of semiotics.

Related papers

A Multimodal Social Semiotic Approach to Translation
Elisabetta Adami

Routledge Handbook of Translation Theory and Concepts, 2023

View PDFchevron_right
Multimodality: A guide for linguists
Jeff Bezemer

This chapter provides a rationale for a multimodal perspective on meaning, communication and discourse. It draws attention to the range of different modes that people use beyond speech and writing and explores the theoretical and methodological implications of multimodality. The chapter addresses two key questions. First, what is multimodality? Why and how is it relevant to linguistics? Second, how has multimodality been taken up? What questions are addressed, what materials are collected, and what methods are used to analyze these materials? Taking one approach – Social Semiotics – and one area of research – online text making – as an example, the chapter defines and illustrates key concepts and steps in multimodal inquiry. It concludes with consideration of future directions.

View PDFchevron_right
Why methods matter: Approaching multimodality in translation research
Catalina Hurtado

Linguistica Antverpiensia, New Series – Themes in Translation Studies

The study of multimodal phenomena calls upon translation scholars to cross disciplinary boundaries and adopt a range of theoretical and methodological approaches. The diversity of the multimodal landscape brings about research challenges that must be carefully addressed to ensure that these research efforts yield useful and credible results. This special issue is dedicated to a discussion on how to engage in multimodal translation research: how traditional research methods can be adapted and what kinds of novel approaches can be adopted or developed in order to deal with a diversity of multimodal data. In this introduction, we first discuss definitions of mode and multimodality and reflect on the nature of multimodality as a topic of research within Translation Studies. We then explain our rationale for dedicating the special issue to research methods and introduce three areas of multimodal translation research that, in our view, merit particular attention from a methodological poin...

View PDFchevron_right
Multimodal Translation: Is Translation only Verbal?
Shubhangi Rao

CSMFL Publications eBooks, 2021

This chapter is based on the Multimodal theory of translation. Although the practice of translation is long-established, the study developed into an academic discipline much later as of the second half of the twentieth century. Before that translation had normally been the element of language learning which was dominated by the Grammar translation method centered on the role study of the grammatical rules and structures of foreign language. The Romantic approach of originality of work has always denied the study of translation as a discipline. The original character of the text has tampered with when it is translated. The idea of Mimesis given by Plato and Aristotle stating all arts as imitative clearly would deny the systematic study of translation. Translation was considered a part of comparative literature but it gained recognition as a separate discipline of study only after the mid-twentieth century along with the emergence of various other disciplines like cultural studies, gender studies, postcolonial studies etc. Since translation studies emerged as an academic discipline, there have been questions about the equivalence of translation from one language to another. But there are also instances in which translation according to the culture is said to be an art in itself. Looking from another perspective, translation from one text to another is entirely

View PDFchevron_right
Multimodality and Semiotics: Language, Image, and Meaning in the Globalized Era
Htein Win

Linguistics, 2025

This paper examines the theoretical and practical implications of multimodal semiotics in contemporary global communication. Drawing on foundational work by Gunther Kress and Theo van Leeuwen, this study explores how meaning-making has evolved from traditional linguistic analysis to encompass visual, auditory, gestural, and digital modes of communication. The research demonstrates that multimodal approaches fundamentally transform our understanding of discourse by revealing how integrated sign systems operate across cultural and technological boundaries. Through critical analysis of theoretical frameworks, digital applications, and crosscultural communication patterns, this paper argues that multimodal semiotics provides essential tools for understanding meaning-making in globalized digital contexts. The findings suggest that contemporary communication increasingly relies on sophisticated interactions between multiple semiotic modes, challenging traditional linguistic paradigms and requiring new analytical frameworks. The study concludes by identifying future research directions in AI-mediated communication and algorithmic semiotics, emphasizing the growing importance of multimodal literacy in digital society.

View PDFchevron_right
Introduction: Multimodality as challenge and resource for translation (2013)
Carol O'Sullivan

Introduction to special issue of the Journal of Specialised Translation on multimodality

View PDFchevron_right
Thinking outside the “Methods Box”: New Avenues for Research in Multimodal Translation
Anne Ketola

Many approaches used in research into the translation of multimodal texts (e.g. van Meerbergen 2009, Taylor 2003), draw from Systemic Functional Linguistics and hence employ concepts developed in linguistics to the analysis of all modes present in a multimodal text. This line of thought tends to foreground the verbal in relation to the other modes. In both of our research, analysing the role of sound in film translation (Kokkola) and the role of images in technical translation (Ketola), we have observed that image and sound play an essential role in the overall meaning construction. We therefore emphasize the need to view image and sound as objects of inquiry in their own right, instead of mere properties of the texts being translated. In this article we propose that the existing methods could be complemented by introducing new methods that address the specificities of image and sound in multimodal translation. Introducing innovative methods requires the discipline to step outside of its “methods comfort zone” and think outside its “methods box” (Hesse-Biber & Leavy 2008: 10), but novel methods can also provide impetus for asking new questions and discovering new realities. Inspired by this idea, we ask if the adoption of such methods might constitute a step towards the enlarged concept of translation (Tymoczko 2007: 57–58), which might eventually lead to reassessment of the borders of Translation Studies.

View PDFchevron_right
Locating the Semiotic Power of Multimodality
Mark Evan Nelson

Written Communication, 2005

This article reports research that attempts to characterize what is powerful about digital multimodal texts. Building from recent theoretical work on understanding the workings and implications of multimodal communication, the authors call for a continuing empirical investigation into the roles that digital multimodal texts play in real-world contexts, and they offer one example of how such investigations might be approached. Drawing on data from the practice of multimedia digital storytelling, specifically a piece titled “Lyfe-N-Rhyme,” created by Oakland, California, artist Randy Young (accessible at https://www.youtube.com/watch?v=yfFg8zNkXZM), the authors detail the method and results of a fine-grained multimodal analysis, revealing semiotic relationships between and among different, copresent modes. It is in these relationships, the authors argue, that the expressive power of multimodality resides.

View PDFchevron_right
Multimodality in English for specific purposes: Reconceptualizing meaning-making practices
Anna Franca Plastina

2013

The rapid evolution from print-based to multimodal information has still not received sufficient attention from the field of English for Specific Purposes (ESP). This paper advocates the need to re-conceptualize ESP through multimodal practice for new opportunities of interactive learner engagement. For the purpose, qualitative, exploratory research was conducted on multimodal ESP practice carried out with post-graduate students at the University of Calabria in Italy. The study addresses the issues of how multimodal environments can affect ESP and how a multimodal assignment can influence learner motivation, engagement and awareness. A theoretical multimodal semiotic approach was combined with multimodal pedagogy to investigate the benefits of learners’ engagement in creating artefacts with content-specific language, as well as developing awareness of their meaning-making processes. A questionnaire survey revealed learners’ active involvement determined by intrinsic, extrinsic and a...

View PDFchevron_right
Multimodality and empiricism: preparing for a corpus- based approach to the study of multimodal meaning-making
John Bateman

2002

Following the ‘visual turn’ in many areas of communication, investigators are increasingly considering explicitly both the presentation of information in forms such as photographs, diagrams, graphics, icons and so on, and interrogating their relationships with linguistically presented information. The majority of analyses currently proposed, however, remain impressionistic and difficult to verify. In this paper, we argue that the study of multimodal meaning-making needs to be placed on a more solid empirical basis in order to move on to detailed theory construction. We describe the state of the art in corpus preparation and show how this can be expanded to be of value for supporting investigative work in the area of multimodality.

View PDFchevron_right

Key takeaways
sparkles

AI

  1. Multimodality explores the synergies between various semiotic resources in meaning-making processes.
  2. Translation studies historically emphasized language, neglecting multimodal concepts and practices.
  3. Structuralist approaches classify translation types but overlook dynamic intermodal relationships.
  4. Social semiotics significantly influences the understanding of meaning through contextually shaped modes.
  5. Future research should integrate cognitive models to enhance comprehension of multimodal texts.
This is an author’s manuscript accepted for publication in: Mona Baker and Gabriela Saldanha (eds) Routledge Encyclopedia of Translation Studies, 3rd edition, London and New York: Routledge, 346-351. https://bit.ly/2PxLkbp Multimodality Luis Pérez-González Multimodality, a term first used in the late 1990s (Jewitt et al. 2016:2), is the study of how we make meaning by combining multiple signifying means or modes – for example, image with writing, music and body movement, speech with gesture – into an integrated whole. While more established disciplines such as linguistics, semiotics and musicology engage primarily with those semiotic resources that fall within their respective conceptual remits, multimodality investigates the synergies between co-occurring semiotic resources. From a multimodal standpoint, each mode “is understood as realizing different communicative work” (Jewitt 2014/2017:16) and is moulded by its own semiotic limitations, potentialities and affordances. Multimodal ensembles – whether they adopt the form of a graphic novel, a film, an illustrated textbook, a museum exhibit or an everyday conversational encounter – result from the interplay between the relevant co-operating modes. Multimodality specialists have “not as yet focused on questions of translation” (Taylor 2016:222), while translation scholars have been slow to engage with multimodal concepts and methods. Theoretical and methodological developments in translation studies until the 1990s were driven by the conceptualization of translation as a process of written language transfer where the printed word is the only signifying means at play. The study of spoken texts as loci of interpreting activity has similarly tended to revolve around their verbal fabric, often glossing over the semiotic contribution of the orality and corporeality of interpreter-mediated speech. This disciplinary emphasis on language-centred meaning-making processes largely derives from the influence of linguistics during the formative period of translation (and interpreting) studies (Baker 2005). It is also consistent with the entrenched prevalence of monomodality, understood as the dominance of one signifying constituent, such as written language, over other types of meaning-making resources. Approaches to multimodality Translation studies has not yet managed to articulate clearly how semiotics and multimodality relate to one another. Stecconi (2009:261) argues that, when used in a lax sense, semiotics encompasses “research that goes beyond verbal language”, including “multimedia/multimodal material”, a term he associates with Gottlieb’s (2005) work. But Gottlieb’s (2017:46) own contribution to this debate identifies multimodality, described as an addition to the more established body of scholarship on multi-channel texts and paraverbal translation, exclusively with the work of multimodality specialists (Baldry and Thibault 2006; Kress 2010). Leaving aside the profusion of terms that translation scholars use to designate their theoretical explorations of meaning-making resources and practices, they would appear to agree that multimodality falls within the wider remit of semiotics. The bulk of the work undertaken to date on the (loosely understood) semiotics of translation aims to draw up inventories of meaning-making signs associated with both verbal and non- verbal sensory channels. Broadly speaking, the structuralist approach to the study of multimodality in translation aims to establish taxonomies in order to “deal systematically with any type of translation encountered in today’s communicative landscape” (Gottlieb 2017:46), whether it concerns text-types such as film where both the original and translated versions are delivered through more than one channel; translations where the target version incorporates new communicative channels that were not used in the original, as in the case of a screen adaptation of a novel; or translations that reduce the number of channels at play in the source, as in a radio adaptation of a film. Jakobson’s intersemiotic translation, defined as the “interpretation of verbal signs by means of signs of nonverbal sign systems” (1959/2000:114) remains a linchpin of these taxonomic explorations, even though scholars have redefined it, to some extent, over the following decades. Remael’s (2001:13-14) use of the term to designate the transfer of meaning across different media, for example, still follows Jakobson closely. By contrast, Fine’s (1984) definition of intersemiotic as the transfer of meaning across medial variants of the same sign – as seen in subtitling, where the meaning of spoken language is conveyed through written text – represents a departure from the original formulation. However, the significance of Jakobson’s contribution to the structuralist strand of multimodal translation research can be best appreciated in Gottlieb’s (2005, 2017) taxonomy of translation types. In this classification, the difference between intersemiotic translations and their intrasemiotic counterparts, the latter understood as textual transfers where “the sign systems used in source and target texts are identical” (2017:51), lies at the heart of a complex web of 34 translation types that encompass “any communicative system working through the combination of sensory signs” (ibid.:50; original emphasis). But while this inclusive theorization of multimodality aspires to account for “any process, or product hereof, in which a text is replaced by another text reflecting, or inspired by, the original entity” (Gottlieb 2017:50; original emphasis), most of the research informed by the structuralist approach has tended to concentrate on translations involving basic transfers of meaning between verbal and non-verbal channels. The collection edited by Poyatos (1997) on various aspects of non-verbal communication and translation, including paralanguage and kinesics, and Chaume’s (2004) structuralist account of signifying means in audiovisual translation are two cases in point. Audiovisual texts, Chaume argues, involve the transfer of meaning along the acoustic and visual channels. The former conveys meaning produced through signs drawn from the verbal code (the spoken word), the paraverbal code (how speech is delivered, rather than what it delivers) and the non-verbal code (including, for example, sound arrangements, special sound effects or music). The visual code, on the other hand, involves the use of signs from the iconographic code (primarily symbols and icons), the photographic code (relating to the use of perspective, colour or light), and the mobility code (operationalized through a classification of proxemic and kinesic cues). While structuralist approaches deliver comprehensive mappings of signifying resources and allow for thorough and systematic descriptions of how meaning is made and transferred in translation, they also have blind spots. On the whole, they privilege the conventional and static meaning of established signs and codes over the potential of semiotic modes to form and negotiate changing relationships with other modes, as required by the specific demands of communicative events. This is precisely what the social semiotic approach to multimodality attempts to do. Among the various strands of multimodal research surveyed by Jewitt et al. (2016:6-13), social semiotics has exerted the greatest influence in translation studies (Kaindl 2012). The central concept in the social semiotic strand of multimodality is mode, defined by Kress (2017:60) as “a socially shaped and culturally given resource for making meaning”. Modes such as “image, writing, layout, music, gesture, moving image, [and] soundtrack” (ibid.) are shaped by the daily social interaction of people, and their semiotic potential is the product of the cultural shaping of the resource in question. A given semiotic resource such as colour may have different meanings in different contexts, as each context “may either have rules or best practices that regulate how specific semiotic resources can be used, or leave the users relatively free in their use of the resource” (van Leeuwen 2005:4). Multimodal texts are therefore composite products resulting from the combination of various modes or modalities that reach the senses of people through media, i.e. “the material resources used in the production of semiotic products and events, including both the tools and the materials used” (Kress and van Leeuwen 2001:22). Examples of media include screens, loudspeakers, paper, fabric, software and clay. Most multimodal texts are also multimedial, as they must be conveyed through various carrier media to be accessed and enjoyed. While structuralist semiotics foregrounds codes and taxonomies, social semiotics is particularly interested in the social uses and the interrelationships of semiotic modes in social practice. It ultimately seeks to understand how people produce and communicate meaning by combining several modes in a specific social setting. Meaning-making resources can be subjected to scrutiny along three analytical dimensions: discourse, genre and style (Kress and van Leeuwen 2001). Discourses are “knowledges” (van Leeuwen 2005:94) of some aspect of reality and are socially constructed through conventionalized combinations of semiotic resources, or genres, to achieve specific purposes in a specific social occasion. Discourses and genres are highly dynamic constructs, as the social settings and communicative purposes they serve change over time and space. Using distinctive combinations of discourses and genres confers a style or identity on sign users that can be exploited for communicative purposes. Chueasuai’s (2013) study of the multimodal shifts that arise in the translation from English into Thai of Cosmopolitan’s ‘Love and Lust’ feature articles is one of the earliest applications of social semiotics to translation studies. By incorporating a range of changes in typography, colour, layout, photographic images and the actual text, Cosmopolitan’s Thai edition delivers a euphemized version of this sexually explicit material. Chueasuai focuses on the cultural shaping of the modes at play to account for these changes. The editorial policy of Cosmopolitan’s Thai edition, local media legislation and socio-cultural norms on femininity are shown to influence the discourses, genres and styles articulated in the translated material. The extent to which social settings influence styles, and hence the semiotic potential of the modes used to make meaning, is also explored in Chang’s (2015) study of note-taking in interpreter-training contexts. While traditional studies of note-taking conventions emphasize the fixed meaning that trainers have arbitrarily allocated to a standard set of signs, the social semiotic approach explains how the interpreters’ personal communicative intentions drive their choice and use of resources drawn from a range of visual modes. Significantly, the fact that interpreters tend to exploit the semiotic potential of their chosen modes in a consistent manner suggests that notes can be conceptualized as a “third visual language with its own logic and meaning-making practices” (ibid.:7). The social semiotic approach is also at the heart of work that seeks to understand how inter- modal connections are established within multimodal ensembles, in order to dissect holistic perceptions of multimodal texts as unified semiotic entities. One example is Remael and Reviers (2018), who focus on the study of filmic multimodal cohesion, defined as “any instance of implicit or explicit ‘sense-relation’ between two or more signs, from the same or different modes, within a given text that helps the reader create a coherent textual semantic unit” (ibid.:260). Maintaining or (re)creating multimodal cohesion is particularly important in audio descriptions and subtitles for the hard of hearing, as these forms of translation can only deliver fully monomodal versions (acoustic and visual, respectively) of a multimodal original. Remael and Reviers’ study foregrounds the importance of genre, as conceptualized within social semiotics, for the perception of multimodal cohesion, given that viewers’ identification, tracking and understanding of the basic narratological building blocks is driven by past and conventionalized spectatorial experiences. The cognitive dimension of multimodal cohesion has also been explored from a social semiotic standpoint by Ketola (2016), who focuses on the translation of illustrated technical texts and aims to gauge the impact of the degree of alignment between images and written texts – i.e. the extent to which they complement or reinforce each other – on translational decisions. She concludes that “different types of images affect the translation of verbal text in different ways” and that “readers themselves decide, whether consciously or unconsciously, to what extent they process images” (ibid.:77, 78). The insight that the semiotic space created by the written word and the illustrations between and around them varies in each case is consistent with the social semioticians’ dynamic conceptualization of signifying resources, whose meaning-making potential is realized differently in each communicative event. The most comprehensive application of social semiotic theory in translation studies is Pérez- González (2014b), who draws on Stöckl’s (2004) operationalization of multimodal theory as a networked system of choices. Here, audiovisual texts are seen as produced by mobilizing those visual and auditory modes that will best realize the creator’s communicative intentions. Among the range of visual and auditory modes identified by multimodal theory, image, language, sound and music are regarded as core modes, as each has more than one medial variant – respectively: static/dynamic images; speech, animated and still writing; sound effects and spectrograms; performed music and score/sheet music. The overarching networked system of modal choices is hierarchically organized, so that each core mode commands a set of associated sub-modes. Deploying a specific mode therefore opens up a new system of sub-modal choices. For example, using speech, one of the media variants of the language mode, prompts more delicate selections in terms of volume, intonation, accent, voice quality, rhythm, speech and pauses that filmmakers can use to enhance dramatic effect or characterization. Drawing on a range of case studies, Pérez-González (ibid.) illustrates the extent to which an enhanced awareness of the multimodal distribution of meaning in audiovisual texts can be crucial in informing translational decisions. Research themes One strand of research has focused on layout and typography. O’Sullivan (2013) draws on Flood’s (1993) study of Martin Luther’s translation of the Bible to demonstrate how the Wittenberg editions published during the Reformation used roman and gothic typefaces in strategic ways, with the former serving to associate “certain [negative] Biblical elements with the Church of Rome” and the latter being the preferred option to deliver “positively connoted words” from a Reformist perspective (O’Sullivan 2013:5-6). In the context of subtitling, layout and typography have been shown to enable new forms of interaction between the diegetic world and audiences in silent films and mainstream television broadcasts (O’Sullivan and Cornu 2018; O’Hagan and Sasamoto 2016); facilitate deaf viewers’ monitoring of interactional structures and turn-taking changes in films and television content (Neves 2005); foster a sense of shared affectivity among fansubbing networks and their audiences (Pérez-González 2007a, 2014b); and articulate new and creative forms of media content production (Fox 2016; Romero-Fresco 2018). A second strand of research examines multimedial spaces such as theatres, cinemas and museums, where a number of carrier media are used to stage multimodal texts. In these contexts, meaning-making activities may adopt different intersemiotic configurations, not all of which entail interlingual transfer. Managing and optimizing the use of multimodal resources is crucial to evaluating the performability of theatre plays, which demand “a dramaturgical capacity to work in several dimensions at once, incorporating visual, gestural, aural and linguistic signifiers” into the (translated) staged version (Hale and Upton 2000:2). Pérez-González (2007b:13-14) offers a multimodal analysis of the Spanish dubbed version of 12 Angry Men (1957) in which he illustrates how camera perspective and the focal length of lenses can be exploited to enhance dramatic tension at crucial junctures of the plot and to evoke emotional responses that influence the translation of the dialogue. Similarly, Maszerowska has drawn attention to the contribution that lighting makes to “the saturation of the audiences’ imaginations, complementing and carrying on the plot, reflecting the characters’ points of view and, at the same time, filling in the gaps between dialogues” (2012:83). The semiotic implications of local spatial constraints, object-text interdependence, overall aesthetic coherence, co-presence of material and virtual artefacts, as well as intertextual relations across individual exhibits are some of the issues that have attracted the attention of translation scholars who theorize museums as multimodal ensembles (Sell 2015; Neather 2012, 2017; Liao 2018). From an audience perspective, subtitling for the hard of hearing, respeaking and audio description have increasingly become foci of applied multimodal research seeking to accelerate the integration of individuals with sensory impairments in mainstream theatre, film and museum audiences (Neves 2018; Romero- Fresco 2011; Jiménez Hurtado and Soler Gallego 2015). The iconic-verbal link in printed texts is one of the most established areas of multimodal research in translation studies. Kress and van Leeuwen’s (1996) grammar of visual design has supported a range of studies on the translation of advertising material (Munday 2004; Millán- Varela 2004; Torresi 2004) and magazines (Chueasuai 2013), all of which conceptualize the partnership between verbal and visual structures as a means to identify particular interpretations of experience and forms of social interaction. The impact of the semiotic dominance of images has also been studied in relation to comics (Zanettin 2008), where humour is often encoded in the images, rather than the speech balloons. The possibility of altering or removing images when translating comics (Zanettin 2011) can be exploited, for example, to reconfigure the semiotic realization of humour across modes and to explore new forms of interaction between images, colour and written text (Kaindl 2004). The multimodal analysis of static images in comics can also yield a better understanding of the semiotic potential of less studied modes, including spatial orientation, body posture, gaze and distance (Borodo 2015). In illustrated literature, including graphic novels, images play a crucial role in restricting or amplifying certain aspects of the meaning conveyed by the written text (Alvstad 2008), to the extent of being constitutive of plot (Leighton and Surridge 2008). The modification or removal of illustrations during the translation of a book have been found to disrupt the text-image connection observed in the original, and affect the readers’ experience of the books in question (Oittinen et al. 2017). A further strand of research has demonstrated that the affordances of digital technology allow for experimental approaches to the translation of polyvocal texts that combine multiple layers of verbal and nonverbal material. Translations of such texts may adopt the form of interactive artefacts such as DVDs which combine written texts, photographs, sound recordings and ethnographic field material in ways that facilitate meaningful reading experiences through hyperlinked connections (Milsom 2008). In her attempt to translate French transgender memoirs into English, which lacks the grammatical resources available in the original language to articulate fluid, sometimes indeterminate, expressions of sex and gender identity, Rose (n.d.) exploits hypertext to display multiple translation options for key markers of transgender discourses, thus fostering a palimpsestuous pluralization of sexual and aesthetic experiences and signifying “the queerness of all identity and text” (ibid.). This “technologically-mediated sense of translation” and the intersemiotic reading experiences that it provides have also been explored by Lee (2013:241) in the context of art installations and cyber-poetry. The emergence of further technological affordances in the future, Lee argues, will pave the way for new forms of creative transposition whose study will have to be informed by a “multimodal perspective on translation and a translational perspective on multimodal expression” (ibid.:254). Alongside printed and digital multimodal ensembles, embodied multimodality has become an important research theme. Studies on paraverbal, prosodic and kinesic behaviour, such as Poyatos (1997), offer insights into the semiotics of the human body, understood as “the use of para-verbal signs – including but not limited to voice qualities, cadence, inflection, or rate of speech – and non-verbal signifiers – such as gestures or movements” (Pérez-González 2014a:122-123). Bosseaux (2015) demonstrates that the interplay between the qualities of characters’ voices (timbre, pitch) and their physical appearance often makes an important contribution to dramatic characterization, as is also the case with a range of sociolinguistic traits embodied in speech and sound, including accent and other markers of linguistic variation. Staging a translated theatre play or dubbing a film often involves decisions that may alter aspects of characterization and erase the sociolinguistic resonances of voice as a marker of identity, which may prove detrimental to the reception of the translated play or film (Queen 2004; Mingant 2010; Bosseaux 2018). In triadic medical encounters, the simultaneous use of various modes can help doctors to elicit patients’ informed consent (Bührig 2004), claim/retain the interactional floor and negotiate conflicting expectations. Understanding these sites of embodied multimodality, where spoken language is complemented by facial expressions, gestures or body positioning, is crucial for the development of effective interpreting strategies through a “co-ordinated manipulation of several semiotic resources” (Pasquandrea 2011:477). Analyses of embodied multimodal resources such as gaze (Davitti 2015), gesture, body position, proxemics and object manipulation have been conducted in pedagogical settings (Davitti 2012), and suggest that dialogue interpreting actively contributes to constructing “different participation frameworks throughout the encounter” within a specific ecology of action, understood as “the relationships between the participants and the surrounding environment” (Davitti and Paquandrea 2017:105). Similar studies have focused on interpreting in business meetings (Bao-Rozée 2016) and group work dialogue among deaf and hearing students (Slettebakk Berge 2018). Future directions Multimodality is still in its infancy and needs to build consensus around key aspects of its core theoretical framework. The feasibility of subsuming all signifying means under a finite range of modes and the formalization of the processes through which individual modes become integrated in a single unified ensemble remain subject to intense debate. Cognitive and neurolinguistic models of translation (Pérez-González 2014b:102-110) are developing the means to understand how translational decisions are influenced by the cognitive salience that individual modes gain against the multimodal whole. For instance, the combined use of a single set of visual resources with variations of the same soundtrack will influence viewers’ gazing trajectories when they try to process the multimodal ensemble (ibid.:105). In addition to multimodal transcriptions (Taylor 2016) and multimodal corpora (Soffritti 2018), eye- tracking is yielding the most effective insights into the reception of multimodal texts. Studies of users’ visual behaviour reveal how they process and comprehend a multimodal text and should, in due course, help translators understand how best to construct a similar coherent whole (Kruger 2012). As it continues to grapple with fast-paced media developments and capitalize on wide-ranging methodological advances, multimodality is delivering increasingly robust accounts of the generative power of semiotic resources and offering valuable insight into ever more complex textualities. Further reading Jewitt, C., J. Bezemer and K. O’Halloran (2016) Introducing Multimodality, London & New York: Routledge. Delivers an accessible overview of multimodality, covering its key concepts and the steps involved in designing a multimodal study. Kaindl, K. (2012) ‘Multimodality in Translation Studies’, in C. Millán-Varela and F. Bartrina (eds) The Routledge Handbook of Translation Studies, London & New York: Routledge, 268- 281. Includes a critique of a range of works by translation studies scholars that predate multimodality but share a number of concerns with social semiotics. Pérez-González, L. (2014a) ‘Multimodality in Translation and Interpreting Studies: Theoretical and methodological perspectives’, in S. Bermann and C. Porter (eds) A Companion to Translation Studies, Chichester: Wiley-Blackwell, 119-131. Explores a number of research themes at the interface of translation and semiotics, including multimodality. References Alvstad, C. (2008) ‘Illustrations and Ambiguity in Eighteen Illustrated Translations of Hans Christian Andersen’s “The Steadfast Tin Soldier”’, in R. Oittinen and K. Kaindl (eds) ‘The Verbal, the Visual, the Translator’, special issue of Meta 53(1): 90–103. Baker, M. (2005) ‘Linguistic Models and Methods in the Study of Translation’, in H. Kittel, A. P. Frank, N. Greiner, T. Hermans, W. Koller, J. Lambert and F. Paul (eds) Übersetzung, Translation, Traduction, Berlin & New York: Walter de Gruyter, 285-294. Baldry, A. and P. J. Thibault (2006) Multimodal Transcription and Text Analysis: A multimedia toolkit and coursebook, London: Equinox. Bao-Rozée, J. (2016) Using Multimodal Analysis to Investigate the Role of the Interpreter, Unpublished Doctoral Thesis, University of Stirling. Available online: https://pdfs.semanticscholar.org/1ea1/9c432bd2431a6eee1556305de6d2eba44c62.p df (accessed 3 August 2018). Borodo, M. (2015) ‘Multimodality, Translation and Comics’, Perspectives 23(1): 22-41. Bosseaux, C. (2015) Dubbing, Film and Performance: Uncanny encounters, Oxford: Peter Lang. Bosseaux, C. (2018) ‘Investigating Dubbing: Learning from the past, looking to the future’, in L. Pérez-González (ed.) The Routledge Handbook of Audiovisual Translation, London & New York, 48-63. Bührig, K. (2004) ‘On the Multimodality of Interpreting in Medical Briefings for Informed Consent. Using diagrams to impart knowledge’, in in E. Ventola, C. Charles and M. Kaltenbacher (eds) Perspectives on Multimodality, Amsterdam & Philadelphia: John Benjamins, 227-241. Chang, L.-W. (2015) Investigating Note-taking in Consecutive Interpreting. Using the concept of visual grammar, Unpublished Doctoral Thesis, University of Manchester. Available online: https://www.research.manchester.ac.uk/portal/files/54565055/full_text.pdf (accessed 3 August 2018). Chaume, F. (2004) Cine y traducción, Madrid: Cátedra. Chueasuai, P. (2011) Translation Shifts in the Love and Lust Section in the Thai Version of Cosmopolitan: A systemic functional perspective, Unpublished Doctoral Thesis, University of Manchester. Available online: https://www.research.manchester.ac.uk/portal/files/54503259/full_text.pdf (accessed 3 August 2018). Cronin, M. (2003) Translation and Globalization, London: Routledge. Davitti, E. (2012) Dialogue Interpreting as Intercultural Mediation: Integrating talk and gaze in the analysis of mediated parent-teacher meetings, Unpublished Doctoral Thesis, University of Manchester. Available online: https://www.research.manchester.ac.uk/portal/files/54519058/full_text.pdf (accessed 3 August 2018). Davitti, E (2015) ‘Gaze’, in F. Pöchhacker (ed.) The Routledge Encyclopedia of Interpreting Studies, London & New York: Routledge, 168. Davitti, E. and S. Pasquandrea (2017) ‘Embodied Participation: What multimodal analysis can tell us about interpreter-mediated encounters in pedagogical settings’, Journal of Pragmatics 107: 105-128. Fine, E. C. (1984) The Folklore Text, Bloomington: Indiana University Press. Flood, J. L. (1993) ‘Nationalistic Currents in Early German Typography’, The Library 15(2): 125- 141. Fox, W. (2016) ‘Integrated Titles: An improved viewing experience?’, in S. Hansen-Schirra and S. Grucza (eds) Eyetracking and Applied Linguistics, Berlin: Language Science Press, 5– 30. Gambier, Y. (2006) ‘Multimodality and Subtitling’, in in H. Gerzymisch-Arbogast and S. Nauert (eds) Challenges of Multidimensional Translation. Proceedings of the Marie Curie Euroconferences. MuTra: Challenges of Multidimensional Translation. Saarbrücken, 2-6 May 2005. Available online: http://euroconferences.info/proceedings/2006_Proceedings/2006_Gambier_Yves.pdf (accessed 3 August 2018). Gambier, Y. and S. Ramos Pinto (2016) ‘Introduction’, in Y. Gambier and S. Ramos Pinto (eds) ‘Audiovisual Translation: Theoretical and methodological challenges’, special issue of Target 28(2): 185–191. Gottlieb, H. (2005) ‘Multidimensional Translation: Semantics turned semiotics’, in H. Gerzymisch-Arbogast and S. Nauert (eds) Challenges of Multidimensional Translation. Proceedings of the Marie Curie Euroconferences. MuTra: Challenges of Multidimensional Translation. Saarbrücken, 2-6 May 2005. Available online: http://www.euroconferences.info/proceedings/2005_Proceedings/2005_Gottlieb_He nrik.pdf (accessed 3 August 2018). Gottlieb, H. (2017) ‘Semiotics and Translation’, in K. Malmkjaer (ed.) The Routledge Handbook of Translation Studies and Linguistics, London & New York: Routledge. Gumbrecht, H. U. (2004) Production of Presence: What meaning cannot convey, Stanford: Stanford University Press. Hale, T. and C.-A. Upton (2000) ‘Introduction’, in C.-A. Upton (ed.) Moving Target: Theatre translation and cultural relocation, Manchester: St Jerome: 1-13. Iedema, R. (2003) ‘Multimodality, Resemiotization: Extending the analysis of discourse as multi-semiotic practice’, Visual Communication 2(1): 29-57. Jakobson, R. (1959/2000) ‘On Linguistic Aspects of Translation’, in R. A. Brower (ed.) On Translation, Cambridge, MA: Harvard University Press, 232-239. Reprinted in L. Venuti (ed.) (2000) The Translation Studies Reader, 2nd edition, London & New York: Routledge, 113-119. Jewitt, C. (2017) ‘An Introduction to Multimodality’, in C. Jewitt (ed.) The Routledge Handbook of Multimodal Analysis, 2nd edition, London & New York: Routledge, 15-30. Jewitt, C., J. Bezemer and K. O’Halloran (2016) Introducing Multimodality, London & New York: Routledge. Jiménez Hurtado, C. and S. Soler Gallego (2015) ‘Museum Accessibility through Translation: A corpus study of pictorial audio description’, in Díaz Cintas, J. and J. Neves (eds) (2015) Audiovisual Translation: Taking stock, Newcastle upon Tyne: Cambridge Scholars Publishing, 277-297. Kaindl, K. (2004) ‘Multimodality in the Translation of Humour in Comics’, in E. Ventola, C. Charles and M. Kaltenbacher (eds) Perspectives on Multimodality, Amsterdam & Philadelphia: John Benjamins, 173–192. Kaindl, K. (2012) ‘Multimodality in Translation Studies’, in C. Millán-Varela and F. Bartrina (eds) The Routledge Handbook of Translation Studies, London & New York: Routledge, 268-281. Ketola, A. (2016) ‘Towards a Multimodally Oriented Theory of Translation: A cognitive framework for the translation of illustrated technical texts’, Translation Studies 9(1): 67- 81. Kress, G. (2010) Multimodality: A social semiotic approach to contemporary communication, London & New York: Routledge. Kress, G. (2017) ‘What is Mode?’, in C. Jewitt (ed.) The Routledge Handbook of Multimodal Analysis, 2nd edition, London & New York: Routledge, 60-75. Kress, G. and T. van Leeuwen (2006) Reading Images: The grammar of visual design, 2nd edition, London & New York: Routledge. Kress, G. and J. Ogborn (1998) ‘Modes of Representation and Local Epistemologies: The presentation of science in education’, Subjectivity in the School Curriculum (Working Paper 2, Institute of Education). London: University of London. Kress, G. and T. van Leeuwen (2001) Multimodal Discourse: The modes and media of contemporary communication, London: Arnold. Lee, T.-K. (2013) ‘Performing Multimodality: Literary translation, intersemioticity and technology’, Perspectives 21:2, 241-256. Leighton, M. E. and L. Surridge (2008) ‘The Plot Thickens: Towards a narratological analysis of illustrated serial fiction in the 1860s’, Victorian Studies 51(1), 65–101. Lemke, J. (2002) ‘Travels in Hypermodality’, Visual Communication 1(3): 299–325. Liao, M.-H. (2018) ‘Museums and Creative Industries: The contribution of translation studies, JoSTrans: The Journal of Specialised Translation 29: 45-62. Available online: https://www.jostrans.org/issue29/art_liao.pdf (accessed 3 August 2018). Littau, Karen (1997) ‘Translation in the Age of Postmodern Production: From text to intertext to hypertext’, Forum for Modern Language Studies 33(1): 81-96. Littau, K. (2016a) ‘Translation and the Materialities of Communication’, Translation Studies 9(1): 82-96. Littau, K. (2016b) ‘Translation’s Histories and Digital Futures’, International Journal of Communication 10: 907-928. Maszerowska, A. (2012) ‘Casting the Light on Cinema: How luminance and contrast patterns create meaning’, in R. Agost, P. Orero and E. di Giovanni (eds) Multidisciplinary in Audiovisual Translation, MonTI 4: 65-85. Millán-Varela, C. (2004) ‘Exploring Advertising in a Global Context: Food for Thought’, in B. Adab and C. Valdés (eds) ‘Key Debates in the Translation of Advertising Material’, special issue of The Translator 10(2): 245-267. Milsom, A.-M. (2008) Picturing Voices,Writing Thickness: A multimodal approach to translating the Afro-Cuban tales of Lydia Cabrera, Unpublished PhD Thesis, Middlesex Universit. Available online: http://eprints.mdx.ac.uk/10750/ (accessed 3 August 2018). Mingant, N. (2010) ‘Tarantino’s “Inglorious Basterds”: A blueprint for dubbing translators?’, Meta 55(4): 712-731. Munday, J. (2004) ‘Advertising: Some challenges to translation theory’, in B. Adab and C. Valdés (eds) ‘Key Debates in the Translation of Advertising Material’, special issue of The Translator 10(2): 199-219. Neather, R. (2012) ‘Intertextuality, Translation and the Semiotics of Museum Presentation: The case of bilingual texts in Chinese museums’, Semiotica 192: 197-218. Neather, R. (2017) ‘Museums, Material Culture and Cultural Representations’, in S.-A. Harding and O. Carbonell Cortés (eds) The Routledge Handbook of Translation and Culture, London & New York: Routledge, 361-378. Neves, J. (2005) Audiovisual Translation: Subtitling for the deaf and hard-of-hearing, Unpublished Doctoral Thesis, University of Roehampton, UK. Available online: http://roehampton.openrepository.com/roehampton/bitstream/10142/12580/1/nev es%2520audiovisual.pdf (accessed 3 August 2018). Neves, J. (2018) ‘Subtitling for Deaf and Hard of Hearing Audiences: Moving forward’, in L. Pérez-González (ed.) The Routledge Handbook of Audiovisual Translation, London & New York, 82-95. O’Hagan, M. and R. Sasamoto (2016) ‘Crazy Japanese Subtitles? Shedding light on the impact of impact captions with a focus on research methodology’, in S. Hansen-Schirra and S. Grucza (eds) Eyetracking and Applied Linguistics, Berlin: Language Science Press, 31–58. Oittinen, R., A. Ketol and M. Garavini (2017) Translating Picturebooks. Revoicing the verbal, the visual and the aural for a child audience, London & New York: Routledge. O’Sullivan, C. (2013) ‘Introduction: Multimodality as challenge and resource for translation’, in C. O’Sullivan (ed.) ‘The Translation of Multimodal Texts’, special issue of JoSTrans: The Journal of Specialised Translation 20: 2-14. Available online: http://www.jostrans.org/issue20/art_osullivan.pdf (accessed 3 August 2018). O’Sullivan, C. and J.-F. Cornu (2018) The Translation of Films 1900-1950, Oxford: Oxford University Press. Pasquandrea, S. (2011) ‘Managing Multiple Actions through Multimodality: Doctors' involvement in interpreter mediated interactions’, Language in Society 40(4): 455-481. Pérez-González, L. (2007a) ‘Intervention in New Amateur Subtitling Cultures: A multimodal account’, Linguistica Antverpiensia 6: 67-80. Pérez-González, L. (2007b) ‘Appraising Dubbed Conversation. Systemic functional insights into the construal of naturalness in translated film dialogue’, The Translator 13(1): 1-38. Pérez-González, L. (2014a) ‘Multimodality in Translation and Interpreting Studies: Theoretical and methodological perspectives’, in S. Bermann and C. Porter (eds) A Companion to Translation Studies, Chichester: Wiley-Blackwell, 119-131. Pérez-González, L. (2014a) Audiovisual Translation: Theories, Methods, Issues, London & New York, Routledge. Poyatos, F. (1007) Non-verbal Communication and Translation, Amsterdam & Philadelphia: John Benjamins. Queen, R. (2004) ‘‘Du hast jar keeneAhnung’: African American English dubbed into German’, Journal of Sociolinguistics 8(4): 515-537. Remael, A. (2001) ‘Some Thoughts on the Study of Multimodal and Multimedia Translation’, in Y. Gambier and H. Gottlieb (eds) (Multi) Media Translation: Concepts, practices and research, Amsterdam & Philadelphia: John Benjamins, 13-22. Remael, A. and N. Reviers (2018) ‘Multimodality and Audiovisual Translation. Cohesion in accessible films’, in L. Pérez-González (ed.) The Routledge Handbook of Audiovisual Translation, London & New York, 260-280. Romero-Fresco, P. (2011) Subtitling Through Speech Recognition: Respeaking, Manchester: St Jerome. Romero Fresco, P. (2018) ‘Accessible Filmmaking: Translation and accessibility from production’, in L. Pérez-González (ed.) The Routledge Handbook of Audiovisual Translation, London & New York, 498-515. Rose, E. (n.d.) Translating Herculine. Memoirs of a French ‘hermaphrodite’ in English. Available online: https://translatingherculine.wordpress.com/ (accessed 3 August 2018). Sell, C. A. (2015) Translation in Japanese Museums: A study of multimodal linguistic landscapes, Unpublished Doctoral Thesis, Monash University. Slettebakk Berge, S. (2018) ‘How sign language interpreters use multimodal actions to coordinate turn-taking in group work between deaf and hearing upper secondary school students’, Interpreting 20(1): 96-125. Stecconi, U. (2009) ‘Semiotics’, in M. Baker & G. Saldanha (eds) Routledge Encyclopedia of Translation Studies, 2nd edition, London & New York: Routledge, 260-263. Stöckl, Hartmut (2004) ‘In Between Modes: Language and Image in Printed Media’, in E. Ventola, C. Charles and M. Kaltenbacher (eds) Perspectives on Multimodality, Amsterdam & Philadelphia: John Benjamins, 9-30. Taylor, C. (2014) ‘Multimodality and Audiovisual Translation’, in Y. Gambier and L. van Doorslaer (eds) Handbook of Translation Studies, vol. 4, Amsterdam & Philadelphia: John Benjamins, 98-104. Taylor, C. (2016) ‘The Multimodal Approach in Audiovisual Translation’, Target 28(2): 222- 236. Torresi, I. (2004) ‘Women, Water and Cleaning Agents: What Advertisements Reveal about the Cultural Stereotype of Cleanliness’, in B. Adab and C. Valdés (eds) ‘Key Debates in the Translation of Advertising Material’, special issue of The Translator 10(2): 269-289. van Leeuwen, T. (2005) Introducing Social Semiotics, London & New York: Routledge. Zanettin, F. (ed.) (2008) Comics in Translation, Manchester: St. Jerome. Zanettin, F. (2011) ‘Comics’, in M. Baker and G. Saldanha (eds) The Routledge Encyclopedia of Translation Studies, 2nd edition, London & New York: Routledge, 37–40.

References (77)

  1. Alvstad, C. (2008) 'Illustrations and Ambiguity in Eighteen Illustrated Translations of Hans Christian Andersen's "The Steadfast Tin Soldier"', in R. Oittinen and K. Kaindl (eds) 'The Verbal, the Visual, the Translator', special issue of Meta 53(1): 90-103.
  2. Baker, M. (2005) 'Linguistic Models and Methods in the Study of Translation', in H. Kittel, A. P. Frank, N. Greiner, T. Hermans, W. Koller, J. Lambert and F. Paul (eds) Übersetzung, Translation, Traduction, Berlin & New York: Walter de Gruyter, 285-294.
  3. Baldry, A. and P. J. Thibault (2006) Multimodal Transcription and Text Analysis: A multimedia toolkit and coursebook, London: Equinox.
  4. Bao-Rozée, J. (2016) Using Multimodal Analysis to Investigate the Role of the Interpreter, Unpublished Doctoral Thesis, University of Stirling. Available online: https://pdfs.semanticscholar.org/1ea1/9c432bd2431a6eee1556305de6d2eba44c62.p df (accessed 3 August 2018).
  5. Borodo, M. (2015) 'Multimodality, Translation and Comics', Perspectives 23(1): 22-41.
  6. Bosseaux, C. (2015) Dubbing, Film and Performance: Uncanny encounters, Oxford: Peter Lang.
  7. Bosseaux, C. (2018) 'Investigating Dubbing: Learning from the past, looking to the future', in L. Pérez-González (ed.) The Routledge Handbook of Audiovisual Translation, London & New York, 48-63.
  8. Bührig, K. (2004) 'On the Multimodality of Interpreting in Medical Briefings for Informed Consent. Using diagrams to impart knowledge', in in E. Ventola, C. Charles and M. Kaltenbacher (eds) Perspectives on Multimodality, Amsterdam & Philadelphia: John Benjamins, 227-241.
  9. Chang, L.-W. (2015) Investigating Note-taking in Consecutive Interpreting. Using the concept of visual grammar, Unpublished Doctoral Thesis, University of Manchester. Available online: https://www.research.manchester.ac.uk/portal/files/54565055/full_text.pdf (accessed 3 August 2018).
  10. Chaume, F. (2004) Cine y traducción, Madrid: Cátedra.
  11. Chueasuai, P. (2011) Translation Shifts in the Love and Lust Section in the Thai Version of Cosmopolitan: A systemic functional perspective, Unpublished Doctoral Thesis, University of Manchester. Available online: https://www.research.manchester.ac.uk/portal/files/54503259/full_text.pdf (accessed 3 August 2018).
  12. Cronin, M. (2003) Translation and Globalization, London: Routledge.
  13. Davitti, E. (2012) Dialogue Interpreting as Intercultural Mediation: Integrating talk and gaze in the analysis of mediated parent-teacher meetings, Unpublished Doctoral Thesis, University of Manchester. Available online: https://www.research.manchester.ac.uk/portal/files/54519058/full_text.pdf (accessed 3 August 2018).
  14. Davitti, E (2015) 'Gaze', in F. Pöchhacker (ed.) The Routledge Encyclopedia of Interpreting Studies, London & New York: Routledge, 168.
  15. Davitti, E. and S. Pasquandrea (2017) 'Embodied Participation: What multimodal analysis can tell us about interpreter-mediated encounters in pedagogical settings', Journal of Pragmatics 107: 105-128.
  16. Fine, E. C. (1984) The Folklore Text, Bloomington: Indiana University Press.
  17. Flood, J. L. (1993) 'Nationalistic Currents in Early German Typography', The Library 15(2): 125- 141.
  18. Fox, W. (2016) 'Integrated Titles: An improved viewing experience?', in S. Hansen-Schirra and S. Grucza (eds) Eyetracking and Applied Linguistics, Berlin: Language Science Press, 5- 30.
  19. Gambier, Y. (2006) 'Multimodality and Subtitling', in in H. Gerzymisch-Arbogast and S. Nauert (eds) Challenges of Multidimensional Translation. Proceedings of the Marie Curie Euroconferences. MuTra: Challenges of Multidimensional Translation. Saarbrücken, 2-6 May 2005. Available online: http://euroconferences.info/proceedings/2006_Proceedings/2006_Gambier_Yves.pdf (accessed 3 August 2018).
  20. Gambier, Y. and S. Ramos Pinto (2016) 'Introduction', in Y. Gambier and S. Ramos Pinto (eds) 'Audiovisual Translation: Theoretical and methodological challenges', special issue of Target 28(2): 185-191.
  21. Gottlieb, H. (2005) 'Multidimensional Translation: Semantics turned semiotics', in H. Gerzymisch-Arbogast and S. Nauert (eds) Challenges of Multidimensional Translation. Proceedings of the Marie Curie Euroconferences. MuTra: Challenges of Multidimensional Translation. Saarbrücken, 2-6 May 2005. Available online: http://www.euroconferences.info/proceedings/2005_Proceedings/2005_Gottlieb_He nrik.pdf (accessed 3 August 2018).
  22. Gottlieb, H. (2017) 'Semiotics and Translation', in K. Malmkjaer (ed.) The Routledge Handbook of Translation Studies and Linguistics, London & New York: Routledge.
  23. Gumbrecht, H. U. (2004) Production of Presence: What meaning cannot convey, Stanford: Stanford University Press.
  24. Hale, T. and C.-A. Upton (2000) 'Introduction', in C.-A. Upton (ed.) Moving Target: Theatre translation and cultural relocation, Manchester: St Jerome: 1-13.
  25. Iedema, R. (2003) 'Multimodality, Resemiotization: Extending the analysis of discourse as multi-semiotic practice', Visual Communication 2(1): 29-57.
  26. Jakobson, R. (1959/2000) 'On Linguistic Aspects of Translation', in R. A. Brower (ed.) On Translation, Cambridge, MA: Harvard University Press, 232-239. Reprinted in L. Venuti (ed.) (2000) The Translation Studies Reader, 2 nd edition, London & New York: Routledge, 113-119.
  27. Jewitt, C. (2017) 'An Introduction to Multimodality', in C. Jewitt (ed.) The Routledge Handbook of Multimodal Analysis, 2 nd edition, London & New York: Routledge, 15-30.
  28. Jewitt, C., J. Bezemer and K. O'Halloran (2016) Introducing Multimodality, London & New York: Routledge.
  29. Jiménez Hurtado, C. and S. Soler Gallego (2015) 'Museum Accessibility through Translation: A corpus study of pictorial audio description', in Díaz Cintas, J. and J. Neves (eds) (2015) Audiovisual Translation: Taking stock, Newcastle upon Tyne: Cambridge Scholars Publishing, 277-297.
  30. Kaindl, K. (2004) 'Multimodality in the Translation of Humour in Comics', in E. Ventola, C. Charles and M. Kaltenbacher (eds) Perspectives on Multimodality, Amsterdam & Philadelphia: John Benjamins, 173-192.
  31. Kaindl, K. (2012) 'Multimodality in Translation Studies', in C. Millán-Varela and F. Bartrina (eds) The Routledge Handbook of Translation Studies, London & New York: Routledge, 268-281.
  32. Ketola, A. (2016) 'Towards a Multimodally Oriented Theory of Translation: A cognitive framework for the translation of illustrated technical texts', Translation Studies 9(1): 67- 81.
  33. Kress, G. (2010) Multimodality: A social semiotic approach to contemporary communication, London & New York: Routledge.
  34. Kress, G. (2017) 'What is Mode?', in C. Jewitt (ed.) The Routledge Handbook of Multimodal Analysis, 2 nd edition, London & New York: Routledge, 60-75.
  35. Kress, G. and T. van Leeuwen (2006) Reading Images: The grammar of visual design, 2 nd edition, London & New York: Routledge.
  36. Kress, G. and J. Ogborn (1998) 'Modes of Representation and Local Epistemologies: The presentation of science in education', Subjectivity in the School Curriculum (Working Paper 2, Institute of Education). London: University of London.
  37. Kress, G. and T. van Leeuwen (2001) Multimodal Discourse: The modes and media of contemporary communication, London: Arnold.
  38. Lee, T.-K. (2013) 'Performing Multimodality: Literary translation, intersemioticity and technology', Perspectives 21:2, 241-256.
  39. Leighton, M. E. and L. Surridge (2008) 'The Plot Thickens: Towards a narratological analysis of illustrated serial fiction in the 1860s', Victorian Studies 51(1), 65-101.
  40. Lemke, J. (2002) 'Travels in Hypermodality', Visual Communication 1(3): 299-325.
  41. Liao, M.-H. (2018) 'Museums and Creative Industries: The contribution of translation studies, JoSTrans: The Journal of Specialised Translation 29: 45-62. Available online: https://www.jostrans.org/issue29/art_liao.pdf (accessed 3 August 2018).
  42. Littau, Karen (1997) 'Translation in the Age of Postmodern Production: From text to intertext to hypertext', Forum for Modern Language Studies 33(1): 81-96.
  43. Littau, K. (2016a) 'Translation and the Materialities of Communication', Translation Studies 9(1): 82-96.
  44. Littau, K. (2016b) 'Translation's Histories and Digital Futures', International Journal of Communication 10: 907-928.
  45. Maszerowska, A. (2012) 'Casting the Light on Cinema: How luminance and contrast patterns create meaning', in R. Agost, P. Orero and E. di Giovanni (eds) Multidisciplinary in Audiovisual Translation, MonTI 4: 65-85.
  46. Millán-Varela, C. (2004) 'Exploring Advertising in a Global Context: Food for Thought', in B. Adab and C. Valdés (eds) 'Key Debates in the Translation of Advertising Material', special issue of The Translator 10(2): 245-267.
  47. Milsom, A.-M. (2008) Picturing Voices,Writing Thickness: A multimodal approach to translating the Afro-Cuban tales of Lydia Cabrera, Unpublished PhD Thesis, Middlesex Universit. Available online: http://eprints.mdx.ac.uk/10750/ (accessed 3 August 2018).
  48. Mingant, N. (2010) 'Tarantino's "Inglorious Basterds": A blueprint for dubbing translators?', Meta 55(4): 712-731.
  49. Munday, J. (2004) 'Advertising: Some challenges to translation theory', in B. Adab and C. Valdés (eds) 'Key Debates in the Translation of Advertising Material', special issue of The Translator 10(2): 199-219.
  50. Neather, R. (2012) 'Intertextuality, Translation and the Semiotics of Museum Presentation: The case of bilingual texts in Chinese museums', Semiotica 192: 197-218.
  51. Neather, R. (2017) 'Museums, Material Culture and Cultural Representations', in S.-A. Harding and O. Carbonell Cortés (eds) The Routledge Handbook of Translation and Culture, London & New York: Routledge, 361-378.
  52. Neves, J. (2005) Audiovisual Translation: Subtitling for the deaf and hard-of-hearing, Unpublished Doctoral Thesis, University of Roehampton, UK. Available online: http://roehampton.openrepository.com/roehampton/bitstream/10142/12580/1/nev es%2520audiovisual.pdf (accessed 3 August 2018).
  53. Neves, J. (2018) 'Subtitling for Deaf and Hard of Hearing Audiences: Moving forward', in L. Pérez-González (ed.) The Routledge Handbook of Audiovisual Translation, London & New York, 82-95.
  54. O'Hagan, M. and R. Sasamoto (2016) 'Crazy Japanese Subtitles? Shedding light on the impact of impact captions with a focus on research methodology', in S. Hansen-Schirra and S. Grucza (eds) Eyetracking and Applied Linguistics, Berlin: Language Science Press, 31-58.
  55. Oittinen, R., A. Ketol and M. Garavini (2017) Translating Picturebooks. Revoicing the verbal, the visual and the aural for a child audience, London & New York: Routledge.
  56. O'Sullivan, C. (2013) 'Introduction: Multimodality as challenge and resource for translation', in C. O'Sullivan (ed.) 'The Translation of Multimodal Texts', special issue of JoSTrans: The Journal of Specialised Translation 20: 2-14. Available online: http://www.jostrans.org/issue20/art_osullivan.pdf (accessed 3 August 2018).
  57. O'Sullivan, C. and J.-F. Cornu (2018) The Translation of Films 1900-1950, Oxford: Oxford University Press.
  58. Pasquandrea, S. (2011) 'Managing Multiple Actions through Multimodality: Doctors' involvement in interpreter mediated interactions', Language in Society 40(4): 455-481.
  59. Pérez-González, L. (2007a) 'Intervention in New Amateur Subtitling Cultures: A multimodal account', Linguistica Antverpiensia 6: 67-80.
  60. Pérez-González, L. (2007b) 'Appraising Dubbed Conversation. Systemic functional insights into the construal of naturalness in translated film dialogue', The Translator 13(1): 1-38.
  61. Pérez-González, L. (2014a) 'Multimodality in Translation and Interpreting Studies: Theoretical and methodological perspectives', in S. Bermann and C. Porter (eds) A Companion to Translation Studies, Chichester: Wiley-Blackwell, 119-131.
  62. Pérez-González, L. (2014a) Audiovisual Translation: Theories, Methods, Issues, London & New York, Routledge.
  63. Poyatos, F. (1007) Non-verbal Communication and Translation, Amsterdam & Philadelphia: John Benjamins.
  64. Queen, R. (2004) ''Du hast jar keeneAhnung': African American English dubbed into German', Journal of Sociolinguistics 8(4): 515-537.
  65. Remael, A. (2001) 'Some Thoughts on the Study of Multimodal and Multimedia Translation', in Y. Gambier and H. Gottlieb (eds) (Multi) Media Translation: Concepts, practices and research, Amsterdam & Philadelphia: John Benjamins, 13-22.
  66. Remael, A. and N. Reviers (2018) 'Multimodality and Audiovisual Translation. Cohesion in accessible films', in L. Pérez-González (ed.) The Routledge Handbook of Audiovisual Translation, London & New York, 260-280.
  67. Romero-Fresco, P. (2011) Subtitling Through Speech Recognition: Respeaking, Manchester: St Jerome. Romero Fresco, P. (2018) 'Accessible Filmmaking: Translation and accessibility from production', in L. Pérez-González (ed.) The Routledge Handbook of Audiovisual Translation, London & New York, 498-515.
  68. Rose, E. (n.d.) Translating Herculine. Memoirs of a French 'hermaphrodite' in English. Available online: https://translatingherculine.wordpress.com/ (accessed 3 August 2018).
  69. Sell, C. A. (2015) Translation in Japanese Museums: A study of multimodal linguistic landscapes, Unpublished Doctoral Thesis, Monash University.
  70. Slettebakk Berge, S. (2018) 'How sign language interpreters use multimodal actions to coordinate turn-taking in group work between deaf and hearing upper secondary school students', Interpreting 20(1): 96-125.
  71. Stecconi, U. (2009) 'Semiotics', in M. Baker & G. Saldanha (eds) Routledge Encyclopedia of Translation Studies, 2 nd edition, London & New York: Routledge, 260-263.
  72. Stöckl, Hartmut (2004) 'In Between Modes: Language and Image in Printed Media', in E. Ventola, C. Charles and M. Kaltenbacher (eds) Perspectives on Multimodality, Amsterdam & Philadelphia: John Benjamins, 9-30.
  73. Taylor, C. (2014) 'Multimodality and Audiovisual Translation', in Y. Gambier and L. van Doorslaer (eds) Handbook of Translation Studies, vol. 4, Amsterdam & Philadelphia: John Benjamins, 98-104.
  74. Taylor, C. (2016) 'The Multimodal Approach in Audiovisual Translation', Target 28(2): 222- 236.
  75. Torresi, I. (2004) 'Women, Water and Cleaning Agents: What Advertisements Reveal about the Cultural Stereotype of Cleanliness', in B. Adab and C. Valdés (eds) 'Key Debates in the Translation of Advertising Material', special issue of The Translator 10(2): 269-289.
  76. van Leeuwen, T. (2005) Introducing Social Semiotics, London & New York: Routledge. Zanettin, F. (ed.) (2008) Comics in Translation, Manchester: St. Jerome.
  77. Zanettin, F. (2011) 'Comics', in M. Baker and G. Saldanha (eds) The Routledge Encyclopedia of Translation Studies, 2 nd edition, London & New York: Routledge, 37-40.

FAQs

sparkles

AI

What explains the relationship between semiotics and multimodality in translation studies?add

The research reveals that while semiotics broadly encompasses meaning-making beyond verbal language, multimodality is defined specifically in terms of multi-channel texts and is influenced by semiotic theory.

How does the structuralist approach methodically categorize translation types?add

The study establishes a taxonomy of 34 translation types focusing on intersemiotic and intrasemiotic transfers, indicating a systematic framework for understanding translational modalities.

What impact does social semiotics have on understanding multimodal texts?add

Social semiotics emphasizes the interrelationships of modes evoked through social practice, elucidating how contextually shaped resources culminate in diverse meaning-making processes.

How does layout influence the interaction between multimodal texts and audiences?add

Research finds that layout and typography in subtitling facilitate audience interactions and enhance comprehension, exemplified in studies of silent films and television broadcasts.

What role does technology play in evolving multimodal translation practices?add

The findings highlight that digital technology fosters innovative translation methodologies, illustrated by projects integrating hypertext and multimedia to enrich reader experiences in polyvocal texts.

About the author
Luis Perez-Gonzalez
University of Agder, Faculty Member

I am Professor of Translation Studies at the Department of Translation and Foreign Languages | www.luisperezgonzalez.org My research interests fall under four main areas: :: The Contestation of Key Political and Scientific Concepts in the Digital Sphere My work in the AHRC-funded project Genealogies of Knowledge: The Evolution and Contestation of Concepts across Time and Space marries my interest in horizontal or deliberative politics with corpus-based translation studies. Drawing on a corpus of Internet discourse in English produced by alternative media and news outlets, I lead a strand of the project exploring how civil society organisations are currently challenging and redefining established meanings and interpretations of key concepts relating to the body politic and to scientific, expert discourse – as radical-democratic projects supersede traditional models of democracy and rationality and their capacity to confer representative authority and canonise knowledge. This 4-year corpus-based project should contribute to raising public awareness of how (re)translation and networked technologies have brought and continue to bring changes to our understanding of key cultural concepts pertaining to these two sets of interconnected concepts. :: Engaged Subtitling and Citizen Media in the Digital Culture :: I am interested in the political dimension of amateur subtitling and its contribution to the production and circulation of citizen media content in the digital culture. Informed by cultural studies, affect theory and narrative theory, this strand of my work examines how activist subtitling agencies at the interface between the actual and the digital are able to escape confinement in essentialist categories of identity politics such as social class, race or gender. The premise underpinning this work is that, in post-industrial societies, self-mediation practices such as activist subtitling mirror the ongoing shift from established representation models of democracy towards deliberative forms of governance with the capacity to mobilise fluid radical constituencies and foster the formation of inter-subjectivity through affective flows. :: Fandom and Audiovisual Translation :: Since the mid-2000s, amateur subtitling networks have become influential fandom-driven agencies of translation. This participatory (sub)cultural communities, typically referred to as fansubbing groups, have engendered significant tensions over the ownership of consumer-generated media content and threatened the economic and industrial foundations of the audiovisual industry. These ‘prosumption’ communities negotiate and hybridise two conflicting logics of the cultural marketplace: the drive to accrue cultural and symbolic capital in the form of recognition and reputation; and the effort, in some cases, to commoditise the content they produce through their immaterial labour. A number of my publications have sought to gauge the extent to which fansubbing is subverting what had so far been regarded as widely accepted standards of professional mediation/intervention, and the capacity of these alternative mediation conventions to resist and challenge the standardising and domesticating effects of mainstream subtitling practices. My work has also explored the potential of such novel practices to leak out of the boundaries of non-mainstream genres into the domain of mainstream commercial content such as films. By addressing the impact of media convergence, co-creation and immaterial labour on the production and consumption of media content, this body of work has also contributed to re-theorising the place of translation in the globalised media landscape. :: Discourse, Translation and the Law :: My doctoral thesis focused on the discursive manifestations of attempted deception in 999 hoax calls, and represented one of the earliest forensic studies of spoken interaction. This work led to the publication of a monograph and a number of papers on the dynamic realisation of genre and discourse modelling in conversation. After completing my thesis, I moved to explore other issues at the interface between language and the law. These included the homogenising effect that globalization is having on courtroom proceedings across legal cultures, as adversarial practices such as trial by jury are adopted in non-adversarial, civil law systems. My publications in this area examine the impact that such changes have had on the performance of translators and courtroom interpreters – given that legal players are often unaware of the pressures that imported Anglo-Saxon practices have on the performance of translators and interpreters trained to work in a non-adversarial environment. More widely, my interest in the study of the interface between language and the law has led me to conduct research into various aspects of legal translator training, including the role that translation technologies play in that process.

Papers
37
Followers
1,920
View all papers from Luis Perez-Gonzalezarrow_forward

Related papers

(2014) Multimodality in Translation and Interpreting Studies
Luis Perez-Gonzalez

Bermann, Sandra and Catherine Porter (eds) A Companion to Translation Studies

""Recent scholarly developments both within and outside translation studies attest to the growing perception among researchers that, in the pursuit of understanding processes of interlingual and intercultural transfer and mediation, analysing language is not enough. The centrality of (mostly written) language in translation studies research is hardly surprising, with linguistics being widely regarded as the discipline which has most informed the study of translation and interpreting since they emerged as a field of academic inquiry in the middle of the twentieth century (Baker 1996). The emphasis of early research on short, often decontextualised stretches of text (Baker and Pérez-González 2010) resulted in an excision of language – understood as text or discourse – from its context that has become the object of growing scrutiny by translation and interpreting scholars over the last two decades. More importantly, this displacement of language from context has favoured the analysis of language and its instantiation in discourse separately from other forms of meaning-making resources. This paper sets out to examine recent theoretical developments seeking to redress the displacement of language from other kinds of meaning-making resources and their impact on the theorisation of translation and interpreting. The starting premise of this chapter is that academic interest in non-verbal semiotic resources and their role in processes of interlingual and intercultural transfer is unevenly spread across different scholarly strands within the discipline. As far as the breadth of this research agenda is concerned, images appear to be the only non-linguistic meaning-making signs showing an increasingly recognised potential to inform research in translation studies. Dialogue interpreting, audiovisual and drama translation, to give but a few examples, still lack the theoretical and methodological concepts and tools to systematically analyse semiotic resources such as the gestures and facial expressions accentuating face-to-face conversation; the choices of fonts, colours and patterns of textual-visual interaction in printed advertisements; or the use of music and lighting in the staging of a drama production, respectively. This paper surveys ongoing research on how different semiotic resources shape translational behaviour in different communicative contexts, including but not limited to the interaction between speech and image in printed media and motion pictures; the modelling of composite semiotic systems, such as movement, gestures and gaze; the representation of identities and ideologies using non-verbal resources; and the conceptualisation of space, interpersonal perspective and salience in a range of settings, such as museums. The paper then moves on to explore how insights imported from multimodal theory, as developed in the field of systemic functional linguistics and social semiotics, may help translation and interpreting scholars to gain new insights into old data. Key notions like ‘multimodal’, ‘multimedial’, ‘mode’, ‘modality’, ‘sub-mode’ and ‘medial realisation’ are introduced and explored in some detail. The contribution of multimodal insights to research in translation studies are also gauged in relation to new data and their contexts of production, as illustrated by the way in which different modes function semiotically when combined in the modern discourse worlds afforded by the computer and the Internet. In these ‘new media’, information is proliferating in forms which push our methods of sharing it effectively; the shape of discourse communities using, assessing and circulating translations is changing with the changing shape of texts; ideological currents engaging with the interpretation of translations are flowing beyond existing linguistic means of analysis and critique; and new amateur phenomena, mainly fandom and political activism, are increasingly appropriating translation and interpreting as a means to effect social change. The final section (before the conclusion) considers the methodological implications of multimodal research in translation and interpreting studies, with particular emphasis on new tools like multimodal transcriptions and multimodal corpora. ""

View PDFchevron_right
Multimodal, polysemiotic, multisemiotic: a necessary demarcation for semiotics and translation studies [target article]
Evangelos Kourdis

Social Semiotics 35(5): 718-732, 2025

Very often in semiotics, but also in translation studies, a conceptual confusion is found between the terms multimodal and polysemiotic/multisemiotic. Certainly, both terms connote the presence of several semiotic systems in a cultural text, but these semiotic systems, although different (e.g. image, color, lay out, typography, graphic design), may belong to the same semiotic mode (e.g. visual). This confusion has resulted in many scholars using one term instead of the other, with the term multimodality gaining ground in their preferences. It is also certain that the occasional conceptual proposals by the French semiotic school (interaction, fusion, jonction, synergie, syncrétisme, intersemioticité, transmutation, transposition, etc.) further complicate the issue of whether a semiotic phenomenon takes place between semiotic systems or modes. This issue is of particular concern to translation scholars who argue that if translation takes place in multimodal environments, why not use the term multimodal translation instead of intersemiotic translation, transmutation, or transposition. The paper aims to present a reflection on the two terms of semiotics, multimodal and polysemiotic/multisemiotic that are also widely used in translation studies and to define their boundaries so that they can be discernible even to scholars of other research fields with which semiotics comes into contact.

View PDFchevron_right
Response to the responses to “Multimodal, polysemiotic, multisemiotic: a necessary demarcation for semiotics and translation studies”
Evangelos Kourdis

Social Semiotics 35(5): 746-751 , 2025

I certainly cannot claim that it is easy to provide a direct answer to the questions I raised in my article, as different traditions and theories influence the content of the terms used in semiotics, translation studies and multimodal studies. I hope, however, that it has been a stimulus for reflection on the historical trajectory and the semantic content of some terms and their use today in different or not-so-different fields of research. I would like to thank the four colleagues for their views and I hope that our collective reflection will contribute to a fuller semiotic consideration of translation phenomena.

View PDFchevron_right
Multimodality, meaning-making, and the issue of “text”. Special Issue of Text & Talk, 34/3, 2014. Adami E., Facchinetti R. and Kress G. (eds.)
Elisabetta Adami
View PDFchevron_right
Translation and Multimodality: Beyond Words
monica boria

2019

One of the first books to explore how translation needs to be redefined and reconfigured in contexts where multiple modes of communication, such as writing, images, gesture, and music, occur simultaneously. Bringing together world-leading experts in translation theory and multimodality, each chapter explores important interconnections among these related, yet distinct, disciplines, with examples from literature, dance, music, TV, film, and the visual arts.

View PDFchevron_right

Related topics

  • Translation Studies
  • Multimodal Discourse Analysis
  • Audiovisual Translation
  • Multimodal Literacy
  • Multimodal Communication
    • 580 California St., Suite 400
    San Francisco, CA, 94104
    © 2026 Academia. All rights reserved