(2014) Multimodality in Translation and Interpreting Studies

9 Multimodality in Translation and Interpreting Studies: Theoretical and Methodological Perspectives Luis Pérez González Since the academic study of translation and interpreting began to accelerate in the middle of the twentieth century, much theorizing has reduced its primary object of investigation to written and oral texts, understood as verbal artifacts. This focus brings to the fore the centrality of linguistics as the discipline that has most informed trans- lation studies from its inception at least through the mid-1980s (Baker and Pérez- González 2011). The emphasis of early translation scholarship on “elaborating taxonomies of different types of equivalence” between decontextualized stretches of text and their translations (2011, 40) effectively excised language from the context that influences translational decisions. In turn, the displacement of language from context favored the study of written and spoken discourse in isolation from other non-verbal meaning-making resources. The analytical and interpretive limitations arising from the first excision (translation from context) became the object of growing scrutiny in the late 1980s. Since then, the emergence and consolidation of alternative disciplinary paradigms – including the “cultural” (Bassnett and Lefevere 1990), the Copyright © 2014. John Wiley & Sons, Incorporated. All rights reserved. “sociological” (Chesterman 2007), and the “medial” (Littau 2011) turns – have shifted attention towards different dimensions of the context where the production and trans- lation of texts are embedded. For all these advances, however, the displacement of language from non-verbal meaning-making and its impact on the theorization of translation and interpreting remains largely unaddressed. The study of the contribution that non-verbal semiotics makes to written and spoken texts as loci of translation and interpreting activity has been patchy. Since A Companion to Translation Studies, First Edition. Edited by Sandra Bermann and Catherine Porter. © 2014 John Wiley & Sons, Ltd. Published 2014 by John Wiley & Sons, Ltd. A Companion to Translation Studies : Companion to Translation Studies, edited by Sandra Bermann, and Catherine Porter, John Wiley & Sons, Incorporated, 2014. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/manchester/detail.action?docID=1598002. Created from manchester on 2021-06-13 22:11:12.  120 Luis Pérez González Roman Jakobson first defined intersemiotic translation as “an interpretation of verbal signs by means of signs of nonverbal sign systems” ([1959] 2000), scholars have pro- posed a range of terms to categorize different types of shifts across sign systems that may arise in translation or interpreting.1 But the lack of a systematic conceptualiza- tion of non-verbal signs and their influence on translational behavior is also apparent in the conflicting definitions of the concepts that those terms designate. While some scholars (Gottlieb 1997, 111; Remael 2001, 13–14) have recently redefined interse- miotic translation as the transfer of meaning across different media (e.g., the filmic adaptation of a literary text), other specialists (Fine 1984) understand it as shifts between two different medial variants of the same sign system (e.g., the change from spoken into written language that takes place in film subtitling). The lack of consen- sus on where the referential boundaries between seemingly interchangeable terms – such as “medium,” “mode,” or “sign system” – lie ultimately exposes the need for a more comprehensive and sophisticated understanding of the semiotic fabric of trans- lated and interpreted texts. Admittedly, awareness of the dialectic between verbal and non-verbal signs has informed typologies of translation based on the nature and scope of the semiotic shifts that arise during the mediation of written or oral texts (e.g., Gottlieb 1998). However, the extent to which translation is influenced by the distri- bution of meaning across various semiotics in the source text has received considerably less attention. The starting premise of this essay is that textual artifacts often encode their message in different meaning-making resources. Translators should therefore give careful con- sideration to the manifold connections between verbal and non-verbal resources in the source text: overlooking them may be detrimental to the target reader’s holistic per- ception of the overall semiotic ensemble. In line with this premise, the first section of this essay surveys a number of interrelated areas of research exploring the dialectic between the physical and signifying structures of traditional textualities. The second section examines the generative potential of digital communication technology as a catalyst for the emergence of new semiotic configurations across a range of texts and communicative encounters. The last section focuses on the disciplinary implications of the growing perception that analyzing language alone does not suffice to understand Copyright © 2014. John Wiley & Sons, Incorporated. All rights reserved. translation. It explores how insights imported from multimodal theory are helping translation and interpreting scholars gain new insights into both old and new data, and addresses the methodological implications of multimodal research in translation and interpreting studies. Non-Verbal Semiotics in Traditional Textualities The study of the impact of non-verbal semiotics on the translation and reception of theatrical texts has been neglected until the recent surge of interest in performability (Bassnett 2000; Zatlin 2005). Traditionally, the staging of plays has been organized as a collaborative effort. The literal annotated translation of the original text produced A Companion to Translation Studies : Companion to Translation Studies, edited by Sandra Bermann, and Catherine Porter, John Wiley & Sons, Incorporated, 2014. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/manchester/detail.action?docID=1598002. Created from manchester on 2021-06-13 22:11:12.  Multimodality in Translation and Interpreting Studies 121 by a translator would normally be rewritten by a non-language-specialist theater practitioner to enhance the performability of the text (Eaton 2008). As adapting for performance “demands a dramaturgical capacity to work in several dimensions at once, incorporating visual, gestural, aural and linguistic signifiers into the translation” (Hale and Upton 2000, 2), negotiating the contribution of extra-linguistic semiotics is crucial during this rewriting stage. Interestingly, a large body of literature on drama translation concedes that decoding the complex of verbal and non-verbal sign systems contained in the source text and re-encoding them in the adapted text falls outside the competence of translators. Only adapters, it is argued, can enable the realization of the “gestic” or inner text that exists within any written play through performance, facilitating the engagement of the director and the actors with the different signifiers of the performed version (Bassnett 2000). This emphasis on written translated plays, to the detriment of translated drama as acted and produced, accounts for the marginal place accorded to the theorization of non-verbal semiotics in the context of traditional drama translation scholarship (Hale and Upton 2000). David Johnston (1996) and Phyllis Zatlin (2005) attribute this conceptualization of translators and theater practitioners as mutually excluding agents to the scarcity of opportunities enjoyed by adapters to reflect and write on the scope of their involve- ment in translation for the stage. This misconception is now being reversed, as the study of semiotic mediation in theatrical texts increasingly focuses on performance as a form of translation realized through the interaction between various sign systems and the different agents involved in the production and reception of the text (Baines and Dalmasso 2007). The analysis of translatorial mediation no longer revolves around structuralist formalizations of on-stage semiotics – such as Tadeusz Kowzan’s (1975) theorization of performance in terms of spoken text, bodily expression, actors’ physical appearance and body language, playing space, and non-spoken sound. Instead, studies on drama translation now examine the extent to which the images of stage set and design reflect the cultural negotiations in the play as expressed through translation (Brodie 2012). With more translators-cum-theater practitioners taking on a reflective role, the debate is shifting towards the role of translators in rerouting the original written text through performance, in a process where the translation of actions and Copyright © 2014. John Wiley & Sons, Incorporated. All rights reserved. the re-creation of non-verbal signifiers become more central than the translation of words (Eaton 2008). Audiovisual textualities – including films, dramas, or videogames – represent another crucial locus of interaction between verbal and non-verbal signifiers. Faced with a complex ensemble of semiotic choices from different sign systems, the transla- tor’s mediation of audiovisual texts is grounded in processes of perceptual hermeneutics. Frederic Chaume Varela’s (2004a, 2004b) theorization of film translation is predicated on the translator’s capacity to interpret the web of interactions between “signifying codes which complement and frame words and linguistic meaning” (2004a, 12). While viewers are neurologically equipped to process filmic artifacts as a single unified gestalt in perception, translators need to be able to dissect this apparently holistic impression. Consequently, by gaining a better understanding of how meaning is A Companion to Translation Studies : Companion to Translation Studies, edited by Sandra Bermann, and Catherine Porter, John Wiley & Sons, Incorporated, 2014. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/manchester/detail.action?docID=1598002. Created from manchester on 2021-06-13 22:11:12.  122 Luis Pérez González distributed across different sign systems, they will be able to mediate spoken dialogue more effectively. According to Chaume Varela’s (2004b) structuralist account, meaning is conveyed to viewers through the acoustic and visual channels along two clusters of semiotic codes. Apart from spoken language, two other codes or sign systems are realized through the acoustic channel: para-verbal signs (not what is said, but how it is said) and non-verbal acoustic signs – including music, special effects, and sound arrangements. The visual channel enables the realization of the iconographic code (through the use of symbols and icons), the photographic code (pertaining to the use of color, light, and perspective), and the mobility code (involving the deployment of proxemic and kinesic cues). Over the last decade, a growing body of interdisciplinary studies has been developed from this same premise: that translators’ familiarity with cinematographic conventions and their acquisition of visual literacy are directly proportional to the quality and sophistication of their mediation. Elsewhere I have examined the influence of visual perspective on the unfolding of cinematic narratives and on the translation of the dialogue that propels narratives forward (Pérez-González 2007a). Shifts in camera angle and variations in the focal length of the lenses used to shoot key scenes in films – one more instance of the semiotic systems at play in cinematographic texts – are found to set the mood for entire filmic sequences by articulating different forms of viewer involvement in the diegetic text and shaping dramatic characterization. Visual perspec- tive, and the emotional responses that it evokes, influence the translator’s interpretation of the filmic semiotic ensemble, and hence the manner in which the translated dialogue interacts with other meaning-making systems. Perceptual hermeneutics also informs Anna Maszerowska’s (2012) work on the impact that luminance and contrast patterns have on the meaning of filmic texts. Lighting “greatly contributes to the saturation of the audiences’ imaginations, complementing and carrying on the plot, reflecting the characters’ points of view and, at the same time, filling in the gaps between dialogues” (2012, 83). Whether the emphasis in any given frame is placed on luminance patterns, the use of color, idiosyncratic camera movements, or directorial editing choices, audiovisual translation calls for an enhanced awareness of the connections between cinematography, Copyright © 2014. John Wiley & Sons, Incorporated. All rights reserved. plot, and dialogue. Against this complex semiotic ensemble, translated language is meant to act as the mortar that cements the rest of the semiotic blocs together, accen- tuating certain messages and/or facilitating the interpretation of other signifiers. Translation and interpreting often interact with the semiotics of the human body. This term designates the use of para-verbal signs (including, but not limited to, voice quality, cadence, inflection, or rate of speech) and non-verbal signifiers (such as ges- tures or movements) (Poyatos 1997). Among the para-verbal means of speech, voice can have a significant impact on the construction and perception of public and fictional personas. Occasionally, the changes in voice quality that arise during translation pro- cesses such as dubbing can be detrimental to dramatic characterization. Changes in pitch or the characters’ control over their vocal delivery may evoke different percep- tions among viewers, thus undermining earlier creative decisions made prior to the A Companion to Translation Studies : Companion to Translation Studies, edited by Sandra Bermann, and Catherine Porter, John Wiley & Sons, Incorporated, 2014. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/manchester/detail.action?docID=1598002. Created from manchester on 2021-06-13 22:11:12.  Multimodality in Translation and Interpreting Studies 123 filming of an audiovisual text and jeopardizing the contribution that the interplay between their characters’ appearance and prosody was meant to deliver (Bosseaux 2008). In other cases, these shifts in perception arise from the mediation of specific sets of prosodic features with distinctive sociolinguistic connotations pertaining, for example, to a character’s accent or dialect. Transferring the resonances of this aspect of para-verbal meaning encoded in the phonetic realization of a character’s dialect is particularly challenging in texts made up of different signifying systems. As transla- tors tend to erase the para-verbal markers of sociolinguistic variation present in the source text, the fact that the original visual effects, gestures, and general plotline remain unchanged in the target version may “shift the social meanings” of those markers, which often proves detrimental to the viewing experience of the target audi- ence (Queen 2004, 531). Mediating para-verbal and non-verbal signifiers effectively is also crucial in inter- preted events, particularly those in institutional settings. Since “dialogue interpret- ing” (Mason 2001) emerged as a distinct paradigm within interpreting studies a decade ago, studies of face-to-face “three-way interaction” between institutional rep- resentatives, service users, and language-cum-culture mediators have drawn upon the semiotics of the human body to reconceptualize the role of interpreters. While this new paradigm recognizes that interpreters remain largely constrained by predeter- mined roles and institutionally sanctioned codes of conduct, it also acknowledges their status as fully ratified interlocutors with the capacity to shape the unfolding of the encounter (Mason and Stewart 2001). By shifting the focus away from the static concept of “role” towards the more dynamic notion of “interpreter positioning,” dia- logue interpreting seeks to better account for the interplay between the language mediators’ discretion and the factors governing the encounter. Under this paradigm, para-verbal and non-verbal aspects of institutional talk are theorized as “contextualiza- tion cues” that prompt changes in the participants’ alignment with one another and facilitate the mutual recognition of their changing role as interlocutors or simple onlookers (Mason 2009). The impact of participants’ gaze on the organization of interpreter-mediated interaction has emerged as one of the most productive areas of study within this research strand. Using highly sophisticated transcription conven- Copyright © 2014. John Wiley & Sons, Incorporated. All rights reserved. tions to encode participants’ gaze vectors, scholars are able to map this non-verbal signifier onto a range of interactional sequences and hence gain a better understanding of the discursive function of participants’ conversational moves in public service encounters (Davitti 2012). The integration of verbal and non-verbal behavior reveals that recurrent patterns of interaction often coincide with specific shifts in gaze direc- tion to pursue preferred responses from fellow participants, to re-engage other parties into the communicative framework at crucial points, and to manage turn-taking mechanisms effectively. Similarly, the use of gestures, facial expressions, and body positioning can help participants to retain control of complex conversational sites where service providers and users negotiate their conflicting expectations through an interpreter (Pasquandrea 2012). Ultimately, dialogue interpreting studies reveal the extent to which A Companion to Translation Studies : Companion to Translation Studies, edited by Sandra Bermann, and Catherine Porter, John Wiley & Sons, Incorporated, 2014. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/manchester/detail.action?docID=1598002. Created from manchester on 2021-06-13 22:11:12.  124 Luis Pérez González [v]erbal and non-verbal semiotic resources constitute an integrated system, which needs to be analyzed as a whole, in order to gain a thorough understanding of the communicative dynamics of interpreter-mediated interaction. (Pasquandrea 2012, 150) The notion of paratexts, on the other hand, illustrates the semiotic contribution of non-linguistic meaning-making resources to the semiotics of written texts. Various applications and critiques of Gérard Genette’s (1997) theorization of paratexts – understood as textual matter which surrounds and mediates the author’s literary text to its readership – are available in the literature (Tahir Gürçağlar 2002; Baker 2006) and are therefore not covered here. Within the wider territory of material (non-lin- guistic) paratexts, the term “visual paratext” designates “features such as illustrated title-pages, woodcut illustrations, frontispiece plates, decorative capital letters, and typographical ornaments” in printed texts (Armstrong 2007, 42). The conceptualiza- tion of these features as paratextual elements is predicated on the premise that pub- lishers’ selection of material or technological resources at any given historical moment and their adherence to or deviation from typographical and mise-en-page conventions are capable of constituting meaning (Pérez-González 2013). Paratextual choices per- taining to the visual and material dimensions of the textual artifact can thus be theo- rized as the outcome of a “complex negotiation of the text’s meaning within the economic, social, political and cultural contexts and conventions current at its moment of production” (Bell 2002, 632). Publishing a new translated edition of a classic, for example, provides all parties, including translators, with a site to inscribe their own narratives and interpretations of the original text, not least through the visual paratextual features of the new artifact. By selecting specific images and illus- trations and opting for certain fonts, types of paper, or layout patterns for the new translation, publishers may seek to frame the classic text, bringing it to bear on current political discourses and debates; alternatively, they may choose to change existing public perceptions of the text in question – for example, by shifting the focus away from its esthetic qualities onto its historical value. Attempts to mediate public reception can also be observed in the film industry, as films have historically contrib- uted to reinforcing or subverting public discourses and attitudes on social class, Copyright © 2014. John Wiley & Sons, Incorporated. All rights reserved. gender, sexuality, and ethnicity. Textually, the very processes of subtitling or dubbing open up opportunities to mediate such discourses through translational decisions, as the speech of characters embodying or resisting specific values or clichés is transferred into the target language(s). Paratextually, the use of specific visual features and resources – such as posters, DVD covers, and captions superimposed thereon – can also play a decisive role in the framing of reception. Through the strategic deploy- ment of visual paratexts, whether they involve replacing the original features or bringing into sharper relief specific aspects of the original representations and their connotations, distributors mediate public perception of films and their characters, managing audiences’ expectations in ways that serve their own commercial, political, or ideological interests (de Marco 2012). A Companion to Translation Studies : Companion to Translation Studies, edited by Sandra Bermann, and Catherine Porter, John Wiley & Sons, Incorporated, 2014. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/manchester/detail.action?docID=1598002. Created from manchester on 2021-06-13 22:11:12.  Multimodality in Translation and Interpreting Studies 125 Non-Verbal Semiotics in Digital Textualities The shift from the age of printed culture and mass media towards the era of electronic and, more recently, digital culture has had a significant impact on the dialectic between verbal and non-verbal semiotic resources in textualities that coordinate text and image, as well as on the consumption of and engagement with such texts. One of the most significant changes pertains to the consolidation of new forms of interse- miotic assistive mediation facilitating access to information and entertainment for sensory impaired people. Capitalizing on the high storage capacity of DVDs, media companies are able to release audiovisual products aimed at mainstream viewers, while simulta- neously allowing additional niche audiences to access the media content in assistive mode – normally by viewing the film in combination with one of the multiple audio and subtitle tracks that this technology affords. Subtitling for the hard of hearing, for example, provides a text display of the characters’ speech interspersed with written descriptions of sound features from the diegetic action that would otherwise not be accessible to deaf viewers. This transfer of information from speech to written subtitles involves the deployment of specialized mediation conventions pertaining to the color, timing, and text positioning of the subtitles (Neves 2005). Audio description, a spoken account of those visual aspects of a film which play a role in conveying its plot, has become equally important in ensuring the accessibility of audiovisual prod- ucts to the visually impaired. While transferring information from the visual to the acoustic channel – from images to the spoken narration that a voice delivers between the stretches of spoken dialogue – the audio describer “engages in a delicate balancing exercise to establish what the needs of the spectator may be, and to ensure the audi- ence is not overburdened with excessive information” (Pérez-González 2009, 16). But the impact of technological developments on our cognitive and perceptual capabilities through changes in our reading, writing, and thinking practices, includ- ing the traditional conceptualization and praxis of translation, is not restricted to the emergence of intersemiotic assistive practices. The influence of computer technology on translational behavior is also being explored in the context of the hypertext (Littau Copyright © 2014. John Wiley & Sons, Incorporated. All rights reserved. 1997). Hypertextual environments enable multiple textual arrangements, for example by embedding texts within wider texts and establishing connections between text and images, hence fostering intertextuality and challenging the seriality of translation. When mediating hypertextual content, translation “can therefore no longer be con- ceived of as the reproduction of an original, but has become subject to reconceptuali- sation as the re-writing of an already pluralised ‘original’ ” (Littau 1997, 81). The less reverent approach to authorship associated with hypertext environments has proved particularly productive for those scholars aiming to politicize the study of translation. Insofar as originals need not be necessarily approached as continuous, coherent texts, engaged scholars are able to resist the dominant discourses encoded in them and open up new and alternative reading positions. The hypertext also helps translation scholars A Companion to Translation Studies : Companion to Translation Studies, edited by Sandra Bermann, and Catherine Porter, John Wiley & Sons, Incorporated, 2014. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/manchester/detail.action?docID=1598002. Created from manchester on 2021-06-13 22:11:12.  126 Luis Pérez González and practitioners articulate and explore the intersemiotic dimension of Kwame Appi- ah’s (2004) notion of “thick translation.”2 Hybrid texts consisting of written and spoken material, straddling singly and multiply authored content, and representing a constellation of participants whose voices need to be acknowledged and conveyed individually can thus be best translated within a hypertextual environment. The mediation of such pluralized and non-linear textual material often results in complex artifacts made up of multiple layers of text, allowing for multiple individual reading experiences through intertextual resonance and the interplay between verbal and non- verbal signifiers (Milsom 2008). Over the last decade, the development of networked and collaborative technologies has fostered the emergence of new forms of participatory citizenship in the new digital economy. Readers and viewers are now able to archive, annotate, and recirculate media content, so their personal copies of audiovisual texts have the potential to provide unique reading experiences (Pérez-González 2013). The relevance of such advances to forms of translation involving the mediation of non-verbal signifiers is twofold. First, collaborative technologies have promoted the proliferation of virtual networks of amateur subtitlers, most of which have articulated and continue to develop new approaches to the mediation of verbal and non-verbal elements in audiovisual texts. Anime fansubbing, a prolific global phenomenon involving the subtitling of Japanese animated cinema by fans, is a good case in point. Unhappy with the cultural insen- sitivity of commercial translations, fansubbing networks originally set out to develop “abusive subtitling” practices (Nornes 1999). Although these require additional pro- cessing effort from viewers, they help preserve the “otherness” of the original films. To safeguard the integrity of the viewing experience, fansubbing networks “exploit traditional meaning-making codes in a creative manner and criss-cross the traditional boundaries between linguistics and visual semiotics in innovative ways” (Baker and Pérez-González 2011, 48) that have been described at length in the literature (Ort- abasi 2006; Pérez-González 2007a, 2007b). Second, the emergence and generalization of participatory textual practices engender new forms of consumption, transforming the discourse communities that use, critique, and circulate translations of those col- laborative texts. The work of participatory translation networks, for example, is closely Copyright © 2014. John Wiley & Sons, Incorporated. All rights reserved. monitored by online communities of users through dedicated websites and forums. In some of these virtual discourse communities, users are able to take part in the actual mediation of texts (Dwyer 2012); in others, translated texts effectively act as “nexus points for discourse around ownership and rights, fan knowledge and ‘subcul- tural capital’ ” or, alternatively, as platforms for users to engage in confrontations with “other mediators and subcultural arbiters” (Denison 2011, 450). From Semiotics to Multimodality The study of the contribution that non-linguistic signs make to translated and inter- preted texts has been informed mainly by semiotics – as adopted in translation studies A Companion to Translation Studies : Companion to Translation Studies, edited by Sandra Bermann, and Catherine Porter, John Wiley & Sons, Incorporated, 2014. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/manchester/detail.action?docID=1598002. Created from manchester on 2021-06-13 22:11:12.  Multimodality in Translation and Interpreting Studies 127 by Roman Jakobson ([1959] 2000) and reformulated by Gideon Toury (1986) and Umberto Eco (2001). The successive iterations of this model have been applied exten- sively not only to the study of translated syncretic texts such as comics (Celotti 2008) and advertisements (Adab and Valdés 2004), where different semiotic systems are co- present and interplay at different levels, but also to the adaptation of written texts “from and into a variety of other art forms, such as . . . cinema (including animated cartoons), painting, music, song, sculpture, pantomime, etc.” (Zanettin 2008, 11). But while this approach to the study of semiotics has made great strides in enhancing our understanding of the iconic-verbal link and the translation thereof, it has received criticism for emphasizing “structures and codes, at the expense of functions and social uses of semiotic systems” (Hodge and Kress 1988, 1). In recent years scholars interested in texts deploying more than one sign system have come to agree that the production and interpretation of semiotic meaning is dynamic and context-dependent. The generative potential of the signs used in each specific context is best encapsulated by the notion of semiotic resource (van Leeuwen 2005). Multimodal theory, a scholarly spin-off of social semiotics and systemic func- tional linguistics, aims to formalize the socially situated nature of meaning-making practices. In this paradigm, the notion of mode (or modality) designates each system of meaning-making resources from which communicators must choose in order to realize their communicative intentions through textual practices (Chandler 2002). As syn- cretic texts draw on several systems of semiotic resources (including, but not limited to, language, image, music, color, and perspective), they are often referred to as mul- timodal texts (Kress and van Leeuwen 2001, 67). Certain modes have more than one medial realization (e.g., language can be used in written or spoken form, while images can be dynamic or static), which will trigger additional choices at the level of sub-modes (Stöckl 2004). The deployment of written (printed) language, for instance, entails sub-modal choices in terms of font type, size, and color, while the use of spoken lan- guage involves choices pertaining to intonation, pitch, and timbre. As I have noted elsewhere, it is the combination of the communicator’s choices out of the options available under Copyright © 2014. John Wiley & Sons, Incorporated. All rights reserved. each sub-model system . . . that ultimately determines the realization of a mode in a multimodal text. (Pérez-González 2007b, 74) Subtitling is the strand of translation studies that has benefited the most from the application of multimodal theory. The reconceptualization of audiovisual texts as “composite products of the combined effect of all the resources used to create and interpret them” (Baldry and Thibault 2006, 18) raises the question of how subtitlers transfer meaning from visual modes onto the written language of subtitles when the overall semiotic fabric of the films requires it (Chuang 2006). In the context of con- ventional film semiotics, teasing out the specific contribution of both linguistic and non-linguistic cinematic signifiers is particularly important in those genres drawing heavily on implied meaning and indirectness – and hence on the viewers’ capacity to A Companion to Translation Studies : Companion to Translation Studies, edited by Sandra Bermann, and Catherine Porter, John Wiley & Sons, Incorporated, 2014. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/manchester/detail.action?docID=1598002. Created from manchester on 2021-06-13 22:11:12.  128 Luis Pérez González process the information encoded in non-verbal modes (Desilla 2012). But technologi- cal advances are paving the way for even more active spectatorial experiences and “encourag[ing] a more multimodal way of watching film” (Ortabasi 2006, 288). Whether it is through the use of hyperlink technology or other systems of multimodal navigation, audiences of certain films can access annotations pertaining to the “histori- cal, cultural and social intertextualities of the film, of which they might otherwise not be aware” (2006, 288). Other applications of multimodal theory in translation studies have revealed the complexity of the textual adaptations that the internationalization of printed media content occasionally calls for (Chueasuai 2010). Multimodal texts created for global consumption can become sensitive when translated for communities professing dif- ferent sociocultural and religious values from those of mainstream Western cultures. To ensure that translated texts remain within the bounds of social and legal accept- ability in the target locale, and hence that corporate profits remain robust, editorial policies promote both verbal and linguistic shifts during the translation process. Constrained by institutional agendas, translators often opt for situated meaning- making practices aiming to minimize potentially offensive political, sexual, or irreli- gious overtones across different modes. New research methods have been developed to help scholars address the complexity of multimodal information processing. Multimodal transcriptions (Thibault 2000) are intended to yield a better understanding of inter-modal relations within texts. In these tabular transcriptions (Taylor 2003), the left-most column typically displays stills of selected frames – with each row devoted to each individual frame. The remaining columns deliver a coded analysis of the semiotic choices deployed by the communica- tor in the frames under scrutiny. The number and ordering of the columns, the range of modes and sub-modes covered in the transcription, and the set of notation conven- tions used for coding purposes depend on the specific needs of the individual project. Computer-held multimodal corpora (Valentini 2006; Sotelo Dios 2011) are also being developed to provide the researcher with quantitative and empirical evidence on the correspondence between certain conflations of multimodal resources and specific trans- lation strategies. Copyright © 2014. John Wiley & Sons, Incorporated. All rights reserved. Concluding Remarks This essay has illustrated how attempts to gain a better understanding of meaning- making practices involving the combination of different types of semiotic resources and their impact on translational decisions are gaining ground within translation studies. Multimodality is bound to become even more central to translation scholar- ship in future years, as technological developments and new forms of amateur and participatory communication and translation move towards the core of mainstream cultural industries. As the kinds of texts featuring interdependent semiotic resources become the norm, new varieties of multimodal literacy will develop, as will the A Companion to Translation Studies : Companion to Translation Studies, edited by Sandra Bermann, and Catherine Porter, John Wiley & Sons, Incorporated, 2014. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/manchester/detail.action?docID=1598002. Created from manchester on 2021-06-13 22:11:12.  Multimodality in Translation and Interpreting Studies 129 theoretical frameworks seeking to articulate and conceptualize their role in social life. Maria Tymoczko’s statement that “future media developments will present additional research questions that we cannot yet even foresee” pre-dates some of the advances surveyed in this chapter, but her claim that such developments may “necessitate the retheorization of various aspects of the entire field of translation studies” (2005, 1090) aptly articulates how multimodality may change the face of the discipline. See also Chapter 1 (Baker), Chapter 4 (Bassnett), Chapter 11 (Dunne), Chapter 23 (Mazzei), Chapter 31 (Lowe), Chapter 37 (Yau), Chapter 38 (Neather), Chapter 45 (Emmerich) Notes 1  For a survey of such terms – including “interse- use of supplementary textual material, e.g., miotic,” “intrasemiotic,” “diasemiotic,” and footnotes, to enhance the reader’s familiarity “intra-/inter-systemic” translation – see Zanet- with the context of production of the primary tin (2008). text. 2  The term “thick translation,” discussed exten- sively by Theo Hermans (2003), designates the References and Further Reading Adab, Beverly, and Cristina Valdés, eds. 2004. Key Bassnett, Susan. 2000. “Theatre and Opera.” In Debates in the Translation of Advertising Material. The Oxford Guide to Literature in English Transla- Special issue, The Translator 10, no. 2. tion, ed. Peter France, 96–103. Oxford: Oxford Appiah, Kwame A. 2004. “Thick Translation.” In University Press. The Translation Studies Reader, ed. Lawrence Venuti, Bassnett, Susan, and André Lefevere, eds. 1990. 2nd edn., 398–401. London: Routledge. Translation, History and Culture. London: Armstrong, Guyda. 2007. “Paratexts and Their Routledge. Functions in Seventeenth-Century English Bell, Maureen. 2002. “Mise-en-page, Illustration, Copyright © 2014. John Wiley & Sons, Incorporated. All rights reserved. ‘Decamerons.’ ” The Modern Language Review Expressive Form. Introduction.” In The Cam- 102, no. 1: 40–57. bridge History of the Book in Britain, vol. 4: Baines, Roger, and Fred Dalmasso. 2007. “A Text 1557–1695, 632–35. Cambridge: Cambridge on Trial: The Translation and Adaptation of University Press. Adel Hakim’s Exécuteur 14.” Social Semiotics 17, Bosseaux, Charlotte. 2008. “Buffy the Vampire no. 2: 229–57. Slayer: Characterization in the Musical Episode Baker, Mona. 2006. Translation and Conflict: A Nar- of the TV Series.” The Translator 14, no. 2: rative Account. London: Routledge. 343–72. Baker, Mona, and Luis Pérez-González. 2011. Brodie, Geraldine. 2012. “Theatre Translation for “Translation and Interpreting.” In Routledge Performance: Conflict of Interests, Conflict of Handbook of Applied Linguistics, ed. James Cultures.” In Words, Images and Performances in Simpson, 39–52. London: Routledge. Translation, ed. Rita Wilson and Brigid Maher, Baldry, Anthony, and Paul. J. Thibault. 2006. 63–81. London: Continuum. Multimodal Transcription and Text Analysis. Celotti, Nadine. 2008. “The Translator of Comics London: Equinox. as a Semiotic Investigator.” In Comics in A Companion to Translation Studies : Companion to Translation Studies, edited by Sandra Bermann, and Catherine Porter, John Wiley & Sons, Incorporated, 2014. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/manchester/detail.action?docID=1598002. Created from manchester on 2021-06-13 22:11:12.  130 Luis Pérez González Translation, ed. Federico Zanettin, 33–49. Man- Gottlieb, Henrik. 1997. Subtitles, Translation chester: St. Jerome. and Idioms. Copenhagen: University of Chandler, Daniel. 2002. Semiotics: The Basics. Copenhagen. London: Routledge. Gottlieb, Henrik. 1998. “Subtitling.” In Routledge Chaume Varela, Frederic. 2004a. “Film Studies Encyclopedia of Translation Studies, ed. Mona and Translation Studies: Two Disciplines at Baker, 1st edn., 244–48. London: Routledge. Stake in Audiovisual Translation.” Meta 49, no. Hale, Terry, and Carole-Ann Upton. 2000. “Intro- 1: 12–24. duction.” In Moving Target: Theatre Translation Chaume Varela, Frederic. 2004b. Cine y traducción. and Cultural Relocation, ed. Carole-Ann Upton, Madrid: Cátedra. 1–13. Manchester: St. Jerome. Chesterman, Andrew. 2007. “Bridge Concepts in Hermans, Theo. 2003. “Cross-Cultural Translation Translation Sociology.” In Constructing a Sociology Studies as Thick Translation.” Bulletin of the of Translation, ed. Michaela Wolf and Alexandra School of Oriental and African Studies 66, no. 3: Fukari, 171–86. Amsterdam: John Benjamins. 380–9. Chuang, Ying-Ting. 2006. “Studying Subtitle Hodge, Robert, and Gunther Kress. 1988. Social Translation from a Multi-Modal Approach.” Semiotics. Cambridge: Polity. Babel 52, no. 4: 372–83. Jakobson, Roman. (1959) 2000. “On Linguistic Chueasuai, Pasakara. 2010. “Translation Shifts in Aspects of Translation.” In Translation Studies the Love and Lust Section in the Thai Version Reader, ed. Lawrence Venuti, 2nd edn., 232–39. of Cosmopolitan: A Systemic Functional Per- London: Routledge. spective.” PhD dissertation, University of Johnston, David. 1996. “Introduction.” In Stages Manchester. of Translation, ed. David Johnston, 5–12. Bath: Davitti, Elena. 2012. “Dialogue Interpreting as Inter- Absolute Classics. cultural Mediation: Integrating Talk and Gaze in Kowzan, Tadeusz. 1975. Littérature et spectacle. The the Analysis of Mediated Parent-Teacher Meet- Hague: Mouton. ings.” PhD dissertation: University of Manchester. Kress, Gunther, and Theo van Leeuwen. 2001. de Marco, Marcella. 2012. Audiovisual Translation Multimodal Discourse: The Modes and Media of through a Gender Lens. Amsterdam: Rodopi. Contemporary Communication. London: Arnold. Denison, Rayna. 2011. “Anime Fandom and the Littau, Karin. 1997. “Translation in the Age of Liminal Spaces between Fan Creativity and Postmodern Production: From Text to Intertext Piracy.” International Journal of Cultural Studies to Hypertext.” Forum for Modern Language Studies 14, no. 5: 449–66. 33, no. 1: 81–96. Desilla, Louisa. 2012. “Implicatures in Film: Con- Littau, Karin. 2011. “First Steps Towards a Media strual and Functions in Bridget Jones Romantic History of Translation.” Translation Studies 4, no. Comedies.” Journal of Pragmatics 44: 30–53. 3: 261–81. Dwyer, Tessa. 2012. “Fansub Dreaming on ViKi: Mason, Ian. 2001. Triadic Exchanges: Studies in Dia- Copyright © 2014. John Wiley & Sons, Incorporated. All rights reserved. ‘Don’t Just Watch but Help when You Are logue Interpreting. Manchester: St. Jerome. Free.’ ” The Translator 18, no. 2: 217–43. Mason, Ian. 2009. “Role, Positioning and Dis- Eaton, Kate. 2008. “You Always Forget Some- course in Face-to-Face Interpreting.” In Inter- thing: Can Practice Make Theory?” New Voices preting and Translating in Public Service Settings. in Translations Studies 4: 53–61. Accessed August Policy, Practice, Pedagogy, ed. Raquel de Pedro 28, 2013. http://www.iatis.org/images/stories/ Ricoy, Isabelle Perez, and Christine Wilson, publications/new-voices/Issue4-2008/ 52–73. Manchester: St. Jerome. Eaton_2008.pdf. Mason, Ian, and Miranda Stewart. 2001. “Interac- Eco, Umberto. 2001. Experiences in Translation. tional Pragmatics, Face and the Dialogue Inter- Toronto: University of Toronto Press. preter.” In Triadic Exchanges: Studies in Dialogue Fine, Elizabeth C. 1984. The Folklore Text. Bloom- Interpreting, ed. Ian Mason, 51–70. Manchester: ington: Indiana University Press. St. Jerome. Genette, Gérard. 1997. Paratexts: Thresholds of Maszerowska, Anna. 2012. “Casting the Light Interpretation, ed. Jane E. Lewin. Cambridge: on Cinema: How Luminance and Contrast Pat- Cambridge University Press. terns Create Meaning.” In Multidisciplinary in A Companion to Translation Studies : Companion to Translation Studies, edited by Sandra Bermann, and Catherine Porter, John Wiley & Sons, Incorporated, 2014. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/manchester/detail.action?docID=1598002. Created from manchester on 2021-06-13 22:11:12.  Multimodality in Translation and Interpreting Studies 131 Audiovisual Translation, ed. Rosa Agost, Pilar tion.” In (Multi) Media Translation: Concepts, Orero, and Elena di Giovanni, 65–85. Special Practices and Research, ed. Yves Gambier and issue, MONTI 4: 65–85. Henrik Gottlieb, 13–22. Amsterdam: John Milsom, Anna-Marjatta. 2008. Picturing Voices, Benjamins. Writing Thickness: A Multimodal Approach to Sotelo Dios, Patricia. 2011. “Using a Multimedia Translating the Afro-Cuban Tales of Lydia Cabrera. Parallel Corpus to Investigate English-Galician PhD dissertation: Middlesex University. Subtitling.” Accessed February 28, 2013. http:// Neves, Josélia. 2005. “Audiovisual Translation: webs.uvigo.es/sli/arquivos/sdh2011.pdf. Subtitling for the Deaf and Hard-of-Hearing.” Stöckl, Hartmut. 2004. “In Between Modes: Lan- PhD dissertation: University of Surrey guage and Image in Printed Media.” In Perspec- Roehampton. tives on Multimodality, ed. Eija Ventola, Cassily Nornes, Abe M. 1999. “For an Abusive Subti- Charles, and Martin Kaltenbacher, 9–30. tling.” Film Quarterly 52, no. 3: 17–34. Amsterdam: John Benjamins. Ortabasi, Melek. 2006. “Indexing the Past: Visual Tahir Gürçağlar, Şehnaz. 2002. “What Texts Language and Translatability in Kon Satoshi’s Mil- Don’t Tell: The Uses of Paratext in Translation lennium Actress.” Perspectives 14, no. 4: 278–91. Research.” In Crosscultural Transgressions: Research Pasquandrea, Sergio. 2012. “Co-constructing Models in Translation Studies, II: Historical and Dyadic Sequences in Healthcare Interpreting: Ideological Issues, ed. Theo Hermans, 44–60. A Multimodal Account.” New Voices in Transla- Manchester: St. Jerome. tion Studies 8: 132–57. Accessed February Taylor, Chris. 2003. “Multimodal Transcription 28, 2013. http://www.iatis.org/images/stories/ in the Analysis, Translation and Subtitling publications/new-voices/Issue8-2012/IPCITI/ of Italian Films.” The Translator 9, no. 2: article-pasquandrea-2012.pdf. 191–205. Pérez-González, Luis. 2007a. “Appraising Dubbed Thibault, Paul. 2000. “The Multimodal Transcrip- Conversation. Systemic Functional Insights into tion of a Television Advertisement: Theory and the Construal of Naturalness in Translated Film Practice.” In Multimodality and Multimediality in Dialogue.” The Translator 13, no. 1: 1–38. the Distance Learning Age, ed. Anthony Baldry, Pérez-González, Luis. 2007b. “Intervention in 311–85. Campobasso: Palladino Editore. New Amateur Subtitling Cultures: A Multi- Toury, Gideon. 1986. “Translation.” In Encyclopedic modal Account.” Linguistica Antverpiensia 6: Dictionary of Semiotics, ed. Thomas Sebeok, vol. 67–80. 2, 1107–24. Berlin: Mouton de Gruyter. Pérez-González, Luis. 2009. “Audiovisual Transla- Tymoczko, Maria. 2005. “Trajectories of Research tion.” In The Routledge Encyclopedia of Translation in Translation Studies.” Meta 50, no. 4: 1082– Studies, ed. Mona Baker and Gabriela Saldanha, 97. 2nd edn., 13–20. London: Routledge. Valentini, Cristina. 2006. “A Multimedia Data- Pérez-González, Luis. 2013. “Co-creational Subti- base for the Training of Audiovisual Transla- Copyright © 2014. John Wiley & Sons, Incorporated. All rights reserved. tling in the Digital Media: Transformative and tors.” Journal of Specialised Translation 6: 68–84. Authorial Practices.” International Journal Cul- Accessed February 28, 2013. http://www tural Studies 16, no. 1: 3–21. .jostrans.org/issue06/art_valentini.pdf. Poyatos, Fernando, ed. 1997. Nonverbal Communi- van Leeuwen, Theo. 2005. Introducing Social Semiot- cation and Translation. Amsterdam: John ics. London: Routledge. Benjamins. Zanettin, Federico. 2008. “Comics in Translation: Queen, Robin. 2004. “‘Du hast jar keene Ahnung’: An Overview.” In Comics in Translation, ed. Fed- African American English Dubbed into German.” erico Zanettin, 1–32. Manchester: St. Jerome. Journal of Sociolinguistics 8, no. 4: 515–37. Zatlin, Phyllis. 2005. Theatrical Translation and Remael, Aline. 2001. “Some Thoughts on the Film Adaptation: A Practitioner’s View. Clevedon: Study of Multimodal and Multimedia Transla- Multilingual Matters. A Companion to Translation Studies : Companion to Translation Studies, edited by Sandra Bermann, and Catherine Porter, John Wiley & Sons, Incorporated, 2014. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/manchester/detail.action?docID=1598002. Created from manchester on 2021-06-13 22:11:12. Copyright © 2014. John Wiley & Sons, Incorporated. All rights reserved. A Companion to Translation Studies : Companion to Translation Studies, edited by Sandra Bermann, and Catherine Porter, John Wiley & Sons, Incorporated, 2014. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/manchester/detail.action?docID=1598002. Created from manchester on 2021-06-13 22:11:12.