2.2.4　Multimodal discourse analysis based on specific genres

With the rise of multimodality as a research field，researchers pay increasingly more attention to the combination of multiple semiotic modes in various discourse genres.The consciousness is further enhanced by the abundance of multimodal artefacts of communication in contemporary society.Researchers try to apply theories of multimodality in a range of discourse genres，such as the technology-mediated interaction like hypertexts（Kok，2004;Djonov，2005），film genres（Baldry and Thibault，2006），advertising genres（Cheong，2004;Stöckl，2004），pedagogic practices and genres（Guo，2004），literacy（Kress，2003;Jewitt，2006;Unsworth，2008），medical genres（Eckkrammer，2004;Bührig，2004），and many other professional genres.Since the English learning advertisements discussed in this book belong to the advertising genre and also bear features of the educational discourses，the review in this section will focus on the studies on advertising genres and educational genres from the multimodal perspective.

2.2.4.1　Advertising genres

Although many studies include advertisements as part of the samples in their theorization of multimodality，the problem is that they are，in most cases，not based on advertisements exclusively，or the advertisements used are not representative of a corpus which has been systematically collected.This problem in methodology is coupled with their research objective：they are mainly theoretically oriented rather than genre-based，that is，their aim is to propose models or frameworks of conceptualizing visual-verbal relations in general（see for example，Royce，1998;Martinec and Salway，2005;O'Halloran，1999）rather than analyzing the way of visual-verbal interaction in a specific genre.

In spite of the scarcity of the semiotic research on inter-semiotic relations based exclusively on advertising genre，there are a few studies that offer implications to the analysis and theorization in the present book.

Bell and Milic（2002）employ theories of Systemic Functional Linguistics，especially Kress and van Leeuwen's system of visual analysis，to identify the semiotic resources which are the basis of gender stereotypes in advertisements.Specifically，resources such as perspectival angle，plane of composition and gaze are used to investigate stereotyped portrayals of males and females in 827 Australian magazine advertisements.The results are discussed with reference to Goffman's work，Gender Advertisements.Their conclusion is that gender stereotyping was still significant in the magazines more than two decades after Goffman's analysis was first published.

Lock（2003）examines the advertisements on the Hong Kong mass transit railway as reflections of Hong Kong's multiple identities as an international，Chinese，Cantonese and post-colonial city.Lock adopts a Hallidayan perspective on the relationship between context and text，and Kress and van Leeuwen's（1996）framework on visual grammar to reveal how semiotic resources may be deployed to create coherent bimodal and often bilingual texts.He argues that the visual and linguistic resources in the ads can be interpreted as having multiple coding，and different readings depend on interaction between the potential meanings in the texts and the meanings available to the readers from their own cultural contexts.

Machin（2004）and Machin&van Leeuwen（2007）study various regional versions of the popular woman magazine Cosmopolitan as an instance of global media.The methodological tools they draw on are systemic-functional discourse analysis（Martin，1992）and modality analysis from social semiotic visual analysis（Kress and van Leeuwen，1996）.They find that these magazines share similar patterns with regard to verbiageimage relations：The text has high modality，uses particular genres and is relatively straight-forward，while the picture has relatively low naturalistic modality，shows women as glamorous，but not being in control of a situation or activity，and overall，has a distinct air of unreality.Thus，they claim that the picture creates contradictions between the fantasy of pleasurable glamour and the reality of the reader's life and work.They conclude that these different versions of magazines perpetuated women's traditional vulnerability and lack of control on the global basis.

Almeida（2009）conducts a visual semiotic account of advertisements for fashion dolls.Drawing on Kress and Van Leeuwen's（1996）grammar of visual design for the analysis of visual features in advertisements，she examines the topics reflected by advertisements for fashion dolls，the roles they play and the type of relationship that is established between the participants involved.Her analyses reveal the contemporary advertised representations of women，and suggest that they foster children's sense of group identity，promote values such as positive self-fulfillment and fashion consumption.

Chen（2016）follows the framework of multimodal critical discourse analysis and analyzes 24 advertisements by four top-selling automobile brands in China.The results reveal a consistent ideological separation between nature and human society in the analyzed advertisements：“nature”is constantly framed as either a valuable commodity for human consumption or an added value for high-end car models that in fact have low energy efficiency.The findings help deconstruct the operations of marketers in specific advertisements and reveal how they contribute to problematic environmental narratives.

Besides the above studies on the multimodal advertisements from the perspective of functional analysis，some works focusing on visual metaphor also have implications for the semiotic analysis in this book.The most relevant ones are Forceville（1996）and McQuarrie and Mick（1999），both of which concern the visual rhetoric in advertisements.The former extended the metaphor theory of Black（1979）from verbal metaphor to visual metaphor in advertising.The latter conducted empirical studies on the impact of visual rhetorical figures in advertisements，including rhyme，antithesis，metaphor，and pun.Sweetser（2017）finds that a complex multimodal ad's meaning lies not just in individual metaphoric mappings or metonymic structures，but in the interrelationship between multiple mappings and mental spaces—that is，in the mental space network built up by the ad.Skilled ad designers involve their audiences by prompting active network construction from implicitly as well as explicitly evoked inputs.

Although these works do not start from the perspective of discourse coherence，they have implications for the analysis of coherence in this book.Specifically，the implication drawn from these works is that the visual mode should be seen as an equal system of expression with the linguistic in the construction of multimodal discourses.The visual mode has features，structures，and styles similar as，or at least comparable to，the linguistic mode and thus can be analyzed in a parallel fashion.This view provides basis for the analysis of coherence in multimodal discourses.Their analytical results on the features and generic structure of pictures and verbal texts are also useful references for the conceptualization of visual-verbal relations in this book.

2.2.4.2　Educational genres

Compared to the studies on multimodal advertising genres，the studies on multimodal educational genres are far more prosperous and better-equipped in terms of methodology.“Although multimodality is underdeveloped as a theory，educational researchers have actively taken it up.Multimodality has been applied to a variety of learning contexts”（Jewitt，2006：3）.The reason may be attributed to the fact that many researchers also work as educators and have much easier access to the discourses produced in the educational contexts.This sub-section will briefly review the studies related to the research topic in this book.

Many researchers start from the print learning materials in various school subjects，such as O'Halloran（1999，2004）on mathematics，Guo（2004）on biology textbooks，and other studies.

With the increasing use of digital technology in education，the studies on electronic digitalized learning materials also spring in a large numbers.For example，Kress（2003）examines how the conceptualization of textuality is changing as images seem to dominate text and as screens overtake paper as the most frequent means of distributing text.He believes that the multiple communication modes，such as image，audio，video，which are brought into learning by the use of new media，are governed by distinct logics which change not only the deeper meanings of textual forms but also the structure of ideas，of conceptual arrangements，and of the structures of our knowledge.

Kaltenbacher（2004）analyzes the strategies for combining visualizations（sound waves，images and video-clips），sound（speech）and written text in the CD-ROMs used for language learning.He finds that，although the purpose of these strategies is ostensibly to train the users'language skills and to help them understand the meaning choices in the target language，the lack of relevant linguistic，semiotic or pedagogic expertise in the design process is too often delivering“multimodality”which actually disrupted learning.

Jewitt（2006）investigates the use of new technology in schools，such as the educational CD-ROMs and computer games，and its impact on the way teachers teach and students learn.The advantages of the technology-facilitated visual communication for learning include its ease and accessibility，its capacity for highlighting overall patterns and trends，its ability to add mood and immediacy，among others.Visuals allow children to interact directly with images or simulations of the real.But they can also be ambiguous and children may struggle to understand them.The implication for literacy and pedagogy is that，literacy should not be fragmented into different kinds of literacy such as visual literacy，digital literacy，and emotional literacy，instead，it should be redefined as design，as an active dynamic process of creating meaning out of multimodal semiotic resources.

From a diachronic perspective，Bezemer and Kress（2008）examine contemporary curriculum materials to investigate the learning gains and losses of different multimodal ensembles.This work draws on a corpus of learning resources for secondary schools in science，mathematics and English from the 1930s，the 1980s and the first decade of the twenty-first century，as well as digitally represented and online learning resources from the year 2000 onward.The project provides a social semiotic account of the changes to the design of e-learning resources and of their epistemological and social/pedagogic significance and shows that image and layout are increasingly meshed in the construction of content.