3.1.2　From linguistic to multimodal：extension and adaptation

Multimodal texts are a carrier of meaning that draws on visual，spatial and verbal presentational modes in combination and co-operation.This kind of meaning is locatable neither within traditional linguistic views of the text nor within traditional views of the image（Bateman，2008：7）.Take the print English learning advertisements for instance，it is impossible to“extract the verbal from its usually highly graphically loaded context and consider it in isolation”（Bateman，2009）.In this new situation，the analysts need to，on the one hand，draw on results and methods from linguistics，and on the other hand，address the special features of other semiotic modes on the pages.It is this demand for new analytical tools and perspectives that make the study on coherence in multimodal discourses a challenging yet rewarding task.Therefore，for this book，the analysis of how a coherent meaning is achieved across different semiotic modes in the English learning advertisements requires a theoretical and analytical framework that is，on the one hand，broad enough to accommodate the commonalities across the verbal and visual modes，and on the other hand，specific enough to fit the special features and principles in the semiotic configurations in multimodal texts.

In order to establish such a framework，we should draw implications from the social semiotic theories which concern all semiotic modes，including verbal language，images，charts，and others.All the semiotic modes are equal means of meaning-making according to multimodal theories，and thus share some basic common principles.In this sense，the three-level division of discourse coherence in the previous section is discussed from such a perspective and can be made use of here.

However，the extension from linguistic to multimodal will not be a direct or selfevident one because these semiotic modes entail distinct features in various aspects，such as signification，structure，function，and form.Consequently considerate adaptation needs to be made for the framework to accommodate the complexity in the means of expression in multimodal discourses.That is，global coherence，local coherence and cohesive devices will all be modified to fit multimodality in the discourses.The genre features of the print English learning advertisements are also taken into consideration in the design of the framework.

This section will contribute to the key issues in the extension of the three-level division of discourse coherence from linguistic sense to multimodal sense，including the necessity，possibility and adaptation.Specifically，Sub-section 3.1.2.1 will explain the reasons why such a three-level framework should and could be extended to account for the coherence construction in multimodal discourses.Sub-section 3.1.2.2 will put forward the adaptations made to the framework according to the characteristics of multimodal discourses.

3.1.2.1　Extension of the three-level framework of coherence

The main challenge for multimodal discourse analysis is that we are surprisingly weak at the understanding of the meaning-making principles of the semiotic modes other than language.The result is a linguistic imperialism that guides us to analyze particular forms or aspects of a multimodal artefact as if they were linguistic in nature without the foundation of adequate knowledge or methodology in them（Bateman，2009;Bateman and Wildfeuer，2014）.Actually，without such understanding of semiotic modes other than language，a grammar of semiosis that can account for the multimodal meaning-making cannot be established.Therefore，the practices of multimodal discourse analysis nowadays run the risk of becoming“running commentaries”on the text instead of being real-sense analysis which should be theory-based.Bateman's remedy for this situation is“to develop frameworks for analysis within which empirical research can be conducted—and this involves concerning ourselves with the issue of just how we can analyze multimodal artefacts in a way that places the analysis on sound empirical and scientific foundations”（Bateman，2009：57）.His research effort yields a GeM framework which stratifies the multimodal artefacts into several layers of analysis and draws much on the Rhetorical Structure Theory and genre theories for the analysis of relations between semiotic elements.

This book is another research effort in this direction，that is，in order to analyze how coherence is constructed in multimodal discourses，a theory-based framework that allows for sound empirical analysis results of the coherence construction in multimodal discourses will be established.

As reviewed in Section 2.2.3，although there have been some studies on the visualverbal relations in various multimodal discourse genres，they cannot meet the abovementioned requirements for the framework.Their limitations are largely rooted in their origin in and over-dependence on the theories and approaches in linguistic discourse analysis.

First of all，the linguistic background of the researchers and the general inadequacy of expertise in image inevitably lead to the priority of language over image in the study of visual-verbal relations.The priority of language can be reflected in the following two aspects：one is that the analysis of language is much more elaborate and systematic than that of image.The other one is that the researchers，consciously or unconsciously，often rely on their interpretation of linguistic information to conceptualize the visual-verbal relations.Therefore，their conceptualization of visual-verbal relations is often influenced by their biases derived in language，and cannot be very objective.The downplay of semiotic resources other than language has resulted in an rather impoverished view of functions and meaning of discourse（O'Halloran，2004a：1）.

Secondly，almost all of the existing frameworks of visual-verbal relations，including those based on Halliday's expansion and projection and those based on semantic sense relations，treat linguistic component and the visual component as self-contained semiotic units（Wang，2010）.However，whether there are links and correspondences in the internal composition and signification between language and image remains largely unknown.

Thirdly，the linguistic theories they draw upon，no matter whether they are drawn from Halliday's framework of clause complexes or Rhetorical Structure Theory，are concerned with the local scope of connectivity between sentences or clause complexes.The ways global coherence and cohesive devices are realized in multimodal discourses remain largely unexplored.In other words，the existing frameworks on visual-verbal relations cover only the relations between the verbal resources and visual resources at a local level，whereas how they relate to the higher-level discourse topic and how they are organized at the surface level of expression have been very little studied.The local and surface aspects of connectivity，as the theories in coherence studies suggest，are also very important for the weaving of texture;the global level of coherence construction is even more important and indispensable than the local level.Therefore，in order to achieve a full understanding of the coherence construction in multimodal discourses，it is necessary and urgent to study the principles and operations of global coherence and surface cohesion in the multimodal sense.

Responding to the above requirements，the present chapter holds that the three-level division of coherence can provide a framework that allows for a systematic，coherent and in-depth analysis of the coherence construction in multimodal discourses since it combines distinct layers of analysis，namely the global，local and surface level in discourse coherence.

3.1.2.2　Adaptation to the framework

Multimodal discourses are different from purely verbal discourses not only in the number of semiotic modes involved，but also，and actually，more，in the“syntax”with which various information are organized in the two-dimensional page and accordingly，the way of perception.More specifically，the text-flow supports a linear unfolding of logical text organization and includes motivations for basic text-formatting options whereas the page-flow draws in the two-dimensional possibilities of the page for expressing rhetorical relations via spatial proximity and grouping（Bateman，2009）.Therefore，the move from linguistic to multimodal is a fundamental one.

Since the data in this book—the English learning advertisements are print materials and thus the involved semiotic modes are mainly the verbal and visual，this book will focus on the differences between the two modes.The meaning-making in verbal language and in images are fundamentally different.Each semiotic has its own specific（systemic）constraints and material affordances.The things we can do with language cannot all be done in visual representation，and vice versa.（Saint-Martin，1995：393，cited in Iedema，2003）.For example，information that displays what the world is like is carried by the image，constant with the logic of the visual as arrangement and display;written language，on the other hand，tends to follow the logic of speech in being oriented to action and event，and is thus oriented to the recording/reporting of actions and events and the ordering of procedures（Unsworth，2008：6）.Another difference in meaning expression between the two modes is that，pictures do not express certain clear propositions due to its lack of a strict syntax.Compared to written sentences which contain relatively unambiguous propositions，“a picture paints a thousand words”（Kaltenbacher，2004：124）.Therefore，the integration of images with language in multimodal discourses presents challenges to logocentric theories of coherence and the traditional primacy of language in texts.

The fundamental differences between linguistic texts and multimodal texts indicate that，the three-level framework of coherence，when extended from linguistic sense to multimodal sense，must undergo major modifications.Specifically，the three levels of coherence，global coherence，local coherence and surface cohesion must be adapted to fit the principles of meaning-making in multimodal discourses.

The adaptations will be done in two aspects：（i）in the aspect of definition，only slight modification in the wording，instead of fundamental changes in conceptualization，will be made since conceptualization of these terms are based upon social semiotic theory of language which applies also to other semiotic modes;（ii）in the aspect of the way of realization and analytical criteria accordingly，major modifications will be done since the monomodal meaning-expressing resource—language—is joined by resources from other semiotic modes.The adaptations made to global coherence，local coherence and surface cohesion in the two aspects will be elaborated in the following：

Global coherence，which has been defined as the macro-control of the discourse topic on the text in traditional linguistic sense，can still hold in the new multimodal situation，but the definition of discourse topic has to undergo some slight modifications to accommodate the complexity in semiotic expression.The new definition of discourse topic will be elaborated in the next section.In contrast to the slight change in the definition，the change in the way of realization will be drastic since a variety of visual resources such as images，graphs and typography are involved.That is，although the discourse topic whose nature is abstract and semantic can hold for both verbal and visual modes，the resources for its realization are multiple in the multimodal situation.The specific patterns of how global coherence is realized in the English learning advertisements will be described and discussed in Chapter 4.

Local coherence，which is defined as the semantic connections between sentences in the monomodal linguistic texts，will be changed to the semantic connections between adjacent visual elements and verbal elements at the local level of multimodal texts.The principles and criteria of analyzing these local coherence relations will be elaborated in detail in Chapter 5.

The change for surface cohesion is the most salient one among the three levels.Since the means of signaling relations between various parts in the surface level of multimodal discourses are largely typographical features instead of the linguistic means，the definition of cohesion will be adapted to the typographical features which signal the relations between the elements and parts in the multimodal discourses.The specific cohesive devices will be discussed in Chapter 6.