3.1.3　A three-level framework of coherence construction in multimodal discourses

The previous sections suggest that the framework for analyzing coherence construction in multimodal discourses will be based on the fusion of the coherence theories in linguistic texts and the special features of multimodal discourses.That is，one the one hand，the concept that discourse coherence should be analyzed in three levels，global，local and surface will be extended to multimodal discourses，and on the other hand，necessary and even major adaptations should be made to meet the requirements of the semiotic complexity in multimodal discourses.Specifically，the establishment of the framework for analyzing coherence construction in multimodal discourses in this chapter is based on considerations in the following three assumptions：

First of all，the three-level division of coherence in the linguistic sense is also reasonable in the multimodal sense.As Louwerse and Graesser（2005）acknowledge，“a complete theory of discourse coherence requires a harmonious layering of several levels，including vocabulary，sentence structure，meaning，discourse context，style，and world knowledge”.Therefore，the model of coherence construction in multimodal discourse will also be a multi-layered one.The central concern in this book is how coherence is built between different semiotic modes，specifically，the verbal and the visual.

Secondly，the definitions，working principles and criteria of the global coherence，local coherence and surface cohesion should be modified to fit the features of multimodal discourses.

Thirdly，because coherence construction cannot be totally completed within the text，but in the collaboration of textual resources and the reader's mental processes in comprehension，as the review in Section 2.1 suggests，a framework that is based exclusively on the text-internal mechanisms of coherence construction will be an incomplete one and run the risk of failing to provide explanations for some of the phenomena and questions concerning the meaning-making and functioning of texts，including both linguistic ones and multimodal ones.Therefore，the framework will take the reader's role into consideration.That is，the reader's mental mechanisms and activities during discourse processing will be added into the framework.

The framework for analyzing coherence construction in multimodal discourses is visualized in the following figure.

Figure 3.1　Framework for analyzing coherence construction in multimodal discourses

As shown in Figure 3.1，there are three levels of analysis，namely global coherence，local coherence and surface cohesion，in the coherence construction of multimodal discourses.When the three-level division is extended from coherence theories from purely verbal texts to multimodal discourses，modifications are made with references to the features in multimodal discourses.

The definition of global coherence as the macro-control of the discourse topic on the whole text can apply to both purely verbal texts and multimodal texts，but the discourse topic needs to be modified in the new multimodal situation.Its definition in the linguistic sense as“the overarching semantic proposition which all the propositions in the text contribute to in some way”is too narrow to accommodate the multiple semiotic modes.Because images are weak in expressing propositions，the word“proposition”should be changed to more general hyponyms，such as conceptual structure or information.The broadened definition of discourse topic will be“the overarching conceptual structure，which all the information in the text contributes to in one way or another”.The patterns of the semiotic configurations by which the discourse topical structure is expressed will be drastically different from those in purely verbal texts since various visual resources，such as pictures，charts and tables，are taken into consideration.

Local coherence is concerned with the semantic connections between adjacent verbal elements and visual elements in the local scope of discourse.Since the English learning advertisements are multi-section discourses with a hierarchical rhetorical structure，the local scope in this book will be the unit of individual sections under the whole discourse.Local coherence is semantic in nature，and not necessarily explicitly signaled by cohesive devices.This book holds that，the set of relations in Rhetorical Structure Theory or Halliday's clause complexes，and the frameworks of visual-verbal relations developed accordingly，as reviewed in Section 2.2.3，are based on the linear information organization in verbal texts，so they are not suitable for the visual-verbal relations in advertisements which are often unconventional and difficult to label and pin down.Therefore，new working principles of identifying and analyzing local coherence will be developed specifically for the English learning advertisements.

As for cohesion，its function and place in the whole coherence system discussed in the linguistic sense are maintained in the multimodal sense.However，its means of realization are totally different because obviously the linguistic means of cohesion cannot work for the two-dimensional multimodal advertisements.Therefore，cohesion in this book will be defined as the typographical features in the layout that serve to signal the logical and semantic relations between various elements and parts in the discourse.The definition of surface cohesion here is proposed on the basis of a large body of documentation on the contribution of layout to the construction of coherence in multimodal discourses.Specifically，it has been found in many studies that the typographical features in the layout affect the establishment of coherence.For example，Stöckl（2005）argues that，meanings constructed from typography may support，reinforce，reinterpret or contradict verbally construed messages;at any rate，these formal and semantic interrelations are intended and aim to create a holistic entity.The reasons for the definition of surface cohesion in a multimodal sense will be elaborated in greater detail in Chapter 6.

The three levels，global coherence，local coherence and surface cohesion，operate in different manners：the global coherence operates through a top-down hierarchical topical network;local coherence is the force drawing adjacent elements together into visual groups or clusters，cohesion in the layout maps the semantic and rhetorical coherence relations onto the two-dimensional visual space.The three levels constitute a unified system in which resources from different semiotic modes are integrated，connected and glued to form a coherent representation of multimodal discourses through semantic and formal means.

The three levels of coherence do not have the same priorities，as discussed for purely verbal texts.First of all，the global coherence has the highest priority.Without the global coherence，the discourse would be a composite of visual and verbal elements placed together that add up to nothing.Actually there are very few such nonsense discourses in natural circumstances which are well-connected between discrete elements but do not contribute to a discourse topic.Weidenmann et al（1999）find that learners show a stable preference for the top-down structuring and sequencing in the presentation of multimodal learning materials，compared to the bottom-up and the presentation as a whole.The top-down sequence in this experiment refers to the presentation of an overview followed by the macro topics one by one.This arrangement is consistent with the conceptualization of discourse topical structure in this chapter.The result indicates that，global coherence，realized through discourse topical structure，should be the automatic cognitive mechanism underlying human practices of discourse processing.

Local coherence is also an indispensable guarantee for the hanging-together of the text and the reader's comprehension.This has been supported in many empirical studies in multimedia learning.When the meaning in pictorial illustrations is consistent with that in the verbal text，a better comprehension will be achieved and the learning result is improved than the verbal text-only version（see for example，Waddill and McDaniel，1992;Carney and Levin，2002;Kools et al，2006）.Reversely，if the picture and verbal text express unrelated topics，or if there is a mismatch between the picture and verbal text in meaning，the comprehension will be impaired（see for example，Mayer et al，2003;Schnotz and Bannert，2003）.

Cohesion，operating on the surface level of discourse，enhances the semantic unity of the text and facilitates comprehension.It is the last one in priority compared to the other two semantic oriented levels of coherence.Empirical studies have shown that the surface level features in the layout can be easily overridden by semantic features in cognitive processing（see for example，Nyström and Holmqvist，2008）.

The difference among the three levels of coherence in priorities does not mean that global coherence is important and surface cohesion is trivial.Instead，each of them plays a significant role in the construction of coherence in the text and in the reader's mental comprehension and only the degree of their importance and indispensability differs.In actual cases the three levels operate simultaneously，and often in a collaborative way.A well-formed discourse has to make use of all the three levels.

This framework provides a way for a systematic and layered analysis of the way multimodal texture is constructed which is beyond the analysis of discursive aspects of multimodal discourses.Therefore it can respond to the limitation in existing studies on multimodality that“the methods employed are often not up to the complexity of the artefacts analyzed and，as a consequence，the results obtained are still too often presented in idiosyncratic ways”（Bateman，2009）.

What needs to be noted is that，the texture in any natural semiotic artefacts，including purely verbal ones and multimodal ones，is a unified whole.What this framework tries to do is to extract analyzable levels from the whole.Therefore，it will be natural if cases arise where no definite boundaries，labels，or classification can be given.In other words，the separation of the three levels is out of the convenience for analysis，instead of the nature of the texture or coherence construction under discussion.