2.2.3　Frameworks of visual-verbal relations

As mentioned above，the research on visual-verbal relations are largely inspired by Barthes'conceptualization，the systemic functional linguistics and Kress and van Leeuwen's visual grammar.In the following，the recent works contributing to the building of frameworks for systematically analyzing visual-verbal relations will be reviewed.

Inspired by the cohesion theory of Halliday and Hasan（1976）and Halliday and Hasan's（1985）lexical cohesion，Royce（1998）formulates the concept of inter-semiotic complementarity and established a system of Multimodal Sense Relations for ideational meaning-making across linguistic and visual choices.He held that the inter-semiotic sense relations shared some characteristics with the linguistic relations，such as repetition，synonymy，antonymy，hyponymy and collocation.

Djonov（1999）adopts systemic functional theory，Kress and van Leeuwen's（1996）visual grammar and the concept of“foregrounding”as the guiding principles to describe the degree of overlap between the experiential，interpersonal and textual meanings realized in the copy（the verbal part）and the display（the visual image）of consumeroriented product advertisements.The relationship between two modes that are used to construct a single text could then be examined in terms of the degree of overlap or even dissonance between the features foregrounded by each mode.

Kress&van Leeuwen（2001）studies how the verbal and visual elements interact in multimodal texts also from the semantic perspective：they may reinforce each other by carrying the same message in different ways，they may fulfill complementary roles by providing extra information about the same thing or they may be hierarchically ordered，meaning that one mode is dominant over the other in the interpretation of a text.

Figure 2.3　Djonov's（2005）network of Logico-Semantic Relations in Hypermedia

Djonov's doctoral dissertation（2005）includes a section on the discussion of visualverbal relations in hypermedia discourse.Her framework（see Figure 2.3）draws upon Halliday's（1994）categories of logical relations in clause complexes，although some subbranches are made on the basis of semantic labeling between different modes，such as exemplifying，generalizing，particularizing，and similarity.

Martinec and Salway（2005）（see Figure 2.4 and Figure 2.5）draws on systemic functional grammar to classify the relations between image and text in terms of the status relations between image and word（equal or unequal）and the nature of the semantic linkage between elements（logic-semantic relations）.Inequality in status means that the two modes stand in a dominance-subordination relationship.When the two modes are equal in status，their logic-semantic relations are more finely conceptualized using Halliday's（1994）categories for analyzing logical relations in clause complexes，like elaboration，extension and enhancement.

●Status relations

Figure 2.4　Status relations of text-image relation system（Martinec and Salway，2005）

●Logic-semantic relations

Figure 2.5　Logic-semantic relations of text-image relation system（Martinec and Salway，2005）

Unsworth（2006）adopts the SFL's social semiotic perspective towards text-image relations and describes them according to the three types of meanings respectively—ideational，interpersonal and textual.However，a systematic framework is raised only in ideational meaning.In terms of ideational meaning，this language-image interaction may be characterized as ideational concurrence，complementarity or connectio n.Each of them has a finer categorization（see Figure 2.6 for illustration）.

Figure 2.6　Ideational meanings at the intersection of language and image（Unsworth，2006）

Matthiesen（2007）attempts to theorize the meaning-making in multimodal discourses with concepts of systemic functional linguistics，arguing that the linguistic and other semiotic resources in a multimodal text share the strata of overall context and semantics，and differ only in the lexico-grammatical stratum.He regards the printed page as a rhetorical mode and is“concerned with division of socio-semiotic labor—with the division of labor between semiotic systems and social systems within a given context，with the division of labor between field and tenor as the main source of organization，and with the division of labor among language and image in the multimodal text and also with the relations that link language and image together”（ibid：32）.The text-image relations are“rhetorical relations”which，in his view，refer to relations of projection and expansion in Halliday's framework of clause complexes.

Liu and O'Halloran（2009）addresse the problem of cohesion and coherence between different modes in multimodal discourses.They first summarize the multiple systems developed by different people and re-organize them into the meta-function framework of Systemic Functional Linguistics（see Table 2.6）.Then based on the examination of Inter-Semiotic Parallelism，Inter-Semiotic Polysemy，and a discourse-based Implication Sequences，the authors propose a“discourse-based”way to approach the different kinds of logical relations between language and images in multimodal discourses“to complement existing grammar-based models（e.g.，Martinec and Salway，2005;Unsworth，2006）”.Their model of inter-semiotic logical relations is shown in Figure 2.6.

Table 2.6　Summary of inter-semiotic cohesive devices in multi-semiotic texts（Liu and O'Halloran，2009）

Table 2.7　Liu and O'Halloran's（2009）model of inter-semiotic logical relations

Bateman（2008）offers an obviously different framework from the above-mentioned ones in that：（i）it draws on RST instead of Halliday's framework of clause complexes，to account for the relations between visual and verbal modes in multimodal texts，and（ii）he puts great emphasis on genre.He proposes a multi-layered framework for the in-depth analysis of multimodal documents.In this model there is a rhetorical base which is defined as“a detailed account of the rhetorical relationships between the content expressed by elements on a page and their communicative purpose”（ibid：108）.It is in this base that visual-verbal relations are discussed in detail.For this purpose he extends Mann and Thompson's Rhetorical Structure Theory from purely linguistic texts to multimodal documents so as to encompass multimodal rhetorical organization.

In summary，Halliday's theories of systemic functional linguistics have been proved to be very adaptable and flexible when they are extended to multimodal discourses.However，despite the fruits and huge theoretical and analytical implications，there are some limitations in the current studies on visual-verbal relations.The popular practice to conceptualize the inter-semiotic relations on the basis of Halliday's framework of clause complexes especially needs to be viewed with caution.