2.2.2　Works on theorizations of inter-semiotic relations

The relationships across and between modes in multimodal texts and interaction are a central area of multimodal research（Jewitt，2009：17）.Substantial theoretical conceptualizations and descriptions of the dynamics of interaction between image and language have been offered.These research efforts can be roughly classified into two groups according to the theoretical origins and approaches：specifically，those based on Barthes'（1977）and those based on systemic functional theories of language.

2.2.2.1　Works in the Barthes'tradition

The first influential view on inter-modal relationship was given by Barthes（1977），which distinguished two functions the linguistic message may have in relation to the pictorial part：anchorage and relay.The former refers to the function of linguistic message when it guides the identification and the interpretation of the pictorial components of the image.Because images are by nature“polysemous”，implying“a‘floating chain’of signifieds”（Barthes，1977：39），language is needed in order to fix both the denoted and the connoted meanings of the visual by identifying and interpreting what the image is showing.The latter means that“language and image stand in a complementary relationship;[whereby]the words，in the same way as the images，are fragments of a more general syntagm and the unity of the message is realized at a higher level”（ibid：41）.The anchoring function of language is claimed to be more common than relay.

Bill Nichols（1976/1981，cited in van Leeuwen，2005：230）proposed a finer classification of inter-modal relationship（See Table 2.5）.This model developed Barthes'notions“anchor”and“relay”by adding several more categories.

Table 2.5　Visual-verbal relations by Nichols

In summary，although the concepts of Barthes provide an important and useful starting point，they give verbal message a clear preference over the visual and effectively deny the possibility of an image having its own，independent structure and meaning（El Refaie，2003）.Their appropriateness for the analysis of contemporary advertising is conditioned because advertising has become much more complicated（Forceville，1996）.Therefore，the tendency in the study of text-image relations is that，more elaborated frameworks of text-image relations are developed drawing upon the SFL theories of language，with the semantic labels in Barthes'terminology frequently embedded.

2.2.2.2　Works in the SFL tradition

The prevalent tenet underlying these works is to extend the meta-functions in Hallidayan Systemic Functional Linguistics to the conceptualization and analysis of modes other than language and the inter-modal interaction.Many researchers exploring imagetext relations explicitly acknowledge the grounding of their work in the SFL metafunctional hypothesis（Unsworth，2006）.In this sub-section the emphasis will be placed on the works on visual-verbal relations that are applicable in static print discourses.

The first breakthrough in this field is the visual grammar proposed in Kress and van Leeuwen（1996）.It draws on the metafunctional principle of SFL and extends it into a systematic visual grammar.The ideational，interpersonal and textual meaning in linguistic analysis correspond to representational，interactive，and compositional meaning in image analysis respectively.For each of these meanings，the realization resources are identified and systematically described（see Figure 2.2 below）.

Although not directly dedicated to text-image relations，this work provides a systematic and unified framework for analyzing meaning in both language and image in multimodal discourses.It makes possible for analysts to draw implications on the meaning-making in verbal texts and images in the same pieces of multimodal discourse in both comparative perspective and collaborative perspective，so it has been widely applied to the analysis of text-image relations in multimodal discourses.A large body of work has been produced with the implications from their framework.It is becoming increasingly definitive of approaches to text analysis and meaning making（Iedema，2003）.

Stöckl（2004）believes that the inter-modal relations like those in the language-image link are anything but simple and pose both theoretical and analytical problems.Despite the complexity，he has managed to sketch out some basic semiotic principles operating across the language and the image in printed media：（i）the three Hallidayan meta-functions，（ii）segmentation，that is the decomposability of larger sign structure or gestalt of perception into their constituent elements，（iii）three interrelated planes of meaning，namely denotation，connotation and association，（iv）semantic relations between concomitant modes，（v）three ways of meaning-making，namely iconic，indexical and symbolic，and（vi）gestalt similarities，that is，inter-modal analogies on both the formal and semantic level.

Figure 2.2　Summary of Kress&van Leeuwen's（1996）framework for visual grammar（reproduced after Almeida，2009）

Van Leeuwen（2005）explores how different semiotic resources were integrated to form multimodal texts and communicative events via rhythm，composition，information linking，and dialogue.

Jewitt points out that the interaction between modes is essential to the meaning of the multimodal texts（Jewitt，2009：25）.When several modes are involved in a communicative event（e.g.a text，a website，a spoken interchange）all of the modes combine to represent a message's meaning.The different aspects of meaning are carried in different ways by each of the modes in the ensemble.Any one mode in that ensemble is carrying a part of the message only：each mode is therefore partial in relation to the whole of the meaning.Multimodal research attends to the interplay between modes to look at the specific work of each mode and how each mode interacts with and contributes to the others in the multimodal ensemble.At times the meaning realized by two modes can be“aligned”，at other times they may be complementary，and at other times each mode may be used to refer to distinct aspects of meaning and be contradictory，or in tension.

Apart from the above mentioned works，new theoretical concepts or principles have also been proposed with the attempt to offer alternative perspectives to the study of intermodal relations in multimodality.For example，Iedema（2003）advances the notion of resemiotization as a complementary perspective to that of multimodality.Resemiotization is about how meaning making shifts from context to context，from practice to practice，or from one stage of a practice to the next.It reflects a sociohistorical perspective to multimodal discourse analysis.Norris（2009）introduces two concepts for thinking about these relationships，namely modal density and modal intensity.Unsworth and Cléirigh（2009）discuss image-text relations and critically engage with early work emphasizing the distinctively different semiotic affordances of image and language and propose a semiotic framework designed to address the synergistic nature of image-language interaction in meaning construction.