3.4.2　Empirical studies

The empirical research on the reader response to the coherence construction resources in the text is motivated by considerations of both theoretical and methodological aspects.

In the theoretical aspect，as reviewed in Chapter 2，coherence is not located within the text;the reader's mental processes also play an indispensible role in the construction of coherence.In other words，“coherence relations are only partly responsible for the perceived coherence of a text;the recognition of coherence relations by the hearer or reader enables them to assign coherence to a text”（Taboada，2006）.Therefore，in order to have an exhaustive investigation of the construction of discourse coherence，both text-internal features and the reader's mental activities during discourse processing need to be taken into consideration.In actual reading process，the two aspects are interwoven in an interactive，dynamic way.The features in the text，as input，can provide cues for the reader and facilitate his or her understanding;at the same time，the information in the text accumulated in the reader's mind will help construct a mental representation，or a schema of what type of the text is，of the genre，topic，theme and macrostructure.Together with the reader's background knowledge，this mental representation will guide the reader to understand the incoming information in the text，in the form of inferences，judgments and filtering.Therefore it is doubtless that“text comprehension is a strategic and constructive process whereby the reader uses information in the text as cues to the structure of the events，images，and inferences drawn from the text”（Jay，2004：273）.If only the textual aspect of coherence is considered，many phenomena in the discourse coherence cannot be eloquently explained.

This is especially true in the case of the English learning advertisements under analysis in this book.Since these advertisements are multimodal，non-linear discourses，their way of information organization will be very different from those linear verbal texts.Moreover，readers can choose from several possible reading paths through the pages instead of following a set path.Consequently，the way discourse coherence is constructed will rely less on the traditional textual cues in the verbal texts，and more on the reader's mental representation in the process of text interpretation.

In the methodological aspect，as above-mentioned and also widely acknowledged（see for example，Jewitt，2009：26），a major weakness of semiotic analysis is its subjectivity.Actually，any semiotic analysis is incomplete and partial because“the interpretation is socially constructed and a function of the hermeneutic，（pre-）understanding of the researchers”（Arnold et al，2001）.The negative consequence of the partial focus on textfocused approach is that the conclusions from text analysis are drawn from theoretical deduction and illustrated by selected samples rather than systematic analysis of a corpus，and cannot be verified by data from empirical research methods.The theoretical assumptions proposed by the text analysts might turn out inconsistent with the cognitive operation reflected in the actual behavior in reading.For example，Holsanova，Rahm and Holmqvist（2006）report that the evidence from eye-tracking data of newspaper reading confirm some assumptions on reading behavior and run against some others raised by semiotic analysts.Therefore，an empirical research on the reader's mental reaction in discourse processing will be especially useful.

For the reasons discussed above，an empirical research part on the reader responses on the coherence construction resources in the English learning advertisements is included in the current book.Specifically，this empirical part includes three component studies：an eye-tracking experiment，a memory recognition test and an investigation of the reader's evaluation of these advertisements.They aim to reveal different aspects of mental activities in discourse processing，namely attention，memory and evaluation.

They were carried out in UiL OTS（Utrecht Institute of Linguistics）at Utrecht University，the Netherlands，during June and July，2010.The three studies were conducted with the same group of participants during a focused period of time on the basis of the same set of materials which were the collection of English learning advertisements in the semiotic analysis.The participants first looked at the advertisements on the screen while their eye movement was recorded.About 10 minutes after the eye-tracking，the participants did the memory recognition test.The final task for the participants was to give their evaluation to the involved advertisements.Each participant spent approximately 30 minutes in the lab doing the three tasks.Therefore the results of the empirical part can be used as support for those in the semiotic analysis.

3.4.2.1　Eye-tracking experiment

Eye-tracking has long been known and used as a method to study the visual attention of individuals.Eye tracking analysis is based on the important assumption that there is a relationship between fixations，our gaze and what we are thinking about.It is a technology often used by psychologists to discover our cognitive and emotional patterns by measuring a person's point of gaze（that is，where they are looking）and determining eye/head position.More specifically，eye tracking works by reflecting invisible infrared light onto an eye，recording the reflection pattern with a sensor system，and then calculating the exact point of gaze using a geometrical model.

Since eye-tracking can provide detailed data on what is being looked at and when and how，it is very revealing in studying the cognitive processes involved the reading/viewing and comprehension of the discourses.As Richardson，Dale and Spivey（2007：324）summarize：

Crucially，this record provides data during the course of cognitive processing，not merely after processing is complete，as is often the case with more conventional measures.Eye-tracking data thus provide not only behavioral end products of our cognitive processes but also clues to the process through which they are achieved...Thus eye-tracking allows for a certain degree of ecological validity in task performance，as the responses it collects are ones that typically occur regardless of experimenters'instructions and participants'intent.

Furthermore，eye-tracking method is not only a good way for studying cognitive processes in reading;it is perhaps the best way compared to other experimental measures.The reason can be found in this paper again，

Eye movements exhibit a unique sensitivity to partially active representations that may not be detectable by most other experimental measures，or even result in any other overt behavior.Since eye movements are extremely fast，quickly corrected，and metabolically cheap，compared to other motor movements，they have a much lower threshold for being triggered.Hence，briefly partially-active representations—that might never elicit reading，speaking，or even internal monolog activity because they fade before reaching those thresholds—can nonetheless occasionally trigger an eye movement that betrays this otherwise-latent momentary consideration of that region of the visual display as being potentially relevant for interpretation and/or action.（Richardson，Dale and Spivey，2007：324）

Therefore，as they concluded，“in cognitive linguistics'concern with the wide array of perceptual，cognitive，and motor faculties that underlie language use，it stands to gain considerably from eye movement techniques”（Richardson，Dale and Spivey，2007：338）.Eye-tracking method will contribute to our knowledge of the actual looking behavior on multimodal discourses and provide ways of testifying the theoretical models and assumptions formed in the semiotic analysis.

Eye-tracking studies can improve theories of multimodal research（Hiippala，2012）.For example，in their study on newspaper reading，Holsanava，Rahm and Holmqvist（2006）find that semiotic or socio-semiotic analyses and eye-tracking studies can be mutually beneficial.If reading behaviors and reading paths are to be mapped in detail，semiotic analysis can be strengthened by being combined with the inductive approach of eye-tracking studies.Conversely，detailed data from eye-tracking and verbal protocols benefit from socio-semiotic analyses，where a deductive approach contributes to understanding.

Another motivation of using eye-tracking technology in the research on these advertisements lies in its possible implications for the theory of cognitive processing in multimodal discourses.As Rayner et al（2001）conclud at the end of their paper，“currently there is no cognitive process theory that makes clear predictions about eye movements and attention while viewing print advertisements”.Therefore the eye movement data drawn from the English learning advertisements have the potential to form the foundation in the search for such a theory.

In conclusion，it will be beneficial to provide empirical evidence for the deductive semiotic analysis through the inductive eye-tracking analysis.It is hopeful in the current study that the inclusion of eye tracking experiment will strengthen the conclusions drawn from the semiotic analysis of the advertisements，and the combination of the two methods will bring insightful implications for the understanding of how a coherent comprehension is achieved in the multimodal English learning advertisements.

3.4.2.2　Memory recognition test

The eye-tracking analysis focuses on the unconscious attention in online processing when readers look at the advertisements，whereas the memory recognition test focuses on the reader's memory of discrete semiotic elements in the advertisements after the reading.

The assumption underlying the memory recognition test is that the short-term memory of an element is closely linked to multiple factors，including the context，content and surface features like size，among others.These factors are linked to global coherence，local coherence and surface cohesion which form the analytical framework of coherence construction in this book.Therefore，through testing the memory recognition of the elements on the multimodal English learning advertisements，the respective effects of these three levels on the reader will be verified.

3.4.2.3　Evaluation

Compared to the above two empirical studies which focus more on the unconscious cognitive level of mind，the evaluation task reveals the reader's appreciation of the multimodal English learning advertisements which is more conscious.That is，the appreciation of the advertisements as complete artefacts is based on the evaluation of the various levels of representation.The assumption underlying the investigation of readers'evaluation is that the arrangement and mobilization of the various semiotic resources in an advertisement through which evaluation is formed are ultimately commanded by its features in global coherence，local coherence and surface cohesion in layout design.Therefore，the hypothesis is that the advertisements whose semiotic representation fits better its requirements in global coherence，local coherence and surface cohesion will be better appreciated.

【注释】

[1]The phrase“globally coherent”is mentioned when Reinhart is discussing the distinction between the linear structure and semantic structure，and that between linear coherence and global coherence in van Dijk（1972）.Thus the phrase actually refers to coherence in a common sense，i.e.，the semantic and pragmatic relations in the text.