2.2.5 Research on multimodality in China
Multimodality as a new research interest has been introduced in China(see for example,Li,2003;Zhu,2007)since the beginning of this new century.It immediately attracts the interest of the community of functional linguists in China.The number of studies contributing to multimodal discourse analysis is growing with an impressive speed.
Scholarly interest on multimodality in China began by the introduction of the important theories and practices in the field of multimodal discourse analysis in Chinese journals.For example,Li(2003)introduced Kress and van Leeuwen's(1996)framework of visual grammar.Zhu(2007)introduced the important theoretical foundations and methodology in the research of multimodality.Hu(2007)reviewed the evolvement of the multimodality theory and discussed some important issues in the multimodality as a research area.Wang(2010)and Zeng(2010)reviewed the researches on visual-verbal relations in multimodal texts.Wang's major concern was the current state and the problems whereas Zeng classified the existing studies on visual-verbal relations into two groups.
Until now,the research of multimodality is generally in the phase of application(see also Cheng and Zhang,2017).Many Chinese researchers are attempting to apply the theoretical concepts and principles of multimodal discourse analysis in a variety of genres in Chinese context.These research efforts can be generally grouped into two types:those on media genres and on educational genres.Among studies on media genres,a large number of sub-genres are studied,including webpages of the People's Daily(Ye,2006),medical reports during the SARS period(Cheng and Tian,2006),Olympic bidding emblem(Zheng and Zhang,2008;Zeng,2017),political cartoons(Zhao and Feng,2017),newspaper report(Wei,2008),film sub-genres such as TV news report(Yi,2015),Istanbul Olympic bidding video(Zhou and He,2016)and Chinese Communist Party image promotional advertisement on CCTV(Chen and Liu,2017).
Apart from media genres,educational genres also attract abundant research attention since many researchers in the multimodality filed work as foreign language teachers at universities.These educational discourses under scrutiny include Chinese science textbooks used in primary schools,middle schools and colleges(Chen and Wang,2008),multimodal English textbooks(Chen and Liu,2016),multimodal essays(Hu,2010),college English classroom teaching(Zhang and Wang,2010),academic English discourse construction of Chinese postgraduate students(Li and Wang,2015),English writing by Chinese college students(Wang,2016),translation and interpretation teaching(Liu,2017;Xu,2017)and other genres related to various kinds of teaching activities.
Besides introduction,review and application,some Chinese scholars also make active efforts to develop the theoretical and analytical concepts and frameworks in multimodal discourse analysis.For example,Zhang(2009)set up a synthetic theoretical framework for multimodal discourse analysis on the basis of systemic functional linguistics.The framework mainly consisted of multimodal discourse systems at five levels:(i)The level of context of culture,(ii)the level of context of situation,(iii)the semantic level,(iv)the formal level and(v)the level of media(substance).This framework was further elaborated in Zhang and Wang(2016)with slight change in terminology and a more expanded and detailed design,such as the third level was changed to“interactional discourse”and the fourth level was changed to“mode”,but the connotation and structure in these levels remained more or less the same.However,as Cheng and Zhang(2017)admit,due to the immaturity in theorization,especially the difficulty in breaking through theoretical obstacles in inter-semiotic synergy,the research on multimodality is facing a cooling-down in China after a boom during 2009-2013.
Compared to theoretical pursuits,the research on methodology in multimodal discourse analysis has not received much attention yet.Gu(2006,2007)and Feng(2010)made research efforts in this regard.Gu's(2006)major concern was the multimodal data which consisted of video streams with synchronized sounds.He outlined and demonstrated a corpus linguistic approach to multimodal text analysis that started from the analytic unit of social situation,to that of activity type,task/episode,and the participants'behavior of talking and doing.After analyzing the relationship between multimedia learning and multimodal learning,Gu(2007)put forward an analytic model for analyzing the two forms of learning.Five hypotheses concerning multimedia and multimodal learning were formulated and discussed.He also outlined how to use agent-based modeling language to data-mine the multimedia and multimodal interactions for computer simulation.Feng(2010)introduced the use of computer software in annotating,describing and analyzing different social semiotic modalities,such as language,sound and visual image.It also showed how software could manipulate video discourse in different ways,such as segmentation,summarization and searching.
In the past decade,some researchers in the multimodality field have discovered the strengths of psychological research methods such as eye-tracking and think-aloud techniques in the exploration of reader reaction of multimodal texts(see for example,Holsanova and Holmqvist,2006;Hiippala,2012).The journal Visual communication published a special issue in August 2012“Methodologies for Multimodal Research”in which several papers employed eye-tracking measurements to address issues in multimodality.However,these studies mainly attempt to explore the general patterns in multimodal discourse processing,such as scan path and attention allocation whereas some deeper issues such as visual-verbal coherence are largely left unsettled.
To summarize,multimodal discourse analysis is a relatively new research interest among the functional linguists in China.Its fast growth and good applicability have been shown in the quickly rising number of studies in this field.However,its development is yet far from mature and need more research efforts in creating valuable insights in theorization,methodology design,application in more diverse genres,and assessment.