Research Article
Print
Research Article
Meaning construal through multimodal clusters in the theatrical discourse
expand article infoElena Loginova‡
‡ Ryazan State University, Ryazan, Russia

Corresponding author: Elena Loginova ( lenlogiya@gmail.com )

Academic editor: Maria Kiose
© 2022 Elena Loginova.
This is an open access article distributed under the terms of the Creative Commons Attribution License (CC BY-NC-ND 4.0), which permits to copy and distribute the article for non-commercial purposes, provided that the article is not altered or modified and the original author and source are credited.
Citation: Loginova E (2022) Meaning construal through multimodal clusters in the theatrical discourse. Languages and Modalities 2: 27-36. https://doi.org/10.3897/lamo.2.78572
Open Access

Abstract

The paper presents the results of an empirical study of monomodal and multimodal data collected from the plays by Russian playwrights of the second half of the 20th century (A. Vampilov, A. Kazantsev, N. Sadur) and the theatrical performances based on these plays. The main aim is to find out what semiotic modalities are more frequently involved in the process of recurrent semiosis and, as such, in the multimodal meaning construal. Following the classical as well as contemporary works in the field explored, the research sets out to elaborate the cognitive-semiotic approach to the analysis of the theatre-specific means of multimodal meaning construal to get insights into dynamicity, embodiment and figurativity of artistic expression typical of theatrical performativity and inaccessible to other arts. The data are explored by means of a complex methodology that comprises quantitative and qualitative methods with the diagraph analysis as a key analytical tool which serves to recognize meaningful correlations within and across modalities resulting from mappings between dialogically juxtaposed independent utterances. This paper demonstrates that the involvement of nonverbal resources is mainly manifested in the use of hand and head gestures, facial movements and body repositioning. Next to the quantitative results, the research shows that co-occurring conceptualizations through verbal and nonverbal components, gestures in particular, can trigger off the amplification of meaning or bring about the emergence of new conceptual projections.

Key Words

diagraph, meaning construal, multimodality, resonance, theatrical discourse

Introduction

I started to analyze plays of British, American and Russian playwrights several years ago sharing the view that in this kind of discourse we deal with mediated perception, a secondary process derived from and based on the primary experience of direct perception, according to J. Gibson’s “ecological” approach to cognition [Gibson 1979]. A related distinction between primary and secondary semiosis (also, primary and secondary modelling systems [Barthes 1989; Lotman 2002]) arises concerning two quite dissimilar – although intimately connected – types of communicative event: the monomodal composed for the theatre and the multimodal produced in the theatre. K. Elam defines them as the written, or “dramatic text” and the theatrical, or “performance text” [Elam 2002].

These two potential focuses of cognitive-semiotic attention account for what can be termed as the dynamism and transformability of the sign, when it happens to be a modification in the secondary material vehicle and/or the mental concept, i.e., a change from the meaning expressed in one mode to the meaning expressed in another mode. In this paper, the term mode which stands for “a semiotic resource for making meaning” [Kress 2010: 79] is used as synonymous with the term modality. The process of modality transformation (also “transmutation” [Jakobson 2004]; “transcodification” [Elan 2002]; “intersemiotic transposition” [Sonesson 2014]) depends on the actors and the director’s interpretative engagement in the interaction with the message made by the playwright; and is targeted at increasing the emotional impact of the “dramatic text” as well as amplifying its meaning.

In this research my task is primarily of quantitative nature. It boils down to the analysis of Russian plays of the second half of the 20th century (A. Vampilov, A. Kazantsev, N. Sadur) and their theatrical interpretations in order to specify what semiotic modes are more frequently involved in the process of recurrent semiosis. The multimodal data are explored with the help of a statistically verified multimodal analysis [O’ Halloran 2011] which can shed light on the process of meaning arising from the integration of language and other meaning-making resources.

The paper is organized as follows: as a starting point I present a brief overview of the main areas of contribution to the present-day multimodal studies. I revise the issue of semiotic modality relying on the notion of theatrical codes in the works by E. Fisher-Lichte and K. Elam. Special attention is given to gestural modality and the types of gestures offered by A. Cienki and C. Műller. Given the theoretical orientations, I formulate the objectives and working hypotheses. After that, I describe the material, methods and tools used in the research. Then I present a quantitative and qualitative discussion of the samples selected for the analysis, zooming in on the major findings which refer to the theoretical framework outlined. Finally, I discuss the implications of the study in light of the multimodal meaning construal in human communication and in the arts as communication and also present some suggestions for further research.

Theoretical framework

Semiotic modalities of a theatrical performance

Recent research in a range of disciplines, including cognitive psychology, linguistics, semiotics and poetics, has focused on the ways in which people pair verbal language with nonverbal expressions, be it in comics, cartoons, memes, demotivators, advertisements, face-to-face interaction, and more. Recognition of multimodal meaning construction mechanisms in spoken discourse is the task tackled by scholars representing different fields of inquiry such as manual co-speech gestures in various types of discourse [Cienki 2013, 2019; Iriskhanova 2018, 2021; Leonteva 2017; Kiose 2021; Zykova 2021]; the interplay of eye gaze, speech, hand and head gestures [Brône et al. 2013; Kibrik, Fedorova 2018; Sonesson 2019]; intonation and also its correlation with eyebrow movement [Yanko 2019; Tsibulya 2021], etc. Corpus-based empirical analysis along with quantitative methods, annotation tools (ELAN, LASG), software programs (Eye-tracking, MoCap, Speech Analyzer, MS Excel, etc.) has become a valid instrument to explore the verbal and non-verbal forms of conceptualization, contributing remarkably to the study of embodiment, iconicity, figurative meaning as well as the semiotic status of gestures and other meaning-making resources in spoken discourse [Zlatev 2007; Sonesson 2010; Oben, Brône 2016; Cienki 2017; Dancygier 2017; Iriskhanova, Cienki 2018; Műller 2019]. The focus on the orchestrations of different semiotic modalities (speech, prosody, gesture, eye gaze, facial expression) help the researchers demonstrate certain specificity in face-to-face interaction, professional activity (e. g., simultaneous interpretation, courtroom or business communication) and also in the cinematic discourse. Surprisingly, nowadays little attention in exploring multimodal meaning construing is given to a theatrical performance which is a vivid example of collaborative multi-media display.

The pioneering works which radically changed the prospects for the scientific analysis of the theatrical performance include books and articles by the Prague School semioticians: J. Mukaŕovskĵ, O. Zich, P. Bogatyrev, R. Jakobson, etc. Their ideas of the theatrical sign as a culturally and socially determined unit of meaning which points beyond the denotative basis to the implicitly represented content (cf., Bogatyrev’s “a sign of a sign and not the sign of a material thing” [Semiotics of Art. Prague School Contributions 1976: 33]) were elaborated by K. Stanislavsky, V. Meyerhold, A. Tairov, J. Grotowski and other theatre directors, who acknowledged the power of actual visual perception. They argued that the actor’s work does not happen without a sense of the body. The processes by which actors move the language of the plays off the pages and into their bodily actions, so that it can be shared with the audience, are rooted in the physical and the sensual, as much as they are in words. And it is exactly for this reason К. Stanislavsky described a successful performance as “the artistic embodiment of inner emotional experience” [Stanislavsky 1964].

Embodied construal of meaning linked to multimodal behavior has defined а theatrical performance since the origins of theatre in prehistoric rituals. Characters, be they gods, people or animals, have always been represented through the bodies of actors with puppet theatre as an exception [Hart 2011]. However, while speaking about embodiment in the context of theatrical signs one is tapping into the scale of embodiment: from the primary level when the body finds its conceptual and expressive forms (the primary cognitive materiality of mind and body) to the secondary level behind which is a focused intentionality that participates along with the body in creating a performance (the second materiality of a theatrical performance as the manifestation of the author’s intent). In the opinion of F. Hart, being a cognitive phenomenon, embodiment creates the instruments of communication, e.g., gestures, through which “a secondary order of constructivity becomes activated” [Ibid: 32].

In this paper, I intend to show that co-existing heterogeneous signs of a theatrical performance converge to enrich the meaning projections and achieve the pragmatic effect required. I proceed from the assumption that a play and its theatrical interpretation can be considered parts of a unique communicative space: the original communicative situation and its multimodal “derivative” respectively. The list of modalities involved in the process of meaning-making in the multimodal theatrical performance includes, but is not reduced to, facial expression, gesture and other movements related to body parts, stage decoration and stage conception, costume, hairstyle, music, lighting (see also the classifications of theatrical codes in [Fisher-Lichte 1992; Elan 2002]). According to E. Fisher-Lichte, all types of signs in a theatrical performance can be classified in terms of general categories based on the following oppositions: acoustic/visual; transient/lasting; actor-related/space-related. The classification of all the theatrical signs elaborated by E. Fisher-Lichte can be seen below (Table 1).

Table 1.
Download as
CSV
XLSX

Signs of a theatrical performance.

Sounds acoustic transient space-related
Music
Linguistic signs actor-related
Paralinguistic signs visual
Mimic signs
Gestural signs
Proxemic signs
Mask long-lasting
Hair
Costume
Stage conception space-related
Stage decoration
Props
Lighting

Source: Fisher-Lichte (1992, p. 18).

The multimodal derivational discourse draws simultaneously upon cognitive processes from across the spectrum given, involving visual and auditory perception together with verbal cognition. So, one and the same concept can be represented by verbal means (in the source discourse) and multimodal clusters of verbal and non-verbal means (in the discourse derivative). What results is a performance founded on the cultural norms of society at large and on secondary regulative rules, peculiar to the theatre as a type of multimodal discourse opposed to all other types both in terms of ontological status of the creative work it involves and with regard to the conditions for its production and interpretation.

Gestural modality and communication on stage

The special nature of theatrical signs that stems from the special ontological status of the performance allows for yet another approach to be taken into consideration. Multimodality on stage is an adaptive process which results in the strengthening of meaning projections between the semiotic resources used. By simultaneously or successively representing complementary and supplementary information in various modalities, resonance arises to highlight or enhance the content of the original communicative situation. This phenomenon cannot be attributed to any sign in isolation. It is a property of discourse relations between verbal or verbal and nonverbal signs, gestures in particular [Du Bois 2014; Iriskhanova 2018; Loginova 2019]. Naturally, this aspect is of interest to the current research.

What I intend to show with my analysis is that meaning construal in the theatrical discourse depends largely on resonance as a kind of interactive alignment which accounts for pragmatic effects and, as such, for communicative efficiency. The analysis concentrates on resonating gestures, particularly actors’ manual movements (iconic or metaphorical) linked to their speech.

For the purposes of the research, I use the classifications of speech-linked gestures offered by A. Cienki and C. Műller. Their classifications include referential gestures which are subdivided into deictic (pointing/touching) and representational gestures. The representation gestures are further specified according to the mode of representation: enacting, embodying, holding/touching, tracing, molding gestures. The other types encompass pragmatic gestures which show one’s stance on the issue under discussion; adaptors (self-adaptors and object-adaptors as non-speech-motivated movements) and beats [Műller 1998; Cienki 2013]. These types of gestures are the ones that can be used in different communicative contexts and different discourses, including a theatrical performance for which gestural signs are of particular importance. As it is noted by E. Fisher-Lichte, “while theatre is certainly possible without language, music or sounds, costumes, decorations, props, or lighting, no form of theatre can completely dispense with the actor’s physical presence, his gestural signs” [Fisher-Lichte 1992: 39].

Generally, it may be a problem for researchers to divide up speech-linked gestures into separate groups. It is so because different pragmatic gestures (e.g., the throwing away gesture to express a negative stance or the palm-up open hand used when presenting an idea) might have the same metaphor in their core being based on mimetic schemas or image-schemas [Cienki 2021]. One of the solutions to the problem of gesture identification is to assume that every gesture involves this or that degree of representational and pragmatic value on a scale from low to high, to display the discursive “profile” of the gesture in a particular usage event. This idea originates in the “semiotic profiles” of gestures, i.e., variable sets of semiotic parameters shaped in concrete contexts, described by O. K. Iriskhanova and A. Cienki. The scholars argue that flexibility of gestures and variability of their semiotic features make them more language-dependent and more exposed to individual differences [Iriskhanova, Cienki 2018].

The situation becomes more complicated when the communicative context analyzed is that of a secondary modelling system in which different functions of gestures can be realized in parallel to allow richer forms of expression. Furthermore, as a rule, gestures in the arts as communication are not characterized by immediacy, nor do actors on stage engage in spontaneous talk. Every verbal and nonverbal sign are the result of rehearsal so that certain images gain intensity and become appealing. How strictly bodily actions in theatrical discourse are regulated depends upon the strength of the conventions at work. E.g., in the classical Chinese and Japanese Noh theatres, the verbal and nonverbal behavior is strictly predetermined. In conventional Western theatre the secondary order embodiment is less tightly constrained with the primary goal for actors to make their choices emotionally charged. Theatrical gestures are regarded as showing body eloquence. Traditionally, they were classified into rhetorical gestures based on antique oratory (motions of the hand and fingers) and gestures-affects used to express surprise, disgust, remorse, censure, etc. The detailed description of these groups, with 24 codified gestures in each, was given in 1646 by J. Bulwer in his treatise on gesture which appeared to be the first book dedicated to the “natural language of the hand” [Kendon 2004: 25–28].

Given this stance, the objectives I have set myself are twofold. On the one hand, I am going to find out what modalities are more frequently realized in a theatrical performance simultaneously with speech and under the guidance of a single communicative intention. On the other hand, the focus of my attention is on the multimodal cluster – speech and gesture – to reveal the functions of gestural component in meaning-making and create a gestural profile for gestures used on stage.

I hypothesize that: a) the gestural modality together with facial movements would be the most active in multimodal meaning construal in the theatrical discourse; b) the pragmatic and representational gestures would prevail in the theatrical discourse; c) gestures in the theatrical discourse would complement the content expressed by speech or represent some new content which can be in supplementary or contrary relations with the semantics of the verbal utterance.

The objectives and the hypotheses account for the design of the research and the methods used.

Data analysis: procedures and methods

Case-study 1

For case-study 1 the source discourse (SD) was A. Vampilov’s “The Elder Son” (1965). I also turned to 22 theatrical performances based on this play staged by the theatres in Moscow, Irkutsk, Nizhniy Novgorod, Perm, Kostroma, Tyumen, Velikie Luki, and more. They all were retrieved from The International on-line Festival “The Elder Son — 55” (November, 2020). The total duration of the discourses derivatives (DDs) is 3110 minutes.

The preliminary step included the analysis of the SD: searching for recurrent cues produced by the same character or different characters, and recurrent communicative situations. The total number included 147 instances of the monomodal data. The next step was to find out whether in the DDs: a) the recurrent cue(s) is(are) modified; b) nonverbal modality(s) accompanying the verbal one is(are) used. Given these tasks, I used the modelling method accompanied by the parametric analysis [Iriskhanova, Cienki 2018; Kiose 2021]. The procedure involves the consistent implementation of three targets.

At first, a list of multimodal parameters was compiled. The list included the verbal modality and also those nonverbal modalities of a theatrical performance that interact with speech, i.e., reflect what is simultaneously produced verbally (gestures, facial movements, body movements, actions, music and/or sound effects, light effects). Each parameter was given a number from 1 to 9. The entire list includes the following parameters (P): Р₁ – verbal component unchanged; Р₂ – verbal component modified; Р₃ – gestural modality (hand gestures) added; Р₄ – head gestures and facial movements added; Р₅ – movements of the legs and/or body movements added (e.g., crossing or uncrossing the arms, legs; changing the seating position; etc.); Р₆ – body movements which indicate a change of place (proxemic signs) added; Р₇ – characters’ actions (also with props) added; Р₈ – music and/or sound effects added; Р₉ – light effects added.

Then the numerical indicator of activity for each parameter in the DD was calculated. In the table below I give the activity indicators on the 22 theatrical performances (DD) supplied with charts (Table 2).

As the next step, I calculated the average activity indicator and the range of activity indicators for each parameter. Normally, the number of the DD, in which the activity of this or that parameter is above the average indicator, and the number of the DD, in which the activity of this or that parameter is below the average indicator, do not exhibit any significant differences. The results are given in the following table (Table 3).

The next step was to create a model of the distributed parameter activity. Schematically, it is presented in the following volume chart (Diagram 1).

Given the need for more insight into the convergence of gesture and speech in the theatrical performance, I proceed with a more detailed research of gesture types used in the theatrical discourse.

Table 2.
Download as
CSV
XLSX

General statistics on multimodal parameter activity in the 22 discourse-derivatives.

DD1 DD2 DD3 DD4 DD5 DD6 DD7 DD8 DD9 DD10 DD11 DD12 DD13 DD14 DD15 DD16 DD17 DD18 DD19 DD20 DD21 DD22
P1 26 22 28 27 30 32 24 29 33 29 21 32 28 25 23 21 32 24 26 27 29 31
P2 74 78 72 73 70 68 76 71 67 71 79 68 72 75 77 79 68 76 74 73 71 69
P3 68 61 66 70 62 59 71 60 74 76 65 70 66 62 68 72 66 70 62 60 72 68
P4 66 67 62 68 70 62 70 69 65 62 60 70 54 59 70 66 58 72 58 61 69 66
P5 45 48 43 47 41 48 48 49 49 51 41 48 50 37 53 40 45 48 52 41 43 44
P6 48 46 50 40 44 52 52 46 53 44 40 55 46 51 54 42 54 49 52 56 48 46
P7 6 5 5 4 2 4 4 2 3 2 4 2 3 5 4 3 6 2 3 5 2 4
P8 3 2 4 3 3 2 2 3 4 3 2 4 3 2 0 3 2 3 2 1 2 3
P9 2 1 2 3 2 2 1 0 1 2 3 1 2 2 3 3 2 1 3 3 1 2
Diagram 1. 

Multimodal parameter activity in the 22 discourse-derivatives.

Table 3.
Download as
CSV
XLSX

General statistics on the average activity indicators and the range of activity indicators for each parameter.

The average activity indicator The range of activity indicators The activity indicator exceeding the average value of the parameter
min max range
P1 27.2 21 33 12 11
P2 72.8 67 79 12 11
P3 66.7 59 76 17 11
P4 64.7 54 72 18 12
P5 46.0 37 53 16 12
P6 48.5 40 56 16 11
P7 3.6 2 6 4 12
P8 2.5 0 4 4 12
P9 1.9 0 3 3 15

Case-study 2

The monomodal dataset used in case-study 2 consists of four SDs – the plays by the 20th century Russian playwrights: A. Vampilov’s “Duck Hunt” (1967); N. Sadur’s “The Odd Woman” (1982) and “Go” (1983); A. Kazantsev’s “The Fleeing Wanderers” (1996). The multimodal dataset comprises six DDs, namely theatrical performances staged by the Moscow Arts Theatre, the Mossovet State Academic Theatre, the Drama School of Yaroslavl State Theatre Institute, Primorsky Drama Theater, “Magic lantern” and “On the chairs” theatres; with a total duration of 620 minutes.

At the first step of the analysis, I selected all the indicators for gestures given in the stage directions. The set of samples comprises 35 indications of representational, pragmatic and deictic (pointing) gestures. There were no indicators for discourse-structuring gestures in the plays analyzed. Then I resorted to the video data to see whether the gestures were realized in the performances or not. To fulfill the task, I used the diagraph analysis suggested by John W. Du Bois in his study of spoken dialogues. The key unit of the analysis is the diagraph recognized as a coupling structure when one speaker (addressee) forms an utterance based on the immediately co-present utterance of an interlocutor (addresser) so that affinities across utterances occur. The following exchange provides an illustration of what Du Bois calls a diagraph [Du Bois 2014: 362]:

Table 4.
Download as
CSV
XLSX

The diagraph.

1 Joanne. it ’s kind of like ˆyou Ken .
2 Ken. that ’s not at ˆall like me Joanne .

Source: Du Bois (2014, p. 362).

To facilitate the analysis of similarities and differences in gestures across the SD and the DD I reformatted the diagraph in order to display the mappings both in rows and columns emerging in dialogic juxtaposition of a monomodal utterance in the SD and its multimodal counterpart in the DD. The multimodal diagraph consists of two parallel sets of rows for the SD and the DD respectively; and also contains a separate column for nonverbal modalities which interact with speech.

The quantitative results obtained with the help of the diagraph analysis are the following: 11 gestures (31.5%) were realized in the DD. The same percentage was for gestures which were modified (the change of gesture type). 13 gestures (37%) were not realized in the DD. The data did not exhibit any correlations between the type of gesture and its usage or non-use in the DD.

The next step of analysis deals with the study of gestures which were added in the DD to cooperate with speech in organizing a performance. The observational method employed made it possible to acquire a dataset of 2577 gestures synchronized with speech or following it. Using the Sampling Analysis tool in MS Excel 100 gestures were selected that were subject to further examination and statistical processing. The quantitative results are distributed as follows: the most frequent are the pragmatic and representative gestures (42% and 31% respectively); then come the deictic gestures (24%); the discourse-structuring gestures are characterized by the lowest frequency (3%).

The final step of the analysis engaged a modelling method based on a number of parameters identified due to the previous studies of gestures as a dynamic system [The multimodal dimensions of the discourse 2021]. They are as follows: regular, intentional, affective, cumulative, synthetic, complementary, additive, contrary. Schematically the multi-vector model for gestures is presented in the diagram below (Diagram 2).

Diagram 2. 

Gestural profile: the theatrical discourse.

Discussion

In this section of the paper I discuss the results of the empirical study to see whether they confirm or refute the hypotheses made.

The first hypothesis was that gestural modality together with facial movements would be the most active in multimodal meaning construal in the theatrical discourse. This hypothesis was partially confirmed. The results obtained show that the activity indicators are really the highest for P₃ and Р₄: hand and head gestures and facial movements such as eyebrows raised, wrinkles across the forehead, eyelids wide open, and more. At the same time, Р₆ (body movements) and Р₂ (verbal component modified) also have relatively high activity indicators. The ways of verbal modification include the repetition or omissions of words or phrases; or enlarging the cue. The following samples illustrate this observation.

In sample 1 the actor repeats the concluding phrase of the cue twice, adding a representational gesture (the stretched index finger hold) and a mimic sign (squinted eyes) which resonate with the speech, making the content salient. The image extracted from the video data (Irkutsk Academic Drama Theater named for N. P. Okhlopkov) and also the arrows in the diagraph explicate the correlations between the verbal and nonverbal components when the information in one modality enhances the information in another modality.

Sample 2 shows that the verbal component added and the gestures synchronized with it indicate a new conceptual projection – the necessity to lie in order to survive in life – contributing to the source utterance content (Moscow Drama Theater “Sphere”).

Sample 1.
Download as
CSV
XLSX
Verbal Non-verbal
SD 1.Sarafanov. I have one thing to complete !
2. just one thing
DD 1.Sarafanov. I have one thing to complete ! the stretched index finger hold;
2. just one thing ↔ squinted eyes
3. one thing
The original text and the script (abridged):
SD DD
Сарафанов. <…> мне надо завершить одну вещь, всего одну вещь! Сарафанов. <…> мне надо завершить одну вещь, всего одну, одну вещь!
Sample 2.
Download as
CSV
XLSX
Verbal Nonverbal
SD 1. Busygin. We have to invent something … -
2. We have to tell a lie , -
3. <…> We have to frighten or -
pity . -
DD 1. Busygin. We have to ,
2. have to invent something … a pragmatic gesture
3. We have to tell a lie ↔ , a pragmatic gesture
4. <…> We have to frighten or a representational gesture
5. pity ↔ . a representational gesture
6. Learn to lie .
7. Learn ↔ ! discourse-structuring gestures
8. Learn ! (head nods)
The original text and the script (abridged):
SD DD
Бусыгин. Надо выдумать что-то такое… <…> Надо соврать как следует, только тогда тебе поверят и посочувствуют. Их надо напугать или разжалобить. Бусыгин. Надо, надо, соврать как следует, только тогда тебе поверят и посочувствуют. Их надо напугать или разжалобить. Так что учись врать. Учись! Учись!

Sometimes a new conceptual projection can be activated with the help of nonverbal modalities only, e.g., body repositioning or body movements and poses that signal some implied meaning. The sample that can illustrate this observation is the final scene set by the actors as an allusion to “The return of the prodigal son”, a famous painting by Rembrandt. The resonating scene setting (The theater company “Free Stage”) emphasizes the role of nonverbal modalities in the process of recurrent semiosis which links the second order embodiment, salience and resonance.

Summarizing the findings, we see that whereas the arts as communication rely on multimodal interaction for meaning construction, the highlighting function is not restricted to speech-accompanying hand and head gestures and facial movements. But being mimetic and characterized by continuously varying forms, exactly these modalities have much in common with prosody in their potential for semantic nuancing [Goldin-Meadow 1999; Wagner 2014]. So, they are usually more adaptive to speech and can reinforce or modify the pragmatic aspect of the utterance.

The second hypothesis was that pragmatic and representational gestures would prevail in the DD due to the ontological nature of a theatrical performance. This hypothesis was confirmed. Besides, I wanted to check whether gestures indicated in the stage directions in the SDs were realized in the DDs. The results of the analysis reveal an approximately equal distribution of the gestures into three groups: gestures that are contextualized, concretized (or modified) and non-used in the DDs. The following samples from A. Vampilov’s “Duck Hunt” and the theatrical performance staged by the Moscow Arts Theatre illustrate the contextualization and concretization of the gestures prescribed by the playwright.

Sample 3.
Download as
CSV
XLSX
Verbal Non-verbal
SD Valeria. Oh, Vadim Andreevich I will … theatrically
I will pray for you .
DD Valeria. Oh, Vadim Andreevich I will … a representational gesture: praying hands
pray for you . In Bulwer’s classification this gesture is described as Oro
I will pray for you .
The original text and the script:
SD DD
Валерия (театрально). О, Вадим Андреич! Я готова… Я готова на Вас молиться. Валерия. О, Вадим Андреич! Я готова просто… (жест «мольба») молиться. На Вас молиться.
Sample 4.
Download as
CSV
XLSX
Verbal Non-verbal
SD Galina. Have you ever killed anything at least a small, this small bird ? shows the size of a bird using the finger
Kuzakov. He can’t get into this one . shows the size of a bird using both hands
DD Galina. Have you ever killed anything at least a small, this small bird ? a representational gesture: with the thumb and index fingers shows the size of the bird
Kuzakov. He can’t get into this one . a representational gesture: stretches both hands up and spreads them to the side to show the size of a bird
The original text and the script:
SD DD
Галина. Ну скажи, убил ли ты что-нибудь хоть раз? Ну хотя бы маленькую, ну хоть вот такую (показывает на пальце) птичку? Галина. Ну скажи, ну признайся, признайся! Убил ли ты что-нибудь хоть раз? Ну хотя бы такую, ну вот такую (большим и указательным пальцами показывает размеры птички) маленькую птичку?
Кузаков. Ну что ты ему показываешь? Он в такую (показывает обеими руками) не попадает. Кузаков. Ну что ты ему показываешь? Он в такую (вытягивает обе руки вверх и разводит в стороны) попасть не может.

As the main result, the analysis showed that 73% of the gestures used in the DDs are those with pragmatic and representational functions. This induces us to ask whether representational gestures (and also discourse-structuring gestures) in the theatrical performance as a second modelling system are deprived of pragmatic function or do they demonstrate different functions (e.g., pragmatic and representational) which complement each other?

While A. Kendon speaks of two main functions of co-speech gestures, namely substantial and pragmatic gestures [Kendon 2004]; the current empirical research makes it possible to define the aforementioned types of gestures in the theatrical discourse as having the pragmatic function in their core. Since the focus of a theatrical performance is on visualization, intensification and expressiveness it accounts for gestural patterns constructing joint meanings with the pragmatic meaning in the forefront. It manifests itself in conveying attitudes, but these are not ad hoc attitudes as in face-to-face interaction [Brône et al. 2013; Cienki 2017; Iriskhanova, Cienki 2018; Sonesson 2019]. These are attitudes organized in advance because the actor’s nonverbal behavior is in a way preprogrammed and integrated into the general semiotics of the theatrical performance together with costume, scenery, prosody, music, lighting, and more.

Consider the following extract from A. Kazantsev’s “The Fleeing Wanderers” staged by the Mossovet Theatre: During this time, many have burst, rolled up, left, and so on… (The original text: Многие за это время лопнули, закатились, уехали и так далее…)

The verbs in the cue are synchronized with a hand gesture (hand swing upward). It is a metaphoric gesture which concurrently fulfills the discourse-structuring function (Fig. 1a–c). It would be possible to use the same type of gesture choosing the opposite direction. But it is the upward movement that accentuates the meaning invested in the multimodal cluster, i.e., to vanish into thin air, disappear without a trace. Thus, the gesture renders the speaker’s attitude which mirrors the intent that the stage director and the actor want to express being the co-authors of the theatrical performance.

Figure1. 

a–c.

The third assumption was that gestures in the DD can complement the content expressed by speech or represent some new content which can be in supplementary or contrary relations with the semantics of the cue. This hypothesis was also confirmed. Gestures and speech in a theatrical performance can contextualize the same image (samples 3–4); the gestural modality can add expressiveness (samples 1–2); or invest some new conceptual projections into the image represented. The additive function of gestures can be illustrated by the following sample from N. Sadur’s “The Odd Woman” staged by “On the chairs” theatre.

Sample 5.
Download as
CSV
XLSX
Verbal Nonverbal
SD Woman 1. I ’m running away
2.You ’re catching me up
3. If you catch me there will be paradise
4. If you don’t catch me there will be the end of the world
DD Woman 1. I ’m running away deictic and representational gestures (acting out the scene)
2.You ’re catching me up
3. If you catch me there will be paradise representational (metaphoric) gestures
4. If you don’t catch me there will be the end of the world

According to the dialogic syntax structure [Du Bois 2014], the verbal correlations between the SD and the DD reflect the frame resonance characterized by straightforward affinities (I : you // ’m : ’re // if you : if you don’t, etc.) and also the focus resonance that serves to lay emphasis on the contrasting images (paradise : the end of the world). In the sample provided the phrase “the end of the world” constitutes the mapping relation with the metaphorical gesture (imagery throat cutting) which implies the meaning of violent death as compared to the meaning of natural death expressed by the phrase.

The original text and the script are the same (abridged):

Баба. <…> Я – убегаю. Ты – догоняешь. Поймаешь – рай, не поймаешь – конец всему свету.

One more observation is that the SD usually has verbal triggers that make it possible to activate the gestural modality or some other semiotic modalities. E.g., the description of a character’s behavior given in a stage direction can account for a certain costume, hairstyle, facial expression and also gait. Some linguistic units can suggest acting out the scene with the help of gestures-illustrators. E.g., phrases like “this kind” or “like this”, intensifiers and textual ellipsis trigger nonverbal completion of an object or an event. Manual action verbs usually co-occur with iconic gestures, e.g., to dial, to switch on, etc. Deictic words in a character’s speech can be accompanied by а pointing gesture on stage or, on the contrary, а pointing gesture can replace the verbal phrase and, as such, represent a nonverbal utterance.

So, the interplay of speech and co-speech gestures which emerges in everyday communication displays specific features and functions when realized in the arts as communication, namely in a theatrical performance in which gestures can simultaneously implement several communicative attitudes assuming a multiplicity of interpretations. As was confirmed, despite the context-dependent, polysemous nature of most hand and head movements the actors employ, it is possible to identify the pragmatic function as a core common to many particular uses.

The results of the multimodal analysis show that one can distinguish two major trajectories of inferencing the original communicative situation: complementary and supplementary. The former means that the co-occurring conceptualizations through verbal and gestural components in the performance can result in the salience of the content and resonance, the pragmatic aspect being preserved. The latter means the emergence of new conceptual projections accompanied by both resonance and the modulation of the pragmatic aspect. Thus, resonance in the arts as communication involves intended additional meaning and is linked to symbolic cognition enfolded with mimetic cognition.

Conclusion

Summing up, the research focus was on the exploration of speech-linked modalities that take part in the multimodal meaning construal in the theatrical discourse. The study has proved the efficiency of the diagraph analysis in organizing information from different modalities and in making the mapping relations explicit. Together with the statistical procedures this analytical tool made it possible to solve the main objectives of the research and confirm the hypotheses set. As was shown, the application of this methodology is relevant for estimating the activity and functions of different modalities in the meaning-making process, which means that it can be applied for the analysis of other types of heterogeneous discourse with the aim of revealing the construal process specificity.

Viewed from the perspective of the dialogic syntax, the original content (a play) and its multimodal derivative (a performance) are engaged in the “pragma-dialogue” [Kecskes 2016], which exploits the characteristics of verbal and nonverbal modalities to realize their potential for creativity and resonance, whether through immediate mappings or evocative allusion to a distant prior communicative event. In accordance with the research findings, the pragma-dialog is characterized by the dynamic strengthening or emergence of meaning projections and expressiveness likely to be caused by cross-activation between semiotic systems and sensory modes in the process of recurrent semiosis.

Being relatively young, the multimodal theory of meaning in different types of discourse needs a more fine-grained exploration before it can answer all the questions that are asked of it. The current research serves as a necessary prelude to a more in-depth study of gestural modality in the theatrical discourse so as to specify the semiotic functions and also variation of gestural patterns as the pivotal semiotic resources of the theatre; and to an overarching research of resonance as a phenomenon that can help us enhance our understanding of embodied meaning construal through different modalities, especially in the arts as communication.

References

  • Barthes R. (1989). Izbrannye raboty. Semiotika. Poetika [Selected works. Semiotics. Poetics]. Trans. from French. Moscow: Progress.
  • Brône G. (2013). Multimodal turn-taking in dialogue: on the interplay of eye gaze, speech and gesture. Proceedings of AFLiCo5: Empirical approaches to multi-modality and to language variation, 21–22.
  • Cienki A. (2013). Cognitive Linguistics: Spoken language and gesture as expressions of conceptualization. In C. Műller, A. Cienki, S. Ladewig, D. McNeill, S. Teßendorf (Eds.), Body – language – communication: An international handbook on multimodality in human interaction (pp.182–201). Vol. 1. Berlin: Mouton de Gruyter. https://doi.org/10.1515/9783110261318.182
  • Cienki A. (2017). Analysing Metaphor in Gesture: A set of Metaphor Identification Guidelines for Gesture (MIG-G). In E. Semino, Z. Demjen (Eds.), The Routledge Handbook of Metaphor and Language (pp.: 131–147). London: Routledge.
  • Cienki A. (2019). Gestures and grammatical constructions. Cognitive Studies of Language, 37, 120–126.
  • Cienki A. (2021). From the finger lift to the palm-up open hand when presenting a point: A methodological exploration of forms and functions. Languages and Modalities, 1, 17–30.
  • Dancygier B. (Ed.) (2017). Language, Body, and Multimodal Communication. In The Cambridge Handbook of Cognitive Linguistics (pp. 91–206). Cambridge, Cambridge University Press. https://doi.org/10.1017/9781316339732
  • Fisher-Lichte E. (1992). The Semiotics of Theatre. Transl. in English by Jeremy Gaines, Doris L. Jones. Indiana University Press.
  • Gibson J. J. (1979). The ecological approach to visual perception. Boston, MA: Houghton Mifflin.
  • Hart F. E. (2011). Performance, phenomenology, and the cognitive turn. In B. McConachie and F. E. Hart (Eds.) Performance and cognition: Theatre studies and the cognitive turn (pp. 29–51). London and New York: Routledge.
  • Iriskhanova O. K. (2018). Gestures as indicators of changes in discourse trajectory. Cognitive Studies of Language, 35, 100–115.
  • Iriskhanova О. К. (2018). Cognitive resonance: verbal and non-verbal dimensions. Cognitive Studies of Language, 32, 334–344.
  • Iriskhanova O. K. et al. (2021). “Gesture turn” in professional communication studies: simultaneous interpretation. Cognitive Studies of Language, 1, 190–202.
  • Iriskhanova O. K. (ed.) (2021). Polimodal’nye izmereniya diskursa [The multimodal dimensions of the discourse]. Moscow: LRC Publishing House.
  • Jakobson R. (2004). Linguistic aspects of translation. In L. Venuti (Ed.). The Translation Studies Reader (pp. 113–118). London and New York. Routledge: Taylor & Francis Group.
  • Kibrik A. A., Fedorova O. V. (2018). Language production and comprehension in face-to-face multichannel communication. Computational Linguistics and Intellectual Technologies, 17, 305–316.
  • Kiose М. I. (2021). Orchestrations of speech and gesture in cinematic shots. Discourse of argumentation and description. Cognitive Studies of Language, 4, 255–270.
  • Kress G. (2010). Multimodality. A social semiotic approach to contemporary communication. London and New York: Routledge.
  • Leonteva A. V. (2017). Strategy of events reconstruction in courtroom discourse: a gesture-with-speech analysis. Cognitive Studies of Language, 29, 643–649.
  • Lotman Yu. M. (2002). Stat’i po semiotike kul’tury i iskusstva [The articles on the semiotics of culture and art]. Saint-Petersburg: Academic Project.
  • Mancing H. (2011). See the play, read the book. In B. McConachie and F.E. Hart (Eds.) Performance and cognition: Theatre studies and the cognitive turn (pp.189–206). London and New York: Routledge.
  • Matejka L., Titunik I. R. (eds. (1976). Semiotics of Art: Prague School Contributions. Cambridge (Mass.); London (MIT Press).
  • Műller C. (1998). Redebegleitende Gesten. Kulturgeschichte – Theorie – Sprachverleich. Berlin: Berlin Verlag.
  • Műller C. (2019). Metaphorizing as embodied interactivity: What gesturing and film viewing can tell us about an ecological view on metaphor. Metaphor and Symbol, 34(1), 61–79. https://doi.org/10.1080/10926488.2019.1591723
  • Halloran K. (2011). Multimodal analysis. In K. Hyland, B. Paltridge (Eds.) Companion to Discourse (pp. 120–137). London and New York.
  • Tsibulya N. B. (2021). Correlation of eyebrow movement and intonation as a marker of cognitive processes. Cognitive Studies of Language, 4, 297–304.
  • Stanislavsky K. (1964). An Actor Prepares. Trans. E. R. Hapgood. New York: Routledge/Theatre Arts Books.
  • Yanko T. E. (2019). Speech acts in discourse: the prosodic strategies of marking the discourse links. Cognitive Studies of Language, 38, 587–594.
  • Zlatev J. (2007). Language, embodiment and mimesis. In T. Ziemke, J. Zlatev, R. Frank (Eds.). Body, language, mind. Vol. 1: Embodiment (pp. 297–337). Berlin: De Gruyter.
  • Zykova I. V. (2021). Poetika kinodiskursa v polimodal’nom izmerenii. In O.K. Iriskhanova (Ed.) Polimodal’nye izmereniya diskursa (pp.153–238) [The poetics of cinematographic discourse: multimodal dimensions. In The multimodal dimensions of the discourse]. Moscow: LRC Publishing House.