Research Article |
Corresponding author: Anna Leonteva ( lentevanja27@gmail.com ) Academic editor: Olga Iriskhanova
© 2022 Maria Kiose, Anna Leonteva, Olga Agafonova.
This is an open access article distributed under the terms of the Creative Commons Attribution License (CC BY-NC-ND 4.0), which permits to copy and distribute the article for non-commercial purposes, provided that the article is not altered or modified and the original author and source are credited.
Citation:
Kiose M, Leonteva A, Agafonova O (2022) Aesthetic multimodality of speech and gesture: Towards its functional framework. Languages and Modalities 2: 1-17. https://doi.org/10.3897/lamo.2.78840
|
The study develops a functional multimodal approach to speech and gesture behavior to explore aestheticism in more and less staged discourse of cinema and interview. We hypothesize that cinema and interview employ the same communicative functions; however, these functions constitute different frameworks which contribute to the higher aesthetic potential of cinema. This approach allows to study the aesthetic via communicative functions frameworks in multimodal discourse.
To establish the function frameworks in cinema and interview, we apply a contrastive functional analysis of speech and gesture in the highly ranked actors’ argumentative and descriptive monologues. With the help of variance and regression analysis, we explore the distribution of pragmatic and discourse-structuring functions (with sub-functions) in speech as contingent on pragmatic, deictic, representational and adaptive functions of gestures. The study confirms that cinematic discourse exploits fewer deictic, representational and adaptive gesture functions, whereas pragmatic gesture functions (especially emphatic ones) appear more frequently and are contingent on several pragmatic and discourse-structuring functions of argumentative and descriptive speech. Interview function frameworks display lower predictability, which shows higher spontaneity of gestures; however, there are specific gestures typical of interview (self-adaptors) which may serve as indicators of pragmatic functions of argumentation. The study also manifests individual variations within the function frameworks. Overall, cinema and interview display variance in replication and regularity of speech and gesture functions, which presumably helps create higher and lower aesthetic effects.
aesthetic multimodality, multimodal discourse, speech functions, gesture functions, function framework, cinema, interview
Until recently, aesthetic semiotics has been the only influential approach exploring aestheticism in multimodal discourse, for instance, in cinema. Aesthetic semiotic studies initiated by G. Shpet and R. Jakobson (Shpet, 1922 (2007),
Multimodal approach to aesthetic experience can offer its own solution to the problem of aesthetic function in discourse. Following Goodrich, we assume that aesthetic (poetic) discourses can hardly possess their own unique aesthetic function represented by some unique markers of aesthetic discourse; however, they definitely possess the communicative functions common for each discourse type, although their aesthetic potential may be different. Therefore, developing a multimodal approach which considers the communicative functions of modalities can become more efficient since it can scale the function frameworks of more and less aesthetic discourse.
The idea of studying higher-order hierarchy of functions (however, termed differently in different works) to explore the dynamicity and modifications of semiotic systems has received theoretical approbation (
Overall, the contributions of this study include: (i) introducing the functional multimodal approach to the study of speech and gesture complexes as specific aesthetic (poetic) means, (ii) revealing the dominant functions of speech and gesture in cinematic discourse as opposed to less aesthetic interview discourse, (iii) disclosing individual variation in multimodal function frameworks that stimulate cinematic aestheticism.
Aesthetic semiotics offers different semiotic instruments to explore the aesthetic (poetic) function in artwork focusing on multiple dimensions of the aesthetic. For instance, in Yu. Stepanov’s semiotic school (Moscow, Russia) the category of the aesthetic appears in constants (invariants) and variations. The semiotic constants (
In aesthetic multimodal discourse, for example, in cinema, exploring these functions in different modalities can become more difficult since their markers will display considerable variation. Besides, multimodality in cinematic discourse is often regarded in two dimensions, the first incorporates the actor’s behavior which is treated on the whole as “Gesture” (
In this study, we explore the aesthetic potential of speech and gesture in cinema and interview via their communicative function frameworks. To reveal them, we 1) assess the activity (frequency) of communicative functions in speech and gesture in two discourse types, 2) reveal the function frameworks of cinematic and interview discourse (applying the methods of variance and regression analysis), 3) explore the individual variance in function frameworks.
In recent years, multimodal (as opposed to semiotic) approach to speech and gesture in cinematic discourse has been applied in different works, for instance in corpus studies (Grishina & Savchuk, 2008). However, they do not aim to reveal the aesthetic potential of discourse. What communicative functions of speech and gesture may serve to explore it?
In terms of speech, we will address the communicative functions outlined in the discourse pragmatic theories; nevertheless, we need to frame these functions on the same grounds. Following T. van Dijk (
In terms of gesture, we will address the communicative functions of gesture manifested in gesture types. The functional approach to gesture analysis initiated in (
We will apply the method of contrastive analysis to reveal the communicative functions of speech and hand gesture in more and less aesthetic discourse, here in cinematic discourse and the discourse of interview. We expect to detect the specificity of speech and gesture functions in cinematic discourse which will allow to establish the function framework stimulating aesthetic discourse potential. To avoid the possible clines in individual multimodal behavior, we select the samples of cinematic and interview discourse with the same highly ranked Russian actors performing in monologues. Cinematic discourse in contrast to the interview is a more staged discourse, employing rehearsed multimodal behavior patterns aimed at enhancing the aesthetic potential. Still, when performing in interviews the actors will hardly avoid applying the familiar patterns, however we expect they will be much influenced by non-staged discourse format. Contrastive analysis of function frameworks in cinematic discourse and in interview will help identify 1) variance in the functions of gesture and speech, 2) variance in their combinations, 3) variance in the individual input in terms of both speech and gesture functions and in their combinations. The research has several constraints, first, the selected fragments of cinematic and interview discourse display different combination of genres, here argumentation and description (although we selected the samples very similar in genre), second, the actors performing will appear in the samples in different age. These constraints will definitely affect the function distribution; however, we consider that the study will only benefit if we manage to identify the steady function frameworks despite the variations which are not systemic.
The research data are the film samples (fragments) with 5 male actors performing monologues in very popular Russian films: «Звонят, откройте дверь», А. Митта (“The doorbell rings, open the door” directed by A. Mitta), 1961, «Они сражались за Родину», С. Бондарчук (“They fought for the Country” directed by S. Bondarchuk), 1975, «Судьба человека», С. Бондарчук (“Man’s fate” directed by S. Bondarchuk), 1959, «Доживем до понедельника», С. Ростоцкий (“Let us live till Monday” directed by S. Rostotskiy), 1968, «Москва слезам не верит», В. Меньшов (“Moscow does not believe in tears” directed by V. Menshov), 1980. They feature the monologues of R. Bykov, Yu. Nikulin, S. Bondarchuk, V. Tikhonov and A. Batalov. To contrast the same actors’ function frameworks in less staged discourse, we selected 5 interviews taken in different environments, with the interviews with V. Bykov, Yu. Nikulin, and V. Tikhonov taken in the studio, the interview with S. Bondarchuk taken in his study, the interview with A. Batalov taken on the river embankment. All the monologues display realistic character; they are both descriptive and argumentative.
Since the research data were the samples of argumentation and description, to compile a list of functions we addressed the studies which looked into general discourse structuring and pragmatic aspects as well as the studies featuring argumentative and descriptive discourse strategies. Regarding argumentation, the studies account for its ability to express opinions and beliefs (
In Table
Argumentation | Description | ||
---|---|---|---|
Pragmatic functions | Pragmatic functions | ||
Opinion | 101 | Achievement | 201 |
(Emotional) assessment | 102 | Process | 202 |
Stating reasons, consequences, conditions | 103 | State | 203 |
Contrast | 104 | Accentuated subject | 204 |
Accusation | 105 | Accentuated object | 205 |
Agreement / Disagreement | 106 | Accentuated action or state | 206 |
Appeal to action | 107 | Accentuated characteristics | 207 |
Promise | 108 | Accentuated time | 208 |
Threat | 109 | Accentuated place | 209 |
Comparison | 110 | ||
Appeal to power | 111 | ||
Discourse-structuring functions | Discourse-structuring functions | ||
Emphasizing opinion or assessment | 112 | Emphasizing discourse component | 210 |
Self-correcting | 113 | Self-correcting | 211 |
Specification | 114 | Specification | 212 |
Generalization | 115 | Generalization | 213 |
Intersubjectivity | 116 | Chain of events | 214 |
Appeal to attention | 117 | New event | 215 |
Rhetorical communication | 118 | Appeal to attention | 216 |
Initializing communication | 119 | Self-quote | 217 |
Chain of arguments | 120 | Quoting others | 218 |
Self-quote | 121 | Figurativity | 219 |
Quoting others | 122 | ||
Figurativity | 123 |
To explore the gesture functions in the current work in both cinema and interview discourse, we will address their four basic communicative functions (with further specification): pragmatic, representational, deictic and adaptive (
Representational gestures, also known as iconic gestures, are based on the idea of similarity between a hand form and / or its movement and the process or object which it refers to (Streek, 2008). There are several modes of representation which are distinguished in this study: Holding, Molding, Acting, Embodying and Tracing (Müller, 2014). In the example taken from the interview (Figure
Another type, deictic gestures, are used to refer to people, objects, notions, places, events, etc. by creating axis in space which connects the speaker and the target of speech (
The last type, adaptors, represent some movements, which can be self-oriented (Self-adaptors) such as rubbing one’s nose, adjusting glasses, fidgeting one’s fingers, etc., or they can be object-oriented (Object adaptors): touching the table in front of the speaker, moving a glass of water, trifling with a pen, etc. These gestures can be used in order to gain control of the situation when the speaker is in the state of distress (
In Table
Gesture functions | |||
---|---|---|---|
Pragmatic functions | Representational functions | ||
Discourse structuring | 308 | Holding | 303 |
Discourse representational | 309 | Molding | 304 |
Discourse emphatic | 310 | Acting | 305 |
Expressing attitude/evaluation | 311 | Embodying | 306 |
Contact establishing | 312 | Tracing | 307 |
Deictic functions | Adaptive functions | ||
Pointing | 301 | Self-adaptors | 313 |
Touching | 302 | Object-adaptors | 314 |
In modelling and processing the data, we will apply the method of variance and regression analysis in discourse profiles construal. The notion of discourse profiles suggested in construction grammar and structure building frameworks (
The study develops a two-stage procedure. First, it detects the variance in the speech and gesture function frameworks in more aesthetic cinematic discourse and less aesthetic interview discourse. Next, we proceed to contrastive analysis of individual specificity of each actor in stimulating this aestheticism, hypothesizing that despite the individual variance these function frameworks will display a steady character.
The main procedural questions were the selection of the unit of analysis and the annotation format. To select the procedural unit for analyzing speech and gesture complexes, we adopt the view that this unit should be able to manifest (and describe) both description and argumentation. Since the smallest unit capable of manifesting description is a word combination displaying predicate or attribute relations and the smallest unit capable of manifesting argumentation must necessarily be a proposition or its modal frame with either predicate or performative relations, we select the unit with a higher information potential which is the proposition or its modal frame.
For instance, Example (1) has 5 propositions which display different argumentation and description potential:
(1) Я никогда не был пионером // Но у нас во дворе был форпост // Но так как я никогда не был первым пионером // то я вам расскажу не про себя а про горниста // который жил у нас во дворе
I have never been a pioneer // But we had a fort-post // But since I have never been a first pioneer // I will tell you not about myself but about a bugler // who lived in our courtyard (R. Bykov, cinematic discourse)
Each of the units was annotated following the same procedure. In case we faced difficulties separating propositions in speech (for instance, when there were hesitations, hedges, interruptions) we adopted the following principle: a unit must necessarily involve either a proposition or a modal frame, therefore all the fragments which do not constitute a proposition or a modal frame are incorporated into the proposition or modal frame that was previously started and was not yet terminated. Example (2) illustrates the described case.
(2) Спасибо вам // что вы именно сюда приехали // потому что как… здесь снимались «Журавли» // Вот … эээ … ну я то … тут финал снимался … вот на этом месте буквально
Thank you // that you came right here // because it was … here… the “Cranes” was filmed // Right here … er… and I … here the final episode was made … at this very place (A. Batalov, interview discourse)
Units 3 (потому что как… здесь снимались «Журавли») and 4 (Вот … эээ … ну я то … тут финал снимался … вот на этом месте буквально) may have combined several propositions, however they are not completed in oral speech.
Annotation was performed in ELAN software, created by Max Plank Institute and used to annotate gestures (https://archive.mpi.nl/tla/elan). We chose it since it allowed to annotate the cinematic and interview shots considering their dynamicity. In Figure
The decision on the gesture function was adopted following the analysis of their form (e.g., the form of the hand: palm up / down, fist, finger extended, etc.; its movement: straight line, circle, wave, etc.; then the direction of it, as well as the space on which it occurred: horizontal or vertical axis, away or towards the speakers, etc. (see Bressem, 2013)) consistent with communicative functions. After that the attention was paid to the semantics of the gesture, which determines its type and corresponding functions in speech, as mere form analysis cannot be used to determine their role in speech due to the polysemantic nature of gestures (
(3) Это слишком серьезное занятие
It is a very serious occupation (S. Bondarchuk, interview discourse).
First, we annotated speech functions in terms of argumentation and description. In the above given proposition, we can see that the actor uses Generalization as a pragmatic function of argumentation. S. Bondarchuk discusses the problems that exist in the cinema and summarizes the point by stating that cinema is a serious type of occupation. We pointed out two pragmatic functions of description: State (since this is a description of a state) and Accentuated characteristics (since the modifier serious is foregrounded). Next, we addressed the co-speech gestures, and specified the gesture function depending on the performed hand movements and speech. In the example given the actor is rubbing his hands which could indicate that he is using Self-adaptors.
We also employed the speech scripts with full annotations in txt-format. To process the data, we applied HETEROSTAT software (Kiose & Efremov, 2020) which allows to identify the annotated functions activity as well as their contingency.
In Figure
As it can be seen, the software checks the data for its consistency with the coded taxonomy, allows to select single or all tiers for further processing. To perform further processing to check the contingency of communication functions of speech and gesture in two discourse types and in individual discourse, we applied JAMOVI software (https://www.jamovi.org).
The analysis of functions in speech and gesture was performed with 10 samples, with 5 of them representing cinematic discourse and 5 representing interview discourse. Each sample lasted approximately 2–3 minutes (min 2:12, max 3:36). The number of annotation units (propositions and modal frames) varied significantly, with min 22, max 66 in cinematic discourse, and min 24, max 71 in interview discourse. The samples displayed finalized communicative events, for instance, in his monologue A. Batalov describes the way the film was made in the very place the interview is taken and presents his arguments on why the film has achieved great success. The total number of the units of analysis in cinematic discourse is 203, and 205 in the interview discourse respectfully, so the data are compatible.
The annotation procedure was carried out by three annotators with two annotators working with the interview discourse, and one annotator working with the cinematic discourse. Then the annotated samples were subjected to crosscheck and Cohen’s Cappa statistical coefficient (
We processed the Cohen’s Cappa separately for two discourse types. Since 56 functions of speech and gesture were annotated and the number of units was 203 and 205 correspondingly, we received a total number of annotation responses equal to 11,368 in cinematic discourse, and 11,480 in interview discourse. In terms of cinematic discourse, both groups of judges agreed to decide 1402 cases in the positive and 9866 cases in the negative with 90 cases decided in the positive by the first annotator and 439 by the other annotator group. The agreement coefficient is 95.52%, and Cohen’s k = 0.82, which is almost perfect agreement. In terms of interview discourse, both groups of judges agreed to decide 1478 cases in the positive and 9720 cases in the negative with 211 cases decided in the positive by the first annotator and 387 by the other annotator group. The agreement coefficient is 94.93%, and Cohen’s k = 0.8, which is also almost perfect agreement. We then voted for including the functions (since there were three annotators) and the final results are the following: the total activity of speech and gesture functions is 863 in cinematic discourse (728 in speech and 135 in gestures), and 1118 in interview discourse (826 in speech and 292 in gestures).
In Tables
Discourse types / Functions | Speech | |||
---|---|---|---|---|
Pragmatic | Discourse-structuring | |||
Argumentation | Description | Argumentation | Description | |
Cinematic | 155 | 402 | 105 | 66 |
Interview | 167 | 461 | 73 | 125 |
Discourse types / Functions | Gesture | |||
---|---|---|---|---|
Deictic | Representational | Pragmatic | Adaptors | |
Cinematic | 17 | 16 | 89 | 13 |
Interview | 36 | 76 | 127 | 53 |
The difference in all functions of speech and gesture seems significant but it is not statistically verified. With F(1, 14) = 0.085 at p = 0.775, we cannot reject the null hypothesis that there is no difference between the functions in cinematic and interview discourse. However, the variance in single functions (56 functions) is significant (F(1, 110) = 4.92, p = 0.027), which proves that specifying functions was an effective solution. It is noticeable that significant differences are observed in gesture only if we consider speech and gesture separately. With F(1, 26) = 7.14, p = 0.008 in gesture and F(1, 26) = 0.947, p = 0.33 in speech, we can claim that gesture distribution is of higher importance in more and less aesthetic discourse multimodal construal.
Therefore, we now move on to discussing single functions of gesture in two discourse types in more detail. Figure
Since these functions might be contingent on the speech functions, we performed regression modelling to reveal the predicting gesture functions in multimodal discourse. Regression modelling is an efficient method to cope with the problem of mixed effects of functions which is a typical case of construal in speech. However, we frequently deal with a problem of aliased coefficients. In the current study, however, these effects were surprisingly scarce which means that functional instrument works well for the needs of multimodal discourse analysis. We will present the model performance summary statistics for the most active functions, first in cinematic discourse, next for the interview discourse.
In cinematic discourse, two gesture functions with the highest activity are the pragmatic functions, Discourse emphatic (310), and Expressing attitude / evaluation (311). Their model performance statistics is given in Table
Regression Model predicting Pragmatic gestures, Discourse emphatic and Expressing attitude.
Predictor | 310 Discourse emphatic R2 = 0.341 | 311 Expressing attitude/evaluation R2 = 0.359 | ||||||
---|---|---|---|---|---|---|---|---|
Estimate | SE | t | p | Estimate | SE | t | P | |
Intercept | 0.077 | 0.091 | 0.853 | 0.395 | -0.133 | 0.078 | -1.7 | 0.091 |
(Emotional) assessment | - | - | - | - | 0.267 | 0.076 | 3.5 | < .001 |
Stating reasons, consequences, conditions | - | - | - | - | 0.181 | 0.072 | 2.507 | 0.013 |
Threat | - | - | - | - | -1.535 | 0.424 | -3.623 | < .001 |
Achievement | - | - | - | - | 0.198 | 0.08 | 2.484 | 0.014 |
Accentuated time | 0.168 | 0.078 | 2.144 | 0.034 | - | - | - | - |
Emphasizing discourse component | 0.358 | 0.122 | 2.938 | 0.004 | - | - | - | - |
Specification | 1.779 | 0.423 | 4.2 | < .001 | - | - | - | - |
As seen from Table
In the following example (Figure
The best predictors for Expressing attitude / evaluation gestures are the pragmatic functions of argumentation and only one function of description. Figure
The received data may provide evidence in favor of a specific functional feature of aesthetic multimodal discourse, which is the rigid (perhaps, because much practiced and rehearsed) correspondences of different gesture functions with different types of speech functions in argumentation and description. We will check this assumption contrasting the results with the model performance statistics of the interview discourse.
Three gesture functions with the highest activity are two pragmatic functions, Discourse representational and Expressing attitude / evaluation; and also one Adaptive function, Self-adaptors. We will present the results on only two functions, Expressing attitude / evaluation (311), which is the same with the cinematic discourse, and Self-adaptors (313), which is of a different function type not active in the cinematic discourse. Their model performance statistics is given in Table
Regression Model predicting Pragmatic gestures, Discourse emphatic and Expressing attitude.
Predictor | 311 Expressing attitude/evaluation R2 = 0.282 | 313 Self-adaptors R2 = 0.256 | ||||||
---|---|---|---|---|---|---|---|---|
Estimate | SE | t | p | Estimate | SE | t | P | |
Intercept | 0.056 | 0.096 | 0.588 | 0.558 | 0.308 | 0.114 | 2.71 | 0.007 |
Opinion | 0.437 | 0.145 | 3.009 | 0.003 | ||||
Intersubjectivity | 1.366 | 0.616 | 2.218 | 0.028 | ||||
Self-correcting | 0.55 | 0.187 | 2.943 | 0.004 |
As seen from Table
The sample given in Figure
There are no other predictors, which makes us think that the interview discourse being less staged allows much more freedom in multimodal function frameworks. In terms of Self-adaptors, the situation is less peculiar. The best predictors are Opinion and Intersubjectivity, which are argumentation Pragmatic and Discourse-structuring functions, and the appearance of Self-adaptors in such situations was most expected.
The example shown in Figure
As far as S. Bondarchuk introduces his own opinion, he is gaining control over the situation by using Self-adaptors.
The contrastive results presented here may suffice to deduce several structural types of function frameworks.
REPLICATION – SINGULARITY. This structure integrates several functions of the same type. It can have three types. Type 1 describes the framework of intensifying the same function in both modalities. In the present study we observed Type 1 with pragmatic function which can be demonstrated in both speech and gesture. Interestingly, in cinematic discourse the gesture functions (at least the ones we explored in terms of predictability) seem to comply with similar functions in speech as if to enhance them additionally, whereas in the interview we did not observe this effect. Therefore, cinematic (and more aesthetic) discourse mostly exploits REPLICATION function structure. Type 2 describes the framework of intensifying the same function within one modality. This happens when the same function is activated several times within one proposition or modal frame. Surprisingly, cinematic discourse shows less preference for multiple pragmatic function expression in description, with very scarce pragmatism of the subject (in contrast with the interview discourse). This result might be explained by the spontaneous nature of the interview discourse and multiple pragmatic patterns chosen by the speakers to describe the events and their participants. Type 3 describes the framework of intensifying multiple sub-functions of one function. The data have shown that this function framework is frequent in both discourse types. Therefore, cinematic discourse mostly exploits REPLICATION function framework, whereas interview discourse shows preference for SINGULARITY.
REINFORCEMENT – SELF-SUFFICIENCY. This structure integrates several contingent functions in multimodal discourse. As we have shown above, it has 2 types of realization. Type 1 describes the framework of reinforcing the function of one modality with the functions of the other modality which display rigid contingency. REINFORCEMENT is more frequently present in cinematic discourse, whereas interview discourse has fewer contingent functions. Interview discourse spontaneity may also account for it. Type 2 describes the framework of reinforcing the functions within one modality. We may notice here that neither discourse exhibits this type of REINFORCEMENT. The functions do not display alignment effects, therefore their individual input into the process of multimodal discourse construal needs no other evidence: these functions are self-sufficient in determining the communicative potential of the modalities considered.
LIMITATION – EXTENSION. This structure is of a different type; it describes the function framework of constraining and widening the number of functions which are manifested in multimodal discourse. Therefore, this structure can be detected only in contrastive analysis. Type 1 appears when the number of functions is significantly smaller in one of the modalities. Unexpectedly, this type is more typical of cinematic discourse which employs far fewer gestures of specific types, for instance, of Self-adaptors. They are mostly present in interview, since they are the most natural and uncontrolled movements, used in our speech when the speaker might feel more stressed and in need of exercising control over the situation (
Next, we will find out whether these function frameworks appear in individual multimodal discourse of the 5 actors. This analysis might also reveal some other function frameworks relevant for assessing the aestheticism of cinematic discourse.
At the second stage we turn to the individual differences in multimodal discourse which may be found if we contrast speech and gesture in two discourse types, cinematic and interview, performed by the same actors. This procedure allows to detect the individual variations which may or may not fall within the function frameworks of either cinematic or interview discourse and consequently, may help specify them and show them in more detail. To proceed, we contrast the multimodal discourse profiles within each discourse. The idea that we entertain is that the profile differences will display some similar tendencies which will suffice to claim that they are more typical of more aesthetic discourse.
At the first step, we identify whether the differences in the functions distribution by each actor are significant and therefore dependent on individual multimodal discourse. In cinematic discourse, variance analysis in three sets, speech functions in argumentation, speech functions in description, functions in gesture revealed the following results: for argumentation (101–123) F(4, 22) = 19.2 at p < .001, for description (201–219) F(4, 18) = 23.9 at p < .001, for gestures (301–314) F(4, 13) = 15.7 at p = 0.003. In interview discourse, variance analysis revealed the following results: for argumentation (101–123) F(4, 22) = 17.9 at p = 0.001, for description (201–219) F(4, 18) = 36.4 at p < .001, for gestures (301–314) F(4, 13) = 19.9 at p < .001. The results suffice to claim that in both discourse types, the functions of speech and gesture display significant variance; interestingly, the highest variance values appeared in the functions of description, especially in the interview discourse. The lowest variance was in the functions of gestures in cinematic discourse, which specifies our earlier findings in the way that gesture functions are not only synchronized with similar speech functions but are also more restricted and allow fewer alternatives in their selection.
At the second step, we find out whether the speech and gesture functions distribution is similar with the actors. To find the answer, we contrasted the function distribution in three data sets (functions in argumentation, functions in description, and functions in gesture) in both discourse types, cinematic and interview. To do this, we introduced a grouping variable, 0 and 1 (for cinematic and interview discourse). In Tables
Actor | F | df1 | df1 | p |
---|---|---|---|---|
A. Batalov | 0.43 | 1 | 43.8 | 0.515 |
S. Bondarchuk | 1.783 | 1 | 44 | 0.189 |
R. Bykov | 1.948 | 1 | 27.6 | 0.174 |
Yu. Nikulin | 2.348 | 1 | 43.5 | 0.133 |
V. Tikhonov | 6.801 | 1 | 26.9 | 0.015 |
Actor | F | df1 | df1 | p |
---|---|---|---|---|
A. Batalov | 0.4352 | 1 | 34.6 | 0.514 |
S. Bondarchuk | 0.0429 | 1 | 33.5 | 0.837 |
R. Bykov | 3.7269 | 1 | 26.8 | 0.064 |
Yu. Nikulin | 0.4049 | 1 | 35.8 | 0.529 |
V. Tikhonov | 0.9128 | 1 | 27 | 0.348 |
In terms of functions in argumentation, there is one actor, V. Tikhonov, whose speech behavior displays significant variance. This may happen only because his monologues display a different communicative character which is true since his speech patterns clearly display fewer opinion and emotional assessment in interview. However, in terms of description functions, all 5 actors displayed similar speech behavior. The most interesting results appear with the distribution of gesture functions. Three actors out of 5 displayed significant differences in gesture functions’ selection in cinematic and interview discourse. The highest variance is attributed to Yu. Nikulin and R. Bykov. The results point out that these are mostly gestures which manifest variance in multimodal discourse of actors in more and less aesthetic discourse and not the speech functions. In a way it supports our previous findings that the inventory of gesture functions in cinematic discourse is more restricted, and there exists higher alignment between gesture and speech functions in cinematic discourse.
Actor | F | df1 | df1 | p |
---|---|---|---|---|
A. Batalov | 4.099 | 1 | 23.3 | 0.055 |
S. Bondarchuk | 3.081 | 1 | 16 | 0.098 |
R. Bykov | 9.252 | 1 | 22.8 | 0.006 |
Yu. Nikulin | 10.045 | 1 | 20.2 | 0.005 |
V. Tikhonov | 0.421 | 1 | 26 | 0.522 |
Therefore, it is possible that these are particular / specific gesture functions that may serve as indicators of aestheticism of cinematic discourse. To find out which functions may fulfill this role and appear in gesture behavior of all the 5 actors, we now turn to analyzing the gesture functions in the actors’ discourse. To perform it, we again introduce a grouping variable, 0 and 1 (for cinematic and interview discourse). In Table
Gesture function | F | df1 | df1 | p |
---|---|---|---|---|
Pointing | 0.935 | 1 | 7.42 | 0.364 |
Touching | 1.514 | 1 | 4.56 | 0.278 |
Holding | 6.881 | 1 | 4.72 | 0.05 |
Molding | 2.667 | 1 | 5.54 | 0.158 |
Acting | 2.359 | 1 | 4.27 | 0.195 |
Embodying | 1.13 | 1 | 4.24 | 0.345 |
Tracing | 3.6 | 1 | 4.57 | 0.122 |
Discourse structuring | 1.561 | 1 | 5.68 | 0.26 |
Discourse representational | 4.971 | 1 | 7.08 | 0.061 |
Discourse emphatic | 2.331 | 1 | 6.95 | 0.171 |
Expressing attitude/evaluation | 0.196 | 1 | 6.41 | 0.672 |
Contact establishing | 6.377 | 1 | 7.86 | 0.036 |
Self-adaptors | 11.157 | 1 | 4.41 | 0.025 |
Object-adaptors | NaN | 1 | NaN | NaN |
The results show that there are three gesture functions that have statistically different distribution with 5 actors and in two discourse types, they are Holding gestures, Contact establishing gestures and Self-adaptors; Discourse representational gestures also play a role. It means that other gesture functions work similarly in both discourse types, and the aesthetic specificity of gesture in cinematic discourse mostly lies within these four types.
The example given in Figure
At the beginning of the sentence (1) while describing the situation, Yu. Nikulin uses Contact establishing gestures to attract attention of the audience to the case being described. Afterwards while giving the description of the fear (2) perceived by him (the German soldier) the actor uses Self-adaptors. At the end of the proposition the actor uses the Holding gesture to show the silence (3), in which the described situation took place. This example demonstrates the variety of co-speech gestures functions used in one proposition.
We selected a fragment employing two propositions with multiple pragmatic functions of description in cinematic discourse with the same actor (Figure
In comparison with interview discourse, Yu. Nikulin does not use many gestures while giving the description in cinematic discourse. In proposition (1) no gestures were identified, whereas in proposition (2) the actor uses one gesture with the function of expressing attitude / evaluation while describing the way a lot of people were forced to sleep in one house.
It is also noticeable that Holding gestures, Contact establishing gestures, Self-adaptors and Discourse representational gestures irregularly appear in cinematic discourse, being common only for the discourse of interview. At the same time, other gestures appear frequently in both discourse types, for instance, Discourse emphatic gestures. Considering this fact, we distinguish one more function framework structure revealed in contrastive analysis of individual difference – that is REGULARITY – IRREGULARITY. It describes the function frameworks with regular or irregular function distribution within the multiple samples of multimodal discourse of the same or contrasting types. At this stage we cannot account for providing a sufficient description of this structure since the research data need to be much extended to become relevant. Nevertheless, even with these data we still managed to detect the regular patterns of function distribution which may evidence in favor of this structure. The function distribution in cinematic discourse has shown more regularity since it was better predicted. As we have revealed, the lowest variance was observed with the functions of gestures in cinematic discourse and this variance displayed regularity among all 5 male actors.
The study develops a methodological approach of contrasting multimodal discourse as displaying higher or lower aestheticism, here manifested in speech and gesture in cinematic and interview discourse. Following R.
We hypothesized that higher and lower aestheticism can be established via communicative functions in multimodal discourse of speech and gesture, and therefore addressed more and less aesthetic discourse types, the cinematic and the interview discourse pursuing an idea of elaborating a suitable instrument for studying their variance. The contrastive study revealed different activity of pragmatic and discourse-structuring functions in speech, and pragmatic, deictic, representational and adaptive functions in gesture. At the same, we found out that the function activity cannot be viewed as a reliable criterion for distinguishing the discourse types in terms of the multimodal behavior, since the activity of all functions does not display significant variance. For this reason, we explored the function framework structure (
The devised function framework may be applicable to contrastive discourse studies exploring multimodal resources and can help describe other social or cultural functions besides the aesthetic function. Additionally, it may reveal significantly more regularities and specifics of the aesthetic function implemented in multiple aesthetic discourse formats. The obtained results may be applicable to studying multimodal behavior as part of different discourses, for instance the cinematic discourse employing not only “Gesture” (speech and gesture of actors) but also “Image” (cinema shots).
This research was supported by the Ministry for Education in Russia, project No. 075-03-2020-013/3 “Multimodal analysis of communicative behavior in different types of spoken discourse” and was carried out at the Centre for SocioCognitive Discourse Studies at Moscow State Linguistic University.
We are grateful to two anonymous reviewers for their critical remarks which have helped to make this study more consistent and convincing. We also thank Olga Iriskhanova, the Head of SCODIS labs of Moscow State Linguistic University for her constructive ideas.