Print
On counterfactuality: a multimodal approach to (apparent) contradictions between positive statements and gestures of negation
expand article infoMaíra Avelar, Beatriz Graça
‡ Universidade Estadual do Sudoeste da Bahia, Vitóra da Conquista, Brazil
Open Access

Abstract

Relying on discussions about recurrent gestures and gestures of negation, in this paper, we aim to demonstrate how the apparent contradiction between negative gestural utterances co-occurring with positive spoken utterances can be explained with the concepts of counterfactuality and epistemic stances, developed in the Mental Spaces Theory framework. To illustrate how gestures of negation can be analyzed as a case of multiple blends and be metaphorically interpreted, we chose three examples of co-occurrences of a positive verbal and negative gestural utterance. Specifically for the discussion proposed here, we selected three videos from the Brazilian TV show “What the hell is this story, Porchat?” (“Que história é essa, Porchat?”). To analyze the data we used the Linguistic Annotation System for Gestures (LASG) and focused on gestural forms and functions, as well as their semantic relation with the speech. The results showed that in all three videos gestures perform a metacommunicative function. Thus, they can be categorized as pragmatic and discursive gestures, realizing specific performative or operational functions. The sweeping away gestures found in two occurrences work on the discursive level to emphasize implicit counterfactuality of the verbal utterance. The throwing away gesture, found in one occurrence, works on the pragmatic level, also to dismiss the positive possibility created by the verbal utterance. In both cases the gesture operates to prevent any possibility of creating an alternative positive mental space, also demonstrating the implicit counterfactuality of the positive verbal utterances.

Key Words

recurrent gestures, conceptual blending, counterfactuality, epistemic stances, negation, multimodality

Introduction

Based on the project “Towards a Grammar of Gesture”, developed for gestures in German, we started the project “Towards a Grammar of Gestures for Brazilian Portuguese” several years ago. In this paper, we present a specific issue that emerged from the construction of the Repertoire of Brazilian Portuguese Gestures of Negation. In most cases, these gestures co-occur with a negative spoken utterance. However, we found some examples with a positive utterance and a gesture of negation. In order to explain this apparent contradiction, we used the Mental Spaces (Fauconnier, 1994) and the Conceptual Blending Theories (Fauconnier & Turner, 2002), demonstrating how counterfactuality, as well as the pragmatic function performed by gestures, can provide a coherent interpretation for this kind of multimodal utterance.

The paper is organized as follows: in the “Theoretical background”, we consider gestures of negation (Bressem & Müller, 2014; Bressem, Stein & Wegener, 2015) as recurrent gestures in Brazilian Portuguese. Furthermore, we present the concepts of negation, counterfactuality and alternativity, relating them to the Mental Space Theory (Fauconnier, 1994) and the Conceptual Blending Theory (Fauconnier & Turner, 2002), and also to epistemic stances (Dancygier, 2012). After that, in “Material and Methods”, we contextualize our data collection and present our methodological procedure, the Linguistic Annotation System for Gestures (Bressem, Ladewig & Müller, 2013). Finally, in “Results and discussion” we present a qualitative discussion of the samples selected for analysis, showing how apparently contradictory verbal and gestural utterances can be blended in a coherent interpretation.

Theoretical background

Gestures of negation: A case of recurrent gestures

Before approaching our object of analysis, gestures of negation, it is worth mentioning that we understand gestures in Kendon’s (2004) terms of visible action. Even though verbal and visual utterances (Kendon, 2004) can be regarded as belonging to different modalities, they are also considered to be part of a single process (Hostetter & Alibali, 2008; Kendon, 2004; Kok & Cienki, 2016; McNeill, 2005). According to Kendon (2004, pp. 1–2), “These two forms of expression [verbal and gestural] are integrated, produced together under the guidance of a single aim”. Taking into consideration that gestures can be visualized as an excursion with phases, and thus can be segmented and also hierarchized and grouped, they have a “linguistic potential” (Bressem, Ladewig & Müller, 2013).

Furthermore, as indicated by Bressem (2007), in linguistic and semiotic approaches, gestures are investigated from the perspective of analyzing the structures and regularities guiding the gestures’ forms. “Contrary to psychological research, gestural forms are conceived of as recurring across speakers whilst sharing stable meanings” (Bressem, 2007, p. 4). Following the linguistic and semiotic approaches, Müller (2018) proposes a continuum, from gesture to sign, in which gestures could be grouped1:

In the present study we focus on the Recurrent Gestures, since gestures of negation can be placed in this segment of the continuum. As part of a conventionalization process, recurrent gestures can be described as intermediate (i.e., lying between lexical and grammatical processes) and hybrid gestural forms. According to Ladewig (2014), recurrent gestures share relatively stable features in the form-meaning relation. Nevertheless, their functional characteristics can vary depending both on the communicative situation and on the cultural norms of the speaker’s community. Even so, their regularities provide the basis for semantic grouping, since the motivation of gestural forms is still transparent (Ladewig, 2014). In other words, common features of form and movement, as well as common semantics, make it possible to establish gesture clusters or families (Kendon, 2004).

In terms of basic functions they perform, recurrent gestures can be grouped into referential, performative or pragmatic, modal and parsing gestures. It is pointed out by Ladewig (2014) that gestures which perform the referential function, can be divided into four modes of representation according to how they conceptually link to the verbal utterance they co-occur with (see the Materials and Methods section below). As it was established by Kendon (2004), gestures with performative functions demonstrate either a discourse move or a speech act performed by the speaker. Finally, gestures with the modal function contribute to marking the structures of spoken discourse, demonstrating how a particular unit of verbal discourse should be interpreted (Kendon, 2004).

In the same line of reasoning, Ladewig (2014) clarifies that the performative or pragmatic function refers to the meta-communicative potential of gestures, as they participate in structuring of spoken discourse. Ladewig explains that recurrent gestures usually perform discursive or modal functions, such as demonstrating a certain attitude or position of the speaker in relation to something that is being discussed, performing an illocutionary act (Austin, 1962) and even serving to maintain speech or to take/assign a speech turn. The repetitive nature of form-function combination makes it possible to group recurrent gestures into gesture families.

Figure 1. 

Continuum from gesture to sign. Source: Müller (2018).

The idea of grouping gestures into families was initially proposed by Kendon (2004), who used the criteria of similarity of movement and shape, as well as common semantic features (semantic theme). To illustrate how the grouping of gestures works, we choose a gesture family proposed by Kendon (2004) – the Proned Hand Family (PHF), since its semantic theme is related to negation.

Kendon (2004) argues that the PHF gestures are used in situations in which an action is denied or interrupted, and also in conversational contexts in which something is being evaluated. Members of this gesture family can be seen in Figure 2:

Figure 2. 

Open Hand Prone Gestures for denial or interruption. Source: Kendon (2004, p. 250).

Depending on the orientation of the palm, Kendon (2004) classifies the Open Hand Prone gestures into two categories:

  1. Vertical Palm Gestures – VP, a category that groups the gestures in which the forearm is vertical and the wrist is extended so that the palm of the hand turns to the opposite side of the speaker. Gestures of this category are used in contexts in which a speaker demonstrates the intention to interrupt or suspend his own line of action or the line of action of his interlocutor(s) (Kendon, 2004).
  2. Horizontal Palm Gestures – ZP, a category that groups gestures in which the palm is suspended and directed downwards directly away from the speaker’s body. Gestures of this category are used in contexts in which a line of action is being interrupted by external conditions that are not under the control of the speaker(s) (Kendon, 2004).

Based on the analysis of representational gestures and Gesture Families initially proposed by Kendon (2004), Müller (2014) reconstructs a gesture family combining semiotic and distributional analyses of recurrent gestural forms in a variety of contexts of use. Within this framework the concept of a semantic theme has developed which, on the one hand, unifies a gestural family and, on the other hand, identifies variations in meaning that distinguish its different members.

Expanding on the ideas introduced by Kendon (2004), Bressem and Müller (2014) offered a study of recurrent gestures for the German language and singled out a gesture family named “the Away Family”. The gestures of this family share an underlying action of removing or keeping away something that is close to or approaching the body, so that the space around the body remains free. According to Kendon, “the ‘open’ hand is moved (…) away from the mid-line of the actor’s body, perhaps deriving from the action of cutting something through, knocking something away or sweeping away irregularities from a surface” (Kendon, 2004, p. 263)

As Bressem and Müller (2014) indicate, the Away Family is semantically connected by the themes of Rejection, Refusal, Negative Assessment and Negation. According to the authors, what unifies the Away Family is not the shape, position or movement of the hands, but rather the act or effect of moving or keeping unwanted objects, entities, ideas or actions away from the immediate gestural space. This effect can be considered as the motivation for gesture forms of negation. The Away Family consists of four recurrent gestures: sweeping away, holding away, brushing away and throwing away gestures (Bressem & Müller, 2014).

Counterfactuality, alternativity and epistemic stances

In the theoretical framework of Mental Spaces, negation is understood as a phenomenon that sets up counterfactual scenarios (Fauconnier, 1994). Counterfactuality can be lexically explained by space builders, such as negative expressions – e.g., “no”, “avoid” etc. For example, the sentences “Luckily, the fire was prevented from crossing the highway. My house would have been destroyed” (Fauconnier, 1994, p. 110) involve two scenarios: an imaginary one, in which the fire was not controlled and the speaker's house was destroyed, and the real one (the Ground or, according to the author, the Real Space), in which the speaker’s house was prevented from being destroyed.

In Fauconnier and Turner’s (2002) terms, counterfactual means an input space that fundamentally has a vital relation of Disanalogy with another input space. Some other vital relations are also recurrent in counterfactuality, such as Identity and Cause and Effect. For example, the sentence “The stack of books has not fallen” (Fauconnier & Turner, 2002, p. 245) presents network that contains one input space, in which nothing happens, and another input space, in which something happens. In the blended space, “nothing happens” becomes an event that is contrasted with the other event in which something happens, that is, the intact stack of books is contrasted with the pile of falling books. In the blend, “nothing happens” becomes an event that is contrasted with the other event: The stack of books stays upright vs. The stack of books falls down.

According to Fauconnier and Turner (2002), counterfactuality, being an important type of conceptual mappings, provides accessibility of the mental spaces involved in blending. For example, to understand even a simple sentence, such as “There is no milk in the refrigerator”, the interlocutors should build a network with counterfactual spaces: there is milk in the refrigerator in the desired space, whereas in the input space of reality the milk is absent. Thus, the blended space has a counterpart for the milk, which results in the compression by disanalogy between the input spaces, corresponding to the property absent in the blend. The blend is counterfactual with respect to the desired input space.

From this perspective, negation is understood as a primary example of alternativity, since negative expressions set up two Mental Spaces instead of one: the negative space described in the sentence and its positive alternative (Dancygier, 2012). In the sentence “There is no milk in the refrigerator”, the particle “no” sets up a negative mental space (“there is no milk”) which makes sense in a context in which the presence of milk in the refrigerator is expected or cognitively accessible, and a positive alternative space in which one “has, had or should have milk in the refrigerator”.

Thus, when negation is used, there are at least two alternatives to which the speaker, as well as his interlocutor(s), can align epistemically (Dancygier, 2012). In the case of an utterance like “I don’t think I’m right; I know I am”, by denying a stance verb in the first sentence (“I don’t think I’m right”) the speaker may initially be aligning epistemically with the negative alternative, that is, the speaker believes him/herself to be wrong. However, the verb “know”, which is another verb of stance-taking used in the second sentence (“I know I am”), breaks the listener’s expectations, since the speaker actually demonstrates that (s)he is sure (s)he is right.

In contrast, in the sentence “I didn’t buy a car. There was no room for it in the garage” (Dancygier, 2012, p. 70, bold type in the original), the speaker maintains what is already expected, since the fact that he did not buy a car, explained in the first sentence, is only corroborated in the next statement. This effect is achieved through the use of the negative particle and the pronoun “it” which refers to the category “car” that belongs to the mental space being denied. Dancygier (2012) states that negative forms can interact in various ways with a series of grammatical constructions. In addition, negative constructions have an intersubjective role and are used for negotiating various viewpoints available in a specific context (Dancygier, 2012).

Mental spaces, gestures of negation and multimodal utterances

Conceptual Blending Theory provides a theoretical framework to deal with how conceptualizations can happen on a human scale. The theory proposes four mental spaces that are projected and interact with each other (Fauconnier & Turner, 2002): i) a generic space; ii) two input spaces; and iii) a blended space. However, conceptual blending is a dynamic operation that can occur in an unlimited number of Mental Spaces that can repeatedly apply their inputs, and blended spaces that can work as other inputs for a new conceptual network, creating, thus, other types of integration (Fauconnier & Turner, 2002). In that sense more complex integration networks correspond to multiple integrations that can set up multiple inputs and successive integrations in which Blended Spaces can function as inputs. This operation is illustrated with the following example:

Figure 3. 

Multimodal representation of a holding away gesture.

By performing a holding away gesture, the speaker creates with his open hand and palm facing away from his body, a metaphoric barrier between him and the rejected topic, and keeps the space around him free from what he qualifies as unwanted. This occurrence, according to the Mental Space Theory, involves multiple integration operations, as illustrated in Figure 4:

Figure 4. 

Multiple integration network in the sentence “I don’t need a tribute!” with a gesture of negation.

In the diagram we present iconic mappings that allow us to understand the speaker’s open hand as the representation of a barrier. In other words, in Input 1 the hand creates a gestural form and the iconic motivation to enact the construction of a barrier that corresponds to Input 2. The correspondence between these two inputs from the generic space results in a blended space A. The emergent frame of this blend, compressed by similarity between input spaces 1 and 2, results in blended space A “hand as a barrier”. In blended space B, which is metaphoric, the source and target domains correspond to input 3 – in fact, to input 2 of blended space A. Thus, blended spaces A and B share one input space that corresponds to the iconic motivation of the gesture. Specifically, in blended space B this input (Input 3) corresponds to the source domain of the metaphor.

In addition, the second input of this blended space (Input 4) corresponds to the verbal content or target domain of the metaphor. Thus, in blended space B the emergence of the blended structure is also a result from compression by similarity which creates the frame of metaphorical interpretation of the barrier as rejection. From these two operations, the two blends (A and B) now function as inputs for the creation of a third blended space – blended space C – in which the open-hand with the palm held away expresses the interruption or rejection of an action or idea.

In general, such gestures co-occur with negative utterances – with the words or prefixes of negation described earlier. In this sense, the third blended space can work as an input space, containing a metaphoric visual utterance of negation, whereas the second input space, with the space builder “no”, would work as a negative mental space evoked by the sentence and its positive counterpart that is negated or rejected. By saying “I don’t need a tribute”, in this specific context of Figure 3, the speaker rejects the idea that was previously mentioned by his interlocutors, of him being honored with a tribute when he dies. In the blended space there is a compression by Analogy (Fauconnier & Turner, 2002), in which the non-event of the tribute emerges as a frame of interpretation.

Materials and methods

The examples selected for analysis were collected from a larger database (Graça, 2021), composed of 53 verbo-gestural occurrences of negation in Brazilian Portuguese. This database consists of TED talks, retrieved from YouTube, face-to-face interactions in interview programs, televised conversations, and news broadcasts on varied subjects. The last two sources of data were collected in the Distributed Little Red Hen Lab library2. Two procedures were adopted to select the video data:

  1. the search for keywords, such as “no”, “never”, and prefixes of negation in Brazilian Portuguese, as described in the following table (Table 1):
  2. the search for gesture forms that compose the German repertoire, not including the verbal utterance at first. We found three occurrences in which gestures of negation co-occur with positive verbal utterances. This is not the default since these kinds of gestures are mostly described as occurring with negative verbal utterances.

Regarding the methodological procedures adopted to analyze and annotate the occurrences of gestures of negation in multimodal data of the Brazilian Portuguese, we used the Linguistic Annotation System for Gestures (LASG), developed by Bressem, Ladewig and Müller (2013), which provides for specific levels of annotation for the analysis of gestural forms and co-speech gestures. This system includes the first three blocks of analysis proposed in the Methods of Gesture Analysis (MGA) (Müller, 2013), which ensure a systematic reconstruction of the fundamental properties of the gestural meaning creation based on form features, by distinguishing four building blocks: i) the form; ii) the sequential structure of gestures in relation to speech and other gestures; iii) the local context of use; iv) the distribution of the gesture over different contexts of use (Bressem et al., 2013, p. 1100).

Table 1.

Keywords used to collect the Red Hen data. Source: Graça, 2021.

Keywords
No (Não)
Neither (Nem)
Never (Nunca)
Nobody (Ninguém)
Never* (Jamais)
None (Nenhuma)
Without (Sem)
Nothing (Nada)
Impossible (Impossível)
Deactivate (Desativar)

The MGA suggests that the meaning of a gesture emerges from the interaction between its shape, its sequential position and its insertion in a context of use (Bressem et al., 2013, p. 1100). This system of gesture analysis is cyclic, which means that the levels of analysis suggested by the MGA do not need to be followed in the order in which they appear (starting from shape to distribution in contexts of use). It is up to the researcher, considering his research question and research goals, to decide at which level to start the analysis. The following figure illustrates the MGA:

The Linguistic Annotation System for Gestures (Bressem, Ladewig & Müller 2013, p. 1100) includes the first three blocks of the MGA (form, sequential position, local context of use) and turns them into annotation tiers applicable to computer annotation tools, such as ELAN (Sloetjes & Wittenburg, 2008). The Linguistic Annotation System for Gestures (LASG) allows the description of gestures, since the system proposes a description of the relation between gesture forms and functions. Furthermore, gestures can be analyzed from the perspective of multimodal grammar, as the system also allows the simultaneous description of gestures and speech.

To narrow down the parameters of form, the authors of the LASG establish four form parameters established for describing sign languages: (1) handshape, such as: open or closed hand, extended or bended index finger; (2) palm-orientation, such as pronated, supinated, horizontal, vertical or diagonal; (3) movement, such as up, down, to the right, to the left, towards and away from the body; and (4) position in the gesture space, such as self-touching gesture (on the speaker’s own body), proximal, medium and distal from the speaker.

According to the authors (Bressem et al., 2013), the system also allows for a description of the gesture modes of representation: (1) enacting, in which the hands move in such way that they represent engagement in a functional act, often one involving manipulating something; (2) embodying, in which the hand stands for the entity it represents by substituting for it; (3) drawing, in which the hand or hands move usually with the tip(s) of the extended finger(s) being the ‘active zone’, moving so as to leave an imagined trace of the form being depicted; (4) holding/molding, in which the hands shape a 3D object; and, for the present research, we added (5) pointing, in which the index finger or the open hand is used to point to something or somewhere (Kendon, 2004; Avelar & Ferrari, 2017).

Regarding the parameters of function, the authors’ proposition is the following: (1) referential gestures that depict concrete objects or abstract entities, ideas, relations or actions; (2) discursive gestures that structure the accompanying verbal utterance, for example by marking emphasis; (3) pragmatic gestures that enact a speech act, such as dismissing an idea (Bressem, Ladewig & Müller, 2013). Finally, considering the distributional context between speech and gesture, the LASG suggests annotating the semantic function of the gesture in relation to the verbal utterance. For this, the system categorizes the semantic functions into: i) function of emphasizing, when the gesture expresses redundant semantic features in relation to speech; ii) function of modifying, when the gesture expresses a complementary semantic features in relation to speech; iii) function of adding, when the gesture carries semantic properties that are contrary to speech; iv) function of replacing, when the gesture expresses a contrary semantic feature in the absence of speech. In sum, gestures can either illustrate or emphasize what was verbally said or modify or replace the verbal meaning (Bressem, Ladewig & Müller, 2013, p. 1112).

To perform the gestural form and function analysis, we created tiers in ELAN (Wittenburg et al., 2008) following the annotation procedures proposed by the system: at first, in order not to be biased by the verbal content, the sound of the video was turned off and the gesture stroke form was described; next, the sound was turned on and, after the orthographic transcription using the GAT2 (2009) system was made, gesture modes of representation were categorized. Finally, gestures were analyzed along with the linguistic context with which they co-occurred, and the gestural functions were identified.

The annotation categories used in the research are presented in Table 2:

Table 2.

Annotation of the data in ELAN according to the parameters suggested in the LASG.

Palm orientation Vertical Down
Movement Straight Straight
Movement Direction Away body Both sides
Movement quality Smooth Sharp
Position Upper right periphery Center
Word class Adverb of negation Adverbial Phrase
Mode of representation Representing Enacting
Motivation - action Action of building a barrier Action of cleaning a surface
Motivation – image schema BLOCKAGE CENTER-PERIPHERY
Context of use Ground: the speaker is talking about football players who are honored when they die and emphatically states that he doesn't need this kind of tribute. Ground: The speaker performs a Sweeping Away Gesture to emphasize an assertive response
Temporal position Parallel Parallel
Gesture function Pragmatic Pragmatic
Semantic relation Complementary Contrary
Semantic function Modifying Emphasizing
Position in turns Beginning Middle
Pragmatic function Modal Operational
Transcription See the transcription in Brazilian Portuguese and in English in Fig. 3. See the transcription in Brazilian Portuguese and in English in Fig. 6.
Palm orientation Down Down
Movement Straight Arced
Movement Direction Both sides Down
Movement quality Sharp Sharp
Position Center Center
Word class Adverb of time Interjaction
Mode of representation Enacting Enacting
Motivation – action Action of cleaning a surface Action of throwing something away
Motivation – image schema CENTER-PERIPHERY SOURCE-PATH-GOAL
Context of use Narrative/description: the presenter describes the format of the guest's show, to ask why her very famous show always had the same format Narrative/Description: the speaker reports what other people often say when they hear a "horror story"
Temporal position Parallel Parallel
Gesture function Pragmatic Pragmatic
Semantic relation Complementary Complementary
Semantic function Modifying Modifying
Position in turns Middle Beginning
Pragmatic function Operational Modal
Transcription See the transcription in Brazilian Portuguese and in English in Fig. 8. See the transcription in Brazilian Portuguese and in English in Fig. 10.

Results and discussion

In this section we present an analysis of three occurrences of positive verbal utterances along with negative gestural utterances. All of them were collected from the YouTube channel of the talk-show "What the hell is this story, Porchat?" (“Que história é essa, Porchat?”3) presented by the actor and comedian Fábio Porchat. In that show three famous people are invited to tell bizarre comic stories that happened to them. The occurrences analyzed are from episodes shown on different days.

The first occurrence consists of a sweeping away gesture, performed by actress Taís Araújo when responding positively to a question about criticism she received during her career. Sweetser (2006, p. 13) states that negative constructions are capable of evoking alternative Mental Spaces in a way that positive constructions do not evoke. However, in the case of the occurrence in question, the speaker makes a statement, but a counterfactual mental space is configured by combining the positive spoken utterance with the gesture of negation that co-occurs with the speech. We present the example below:

Figure 5. 

Diagram of the Methods of Gesture Analysis. Source: Barbosa (2020), adapted from Müller (2019).

According to the representation proposed in the Blending Space Theory, the verbal and gestural input spaces, as well as the blended space, can be represented in the following way:

The actress states, with a positive assertive stance (Dancygier, 2012), that she will “for sure” continue receiving criticism. The verbal utterance is jointly performed with a sweeping gesture utterance. By doing that, the actress triggers, in the gestural input space, a counterfactual mental space. However, this gesture operates on the discursive level, exercising an operational function (Kendon, 2013). The gesture excludes any contrary possibility to what it is being argued. Thus, the speaker expresses through the gesture that there is not the slightest chance she will not be criticized again. This multimodal construction, composed of gesture and spoken utterance, reveals an assertive stance by which the speaker takes a stand (Kendon, 2013, p. 171). Moreover, as proposed by the LASG, this occurrence is evidence of the semantic relationship of emphasizing the spoken utterance. This leads to the interpretation of the verbal utterance having an implicit counterfactual relation, in which the “absence of possibility”, that is, the impossibility is emphasized by the gesture. It is worth mentioning that the two-handed gesture also marks emphasis given by the speaker to the utterance.

Figure 6. 

Multimodal representation of a sweeping away gesture (sample 1).

Figure 7. 

Diagram representing the mental operation of sample 1.

In the second example host Fábio Porchat asks a question to Marília Gabriela, a famous Brazilian TV interviewer, and performs a sweeping away gesture combined with the statement that her shows “always” had the same format. The example is presented in Figure 8:

Figure 8. 

Multimodal representation of a sweeping away gesture (sample 2).

By using the word “always” combined with the sweeping away gesture, the host configures a counterfactual alternative mental space, since the multimodal construction conveys a negative sense, that is, the fact that Marília Gabriela’s shows have never had a different format. In that sense, in the counterfactual mental space, there is a positive counterpart, denied by the gesture, which indicates the possibility of Marília Gabriela’s programs having other formats different from the formats already known. Thus, it is possible to assert that the meaning of the gesture remains the same, despite what is being said. That is, the sweeping away gesture, in this occurrence, is still a gesture of negation with an operational function, even co-occurring with a positive utterance, because it operates at the discursive level and not on the structure of the spoken utterance; the speaker implicitly builds a counterfactual space, referring to other possible formats of the show presented by Marília Gabriela and using the gesture eliminates these possibilities (Kendon, 2013, p. 15). As in the previous example, emphasis is marked through the use of both hands.

In this occurrence, the representation proposed by the Blending Space Theory, the verbal and gestural input spaces, as well as the blended space can be represented as in Figure 9:

Figure 9. 

Diagram representing the mental operation of sample 2.

Kendon (2004) also discusses occurrences of this kind and argues that a sweeping away gesture can be used when we make a positive assessment. This can be considered contradictory, and the gesture, in this case, could be just an intensifier. However, the author (2004) argues that in these occurrences gestures function as intensifiers, but only because they carry out an implicit negative meaning. A positive assessment which carries the implication that nothing else is as good as the object evaluated is an intensified evaluation. If the speaker uses a ZP gesture or a head nodding (or both) when making a positive assessment of something, the gesture intensifies their assessment. When using them, the speaker implies that there is nothing else that can be similar in terms of value.

The following occurrence is taken from another episode of the program and consists of a story told by an audience participant about a supernatural experience she had as a child.

Figure 10. 

Multimodal representation of a throwing away gesture (sample 3).

In this episode, the speaker states that people do not usually believe her when she tells this type of story. In doing so, she enacts the type of reaction that people usually have in situations like that and performs the throwing away gesture with a performative function. In this example, analyzed in terms of the Blending Theory, the spaces of verbal and gestural input, as well as the blended space can be represented as in Figure 11:

Figure 11. 

Diagram representing the mental operation of sample 3.

The gesture in question reveals a negative epistemic stance conveyed in the speaker’s reported speech and provides a frame of interpretation for the verbal utterance, manifesting the illocutionary force (Austin, 1962) of what is being verbally conveyed. In this case, the utterance, which corresponds to the speech of a third person, is a positive utterance: “Oh, nice story”. However, the gesture that accompanies the reported speech, indicates that something in this specific example, the supernatural experience reported by the speaker, is regarded as irrelevant by the character she performs, representing the viewpoint of other people who listened to the story. The performative function of dismissal performs an illocutionary force upon the positive locution, demonstrating the implicit counterfactuality of the positive statement. As proposed by the LASG, the gesture has an additive semantic property, pragmatically marked by the illocutionary act. In this case, the gestural utterance demonstrates the dismissal of the positive assessment expressed by the verbal utterance, and, consequently, the dismissal of the story told by the speaker. In the blended space, the negative act of dismissing something is highlighted – that is the fact that the narrative is treated as irrelevant.

Conclusion

As it was shown by the analysis, recurrent gestures, in general, are gestures employed in a metacommunicative way that operate on the speaker’s utterance. They can serve an operational, a parsing, a performative or a modal function. When they perform an operational function, these gestures act as an operator in relation to the spoken utterance (Kendon, 2013, p. 15). A very common example of a gesture working as an operator, as shown by the data, is a sweeping gesture - a gesture commonly used in conjunction with negative constructions or constructions that imply a negative circumstance, but which is also performed in conjunction with strong positive statements, as if the hands served to prevent any attempt to deny what was said (e.g., samples 1 and 2).

When they perform modal and performative functions, recurrent gestures provide an interpretation frame for a certain excerpt of the spoken discourse and may also reveal an attitude or stance of the speaker in relation to what is being said or done. An example of this is the quotation mark gesture, commonly used to highlight a portion of the spoken utterance (Kendon, 2013, p. 16) and mark it as irony or sarcasm. Gestures with modal function are strongly connected with the verbal unit on which they operate (sample 3).

In general, the study has confirmed that recurrent gestures convey semantic and pragmatic information and can work as discursive markers or even as grammatical and lexical morphemes (Ladewig, 2014, p. 1567). In addition to these properties, the stable structure of form and meaning allows recurrent gestures used in different cultural contexts to compose gesture families such as the “Away Family” addressed throughout this article, which focused specifically on what we named as gestures of negation that co-occurred with positive verbal statements.

In addition, the Conceptual Blending Theory was shown to be a productive model to approach the apparent contradiction between positive verbal utterances and negative gestural utterances, since the notions of counterfactuality and alternativity demonstrate that gestures of negation can perform operational functions along with positive statements, working as intensifiers of the verbal utterance. They can also carry out performative functions, when co-occurring with verbal utterances that reveal epistemic stances of negative assessment.

In both kinds of gestures discussed here, the gestures perform pragmatic and discursive functions, operating in a metacommunicative way. In sum, recurrent gestures convey conventionalized gestural patterns that are very important to the meaning of multimodal constructions.

Acknowledgements

The majority of this research was conducted with the support of the Coordination for Improvement for Higher Education (CAPES Foundation, Brazil). We are grateful for the grant awarded to us (Code 001).

References

  • Austin J. (1962). How to do things with words. Oxford: Clarendon Press.
  • Bressem J. (2007). Recurrent form features in coverbal gestures. Talk at the Third International Conf. of the International Society for Gesture Studies (ISGS).
  • Bressem J., Ladewig S. H., Müller C. (2013). Linguistic Annotation System for Gestures. In C. Müller, A. Cienki, E. Fricke, S. H. Ladewig, D. McNeill, & S. Teßendorf (Eds.), Body – language – communication. An international handbook on multimodality in human interaction, pp. 1098–1024. Berlin/Boston: De Gruyter Mouton. https://doi.org/10.1515/9783110261318.1098
  • J. , Müller C. (2014). The family of Away gestures: Negation, refusal, and negative assessment. In C. Müller, E. Fricke, A. Cienki, S. H. Ladewig, D. McNeill, & J. Bressem (Eds.), Body – language – communication: An international handbook on multimodality in human interaction, pp. 1592–1604. Berlin/Boston: De Gruyter Mouton. https://doi.org/10.1515/9783110302028.1592
  • Bressem J., Stein N., Wegener C. (2015). Structuring and highlighting speech–discursive functions of holding away gestures in Savosavo. In G. Ferré & M. Tutton (Eds.), Proceedings of GESPIN, Nantes. https://hal.archives-ouvertes.fr/hal-01195646
  • Dancygier B. (2012). Negation, stance verbs, and intersubjectivity. In B. Dancygier & E. Sweetser (Eds.), Figurative Language, pp. 69–93. Cambridge: Cambridge University Press. https://doi.org/10.1017/CBO9781139084727.006
  • Fauconnier G., Turner M. (2002). The way we think: Conceptual blending and the mind’s hidden complexities. New York: Basic Books.
  • Graça B. (2021). A construção de um repertório de gestos de negação do Português Brasileiro: uma proposta cognitivo-gestual (master’s thesis). Universidade Estadual do Sudoeste da Bahia, Vitória da Conquista, Bahia, Brazil.
  • Kendon A. (2013). Exploring the utterance roles of visible bodily action: A personal account. In C. Müller, A. Cienki, E. Fricke, S. H. Ladewig, D. McNeill, & S. Teßendorf (Eds.), Body – language – communication. An international handbook on multimodality in human interaction, pp. 7–28. Berlin/Boston: De Gruyter Mouton. https://doi.org/10.1515/9783110261318.7
  • Ladewig S. H. (2014). Recurrent gestures. In C. Müller, E. Fricke, A. Cienki, S. H. Ladewig, D. McNeill, & J. Bressem (Eds.), Body – language – communication: An international handbook on multimodality in human interaction, pp. 1558–1574. Berlin/Boston: De Gruyter Mouton.
  • Müller C. (2013). Gestures as a medium of expression: The linguistic potential of gestures. In C. Müller, A. Cienki, E. Fricke, S. H. Ladewig, D. McNeill, & S. Teßendorf (Eds.), Body – language – communication. An international handbook on multimodality in human interaction, pp. 202–217. Berlin/Boston: De Gruyter Mouton. https://doi.org/10.1515/9783110261318.202
  • Selting M. et al. (2009) Gesprächsanalytisches Transkriptionssystem 2 (GAT2). In: Gesprächsforschung 10, 353–402. [Retrieved from:] www.gespraechsforschung-ozs.de
  • Sloetjes H., Wittenburg P. (2008). Annotation by category - ELAN and ISO DCR. In: Proceedings of the 6th International Conference on Language Resources and Evaluation (LREC 2008).
  • Sweetser E. (2006) Negative space: levels of negation and kinds of spaces. In Bonefille, S.; Salbayre, S. (eds.). Proceedings of the conference “Negation: Form, figure of speech, conceptualization”. Publication du groupe de recherches angloamericaines de l’Universit´e de Tours. Tours: Publications Universitaires François. Rabelais.

1 Kendon (1980) proposed a basic continuum, which McNeill (1992) popularized, calling it “Kendon’s continuum”. Cienki (2015) already proposed that recurrent gestures should be seen as having an intermediate place on the Kendon’s continuum as well.
3 In informal Brazilian Portuguese, the expression “What is this story?” has the meaning of doubting a little what is being said or questioning another’s attitude. In the context of the show’s name, it has a more literal meaning, and also the meaning explained above.
login to comment