Relying on discussions about recurrent gestures and gestures of negation, in this paper, we aim to demonstrate how the apparent contradiction between negative gestural utterances co-occurring with positive spoken utterances can be explained with the concepts of counterfactuality and epistemic stances, developed in the Mental Spaces Theory framework. To illustrate how gestures of negation can be analyzed as a case of multiple blends and be metaphorically interpreted, we chose three examples of co-occurrences of a positive verbal and negative gestural utterance. Specifically for the discussion proposed here, we selected three videos from the Brazilian TV show “What the hell is this story, Porchat?” (“Que história é essa, Porchat?”). To analyze the data we used the Linguistic Annotation System for Gestures (LASG) and focused on gestural forms and functions, as well as their semantic relation with the speech. The results showed that in all three videos gestures perform a metacommunicative function. Thus, they can be categorized as pragmatic and discursive gestures, realizing specific performative or operational functions. The sweeping away gestures found in two occurrences work on the discursive level to emphasize implicit counterfactuality of the verbal utterance. The throwing away gesture, found in one occurrence, works on the pragmatic level, also to dismiss the positive possibility created by the verbal utterance. In both cases the gesture operates to prevent any possibility of creating an alternative positive mental space, also demonstrating the implicit counterfactuality of the positive verbal utterances.

recurrent gestures, conceptual blending, counterfactuality, epistemic stances, negation, multimodality

Based on the project “Towards a Grammar of Gesture”, developed for gestures in German, we started the project “Towards a Grammar of Gestures for Brazilian Portuguese” several years ago. In this paper, we present a specific issue that emerged from the construction of the Repertoire of Brazilian Portuguese Gestures of Negation. In most cases, these gestures co-occur with a negative spoken utterance. However, we found some examples with a positive utterance and a gesture of negation. In order to explain this apparent contradiction, we used the Mental Spaces (Fauconnier, 1994) and the Conceptual Blending Theories (Fauconnier & Turner, 2002), demonstrating how counterfactuality, as well as the pragmatic function performed by gestures, can provide a coherent interpretation for this kind of multimodal utterance.

The paper is organized as follows: in the “Theoretical background”, we consider gestures of negation (Bressem & Müller, 2014; Bressem, Stein & Wegener, 2015) as recurrent gestures in Brazilian Portuguese. Furthermore, we present the concepts of negation, counterfactuality and alternativity, relating them to the Mental Space Theory (Fauconnier, 1994) and the Conceptual Blending Theory (Fauconnier & Turner, 2002), and also to epistemic stances (Dancygier, 2012). After that, in “Material and Methods”, we contextualize our data collection and present our methodological procedure, the Linguistic Annotation System for Gestures (Bressem, Ladewig & Müller, 2013). Finally, in “Results and discussion” we present a qualitative discussion of the samples selected for analysis, showing how apparently contradictory verbal and gestural utterances can be blended in a coherent interpretation.

The examples selected for analysis were collected from a larger database (Graça, 2021), composed of 53 verbo-gestural occurrences of negation in Brazilian Portuguese. This database consists of TED talks, retrieved from YouTube, face-to-face interactions in interview programs, televised conversations, and news broadcasts on varied subjects. The last two sources of data were collected in the Distributed Little Red Hen Lab library². Two procedures were adopted to select the video data:

1. the search for keywords, such as “no”, “never”, and prefixes of negation in Brazilian Portuguese, as described in the following table (Table 1):
2. the search for gesture forms that compose the German repertoire, not including the verbal utterance at first. We found three occurrences in which gestures of negation co-occur with positive verbal utterances. This is not the default since these kinds of gestures are mostly described as occurring with negative verbal utterances.

Regarding the methodological procedures adopted to analyze and annotate the occurrences of gestures of negation in multimodal data of the Brazilian Portuguese, we used the Linguistic Annotation System for Gestures (LASG), developed by Bressem, Ladewig and Müller (2013), which provides for specific levels of annotation for the analysis of gestural forms and co-speech gestures. This system includes the first three blocks of analysis proposed in the Methods of Gesture Analysis (MGA) (Müller, 2013), which ensure a systematic reconstruction of the fundamental properties of the gestural meaning creation based on form features, by distinguishing four building blocks: i) the form; ii) the sequential structure of gestures in relation to speech and other gestures; iii) the local context of use; iv) the distribution of the gesture over different contexts of use (Bressem et al., 2013, p. 1100).

The MGA suggests that the meaning of a gesture emerges from the interaction between its shape, its sequential position and its insertion in a context of use (Bressem et al., 2013, p. 1100). This system of gesture analysis is cyclic, which means that the levels of analysis suggested by the MGA do not need to be followed in the order in which they appear (starting from shape to distribution in contexts of use). It is up to the researcher, considering his research question and research goals, to decide at which level to start the analysis. The following figure illustrates the MGA:

The Linguistic Annotation System for Gestures (Bressem, Ladewig & Müller 2013, p. 1100) includes the first three blocks of the MGA (form, sequential position, local context of use) and turns them into annotation tiers applicable to computer annotation tools, such as ELAN (Sloetjes & Wittenburg, 2008). The Linguistic Annotation System for Gestures (LASG) allows the description of gestures, since the system proposes a description of the relation between gesture forms and functions. Furthermore, gestures can be analyzed from the perspective of multimodal grammar, as the system also allows the simultaneous description of gestures and speech.

To narrow down the parameters of form, the authors of the LASG establish four form parameters established for describing sign languages: (1) handshape, such as: open or closed hand, extended or bended index finger; (2) palm-orientation, such as pronated, supinated, horizontal, vertical or diagonal; (3) movement, such as up, down, to the right, to the left, towards and away from the body; and (4) position in the gesture space, such as self-touching gesture (on the speaker’s own body), proximal, medium and distal from the speaker.

According to the authors (Bressem et al., 2013), the system also allows for a description of the gesture modes of representation: (1) enacting, in which the hands move in such way that they represent engagement in a functional act, often one involving manipulating something; (2) embodying, in which the hand stands for the entity it represents by substituting for it; (3) drawing, in which the hand or hands move usually with the tip(s) of the extended finger(s) being the ‘active zone’, moving so as to leave an imagined trace of the form being depicted; (4) holding/molding, in which the hands shape a 3D object; and, for the present research, we added (5) pointing, in which the index finger or the open hand is used to point to something or somewhere (Kendon, 2004; Avelar & Ferrari, 2017).

Regarding the parameters of function, the authors’ proposition is the following: (1) referential gestures that depict concrete objects or abstract entities, ideas, relations or actions; (2) discursive gestures that structure the accompanying verbal utterance, for example by marking emphasis; (3) pragmatic gestures that enact a speech act, such as dismissing an idea (Bressem, Ladewig & Müller, 2013). Finally, considering the distributional context between speech and gesture, the LASG suggests annotating the semantic function of the gesture in relation to the verbal utterance. For this, the system categorizes the semantic functions into: i) function of emphasizing, when the gesture expresses redundant semantic features in relation to speech; ii) function of modifying, when the gesture expresses a complementary semantic features in relation to speech; iii) function of adding, when the gesture carries semantic properties that are contrary to speech; iv) function of replacing, when the gesture expresses a contrary semantic feature in the absence of speech. In sum, gestures can either illustrate or emphasize what was verbally said or modify or replace the verbal meaning (Bressem, Ladewig & Müller, 2013, p. 1112).

To perform the gestural form and function analysis, we created tiers in ELAN (Wittenburg et al., 2008) following the annotation procedures proposed by the system: at first, in order not to be biased by the verbal content, the sound of the video was turned off and the gesture stroke form was described; next, the sound was turned on and, after the orthographic transcription using the GAT2 (2009) system was made, gesture modes of representation were categorized. Finally, gestures were analyzed along with the linguistic context with which they co-occurred, and the gestural functions were identified.

The annotation categories used in the research are presented in Table 2:

Table 2.

Download as

CSV

XLSX

Annotation of the data in ELAN according to the parameters suggested in the LASG.

Palm orientation	Vertical	Down
Movement	Straight	Straight
Movement Direction	Away body	Both sides
Movement quality	Smooth	Sharp
Position	Upper right periphery	Center
Word class	Adverb of negation	Adverbial Phrase
Mode of representation	Representing	Enacting
Motivation - action	Action of building a barrier	Action of cleaning a surface
Motivation – image schema	BLOCKAGE	CENTER-PERIPHERY
Context of use	Ground: the speaker is talking about football players who are honored when they die and emphatically states that he doesn't need this kind of tribute.	Ground: The speaker performs a Sweeping Away Gesture to emphasize an assertive response
Temporal position	Parallel	Parallel
Gesture function	Pragmatic	Pragmatic
Semantic relation	Complementary	Contrary
Semantic function	Modifying	Emphasizing
Position in turns	Beginning	Middle
Pragmatic function	Modal	Operational
Transcription	See the transcription in Brazilian Portuguese and in English in Fig. 3.	See the transcription in Brazilian Portuguese and in English in Fig. 6.
Palm orientation	Down	Down
Movement	Straight	Arced
Movement Direction	Both sides	Down
Movement quality	Sharp	Sharp
Position	Center	Center
Word class	Adverb of time	Interjaction
Mode of representation	Enacting	Enacting
Motivation – action	Action of cleaning a surface	Action of throwing something away
Motivation – image schema	CENTER-PERIPHERY	SOURCE-PATH-GOAL
Context of use	Narrative/description: the presenter describes the format of the guest's show, to ask why her very famous show always had the same format	Narrative/Description: the speaker reports what other people often say when they hear a "horror story"
Temporal position	Parallel	Parallel
Gesture function	Pragmatic	Pragmatic
Semantic relation	Complementary	Complementary
Semantic function	Modifying	Modifying
Position in turns	Middle	Beginning
Pragmatic function	Operational	Modal
Transcription	See the transcription in Brazilian Portuguese and in English in Fig. 8.	See the transcription in Brazilian Portuguese and in English in Fig. 10.

In this section we present an analysis of three occurrences of positive verbal utterances along with negative gestural utterances. All of them were collected from the YouTube channel of the talk-show "What the hell is this story, Porchat?" (“Que história é essa, Porchat?”³) presented by the actor and comedian Fábio Porchat. In that show three famous people are invited to tell bizarre comic stories that happened to them. The occurrences analyzed are from episodes shown on different days.

The first occurrence consists of a sweeping away gesture, performed by actress Taís Araújo when responding positively to a question about criticism she received during her career. Sweetser (2006, p. 13) states that negative constructions are capable of evoking alternative Mental Spaces in a way that positive constructions do not evoke. However, in the case of the occurrence in question, the speaker makes a statement, but a counterfactual mental space is configured by combining the positive spoken utterance with the gesture of negation that co-occurs with the speech. We present the example below:

Figure 5. 

Diagram of the Methods of Gesture Analysis. Source: Barbosa (2020), adapted from Müller (2019).

According to the representation proposed in the Blending Space Theory, the verbal and gestural input spaces, as well as the blended space, can be represented in the following way:

The actress states, with a positive assertive stance (Dancygier, 2012), that she will “for sure” continue receiving criticism. The verbal utterance is jointly performed with a sweeping gesture utterance. By doing that, the actress triggers, in the gestural input space, a counterfactual mental space. However, this gesture operates on the discursive level, exercising an operational function (Kendon, 2013). The gesture excludes any contrary possibility to what it is being argued. Thus, the speaker expresses through the gesture that there is not the slightest chance she will not be criticized again. This multimodal construction, composed of gesture and spoken utterance, reveals an assertive stance by which the speaker takes a stand (Kendon, 2013, p. 171). Moreover, as proposed by the LASG, this occurrence is evidence of the semantic relationship of emphasizing the spoken utterance. This leads to the interpretation of the verbal utterance having an implicit counterfactual relation, in which the “absence of possibility”, that is, the impossibility is emphasized by the gesture. It is worth mentioning that the two-handed gesture also marks emphasis given by the speaker to the utterance.

Figure 6. 

Multimodal representation of a sweeping away gesture (sample 1).

Figure 7. 

Diagram representing the mental operation of sample 1.

In the second example host Fábio Porchat asks a question to Marília Gabriela, a famous Brazilian TV interviewer, and performs a sweeping away gesture combined with the statement that her shows “always” had the same format. The example is presented in Figure 8:

Figure 8. 

Multimodal representation of a sweeping away gesture (sample 2).

By using the word “always” combined with the sweeping away gesture, the host configures a counterfactual alternative mental space, since the multimodal construction conveys a negative sense, that is, the fact that Marília Gabriela’s shows have never had a different format. In that sense, in the counterfactual mental space, there is a positive counterpart, denied by the gesture, which indicates the possibility of Marília Gabriela’s programs having other formats different from the formats already known. Thus, it is possible to assert that the meaning of the gesture remains the same, despite what is being said. That is, the sweeping away gesture, in this occurrence, is still a gesture of negation with an operational function, even co-occurring with a positive utterance, because it operates at the discursive level and not on the structure of the spoken utterance; the speaker implicitly builds a counterfactual space, referring to other possible formats of the show presented by Marília Gabriela and using the gesture eliminates these possibilities (Kendon, 2013, p. 15). As in the previous example, emphasis is marked through the use of both hands.

In this occurrence, the representation proposed by the Blending Space Theory, the verbal and gestural input spaces, as well as the blended space can be represented as in Figure 9:

Figure 9. 

Diagram representing the mental operation of sample 2.

Kendon (2004) also discusses occurrences of this kind and argues that a sweeping away gesture can be used when we make a positive assessment. This can be considered contradictory, and the gesture, in this case, could be just an intensifier. However, the author (2004) argues that in these occurrences gestures function as intensifiers, but only because they carry out an implicit negative meaning. A positive assessment which carries the implication that nothing else is as good as the object evaluated is an intensified evaluation. If the speaker uses a ZP gesture or a head nodding (or both) when making a positive assessment of something, the gesture intensifies their assessment. When using them, the speaker implies that there is nothing else that can be similar in terms of value.

The following occurrence is taken from another episode of the program and consists of a story told by an audience participant about a supernatural experience she had as a child.

Figure 10. 

Multimodal representation of a throwing away gesture (sample 3).

In this episode, the speaker states that people do not usually believe her when she tells this type of story. In doing so, she enacts the type of reaction that people usually have in situations like that and performs the throwing away gesture with a performative function. In this example, analyzed in terms of the Blending Theory, the spaces of verbal and gestural input, as well as the blended space can be represented as in Figure 11:

Figure 11. 

Diagram representing the mental operation of sample 3.

The gesture in question reveals a negative epistemic stance conveyed in the speaker’s reported speech and provides a frame of interpretation for the verbal utterance, manifesting the illocutionary force (Austin, 1962) of what is being verbally conveyed. In this case, the utterance, which corresponds to the speech of a third person, is a positive utterance: “Oh, nice story”. However, the gesture that accompanies the reported speech, indicates that something in this specific example, the supernatural experience reported by the speaker, is regarded as irrelevant by the character she performs, representing the viewpoint of other people who listened to the story. The performative function of dismissal performs an illocutionary force upon the positive locution, demonstrating the implicit counterfactuality of the positive statement. As proposed by the LASG, the gesture has an additive semantic property, pragmatically marked by the illocutionary act. In this case, the gestural utterance demonstrates the dismissal of the positive assessment expressed by the verbal utterance, and, consequently, the dismissal of the story told by the speaker. In the blended space, the negative act of dismissing something is highlighted – that is the fact that the narrative is treated as irrelevant.

As it was shown by the analysis, recurrent gestures, in general, are gestures employed in a metacommunicative way that operate on the speaker’s utterance. They can serve an operational, a parsing, a performative or a modal function. When they perform an operational function, these gestures act as an operator in relation to the spoken utterance (Kendon, 2013, p. 15). A very common example of a gesture working as an operator, as shown by the data, is a sweeping gesture - a gesture commonly used in conjunction with negative constructions or constructions that imply a negative circumstance, but which is also performed in conjunction with strong positive statements, as if the hands served to prevent any attempt to deny what was said (e.g., samples 1 and 2).

When they perform modal and performative functions, recurrent gestures provide an interpretation frame for a certain excerpt of the spoken discourse and may also reveal an attitude or stance of the speaker in relation to what is being said or done. An example of this is the quotation mark gesture, commonly used to highlight a portion of the spoken utterance (Kendon, 2013, p. 16) and mark it as irony or sarcasm. Gestures with modal function are strongly connected with the verbal unit on which they operate (sample 3).

In general, the study has confirmed that recurrent gestures convey semantic and pragmatic information and can work as discursive markers or even as grammatical and lexical morphemes (Ladewig, 2014, p. 1567). In addition to these properties, the stable structure of form and meaning allows recurrent gestures used in different cultural contexts to compose gesture families such as the “Away Family” addressed throughout this article, which focused specifically on what we named as gestures of negation that co-occurred with positive verbal statements.

In addition, the Conceptual Blending Theory was shown to be a productive model to approach the apparent contradiction between positive verbal utterances and negative gestural utterances, since the notions of counterfactuality and alternativity demonstrate that gestures of negation can perform operational functions along with positive statements, working as intensifiers of the verbal utterance. They can also carry out performative functions, when co-occurring with verbal utterances that reveal epistemic stances of negative assessment.

In both kinds of gestures discussed here, the gestures perform pragmatic and discursive functions, operating in a metacommunicative way. In sum, recurrent gestures convey conventionalized gestural patterns that are very important to the meaning of multimodal constructions.

The majority of this research was conducted with the support of the Coordination for Improvement for Higher Education (CAPES Foundation, Brazil). We are grateful for the grant awarded to us (Code 001).