Kendon (2004), inspired by David Efron (1941/1972) and Wilhem Wundt (1921/1973) made a call to study the use of gestures in context. The final objective of the collective research carried out through several projects in which we actively collaborated with Dominique Boutet was to grasp language in its environment in order to articulate its actional roots and its symbolic functions. We therefore analyzed the bodies of the participants both as what Dominique Boutet called a substrate (that is to say, what constitutes it and gives it its structure) and as a support (an instrument) for “languaging.” If we owe this term to Maturana (1978), we prefer to borrow the meaning from Linell (2009: 274): “linguistic actions and activities in actual communication and thinking” in order to refer to the act of multimodal use of language in interaction.

We therefore analyze human interaction in an approach that allows language to be included in embodied action rather than considering it as the use of a code or a symbolic system (Bottineau 2012). This foundation enables us to apply what Slobin (1987: 443) calls “thinking for speaking” to “thinking for languaging” with truly multimodal analyses. According to Slobin, each of us can choose to express an experience with the linguistic means available offered by our mother tongue. This theoretical foundation can be adapted to plurilingualism and to what is called “translanguaging” (Lewis et al. 2012) which allows multilingual subjects to navigate between their languages or to spin them together. But it is also adaptable to all the modalities of expression that are most relevant at the time of production, depending on the context, the participants, the situation, or to use Cienki’s formulation (2012), “the dynamic scope of relevant behaviors”. “Languaging” or “multimodal languaging”, is thus not only related to the languages and cultures to which we belong, but also to the available and coordinated semiotic resources we can use, which, of course, allow our mental constructions to be embodied but which reciprocally and continuously inform them, construct them, give them meaning. Multimodal analyses of language (Morgenstern 2014) as it is practiced in a situation and as it is informed by moving bodies might in turn transform our linguistic theories and bring about profound changes.

The analysis of “co-verbal” gesture (we could also describe verbal production as being “co-gestural”) that we will call expressive and interactional, is often “contaminated” (Boutet 2010: 77) by co-occurring speech (or located in its close temporal environment). However, it is important to carry out multimodal or plurisemiotic analyses without being influenced by the linearity of the verbal flow. Indeed, gesture is compositional. Not only is gesture co-articulated with speech (in the case of subjects producing a vocal language – signed productions are themselves compositional), and co-articulated with gaze, facial expressions, posture, but each gesture produced with one of the upper limbs is potentially composed of movements/motion of the shoulder, arms, forearms, hands, fingers and is often coordinated with the movements/motion of the other upper limb. When we study gesture, we therefore analyze the fine and complex orchestration of all our body segments, and our multilinear way of expressing meaning (Boutet 2010: 77). In order to capture the complexity of this dynamic orchestration, we need to understand that “the body does not merely carry gestures, it informs them. Rather than being a support for gesture, body is its substrate” (Boutet and Cuxac 2008; Boutet 2010: 78). Recent research on protactile language used by the DeafBlind makes this approach all the more relevant. As Edwards and Brentari (2020) demonstrate, DeafBlind uses of Tactile American Sign Language have progressively transformed its very structure to fit the affordances of another channel of communication and other types of users among each other. The recurrent engagement of at least two bodies (involving four hands and four arms) in the reciprocal tactile communication of the participants serve as the support and the substrate of this new language in the tactile proprioceptive modality against the preservation of the more visual aspects of ASL grammar.

According to Boutet (2018), “Kendon’s continuum” (1988) proposed by McNeill (1992) linearizes gestural phenomena by linking them but according to their degree of conventionality (among other parameters):

Gesticulation > Language-like gestures > Pantomime > Emblem > Sign language

This continuum largely takes into account the presence or absence of co-articulated speech. Emblems are considered as the most “lexicalized,” the most “linguistic,” the most “conventional”. If we stop considering the body as a simple support for gestuality, and we take it into account as a substrate, then, this continuum can be reversed. Within gesture studies, it is customary to categorize interactional gestures into 1) iconic or representational gestures which are considered as the least conventional and the most “imagistic”, expressive and individualized, and are opposed to 2) beats which have a “prosodic” role as they structure and punctuate the gestural flow, 3) deictic gestures (including pointing) considered as transparent and linked to their referents, and 4) pragmatic gestures (also called “recurrent gestures” by Ladewig 2014 as for example shrugs interpreted as incapacity or ignorance, index-wagging interpreted as negation) which present a high degree of conventionality within the same linguistic community. Boutet (2010) proposed that iconic gestures as well as beats, two categories of gestures considered to be non-conventional or non-linguistic and which appear visually shapeless and idiosyncratic, are sketches of emblems. Because their production and economy are punctuated by the discursive flow, they cannot be fully performed (or are not in full-bloom). He therefore suggested that there is a true formal relationship between the “most conventional” gestures, and those which are less conventional, based on their physiological structure, despite significant visual differences due to the part of the body in motion.

For Boutet, our cultural filters and our restriction of gesture to the visual modality have thus largely limited our analyses.

The kinesiological approach proposed by Boutet (2008, 2010) makes it possible to analyze gesture and sign in a radically different way from what is presented in most gesture studies. It was initially focused on the production of each single subject rather than on the interactional level because of the technical constraints of the analyses that were conducted. The end goal of the projects Dominique participated in however, was to better understand and capture interactions in their natural ecosystem. This involves a focus on the language structuring provided by the coordination of participants’ bodies during interactions, within the space they inhabit and according to the constraints exerted by the ongoing activities in which they are involved as they use language.

The kinesiological approach reconciles actional and symbolic human activities (Boutet 2018: 17). It shares some foundations with the form-based ToGoG approach (Müller, Bressem and Ladewig 2013) in which a detailed analysis of the movement of gestures is the starting point for the analysis of their meaning. Müller (2005) has shown that gesture can be representational and performed via drawing, modeling, enacting or embodying through imitation. This therefore makes it possible to summon the praxic dimension of gesturality (Boutet 2018: 400). It highlights how much symbolic gesturality has actional roots. These common ideas allowed us to work harmoniously with Cornelia Müller herself as well as with Alan Cienki within a project on aspectuality and gesture funded by the National Science Foundation for five years (Cienki and Iriskhanova 2018), in which we defended the idea that gestures take on or complement some of the linguistic functions of vocal units and take part in the syntactic structure of the multimodal utterance (Ladewig 2014; Cienki 2016).

The orchestration of the semiotic resources deployed by interacting subjects depends on their intersubjective relationships, their reactions to each other, the context, the environment, the activities in progress, the time of day ... according to the dynamic scope of relevant behaviors presented by Cienki (2012).

To describe and analyze the gestural component of multimodal constructions, we need a gesture-based approach in which principles govern our potential for “languaging” and where formal differences are relevant and allow subtle variations in meaning to be grasped. Boutet’s kinesiological approach provides the relevant formal foundations for such an approach.

Most studies and categorizations of communicative gestures focus on the hands because they are the most complex and salient articulators that are mobilized in the visual-gestural mode during multimodal communication. However, we mobilize other articulators (head, face, shoulders and trunk orientation) to communicate (Filhol, Hadjadj and Choisier 2014). This focus on the most distal parts of our body (those which are furthest from the trunk), our hands, is criticized by Boutet (2018) who devoted his research to demonstrating how physiological constraints structure meaning and how all segments of our body, in particular the arms and their dynamics, must be analyzed.

The kinesiological approach to gesture is formal: gestures’ formal components shape their meaning or function. According to Boutet, production, performance, process, seem to have been somewhat forgotten in the analysis of symbolic gesture. “The body has been viewed as a location in which movements simply appear without there being any materiality attributed to it” (Boutet 2008: 82).

However, there is a difference between the phonatory equipment used for vocal languages and the articular equipment used for signed languages: The phonatory tract has only one function, while the upper limbs have several. Except for a few screams or throat clearing, phonatory equipment is used for speaking and singing. Of course, the voice is constantly adjusting and modulating in response to the environment and the communication situation, but the vocal cords only serve this verbal communication and artistic expression. It is therefore easy to understand that the phonology of vocal languages is not influenced by our physical environment. Conversely, the manual equipment at work in sign languages is used daily to interact with the world in an array of activities. The materiality of the body has had the potential to shape our environment, our tools, our objects, the spaces we inhabit (Leroi-Gourhan 1993). Praxic gesture is omnipresent, whether for hearing people or for deaf signers. All this experimentation of the artifactual world made with and for our upper limbs has every reason to exert an influence on co-verbal gestures and on the realization of signs. For Boutet (2018), the structuring of these artifacts is closely linked to praxic gestures, and is in full continuity with symbolic gestures. However, this strong link constitutes a theoretical approach that has never been applied in depth to co-verbal gestures and sign languages (a preliminary study was carried out by Boutet, Chevrefils and Thomas (2019) on manual configurations which were taken as privileged receptacles for this influence of praxic body language on symbolic body language). For Boutet, referents are not primarily associated with gestures through the human capacity to construct visual analogies, but their meaning is directly shaped by the gestures we produce and their dynamics. Boutet describes gestural units on the basis of their formal characteristics and physiological constraints rather than their imagistic iconicity. Indeed, vision has long been considered as the only modality to be taken into account in the analysis of expressive or interactional gestures because our eyes make it possible to perceive, capture and recognize forms. Boutet (2018) shows to what extent gesture studies have ignored art history, but at the same time, he also criticizes how much our apprehension of gesture seems unconsciously linked to the postures represented in art forms that freeze the dynamics of gesturality. By grounding their perception on vision, unconsciously removing movement and its flow from our system of analysis, gesture specialists have not been able to grasp what constitutes the true essence of gesturality. Despite the impact of using video, our analyses of gestures are too often based on the analysis of their static visual representations derived from art and we simply analyze images through a succession of postures (often represented by screen captures or drawings). Movement with its rhythmic variations and its timing is too often discarded.

As opposed to Boutet’s approach (2008, 2010, 2018), in most of the research conducted in gesture studies, praxis and proprioception, sometimes called upon by artists as well as researchers, are not at the heart of the analyses. But if we only take into account the visual modality, each gestural unit may seem to involve multiple, discontinuous formal elements. If we change our approach and place the proprioceptive modality at the core of our analysis, we then find a unique and continuous formal envelope (Boutet 2008: 82). Boutet therefore has offered to use a true phonology of gesture based on human physiology. But this phonology, unlike the analyses proposed by McNeill (1992), is not based on a three-dimensional egocentric frame of reference in which gestures are taken into account in a rather holistic way and within which it is impossible to analyze the movement of each of the segments independently, and thus to capture both the links between them and their variations. When we use an egocentric frame of reference which is three-dimensional, we can take into account whether a movement is upward or downward, forward or backward, or is performed on the left or on the right in relation to the moving subject. That frame of reference allows us to identify general directions, but it remains relative, static … and “’impressionistic” (Boutet 2008: 83). There have been a number of studies in which the body is invited to “play a role” (Boutet 2018: 5) such as in the research conducted by Goodwin (2000) or Gibbs (2005). The egocentric frame of reference has enabled authors to make sense of many metaphors (Lakoff and Johnson 1985). Action can be described as simulated (Hostetter and Alibali 2008). Thus, in these studies the body is “admitted, but must keep its rank” (Boutet 2018: 5) and analyses remain under the influence of Cartesian dualism. A unitary and disembodied conception of the body takes precedence over its segmentation and its materiality (Boutet 2018: 11). In his own work, Boutet proposes to fully take the body into account in all its materiality and to place it at the heart of the very fabric of language and of collective language practices with an intrinsic point of view.

In order to give us the possibility to describe gestures, to locate them as precisely as possible, to account for the movement of each of the segments with both their own dynamics and the links between them, the kinesiological approach requires a true articulatory frame of reference. The unit Dominique Boutet chose is the degree of freedom. For example, for the hand, there are two degrees of freedom, flexion/extension and abduction/adduction. Boutet was inspired here by human articulatory biomechanics. The degree of freedom is defined as the independent relative movement of a segment with respect to its adjacent and more proximal counterpart (Boutet 2018: 24). This means that the movement of the hand is analyzed in relation to the position of the forearm, the forearm to the arm and the arm to the trunk. Taking degrees of freedom into account makes it possible at each moment of the gesture to describe the positions and movements of each of the segments (fingers, hand, forearm, arms and shoulders for the upper limb). The analysis can also extend to the face, head and trunk. This frame of reference makes it possible to measure all the movements of all the segments, and is centered on each joint. This involves proposing a multiple intrinsic analysis centered on what the degrees of freedom afford in terms of movement and position for each single segment (Boutet 2018: 11).

Throughout his research, Boutet has mainly worked on the upper limbs by focusing on their position and movement⁵ but an analysis of facial expressions is also in progress⁶.

There are 28 degrees of freedom from the fingers to the shoulder. Boutet reminds us that they are defined in relation to an anatomical reference position: the body is upright, the head, the torso, the knees, the toes are directed forward. The upper limbs are dangling alongside the body, the palms facing forward.

This makes it possible to build a 28-dimensional space, with 4 degrees of freedom for each finger except the thumb which has 5: there are 2 for each hand, 2 for each forearm, 3 for each arm with the shoulder joint (for details see Boutet 2008, 2018).

This is the reference position on which were established, among others, the occidental notions of “front,” “back,” “left” and “right” (Fig. 1). Through the addition of axes to this posture, all the movements a body can perform have been conventionalized according to degrees of freedom. The median plane divides the body in its height, it separates the body equally between the left and the right and differentiates the movements in which the segments move away from the axis (grouped under the abduction category), from those in which the segments get closer to the axis (adduction category). The frontal plane separates the movements which are located in front of the axis (flexion category) from those which are located behind the axis (extension category).

Figure 1. 

Representation of two anatomical planes as a reference to the abduction/adduction and flexion/extension degrees of freedom (image is taken from https://www.shutterstock.com/fr/search/similar/1373139335 and slightly modified).

We can observe that rotations are missing from this categorization. However, rotations affect two segments: the arm (external and internal rotation) and the forearm (pronation and supination). Unlike the others, these two degrees of freedom are not located on a joint – shoulder, elbow or wrist – but along the bones (humerus for the arm, ulna and radius for the forearm) (Fig. 2). When the human body is in its anatomical reference position, the axis is located inside the upper limb: the segments rotate on themselves.

Figure 2. 

Representation of two parallel rotations in anatomical reference position, external/internal rotation on the arm and pronation/supination on the forearm (image is taken from https://www.shutterstock.com/fr/search/similar/1373139335 and slightly modified).

These degrees of freedom can be added to each other; they can operate at the same time, and even influence each other. Indeed, this contiguity between the segments (they are all attached to each other) allows the body to be used in three-dimensional space, but also creates constraints in the possibilities offered for movement to be performed. So, it is necessary to take into account all possible physiological constraints in order to understand that some have a reduced amplitude. Indeed, if we consider that the neutral position is the one in which we have our arms hanging along the sides of the body and if we start from there to place the forearm in the position of maximum flexion (hand raised at the height of the neck), the pronation of the hand is then only possible at a 45° angle and not at an 85° angle (Boutet 2008: 90). Thus, depending on the position of each segment, the movements of the other segments will have different constraints. Furthermore, the axes of rotation are variable and also depend on the position of each segment. The two degrees of freedom of the hand are not independent. They result from limited amplitudes which structure their relationships and must truly be integrated into the analysis of the movements of each gesture studied. Boutet therefore tried to show that the visual modality is quite insufficient to follow and identify each type of movement and that we must fully understand the physiological constraints of the human body in order to analyze gesture.

Because the articulators used to perform gestures are much more visible and therefore accessible than those which allow us to produce sound (only the acoustic effects of sounds are perceptible apart from accentuated mouth movements), Boutet (2010) proposed to adopt a “bottom-up” approach based both on perception and on our knowledge of physiological processes. Gestures are analyzed as movement that flows from one segment of the body to the next. The trunk of the body is used to construct the frame of reference for other parts of the body which thus can either be called proximal (closest to the trunk) or distal (farthest from the trunk). The distal or proximal nature of moving segments permeates gestures with meaning, rooted in experience.

Movement can be made up of the effective mobility of a segment (which we have called in English “movement”) or of the displacement of a segment which is not affected by a specific movement (which we have called in English “motion”). When we carry out a detailed analysis of gestures, we can capture movement transfers from one segment to another. On the other hand, it is difficult to identify and describe them in depth without resorting to motion capture. However, for Boutet (2018: 44) when there is displacement (or motion), the movement comes from a more proximal segment (closer to the trunk) and meaning is not to be constructed as rooted in the segment that is simply set in motion but in the segment that carries effective movement (for a detailed analysis of these differences, see Boutet 2015).

As far as the upper limbs are concerned, several segments are set in motion. If the flow that connects one segment to the next travels from the shoulder to the fingertips, the flow is called “proximal-distal”, if the flow travels from the hands to the shoulders, it is called “distal-proximal”. But if the movement is only localized in one part of the body (hand, shoulder, head ...), then there will be no apparent flow. In addition, the inertias of each segment must be taken into account in the analysis of gestures. The proximal-distal flow will be impacted by gravity and will therefore require less energy than the distal-proximal flow for which it will be necessary to go against gravity. “The difference between certain gestures only concerns their flow. These gestures constitute what seems closest to a minimal pair since, apart from orientation or configuration, the only difference lies in the sequence in which the segments flow one after the other.” (Boutet 2018: 9).

As an example, let us analyze the gesture in which the subject holds out her arm with a palm up open hand. That gesture can be either identified as a presentation gesture —translated in words with “c’est ça” (that’s it) by Calbris (1990) – or as an epistemic negation—that we could formulate into words as “I don’t know” or “never mind”, see Debras 2017 for a variety of meanings in adults, or Beaupoil-Hourdel and Debras 2017 as well as Morgenstern et al. 2017, in children.

By focusing on the hand and on a frozen image, the analysis leads to two different interpretations which clearly demonstrates that the method is insufficient. The gesture must be analyzed in the context of the movement of the whole body (see section 3) and not via a focus on the hand(s). Gestures are not frozen postures and must be considered as meaningful movement.

Flow therefore organizes meaning. The great value given to flow in the kinesiological approach focuses on how movement and its dynamics are essential for the analysis of gesture. Gestural forms are not considered simply according to their resemblance to a referent (Boutet 2005) but as informing meaning.

Movement has not often been placed at the core of analyses in gesture or sign studies as the various methods used were focused on the trajectory and could not uncover physiological invariants. Dominique Boutet’s kinesiological approach, gives us the possibility to analyze movement through its formal features, to fully grasp it, to reveal all the information that it contains. It shows us all the work that remains to be done within linguistics, which are still mostly dominated by models provided at first by written forms, then by vocal forms. Boutet’s goal was to reinstate the body as both the vehicle and the source of meaning.

Boutet (2010) proposed a physiological structuring of gestural units in the form of action patterns. These action patterns include all movements involving transfers over other degrees of freedom that can be grouped together for each gesture. The difference between action patterns is based on physiological and physical parameters. For Boutet (2018), the structuring of gestures is not based on meaning but is grounded in forms. The repeated use of those forms lead to their stabilization. In order to study gestures, as they are rooted in their own materiality (Boutet 2018: 79), it is necessary to carry out low-level analyses by taking into account the segments engaged in a gesture, their position, the movement of each segment and their flow as they unfold in time. The semiotics of gestures emerge for the person who produces the gestures as well as for the person who perceives them thanks to their action patterns. Those action patterns can be tested by judges and categorized in the form of gestural units (GU). A GU can be performed in a variety of ways, it can thus be considered as an envelope for potential gestures (Boutet 2018: 80). By using an experimental design in which judges recognize labels associated with videos of gestures, it is possible to validate the semantics of gestures and the physiological structuring of their meaning.

Thanks to his method which combined corpus analyses and perception tests, Boutet analyzed and isolated gestures with very similar forms and sought to clarify a network of semantic relationships based on the constraints and affordances of the body.

In order to conduct corpus analyses, Dominique Boutet constantly created detailed “templates” using the ELAN software (https://archive.mpi.nl/tla/elan; Wittenburg et al. 2006) which made it possible to annotate all the kinesiological parameters he highlighted. This enabled him and the various colleagues who worked with him to capture the internal composition of gestures as well as their differences, however slight they may be, using internal criteria based on the specificities of the part (or parts) of the body that are in movement (or set in motion). This is why Dominique Boutet naturally turned to motion capture and new technologies and enthusiastically followed the advice given by Goldin-Meadow and Brentari (2017). This allowed us to explore form-function pairings and to conduct automatic extraction with formal criteria with much more rigor and without depending solely on our visual perception. Dominique Boutet focused his most recent activities (the Polimod Project, the Typannot Project, the PhD theses of Léa Chevrefils and Chloé Thomas) on a light, portable system called the “Perception Neuron” (https://neuronmocap.com), made of inertial units, which enabled us to collect an interactional corpus. By coupling the Neuron with video collection, it is possible to compare the data from the motion capture with the manual annotations carried out in ELAN. The objective is to arrive at form / function pairings that could allow a semi-automatic annotation of gestures.

Dominique Boutet had also started working with OpenPose and OpenFace which offer the possibility to use videos collected in a more ecological environment and thus to automatically detect postures and facial expressions and analyze, for example, family interactions⁷ without the participants having to wear motion sensors, which can be invasive for the speaker or signer.

The kinesiological approach has made it possible for Boutet and his colleagues in their research on the gestures performed with the upper limbs, to show that arms are the primary substrate of gestural meaning which subsequently extends all the way to the hands (Boutet 2018: 115). Meaning spreads over several segments. The method implemented by Boutet gives us the tools to grasp how movement propagates over all the segments of the upper limbs and thus how meaning is constructed through the gestural flow. Because they are first determined kinesiologically, rather than being analyzed through the visual traces they leave as they are perceived in reality or on video, gestures are analyzed within the course of their movement. The deployment of gestures takes precedence over their trace. “The temporal unfolding of gestures is essential for the understanding of what is at stake in each gesture” (Boutet 2018: 116). Boutet’s first studies also show that motor control plays an important role in gestures’ degree of conventionalization. Lesser motor control could correspond to pragmatic (or recurrent) gestures which are thus also cognitively more economical. The greater the motor control, the more the idiosyncratic character of gestures might increase (Boutet 2018: 226).

Moreover, unlike approaches centered on iconicity in which representation is secondary, in the collective work inspired by the kinesiological approach, the body is not a vector but structures gestures, and flow functions as a distributor of meaning (Boutet 2018: 119).

As an extension of the ANR CoLaJE project (Morgenstern and Parisse 2012) in which a team had worked on negation gestures, we were interested in recurrent gestures in adults and children. “Recurrent” gestures (Müller et al. 2013) are the result of a gradual sedimentation of our sensorimotor system. They are culturally shared, embedded in conventional, embodied experiential frameworks. They are precipitates of experience that have given us the potential to construct multimodal scripts by taking into account multiple temporalities: that of a single community that shares a culture and language, that of each individual’s development, and the moments spent with our conversational partners. The meaning of recurrent gestures is not as conventionalized as the meaning of emblems and it is only possible to associate forms with functions in a conversational context by taking into account the verbal content of the utterances, prosody, gaze, facial expressions and all the gestures produced. In our perspective, recurrent gestures have the potential to express a substrate or a meaning that can only be deployed in association with other semiotic resources according to the affordances of the situation. However, it is possible to uncover formal differences between seemingly very similar gestures, which is what we are trying to do in order to differentiate the presentation gesture and the gesture for “epistemic negation”.

The presentation gesture offers a visual resemblance to epistemic gestures. As far as their function is concerned, the presentation gesture is rhematic, it often allows the speaker to add new information, whereas “epistemic negation” is thematic and brings a subjective perspective on a shared object of discourse. They have been analyzed as similar in form (Calbris 1990; Kendon 2004) whereas for Boutet (2018: 16) the presentation gesture is informed by a distal-proximal flow (Fig. 3) and the gesture of epistemic negation, by a proximal-distal flow (Fig. 4). To be more precise the epistemic gesture emanates from the arm. The transfer of movement can affect the forearm, the hand and then the fingers. But the gesture can propagate towards the shoulder and be transformed or completed by a shrug of the shoulder (to which a shrug of the eyebrows and a puckering of the mouth could be added). That gesture has been described as a composite gesture (Streek 2009; Debras 2017). One of its parts can metonymically represent the whole. The velocity of these two gestures, impacted by inertial differences can also be different. Gesture typologies that are based solely on posture are insufficient to accurately analyze the differences between these gestures. The analysis of gestures in terms of images (McNeill 1992), which is based on the visual modality, does not take into account each of the gestural articulators, their organization, their dynamics, but they are necessary to highlight the structural differences that inform their semantic differences.

Figure 3. 

Articular decomposition of an epistemic negation performed in context. Screen shots (Michael Jordan) are extracted from the video on line https://www.youtube.com/watch?v=gbngKR8UHLk&ab_channel=ESPNESPNValid%C3%A9 (Timing around 47 seconds).

Thanks to a series of similar analyses, it is possible to uncover gestural invariants, with identic features throughout different instances in a range of contexts. In these cases, the extension of the arm, the rotation of the forearm and the supination of the hand according to a proximal-distal flow constitute the invariants expressing epistemic negation, non-existence, absence and/or incapacity. In the presentation gesture, the same degrees of freedom are used, but the flow is distal-proximal and the hand is the actual leader of the movement.

Figure 4. 

Articular decomposition of a presentation gesture performed in context. These screen captures are extracted from the data set collected in France for the Polimod project funded by the Russian Science Foundation (Cienki and Iriskhanova 2018; Boutet et al. 2018a–c; Morgenstern et al. 2017).

For a gesture to be identified as a presentation gesture, the hand must at least be in a supine position. The motion of the forearm (and possibly the arm) is triggered by the movement of the hand. The forearm and arm are impacted by the hand’s own movement. In the case of epistemic negation, it is the hand that is set in motion thanks to the movement of the forearm, or even the arm.

We must change our perspective on the unfolding of a gesture of the upper limb to determine the flow of the movement, which thus makes it possible to distinguish forms that are visually close but have different meanings. Gestures of “presentation” as opposed to “incapacity” or “ignorance” take place according to opposite flows.

Moreover, if the flow structures the unfolding of the gesture, meaning can be instantiated on a segment far from the origin of the gesture. There is no form/function pairing independently of the degree(s) of freedom that generate(s) the form, and a form cannot be studied without analyzing the flow of the movement. It is therefore important to identify the segment that generates the form and then to grasp the flow of the movement and the segments on which it propagates.

All of this makes it possible to understand that forms which are visually very different, such as the “shrug” and Palm Up Open Hand are closely linked. A number of formal characteristics distinguish these two gestures. However, there is a causal link between the two. Shoulder shrugs and Palm Up Open Hand seem to move in opposite directions (up and down), they have divergent trajectories and different amplitudes but they are linked through a transfer of movement and can be considered as forming two instances of the same gesture. Shoulder shrugs and Palm Up Open Hand are indeed connected by many authors (Streeck 2009; Debras 2017; Cooperider, Abner and Goldin-Meadow 2018).

If these differences between epistemic and presentation gestures, which were brought to light by Boutet’s early work, have been confirmed in our first analyses of a few coded sessions of family dinners, of dyadic adult-child and adult-adult corpora in English and in French, more coding must be carried out in order to ground Dominique Boutet’s dazzling intuitions, linked to his mastery of the bio-mechanical functioning of gestuality, on solid statistical results. Ideally, fine manual annotations should be coupled with motion capture data. But our preliminary results, which are the first to our knowledge to deal with this type of differentiation in context on two languages, are promising. The flow is possibly a significant parameter in the pairing of gesture forms and functions. An analysis of equivalent functions in French Sign Language, in which the semantic stakes are higher, also allows a more detailed analysis of the involvement of distal segments in meaning variations.

The signs [PLACE] and [CATCH] in French sign language are carried out with the same degrees of freedom: extension of the forearm and closing of the fingers. But the order of activation of these degrees of freedom is reversed. For [PLACE], the flow is proximal-distal (Fig. 5) with forearm extension first; for [CATCH] the flow is distal-proximal (Fig. 6) with the closing of the fingers first. Therefore, there is a specific way to distinguish the movement of these two signs: their flow.

Figure 5. 

Articular decomposition of the sign [PLACE] in French Sign Language (spreadthesign.com).

Figure 6. 

Articular decomposition of the sign [TO CATCH] in French Sign Language (spreadthesign.com).

In the research on grammatical aspect and gestures conducted as part of the Polimod project funded by the Russian National Science Foundation and coordinated by Alan Cienki (Cienki and Iriskhanova 2018), we illustrate how research on gestuality in conversation provides new ways to study participants’ conceptualization of grammatical notions during multimodal “languaging” (Linell 2009). We analyzed the expression of aspect in French, Russian and German by correlating the analysis of verbal forms with that of synchronously produced gestural forms. Our hypothesis was that the differences between perfective and imperfective aspects could be correlated with kinesiological aspects of gestures. The idea was that certain kinesiological features, and in particular the quality of the movement, could make the semantic differences truly tangible. The nature of the co-verbal gestures used by speakers could thus be linked to aspectuality and their specific features might be captured in our detailed annotations.

By coding ten dyadic interviews between French students, each lasting 15 minutes, we uncovered a strong correlation between gestures that we called bounded (with a strong punctual acceleration during the unfolding of the gesture) and the passé composé (perfective) and unbounded gestures (whose unfolding was performed with a stable speed) and the imparfait (imperfective aspect) (Morgenstern et al. 2017; Boutet et al. 2018a, b, c). We wanted to confirm this first study by analyzing the kinesiological parameters involved and in particular not only speed but also flow as well as the segments involved (fingers, hands, forearm, arm, shoulder) (Boutet et al. 2016a). We found that the bounded quality of a gesture that is grounded in a sudden variation in velocity is much more likely to be produced with a distal-proximal flow and that unbounded gestures were performed with a proximal-distal flow. According to the kinesiological approach, velocity is greater when the movement originates on distal segments (hands and fingers). In the case of a movement that originates on the hand and propagates to the forearm, the hand acquires a certain speed with accelerations. The speed depends on the length of the segment. The forearm is three times longer than the hand. When the motion is transferred from the hand to the forearm, the acceleration is potentially higher, and this is often visible in the video. This explains why the bounded quality of gestures with its sudden changes in velocity could be related to the distal-proximal flow.

The results of our detailed coding of 4 sessions (8 French participants) showed that 81.3% of the gestures accompanying a verb in the imparfait (imperfective aspect) were produced with proximal-distal flow (unbounded gestures) and 74% of the gestures accompanying a passé composé (perfective past tense) were performed with a distal-proximal flow. We also showed that gestures accompanying the passé composé were significantly faster than those accompanying the imparfait and involved fewer segments. Perfectivity thus seems to be associated with well-determined kinesiological parameters and in particular the flow of the movement which draws on both biomechanical properties and their cause. We need to continue these analyses and in particular on our data collections that involved both video-recording and motion capture with the Neuron, in order to be able to understand if the biomechanical properties of the movements embodying aspectuality are indeed the link between form and function of the gestures we have analyzed.

Dominique Boutet’s personality, his energy and his multidisciplinary span were a source of inspiration for a variety of projects and studies.

Dominique Boutet and Catherine Bolly had long highlighted the importance of taking gestures into account in elderly people’s communicative expression. They were part of an international network interested in the effects of aging on language. With the SignAge project, we wanted to explore several hypotheses presented in the literature as characterizing elderly people’s multimodal communication: i) for both signers and non-signers amplitude of gestures are reduced because angles of rotation are reduced, ii) for signers, there is a grammatical use of space specific to this age range; iii) for both signers and non-signers, vocal-gestural prosodic markers, and a specific management of speech turns can be identified. Dominique emphasized the value of using motion capture to measure rotations, duration, speed, acceleration and jerks, and to test his model based on movement distribution according to segments and their degrees of freedom. He convinced us to use mocap, with humility and realism, insofar as these tools were to allow us to objectify and semi-automate annotation and processing.

We started with a minimally invasive and inexpensive tool, the Kinect. First⁸, we studied the movements of the head, which we classified according to the three rotation axes: nodding, headshaking, and tilting. We used two coding methods in parallel: 1) we conducted manual annotations with ELAN based on the video, thus in 2D. 2) We extracted the 3D positions recorded by the Kinect in a spreadsheet and applied thresholds. These thresholds concerned the deviation of the positions on each axis, for a defined number of successive images, and enabled us to automatically identify the three types of head movements. We wanted to confront our manual annotations and the annotations derived from the Kinect (filtered thanks to our thresholds), in order to progressively adjust the two types of inputs. This approach makes it possible to check the annotator’s perception in 2D and to adjust the thresholds so that the two coincide as much as possible in order to objectify and facilitate annotation in the future.

We then acquired⁹ a more precise piece of equipment with IMU, the Neuron, (see description above). Although the tool is more invasive, our older signing participants seemed to cope with it very well. As in the previous study, we worked on the video data in parallel, segmenting and decomposing the lexical units in ELAN, in order to calculate signing rate (average duration of a sign, average duration of its gestural core, the stroke). Here again, we hypothesized that the flow rate slows down with age. But, if the average number of signs per unit of time seems to decrease, and the average duration of the sign increases, the average duration of the stroke seems to remain relatively stable and conforms to the average described in sign language metrics. We therefore hypothesized that the slowing down occurs in the transitions between strokes. This is one of the aspects that could clearly confirm Dominique’s proposals on the distribution or propagation of movement according to the segment and to its degrees of freedom. Using other scripts developed by the team, we also created visualizations that indicate the distance between the dominant hand and the torso and help us identify patterns or profiles of signers related to the mocap data.

Finally, more recently still, Chloé Thomas has begun a PhD project on non-manuals in French Sign Language, under the supervision of Dominique Boutet. This has offered us the opportunity to test yet another system, OpenFace. This facial recognition software enables us to trace the face in real time or after recordings, to detect the different landmarks of the face, to identify the position of the head, to recognize facial action units such as the ones defined by Ekman and Friesen (1972), but also gaze direction, as well as 3D variations of each point on the face. The objective is to link the data derived from this mocap system and our manual annotations, in order to establish thresholds that allow us to characterize the movements of the face articulators. We can for example distinguish relevant differences between the various positions of the eyebrows in order to identify “eyebrow raising” according to a scale specific to each individual signer, and to check that this movement has indeed been identified as such via our manual annotation, thanks to the Typannot system (see Chevrefils et al. in this issue).

By systematically applying this process to non-manual gestures, our objective is to identify formal patterns, invariants, and take into account all the segments and articulatory flows based on the model proposed by Dominique Boutet. This should also help us automate the annotation of non-manuals in sign languages.

As part of his multidisciplinary approach, Dominique Boutet developed many connections with the arts. He naturally searched for the influence of art in his own studies but his different models have also influenced some recent artistic creations. Three of his out-of-the-box contributions are presented here: a) the vision-centered gesture analysis paradigm linked to art history, b) the gesture annotation of an expressive virtual actor in an interactive digital art installation, and c) tools developed first for motion capture analysis applied recently to a live piano performance using augmented reality.

1. Since the revival of gesture studies in the ’80s, according to Boutet (2018), we have been analyzing the representation of gestures rather than gestures for themselves. In several talks and workshops (Boutet and Jego 2016, 2017), we investigated how gesture analysis – which is mostly based on vision – could be influenced by art history. In many artworks from prehistory to the modern period, gestures have been studied by painters, sculptors or photographers not only as simple postures but also as movement. Artists have been representing movement by playing with anatomy, with proportions or even with the different body parts of the subject: they were able to show different moments of an action in the same picture or could use superposition as in chrono-photography. Regarding composition, it seems interesting to consider it not only as pictorial but also as semiotic (Boutet 2018). This reveals that the dominant models used to represent and analyze gestures are mostly vision-centered, whereas artists tend to conceive/perceive movement from the inside with a more embodied perspective. We started investigating how to explore gesture from other perspectives. For instance, we used motion capture which can combine different modalities involving proprioception or kinesthesia and free our perspective from a vision-centered analysis (Jego, Meyrueis and Boutet 2019).
2. In the field of digital arts, (Boutet, Tramus, Blondel, et al. 2016) proposed an unconventional contribution to gesture annotation: they analyzed improvisations between a participant and a virtual agent in an interactive art installation InterACTE (Batras et al. 2016). The participant was invited to interact with the shadow of a virtual character using upper-limb gestures. This character was able to generate gestures picked in a database and also to imitate the participant’s movements using a Kinect motion capture camera which tracks the body but not the fingers. The arm movements of the virtual character appeared to be smooth and did not include finger movements. This happens to look similar to children’s gestures before 17 months. Children use relational gestures initiated on the arms, as opposed to manipulative gestures, initiated on the hands. Thus, participants tend to adapt their behavior to this limitation as if the virtual character were a child. This adaptation and this type of gestural exchange based on imitation push the participant to integrate the limitations of the virtual actor’s gestures. Moreover, since the participant’s emergent gestural style appears even in non-imitative productions, a form of empathy could thus be observed. This empathy favors movements of the arms and forearms and thus triggers the participant to gesture mostly with proximal segments. This experiment sheds new light on gestural exchanges and raises new questions: Is it the absence of vocal production and the limitation of gestures and/or mimicry that promotes empathy in these interactions with the virtual actor? Because of this type of constrained gestuality, is the relationship with the virtual agent similar to what we find in adult-child interactions?
3. In the Polimod project detailed above, we explored how to study and measure the aspectuality of gestures across three languages (Russian, German and French). The research project focused on the quality of the movement and considered the gesture’s boundedness (bounded or unbounded) in terms of kinetic “pulse of effort” as defined by Boutet, Morgenstern and Cienki (2016). In order to evaluate the “pulse of effort” which is based on gesture kinematics, we used motion capture to observe the different frames of reference of the participants’ gestures and also the different levels of complexity and the boundedness of the “pulse of effort”. We developed a tool to visualize in real time different gesture descriptors based on kinematics such as velocity/acceleration/jerk of any body part position or orientation using a motion capture system (Jégo, Meyrueis and Boutet 2019). It helped to assess the range or motion of each degree of freedom for each segment (arm, forearm, hand) displaying graphs for the frame of reference of the desired body part and with the desired perspective (first- or third-person viewpoint).

Following this study, the tool has recently been used in order to visualize different kinematics of the complex body movements of a pianist playing the contemporary piece Evryali by the composer I. Xenakis (Antoniadis et al. 2021). Since Evryali evokes “the open sea,” we opted for aesthetics related to wave forms and light, aiming to create a synesthetic experience combining sound, visuals and motion visualization. We thus designed an augmented reality stage made of several interactive video projections of a virtual avatar of the performer. The movements of his chosen body parts and the physical efforts required to perform Evryali were represented in virtual space using trails and curves. We were also able to switch from egocentric to allocentric points of view for the avatar of the pianist in real time. The performance was presented in March 2021 in France.

These back-and-forth movements between disciplines are clearly visible in the conclusion of Dominique Boutet’s habilitation thesis (Boutet 2018). He underlines that gestural forms are not essentially limited to traces left in space. He additionally identifies the importance of the degrees of freedom or the poles of each body segment, and proposed to explain that the diversity of gesture analyses is due to the various frames of reference, the flows and the levels of structuring. Thus, he opened new avenues to explore gesture. We have barely started to implement these crucial new aspects in our various disciplines, but these groundbreaking perspectives will remain true inspirations to make the invisible visible.