Print
“Of thee I sing”: An opening to Dominique Boutet’s kinesiological approach to gesture1
expand article infoAliyah Morgenstern, Lea Chevrefils§, Marion Blondel|, Coralie Vincent|, Chloé Thomas§, Jean-François Jego, Dominique Boutet§
‡ Sorbonne Nouvelle University, Paris, France
§ Université de Rouen Normandie, Rouen, France
| CNRS, Paris, France
¶ Université Paris 8, Paris, France
Open Access

Abstract

In this tribute to Dominique Boutet and the kinesiological approach he founded, the authors have tried to make their memories of the scientific collective projects they worked on together resonate with the written work of this extraordinary scientific partner, gathered in his published articles and his habilitation document (Boutet 2018). His approach based on an intimate knowledge of the biomechanics of the human body is centered on the structuring role of the body in gestures (and signed languages). It is form-based: the form of gestures shapes their meaning or function. Gestural units are described on the basis of their formal characteristics and physiological constraints rather than their imagistic iconicity.

The article presents the foundations of the approach, a synoptic description and some examples of its application. The originality of the kinesiological approach lies in the double revolution that it allows us to operate: on the one hand, gesture is not simply an appendix of speech; on the other hand; it is shaped by bodily physiology. The approach is based on the movements of the human body analyzed from a biomechanical point of view. The meaning of our gestural productions is the produce of our body, as it is naturally articulated, imprinted as it is by our past experiences.

Key Words

flow, gesture, kinesiology, motion capture, movement, sign languages

Introduction

Dominique Boutet was extremely erudite, passionate and curious. He savored life and all human cultural and scientific productions to the fullest. His kinesiological approach to communicative gesture and sign is difficult to access as it was conceived by combining his knowledge of biology, medicine, biomechanics, art history, linguistics, philosophy, anthropology, archaeology and many other fields.

This article is an attempt to give an account of the sessions shared with Dominique in Moscow, Rouen, Paris and of our work together. It resonates with his Habilitation thesis (2018) and all his publications. We are not immune to a subjective appropriation of his approach, which took us some time to penetrate and which is reconstructed through our own theoretical filters and experiences. But this is what made our relationship with Dominique and our mutual conversions to each other’s ideas and passions so special. We therefore tried to express the fruit of the meeting of his and our subjectivity.

By writing this article grounded in our shared experiences, our memories and his publications, so soon after the departure of this vital friend who died of COVID in May 2020, this scientific brother/father, and partner with unique qualities that Dominique was to us, we first wanted to express our gratitude.

This article is only an “opening” because it was not possible to uncover all the contours and contents of the theoretical and methodological work accomplished by this multidimensional researcher in one paper. It is also an opening to our collective continuation of Dominique’s theoretical and methodological approach, and his scientific findings through a series of projects we will pursue to embody his prolific ideas.

In his combat against the cultural filters that led many linguists to deny the role of the body and its dynamics, Dominique Boutet constantly sought to place the body at the center of all language activity. In order to illuminate how language is structured by our bodily activity in all its materiality, he analyzed sign languages, spoken languages including gestures, facial expressions, gaze and postures, in everyday interactions as well as in more experimental or artistic settings.

The originality of the kinesiological approach (from the Greek kinesis, movement and logos, speech, science) lies in the double revolution that it allows us to operate: on the one hand, gesture is not simply an annex to speech; on the other hand, it is shaped by our body’s physiology. Gestures derive from the movements of the human body analyzed from a biomechanical and articular point of view. The meaning of our gestural productions is articulated out of our body, permeated with our past experiences and our relationships to others. Our gestures are the result of the hybridization between what is universal, generic through the setting in motion of human bodies as they are all naturally structured, and what belongs to each and every single individual, their development, and their history.

The starting point of our work together was our questioning of disembodied linguistics. The forms produced by human beings in interaction are the main object on which we work as linguists. Dominique considered that a merely semiotic approach to language evacuates the signifier as a mode of expression. His kinesiological approach, on the other hand, is based on the signifier, the body and all the segments whose constraints but also whose “affordances2” (Gibson 1977) make it possible to produce the signified and reveal how it structures meaning, how it can limit but also strengthen our expressivity.

We will first lay the foundations of the kinesiological approach, make a synoptic description of the approach itself and its integration into collective research projects. We will then give some examples of how it was applied in projects in which we analyzed gestures and signs with a variety of perspectives and in connection with an array of scientific fields.

Foundations of the kinesiological approach to gesture

The theoretical and methodological framework proposed by Boutet (2008, 2010) was naturally integrated by the teams of the ANR ColaJE3 project and the Polimod4 project, which proposed to analyze interactions with a multimodal, situated and embodied approach.

Multimodal situated and embodied “languaging”

Kendon (2004), inspired by David Efron (1941/1972) and Wilhem Wundt (1921/1973) made a call to study the use of gestures in context. The final objective of the collective research carried out through several projects in which we actively collaborated with Dominique Boutet was to grasp language in its environment in order to articulate its actional roots and its symbolic functions. We therefore analyzed the bodies of the participants both as what Dominique Boutet called a substrate (that is to say, what constitutes it and gives it its structure) and as a support (an instrument) for “languaging.” If we owe this term to Maturana (1978), we prefer to borrow the meaning from Linell (2009: 274): “linguistic actions and activities in actual communication and thinking” in order to refer to the act of multimodal use of language in interaction.

We therefore analyze human interaction in an approach that allows language to be included in embodied action rather than considering it as the use of a code or a symbolic system (Bottineau 2012). This foundation enables us to apply what Slobin (1987: 443) calls “thinking for speaking” to “thinking for languaging” with truly multimodal analyses. According to Slobin, each of us can choose to express an experience with the linguistic means available offered by our mother tongue. This theoretical foundation can be adapted to plurilingualism and to what is called “translanguaging” (Lewis et al. 2012) which allows multilingual subjects to navigate between their languages or to spin them together. But it is also adaptable to all the modalities of expression that are most relevant at the time of production, depending on the context, the participants, the situation, or to use Cienki’s formulation (2012), “the dynamic scope of relevant behaviors”. “Languaging” or “multimodal languaging”, is thus not only related to the languages and cultures to which we belong, but also to the available and coordinated semiotic resources we can use, which, of course, allow our mental constructions to be embodied but which reciprocally and continuously inform them, construct them, give them meaning. Multimodal analyses of language (Morgenstern 2014) as it is practiced in a situation and as it is informed by moving bodies might in turn transform our linguistic theories and bring about profound changes.

A compositional mode of expression

The analysis of “co-verbal” gesture (we could also describe verbal production as being “co-gestural”) that we will call expressive and interactional, is often “contaminated” (Boutet 2010: 77) by co-occurring speech (or located in its close temporal environment). However, it is important to carry out multimodal or plurisemiotic analyses without being influenced by the linearity of the verbal flow. Indeed, gesture is compositional. Not only is gesture co-articulated with speech (in the case of subjects producing a vocal language – signed productions are themselves compositional), and co-articulated with gaze, facial expressions, posture, but each gesture produced with one of the upper limbs is potentially composed of movements/motion of the shoulder, arms, forearms, hands, fingers and is often coordinated with the movements/motion of the other upper limb. When we study gesture, we therefore analyze the fine and complex orchestration of all our body segments, and our multilinear way of expressing meaning (Boutet 2010: 77). In order to capture the complexity of this dynamic orchestration, we need to understand that “the body does not merely carry gestures, it informs them. Rather than being a support for gesture, body is its substrate” (Boutet and Cuxac 2008; Boutet 2010: 78). Recent research on protactile language used by the DeafBlind makes this approach all the more relevant. As Edwards and Brentari (2020) demonstrate, DeafBlind uses of Tactile American Sign Language have progressively transformed its very structure to fit the affordances of another channel of communication and other types of users among each other. The recurrent engagement of at least two bodies (involving four hands and four arms) in the reciprocal tactile communication of the participants serve as the support and the substrate of this new language in the tactile proprioceptive modality against the preservation of the more visual aspects of ASL grammar.

Gesture categories

According to Boutet (2018), “Kendon’s continuum” (1988) proposed by McNeill (1992) linearizes gestural phenomena by linking them but according to their degree of conventionality (among other parameters):

Gesticulation > Language-like gestures > Pantomime > Emblem > Sign language

This continuum largely takes into account the presence or absence of co-articulated speech. Emblems are considered as the most “lexicalized,” the most “linguistic,” the most “conventional”. If we stop considering the body as a simple support for gestuality, and we take it into account as a substrate, then, this continuum can be reversed. Within gesture studies, it is customary to categorize interactional gestures into 1) iconic or representational gestures which are considered as the least conventional and the most “imagistic”, expressive and individualized, and are opposed to 2) beats which have a “prosodic” role as they structure and punctuate the gestural flow, 3) deictic gestures (including pointing) considered as transparent and linked to their referents, and 4) pragmatic gestures (also called “recurrent gestures” by Ladewig 2014 as for example shrugs interpreted as incapacity or ignorance, index-wagging interpreted as negation) which present a high degree of conventionality within the same linguistic community. Boutet (2010) proposed that iconic gestures as well as beats, two categories of gestures considered to be non-conventional or non-linguistic and which appear visually shapeless and idiosyncratic, are sketches of emblems. Because their production and economy are punctuated by the discursive flow, they cannot be fully performed (or are not in full-bloom). He therefore suggested that there is a true formal relationship between the “most conventional” gestures, and those which are less conventional, based on their physiological structure, despite significant visual differences due to the part of the body in motion.

For Boutet, our cultural filters and our restriction of gesture to the visual modality have thus largely limited our analyses.

A synoptic description of the kinesiological approach

A form-based approach

The kinesiological approach proposed by Boutet (2008, 2010) makes it possible to analyze gesture and sign in a radically different way from what is presented in most gesture studies. It was initially focused on the production of each single subject rather than on the interactional level because of the technical constraints of the analyses that were conducted. The end goal of the projects Dominique participated in however, was to better understand and capture interactions in their natural ecosystem. This involves a focus on the language structuring provided by the coordination of participants’ bodies during interactions, within the space they inhabit and according to the constraints exerted by the ongoing activities in which they are involved as they use language.

The kinesiological approach reconciles actional and symbolic human activities (Boutet 2018: 17). It shares some foundations with the form-based ToGoG approach (Müller, Bressem and Ladewig 2013) in which a detailed analysis of the movement of gestures is the starting point for the analysis of their meaning. Müller (2005) has shown that gesture can be representational and performed via drawing, modeling, enacting or embodying through imitation. This therefore makes it possible to summon the praxic dimension of gesturality (Boutet 2018: 400). It highlights how much symbolic gesturality has actional roots. These common ideas allowed us to work harmoniously with Cornelia Müller herself as well as with Alan Cienki within a project on aspectuality and gesture funded by the National Science Foundation for five years (Cienki and Iriskhanova 2018), in which we defended the idea that gestures take on or complement some of the linguistic functions of vocal units and take part in the syntactic structure of the multimodal utterance (Ladewig 2014; Cienki 2016).

The orchestration of the semiotic resources deployed by interacting subjects depends on their intersubjective relationships, their reactions to each other, the context, the environment, the activities in progress, the time of day ... according to the dynamic scope of relevant behaviors presented by Cienki (2012).

To describe and analyze the gestural component of multimodal constructions, we need a gesture-based approach in which principles govern our potential for “languaging” and where formal differences are relevant and allow subtle variations in meaning to be grasped. Boutet’s kinesiological approach provides the relevant formal foundations for such an approach.

The body as structure

Most studies and categorizations of communicative gestures focus on the hands because they are the most complex and salient articulators that are mobilized in the visual-gestural mode during multimodal communication. However, we mobilize other articulators (head, face, shoulders and trunk orientation) to communicate (Filhol, Hadjadj and Choisier 2014). This focus on the most distal parts of our body (those which are furthest from the trunk), our hands, is criticized by Boutet (2018) who devoted his research to demonstrating how physiological constraints structure meaning and how all segments of our body, in particular the arms and their dynamics, must be analyzed.

The kinesiological approach to gesture is formal: gestures’ formal components shape their meaning or function. According to Boutet, production, performance, process, seem to have been somewhat forgotten in the analysis of symbolic gesture. “The body has been viewed as a location in which movements simply appear without there being any materiality attributed to it” (Boutet 2008: 82).

However, there is a difference between the phonatory equipment used for vocal languages and the articular equipment used for signed languages: The phonatory tract has only one function, while the upper limbs have several. Except for a few screams or throat clearing, phonatory equipment is used for speaking and singing. Of course, the voice is constantly adjusting and modulating in response to the environment and the communication situation, but the vocal cords only serve this verbal communication and artistic expression. It is therefore easy to understand that the phonology of vocal languages is not influenced by our physical environment. Conversely, the manual equipment at work in sign languages is used daily to interact with the world in an array of activities. The materiality of the body has had the potential to shape our environment, our tools, our objects, the spaces we inhabit (Leroi-Gourhan 1993). Praxic gesture is omnipresent, whether for hearing people or for deaf signers. All this experimentation of the artifactual world made with and for our upper limbs has every reason to exert an influence on co-verbal gestures and on the realization of signs. For Boutet (2018), the structuring of these artifacts is closely linked to praxic gestures, and is in full continuity with symbolic gestures. However, this strong link constitutes a theoretical approach that has never been applied in depth to co-verbal gestures and sign languages (a preliminary study was carried out by Boutet, Chevrefils and Thomas (2019) on manual configurations which were taken as privileged receptacles for this influence of praxic body language on symbolic body language). For Boutet, referents are not primarily associated with gestures through the human capacity to construct visual analogies, but their meaning is directly shaped by the gestures we produce and their dynamics. Boutet describes gestural units on the basis of their formal characteristics and physiological constraints rather than their imagistic iconicity. Indeed, vision has long been considered as the only modality to be taken into account in the analysis of expressive or interactional gestures because our eyes make it possible to perceive, capture and recognize forms. Boutet (2018) shows to what extent gesture studies have ignored art history, but at the same time, he also criticizes how much our apprehension of gesture seems unconsciously linked to the postures represented in art forms that freeze the dynamics of gesturality. By grounding their perception on vision, unconsciously removing movement and its flow from our system of analysis, gesture specialists have not been able to grasp what constitutes the true essence of gesturality. Despite the impact of using video, our analyses of gestures are too often based on the analysis of their static visual representations derived from art and we simply analyze images through a succession of postures (often represented by screen captures or drawings). Movement with its rhythmic variations and its timing is too often discarded.

As opposed to Boutet’s approach (2008, 2010, 2018), in most of the research conducted in gesture studies, praxis and proprioception, sometimes called upon by artists as well as researchers, are not at the heart of the analyses. But if we only take into account the visual modality, each gestural unit may seem to involve multiple, discontinuous formal elements. If we change our approach and place the proprioceptive modality at the core of our analysis, we then find a unique and continuous formal envelope (Boutet 2008: 82). Boutet therefore has offered to use a true phonology of gesture based on human physiology. But this phonology, unlike the analyses proposed by McNeill (1992), is not based on a three-dimensional egocentric frame of reference in which gestures are taken into account in a rather holistic way and within which it is impossible to analyze the movement of each of the segments independently, and thus to capture both the links between them and their variations. When we use an egocentric frame of reference which is three-dimensional, we can take into account whether a movement is upward or downward, forward or backward, or is performed on the left or on the right in relation to the moving subject. That frame of reference allows us to identify general directions, but it remains relative, static … and “’impressionistic” (Boutet 2008: 83). There have been a number of studies in which the body is invited to “play a role” (Boutet 2018: 5) such as in the research conducted by Goodwin (2000) or Gibbs (2005). The egocentric frame of reference has enabled authors to make sense of many metaphors (Lakoff and Johnson 1985). Action can be described as simulated (Hostetter and Alibali 2008). Thus, in these studies the body is “admitted, but must keep its rank” (Boutet 2018: 5) and analyses remain under the influence of Cartesian dualism. A unitary and disembodied conception of the body takes precedence over its segmentation and its materiality (Boutet 2018: 11). In his own work, Boutet proposes to fully take the body into account in all its materiality and to place it at the heart of the very fabric of language and of collective language practices with an intrinsic point of view.

An articulatory frame of reference

In order to give us the possibility to describe gestures, to locate them as precisely as possible, to account for the movement of each of the segments with both their own dynamics and the links between them, the kinesiological approach requires a true articulatory frame of reference. The unit Dominique Boutet chose is the degree of freedom. For example, for the hand, there are two degrees of freedom, flexion/extension and abduction/adduction. Boutet was inspired here by human articulatory biomechanics. The degree of freedom is defined as the independent relative movement of a segment with respect to its adjacent and more proximal counterpart (Boutet 2018: 24). This means that the movement of the hand is analyzed in relation to the position of the forearm, the forearm to the arm and the arm to the trunk. Taking degrees of freedom into account makes it possible at each moment of the gesture to describe the positions and movements of each of the segments (fingers, hand, forearm, arms and shoulders for the upper limb). The analysis can also extend to the face, head and trunk. This frame of reference makes it possible to measure all the movements of all the segments, and is centered on each joint. This involves proposing a multiple intrinsic analysis centered on what the degrees of freedom afford in terms of movement and position for each single segment (Boutet 2018: 11).

Throughout his research, Boutet has mainly worked on the upper limbs by focusing on their position and movement5 but an analysis of facial expressions is also in progress6.

There are 28 degrees of freedom from the fingers to the shoulder. Boutet reminds us that they are defined in relation to an anatomical reference position: the body is upright, the head, the torso, the knees, the toes are directed forward. The upper limbs are dangling alongside the body, the palms facing forward.

This makes it possible to build a 28-dimensional space, with 4 degrees of freedom for each finger except the thumb which has 5: there are 2 for each hand, 2 for each forearm, 3 for each arm with the shoulder joint (for details see Boutet 2008, 2018).

This is the reference position on which were established, among others, the occidental notions of “front,” “back,” “left” and “right” (Fig. 1). Through the addition of axes to this posture, all the movements a body can perform have been conventionalized according to degrees of freedom. The median plane divides the body in its height, it separates the body equally between the left and the right and differentiates the movements in which the segments move away from the axis (grouped under the abduction category), from those in which the segments get closer to the axis (adduction category). The frontal plane separates the movements which are located in front of the axis (flexion category) from those which are located behind the axis (extension category).

Figure 1. 

Representation of two anatomical planes as a reference to the abduction/adduction and flexion/extension degrees of freedom (image is taken from https://www.shutterstock.com/fr/search/similar/1373139335 and slightly modified).

We can observe that rotations are missing from this categorization. However, rotations affect two segments: the arm (external and internal rotation) and the forearm (pronation and supination). Unlike the others, these two degrees of freedom are not located on a joint – shoulder, elbow or wrist – but along the bones (humerus for the arm, ulna and radius for the forearm) (Fig. 2). When the human body is in its anatomical reference position, the axis is located inside the upper limb: the segments rotate on themselves.

Figure 2. 

Representation of two parallel rotations in anatomical reference position, external/internal rotation on the arm and pronation/supination on the forearm (image is taken from https://www.shutterstock.com/fr/search/similar/1373139335 and slightly modified).

These degrees of freedom can be added to each other; they can operate at the same time, and even influence each other. Indeed, this contiguity between the segments (they are all attached to each other) allows the body to be used in three-dimensional space, but also creates constraints in the possibilities offered for movement to be performed. So, it is necessary to take into account all possible physiological constraints in order to understand that some have a reduced amplitude. Indeed, if we consider that the neutral position is the one in which we have our arms hanging along the sides of the body and if we start from there to place the forearm in the position of maximum flexion (hand raised at the height of the neck), the pronation of the hand is then only possible at a 45° angle and not at an 85° angle (Boutet 2008: 90). Thus, depending on the position of each segment, the movements of the other segments will have different constraints. Furthermore, the axes of rotation are variable and also depend on the position of each segment. The two degrees of freedom of the hand are not independent. They result from limited amplitudes which structure their relationships and must truly be integrated into the analysis of the movements of each gesture studied. Boutet therefore tried to show that the visual modality is quite insufficient to follow and identify each type of movement and that we must fully understand the physiological constraints of the human body in order to analyze gesture.

Movement and propagation of movement: importance of the flow

Because the articulators used to perform gestures are much more visible and therefore accessible than those which allow us to produce sound (only the acoustic effects of sounds are perceptible apart from accentuated mouth movements), Boutet (2010) proposed to adopt a “bottom-up” approach based both on perception and on our knowledge of physiological processes. Gestures are analyzed as movement that flows from one segment of the body to the next. The trunk of the body is used to construct the frame of reference for other parts of the body which thus can either be called proximal (closest to the trunk) or distal (farthest from the trunk). The distal or proximal nature of moving segments permeates gestures with meaning, rooted in experience.

Movement can be made up of the effective mobility of a segment (which we have called in English “movement”) or of the displacement of a segment which is not affected by a specific movement (which we have called in English “motion”). When we carry out a detailed analysis of gestures, we can capture movement transfers from one segment to another. On the other hand, it is difficult to identify and describe them in depth without resorting to motion capture. However, for Boutet (2018: 44) when there is displacement (or motion), the movement comes from a more proximal segment (closer to the trunk) and meaning is not to be constructed as rooted in the segment that is simply set in motion but in the segment that carries effective movement (for a detailed analysis of these differences, see Boutet 2015).

As far as the upper limbs are concerned, several segments are set in motion. If the flow that connects one segment to the next travels from the shoulder to the fingertips, the flow is called “proximal-distal”, if the flow travels from the hands to the shoulders, it is called “distal-proximal”. But if the movement is only localized in one part of the body (hand, shoulder, head ...), then there will be no apparent flow. In addition, the inertias of each segment must be taken into account in the analysis of gestures. The proximal-distal flow will be impacted by gravity and will therefore require less energy than the distal-proximal flow for which it will be necessary to go against gravity. “The difference between certain gestures only concerns their flow. These gestures constitute what seems closest to a minimal pair since, apart from orientation or configuration, the only difference lies in the sequence in which the segments flow one after the other.” (Boutet 2018: 9).

As an example, let us analyze the gesture in which the subject holds out her arm with a palm up open hand. That gesture can be either identified as a presentation gesture —translated in words with “c’est ça” (that’s it) by Calbris (1990) – or as an epistemic negation—that we could formulate into words as “I don’t know” or “never mind”, see Debras 2017 for a variety of meanings in adults, or Beaupoil-Hourdel and Debras 2017 as well as Morgenstern et al. 2017, in children.

By focusing on the hand and on a frozen image, the analysis leads to two different interpretations which clearly demonstrates that the method is insufficient. The gesture must be analyzed in the context of the movement of the whole body (see section 3) and not via a focus on the hand(s). Gestures are not frozen postures and must be considered as meaningful movement.

Flow therefore organizes meaning. The great value given to flow in the kinesiological approach focuses on how movement and its dynamics are essential for the analysis of gesture. Gestural forms are not considered simply according to their resemblance to a referent (Boutet 2005) but as informing meaning.

Methods of analysis

Movement has not often been placed at the core of analyses in gesture or sign studies as the various methods used were focused on the trajectory and could not uncover physiological invariants. Dominique Boutet’s kinesiological approach, gives us the possibility to analyze movement through its formal features, to fully grasp it, to reveal all the information that it contains. It shows us all the work that remains to be done within linguistics, which are still mostly dominated by models provided at first by written forms, then by vocal forms. Boutet’s goal was to reinstate the body as both the vehicle and the source of meaning.

Boutet (2010) proposed a physiological structuring of gestural units in the form of action patterns. These action patterns include all movements involving transfers over other degrees of freedom that can be grouped together for each gesture. The difference between action patterns is based on physiological and physical parameters. For Boutet (2018), the structuring of gestures is not based on meaning but is grounded in forms. The repeated use of those forms lead to their stabilization. In order to study gestures, as they are rooted in their own materiality (Boutet 2018: 79), it is necessary to carry out low-level analyses by taking into account the segments engaged in a gesture, their position, the movement of each segment and their flow as they unfold in time. The semiotics of gestures emerge for the person who produces the gestures as well as for the person who perceives them thanks to their action patterns. Those action patterns can be tested by judges and categorized in the form of gestural units (GU). A GU can be performed in a variety of ways, it can thus be considered as an envelope for potential gestures (Boutet 2018: 80). By using an experimental design in which judges recognize labels associated with videos of gestures, it is possible to validate the semantics of gestures and the physiological structuring of their meaning.

Thanks to his method which combined corpus analyses and perception tests, Boutet analyzed and isolated gestures with very similar forms and sought to clarify a network of semantic relationships based on the constraints and affordances of the body.

In order to conduct corpus analyses, Dominique Boutet constantly created detailed “templates” using the ELAN software (https://archive.mpi.nl/tla/elan; Wittenburg et al. 2006) which made it possible to annotate all the kinesiological parameters he highlighted. This enabled him and the various colleagues who worked with him to capture the internal composition of gestures as well as their differences, however slight they may be, using internal criteria based on the specificities of the part (or parts) of the body that are in movement (or set in motion). This is why Dominique Boutet naturally turned to motion capture and new technologies and enthusiastically followed the advice given by Goldin-Meadow and Brentari (2017). This allowed us to explore form-function pairings and to conduct automatic extraction with formal criteria with much more rigor and without depending solely on our visual perception. Dominique Boutet focused his most recent activities (the Polimod Project, the Typannot Project, the PhD theses of Léa Chevrefils and Chloé Thomas) on a light, portable system called the “Perception Neuron” (https://neuronmocap.com), made of inertial units, which enabled us to collect an interactional corpus. By coupling the Neuron with video collection, it is possible to compare the data from the motion capture with the manual annotations carried out in ELAN. The objective is to arrive at form / function pairings that could allow a semi-automatic annotation of gestures.

Dominique Boutet had also started working with OpenPose and OpenFace which offer the possibility to use videos collected in a more ecological environment and thus to automatically detect postures and facial expressions and analyze, for example, family interactions7 without the participants having to wear motion sensors, which can be invasive for the speaker or signer.

Results

The kinesiological approach has made it possible for Boutet and his colleagues in their research on the gestures performed with the upper limbs, to show that arms are the primary substrate of gestural meaning which subsequently extends all the way to the hands (Boutet 2018: 115). Meaning spreads over several segments. The method implemented by Boutet gives us the tools to grasp how movement propagates over all the segments of the upper limbs and thus how meaning is constructed through the gestural flow. Because they are first determined kinesiologically, rather than being analyzed through the visual traces they leave as they are perceived in reality or on video, gestures are analyzed within the course of their movement. The deployment of gestures takes precedence over their trace. “The temporal unfolding of gestures is essential for the understanding of what is at stake in each gesture” (Boutet 2018: 116). Boutet’s first studies also show that motor control plays an important role in gestures’ degree of conventionalization. Lesser motor control could correspond to pragmatic (or recurrent) gestures which are thus also cognitively more economical. The greater the motor control, the more the idiosyncratic character of gestures might increase (Boutet 2018: 226).

Moreover, unlike approaches centered on iconicity in which representation is secondary, in the collective work inspired by the kinesiological approach, the body is not a vector but structures gestures, and flow functions as a distributor of meaning (Boutet 2018: 119).

Applications

Presentation gesture / “epistemic negation”

As an extension of the ANR CoLaJE project (Morgenstern and Parisse 2012) in which a team had worked on negation gestures, we were interested in recurrent gestures in adults and children. “Recurrent” gestures (Müller et al. 2013) are the result of a gradual sedimentation of our sensorimotor system. They are culturally shared, embedded in conventional, embodied experiential frameworks. They are precipitates of experience that have given us the potential to construct multimodal scripts by taking into account multiple temporalities: that of a single community that shares a culture and language, that of each individual’s development, and the moments spent with our conversational partners. The meaning of recurrent gestures is not as conventionalized as the meaning of emblems and it is only possible to associate forms with functions in a conversational context by taking into account the verbal content of the utterances, prosody, gaze, facial expressions and all the gestures produced. In our perspective, recurrent gestures have the potential to express a substrate or a meaning that can only be deployed in association with other semiotic resources according to the affordances of the situation. However, it is possible to uncover formal differences between seemingly very similar gestures, which is what we are trying to do in order to differentiate the presentation gesture and the gesture for “epistemic negation”.

The presentation gesture offers a visual resemblance to epistemic gestures. As far as their function is concerned, the presentation gesture is rhematic, it often allows the speaker to add new information, whereas “epistemic negation” is thematic and brings a subjective perspective on a shared object of discourse. They have been analyzed as similar in form (Calbris 1990; Kendon 2004) whereas for Boutet (2018: 16) the presentation gesture is informed by a distal-proximal flow (Fig. 3) and the gesture of epistemic negation, by a proximal-distal flow (Fig. 4). To be more precise the epistemic gesture emanates from the arm. The transfer of movement can affect the forearm, the hand and then the fingers. But the gesture can propagate towards the shoulder and be transformed or completed by a shrug of the shoulder (to which a shrug of the eyebrows and a puckering of the mouth could be added). That gesture has been described as a composite gesture (Streek 2009; Debras 2017). One of its parts can metonymically represent the whole. The velocity of these two gestures, impacted by inertial differences can also be different. Gesture typologies that are based solely on posture are insufficient to accurately analyze the differences between these gestures. The analysis of gestures in terms of images (McNeill 1992), which is based on the visual modality, does not take into account each of the gestural articulators, their organization, their dynamics, but they are necessary to highlight the structural differences that inform their semantic differences.

Figure 3. 

Articular decomposition of an epistemic negation performed in context. Screen shots (Michael Jordan) are extracted from the video on line https://www.youtube.com/watch?v=gbngKR8UHLk&ab_channel=ESPNESPNValid%C3%A9 (Timing around 47 seconds).

Thanks to a series of similar analyses, it is possible to uncover gestural invariants, with identic features throughout different instances in a range of contexts. In these cases, the extension of the arm, the rotation of the forearm and the supination of the hand according to a proximal-distal flow constitute the invariants expressing epistemic negation, non-existence, absence and/or incapacity. In the presentation gesture, the same degrees of freedom are used, but the flow is distal-proximal and the hand is the actual leader of the movement.

Figure 4. 

Articular decomposition of a presentation gesture performed in context. These screen captures are extracted from the data set collected in France for the Polimod project funded by the Russian Science Foundation (Cienki and Iriskhanova 2018; Boutet et al. 2018ac; Morgenstern et al. 2017).

For a gesture to be identified as a presentation gesture, the hand must at least be in a supine position. The motion of the forearm (and possibly the arm) is triggered by the movement of the hand. The forearm and arm are impacted by the hand’s own movement. In the case of epistemic negation, it is the hand that is set in motion thanks to the movement of the forearm, or even the arm.

We must change our perspective on the unfolding of a gesture of the upper limb to determine the flow of the movement, which thus makes it possible to distinguish forms that are visually close but have different meanings. Gestures of “presentation” as opposed to “incapacity” or “ignorance” take place according to opposite flows.

Moreover, if the flow structures the unfolding of the gesture, meaning can be instantiated on a segment far from the origin of the gesture. There is no form/function pairing independently of the degree(s) of freedom that generate(s) the form, and a form cannot be studied without analyzing the flow of the movement. It is therefore important to identify the segment that generates the form and then to grasp the flow of the movement and the segments on which it propagates.

All of this makes it possible to understand that forms which are visually very different, such as the “shrug” and Palm Up Open Hand are closely linked. A number of formal characteristics distinguish these two gestures. However, there is a causal link between the two. Shoulder shrugs and Palm Up Open Hand seem to move in opposite directions (up and down), they have divergent trajectories and different amplitudes but they are linked through a transfer of movement and can be considered as forming two instances of the same gesture. Shoulder shrugs and Palm Up Open Hand are indeed connected by many authors (Streeck 2009; Debras 2017; Cooperider, Abner and Goldin-Meadow 2018).

If these differences between epistemic and presentation gestures, which were brought to light by Boutet’s early work, have been confirmed in our first analyses of a few coded sessions of family dinners, of dyadic adult-child and adult-adult corpora in English and in French, more coding must be carried out in order to ground Dominique Boutet’s dazzling intuitions, linked to his mastery of the bio-mechanical functioning of gestuality, on solid statistical results. Ideally, fine manual annotations should be coupled with motion capture data. But our preliminary results, which are the first to our knowledge to deal with this type of differentiation in context on two languages, are promising. The flow is possibly a significant parameter in the pairing of gesture forms and functions. An analysis of equivalent functions in French Sign Language, in which the semantic stakes are higher, also allows a more detailed analysis of the involvement of distal segments in meaning variations.

The signs [PLACE] and [CATCH] in French sign language are carried out with the same degrees of freedom: extension of the forearm and closing of the fingers. But the order of activation of these degrees of freedom is reversed. For [PLACE], the flow is proximal-distal (Fig. 5) with forearm extension first; for [CATCH] the flow is distal-proximal (Fig. 6) with the closing of the fingers first. Therefore, there is a specific way to distinguish the movement of these two signs: their flow.

Figure 5. 

Articular decomposition of the sign [PLACE] in French Sign Language (spreadthesign.com).

Figure 6. 

Articular decomposition of the sign [TO CATCH] in French Sign Language (spreadthesign.com).

Perfective/imperfective aspect

In the research on grammatical aspect and gestures conducted as part of the Polimod project funded by the Russian National Science Foundation and coordinated by Alan Cienki (Cienki and Iriskhanova 2018), we illustrate how research on gestuality in conversation provides new ways to study participants’ conceptualization of grammatical notions during multimodal “languaging” (Linell 2009). We analyzed the expression of aspect in French, Russian and German by correlating the analysis of verbal forms with that of synchronously produced gestural forms. Our hypothesis was that the differences between perfective and imperfective aspects could be correlated with kinesiological aspects of gestures. The idea was that certain kinesiological features, and in particular the quality of the movement, could make the semantic differences truly tangible. The nature of the co-verbal gestures used by speakers could thus be linked to aspectuality and their specific features might be captured in our detailed annotations.

By coding ten dyadic interviews between French students, each lasting 15 minutes, we uncovered a strong correlation between gestures that we called bounded (with a strong punctual acceleration during the unfolding of the gesture) and the passé composé (perfective) and unbounded gestures (whose unfolding was performed with a stable speed) and the imparfait (imperfective aspect) (Morgenstern et al. 2017; Boutet et al. 2018a, b, c). We wanted to confirm this first study by analyzing the kinesiological parameters involved and in particular not only speed but also flow as well as the segments involved (fingers, hands, forearm, arm, shoulder) (Boutet et al. 2016a). We found that the bounded quality of a gesture that is grounded in a sudden variation in velocity is much more likely to be produced with a distal-proximal flow and that unbounded gestures were performed with a proximal-distal flow. According to the kinesiological approach, velocity is greater when the movement originates on distal segments (hands and fingers). In the case of a movement that originates on the hand and propagates to the forearm, the hand acquires a certain speed with accelerations. The speed depends on the length of the segment. The forearm is three times longer than the hand. When the motion is transferred from the hand to the forearm, the acceleration is potentially higher, and this is often visible in the video. This explains why the bounded quality of gestures with its sudden changes in velocity could be related to the distal-proximal flow.

The results of our detailed coding of 4 sessions (8 French participants) showed that 81.3% of the gestures accompanying a verb in the imparfait (imperfective aspect) were produced with proximal-distal flow (unbounded gestures) and 74% of the gestures accompanying a passé composé (perfective past tense) were performed with a distal-proximal flow. We also showed that gestures accompanying the passé composé were significantly faster than those accompanying the imparfait and involved fewer segments. Perfectivity thus seems to be associated with well-determined kinesiological parameters and in particular the flow of the movement which draws on both biomechanical properties and their cause. We need to continue these analyses and in particular on our data collections that involved both video-recording and motion capture with the Neuron, in order to be able to understand if the biomechanical properties of the movements embodying aspectuality are indeed the link between form and function of the gestures we have analyzed.

Dominique Boutet as an inspiration for new methods and approaches

Dominique Boutet’s personality, his energy and his multidisciplinary span were a source of inspiration for a variety of projects and studies.

Linguistics and insights from motion capture

Dominique Boutet and Catherine Bolly had long highlighted the importance of taking gestures into account in elderly people’s communicative expression. They were part of an international network interested in the effects of aging on language. With the SignAge project, we wanted to explore several hypotheses presented in the literature as characterizing elderly people’s multimodal communication: i) for both signers and non-signers amplitude of gestures are reduced because angles of rotation are reduced, ii) for signers, there is a grammatical use of space specific to this age range; iii) for both signers and non-signers, vocal-gestural prosodic markers, and a specific management of speech turns can be identified. Dominique emphasized the value of using motion capture to measure rotations, duration, speed, acceleration and jerks, and to test his model based on movement distribution according to segments and their degrees of freedom. He convinced us to use mocap, with humility and realism, insofar as these tools were to allow us to objectify and semi-automate annotation and processing.

We started with a minimally invasive and inexpensive tool, the Kinect. First8, we studied the movements of the head, which we classified according to the three rotation axes: nodding, headshaking, and tilting. We used two coding methods in parallel: 1) we conducted manual annotations with ELAN based on the video, thus in 2D. 2) We extracted the 3D positions recorded by the Kinect in a spreadsheet and applied thresholds. These thresholds concerned the deviation of the positions on each axis, for a defined number of successive images, and enabled us to automatically identify the three types of head movements. We wanted to confront our manual annotations and the annotations derived from the Kinect (filtered thanks to our thresholds), in order to progressively adjust the two types of inputs. This approach makes it possible to check the annotator’s perception in 2D and to adjust the thresholds so that the two coincide as much as possible in order to objectify and facilitate annotation in the future.

We then acquired9 a more precise piece of equipment with IMU, the Neuron, (see description above). Although the tool is more invasive, our older signing participants seemed to cope with it very well. As in the previous study, we worked on the video data in parallel, segmenting and decomposing the lexical units in ELAN, in order to calculate signing rate (average duration of a sign, average duration of its gestural core, the stroke). Here again, we hypothesized that the flow rate slows down with age. But, if the average number of signs per unit of time seems to decrease, and the average duration of the sign increases, the average duration of the stroke seems to remain relatively stable and conforms to the average described in sign language metrics. We therefore hypothesized that the slowing down occurs in the transitions between strokes. This is one of the aspects that could clearly confirm Dominique’s proposals on the distribution or propagation of movement according to the segment and to its degrees of freedom. Using other scripts developed by the team, we also created visualizations that indicate the distance between the dominant hand and the torso and help us identify patterns or profiles of signers related to the mocap data.

Finally, more recently still, Chloé Thomas has begun a PhD project on non-manuals in French Sign Language, under the supervision of Dominique Boutet. This has offered us the opportunity to test yet another system, OpenFace. This facial recognition software enables us to trace the face in real time or after recordings, to detect the different landmarks of the face, to identify the position of the head, to recognize facial action units such as the ones defined by Ekman and Friesen (1972), but also gaze direction, as well as 3D variations of each point on the face. The objective is to link the data derived from this mocap system and our manual annotations, in order to establish thresholds that allow us to characterize the movements of the face articulators. We can for example distinguish relevant differences between the various positions of the eyebrows in order to identify “eyebrow raising” according to a scale specific to each individual signer, and to check that this movement has indeed been identified as such via our manual annotation, thanks to the Typannot system (see Chevrefils et al. in this issue).

By systematically applying this process to non-manual gestures, our objective is to identify formal patterns, invariants, and take into account all the segments and articulatory flows based on the model proposed by Dominique Boutet. This should also help us automate the annotation of non-manuals in sign languages.

Dominique Boutet’s Reciprocal Relation With the Arts

As part of his multidisciplinary approach, Dominique Boutet developed many connections with the arts. He naturally searched for the influence of art in his own studies but his different models have also influenced some recent artistic creations. Three of his out-of-the-box contributions are presented here: a) the vision-centered gesture analysis paradigm linked to art history, b) the gesture annotation of an expressive virtual actor in an interactive digital art installation, and c) tools developed first for motion capture analysis applied recently to a live piano performance using augmented reality.

  1. Since the revival of gesture studies in the ’80s, according to Boutet (2018), we have been analyzing the representation of gestures rather than gestures for themselves. In several talks and workshops (Boutet and Jego 2016, 2017), we investigated how gesture analysis – which is mostly based on vision – could be influenced by art history. In many artworks from prehistory to the modern period, gestures have been studied by painters, sculptors or photographers not only as simple postures but also as movement. Artists have been representing movement by playing with anatomy, with proportions or even with the different body parts of the subject: they were able to show different moments of an action in the same picture or could use superposition as in chrono-photography. Regarding composition, it seems interesting to consider it not only as pictorial but also as semiotic (Boutet 2018). This reveals that the dominant models used to represent and analyze gestures are mostly vision-centered, whereas artists tend to conceive/perceive movement from the inside with a more embodied perspective. We started investigating how to explore gesture from other perspectives. For instance, we used motion capture which can combine different modalities involving proprioception or kinesthesia and free our perspective from a vision-centered analysis (Jego, Meyrueis and Boutet 2019).
  2. In the field of digital arts, (Boutet, Tramus, Blondel, et al. 2016) proposed an unconventional contribution to gesture annotation: they analyzed improvisations between a participant and a virtual agent in an interactive art installation InterACTE (Batras et al. 2016). The participant was invited to interact with the shadow of a virtual character using upper-limb gestures. This character was able to generate gestures picked in a database and also to imitate the participant’s movements using a Kinect motion capture camera which tracks the body but not the fingers. The arm movements of the virtual character appeared to be smooth and did not include finger movements. This happens to look similar to children’s gestures before 17 months. Children use relational gestures initiated on the arms, as opposed to manipulative gestures, initiated on the hands. Thus, participants tend to adapt their behavior to this limitation as if the virtual character were a child. This adaptation and this type of gestural exchange based on imitation push the participant to integrate the limitations of the virtual actor’s gestures. Moreover, since the participant’s emergent gestural style appears even in non-imitative productions, a form of empathy could thus be observed. This empathy favors movements of the arms and forearms and thus triggers the participant to gesture mostly with proximal segments. This experiment sheds new light on gestural exchanges and raises new questions: Is it the absence of vocal production and the limitation of gestures and/or mimicry that promotes empathy in these interactions with the virtual actor? Because of this type of constrained gestuality, is the relationship with the virtual agent similar to what we find in adult-child interactions?
  3. In the Polimod project detailed above, we explored how to study and measure the aspectuality of gestures across three languages (Russian, German and French). The research project focused on the quality of the movement and considered the gesture’s boundedness (bounded or unbounded) in terms of kinetic “pulse of effort” as defined by Boutet, Morgenstern and Cienki (2016). In order to evaluate the “pulse of effort” which is based on gesture kinematics, we used motion capture to observe the different frames of reference of the participants’ gestures and also the different levels of complexity and the boundedness of the “pulse of effort”. We developed a tool to visualize in real time different gesture descriptors based on kinematics such as velocity/acceleration/jerk of any body part position or orientation using a motion capture system (Jégo, Meyrueis and Boutet 2019). It helped to assess the range or motion of each degree of freedom for each segment (arm, forearm, hand) displaying graphs for the frame of reference of the desired body part and with the desired perspective (first- or third-person viewpoint).

Following this study, the tool has recently been used in order to visualize different kinematics of the complex body movements of a pianist playing the contemporary piece Evryali by the composer I. Xenakis (Antoniadis et al. 2021). Since Evryali evokes “the open sea,” we opted for aesthetics related to wave forms and light, aiming to create a synesthetic experience combining sound, visuals and motion visualization. We thus designed an augmented reality stage made of several interactive video projections of a virtual avatar of the performer. The movements of his chosen body parts and the physical efforts required to perform Evryali were represented in virtual space using trails and curves. We were also able to switch from egocentric to allocentric points of view for the avatar of the pianist in real time. The performance was presented in March 2021 in France.

These back-and-forth movements between disciplines are clearly visible in the conclusion of Dominique Boutet’s habilitation thesis (Boutet 2018). He underlines that gestural forms are not essentially limited to traces left in space. He additionally identifies the importance of the degrees of freedom or the poles of each body segment, and proposed to explain that the diversity of gesture analyses is due to the various frames of reference, the flows and the levels of structuring. Thus, he opened new avenues to explore gesture. We have barely started to implement these crucial new aspects in our various disciplines, but these groundbreaking perspectives will remain true inspirations to make the invisible visible.

Conclusion

Boutet (2018) has shown that in Kendon’s continuum (McNeill 1992) speech and its monolinearity have continued to divert analyses from body gestures and their role in structuring meaning. Under the influence of our analyses of the vocal-acoustic modality, we have applied a filter on language practices that must now be removed. For Boutet, movement and its flow need to be placed at the heart of our analyses of gesture and not their static and mainly visual representation. Kinesiology, with its intrinsic and multiple frame of reference, makes it possible to analyze the formal link between different parts of the body by capturing the flow of the movement. We have been influenced by the fixed images of drawing, painting, photography and sculpture. Images have such an influence on our gestures that they freeze them into postures. The proposed typologies emerge from our visual modality (McNeill 1992) and even when they are based on configuration and orientation (Kendon 2004), physical grip (Calbris 1990), or simulation of actions (Hostetter and Alibali 2008), none of them interrogate the physical and physiological constituents of gestures and they leave little room for movement (Boutet 2018: 218).

By applying a kinesiological approach and combining the possibilities offered by new technologies and the collective skills of multidisciplinary teams of researchers, it will be possible to place movement at the heart of our analyses of gesture and shift paradigms.

Boutet has responded in his own way to the call launched in 1998 by Cornelia Müller to make gestural studies a discipline in its own right and to entirely renew our approaches and methods. In his work, he combined a wide variety of scientific and artistic cultures that put the body at the center of his analyses. As we continue to bring together several communities and disciplines and we reconstruct the synergies that he was able to impulse with dynamism and creativity, we hope that his research program will be implemented collectively.

This paper is based on the introduction written in French by Morgenstern in company of Dominique Boutet’s voice and works after his death (Boutet and Morgenstern, 2020). It was enriched and adapted thanks to the participation of the other co-authors. The Journal TIPA gave Morgenstern special permission to supervise this adaptation to honor Boutet’s scientific approach and his charismatic personality.
2 Affordances can be defined here as the potential offered by physiological constraints that shape the possible movements of each part of the human body.
3 ANR-08-COM-021 Project, Communication Langagière chez le Jeune Enfant, Principal Investigator, Aliyah Morgenstern, Morgenstern and Parisse 2012; https://colaje.scicog.fr.
4 RSF project 14-48-00067 on aspectuality and gestures in Russian, French and German, subsidized by the Russian Scientific Research Foundation, scientific manager Alan Cienki, French partners Aliyah Morgenstern and Dominique Boutet, see Cienki and Iriskhanova 2018; http://scodis.com/our-projects/events-and-gestures.
5 As will be demonstrated in the PhD research Léa Chevrefils is conducting.
6 Dissertation by Chloé Thomas.
7 We will use these technique in the DinLang project – Multimodal LANGuage practices in French family DINners – funded by the French Research Agency (2021–2025 project coordinated by Aliyah Morgenstern with Boutet as co-PI as well as Blondel and Parisse).
8 Blondel, Boutet, Catteau and Vincent (2017).
9 Blondel, Boutet, Catteau and Vincent (2019).

References

  • Antoniadis P., Duval A., Jégo J.-F., Solomos M. (2021, To appear). Dwelling Xenakis. An augmented reality project on Evryali for piano solo. Conference Nova Contemporary Music Meeting (NCMM 2021), Lisbon, Portugal.
  • Batras D., Guez J., Jégo J.-F., Tramus M.-H. (2016). A virtual reality agent-based platform for improvisation between real and virtual actors using gestures. VRIC ’16: Proceedings of the 2016 Virtual Reality International Conference, 1–4. https://doi.org/10.1145/2927929.2927947
  • Blondel M., Boutet D., Vincent C. (2017). LSF interactions in elderly signers: Insights from motion capture? Corpora for Language and Aging Research (CLARe3), Mar 2017, Berlin, Germany. http://wikis.fu-berlin.de/display/clare/HOME
  • Blondel M., Boutet D., Catteau F., Vincent C. (2019). Signing amplitude in older signers of the SignAge Corpus: Insights from motion capture. CLARe4, Helsinki. hal-02065437 HAL Id : hal-02065437.
  • Bottineau D. (2012). Parole, corporéité, individu et société—L’embodiment dans les linguistiques cognitives. Texto !, 17 (1–2), en ligne.
  • Boutet D. (2008). Une morphologie de la gestualité : Structuration articulaire. Cahiers de Linguistique Analogique, 5, 81–115.
  • Boutet D. (2010). Structuration physiologique de la gestuelle : Modèle et tests. Lidil. Revue de linguistique et de didactique des langues, 42, 77–96. https://doi.org/10.4000/lidil.3070
  • Boutet D. (2015). Conditions formelles d’une analyse de la négation gestuelle. Vestnik of Moscow State Linguistic University, Discourse as social practice: priorities and prospects, 6 (717), 116–129.
  • Boutet D., Chevrefils L., Thomas C. (2019, October). Iconicité dans les langues des signes au niveau phonologique. L’Arbitraire et la Motivation, Sorbonne Université, Paris, France. https://hal.archives-ouvertes.fr/hal-02342429
  • Boutet D., Cuxac C. (2008). Le signifiant gestuel : Langue des signes et gestualité ; Avant-propos. Cahiers de Linguistique Analogique, 5, 2–15.
  • Boutet D., Jégo J.-F. (2016). Analyzing Gestures: The Influence of Art History and how MOCAP could change our analyses. MeMuMo: Methods in Multimodal Communication Research, Moscow, Russia.
  • Boutet D., Jégo J.-F. (2017). How to record, analyze and annotate gestures and sign languages with MoCap technologies. MeMuMo: Methods in Multimodal Communication Research, Moscow, Russia.
  • Boutet D., Morgenstern A. (2020). Prélude et Ode à l’approche kinésiologique de la gestualité. TIPA. Travaux interdisciplinaires sur la parole et le langage, 36, Article 36. https://doi.org/10.4000/tipa.3892
  • Boutet D., Morgenstern A., Cienki A. (2016). Grammatical Aspect and Gesture in French: A kinesiological approach. Russian Journal of Linguistics, 20 (3), 132–151.
  • Boutet D., Morgenstern A., Cienki A. (2018a). Looking ahead: Kinesiological analysis. In A. Cienki & O. K. Iriskhanova (Eds.), Aspectuality across Languages (pp. 143–158). John Benjamins.
  • Boutet D., Morgenstern A., Cienki A. (2018b). Summing up: Aspect as amodal or as modality-dependent. In A. Cienki & O. K. Iriskhanova (Eds.), Aspectuality across Languages (pp. 58–60). John Benjamins.
  • Boutet D., Müller C., Morgenstern A. (2018c). Categories used for the analysis of event construal in gesture. In A. Cienki & O. K. Iriskhanova (Eds.), Aspectuality across Languages (pp. 67–76). John Benjamins.
  • Boutet D., Tramus M.-H., Blondel M., Jégo J.-F., Batras D., Guez J., Catteau F., Chen C.-Y., Vincent C. (2016). Gestuelle contrainte et empathie dans l’interaction homme-machine. Rencontres Internationales d’Anthropologie Linguistique (RIAL 2016) “L’être de langage, entre corps et technique : nouvelles données, nouvelle donne ?”, Montpellier, France. https://hal.archives-ouvertes.fr/hal-01293427
  • Calbris G. (1990). The Semiotics of French Gestures. Indiana University Press.
  • Cienki A. (2012). Usage events of spoken language and the symbolic units we (may) abstract from them. In J. Badio & K. Kosecki (Eds.), Cognitive Processes in Language (pp. 149–158). Peter Lang. https://doi.org/10.3726/978-3-653-04562-8
  • Cienki A., Iriskhanova O. K. (2018). Aspectuality across Languages: Event construal in speech and gesture. In Hcp.62. John Benjamins. https://doi.org/10.1075/hcp.62
  • Cooperrider K., Abner N., Goldin-Meadow S. (2018). The Palm-Up Puzzle: Meanings and Origins of a Widespread Form in Gesture and Sign. Frontiers in Communication 3:23. https://doi.org/10.3389/fcomm.2018.00023
  • Edwards T., Brentari D. (2020). Feeling phonology: the emergence of tactile phonological patterns in protactile communities in the United States. Language 96, 1–22. https://doi.org/10.1353/lan.0.0248
  • Ekman P., Friesen W. V., Ellsworth P. (1972). Emotion in the Human Face. Pergamon Press.
  • Filhol M., Hadjadj M. N., Choisier A. (2014). Non-manual features: The right to indifference. Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC 2014). Ninth International Conference on Language Resources and Evaluation (LREC’14), Reykjavik, Iceland. https://hal.archives-ouvertes.fr/hal-01849040
  • Gibson J. J. (1977). The theory of affordances. In R. E. Shaw & J. Bransford (Eds.), Perceiving, acting, and knowing: Toward an ecological psychology (pp. 67–82). Lawrence Erlbaum Associates. https://hal.archives-ouvertes.fr/hal-00692033
  • Goldin-Meadow S., Brentari D. (2017). Gesture, sign, and language: The coming of age of sign language and gesture studies. The Behavioral and Brain Sciences, 40, 1–82. https://doi.org/10.1017/S0140525X15001247
  • Jégo J.-F., Meyrueis V., Boutet D. (2019). A Workflow for Real-Time Visualization and Data Analysis of Gesture using Motion Capture. Proceedings of the 6th International Conference on Movement and Computing, MOCO ’19, 1–6. https://doi.org/10.1145/3347122.3359598
  • Ladewig S. H. (2014). Recurrent gestures. In C. Müller, A. Cienki, E. Fricke, S. H. Ladewig, D. McNeill, & J. Bressem (Eds.), Body – Language – Communication: An International Handbook on Multimodality in Human Interaction (Vol. 2, pp. 1558–1574). De Gruyter Mouton. https://doi.org/10.1515/9783110302028.1558
  • Lakoff G., Johnson M. (1986). Les métaphores dans la vie quotidienne (M. de Fornel, Trans.). Editions de Minuit.
  • Leroi-Gourhan A. (1964). Le geste et la parole, tome 1 : Technique et Langage. Albin Michel.
  • Leroi-Gourhan A. (1993). L’homme et la matière, tome 1 : Évolution et techniques. Albin Michel.
  • Linell P. (2009). Rethinking Language, Mind, and World dialogically: Interactional and Contextual Theories of Human Sense-Making. Information Age Publishing.
  • Maturana H. R. (1978). Biology of language: The epistemology of reality. In G. A. Miller & E. Lenneberg (Eds.), Psychology and Biology of Language and Thought: Essays in Honor of Eric Lenneberg (pp. 27–63). Academic Press.
  • McNeill D. (1992). Hand and Mind: What Gestures Reveal about Thought. University of Chicago Press.
  • Morgenstern A., Blondel M., Beaupoil-Hourdel P., Benazzo S., Boutet D., Kochan A., Limousin F. (2017). The blossoming of negation in gesture, sign and oral production. In M. Hickmann, E. Veneziano, & H. Jisa (Eds.), Sources of variation in first language acquisition: Languages, contexts, and learners (pp. 339–363). John Benjamins. https://doi.org/10.1075/tilar.22.17mor
  • Morgenstern A., Boutet D., Debras C. (2018). Chapter 3.2. The French speakers’ verbal expression of event construal. In A. Cienki & O. K. Iriskhanova (Eds.), Aspectuality across Languages (pp. 81–90). John Benjamins.
  • Müller C., Bressem J., Ladewig S. H. (2013). Towards a grammar of gesture: A form-based view. In C. Müller, A. Cienki, E. Fricke, S. H. Ladewig, D. McNeill, & S. Tessendorf (Eds.), Body – Language – Communication: An International Handbook on Multimodality in Human Interaction (Vol. 1, pp. 707–733). De Gruyter Mouton. https://doi.org/10.1515/9783110261318.707
  • Wittenburg P., Brugman H., Russel A., Klassmann A., Sloetjes H. (2006). ELAN: A Professional Framework for Multimodality Research. Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC 2006), 1556–1559.