Corresponding author: Aliyah Morgenstern ( aliyah.morgenstern@sorbonne-nouvelle.fr ) Academic editor: Olga Iriskhanova
© 2021 Aliyah Morgenstern, Lea Chevrefils, Marion Blondel, Coralie Vincent, Chloé Thomas, Jean-François Jego, Dominique Boutet.
This is an open access article distributed under the terms of the Creative Commons Attribution License (CC BY-NC-ND 4.0), which permits to copy and distribute the article for non-commercial purposes, provided that the article is not altered or modified and the original author and source are credited.
Citation:
Morgenstern A, Chevrefils L, Blondel M, Vincent C, Thomas C, Jego J-F, Boutet D (2021) “Of thee I sing”: An opening to Dominique Boutet’s kinesiological approach to gesture. Languages and Modalities 1: 3-16. https://doi.org/10.3897/lamo.1.68148
|
In this tribute to Dominique Boutet and the kinesiological approach he founded, the authors have tried to make their memories of the scientific collective projects they worked on together resonate with the written work of this extraordinary scientific partner, gathered in his published articles and his habilitation document (
The article presents the foundations of the approach, a synoptic description and some examples of its application. The originality of the kinesiological approach lies in the double revolution that it allows us to operate: on the one hand, gesture is not simply an appendix of speech; on the other hand; it is shaped by bodily physiology. The approach is based on the movements of the human body analyzed from a biomechanical point of view. The meaning of our gestural productions is the produce of our body, as it is naturally articulated, imprinted as it is by our past experiences.
flow, gesture, kinesiology, motion capture, movement, sign languages
Dominique Boutet was extremely erudite, passionate and curious. He savored life and all human cultural and scientific productions to the fullest. His kinesiological approach to communicative gesture and sign is difficult to access as it was conceived by combining his knowledge of biology, medicine, biomechanics, art history, linguistics, philosophy, anthropology, archaeology and many other fields.
This article is an attempt to give an account of the sessions shared with Dominique in Moscow, Rouen, Paris and of our work together. It resonates with his Habilitation thesis (2018) and all his publications. We are not immune to a subjective appropriation of his approach, which took us some time to penetrate and which is reconstructed through our own theoretical filters and experiences. But this is what made our relationship with Dominique and our mutual conversions to each other’s ideas and passions so special. We therefore tried to express the fruit of the meeting of his and our subjectivity.
By writing this article grounded in our shared experiences, our memories and his publications, so soon after the departure of this vital friend who died of COVID in May 2020, this scientific brother/father, and partner with unique qualities that Dominique was to us, we first wanted to express our gratitude.
This article is only an “opening” because it was not possible to uncover all the contours and contents of the theoretical and methodological work accomplished by this multidimensional researcher in one paper. It is also an opening to our collective continuation of Dominique’s theoretical and methodological approach, and his scientific findings through a series of projects we will pursue to embody his prolific ideas.
In his combat against the cultural filters that led many linguists to deny the role of the body and its dynamics, Dominique Boutet constantly sought to place the body at the center of all language activity. In order to illuminate how language is structured by our bodily activity in all its materiality, he analyzed sign languages, spoken languages including gestures, facial expressions, gaze and postures, in everyday interactions as well as in more experimental or artistic settings.
The originality of the kinesiological approach (from the Greek kinesis, movement and logos, speech, science) lies in the double revolution that it allows us to operate: on the one hand, gesture is not simply an annex to speech; on the other hand, it is shaped by our body’s physiology. Gestures derive from the movements of the human body analyzed from a biomechanical and articular point of view. The meaning of our gestural productions is articulated out of our body, permeated with our past experiences and our relationships to others. Our gestures are the result of the hybridization between what is universal, generic through the setting in motion of human bodies as they are all naturally structured, and what belongs to each and every single individual, their development, and their history.
The starting point of our work together was our questioning of disembodied linguistics. The forms produced by human beings in interaction are the main object on which we work as linguists. Dominique considered that a merely semiotic approach to language evacuates the signifier as a mode of expression. His kinesiological approach, on the other hand, is based on the signifier, the body and all the segments whose constraints but also whose “affordances
We will first lay the foundations of the kinesiological approach, make a synoptic description of the approach itself and its integration into collective research projects. We will then give some examples of how it was applied in projects in which we analyzed gestures and signs with a variety of perspectives and in connection with an array of scientific fields.
The theoretical and methodological framework proposed by
We therefore analyze human interaction in an approach that allows language to be included in embodied action rather than considering it as the use of a code or a symbolic system (
The analysis of “co-verbal” gesture (we could also describe verbal production as being “co-gestural”) that we will call expressive and interactional, is often “contaminated” (
According to
Gesticulation > Language-like gestures > Pantomime > Emblem > Sign language
This continuum largely takes into account the presence or absence of co-articulated speech. Emblems are considered as the most “lexicalized,” the most “linguistic,” the most “conventional”. If we stop considering the body as a simple support for gestuality, and we take it into account as a substrate, then, this continuum can be reversed. Within gesture studies, it is customary to categorize interactional gestures into 1) iconic or representational gestures which are considered as the least conventional and the most “imagistic”, expressive and individualized, and are opposed to 2) beats which have a “prosodic” role as they structure and punctuate the gestural flow, 3) deictic gestures (including pointing) considered as transparent and linked to their referents, and 4) pragmatic gestures (also called “recurrent gestures” by
For Boutet, our cultural filters and our restriction of gesture to the visual modality have thus largely limited our analyses.
The kinesiological approach proposed by
The kinesiological approach reconciles actional and symbolic human activities (
The orchestration of the semiotic resources deployed by interacting subjects depends on their intersubjective relationships, their reactions to each other, the context, the environment, the activities in progress, the time of day ... according to the dynamic scope of relevant behaviors presented by
To describe and analyze the gestural component of multimodal constructions, we need a gesture-based approach in which principles govern our potential for “languaging” and where formal differences are relevant and allow subtle variations in meaning to be grasped. Boutet’s kinesiological approach provides the relevant formal foundations for such an approach.
Most studies and categorizations of communicative gestures focus on the hands because they are the most complex and salient articulators that are mobilized in the visual-gestural mode during multimodal communication. However, we mobilize other articulators (head, face, shoulders and trunk orientation) to communicate (
The kinesiological approach to gesture is formal: gestures’ formal components shape their meaning or function. According to Boutet, production, performance, process, seem to have been somewhat forgotten in the analysis of symbolic gesture. “The body has been viewed as a location in which movements simply appear without there being any materiality attributed to it” (
However, there is a difference between the phonatory equipment used for vocal languages and the articular equipment used for signed languages: The phonatory tract has only one function, while the upper limbs have several. Except for a few screams or throat clearing, phonatory equipment is used for speaking and singing. Of course, the voice is constantly adjusting and modulating in response to the environment and the communication situation, but the vocal cords only serve this verbal communication and artistic expression. It is therefore easy to understand that the phonology of vocal languages is not influenced by our physical environment. Conversely, the manual equipment at work in sign languages is used daily to interact with the world in an array of activities. The materiality of the body has had the potential to shape our environment, our tools, our objects, the spaces we inhabit (
As opposed to Boutet’s approach (2008, 2010, 2018), in most of the research conducted in gesture studies, praxis and proprioception, sometimes called upon by artists as well as researchers, are not at the heart of the analyses. But if we only take into account the visual modality, each gestural unit may seem to involve multiple, discontinuous formal elements. If we change our approach and place the proprioceptive modality at the core of our analysis, we then find a unique and continuous formal envelope (
In order to give us the possibility to describe gestures, to locate them as precisely as possible, to account for the movement of each of the segments with both their own dynamics and the links between them, the kinesiological approach requires a true articulatory frame of reference. The unit Dominique Boutet chose is the degree of freedom. For example, for the hand, there are two degrees of freedom, flexion/extension and abduction/adduction. Boutet was inspired here by human articulatory biomechanics. The degree of freedom is defined as the independent relative movement of a segment with respect to its adjacent and more proximal counterpart (
Throughout his research, Boutet has mainly worked on the upper limbs by focusing on their position and movement
There are 28 degrees of freedom from the fingers to the shoulder. Boutet reminds us that they are defined in relation to an anatomical reference position: the body is upright, the head, the torso, the knees, the toes are directed forward. The upper limbs are dangling alongside the body, the palms facing forward.
This makes it possible to build a 28-dimensional space, with 4 degrees of freedom for each finger except the thumb which has 5: there are 2 for each hand, 2 for each forearm, 3 for each arm with the shoulder joint (for details see
This is the reference position on which were established, among others, the occidental notions of “front,” “back,” “left” and “right” (Fig.
Representation of two anatomical planes as a reference to the abduction/adduction and flexion/extension degrees of freedom (image is taken from https://www.shutterstock.com/fr/search/similar/1373139335 and slightly modified).
We can observe that rotations are missing from this categorization. However, rotations affect two segments: the arm (external and internal rotation) and the forearm (pronation and supination). Unlike the others, these two degrees of freedom are not located on a joint – shoulder, elbow or wrist – but along the bones (humerus for the arm, ulna and radius for the forearm) (Fig.
Representation of two parallel rotations in anatomical reference position, external/internal rotation on the arm and pronation/supination on the forearm (image is taken from https://www.shutterstock.com/fr/search/similar/1373139335 and slightly modified).
These degrees of freedom can be added to each other; they can operate at the same time, and even influence each other. Indeed, this contiguity between the segments (they are all attached to each other) allows the body to be used in three-dimensional space, but also creates constraints in the possibilities offered for movement to be performed. So, it is necessary to take into account all possible physiological constraints in order to understand that some have a reduced amplitude. Indeed, if we consider that the neutral position is the one in which we have our arms hanging along the sides of the body and if we start from there to place the forearm in the position of maximum flexion (hand raised at the height of the neck), the pronation of the hand is then only possible at a 45° angle and not at an 85° angle (
Because the articulators used to perform gestures are much more visible and therefore accessible than those which allow us to produce sound (only the acoustic effects of sounds are perceptible apart from accentuated mouth movements),
Movement can be made up of the effective mobility of a segment (which we have called in English “movement”) or of the displacement of a segment which is not affected by a specific movement (which we have called in English “motion”). When we carry out a detailed analysis of gestures, we can capture movement transfers from one segment to another. On the other hand, it is difficult to identify and describe them in depth without resorting to motion capture. However, for
As far as the upper limbs are concerned, several segments are set in motion. If the flow that connects one segment to the next travels from the shoulder to the fingertips, the flow is called “proximal-distal”, if the flow travels from the hands to the shoulders, it is called “distal-proximal”. But if the movement is only localized in one part of the body (hand, shoulder, head ...), then there will be no apparent flow. In addition, the inertias of each segment must be taken into account in the analysis of gestures. The proximal-distal flow will be impacted by gravity and will therefore require less energy than the distal-proximal flow for which it will be necessary to go against gravity. “The difference between certain gestures only concerns their flow. These gestures constitute what seems closest to a minimal pair since, apart from orientation or configuration, the only difference lies in the sequence in which the segments flow one after the other.” (
As an example, let us analyze the gesture in which the subject holds out her arm with a palm up open hand. That gesture can be either identified as a presentation gesture —translated in words with “c’est ça” (that’s it) by
By focusing on the hand and on a frozen image, the analysis leads to two different interpretations which clearly demonstrates that the method is insufficient. The gesture must be analyzed in the context of the movement of the whole body (see section 3) and not via a focus on the hand(s). Gestures are not frozen postures and must be considered as meaningful movement.
Flow therefore organizes meaning. The great value given to flow in the kinesiological approach focuses on how movement and its dynamics are essential for the analysis of gesture. Gestural forms are not considered simply according to their resemblance to a referent (
Movement has not often been placed at the core of analyses in gesture or sign studies as the various methods used were focused on the trajectory and could not uncover physiological invariants. Dominique Boutet’s kinesiological approach, gives us the possibility to analyze movement through its formal features, to fully grasp it, to reveal all the information that it contains. It shows us all the work that remains to be done within linguistics, which are still mostly dominated by models provided at first by written forms, then by vocal forms. Boutet’s goal was to reinstate the body as both the vehicle and the source of meaning.
Thanks to his method which combined corpus analyses and perception tests, Boutet analyzed and isolated gestures with very similar forms and sought to clarify a network of semantic relationships based on the constraints and affordances of the body.
In order to conduct corpus analyses, Dominique Boutet constantly created detailed “templates” using the ELAN software (https://archive.mpi.nl/tla/elan;
Dominique Boutet had also started working with OpenPose and OpenFace which offer the possibility to use videos collected in a more ecological environment and thus to automatically detect postures and facial expressions and analyze, for example, family interactions
The kinesiological approach has made it possible for Boutet and his colleagues in their research on the gestures performed with the upper limbs, to show that arms are the primary substrate of gestural meaning which subsequently extends all the way to the hands (
Moreover, unlike approaches centered on iconicity in which representation is secondary, in the collective work inspired by the kinesiological approach, the body is not a vector but structures gestures, and flow functions as a distributor of meaning (
As an extension of the ANR CoLaJE project (
The presentation gesture offers a visual resemblance to epistemic gestures. As far as their function is concerned, the presentation gesture is rhematic, it often allows the speaker to add new information, whereas “epistemic negation” is thematic and brings a subjective perspective on a shared object of discourse. They have been analyzed as similar in form (
Articular decomposition of an epistemic negation performed in context. Screen shots (Michael Jordan) are extracted from the video on line https://www.youtube.com/watch?v=gbngKR8UHLk&ab_channel=ESPNESPNValid%C3%A9 (Timing around 47 seconds).
Thanks to a series of similar analyses, it is possible to uncover gestural invariants, with identic features throughout different instances in a range of contexts. In these cases, the extension of the arm, the rotation of the forearm and the supination of the hand according to a proximal-distal flow constitute the invariants expressing epistemic negation, non-existence, absence and/or incapacity. In the presentation gesture, the same degrees of freedom are used, but the flow is distal-proximal and the hand is the actual leader of the movement.
For a gesture to be identified as a presentation gesture, the hand must at least be in a supine position. The motion of the forearm (and possibly the arm) is triggered by the movement of the hand. The forearm and arm are impacted by the hand’s own movement. In the case of epistemic negation, it is the hand that is set in motion thanks to the movement of the forearm, or even the arm.
We must change our perspective on the unfolding of a gesture of the upper limb to determine the flow of the movement, which thus makes it possible to distinguish forms that are visually close but have different meanings. Gestures of “presentation” as opposed to “incapacity” or “ignorance” take place according to opposite flows.
Moreover, if the flow structures the unfolding of the gesture, meaning can be instantiated on a segment far from the origin of the gesture. There is no form/function pairing independently of the degree(s) of freedom that generate(s) the form, and a form cannot be studied without analyzing the flow of the movement. It is therefore important to identify the segment that generates the form and then to grasp the flow of the movement and the segments on which it propagates.
All of this makes it possible to understand that forms which are visually very different, such as the “shrug” and Palm Up Open Hand are closely linked. A number of formal characteristics distinguish these two gestures. However, there is a causal link between the two. Shoulder shrugs and Palm Up Open Hand seem to move in opposite directions (up and down), they have divergent trajectories and different amplitudes but they are linked through a transfer of movement and can be considered as forming two instances of the same gesture. Shoulder shrugs and Palm Up Open Hand are indeed connected by many authors (
If these differences between epistemic and presentation gestures, which were brought to light by Boutet’s early work, have been confirmed in our first analyses of a few coded sessions of family dinners, of dyadic adult-child and adult-adult corpora in English and in French, more coding must be carried out in order to ground Dominique Boutet’s dazzling intuitions, linked to his mastery of the bio-mechanical functioning of gestuality, on solid statistical results. Ideally, fine manual annotations should be coupled with motion capture data. But our preliminary results, which are the first to our knowledge to deal with this type of differentiation in context on two languages, are promising. The flow is possibly a significant parameter in the pairing of gesture forms and functions. An analysis of equivalent functions in French Sign Language, in which the semantic stakes are higher, also allows a more detailed analysis of the involvement of distal segments in meaning variations.
The signs [PLACE] and [CATCH] in French sign language are carried out with the same degrees of freedom: extension of the forearm and closing of the fingers. But the order of activation of these degrees of freedom is reversed. For [PLACE], the flow is proximal-distal (Fig.
Articular decomposition of the sign [PLACE] in French Sign Language (spreadthesign.com).
Articular decomposition of the sign [TO CATCH] in French Sign Language (spreadthesign.com).
In the research on grammatical aspect and gestures conducted as part of the Polimod project funded by the Russian National Science Foundation and coordinated by Alan Cienki (
By coding ten dyadic interviews between French students, each lasting 15 minutes, we uncovered a strong correlation between gestures that we called bounded (with a strong punctual acceleration during the unfolding of the gesture) and the passé composé (perfective) and unbounded gestures (whose unfolding was performed with a stable speed) and the imparfait (imperfective aspect) (
The results of our detailed coding of 4 sessions (8 French participants) showed that 81.3% of the gestures accompanying a verb in the imparfait (imperfective aspect) were produced with proximal-distal flow (unbounded gestures) and 74% of the gestures accompanying a passé composé (perfective past tense) were performed with a distal-proximal flow. We also showed that gestures accompanying the passé composé were significantly faster than those accompanying the imparfait and involved fewer segments. Perfectivity thus seems to be associated with well-determined kinesiological parameters and in particular the flow of the movement which draws on both biomechanical properties and their cause. We need to continue these analyses and in particular on our data collections that involved both video-recording and motion capture with the Neuron, in order to be able to understand if the biomechanical properties of the movements embodying aspectuality are indeed the link between form and function of the gestures we have analyzed.
Dominique Boutet’s personality, his energy and his multidisciplinary span were a source of inspiration for a variety of projects and studies.
Dominique Boutet and Catherine Bolly had long highlighted the importance of taking gestures into account in elderly people’s communicative expression. They were part of an international network interested in the effects of aging on language. With the SignAge project, we wanted to explore several hypotheses presented in the literature as characterizing elderly people’s multimodal communication: i) for both signers and non-signers amplitude of gestures are reduced because angles of rotation are reduced, ii) for signers, there is a grammatical use of space specific to this age range; iii) for both signers and non-signers, vocal-gestural prosodic markers, and a specific management of speech turns can be identified. Dominique emphasized the value of using motion capture to measure rotations, duration, speed, acceleration and jerks, and to test his model based on movement distribution according to segments and their degrees of freedom. He convinced us to use mocap, with humility and realism, insofar as these tools were to allow us to objectify and semi-automate annotation and processing.
We started with a minimally invasive and inexpensive tool, the Kinect. First
We then acquired
Finally, more recently still, Chloé Thomas has begun a PhD project on non-manuals in French Sign Language, under the supervision of Dominique Boutet. This has offered us the opportunity to test yet another system, OpenFace. This facial recognition software enables us to trace the face in real time or after recordings, to detect the different landmarks of the face, to identify the position of the head, to recognize facial action units such as the ones defined by
By systematically applying this process to non-manual gestures, our objective is to identify formal patterns, invariants, and take into account all the segments and articulatory flows based on the model proposed by Dominique Boutet. This should also help us automate the annotation of non-manuals in sign languages.
As part of his multidisciplinary approach, Dominique Boutet developed many connections with the arts. He naturally searched for the influence of art in his own studies but his different models have also influenced some recent artistic creations. Three of his out-of-the-box contributions are presented here: a) the vision-centered gesture analysis paradigm linked to art history, b) the gesture annotation of an expressive virtual actor in an interactive digital art installation, and c) tools developed first for motion capture analysis applied recently to a live piano performance using augmented reality.
Following this study, the tool has recently been used in order to visualize different kinematics of the complex body movements of a pianist playing the contemporary piece Evryali by the composer I. Xenakis (
These back-and-forth movements between disciplines are clearly visible in the conclusion of Dominique Boutet’s habilitation thesis (
By applying a kinesiological approach and combining the possibilities offered by new technologies and the collective skills of multidisciplinary teams of researchers, it will be possible to place movement at the heart of our analyses of gesture and shift paradigms.
Boutet has responded in his own way to the call launched in 1998 by Cornelia Müller to make gestural studies a discipline in its own right and to entirely renew our approaches and methods. In his work, he combined a wide variety of scientific and artistic cultures that put the body at the center of his analyses. As we continue to bring together several communities and disciplines and we reconstruct the synergies that he was able to impulse with dynamism and creativity, we hope that his research program will be implemented collectively.