Long considered as a very elaborate pantomime, SLs have, since the founding works of Stokoe (1960), been recognized as languages in their own right, thus fully falling within the scope of linguistic studies.¹

The argument making it possible to demonstrate the linguistic nature of SLs goes through the validation of the criteria which, at the time of Stokoe, were considered as defining a language (Benveniste, 1966): 1) there is no voiceless language; 2) linguistic signs must be arbitrary; 3) language must be decomposable into combinable elements, according to defined rules; 4) language must allow dialogue; 5) language must have double articulation. It is possible to verify statements (1) by specifying “gestural voice” to “voice” and (4) by simply observing deaf communication.

The validation of statements (3) and (5) results from the work of Stokoe (1960; Battison et al., 1965), identifying cheremes (i.e., phonemes) in manual parameters (hand shape, orientation, location and movement) and kinemes (i.e., morphemes) in whole signs (note that, for Stokoe, meaning lies exclusively in the hand). However, the validation of statement (2) is problematic: observing SLs reveals a strong iconic motivation linking referent to signifier, which seems to go against the arbitrariness of linguistic signs (Fig. 1).

Figure 1. 

The shape of the sign [TREE] in LSF is justified in relation to its referent: the upper right limb refers to the trunk (forearm) and to the tree branches (hand and fingers) (extract from the corpus DyLIS, 2020).

Early SLs research therefore strives to demonstrate that iconicity is irrelevant, which is tantamount to showing that the signifying form of signs does not matter.

Shortly after Stokoe’s research, SLs captured the interest of researchers (e.g., Klima and Bellugi [1979]) in Chomsky’s rising generativist paradigm (1968). They intended to test their theories on universal grammar using languages which, by their visual-gestural nature, are different from all the other vocal languages (VLs). Their aim was to demonstrate the universality of grammar, independently of the way in which the language is produced: the SLs corporeality, after having captured the generativists’ interest, became an element whose linguistic irrelevance must be demonstrated.

It was therefore within the framework of a body denial that SLs research developed from the ‘60s. In the ‘80s, while generativist research was still in full swing in the USA and in a large part of Europe, the work carried out in France by Cuxac and his team (Cuxac, 2000; Sallandre, 2014) brought back the analysis of iconicity to the center, while underlining the relevance of the visuo-gestural channel for understanding the SLs functioning. However, this research, based on discourse analysis, focused on the functions of signs and not on their signifying form: the signer’s body is considered relevant but only the bodily elements identified as functionally distinctive (e.g., eye gaze, facial expressions, body shifting) are taken into account for the analysis. The signer’s body is considered pertinent as a necessary support for creating iconicity, but it is not recognized as a motor for the creation of meaning.

One explanation for the lack of interest in the signifying form of signs, unrelated to the linguistic theories, lies in the difficulty to graphically represent SLs. Indeed, like most languages in the world, SLs are exclusively oral languages, that is to say they do not have a writing system (although there are many that tried, in vain, to be accepted by the Deaf; see Bianchini, 2012); moreover, unlike VLs, given the specificities of the visuo-gestural modality, SLs cannot be represented via the adaptation of existing phonographic systems (e.g., the International Phonetic Alphabet [IPA]).

The comparison between the processing phases of VLs and SLs oral corpora highlights the implications that representation problems have on linguistic analyses. Researchers working on corpora of oral VLs record their raw data on audio or audiovisual media; thereafter, they can, through IPA or other adaptations of existing (phonographic) systems, produce a transcription of the meaningful form of the language; if they do not master the language in question or if they want to make the corpus more accessible, they may add a word-by-word and/or a sentence-by-sentence translation; finally, based on the transcription, they produce annotations – i.e., labels, reflecting their theories, assumptions and methodologies – on which their analyses will be based (Table 1).

Notwithstanding the researcher’s best intentions of objectivity, all pre-analysis operations apply filters that attach subjectivity to the data examined by the researcher. Even if “transcription and its notation system [...] incorporate the theoretical presuppositions of the transcriptor on the written modes of oral representation” (Mondada, 2020), it is indeed this operation that allows maintaining the link between raw data and annotations.

Like those working on VLs, SLs researchers record data in video format. However, not being able to transcribe the signifying forms of the signs, given the absence of a graphic system, they replace them with “glosses”, sign-by-word translations in the researcher’s reference VL (Pizzuto and Pietrandrea, 2001). Then, they carry out their analyses by producing annotations which, for lack of being able to rely on the transcription, rely on glosses, and therefore on the signs translation of signs. Researchers interested in the signifying form of signs may add labels annotating some formal characteristics considered relevant (e.g., gaze, hand shape, etc.) (Table 2).

The lack of a graphical representation system makes it necessary to switch from a “transcription” to a “translation” operation, thus losing the connection for linking raw data to annotations. The researcher then bases his analyzes not only on the written form of an oral language, but on the written form of a language which is not only another language – which already creates biases –, but which in addition does not even share the modality of production-reception of the language object of the analysis. So, it is not just a question of a link absence, but of a true rift between raw data and the analyses carried out on those data.

Although many SLs researchers persist in using this practice – once again showing the influence of body denial on SLs analysis –, since the 2000s the widespread use of software like ELAN (Wittenburg et al., 2006) allows the temporal alignment of videos and annotations. Therefore, even if the glossing of signs is still common practice, and glosses are still sometimes used as a basis for analyses, the annotations can be related to the signifying form of the video-recorded signs. However, this solution only narrows the rift, without filling it: indeed, if the analyses are carried out taking into account the gestural form of the signs, this cannot be queried, unless it is represented, again, by annotations describing, through the transcriptionist’s reference LV, the sign shape (e.g.: “flat hand turned upwards”, “flat hand with spread fingers”, “5 hand”, “B hand”, “duckbill hand”; etc.). Doing so makes it difficult to compare similar signifying forms; moreover, it is never the whole shape of the signs that is described, but only a small part considered relevant for the analysis in progress, which amounts, in most cases, to being limited to hand representation.

There is nevertheless a minority of researchers (among whom it is possible to remember Antinoro Pizzuto, Garcia, Crasborn, Prillwitz, and the GestualScript team) who consider that the absence of this line of transcription is a major problem, to be solved through the development of specific instruments to graphically represent SLs. The approaches among these researchers are very different: some try to transcribe SLs using systems invented to write them (or other body practices, such as dance); others think about the theoretic characteristics that a SL graphic system should have to satisfy the various writing and transcription functions without, however, proposing concrete graphic solutions; finally, others get practical and try to build a transcription system to meet their research requirements (see Bianchini, 2012). Whilst these researchers are working on different SLs, finding a solution for a specific SL would open the door to the development of a transcription system for all body expressive forms, i.e. not only every SL but also the human gestures, co-verbal or non-verbal.

Although – as it has been said – solutions do exist, they are only adopted by a small part of SLs linguists and not just only because of a lack of interest in the question: existing systems have often been developed to meet specific descriptive requirements of a research project, and their graphic principles raise many criticalities. For example, these systems have problems with the correspondence between number of characters and information provided (some do not have enough characters to represent all the SL finesse, while others have so many that there are multiple ways to write the same signifying form, hampering the learning process and making it impossible to compare transcriptions); they are either difficult to read for the researcher (preventing a qualitative assessment of the corpus content) or by linguistic analysis software (making quantitative analysis of the corpus impossible); they sometimes seek to be exhaustive without however being easily extendable to unforeseen cases, blocking their opening to other SLs or to human gesture; finally, the use of these systems is extremely time-consuming because of the complexity inherent in them and/or the lack of computer tools to facilitate their use. For more details on the different systems, see Bianchini (2012) and Crasborn (2015).

The sum of body denial and difficulties to graphically represent the signifying forms imposes many biases on SLs linguistic studies. The consequences are noticeable, especially in the study of movement. The absence of a graphic system to represent it, but also of a descriptive model allowing to understand its formal characteristics, means that movement – if even considered – is expounded as a complex and unstable parameter, difficult to analyze and model.

Bypassing the search for reasons preventing the adoption of these graphic systems by linguists, their comparison brings out another common characteristic: although they were created with the aim of representing SLs shape, these systems offer a limited and frozen vision of signs. Moreover, all these systems are organized around the hand: its shape, location, orientation (all grouped under the revealing name of “manual parameters”) are described through their superficial manifestations, offering the vision of a hand “xeroxed” in the graphic space (Boutet et al., 2018). Apart from the hands, any other body element, whether facial expressions or proper body parts (head, bust, arms, forearms, etc.), is described through its posture and not its dynamics (arms and forearms are represented only if they constitute the anchor point of a manual sign). The movement, always described only in terms of “manual parameter”, is represented by the hand trajectory in space, and not as a dynamic process of gestural unfolding: here too, the dynamics of body transformations are not truly reported, being replaced by the superficial and visual effect of movement. The signifying form emerges, but the body and its dynamics are lost!

This trend, as already noted in §1, is not specific to graphic systems: it has permeated SLs research since 1960. Even research on phonology, for which the sign articular component is fundamental, is limited to the hand, reducing movement to a series of successive postures. This is the case, for example, with the “Hold and Movement” descriptive model of Liddell and Johnson (1989; Johnson and Liddell, 2011), where a separation is made between the “hold” phase (static phase common to manual and non-manual parameters) and that of “movement”, which is annotated as “changing” without description (Fig. 2b). For these authors, signs are reduced to “postural gestures [which constitute the] basic building blocks of signs” (p. 417).

Figure 2. 

Analysis of the sign [CHICAGO] (Johnson and Liddell, 2011): only three phases of the sign are (a) retained and (b) described; the ∞ symbol indicates that the parameter, being modified, cannot be analyzed.

Departing from these research practices which entail a hierarchy between parameters - with movement downgraded to a secondary role - and reduce the body to only one of its segments (the hand), Boutet’s KinApp was developed (2005; 2008; and more works; see also the article by Morgenstern and Chevrefils, in this issue).

For Boutet, meaning arises from the body articular capacities, and it is therefore necessary to reposition the body and its dynamics as the core of SLs description: KinApp offers, in addition to an analysis of the perceptual form of signs (the trace drawn in the signing space by the body, seen from the outside), an articular-skeletal and intrinsic description of the upper limbs creating them (the way in which the body is articulated to create this trace, the point of view is then “internal” to the signer’s own segments); this second descriptive scheme is the most innovative aspect of KinApp.

Existing phonological descriptions focus on the hand, as the only segment carrying meaning, and restrict the description of movement to this same segment: the sign [NEVER]² (Fig. 3) in French SL (LSF) is described as “a hand, fist closed and auricular extended, tracing in space a horizontal line which moves away from the signer”. A simple biomechanical analysis of this sign shows, however, that the hand is not an agent of movement: in fact, the wrist joint does not modify any of the Degrees of Freedom (DoF) assigned to it³, the hand is as if welded to the forearm; conversely, there is a rotation of the forearm⁴ at the elbow. The movement in [NEVER] is therefore carried by the forearm, and not by the hand, even if visually it seems that the latter is tracing the sign in space. Using the terminology proposed by Boutet, the hand is in “displacement” (it moves in space without variations of its DoF), while the forearm has a “proper movement” because one of the DoF assigned to it does vary.

Figure 3. 

The sign [NEVER] performed with an outward rotation of the forearm (extract from the corpus DyLIS, 2020).

The analysis in biomechanical terms requires examining, one by one, the possible variations of DoF of each segment on the upper limb; this analysis must adopt a frame of reference intrinsic to each segment⁵. In addition to more accurately describing the physiological progression of a sign, taking into account the entire upper limb helps to highlight the movement “flow” (see Morgenstern and Chevrefils, in this issue). These two strong paradigmatic shifts – taking into account the body as an inseparable whole (i.e., the entire upper limb and not just the hand) that must be analyzed internally (i.e., using an intrinsic frame of reference) – induce a third transformation of the descriptive SLs principles: the geometry of reference.

In Euclidean geometry, the properties of 2D shapes are studied on plane surfaces. Still governing our normal way of conceiving, measuring and quantifying space, this geometry is adopted by most descriptive models of SLs and gestures: using this approach, the extrinsic frame of reference and the focus on hand, the simplest form a signer can produce is a straight line with the hand. For a signer, however, drawing this line requires in reality the setting in motion of a large number of DoF distributed over several segments (Fig. 4a), thus adding up to a great articular complexity.

Figure 4. 

The straight line requires setting in motion 3 degrees of freedom, distributed over 3 segments (a); the curved line requires setting in motion a single degree of freedom on a single segment (b); orange dots represent moving joints.

New geometries, called “non-Euclidean”, were developed since the 19^th century, among which are spherical geometries, where 2D shapes are put on the surface of a sphere. Using this approach and analyzing each segment within a frame of reference intrinsic to the segment, the simplest form a signer can produce is a curve with the segment extremity. Drawing this curve only requires setting in motion a single DoF of the segment (Fig. 4b), thus achieving a great articular simplicity.

KinApp adopts a spherical geometry, allowing a simple geometrical description of what is simple from the articular (but not the visual) point of view, thus realizing a descriptive economy which helps to “decomplexify” the movement analysis: so, movement becomes “simple” to describe and model.

These three paradigmatic changes empowered Boutet to explore in a new way the gestural phenomena, in particular those linked to negation. Although this type of gesture can be carried out in multiple ways, all manners allow to decode the same message: this is because the different realizations contain invariants, i.e. stable and recurring parameters (Boutet, 2015; Morgenstern and Chevrefils, in this issue). KinApp allows to distinguish information carrying meaning from information that may be disregarded, namely modulations not affecting meaning.

A question then arises for SLs (and, by extension, for body language as a whole): within the movement dynamics, is it possible to distinguish formal invariants (which, in Boutet’s hypothesis, constitute the core of meaning creation)? The SL analysis offered by existing systems is too coarse for uncovering the linguistic traits necessary for this exercise. This situation led Boutet to restructure the classic manual parameters (movement, orientation, location), replacing them by the notions of initial location (LOCini) and movement (MOV)⁶. LOCini describes the position taken by the upper limb segments before the sign deployment: it concentrates the information relating to orientation AND location and applies them not only to the hand but also to the forearm and arm. For MOV it is not the name but the very concept of movement that is revolutionized. Movement is usually considered to be a complex parameter (Sandler, 1989; Brentari, 1998; Johnson and Liddell, 2011); in KinApp, in addition to extending movement to all upper limb segments, MOV is described as simply putting into action a body configuration already present in LOCini. The subordination hence assumed renders LOCini as already carrying a semiotic value (Boutet, 2018: 72). LOCini and MOV would then function as an inseparable binomial, the first grabbing – through the installation and holding of articulators – the creation of meaning, realized and completed by the dynamics of the second. It is then possible to reverse the way of collecting data: it is no longer a question of deducing the sign meaning from its movement, but of querying the gestural form to understand how it allows this meaning to emerge.

LOCini and MOV are central for understanding the corporeality that structures SLs, i.e. what is at stake in sign deployment. While their relationship is still a hypothesis, it is important to unveil the links between LOCini and MOV, because their understanding may prove the movement stabilization, a parameter which will thereafter become simple to describe and to model. This search is currently being carried out as part of Chevrefils’ thesis (forthcoming).

The hypotheses concerning the relationship between LOCini and MOV, as well as the methodology for deepening and testing them, are presented in the following section (§II.B).

Through KinApp, Boutet proposes to study movement as a stabilized process governed by economic principles: this statement is based on two hypotheses, one biomechanical and the other cognitive.

The biomechanical hypothesis is based on the “movement flow” notion (see Morgenstern and Chevrefils, in this issue). The MOV unfurling through the upper limb does not happen randomly but following rules of inertia transfer and geometric relationships (Boutet, 2018: 40–49). One of these rules⁷ states that the closer a segment is to its joint stop, the more likely is a partial transfer to the adjacent segment: in other words, the lower the DoF amplitude, the greater the transfer potential, this being the case of the hand, whose DoF are the smallest⁸. In the context of a transformation of a hand DoF, a movement transfer to the forearm is therefore predictable: but the latter having DoF that are much larger⁹, it is rare for them to reach the joint stops, and thus the movement transfer ends in the forearm and does not continue on the arm. If the flow is mainly distal-proximal – and this is the core of the hypothesis – then MOV is likely to fade after a single transfer, directly contributing to the “economy” of the sign signifying form. A preliminary study, covering just a few minutes of corpus in 3 SLs (French LSF, British BSL and Italian LIS), seems to corroborate these considerations (Danet et al., 2017).

The formal sign stabilization is also based on a cognitive hypothesis. This part of the “economy” is linked to the fact that in order to be able to carry out a gesture or a sign, the central nervous system (CNS) must have already set up a precise motor program, i.e. the instructions necessary for its smooth running. Boutet’s hypothesis relates to the nature of these instructions: the observation of signers shows that the realization of a MOV does not require any feedback, readjustment, or special attention, and takes place in one straight whack. The absence of a feedback loop assumes that the motor program readied by the CNS is reduced to its simplest configuration, thus leading to highly economical MOV instructions. Boutet compares this cognitive functioning to a “Mobile” by the sculptor Alexander Calder: “The motor program (Schmidt, 2003) is in some ways already contained in the mobile structure, its setting in motion requiring only a more or less vigorous fillip on one of the plates, without any further intervention during the deployment of the new mobile disposition (i.e., the motor control is restricted just to the initial setting in motion)” (Boutet, 2018: 225 - translated from French).

Although the “motor program” notion comes from cognitivist theories – which consider the control of gestures as a centralized task of information processing (Adams, 1971) – Boutet’s analysis is more akin to dynamic and emergentist approaches of motor control, the main difference between these theories being the CNS role. Cognitivist theories present the CNS as being able to figure out the movement to be made, to plan it in detail, then to send all the commands necessary for its execution. From this perspective, a precise hierarchy is established between CNS, which plans and orders, and muscles, which execute (Fig. 5a). This presupposes that the CNS is capable of storing an enormous number of data, because all these steps must be repeated at the slightest gesture. For Turvey (1990), the CNS does activate by neural impulses the segments participating in movement, but these have an organization of their own: the articulators self-organize (Fig. 5b), and the system is decentralized.

Figure 5. 

Metaphoric images of (a) the cognitivist approach with the CNS represented by the hand which operates the puppet and (b) of the dynamic approach with the puppet functioning by itself in a self-organized form (Turvey, 1990: 939).

Kugler and Turvey (1987) compare the functioning of this self-organization to that of a termite mound: a swarm is made up of hundreds of termites which take part in the construction of the common shelter, the termite mound; the existence of a CNS governing all movements would be tantamount to assuming the existence of a termite capable of designing the complete architectural ensemble of the mound, and then of distributing precise roles to each of its building termites; but such a termite simply does not exist! In order to build their shelters, termites randomly deposit small piles of material; on these deposits they leave pheromones pushing the other termites to deposit their own clusters of matter in the same places, thus forming pillars which join together to form cupolas; to achieve the construction of their complex structures, termites therefore act by themselves, according to certain affordances (i.e., pheromones) (Gibson, 1979); the same goes for human articulators, who organize themselves to interact with the surrounding natural or artifactual environment, or for communication purposes; this coordination is not pre-determined at the start, but “the details are contributed gradually, by many subsystems¹⁰ working together” (Turvey, 1990: 939).

This dynamic theory of motor control is particularly interesting because it emphasizes the relationships among segments or limbs, echoing the body uniqueness of KinApp. Moreover, the self-organization idea is closely linked to that of motor economy, a founding element of Boutet’s thought. Indeed, the two puppets in Fig. 5 present a revealing difference: the number of threads needed for movement is significantly greater in the first (a) than in the second (b). This is because segments self-organize at the local level (between adjacent segments) and also at the global level (between non-adjacent segments, whose dependence is marked by the threads) around a common objective, i.e. to perform a gesture in the most economic¹¹ possible way. The collective sought-after optimization, whether related to the body segments, a puppet or the termite swarm, is based on a principle of economy.

It is therefore relevant to link KinApp to the dynamic theories of motor control to explain MOV produced in SLs and, more generally, in gestures. In order to find answers to Boutet’s cognitive and motor hypotheses (see beginning of §II.B), they are further specified in the context of Chevrefils’ thesis (to be published). These sub-hypotheses, one of which relates to LOCini (Fig. 6a) and the others to MOV (Fig. 6b–d), are summarized in the following diagram.

Figure 6. 

Sub-hypotheses highlighting the relationship between LOCini and MOV.

The green and blue hatched area in Fig. 6 represents the potential links of easing complementarity that the two parameters maintain: LOCini would already carry the semantic information of the sign that MOV only have to update. This link can be formally characterized once the 4 sub-hypotheses have been tested. Indeed, the fact that LOCini follows recurrent patterns (sub-hypothesis a); that MOV concerns only two, or even just one, DoF on a single segment¹² (sub-hypothesis b); and that the predominant flow is distal-proximal (sub-hypothesis c) directly participates in redefining the parameter as an independent unit (§II.A), and simultaneously identifies the nature of this supposedly predictive link. Sub-hypothesis d), which postulates the existence of recurrent kinematic constants that may be recognizable within different realizations of the same sign, makes it possible to differentiate the shape fluctuations from invariants, as Boutet had already started to study (§II.A).

Testing these hypotheses requires a fine and reliable graphic representation of the SLs signifying form: Typannot, to be described in §III, meets this requirement. However, the complexity of the task also requires a means for processing kinematic data in a quantitative and objective manner, without the latent subjectivity that permeates all intellectual enterprises. In order to efficiently explore the LSF structuring and to test KinApp, a hybrid corpus – video and MoCap–¹³ has been collected as part of Chevrefils’ thesis (forthcoming); covering the sentences of 10 signers¹⁴ for a total of 5552 signs; since the data set is double, the procedure must also be doubled.

The first processing, on the video corpus, has been mainly a data segmentation work, with the aim to differentiate one sign from another, but also to distinguish the preparation phase (articulators placement leading to LOCini) from the signifying phase (MOV) of each sign (Fig. 7b.1); thereafter, a Typannot transcription focused on these two parameters was carried out (Fig. 7c.1) on the ELAN annotation software.

Speaking of the MoCap recordings – which provide values of relative positions along the 3 orthogonal axes of each body segment – the exported row data is like a digital time series: since it is impossible to distinguish the relevant information (intra-sign movements) from what is irrelevant (extra-sign movements), it has been necessary to synchronize the recordings and to merge them with the previously established segmentation, in order to recover the areas of interest in the kinematic data (Fig. 7b.2); thereafter, processing these data with the MATLAB software (The MathWorks, Natick Inc., USA) allowed to read them as drawn curves (Fig. 7c.2).

Figure 7. 

Processing steps within the video/MoCap hybrid corpus.

The last step (Fig. 7d), still in progress, consists of bringing together and comparing the two types of results (Typannot transcriptions and MoCap data) which ensue from this double process, in order to find answers to the stated hypotheses and with the goal of developing a semi-automatic transcription interface (§III.A).

The comparisons arising from this “double” corpus will be examined on a selection of signs classified according to their degrees of complexity: e.g., for each sign, the vector of motion of the segments will be broken down into components, determining what is proper motion and what may belong to the transfer motion from another segment; and compare this to the Typannot transcriptions. That each level of phonological description may find its kinematic correspondence is directly related to the scientific framework set up by KinApp: Boutet’s choices were guided by his willingness to participate in the deployment of MoCap technologies, still not widespread in SLs labs.

These thesis activities thus propose to bring to light and clarify the motor coupling of LOCini and MOV through a double analysis: better understanding the functioning of these two parameters – central to all SLs – can lead to typifying the sign shape from the start of its deployment; in this sense, this applied research is a direct filiation of the approach built by Boutet, contributing to its development and dissemination.

The work carried out by Chevrefils (forthcoming) is not the only one cascading from Boutet’s work; Thomas (forthcoming) is also working on a thesis which, focusing on the analysis of head movements and facial expressions, fits into KinApp.

Still, proving KinApp various hypotheses requires a SLs transcription system allowing the description of upper limb segments, essential for these reflections: the need to create a new form of graphic representation dedicated to SLs, as well as gestures, which incorporates KinApp principles, is therefore unavoidable.

The KinApp principles were adopted by the GestualScript research group¹⁵ to develop Typannot (Bianchini et al., 2018; Boutet et al., 2018; Doan et al., 2019; Boutet et al., 2020; Danet et al., 2021), a set of typographic characters, compatible with Unicode and associated with a font family in OpenType format (*.otf), which constitute a system of transcription for SLs and gestures. These characters represent LOCini and MOV, but also HandShape (HS), MouthAction (MAct) and EyeAction (EAct), while other components with similar principles of creation and operation may emerge in the future.

Typannot development is made up of 3 parts, to be exemplified hereafter:

1. Establishment of a “generic characters inventory”: i.e. a list of primitives, non-decomposable formal features, which allow to describe the attributes of the different sign components (HS, LOCini, MOV, etc.). These abstract characters are made visible, in Typannot fonts, through “generic glyphs”; in order to account for a parameter occurrence, the generics are organized in a “generic formula” with rigid syntax, easily usable by researchers and computer queryable.
2. Creation of a “family” of typographic fonts: in fact, 5 fonts allowing to relate the descriptive level of generic characters (see point 1) with the representational level of composed glyphs, which synthesize the intrinsic characteristics of the SLs sign in a familiar form, i.e. a body articulated image. The compound glyphs replace the generics when the text ligature feature – built into the OpenType font format – is activated, and are automatically generated assembling glyph modules designed to increase the glyphs readability.
3. Creation of an input interface: a “virtual keyboard”, specially conceived for Typannot, whose aim is to facilitate and speed up transcription; the interface is also designed to be a system learning tool.

Prior to any explanation of Typannot, it is fundamental to clarify the difference between “typographic character” and “typographic glyph”: a character is an abstract distinctive graphic unit whose informative content is independent of its form; in order to be recognized by software, characters must be linked to a code assigned by the Unicode Consortium; a glyph is the concrete graphic realization of a character or a comb of characters (in the case of typographic ligatures) in a defined font. By pressing the SHIFT+A keys on a keyboard, the command “writes the Unicode character 0041” is sent to the word processor which then displays A, A or A, depending on the chosen font (here: Arial, Impact and Curlz): these 3 different A’s all correspond to the same typographic character (A), but all have different shapes because the displayed glyph is different.

In order to briefly and concretely present how Typannot works, the LOCini component is described below.

LOCini inventory consists of 18 generic characters which, to be displayed on screen, need to be associated with generic glyphs (Fig. 8). The list of selected characters comes from KinApp and is subdivided into lines identifying the upper limb concerned (), the segments to be described (), the possible DoF () and the rotation angles of the different DoF, expressed as notches ().

To describe a particular LOCini, the generics are organized in a rigid syntax formula (Fig. 9), which is based on the principle of grouping the various possible values (notches) into variables (DoF) and associating them with each part (segment) of a parameter (right or left segment). This procedure also applies to HS, MOV, MAct and EAct.

Figure 8. 

List of LOCini generics (associated with their generic glyphs) subdivided into 5 types of information: font, parameter, parts, variables and values.

Figure 9. 

Generic formula of LOCini of the upper right limb for the sign [NEVER] in LSF; colors distinguish font, parameter, parts, variables and values.

From just 18 characters, it is possible to encode more than 4 million distinct LOCini, i.e. as many as the possible generic combinations; moreover, the generics inventory is made of characters specific to LOCini, but other traits are common to several sign components, e.g. distinguishes left and right sides in LOCini, in HS, in EAct. This transversality among characters greatly limits the number of generics necessary for the description of all the SLs components: this opens the possibility for recognition of Typannot by the Unicode Consortium, the international organization which ensures the compatibility of a graphics system with all software, operating systems and browsers¹⁶. Indeed, although it would not be possible to ask Unicode to register several million of characters¹⁷, it is possible to ask it to list a hundred, and that is all Typannot needs to encode manual and non-manual parameters of all SLs and gestures.

The transcription carried out with Typannot results in a description queryable on several levels, e.g. searching for a whole LOCini, but also for all inflections regardless of the segment concerned and/or the flexion amplitude, or looking for co-occurrence of hand flexion and arm adduction, and so on. In the KinApp context, this allows to search for gestural invariants, stable and recurring values within signs (and co-verbal gestures), with the aim of finding what is in common among different realizations of the same sign and of identifying the origin of meaning creation.

The generics arrangement in the formula guarantees information consistency, completeness and queryability. The typographical approach to the transcription problems, via the separation between information imbedded in the character and its glyphic manifestation, as well as the recognition of these characters by the Unicode Consortium, not only ensures that Typannot is compatible with all current software and OS, but that it will remains compatible as new writing technologies are deployed. However, although deciphering these formulas is quite simple, this part of Typannot does not guarantee a good readability for the human researcher.

The second part of Typannot consists of the deployment of a family of typographic fonts including on the one hand all generic glyphs, and on the other a series of compound glyphs, which are visual, synthetic representations of all information encoded by generics in a formula (Fig. 10). The principle beyond these glyphs is a complex typographic ligature, possible by the OpenType font format: just as it is possible to produce the glyph [œ] by composing one after the other the characters “o” and “e” or to show [] by composing “” and “”, the same is true to make appear the glyph [] by composing the formula “”. This procedure allows a multitude of glyphs to appear from a very limited number of characters; moreover, it ensures the queryability of the information contained in the formula while providing a more readable version of this information. Indeed, as it is possible to find [œ] by searching for “o”, it is possible to find [] by searching for “” because the information is contained in the abstract characters and not in their visible, actual glyphic visualization.

Figure 10. 

Association between the generic formula of LOCini of the upper right limb for the sign [NEVER] in LSF and its corresponding compound glyph.

Creating the compound glyphs is not just a matter of drawing bodies and hands, it is a real design research process (to identify glyphic modules that can be combined in order to obtain a distinct graphic representation for each formula, and to determine the way to combine them) and in linguistics (to identify which combinations to represent¹⁸): since it is impossible to describe here this part of the research, see article by Danet et al. (2021) for more details.

As readable, coherent, exhaustive and searchable as it may be, a good transcription system must be easy to write: indeed, a poorly scriptable system disproportionately increases the time required for transcription. Therefore, Typannot approach was to create a computer instrument that could facilitate the generics insertion: its virtual keyboard (TypannotKB) not only allows the generics selection and their arrangement in the formula without syntax errors, but also checking the transcription in real time, by displaying the compound glyph and an avatar showing the pictured form.

Thus, for LOCini, TypannotKB allows, through a series of cursors, to assign a notch to each segment DoF; the glyph dynamic display visualizes in real time the effects of a DoF variation on a particular segment (Fig. 11), thus allowing the transcriptor to verify¹⁹ the transcription accuracy. Once validated, the LOCini description may be exported to any Unicode-supporting software: generic and compound glyphs can then be viewed by installing the Typannot font family.

Figure 11. 

TypannotKB: (a) the sign [NEVER] in LSF written with TypannotKB; (b) by modifying the FlxExt of the forearm by one notch, there is an automatic change in both the formula and the compound glyph.

The presence of an immediate feedback on the values manipulation is, in addition to a means of transcription verification, a pedagogic instrument for Typannot learning. Indeed, the system can be managed forthwith by manipulating the virtual keyboard: the avatar visualization of the various attempts details the exact informational content of each generic; their display as a compound glyph allows learning to read Typannot; their display as generic formula frames makes it possible to memorize the syntax but also the generic glyphs.

In order to further improve Typannot scriptability, the virtual keyboard shall be equipped with a second interface designed to automatically convert data from MoCap into Typannot generic characters (and therefore into compound glyphs too). This innovation, still at an early stage, will drastically reduce the SLs transcription time, from the current 5 hours per minute transcribed (Crasborn, 2015) to almost real-time. It would indeed be possible, by using the body (equipped with MoCap devices) as a paintbrush (Boutet et al., 2018) to simultaneously transcribe information on HS, LOCini, MOV, MAct and EAct. This research phase does not only consist in finding algorithms transforming MoCap data into Typannot characters, but also in identifying a MoCap system combining device portability, non-invasiveness for the “captured” person, ease of handling by the researcher and limited cost.

Typannot wants to provide a body description and representation, which allows detailed analyses of SLs and gestures regardless of the researcher’s theoretical framework; still, Typannot is not born “out of theory”, but is the result of representation necessities essential for the kinesiological reflection of Boutet, founding member and co-coordinator of the GestualScript team till March 2020. It is therefore in the KinApp framework that Typannot MOV is issued, the famous “last but not least” piece of this ten-year project, on which the GestualScript team has been working since January 2021.

The GestualScript current step is to establish the inventory of the generic characters, starting from an in-depth analysis of Boutet’s work to extract the different elements he had identified to describe MOV. The balance between Typannot fonts usable by everybody and its insertion in KinApp requires substantial efforts to fit Boutet’s approach, in particular to make its descriptive levels accessible without forcing the transcriptor to acquire peculiar skills in the study of SLs, gestures and human physiology, which were at the origin of KinApp. It is therefore necessary to detect generics that make it possible to describe MOV from a skeletal and articular point of view (by describing the DoF variations of the different segments, by distinguishing the proper movements from simple displacements, by identifying the gestural flows, etc.) as well as from its visual perception (the destination of pointings, the shapes of the traces left by the movement, etc.), but also to find a way to transmit the informational content of the different generics to the transcriptor.

The idea behind this development step is to use TypannotKB as an instrument for its simplification: the generics remain complex and linked to KinApp, but the keyboard permits to inspect MOV by returning simple questions, which then leads to the automatic selection of generics.

Preliminary results indicate that KinApp requires, among other things, to determine both the motor (i.e., the movement creation) and the semiotic contributions (i.e., the meaning creation) of each segment; following Boutet (2018), these elements are identified by the “portée” (scope) and the “type d’emprise” (type of involvement) criteria²⁰. TypannotKB allows the transcriptor to identify, without hesitation, both scope and type of involvement²¹ via 4 questions, two relating to motor skills (does the segment draw a trace in space? does it include variation in its DoF?) and two concerning the meaning creation (is the segment involved in meaning creation? if so, is it active or passive?). The reasoning also shows that these 4 questions, together with a further question on which segment(s) appeared to trigger the MOV progress, make it possible to automatically determine the flow. The relevance of these 5 questions, as well as their exact formulation, is currently being tested as part of Chevrefils’ thesis (forthcoming) (§II.B), which uses them to annotate scope, type of involvement and flow in DyLIS corpus.

Once the list of generics and the list of questions allowing TypannotKB to guide the transcriptor have been completed, it will be necessary to determine how all this information can be combined in a generic formula with rigid syntax. Further work will then start for creating the MOV font (with the challenges of drawing composite glyphs) and the computer implementation of TypannotKB (with its questions, its composite glyphs, its avatar and its parametric and gestural interface).

The use of Typannot will allow the researcher to achieve a very fine transcription of the movement parameter, but also to enter smoothly into the descriptive model proposed by KinApp through, among other things, the use of technology.