Professional Documents
Culture Documents
In this study, we verify the observation that signs for emotion related concepts
are articulated with the congruent facial movements in German Sign Language
using a corpus. We propose an account for the function of these facial movements in the language that also explains the function of mouthings and other
facial movements at the lexical level. Our data, taken from 20 signers in three
different conditions, show that for the disgust related signs, a disgust related
facial movement with temporal scope only over the individual sign occurred
in most cases. These movements often occurred in addition to disgust related
facial movements that had temporal scope over the entire clause. Using the Facial
Action Coding System, we found some variability in how exactly the facial movement was instantiated, but most commonly, it consisted of tongue protrusion
and an open mouth. We propose that these lexically related facial movements be
regarded as an additional layer of communication with both phonological and
morphological properties, and we extend this proposal to mouthings as well. The
relationship between this layer and manual lexical items is analogous in some
ways to the gesture-word relationship, and the intonation-word relationship.
Keywords: facial expressions, non-manual features, phonology, mouthings
1.
It has been reported for ASL that manual signs for some emotion related concepts temporally co-occur with facial movements that have the congruent meaning (Liddell 1980; McIntire & Reilly 1988; Reilly, McIntire & Bellugi 1990). For
example, the sign sad in ASL is reported to consist both of a manual component
and a temporally co-occurring frown. It is this phenomenon, that is, the frown
part of sad, that we attempt to explain in our study.
Sign Language & Linguistics 17:2 (2014), 123180. doi 10.1075/sll.17.2.01ell
issn 13879316 / e-issn 1569996x John Benjamins Publishing Company
The function of facial movements such as the frown in sad, whether phonological, morphological, or something else, is as yet unclear. The indeterminate nature of these facial movements is reflected in the terminological choices
in the literature. The above cited authors do not use either the term phonological or morphological for these facial movements. Rather, they refer to them with
some form of the term lexically related. For German Sign Language (Deutsche
Gebrdensprache: DGS), they are described under the header phonology in
Happ & Vorkper (2006).
Emotion concept signs often appear to have a lexically related facial movement, although they are not the only semantic category of signs to have one; an
example of a non-emotion related sign in ASL with a facial component is not-yet
(Liddell 1980: 17). In this study, we only focus on facial movements that are lexically related to signs for emotion concepts. We focus on this phenomenon for the
following reason: There is a long tradition of research on facial movements associated with emotions going back at least as far as Darwin (1904 [1872]). However,
despite over a century of scientific investigation on the subject, why certain facial
movements are often associated with emotions is still a controversial issue (Russell
& Fernandez-Dols 1997; Ekman 2004; Wierzbicka 1999; Izard 2010). Some of the
main controversies surround the questions of whether facial movements are indices of emotions or rather of intentions, of identifying what is cultural and what
is innate, and of what the scientific definition of emotion should be. Comparing
emotion related facial movements with their lexically related facial movement
counterparts in sign languages may contribute to clarifying what exactly is unique
to each in their form, function, and meaning.
1.1 The goal of our investigation
The goal of our investigation is to understand what is the function of facial movements that (a) temporally co-occur with emotion related signs and (b) are semantically congruent with the manual component (e.g. sad + frown). Are they phonemes with some morphological properties as we propose below?
To achieve this goal using a corpus of DGS disgust signs, we ask the following
three empirical questions:
1. Does an emotion related facial movement consistently temporally co-occur
with disgust signs produced as single signs?
2. Does an emotion related facial movement consistently temporally co-occur
with disgust signs produced in direct and reported sentence types?
3. Which facial muscles are used by each signer when producing an emotion
related facial movement temporally co-occurring with a disgust sign?
intends something. For example, a face in which ones teeth are bared does not
mean I am angry it means I intend to aggress. For a short review on the topic
see Elliott & Jacobs (2013).
Our approach to the meaning of facial movements, based on many of the results
of Wierzbickas (1999) semantic analysis of facial movements, is as follows: whichever framework is adopted, we take it as uncontroversial to state that for humans,
facial movements serve as semiotic units (form-meaning pairings) that are used to
communicate information about oneself (whether ones intentions or emotions) in
the present. This communication system about ones present emotions/intentions
appears to have some innate facial movement components and develops before
other modes of communication such as gesture and speech/sign. It is commonly
observed to be used both in the presence and absence of gestures and speech/sign.
It appears implicit both to the facial expression program and the behavioral
ecology view that facial semiotic units are in the first person present tense orientation. For example, the bared teeth face could mean I am angry or I intend to
aggress, but what is common to both expressions is the template I X now,
where X is an emotion or intention. Wierzbicka (1999: 175) makes this semantic
feature explicit in her analysis of the meaning of facial movements and it is easily
verifiable by observation. When we smile or frown at each other, or when we hear
an unexpected loud noise and rapidly raise our brows and widen our eyes, it never
signifies you are happy/sad/afraid or I will be happy/sad/afraid. Wierzbicka notes
that in this semantic feature, facial movements are akin to exclamations, interjections, and performative verbs. This is also true when they are used as enactments.
For example, if I am telling someone about a past event, and I say She went [nose
wrinkle] when she saw the food even though I am referring to someone elses past
reaction to food through an enactment, the enactment itself is to be interpreted in
present tense first person orientation in which I represent the character she and
the movements I make in character are to be interpreted as if in the present. The
facial movements used in speaker attitude depiction and role-shift in ASL also appear to have a first person present tense orientation (Liddell 1980: 5358).
1.3 Facial movements in sign languages
Research into sign languages has shown that facial movements are not only used
for communication of emotion/intention in the first person present tense orientation they can also become part of signing. Facial and head movements are
used in sign languages at all levels of structure. At the lexical level, some signs
have a facial component in their citation form (Liddell 1980; Woll 2001). There
are facial morphemes (Lewin & Schembri 2011; Crasborn et al. 2008; McIntire
& Reilly 1988; Liddell 1980) such as, for instance, the ASL adverbial th meaning
Information type
Emotion/
intention
No duality of patterning
Non-combinatoric
Possibly innate
Gesture
No duality of patterning
Non-combinatoric
Idiosyncratic
Intonation
Sentence type,
No duality of patspeech act, topic and terning
focus
Combinatoric
Conventionalized
Timed to onsets
Vocal folds
and offsets of
Face
lexical items in prosodic constituents
Words
Culture-specific
semantic categories
Duality of Pattering
Combinatoric
Conventionalized
Articulators
Hands
Face
Hands
Vocal tract
detail the four information types, what is known about their properties, and how
tightly they seem to be connected to lexical items.
Emotion/intention signals appear early in acquisition, before speech and
gesture (Izard et al. 1995). Besides the face, they can be encoded in the voice
(Pittam & Scherer 1993) and the hands (Reilly, McIntire & Seago 1992). There is
cross-cultural evidence (Ekman 1972) and evidence from the congenitally blind
(Matsumoto & Willingham 2009) suggesting that some of the facial movements
for this information type are innate and universal. These types of units do not seem
to temporally align with lexical items as seen in our data below (see Figure 13) and
in Baker-Shenk (1983), for example.
We use the term gesture in McNeills sense as referring to hand movements
that accompany speech and are idiosyncratic and spontaneous (McNeill 1992: 37).
According to McNeill (1992), co-speech gestures encode visual-imagistic information such as direction, path, manner, size, and shape. They do not display duality of patterning. (The term duality of patterning (Hockett 1960) is used to refer
to the combined properties of the meaninglessness of form units, and their independent patterning. Some forms, such as e.g. the smile, cannot be broken down
into smaller units and are always associated with the same meaning.) They are
non-combinatoric. They are idiosyncratic in the sense that there are no standards
of well formedness for a particular gesture. Gestures are tightly synchronized with
speech. There is usually one gesture per clause. The preparation phase (hands
moving into position) of a gesture anticipates speech, and then synchronizes with
it during the stroke (the execution of the gesture). The stroke is timed to end at or
before, but not after, the peak syllable (McNeill 1992: 42, 85). Speech-gesture synchrony is not disrupted by stuttering (Mayberry et al. 1998) or delayed auditory
feedback (McNeill 1992: 273294).
Intonational units following Ladds (1996) definition encode post-lexical information, namely information about sentence type, speech act, and information
structure such as topic and focus. They can be encoded by facial movements or
vocal fold movements (Dachkovsky & Sandler 2009; Krahmer & Swerts 2009;
Crasborn & van der Kooij 2013). The properties of intonational units, according
to a componential model (e.g. Bartels 1999; Dachkovsky & Sandler 2009), are as
follows: there exists a finite set of intonational primitives, and these primitives
encode a meaning. The primitives of intonation, like gestures, do not display duality of patterning, but unlike gestures they are combinatoric. Intonational units
appear to be timed to word onsets and offsets and their scope to be determined
by reference to the relevant level in the prosodic hierarchy. Furthermore, they are
conventionalized (Dachkovsky & Sandler 2009).
The words of a language capture the semantic categories created by a particular culture. These categories can vary greatly between cultures, even for semantic
domains that are thought to be based on universal human experiences such as
color vision or feelings (Wierzbicka 1996, 1999). Words generally display duality
of patterning, although this may not be a strictly universal property of words as
previously assumed (de Boer, Sandler & Kirby 2012; Aryani et al. 2013). Words
are combinatoric; that is, they combine with each other in hierarchical patterns to
form more complex semiotic structures.
In what layer of information do lexically related facial movements, particularly
facial movements that are lexically related to signs for emotion concepts, belong?
Do they encode information about emotions/intentions in the first person present
tense orientation? What are their combinatoric properties? Are they idiosyncratic
like gestures, or more conventionalized like words and intonational units? In the
following section, we discuss the findings that are relevant to these questions.
3. Lexically related facial movements phonemes or morphemes?
In this section, we present extant findings on the properties of facial movements
that are lexically related to signs for emotion concepts and argue that so far the evidence suggests that they are phonological elements mental representations of
form units whose primary function is to provide perceptually salient cues for identification of the sign with some morphological (meaning bearing) properties.
More precisely, the movements themselves are of course the phonetic instantiations of phonological elements (mental representations). In our arguments below,
we attempt to make clear how a facial movement can have both a phonological and
morphological function and provide examples of other elements both in signed
and spoken languages that also display such dual functional properties.
3.1 Properties of facial movements lexically related to signs for emotion
concepts
What is known about the properties of facial movements that are lexically related
to signs for emotion concepts? It is reported that they are related to manual lexical
signs in the following way: (a) they consistently co-occur with particular manual
signs (Liddell 1980; Reilly, McIntire & Bellugi 1990; McIntire & Reilly 1988); (b)
some signs are considered ill-formed without the facial movement (Sandler &
Lillo-Martin 2006: 61); (c) some facial movements can act as minimally distinctive features (Sandler & Lillo-Martin 2006: 61). They differ from facial adverbials
by the fact that they are not systematically used to modify the meaning of entire
classes of signs. They differ from depictions of speaker attitude and from role shift
in their temporal scope and in that they are produced in the presence of a manual
component with a congruent meaning (Liddell 1980).
Property (a), consistency of co-occurrence, might suggest that these facial
movements are phonological elements. However, if the consistency of co-occurrence is less than 100%, such movements might rather be optional reinforcers
(Liddell 1980: 16). Property (b), ill-formedness, can be regarded as a stronger form
of consistency of co-occurrence in which the consistency is 100% or very close
to 100% when allowing for production errors. If property (b) can be verified for
particular manual-facial pairings, we think this is a reason to regard that facial
movement as a phonological element. Similarly, the finding of minimal pairs (c),
the classic diagnostic for phonological features, is also a reason to regard a facial
movement as a phonological element.
Is there a reason to consider a facial movement that has been established as
necessary for the well-formedness of a sign as not being a phonological element,
i.e. part of the stored mental representation of the form of the sign? We think not,
but if one defines phonological elements as discrete meaningless units, as is generally held (e.g. Sandler & Lillo-Martin 2006), than facial movements that have a
meaning, such as smiles and frowns, cannot be considered phonemes.
We think there are good reasons not to make the theoretical assumption
that phonemes are necessarily meaningless. Rather, we regard phonemes as units
whose primary function is to create perceptual distinctions but which may sometimes also be associated with a meaning. As we already mentioned above, studies
on both signed and spoken languages (de Boer, Sandler & Kirby 2012; Aryani et
al. 2013) suggest that the smallest units of a language are not always meaningless.
The manual parameters of handshape, location, and movement, regarded as the
discrete meaningless units of sign language phonology, behave like independently
meaningful morphemes in classifier constructions (Sandler 2009: 261), and the
distribution of certain handshapes within the lexicon has been shown to be better
explained by their meaning (i.e. their morphological potential) than their form
(Fuks & Tobin 2008). Evidence cited in Blevins (2012) indicates that for spoken
languages, too, there are cases in the lexicon in which duality of patterning is not
found that is, there exist morphemes that cannot be further broken down into
meaningless units. One example from Blevins (2012) is the phonological feature
of palatalization which consistently means uncontrolledness/child-likeness in
Japanese mimetic constructions.
Corpus data will help in establishing consistency of co-occurrence, establishing environments in which particular facial movements appear, establishing
degree of conventionalization of facial movements, and establishing whether all
emotion-concept related signs are produced with facial movements.
3.2 Crasborn et al.s (2008) typology of mouth movements
In this section, we describe Crasborn et al.s (2008) corpus based findings on mouth
actions. Crasborn et al. (2008) created a typology of mouth actions based on data
signed by six people from three different European sign languages (British, Dutch,
and Swedish). They found that between 5080% of the manual signs across the three
languages had a facial movement component. They divided these facial movements
into five categories: mouthings, adverbials, whole face actions, enacting mouth actions, and semantically empty mouth actions. Mouthings are the lip movements
used in sign languages that have been adopted from the articulation of words from
ambient spoken languages. Adverbials are facial movements that are used to modify temporally co-occurring manually articulated verbs. Whole face actions are facial movements that use both upper and lower face muscles and would include
some movements that are lexically related to signs for emotion concepts. Enacting
mouth actions include, for example, depictions of kissing or chewing. Semantically
empty types are mouth movements that are neither enactments nor mouthings and
that do not seem to have any semantic content. The frequency ranking of each type
taken from Crasborn et al. (2008: 5152) is given in Table 2 below:
131
Adverbials
Whole face
Enacting + Empty
combined
3957%
1430%
1620%
814%
The frequency scores above demonstrate that mouth and whole face movements
temporally co-occurring with lexical signs are ubiquitous, suggesting they have an
important function but what is this function?
3.3 Functions of facial movements that temporally co-occur with manual
lexical items
Before discussing functions, a word on the notation conventions we use for mouthings is necessary. Mouthings are written using the visemic transcription proposed
in Elliott et al. (2012) and represented in Table 3 below. It is similar to Kellers
(2001) kinematic description of mouthings. This notation is fairly transparent as
upper case Roman letters, familiar to any English reader, are used to represent
classes of phonemes that share visual appearance. For example the viseme /p/, described as lips compressed, maps onto the phonemes /p, b, m/. Thus, readers can
identify the viseme intended in the transcription, with minimal reference to the
table, by attempting to articulate the letter. We transcribe the mouthing component to the right of the sign gloss with a + symbol, e.g. for the DGS sign bruder
(brother) the gloss is bruder + p-ut-.
Table 3. Visemic transcription of mouthings
Phoneme
(CELEX notation)
Viseme
Description
bmp
Lips compressed
dnstz
@Nghkrx
fv
Iij
Ee
&OQo
Rounding of lips
UYuy
Pouting of lips
mouth movement in their data, the authors of the cited study did not consider
them further as their research questions focused on the mouth only.
For the purpose of our study, the most important finding from Crasborn et al.
(2008) is that not only mouthings have a mixed morphological/phonological profile. Enacting movements behave this way, too; that is, in some cases, they appear
to be part of the phonological form of the word, e.g. chew + chew_mouth above.
In other cases, they function as morphemes by adding new semantic content, e.g.
run + shouting_mouth above.
3.4 Lexically related facial movements as a class
Based on the similarities in behavior of mouthings and enacting actions (both
mouth-only and whole-face movements), we propose that these two facial actions
are one class of phenomenon: elements that have the function of adding phonological and semantic information to lexical items. Furthermore, we propose that
empties and adverbials also belong to this class of elements, but they are extreme
points on a phonological-morphological continuum within this class. The elements in this class vary in (a) the degree in which they function as phonological
elements (adding perceptual information) or morphological elements (adding semantic information) and (b) their origin (ambient spoken language movements,
instrumental and emotion/intention facial movements, echoes of manual movements) as presented in Table 4 below.
Table 4. Properties of facial movements related to single lexical signs
Face Movement
Function
Origin
Empties
Phonological
Mouthings
Phonological/Morphological
Spoken languages
Enactings
Phonological/Morphological
Adverbials
Morphological
Information
Transmitted
Enactings
Culture-specific
semantic categories of
emotion and action
and phonological
information
No duality of patterning
Non-combinatoric
Conventionalization
unknown
Mouthings
Culture-specific
semantic categories of
any kind and phonological information
Duality of patterning
Non-combinatoric
Conventionalized
Empties
Adverbials
Culture-specific
semantic categories of
manner
No duality of patterning
Combinatoric
Conventionalized
Articulators
We wish to make clear why enactings convey culture-specific semantic categories rather than emotions/intentions. By culture-specific semantic categories we
mean that lexicons reflect the conceptual categories created by a particular speech
community. Even if some emotions or instrumental actions (such as chewing and
biting) are universal, not all cultures name these acts or experiences. For example,
the Dani people studied by Ekman do not have words for what Ekman proposes
are the six basic universal emotions: happiness, sadness, anger, disgust, fear, and
surprise (Ekman 1975: 39, cited in Wierzbicka 1999: 25). Therefore, using a frown
to refer to a concept of sadness, rather than to the current state of the one making
the frown, is a culture-specific act of semantic categorization.
135
Furthermore, there is some evidence that emotion related enactings have undergone some semantic bleaching and lost their first person present tense orientation, unlike role-shift elements and attitude depicting enactments. Emotion related enactings occur in the citation forms of the lexical items in question (i.e. even
when not in a context in which there could be a first-person subject to predicate
sadness over in the present). Therefore, the frown component of sad would appear
to simply mean sad as opposed to I am sad. Additionally, three DGS consultants confirmed that emotion signs such as traurig (sad) and ekel (disgust) are
made with semantically congruent facial movements even in negated sentences
such as I am not sad. Enactments of instrumental movements similarly remain
present even when the sign is negated as per Liddell (1980: 17) for the ASL sign
bite. Additionally, enactings have a different temporal relationship to lexical items
than emotion/intention signals.
5. Methods
5.1 Defining an emotion related sign
We needed to define emotion related sign for the purposes of our study. As there
is currently no consensus on the scientific definition of emotion, we decided to
take into account two dominant types of emotion theory: basic emotions theories such as Ekman (1972, 1992, 2004) and dimensional theories such as Russell
(1980). Using the basic emotions approach, we defined emotion related signs as
signs corresponding to Ekmans basic emotions happiness, sadness, anger, disgust, fear, and surprise. Using the dimensional approach, we defined an emotion
related sign as one with a particular rating on the dimensions valence and arousal
as done e.g. in Hofmann et al. (2009) and Vo et al. (2009). In our complete corpus, we used both definitions, but in this paper, we only present the disgust items
from the basic emotions data. We then created our emotion signs corpus following
many suggestions in Johnston & Schembri (2006).
5.2 Collecting emotion related signs
In order to see whether speakers of DGS make semantically corresponding facial movements with emotion related signs, we use both a translation task (from
German to DGS) and a free speech task. A translation task is practical but artificial. A free speech task is natural, but less practical because we cannot guarantee
that DGS speakers will produce a large number of emotion related signs in their
free speech. In this paper, we only present the translation task data associated with
5.3 Participants
There were 20 participants, (nine male, average age 26: Max = 39, Min = 23). 15
reported that DGS is their first language. Eight acquired DGS from birth, seven
were early acquirers (between ages 15), four acquired DGS after the age of five
(Max = 21). Data on age of acquisition is missing for one participant. Nine participants had at least one deaf parent. They were recruited by two Deaf native DGS
signers through personal contacts and advertisements. They received monetary
compensation for their time.
Description
Tokens
Handshape
Location
Movement
5-handshape
moves outwards
11
5-handshape
bi-manual
local movement
11
5-handshape
10
Region: Berlin
Subjective Frequency: 4
Meaning: Used mainly to describe dislike of a food, or to say that a person does not
look good.
yuck
I-handshape
sideways movement
away from body
5-handshape
B-handshape
palm inwards
contacts chin
repeated outward
movement
5-handshape
bi-manual
at stomach level
upward movement
Table 6. (continued)
Sign Type Label Description
vomit
Tokens
Handshape
Location
Movement
5-Handshape
bi-manual palm
upwards
mouth area
outward movement
Region: Hamburg
Subjective Frequency: Meaning: Often transcribed as erbrechen (vomit), but it does not mean to physically
vomit. There is another DGS sign for physical vomiting.
touch
F-handshape
local movement
Region: Saxony
Subjective Frequency: 2
Meaning: It describes the sensation of disgust one would have touching something
unpleasant.
neck
inward movement
downward movement
outward movement
Region: Cologne
Subjective Frequency: 2
Meaning: It is used to say of a person or a thing that they look bad. Made with a pff
mouth action.
Total:
63
eliciting conditions (direct speech, reported speech, single word). Figure 1 below,
shows that most sign types (7/12) appeared at least once in each condition.
The five signs that only appear in one or two of the conditions were yuck,
polite, neck, slang, and pff. Since yuck is a high frequency sign in our corpus
(fourth most frequent sign; 11% of the tokens), its complete absence from the
single sign condition appears meaningful. yuck seems to be an interjection, like
the German iii it is associated with. This may be the reason why participants did
Direct speech
Reportive speech
Single word
3
2
1
ff
.p
12
t
yu
os
ck
ebu
m
ps
6.
po
lit
e
7.
na
us
ea
8.
vo
m
it
9.
to
uc
h
10
.n
ec
11 k
.sl
an
g
5.
go
4.
oa
t
us
th
r
3.
di
sg
2.
1.
ha
te
not produce it when translating the single German word Ekel. However, in a sentence the interjection appears to be able to serve as a verb or predicative adjective.
The signs polite (frequency = 6%) occurred both in the single word condition
and in a clause. slang (frequency = 3%) and pff (frequency = 1%) only occurred
in clauses. neck (frequency = 3%) only appeared in the single word condition, and
never in a clause.
We also examined the distribution of the 20 participants per sign type in order
to see if a particular sign was uniquely associated with a particular signer. Figure 2
below shows that 75% of the sign types were produced by more than one signer. The three signs that were uniquely associated with one participant vomit,
9
8
7
6
5
Number of Participants
4
3
2
1
2.
ff
.p
12
1.
ha
te
di
sg
us
t
3.
th
ro
at
5.
go 4.y
uc
os
k
ebu
m
ps
6.
po
lit
e
7.
na
us
ea
8.
vo
m
it
9.
to
uc
h
10
.n
ec
11 k
.sl
an
g
10
8
6
Number of Participants
4
2
0
1 sign type 2 sign types 3 sign types 4 sign types
touch, and pff were those associated with a particular region in Germany. As
shown in Table 6 above, vomit is a Hamburg variant, and the participant using
vomit is originally from Hamburg. Similarly, touch was signed by a participant
originally from Saxony, and pff by a participant originally from Cologne. The
other sign that was reported to be region-specific (throat) was signed by five different participants; however, this is not surprising since the region it is reported to
be specific to is Berlin, which is where the interviews took place.
To check to what extent individual signers were consistent in their sign type
choice across conditions, we looked at the frequency distribution of number of
sign types used per participant. Figure 3 below shows that 70% of the participants
used two or more sign types across the conditions.
The frequency distribution of our sign type data in the three eliciting conditions and across participants suggests that DGS has at least seven disgust related
concepts used across Germany and five that appear to be region-specific or slang.
The sign yuck only appears in a sentence context (i.e. in our first two conditions).
Based on our preliminary semantic analysis, the various signs appear to distinguish between whether a sense of disgust is due more to seeing, touching or tasting, and also whether the referent is human or non-human. Note that empirically,
all the sign types (except perhaps neck) are compatible with the meaning of a
disgust sensation towards a non-human referent as they all (again except neck)
appeared at least once in a clause in which the object of disgust is a worm.
The iconic structure of the signs also suggests differential emphases on seeing,
touching and tasting. Iconicity is a form-meaning resemblance. For example, the
ASL sign tree visually resembles an image of a prototypical tree. Elements of the
phonological form are mapped onto elements of the schematic image of the tree in
an analogue building process (Taub 2001): branches are mapped onto the hand;
trunk is mapped onto the arm; ground is mapped onto the non-dominant arm.
Less concrete concept signs, such as religion, can also resemble their meaning
through a metonymic or metaphorical relationship between the iconically represented image and the meaning. The DGS sign religion is made with an alternating movement of tip of the middle finger of one hand contacting the middle of
the palm of the other hand. This is an iconic metonym of the concept religion as it
associates Christs crucifixion wounds with the general concept of religion.
The signs hate, throat, polite, nausea, and vomit all select a particular
image of vomiting as an iconic metaphor for the concept disgust. Features of the
schematic image of vomiting are mapped onto phonological features such that
location of bodily sensation is mapped onto the location features stomach, throat
or mouth. The feature of vomited matter is mapped onto the handshape, and the
trajectory of the vomit is mapped onto the movement feature. The sign touch
is made with the f-handshape which resembles an index finger-thumb grip one
might use when picking up something that one would rather not touch. For the
sign disgust, the 5-handshape placed at approximately face level creates a sight
barrier which is an iconic depiction of dont want to see.
Figure 4 below shows the sign types from left to right: hate, disgust, throat,
yuck, goose-bumps, nausea, vomit, neck, pff. Not all sign types are represented due to lack of permissions from participants.
meaning congruent to disgust, such as a smile, need to be excluded. Our data set
is available in Appendix 1.
The action units started at onset or in 4/62 cases at the stroke phase of disgust
signs and faded with the retraction. It was clear that the face and the mouth in
particular, were engaged in rapid movements timed to the segmental actions of the
hands throughout the sentences and not just on disgust signs. Average mouthing
rate over all three conditions, calculated as number of signs containing a mouthing
divided by total signs, was 0.41 (SD 0.17). That means that for almost every second
sign, the mouth was engaged in our data set. This mouthing rate for DGS is similar
to that reported in Ebbinghaus & Hemann (2001).
6.5 Mouthings
In four cases, the facial action that occurred with disgust signs was a mouthing
or mouth gesture, coded as AU50, with no other detectable movements. In nine
cases, there were both mouthings and additional facial actions. There were two
different mouthings that occurred: /sle-t/ and /e-el/ derived from the German
words schlecht (nausea) and Ekel (disgust), respectively. There was one case of
the DGS mouth gesture pff . Table 7 below gives the mouthed and mouth gestured
signs, with data on participant, eliciting condition, mouthing type, and any additional action units that co-occurred with the mouthing. The table shows that 7/12
sign types were mouthed at least once. The most frequently mouthed sign type
was nausea. Nine different participants used a mouthing on a disgust sign at least
once. Most occurrences of mouthing (8/12 mouthed signs) happened in the single
word condition.
Figure 5 below of the clause worm index1 pff (German gloss wurm ich
ekelig) shows an example of a mouth gesture occurring over a disgust related
sign with no additional action units. It is clear from the figure that facial movements with temporal scope over larger constituents than the morpho-syntactic
signs for disgust occurred, even though we did not code them. In this example,
there are three groups of facial movement with regards to temporal scope. First,
there is a facial movement with temporal scope over the entire clause: the brows
go down at the stroke phase of the first sign worm (frame 1) and stay down until
the end of the clause (frame 3). Second, there is a facial movement with temporal
scope over two signs: the lip corners are pulled laterally and slightly down from
the articulation of the second sign index1 (frame 2) and stay in that position until
the end of the clause (frame 3). Third, there are facial movements that only have
temporal scope over individual signs: there is a mouthing over the first sign worm
(frame 1), and there is a mouth gesture pff (expulsion of air through compressed
lips) over the disgust related sign we labeled pff (frame 3). In coding the action
single
neck
e-el
single
polite
e-el
13
single
hate
e-el
18
reported
yuck
e-el
18
single
neck
e-el
19
single
disgust
e-el
20
reported
nausea
sle-t
20
single
nausea
e-el
direct
nausea
sle-t
direct
nausea
sle-t
direct
pff
pff
single
throat
e-el
e-el
polite
single
Mouthing
1
1
1
1
1
1
1
1
units that occurred over the disgust related sign, we excluded any pre-existing
facial movements such as the brow furrow and lateral pulling of the lips. We only
coded AU50 for the mouth gesture pff .
Figure 5. Mouth gesture over disgust sign in the sentence worm index1 pff (German
gloss wurm ich ekelig)
dr
op
AU pre
s
7.
lid sor
A
AU U17 s t
10 .ch igh
t
.u
pp in ra
ise
er
AU lip
r
2
r
AU 0.li aise
p
r
4.
br stre
ow tc
h
l
A ow
AU U50 ere
r
.
s
9.
no pe
ec
AU se
h
43 wri
nk
.e
y
l
e
AU e
c
AU 6.c losu
21 hee re
kr
AU .ne
ai
27 ck
se
AU .mo tigh
ut ten
2.
o
h
e
AU ute str r
1. r br etc
AU inn ow h
16 er
r
.lo br aise
w ow
er
lip rais
de e
pr
es
s
de
er
.ja
26
AU
15
.lip
co
AU
rn
t
ar
sh
sp
ue
.lip
ng
25
.to
AU
19
AU
ow
100
90
80
70
60
50
40
30
20
10
0
Frequency (%)
To see whether there was 100% consistency of co-occurrence for an action unit
to a sign type, we present the frequency distributions of the action units per sign
type. Figure 7 shows the four most frequent signs together. None of the four highest frequency sign types has a 100% consistency of co-occurrence with any action
unit. hate, disgust, and throat show similar AU frequency distribution patterns, but yuck seems to have a unique profile. Specifically, AU19 tongue show
is completely absent from yuck while it is the second most consistent AU for the
other three signs. Note, however, that there is an un-equal amount of tokens per
sign type, and participant numbers, and eliciting conditions are not constant for
each type, therefore strong conclusions about patterns cannot be drawn.
1
AU
ue
ng
to
9.
rt
pa
lip
5.
1
AU
ht
ig
pe
up
0.
se
i
ra
1
AU
in
.ch
17
AU
t
ds
li
7.
AU
so
s
re
ep
rd
e
rn
op
dr
co
w
.ja
2
AU
ow
sh
r
lip
h
tc
re
st
r
s
e
h
er
ch
re
ise
ise
ise
kl
es
tc
ne
er
ra
ra
ra
su
ee
re
pr
te
rin
w
t
o
p
k
e
h
l
s
w
w
o
s
w
l
.
c
d
o
o
ee
e
tig
th
50
ip
br
br
.lip
ye
ch
os
ow
ck
rl
ou
.e
6.
er
er
20
AU
.n
e
e
3
t
br
m
n
.
U
U
9
.
4
w
.n
4
A
A
in
ou
27
AU
.lo
21
AU
1.
2.
AU
16
AU
AU
AU
AU
U
A
r
se
i
ra
Figure 7. Frequency distribution for sign types hate, disgust, throat, and yuck
2
AU
s
.lip
10
20
30
40
50
60
70
80
90
100
HATE
DISGUST
THROAT
YUCK
.lip
15
AU
de op
AU pre
7. sso
li
A
r
AU U17 ds t
10 .ch igh
t
.u
pp in ra
er
ise
AU lip
r
2
r
AU 0.li aise
4. p st r
br
ow retc
h
AU low
e
AU 50 re
9. .sp r
n
e
AU ose ech
43 wr
in
.e
k
AU ye c le
AU 6.c los
21 he ure
e
AU .nec k ra
ise
27 k t
AU .mo igh
t
ut en
2.
o
h
e
AU ute str r
e
r
t
1
b
AU .inn row ch
16 er
r
.lo br aise
w ow
er
lip rais
de e
pr
es
s
ow
dr
er
co
rn
26
.ja
sh
ue
AU
ng
.to
19
AU
AU
25
.lip
sp
ar
100
90
80
70
60
50
40
30
20
10
0
GOOSEBUMPS
POLITE
NAUSEA
AU AU
19 25.
AU
. t lip
on s
15
p
A
. li U gue art
p
co 26. sho
rn jaw w
er
d d
AU ep rop
7. res
A
AU U1 lid so
10 7. s t r
. u ch igh
pp in
t
AU er rais
lip er
2
AU 0. ra
4. lip ise
br st r
ow ret
AU lo ch
AU 5 w
9. 0. s ere
AU nos pee r
43 e w ch
AU . ey rink
AU 6. e c le
c lo
2
AU 1. n hee sur
k e
2 e
AU 7. m ck t rais
2. ou igh e
AU out th ten
e s
AU 1. i r b tre er
16 nne row tch
. lo r b r
w ro aise
er w
lip ra
de ise
pr
es
s
VOMIT
TOUCH
151
per type, however, the same participant signed all the vomit tokens, and the same
participant signed all the touch tokens. Therefore, it is impossible to attribute
any consistencies to the sign type, as they could equally well be due to the habitual
facial movements of the signer. Also, bear in mind that the frequencies are calculated over three tokens only.
The remaining low frequency sign types neck, slang, and pff only have 12
tokens each and therefore, a chart of the frequency distributions of their co-occurring AUs would not be meaningful.
Figure 10 below shows the occurrence of a disgust related AU configuration
occurring over a disgust related sign. When signing the sentence index1 worm
index1 disgust (German gloss ich wurm ich ekelig), there are three different
facial movements with regards to temporal scope. First, there is a facial movement with temporal scope over the entire clause: the eyes are narrowed from the
first sign index1 (frame 1) and stay this way until the end of the clause (frame 4).
Second, there are facial movements with temporal scope over several signs: the
brows are lowered from the second sign worm (frame 2) and stay in that position
until the end of the clause (frame 4); the lip corners are pulled down from the
third sign index1 (frame 3) and stay in that position until the end of the clause
(frame 4). Third, there are facial movements that only have temporal scope over
individual signs: there is a mouthing over the second sign worm (frame 2), and
there is tongue protrusion through an open mouth over the last sign disgust
Figure 10. Emotion related action units over disgust sign in the sentence index1 worm
index1 disgust (German gloss ich wurm ich ekelig)
(frame 4). Note that when coding the action units for this disgust related sign, we
excluded all the pre-existing action units such as the lowered brows.
6.7 Comparison to Ekmans prototypical disgust face
According to Ekman et al. (2002a: 174), the prototypical disgust face could be any
of the six following action unit combinations:
i. AU9 nose wrinkle
ii. AU9 nose wrinkle + AU16 lower lip depress + AU15 lip corner depressor +
AU26 jaw drop
iii. AU9 nose wrinkle +AU17 chin raiser
iv. AU10 upper lip raiser
v. AU10 upper lip raiser + AU16 lower lip depress + AU15 lip corner depressor + AU26 jaw drop
vi. AU10 upper lip raiser +AU17 chin raiser
As shown in Figure 6, all of these action units occurred at least once in our data.
However, AU9 nose wrinkle and AU10 upper lip raiser, which are the core of
Ekmans prototypical disgust faces, are not the highest frequency action units in
our data. The core disgust face for our lexical items is a combination of AU19
tongue show, AU25 lips part, and AU26 jaw drop. Figure 11 below contrasts
Ekmans prototypical disgust face (version ii) with the prototypical disgust face we
found associated with disgust signs in our data:
Figure 11. Facial movements most typically associated with disgust related signs in DGS
(left picture) compared to Ekmans prototypical disgust face (right picture)
153
Figure 12. No action units over disgust sign in the sentence worm index1 yuck
(German gloss wurm ich ekel)
Third, there is a facial movement that only has temporal scope over an individual
sign: there is a mouthing over the first sign worm. This is the only sign in this
clause to have a facial action unique to its temporal scope. The disgust related
sign in this clause (yuck) has no additional marking besides the pre-existing brow
lowering, eye closure, and mouth corner pulling. One might interpret the mouth
movements over index1 and yuck (frames 2 and 3) as the spreading of a phonological facial feature over the prosodic word domain; however, it does not seem
to be the case that the first person pronoun was cliticized onto yuck since there
was no reduction in form of the pronoun comparable to that described in Sandler
(1999a). This issue remains to be addressed by future research.
6.10 Smiles
In six cases, a smile had onset before or during the sign analyzed, but it remained
on the face well after offset of the sentence; 3864ms on average. This smile seemed
to be directed at the interviewer and to express the amusement of the participant
with the task. Figure 13 below shows that when producing the sentence index1
worm throat (German gloss ich wurm ekel), the signer made three different
facial movements with regards to temporal scope. First, there is a facial movement with temporal scope over the entire clause: the brows go down and the eyes
are narrowed from the onset of the first sign worm (frame 1) and stay this way
until the end of the clause (frame 4). Second, there are facial movements that only
have temporal scope over individual lexical signs: the sign worm (frame 2) has
a mouthing, and the sign throat has tongue protrusion. Third, there is a facial
Figure 13. An action unit with temporal scope larger than the clause in the sentence
index1 worm throat (German gloss ich wurm ekel)
155
movement with temporal scope over a time period longer than the clause: the lip
corners are pulled upwards in a smile from the start of the clause (frame 1) and
the upward pull is maintained even though antagonistic muscles pull the mouth in
other directions in frames 2 and 3; the upward pull remains as the hands return to
their resting position in frame 4; the upward pull remains for some seconds after
the hands have reached their resting position (frames 5 and 6).
6.11 The answers to our three questions
Does an emotion related facial movement consistently temporally co-occur with
disgust signs as single words? Emotion related movements occurred with 95%
consistency in the single word condition. In one case in the single word condition,
a disgust sign occurred with a mouthing but no emotion related facial movement.
Does an emotion related facial movement consistently temporally co-occur
with disgust signs in direct and reported sentence types? An emotion related facial movement occurred with 82% consistency in direct sentence types and 100%
consistency in reported sentence types. In one direct speech sentence, a disgust
sign occurred with no facial movement that was unique to the time window of the
morpho-syntactic sign. In three direct speech cases, a sign occurred with a mouthing or mouth gesture but no emotion related facial movement.
Which facial muscles are used by each signer when producing an emotion
related facial movement temporally co-occurring with a disgust sign? 18 different
action units occurred in our data and none of them was 100% consistent with any
sign type. The most frequent facial actions were AU25 lips part, AU19 tongue
show, and AU26 jaw drop.
What can this data tell us about the function of the facial movements that are
lexically related to signs for emotion concepts? We answer this question in the
discussion section below.
7.
Discussion
at all on disgust related signs. Our data confirms that most disgust related signs
are articulated with a disgust related face that is unique to the time frame of the
morpho-syntactic sign. However, in order to check whether compounds are possible, we did ask three native DGS consultants whether it is possible to combine
a disgust face or sad face with the verb laufen (run/walk) to mean run while
being sad/disgusted and they confirmed that it is.
7.2 Phonological function
As we stated above, our diagnostic heuristics for the phonological status of a facial
movement are: temporal scope only over the morpho-syntactic or possibly the
phonological word and 100% consistency of co-occurrence with a particular lexical sign. We found that for almost all our disgust related tokens (62/63), there was
a facial movement unique to the time span of the morpho-syntactic sign, and that
in 58/63 cases, there was a facial movement configuration unique to the time span
of the morpho-syntactic sign that could mean disgust. However, no one particular
movement was 100% consistent with any particular disgust related sign. Does this
mean that disgust related signs in DGS do not have a facial phonological component?
Given that disgust related movements occurred on the morpho-syntactic disgust related signs even on top of already existing attitude depicting disgust related
movements, as shown in Figures 10 and 13 above, we think it is highly likely that
the disgust signs in our corpus are specified for a facial phonological component.
The variation in the articulation of the facial component could be due to at
least two factors. Since tongue protrusion with open mouth was the most common
facial movement associated with disgust, it might be the case that the facial phonological component of some disgust signs in DGS is in the process of lexicalization
and that in the future, there will be more homogeneity in production per lexical
item across the DGS speaking community.
It is also possible that the variations in production of the disgust related faces
reflect allophones of a schematic disgust face phoneme.
As we continue to analyze the many emotion signs in our corpus, we will see
whether other emotion signs are more consistently produced with emotion related
faces, or whether disgust signs are unique in this respect.
The phonological function of disgust facial movements can be tested in future
single sign identification tasks in which it can be seen whether the presence of a
facial component facilitates participant reaction time and reduces error rates compared to signs with no facial component.
However, one may ask if the disgust related faces are phonemes, why do they
not belong to the words layer, as part of the form of the word? If they are morphemes that convey the same type of information as words, why are they simply
not words?
As we stated in our introduction, there are probably more than four layers of
information in face-to-face conversation, and there may be some semiotic units
which are on the border between two layers. Enactings and mouthings have slightly different properties to any of the four information types that we mentioned
in this paper, as shown in Table 5 above. Therefore, we place them in their own
information type layer and call it the parallel lexicon. Do the disgust faces in our
data, which should belong to the enacting category, also show the properties of
enactings reported in the literature? In this study, we did not look for evidence
that these movements can combine with each other in hierarchical structures, or
with signs other than disgust related signs. From our evidence alone, it could well
be that the disgust related faces in our data should be regarded as phonological
features that are part of the form of words in the words layer, like empties, except
that they also happen to have semantic content. However, if as attested by our
informants, they can also combine with non-disgust words, we think that these
semiotic units, together with mouthings, make up their own sub-system of communication that is interdependent on words.
7.5 Interdependent systems
Are the disgust facial movements in our data paralinguistic? In the non-manuals
literature, it is common to find a distinction made between linguistic and paralinguistic markers, or some variation of these terms. Reilly and her colleagues
(see Reilly (2005) for an overview), for instance, use the following four oppositions: grammatical vs. non-grammatical, linguistic vs. affective, linguistic vs. communicative, grammatical vs. communicative. Crasborn & Van der Kooij (2013)
make the following distinctions: linguistic, paralinguistic (which includes emotion/intention signals), and extra-linguistic (non-communicative actions). The
linguistic/paralinguistic distinction is not incompatible with our analyses, but it
would be an unnecessary add-on from our perspective. Given that there is still
no consensus on what exactly is emotion or paralinguistic on the one hand, and
what is linguistic on the other hand, we prefer to avoid such labels and rather
examine the various co-existing communicative signals and attempt to categorize
them according to differences and similarities in properties such as form-meaning
relationship, combinatoricity, etc. We do this so as to not prematurely exclude relevant communicative behavior from analysis, as has historically happened in the
case of sign languages, as in the case of intonation and co-speech gestures (Liddell
2003: 358) and in the case of iconicity (Taub 2001). We recognize that some of
our basic theoretical assumptions are not compatible with all linguistic theories.
However, like Liddell we think
[] that spoken and signed languages both make use of multiple types of semiotic elements in the language signal, but that our understanding of what constitutes
language has been too narrow. (Liddell 2003: 362)
The systems (by system we mean a set of units that interact with each other and
that this interacting set of units together serves a particular function) we listed
in our information type layers are to some degree independent in the sense that
they each serve a unique function. However, some of them, such as intonation
and words, interact with each other and in fact seem interdependent in that the
interpretation of a sentence (e.g. as question, statement, irony) is not possible
without both types of information. Together intonational units and lexical units
serve a more general communicative function; therefore, they are sub-systems of
the same larger system. Co-speech gestures also have a special communicative
function, and they interact with words. Co-speech gesture and words then are
also sub-systems of a larger communicative system. Apart from these four layers,
there are also emblematic gestures, the parallel lexicon layer that we propose in
this paper, and other kinds of communicative behavior which have not yet been
intensively studied such as the conversational facial movements made in spoken
languages. Which of the sub-systems should be considered linguistic and which
paralinguistic depends on ones theory of language and ones theory of emotion,
but the behaviors of these systems are describable facts.
7.6 Economy, effort, and redundancy
Are the facial movements we found on the disgust related signs redundant? It is
proposed in Hohenberger & Happ (2001) that mouthings are redundant in DGS.
Since the disgust signs in our data repeat the meaning of the manual part of the
lexical item, just like mouthings often do, we consider the question: are they redundant?
In discussing the meaning of redundancy, Hohenberger & Happ (2001) note
that even though languages are designed to be economic, there is redundancy built
into them at all levels. The redundancy is there to guarantee information transfer.
This kind of redundancy can perhaps be compared to the redundancy of a humans second kidney. They argue, however, that there can also exist too much redundancy, which they term profligacy. They state that given the economic design
of languages, profligate elements will eventually disappear.
The economy of language is described by Zipf as the Principle of Least Effort:
In simple terms, the Principle of Least Effort means, for example, that a person
in solving his immediate problems will view these against the background of
his future problems, as estimated by himself. Moreover he will strive to solve his
problems in such a way as to minimize the total work that he must expend in
solving both his immediate problems and his probable future problems. That in
turn means that the person will strive to minimize the probable average rate of his
work-expenditure (over time). And in so doing he will be minimizing his effort, by
our definition of effort. Least effort, therefore, is a variant of least work. (Zipf 2012
[1949]: 1; emphases in original)
We are not convinced that mouthings are profligate elements in sign languages.
They appear in other sign languages besides DGS (Crasborn et al. 2008), and they
still have not fallen out of use as attested by the mouthing rate (40%) in the disgust
section of our corpus. Furthermore, besides fulfilling a redundant function of the
good sort (ensuring transmission of the signal by adding salient perceptual cues,
i.e. functioning as phonological elements), they also seem to take on morphological and information structure functions (Crasborn et al. 2008; Crasborn & Van
der Kooij 2013).
Like mouthings, we think that the enacting actions, including the disgust related facial movements in our data, ensure successful transmission of the signed
signal, and perhaps also take on additional functions. By investing more work in
creating a robust signal, one decreases the chances of having to repeat ones self or
of being catastrophically misunderstood, therefore minimizing effort as defined
by Zipf: probable average rate of work-expenditure over time.
Beyond the articulatory work needed to ensure information transfer in communication (see e.g. Tobin 1997), one can invest even more articulatory work on
some words than others, and this indicates to the interlocutor what information is
most important. Gussenhoven (2002) calls this the Effort Code; however, his use
of the word effort is not to be confused with Zipfs definition of effort. In Zipfs
terms, Gussenhovens effort is work.
To the best of our knowledge, enacting facial actions and mouth gestures
have never come under suspicion of profligacy because they are not derived from
spoken languages and are therefore considered more native to sign languages.
However, many mouth gestures and all enacting actions are derived from emotion/intention faces or instrumental facial movements, most of which a human
masters before she produces her first words (Izard et al. 1995). We regard these
items to be borrowings into DGS to the same extent as mouthings.
By only coding the action units that were timed to start and end within the
time frame of the morpho-syntactic sign, we were able to filter out facial movements that have temporal scope over the entire clause or over constituents larger
than single signs.
The disgust related movements with temporal scope only over the lexical sign
for disgust were often made in addition to disgust related movements with temporal scope over larger constituents, which appear to convey speaker/role attitude.
That is, often the brows would be lowered and the lip corners turned down for
most of the clause, and during the production of the disgust sign, additional facial
movements were made, such as tongue show. We propose that these facial movements function is not to depict attitude but rather to add phonological information to the disgust sign in order to aid identification by the interlocutor.
We found that 18 different action units were used in total in our data set. No
action unit was 100% consistent with any particular disgust sign type, however, the
most common disgust face was an open mouthed tongue show (AUs 19, 26, 25).
Data from consultants suggests that such facial movements can also function
as a modifier on signs other than disgust signs. We propose that this facial movement element is part of an information layer temporally parallel to words/signs,
in the same way that intonation and gesture exist as information layers temporally
parallel to words/signs and interdependent on them.
Acknowledgments
This work was supported by the International Max Planck Research School The Life Course:
Evolutionary and Ontogenetic Dynamics (LIFE), and the Cluster of Excellence, Languages of
Emotion, Freie Universitt Berlin.
References
Aarons, Debra, Ben Bahan, Judy Kegl & Carol Neidle. 1992. Clausal structure and a tier for grammatical marking in American Sign Language. Nordic Journal of Linguistics 15. 103142.
Anderson, Diane E. & Judy S. Reilly. 1997. The puzzle of negation: How children move from
communicative to grammatical negation in ASL. Applied Psycholinguistics 18(4). 411429.
DOI: 10.1017/S0142716400010912
Aryani, Arash, Markus Conrad & Arthur M. Jacobs. 2013. Extracting salient sublexical units
from written texts: Emophon, a corpus-based approach to phonological iconicity. Frontiers
in Psychology 4.654. DOI: 10.3389/fpsyg.2013.00654.
Baker-Shenk, Charlotte. 1983. A micro-analysis of the non-manual components of questions in
American Sign Language. PhD dissertation, University of California, Berkeley, CA.
12
13
14
15
16
17
18
10
11
11
mouthing of
ekel
hate
disgust
neck
hate
mouthing ekel
slang
polite
polite
slang
goose-bumps
goose-bumps
nausea mouthing of
schlecht
goose-bumps
1
4
AU17
11
AU17Int
10
goose-bumps
goose-bumps
AU15
AU15Int
AU6
1
AU19
AU4Int
4
AU26
AU4
1
AU6Int
AU7
AU7Int
AU43
AU1
yuck
AU1Int
4
AU12
AU2
1
AU12Int
AU2Int
3
AU50
AU9
1
AU9Int
AU10
AU10Int
AU16
AU16Int
started
before
word
onset
AU20
yuck
AU20Int
AU25
Notes
Appendices
AU27
AU21
10
10
10
11
29
30
31
32
12
12
13
13
13
14
14
14
36
38
39
40
41
42
43
throat
throat
throat
hate
disgust
disgust
mouthing ekel
hate
yuck
yuck
touch
touch
touch
AU21
polite
yuck
AU2Int
37
AU9
AU9Int
polite
AU12
12
28
AU12Int
35
27
hate
AU20
11
26
throat
AU20Int
11
25
hate
hate
AU27
33
24
AU43
34
23
AU1
AU1Int
AU2
AU4
21
AU4Int
22
AU6
1
AU6Int
AU7
AU7Int
AU10
hate
AU10Int
pff
AU15
AU15Int
12
AU17
AU50
AU17Int
AU19
AU25
19
Notes
AU26
20
AU16Int
AU16
18
18
18
19
19
19
54
55
56
57
58
59
mouthing ekel
hate
disgust
disgust
disgust
neck
mouthing ekel
AU6
10
yuck
yuck
AU6Int
AU10
AU4
1
AU4Int
AU9
disgust
disgust
AU9Int
AU10Int
20
18
53
AU15
63
17
52
AU15Int
20
17
51
disgust
AU16
hate
1
AU16Int
hate
AU17
62
17
50
AU17Int
19
16
49
AU21
20
16
48
vomit
vomit
AU7
AU7Int
AU27
60
15
47
AU50
61
15
46
AU1
throat
AU1Int
vomit
AU2
AU19
AU2Int
AU26
AU20
14
AU20Int
15
AU25
44
Notes
AU43
45
AU12Int
AU12
171
direct_speech 3
reportive_
speech
single_word
direct_speech 3
reportive_
speech
single_word
direct_speech 2
reportive_
speech
single_word
eliciting_con- total_signs_
dition
sentence
participant
929
3709
1564
897
2035
1484
1612
2927
2184
sentence_
length_ms
participant
condition word_
length_ms
my
nausea
ekel
929
freund
friend
er-index
him-index
460
worm
508
ekelig
wurm
770
goose-bumps
ekel
794
him-index
friend
897
person
person
566
say
sagen
346
190
er-index
freund
goose-bumps worm
305
770
wurm
520
ekel
450
wurm
717
worm
wurm
like
mag
664
not
nicht
190
goose-bumps worm
ekel
477
self-him
he-tell-me
337
friend
ich
194
throat
ekel
1612
433
yuck
er-bescheid-ich selbst-er
383
219
ekelig
953
freund
mein
ich
worm
356
wurm
875
380
worm
wurm
346
also
auch
725
goosebumps
ekelig
975
yuck
ekelig
direct_speech 3
reportive_
speech
single_word
direct_speech 5
reportive_
speech
single_word
direct_speech 4
reportive_
speech
single_word
10
eliciting_con- total_signs_
dition
sentence
participant
734
2374
1817
1058
5691
3137
660
2249
1470
sentence_
length_ms
participant
condition word_
length_ms
creeping
worm
friend
my
ekel
734
freund
mein
266
i
283
wurm
ich
241
neck
ekel
650
friend
my
1058
freund
mein
362
worm
528
schwirren
wurm
563
polite
ekel
523
person
friend
660
person
freund
227
worm
i
487
wurm
410
356
ich
251
goose-bumps
875
173
146
disgust
ekelig
779
so
so
589
slang
ekelig
800
he-index
he-tell-me
he-index
er-bescheid-ich er-index
354
ich
147
self-freind
selbst-freund
472
self-i
selbst-ich
470
he-tell-me
er-bescheid-ich er-index
353
slang
hass
809
953
worm
wurm
500
tell
erzhlen
380
nausea
unwohl
781
polite
hass
470
902
hate
hass
825
he-index hate
er-index hass
522
worm
wurm
539
what
was
454
1108
worm creeping
wurm schwirren
374
direct_speech 3
reportive_
speech
single_word
direct_speech 4
reportive_
speech
single_word
direct_speech
reportive_
speech
single_word
eliciting_con- total_signs_
dition
sentence
participant
860
2406
813
2570
2207
970
3211
1853
sentence_
length_ms
participant
condition word_
length_ms
friend
my
ekel
860
freund
mein
187
throat
ekel
183
friend
my
813
freund
mein
337
worm
342
676
wurm
ich
399
hate
ekel
970
freund
friend
mein
my
517
worm
214
ich
327
356
wurm
725
hate
875
317
267
throat
ekel
912
this
550
he
he-tell-me
polite
er-bescheid-ich ekel
254
he-tell-me
er-bescheid-ich er
323
ich
220
he-tell-me
er-bescheid-ich dies
483
pff
ekelig
801
953
worm
wurm
643
worm
wurm
626
worm
wurm
533
yuck
iiiiiii
589
hate
hass
675
he
er
294
hate
hass
853
direct_speech 2
reportive_
speech
single_word
direct_speech 3
reportive_
speech
single_word
direct_speech 3
reportive_
speech
single_word
10
10
10
11
11
11
12
12
12
eliciting_con- total_signs_
dition
sentence
participant
1191
2460
1480
1320
2807
2890
1010
3229
1463
sentence_
length_ms
12
12
12
11
11
11
10
10
10
participant
condition word_
length_ms
yuck
darling
my
ekel
1191
schatz
mein
323
worm
i
242
wurm
570
ich
112
throat
ekel
1320
freund
friend
mein
my
267
i
172
igitt
654
ich
424
touch
ekel
1010
freund
friend
mein
my
597
touch
402
ekel
worm
911
356
wurm
552
polite
875
index-darling
index-schatz
150
throat
ekel
798
also
auch
377
worm
wurm
1812
say
sag
886
953
self
selbst
240
yuck
igitt
690
worm
wurm
404
hate
ekel
690
worm
wurm
1301
touch
ekel
940
worm
wurm
815
direct_speech 3
reportive_
speech
single_word
direct_speech 3
reportive_
speech
single_word
direct_speech 4
reportive_
speech
13
13
13
14
14
14
15
15
eliciting_con- total_signs_
dition
sentence
participant
3714
2648
1600
4041
2510
889
2773
1967
sentence_
length_ms
15
14
14
14
13
13
13
participant
condition word_
length_ms
self
344
i
319
selbst
ich
296
throat
ekel
273
friend
my
1600
freund
mein
414
worm
about-worm
400
wurm
auf-wurm
319
hate
ekel
550
freind
my
889
freund
mein
284
disgust
i
189
ekel
564
356
ich
670
hate
875
516
worm
wurm
310
say
sag
303
throat
ekel
1641
about-freind
auf-freund
283
worm
wurm
733
953
310
vomit
erbrech
1769
to-me
auf-ich
417
say
sag
246
ekel
576
disgust
ekel
584
880
1345
index-friendthroat
indexfreund
210
worm
wurm
547
931
worm
throat
wurm ekel
790
forfriend
fuerfreund
640
single_word
direct_speech 3
reportive_
speech
single_word
direct_speech 3
reportive_
speech
single_word
direct_speech 5
15
16
16
16
17
17
17
18
eliciting_con- total_signs_
dition
sentence
participant
3068
607
2508
1448
4133
2183
1204
sentence_
length_ms
18
17
17
17
16
16
16
15
15
condition word_
length_ms
participant
ekel
disgust
ich
i
727
disgust
ekel
793
darling
my
607
schatz
mein
340
really
worm
307
echt
227
wurm
578
bescheid
tell
freund
friend
1290
hate
i
1289
hass
ich
301
vomit
ekel
787
friend
my
1204
freund
356
mein
875
over
ueber
467
say
sag
348
disgust
ekel
643
index-friend
index-freund
273
worm
wurm
1095
say
sag
953
456
hate
hass
898
460
worm
yuck
fa
621
index-darling disgust
wurm
auf-wurm
worm
wurm
945
friendaboutvomit-wormworm
freunderbrechwurm
index-schatz ekel
112
worm
wurm
383
self
selbst
reportive_
speech
single_word
direct_speech 4
reportive_
speech
single_word
direct_speech 3
reportive_
speech
single_word
18
18
19
19
19
20
20
20
eliciting_con- total_signs_
dition
sentence
participant
700
3314
1847
767
2440
1328
680
3062
sentence_
length_ms
20
20
20
19
19
19
18
18
participant
condition word_
length_ms
disgust
nausea
ekel
700
erzaehl
tell
friend
index-freund freund
470
worm
wurm
821
index-friend
747
hate
i
241
hass
794
ich
232
disgust
ekel
767
sag
say
friend
index_freund freund
372
worm
wurm
417
index-friend
index-freund
535
953
index-friend
547
i
341
ekel
ich
178
neck
ekel
351
friend
680
freund
my
535
356
mein
306
875
717
disgust
ekel
790
over
ueber
363
index-friend nausea
index-freund schlecht
213
worm
wurm
390
disgust
ekel
382
yuck
fa
570
796
to-worm worm
auf-wurm wurm
130
worm
wurm
753
Appendix 3. Gloss choices of transcribers for each disgust related sign type
Sign Type
Transcription Variations
hate
disgust
throat
yuck
goose-bumps
polite
hass, ekel
nausea
vomit
erbrech, erbrech
touch
ekel, ekel
10
neck
n/a
11
slang
hass, ekelig
12
pff
ekelig