Review

International Journal of Audiology 2003; 42:S68–S76

Jerker Rönnberg Cognition in the hearing impaired and

Department of Behavioural Sciences,
Linköping University and deaf as a bridge between signal and
The Swedish Institute for Disability
Research,
dialogue: a framework and a model
Linköping, Sweden

Key Words Abstract

Cognition This paper focuses on the role of cognition in visual language processing in the deaf and hard of
Working memory hearing. Although there are modality-specific cognitive findings in the literature on comparisons across
speech communication modes and language (sign and speech), there is an impressive bulk of evidence
Speech-reading that supports the notion of general modality-free cognitive functions in speech and sign processing. A
Speech understanding working-memory framework is proposed for the cognitive involvement in language understanding (sign
Sign language and speech). On the basis of multiple sources of behavioural and neuroscience data, four important
Modality-free parameters for language understanding are described in some detail: quality and precision of phonology,
long-term memory access speed, degree of explicit processing, and general processing and storage
Working memory model capacity. Their interaction forms an important parameter space, and general predictions and applications
can be derived for both spoken and signed language conditions. The model is mathematically
formulated at a general level, hypothetical ease-of-language-understanding (ELU) functions are
presented, and similarities and differences from current working-memory and speech perception
formulations are pointed out.

Introduction speech signals (i.e. auditory, visual, or tactile) are poorly

Individual-difference research on cognitive abilities associated
with visual communication skills such as speech-reading and
speech understanding has a relative stronghold in aural
rehabilitation and audiology (Jeffers & Barley, 1971). Charac-
teristic of previous work is the tendency to associate single
perceptual or cognitive predictors with single language-test
variables. More recent research has started to focus on the
multicomponent nature of cognitive functions that support
speech reading (e.g. Cowie & Douglas-Cowie, 1992; Gailey,
1987). We have been able to show that, for adventitiously
hearing-impaired persons, certain individual cognitive abilities
that together form an interacting cognitive architecture mediate
effective visual speech perception and understanding (Rönnberg,
1995, 2003). The fact that subsets of individual cognitive func-
tions support speech understanding under different conditions,
with or without sensory aids (e.g. Lyxell et al, 1996; Rönnberg,
Andersson, Lyxell & Spens, 1998a; Lunner, 2002), represents an
important first cornerstone of a general, comprehensive multi-
modal model of speech understanding with poorly specified
linguistic input (Rönnberg et al, 1998b). In this article, it is
claimed that the actual interface between perception and cog-
nition in speech understanding constitutes the bridge between
signal processing and dialogue. Theoretical and practical implic-
ations are likely to result as a consequence of a model which
addresses this important interface.
It is here emphasized that the term speech understanding
rather than speech perception should be used, because when
specified or distorted, there is always the inherent task demand
of substituting missing information with inference-making to
reach understanding (Lyxell et al, 1998; Rönnberg et al, 1998b).
In this sense, speech understanding is a broader term that
includes both bottom-up and top-down processes. Task demands
will always contribute to the relative weights of and interactions
between the two types of process.
Understanding speech implies extraction of meaning and
overall comprehension of a message. In a dialogue this becomes
especially crucial, because meaning and meaning-extracting
activities always rely on negotiations and co-constructions
between the two parties (e.g. Markova & Linell, 1996). The
general appraisal, then, is that speech-understanding tests are
much more cognitively embedded or demanding than speech
perception tests, and, typically, test materials are more complex
and ecologically relevant (Borg, this supplement; Larsby &
Arlinger, 1994). Speech understanding—especially in dialogue-
like situations—is therefore likely to be dependent on the
interaction between implicit, bottom-up processes and explicit,
top-down processes, and not only on the particulars of bottom-
up word recognition processes (Bernstein & Auer, 2003;
Berstein, Demorest & Tucker, 2000).
The cognitive framework
A framework for a general working-memory system for the
processing of poorly specified language input was presented a

Jerker Rönnberg
Department of Behavioural Sciences,
Linköping University,
S-581 83 Linköping, Sweden
E-mail: JR@IBV.LIU.SE
few years ago (Rönnberg et al, 1998b). The current formulation
is an attempt to develop this framework a step further, and to
specify the general parameters of a model, and some of their
interactions.
The components involved in the general formulation of the
framework are: perceptual input channels, working memory,
and long-term memory (Rönnberg, 2003). The model acknow-
ledges top-down processes, in the sense that it allows for inform-
ation retrieved from long-term memory affecting processing of
the perceptual input, specifically utilizing script-related and
semantic information about the particular communicative
situation or topic (Samuelsson & Rönnberg, 1993). As a rule of
thumb, then, the more degraded, distorted or unintelligible the
perceived signal, the greater the contribution from long-term
memory, and, conversely, the more specified the perceptual
input (sometimes conveyed by several poorly specified inform-
ation channels in concert), the less the need for top-down
processing. The reciprocal relationship between top-down and
poorly specified bottom-up processing in speech understanding
is also assumed to be modulated by the general capacity for
processing and storage of information in an on-line working-
memory system. Lexical access or word recognition in long-term
memory is assumed to be achieved through phonological,
and/or contextual, successive specification in real time (see Luce
& Pisoni (1998) and Marslen-Wilson (1987) for spoken word
identification).
It is important to note that interaction (i.e. bottom-up
processing interacting with top-down processing) in the general
sense of other auditory speech perception models such as the
TRACE model (McClelland & Elman, 1986), the Cohort model
(Marslen-Wilson, 1987) and the NAM model (Luce & Pisoni,
1998) is assumed. Owing to the general precondition of poor
language input and assumption of multi-modal interaction
(Rönnberg et al, 1998b), the framework emphasizes prosodic and
syllabic aspects as one means of disambiguating the signal into
word boundaries for lexical activation, a feature that has not
been made explicit in the above speech perception models
(Andersson, 2001). In the word-initial cohort (Marslen-Wilson,
1987), for example, reliance is put on similarities among
candidate items that share initial phonetic information. Gener-
ally, the present framework is more similar to the NAM,
because it also assumes a matching process between input and
representations in memory that takes into account both phono-
logical similarity and word frequency (i.e. semantic constraints).
The framework takes the whole speech-understanding process
a step further by assuming that explicit working-memory
capacity is called for only when there is a mismatch in the word
decision or lexical activation process. This assumption concerns
processing economy (for details, see below). The present frame-
work also seeks to expand the domain of speech understanding
into sign understanding. In general, the framework and the
model are about the ease with which the processing of multi-
modal input and language understanding is accomplished. The
model is not a model of language understanding per se.
Classes of working memory model
There are several classes of working-memory model, developed
for different theoretical and applied purposes, supported by
different kinds of data (see Richardson et al (1996) for an
overview). In this context, it is important to note that the
Cognition in the hearing impaired and deaf as a bridge between
signal and dialogue: a framework and a model
tradition following the model of Baddeley and colleagues (e.g.
Baddeley, 2000) emphasizes that working-memory resources
comprise amodal as well as modality-specific components. Thus,
initially a central, executive component as well as two modality-
specific slave systems (i.e. the phonological and visuospatial
systems), meeting different attention, storage and processing
demands, were assumed (Baddeley & Hitch, 1974). Later
research on, for example, visual similarity on verbal recall and
on the impact of meaning on sentence recall has prompted
Baddeley (2000) to suggest a synergistic, episodic memory buffer,
capable of binding together information from long-term memory
and from the slave systems.
An alternative type of model follows Just and Carpenter and
colleagues (e.g. Just & Carpenter, 1992), where language com-
putation resources are seen as more global processing and
storage capacities. This model contrasted more sharply with the
Baddeley & Hitch (1974) model before the latest revision by
Baddeley (2000). Also, recognizing both the modality-specific
and modality-free nature of embedded cognition, Wilson (2001)
has discussed the similarities and differences that exist when
working-memory systems for signed and spoken language are
compared. It is with her reasoning and arguments as back-
ground that we wish to pursue the general hypothesis that it is
reasonable to formulate a general model as a more explicit
starting point for comparisons across communication modes
and languages.
Thus, when we examine the data and arguments that are
pertinent to determining the level and kind of working-memory
model for sign and speech which are most appropriate, there
seems to be sufficient support to initially test a model where
modality specificity is less emphasized, and integration of
information is favoured (see below for arguments). Furthermore,
attempting to generalize to a cross-language, comparative
approach, we will necessarily have to consider potentially
common, amodal bases for such a working-memory system.
Empirical arguments for the modality-free speech
communication modes aspect of working memory
1. Cross-modal and ‘auditory’ cortical activity may come about
as a result of interventions by means of cochlear implants in
postlingually deaf users (Zatorre, 2001), and auditory cortical
overcompensations, as well as visual cortical recruitment, may
represent the resulting, new perceptual strategies (Giraud et al,
2000). Primary visual cortex areas (V1 and V2) are recruited in
the cochlear implant user, and the degree to which this is done is
also a function of the length of the post-implant phase (Giraud,
Price, Graham, Truy & Frackowiak, 2001). Tactile stimuli in the
congenitally deaf tactile aid user activate secondary auditory
areas (Levänen, 1998). Silent speech-reading activates the audi-
tory cortex (Calvert et al, 1997; Ludman et al, 2000; MacSweeney
et al, 2001), and the activation is dependent on speech-reading
skill (Ludman et al, 2000) and auditory deprivation (MacSweeney
et al, 2001).
Thus, there is strong neurophysiological evidence that in
many cases the interdependence, plasticity, and substitutability
of brain tissue for different speech-processing modes are
commonplace. Depending on the type of intervention and
alternative sensory input used, new ‘audio-visual–tactile’,
speech-processing strategies may thus be developed. The audi-
tory input channel does not seem to be privileged for speech
Rönnberg S69
understanding. Speech-relevant information may arrive in
different ways (Leybaert, 1998). In general, it seems to be the
case that compensatory pressure is at hand when the perceptual
input is poorly specified (Rönnberg et al, 1998b), or needs to be
recombined to represent a ‘new’ perceptual platform for the
profoundly hearing-impaired or deaf individual. This sets the
stage for a more general and abstract working-memory model
that capitalizes on common bases and potentials of a system
that deals with poorly specified input.
2. The generality of certain cognitive and working-memory-
related functions has been empirically demonstrated and
successfully applied to several communicative domains, such as
tactile aids (Rönnberg et al, 1998a; Andersson, Lyxell, Rönnberg
& Spens, 2001a,b), cochlear implants (Lyxell et al, 1996, 1998),
and visual speech understanding (review: Rönnberg, Samuelsson
& Lyxell, 1998c). Briefly, the predictors are: (1) phonology,
measured by rhyme tests (i.e. tapping the syllable level); (2)
lexical access speed; (3) verbal processing and storage capacity;
and (4) verbal inference-making. It is here ventured that
communicative proficiency mediated by such predictors, across
visual, non-auditory and alternative communication systems,
adds to the plausibility of a rather abstract design of a working-
memory system. It is clear that speed of lexical access, the nature
and quality of phonological representations, and sometimes
more complex information-processing indices, represent the
predictors both within and across sensory aid domains, and
sometimes also across pre- and post-training of speech-under-
standing performance (Rönnberg et al, 1998b).
3. Research on speech-reading expertise (Lyxell, 1994; Rönnberg,
1993; Rönnberg et al, 1999) shows that, despite differences in
onset of hearing impairment or deafness, and despite differences
in communicative habits, language use and strategy, there is
convergence on the fact that, for successful speech-readers,
bottom-up cognitive functions such as lexical-access speed and
phonology are relatively normal, whereas top-down cognitive
skills, such as complex storage and processing of information, as
well as inference-making, are exceptional compared to normal
participants. Thus, ‘higher-order’ cognitive and working-memory-
related skills are required for speech-understanding expertise,
beyond the threshold that is set by the lower-order functions
(Rönnberg et al, 1998a), and they operate independently of
communication mode.
Empirical arguments for the modality-free language
aspect of working memory
1. Lesion data and neurophysiological data seem to suggest
similarities in the ways in which left-hemisphere neural networks
are active during language processing. What can also be seen is
that the similarities hold true across different levels of language
and imaging technique (reviews: Rönnberg, Söderfeldt & Risberg,
2000; Wilson, 2001). As regards ‘inner speech’, it was found in a
recent imaging study that ‘inner signing’ of sentences engages
similar functional networks in the brain to those associated with
the activation of the phonological loop: left inferior frontal
cortex, rather than visuospatial areas (McGuire et al, 1997).
2. Additional neurophysiological evidence seems to support the
notion of an ‘abstract’ site for carrying phonological, syllable-
S70
like representations, i.e. planum temporale (Petitto et al, 2000).
The syllable may be a reasonably good approximation for the
unit that a cross-language working-memory system is based on.
This empirical finding undoubtedly fits with the linguistic
argument that there are organizational similarities between sign
and spoken syllables: positions and movements in sign corres-
pond to margins and nuclei in speech (Brentari, Poizner & Kegl,
1995).
3. Behavioural data also suggest rather strongly that there are
similarities across signed and spoken languages with respect to
‘phonological similarity’ effects in working memory (Wilson &
Emmorey, 1997), and with respect to the capacity of the ‘phono-
logical’ loop (Marschark & Mayer, 1998). Phonological loop
effects have also been observed with cued speech (Leybaert &
Lechat, 2001), and hence seem to constitute a rather general
operating characteristic of working memory for language. On
the whole, many classical working-memory effects are analogous
for sign and speech (i.e. similarity, length and suppression effects
(Wilson, 2001)).
Given the empirical arguments for the modality-free nature
of working memory in terms of neurophysiological, behavioural
and cognitive similarities—across speech communication modes
and languages—a tentative model can be formulated. The model
is abstract in the sense of modality and language independence,
and assumes that four parameters are especially crucial in
predictions of the ease with which speech and language can be
understood. As already indicated, it further assumes that
perceptual input channels, long-term memory and working-
memory interactions set the structural frame for interactions
among parameters.
Model parameters: across communication modes and
languages
1. The first assumption is that the interface between perception/
perceptual input systems and cognition for speech and sign
processing is at the sublexical, syllable level. The accuracy and
quality of phonological, syllable-like representations determine
parameter fp(P). This assumption is supported by both theory
and data. Four points can be adduced here, as follows.
(a) The assumption is founded on the facts that phonological
representations stored in long-term memory deteriorate over
time and are critical for initial processing of spoken, visual
language (i.e. indicated by a text-based, rhyme test (Lyxell,
Rönnberg & Samuelsson, 1994; Andersson, 2001)). Phone-
mically based processing in letter span tasks (i.e. tapping the
phonological loop in working memory) does not deteriorate
(Andersson, 2001). In addition, the interpretation of
correlations with this test is less clear, as the similarity index
was negatively correlated with visual speech understanding,
suggesting also a visual component (Andersson et al,
2001a). In a phonological lexical-access test, where the
subject is instructed to decide as quickly as possible whether
a non-word (sometimes a pseudohomophone) sounds like a
real word, processing is more explicitly geared toward a
phonemic analysis—otherwise the task cannot be solved
correctly (Andersson et al, 2001a). Again, the point here is
that this task is not as sensitive as a predictor of visual,
International Journal of Audiology, Volume 42 Supplement 1
 sentence-based speech understanding as the rhyme tasks.
One reason why the syllable level of phonology is important
may be that not only is initial phonemic information
involved in a rhyme test, but also the suprasegmental and
syllabic stress information are important (cf. the NAM
(Luce & Pisoni, 1998)). The latter aspects of the test relate to
intonation and prosody, at both the word and sentence levels
(Kjelgaard & Speer, 1999; Lindfield, Wingfield & Goodglass,
1999), which under any circumstances may represent com-
pensatory cues, when we know that certain phonemes are
not easily distinguished from lip movements only (Rönnberg
et al, 1998c).
(b) A further aspect of this assumption of syllabic primacy of
the conceptualized interface is that the phonological pro-
cessor must be relatively ‘abstract’; that is, it is assumed to
work with non-sound, tactile and visual input, as well as with
auditory input. When perceptually amalgamating tactile,
prosodic information with visual, lip-read information, it is
hard to see that the perceptual–cognitive bridge is phonemic
when the initial word information offered is less precise to
begin with (Rönnberg, 1993). Therefore, two alternative
interpretations remain: either visual–tactile perception works
because phonemic information is derived before lexical
access, as a result of some further preprocessing of
information, or it is assumed that sublexical, syllable-like
information is represented in long-term memory and is
sufficient to accomplish lexical access. As already implied, we
opt for the latter alternative, because it is more parsimonious
and functional, and in conjunction with script-related
constraints, speech understanding may run smoothly.
(c) The syllable assumption is also compatible with the idea
that syllables represent an important perceptual–cognitive
bridge for natural languages, but not necessarily for arte-
facts, like script. The written language bias is based on a
long history of research, e.g. on reading development. Here,
it is clear that phonemes are crucial, because phonemic tests,
but not rhyme tests, are crucial indices for beginning readers
(Höien, Lundberg, Stanovich & Bjaalid, 1995). However,
this is not necessarily true for dynamic, non-sound stimuli
that drive language-understanding processes.
(d) It is conceivable that the syllable is also an important unit of
analysis when it comes to possible generalization of this
model to signed languages. This possibility is supported by
some neurophysiological data on sign language (Petitto et
al, 2000).
In our view (Rönnberg et al, 1998a), then, the above data imply
that the brain has the capacity for a variety of ‘phonological’
codes—coding schemes that capitalize on a rapid linguistic
combinatorial capacity for sublexical units such as the syllable.
2. The second assumption is that long-term memory access
speed is important for perceptual decoding and lexical and
semantic access, giving parameter fs(S) (Pichora-Fuller,
Schneider & Daneman, 1995; Pichora-Fuller, this supplement;
Rönnberg, 1990; Tun & Wingfield, 1999).
Corollary: fp(P) and fs(S) are assumed to interact. The inference
is that the interactions typically take place in the implicit and
Cognition in the hearing impaired and deaf as a bridge between
signal and dialogue: a framework and a model
automatic, on-line mode of language processing. Given that the
input level of processing in the cognitive system is at the syllable
level, and that representations in long-term memory carry
sufficient syllable-like properties, then it is also possible to argue
that processing is implicit with that unit of analysis. This is
because rhyming skills are developed earlier than phonemic
skills (e.g. Wagner & Torgesen, 1987), and rhyming tasks are
typically executed implicitly rather than explicitly, compared to
other more analytical phonological lexical-access tasks
(Andersson et al, 2001a).
3. The mechanism whereby processing is switched over to an
explicit, controlled mode, fe(E), is assumed to be determined by a
general mismatch mechanism (Näätänen, 1986; Näätänen &
Escera, 2000)—a mechanism that is assumed to operate at two
levels of representation in long-term memory: The phonological
and the semantic script levels both contribute to lexical access,
and therefore set limits for language understanding in the
dialogue. When there is a mismatch signal from long-term
memory, working-memory resources are assumed to be allocated
to inference-making and disambiguation of the message.
(a) When lexical and semantic access are hindered by a less
favourable fp(P)fs(S) interaction, then explicit, controlled
processing resources need to be invoked. Put in other
words, when the perceived signals are too distorted (due to
the impairment or signal properties) to activate long-term
memory phonological representations, the probability of a
mismatch signal occurring is higher. For example, either
speed of processing hampers understanding at a rate suffici-
ent to keep up with a conversation, or poor phonological
representations, or the interaction between them (or inter-
action with the peripheral hearing loss in the case of
speech) affect lexical access to such an extent that lexical
entry and meaning of the message are lost. Hence, a
mismatch signal is produced.
(b) The expectancy or prediction given by the context and
discourse (Samuelsson & Rönnberg, 1993) is a powerful
determinant of visual speech understanding. For example, as
long as the materials are typical of a particular script or
communicative situation, activation of the lexicon proceeds
with sufficient efficiency, and, hence, processing proceeds
implicitly. When materials to be understood are below the
level of prediction (i.e. are less typical), the probability of a
mismatch signal increases. For materials at or above the level
of prediction, processing is assumed to proceed implicitly.
Corollary: Working memory is continuously testing predictions
and the quality/speed of phonological lexical access in its
interactions with long-term memory and perceptual input.
Processing is implicit as long as no mismatch occurs. Thus, the
system is conservative and economical in the sense that working-
memory resources should not be unnecessarily deployed. This
assumption is not made explicit in current models of working
memory (Richardson et al, 1996). When processing becomes
explicit, the degree of mismatch determines the parameter fe(E)
(Samuelsson & Rönnberg, 1993). The mismatch signal can be
derived from two sources: one phonological and the other
contextual, both relying on the relationship between long-term
memory representations and input signal. The exact weighting of
Rönnberg S71
the two sources of potential mismatch generally depends on the
quality of the input signal and contextual specification of script
use—in general, the mismatch depends on the way in which
bottom-up and top-down processing balance each other in the
language-understanding task. Experimentally, one could, for
example, keep a well-defined context, with highly predicted
materials, manipulating only the quality of the phonological
input to tease out the contribution of the long-term phonological
representations. The next logical step would be to offer less
specified contextual support to estimate parameter fe(E).
4. Given a high value of parameter fe(E), the assumption is that
explicit resources are invoked, named the capacity parameter,
fc(C), in working memory; when fc(C) is high, explicit process-
ing, inference-making and complex information processing are
made easier. fc(C) represents the general, amodal capacity of the
working-memory system. It is not tied to a specific loop
function, but represents a general processing and storage
capacity (Daneman & Merikle, 1996).
Corollary: The fp(P)fs(S) interaction sets the perceptual, implicit
platform (threshold) (Rönnberg et al, 1998c) for language
understanding, on the basis of which explicit resources may be
needed, for which the fe(E)fc(C) interaction represents an
important moderator function.
Expectations and applications
Based on the model, a few expectations and applications that
follow and are compatible with existing data are presented
below.
Across communication modes
1. Tests that assess phonological, syllable-based representations
in long-term memory should have a high predictive power for
speech understanding, irrespective of communicative form
(visual, auditory, or tactile, or combinations; see Andersson et al
(2001a,b) and Rönnberg et al (1998a,c).
2. Cognitive ageing affects the speed of information processing.
It can be expected that the fp(P)fs(S) interaction will be further
reinforced in elderly, hearing-impaired listeners/speech-readers
(Pichora-Fuller, this supplement).
3. The fe(E)fc(C) interaction can account for the facts that (a)
skilled speech-readers capitalize better on contextual constraints
(i.e. they have a higher fc(C) (Samuelsson & Rönnberg (1991)),
(2) younger listeners (compared to older listeners with similar
audiograms) do better in noise conditions (Pichora-Fuller et al,
1995; Tun & Wingfield, 1999), and (3) high-capacity subjects
adapt more easily to different processing strategies in hearing
aids (Lunner, this supplement).
4. One way of conceiving acclimatization effects in hearing aid
fitting, or with other sensory aids, is to assume that the overall
ratio between explicit fe(E) and implicit processes (fp(P)fs(S) is
reduced as acclimatization proceeds. Fc(C) may be tied to
further adaptation of the processing scheme in the hearing aid
(Lunner, this supplement). Procedures that vary the balance
between explicit and implicit processing need to be developed to
estimate these processes with standardized testing for a given
S72
type of hearing aid. The ratio may be a useful indicator for long-
term evaluations.
5. Various contextual effects determine working-memory pre-
dictions and the degree of mismatch: this holds true for dialogue
and experimental sentences; context and lexical activation also
interact, except for the phonology–lexical-access interaction.
Materials for assessing and estimating explicit and implicit
processes can be constructed and developed for different
communication modes.
Across languages
1. The existence of phonological problems in sign language
users is expected. For example, specific language impairments in
sign, with phonological, ‘non-word repetition’ problems, should
be observed (Bishop, North & Donlan, 1996; Briscoe, Bishop &
Norbury, 2001), particular phonological awareness problems in
‘phonological dyslexia’ (i.e. ordering and composition of
sublexical syllable-like units, problems with analysis of lexical
into sublexical units, and synthesis of sublexical units into
lexical units (Höien et al, 1995)), as well as the possibility that
phonological loop deficits in sign will determine lower rates of
acquisition of a foreign sign language vocabulary (Baddeley,
Gathercole & Papagno, 1988).
2. fs(S)fp(P) interactions in parkinsonian patients using sign
may demand high fc(C) estimates to allow fluency of dialogue
and to compensate for the ‘whispering’, slow sign production
that is typically employed by the parkinsonian signer (Kegl,
Cohen & Poizner, 1999).
3. fe(E)fc(C) interactions are expected in contextually un-
expected or poorly specified, signed situations, especially in
elderly signers.
4. Similar neuronal networks are expected to be activated for
signed and spoken working-memory tasks.
Discussion
In the overall evaluation of generality across speech com-
munication modes, points 1–3 hold up to experimental scrutiny,
whereas points 4 and 5 have not been properly assessed and
evaluated. When it comes to generality across languages, points
1, 3 and 4 have not been properly researched. While we await a
firmer basis for decision in this respect, it can be stated that the
model parameters put forward here are not crucially dependent
on modality specificity per se (Wilson, 2001). It is quite con-
ceivable that all parameters are equally applicable to both signed
and spoken languages.
However, there is also a set of modality-specific findings:
recall order effects are modality specific, spatial rehearsal and
an irrelevant-sign effect are de facto found for sign language
(Wilson, 2001), as well as modality-specific recency and recall
preferences (Rönnberg & Nilsson, 1987). There is sign-specific
activity (e.g. in the right hemisphere (Rönnberg et al, 2000)), as
well as other domain-specific working-memory effects that
oppose the simple, language-neutral, amodal view (Smith &
Jonides, 1997).
A compromise view at this early stage of testing of the
memory part of the model would be to use Baddeley’s recent
International Journal of Audiology, Volume 42 Supplement 1
development of the working-memory model, which is very clear
on the possibility of integration of sensory input in the episodic
buffer (Baddeley, 2000). One way of deciding the viability of the
opposing arguments would be to design experiments which
allow for modality-free and modality-specific classical working-
memory effects—across languages in bilingual individuals—and
to do this as a neuroscience, imaging project. Specificity versus
generality in working-memory representations would thus be
tested explicitly.
It should also be noted that current working-memory models
have not relied on the processing economy inherent in the
assumption of implicit processing, given a lack of mismatch
signals. For example, the models by Logie (1996) and Baddeley
(2000) seem to rely on automatic activation of both long-term
memory representations and corresponding modality-specific
‘slave systems’. This non-constrained activity in working
memory seems appropriate for memory and cognitive tasks as
such, but less so when cognition is used in time-pressed, on-line,
language-processing tasks. In the present model, selective
activation of working-memory resources is time-consuming and
should be done only when mismatch signals occur.
As a summary and stricter formulation, the model can also
be couched in mathematical terms, partly because it is quite
possible to estimate effect sizes through proper experimentation.
Thus, one needs to determine the function(s) that relate(s) ease
of language understanding (ELU) to functions of the four
(standardized) parameters involved: ELU refers primarily to
understanding messages at the sentence level, but is assumed to
be useful also for the discourse level of understanding. fp(P)
refers to the accuracy and quality of syllable-like representations
in long-term memory; fs(S) refers to long-term memory access
speed; the degree of mismatch, phonologically or semantically,
determines fe(E); and, finally, fc(C) represents the general
processing and storage capacity of working memory. Proper
estimation of ELU demands experimentation to evaluate
interactions among variables, as well as separate testing of
abilities, such as parameter fc(C). Thus, each parameter is in
turn a function of type of test, potential subcomponents, and
linear and non-linear interactions among subcomponents.
Nevertheless, one general point of departure is the funda-
mental relationship between the implicit and explicit com-
ponents in the general formulation of the model. The general
form may be stated as follows: ELU = [fp(P)fs(S)]/[fe(E)fc(C)].
When fe(E) is small, the three remaining individual parameters
point to a high ELU value. This represents the general case
when input is typical and expected and the perceptual
mechanism works with sufficient precision and automaticity in
relation to the degree of specification of the input. However, in
many instances, the above conditions do not hold true for the
hearing-impaired listener or language user. Therefore, it is
important to address the different conditions, or hypothetical
outcomes of principal interest, when the ratio between the
implicit and explicit components varies.
When [fp(P)fs(S)]/fe(E) is >1, assuming standardized vari-
ables, ELU is generally better than when [fp(P)fs(S)]/fe(E) < 1.
In addition, for low [fp(P)fs(S)]/fe(E) ratios, e.g. when fe(E) is
relatively high, it is expected that fc(C)—in accordance with the
general formulation—includes an added compensatory function
such that performance improves in a U-shaped fashion. Previous
case studies support such a supposition (Rönnberg et al, 1998c).
Cognition in the hearing impaired and deaf as a bridge between
signal and dialogue: a framework and a model
A further general restriction of the model is that this is only
possible given a sufficiently precise (but still distorted) per-
ceptual input; that is, we must assume a family of functions
where a minimal threshold (Mt) of fp(P)fs(S) is assumed (e.g.
Mt = 20% accuracy). For fp(P)fs(S) > Mt, the U-shaped function
is generally true, and for f(P)f(S) < Mt, the [f(P)f(S)]/f(E) ratio
gives essentially very small possibilities for compensation by
means of fc(C).
Furthermore, for intermediate levels of fp(P)fs(S) (Figure 1),
the most pronounced U-shaped, compensatory function is
expected for high-capacity individuals when the ratio of
[f(P)f(S)]/f(E) is allowed to vary from >1 to <1. This is assumed
to be possible because (1) there is still room for explicit
compensation when fc(C) is high, and (2) the phonological input
is sufficiently poor to trigger a mismatch signal—demanding a
higher fe(E)—but at the same time sufficiently precise to support
the further interaction with fc(C). For high levels of initial
fp(P)fs(S) (Figure 2), the variation of fc(C) will presumably—as
in the case for levels above, but close to an Mt of fp(P)fs(S)
(Figure 3)—result in a less pronounced U-function for high-
capacity individuals. In the high-fp(P)fs(S) case, this is because
there is less room for improvement; in the low-fp(P)fs(S) case,
this happens because there are very few elements on which to
base further processing. In addition, it may be the case that the
monotonic decrease in ELU for all levels of initial fp(P)fs(S) (as a
function of the decrease in ratio) vary as to when the interaction
with fc(C) starts. For high compared to low levels of initial
fp(P)fs(S), it is expected that processing is more resistant to
disruption; for example, it takes more noise to disrupt an initially
stable and implicit processing mode; that is, in principle, it takes
a lower fp(P)fs(S)/f(E) ratio to initiate the interaction with fc(C)
(Figure 2), whereas the reverse is assumed to hold true for low
levels of initial fp(P)fs(S) (Figure 3).
Testing and simulation of the model will have to entail
parameter estimation, standardization, and variation of the
actual functions relating ELU to the critical zones above and
below [fp(P)fs(S)]/fe(E) = 1, for different levels of fp(P)fs(S),
combined with a variation of fe(E)fc(C).
High
High C
ELU
Moderate
Low C
Low
>1 1 <1
[fp(P)fs(S)]/fe(E)
[fp(P)fs(S)]/fe(E)
Figure 1. Hypothetical ELU functions for high- and low-fc(C)
subjects, given an intermediate level of fp(P)fs(S) and a variation
of the [fp(P)fs(S)]/fe(E) ratios above and below 1.
Rönnberg S73
 High
High C
Low C
ELU
Moderate
Low
>1 1 <1
[fp(P)fs(S)]/fe(E)
[fp(P)fs(S)]/fe(E)
Figure 2. Hypothetical ELU functions for high- and low-fc(C)
subjects, given a high level of fp(P)fs(S) and a variation of the
[fp(P)fs(S)]/fe(E) ratios above and below 1.
Critical empirical tests of communication mode and language
specificity are needed to further shape the model (Wilson, 2001).
For example, the weighting of fp(P) and fs(S) may vary across
language, and the particular phonological tests used. Presently, it is
expected that syllable-based rhyming tests represent straightforward
independent tests that capture fp(P) (Andersson, 2001), lexical-
access speed is a good approximation of a general cognitive speed
component, fs(S) (Rönnberg, 1990; Pichora-Fuller, this supple-
ment; Tun & Wingfield, 1999), and the reading (or listening) span
test is a good first estimate of explicit capacity in general, fc(C) (e.g.
Lunner, this supplement; Rönnberg, 1995). fc(C) is also highly
correlated with the actual ability to make intelligent guesses and
linguistic inferences (Lyxell & Rönnberg, 1989). Such a test can be
developed with different languages and communication modes.
Conclusions
This article builds on the impressive empirical evidence,
behavioural and neurophysiological, that exists for modality-free
speech and language understanding. A framework, which
assumes a continuous interaction between perceptual input
channels, long-term memory, and working memory, was assumed
as a starting point for conceptualizing the bottlenecks that
determine the ease with which language can be understood,
spoken, or signed. The theoretical context of speech perception
models and working-memory models have been addressed.
On the basis of this general framework for language under-
standing, an explicit model was proposed. In brief, the model
assumes four interacting, modality-free parameters: phonology
(quality and precision) fp(P), speed fs(S), explicit processing
fe(E), and general storage and processing capacity fc(C) in
working memory. A general formalized description was also
proposed, where ease of language understanding ELU =
[fp(P)fs(S)]/[fe(E)fc(C)]. Hypothetical graphs were used to
illustrate some of the interacting features and limiting
conditions of the model. Applications and expectations across
speech communication modes and language were pointed out.
S74
High
ELU
Moderate
High C
Low Low C
>1 1 <1
[fp(P)fs(S)]/fe(E)
[fp(P)fs(S)]/fe(E)
Figure 3. Hypothetical ELU functions for high- and low-fc(C)
subjects, given a low level of fp(P)fs(S) and a variation of the
[fp(P)fs(S)]/fe(E) ratios above and below 1.
Experimentation, combined with neuroimaging, is needed to
evaluate the modality specificity of the model (e.g. comparing
working memory for sign and speech). Systematic manipul-
ation of parameters is needed to derive empirically founded
ELU functions, functions which may assist the development
of new tests and the future conceptualization of ‘cognitive’
sensory aids.
Acknowledgments
This research is supported by a grant from the Swedish Council
for Social Research (30305108).
References
Andersson, U. (2001). Cognitive deafness. The deterioration of
phonological representations in adults with an acquired severe
hearing loss and its implications for speech understanding.
Dissertation. Studies from the Swedish Institute for Disability
Research No. 3.
Andersson, U., Lyxell, B., Rönnberg, J. & Spens, K.-E. (2001a).
Cognitive predictors of visual speech understanding. Journal of
Deaf Studies and Deaf Education, 6, 116–129.
Andersson, U., Lyxell, B., Rönnberg, J. & Spens, K.-E. (2001b). A
follow-up study on the effects of speech tracking training on visual
speechreading of sentences and words: cognitive prerequisites and
chronological age. Journal of Deaf Studies and Deaf Education, 6,
103–116.
Baddeley, A.D. (2000). The episodic buffer: a new component of
working memory? Trends in Cognitive Neuroscience, 4, 417–423.
Baddeley, A., Gathercole, S. & Papagno, C. (1998). The phonological
loop as a language learning device. Psychological Review, 105,
158–173.
Baddeley, A.D. & Hitch, G. (1974). Working memory. In G.A. Bower
(Ed.), The psychology of learning and motivation (pp. 47–89).
London: Academic Press.
Bernstein, L. & Auer, E. (2003). Speech perception and spoken word
recognition. In: M. Marschark & P.E. Spencer (Eds.), The handbook
of deaf studies, language, and education (pp. 379–391). Oxford:
Oxford University Press.
Bernstein, L.E., Demorest, M.E. & Tucker, P.E. (2000). Speech perception
without hearing. Perception & Psychophysics, 62, 233–252.
International Journal of Audiology, Volume 42 Supplement 1
Bishop, D.V.M., North, T. & Donlan, C. (1996). Nonword repetition as
a behavioural marker for inherited language impairment: evidence
from a twin study. Journal of Child Psychology and Psychiatry and
Allied Disciplines, 37, 391–403.
Brentari, D., Poizner, H. & Kegl, J. (1995). Aphasic and Parkinsonian
signing: differences in phonological disruption. Brain and Language,
48, 69–105.
Briscoe, J., Bishop, D.V.M. & Norbury, C.F. (2001). Phonological
processing, language, and literacy: a comparison of children with
mild-to-moderate sensorineural hearing loss and those with specific
language impairment. Journal of Child Psychology and Psychiatry
and Allied Disciplines, 42, 329–340.
Calvert, G., Bullmore, E., Brammer, M., Campbell, R., Woodruff, P.,
McGuire, P., et al (1997). Activation of auditory cortex during silent
speechreading. Science, 276, 593–596.
Cowie, R.E. & Douglas-Cowie, E. (1992). Postlingually acquired
deafness: speech deterioration and the wider consequences. New
York: Mouton de Gruyter.
Daneman, M. & Merikle, P.M. (1996). Working memory and language
comprehension: a meta-analysis. Psychonomic Bulletin & Review,
3(4), 422–433.
Gailey, L. (1987). Psychological parameters of lipreading skill. In B.
Dodd & R. Campbell (Eds.), Hearing by eye: the psychology of
lipreading (pp. 115–141). London: Lawrence Erlbaum.
Giraud, A.-L., Price, C.J., Graham, J.M., Truy, E. & Frackowiak, R.S.J.
(2001). Cross-modal plasticity underpins language recovery after
cochlear implantation. Neuron, 30, 657–663.
Giraud, A.-L., Truy, E., Frackowiak, R.S.J., Grégoire, M.-C., Pujol, J.-F.
& Collet, L. (2000). Differential recruitment of the speech
processing system in healthy subjects and rehabilitated cochlear
implant patients. Brain, 123, 1391–1402.
Höien, T., Lundberg, I., Stanovich, K.E. & Bjaalid, I.-K. (1995).
Components of phonological awareness. Reading and Writing, 7,
171–188.
Jeffers, J. & Barley, M. (1971). Speechreading. Springfield, IL: Charles
C. Thomas Publishers.
Just, M.A. & Carpenter, P.A. (1992). A capacity theory of compre-
hension—individual differences in working memory. Psychological
Review, 99, 122–149.
Kegl, J., Cohen, H. & Poizner, H. (1999). Articulatory consequences of
Parkinson’s disease: perspectives from two modalities. Brain and
Cognition, 40, 355–386.
Kjelgaard, M.M. & Speer, S.H. (1999). Prosodic facilitation and
interference in the resolution of temporary syntactic closure
ambiguity. Journal of Memory & Language, 40, 153–194.
Larsby, B. & Arlinger, S. (1994). Speech recognition and just-follow-
conversation tasks for normal-hearing and hearing-impaired
listeners with different maskers. Audiology, 33, 165–176.
Leybaert, J. (1998). Phonological representations in deaf children: the
importance of early linguistic experience. Scandinavian Journal of
Psychology, 39, 169–173.
Leybaert, J. & Lechat, J. (2001). Phonological similarity effects in
memory for serial order of cued speech. Journal of Speech, Language,
and Hearing Research, 44, 949–963.
Levänen, S. (1998). Neuromagnetic studies of human auditory cortex
function and reorganisation. Scandinavian Audiology, 27(suppl 49),
1–6.
Lindfield, K.C., Wingfield, A. & Goodglass, H. (1999). The role of
prosody in the mental lexicon. Brain & Language, 68, 312–317.
Logie, R.H. (1996). The seven ages of working memory. In J.T.E.
Richardson, R.W. Engle, L. Hasher, R.H. Logie, E.R. Stolzfus &
R.I. Zacks (Eds.), Working memory and human cognition. Oxford:
Oxford university Press.
Ludman, C.N., Summerfield, A.Q., Hall, D., Elliott, M., Foster, J.,
Hykin, J.L., et al (2000). Lip-reading ability and patterns of cortical
activation studied using fMRI. British Journal of Audiology, 34,
225–230.
Luce, P.A. & Pisoni, D.A. (1998). Recognising spoken words: the
neighborhood activation model. Ear & Hearing, 19, 1–36.
Lyxell, B. (1994). Skilled speechreading—a single case study.
Scandinavian Journal of Psychology, 35, 212–219.
Lyxell, B., Arlinger, S., Andersson, J., Bredberg, G., Harder, H. &
Rönnberg, J. (1998). Phonological representation and speech under-
Cognition in the hearing impaired and deaf as a bridge between
signal and dialogue: a framework and a model
standing with cochlear implants in deafened adults. Scandinavian
Journal of Psychology, 39, 175–179.
Lyxell, B., Arlinger, S., Andersson, J., Harder, H., Näsström, E.,
Svensson, H., et al (1996). Information-processing capabilities and
cochlear implants: pre-operative predictors for speech understand-
ing. Journal of Deaf Studies & Deaf Education, 1, 190–201.
Lyxell, B. & Rönnberg, J. (1989). Information-processing skills and
speechreading. British Journal of Audiology, 23, 339–347.
Lyxell, B., Rönnberg, J. & Samuelsson, S. (1994). Internal speech
functioning and speechreading in deafened and normal hearing
adults. Scandinavian Audiology, 23, 181–185.
MacSweeney, M., Campbell, R., Calvert, G.A., McGuire, P.K., David,
A.S., Suckling, J., et al (2001). Dispersed activation in the left
temporal cortex for speech-reading in congenitally deaf people.
Proceedings of the Royal Society of London Series B—Biological
Sciences, 268, 451–457.
Markova, I. & Linell, P. (1996). Coding elementary contributions to
dialogue: individual acts versus dialogical interactions. Journal for
the Theory of Social Behavior, 26, 353–373.
Marschark, M. & Mayer, T.S. (1998). Interactions of language and
memory in deaf children and adults. Scandinavian Journal of
Psychology, 39, 145–148.
Marslen-Wilson, W. (1987). Functional parallelism in spoken word
recognition. Cognition, 25, 71–103.
McClelland, J.L. & Elman, J.L. (1986). The TRACE model of speech
perception. Cognitive Psychology, 18, 1–86.
McGuire, P.K., Robertson, D., Thacker, A., David, A.S., Kitson, N.,
Frackowiak, R.S.J., et al (1997). Neural correlates of thinking in
sign language. Neuroreport, 8, 695–698.
Näätänen, R. (1996). Neurophysiological basis of the echoic memory as
suggested by event-related potentials and magnetoencephalogram.
In F. Klix & H. Hagendorf (Eds.), Human memory and cognitive
capabilities (pp. 615–628). Amsterdam: North Holland.
Näätänen, R. & Escera, C. (2000). Mismatch negativity: clinical and
other applications. Audiology and Neuro-otology, 5, 105–110.
Petitto, L.A., Zatorre, R.J., Gauna, K., Nikelski, E.J., Dostie, D. &
Evans, A.C. (2000). Speech-like cerebral activity in profoundly deaf
people processing signed languages: implications for the neural basis
of human language. Proceedings of the National Academy of
Sciences of the United States of America, 97(25), 13961–13966.
Pichora-Fuller, M.K., Schneider, B.A. & Daneman, M. (1995). How
young and old adults listen to and remember speech in noise.
Journal of the Acoustical Society of America, 97, 593–608.
Richardson, J.T.E., Engle, R.W., Hasher, L., Logie, R.H., Stoltzfus, E.R.
& Zacks, R.T. (1996). Working memory and human cognition.
Oxford: Oxford university Press.
Rönnberg, J. (1990). Cognitive and communicative function: the effects
of chronological age and ‘handicap age’. European Journal of
Cognitive Psychology, 2, 253–273.
Rönnberg, J. (1993). Cognitive characteristics of skilled tactiling: the
case of GS. European Journal of Cognitive Psychology, 5, 19–33.
Rönnberg, J. (1995). What makes a skilled speechreader? In G. Plant &
K. Spens (Eds.), Profound deafness and speech communication (pp.
393–416). London: Whurr Publishers.
Rönnberg, J. (2003). Working memory, neuroscience, and language:
evidence from the deaf and hard of hearing individuals. In M.
Marschark & P.E. Spencer (Eds.), The handbook of deaf studies,
language and education (pp. 478–489). Oxford: Oxford University
Press.
Rönnberg, J., Andersson, J., Andersson, U., Johansson, K., Lyxell, B. &
Samuelsson, S. (1998b). Cognition as a bridge between signal and
dialogue: communication in the hearing impaired and deaf.
Scandinavian Audiology, 27(suppl 49), 101–108.
Rönnberg, J., Andersson, U., Lyxell, B., & Spens, K. (1998a). Vibro-
tactile speechreading support: cognitive prerequisites for training.
Journal of Deaf Studies & Deaf Education, 3, 143–156.
Rönnberg, J., Andersson, J., Samuelsson, S., Söderfeldt, B., Lyxell, B. &
Risberg, J. (1999). A speechreading expert: the case of MM. Journal
of Speech, Language and Hearing Research, 42, 5–20.
Rönnberg, J. & Nilsson, L-G. (1987). The modality effect, sensory handi-
cap, and compensatory functions. Acta Psychologica, 65, 263–283.
Rönnberg, J., Samuelsson, S. & Lyxell, B. (1998c). Conceptual
constraints in sentence-based lipreading in the hearing impaired. In
Rönnberg S75
 R. Campbell, B. Dodd & D. Burnham (Eds.), Hearing by eye: Part
II. Advances in the psychology of speechreading and audiovisual
speech (pp. 143–153). London: Lawrence Erlbaum Associates.
Rönnberg, J., Söderfeldt, B. & Risberg, J. (2000). The cognitive neuro-
science of signed language. Acta Psychologica, 105, 237–254.
Samuelsson, S. & Rönnberg, J. (1991). Script activation in lipreading.
Scandinavian Journal of Psychology, 32, 124–143.
Samuelsson, S. & Rönnberg, J. (1993). Implicit and explicit use of
scripted constraints in lipreading. European Journal of Cognitive
Psychology, 5, 201–233.
Smith, E.E. & Jonides, J. (1997). Working memory: a view from neuro-
imaging. Cognitive Psychology, 33, 5–42.
Tun, P.A. & Wingfield, A. (1999). One voice too many: adult age differ-
S76
ences in language processing with different types of distracting sounds.
Journal of Gerontology B Psychological Sciences, 54, 317–327.
Wagner, R.K. & Torgesen, J.K. (1987). The nature of phonological
processing and its causal role in the acquisition of reading skills.
Psychological Bulletin, 101, 192–212.
Wilson, M. (2001). The case for sensorimotor coding in working
memory. Psychonomic Bulletin & Review, 8, 44–57.
Wilson, K & Emmorey, K. (1997). Working memory for sign language: a
window into the architecture of the working memory system.
Journal of Deaf Studies and Deaf Education, 2, 121–130.
Zatorre, R.J. (2001). Do you see what I’m saying? Interactions between
auditory and visual cortices in cochlear implant users. Neuron, 31,
13–14.
International Journal of Audiology, Volume 42 Supplement 1