Professional Documents
Culture Documents
Chapter 7
Human Behaviour
Recognition in Ambient
Intelligent Environments
Hans W. Guesgen
Massey University, New Zealand
Stephen Marsland
Massey University, New Zealand
ABSTRACT
The recognition of human behaviour from sensor observations is an important area of research in smart
homes and ambient intelligence. In this chapter, the authors introduce the idea of spatio-temporal foot-
prints, which are local patterns in space and time that should be similar across repeated occurrences
of the same behaviour. They discuss the spatial and temporal mapping requirements of these footprints,
together with how they may be used. As possible formalisms for implementing spatio-temporal footprints,
the authors discuss and evaluate probability theory, fuzzy sets, and the Dempster-Shafer theory.
Copyright 2015, IGI Global. Copying or distributing in print or electronic forms without written permission of IGI Global is prohibited.
Human Behaviour Recognition in Ambient Intelligent Environments
2009; Duong, Bui, Phung, & Venkatesh, 2005; that some behaviours repeat more-or-less identi-
Gopalratnam & Cook, 2004; Rivera-Illingworth, cally over the three days, while others only occur
Callaghan, & Hagras, 2007; Tapia, Intille, & once. The challenge with such representation of
Larson, 2004). Although the reported successes behaviour as footprints is to identify and recognise
are promising, it has become clear that all ap- the various behaviours that are represented.
proaches fall short of being perfect. Due to the The rest of this article investigates the role
limited information that is in the sensor data, of spatio-temporal footprints in more detail. We
noise, and the inherently complexity of human start with a discussion of what space-time means
behaviours, it is often impossible to determine in the context of smart homes, arguing that there
the correct behaviour from the sensor data alone, is more than one space-time (or more precisely,
in particular if behaviours are overlapping or are representation of space-time). We then look at
being executed by more than one person. how behaviours leave footprints in space-time
Several researchers have realised that ad- and explore invariants in these footprints, with
ditional information can be useful to boost the the goal of classifying different forms of invari-
behaviour recognition process (Aztiria, Augusto, ants and looking into how the footprints are dis-
Izaguirre, & Cook, 2008; Jakkula & Cook, 2008; tributed in space-time and what influences this
Tavenard, Salah, & Pauwels, 2007). In this article, distribution. Finally we discuss the advantages
we focus on how spatio-temporal information, and disadvantage of using fuzzy sets, respectively
enriched with context information, can be used for the Dempster-Shafer theory, instead of probabil-
this purpose. When a particular activity occurs, ity theory to capture the idea of spatio-temporal
like preparing breakfast, it leaves a footprint footprints.
in space-time, i.e., a particular pattern of sen-
sor observations in some set of locations over
some period of time. The activity starts at some SPACE-TIME
specific time and in some specific location, goes
on for a specific duration in some specific area, When reasoning about time, we usually associate
and terminates at a specific time at some specific a time axis with the data. The time axis might use
location. Since footprints differ from behaviour a calendar as reference system and absolute dates/
to behaviourbut often relatively little between times to refer to points on the axis. Or it might
different instances of the same behaviourwe use some artificial start point as zero time, such
can use these to inform the behaviour recognition as the time when the smart home became opera-
process: if something is happening at 07:00 in the tional, and some counter to advance time along
kitchen, it is more likely to be preparing breakfast the time axis. In the latter case, we would not be
than taking a shower. We can also use them to able to refer back to times before the birth of the
detect abnormal behaviour: if the inhabitant of smart home, while the first case would provide
the smart home uses the shower at 03:00 (when an infinite extension of time into both the past
usually this is not a footprint that is seen), then and the future.
this can be interpreted as abnormal. As we will see later when discussing footprint
Figure 1 shows an example of a possible set of invariants, it may sometimes be advantageous to
footprints over three days, with a linear time axis view the time axis as a circular reoccurrence of
that repeats each day, and a single space axis that time points. For example, if we are only interested
could identify rooms, or similar (this is discussed in when behaviours occur during the day, then we
in more detail in the next section). It can be seen might want to abstract from years, months, and
108
Human Behaviour Recognition in Ambient Intelligent Environments
days, which would leave us with references to space, but not in time), it has many similarities to
times of the day. At the end of the day, we would time. In the simplest case, it has the same dimen-
warp time and start at the beginning of the time sionality, for example if we can move only along
axis again. a predefined trajectory (Mukerjee & Joe, 1990).
Independently of whether we use linear or The trajectory can be viewed as continuous, in
circular time, we still have to decide whether we which case we would associate the distance from
view time as continuous or discrete. For example, the origin with locations on the spatial axis, or it
when referring to 13:00, do we really mean this can be discrete, in which case we need a way to
exact point on a continuous time axis? If this is associate distances with reference points on the
the case, then a behaviour that occurs one mil- spatial axis.
lisecond after this time point would not match As with time, we can envision different rep-
13:00, unless we allow for fuzzy matches. On resentations of space. Not only can we extend
the other hand, if we view it as a discrete time the dimensionality of space to two or three, we
stamp, surrounded by, say, 12:55 and 13:05, then can also move away from a canonical Euclidean
it would make sense to associate the behaviour space to a more abstract space. For example, we
with 13:00 rather than 12:55 or 13:05. This can can use the rooms of the smart home to define
be thought of as temporal resolution. space points, or the areas covered by the sensors
Similar considerations can be made when re- of the smart home. Given some knowledge of
ferring to space. Although space is more complex the physical locations of sensors, the house can
than time (partially because we can move freely in infer that when sensor events occur, the house
109
Human Behaviour Recognition in Ambient Intelligent Environments
inhabitant must be in the physical vicinity of that As an example of the first of these, consider
sensor (obviously, there are exceptions to this the case that breakfast always takes place in the
for certain sensors such as thermometers and for kitchen at 08:00. Then the mapping between this
remote-controlled or time-controlled devices). footprint on different days is simply the identity,
The spatial pattern could then be some mapping as it leaves the same footprint in space-time each
between sensor locations, which could be based time it occurs. Of course, this assumes that we
on the underlying physical layout of the house, have chosen a suitable representation for space-
but does not have to be. time (e.g., one that only looks at the time of the
The importance of this is that different resolu- day in a discrete way and uses the rooms of the
tions can make recognition of particular footprints smart home as spatial entities). It is unlikely that
easier or more difficult. As the resolution becomes we observe this invariant very often, as it would
finer, footprints that appear to be exactly aligned require very rigid patterns of behaviour and a
start to separate, meaning that identifying them relatively abstract form of space-time.
as examples of the same behaviour can require a The next type of transformation is that where
clustering algorithm. Breakfast might not occur we match patterns despite shifts in space-time.
exactly at 08:00 each day, but it is likely that it Here we can distinguish among three different
happens at times that we associate with morn- types of shift:
ing. Or reading a book might always occur in the
lounge, but sometimes while sitting on the sofa Shifts on the time axis only,
and sometimes while lying back in an easy chair. Shifts on the space axes, and
Shifts on all axes.
110
Human Behaviour Recognition in Ambient Intelligent Environments
There is one example of footprint change that closely related to the one outlined above. Similar
we might need to be particularly careful about, discussions can be found around the region con-
which is the deformation of footprints in space- nection calculus, which is used for reasoning about
time. This again might be restricted to particular spatial relations (Randell, Cui, & Cohn, 1992).
axes such as time, or might include all axes. We Thus, we are lead to consider two related
start the discussion of this by restricting ourselves methods of footprint identification: identifying
to the time axis. The projection of a footprint the same footprint occurring in different places
onto the time axis is a time interval, consisting in space-time, and distinguishing between dif-
of a start point and an end point. We can restrict ferent footprints that occur at the same place in
the deformation of the time interval to one of its space-time.
boundaries, either the start or the end point, but not
both. An example would be breakfast that always
starts at 08:00 but might last between 10 and 20 FOOTPRINT PROBABILITY
minutes, or doing the dishes after dinner might DISTRIBUTIONS
start at any time, but always finishes in time to be
ready for the soap opera on TV at 18:00. If we do The examples of the previous sections indicate
not restrict ourselves to one of the boundaries, we that footprints are not necessarily distributed
obtain a deformation which (more or less) keeps evenly in space-time. A straightforward way to
the centre of the interval invariant. For example, find the distribution of footprints is to empirically
taking a shower might always occur at around approximate for each behaviour the probability of
20:00, starting about 10 minutes before that time a footprint occurring at a particular location (or
and ending about 10 minutes after that time. within a particular cluster) in space-time. This
This same effect can be seen for space. In this does not require any extra knowledge about the be-
case, we have to consider boundaries of regions in haviour or about particular regions in space-time,
(usually) multi-dimensional space rather than start but it does require enough data to approximate
and end points of time intervals, which means that the probabilities within reasonable error margins.
instead of considering two classes of deformation As an alternative to this approach, we can anal-
(one boundary vs. two boundaries), we have to yse the regions occupied by a particular behaviour
consider an infinite number of classes. One way in space-time in order to find the distributions of the
to achieve this is by determining the percentage corresponding footprints. Breakfast on weekends
of the boundary that stays the same and associat- as opposed to weekdays is an example for that, as
ing the behaviours with a finite set of classes that illustrated previously. Obviously, the footprints are
are given by a range of percentages. For example, not distributed evenly over the two clusters, but
we might want to distinguish just between those have a higher density in the weekday breakfast
deformations that change more than 50% of the cluster than in the weekend breakfast cluster. We
boundary and those that keep at least 50% of the know that there are five weekdays per week, but
boundary invariant. only two weekend days (for the average working
It should be noted that our discussion of foot- person). Assuming that the breakfast behaviour
print invariants is closely related to the discussion occurs exactly once per day, we conclude that
around neighbourhood graphs in (Freksa, 1992), the probability of a breakfast footprint being in
where Freksa introduces three forms of neighbour- the weekend cluster of breakfast footprints is 2/7,
hood graphs for Allens temporal logic (Allen, whereas the probability of it being in the weekday
1983). The graphs are based on shifts and two one is 5/7. Taking this approach a step further, we
forms of deformations of time interval, in a way can then compute conditional probabilities, which
111
Human Behaviour Recognition in Ambient Intelligent Environments
give us further insides into where a footprint is understanding of the relationship between be-
located in space-time. For example, if we know haviours and their space-time footprints, it is not
that a behaviour occurred on a weekday, then the practical to use context information excessively.
conditional probability of it being in the weekday Each bit of information requires us to explicitly
cluster is 1 (and 0 for the weekend cluster). We will model the correlation between the information
discuss this approach further in the next section. and its impact on the behaviourfootprint rela-
In general, this leads to an approach where tion, which requires a significant amount of world
context information is taken into consideration knowledge or data. In other words, we trade off the
when behaviours are related to space-time foot- need for sufficient training data against the need
prints. In the example above, the context is of a for explicit modelling. In the next sections, we
spatio-temporal nature, but this does not have to discuss the pros and cons of some formalisms that
be the case, as it can make sense to utilise other can be used to model spatio-temporal footprints.
types of context information as well, such as:
112
Human Behaviour Recognition in Ambient Intelligent Environments
behaviours occurring in particular contexts), this Rather than asking the question of what is the
may lead to counterintuitive results. For example, probability of a certain behaviour occurring in a
if we do not have any information that a behaviour particular context, we now pose the question as fol-
normally occurs in the kitchen, we cannot assign lows. Given some context information C, to which
a probability of 0 to P(B|C), since this would degree is a particular behaviour a C-behaviour?
imply that P(B|C) equals 1, which would mean For example, if C is the day of the week, then we
that the behaviour happens outside the kitchen (if can ask for the degree of the behaviour to be a
it happens at all). Monday behaviour, Tuesday behaviour, and so
The Dempster-Shafer theory (Dempster, 1967; on. In terms of fuzzy sets, we define D as the
Shafer, 1976) offers a way out of this dilemma. It set of the seven days of the week and as the
uses the concepts of belief (Bel) and plausibility membership function that determines to which
(Pl) instead of probability to formulate uncertainty, degree the behaviour occurs on a particular day.
where classical probability lies between belief For example, we might want to define a fuzzy
and plausibility: set that reflects the degree to which a restaurant
visit falls on a particular day. The membership
Bel(B|C) P(B|C) Pl(B|C) function of such a fuzzy set is shown graphically
in Figure 2. Unlike probabilities, the membership
Although this solves the problem of evidence grades do not need to add up to one.
not adding up to 1, it still leaves us with the prob- In the example above, the context information
lem of assigning appropriate basic probabilities is still crisp information, despite the fact that it is
to all subsets of the frame of discernment, which used in a fuzzy set: for any restaurant visit, we can
might not be possible in real-world scenarios. A determine precisely on which day of the week it
qualitative approach helps to mitigate this problem. occurs. Other context information might not be
We introduce such an approach in the next sec- precise, but rather conveys some vague informa-
tion, but before that we take a detour and discuss tion. For example, if we know that a behaviour
a simpler framework, which is not as rigorous occurs near the kitchen, we usually do not know
from the mathematical point of view, but which exactly how many metres away from the kitchen
provides a simple and robust way to deal with the behaviour occurs. In this case, we can repre-
context information. This approach is based on sent the context information itself as a fuzzy set,
fuzzy sets. as illustrated in Figure 3.
Unlike traditional sets, fuzzy sets allow their Similarly, we can define a fuzzy set that ex-
elements to belong to the set with a certain de- presses distances by rounding them to the closest
gree. Rather than deciding whether an element d half metre something we as humans often do
does or does not belong to a set A of a domain D, when we perceive distances, although not neces-
we determine for each element of D the degree sarily always on the same scale (see Figure 4).
with which it belongs to the fuzzy set . In other As the examples have shown, fuzzy sets can
words, a fuzzy subset of a domain D is a set of be used for associating behaviours with context
ordered pairs, (d, (d)), where d D and : D information and for representing imprecise context
[0, 1] is the membership function of . The information. Fuzzy set theory also provides us with
membership function replaces the characteristic a means to convert fuzzy sets back to crisp sets,
function of a classical subset A D. which is achieved with the notion of an -level
set. Let be a fuzzy subset in D, then the (crisp)
113
Human Behaviour Recognition in Ambient Intelligent Environments
Figure 2. Graphical representation of a membership function that determines the degree to which a
restaurant visit falls on a particular day of the week
set of elements that belong to the fuzzy set If the membership grade is strictly greater than
with a membership grade of at least is called , then the set is referred to as a strong -level set.
the -level set of : To reason about fuzzy sets, we adopt one of
the schemes for combining fuzzy sets that was
A = {d D | (d) } originally proposed by Zadeh (1965). Given two
fuzzy sets 1 and 2 with membership functions
Figure 3. A fuzzy set that maps distances to the qualitative values very near, near, far, and very far
114
Human Behaviour Recognition in Ambient Intelligent Environments
Figure 4. A fuzzy set that approximates distances with a granularity of half a metre
1(d) and 2(d), respectively, then the member- Kreinovich, & Tolbert, 1993) proved that the min/
ship function of the intersection 3 = 1 2 is max operations are the most robust operations for
pointwise defined by: combining fuzzy sets, where robustness is defined
in terms of how much impact uncertainty in the
3(d) = min{1(d), 2(d)} input has on the error in the output.
Regardless of which combination scheme is
Analogously, the membership function of the used, determining the fuzzy sets themselves is not
union 3 = 1 2 is pointwise defined by: a trivial exercise. There have been proposals to
learn the fuzzy sets, mainly in the area of fuzzy
3(d) = max{1(d), 2(d)} control (Berenji & Khedkar, 1992; Nauck & Kruse,
1993; Van Cleave & Rattan, 2000). Adopting these
The membership grade for the complement approaches for human behaviour recognition and
of a fuzzy set , denoted as , is defined in the context-awareness is a possibility, but it would
same way as the complement in probability theory: require a large amount of data. For that reason,
explicit modelling seems to be the best option.
(d) = 1 (d) However, this requires a good understanding of
when and where particular behaviours occur, and
Zadeh (1965) stresses that the min/max combi- that depends to a large extent on who exhibits the
nation scheme is not the only scheme for defining behaviours.
intersection and union of fuzzy sets, and that it
depends on the context which scheme is the most
appropriate. While some of the schemes are based
on empirical investigations, others are the result
of theoretical considerations (Dubois & Prade,
1980; Klir & Folger, 1988). However, (Nguyen,
115
Human Behaviour Recognition in Ambient Intelligent Environments
QUALITATIVE DEMPSTER-
SHAFER THEORY
[m1 m2 ](X) =
VW=Xm1(V) m2 (W)
Before discussing a qualitative version of the 1 m1 (V) m2 (W)
V W=
Dempster-Shafer theory, we briefly review the
main concepts of the theory, which uses the no-
tions of belief (Bel) and plausibility (Pl) instead for X
of probability to formulate uncertainty. Belief and
plausibility are defined in the Dempster-Shafer [m1 m2 ]() = 0
theory on the basis of a mass assignment func-
tion, which assigns basic probabilities to any set
This assumes that the weight of conflict is not
that is a member of the power set of the universe
equal to 1, which means:
U (which is also called the frame of discernment
or set of hypotheses):
v w =
m1 (V )m2 (W) 1
m: 2 [0,1]
U
116
Human Behaviour Recognition in Ambient Intelligent Environments
Table 1. Combining mass assignment functions Table 2. Combining mass assignment functions
for the context example for the context example
117
Human Behaviour Recognition in Ambient Intelligent Environments
118
Human Behaviour Recognition in Ambient Intelligent Environments
is possible. This means that we need to adapt Berenji, H., & Khedkar, P. (1992). Learning and
the system to each person. If we have sufficient tuning fuzzy logic controllers through reinforce-
knowledge about this person, we might be able ments. IEEE Transactions on Neural Networks,
to explicitly model the persons behaviours and 3(5), 724740. PMID:18276471
tie them in with context information. In this case,
Chua, S.-L., Marsland, S., & Guesgen, H. W.
fuzzy sets might be the appropriate approach, in
(2009). Spatio- temporal and context reasoning
particular if we have some guarantee of robustness
in smart homes. Paper presented at the COSIT-09
(through the appropriate choice of combination
Workshop on Spatial and Temporal Reasoning
schemes) so that any mistakes that we make do
for Ambient Intelligence Systems. Aber Wrach,
not have a significant impact on the correctness
France.
of the system.
The question, of course, then is whether such Cook, D. (2006). Health monitoring and assistance
a scenario is realistic. If we aim at an out-of-the- to support aging in place. Journal of Universal
box system that is easily available from retailers, Computer Science, 12(1), 1529.
then the expectation most likely will be that such
Dempster, A. (1967). Upper and lower probabili-
a system can be set up easily and without a large
ties induced by a multivalued mapping. Annals of
amount of pre-knowledge. Ideally, the system
Mathematical Statistics, 38(2), 325339.
should learn and adapt automatically to the be-
haviours of a particular person. A probabilistic Dubois, D., & Prade, H. (1980). Fuzzy Sets and
approach, either in its nave form or in the form Systems: Theory and Applications. London:
of the Dempster-Shafer theory, seems to be the Academic Press.
obvious choice in this case. However, such an
Duong, T., Bui, H., Phung, D., & Venkatesh, S.
approach requires a large amount of data which
(2005). Activity recognition and abnormality
we first have to acquire.
detection with the switching hidden semi-Markov
model. In Proceedings of IEEE Computer Soci-
ety Conference on Computer Vision and Pattern
REFERENCES
Recognition (pp. 838845). Los Alamitos, CA:
Allen, J. (1983). Maintaining knowledge about IEEE Computer Society.
temporal intervals. Communications of the ACM, Freksa, C. (1992). Temporal reasoning based
26, 832843. on semi-intervals. Artificial Intelligence, 54,
Augusto, J., & Nugent, C. (2004). The use of 199227.
temporal reasoning and management of complex Gopalratnam, K., & Cook, D. (2004). Active LeZi:
events in smart homes. In Proceedings of the 16th An incremental parsing algorithm for sequential
European Conference on Artificial Intelligence prediction. International Journal of Artificial
(pp. 778782). Amsterdam, The Netherlands: Intelligence Tools, 14(12), 917930.
IOS Press.
Jakkula, V., & Cook, D. (2008). Anomaly detec-
Aztiria, A., Augusto, J., Izaguirre, A., & Cook, tion using temporal data mining in a smart home
D. (2008). Learning accurate temporal relations environment. Methods of Information in Medicine,
from user actions in intelligent environments. In 47(1), 7075. PMID:18213431
Proceedings of the 3rd Symposium of Ubiqui-
tous Computing and Ambient Intelligence (pp.
274283). Berlin, Germany: Springer.
119
Human Behaviour Recognition in Ambient Intelligent Environments
Kiecolt-Glaser, J., & Glaser, R. (1999). Chronic Shafer, G. (1976). A Mathematical Theory of Evi-
stress and mortality among older adults. Jour- dence. Princeton, NJ: Princeton University Press.
nal of the American Medical Association, 282,
Tapia, E., Intille, S., & Larson, K. (2004). Ac-
22592260. PMID:10605979
tivity recognition in the home using simple and
Klir, G., & Folger, T. (1988). Fuzzy Sets, Uncer- ubiquitous sensors. In D. Hutchison, T. Kanade,
tainty, and Information. Englewood Cliffs, NJ: J. Kittler, J. M. Kleinberg, A. Kobsa, F. Mattern,
Prentice Hall. ... G. Weikum (Eds.), Pervasive Computing: 2nd
International Conference, PERVASIVE 2004
Mozer, M. (2005). Lessons from an adaptive house.
(LNCS) (Vol. 3001, pp. 158175). Berlin, Ger-
In D. Cook & R. Das (Eds.), Smart Environments:
many: Springer.
Technologies, Protocols, and Applications (pp.
273294). Hoboken, NJ: Wiley. Tavenard, R., Salah, A., & Pauwels, E. (2007).
Searching for temporal patterns in AmI sensor
Mukerjee, A., & Joe, G. (1990). A qualitative
data. In B. Schiele, A. K. Dey, H. Gellersen, M.
model for space. In Proceedings of the 8th Na-
Tscheligi, R. Wichert, E. Aerts, & A. Buchmann
tional Conference on Artificial Intelligence (pp.
(Eds.), Ambient Intelligence: European Confer-
721727). Palo Alto, CA: AAAI Press.
ence, AmI 2007 (LNCS) (Vol. 4794, pp. 5362).
Nauck, D., & Kruse, R. (1993). A fuzzy neural Berlin, Germany: Springer.
network learning fuzzy control rules and member-
Van Cleave, D., & Rattan, K. (2000). Tuning of
ship functions by fuzzy error backpropagation. In
fuzzy logic controller using neural network. In
Proceedings of the IEEE International Confer-
Proceedings of the IEEE National Conference
ence on Neural Networks (pp. 10221027). Los
Aerospace and Electronics (pp. 305312). Los
Alamitos, CA: IEEE Computer Society.
Alamitos, CA: IEEE Computer Society.
Nguyen, H., Kreinovich, V., & Tolbert, D. (1993).
On robustness of fuzzy logics. In Proceedings
of the IEEE International Conference on Fuzzy
KEY TERMS AND DEFINITIONS
Systems (pp. 543547). Los Alamitos, CA: IEEE
Computer Society. Ambient Intelligence: Ubiquitous computer
Parsons, S. (1994). Some qualitative approaches to system networked with intelligent embedded
applying the Dempster-Shafer theory. Information devices that can sense the environment and react
and Decision Technologies, 19, 321337. appropriately to it.
Behaviour Recognition: The process of de-
Randell, D., Cui, Z., & Cohn, A. (1992). A spa- termining which activity is performed in a smart
tial logic based on regions and connection. In home. Usually based on information produced by
Proceedings of the 3rd International Conference sensors or video cameras.
on Principles of Knowledge Representation and Context Awareness: Taking context informa-
Reasoning (pp. 165176). San Francisco, CA: tion such as time, location, temperature, etc. into
Morgan Kaufmann. consideration in the reasoning process.
Rivera-Illingworth, F., Callaghan, V., & Hagras, Dempster-Shafer Theory: Formalism for
H. (2010). Detection of normal and novel behav- modelling belief based on the concepts of frame
iours in ubiquitous domestic environments. The of discernment and mass assignment.
Computer Journal, 53(2), 142151. Fuzzy Sets: Extension of set theory based on
the concept of degree of membership.
120
Human Behaviour Recognition in Ambient Intelligent Environments
121