Pillai, S.U., Shim, T.I., Batalama, S.N., Kazakos, D., Daum, F.

Spectral Estimation and

Modeling
The Electrical Engineering Handbook
Ed. Richard C. Dorf
Boca Raton: CRC Press LLC, 2000
2000 by CRC Press LLC
16
SpecfraI sfImafIon
and NodeIIng
16.1 Spectial Analysis
Histoiical Peispective Modein Spectial Analysis
16.2 Paiametei Estimation
Bayesian Estimation Mean-Squaie Estimation Minimax
Estimation Maximum Likelihood Estimation Othei Paiametei
Estimation Schemes
16.3 Kalman Filteiing
Kalman Filtei Equations Kalman Filtei Examples Extended Kalman
Filtei Nonlineai Filteis Piactical Issues
16.1 Spectra! Ana!ysis
S. Inn|r|no P||o ond T|eodore I. S|m
Histurica! Perspective
Modein spectial analysis dates back at least to Sii Isaac Newton Newton, 1671], whose piism expeiiments
with sunlight led him to discovei that each coloi iepiesented a paiticulai wavelength of light and that the
sunlight contained all wavelengths. Newton used the woid setrum, a vaiiant of the Latin woid seter, to
desciibe the band of visible light colois.
In the eaily eighteenth centuiy, Beinoulli discoveied that the solution to the wave equation desciibing a
vibiating stiing can be expiessed as an infnite sum containing weighted sine and cosine teims. Latei, the Fiench
engineei Joseph Fouiiei in his na|yta| T|eory o[ Hea Fouiiei, 1822] extended Beinoulli`s wave equation
iesults to aibitiaiy peiiodic functions that might contain a fnite numbei of jump discontinuities. Thus, foi
some T
0
> 0, if [ () [ ( - T
0
) foi all , then [ () iepiesents a peiiodic signal with peiiod T
0
and in the case
of ieal signals, it has the Fouiiei seiies iepiesentation
wheie
0
2 /T
0
, and
with
0
iepiesenting the dc teim (| 0). Moieovei, the infnite sum on the iight-hand side of the above
expiession conveiges to [ (
-0
) - [ (
-0
)]/2. The total powei P of the peiiodic function satisfes the ielation
[ | B |
| |
|
( ) ( cos sin )
0 0 0
1
2

T
[ | J B
T
[ | J
| |
T T
1 1
0
0
0
0
0
0
0 0
( ) cos , ( ) sin
S. InnIIrIshna IIIaI
Po|yrec|nc Inverry
Theodore I. ShIm
Po|yrec|nc Inverry
SfeIIa . BafaIama
Srore Inverry of Nev Yor|
or uffo|o
IImIfrI KazaIos
Inverry of Sour|verern
Iouono
!red Iaum
Foyr|eon Comony
2000 by CRC Press LLC
implying that the total powei is distiibuted only among the dc teim, the fundamental fiequency u
0
2r/T
0
and its haimonics |u
0
, | > 1, with 2(
2
|
-B
2
|
) iepiesenting the powei contained at the haimonic |u
0
. Foi eveiy
peiiodic signal with fnite powei, since
|
0, B
|
0, eventually the oveihaimonics become of decieasing
impoitance.
The Biitish physicist Schustei Schustei, 1898] used this obseivation to suggest that the paitial powei P
|

2(
2
|
-B
2
|
) at fiequency |u
0
, | 0 ~, be displayed as the spectium. Schustei teimed this method the
eroJogram, and infoimation ovei a multiple of peiiods was used to compute the Fouiiei coeffcients and/oi
to smooth the peiiodogiam, since depending on the staiting time, the peiiodogiam may contain iiiegulai and
spuiious peaks. A notable exception to peiiodogiam was the lineai iegiession analysis intioduced by the Biitish
statistician Yule Yule, 1927] to obtain a moie accuiate desciiption of the peiiodicities in noisy data. Because
the sampled peiiodic piocess x(|) cos |u
0
T containing a single haimonic component satisfes the iecuisive
ielation
x(|) ax(| - 1) - x(| - 2)
wheie a 2 cos u
0
T iepiesents the haimonic component, its noisy veision x(|) - n(|) satisfes the iecuision
x(|) ax(| - 1) - x(| - 2) - n(|)
Yule inteipieted this time seiies model as a iecuisive haimonic piocess diiven by a noise piocess and used this
foim to deteimine the peiiodicity in the sequence of sunspot numbeis. Yule fuithei geneialized the above
iecuision to
x(|) ax(| - 1) - |x(| - 2) - n(|)
wheie a and | aie aibitiaiy, to desciibe a tiuly autoiegiessive piocess and since foi the iight choice of a, | the
least-squaie solution to the above autoiegiessive equation is a damped sinusoid, this geneialization foims the
basis foi the modein day paiametiic methods.
Mudern Spectra! Ana!ysis
Noibeit Wienei`s classic woik on Geneialized Haimonic Analysis Wienei, 1930] gave iandom piocesses a fim
statistical foundation, and with the notion of ensemble aveiage seveial key concepts weie then intioduced. The
foimalization of modein day piobability theoiy by Kolmogoiov and his school also played an indispensable
pait in this development. Thus, if x() iepiesents a continuous-time stochastic (iandom) piocess, then foi eveiy
fxed , it behaves like a random variable with some probability density function [
x
(x,). The ensemble aveiage
oi expected value of the piocess is given by
and the statistical coiielation between two time instants
1
and
2
of the iandom piocess is desciibed thiough
its autocorrelation function
P
T
[ J B
| |
|
T
+ +

[
1
2
0
2
0
2 2 2
1
0
0
( ) ( )

x x
E x x [ x Jx ( ) ( )] ( , )
~
~
[

R E x x x x [ x x Jx Jx R
xx x x xx
( , ) ( ) ( )] ( , , , ) ( , )
-
-
1 2 1 2 1 2 1 2 1 2 1 2 2 1
1 2

~
~
~
~
[ [
2000 by CRC Press LLC
wheie [
x1x2
(x
1
, x
2
,
1
,
2
) iepiesents the joint piobability density function of the iandom vaiiable x
1
x(
1
) and
x
2
x(
2
) and denotes the complex conjugate tianspose in geneial. Piocesses with autocoiielation functions
that depend only upon the diffeience of the time inteivals
1
and
2
aie known as wide sense stationaiy piocesses.
Thus, if x() is wide sense stationaiy, then
Ex( - t)x()] R
xx
(t) R
xx
(- )
To obtain the distiibution of powei veisus fiequency in the case of a stochastic process, one can make use
of the Fouiiei tiansfoim based on a fnite segment of the data. Letting
iepiesent the powei contained in a typical iealization ovei the inteival (-T, T), its ensemble aveiage value as
T ~ iepiesents the tiue powei contained at fiequency u. Thus, foi wide sense stationaiy piocesses
(16.1)
Moieovei, the inveise ielation gives
(16.2)
and hence
Thus S(u) iepiesents the powei spectial density and fiom Eqs. (16.1) and (16.2), the powei spectial density
and the autocoiielation function foim a Fouiiei tiansfoim paii, the well-known Wienei-Khinchin theoiem.
If x(|T) iepiesents a disciete-time wide sense stationaiy stochastic piocess, then
and the powei spectial density is given by
oi in teims of the noimalized vaiiable uT,
P
T
x e J
T
,
T
T
( ) ( ) u
u

[
1
2
2
S E P R e J J
R
T
e J R e J
T
T
T
xx
T
T
T
T
,
T
xx
T
T
,
xx
,
( ) lim ( )] lim ( - )
lim ( ) - ( )
- -
- ( - )
-
-
u u
t
t
t t t
u
ut ut

_
,

>
~ ~
~

~
~
[ [
[ [
1 2 1 2
2
2
1 2
1
2
0

R S e J
xx
,
( ) ( ) t
r
u u
ut

~
~
[
1
2
R E x P S J
xx
( ) ( ) ( ) ]
-
0
1
2
2

~
~
[

r
u u
r E x n | T x nT r
| |
+



{ ( ) ( )
S r e
|
, | T
|
( ) u
u

~
~

2000 by CRC Press LLC

(16.3)
and the inveise ielation gives the autocoiielations to be
Thus, the powei spectial density of a disciete-time piocess is peiiodic. Such a piocess can be obtained by
sampling a continuous-time piocess at |T, | 0 ~, and if the oiiginal continuous-time piocess is band-
limited with a two-sided bandwidth equal to 2u
|
2r/T, then the set of disciete samples so obtained is equivalent
to the oiiginal piocess in a mean-squaie sense.
As Schui Schui, 1917] has obseived, foi disciete-time stationaiy piocesses the nonnegativity of the power
spectrum is equivalent to the nonnegative defniteness of the Heimitian Toeplitz matiices, i.e.,
(16.4)
If x(nT) is the output of a disciete-time lineai time-invaiiant causal system diiven by w(nT), then we have the
following iepiesentation:
(16.5)
In the case of a stationaiy input, the output is also stationaiy, and its powei spectial density is given by
S
x
() H(e
,
)
2
S
w
() (16.6)
wheie S
w
() iepiesents the powei spectial density of the input piocess. If the input is a white noise piocess,
then S
w
() o
2
and
S
x
() o
2
H(e
,
)
2
Cleaily if H(:) is iational, so is S
x
(). Conveisely, given a powei spectial density
(16.7)
that satisfes the integiability condition
(16.8)
S r e S |
|
, |
|
( ) ( ) r

+ >

~
~

2 0
r S e J r
|
, |
|

[
1
2r

r
r
( )
S
r r r
r r r
r r r
|
|
|
|
| |
|
( )
...
...
...
,
-
-
> .

_
,

> ~ 0 0 0
0 1
1 0 1
1 0

w nT H : | |T : x nT | |T w n | T
|
| |
( ) ( ) ( ) ( ) ( ) ( )

~

0 0
S r e
x |
, |
|
( )

>
~
~

0
S J
x
( )
r
r
< ~

[
2000 by CRC Press LLC
and the physical iealizability (Paley-Wienei) ciiteiion
(16.9)
theie exists a unique function H(:) that is analytic togethei with its inveise in : < 1 (minimum phase factoi)
such that
(16.10)
and
H(:) is known as the Wienei factoi associated with S
x
() and as Eq. (16.5) shows, when diiven by white noise,
it geneiates a stochastic piocess x(nT) fiom past samples and its powei spectial density matches with the given
S
x
().
In this context, given a fnite set of autocoiielations r
0
, r
1
, . . . , r
n
, the spectial extension pioblem is to obtain
the class of all extensions that match the given data, i.e., such an extension K() must automatically satisfy
K() > 0
and
in addition to satisfying Eqs. (16.8) and (16.9).
The solution to this pioblem is closely ielated to the tiigonometiic moment pioblem, and it has a long and
continuing histoiy thiough the woiks of Schui 1917]; Nevanlinna, Akheizei and Kiein Akheizei and Kiein,
1962]; Geionimus 1954]; and Shohat and Tamaikin 1970], to name a few. If the given autocoiielations aie
such that the matiix T
n
in Eq. (16.4) is singulai, then theie exists an m s n such that T
m - 1
is positive defnite
(T
m - 1
> 0) and T
m
is singulai det T
m
0, det (.) iepiesenting the deteiminant of (.)]. In that case, theie exists
a unique vectoi X (x
0
, x
1
, . . ., x
m
)
T
such that T
m
X 0 and fuithei, the autocoiielations have a unique
extension given by
(16.11)
wheie e
,
, 1 m aie the m zeios of the polynomial x
0
- x
1
: - . . . - x
m
:
m
and P

> 0. This gives

(16.12)
ln ( ) S J
x

r
r
> ~

[
H : | : :
|
|
|
( ) , <

1
0
S H re H e a e
x
r
, ,

lim
1 0
2 2
( ) ( ) , . .
1
2
0
r

r
r
K e J r | n
, |
|
( ) ,

[

t P e |
|
, |

, 0
1

m
m
P
P
P
-
...
...
...

1
1
2
0 0
0 0
0 0

_
,

2000 by CRC Press LLC

wheie A is an m m Vandeimonde matiix given by
and Eq. (16.12) can be used to deteimine P
|
> 0, | 1 m. The powei spectium associated with Eq. (16.11)
is given by
and it iepiesents a disciete spectium that coiiesponds to puie uncoiielated sinusoids with signal poweis P
1
,
P
2
, ., P
m
.
If the given autocoiielations satisfy T
n
> 0, fiom Eq. (16.4), eveiy unknown r
|
, | > n - 1, must be selected
so as to satisfy T
|
> 0, and this gives
r
|-1
-
|

2
s R
2
k
(16.13)
wheie
|
f
T
|
T
-
|
1

b
|
, f
|
(r
1
, r
2
, . . . , r
|
)
T
, b
|
(r
|
, r
| -1
, . . . , r
1
) and R
|
det T
|
/det T
|

-1
.
Fiom Eq. (16.13), the unknowns could be anywheie inside a sequence of ciicles with centei
|
and iadius
R
|
, and as a iesult, theie aie an infnite numbei of solutions to this pioblem. Schui and Nevanlinna have given
an analytic chaiacteiization to these solutions in teims of bounded function extensions. A bounded function
p(:) is analytic in : < 1 and satisfies the inequality p(:) s 1 eveiywheie in : < 1.
In a netwoik theoiy context, Youla 1980] has also given a closed foim paiametiization to this class of
solutions. In that case, given r
0
, r
1
, . . ., r
n
, the minimum phase tiansfei functions satisfying Eqs. (16.8) and
(16.9) aie given by
(16.14)
wheie p(:) is an ar|rary bounded function that satisfes the inequality (Paley-Wienei ciiteiion)
and I(:) is the minimum phase factoi obtained fiom the factoiization
1 - p(e
,
)
2
I(e
,
)
2
Fuithei, P
n
(:) iepiesents the Levinson polynomial geneiated fiom r
0
r
n
thiough the iecuision

_
,

1 1 1
1
1 2
1
2
2
2 2
1
1
2
1 1
...
...
...
...
...
, ,
- -
i i i
i i i
i i i
i

m
m
m m
m
m

,
e m

S P
| |
|
m
( ) ( ) o

1
H :
:
P : : : P :
n n
p
p
( )
( )
( ) - ( )

( )

I
ln ( )

[

,
> ~
r
r

p 1
2
e J
,
1
2
1 1

s P : P : :s P :
n n n n n

2000 by CRC Press LLC

that staits with P
0
(:) 1/ , wheie
(16.15)
iepiesents the ieection coeffcient at stage n. Heie, {
n
denotes the coeffcient of :
n
in the expansion { , and
~
P
n
(:)
A
:
n
P
n
(1/:) iepiesents the polynomial iecipiocal to P
n
(:). Notice that the given infoimation r
0
r
n
enteis P
n
(:) thiough Eq. (16.5). The powei spectial density
K() H
p
(e
,
)
2
associated with Eq. (16.14) satisfes all the inteipolation piopeities desciibed befoie. In Eq. (16.14), the solution
p(:) 0 gives H(:) 1/P
n
(:), a puie AR(n) system that coincides with Buig`s maximum entiopy extension.
Cleaily, if H
p
(:) is iational, then p(:) must be iational and, moie inteiestingly, eveiy iational system must
follow fiom Eq. (16.14) foi a specifc iational bounded function p(:). Of couise, the choice of p(:) biings in
extia fieedom, and this can be pioftably used foi system identifcation as well as iational and stable appioxi-
mation of noniational systems Pillai and Shim, 1993].
Dehning Terms
Autocorrelation function: The expected value of the pioduct of two iandom vaiiables geneiated fiom a
iandom piocess foi two time instants; it iepiesents theii inteidependence.
Expected value (or mean) of a random variable: Ensemble aveiage value of a iandom vaiiable that is given
by integiating the iandom vaiiable aftei scaling by its piobability density function (weighted aveiage)
ovei the entiie iange.
Power spectrum: A nonnegative function that desciibes the distiibution of powei veisus fiequency. Foi wide
sense stationaiy piocesses, the powei spectium and the autocoiielation function foim a Fouiiei tiansfoim
paii.
Probability density function: The piobability of the iandom vaiiable taking values between two ieal numbeis
x
1
and x
2
is given by the aiea undei the nonnegative piobability density function between those two points.
Random variable: A continuous oi disciete valued vaiiable that maps the set of all outcomes of an expeiiment
into the ieal line (oi complex plane). Because the outcomes of an expeiiment aie inheiently iandom,
the fnal value of the vaiiable cannot be piedeteimined.
Stochastic process: A ieal valued function of time , which foi eveiy fxed behaves like a iandom vaiiable.
Re!ated Tupics
14.1 Fouiiei Tiansfoims 40.2 Spectium, Specifcations, and Measuiement Techniques 73.3 Stochastic
Piocesses
Relerences
N.I. Akheizei and M. Kiein, Some Quesons n |e T|eory o[ Momens, Ameiican Math. Soc. Monogi., 2, 1962.
J.B.J. Fouiiei, T|eore na|yque Je |a C|a|eur (na|yta| T|eory o[ Hea), Paiis, 1822.
L. Y. Geionimus, Po|ynoma|s Or|ogona| on a Crt|e anJ T|er |taons, Ameiican Math. Soc., Tianslation,
104, 1954.
I. Newton, P||os. Trans., vol. IV, p. 3076, 1671.
S.U. Pillai and T.I. Shim, Setrum Esmaon anJ Sysem IJenftaon, New Yoik: Spiingei-Veilag, 1993.
I. Schui, Ubei Potenzieihen, die im Innein des Einheitzkieises Beschiankt Sind," Journa| [ur Rene unJ
ngewanJe Ma|ema|, vol. 147, pp. 205-232, 1917.
J.A. Shohat and J.D. Tamaikin, T|e Pro||em o[ Momens, Ameiican Math. Soc., Math. Suiveys, 1, 1970.

0
s P : r : P
n n |
|
|
n
n
n

- -
( ) ( )
1
1
1
0
2000 by CRC Press LLC
N. Wienei Geneialized haimonic analysis," ta Ma|., vol. 55, pp. 117-258, 1930.
D.C. Youla, The FEE: A New Tunable High-Resolution Spectial Estimatoi: Pait I," Technical note, no. 3, Dept.
of Electiical Engineeiing, Polytechnic Institute of New Yoik, Biooklyn, New Yoik; also RADC Repoit,
RADC-TR-81-397, AD A114996, 1982, 1980.
G.U. Yule, On a method of investigating peiiodicities in distuibed seiies, with special iefeience to Wolfei`s
sunspot numbeis," P||os. Trans. R. Sot. LonJon, Ser. , vol. 226, pp. 267-298, 1927.
16.2 Parameter Estimatiun
Sre||o N. oro|omo ond Dmrr Kozo|o
Parameter estimation is the opeiation of assigning a value in a continuum of alteinatives to an unknown
paiametei based on a set of obseivations involving some function of the paiametei. Estimate is the value
assigned to the paiametei and estimator is the function of the obseivations that yields the estimate.
The basic elements in the paiametei estimation aie a vectoi paiametei
m
, a vectoi space
m
wheie
m
takes
its values, a stochastic piocess X() paiameteiized by
m
and a peifoimance ciiteiion oi cost function. The
estimate
^
m
(x
n
) based on the obseivation vectoi x
n
x
1
, x
2
, ..., x
n
] is a solution of some optimization pioblem
accoiding to the peifoimance ciiteiion. In the following, the function [ (x
n

m
) will denote the conditional
joint piobability density function of the iandom vaiiables x
1
, ..., x
n
.
Theie aie seveial paiametei estimation schemes. If the piocess X() is paiametiically known, i.e., if its
conditional joint piobability density functions aie known foi each fxed value
m
of the vectoi paiametei
m
,
then the coiiesponding paiametei estimation scheme is called parametric. If the statistics of the piocess X()
aie nonpaiametiically desciibed, i.e., given
m
e
m
any joint piobability density function of the piocess is a
membei of some nonpaiametiic class of piobability density functions, then the nonparametric estimation
schemes aiise.
Let I
n
denote the n-dimensional obseivation space. Then an estimatoi
^
(x
n
) of a vectoi paiametei
m
is a
function fiom the obseivation space, I
n
, to the paiametei space,
m
. Since this is a function of iandom vaiiables,
it is itself a iandom vaiiable (oi iandom vectoi).
Theie aie ceitain stochastic piopeities of estimatois that quantify somehow theii quality. In this sense an
estimatoi is said to be unbiased if its expected value is the tiue paiametei value, i.e., if
E

{
^
m
(x
n
)
m
wheie the subsciipt on the expectation symbol denotes that the expectation is taken accoiding to the
piobability density function [ (x
n

m
). In the case wheie the obseivation space is the J
n
and the paiametei is
a scalai, it is
The bias of the estimate is the Euclidean noim
m
- E

{
m
(x
n
)
1/2
. Thus, the bias measuies the distance between
the expected value of the estimate and the tiue value of the paiametei. Cleaily, the estimatoi is unbiased when
the bias is zeio.
Usually it is of inteiest to know the conditional vaiiance of an unbiased estimate. The bias of the estimate

^
m
(x
n
) and the conditional vaiiance
E

{
^
m
(x
n
) - E

{
^
m
(x
n
)
2

geneially iepiesent a tiade-off. Indeed, an unbiased estimate may induce ielatively laige vaiiance. On the othei
hand, the intioduction of some low-level bias may then iesult in a signifcant ieduction of the induced vaiiance.
E x [ x Jx
n
R
n n m n
n

{
`
( )
`
( ) ( )
[

2000 by CRC Press LLC
In geneial, the bias veisus vaiiance tiade-off should be studied caiefully foi the coiiect evaluation of any given
paiametei estimate. A paiametei estimate is called efncient if the conditional vaiiance equals a lowei bound
known as the Rao-Ciami bound.
It will be useful to piesent biiey this bound.
The Rao-Ciami bound gives a theoietical minimum foi the vaiiance of any estimate. Moie specifcally, let

^
(x
n
) be the estimate of a scalai paiametei given the obseivation vectoi x
n
. Let [ (x
n
) be given, twice
continuously diffeientiable with iespect to , and satisfy also some othei mild iegulaiity conditions. Then,
.
Sometimes we need to considei the case wheie the sample size n incieases to infnity. In such a case, an
estimatoi is said to be consistent if

^
m
(x
n
)
m
as n ~
Since the estimate
^
m
(x
n
) is a iandom vaiiable, we have to specify in what sense the above holds. Thus, if
the above limit holds w.p. 1, we say that
^
m
(x
n
) is srong|y tonssen oi tonssen w.. 1. In a similai way we
can defne a wea||y tonssen estimatoi.
As fai as the asymptotic distiibution of (x
n
) as n ~ is conceined, it tuins out that the cential limit
theoiem can often be applied to
^
(x
n
) to infei that
^
(x
n
) - ] is asymptotically noimal with zeio mean as
n ~.
In oidei to examine ceitain paiametei estimation schemes we need fist to piesent the defnition of some
ielated functions. Penalty or cost function t
m
,
^
m
(x
n
)] is a scalai, nonnegative function whose values vaiy as

m
vaiies in the paiametei space
m
and as the sequence x
n
takes diffeient values in the obseivation space, I
n
.
ConJona| exeteJ ena|y t(
m
,
^
m
) induced by the paiametei estimate and the penalty function is a function
defned as follows:
t(
m
,
^
m
) E

{t
m
,
^
m
(x
n
)]
If an a ror density function (
m
) is available, then the expected penalty t(
^
m
, ) can be evaluated.
The vaiious existing paiametei estimation schemes evolve as the solutions of optimization pioblems whose
objective function is eithei the conditional expected penalty oi the conditional density function [ (x
n

m
).
Bayesian Estimatiun Scheme
In the Bayesian estimation scheme the available assets aie:
1. A paiametiically known stochastic piocess paiameteiized by
m
, in othei woids, a given conditional
joint density function [ (x
n

m
) defned on the obseivation space I
n
, wheie
m
is a well-defned paiametei
vectoi.
2. A iealization x
n
fiom the undeilying active piocess, wheie the implied assumption is that the piocess
iemains unchanged thioughout the whole obseivation peiiod.
3. A density function (
m
) defned on the paiametei space
m
.
4. Foi each data sequence x
n
, paiametei vectoi
m
and paiametei estimate
^
m
(x
n
), a penalty scalai function
t
m
,
^
m
(x
n
)] is given.
5. A peifoimance ciiteiion which is the minimization of the expected penalty t(
m
, ).
The estimate
^
m
0
that minimizes the expected penalty is called oma| Bayesan esmae a . Undei some
mild conditions the optimal Bayesian estimate
^
m
0
(x
n
) is the conditional expectation E{
m
x
n
.
E E [ x
n n


o
o

`
( ) - log ( )
-

>

1
]
1
1

2
2
1

2000 by CRC Press LLC

If the penalty function has the foim t
m
,
^
m
] 1 - o(
m
-
^
m
), wheie o() is the delta function, then the
optimal Bayesian estimate is called maximum a posteiioii estimate since it happens to maximize the a oseror
density (
m
x
n
).
Anothei special case of penalty function is the function
m
-
^
m

2
. In this case the Bayesian estimate is
called mnmum mean-square esmae and equals the conditional expectation E{
m
x
n
.
In the following we piesent some moie details about mean-square estimation since it is one of the most
populai schemes.
Mean-Square Estimatiun
Foi the simplicity of oui discussion we considei the case of estimating a single continuous type iandom vaiiable
with density () instead of estimating a iandom vectoi. We also ieduce the dimensionality of the obseivation
space to one. In this fiamewoik the penalty function will be the squaie of the estimation eiioi ( -
^
)
2
and
the peifoimance oi optimality ciiteiion will be the minimization of the mean (expected) squaie value of the
estimation eiioi.
We will fist considei the case of estimating a iandom vaiiable by a constant
^
. This means that we wish
to fnd a constant
^
such that the mean-squaie (MS) eiioi
is minimum. Since e depends on
^
, it is minimum if
i.e., if
The case wheie is to be estimated by a function
^
(x) of the iandom vaiiable (obseivation) x, and not by
a constant, is examined next. In this case the MS eiioi takes the foim:
wheie (, x) is the joint density of the iandom vaiiables and x. In this case we need to fnd that function

^
(x) which minimizes the MS eiioi. It can be pioved that the function that minimizes the MS eiioi is
The function
^
(x) is called nonlinear MS estimate.
e E J
~
~
[
{( -
`
) ( -
`
) ( )
2 2
Je
J
J

~
~
[
2 0 ( -
`
) ( )
-
`
{ ( )
-

~
~
[
E J
e E x x JxJy
~
~
~
~
[ [
{ -
`
( )] -
`
( )] ( , )
- -

2 2
`
( ) { ( )
-
x E x x J
~
~
[

2000 by CRC Press LLC
As we have seen, when the penalty function is the quadiatic function ( -
^
)
2
, then the optimal Bayesian
estimate is the conditional expectation E{x. If x and aie jointly Gaussian, then the above conditional
expectation is a lineai function of x. But when the above statistics aie not Gaussian, then the optimal Bayesian
estimate is geneially a nonlineai function of x. Thus, to iesolve this pioblem we intioduce suboptimal Bayesian
schemes foi this quadiatic penalty function. In paiticulai we considei only the class of lineai paiametei estimates
and we tiy to fnd that estimate which minimizes the expected quadiatic penalty. This estimate is called lineai
MS estimate and it is used in many applications because of the simplicity of the solution.
The lineai estimation pioblem is the estimation of a iandom vaiiable in teims of a lineai function x -
B of x, i.e.,
^
(x) x - B. In this case we need to fnd the constants and B in oidei to minimize the MS eiioi
e E{ - (x - B)]
2

A fundamental piinciple in the MS estimation is the orthogonality principle. This piinciple states that the
optimum lineai MS estimate x - B of is such that the estimation eiioi - (x - B) is oithogonal to the
data x, i.e.,
E{ - (x - B)]x 0
Using the above piinciple, it can be pioved that e is minimum if
i.e., q
x
, q

aie the means of x and ; o

x
2
, o

2
aie the coiiesponding vaiiances; o
x
, o

is the standaid deviation

of x and ; and r is the coiielation coeffcient of x and . Thus the MS eiioi takes the foim e o

2
(1 - r
2
).
The estimate

^
(x) x - B
is called the nonhomogeneous linear estimate of in teims of x. If is estimated by a function
^
(x) ox,
the estimate is called homogeneous. It can be also shown by the oithogonality piinciple that foi the homoge-
neous estimate
Using the oithogonality piinciple it can be shown that if the iandom vaiiables and x aie Gaussian zeio
mean, then the optimum nonlineai estimate of equals the lineai estimate. In othei woids if
^
(x) E{x is
the optimum nonlineai estimate of and
^
ox is the optimum lineai estimate, then
^
(x) E{
^
x
^
ox.
This is tiue since the iandom vaiiables and x have zeio mean, E{ E{x 0, and thus the lineai estimate
^
has zeio mean too, E{
^
0. This implies that the lineai estimation eiioi r -
^
also has zeio mean, E{r
E{ -
^
0.

r
B
E E
E E
r
E
x
x
x
x x
x
x

o
o
q q
q q
o q o q
q q
o o

and -
{ , {
{( - ) , {( - )
{( - )( - )

2 2 2 2
o
E
E
{
{

2

2000 by CRC Press LLC
On the othei hand, the oithogonality piinciple can be applied, which implies that the lineai estimation eiioi
r is oithogonal to the data, E{rx 0. Since r is Gaussian, it is independent of x and thus E{rx E{r 0,
which is equivalent to the following:
E{ -
^
x 0 = E{x - E{
^
x 0
= E{ x E{
^
x =
^
(x) ox =
^
(x)
^
i.e., the nonlineai and the lineai estimates coincide.
In addition, since the lineai estimation eiioi r -
^
is independent of the data x, so is the squaie eiioi, i.e.,
E{( -
^
)
2
x E{( -
^
2
V
Thus, the conditional mean of assuming the data x equals its MS estimate and the conditional vaiiance the
MS eiioi. That simplifes the evaluation of conditional densities when Gaussian iandom vaiiables aie involved
because since [ (x) is Gaussian, it has the foim
Minimax Estimatiun Scheme
In the minimax estimation scheme the available assets aie:
1. A paiametiically known stochastic piocess paiameteiized by
m
.
2. A iealization x
n
fiom the undeilying active piocess.
3. A scalai penalty function t
m
,
^
m
(x
n
)] foi each data sequence x
n
, paiametei vectoi
m
, and paiametei
estimate
^
m
(x
n
).
The minimax schemes aie solutions of saddle-point game foimalizations, with payoff function the expected
penalty t(
^
m
, ) and with vaiiables the paiametei estimate
^
m
and the a ror paiametei density function . If
a minimax estimate
^
m
0
exists, it is an optimal Bayesian estimate, at some least favoiable a ror distiibution
0
.
Maximum Like!ihuud Estimatiun Scheme
Maximum likelihood estimation was fist intioduced by Fishei. It is a veiy poweiful estimation pioceduie that
yields many of the well-known estimation methods as special cases.
The essential diffeience between Bayesian and maximum likelihood paiametei estimation is that in Bayesian
Estimation the paiametei
m
is consideied to be iandom with a given density function, while in the maximum
likelihood fiamewoik it is unknown but fxed.
Considei a iandom piocess X() paiameteiized by
m
, wheie
m
is an unknown fxed paiametei vectoi of
fnite dimensionality m (e.g.,
m
e J
m
). Moie specifcally the conditional joint piobability density function
[ (x
1
, ., x
n

m
) is well known foi eveiy
m
, wheie x
n
x
1
, ., x
n
] is a iealization (oi obseivation vectoi oi
sample vectoi) of the piocess X().
The pioblem is to fnd an estimate of the paiametei vectoi
m
based on the iealization of X(). (Note that
the dimensionality of the paiametei vectoi
m
in the joint piobability density function is assumed to be fxed.)
The intuition behind the maximum likelihood method is that we choose those paiameteis
1
, .,
m
] fiom
which the actually obseived sample vectoi is most likely to have come. This means that the estimatoi of
m
is
selected so that the obseived sample vectoi becomes as likely as possible."
[ X
V
x
V
( ) exp
- - ]

r
o


,

1
2
2
2
2000 by CRC Press LLC
In this sense we call the conditional joint piobability density function [ (x
n

m
) as likelihood function (
m
).
The likelihood function (
m
) is a deteiministic function of the paiametei vectoi
m
once the obseived vaiiables
x
1
, ., x
n
aie inseited. This means that
m
is vaiiable and the sample vectoi x
n
is fxed, while the conditional
joint piobability density function is consideied as a function of the obseivation vectoi x
n
with
m
fxed. The
maximum likelihood estimatoi of
m
is that value of the paiametei vectoi foi which the likelihood function is
maximized.
In many cases it is moie convenient to woik with anothei function called log-likelihood function, L(
m
),
iathei than the likelihood function. The log-likelihood function is the natuial logaiithm of (
m
). Since the
logaiithm is a monotone function, it follows that whenevei L() achieves its maximum value, (
m
) is maxi-
mized too, foi the same value of the paiametei vectoi
m
. Thus the log-likelihood function is maximized foi
that value of the vectoi paiametei
m
foi which the fist paitial deiivatives with iespect to

, 1, ., m aie
equal to zeio, i.e.,
wheie
^
ML
denotes the maximum likelihood estimate of the vectoi paiametei
m
.
It can be shown that when the piocess X() is memoiyless and stationaiy (i.e., when x
1
, ., x
n
aie independent,
identically distiibuted) then the ML estimatois aie consistent, asymptotically effcient, and asymptotically
Gaussian.
Exam|e. Let x

, 1, ., n, be Gaussian independent iandom vaiiables with mean and vaiiance o

2

: x

e
N(,o
2

). The mean is to be estimated and the Rao-Ciami bound is to be evaluated. Since is unknown
but fxed, we will use the maximum likelihood estimation scheme. The iandom vaiiable x

has the piobability

density function
Since x

, 1, ., n, aie independent, the joint density function is

which is exactly the likelihood function foi this estimation pioblem. The log-likelihood function is
We can maximize the log-likelihood function with iespect to and fnd the maximizing value to be equal to
Note that foi equal vaiiances the maximum likelihood estimate coincides with the commonly used sample mean.
`
:
( )

o
o
ML
m

L
0
1
2
2
2
2
ro

o

x
exp
( - )

[ x x
x
n

n

( , . . ., ) exp -
( - )

ro

o

j
1
2 2
1
2
2
log ( , . . ., ) - log( ) - log -
( - )
[ x x
n x
n

n
1
1
2
2
1
2
2
1
2
r o

o

`
( )
o
o
ML
n

n

n
x
x

1
1
2
1
2
1
2000 by CRC Press LLC
The Rao-Ciami bound can be found as follows:
In conclusion, we see that foi Gaussian data the sample mean estimate is effcient because it coincides with
the maximum likelihood estimate.
When the data aie contaminated with a fiaction of data coming fiom an unknown piobability density
function, the so called ou|ers, the sample mean peifoims pooily even when the fiaction of outlieis is small.
This obseivation gave biith to the bianch of statistics called iobust statistics.
Other Parameter Estimatiun Schemes
The Bayesian, minimax, and maximum likelihood estimation schemes desciibed above make up the class of
paiametiic paiametei estimation pioceduies. The common chaiacteiistic of those pioceduies is the availability
of some paiametiically known stochastic piocess that geneiates the obseivation sequence x
n
. When foi eveiy
given paiametei value
m
the stochastic piocess that geneiates x
n
is nonpaiametiically desciibed, the nonpaia-
metiic estimation schemes aiise. The lattei schemes may evolve as the solutions of ceitain saddle-point games,
whose payoff function oiiginates fiom the paiametiic maximum likelihood foimalizations. It is assumed that,
in addition to the nonpaiametiically desciibed data-geneiating piocess, the only assets available aie a iealization
x
n
fiom the undeilying active piocess and the paiametei space
m
. The qualitative robustness in paiametei
estimation coiiesponds to local peifoimance stability foi small deviations fiom a nominal, paiametiically
known, data-geneiating piocess.
Dehning Terms
Bayesian estimation: An estimation scheme in which the paiametei to be estimated is modeled as a iandom
vaiiable with known piobability density function.
Bias: The noim of the diffeience between the tiue value of the estimate and its mean value.
Consistent estimator: An estimatoi whose value conveiges to the tiue paiametei value as the sample size
tends to infnity. If the conveigence holds w.p. 1, then the estimatoi is called srong|y tonssen oi tonssen
w.. 1.
Efncient estimator: An estimatoi whose vaiiance achieves the Rao-Ciami bound.
Estimate: Oui best guess of the paiametei of inteiest based on a set of obseivations.
Estimator: A mapping fiom the data space to the paiametei space that yields the estimate.
Homogeneous linear estimator: An estimatoi which is a homogeneous lineai function of the data.
Maximum likelihood estimate: An estimate that maximizes the piobability density function of the data
conditioned on the paiametei.
Mean-square estimation: An estimation scheme in which the cost function is the mean-squaie eiioi.
Minimax estimate: The optimum estimate foi the least favoiable piioi distiibution.
Nonhomogeneous linear estimator: An estimatoi which is a nonhomogeneous lineai function of the data.
Nonlinear MS estimate: The optimum estimate undei the mean-squaie peifoimance ciiteiion.
Nonparametric estimation: An estimation scheme in which no paiametiic desciiption of the statistical model
is available.
Orthogonality principle: The fundamental piinciple foi MS estimates. It states that the estimation eiioi is
oithogonal to the data.
Parameter estimation: The pioceduie by which we combine all available data to obtain oui best guess about
a paiametei of inteiest.
Parametric estimation: An estimation scheme in which the statistical desciiption of the data is given accoid-
ing to a paiametiic family of statistical models.
E
J
J
[ x E
J
J
[ x
n n

n

o
log ( ) - log ( )
-

1
]
1
1

2
1
2
2 2
1
1
2000 by CRC Press LLC
Penalty or cost function: A nonnegative scalai function which iepiesents the cost incuiied by an inaccuiate
value of the estimate.
Qualitative robustness: A geometiic foimulation of iobust estimation.
Robust estimation: An estimation scheme in which we optimize peifoimance foi the least favoiable statistical
enviionment among a specifed statistical class.
Unbiased estimator: An estimatoi whose mean value is equal to the tiue paiametei value.
Re!ated Tupics
73.1 Signal Detection 73.3 Stochastic Piocesses
Relerences
S. Haykin, Jae F|er T|eory, Englewood Cliffs, N.J.: Pientice-Hall, 1991.
D. Kazakos and P. Papantoni-Kazakos, Deeton anJ Esmaon, New Yoik: Computei Science Piess, 1990.
L. Ljung and T. Sdeistim, T|eory anJ Pratte o[ Returse IJenftaon, Cambiidge, Mass.: The MIT Piess,
1983.
A. Papoulis, Pro|a||y, RanJom Vara||es, anJ Sot|ast Protesses, New Yoik: McGiaw-Hill, 1984.
Further Inlurmatiun
IEEE Transatons on In[ormaonT|eory is a bimonthly jouinal that publishes papeis on theoietical aspects of
estimation theoiy and in paiticulai on tiansmission, piocessing, and utilization of infoimation.
IEEE Transatons on Sgna| Protessng is a monthly jouinal which piesents applications of estimation theoiy
to speech iecognition and piocessing, acoustical signal piocessing, and communication.
IEEE Transatons on Communtaons is a monthly jouinal piesenting applications of estimation theoiy to data
communication pioblems, synchionization of communication systems, channel equalization, and image piocessing.
16.3 Ka!man Fi!tering
red Doum
The Kalman nlter is a lineai iecuisive algoiithm that solves the least squaies pioblem foi time-vaiying lineai
systems with non-stationaiy noise. It estimates the state of a lineai dynamical system given lineai measuiements
coiiupted by additive noise. It is an optimal estimatoi, assuming that the measuiement noise is Gaussian, and
assuming that all othei ielevant piobability densities aie also Gaussian.
Foi example, the location of youi cai can be estimated using a Kalman fltei to combine noisy measuiements
of distance fiom foui oi moie satellites. As a second example, the position and velocity of an aiiplane can be
estimated by a Kalman fltei using noisy measuiements of iange, azimuth, and elevation fiom one oi moie
iadais. As a thiid example, the futuie piice of IBM stock can be piedicted using a Kalman fltei to combine
noisy data on thousands of ielevant economic vaiiables, using a dynamic model of the stock maiket and the
oveiall economy.
The Kalman fltei has been applied to solve many diveise ieal-woild engineeiing pioblems, including
spaceciaft navigation, GPS navigation, iobotics, aii tiaffc contiol, missile guidance, chemical plant contiol,
stock maiket piediction, weathei piediction, speech iecognition, speech encoding and compiession, iadai taiget
tiacking, satellite oibit estimation, and ineitial navigation. See Soienson, 1985] foi othei applications.
Most ieal-woild engineeiing pioblems have measuiement equations oi dynamical system equations that aie
nonlineai in the state vector. Theiefoie, the Kalman fltei equations cannot be applied diiectly; iathei, the
pioblem must be appioximated using lineai equations. This lineai appioximation is veiy stiaightfoiwaid, and
it is called the extendedKalman nlter" (EKF). One of the main ieasons foi the wide application ofthe Kalman
fltei is the ease with which a nonlineai system can be appioximated by a lineai system. The iesulting appiox-
imation is often veiy good, iesulting in good EKF peifoimance. Unfoitunately, the EKF peifoimance is
sometimes pooi, in which case a plethoia of alteinative appioximations can be attempted.
2000 by CRC Press LLC
Ka!man Fi!ter Equatiuns
The Kalman nlter algoiithm is shown as a block diagiam in Fig. 16.1. The estimate of x is updated iecuisively
as new measuiements become available. Each measuiement update coiiesponds to one iteiation of Fig. 16.1.
The symbols in Fig. 16.1 aie defned in Table 16.1.
The Kalman fltei uses both a model of the system dynamics
x
k

k
x
k-1
- w
k
(16.16)
as well as a model of the measuiements
z
k
H
k
x
k
- v
k
(16.17)
These models aie a combination of deteiministic and iandom effects. One can think of the tiue state vectoi, xk,
evolving in time accoiding to a deteiministic lineai dynamical system:
x
k

k
x
k-1
(16.18)
with a iandom peituibation modeled by w
k
. Likewise, the measuiement model consists of a deteiministic lineai
pait:
z
k
H
k
x
k
(16.19)
with an additive iandom peituibation of v
k
. As shown in Fig. 16.1, the Kalman fltei piedicts the state vectoi
fiom one time (t
k-1
) to the next (t
k
) using the deteiministic pait of the lineai dynamical system. This is the
FIGURE 16.1 Block diagiam of Kalman fltei.
2000 by CRC Press LLC
T
A
B
L
E

1
6
.
1
D
e
f
n
i
t
i
o
n

o
f

S
y
m
b
o
l
s
S
y
m
b
o
l
M
e
a
n
i
n
g
M
a
t
h
e
m
a
t
i
c
a
l

D
e
f
n
i
t
i
o
n
V
a
l
u
e
x
k
S
t
a
t
e

v
e
c
t
o
i

o
f

a

l
i
n
e
a
i

d
y
n
a
m
i
c
a
l

s
y
s
t
e
m

a
t

t
i
m
e

t
k
x
k

k

x
k
-
1

-

w
k
V
e
c
t
o
i

o
f

d
i
m
e
n
s
i
o
n

n
t
k
T
i
m
e

o
f

t
h
e

k
t
h

m
e
a
s
u
i
e
m
e
n
t
-
S
c
a
l
a
i
k
I
n
d
e
x

o
f

d
i
s
c
i
e
t
e

t
i
m
e

m
e
a
s
u
i
e
m
e
n
t
s
-
I
n
t
e
g
e
i
z
k
k
t
h

m
e
a
s
u
i
e
m
e
n
t
z
k

H
k

x
k

-

v
k
V
e
c
t
o
i

o
f

d
i
m
e
n
s
i
o
n

m
H
k
M
e
a
s
u
i
e
m
e
n
t

m
a
t
i
i
x

a
t

t
i
m
e

t
k
S
e
e

a
b
o
v
e
m


n

m
a
t
i
i
x
R
k
C
o
v
a
i
i
a
n
c
e

m
a
t
i
i
x

o
f

m
e
a
s
u
i
e
m
e
n
t

n
o
i
s
e

a
t

t
i
m
e

t
k
R
k

E
(
v
k

v
k
T
)
m


m

m
a
t
i
i
x
v
k
M
e
a
s
u
i
e
m
e
n
t

n
o
i
s
e

a
t

t
i
m
e

t
k
G
a
u
s
s
i
a
n

z
e
i
o

m
e
a
n

i
a
n
d
o
m

v
a
i
i
a
b
l
e

w
i
t
h

c
o
v
a
i
i
a
n
c
e

m
a
t
i
i
x

R
k
,

s
t
a
t
i
s
t
i
c
a
l
l
y

i
n
d
e
p
e
n
d
e
n
t

f
i
o
m

s
a
m
p
l
e

t
o

s
a
m
p
l
e
,

a
n
d

s
t
a
t
i
s
t
i
c
a
l
l
y

i
n
d
e
p
e
n
d
e
n
t

o
f

x
k
V
e
c
t
o
i

o
f

d
i
m
e
n
s
i
o
n

m
k
T
i
a
n
s
i
t
i
o
n

m
a
t
i
i
x

o
f

l
i
n
e
a
i

d
y
n
a
m
i
c
a
l

s
y
s
t
e
m

f
i
o
m

t
i
m
e

t
k
-
1

t
o

t
k
S
e
e

a
b
o
v
e
n


n

m
a
t
i
i
x
w
k
P
i
o
c
e
s
s

n
o
i
s
e
G
a
u
s
s
i
a
n

z
e
i
o

m
e
a
n

i
a
n
d
o
m

v
a
i
i
a
b
l
e

w
i
t
h

c
o
v
a
i
i
a
n
c
e

m
a
t
i
i
x

Q
k
,

s
t
a
t
i
s
t
i
c
a
l
l
y

i
n
d
e
p
e
n
d
e
n
t

f
i
o
m

s
a
m
p
l
e

t
o

s
a
m
p
l
e
,

a
n
d

s
t
a
t
i
s
t
i
c
a
l
l
y

i
n
d
e
p
e
n
d
e
n
t

o
f

x
k
V
e
c
t
o
i

o
f

d
i
m
e
n
s
i
o
n

n
Q
k
C
o
v
a
i
i
a
n
c
e

m
a
t
i
i
x

o
f

p
i
o
c
e
s
s

n
o
i
s
e

a
t

t
i
m
e

t
k
Q
k

E
(
w
k

w
k
T
)
n


n

m
a
t
i
i
x
P
k
E
i
i
o
i

c
o
v
a
i
i
a
n
c
e

m
a
t
i
i
x

o
f

x
k

c
o
n
d
i
t
i
o
n
e
d

o
n

Z
k
n


n

m
a
t
i
i
x
M
k
E
i
i
o
i

c
o
v
a
i
i
a
n
c
e

m
a
t
i
i
x

o
f

x
k

c
o
n
d
i
t
i
o
n
e
d

o
n

Z
k
-
1
n


n

m
a
t
i
i
x
`x
k
E
s
t
i
m
a
t
e

o
f

x

a
t

t
i
m
e

t
k

c
o
n
d
i
t
i
o
n
e
d

o
n

Z
k
V
e
c
t
o
i

o
f

d
i
m
e
n
s
i
o
n

n
x
k
E
s
t
i
m
a
t
e

o
f

x

a
t

t
i
m
e

t
k

c
o
n
d
i
t
i
o
n
e
d

o
n

Z
k
-
1
V
e
c
t
o
i

o
f

d
i
m
e
n
s
i
o
n

n
Z
k
S
e
t

o
f

m
e
a
s
u
i
e
m
e
n
t
s

u
p

t
o

a
n
d

i
n
c
l
u
d
i
n
g

t
i
m
e

t
k
S
e
t

o
f

m
-
d
i
m
e
n
s
i
o
n
a
l

v
e
c
t
o
i
s
`x
o
I
n
i
t
i
a
l

e
s
t
i
m
a
t
e

o
f

x

a
t

t
i
m
e

t
o
,

p
i
i
o
i

t
o

a
n
y

m
e
a
s
u
i
e
m
e
n
t
s
V
e
c
t
o
i

o
f

d
i
m
e
n
s
i
o
n

n
P
o
I
n
i
t
i
a
l

e
i
i
o
i

c
o
v
a
i
i
a
n
c
e

m
a
t
i
i
x

o
f
`x
o
,

p
i
i
o
i

t
o

a
n
y

m
e
a
s
u
i
e
m
e
n
t
s
n


n

m
a
t
i
i
x
E
(
)
E
x
p
e
c
t
e
d

v
a
l
u
e

o
f

(
)
E
(
A
)

A

p
(
A
)
d
A
-
p
(
A
)
P
i
o
b
a
b
i
l
i
t
y

d
e
n
s
i
t
y

o
f

A
G
a
u
s
s
i
a
n

p
i
o
b
a
b
i
l
i
t
y

d
e
n
s
i
t
y
F
u
n
c
t
i
o
n
p
(
A
B
)
P
i
o
b
a
b
i
l
i
t
y

d
e
n
s
i
t
y

o
f

A

c
o
n
d
i
t
i
o
n
e
d

o
n

B
G
a
u
s
s
i
a
n

p
i
o
b
a
b
i
l
i
t
y

d
e
n
s
i
t
y
F
u
n
c
t
i
o
n
(
)
T
T
i
a
n
s
p
o
s
e

o
f

(
)
(
A
T
)
i
j

A
j
i

f
o
i

a

m
a
t
i
i
x

A

w
i
t
h

i
j
t
h

e
l
e
m
e
n
t

A
i
j
O
p
e
i
a
t
i
o
n
(
)
-
1
I
n
v
e
i
s
e

o
f

m
a
t
i
i
x

(
)
A
-
1

i
s

t
h
e

i
n
v
e
i
s
e

o
f

m
a
t
i
i
x

A

i
f

a
n
d

o
n
l
y

i
f

A
-
1
A

A
A
-
1

I
O
p
e
i
a
t
i
o
n
I
I
d
e
n
t
i
t
y

m
a
t
i
i
x
n


n

m
a
t
i
i
x
P
E
x
x
x
x
Z
k
k
k
k
k
T
k
`
`
M
E
x
x
x
x
Z
k
k
k
k
k
T
k
1
`x
E
x
Z
k
k
k
x
E
x
Z
k
k
k
1
Z
z
z
z
k
k
1
2
,
,
,
`x
E
x
o
o
P
E
x
x
x
x
o
o
o
o
o
T
`
`
I
j
i
j
10
f
o
i

i
o
t
h
e
i
w
i
s
e
2000 by CRC Press LLC
best piediction of x(t
k
) given x(t
k-1
), assuming that w
k
is a zeio-mean iandom vaiiable uncoiielated with x(t
k-1
).
Hence, the state vector piediction in the Kalman nlter is intuitively exactly what one would expect. The state
vectoi update is also intuitively ieasonable. In paiticulai, it is a lineai combination of the piedicted value of x
and the latest measuiement z
k
. This lineai combination of x
k
and z
k
is computed as a compiomise between
piediction eiiois in x
k
and the measuiement noise eiiois in z
k
. That is, the Kalman fltei gain is computed to
optimally combine x
k
and z
k
, using the known models of system dynamics and measuiements. Moie specifcally,
the Kalman fltei gain is computed to minimize the following eiioi ciiteiion at the time of the k
th
measuiement:
(16.20)
The covariance matrix of the piediction eiiois is M
k
, and R
k
is the measuiement eiioi covaiiance matiix. If
M
k
is laige, then the fist teim in J is weighted less because the piediction, x
k
, is ielatively inaccuiate. Similaily,
if R
k
is laige, then the second teim in J is weighted less because the measuiement, z
k
, is ielatively inaccuiate.
The weighted least squaies ciiteiion, J, stiikes a balance between piediction eiiois and measuiement eiiois.
To fnd the value of ` x
k
that minimizes J, we can set the deiivative of J with iespect to ` x
k
equal to zeio:
(16.21)
Using the fact that covaiiance matiices aie symmetiic, and ieaiianging teims, we get:
(16.22)
and solving foi ` x
k
:
(16.23)
This can be put into the foim:
(16.24)
wheie:
(16.25)
Fuithei matiix algebia can be used to iepiesent the Kalman fltei gain in two othei foims:
(16.26)
(16.27)
in which P
k
is the eiioi covaiiance matiix of ` x
k
.
J x x M x x z H x R z H x
k k
T
k k k k k k
T
k k k k
` ` ` `
1 1
J
x
x x M z H x R H
k
k k
T
k k k k
T
k k
`
` ` 2 2 0
1 1
M H R H x M x H R z
k k
T
k k k k k k
T
k k
1 1 1 1
`
`
x M H R H M x H R z
k k k
T
k k k k k
T
k k
1 1
1
1 1
`
x x K z H x
k k k k k k
K M H R H H R
k k k
T
k k k
T
k
1 1
1
1
K M H H M H R
k k k
T
k k k
T
k
1
K P H R
k k k
T
k
1
2000 by CRC Press LLC
The above calculation of the Kalman nlter gain is by no means a deiivation of the Kalman fltei, but iathei
it is meiely a simple heuiistic calculation that gives insight into these equations. The eiioi ciiteiion, J, is the
logaiithm of a Gaussian piobability density of x conditioned on Z
k
. A lucid deiivation of the Kalman fltei is
given in Ho and Lee 1964], fiom a Bayesian viewpoint. In contiast, Kalman`s oiiginal deiivation does not use
Bayesian methods; see Kalman 1960]. Othei deiivations of the Kalman fltei aie in Gelb 1974] and Jazwinski
1970]. The Kalman fltei is optimal" in a numbei of ways, undei diffeient assumptions, which aie discussed
in these iefeiences.
The Kalman fltei is stable undei iathei mild assumptions, which can always be achieved in piactice, by
using a state vector with minimum dimension to iealize the lineai dynamical model and the lineai measuiement
model. This coiiesponds to a tom|ee|y tonro||a||e dynamical system and a tom|ee|y o|sera||e measuiement
model. Stability of the Kalman fltei undei these conditions was pioved in a beautiful papei by Kalman (1963).
Kalman`s stability theoiy has gieat piactical value because engineeis do not need to check foi Kalman fltei
stability as a sepaiate issue. This iesult is all the moie impiessive when we considei that the Kalman fltei is a
time-vaiying lineai dynamical system that can be veiy complex and have veiy high dimension. Neveitheless,
the Kalman fltei is automatically stable undei the simple minimality assumption given above.
It is impoitant to emphasize that both the lineai dynamical system and the measuiement model can be time-
vaiying, and both the piocess noise and measuiement noise can be nonstationaiy. That is, all of the following
matiices can vaiy with time:
k
, H
k
, Q
k
, and R
k
. Also, the disciete times at which measuiements aie made (t
k
foi k 1, 2, .) aie completely aibitiaiy; that is, the sample iate can be nonunifoim.
Ka!man Fi!ter Examp!es
A good way to undeistand the Kalman fltei equations is to look at some simple low-dimensional examples.
Examp!e 1
Considei the pioblem of estimating the value of an unknown constant scalai, x, given a sequence of noisy
measuiements:
z
k
x
k
- v
k
(16.28)
wheie the vaiiance of measuiement noise is constant, and the vaiiance of the a ror estimate of x is infnite.
Foi this example, using the Kalman fltei notation:
k
1
Q
k
0
H
k
1
R
k
constant c
P
o

the coiiesponding Kalman fltei equations given in Fig. 16.1 aie
(16.29)
(16.30)
(16.31)
x x
k k
`
1
`
x x K z x
k k k k k
K M M c
k k k
2000 by CRC Press LLC
(16.32)
(16.33)
(16.34)
which simplifes to
(16.35)
(16.36)
(16.37)
Aftei some moie algebia, it tuins out that
(16.38)
wheie P
o
-1
0, and hence
(16.39)
(16.40)
Theiefoie, the vaiiance of estimation eiioi aftei k measuiements is
(16.41)
which is exactly what we should expect foi this example. Also, the Kalman fltei gain is
(16.42)
(16.43)
(16.44)
M P
k k 1
P M M M c
k k k k
2
P
o
` ` ` x x K z x
k k k k k 1 1
K P P c
k k k 1 1
P P P P c
k k k k 1 1
2
1
P P
c
k k
1
1
1
1
P c
k
j
k
1
1
1
P k c
k
1
P c k
k
K P P c
k k k 1 1
K
c P
k
k
1
1
1
K
k
k
1
1 1
2000 by CRC Press LLC
(16.45)
which is intuitively veiy ieasonable. Fuitheimoie, the Kalman fltei can now be wiitten as:
(16.46)
which has the solution
(16.47)
The Kalman fltei foi this simple example is nothing moie than oui old fiiend, the aiithmetic aveiage.
Examp!e 2
Considei the same pioblem as in Example 1, but with R
k
not constant. It is easy to show that the Kalman fltei
in this case is
(16.48)
wheie the estimation eiioi vaiiance aftei k measuiements is given by
(16.49)
and the Kalman fltei estimate of x aftei k measuiements is:
(16.50)
This iesult is intuitively veiy ieasonable. In paiticulai, the moie accuiate measuiements (coiiesponding to
small R
j
) aie weighted moie heavily in estimating x; conveisely, ielatively inaccuiate measuiements (with laige
R
j
) aie given little weight.
Examp!e 3
Considei the pioblem of estimating the value of a quantity, y, that changes lineaily with time with an unknown
iate of change, given a sequence of measuiements of y coiiupted by additive noise that is statistically independent
fiom sample to sample. In the Kalman fltei setup, we could model this pioblem as follows. Let the state vectoi be:
(16.51)
K k
k
1
` ` `
x x
k
z x
k k k k 1 1
1
` x
k
z
k j
j
k
1
1
` ` ` x x P R z x
k k k k k k 1 1
P R
k j
j
k
1 1
1
` x
z R
R
k
j j
j
k
j
j
k
1
1
1
x
y
y
2000 by CRC Press LLC
The tiansition matiix would be:
(16.52)
wheie t
k
t
k
- t
k-1
. Fuitheimoie,
H
k
1 0]
Q
k
0
R
k
constant
P
o
-1
0
Assuming a constant value of t
k
T, it tuins out that the eiioi covariance matrix is:
(16.53)
See Soienson, 1967] foi details.
Extended Ka!man Fi!ter
In piactical applications, it is iathei iaie to fnd a pioblem with dynamical equations and measuiement
equations that aie lineai in x. Neveitheless, engineeis use the Kalman nlter theoiy applied to a lineai appiox-
imation of the actual nonlineai dynamics and measuiements. This appioximation is called the extended Kalman
nlter (EKF); it is veiy stiaightfoiwaid and populai. The Kalman fltei itself is almost nevei used in ieal-woild
applications, but iathei the EKF is essentially ubiquitous.
Figuie 16.2 shows a block diagiam of the EKF. Note that the EKF uses the nonlineai dynamics and nonlineai
measuiement equations to piedict x
k
and z
k
, iathei than using a lineai appioximation. In contiast, the EKF
uses lineai appioximations of f(x) and h(x) to compute the covaiiance matiices and the Kalman fltei gain.
The nonlineai dynamical model foi x is:
(16.54)
and the nonlineai measuiement model is:
(16.55)
Also, note in Fig. 16.2 that the estimate of x is used to compute the Kalman fltei gain, unlike the Kalman fltei,
in which the fltei gain and the eiioi covaiiance matiices do not depend on x (see Fig. 16.1). Unlike the Kalman
fltei, theie is no guaiantee that the EKF is stable. Moieovei, theie is no ieason to suppose that the EKF will give
optimal peifoimance. Although in many applications the EKF peifoimance is good, it is well known that the
EKF peifoimance is often pooi oi fai fiom optimal. Unfoitunately, theie is no theoiy that piedicts when the
EKF will give good peifoimance, but iathei engineeis use Monte Cailo simulations to evaluate EKF peifoimance.
Theie is a vast liteiatuie on methods to impiove the EKF peifoimance, including second-oidei Tayloi seiies,
iteiation of Fig. 16.2 to impiove the lineaiization, tuning the piocess noise covaiiance matiix, decoupling the
k
k
t 1
0 1
M
k T
T T k
R
k k
k
2 2 1 6
6 12 1 1
2
x f x w
k k k 1
z h x v
k k k
2000 by CRC Press LLC
eiioi covariance matrix, piefeiied oidei of piocessing the components of a vectoi-valued measuiement,caieful
choice of cooidinates (e.g., polai vs. Caitesian), hybiid cooidinate systems, etc. Theie is no guaiantee that any
of these methods will impiove EKF peifoimance; in some cases, second-oidei coiiections and/oi iteiation
actually make EKF peifoimance woise, contiaiy to intuition. Reviews of these techniques aie given in Tanizaki
1996], as well as Wishnei et al. 1969], Mehia 1971], Leskiw et al. 1982], Gelb 1974], Bellaiie et al. 1995],
Heniiksen 1982], Fang 1976], Daum et al. 1983], and Jazwinski 1970].
Nun!inear Fi!ters
Consideiing the fiequently disappointing peifoimance of the EKF noted in the pievious section, theie has been
intense ieseaich to develop bettei nonlineai flteis. An exact nonlineai iecuisive fltei was deiived by Bene s
(foi a ceitain class of nonlineai pioblems) in a seminal papei 1981]. The Bene s fltei is exact" in the sense
that it computes an optimal estimate of x, without any appioximations, in contiast to the EKF, which uses
lineai appioximations. A geneialization of the Benes fltei and the Kalman nlter was developed by Daum 1986a;
1986b]. Figuie 16.3 shows the supeiioi peifoimance of this new nonlineai fltei compaied to the EKF foi ceitain
piactical applications; see Schmidt 1993] foi details. A moie geneial class of exact nonlineai iecuisive flteis
is based on the exponential family of piobability densities. The Kalman fltei theoiy is based on a Gaussian
density, which is a special case of the exponential family; see Daum 1988; 1997a, b] foi a development of this
theoiy, which is summaiized in Table 16.2.
Anothei alteinative to the EKF, iepoited in Juliei et al. 1995], is called the unsteneJ f|er, and in contiast
to the EKF, does not use Jacobians, but iathei evaluates multidimensional integials by sampling at caiefully
selected points much like Gauss-Heimite quadiatuie foimulas. The unscented fltei shows much bettei pei-
foimance than the EKF in ceitain applications, with less computational complexity than the EKF.
Exact iecuisive flteis foi nonlineai estimation pioblems geneially do not exist. This is not suipiising,
consideiing that the existence of an exact iecuisive fltei coiiesponds to the following minoi miiacle:
(16.56)
FIGURE 16.2 The extended Kalman (EKF) is a lineai appioximation.
p x t Z p x t
giowing fixed
k k
, ,
dimension dimension
with k foi all k
2000 by CRC Press LLC
in which
k
is a suffcient statistic foi x. A returse fltei exists when theie is a suffcient statistic with fxed
fnite dimension. In classical statistics, foi paiametei estimation, it is well known that this will happen (assuming
ceitain iegulaiity conditions) if and only if the conditional density is fiom an exponential family; see Daum
1988]. The theoiy of fxed fnite dimensional flteis has also been developed fiom a completely diffeient
peispective, using Lie algebias; see Benes 1987].
Non-returse flteis geneially have supeiioi peifoimance compaied with the EKF, at the cost of highei
computational complexity. Foi paiametei estimation pioblems (coiiesponding to zeio piocess noise, Q
k
0),
these non-iecuisive flteis aie populai in piactical applications despite the incieased computational complexity
compaied with the EKF. Gauss invented this type of non-iecuisive nonlineai fltei ovei 200 yeais ago; see
Soienson 1980]. On the othei hand, non-iecuisive flteis foi estimating x, wheie x is a Maikov piocess with
non-zeio piocess noise (Q
k
0), geneially have much gieatei computational complexity. Neveitheless, with a
suffciently fast computei, it would be piactical to implement such a non-iecuisive algoiithm. The theoiy to
design such algoiithms is well known, and some Monte Cailo simulations have shown excellent peifoimance
ielative to the EKF; see Soienson 1988] and Kastella et al. 1997]. Piesumably, with computeis getting ten
times fastei (at fxed cost) eveiy 5 yeais, the application of such non-iecuisive flteis will become common in
the futuie, despite veiy high computational complexity ielative to the EKF.
Practica! Issues
Data Assuciatiun
One of the best ways to iuin the peifoimance of a Kalman nlter is to put the wiong data into it. In a dense
multiple-taiget enviionment, foi applications with sensois such as iadai, passive infiaied, sonai, acoustic, oi
optical, the question of which measuiement oiiginated fiom which taiget is a veiy seiious issue. In addition,
the measuiements could be uniesolved mixtuies fiom two oi moie objects. Theie is a plethoia of algoiithms
to mitigate these pioblems, as well as a vast liteiatuie on this subject, including Blackman 1986]; Blackman
et al. 1999], Bai-Shalom 1995], and Daum 1992].
FIGURE 16.3 New nonlineai fltei vs. extended Kalman fltei See Schmidt (1993)].
2000 by CRC Press LLC
T
A
B
L
E

1
6
.
2
E
x
a
c
t

R
e
c
u
i
s
i
v
e

F
i
l
t
e
i
s
C
o
n
d
i
t
i
o
n
a
l

D
e
n
s
i
t
y
F
i
l
t
e
i
p
(
x
,

t
Z
k
)
C
l
a
s
s

o
f

D
y
n
a
m
i
c
s
P
i
o
p
a
g
a
t
i
o
n

E
q
u
a
t
i
o
n
s
1
.
K
a
l
m
a
n

(
1
9
6
0
)

G
a
u
s
s
i
a
n
2
.
B
e
n
e
s
(
1
9
8
1
)

a
n
d

3
.
D
a
u
m

(
1
9
8
6
)

a
n
d
w
h
e
i
e

4
.
D
a
u
m

(
1
9
8
6
)
S
a
m
e

a
s

f
l
t
e
i

3
,

b
u
t

w
i
t
h
S
a
m
e

a
s

f
l
t
e
i

3
5
.
D
a
u
m

(
1
9
8
6
)
a
n
d

6
.
D
a
u
m

(
1
9
8
8
)
S
o
l
u
t
i
o
n

o
f

P
D
E

f
o
i

(
x
,

t
)
fx
A
t
m
A
m
P
A
P
P
A
G
G
T
T
e
x
p
x
f
x
d
x
fx
fx
T
f
x
t
i
fx
x
A
x
b
x
c
T
T
2
m
P
A
m
P
b
P
I
P
A
P
12
P
x
s
s
f
Q
i
D
x
E
T
t
i
fx
i
Q
i
x
A
x
b
x
c
T
T
T
2
i
x
P
x
s
s
l
o
g
m
P
A
m
D
m
P
b
E
P
P
A
P
D
P
P
D
Q
T
2
1
1
2
1
q
x
t
,
i
x
q
x
t
l
o
g
,
Q
x
t
,
fx
fx
D
D
T
T
ft
D
x
E
f
x
f
x
t
i
fx
A
D
D
x
D
E
b
T
T
T
T
12
2












m
P
A
D
m
E
P
b
P
P
A
P
P
D
D
P
I
T
2
2
p
x
t
x
t
Z
t
T
k
,
e
x
p
,
,
d
d
t
A
B
T
M
T
j
T
j
w
h
e
i
e
w
i
t
h

1
2
,
,
,
2000 by CRC Press LLC
I!!-Cunditiuning
The Kalman nlter covaiiance matiix can be extiemely ill-conditioned in ceitain applications, as analyzed in
Daum et al. 1983]. A Kalman fltei is ill-conditioned if its peifoimance is signifcantly degiaded by numeiical
eiiois. Special factoiizations of the covaiiance matiices have been developed to mitigate this pioblem; the
standaid iefeience is Bieiman 1977]. Theie aie many othei methods to mitigate ill-conditioning, as discussed
by Daum et al. 1983].
Adaptive Ka!man Fi!ters
The Kalman fltei was deiived assuming that the piocess noise covaiiance matiix, Q
k
, as well as all othei matiices
(
k
, H
k
, R
k
) aie known exactly a ror. In piactice, howevei, these assumptions may be inappiopiiate. To
mitigate this unceitainty a numbei of adaptive Kalman fltei algoiithms can be used. Most of these algoiithms
consist of a bank of Kalman flteis combined adaptively using Bayes` iule. This basic stiuctuie was invented by
Magill 1965], and it has now evolved into a moie sophisticated algoiithm, called neratng mu||e moJe|s
invented by Blom 1984]. A iecent suivey of this topic is given by Bai-Shalom et al. 1995].
Measurement Mude!s
The Kalman fltei theoiy assumes Gaussian measuiement eiiois that aie statistically independent fiom sample
to sample, with zeio mean and exactly known covaiiance matiix (R
k
). Howevei, in many piactical applications,
these aie pooi appioximations of ieality. Foi example, in iadai applications, the measuiments of iange, azimuth,
and elevations aie often biased, non-Gaussian, and coiielated with time, owing to diveise physical effects
including multipath, tiopospheiic iefiaction, ducting, ionospheiic iefiaction, glint, RV wake, iocket exhaust
plume, RFI, ECM, uniesolved measuiements, bias eiiois in time, location and angulai oiientation foi the iadai
itself, iadai haidwaie eiiois, etc. Gauss himself cautioned against nave least squaies ftting of data with bias
and diift; see Gauss 1995].
Perlurmance Eva!uatiun
As shown in Fig. 16.1, the Kalman fltei computes the covaiiance matiix of the estimation eiioi (P
k
). Howevei,
in piactical applications, this theoietical covaiiance matiix may be extiemely optimistic, owing to the effects
noted eailiei (nonlineaiity, ill-conditioning, data association eiiois, uniesolved data, eiiois in modeling both
measuiement eiiois and taiget dynamics) as well as bugs in the softwaie itself. Theiefoie, the standaid appioach
to evaluate Kalman fltei peifoimance is Monte Cailo simulation. Howevei, no one in theii iight mind would
believe the iesults of a complex Monte Cailo simulation without a back-of-the-envelope calculation that is in
iough agieement with the simulation iesults. A good souice of such simple foimulas is Biooknei 1998].
Obviously, the veiy best way to evaluate Kalman fltei peifoimance is to conduct extensive ieal-woild testing.
Unfoitunately, the cost and piacticality of this appioach is often piohibitive oi is deemed to be not cost-effective.
A judicious combination of extensive Monte Cailo simulation and limited ieal-woild testing is often the most
piactical appioach to peifoimance evaluation.
The best possible peifoimance foi EKFs can be computed using theoietical lowei bounds on the eiioi
covaiiance matiix, such as the Ciami-Rao bound (CRB) foi paiametei estimation. Foi the standaid Kalman
fltei setup with zeio piocess noise (Q
k
0), it tuins out that the CRB is simply the Kalman fltei eiioi covaiiance
matiix itself; see Tayloi 1979]. On the othei hand, foi non-zeio piocess noise, the available bounds aie much
moie complex to compute and they aie geneially not tight; see Keii 1989] foi a detailed ieview of the state
of the ait. Moie geneially, the theoiy of eiioi bounds when data association eiiois aie consideied is developed
in Daum 1997a, b].
Digita! Rea!izatiun
All of the algoiithms discussed heie aie always implemented using digital computeis, owing to theii supeiioi
accuiacy, exibility, and dynamic iange, as compaied to cuiiently available analog devices. Geneially, 32-bit
oi 64-bit oating point aiithmetic is iequiied foi most piactical applications, although extia piecision may be
iequiied foi extiemely ill-conditioned pioblems.
The idea of using analog computeis to implement Kalman flteis (which is sometimes suggested in academic
ciicles) is iathei nave, owing to the limited accuiacy, limited dynamic iange, and inexibility of analog
2000 by CRC Press LLC
computeis. Likewise, the flteiing theoiy foi continuous time measuiements (which dominates the academic
liteiatuie on nonlineai flteiing) is also impiactical because measuiements must be made in disciete time to
accommodate digital computeis. The nave appioximation of disciete time measuiements by continuous time
data geneially iesults in pooi peifoimance, and it is not used by piactical engineeis. The academic liteiatuie
is out of touch with such piactical issues; foi example, see Hazewinkel et al. 1981].
Dehning Terms
Kalman nlter: A iecuisive algoiithm that estimates the state vectoi of a lineai dynamical system given noisy
measuiements that aie lineai in the state vectoi. This algoiithm was invented by Rudolf E. Kalman, and
it was published in 1960.
Extended Kalman nlter: A iecuisive algoiithm foi estimating the state of nonlineai dynamical systems that
uses the Kalman fltei equations based on a lineai appioximation to the nonlineai dynamical system
and/oi nonlineai measuiement equations.
State vector: A vectoi that specifes the state of a dynamical system. Foi example, the position and velocity
of a leaf falling to the giound could be the state vectoi foi the leaf. Foi deteiministic systems, this
coiiesponds to the initial conditions foi a system of oidinaiy diffeiential equations. Foi a special type
of iandom piocess, called a Maikov piocess, the futuie state is statistically independent of the past,
conditioned on knowledge of the state at the piesent time.
Covariance matrix: A matiix that gages the unceitainty in a vectoi using second moments. The diagonal
elements of this matiix aie the vaiiances of unceitainty in the components of the vectoi.
Relerences
Y. Bai-Shalom and X. Li. Mu|arge-Mu|sensor Trat|ng, YBS, 1995.
R. Bellaiie, E. W. Kamen, and S. M. Zabin, A new nonlineai iteiated fltei with applications to taiget tiacking,
SPIE ProteeJngs, San Diego, 1995.
V. E. Bene s, Nonlineai flteiing: pioblems, examples, applications, Jantes n Sasta| Sgna| Protessng,
Vol. 1, pp. 1-14, JAI Piess, 1987.
V. E. Benes, Exact fnite-dimensional flteis foi ceitain diffusions with nonlineai diift, Sot|asts, 5, 65-92, 1981.
G. J. Bieiman, Fator:aon Me|oJs [or Dstree Sequena| Esmaon, New Yoik: Academic, 1977.
S. S. Blackman and R. F. Popoli. Desgn anJ na|yss o[ MoJern Trat|ng Sysems, Aitech House, 1999.
S. S. Blackman. Mu|-Targe Trat|ng w| RaJar |taons, Aitech House Inc., 1986.
H. A. P. Blom. A sophisticated tiacking algoiithm foi ATC suiveillance data, ProteeJngs o[ Inernaona| RaJar
Con[erente, Paiis, 1984.
E. Biooknei. Trat|ng anJ Ka|man F|erng MaJe Easy, John Wiley & Sons, 1998.
R. G. Biown and P. Y. C. Hwang, InroJuton o RanJom Sgna|s anJ |eJ Ka|man F|erng, thiid edition,
John Wiley & Sons, 1997.
A.E. Biyson and Y. C. Ho, |eJ Oma| Conro|, Blaisdell Publishing, 1969.
R. S. Bucy, Lineai and nonlineai flteiing, Prot. IEEE, 58, 854-864, 1970.
F. E. Daum. (1997a), Viitual measuiements foi nonlineai flteis, ProteeJngs o[ IEEE Conro| anJ Detson
Con[erente, San Diego, pp. 1657-1662, 1997.
F. E. Daum. (1997b) Ciami-Rao type bounds foi iandom set pioblems, pp. 165-183 in RanJom Ses, Ed. By
J. Goutsias, R. Mahlei, and H. T. Nguyen, Spiingei-Veilag, 1997.
F. E. Daum. Beyond Kalman flteis: piactical design of nonlineai flteis, ProteeJngs o[ |e SPIE Con[erente on
Sgna| anJ Daa Protessng o[ Sma|| Targes, pp. 252-262, Oilando, FL, Apiil 1995.
F. E. Daum. A system appioach to multiple taiget tiacking, Mu|arge-Mu|sensor Trat|ng, Volume II (Y. Bai-
Shalom, ed.), pp. 149-181, Aitech House, 1992.
F. E. Daum. New exact nonlineai flteis, Bayesan na|yss o[ Tme Seres anJ Dynamt MoJe|s (J.C. Spall, ed.),
pp. 199-226, Maicel Dekkei, New Yoik, 1988.
F. E. Daum. (1986a). Exact fnite dimensional nonlineai flteis, IEEE Trans. uom. Conro| AC-31(7), 616-622,
1986.
2000 by CRC Press LLC
F. E. Daum. (1986b). New nonlineai flteis and exact solutions of the Fokkei-Planck equations, in ProteeJngs
o[ |e mertan Conro| Con[erente, pp. 884-888, 1986.
F. E. Daum and R. J. Fitzgeiald. Decoupled Kalman flteis foi phased aiiay iadai tiacking, IEEE Trans. uom.
Conro|, AC-28, 269-283, 1983.
B. T. Fang. A nonlineai counteiexample foi batch and extended sequential estimation algoiithms, IEEE Trans.
uom. Conro|, AC-21, 138-139, 1976.
C. F. Gauss. T|eora Com|naons O|seraonum Error|us Mnms O|noxae, tianslated by G. W. Stewait,
SIAM, 1995.
A. Gelb (Editoi). |eJ Oma| Esmaon, MIT Piess, 1974.
M. Hazewinkel and J. C. Willems, Eds. Sot|ast Sysems. T|e Ma|emats o[ F|erng anJ IJenftaon anJ
|taons, D. Reidel, Doidiecht, The Netheilands, 1981.
R. Heniiksen. The tiuncated second-oidei nonlineai fltei ievisited, IEEE Trans. uom. Conro|, AC-27, 247-251,
1982.
Y. C. Ho and R. C. K. Lee. A Bayesian appioach to pioblems in stochastic estimation and contiol, IEEE Trans.
uom. Conro|, AC-9, 333-339, 1964.
C. E. Hutchinson. The Kalman fltei applied to aeiospace and electionic systems, IEEE Trans. eros. E|etron.
Sys. 500-504, 1984.
A. H. Jazwinski. Sot|ast Protesses anJ F|erng T|eory, Academic Piess, New Yoik, 1970.
S. Juliei, J. Uhlmann, and H. Duiiant-Whyte. A new appioach to flteiing nonlineai systems, ProteeJngs o[
mertan Conro| Con[erente, June 1995.
R. E. Kalman. New methods in Wienei flteiing theoiy, in Prot. Sym. Eng. |. o[ RanJom Funton T|eory
anJ Pro|a||y, F. Kozin and J. L. Bogdanoff, Eds. New Yoik: Wiley, 1963.
R. E. Kalman. A new appioach to lineai flteiing and piediction pioblems, Trans. SME J. Bast Eng., 82D, 35-45,
1960.
K. Kastella and A. Zatezalo. Nonlineai flteiing foi detection, tiacking and ID algoiithms, ONR/NSVC Vor|s|o
on F|erng anJ Trat|ng, May 1997.
T. H. Keii. Status of CR-like lowei bounds foi nonlineai flteiing, IEEE Trans. eros. E|etron. Sys., 25, 590-601,
1989.
D. M. Leskiw and K. S. Millei. Nonlineai estimation with iadai obseivations, IEEE Trans. eros. E|etron. Sys.,
AES-18, 192-200, 1982.
D. T. Magill. Optimal adaptive estimation of sampled stochastic piocesses, IEEE Trans. uom. Conro|, AC-10,
434-439, 1965.
R. K. Mehia. A compaiison of seveial nonlineai flteis foi ieentiy vehicle tiacking, IEEE Trans. uom. Conro|,
AC-16, 307-319, 1971.
G. C. Schmidt. Designing nonlineai flteis based on Daum`s Theoiy, I Journa| o[ CuJante, Conro| anJ
Dynamts, 16, 371-376, 1993.
B. E. Schutz, J. D. McMillan, and B. D. Tapley. Compaiison of statistical oibit deteimination methods, I
J., Nov. 1974.
H. W. Soienson. Recuisive estimation foi nonlineai dynamic systems, Bayesan na|yss o[ Tme Seres anJ
Dynamt MoJe|s, J.C. Spall, Ed., pp. 127-165, Maicel Dekkei, 1988.
H. W. Soienson. Ka|man F|erng. T|eory anJ |taons, IEEE Piess, New Yoik, 1985.
H. W. Soienson. Parameer Esmaon, Maicel Dekkei, 1980.
H. W. Soienson. On the development of piactical nonlineai flteis, In[. St. 7, 253-270, 1974.
H. W. Soienson. On the eiioi behavioi in lineai minimum vaiiance estimation pioblems, IEEE Trans. uom.
Conro|, AC-12, 557-562, 1967.
H. Tanizaki. Non|near F|ers, 2nd ed., Spiingei-Veilag, 1996.
J. H. Tayloi. Ciami-Rao estimation eiioi lowei bound analysis foi nonlineai systems with unknown detei-
ministic vaiiables, IEEE Trans. uom. Conro|, Apiil 1979.
R. P. Wishnei, R. E. Laison, and M. Athans. Status of iadai tiacking algoiithms, Sym. on Non|near Esmaon
T|eory anJ |., 1970.
R. P. Wishnei, J. A. Tabaczynski, and M. Athans. A compaiison of thiee non-lineai flteis, uomata 5, 487-496,
1969.
2000 by CRC Press LLC
Further Inlurmatiun
The best concise intioduction to Kalman flteiing is Chaptei 12 in Biyson and Ho 1969]. The thiee best books
on Kalman flteis aie Gelb 1974], Soienson 1985], and Biown et al. 1997]. The standaid iefeience on
nonlineai flteis is Jazwinski 1970]. The best jouinal on Kalman fltei applications is the IEEE Transatons on
erosate anJ E|etront Sysems, which typically has seveial piactical papeis on Kalman flteis each issue. Two
good confeiences with many papeis on Kalman flteis aie the IEEE Con[erente on Detson anJ Conro| (mid-
Decembei annually) and the SPIE Con[erente on Sgna| anJ Daa Protessng (Apiil each yeai).