Introduction to Time Series Analysis. Lecture 6.

Peter Bartlett
www.stat.berkeley.edu/bartlett/courses/153-fall2010
Last lecture:
1. Causality
2. Invertibility
3. AR(p) models
4. ARMA(p,q) models
1
Introduction to Time Series Analysis. Lecture 6.
Peter Bartlett
www.stat.berkeley.edu/bartlett/courses/153-fall2010
1. ARMA(p,q) models
2. Stationarity, causality and invertibility
3. The linear process representation of ARMA processes: .
4. Autocovariance of an ARMA process.
5. Homogeneous linear difference equations.
2
Review: Causality
A linear process {X
t
} is causal (strictly, a causal function
of {W
t
}) if there is a
(B) =
0
+
1
B +
2
B
2
+
with

j=0
|
j
| <
and X
t
= (B)W
t
.
3
Review: Invertibility
A linear process {X
t
} is invertible (strictly, an invertible
function of {W
t
}) if there is a
(B) =
0
+
1
B +
2
B
2
+
with

j=0
|
j
| <
and W
t
= (B)X
t
.
4
Review: AR(p), Autoregressive models of order p
An AR(p) process {X
t
} is a stationary process that satises
X
t

1
X
t1

p
X
tp
= W
t
,
where {W
t
} WN(0,
2
).
Equivalently, (B)X
t
= W
t
,
where (B) = 1
1
B
p
B
p
.
5
Review: AR(p), Autoregressive models of order p
Theorem: A (unique) stationary solution to (B)X
t
= W
t
exists iff the roots of (z) avoid the unit circle:
|z| = 1 (z) = 1
1
z
p
z
p
= 0.
This AR(p) process is causal iff the roots of (z) are outside
the unit circle:
|z| 1 (z) = 1
1
z
p
z
p
= 0.
6
Reminder: Polynomials of a complex variable
Every degree p polynomial a(z) can be factorized as
a(z) = a
0
+ a
1
z + + a
p
z
p
= a
p
(z z
1
)(z z
2
) (z z
p
),
where z
1
, . . . , z
p
C are the roots of a(z). If the coefcients a
0
, a
1
, . . . , a
p
are all real, then the roots are all either real or come in complex conjugate
pairs, z
i
= z
j
.
Example: z + z
3
= z(1 + z
2
) = (z 0)(z i)(z + i),
that is, z
1
= 0, z
2
= i, z
3
= i. So z
1
R; z
2
, z
3
R; z
2
= z
3
.
Recall notation: A complex number z = a + ib has Re(z) = a, Im(z) = b,
z = a ib, |z| =

a
2
+ b
2
, arg(z) = tan
1
(b/a) (, ].
7
Review: Calculating for an AR(p): general case
(B)X
t
= W
t
, X
t
= (B)W
t
so 1 = (B)(B)
1 = (
0
+
1
B + )(1
1
B
p
B
p
)
1 =
0
, 0 =
j
(j < 0),
0 = (B)
j
(j > 0).
We can solve these linear difference equations in several ways:
numerically, or
by guessing the form of a solution and using an inductive proof, or
by using the theory of linear difference equations.
8
Introduction to Time Series Analysis. Lecture 6.
1. Review: Causality, invertibility, AR(p) models
2. ARMA(p,q) models
3. Stationarity, causality and invertibility
4. The linear process representation of ARMA processes: .
5. Autocovariance of an ARMA process.
6. Homogeneous linear difference equations.
9
ARMA(p,q): Autoregressive moving average models
An ARMA(p,q) process {X
t
} is a stationary process that
satises
X
t

1
X
t1

p
X
tp
= W
t
+
1
W
t1
+ +
q
W
tq
,
where {W
t
} WN(0,
2
).
AR(p) = ARMA(p,0): (B) = 1.
MA(q) = ARMA(0,q): (B) = 1.
10
ARMA(p,q): Autoregressive moving average models
An ARMA(p,q) process {X
t
} is a stationary process that
satises
X
t

1
X
t1

p
X
tp
= W
t
+
1
W
t1
+ +
q
W
tq
,
where {W
t
} WN(0,
2
).
Usually, we insist that
p
,
q
= 0 and that the polynomials
(z) = 1
1
z
p
z
p
, (z) = 1 +
1
z + +
q
z
q
have no common factors. This implies it is not a lower order ARMA model.
11
ARMA(p,q): An example of parameter redundancy
Consider a white noise process W
t
. We can write
X
t
= W
t
X
t
X
t1
+ 0.25X
t2
= W
t
W
t1
+ 0.25W
t2
(1 B + 0.25B
2
)X
t
= (1 B + 0.25B
2
)W
t
This is in the form of an ARMA(2,2) process, with
(B) = 1 B + 0.25B
2
, (B) = 1 B + 0.25B
2
.
But it is white noise.
12
ARMA(p,q): An example of parameter redundancy
ARMA model: (B)X
t
= (B)W
t
,
with (B) = 1 B + 0.25B
2
,
(B) = 1 B + 0.25B
2
X
t
= (B)W
t
(B) =
(B)
(B)
= 1.
i.e., X
t
= W
t
.
13
Introduction to Time Series Analysis. Lecture 6.
1. Review: Causality, invertibility, AR(p) models
2. ARMA(p,q) models
3. Stationarity, causality and invertibility
4. The linear process representation of ARMA processes: .
5. Autocovariance of an ARMA process.
6. Homogeneous linear difference equations.
14
Recall: Causality and Invertibility
A linear process {X
t
} is causal if there is a
(B) =
0
+
1
B +
2
B
2
+
with

j=0
|
j
| < and X
t
= (B)W
t
.
It is invertible if there is a
(B) =
0
+
1
B +
2
B
2
+
with

j=0
|
j
| < and W
t
= (B)X
t
.
15
ARMA(p,q): Stationarity, causality, and invertibility
Theorem: If and have no common factors, a (unique) sta-
tionary solution to (B)X
t
= (B)W
t
exists iff the roots of
(z) avoid the unit circle:
|z| = 1 (z) = 1
1
z
p
z
p
= 0.
This ARMA(p,q) process is causal iff the roots of (z) are out-
side the unit circle:
|z| 1 (z) = 1
1
z
p
z
p
= 0.
It is invertible iff the roots of (z) are outside the unit circle:
|z| 1 (z) = 1 +
1
z + +
q
z
q
= 0.
16
ARMA(p,q): Stationarity, causality, and invertibility
Example: (1 1.5B)X
t
= (1 + 0.2B)W
t
.
(z) = 1 1.5z =
3
2
_
z
2
3
_
,
(z) = 1 + 0.2z =
1
5
(z + 5) .
1. and have no common factors, and s root is at 2/3, which is not on
the unit circle, so {X
t
} is an ARMA(1,1) process.
2. s root (at 2/3) is inside the unit circle, so {X
t
} is not causal.
3. s root is at 5, which is outside the unit circle, so {X
t
} is invertible.
17
ARMA(p,q): Stationarity, causality, and invertibility
Example: (1 + 0.25B
2
)X
t
= (1 + 2B)W
t
.
(z) = 1 + 0.25z
2
=
1
4
_
z
2
+ 4
_
=
1
4
(z + 2i)(z 2i),
(z) = 1 + 2z = 2
_
z +
1
2
_
.
1. and have no common factors, and s roots are at 2i, which is not
on the unit circle, so {X
t
} is an ARMA(2,1) process.
2. s roots (at 2i) are outside the unit circle, so {X
t
} is causal.
3. s root (at 1/2) is inside the unit circle, so {X
t
} is not invertible.
18
Causality and Invertibility
Theorem: Let {X
t
} be an ARMA process dened by
(B)X
t
= (B)W
t
. If all |z| = 1 have (z) = 0, then there
are polynomials

and

and a white noise sequence

W
t
such
that {X
t
} satises

(B)X
t
=

(B)

W
t
, and this is a causal,
invertible ARMA process.
So well stick to causal, invertible ARMA processes.
19
Introduction to Time Series Analysis. Lecture 6.
1. Review: Causality, invertibility, AR(p) models
2. ARMA(p,q) models
3. Stationarity, causality and invertibility
4. The linear process representation of ARMA processes: .
5. Autocovariance of an ARMA process.
6. Homogeneous linear difference equations.
20
Calculating for an ARMA(p,q): matching coefcients
Example: X
t
= (B)W
t
(1 + 0.25B
2
)X
t
= (1 + 0.2B)W
t
,
so 1 + 0.2B = (1 + 0.25B
2
)(B)
1 + 0.2B = (1 + 0.25B
2
)(
0
+
1
B +
2
B
2
+ )
1 =
0
,
0.2 =
1
,
0 =
2
+ 0.25
0
,
0 =
3
+ 0.25
1
,
.
.
.
21
Calculating for an ARMA(p,q): example
1 =
0
, 0.2 =
1
,
0 =
j
+ 0.25
j2
(j 2).
We can think of this as
j
= (B)
j
, with
0
= 1,
j
= 0 for j < 0, j > q.
This is a rst order difference equation in the
j
s.
We can use the
j
s to give the initial conditions and solve it using the theory
of homogeneous difference equations.

j
=
_
1,
1
5
,
1
4
,
1
20
,
1
16
,
1
80
,
1
64
,
1
320
, . . .
_
.
22
Calculating for an ARMA(p,q): general case
(B)X
t
= (B)W
t
, X
t
= (B)W
t
so (B) = (B)(B)
1 +
1
B + +
q
B
q
= (
0
+
1
B + )(1
1
B
p
B
p
)
1 =
0
,

1
=
1

0
,

2
=
2

1

2

0
,
.
.
.
This is equivalent to
j
= (B)
j
, with
0
= 1,
j
= 0 for j < 0, j > q.
23
Introduction to Time Series Analysis. Lecture 6.
1. Review: Causality, invertibility, AR(p) models
2. ARMA(p,q) models
3. Stationarity, causality and invertibility
4. The linear process representation of ARMA processes: .
5. Autocovariance of an ARMA process.
6. Homogeneous linear difference equations.
24
Autocovariance functions of linear processes
Consider a (mean 0) linear process {X
t
} dened by X
t
= (B)W
t
.
(h) = E(X
t
X
t+h
)
= E(
0
W
t
+
1
W
t1
+
2
W
t2
+ )
(
0
W
t+h
+
1
W
t+h1
+
2
W
t+h2
+ )
=
2
w
(
0

h
+
1

h+1
+
2

h+2
+ ) .
25
Autocovariance functions of MA processes
Consider an MA(q) process {X
t
} dened by X
t
= (B)W
t
.
(h) =
_
_
_

2
w

qh
j=0

j

j+h
if h q,
0 if h > q.
26
Autocovariance functions of ARMA processes
ARMA process: (B)X
t
= (B)W
t
.
To compute , we can compute , and then use
(h) =
2
w
(
0

h
+
1

h+1
+
2

h+2
+ ) .
27
Autocovariance functions of ARMA processes
An alternative approach:
X
t

1
X
t1

p
X
tp
= W
t
+
1
W
t1
+ +
q
W
tq
,
so E((X
t

1
X
t1

p
X
tp
) X
th
)
= E((W
t
+
1
W
t1
+ +
q
W
tq
) X
th
) ,
that is, (h)
1
(h 1)
p
(h p)
= E(
h
W
th
X
th
+ +
q
W
tq
X
th
)
=
2
w
qh

j=0

h+j

j
. (Write
0
= 1).
This is a linear difference equation.
28
Autocovariance functions of ARMA processes: Example
(1 + 0.25B
2
)X
t
= (1 + 0.2B)W
t
, X
t
= (B)W
t
,

j
=
_
1,
1
5
,
1
4
,
1
20
,
1
16
,
1
80
,
1
64
,
1
320
, . . .
_
.
(h)
1
(h 1)
2
(h 2) =
2
w
qh

j=0

h+j

j
(h) + 0.25(h 2) =
_

2
w
(
0
+ 0.2
1
) if h = 0,
0.2
2
w

0
if h = 1,
0 otherwise.
29
Autocovariance functions of ARMA processes: Example
We have the homogeneous linear difference equation
(h) + 0.25(h 2) = 0
for h 2, with initial conditions
(0) + 0.25(2) =
2
w
(1 + 1/25)
(1) + 0.25(1) =
2
w
/5.
We can solve these linear equations to determine .
Or we can use the theory of linear difference equations...
30
Introduction to Time Series Analysis. Lecture 6.
Peter Bartlett
www.stat.berkeley.edu/bartlett/courses/153-fall2010
1. ARMA(p,q) models
2. Stationarity, causality and invertibility
3. The linear process representation of ARMA processes: .
4. Autocovariance of an ARMA process.
5. Homogeneous linear difference equations.
31
Difference equations
Examples:
x
t
3x
t1
= 0 (rst order, linear)
x
t
x
t1
x
t2
= 0 (2nd order, nonlinear)
x
t
+ 2x
t1
x
2
t3
= 0 (3rd order, nonlinear)
32
Homogeneous linear diff eqns with constant coefcients
a
0
x
t
+ a
1
x
t1
+ + a
k
x
tk
= 0

_
a
0
+ a
1
B + + a
k
B
k
_
x
t
= 0
a(B)x
t
= 0
auxiliary equation: a
0
+ a
1
z + + a
k
z
k
= 0
(z z
1
)(z z
2
) (z z
k
) = 0
where z
1
, z
2
, . . . , z
k
C are the roots of this characteristic polynomial.
Thus,
a(B)x
t
= 0 (B z
1
)(B z
2
) (B z
k
)x
t
= 0.
33
Homogeneous linear diff eqns with constant coefcients
a(B)x
t
= 0 (B z
1
)(B z
2
) (B z
k
)x
t
= 0.
So any {x
t
} satisfying (Bz
i
)x
t
= 0 for some i also satises a(B)x
t
= 0.
Three cases:
1. The z
i
are real and distinct.
2. The z
i
are complex and distinct.
3. Some z
i
are repeated.
34
Homogeneous linear diff eqns with constant coefcients
1. The z
i
are real and distinct.
a(B)x
t
= 0
(B z
1
)(B z
2
) (B z
k
)x
t
= 0
x
t
is a linear combination of solutions to
(B z
1
)x
t
= 0, (B z
2
)x
t
= 0, . . . , (B z
k
)x
t
= 0
x
t
= c
1
z
t
1
+ c
2
z
t
2
+ + c
k
z
t
k
,
for some constants c
1
, . . . , c
k
.
35
Homogeneous linear diff eqns with constant coefcients
1. The z
i
are real and distinct. e.g., z
1
= 1.2, z
2
= 1.3
0 2 4 6 8 10 12 14 16 18 20
0.8
0.6
0.4
0.2
0
0.2
0.4
0.6
0.8
1
c
1
z
1
t
+ c
2
z
2
t
c
1
=1, c
2
=0
c
1
=0, c
2
=1
c
1
=0.8, c
2
=0.2
36
Reminder: Complex exponentials
a + ib = re
i
= r(cos + i sin),
where r = |a + ib| =
_
a
2
+ b
2
= tan
1
_
b
a
_
(, ].
Thus, r
1
e
i
1
r
2
e
i
2
= (r
1
r
2
)e
i(
1
+
2
)
,
z z = |z|
2
.
37
Homogeneous linear diff eqns with constant coefcients
2. The z
i
are complex and distinct.
As before, a(B)x
t
= 0
x
t
= c
1
z
t
1
+ c
2
z
t
2
+ + c
k
z
t
k
.
If z
1
R, since a
1
, . . . , a
k
are real, we must have the complex conjugate
root, z
j
= z
1
. And for x
t
to be real, we must have c
j
= c
1
. For example:
x
t
= c z
t
1
+ c z
1
t
= r e
i
|z
1
|
t
e
it
+ r e
i
|z
1
|
t
e
it
= r|z
1
|
t
_
e
i(t)
+ e
i(t)
_
= 2r|z
1
|
t
cos(t )
where z
1
= |z
1
|e
i
and c = re
i
.
38
Homogeneous linear diff eqns with constant coefcients
2. The z
i
are complex and distinct. e.g., z
1
= 1.2 + i, z
2
= 1.2 i
0 2 4 6 8 10 12 14 16 18 20
1
0.8
0.6
0.4
0.2
0
0.2
0.4
0.6
0.8
1
c
1
z
1
t
+ c
2
z
2
t
c=1.0+0.0i
c=0.0+1.0i
c=0.80.2i
39
Homogeneous linear diff eqns with constant coefcients
2. The z
i
are complex and distinct. e.g., z
1
= 1 + 0.1i, z
2
= 1 0.1i
0 10 20 30 40 50 60 70
2
1.5
1
0.5
0
0.5
1
1.5
2
c
1
z
1
t
+ c
2
z
2
t
c=1.0+0.0i
c=0.0+1.0i
c=0.80.2i
40