Structural Break Detection in Time Series

Richard A. Davis Thomas Lee Gabriel Rodriguez-Yam

Colorado State University (http://www.stat.colostate.edu/~rdavis/lectures)

This research supported in part by an IBM faculty award.

1

Introduction Background Setup for AR models Model Selection Using Minimum Description Length (MDL) General principles Application to AR models with breaks Optimization using Genetic Algorithms Basics Implementation for structural break estimation Simulation examples Piecewise autoregressions Slowly varying autoregressions Applications Multivariate time series EEG example Application to nonlinear models (parameter-driven state-space models) Poisson model Stochastic volatility model
2

Introduction
Structural breaks: Kitagawa and Akaike (1978) fitting locally stationary autoregressive models using AIC computations facilitated by the use of the Householder transformation Davis, Huang, and Yao (1995) likelihood ratio test for testing a change in the parameters and/or order of an AR process. Kitagawa, Takanami, and Matsumoto (2001) signal extraction in seismology-estimate the arrival time of a seismic signal. Ombao, Raz, von Sachs, and Malow (2001) orthogonal complex-valued transforms that are localized in time and frequency- smooth localized complex exponential (SLEX) transform. applications to EEG time series and speech data.
3

Introduction (cont)
Locally stationary: Dahlhaus (1997, 2000,) locally stationary processes estimation Adak (1998) piecewise stationary applications to seismology and biomedical signal processing MDL and coding theory: Lee (2001, 2002) estimation of discontinuous regression functions Hansen and Yu (2001) model selection
4

Introduction (cont)
Time Series: y1, . . . , yn Piecewise AR model:

Yt = j + j1Yt 1 + L + jp j Yt p j + j t , if j-1 t < j ,

where 0 = 1 < 1 < . . . < m-1 < m = n + 1, and {t} is IID(0,1). Goal: Estimate m = number of segments j = location of jth break point j = level in jth epoch pj = order of AR process in jth epoch

( j1 ,K, jp j ) = AR coefficients in jth epoch

j = scale in jth epoch
5

Introduction (cont)
Motivation for using piecewise AR models: Piecewise AR is a special case of a piecewise stationary process (see Adak 1998),
m ~ Yt ,n = Yt j I [ j 1 , j ) (t / n ),

where {Yt }, j = 1, . . . , m is a sequence of stationary processes. It is argued in Ombao et al. (2001), that if {Yt,n} is a locally stationary process (in the sense of ~ Dahlhaus), then there exists a piecewise stationary process {Yt ,n } with
mn with mn / n 0, as n ,

j =1

that approximates {Yt,n} (in average mean square). Roughly speaking: {Yt,n} is a locally stationary process if it has a time-varying spectrum that is approximately |A(t/n,)|2 , where A(u,) is a continuous function in u.
6

Example--Monthly Deaths & Serious Injuries, UK

Data: yt = number of monthly deaths and serious injuries in UK, Jan `75 Dec `84, (t = 1,, 120) Remark: Seat belt legislation introduced in Feb `83 (t = 99).
2200 Counts 1200 1400 1600 1800 2000

1976

1978

1980 Year

1982

1984

Example -- Monthly Deaths & Serious Injuries, UK (cont)

Data: xt = number of monthly deaths and serious injuries in UK, differenced at lag 12; Jan `75 Dec `84, (t = 13,, 120) Remark: Seat belt legislation introduced in Feb `83 (t = 99). Traditional regression analysis:

200

Differenced Counts

Yt = a + bf (t ) + Wt , 0, if 1 t 98, f(t) = 1, if 98 < t 120.

X tt = Ytt Ytt12 X = Y Y 12
0 -200

= bg + N = bg ((tt)) + N tt
1976 1978 1980 Year 1982 1984

-400

1 if 99 110 1,, if 99 tt 110,, g(t) = g(t) = 0, otherwise 0, otherwise..

-600

Model: b=373.4, {Nt}~AR(13).

8

Model Selection Using Minimum Description Length

Basics of MDL: Choose the model which maximizes the compression of the data or, equivalently, select the model that minimizes the code length of the data (i.e., amount of memory required to encode the data).
M = class of operating models for y = (y1, . . . , yn)

LF (y) = code length of y relative to F M Typically, this term can be decomposed into two pieces (two-part code),

LF ( y ) = L(F|y) + L(e | F),

where

L(F|y) = code length of the fitted model for F L( e|F) = code length of the residuals based on the fitted model
9

Model Selection Using Minimum Description Length (cont)

Applied to the segmented AR model: Yt = j + j1Yt 1 + L + jp j Yt p j + j t , if j-1 t < j ,

First term L(F|y) : Let nj = j j-1 and j = ( j , j1 ,K, jp j , j ) denote the length of the jth segment and the parameter vector of the jth AR process,
respectively. Then

L(F|y) = L(m) + L( 1 ,K, m ) + L( p1 ,K, pm ) + L(1 | y ) + L + L( m | y ) = L(m) + L(n1 ,K, nm ) + L( p1 ,K, pm ) + L(1 | y ) + L + L( m | y )
Encoding: integer I : log2 I bits (if I unbounded) log2 IU bits (if I bounded by IU)

MLE : log2N bits (where N = number of observations used to compute ; Rissanen (1989))
10

Model Selection Using Minimum Description Length (cont)

So,

|y) = log2 m + m log2 n + log2 p j + p j + 2 log2 n j L(F 2 j =1 j =1

m m

Second term L( e | F) : Using Shannons classical results on information theory,

Rissanen demonstrates that the code length of e can be approximated by the

negative of the log-likelihood of the fitted model, i.e., by

n L( e | F) j (log2 ( 22j ) + 1) j =1 2 For fixed values of m, (1,p1),. . . , (m,pm), we define the MDL as
MDL(m, ( 1 , p1 ),K, ( m , pm )) = log2 m + m log2 n + log2 p j +
j =1 j =1 m m

pj + 2 2

log2 n j +
j =1

nj 2

log2 (22j ) +

n 2

The strategy is to find the best segmentation that minimizes MDL(m,1,p1,, m,pm). To speed things up, we use Y-W estimates of AR parameters.
11

Optimization Using Genetic Algorithms

Basics of GA: Class of optimization algorithms that mimic natural evolution. Start with an initial set of chromosomes, or population, of possible solutions to the optimization problem. Parent chromosomes are randomly selected (proportional to the rank of their objective function values), and produce offspring using crossover or
mutation operations.

After a sufficient number of offspring are produced to form a second generation, the process then restarts to produce a third generation. Based on Darwins theory of natural selection, the process should produce future generations that give a smaller (or larger) objective function.
12

Application to Structural Breaks(cont)

Genetic Algorithm: Chromosome consists of n genes, each taking the value of 1 (no break) or p (order of AR process). Use natural selection to find a near optimal solution. Map the break points with a chromosome c via
( m, ( 1 , p1 ) K, ( m , pm )) c = ( 1 ,K, n ),

where

1,, if no break point at tt,, 1 if no break point at tt = = p jj,, if break point at time tt = jj11 and AR order is p jj.. p if break point at time = and AR order is p

For example,
c = (2, -1, -1, -1, -1, 0, -1, -1, -1, -1, 0, -1, -1, -1, 3, -1, -1, -1, -1,-1) t: 1 6 11 15

would correspond to a process as follows: AR(2), t=1:5; AR(0), t=6:10; AR(0), t=11:14; AR(3), t=15:20
13

Implementation of Genetic Algorithm(cont)

Generation 0: Start with L (200) randomly generated chromosomes, c1, . . . ,cL with associated MDL values, MDL(c1), . . . , MDL(cL). Generation 1: A new child in the next generation is formed from the chromosomes c1, . . . , cL of the previous generation as follows: with probability c, crossover occurs. two parent chromosomes ci and cj are selected at random with probabilities proportional to the ranks of MDL(ci).
kth gene of child is k = i,k w.p. and j,k w.p.

with probability 1 c, mutation occurs. a parent chromosome ci is selected

kth gene of child is k = i,k w.p. 1 ; 1 w.p. 2 ; and p w.p. 1 12.
14

Implementation of Genetic Algorithm(cont)

Execution of GA: Run GA until convergence or until a maximum number of
generations has been reached. .

Various Strategies: include the top ten chromosomes from last generation in next generation. use multiple islands, in which populations run independently, and then allow migration after a fixed number of generations. This implementation is amenable to parallel computing.

15

Simulation Examples-based on Ombao et al. (2001) test cases

1. Piecewise stationary: Consider a time series following the model,
if 1 t < 513, .9Yt 1 + t , Yt = 1.69Yt 1 .81Yt 2 + t , if 513 t < 769, 1.32Y .81Y + , if 769 t 1024, t 1 t 2 t where {t} ~ IID N(0,1).
10 -10 1 -5 0 5

200

400 Time

600

800

1000

16

1. Piecewise stat (cont)

Implementation: Start with NI = 50 islands, each with population size L = 200. After every Mi = 5 generations, allow migration. Replace worst 2 in Island 3 2 1 4 with best 2 from Island 2. 1. 4. 3. Stopping rule: Stop when the max MDL does not change for 10 consecutive migrations or after 100 migrations. 4

Span configuration for model selection: Max AR order K = 10, p mp p 0 25 .08 1 25 .1 2 30 .1 3 35 .09 4 40 .09 5 45 .09 6-10 50 .09
17

1. Piecewise stat (cont)

GA results: 3 pieces with breaks at 1=513 and 2=769. Total run time 16.31 secs 2 2 Fitted model: 1 1- 512: .857 .9945 513- 768: 1.68 -0.801 1.1134 769-1024: 1.36 -0.801 1.1300 True Model
0.5 0.4 0.4 0.5

Fitted Model

0.3

0.2

0.1

0.0

0.0

0.2

0.4 Time

0.6

0.8

1.0

0.0 0.0

0.1

0.2

0.3

0.2

0.4 Time

0.6

0.8

1.0

18

1. Piecewise stat (cont)

Simulation: 200 replicates of time series of length 1024 were generated. (SLEX results from Ombao et al.)

ASE = n ((1//33)) {log ff ((tt//n,, jj)) log ff ((tt//n,, jj)}22,, ASE = n 1 33 {log n log n )}
11 =1 =0 t t =1 j j =0

nn 32 32

j j = 2jj//64.. = 2 64
GA

# of segments 2 3

% 0 60.0

Auto-SLEX Change Points 1/2 1/4 , 3/4

ASE 25.42 (4.56) 34.09 (6.74) 32.77 (5.20) 50.01 (6.25)

% 0 82.0

mean .500 .749 .476 .616 .761

std .006 .006 .080 .110 .037

ASE 3.64 (.13) 3.73 (.13)

34.0

1/4, 2/4, 3/4

17.5

5 6

5.0 1.0

2/8, 4/8, 5/8, 6/8, 7/8

0 0.5 3.83
19

1. Piecewise stat (cont)

Simulation (cont): True model: if 1 t < 513, .9Yt 1 + t , Yt = 1.69Yt 1 .81Yt 2 + t , if 513 t < 769, 1.32Y .81Y + , if 769 t 1024, t 1 t 2 t AR orders selected (percent): Order p1 p2 p3 0 0 0 0 1
99.4

2 0.60
86.0 89.0

3 0 11.6 10.4

4 0 1.8 0.6

5 0 0.6 0

6 0 0 0

0 0

20

Simulation Examples (cont)

2. Piecewise stationary:

.9Yt 1 + t , if 1 t < 198 Yt = .9Yt 1 + t , if 198 t 1024 where {t} ~ IID N(0,1).

-5 1

200

400 Time

600

800

1000

21

2. Piecewise stationary (cont)

GA results: 2 pieces with break at 1=196. Total run time 11.96 secs Fitted model: 1 2 1- 195: .872 1.081 196 - 1024: -.883 1.078 Fitted Model
0.4 0.0 0.2 0.4 Time 0.6 0.8 1.0 0.0 0.0 0.1 0.2 0.3 0.5

True Model
0.4 0.0 0.1 0.2 0.3 0.5

0.2

0.4 Time

0.6

0.8

1.0

22

2. Piecewise stationary (cont)

Simulation: 200 replicates of time series of length 1024 were generated. AR(1) models with break at 196/1024 =.191. # of segments
4 5

Auto-SLEX % ASE
0 44.0 1.81 (.039) .220 (.056) .211 (.047) .290 (.096) .255 (.100)

# of segmnts
1 2

change points % mean std

0 97.0 .192 .154 .323 .002 .023 .320

ASE
.017 (.013) .060 (.044)

37.0

3 4

3.0

7 8 9

6.0 12.0 1.0

23

Simulation Examples (cont)

3. Piecewise stationary:
.9Yt 1 + t , if 1 t < 51 Yt = .25Yt 1 + t , if 50 t 500

where {t} ~ IID N(0,1). GA results: 2 pieces with break at 1=47

-2 1

100

200 Time

300

400

500

24

3. Piecewise stationary (cont)

Simulation results: Change occurred at time 1 = 51; 51/500=.1

# of segments
1

change points % mean std

9.0 .

89.0

.096

.017

3 4

2.0

.048 .092

.020 .011

25

Simulation Examples (cont)

4. Slowly varying AR(2) model:
Yt = atYt 1 .81Yt 2 + t if 1 t 1024

where at = .8[1 0.5 cos( t / 1024)], and {t} ~ IID N(0,1).

-2

-6

-4

0.4 0

0.6

0.8

a_t

1.0

1.2

200

400 Time

600

800

1000

200

400 time

600

800

1000

26

4. Slowly varying AR(2) (cont)

GA results: 3 pieces with breaks at 1=293 and 2=615. Total run time 27.45 secs Fitted model: 1 2 2 1- 292: .365 -0.753 1.149 293- 614: .821 -0.790 1.176 615-1024: 1.084 -0.760 0.960 True Model
0.5 0.4 0.5

Fitted Model

0.4

0.3

0.2

0.1

0.0

0.0 0.0

0.1

0.2

0.3

0.0

0.2

0.4 Time

0.6

0.8

1.0

0.2

0.4 Time

0.6

0.8

1.0

27

4. Slowly varying AR(2) (cont)

Simulation: 200 replicates of time series of length 1024 were generated. # of segments
4 5

Auto-SLEX % ASE
14.0 27.0 .238 (.030) .228 (.025) .232 (.029) .243 (.033) .269 (.040) .308

# of segments
1 2

change points % mean std

0 36.0 .502 .258 .661 .309 .550 .860 .044 .071 .075

ASE

.127 (.014) .080 (.016)

35.0

63.0

7 8 9

18.0 15.0 1.0

1.0

0
28

4. Slowly varying AR(2) (cont)

Simulation (cont): True model: Yt = atYt 1 .81Yt 2 + t if 1 t 1024

AR orders selected (percent): (2 segment realizations) Order p1 p2 0 0 0 1 0 0 2

97.2 97.2

3 1.4 2.8

4 1.4 0

5 0 0

AR orders selected (percent): (3 segment realizations) Order p1 p2 p3 0 0 0 0 1 0 0 0 2

100 98.4 97.6

3 0 1.6 1.6

4 0 0 0.8

5 0 0 0
29

4. Slowly varying AR(2) (cont)

In the graph below right, we average the spectogram over the GA fitted models generated from each of the 200 simulated realizations.

True Model
0.5
0.5

Average Model

0.4

0.3

0.2

Frequency

0.1

0.0

0.0 0.0

0.1

0.2

0.3

0.4

0.0

0.2

0.4 Time

0.6

0.8

1.0

0.2

0.4 Time

0.6

0.8

1.0

30

Example: Monthly Deaths & Serious Injuries, UK

Data: Yt = number of monthly deaths and serious injuries in UK, Jan `75 Dec `84, (t = 1,, 120) Remark: Seat belt legislation introduced in Feb `83 (t = 99).

Counts

1200

1400

1600

1800

2000

2200

1976

1978

1980 Year

1982

1984

31

Example: Monthly Deaths & Serious Injuries, UK

Data: Yt = number of monthly deaths and serious injuries in UK, Jan `75 Dec `84, (t = 1,, 120) Remark: Seat belt legislation introduced in Feb `83 (t = 99).
200 Differenced Counts -600 1976 -400 -200 0

1978

1980 Year

1982

1984

Results from GA: 3 pieces; time = 4.4secs Piece 1: (t=1,, 98) IID; Piece 2: (t=99,108) IID; Piece 3: t=109,,120 AR(1)
32

Application to Multivariate Time Series

Multivariate time series (d-dimensional): y1, . . . , yn Piecewise AR model:
Yt = j + j1Yt 1 + L + jp j Yt p j + 1j/ 2 Zt , if j-1 t < j ,

where 0 = 1 < 1 < . . . < m-1 < m = n + 1, and {t} is IID(0, Id). In this case,

MDL( m, ( 1 , p1 ),K, ( m , pm )) = log m + m log n + log p j

j =1

+
j =1

p j d 2 + d + d ( d + 1) / 2 2

1 j log n j + log(| Vt |) + ( Yt Yt )T Vt 1 ( Yt Yt ) , j =1 2 t = j 1

where Yt = E ( Yt | Yt ,K Y1 ) and Vt = E ( Yt Yt )2 and the AR parameters are

estimated by the multivariate Y-W equations based on Whittles generalization of the Durbin-Levinson algorithm.
33

Example: Bivariate Time Series

{Yt1} same as the series in Example 2 (3 segments: AR(1), AR(3), AR(2)) {Yt2} same as the series in Example 1 (2 segments: AR(1), AR(1)) GA results: TS 1: 3 pieces with breaks at 1=513 and 2=769. Total run time 16.31 secs TS 2: 2 pieces with break at 1=196. Total run time 11.96 secs Bivariate: 4 pieces with breaks at 1=197, 2=519, 3=769: AR(1), AR(1), AR(2), AR(2) Total run time 1126 secs
10

-5

-10

-5 1

200

400 Time

600

800

1000

200

400 Time

600

800

1000

34

Example: EEG Time series

Data: Bivariate EEG time series at channels T3 (left temporal) and P3 (left parietal). Female subject was diagnosed with left temporal lobe epilepsy. Data collected by Dr. Beth Malow and analyzed in Ombao et al (2001). (n=32,768; sampling rate of 100H). Seizure started at about 1.85 seconds. GA univariateresults: 11 pieces with AR orders, 17, 2, 6 15, 2,for 5, 9, 5, 4, 1 bivariate results: 14 breakpoints for T3; 11 breakpoints 3, P3 T3 Channel
200 0

P3 Channel

EEG P3 channel 1 50 100 150 200 250 300

EEG T3 channel

-200

-400

-600

-400 1

-300

-200

-100

50

100

150

200

250

300

Time in seconds

Time in seconds

35

Example: EEG Time series (cont)

Remarks: the general conclusions of this analysis are similar to those reached in Ombao et al. prior to seizure, power concentrated at lower frequencies and then spread to high frequencies. power returned to the lower frequencies at conclusion of seizure. T3 Channel
50 40 40 50

P3 Channel

Frequency (Hertz)

Frequency (Hertz) 1 50 100 150 200 250 300

30

20

10

0 1

10

20

30

50

100

150

200

250

300

Time in seconds

Time in seconds

36

Example: EEG Time series (cont)

Remarks (cont): T3 and P3 strongly coherent at 9-12 Hz prior to seizure. strong coherence at low frequencies just after onset of seizure. strong coherence shifted to high frequencies during the seizure. T3/P3 Coherency
50 Frequency (Hertz) 0 1 10 20 30 40

50

100

150

200

250

300

37

Time in seconds

Application to Structural Breaks (Davis, Lee, Rodriguez-Yam)

State Space Model Setup: Observation equation: p(yt | t) = exp{t yt b(t) + c(yt)}. State equation: {t} follows the piecewise AR(1) model given by t = k + kt-1 + kt , if k-1 t < k , where 1 = 0 < 1 < < m < n, and {t } ~ IID N(0,1). Parameters: m = number of break points k = location of break points k = level in kth epoch k = AR coefficients kth epoch k = scale in kth epoch
38

Application to Structural Breaks(cont)

Estimation: For (m, 1, . . . , m) fixed, calculate the approximate likelihood evaluated at the MLE, i.e.,
| Gn |1 / 2 exp{yT * 1T {b( * ) c( y n )} ( * )T Gn ( * ) / 2}, La (; y n ) = n (K + Gn )1 / 2
2 2 where = ( 1 ,K , m , 1 ,K , m , 1 ,K , m ) is the MLE.

Goal: Optimize an objective function over (m, 1, . . . , m).

use minimum description length (MDL) as an objective function use genetic algorithm for optimization

39

Application to Structural Breaks(cont)

Minimum Description Length (MDL): Choose the model which maximizes the compression of the data or, equivalently, select the model that minimizes the code length of the data (i.e., amount of memory required to store the data). Code Length(data) = CL(fitted model) + CL(data | fitted model) ~ CL(parameters) + CL(residuals)
MDL m K m MDL((m,,11,,K,,m ))
= log(m ))+ m log(n ))+ 1..5 log( jj jj11)) log((Laa(( jj;;y j 11:j j)) = log(m + m log(n + 1 5 log( log L y j : )) =1 j =1 1444444 24444444 1144424443 4 3 jj==1 1444444 24j444444 144424443 4 3 CL(" Parameters" )) CL(" residuals" )) CL(" Parameters" CL(" residuals"
m m m m

Generalization: AR(p) segments can have unknown order.

MDL m p ),K m pm )) MDL((m,,((11,, p11),K ,,(( m ,, pm ))
= log(m ))+ m log(n ))+ 0..5 ((p jj + 2))log( jj jj11)) log((Laa(( jj;;y j 11:j j)) = log(m + m log(n + 0 5 p + 2 log( log L y j : ))
=1 jj =1 =1 jj =1 m m m m

40

Count Data Example

Model: Yt | t Pois(exp{ + t }), t = t-1+ t , {t}~IID N(0, 2)
15 1014

blue =AL red = IS

MDL
5 0 1 100 200 300 400 500

1002 1

1004

1006

1008

1010

10

1012

100

200

300

400

500

time

Breaking Point

True model: Yt | t Pois(exp{.7 + t }), t = .5t-1+ t , {t}~IID N(0, .3), t < 250 Yt | t Pois(exp{.7 + t }), t = .5t-1+ t , {t}~IID N(0, .3), t > 250. GA estimate 251, time 267secs
41

SV Process Example
Model: Yt | t N(0,exp{t}), t = + t-1+ t , {t}~IID N(0, 2)
1315 3 2

MDL
1 500 1000 1500 2000 2500 3000

-2

1295 1

1300

-1

1305

1310

500

1000

1500

2000

2500

3000

time

Breaking Point

True model: Yt | t N(0, exp{t}), t = -.05 + .975t-1+ t , {t}~IID N(0, .05), t 750 Yt | t N(0, exp{t }), t = -.25 +.900t-1+ t , {t}~IID N(0, .25), t > 750. GA estimate 754, time 1053 secs
42

SV Process Example
Model: Yt | t N(0,exp{t}), t = + t-1+ t , {t}~IID N(0, 2)
-505
0.5

MDL
1 100 200 300 400 500

0.0

-0.5

-530 1

-525

-520

-515

-510

-500

100

200

300

400

500

time

Breaking Point

True model: Yt | t N(0, exp{t}), t = -.175 + .977t-1+ t , {t}~IID N(0, .1810), t 250 Yt | t N(0, exp{t }), t = -.010 +.996t-1+ t , {t}~IID N(0, .0089), t > 250. GA estimate 251, time 269s
43

SV Process Example-(cont)
True model: Yt | t N(0, exp{t}), t = -.175 + .977t-1+ t , {t}~IID N(0, .1810), t 250 Yt | t N(0, exp{t }), t = -.010 +.996t-1+ t , {t}~IID N(0, .0089), t > 250. Fitted model based on no structural break: Yt | t N(0, exp{t}), t = -.0645 + .9889t-1+ t , {t}~IID N(0, .0935) original series
0.5

simulated series

0.0

y
-0.5 1 0.0

-0.5

0.5

1.0

100

200

300

400

500

100

200

300

400

500

time

time

44

SV Process Example-(cont)
Fitted model based on no structural break: Yt | t N(0, exp{t}), t = -.0645 + .9889t-1+ t , {t}~IID N(0, .0935) simulated series
1.0

0.5

MDL
1 100 200 300 400 500

0.0

-478 1

-476

-0.5

-474

-472

-470

-468

-466

100

200

300

400

500

time

Breaking Point

45

Summary Remarks
1. MDL appears to be a good criterion for detecting structural breaks. 2. Optimization using a genetic algorithm is well suited to find a near optimal value of MDL. 3. This procedure extends easily to multivariate problems. 4. While estimating structural breaks for nonlinear time series models is more challenging, this paradigm of using MDL together GA holds promise for break detection in parameter-driven models and other nonlinear models.

46