Lecture 21 : Continuous Time Markov Chains

STAT 150 Spring 2006 Lecturer: Jim Pitman Scribe: Stephen Bianchi <>

(These notes also include material from the subsequent guest lecture given by Ani Adhikari.) Consider a continuous time stochastic process (Xt , t 0) taking on values in the nite state space S = {0, 1, 2, . . . , N }. Recall that in discrete time, given a transition matrix P = p(i, j ) with i, j S , the n-step transition matrix is simply P n = pn (i, j ), and that the following relationship holds: P n P m = P n+m . Where pn (i, j ) is the probability that the process moves from state i to state j in n transitions. That is, pn (i, j ) = Pi (Xn = j ), pn (i, j ) = P(Xn = j |X0 = i), pn (i, j ) = P(Xm+n = j |Xm = i). Now moving to continuous time, we say that the process (Xt , t 0) is a continuous time markov chain if the following properties hold for all i, j S , t, s 0: Pt (i, j ) 0,
N j =0

Pt (i, j ) = 1,
N k =0 N k =0

P(Xt+s = j |X0 = i) = P(Xt+s = j |X0 = i) =

P(Xt+s = j |Xs = k )P(Xs = k |X0 = i), or Ps (i, k )Pt (k, j ).

This last property can be written in matrix form as Ps+t = Ps Pt . This is known as the Chapman-Kolmogorov equation (the semi-group property). Example 21.1 (Poissonize a Discrete Time Markov Chain) Consider a pois , which son process (Nt , t 0) with rate , and a jump chain with transition matrix P 21-1

Lecture 21: Continuous Time Markov Chains

21-2

makes jumps at times (Nt , t 0). Let Xt = YNt (YNt is the value of the jump chain at time Nt ), where Xt takes values in the discrete state space S . Assume (Y0 , Y1 , . . .) and Nt are independent. Find Pt for the process Xt . By denition Pt (i, j ) = P(Xt = j |X0 = i). Condition on Nt to give,

Pt (i, j ) =
n=0

n (i, j ) P(Nt = n)P et (t)n n P (i, j ) n!

=
n=0

= e = e Dropping indices then gives,

t n=0

)n (i, j ) (tP n! .

t tP

Pt = et(P I ) . Pt is a paradigm example of a continuous time semi-group of transition matrices. It is easy to check that Pt Ps = Pt+s , since eA+B = eA eB .

21.1

Generalization

Let A be a matrix of transition rates, where qij = A(i, j ) is the rate of transitions from state i to state j (i = j ), qij 0, and qi = A(i, i) is the total rate of transitions out of state i. Hence, qi = N j =0,j =i qij and each row of A sums to 0. Also, since Pt is continuous and dierentiable we have qij = lim qi Ph (i, j ) h0 h 1 Ph (i, i) . = lim h0 h

In words, qij is the probability per unit time of transitioning from state i to state j in a small time increment (h). The limit relations above can be expressed in matrix form, Pt+h Pt Pt Ph P t Pt [Ph I ] d Pt = lim = lim = lim h0+ h0+ h0+ dt h h h

Lecture 21: Continuous Time Markov Chains

21-3

hence

Ph I d d Pt = Pt lim = Pt P0 = Pt A = APt , h 0+ dt h dt
d P. dt 0

where A =

Given the initial condition P0 = I , the solution to the equation d Pt = Pt A = APt dt

d d is Pt = eAt . dt Pt = Pt A is known as the Kolmogorov forward equations and dt Pt = APt is known as the Kolmogorov backward equations. We claim that (Pt , t 0) is a collection of transition matrices of some continuous time markov chain.

In the example given above we have, I) A = (P (i, i) 1) A(i, i) = (P (i, j ). A(i, j ) = P Example 21.2 Consider the two state markov chain with states {0, 1} and A= .

The sojourn times in state 0 and 1 are independent and exponentially distributed with parameters and , respectively. Find Pt for this chain. First note that d Pt = APt = dt Then Pt (0, 0) + Pt (1, 0) Pt (0, 1) + Pt (1, 1) Pt (0, 0) Pt (1, 0) Pt (0, 1) Pt (1, 1) .

d (Pt (0, 0) Pt (1, 0)) = ( + )(Pt (0, 0) Pt (1, 0)). dt This equation has the form f (t) = cf (t), and a solution is Pt (0, 0) Pt (1, 0) = ce(+)t . Since c evaluated at t = 0 is one (i.e., P0 (0, 0) P0 (1, 0) = 1)), we have Pt (0, 0) Pt (1, 0) = e(+)t , which implies d Pt (0, 0) = e(+)t . dt

Lecture 21: Continuous Time Markov Chains

21-4

Then
t

Pt (0, 0) = P0 (0, 0) +
0

e(+)s ds

(+)t Pt (0, 0) = 1 + e + + (+)t + e . Pt (0, 0) = + + Likewise (+)t e + + (+)t e Pt (0, 1) = + + (+)t Pt (1, 1) = + e . + + Pt (1, 0) = Notice that as t Pt (0, 0) = Pt (1, 0) = + Pt (0, 1) = Pt (1, 1) = . +

These limits are independent of the initial state. The stationary distribution of the chain is 0 = + . 1 = +

21.2

Jump Hold Description

Let (Xt , t 0) be a markov chain with transition rate matrix A, such that A(i, j ) 0 for i = j and A(i, i) = i=j A(i, j ). Let Hi be the holding time in state i (i.e., the process holds in state i for an amount of time Hi and then jumps to another state). The markov property implies Pi (Hi > s + t|Hi > s) = Pi (Hi > t). We also recognize this as the memoryless property of the exponential distribution (fact: the only continuous distribution with the memoryless property is the exponential).

Lecture 21: Continuous Time Markov Chains

21-5

By the idea of competing risks we will see that Pi (Hi > t) = eA(i,i)t . Imagine that each state j = i has a random variable j exp(A(i, i)). Think of a bell that rings in state j at time j , and from state i the process moves to the state where the bell rings rst. Construct Hi = mini=j j , then Pi (Hi > t) = P(
i=j

j > t)

= i=j eA(i,j )t P = e( i=j A(i,j ))t = eA(i,i)t , where A(i, i) < 0. What is the probability is that the rst transition out of state i is to state k ? This is given by

P(min j = k ) =
i=j 0

P(min dt, min = k )dt P(k dt, all the others are > t)dt
0

= =
0

A(i, k )eA(i,k)t e( A(i, k )e(

0 P
i=j

i=k,j

A(i,k ))t

dt

= =

A(i,j ))t

dt

A(i, k ) A(i, k ) = . A(i, i) i=j A(i, j )

Observe from this calculation that the following mechanisms are equivalent: 1. (a) competing random variables j exp(A(i, j )) for state j = i (b) Hi = minj =i j (c) jump to the state j which attains the minimum 2. (a) Hi exp(A(i, i) =
i=j

A(i, j ))
A(i,k ) A(i,i)

(b) at time Hi jump to k with probability

Example 21.3 (Hold Rates for a Poissonized Chain) Consider a general tran , where some of the diagonal entries maybe greater than zero. So we sition matrix P have Pt A A(i, i) A(i, j ) = = = = et(P I ) I) (P (i, i) 1) = (1 P (i, i)) (P (i, j )). (P

Lecture 21: Continuous Time Markov Chains

21-6

(i, i)) = Hence A(i, i) = (1 P i=j P (i, j ) = i=j A(i, j ) is the total rate of (i, j )) is the rate of i to j transitions. Let transitions out of state i, and A(i, j ) = (P Hi = S 1 + S 2 + + S G , where S1 , S2 , . . . exp() are independent. Dene G to be the number of steps of the chain until the process leaves state i. Then P (i, i)n1 (1 P (i, i)). P(G = n) = P (i, i) and Hi has an exponential So G has a geometric distribution with parameter 1 P distribution with parameter (1 P (i, i)). It is easy to check the expectations, since (i, i)), and E[Hi ] = 1/(1 P (i, i)). Then by Walds E[Si ] = 1/, E[G] = 1/(1 P identity 1 1 1 . = E[Hi ] = E[Si ]E[G] = (i, i) (i, i)) 1P (1 P

21.3

Theorem

Theorem 21.4 (for nite state irreducible chain) There is a unique probability distribution so that 1. limt Pt (i, j ) = j for all i. 2. solves the balance equations A = 0 (equivalently, Pt = for all t). 3. j is the long-run proportion of time spent in state j 4. the expected return time is Ej [Tj ] = Proof: [for part (2)] Assume (1) is true. Ps+t (i, j ) =
k

1 . q j j

Ps (i, k )Pt (k, j )

Letting s we get j =
k

k Pt (k, j )

Therefore, = Pt for all t. Dierentiating with respect to t then gives 0 = A.

Lecture 21: Continuous Time Markov Chains

21-7

Look at the j th element of A (which we get by multiplying by the j th column of A). We see that i qij j qj = 0
i=j

which gives i qij = j qj .

i=j

This is the reason these equations are referred to as balance equations. Computations can often be simplied if there is also detailed balance, which simply means that for all pairs i, j (i = j ) we have i qij = j qji . This is the condition of reversibility of the markov chain. Example 21.5 (Birth-Death Process) Consider a process that, from state i, can only move to state i + 1 or state i 1. That is qi,i+1 = i qi,i1 = i qi,j = 0, j {i + 1, i 1}. So i is the birth rate and i is the death rate. We try to nd so that 0 q0,1 = 1 q1,0 1 q1,2 = 2 q2,1 . . . i1 qi1,i = i qi,i1 . Then 0 q0,1 = 1 q1,0 0 0 = 1 1 0 1 = 0 . 1 And 1 q1,2 = 2 q2,1 1 1 = 2 2 1 0 1 1 = 0 . 2 = 2 1 2

Lecture 21: Continuous Time Markov Chains

21-8

Hence k = and 0 =

0 1 k1 0 , 1 2 k 1
0 1 k1 k =1 1 2 k

1+