Dierence Equation and Markov Chain Applications

A dierence equation is an equation of the form xk = Axk1 where x is a vector and A is an n n matrix. A Markov chain is a process where the future state depends only on the most recent state; such a processes can be encoded as a dierence equation where the entries in each column of A sum to 1. The book mentions that dierence equations . . . occur in a variety of applications and serve as mathematical models to describe population growth, ecological systems, radar tracking of airborne objects, digital control of chemical processes, and the like. Given this wealth of applications, the book has amazingly few examples and no interesting homework problems. Here is a small selection of problems to show o some applications.

Recurrence Relations

A recurrence relation is a sequence of numbers dened in terms of previous numbers. For example,

x0 = 0,

x1 = 1,

xn = 5xn1 6xn2

You can use the recurrence to compute new terms in the sequence iteratively: x0 = 0 x1 = 1 x2 = 5x1 6x0 = 5(1) 6(0) = 5 x3 = 5x2 6x1 = 5(5) 6(1) = 19 x4 = 5x3 6x2 = 5(19) 6(5) = 65 x5 = 5x4 6x3 = 5(65) 6(19) = 211 . . . 1

We can nd a formula for a recurrence relation by writing it as a dierence equation: xn = 5xn1 6xn2 is the same as writing: xn = 5xn1 6xn2 xn1 = xn1 or in matrix form: 5 6 xn = 1 0 xn1 xn1 xn2

(Note that if the recurrence relation has more terms in it, you just make a bigger matrix: for example, if the recurrence was rn = 2rn1 + 5rn2 6rn3 + 4rn4 , then we just add three more equations so well have A be a 4 4 matrix: rn rn1 rn2 rn3 = 2rn1 + 5rn2 6rn3 + 4rn4 = rn1 = rn2 = rn3

so

2 r 1 n1 = rn2 0 0 rn3

rn

5 6 4 rn1 0 0 0 rn2 . 1 0 0 rn3 0 1 0 rn4

Now back to the example.) Let A = 5 6 x 1 . Our initial condition is x0 = 1 = so then 1 0 x0 0 x1 = x2 = . . . xn = xn+1 = An x0 , xn x2 = Ax0 , x1 x3 = Ax1 = A2 x0 , x2

If we can nd a formula for An , then we can get a formula for the sequence. We compute the eigenvalues of A are 1 = 2 and 2 = 3. The eigenvector for = 2 is v1 = the eigenvector for = 3 is v2 = 3 . So we can diagonalize A as A = SDS 1 , where 1 1 0 2 0 = 0 2 0 3 2 3 1 1 2 ; 1

D=

S = v1 v2 = S 1 =

1 1 3 1 3 = 1 2 2 3 1 2 2

Then we nd that xn+1 = xn = An x0 xn = (SDS 1 )n x0 = SDn S 1 x0 = muliplying everything out gives 2n+1 + 3n+1 xn+1 = xn 2n + 3n So the nth term in the sequence is xn = 3n 2n . Its now easy to check x5 = 35 25 = 211 matches what we found earlier, and now we can compute things like x20 = 320 220 = 3485735825, without having to compute x1 , x2 , . . . , x19 rst. Heres another example you can try for yourself: (this is much easier than the homework problem of deriving the formula for the Fibonacci numbers.) y0 = 4, y1 = 1, yn = yn1 + 2yn2 2 3 1 1 2n 0 0 3n 1 3 1 2 1 0

Population Modeling

Heres a predator-prey system with rabbits and foxes. Let rt be the number of rabbits at time t, and ft the number of foxes at time t. Suppose we start with r0 = 200 rabbits and f0 = 11 foxes. The following equation models the population: rt+1 = art bft ft+1 = crt + dft The parameters a, b, c, and d have the following interpretation: parameter a b c d value 1.14 0.17 0.10 0.82 meaning with no foxes, the rabbit population will increase by this factor each month the decrease in rabbit population due to foxes the increase in fox population due to rabbits with no rabbits, the foxes will die out at this rate each month

We can show the process graphically with a transition graph:

0.17 1.14 rabbits (r) 0.10

We can write this as a dierence equation: rt+1 1.14 0.17 = ft+1 0.10 0.82
xt+1 A

foxes (f )

0.82

rt , ft
xt

200 r0 = 11 f0
x0

The matrix A is known as the transition matrix. The number of rabbits and foxes at any time t is then given by xt = At x0 . To see what happens long-term, we can diagonalize A. Heres how to do it in Octave/Matlab. (This is also a 2 2 matrix, so we can do it by hand using the quadratic formula.) A = [ 1.14 -0.17; 0.10 0.82 ] [S, D] = eig(A) Sinv = inv(S) We get A = SDS 1 , where S= 0.930 0.558 0.368 0.830 D= 1.073 0 0 0.887 S 1 = 1.465 0.986 0.650 1.642

Now, what happens long term? At = (SDS 1 )t = SDt S 1 . As t , the 1.073t in D grows increasingly large (so we cant ignore it), while 0.887t 0. So for big t, we have xt = At x0 = SDt S 1 x0 = 0.930 0.558 0.368 0.830 0.930 0.558 0.368 0.830 1.073t 0 0 0.887t 1.073t 0 0 0 1 0 0 0 1.465 0.986 0.650 1.642 200 11 200 11 200 11

1.465 0.986 0.650 1.642 1.465 0.986 0.650 1.642

= 1.073t = 1.073t

0.930 0.558 0.368 0.830 262.45 103.84

So we can see that (1) both the numbers of foxes and rabbits increase exponentially (increasing at 7.3 percent per month) and (2) the ratio of rabbits to foxes becomes 262.45 : 103.84, or about 2.5 rabbits per fox. 4

Disease Modeling

The SIR model measures the number of susceptible, infected, and recovered individuals in a host population as a disease spreads. Given a xed population, let St be the number of people susceptible to an infectious, but not deadly, disease at time t; let It be the number that is infected at time t; and let Rt be the number that has recovered. (A single epidemic outbreak is usually far more rapid than the vital dynamics of a population, so we can neglect birth-death processes.) Let be the rate of infections. (This rate is relatively constant at the beginning of a disease outbreak.) If there are It infected people, then at the next time step, there will be It susceptible people become infected. this means at the next step there will only be St+1 = St It people susceptible. Let be the rate at which people recover; that is, if there are currently It infected, then at the next step It of them will change from infected to recovered. These transitions are neatly summarized in the following diagram:

1+

S
The transition matrix is

St+1 1 0 St It+1 = 0 1 + 0 It Rt+1 0 1 Rt We cant ask what happens as t , since the infection rate changes once a signicant fraction of the population becomes infected. (We would need a nonlinear model for this.) Nonetheless, we can still make predictions for the early spread of disease. Example: Hong Kong Flu. Suppose there are 8 million people in Hong Kong, and we start with 10 people infected with the u. Its estimated that each infected person infects one person every other day, so = 1/2, also, the estimated time to recover is 3 days, so = 1/3. Homework: what are the estimated number of people infected and recovered after 30 days?

ChemistryChemical Equilibrium

Often, chemical reactions can go both ways, for example, the reaction H2 + I2 2HI is better written as H2 + I 2 2HI Depending on the relative rates of the forward and backward reactions, the system will typically attain some equilibrium where there is some mixture of hydrogen, iodine, and hydrogen iodide. This reaction was studied by Taylor and Krist (J. Am. Chem. Soc. 1941: 1377). Suppose that we have x0 moles of hydrogen iodide (HI) and y0 moles each of the components hydrogen (H2 ) and iodine (I2 ). Each minute, 23.3% of the hydrogen iodide (HI) changes to the separate components (H2 and I2 ), and 38.8% of the components change back to HI. Whats the eventual proportions of HI compared to the components (H2 and I2 )? That is, whats the equilibrium? To help clarify whats going on, there are basically two states: the rst is x, the amount of hydrogen iodide; the other is y, the amount of the other components. If 23.3% of x changes to y, then this means that 76.7% stays as x. Similarly, if only 38.8% of y change to x, then 61.2% stays as y. The transition diagram below shows the situation clearly:

0.388 0.767 x 0.233 HI

Writing the process as a dierence equation, we have xt+1 0.767 0.388 = yt+1 0.233 0.612 xt yt

0.612

H2 and I2

Notice that since the quantities xt and yt are percentages, the columns of the matrix must sum to 1. When this happens, this is known as a Markov Chain. When this happens, = 1 should be an eigenvalue of the matrix, and the eigenvector corresponding to = 1 determines the steady state. To see this, consider what happens when the system is at equilibrium. Going to the next time step will not change the system. Then we would have Ax = x, which means that = 1 is an eigenvalue, and x, the eigenvector, must be the steady state. Knowing this, we can nd the steady state much more easily. Lets try this: A 1I = eigenvector x = 0.233 0.388 1 1.665 , which row reduces to . So = 1 has an 0.233 0.388 0 0

1.665 . Thus at equilibrium there will be 1.665 times as much HI as H2 and I2 . 1 6

To get quantities back in terms of percentages, we can scale the eigenvector so that the sum of the entries is 1 (100%). The sum of entries is 1.665 + 1 = 2.665, so dividing by 2.665 gives an equivalent 1.665 0.625 1 eigenvector x = 2.665 = . So at equilibrium there is 62.5% HI and 37.5% H2 and I2 . 1 0.375

ProbabilityGamblers Ruin

This is a famous problem in probability. Suppose theres a gambler who comes into a casino with $20. He plays a game where the chance of winning is p and the chance of losing is 1 p. Each time he wins or loses $10. He continues gambling until he has $40 (doubling his original amount) or goes broke. We can model this with a Markov chain, using the dierent states x0 , x1 , x2 , x3 , x4 for how much money he has ($0, $10, $20, $30, $40). The transition graph looks like this:

p 1 x0 1p x1 1p x2

p x3 1p

p x4 1

The transition matrix that describes the probabilities of how much money the gambler will have after the next game is:

x0,t+1 1 1p 0 0 x 0 0 1p 0 1,t+1 p 0 1p x2,t+1 = 0 x3,t+1 0 0 p 0 0 0 0 p x4,t+1

0 x0,t 0 x1,t 0 x2,t 0 x3,t 1 x4,t

Lets suppose the chance of winning is p = 1/3; lets nd the probability that the gambler will walk away with $40 (assuming he starts with $20). When p = 1/3, the transition matrix is 1 2/3 0 0 0 0 0 2/3 0 0 A = 0 1/3 0 2/3 0 0 0 1/3 0 0 0 0 0 1/3 1 x0,t 0 x 0 1,t Let xt = x2,t . Since the gambler starts with $20, the initial vector is x0 = 1. From here on, x3,t 0 x4,t 0 7

we have xt+1 = Axt , so that xt = At x0 . Using Octave (or careful work by hand) we nd the following eigenvalues and eigenvectors of A: 1 = 1 2 = 1 3 = 2/3 4 = 2/3 5 = 0 So A diagonalizes as A = SDS 1 where 1 0 S = 0 0 0

v1 = 1 0 0 0 0 v2 = 0 0 0 0 1

T T T T

v3 = 4 2 2 1 1

v4 = 4 10 10 5 1 v5 = 4 6 0 3 1
T

0 4 4 4 0 2 10 6 0 2 10 0 , 0 1 5 3 1 1 1 1

1 0 D = 0 0 0

0 0 0 0 1 0 0 0 0 2/3 0 0 0 0 2/3 0 0 0 0 0

S 1

120 112 96 64 0 0 8 24 56 120 1 15 30 30 0 = 0 120 0 3 6 6 0 0 10 0 20 0

Now we have the limiting probabilities are: x = lim xt

t t t t

= lim At x0 = lim (SDS 1 )t x0 = lim SDt S 1 x0 1 0 = lim 0 t 0 0

0 0 0 4 4 4 1t 0 0 1t 0 0 0 2 10 6 0 0 2 10 0 0 0 (2/3)t 0 (2/3)t 0 1 5 3 0 0 0 0 0 0 1 1 1 1

0 0 120 112 96 64 0 0 0 0 8 24 56 120 1 0 15 30 30 0 1 0 120 0 0 3 6 6 0 0 0 0 0 10 0 20 0

as t , we have (2/3)t 0 and (2/3)t 0, so we get 1 0 1 = 0 120 0 0 4/5 0 = 0 0 1/5 So if the gambler starts with $20, he has an 80% chance of losing all his money and a 20% chance of leaving with $40. 8

0 4 4 4 1 0 2 10 6 0 0 2 10 0 0 0 1 5 3 0 1 1 1 1 0

0 1 0 0 0

0 0 0 0 0

0 0 0 0 0

0 120 112 96 64 0 0 0 0 8 24 56 120 0 0 0 15 30 30 0 1 0 0 3 6 6 0 0 0 0 10 0 20 0 0