Discrete Time Markov Chains, Definition

and classification
Bo Friis Nielsen1
1 Applied

Mathematics and Computer Science

02407 Stochastic Processes 1, September 1 2015

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Discrete time Markov chains

Today:
I
I
I
I

Short recap of probability theory

Markov chain introduction (Markov property)
Chapmann-Kolmogorov equations
First step analysis

Next week
I
I
I
I

Random walks
First step analysis revisited
Branching processes
Generating functions

Two weeks from now

I
I
I

Classification of states
Classification of chains
Discrete time Markov chains - invariant probability
distribution
Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Basic concepts in probability

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Basic concepts in probability

Sample space

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Basic concepts in probability

Sample space

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Basic concepts in probability

Sample space
Outcome

Bo Friis Nielsen

set of all possible outcomes

Discrete Time Markov Chains, Definition and classification

Basic concepts in probability

Sample space
Outcome

Bo Friis Nielsen

set of all possible outcomes

Discrete Time Markov Chains, Definition and classification

Basic concepts in probability

Sample space
Outcome
Event

Bo Friis Nielsen

set of all possible outcomes

Discrete Time Markov Chains, Definition and classification

Basic concepts in probability

Sample space
Outcome
Event

A, B

Bo Friis Nielsen

set of all possible outcomes

Discrete Time Markov Chains, Definition and classification

Basic concepts in probability

Sample space
Outcome
Event
Complementary event

A, B

Bo Friis Nielsen

set of all possible outcomes

Discrete Time Markov Chains, Definition and classification

Basic concepts in probability

Sample space
Outcome
Event
Complementary event

A, B
Ac =

Bo Friis Nielsen

set of all possible outcomes

Discrete Time Markov Chains, Definition and classification

Basic concepts in probability

Sample space
Outcome
Event
Complementary event

A, B
Ac = \A

Bo Friis Nielsen

set of all possible outcomes

Discrete Time Markov Chains, Definition and classification

Basic concepts in probability

Sample space
Outcome
Event
Complementary event

A, B
Ac = \A

set of all possible outcomes

Union

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Basic concepts in probability

Sample space
Outcome
Event
Complementary event
Union

A, B
Ac = \A

set of all possible outcomes

AB

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Basic concepts in probability

Sample space
Outcome
Event
Complementary event
Union

A, B
Ac = \A

set of all possible outcomes

AB

outcome in at least one of

A or B

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Basic concepts in probability

Sample space
Outcome
Event
Complementary event
Union

A, B
Ac = \A

set of all possible outcomes

AB

outcome in at least one of

A or B

Intersection

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Basic concepts in probability

Sample space
Outcome
Event
Complementary event

A, B
Ac = \A

set of all possible outcomes

Union

AB

outcome in at least one of

A or B

Intersection

AB

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Basic concepts in probability

Sample space
Outcome
Event
Complementary event

A, B
Ac = \A

set of all possible outcomes

Union

AB

Intersection

AB

outcome in at least one of

A or B
Outcome is in
both A and B

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Basic concepts in probability

Sample space
Outcome
Event
Complementary event

A, B
Ac = \A

set of all possible outcomes

Union

AB

Intersection

AB

outcome in at least one of

A or B
Outcome is in
both A and B

(Empty) or impossible event

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Basic concepts in probability

Sample space
Outcome
Event
Complementary event

A, B
Ac = \A

set of all possible outcomes

Union

AB

Intersection

AB

outcome in at least one of

A or B
Outcome is in
both A and B

(Empty) or impossible event

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Probability axioms and first results

0 P(A) 1,

Bo Friis Nielsen

P() = 1

Discrete Time Markov Chains, Definition and classification

Probability axioms and first results

0 P(A) 1,

P() = 1

P(A B) = P(A) + P(B)

Bo Friis Nielsen

for A B =

Discrete Time Markov Chains, Definition and classification

Probability axioms and first results

0 P(A) 1,

P() = 1

P(A B) = P(A) + P(B)

for A B =

Leading to

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Probability axioms and first results

0 P(A) 1,

P() = 1

P(A B) = P(A) + P(B)

for A B =

Leading to
P() = 0,

P(Ac ) = 1 P(A)

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Probability axioms and first results

0 P(A) 1,

P() = 1

P(A B) = P(A) + P(B)

for A B =

Leading to
P() = 0,

P(Ac ) = 1 P(A)

P(A B) = P(A) + P(B) P(A B)

(inclusion- exlusion)

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Conditional probability and independence

P(A|B) =

P(A B)
P(B)

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Conditional probability and independence

P(A|B) =

P(A B)
P(A B) = P(A|B)P(B)
P(B)

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Conditional probability and independence

P(A|B) =

P(A B)
P(A B) = P(A|B)P(B)
P(B)

(multiplication rule)
i Bi =

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Conditional probability and independence

P(A|B) =

P(A B)
P(A B) = P(A|B)P(B)
P(B)

(multiplication rule)
i Bi =

Bi Bj =

Bo Friis Nielsen

i 6= j

Discrete Time Markov Chains, Definition and classification

Conditional probability and independence

P(A|B) =

P(A B)
P(A B) = P(A|B)P(B)
P(B)

(multiplication rule)
i Bi =
P(A) =

Bi Bj =

P(A|Bi )P(Bi )

i 6= j

(law of total probability)

Independence:
P(A|B) = P(A|B c ) = P(A)

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Conditional probability and independence

P(A|B) =

P(A B)
P(A B) = P(A|B)P(B)
P(B)

(multiplication rule)
i Bi =
P(A) =

Bi Bj =

P(A|Bi )P(Bi )

i 6= j

(law of total probability)

Independence:
P(A|B) = P(A|B c ) = P(A) P(A B) = P(A)P(B)

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Discrete random variables

Mapping from sample space to metric space
(Read: Real space)
Probability mass function
f (x) = P(X = x) = P ({|X () = x})
Distribution function
F (x) = P(X x) = P ({|X () x}) =

f (t)

tx

Expectation
X
X
X
xP(X = x), E(g(X )) =
g(x)P(X = x) =
g(x)f (x)
E(X ) =
x

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Joint distribution

f (x1 , x2 ) = P(X1 = x1 , X2 = x2 ),
F (x1 , x2 ) = P(X1 x1 , X2 x2 )
X
X
fX1 (x1 ) = P(X1 = x1 ) =
P(X1 = x1 , X2 = x2 ) =
f (x1 , x2 )
x2

FX1 (x1 ) =

x2

P(X1 = t1 , X2 = x2 ) = F (x1 , )

tx1 ,x2

Straightforward to extend to n variables

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

We can define the joint distribution of (X0 , X1 ) through

P(X0 = x0 , X1 = x1 ) = P(X0 = x0 )P(X1 = x1 |X0 = x0 ) = P(X0 = x0 )Px0 ,x1
Suppose now some stationarity in addition that X2 conditioned
on X1 is independent on X0
P(X0 = x0 , X1 = x1 , X2 = x2 ) =
P(X0 = x0 )P(X1 = x1 |X0 = x0 )P(X2 = x2 |X0 = x0 , X1 = x1 ) =
P(X0 = x0 )P(X1 = x1 |X0 = x0 )P(X2 = x2 |X1 = x1 ) =
px0 Px0 ,x1 Px1 ,x2
which generalizes to arbitrary n.

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Markov property
P (Xn = xn |H)

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Markov property
P (Xn = xn |H)
= P (Xn = xn |X0 = x0 , X1 = x1 , X2 = x2 , . . . Xn1 = xn1 )

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Markov property
P (Xn = xn |H)
= P (Xn = xn |X0 = x0 , X1 = x1 , X2 = x2 , . . . Xn1 = xn1 )
= P (Xn = xn |Xn1 = xn1 )

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Markov property
P (Xn = xn |H)
= P (Xn = xn |X0 = x0 , X1 = x1 , X2 = x2 , . . . Xn1 = xn1 )
I

= P (Xn = xn |Xn1 = xn1 )

Generally the next state depends on the current state and
the time

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Markov property
P (Xn = xn |H)
= P (Xn = xn |X0 = x0 , X1 = x1 , X2 = x2 , . . . Xn1 = xn1 )
I

= P (Xn = xn |Xn1 = xn1 )

Generally the next state depends on the current state and
the time
In most applications the chain is assumed to be time
homogeneous, i.e. it does not depend on time

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Markov property
P (Xn = xn |H)
= P (Xn = xn |X0 = x0 , X1 = x1 , X2 = x2 , . . . Xn1 = xn1 )
I

= P (Xn = xn |Xn1 = xn1 )

Generally the next state depends on the current state and
the time

In most applications the chain is assumed to be time

homogeneous, i.e. it does not depend on time

The only parameters needed are

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Markov property
P (Xn = xn |H)
= P (Xn = xn |X0 = x0 , X1 = x1 , X2 = x2 , . . . Xn1 = xn1 )
I

= P (Xn = xn |Xn1 = xn1 )

Generally the next state depends on the current state and
the time

In most applications the chain is assumed to be time

homogeneous, i.e. it does not depend on time

The only parameters needed are P (Xn = j|Xn1 = i)

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Markov property
P (Xn = xn |H)
= P (Xn = xn |X0 = x0 , X1 = x1 , X2 = x2 , . . . Xn1 = xn1 )
I

= P (Xn = xn |Xn1 = xn1 )

Generally the next state depends on the current state and
the time

In most applications the chain is assumed to be time

homogeneous, i.e. it does not depend on time

The only parameters needed are P (Xn = j|Xn1 = i) = pij

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Markov property
P (Xn = xn |H)
= P (Xn = xn |X0 = x0 , X1 = x1 , X2 = x2 , . . . Xn1 = xn1 )
I

= P (Xn = xn |Xn1 = xn1 )

Generally the next state depends on the current state and
the time

In most applications the chain is assumed to be time

homogeneous, i.e. it does not depend on time

The only parameters needed are P (Xn = j|Xn1 = i) = pij

We collect these parameters in a matrix

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Markov property
P (Xn = xn |H)
= P (Xn = xn |X0 = x0 , X1 = x1 , X2 = x2 , . . . Xn1 = xn1 )
I

= P (Xn = xn |Xn1 = xn1 )

Generally the next state depends on the current state and
the time

In most applications the chain is assumed to be time

homogeneous, i.e. it does not depend on time

The only parameters needed are P (Xn = j|Xn1 = i) = pij

We collect these parameters in a matrix P = {pij }

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Markov property
P (Xn = xn |H)
= P (Xn = xn |X0 = x0 , X1 = x1 , X2 = x2 , . . . Xn1 = xn1 )
I

= P (Xn = xn |Xn1 = xn1 )

Generally the next state depends on the current state and
the time

In most applications the chain is assumed to be time

homogeneous, i.e. it does not depend on time

The only parameters needed are P (Xn = j|Xn1 = i) = pij

We collect these parameters in a matrix P = {pij }

The joint probability of the first n occurrences is

P(X0 = x0 , X1 = x1 , X2 = x2 . . . , Xn = xn ) = px0 Px0 ,x1 Px1 ,x2 . . . Pxn1 ,xn

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

A profuse number of applications

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

A profuse number of applications

Storage/inventory models

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

A profuse number of applications

Storage/inventory models

Telecommunications systems

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

A profuse number of applications

Storage/inventory models

Telecommunications systems

Biological models

Xn the value attained at time n

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

A profuse number of applications

Storage/inventory models

Telecommunications systems

Biological models

Xn the value attained at time n

Xn could be

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

A profuse number of applications

Storage/inventory models

Telecommunications systems

Biological models

Xn the value attained at time n

Xn could be

The number of cars in stock

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

A profuse number of applications

Storage/inventory models

Telecommunications systems

Biological models

Xn the value attained at time n

Xn could be

I
I

The number of cars in stock

The number of days since last rainfall

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

A profuse number of applications

Storage/inventory models

Telecommunications systems

Biological models

Xn the value attained at time n

Xn could be

I
I
I

The number of cars in stock

The number of days since last rainfall
The number of passengers booked for a flight

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

A profuse number of applications

Storage/inventory models

Telecommunications systems

Biological models

Xn the value attained at time n

Xn could be

I
I
I
I

The number of cars in stock

The number of days since last rainfall
The number of passengers booked for a flight
See textbook for further examples

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Example 1: Random walk with two

reflecting barriers 0 and N

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Example 1: Random walk with two

reflecting barriers 0 and N

P=

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Example 1: Random walk with two

reflecting barriers 0 and N


 1 p





P = 






Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Example 1: Random walk with two

reflecting barriers 0 and N


 1 p





P = 






Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Example 1: Random walk with two

reflecting barriers 0 and N


 1 p





P = 






Bo Friis Nielsen

...

Discrete Time Markov Chains, Definition and classification

Example 1: Random walk with two

reflecting barriers 0 and N


 1 p

 q



P = 






Bo Friis Nielsen

...

Discrete Time Markov Chains, Definition and classification

Example 1: Random walk with two

reflecting barriers 0 and N


 1 p
p

 q
1

p
q



P = 






Bo Friis Nielsen

...

Discrete Time Markov Chains, Definition and classification

Example 1: Random walk with two

reflecting barriers 0 and N


 1 p
p

 q
1

p
q



P = 






0
p

Bo Friis Nielsen

...

Discrete Time Markov Chains, Definition and classification

Example 1: Random walk with two

reflecting barriers 0 and N


 1 p
p

 q
1

p
q



P = 






0
p

Bo Friis Nielsen

...
...

0
0

0
0

0
0

Discrete Time Markov Chains, Definition and classification

Example 1: Random walk with two

reflecting barriers 0 and N


 1 p
p
0

 q
1

q
p

 0
q
1pq

P =  .
..
..
.
 .
.
.





Bo Friis Nielsen

...
...
...
..
.

0
0
0
..
.

0
0
0
..
.

0
0
0
..
.

Discrete Time Markov Chains, Definition and classification

Example 1: Random walk with two

reflecting barriers 0 and N


 1 p
p
0

 q
1

q
p

 0
q
1pq

P =  .
..
..
.
 .
.
.

 0
0
0



Bo Friis Nielsen

...
...
...
..
.

0
0
0
..
.

0
0
0
..
.

0
0
0
..
.

...

Discrete Time Markov Chains, Definition and classification

Example 1: Random walk with two

reflecting barriers 0 and N


 1 p
p
0

 q
1

q
p

 0
q
1pq

P =  .
..
..
.
 .
.
.

 0
0
0



Bo Friis Nielsen

...
...
...
..
.

0
0
0
..
.

0
0
0
..
.

...

q 1pq

0
0
0
..
.
p

Discrete Time Markov Chains, Definition and classification

Example 1: Random walk with two

reflecting barriers 0 and N


 1 p
p
0

 q
1

q
p

 0
q
1pq

P =  .
..
..
.
 .
.
.

 0
0
0

 0
0
0

Bo Friis Nielsen

...
...
...
..
.

0
0
0
..
.

0
0
0
..
.

...
...

q 1pq
0

0
0
0
..
.
p

Discrete Time Markov Chains, Definition and classification

Example 1: Random walk with two

reflecting barriers 0 and N


 1 p
p
0

 q
1

q
p

 0
q
1pq

P =  .
..
..
.
 .
.
.

 0
0
0

 0
0
0

Bo Friis Nielsen

...
...
...
..
.

0
0
0
..
.

0
0
0
..
.

...
...

q 1pq
0
q

0
0
0
..
.
p

Discrete Time Markov Chains, Definition and classification

Example 1: Random walk with two

reflecting barriers 0 and N


 1 p
p
0

 q
1

q
p

 0
q
1pq

P =  .
..
..
.
 .
.
.

 0
0
0

 0
0
0

Bo Friis Nielsen

...
...
...
..
.
...
...











q 1pq
p 
0
q
1 q 
0
0
0
..
.

0
0
0
..
.

0
0
0
..
.

Discrete Time Markov Chains, Definition and classification

Example 2: Random walk with one

reflecting barrier at 0

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Example 2: Random walk with one

reflecting barrier at 0

P=

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Example 2: Random walk with one

reflecting barrier at 0


 1 p




P = 





Bo Friis Nielsen

0 ...

Discrete Time Markov Chains, Definition and classification

Example 2: Random walk with one

reflecting barrier at 0


 1 p
p

 q
1

p
q


P = 





Bo Friis Nielsen

0
p

0
0

0 ...
0 ...

Discrete Time Markov Chains, Definition and classification

Example 2: Random walk with one

reflecting barrier at 0


 1 p
p
0
0

 q
1

q
p
0

 0
q
1pq
p
P = 
 0
0
q
1

p
q




Bo Friis Nielsen

0
0
0
p

...
...
...
...

Discrete Time Markov Chains, Definition and classification

Example 2: Random walk with one

reflecting barrier at 0


 1 p
p
0
0

 q
1

q
p
0

 0
q
1pq
p
P = 
 0
0
q
1

p
q

 .
.
.
.
..
..
..
 ..

Bo Friis Nielsen

0
0
0
p
..
.

...
...
...
...
...

Discrete Time Markov Chains, Definition and classification

Example 3: Random walk with two

absorbing barriers

P=

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Example 3: Random walk with two

absorbing barriers


 1






P = 







Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Example 3: Random walk with two

absorbing barriers


 1






P = 







Bo Friis Nielsen

0 ...

Discrete Time Markov Chains, Definition and classification

Example 3: Random walk with two

absorbing barriers


 1
0

 q 1 p q




P = 







0
p

Bo Friis Nielsen

0
0

0 ...
0 ...

0
0

0
0

0
0

Discrete Time Markov Chains, Definition and classification

Example 3: Random walk with two

absorbing barriers









P = 







0
0
0
1
q 1pq
p
0
0
q
1pq
p
0
0
q
1pq
..
..
..
..
.
.
.
.

Bo Friis Nielsen

0
0
0
p
..
.

...
...
...
...
..
.

0
0
0
0
..
.

0
0
0
0
..
.

0
0
0
0
..
.

Discrete Time Markov Chains, Definition and classification

Example 3: Random walk with two

absorbing barriers









P = 







0
0
0
1
q 1pq
p
0
0
q
1pq
p
0
0
q
1pq
..
..
..
..
.
.
.
.

0
0
0
p
..
.

0 ...

Bo Friis Nielsen

...
...
...
...
..
.

0
0
0
0
..
.

0
0
0
0
..
.

0
0
0
0
..
.

q 1pq p

Discrete Time Markov Chains, Definition and classification

Example 3: Random walk with two

absorbing barriers









P = 







0
0
0
1
q 1pq
p
0
0
q
1pq
p
0
0
q
1pq
..
..
..
..
.
.
.
.

0
0
0
p
..
.

0
0

0 ...
0 ...

0
0

0
0

Bo Friis Nielsen

0
0

...
...
...
...
..
.













q 1 p q p 
0
0
1 
0
0
0
0
..
.

0
0
0
0
..
.

0
0
0
0
..
.

Discrete Time Markov Chains, Definition and classification

The matrix can be finite

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

The matrix can be finite (if the Markov chain is finite)

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

The matrix can be finite (if the Markov chain is finite)



 p1,1 p1,2 p1,3 . . . p1,n 


 p2,1 p2,2 p2,3 . . . p2,n 




P =  p3,1 p3,2 p3,3 . . . p3,n 
 ..
..
..
..
.. 
 .
.
.
.
. 

 pn,1 pn,2 pn,3 . . . pn,n 

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

The matrix can be finite (if the Markov chain is finite)



 p1,1 p1,2 p1,3 . . . p1,n 


 p2,1 p2,2 p2,3 . . . p2,n 




P =  p3,1 p3,2 p3,3 . . . p3,n 
 ..
..
..
..
.. 
 .
.
.
.
. 

 pn,1 pn,2 pn,3 . . . pn,n 
Two reflecting/absorbing barriers

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

or infinite

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

or infinite (if the Markov chain is infinite)

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

or infinite (if the Markov chain is infinite)


 p1,1 p1,2 p1,3 . . .

 p2,1 p2,2 p2,3 . . .

 p3,1 p3,2 p3,3 . . .

..
..
..
P =  ..
.
.
.
 .
 pn,1 pn,2 pn,3 . . .

 .
..
..
..
 ..
.
.
.

Bo Friis Nielsen

p1,n . . .
p2,n . . .
p3,n . . .
..
..
.
.
pn,n . . .
..
..
.
.

Discrete Time Markov Chains, Definition and classification

or infinite (if the Markov chain is infinite)


 p1,1 p1,2 p1,3 . . .

 p2,1 p2,2 p2,3 . . .

 p3,1 p3,2 p3,3 . . .

..
..
..
P =  ..
.
.
.
 .
 pn,1 pn,2 pn,3 . . .

 .
..
..
..
 ..
.
.
.

p1,n . . .
p2,n . . .
p3,n . . .
..
..
.
.
pn,n . . .
..
..
.
.

At most one barrier

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

The matrix P can be interpreted as

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

The matrix P can be interpreted as

the engine that drives the process

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

The matrix P can be interpreted as

the engine that drives the process

the statistical descriptor of the quantitative behaviour

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

The matrix P can be interpreted as

the engine that drives the process

the statistical descriptor of the quantitative behaviour

a collection of discrete probability distributions

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

The matrix P can be interpreted as

the engine that drives the process

the statistical descriptor of the quantitative behaviour

a collection of discrete probability distributions

For each i we have a conditional distribution

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

The matrix P can be interpreted as

the engine that drives the process

the statistical descriptor of the quantitative behaviour

a collection of discrete probability distributions

I
I

For each i we have a conditional distribution

What is the probability of the next state being j knowing that
the current state is i

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

The matrix P can be interpreted as

the engine that drives the process

the statistical descriptor of the quantitative behaviour

a collection of discrete probability distributions

I
I

For each i we have a conditional distribution

What is the probability of the next state being j knowing that
the current state is i pij = P (Xn = j|Xn1 = i)

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

The matrix P can be interpreted as

the engine that drives the process

the statistical descriptor of the quantitative behaviour

a collection of discrete probability distributions

I
I

For each i we have a conditional distribution

What is the probability of the next state being j knowing that
the
P current state is i pij = P (Xn = j|Xn1 = i)
j pij = 1

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

The matrix P can be interpreted as

the engine that drives the process

the statistical descriptor of the quantitative behaviour

a collection of discrete probability distributions

I
I

I
I

For each i we have a conditional distribution

What is the probability of the next state being j knowing that
the
P current state is i pij = P (Xn = j|Xn1 = i)
j pij = 1
We say that P is a stochastic matrix

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

More definitions and the first properties

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

More definitions and the first properties

We have defined rules for the behaviour from one value

and onwards

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

More definitions and the first properties

We have defined rules for the behaviour from one value

and onwards
Boundary conditions

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

More definitions and the first properties

We have defined rules for the behaviour from one value

and onwards
Boundary conditions specify e.g. behaviour of X0

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

More definitions and the first properties

We have defined rules for the behaviour from one value

and onwards
Boundary conditions specify e.g. behaviour of X0
I

X0 could be certain

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

More definitions and the first properties

We have defined rules for the behaviour from one value

and onwards
Boundary conditions specify e.g. behaviour of X0
I

X0 could be certain X0 = a

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

More definitions and the first properties

We have defined rules for the behaviour from one value

and onwards
Boundary conditions specify e.g. behaviour of X0
I
I

X0 could be certain X0 = a
or random

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

More definitions and the first properties

We have defined rules for the behaviour from one value

and onwards
Boundary conditions specify e.g. behaviour of X0
I
I

X0 could be certain X0 = a
or random P (X0 = i) = pi

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

More definitions and the first properties

We have defined rules for the behaviour from one value

and onwards
Boundary conditions specify e.g. behaviour of X0
I
I
I

X0 could be certain X0 = a
or random P (X0 = i) = pi
Once again we collect the possibly infinite many
parameters in a vector p

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

n step transition probabilities

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

n step transition probabilities

P (Xn = j|X0 = i)

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

n step transition probabilities

(n)

P (Xn = j|X0 = i) = Pij

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

n step transition probabilities

(n)

P (Xn = j|X0 = i) = Pij

I

the probability of being in j at the nth transition having

started in i

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

n step transition probabilities

(n)

P (Xn = j|X0 = i) = Pij

I

the probability of being in j at the nth transition having

started in i

Once again collected in a matrix

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

n step transition probabilities

(n)

P (Xn = j|X0 = i) = Pij

I

the probability of being in j at the nth transition having

started in i

Once again collected in a matrix P (n) = {Pij }

(n)

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

n step transition probabilities

(n)

P (Xn = j|X0 = i) = Pij

I

the probability of being in j at the nth transition having

started in i

Once again collected in a matrix P (n) = {Pij }

The rows of P (n) can be interpreted like the rows of P

(n)

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

n step transition probabilities

(n)

P (Xn = j|X0 = i) = Pij

I

the probability of being in j at the nth transition having

started in i

Once again collected in a matrix P (n) = {Pij }

The rows of P (n) can be interpreted like the rows of P

We can define a new Markov chain on a larger time scale

(n)

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

n step transition probabilities

(n)

P (Xn = j|X0 = i) = Pij

I

the probability of being in j at the nth transition having

started in i

Once again collected in a matrix P (n) = {Pij }

The rows of P (n) can be interpreted like the rows of P

We can define a new Markov chain on a larger time scale

(P n )

(n)

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Small example


 1 p

 q
P = 
 0
 0

p
0
q
0

0
0
p
0
0
p
q 1q

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Small example


 1 p

 q
P = 
 0
 0

p
0
q
0

0
0
p
0
0
p
q 1q

P (2) =

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Small example


 1 p

 q
P = 
 0
 0

P (2)

p
0
q
0

0
0
p
0
0
p
q 1q


 (1 p)2


= 



Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Small example


 1 p

 q
P = 
 0
 0

P (2)

p
0
q
0

0
0
p
0
0
p
q 1q


 (1 p)2 + pq


= 



Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Small example


 1 p

 q
P = 
 0
 0

P (2)

p
0
q
0

0
0
p
0
0
p
q 1q


 (1 p)2 + pq (1 p)p


= 



Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Small example


 1 p

 q
P = 
 0
 0

P (2)

p
0
q
0

0
0
p
0
0
p
q 1q


 (1 p)2 + pq (1 p)p


= 



Bo Friis Nielsen









p2

Discrete Time Markov Chains, Definition and classification

Small example


 1 p

 q
P = 
 0
 0

(2)

p
0
q
0

0
0
p
0
0
p
q 1q


 (1 p)2 + pq (1 p)p


= 



Bo Friis Nielsen









p2

Discrete Time Markov Chains, Definition and classification

Small example


 1 p

 q
P = 
 0
 0

(2)

p
0
q
0

0
0
p
0
0
p
q 1q


 (1 p)2 + pq (1 p)p


q(1 p)
= 



Bo Friis Nielsen









p2

Discrete Time Markov Chains, Definition and classification

Small example


 1 p

 q
P = 
 0
 0

(2)

p
0
q
0

0
0
p
0
0
p
q 1q


 (1 p)2 + pq (1 p)p


q(1 p)
2qp
= 



Bo Friis Nielsen









p2

Discrete Time Markov Chains, Definition and classification

Small example


 1 p

 q
P = 
 0
 0

(2)

p
0
q
0

0
0
p
0
0
p
q 1q


 (1 p)2 + pq (1 p)p


q(1 p)
2qp
= 



Bo Friis Nielsen









p2
0

Discrete Time Markov Chains, Definition and classification

Small example


 1 p

 q
P = 
 0
 0

(2)

p
0
q
0

0
0
p
0
0
p
q 1q


 (1 p)2 + pq (1 p)p


q(1 p)
2qp
= 



Bo Friis Nielsen









p2
0

0
p2

Discrete Time Markov Chains, Definition and classification

Small example


 1 p

 q
P = 
 0
 0

(2)

p
0
q
0

0
0
p
0
0
p
q 1q


 (1 p)2 + pq (1 p)p


q(1 p)
2qp
= 
2
q
0



Bo Friis Nielsen









p2
0
2qp

0
p2
p(1 q)

Discrete Time Markov Chains, Definition and classification

Small example


 1 p

 q
P = 
 0
 0

P (2)

p
0
q
0

0
0
p
0
0
p
q 1q


 (1 p)2 + pq (1 p)p
p2
0

2

q(1

p)
2qp
0
p
= 
2
q
0
2qp
p(1 q)

2

0
q
(1 q)q (1 q)2 + qp

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Chapmann Kolmogorov equations

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Chapmann Kolmogorov equations

There is a generalisation of the example above

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Chapmann Kolmogorov equations

There is a generalisation of the example above

Suppose we start in i at time 0 and wants to get to j at time

n+m

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Chapmann Kolmogorov equations

There is a generalisation of the example above

Suppose we start in i at time 0 and wants to get to j at time

n+m

At some intermediate time n we must be in some state k

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Chapmann Kolmogorov equations

There is a generalisation of the example above

Suppose we start in i at time 0 and wants to get to j at time

n+m

At some intermediate time n we must be in some state k

We apply the law of total probability

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Chapmann Kolmogorov equations

There is a generalisation of the example above

Suppose we start in i at time 0 and wants to get to j at time

n+m

At some intermediate time n we must be in some state k

We apply the law of total probability

P (B) =

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Chapmann Kolmogorov equations

There is a generalisation of the example above

Suppose we start in i at time 0 and wants to get to j at time

n+m

At some intermediate time n we must be in some state k

We applyP
the law of total probability
P (B) = k P (B|Ak )

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Chapmann Kolmogorov equations

There is a generalisation of the example above

Suppose we start in i at time 0 and wants to get to j at time

n+m

At some intermediate time n we must be in some state k

We applyP
the law of total probability
P (B) = k P (B|Ak ) P (Ak )
P (Xn+m = j|X0 = i)

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Chapmann Kolmogorov equations

There is a generalisation of the example above

Suppose we start in i at time 0 and wants to get to j at time

n+m

At some intermediate time n we must be in some state k

We applyP
the law of total probability
P (B) = k P (B|Ak ) P (Ak )
P (Xn+m = j|X0 = i)
=

P (Xn+m = j|X0 = i, Xn = k )

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Chapmann Kolmogorov equations

There is a generalisation of the example above

Suppose we start in i at time 0 and wants to get to j at time

n+m

At some intermediate time n we must be in some state k

We applyP
the law of total probability
P (B) = k P (B|Ak ) P (Ak )
P (Xn+m = j|X0 = i)
=

P (Xn+m = j|X0 = i, Xn = k ) P (Xn = k |X0 = i)

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

P (Xn+m = j|X0 = i, Xn = k )

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

P (Xn+m = j|X0 = i, Xn = k ) P (Xn = k |X0 = i)

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

P (Xn+m = j|X0 = i, Xn = k ) P (Xn = k |X0 = i)

by the Markov property we get

X
P (Xn+m = j|Xn = k )
k

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

P (Xn+m = j|X0 = i, Xn = k ) P (Xn = k |X0 = i)

by the Markov property we get

X
P (Xn+m = j|Xn = k ) P (Xn = k |X0 = i)
k

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

P (Xn+m = j|X0 = i, Xn = k ) P (Xn = k |X0 = i)

by the Markov property we get

X
P (Xn+m = j|Xn = k ) P (Xn = k |X0 = i)
k

(m)

(n)

Pkj Pik =

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

P (Xn+m = j|X0 = i, Xn = k ) P (Xn = k |X0 = i)

by the Markov property we get

X
P (Xn+m = j|Xn = k ) P (Xn = k |X0 = i)
k

(m)

(n)

Pkj Pik =

Bo Friis Nielsen

(n)

(m)

Pik Pkj

Discrete Time Markov Chains, Definition and classification

P (Xn+m = j|X0 = i, Xn = k ) P (Xn = k |X0 = i)

by the Markov property we get

X
P (Xn+m = j|Xn = k ) P (Xn = k |X0 = i)
k

(m)

(n)

Pkj Pik =

(n)

(m)

Pik Pkj

which in matrix formulation is

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

P (Xn+m = j|X0 = i, Xn = k ) P (Xn = k |X0 = i)

by the Markov property we get

X
P (Xn+m = j|Xn = k ) P (Xn = k |X0 = i)
k

(m)

(n)

Pkj Pik =

(n)

(m)

Pik Pkj

which in matrix formulation is

P (n+m) = P (n) P (m)

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

P (Xn+m = j|X0 = i, Xn = k ) P (Xn = k |X0 = i)

by the Markov property we get

X
P (Xn+m = j|Xn = k ) P (Xn = k |X0 = i)
k

(m)

(n)

Pkj Pik =

(n)

(m)

Pik Pkj

which in matrix formulation is

P (n+m) = P (n) P (m) = P n+m

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

The probability of Xn

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

The probability of Xn

The behaviour of the process itself - Xn

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

The probability of Xn

The behaviour of the process itself - Xn

The behaviour conditional on X0 = i is known (Pij )

(n)

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

The probability of Xn

The behaviour of the process itself - Xn

The behaviour conditional on X0 = i is known (Pij )

Define P (Xn = j)

(n)

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

The probability of Xn

The behaviour of the process itself - Xn

The behaviour conditional on X0 = i is known (Pij )

Define P (Xn = j) = pj

(n)

(n)

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

The probability of Xn

The behaviour of the process itself - Xn

The behaviour conditional on X0 = i is known (Pij )

Define P (Xn = j) = pj

with p (n) = {pj }

(n)

(n)

(n)

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

The probability of Xn

The behaviour of the process itself - Xn

The behaviour conditional on X0 = i is known (Pij )

Define P (Xn = j) = pj

with p (n) = {pj } we find

(n)

(n)

(n)

p (n)

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

The probability of Xn

The behaviour of the process itself - Xn

The behaviour conditional on X0 = i is known (Pij )

Define P (Xn = j) = pj

with p (n) = {pj } we find

(n)

(n)

(n)

p (n) = p

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

The probability of Xn

The behaviour of the process itself - Xn

The behaviour conditional on X0 = i is known (Pij )

Define P (Xn = j) = pj

with p (n) = {pj } we find

(n)

(n)

(n)

p (n) = pP (n) = pP n

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Small example - revisited


 1 p

 q
P = 
 0
 0

p
0
q
0

Bo Friis Nielsen

0
0
p
0
0
p
q 1q

Discrete Time Markov Chains, Definition and classification

Small example - revisited


 1 p

 q
P = 
 0
 0


with p = 13 , 0, 0, 23

p
0
q
0

Bo Friis Nielsen

0
0
p
0
0
p
q 1q

Discrete Time Markov Chains, Definition and classification

Small example - revisited


 1 p p

 q
0
P = 
q
 0
 0
0


with p = 13 , 0, 0, 23 we get

0
0
p
0
0
p
q 1q

p (1)

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Small example - revisited


 1 p p

 q
0
P = 
q
 0
 0
0


with p = 13 , 0, 0, 23 we get

(1)


=

1
2
, 0, 0,
3
3

0
0
p
0
0
p
q 1q

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Small example - revisited


 1 p p

 q
0
P = 
q
 0
 0
0


with p = 13 , 0, 0, 23 we get

p (1) =



  1 p
1
2  q
, 0, 0,
3
3  0
 0

p
0
q
0

0
0
p
0
0
p
q 1q

0
0
p
0
0
p
q 1q

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Small example - revisited


 1 p p

 q
0
P = 
q
 0
 0
0


with p = 13 , 0, 0, 23 we get

p (1) =



  1 p
1
2  q
, 0, 0,
3
3  0
 0

p
0
q
0

0
0
p
0
0
p
q 1q

0
0
p
0
0
p
q 1q

Bo Friis Nielsen






 = 1 p ,

3



Discrete Time Markov Chains, Definition and classification

Small example - revisited


 1 p p

 q
0
P = 
q
 0
 0
0


with p = 13 , 0, 0, 23 we get

p (1) =



  1 p
1
2  q
, 0, 0,
3
3  0
 0

p
0
q
0

0
0
p
0
0
p
q 1q

0
0
p
0
0
p
q 1q

Bo Friis Nielsen






 = 1 p , p ,

3
3



Discrete Time Markov Chains, Definition and classification

Small example - revisited


 1 p p

 q
0
P = 
q
 0
 0
0


with p = 13 , 0, 0, 23 we get

p (1) =



  1 p
1
2  q
, 0, 0,
3
3  0
 0

p
0
q
0

0
0
p
0
0
p
q 1q

0
0
p
0
0
p
q 1q

Bo Friis Nielsen






 = 1 p , p , 2q ,

3
3 3



Discrete Time Markov Chains, Definition and classification

Small example - revisited


 1 p p

 q
0
P = 
q
 0
 0
0


with p = 13 , 0, 0, 23 we get

p (1) =



  1 p
1
2  q
, 0, 0,
3
3  0
 0

p
0
q
0

0
0
p
0
0
p
q 1q

0
0
p
0
0
p
q 1q

Bo Friis Nielsen







 = 1 p , p , 2q , 2(1 q)

3
3 3
3



Discrete Time Markov Chains, Definition and classification


p=

2
1
, 0, 0,
3
3


,


 (1 p)2 + pq (1 p)p
p2
0

2


q(1

p)
2qp
0
p
P 2 = 
q2
0
2qp
p(1 q)


0
q2
(1 q)q (1 q)2 + qp

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

p (2)

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

(2)


=

Bo Friis Nielsen

1
2
, 0, 0,
3
3

Discrete Time Markov Chains, Definition and classification

(2)


=

1
2
, 0, 0,
3
3


 (1 p)2 + pq (1 p)p
p2
0


q(1 p)
2qp
0
p2

2

q
0
2qp
p(1 q)

2

0
q
(1 q)q (1 q)2 + qp

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

(2)


=

1
2
, 0, 0,
3
3


 (1 p)2 + pq (1 p)p
p2
0


q(1 p)
2qp
0
p2

2

q
0
2qp
p(1 q)

2

0
q
(1 q)q (1 q)2 + qp


=

(1 p)2 + pq
,
3

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

(2)


=

1
2
, 0, 0,
3
3


 (1 p)2 + pq (1 p)p
p2
0


q(1 p)
2qp
0
p2

2

q
0
2qp
p(1 q)

2

0
q
(1 q)q (1 q)2 + qp


=

(1 p)2 + pq (1 p)p
,
,
3
3

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

(2)


=

1
2
, 0, 0,
3
3


 (1 p)2 + pq (1 p)p
p2
0


q(1 p)
2qp
0
p2

2

q
0
2qp
p(1 q)

2

0
q
(1 q)q (1 q)2 + qp


=

(1 p)2 + pq (1 p)p 4qp

,
,
,
3
3
3

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

(2)


=

1
2
, 0, 0,
3
3


 (1 p)2 + pq (1 p)p
p2
0


q(1 p)
2qp
0
p2

2

q
0
2qp
p(1 q)

2

0
q
(1 q)q (1 q)2 + qp


=

(1 p)2 + pq (1 p)p 4qp 2p(1 q)

,
,
,
3
3
3
3

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

First step analysis - setup

Consider the transition probability matrix



 1 0 0 


P =  
 0 0 1 
Define
T = min {n 0 : Xn = 0 or Xn = 2}
and
u = P(XT = 0|X0 = 1)

Bo Friis Nielsen

v = E(T |X0 = 1)

Discrete Time Markov Chains, Definition and classification

First step analysis - absorption probability

u = P(XT = 0|X0 = 1)
=

2
X

P(X1 = k |X0 = 1)P(XT = 0|X0 = 1, X1 = k )

k =0

2
X

P(X1 = k |X0 = 1)P(XT = 0|X1 = k )

k =0

= P1,0 1 + P1,1 u + P1,2 0.

And we find
u=

P1,0

=
1 P1,1
+

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

First step analysis - time to absorption

= E(T |X0 = 1)
=

2
X

P(X1 = k |X0 = 1)E(T |X0 = 1, X1 = k )

k =0

= 1+

2
X

P(X1 = k |X0 = 1)E(T |X0 = k )( NB!

k =0

= 1 + P1,0 0 + P1,1 v + P1,2 0.

And we find
v=

1
1
=
1 P1,1
1

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

More than one transient state


 1
0
0
0

 P1,0 P1,1 P1,2 P1,3
P = 
 P2,0 P2,1 P2,2 P2,3
 0
0
0
1

Here we will need conditional probabilities

ui = P(XT = 0|X0 = i)

and conditional expectations vi = E(T |X0 = i)

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Leading to

u1 = P1,0 + P1,1 u1 + P1,2 u2

u2 = P2,0 + P2,1 u1 + P2,2 u2
and
v1 = 1 + P1,1 v1 + P1,2 v2
v2 = 1 + P2,1 v1 + P2,2 v2

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

General finite state Markov chain



 Q R 



P = 
0 I 

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

General absorbing Markov chain

T = min {n 0, Xn r }
In state j we accumulate reward g(j), wi is expected total
reward conditioned on start in state i
!
T
1
X
wi = E
g(Xn )|X0 = i
n=0

leading to
wi = g(i) +

Pi,j wj

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Special cases of general absorbing Markov chain

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Special cases of general absorbing Markov chain

g(i) = 1 expected time to absorption (vi )

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Special cases of general absorbing Markov chain

g(i) = 1 expected time to absorption (vi )

g(i) = ik expected visits to state k before absorption

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification