2 views

Original Title: Lecture 32

Uploaded by Ed Z

- Eigenvalues and Eigenvectors
- dm07_013Meila3
- Math 275 Spring 2012 3-7
- Global CMT Search Results.2pdf.pdf
- MATH2019 Engineering Mathematics 2E. (3)
- Heat Problems
- On the Maxwell Stefan Approach to Multi
- kb-Ritzvs.Eigenvectors-080815-0828-26052
- Paper2005.pdf
- gennPrac11
- hw6sol
- 4310_HW7
- UOL Statistic 2 Exam Commentary Zone A
- Secure Relay and Jammer Selection for Physical Layer Security
- Neuralnt Code
- Talking head - OA interview questions.docx
- Time-Varying Informed and Uninformed Trading Activities Qin_Lei_JFM_2005
- nipsdlufl10-RandomWeights
- Ch3
- TRATAMIENTO_DE_AGUAS_RESIDUALES_DE_USO_AGRICOLA_EN.pdf

You are on page 1of 53

600: Lecture 32

Markov Chains

Scott Sheffield

MIT

Outline

Markov chains

Examples

Outline

Markov chains

Examples

Markov chains

taking values in the same state space, which for now we take

to be a finite set that we label by {0, 1, . . . , M}.

Markov chains

taking values in the same state space, which for now we take

to be a finite set that we label by {0, 1, . . . , M}.

I Interpret Xn as state of the system at time n.

Markov chains

taking values in the same state space, which for now we take

to be a finite set that we label by {0, 1, . . . , M}.

I Interpret Xn as state of the system at time n.

I Sequence is called a Markov chain if we have a fixed

collection of numbers Pij (one for each pair

i, j {0, 1, . . . , M}) such that whenever the system is in state

i, there is probability Pij that system will next be in state j.

Markov chains

taking values in the same state space, which for now we take

to be a finite set that we label by {0, 1, . . . , M}.

I Interpret Xn as state of the system at time n.

I Sequence is called a Markov chain if we have a fixed

collection of numbers Pij (one for each pair

i, j {0, 1, . . . , M}) such that whenever the system is in state

i, there is probability Pij that system will next be in state j.

I Precisely,

P{Xn+1 = j|Xn = i, Xn1 = in1 , . . . , X1 = i1 , X0 = i0 } = Pij .

Markov chains

taking values in the same state space, which for now we take

to be a finite set that we label by {0, 1, . . . , M}.

I Interpret Xn as state of the system at time n.

I Sequence is called a Markov chain if we have a fixed

collection of numbers Pij (one for each pair

i, j {0, 1, . . . , M}) such that whenever the system is in state

i, there is probability Pij that system will next be in state j.

I Precisely,

P{Xn+1 = j|Xn = i, Xn1 = in1 , . . . , X1 = i1 , X0 = i0 } = Pij .

I Kind of an almost memoryless property. Probability

distribution for next state depends only on the current state

(and not on the rest of the state history).

Simple example

rainy and sunny.

Simple example

rainy and sunny.

I If its rainy one day, theres a .5 chance it will be rainy the

next day, a .5 chance it will be sunny.

Simple example

rainy and sunny.

I If its rainy one day, theres a .5 chance it will be rainy the

next day, a .5 chance it will be sunny.

I If its sunny one day, theres a .8 chance it will be sunny the

next day, a .2 chance it will be rainy.

Simple example

rainy and sunny.

I If its rainy one day, theres a .5 chance it will be rainy the

next day, a .5 chance it will be sunny.

I If its sunny one day, theres a .8 chance it will be sunny the

next day, a .2 chance it will be rainy.

I In this climate, sun tends to last longer than rain.

Simple example

rainy and sunny.

I If its rainy one day, theres a .5 chance it will be rainy the

next day, a .5 chance it will be sunny.

I If its sunny one day, theres a .8 chance it will be sunny the

next day, a .2 chance it will be rainy.

I In this climate, sun tends to last longer than rain.

I Given that it is rainy today, how many days to I expect to

have to wait to see a sunny day?

Simple example

rainy and sunny.

I If its rainy one day, theres a .5 chance it will be rainy the

next day, a .5 chance it will be sunny.

I If its sunny one day, theres a .8 chance it will be sunny the

next day, a .2 chance it will be rainy.

I In this climate, sun tends to last longer than rain.

I Given that it is rainy today, how many days to I expect to

have to wait to see a sunny day?

I Given that it is sunny today, how many days to I expect to

have to wait to see a rainy day?

Simple example

rainy and sunny.

I If its rainy one day, theres a .5 chance it will be rainy the

next day, a .5 chance it will be sunny.

I If its sunny one day, theres a .8 chance it will be sunny the

next day, a .2 chance it will be rainy.

I In this climate, sun tends to last longer than rain.

I Given that it is rainy today, how many days to I expect to

have to wait to see a sunny day?

I Given that it is sunny today, how many days to I expect to

have to wait to see a rainy day?

I Over the long haul, what fraction of days are sunny?

Matrix representation

i, j {0, 1, . . . , M}.

Matrix representation

i, j {0, 1, . . . , M}.

I It is convenient to represent the collection of transition

probabilities Pij as a matrix:

P00 P01 . . . P0M

P10 P11 . . . P1M

A=

PM0 PM1 . . . PMM

Matrix representation

i, j {0, 1, . . . , M}.

I It is convenient to represent the collection of transition

probabilities Pij as a matrix:

P00 P01 . . . P0M

P10 P11 . . . P1M

A=

PM0 PM1 . . . PMM

PM

j=0 Pij = 1 for each i. That is, the rows sum to one.

Transitions via matrices

I Suppose that pi is the probability that system is in state i at

time zero.

Transitions via matrices

I Suppose that pi is the probability that system is in state i at

time zero.

I What does the following product represent?

P00 P01 . . . P0M

P10 P11 . . . P1M

p0 p1 . . . pM

PM0 PM1 . . . PMM

Transitions via matrices

I Suppose that pi is the probability that system is in state i at

time zero.

I What does the following product represent?

P00 P01 . . . P0M

P10 P11 . . . P1M

p0 p1 . . . pM

PM0 PM1 . . . PMM

Transitions via matrices

I Suppose that pi is the probability that system is in state i at

time zero.

I What does the following product represent?

P00 P01 . . . P0M

P10 P11 . . . P1M

p0 p1 . . . pM

PM0 PM1 . . . PMM

I How about the following product?

An

p0 p1 . . . pM

Transitions via matrices

I Suppose that pi is the probability that system is in state i at

time zero.

I What does the following product represent?

P00 P01 . . . P0M

P10 P11 . . . P1M

p0 p1 . . . pM

PM0 PM1 . . . PMM

I How about the following product?

An

p0 p1 . . . pM

Powers of transition matrix

(n)

I We write Pij for the probability to go from state i to state j

over n steps.

Powers of transition matrix

(n)

I We write Pij for the probability to go from state i to state j

over n steps.

I From the matrix point of view

(n) (n) (n)

n

P00 P01 . . . P0M P00 P01 . . . P0M

(n) (n) (n)

P10 P11 . . . P1M 10 P11 . . . P1M

P

=

(n) (n) (n) PM0 PM1 . . . PMM

PM0 P M1 ... P MM

Powers of transition matrix

(n)

I We write Pij for the probability to go from state i to state j

over n steps.

I From the matrix point of view

(n) (n) (n)

n

P00 P01 . . . P0M P00 P01 . . . P0M

(n) (n) (n)

P10 P11 . . . P1M 10 P11 . . . P1M

P

=

(n) (n) (n) PM0 PM1 . . . PMM

PM0 P M1 ... P MM

transition matrix.

Questions

Questions

I Answer: state sequence Xi consists of i.i.d. random variables.

Questions

I Answer: state sequence Xi consists of i.i.d. random variables.

I What if matrix is the identity?

Questions

I Answer: state sequence Xi consists of i.i.d. random variables.

I What if matrix is the identity?

I Answer: states never change.

Questions

I Answer: state sequence Xi consists of i.i.d. random variables.

I What if matrix is the identity?

I Answer: states never change.

I What if each Pij is either one or zero?

Questions

I Answer: state sequence Xi consists of i.i.d. random variables.

I What if matrix is the identity?

I Answer: states never change.

I What if each Pij is either one or zero?

I Answer: state evolution is deterministic.

Outline

Markov chains

Examples

Outline

Markov chains

Examples

Simple example

theres a .5 chance it will be rainy the next day, a .5 chance it

will be sunny. If its sunny one day, theres a .8 chance it will

be sunny the next day, a .2 chance it will be rainy.

Simple example

theres a .5 chance it will be rainy the next day, a .5 chance it

will be sunny. If its sunny one day, theres a .8 chance it will

be sunny the next day, a .2 chance it will be rainy.

I Let rainy be state zero, sunny state one, and write the

transition matrix by

.5 .5

A=

.2 .8

Simple example

theres a .5 chance it will be rainy the next day, a .5 chance it

will be sunny. If its sunny one day, theres a .8 chance it will

be sunny the next day, a .2 chance it will be rainy.

I Let rainy be state zero, sunny state one, and write the

transition matrix by

.5 .5

A=

.2 .8

I Note that

2 .64 .35

A =

.26 .74

Simple example

theres a .5 chance it will be rainy the next day, a .5 chance it

will be sunny. If its sunny one day, theres a .8 chance it will

be sunny the next day, a .2 chance it will be rainy.

I Let rainy be state zero, sunny state one, and write the

transition matrix by

.5 .5

A=

.2 .8

I Note that

2 .64 .35

A =

.26 .74

.285719 .714281

I Can compute A10 =

.285713 .714287

Does relationship status have the Markov property?

In a relationship

Married Engaged

Does relationship status have the Markov property?

In a relationship

Married Engaged

Does relationship status have the Markov property?

In a relationship

Married Engaged

I Markov model implies time spent in any state (e.g., a

marriage) before leaving is a geometric random variable.

Does relationship status have the Markov property?

In a relationship

Married Engaged

I Markov model implies time spent in any state (e.g., a

marriage) before leaving is a geometric random variable.

I Not true... Can we make a better model with more states?

Outline

Markov chains

Examples

Outline

Markov chains

Examples

Ergodic Markov chains

I Say Markov chain is ergodic if some power of the transition

matrix has all non-zero entries.

Ergodic Markov chains

I Say Markov chain is ergodic if some power of the transition

matrix has all non-zero entries.

I Turns out that if chain has this property, then

(n)

j := limn Pij exists and the j are the unique

non-negative solutions of j = M

P

k=0 k Pkj that sum to one.

Ergodic Markov chains

I Say Markov chain is ergodic if some power of the transition

matrix has all non-zero entries.

I Turns out that if chain has this property, then

(n)

j := limn Pij exists and the j are the unique

non-negative solutions of j = M

P

k=0 k Pkj that sum to one.

I This means that the row vector

= 0 1 . . . M

Ergodic Markov chains

I Say Markov chain is ergodic if some power of the transition

matrix has all non-zero entries.

I Turns out that if chain has this property, then

(n)

j := limn Pij exists and the j are the unique

non-negative solutions of j = M

P

k=0 k Pkj that sum to one.

I This means that the row vector

= 0 1 . . . M

I We call the stationary distribution of the Markov chain.

Ergodic Markov chains

I Say Markov chain is ergodic if some power of the transition

matrix has all non-zero entries.

I Turns out that if chain has this property, then

(n)

j := limn Pij exists and the j are the unique

non-negative solutions of j = M

P

k=0 k Pkj that sum to one.

I This means that the row vector

= 0 1 . . . M

I We call the stationary distribution of the Markov chain.

I One can

P solve the system of linear equations

j = M k=0 k Pkj to compute the values j . Equivalent to

considering A fixed and solving A = . Or solving

(A I ) = 0. This determines

P up to a multiplicative

constant, and fact that j = 1 determines the constant.

Simple example

.5 .5

I If A = , then we know

.2 .8

.5 .5

A = 0 1 = 0 1 = .

.2 .8

Simple example

.5 .5

I If A = , then we know

.2 .8

.5 .5

A = 0 1 = 0 1 = .

.2 .8

we also know that 0 + 1 = 1. Solving these equations gives

0 = 2/7 and 1 = 5/7, so = 2/7 5/7 .

Simple example

.5 .5

I If A = , then we know

.2 .8

.5 .5

A = 0 1 = 0 1 = .

.2 .8

we also know that 0 + 1 = 1. Solving these equations gives

0 = 2/7 and 1 = 5/7, so = 2/7 5/7 .

I Indeed,

.5 .5

A = 2/7 5/7 = 2/7 5/7 = .

.2 .8

Simple example

.5 .5

I If A = , then we know

.2 .8

.5 .5

A = 0 1 = 0 1 = .

.2 .8

we also know that 0 + 1 = 1. Solving these equations gives

0 = 2/7 and 1 = 5/7, so = 2/7 5/7 .

I Indeed,

.5 .5

A = 2/7 5/7 = 2/7 5/7 = .

.2 .8

I Recall

that

10 .285719 .714281 2/7 5/7

A = =

.285713 .714287 2/7 5/7

- Eigenvalues and EigenvectorsUploaded byRicardo Colosimo
- dm07_013Meila3Uploaded bynagator
- Math 275 Spring 2012 3-7Uploaded byMutt3012
- Global CMT Search Results.2pdf.pdfUploaded byMetar Yoseplin Hutauruk
- MATH2019 Engineering Mathematics 2E. (3)Uploaded byMaria DeeTee Nguyen
- Heat ProblemsUploaded byIrv Tololoche
- On the Maxwell Stefan Approach to MultiUploaded byshubhankar_ps
- kb-Ritzvs.Eigenvectors-080815-0828-26052Uploaded byjwanro
- Paper2005.pdfUploaded byJoshua Gates
- gennPrac11Uploaded bykbaah0
- hw6solUploaded bySandeep Badiganti
- 4310_HW7Uploaded byGag Paf
- UOL Statistic 2 Exam Commentary Zone AUploaded byBoon Kiat Teh
- Secure Relay and Jammer Selection for Physical Layer SecurityUploaded byhendra lam
- Neuralnt CodeUploaded byKapil Mishra
- Talking head - OA interview questions.docxUploaded bysaem mdued
- Time-Varying Informed and Uninformed Trading Activities Qin_Lei_JFM_2005Uploaded byalexa_sherpy
- nipsdlufl10-RandomWeightsUploaded byAbbé Busoni
- Ch3Uploaded bySteven Robert Tseng
- TRATAMIENTO_DE_AGUAS_RESIDUALES_DE_USO_AGRICOLA_EN.pdfUploaded byJuan
- SymmetryUploaded bysevkax
- [P] [Ko] Method for 3D measuring CL.pdfUploaded byDũng Lê
- How to Prepare for Financial Engineering Programs - Part 2Uploaded bycharles luis
- Chap14Uploaded byVIMAL
- mat3607_2012_assign2Uploaded byGhettoopressed
- TIPS FOR GATE PREPARATIONUploaded bysandeep
- 2 D.O.F Free VibrationUploaded bySalvador Silveyra
- Analyzing biosignals using the R freewareUploaded byergoshaun
- Driving Style Recognition Based.pdfUploaded bySiva Ranjani
- Curri2004 MechanicalUploaded byNaveen Baskar

- File 0091.pdfUploaded byEd Z
- File 0099.pdfUploaded byEd Z
- File 0092.pdfUploaded byEd Z
- File 0085.pdfUploaded byEd Z
- File 0078.pdfUploaded byEd Z
- File 0097.pdfUploaded byEd Z
- File 0094.pdfUploaded byEd Z
- File 0083.pdfUploaded byEd Z
- File 0077.pdfUploaded byEd Z
- File 0096.pdfUploaded byEd Z
- File 0089.pdfUploaded byEd Z
- File 0100.pdfUploaded byEd Z
- File 0095.pdfUploaded byEd Z
- File 0090.pdfUploaded byEd Z
- File 0098.pdfUploaded byEd Z
- File 0087.pdfUploaded byEd Z
- File 0068.pdfUploaded byEd Z
- File 0088.pdfUploaded byEd Z
- File 0063.pdfUploaded byEd Z
- File 0061.pdfUploaded byEd Z
- File 0081.pdfUploaded byEd Z
- File 0075.pdfUploaded byEd Z
- File 0080.pdfUploaded byEd Z
- File 0082.pdfUploaded byEd Z
- File 0079.pdfUploaded byEd Z
- File 0069.pdfUploaded byEd Z
- File 0086.pdfUploaded byEd Z
- File 0093.pdfUploaded byEd Z
- File 0084.pdfUploaded byEd Z
- File 0071.pdfUploaded byEd Z

- Physics of Music - NotesUploaded byadrisor1409
- Fundation of Asset Pricing. Ronal Balver ContentUploaded byArShida
- Results GraphUploaded byHomero Castilla
- Mobile and Wireless Communication Complete Lecture Notes #3Uploaded byStudent Lecture Notes
- CHAP09 Finitely Generated Abelian GroupsUploaded byDaniél Cárdenas
- A Study on Column Discontinuity on Behavior of StructureUploaded byInternational Journal for Scientific Research and Development - IJSRD
- Torrence Student's Introduction to MATHEMATICA 2nd TxtbkUploaded byKabalks
- Numerical S-Parameter Extraction and Characterization of Inhomogeneously Filled WaveguidesUploaded byPedro Barba Medina
- D AlgorithmUploaded byKiran Kumar
- Vehicle License Plate Recognition SystemUploaded bydips032
- 125913139-Winches-and-Reels-Tension-in-Line-and-Pressure-on-the-Drum-and-Flanges.pdfUploaded byjosebernal_mza
- 1st Sem Mid QuestionsUploaded byAkhil Aaron
- Centrifugal Pump Alfred FranklinUploaded byAlfred Franklin V
- Why the Math of Correlation Matters WSJ 10-4-10Uploaded byaureliotiziano
- Ascension - The Infinite PathwayUploaded byapi-3839806
- Introduction to Computer Progamming in Qbasic LanguageUploaded byYoshitomi Maehara Aliaga
- 01 COMPLETE Advanced Course Offshore All LectureNotesUploaded byMiguel Teixeira Couteiro
- Airscrews for the AeromodellerUploaded byRussell Hartill
- full_textUploaded byRohadi Hashim
- Alshehri Naeem Spring 2010 -Quantification of Reservoir Uncertainty for Optimal Decision MakingUploaded byWladimir Rondan
- Week 1.pdfUploaded byAiman Alammari
- 225927983 Neural Network Predistortion Tech for RF AmplifiersUploaded byRafael Benitez
- Intro to Hypo TestingUploaded byfocus16hoursgmailcom
- harmonic pattern by jay purohitUploaded byNickKr
- MemristiveDevicesAndSystems.pdfUploaded byMuhammet Ateş
- 1.BinomialModelUploaded bylowchangsong
- Command Terms ActivityUploaded byAttila Dudas
- Weka SampleUploaded byLoveAstro
- bearing-capacity-evaluation-of-footing-on-a-layered-soil-using-abaqus-2157-7617-1000264.pdfUploaded byFreddy Rodriguez Badillo
- GEC225-Applied Computer Programming-Storage Classes and Scope No 4Uploaded byHenry Nwankwo