The Baseball Problem (Howard )

Dynamic Programming and Markov Processes by Ronald A. Howard, (MIT Press, Cambridge, Mas

Paul A. Jensen

This is a model of an inning of baseball from the offense point of view.

It is modeled in the DP Models add-in and solved as a Markov Decision Process.

Use the Start command to add or replace buttons on a worksheet.

Use the Finish command to remove buttons from the worksheet.
T Press, Cambridge, Massachusetts, 1960)
Markov Chain Transition Matrix
Type: DTMC Step Matrix Analyzed.
Title: Bulb Calculate Measure 5 Recurrent States
Change Month 1 Recurrent State Class
Analyze 0 Transient States
State 5 0 1 2 3
Index Names New 1-mo 2-mo 3-mo
Economics 0 New New 0.4 0.6 0 0
1 1-mo 1-mo 0.3 0 0.7 0
Transient 2 2-mo 2-mo 0.5 0 0 0.5
3 3-mo 3-mo 0.7 0 0 0
Steady State 4 4-mo 4-mo 0.8 0 0 0
Sum 2.7 0.6 0.7 0.5
n-step Probabilities

First Pass

Simulate

Absorbing States
 4
4-mo Sum Status
0 1 Class-1
0 1 Class-1
0 1 Class-1
0.3 1 Class-1
0.2 1 Class-1
0.5
Economic Data
Type: DTMC Transition Cost Matrix
Title: Bulb 0 1
State Combine
State Cost d Cost New 1-mo
0 New 2.5 2.5 0 0
Calculate 1 1-mo 0.5 0.5 0 0
2 2-mo 0.5 0.5 0 0
Matrix 3 3-mo 0.5 0.5 0 0
4 4-mo 0.5 0.5 0 0

conomic Measure Cost

Discount Rate 1.00%
Transition Cost Matrix
2 3 4

2-mo 3-mo 4-mo

0 0 0
0 0 0
0 0 0
0 0 0
0 0 0
Steady State Analysis
Type: DTMC Average
Title: Bulb 0 1 2 3 4 Cost
Solve by: Inverse New 1-mo 2-mo 3-mo 4-mo per period
Steady State Dist. 0.4331 0.2599 0.1819 0.091 0.0341 1.36626963

Expected NPW 138.9 137.01 137.43 137.7 137.81

Matrix
Markov Chain Transition Matrix
Type: DTMC Step Matrix Analyzed.
Title: Bulb_A Calculate Measure 1 Recurrent State
Change Month 1 Recurrent State Class
Analyze 4 Transient States
State 5 0 1 2 3
Index Names New 1-mo 2-mo 3-mo
Economics 0 New New 1 0 0 0
1 1-mo 1-mo 1 0 0 0
Transient 2 2-mo 2-mo 1 0 0 0
3 3-mo 3-mo 1 0 0 0
Steady State 4 4-mo 4-mo 1 0 0 0
Sum 5 0 0 0
n-step Probabilities

First Pass

Simulate

Absorbing States
 4
4-mo Sum Status
0 1 Class-1
0 1 Transient
0 1 Transient
0 1 Transient
0 1 Transient
0
Economic Data
Type: DTMC Transition Cost Matrix
Title: Bulb_A 0 1
State Combine
State Cost d Cost New 1-mo
0 New 1.7 1.7 0 0
Calculate 1 1-mo 0 0 0 0
2 2-mo 0 0 0 0
Matrix 3 3-mo 0 0 0 0
4 4-mo 0 0 0 0

conomic Measure Cost

Discount Rate 1.00%
Transition Cost Matrix
2 3 4

2-mo 3-mo 4-mo

0 0 0
0 0 0
0 0 0
0 0 0
0 0 0
Steady State Analysis
Type: DTMC Average
Title: Bulb_A 0 1 2 3 4 Cost
Solve by: Inverse New 1-mo 2-mo 3-mo 4-mo per period
Steady State Dist. 1 0 0 0 0 1.7

Expected NPW 171.7 170 170 170 170

Matrix
DP Solver
Type MDP
Title Bulb
Goal Min Solve Make Markov Chain
States 5
Actions 2 Change Make LP Model
Actions/State 2
Events 2 Equations
Events/Action 2
Iteration Type Value Gain
Policy Steps 10 0
Max. Val. Difference 1.2048 State List
Inde State State Decisio Action State
Sum Prob. Difference 0.0782 x Name Cost n Index Name Value
Time Measure Month 1 New 2 1 Inspect 14.1604
Economic Measure Cost 2 1-mo 0 3 Inspect 12.1947
Discount Rate 1.000% 3 2-mo 0 3 Inspect 12.523
Step Interval 1 4 3-mo 0 4 Replace 12.5294
5 4-mo 0 4 Replace 12.5294
 1 2
Next Value 12.958 10.99
State Prob. 0.2598 0.2728
Exp. Value
1.31219 Transition Probability
Decision List 1 2

Last Decisio State Action Decision Decisio Last

Prob. n Index Index Index Name n Value Prob. New 1-mo
0.45459 1 1 1 New / Inspect 12.1604 0.4546 0.4 0.6
0.25995 2 1 2 New / Replace 12.5294 0 1 0
0.16423 3 2 1 1-mo / Inspect 12.1947 0.2599 0.3 0
0.12124 4 2 2 1-mo / Replace 12.5294 0 1 0
0 5 3 1 2-mo / Inspect 12.523 0 0.5 0
6 3 2 2-mo / Replace 12.5294 0 1 0
7 4 1 3-mo / Inspect 12.8456 0 0.7 0
8 4 2 3-mo / Replace 12.5294 0 1 0
9 5 1 4-mo / Inspect 13.0069 0 0.8 0
10 5 2 4-mo / Replace 12.5294 0 1 0

SolutionValue Solution Matrix

134.47 0.604 -0.594
132.5 -0.297 1
132.83 -0.495 0
132.83 -0.99 0
132.83 -0.99 0

Probability Solution Matrix

-0.6 0.6
0.3 -1
0.5 0
1 0
1 0

Const. 0 0

SS Prob. 0.4484 0.2691

 3 4 5
11.32 11.329 11.329
0.182 0 0

sition Probability Transition Cost

3 4 5 1 2 3 4 5

2-mo 3-mo 4-mo Row Sum New 1-mo 2-mo 3-mo 4-mo
0 0 0 1 0 0 0 0 0
0 0 0 1 0 0 0 0 0
0.7 0 0 1 0 0 0 0 0
0 0 0 1 0 0 0 0 0
0 0.5 0 1 0 0 0 0 0
0 0 0 1 0 0 0 0 0
0 0 0.3 1 0 0 0 0 0
0 0 0 1 0 0 0 0 0
0 0 0.2 1 0 0 0 0 0
0 0 0 1 0 0 0 0 0
Min Sum 1
Max Sum 1
Min Prob. 0

e Solution Matrix Reward

0 0 0 2.5
-0.693 0 0 0.5
1 -0.495 0 0.5
0 1 0 -0.3
0 0 1 -0.3

bability Solution Matrix

0 0 1
0.7 0 1
-1 0.5 1
0 -1 1
0 0 1

0 0 1

0.1883 0.0942 0
DP Solver
Type MDP
Title Bulb_DP1
Goal Min Solve Make Markov Chain
States 5
Actions 2 Change Make LP Model
Actions/State 2
Events 2 Equations
Events/Action 2
Iteration Type Value
Iteration Steps 0
Stop Dif. 0.00001 State List Action List
State State Final
Value Error 99999 Index Name Cost Cost Index
Prob. Error 99999 1 New 2 0 1
Time Measure Month 2 1-mo 0 0 2
Economic Measure Cost 3 2-mo 0 0 3
Discount Rate 1.000% 4 3-mo 0 0 4
Discount Factor 99.0% 5 4-mo 0 0
Step Interval 1
Action List Event List Decision List
Action Action Event Event Decisio State
Name Cost Index Name Cost Prob. n Index Index
Inspect 0.5 1 Survive 0 1 1 1
Replace -0.3 2 Fail 0 0 2 1
Null 0 3 Null 0 0 3 2
NA 999999 4 NA 999999 0 4 2
5 3
6 3
7 4
8 4
9 5
10 5
 1 2 3 4 5
0 0 0 0 0
1 0 0 0 0

Transition Probability Transition Cost

sion List 1 2 3 4 5 1 2 3

Action Decision
Index Name New 1-mo 2-mo 3-mo 4-mo Row Sum New 1-mo 2-mo
1 New / Inspect 0.4 0.6 0 0 0 1 0 0 0
2 New / Replace 1 0 0 0 0 1 0 0 0
1 1-mo / Inspect 0.3 0 0.7 0 0 1 0 0 0
2 1-mo / Replace 1 0 0 0 0 1 0 0 0
1 2-mo / Inspect 0.5 0 0 0.5 0 1 0 0 0
2 2-mo / Replace 1 0 0 0 0 1 0 0 0
1 3-mo / Inspect 0.7 0 0 0 0.3 1 0 0 0
2 3-mo / Replace 1 0 0 0 0 1 0 0 0
1 4-mo / Inspect 0.8 0 0 0 0.2 1 0 0 0
2 4-mo / Replace 1 0 0 0 0 1 0 0 0
Min Sum 1
Max Sum 1
Min Prob. 0
 4 5

3-mo 4-mo
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
 Final Step 10 Initial
Sum State Action Action State State Next
Prob. Index Name Index Name Value Prob. Value
6.97391 1 New 1 Inspect 14.1604 0.46317 0
3.30129 2 1-mo 1 Inspect 12.1947 0.27275 0
2.11101 3 2-mo 1 Inspect 12.523 0.18196 0
0.61378 4 3-mo 2 Replace 12.5294 0.08212 0
0 5 4-mo 2 Replace 12.5294 0 0
 Value
Step 0 Step 1
Last Action Action State State Action Action State State
Prob. Index Name Value Prob. Index Name Value Prob.
1 2 Replace 1.7 1 1 Inspect 2.99505 0.4
0 2 Replace -0.3 0 1 Inspect 0.79703 0.6
0 2 Replace -0.3 0 1 Inspect 1.19307 0
0 2 Replace -0.3 0 2 Replace 1.38317 0
0 2 Replace -0.3 0 2 Replace 1.38317 0
Step 2 Step 3 Step 4
Action Action State State Action Action State State Action
Index Name Value Prob. Index Name Value Prob. Index
1 Inspect 4.15964 0.34 1 Inspect 5.46411 0.628 1
1 Inspect 2.2165 0.24 1 Inspect 3.58284 0.204 1
2 Replace 2.6654 0.42 2 Replace 3.81846 0.168 1
2 Replace 2.6654 0 2 Replace 3.81846 0 2
2 Replace 2.6654 0 2 Replace 3.81846 0 2
 Step 5 Step 6
Action State State Action Action State State Action Action
Name Value Prob. Index Name Value Prob. Index Name
Inspect 6.79243 0.3964 1 Inspect 8.02341 0.427 1 Inspect
Inspect 4.76946 0.3768 1 Inspect 6.04897 0.23784 1 Inspect
Inspect 5.09533 0.1428 1 Inspect 6.3923 0.26376 2 Replace
Replace 5.11001 0.084 2 Replace 6.42517 0.0714 2 Replace
Replace 5.11001 0 2 Replace 6.42517 0 2 Replace
 Step 7 Step 8
State State Action Action State State Action Action State
Value Prob. Index Name Value Prob. Index Name Value
9.27104 0.57731 1 Inspect 10.5163 0.39103 1 Inspect 11.745
7.3135 0.2562 1 Inspect 8.55158 0.34639 1 Inspect 9.7738
7.64397 0.16649 1 Inspect 8.87377 0.17934 1 Inspect 10.1018
7.64397 0 2 Replace 8.87924 0.08324 2 Replace 10.1122
7.64397 0 2 Replace 8.87924 0 2 Replace 10.1122
 Step 9 Step 10
State Action Action State State Action Action State State
Prob. Index Name Value Prob. Index Name Value Prob.
0.43324 1 Inspect 12.9577 0.45459 1 Inspect 14.1604 0.46317
0.23462 1 Inspect 10.9899 0.25995 1 Inspect 12.1947 0.27275
0.24247 1 Inspect 11.3204 0.16423 1 Inspect 12.523 0.18196
0.08967 2 Replace 11.3287 0.12124 2 Replace 12.5294 0.08212
0 2 Replace 11.3287 0 2 Replace 12.5294 0
Markov Chain Transition Matrix
Type: DTMC Step Matrix Analyzed.
Title: Bulb1 Calculate Measure 4 Recurrent States
Change Month 1 Recurrent State Class
Analyze 1 Transient State
State 5 0 1 2 3
Index Names New 1-mo 2-mo 3-mo
Economics 0 New New 0.4 0.6 0 0
1 1-mo 1-mo 0.3 0 0.7 0
Transient 2 2-mo 2-mo 0.5 0 0 0.5
3 3-mo 3-mo 1 0 0 0
Steady State 4 4-mo 4-mo 1 0 0 0
Sum 3.2 0.6 0.7 0.5
n-step Probabilities

First Pass

Simulate

Absorbing States
 4
4-mo Sum Status
0 1 Class-1
0 1 Class-1
0 1 Class-1
0 1 Class-1
0 1 Transient
0
Economic Data
Type: DTMC Transition Cost Matrix
Title: Bulb1 0 1
State Combine
State Cost d Cost New 1-mo
0 New 2.5 2.5 0 0
Calculate 1 1-mo 0.5 0.5 0 0
2 2-mo 0.5 0.5 0 0
Matrix 3 3-mo -0.3 -0.3 0 0
4 4-mo -0.3 -0.3 0 0

conomic Measure Cost

Discount Rate 1.00%
Transition Cost Matrix
2 3 4

2-mo 3-mo 4-mo

0 0 0
0 0 0
0 0 0
0 0 0
0 0 0
Steady State Analysis
Type: DTMC Average
Title: Bulb1 0 1 2 3 4 Cost
Solve by: Inverse New 1-mo 2-mo 3-mo 4-mo per period
Steady State Dist. 0.4484 0.2691 0.1883 0.0942 0 1.32152466

Expected NPW 134.47 132.5 132.83 132.83 132.83

Matrix
DP Solver
Type MDP
Title Bulb1
Goal Min Solve Make Markov Chain
States 5
Actions 2 Change Make LP Model
Actions/State 2
Events 2 Equations
Events/Action 2
Iteration Type Policy
Policy Steps 2
Max. Val. Difference 3E-014 State List

Decisio Decisio
State State n n Action
Sum Prob. Difference 0 Index Name Cost Index Range Index
Time Measure Month 1 New 2 1 2 1
Economic Measure Cost 2 1-mo 0 3 2 1
Discount Rate 1.000% 3 2-mo 0 3 2 1
Step Interval 1 4 3-mo 0 4 2 2
5 4-mo 0 4 2 2
Beta 0.9901

=SUMPRODUCT(Bulb_DPDecProb,AH14:AH23
)
 =SUMPRODUCT(Bulb_DPStateNextValue,AH14:AL14
)
=INDEX(Bulb_DPActionReward,Bulb_DPDecActionIndex)

=Bulb_DPDecReward+Bulb_DPDecTransReward+(Bulb_DPDecFuture/
(1+Bulb_DPDiscount))
 Gain Exp. Value
0 1.3215
Action List Event List

Decisio
Action n Step State Last Action Action
Name Value Value Value Prob. Index Name Cost Index
Inspect 132.47 2.5 134.47 0.4484 1 Inspect 0.5 1
Inspect 132.5 0.5 132.5 0.2691 2 Replace -0.3 2
Inspect 132.83 0.5 132.83 0.1883 3 Null 0 3
Replace 132.83 -0.3 132.83 0.0942 4 NA 999999 4
Replace 132.83 -0.3 132.83 0
e/
Event List Decision List Expect
ed
Decisio Transit
Event Event n State Action Decision Decisio ion
Name Cost Prob. Index Index Index Name n Cost Cost
Survive 0 1 1 1 1 New / Inspect 0.5 0
Fail 0 0 2 1 2 New / Replace -0.3 0
Null 0 0 3 2 1 1-mo / Inspect 0.5 0
NA 999999 0 4 2 2 1-mo / Replace -0.3 0
5 3 1 2-mo / Inspect 0.5 0
6 3 2 2-mo / Replace -0.3 0
7 4 1 3-mo / Inspect 0.5 0
8 4 2 3-mo / Replace -0.3 0
9 5 1 4-mo / Inspect 0.5 0
10 5 2 4-mo / Replace -0.3 0
 1 2 3 4 5
Next Value 134.47 132.5 132.83 132.83 132.83
State Prob. 0.2601 0.2691 0.1883 0 0

Transition Probability
1 2 3 4 5
Expect
ed Decisio
Next n Last
Cost Value Prob. New 1-mo 2-mo 3-mo 4-mo Row Sum
133.29 132.47 0.4484 0.4 0.6 0 0 0 1
134.47 132.83 0 1 0 0 0 0 1
133.32 132.5 0.2691 0.3 0 0.7 0 0 1
134.47 132.83 0 1 0 0 0 0 1
133.65 132.83 0 0.5 0 0 0.5 0 1
134.47 132.83 0 1 0 0 0 0 1
133.98 133.15 0 0.7 0 0 0 0.3 1
134.47 132.83 0 1 0 0 0 0 1
134.14 133.31 0 0.8 0 0 0 0.2 1
134.47 132.83 0 1 0 0 0 0 1
Min Sum 1
Max Sum 1
Min Prob. 0

Solution Value Solution Matrix Reward

134.47 0.604 -0.5941 0 0 0 2.5
132.5 -0.297 1 -0.6931 0 0 0.5
132.83 -0.495 0 1 -0.495 0 0.5
132.83 -0.9901 0 0 1 0 -0.3

132.83 -0.9901 0 0 0 1 -0.3

Probability Solution Matrix

-0.6 0.6 0 0 1
0.3 -1 0.7 0 1
0.5 0 -1 0.5 1
1 0 0 -1 1
1 0 0 0 1
 Const. 0 0 0 0 1

SS Prob. 0.4484 0.2691 0.1883 0.0942 0

 Discount rate 0.01 Beta 0.9901

Variable V1 V2 V3 V4 V5
Decisio Decisio
n State Action n
Index Index Index Name New 1-mo 2-mo 3-mo 4-mo
1 1 1 New / Inspect 0.604 -0.5941 0 0 0
2 1 2 New / Replace0.0099 0 0 0 0
3 2 1 1-mo / Inspect
-0.297 1 -0.6931 0 0
4 2 2 1-mo / Replace
-0.9901 1 0 0 0
5 3 1 2-mo / Inspect
-0.495 0 1 -0.495 0
6 3 2 2-mo / Replace
-0.9901 0 1 0 0
7 4 1 3-mo / Inspect
-0.6931 0 0 1 -0.297
8 4 2 3-mo / Replace
-0.9901 0 0 1 0
9 5 1 4-mo / Inspect
-0.7921 0 0 0 0.802
10 5 2 4-mo / Replace
-0.9901 0 0 0 1
Immed
iate
Cost
2.5
1.7
0.5
-0.3
0.5
-0.3
0.5
-0.3
0.5
-0.3
Linear Model Name:Bulb1_LP Solver:
1 Type: LP1 Type:
0 Change Goal: Max Sens.:
1 Profit: 134.47 Side:
0 Solve
0 Variables 1 2 3
100 Vary Name: New 1-mo 2-mo
100 Values: 134.466 132.498 132.827
0 Change Relation Lower Bounds: -10000 -10000 -10000
60 Upper Bounds: 10000 10000 10000

Linear Obj. Coef.: 1 0 0

Constraints
Num. Name Value Rel. RHS Linear Constraint Coefficients
1 New / Inspect 2.5 <= 2.5 0.60396 -0.5941 0
2 New / Replace 1.3313 <= 1.7 0.0099 0 0
3 1-mo / Inspect 0.5 <= 0.5 -0.297 1 -0.6931
4 1-mo / Replace -0.636 <= -0.3 -0.9901 1 0
5 2-mo / Inspect 0.5 <= 0.5 -0.495 0 1
6 2-mo / Replace -0.308 <= -0.3 -0.9901 0 1
7 3-mo / Inspect 0.1846 <= 0.5 -0.6931 0 0
8 3-mo / Replace -0.3 <= -0.3 -0.9901 0 0
9 4-mo / Inspect 0.023 <= 0.5 -0.7921 0 0
10 4-mo / Replace -0.3 <= -0.3 -0.9901 0 0

New 1-mo 2-mo

0.60396 -0.5941 0
0.0099 0 0
-0.297 1 -0.6931
-0.9901 1 0
-0.495 0 1
-0.9901 0 1
-0.6931 0 0
-0.9901 0 0
-0.7921 0 0
-0.9901 0 0
 Jensen LP/IP Ph. 1 Iter. 4
Linear Total Iter. 7
Yes Comp. Time 00:01
No Status Optimal

4 5
3-mo 4-mo
132.834 132.834
-10000 -10000
10000 10000

0 0

Coefficients
0 0
0 0
0 0
0 0
-0.495 0
0 0
1 -0.297
1 0
0 0.80198
0 1

3-mo 4-mo
0 0
0 0
0 0
0 0
-0.495 0
0 0
1 -0.297
1 0
0 0.80198
0 1
Sensitivity Analysis for Worksheet Bulb1_LP

Variable Analysis Objective Value: 134.4656

Reduced Objective Range Range
Num. Name Value Status Cost Coefficient Lower Limit Upper Limit
1 New 134.4656 Basic 0 1 0 ---
2 1-mo 132.4984 Basic 0 0 -0.9836 ---
3 2-mo 132.8267 Basic 0 0 -0.9631 ---
4 3-mo 132.8343 Basic 0 0 -0.9112 ---
5 4-mo 132.8343 Basic 0 0 0 0

Constraint Analysis
Shadow Constraint Range Range
Num. Name Value Status Price Limit Lower Limit Upper Limit
1 New / Inspect 2.5 Upper 45.7095 2.5 2.4661 3.3146
2 New / Replace 1.3313 Basic 0 1.7 1.3313 ---
3 1-mo / Inspect 0.5 Upper 27.1541 0.5 0.4429 0.9656
4 1-mo / Replace -0.6359 Basic 0 -0.3 -0.6359 ---
5 2-mo / Inspect 0.5 Upper 18.8197 0.5 -518.0384 0.5084
6 2-mo / Replace -0.3076 Basic 0 -0.3 -0.3076 ---
7 3-mo / Inspect 0.1846 Basic 0 0.5 0.1846 ---
8 3-mo / Replace -0.3 Upper 9.3167 -0.3 -991.3403 -0.2831
9 4-mo / Inspect 0.023 Basic 0 0.5 0.023 ---
10 4-mo / Replace -0.3 Upper 0 -0.3 -1.362 0.2947
DP Solver
Type MDP
Title Bulb_DP2
Goal Min Solve Make Markov Chain
States 5
Actions 2 Change Make LP Model
Actions/State 2
Events 2 Equations
Events/Action 2
Iteration Type Fixed/Value
Iteration Steps 37
Stop Dif. 0.00001 State List
State State Final Decisio
Value Error 8.73E-006 Index Name Cost Cost n Index
Prob. Error 8.75E-008 1 New 2 0 1
Time Measure Month 2 1-mo 0 0 3
Economic Measure Cost 3 2-mo 0 0 5
Discount Rate 1.000% 4 3-mo 0 0 8
Discount Factor 99.0% 5 4-mo 0 0 10
Step Interval 0

Index
1
2
3
4
5
Max.

C_L
C_U
Val. Error
Prob. Error
 Gain Exp. Value
0 1.32152
Decision List
Decisio
n Action Action Decisio Step Discoun Last Decisio State
Range Index Name n Value Value t Value Prob. n Index Index
2 1 Inspect 132.466 2.5 134.466 0.44843 1 1
2 1 Inspect 132.498 0.5 132.498 0.26906 2 1
2 1 Inspect 132.827 0.5 132.827 0.18834 3 2
2 2 Replace 132.834 -0.3 132.834 0.09417 4 2
2 2 Replace 132.834 -0.3 132.834 0 5 3
6 3
7 4
State Value Probability 8 4
Value Dif.Abs. Dif. Lower Upper Next Val.Abs. Dif. 9 5
4E-008 4E-008 134.466 134.466 134.466 2E-008 10 5
-3E-009 3E-009 132.498 132.498 132.498 2E-008
-5E-008 5E-008 132.827 132.827 132.827 2E-008
-2E-008 2E-008 132.834 132.834 132.834 2E-008
-2E-008 2E-008 132.834 132.834 132.834 0
5E-008 2E-008

-5E-006
4E-006
9E-006
9E-008
 1 2 3 4 5
Next Value 134.5 132.5 132.8 132.8 132.8
State Prob. 0.448 0.269 0.188 0.094 0

Transition Probability
sion List 1 2 3 4 5

Action Decisio Last

Index n Value Prob. New 1-mo 2-mo 3-mo 4-mo Row Sum
1 132.466 0.44843 0.4 0.6 0 0 0 1
2 132.834 0 1 0 0 0 0 1
1 132.498 0.26906 0.3 0 0.7 0 0 1
2 132.834 0 1 0 0 0 0 1
1 132.827 0.18834 0.5 0 0 0.5 0 1
2 132.834 0 1 0 0 0 0 1
1 133.15 0 0.7 0 0 0 0.3 1
2 132.834 0.09417 1 0 0 0 0 1
1 133.311 0 0.8 0 0 0 0.2 1
2 132.834 0 1 0 0 0 0 1
Min Sum 1
Max Sum 1
Min Prob. 0
 Exp.
Gain Value
0 1.3087

Final Step 8 Initial

State Action Discoun State Next Last
Index Name Name t Value Prob. Value Prob.
1 New Inspect 11.18928 0.45854 0 1
2 1-mo Inspect 9.255527 0.26894 0 0
3 2-mo Inspect 9.577884 0.1802 0 0
4 3-mo Replace 9.569928 0.09232 0 0
5 4-mo Replace 9.569928 0 0 0
 Exp. Exp. Exp.
Gain Value Gain Value Gain Value
0 2.5 0 1.3 0 1.18
Fixed/Value Iterations Fixed/Value Iterations
Step 1 Step 2 Step 3
Action Discoun State Action Discoun State Action Discoun State
Name t Value Prob. Name t Value Prob. Name t Value Prob.
Inspect 2.5 0.4 Inspect 3.787129 0.34 Inspect 4.943878 0.418
Inspect 0.5 0.6 Inspect 1.589109 0.24 Inspect 2.726252 0.204
Inspect 0.5 0 Inspect 1.589109 0.42 Inspect 3.451671 0.168
Replace -0.3 0 Replace 2.175248 0 Replace 3.449632 0.21
Replace -0.3 0 Replace 2.175248 0 Replace 3.449632 0
 Exp. Exp. Exp.
Gain Value Gain Value Gain Value
0 1.168 0 1.4776 0 1.32208

Step 4 Step 5 Step 6

Action Discoun State Action Discoun State Action Discoun State
Name t Value Prob. Name t Value Prob. Name t Value Prob.
Inspect 6.077527 0.5224 Inspect 7.497472 0.4396 Inspect 8.755386 0.42905
Inspect 4.360726 0.2508 Inspect 5.531585 0.31344 Inspect 6.735265 0.26376
Inspect 4.655203 0.1428 Inspect 5.783394 0.17556 Inspect 7.041993 0.21941
Replace 4.594929 0.084 Replace 5.717354 0.0714 Replace 7.12324 0.08778
Replace 4.594929 0 Replace 5.717354 0 Replace 7.12324 0
 Exp. Exp.
Gain Value Gain Value
0 1.28788 0 1.3087

Step 7 Step 8
Action Discoun State Action Discoun State
Name t Value Prob. Name t Value Prob.
Inspect 9.968627 0.44823 Inspect 11.18928 0.45854
Inspect 7.981199 0.25743 Inspect 9.255527 0.26894
Inspect 8.360706 0.18463 Inspect 9.577884 0.1802
Replace 8.368699 0.1097 Replace 9.569928 0.09232
Replace 8.368699 0 Replace 9.569928 0
DP Solver
Type MDP
Title Bulb2
Goal Min Solve Make Markov Chain
States 5
Actions 2 Change Make LP Model
Actions/State 2
Events 2 Equations
Events/Action 2
Iteration Type Policy
Policy Steps 2
Max. Val. Difference 2E-016 State List
Decisio
State State Action Action n
Sum Prob. Difference 0 Index Name Cost Index Name Value
Time Measure Month 1 New 2 1 Inspect 0.943
Economic Measure Cost 2 1-mo 0 1 Inspect 0.9789
Discount Rate 0.000% 3 2-mo 0 1 Inspect 1.3108
Step Interval 1 4 3-mo 0 2 Replace 1.3215
5 4-mo 0 2 Replace 1.3215
 Next Value
State Prob.
Gain Exp. Value
1.3215 1.32152
Decision List
Step State Last Decisio State Action Decision Decisio Last
Value Value Prob. n Index Index Index Name n Cost Prob.
2.5 2.943 0.44843 1 1 1 New / Inspect 0.5 0.4484
0.5 0.9789 0.26906 2 1 2 New / Replace -0.3 0
0.5 1.3108 0.18834 3 2 1 1-mo / Inspect 0.5 0.2691
-0.3 1.3215 0.09417 4 2 2 1-mo / Replace -0.3 0
-0.3 1.3215 0 5 3 1 2-mo / Inspect 0.5 0
6 3 2 2-mo / Replace -0.3 0
7 4 1 3-mo / Inspect 0.5 0
8 4 2 3-mo / Replace -0.3 0
9 5 1 4-mo / Inspect 0.5 0
10 5 2 4-mo / Replace -0.3 0

Solution
1.6215
-0.3426
-0.0108
0
1.3215

Const.

SS Prob.
 1 2 3 4 5
1.6215 -0.3426 -0.0108 0 0
0.2601 0.2691 0.1883 0 0

Transition Probability
1 2 3 4 5

New 1-mo 2-mo 3-mo 4-mo Row Sum

0.4 0.6 0 0 0 1
1 0 0 0 0 1
0.3 0 0.7 0 0 1
1 0 0 0 0 1
0.5 0 0 0.5 0 1
1 0 0 0 0 1
0.7 0 0 0 0.3 1
1 0 0 0 0 1
0.8 0 0 0 0.2 1
1 0 0 0 0 1
Min Sum 1
Max Sum 1
Min Prob. 0

Value Solution Matrix Reward

0.6 -0.6 0 0 1 2.5
-0.3 1 -0.7 0 1 0.5
-0.5 0 1 -0.5 1 0.5
-1 0 0 1 1 -0.3
-1 0 0 0 1 -0.3

Probability Solution Matrix

-0.6 0.6 0 0 1
0.3 -1 0.7 0 1
0.5 0 -1 0.5 1
1 0 0 -1 1
1 0 0 0 1

0 0 0 0 1

0.4484 0.2691 0.1883 0.0942 0

 Exp.
Gain Value
1.32152 1.32152

Final Step 2
State Action Action Step Next Last
Index Name Index Name Value Value Prob.
1 New 1 Inspect 2.5 1.62152 0.44843
2 1-mo 1 Inspect 0.5 -0.3426 0.26906
3 2-mo 1 Inspect 0.5 -0.0108 0.18834
4 3-mo 2 Replace -0.3 0 0.09417
5 4-mo 2 Replace -0.3 0 0
 Exp.
Gain Value Gain
1.7 1.7 1.32152
Policy
Step 1 Step 2
Action Action Step Next Last Action Action Step Next
Index Name Value Value Prob. Index Name Value Value
2 Replace 1.7 2 1 1 Inspect 2.5 1.62152
2 Replace -0.3 0 0 1 Inspect 0.5 -0.3426
2 Replace -0.3 0 0 1 Inspect 0.5 -0.0108
2 Replace -0.3 0 0 2 Replace -0.3 0
2 Replace -0.3 0 0 2 Replace -0.3 0
 Exp.
Value
1.32152

Last
Prob.
0.44843
0.26906
0.18834
0.09417
0
DP Solver
Type MDP
Title Bulb_DP3
Goal Min Solve Make Markov Chain
States 5
Actions 2 Change Make LP Model
Actions/State 2
Events 3 Equations
Events/Action 2
Iteration Type Value
Iteration Steps 0
Stop Dif. 0.00001 State List
State State Final Decision
Value Error 99999 Index Name Cost Cost Index
Prob. Error 99999 1 New 2 0 2
Time Measure Month 2 1-mo 0 0 4
Economic Measure Cost 3 2-mo 0 0 4
Discount Rate 1.000% 4 3-mo 0 0 4
Discount Factor 99.0% 5 4-mo 0 0 4
Step Interval 0
 Gain Exp. Value
0 1.7

Decision Action Action Decision Step State Next State Last

Range Index Name Value Value Value Value Prob. Prob.
2 2 Replace -0.3 1.7 1.7 0 1 1
2 2 Replace -0.3 -0.3 -0.3 0 0 0
2 2 Replace -0.3 -0.3 -0.3 0 0 0
2 2 Replace -0.3 -0.3 -0.3 0 0 0
2 2 Replace -0.3 -0.3 -0.3 0 0 0
 Action List Event List Decision List
Action Action Event Event Decision
Index Name Cost Index Name Cost Prob. Index
1 Inspect 0.5 1 Survive 0 1 1
2 Replace -0.3 2 Fail 0 0 2
3 Null 0 3 New 0 0 3
4 NA 999999 4 Null 0 0 4
5 NA 999999 0 5
6
7
8
9
10
Decision List
Expected Expected
State Action Decision Decision Transitio Transitio Next Decision
Index Index Name Cost n Range n Cost Cost Value
1 1 New / Inspect 0.5 2 0 0 0.5
1 2 New / Replace -0.3 2 0 0 -0.3
2 1 1-mo / Inspect 0.5 2 0 0 0.5
2 2 1-mo / Replace -0.3 2 0 0 -0.3
3 1 2-mo / Inspect 0.5 2 0 0 0.5
3 2 2-mo / Replace -0.3 2 0 0 -0.3
4 1 3-mo / Inspect 0.5 2 0 0 0.5
4 2 3-mo / Replace -0.3 2 0 0 -0.3
5 1 4-mo / Inspect 0.5 2 0 0 0.5
5 2 4-mo / Replace -0.3 2 0 0 -0.3
 Transition List
Last Decisio Event Transitio Transitio
Prob. Index n Index Index Transition Name n Cost n Prob.
0 1 1 1 New / Inspect / Survive 0 0.6
1 2 1 2 New / Inspect / Fail 0 0.4
0 3 2 3 New / Replace / New 0 1
0 4 3 1 1-mo / Inspect / Survive 0 0.7
0 5 3 2 1-mo / Inspect / Fail 0 0.3
0 6 4 3 1-mo / Replace / New 0 1
0 7 5 1 2-mo / Inspect / Survive 0 0.5
0 8 5 2 2-mo / Inspect / Fail 0 0.5
0 9 6 3 2-mo / Replace / New 0 1
0 10 7 1 3-mo / Inspect / Survive 0 0.3
11 7 2 3-mo / Inspect / Fail 0 0.7
12 8 3 3-mo / Replace / New 0 1
13 9 1 4-mo / Inspect / Survive 0 0.2
14 9 2 4-mo / Inspect / Fail 0 0.8
15 10 3 4-mo / Replace / New 0 1
Next Next Next Next
State Name Cost Prob.
2 1-mo 0 0
1 New 0 0
1 New 0 1
3 2-mo 0 0
1 New 0 0
1 New 0 0
4 3-mo 0 0
1 New 0 0
1 New 0 0
5 4-mo 0 0
1 New 0 0
1 New 0 0
5 4-mo 0 0
1 New 0 0
1 New 0 0
DP Solver
Type MDP
Title Bulb3
Goal Min Solve Make Markov Chain
States 5
Actions 2 Change Make LP Model
Actions/State 2
Events 3 Equations
Events/Action 2
Iteration Type Policy
Policy Steps 2
Max. Val. Difference 3E-014 State List
State State Decision Decision
Sum Prob. Difference 0 Index Name Cost Index Range
Time Measure Month 1 New 2 1 2
Economic Measure Cost 2 1-mo 0 3 2
Discount Rate 1.000% 3 2-mo 0 3 2
Step Interval 1 4 3-mo 0 4 2
5 4-mo 0 4 2
 Gain Exp. Value
0 1.321525

Action Action Decision Step State Next State Last

Index Name Value Value Value Value Prob. Prob.
1 Inspect 132.4656 2.5 134.4656 134.4656 0.26009 0.44843
1 Inspect 132.4984 0.5 132.4984 132.4984 0.269058 0.269058
1 Inspect 132.8267 0.5 132.8267 132.8267 0.188341 0.188341
2 Replace 132.8343 -0.3 132.8343 132.8343 0 0.09417
2 Replace 132.8343 -0.3 132.8343 132.8343 0 0
 Action List Event List Decision List
Action Action Event Event Decisio
Index Name Cost Index Name Cost Prob. n Index
1 Inspect 0.5 1 Survive 0 1 1
2 Replace -0.3 2 Fail 0 0 2
3 Null 0 3 New 0 0 3
4 NA 999999 4 Null 0 0 4
5 NA 999999 0 5
6
7
8
9
10
Decision List Transition List
Expected
State Action Decision Decision Transitio Decisio Event
Index Index Name Cost n Cost Index n Index Index
1 1 New / Inspect 0.5 0 1 1 1
1 2 New / Replace -0.3 0 2 1 2
2 1 1-mo / Inspect 0.5 0 3 2 3
2 2 1-mo / Replace -0.3 0 4 3 1
3 1 2-mo / Inspect 0.5 0 5 3 2
3 2 2-mo / Replace -0.3 0 6 4 3
4 1 3-mo / Inspect 0.5 0 7 5 1
4 2 3-mo / Replace -0.3 0 8 5 2
5 1 4-mo / Inspect 0.5 0 9 6 3
5 2 4-mo / Replace -0.3 0 10 7 1
11 7 2
12 8 3
13 9 1
14 9 2
15 10 3
 Transitio Transitio Next Next Next Next
Transition Name n Cost n Prob. State Name Cost Prob.
New / Inspect / Survive 0 0.6 2 1-mo 132.4984 0.269058
New / Inspect / Fail 0 0.4 1 New 134.4656 0.179372
New / Replace / New 0 1 1 New 134.4656 0
1-mo / Inspect / Survive 0 0.7 3 2-mo 132.8267 0.188341
1-mo / Inspect / Fail 0 0.3 1 New 134.4656 0.080717
1-mo / Replace / New 0 1 1 New 134.4656 0
2-mo / Inspect / Survive 0 0.5 4 3-mo 132.8343 0
2-mo / Inspect / Fail 0 0.5 1 New 134.4656 0
2-mo / Replace / New 0 1 1 New 134.4656 0
3-mo / Inspect / Survive 0 0.3 5 4-mo 132.8343 0
3-mo / Inspect / Fail 0 0.7 1 New 134.4656 0
3-mo / Replace / New 0 1 1 New 134.4656 0
4-mo / Inspect / Survive 0 0.2 5 4-mo 132.8343 0
4-mo / Inspect / Fail 0 0.8 1 New 134.4656 0
4-mo / Replace / New 0 1 1 New 134.4656 0
Solution Value Solution Matrix
134.4656 0.60396 -0.59406 0 0
132.4984 -0.29703 1 -0.69307 0
132.8267 -0.49505 0 1 -0.49505
132.8343 -0.9901 0 0 1
132.8343 -0.9901 0 0 0

Probability Solution Matrix

-0.6 0.6 0 0
0.3 -1 0.7 0
0.5 0 -1 0.5
1 0 0 -1
1 0 0 0

Const. 0 0 0 0

SS Prob. 0.44843 0.269058 0.188341 0.09417

 Reward
0 2.5
0 0.5
0 0.5
0 -0.3
1 -0.3

1
1
1
1
1

0
Transition Probabilities

New 1-mo 2-mo 3-mo 4-mo

New 0.4 0.6 0 0 0
1-mo 0.3 0 0.7 0 0
2-mo 0.5 0 0 0.5 0
3-mo 0.7 0 0 0 0.3
4-mo 0.8 0 0 0 0.2

Transition Probabilities

New 1-mo 2-mo 3-mo 4-mo

New 1 0 0 0 0
1-mo 1 0 0 0 0
2-mo 1 0 0 0 0
3-mo 1 0 0 0 0
4-mo 1 0 0 0 0

Decision
Name New 1-mo 2-mo 3-mo 4-mo
New / Inspect 0.4 0.6 0 0 0
1-mo / Inspect 0.3 0 0.7 0 0
2-mo / Inspect 0.5 0 0 0.5 0
3-mo / Replace 1 0 0 0 0
4-mo / Replace 1 0 0 0 0
Costs
State
State Cost
New 2.5
1-mo 0.5
2-mo 0.5
3-mo 0.5
4-mo 0.5

Costs
State Decision
State Cost Name New 1-mo 2-mo 3-mo
New 1.7 New / Inspect 0.4 0.6 0 0
1-mo 0 New / Replace 1 0 0 0
2-mo 0 1-mo / Inspect 0.3 0 0.7 0
3-mo 0 1-mo / Replace 1 0 0 0
4-mo 0 2-mo / Inspect 0.5 0 0 0.5
2-mo / Replace 1 0 0 0
3-mo / Inspect 0.7 0 0 0
State Action
State Cost Cost 3-mo / Replace 1 0 0 0
New 2 0.5 4-mo / Inspect 0.8 0 0 0
1-mo 0 0.5 4-mo / Replace 1 0 0 0
2-mo 0 0.5
3-mo 0 -0.3
4-mo 0 -0.3
 Decision Decision
4-mo Cost Name New
0 0.5 New / Inspect 0.4
0 -0.3 1-mo / Inspect 0.3
0 0.5 2-mo / Inspect 0.5
0 -0.3 3-mo / Replace 1
0 0.5 4-mo / Replace 1
0 -0.3
0.3 0.5

0 -0.3
0.2 0.5
0 -0.3
 Decision
1-mo 2-mo 3-mo 4-mo Cost
0.6 0 0 0 New / Inspect 0.5
0 0.7 0 0 1-mo / Inspect 0.5
0 0 0.5 0 2-mo / Inspect 0.5
0 0 0 0 3-mo / Replace -0.3
0 0 0 0 4-mo / Replace -0.3
 State Action Event
Variabl Variabl Variabl State
Problem es es es Blocks
Cab - MDP x x x x
Baseball - MDP 4 1 1 2
Replacement - M 1 1 1 1
Sequence - MDP 6 1 1 0
Birth-Death-MC 1 0 2 1
Birth-Death-MDP 1 1 2 1
Investment DDP 2 1 0 2
Queue - MDP 2 1 1 0
Doors - MDP 2 1 1 3
Decisio
n Transitio Transition
Blocks n Blocks States Actions Events Decisions s
x x 3 3 3 6 27
8 26 33 5 14 78 402
1 5 40 41 2 1640 3239
2 1 729 7 2 2188 3646
0 3 11 0 2 0 32
3 3 11 2 2 16 47
0 1 100 5 0 372 372
2 2 55 3 3 143 416
0 0 17 5 4 73 227