3/15/2017 Sensitivityanalysis:strategies,methods,concepts,examples

Sensitivityanalysis:strategies,methods,concepts,examples
DavidJ.Pannell
SchoolofAgriculturalandResourceEconomics,UniversityofWesternAustralia,Crawley6009,Australia

Abstract
The parameter values and assumptions of any model are subject to change and
error. Sensitivity analysis (SA), broadly defined, is the investigation of these
potentialchangesanderrorsandtheirimpactsonconclusionstobedrawnfromthe
model. There is a very large literature on procedures and techniques for SA. This
paperisaselectivereviewandoverviewoftheoreticalandmethodologicalissuesin
SA. There are many possible uses of SA, described here within the categories of
decision support, communication, increased understanding or quantification of the
system,andmodeldevelopment.Thepaperfocusessomewhatondecisionsupport.
It is argued that even the simplest approaches to SA can be theoretically
respectable in decision support if they are done well. Many different approaches to
SAaredescribed,varyingintheexperimentaldesignusedandinthewayresultsare
processed. Possible overall strategies for conducting SA are suggested. It is
proposed that when using SA for decision support, it can be very helpful to attempt
toidentifywhichofthefollowingformsofrecommendationisthebestwaytosumup
the implications of the model: (a) do X, (b) do either X or Y depending on the
circumstances, (c) do either X or Y, whichever you like, or (d) if in doubt, do X. A
systemforreportinganddiscussingSAresultsisrecommended.

1.Introduction
Theparametervaluesandassumptionsofanymodelaresubjecttochangeanderror.Sensitivity
analysis(SA),broadlydefined,istheinvestigationofthesepotentialchangesanderrorsandtheir
impactsonconclusionstobedrawnfromthemodel(e.g.Baird,1989).SAcanbeeasytodo,easy
to understand, and easy to communicate. It is possibly the most useful and most widely used
technique available to modellers who wish to support decision makers. The importance and
usefulnessofSAiswidelyrecognised:

"A methodology for conducting a [sensitivity] analysis ... is a well established

requirementofanyscientificdiscipline.A sensitivityandstabilityanalysisshouldbe
an integral part of any solution methodology. The status of a solution cannot be
understood without such information. This has been well recognised since the
inceptionofscientificinquiryandhasbeenexplicitlyaddressedfromthebeginningof
mathematics".(Fiacco,1983,p3).

ThereisaverylargeanddiverseliteratureonSA,includinganumberofreviews(e.g.Clemsonet
al., 1995 Eschenbach and Gimpel, 1990 Hamby, 1994 Lomas and Eppel, 1992 Rios Insua,
1990 SobieszczanskiSobieski, 1990 Tzafestas et al., 1988). However, the existing literature is
limitedinanumberofrespects.Mostofwhathasbeenwrittenaboutsensitivityanalysishastaken
averynarrowviewofwhatitisandwhatitcanbeusefulfor.Alargeproportionoftheliteratureis
highly mathematical and rather theoretical in nature. Even those papers with a focus on applied
methodology have tended to concentrate on systems and procedures which are relatively time
consuming and complex to implement. There has been almost no discussion of procedures and
methodologicalissuesforsimpleapproachestosensitivityanalysis.(EschenbachandMcKeague,
1989, is a rare exception). This is remarkable, considering the usefulness and extremely wide
usageofsimpleapproaches.

Myaiminthispaperis,inpart,tofillthisgap.Manytechniquesandprocedureswillbediscussed,
ranging from simple to complex. While it is acknowledged that some of the complex procedures
which have been proposed are potentially of high value, the primary objective of this paper is to
provideguidanceandadvicetoimprovetherigourandvalueofrelativelysimpleapproaches.Itwill
be argued that even the simplest approaches to SA can be theoretically respectable in decision
support. The paper is relevant to both optimisation and simulation models used for decision
support,althoughthereisagreateremphasisonoptimisationmodelsinthediscussion.

2.UsesofSensitivityAnalysis
http://dpannell.fnas.uwa.edu.au/dpap971f.htm 1/16
3/15/2017 Sensitivityanalysis:strategies,methods,concepts,examples

Thereisaverywiderangeofusestowhichsensitivityanalysisisput.Anincompletelistisgiven
in Table 1. The uses are grouped into four main categories: decision making or development of
recommendations for decision makers, communication, increased understanding or quantification
ofthesystem,andmodeldevelopment.Whilealltheseusesarepotentiallyimportant,theprimary
focusofthispaperisonmakingdecisionsorrecommendations.

Table1.Usesofsensitivityanalysis

1. DecisionMakingorDevelopmentofRecommendationsforDecisionMakers
1.1 Testingtherobustnessofanoptimalsolution.
Identifyingcriticalvalues,thresholdsorbreakevenvalueswheretheoptimal
1.2
strategychanges.
1.3 Identifyingsensitiveorimportantvariables.
1.4 Investigatingsuboptimalsolutions.
1.5 Developingflexiblerecommendationswhichdependoncircumstances.
1.6 Comparingthevaluesofsimpleandcomplexdecisionstrategies.
1.7 Assessingthe"riskiness"ofastrategyorscenario.

2. Communication
Makingrecommendationsmorecredible,understandable,compellingor
2.1
persuasive.
2.2 Allowingdecisionmakerstoselectassumptions.
2.3 Conveyinglackofcommitmenttoanysinglestrategy.

3. IncreasedUnderstandingorQuantificationoftheSystem
3.1 Estimatingrelationshipsbetweeninputandoutputvariables.
3.2 Understandingrelationshipsbetweeninputandoutputvariables.
3.3 Developinghypothesesfortesting

4. ModelDevelopment
4.1 Testingthemodelforvalidityoraccuracy.
4.2 Searchingforerrorsinthemodel.
4.3 Simplifyingthemodel.
4.4 Calibratingthemodel.
4.5 Copingwithpoorormissingdata.
4.6 Prioritisingacquisitionofinformation.

In all models, parameters are moreorless uncertain. The modeller is likely to be unsure of their
currentvaluesandtobeevenmoreuncertainabouttheirfuturevalues.Thisappliestothingssuch
as prices, costs, productivity, and technology. Uncertainty is one of the primary reasons why
sensitivity analysis is helpful in making decisions or recommendations. If parameters are
uncertain,sensitivityanalysiscangiveinformationsuchas:

a. how robust the optimal solution is in the face of different parameter values (use
1.1fromTable1),

b.underwhatcircumstancestheoptimalsolutionwouldchange(uses1.2,1.3,1.5),

c.howtheoptimalsolutionchangesindifferentcircumstances(use3.1),

d. how much worse off would the decision makers be if they ignored the changed
circumstances and stayed with the original optimal strategy or some other
strategy(uses1.4,1.6),

This information is extremely valuable in making a decision or recommendation. If the optimal

strategyisrobust(insensitivetochangesinparameters),thisallowsconfidenceinimplementingor
http://dpannell.fnas.uwa.edu.au/dpap971f.htm 2/16
3/15/2017 Sensitivityanalysis:strategies,methods,concepts,examples
recommendingit.Ontheotherhandifitisnotrobust,sensitivityanalysiscanbeusedtoindicate
how important it is to make the changes to management suggested by the changing optimal
solution. Perhaps the basecase solution is only slightly suboptimal in the plausible range of
circumstances, so that it is reasonable to adopt it anyway. Even if the levels of variables in the
optimal solution are changed dramatically by a higher or lower parameter value, one should
examine the difference in profit (or another relevant objective) between these solutions and the
basecase solution. If the objective is hardly affected by these changes in management, a
decision maker may be willing to bear the small cost of not altering the strategy for the sake of
simplicity.

If the basecase solution is not always acceptable, maybe there is another strategy which is not
optimalintheoriginalmodelbutwhichperformswellacrosstherelevantrangeofcircumstances.If
there is no single strategy which performs well in all circumstances, SA identifies different
strategies for different circumstances and the circumstances (the sets of parameter values) in
whichthestrategyshouldbechanged.

Evenifthereisnouncertaintyabouttheparametervalues,itmaybecompletelycertainthatthey
willchangeinparticularwaysindifferenttimesorplaces.Inasimilarwaytothatoutlinedabove,
sensitivityanalysiscanbeusedtotestwhetherasimpledecisionstrategyisadequateorwhether
acomplexconditionalstrategyisworththetrouble.

SA can be used to assess the "riskiness" of a strategy or scenario (use 1.7). By observing the
range of objective function values for the two strategies in different circumstances, the extent of
the difference in riskiness can be estimated and subjectively factored into the decision. It is also
possibletoexplicitlyrepresentthetradeoffbetweenriskandbenefitwithinthemodel.

3. Theoretical Framework for Using Sensitivity Analysis for Decision

Making
In this discussion, a decision variable is a variable over which the decision maker has control
and wishes to select a level, whereas a strategy refers to a set of values for all the decision
variablesofamodel.Anoptimalstrategyisthestrategywhichisbestfromthepointofviewofthe
decision maker it optimises the value of the decision maker's objective function (e.g. profit,
social welfare, expected utility, environmental outcome). Suppose that the modeller knows the
objectiveofthedecisionmakerwhowillusetheinformationgeneratedbythemodel.Themodeller
willbeabletoformsubjectivebeliefs(internalbeliefs,hunchesorguesses)abouttheperformance
ofdifferentstrategiesfromtheperspectiveofthedecisionmaker.Themodeller'ssubjectivebeliefs
areinfluencedbythemodelbutalsobyotherfactorsthesebeliefsmayormaynotbeclosetothe
objectivetruth.

SAisaprocessofcreatingnewinformationaboutalternativestrategies.Itallows the modeller to

improvethequalityoftheirsubjectivebeliefsaboutthemeritsofdifferentstrategies.

Conceptually, the process of conducting a SA to choose an optimal strategy can proceed as

follows. Following an initial run with a "basecase" model which incorporates "bestbet" values of
parameters, a belief about the optimal strategy can be formed. This belief is based on the
modeller's perceptions of the probability distributions of profit (or another measure of benefit or
welfare)forthepreferredstrategyandotherstrategies.Theinitialoptimalstrategyistheonewhich
maximisestheexpectedvalueoftheobjectivefunction(i.e.itsweightedaveragevalue),giventhe
modeller'sinitialbeliefsaboutprobabilitydistributionsofprofitfordifferentstrategies.Theseinitial
beliefs could also be used to make statements about the modeller's level of confidence that the
initialstrategyisoptimal.

Followingasensitivityanalysisbasedononeormoreofthetechniquesoutlinedlater,themodeller
reviseshisorhersubjectivebeliefsabouttheprofitabilityof different strategies. (More rigorously,
the modeller's subjective perceptions about the probability distributions of profit for each strategy
aremodified.)Dependingonhowtheperceptionschange,theoptimalstrategymayormaynotbe
altered. The modified distributions are likely to be less uncertain (although not necessarily less
risky), due to the information obtained from the SA, so the modeller can make improved
statementsabouthisorherconfidenceintheoptimalstrategy.

This view of the SA process is highly consistent with "Bayesian decision theory", a powerful
approach for making the best possible use of information for decision making under risk and
uncertainty. Even if the modeller does not literally use a Bayesian approach, merely
conceptualising the process in the way described above will probably improve the rigour and

http://dpannell.fnas.uwa.edu.au/dpap971f.htm 3/16
3/15/2017 Sensitivityanalysis:strategies,methods,concepts,examples

consistency of the SA. If the modeller is thinking with rigour and consistency, it may be that an
unstructured "what if?" approach to the SA is adequate for some studies. On the other hand, the
modeller may be encouraged to adopt a structured, explicitly probabilistic approach to SA based
onBayesiandecisiontheory.

A conceptual difficulty with this theoretical framework when using an optimisation model is
outlinedinanappendix.

4.ApproachestoSensitivityAnalysis
Inprinciple,sensitivityanalysisisasimpleidea:changethemodelandobserveitsbehaviour.In
practice there are many different possible ways to go about changing and observing the model.
Thesectioncoverswhattovary,whattoobserveandtheexperimentaldesignoftheSA.

4.1Whattovary

Onemightchoosetovaryanyorallofthefollowing:

a.thecontributionofanactivitytotheobjective,

b.theobjective(e.g.minimiseriskoffailureinsteadofmaximisingprofit),

c.aconstraintlimit(e.g.themaximumavailabilityofaresource),

d. the number of constraints (e.g. add or remove a constraint designed to express

personalpreferencesofthedecisionmakerfororagainstaparticularactivity),

e.thenumberofactivities(e.g.addorremoveanactivity),or

f.technicalparameters.

Commonly, the approach is to vary the value of a numerical parameter through several levels. In
othercasesthereisuncertaintyaboutasituationwithonlytwopossibleoutcomeseitheracertain
situationwilloccuroritwillnot.Examplesinclude:

Whatifthegovernmentlegislatestobanaparticulartechnologyforenvironmental
reasons?

In a shortest route problem, what if a new freeway were built between two major
centres?

Whatifanewinputoringredientwithuniquepropertiesbecomesavailable?

Often this type of question requires some structural changes to the model. Once these changes
aremade,outputfromtherevisedmodelcanbecomparedwiththeoriginalsolution,ortherevised
model can be used in a sensitivity analysis of uncertain parameters to investigate wider
implicationsofthechange.

4.2Whattoobserve

Whicheveritemsthemodellerchoosestovary,therearemanydifferentaspectsofamodeloutput
towhichattentionmightbepaid:

a.thevalueoftheobjectivefunctionfortheoptimalstrategy,

b.thevalueoftheobjectivefunctionforsuboptimalstrategies(e.g.strategies which
areoptimalforotherscenarios,orparticular strategies suggested by the decision
maker),

c. the difference in objective function values between two strategies (e.g. between
theoptimalstrategyandaparticularstrategysuggestedbythedecisionmaker),

d.thevaluesofdecisionvariables,

e.inanoptimisationmodel,thevaluesofshadowcosts,constraintslacksorshadow
prices,or

http://dpannell.fnas.uwa.edu.au/dpap971f.htm 4/16
3/15/2017 Sensitivityanalysis:strategies,methods,concepts,examples

f.therankingsofdecisionvariables,shadowcosts,etc.

4.3Experimentaldesign

The experimental design is the combinations of parameters which will be varied and the levels at
whichtheywillbeset.Themodellermustdecidewhethertovaryparametersoneatatime,leaving
all others at standard or base values, or whether to examine combinations of changes. An
important issue in this decision is the relative likelihood of combinations of changes. If two
parameters tend to be positively correlated (e.g. the prices of two similar outputs) the possibility
thattheywillbothtakeonrelativelyhighvaluesatthesametimeisworthconsidering.Conversely
if two parameters are negatively correlated, the modeller should examine high values of one in
combination with low values of the other. If there is no systematic relationship between
parameters,itmaybereasonabletoignorethelowriskthattheywillbothdiffersubstantiallyfrom
theirbasevaluesatthesametime,especiallyiftheyarenotexpectedtovarywidely.

In selecting the parameter levels which will be used in the sensitivity analysis, a common and
normally adequate approach is to specify values in advance, usually with equal sized intervals
between the levels (e.g. Nordblom et al., 1994). The levels selected for each parameter should
encompass the range of possible outcomes for that variable, or at least the "reasonably likely"
range.Whatconstitutes"reasonablylikely"isasubjectivechoiceofthemodeller,butonepossible
approach is to select the maximum and minimum levels such that the probability of an actual
valuebeingoutsidetheselectedrangeis10percent.

Ifcombinationsofchangestotwoormoreparametersarebeinganalysed,apotentialapproachis
to use a "complete factorial" experimental design, in which the model is solved for all possible
combinationsoftheparameters.Whilethisprovidesawealthofinformation,ifthereareanumber
of parameters to analyse, the number of model solutions which must be obtained can be
enormous.Toconductacompletefactorialsensitivityanalysisforeightparameterseachwithfive
levels would require 390,625 solutions. If these take one minute each to process, the task would
take nine months, after which the volume of output created would be too large to be used
effectively. In practice one must compromise by reducing the number of variables and/or the
number of levels which are included in the complete factorial. Preliminary sensitivity analyses on
individualparametersarehelpfulindecidingwhicharethemostimportantparametersforinclusion
inacompletefactorialexperiment.(Seelatercommentson"screening".)

Alternatively one may reduce the number of model solutions required by adopting an incomplete
design with only a subset of the possible combinations included. Possibilities include central
composite designs (e.g. Hall and Menz, 1985), Taguchi methods (e.g. Clemson et al., 1995) or
some system of random sampling or "Monte Carlo" analysis (e.g. Clemson et al., 1995 Uyeno,
1992).

5.ProcessingofSensitivityAnalysisResults
Agreatdealofinformationcanbegeneratedinsensitivityanalysis,somuchsothatthereisarisk
ofthevolumeofdataobscuringtheimportantissues(EschenbachandMcKeague,1989).Forthis
reason, the modeller must process and/or summarise the information to allow decision makers to
identifythekeyissues.Thefollowingsubsectionscovervariouspossiblemethodsforprocessing
results of a sensitivity analysis, ranging from very simple to very complex. For many of the
methods of analysis, I suggest possible layouts for graphs and tables. There are many other
layoutswhichmaybemoresuitablethantheseforparticularpurposes.Anumberofexamplesare
drawnfrommyresearchinagriculturaleconomics.

5.1Summariesofactivitylevelsorobjectivefunctionvalues:onedimension

The simplest approach to analysis of SA results is to present summaries of activity levels or

objective function values for different parameter values. It may be unnecessary to conduct any
furtheranalysisoftheresults.

A simple example of such a summary is presented in Figure 1. This example (like several which
follow)isfromMIDAS,alinearprogrammingmodelwhichselectsoptimalcombinationsoffarming
enterprises for representative farms in a region of Western Australia (Morrison et al., 1986
KingwellandPannell,1987).Figure1showshowtheoptimalareaofwheatvariesasanumberof
parametersarevariedeithersideoftheirstandardvalues.Eachoftheparametersinthisexample
is varied up or down by amounts reflecting their realistic possible ranges. The format in Figure 1
allows results from several parameters to be presented on a single graph. This allows easy
comparisonoftherelativeimpactsoftheseparameterswhenvariedovertheirrealisticranges,and
http://dpannell.fnas.uwa.edu.au/dpap971f.htm 5/16
3/15/2017 Sensitivityanalysis:strategies,methods,concepts,examples
these ranges are communicated by the horizontal span of the lines. In this example one can see
that wheat yields have the biggest impact on the optimal area of wheat. Eschenbach and
McKeague (1989) refer to this type of graph as a "spider diagram", for obvious reasons. Another
variation is to also plot the vertical axis in percentage terms so that the graph illustrates
"elasticities"(seeSubsection5.3).

Figure1.Graphingchangesinmultipleparametersforasingleoutputvariable.

Spiderdiagramslikethesecanalsobeconstructedwiththeobjectivefunctionvalueratherthanan
activity level as the dependent variable, allowing the decision maker to assess the sensitivity of
the objective function value to parameter changes. For example if the objective is to maximise
profit, this type of diagram reveals whether any parameter changes would result in a negative
profit.

Apotentialproblemwiththeuseofpercentagechangesinspiderdiagramsisthatiftheparameter
issmall(e.g.variationiscentredaroundzero),percentagechangesmaybelargerelativetothose
for other variables. In fact, if the initial parameter value is zero, percentage changes to the
parameterarenotdefined.Fortheseparameters,itmaybeappropriatetouseanabsolutechange.

Spiderdiagramsareusuallypracticalonlyfordisplayingthelevelsofasingleactivity.Wherethere
are several important variables to display, one normally needs to limit results to changes in a
single parameter. Figure 2 is an example from MIDAS showing production of wheat grain, lupin
grain, pea grain and wool as a function of wheat price. Because of the different scales of
production, wool is shown on the right hand axis. This graph reveals that the main effect of
increasing wheat price is to increase wheat production at the expense of wool. There are also
smallerchangesintheproductionoflupingrainandpeagrain.

Figure2.Graphingmultipleoutputvariablesforchangesinasingleparameter.

http://dpannell.fnas.uwa.edu.au/dpap971f.htm 6/16
3/15/2017 Sensitivityanalysis:strategies,methods,concepts,examples

A different way of summarising the same model results is to show the allocation of a particular
input or resource to the different possible outputs. The way these allocations vary can be
effectivelydisplayedbystackingthelinesorbars,asshowninFigure3.Thisshowstheallocation
oflandtoproductionofeachofthefourproducts,withtheallocationsmirroringthetrendsinFigure
2.

Figure 3. Graphing the allocation of a resource among alternative uses for changes in a single
parameter.

http://dpannell.fnas.uwa.edu.au/dpap971f.htm 7/16
3/15/2017 Sensitivityanalysis:strategies,methods,concepts,examples

5.2Summariesofactivitylevelsorobjectivefunctionvalues:higherdimensions

InFigure1,becauseallparametersbutonewerewereheldconstantforeachlineonthegraph,it
waspossibletodisplayresultsforseveralparametersonthesamegraph.Indisplayingtheresults
of changing parameters simultaneously, it is difficult to handle more than two parameters in a
graphwithoutitbecomingcomplexanddifficulttofollow.Figure4showsanexampleofamethod
fordisplayingresultsfromsensitivityanalysesontwoparameters.Thisfigureshowstheimpacts
ofchangingwheatpriceandwoolpriceontheoptimalareaofwheatselectedbyMIDAS.Thereare
manyotherformatsforthreedimensionalgraphswhichcanbeusedforthispurpose.

Figure4.Graphingcombinationsofparameterchanges.

http://dpannell.fnas.uwa.edu.au/dpap971f.htm 8/16
3/15/2017 Sensitivityanalysis:strategies,methods,concepts,examples

Resultsformorethantwoparametersrequireaseriesofgraphsoratable.Wellstructuredtables
are probably the better option. Another approach is to develop an interactive database of model
results,allowingdecisionmakerstoselecttheparametervaluesanddisplayingthecorresponding
optimalsolution.Thistypeofdatabaseactsasasimplified(andmuchquicker)versionofthefull
model.

A final possible approach to the analysis of multidimensional sensitivity analysis is to use

statisticalregressiontechniquestofitasmoothsurfacetotheresults(Kleijnen,1992,1995b).This
approach provides an equation which approximates the functional relationship between the
parameter values and the dependent variable (e.g. the activity level or objective function value).
Such an equation will be smoother than the step functions often produced by mathematical
programming models and this may be useful for producing graphs or for conducting some of the
analysesoutlinedbelow.

5.3Slopesandelasticities

The rate of change (the slope) of an activity level or of the objective function with respect to
changesinaparameterisanevenbriefersummaryoftheissuethanthegraphsshownsofar.An
issue is the need to compare slopes for different parameters. The units of measure of different
parameters are not necessarily comparable, so neither are absolute slopes with respect to
changesindifferentparameters.Onecanoftenovercomethisproblembycalculating"elasticities",
which are measures of the percentage change in a dependent variable (e.g. an activity level)
dividedbythepercentagechangeinanindependentvariable(e.g.aparameter).

(1)e=%Y/%X

or

(2)e=Y/X.X/Y

A comparison of elasticities of an activity level with respect to different parameters provides a

goodindicationoftheparameterstowhichtheactivityismostsensitive.Table2isanexampleof
such a comparison for MIDAS. The elasticities have been calculated assuming base values for
parametersotherthantheoneinquestion.Resultshavebeensmoothedusingregressionanalysis
andelasticitieshavebeencalculatedfromthefittedsmoothcurves.

http://dpannell.fnas.uwa.edu.au/dpap971f.htm 9/16
3/15/2017 Sensitivityanalysis:strategies,methods,concepts,examples

Table2.Elasticitiesofoptimalwheatareawithrespecttochangesinvariousparameters

Parameter Elasticityofoptimalwheatarea
Wheatprice 1.5
Wheatyield 1.4
Woolprice 0.5
Lupinprice 0.3
Machinerysize 0.0

Analternativetotheuseofelasticitiesistostandardiseparametersasfollows:

(3)Z=(Xb)/a

wherebisthebasevalueforXandaistherange(i.e.,XmaxXmin)(Kleijnen,1995a).

5.4Sensitivityindices

A sensitivity index is a number calculated by a defined procedure which gives information about
the relative sensitivity of results to different parameters of the model. A simple example of a
sensitivity index is the elasticity of a variable with respect to a parameter (Subsection 5.3). The
higher the elasticity, the higher the sensitivity of results to changes in that parameter. Hamby
(1994)outlined14possiblesensitivityindicesforcaseswhereonlyasingleoutputvariableistobe
evaluated, including the "importance index", the "relative deviation" index, the "partial rank
correlation coefficient", the Smirnov test, the Cramervon Mises test, and a number of others.
These are not outlined in detail here because many of them are complex and timeconsuming to
calculate. Furthermore, Hamby (1995) conducted a detailed comparison of the performance of
each of the indices relative to a composite index based on ten of them. None of the complex
indicestestedperformedaswellasimpleindexproposedbyHoffmanandGardner(1983):

(4)SI=(DmaxDmin)/Dmax

where Dmax is the output result when the parameter in question is set at its maximum value and
Dministheresultfortheminimumparametervalue.Incaseswherecomparisonsbetweendifferent
models are not important, the following even simpler sensitivity index can be perfectly adequate
(andperhapsevenpreferable).

(5)SI=(DmaxDmin)

Alexander(1989)suggestedanumberofcomplexindicesforuseinsituationswherethemodeller
wishes to assess the sensitivity of several output variables simultaneously. For example, for
cases where the result of interest is a ranking of several variables, Alexander provides an index
whichindicatesthesensitivityoftherankingtochangesinaparameter.

5.5Breakevenvalues

Considerthequestion:"IfparameterXweretochangefromitscurrentvalue,byhowmuchwould
ithavetochangeinorderfortheoptimalsolutiontochangeinaparticularway?"Thisbreakeven
approach addresses the issue of uncertainty about parameter values in a way which is often
particularly helpful to decision makers. It helps in the assessment of whether the critical value of
the variable falls within the range of values considered reasonable for the variable. If not, the
decision maker can be advised (for the purposes of planning) to disregard the possibility of the
variable taking a different value. If the breakeven value is within the realistic range, this
information can be used to justify collection of additional information to help predict the actual
valueoftheparameter.

Table3showsanexamplefromMIDAS.Inthestandardversionofthismodel,theoptimaluseof
land of a particular type (soil type 1) is to grow pasture for grazing by sheep. The aim is to
determine the circumstances in which cropping would be as good as or better than pasture. The
table shows breakeven percentage changes in various parameters (changes needed for the
profitability of cropping on soil type 1 to equal that for pasture). By judging whether parameter
http://dpannell.fnas.uwa.edu.au/dpap971f.htm 10/16
3/15/2017 Sensitivityanalysis:strategies,methods,concepts,examples

changes of at least this magnitude are ever likely to occur, the modeller can judge whether
croppingiseverlikelytoberecommendedonthissoiltype.

Table 3. Breakeven changes in parameter values for cropping to be as profitable as pasture

productiononsoiltype1

Parameter Breakevenparameterchange
Wheatprice +50%
Wheatyieldonsoiltype1 +40%
Woolprice 80%
Pastureyieldonsoiltype1 70%
Lupinprice +130%
Lupinyieldonsoiltype1 +120%

5.6Comparingconstrainedandunconstrainedsolutions

Theapproachesdiscussedsofararebasedonassessingthesensitivityofthemodeltochanges
inparameters.Adifferentapproachistoaddconstraintstothemodelsothatitisforcedtoadopt
otherinterestingstrategies.Itisoftenveryvaluabletoknowhowotherstrategiesperformrelative
to the optimum. Figure 5 shows an example, where the MIDAS model has been constrained to
plant crops on various percentages of the farm area. Such a graph is valuable if the decision
makerwishestoconsiderotherstrategieswhichachieveobjectivesotherthanthatrepresentedin
themodel.Figure5showshowmuchprofitmustbesacrificedifthefarmerwishestodeviatefrom
theoptimalcroppingareaof60percent.

Figure5.Comparingoptimalwithsuboptimalstrategies.

A useful way of indicating the flexibility available to the decision maker is to report the set of
strategies with objective function values within a certain distance of the optimum. For example,
http://dpannell.fnas.uwa.edu.au/dpap971f.htm 11/16
3/15/2017 Sensitivityanalysis:strategies,methods,concepts,examples

any area of crop between 40 and 70 percent of the farm is within $5000 of the maximum profit.
This is an example of one approach to testing the "robustness" of a solution (one of the uses of
SAlistedinTable1).

Sometimes it is useful to constrain the model to exclude an option in order to calculate the total
contributionofthisoptiontotheobjective,andtoidentifythebeststrategywhichdoesnotinclude
it. Table 4 shows a summary of the MIDAS solutions which include and exclude the option of
growinglupinsonthefarm.Itisapparentthattheinclusionoflupinsincreasesprofitsbyaround66
percent.

Table4.Profitandoptimalrotationswithandwithoutlupins

Lupins Lupins
included excluded
Wholefarmprofit($) 40,870 24,553
Rotationselected*
Soiltype1 PPPP PPPP
Soiltype2 CL PPPC
Soiltype3 CCL CCCC
Soiltype4 CCL CCCC
Soiltype5 CCF CCF
Soiltype6 PPPP PPPP
Soiltype7 CCF CCF

*C=cerealcropP=pastureL=lupinsF=fieldpeas

5.7Usingprobabilities

A common characteristic of the methods of analysis presented above is that they do not require
the modeller to explicitly specify probabilities of different situations. Sensitivity analysis can be
extremely effective and useful even without taking this extra step to a more formal and complex
analysis of results. In excluding probabilities from the analysis, the modeller is relying on the
decisionmakertogiveappropriateweighttoeachscenario.Ontheotherhand,ananalysisusing
probabilities may be unnecessarily difficult and time consuming to conduct, and is likely to be
more difficult to explain to the decision maker. The potential simplicity of sensitivity analysis is
oneofitsattractions:ananalysiswhichisnotunderstoodisunlikelytobebelieved.Dependingon
the importance of the issue and the attitudes and knowledge of the decision maker, the best
approachtosensitivityanalysismightnotinvolveformalandexplicituseofprobabilities.Evenifa
probabilistic sensitivity analysis is to be conducted, a simpler preliminary analysis may be useful
inplanningthefinalanalysis.

6.OverallStrategiesforSensitivityAnalysis
Thetechniquesoutlinedaboveareapowerfulsetoftoolsforassistingadecisionmaker.However
the modeller needs to avoid conducting sensitivity analysis in an aimless or mechanical fashion.
The approach should be adjusted to suit the decision problem. As the analysis proceeds, the
resultsobtainedmayleadtofurthermodelrunstotestideasoranswerquestionswhicharise.Ina
thoroughsensitivityanalysis,anumberoftheapproachessuggestedintheprevioussectionmight
beused.

Withinthesebroadguidelines,thereareverymanyoverallstrategiesforsensitivityanalysiswhich
might be adopted. Here are three systematic suggestions of overall strategies which are likely to
beeffectiveincaseswheretheanalysisisusedtohelpmakeadecisionorrecommendationabout
theoptimalstrategy.

StrategyA(themostcomprehensive)isasfollows.

http://dpannell.fnas.uwa.edu.au/dpap971f.htm 12/16
3/15/2017 Sensitivityanalysis:strategies,methods,concepts,examples

1. Select the parameters to be varied. Identify a range for each parameter which realistically
reflectsitspossiblerange. For example, use maximum and minimum values, or an 80 percent
confidence interval, but not a uniform 10 or 20 percent either side of the expected value. Also
identifyotherpossiblediscretescenariosrequiringchangestothemodelstructureorformulation
(e.g.changesintheobjectivetobeoptimised,inclusionofadditionalconstraints).

2.Conductsensitivity analyses for each parameter individually, using two parameter values (high
and low or maximum and minimum). Conduct sensitivity analysis for each discrete scenario
individually.

3. Identify parameters and discrete scenarios to which the key decision variables are relatively
unresponsive,usingoneofthesensitivityindicespresentedinSubsection5.4(e.g.,equation5).

4. Exclude unresponsive parameters and scenarios from further analysis. For the remaining
parameters, consider whether they are likely to have high positive, high negative or low
correlationwitheachother.Ifitisintendedtouseprobability distributions for random sampling
of scenarios or for summarisation of results, estimate the distribution for each parameter and,
for cases of high correlation, estimate the joint probability distribution. Possibly also estimate
probabilitiesforthediscretescenariosselectedinstep1.

5. Design and conduct a modelling experiment which allows appropriately for combinations of
parameter changes, paying particular attention to the cases of high correlation between
parameters.PossiblyuseLatinhypercubesampling(Clemsonetal.,1995)or,ifthenumberof
combinations is manageable, a complete factorial design. Repeat this for each of the discrete
scenariosindividually,orifpractical,forallcombinationsofthediscretescenarios.

6. Summarise results. For each key decision variable, calculate the values of a sensitivity index
forallparametersanddiscrete scenarios, and rank them by absolute value. These results can
be reported directly or used to select which parameters will be examined in graphs and tables
(e.g. spider diagrams). This approach helps to prioritise the presentation of results which is
essentialtoavoidanoverloadofgraphsandtables.Italsoallowsthedecision maker to focus
onimportantparametersandrelationships.Calculatebreakevenparametervaluesforparticular
circumstancesofinterest.

7. On the basis of results so far, identify a tentative bestbet strategy and several others of
interest.Theotherstrategiesmightbechosenbecausetheycontributetoobjectivesother than
thoserepresentedinthemodel,orbecausetheyareofpersonalinteresttothedecisionmaker.
French (1992) suggested focusing on "adjacent potentially optimal" alternative solutions,
meaningstrategieswhichareclosetothebasecaseoptimumandwhichwouldbecomeoptimal
ifparameterschangedsufficiently.Itisnotnecessarytolimittheanalysistosuchanarrowset
of strategies, although one should be mindful of the number of solutions required in the next
step.

8.Repeattheexperiment(step5)withthemodelconstrainedtoeachofthe strategies (from step

7). Summarise these results. Identify scenarios (if any) where each strategy is optimal.
Calculate the cost of each strategy relative to the bestbet. Possibly repeat this with another
strategyasthebestbet.Atthisstagethemodellermaywishtouse probability distributions to
makeprobabilisticstatementsabouttheresults.

9.Attempttodrawconclusions.Itcanbehelpfultotheanalyststofocustheirthinkingbytryingto
couchconclusionsintermssimilartooneofthefollowingexamples.

a.Theoptimal strategy is X in almost any plausible scenario, so X is a safe best

betstrategy.

b.InsomescenariostheoptimalstrategyisX,whereasintheseotherscenariosthe
optimalstrategyisY.Ifyoucanpredictoridentifythescenario,itisimportant to
dotherightstrategyintherightscenario.

c.InsomescenariostheoptimalstrategyisX,whereasintheseotherscenariosthe
optimalstrategyisY.However,thecostofdoingthewrongstrategyisverylow,
so it is not very important to worry about doing the right strategy in the right
scenario.

d.InsomescenariostheoptimalstrategyisX,whereasintheseotherscenariosthe
optimal strategy is Y. The cost of doing the wrong strategy when the decision

http://dpannell.fnas.uwa.edu.au/dpap971f.htm 13/16
3/15/2017 Sensitivityanalysis:strategies,methods,concepts,examples

maker should be doing Y is low, but the cost of doing the wrong strategy when
thedecisionmakershouldbedoingXishigh,soifitisnotpossibletopredictor
identifythescenario,Xisasafebestbetstrategy.

These conclusions correspond to the following recommendations: (a) do X, (b) do either X or Y

dependingonspecificcircumstances,(c)doeitherXorY, it doesnt matter which, (d) if in doubt,
doX.Inadditionthereisaconversesetofconclusionsaboutwhichstrategiesareneverlikelyto
beoptimal:(e)neverdoZ,(f)incertaincircumstancesdonotdoZ,(g)ifindoubt,donotdoZ.Try
to identify which of the categories (a) to (d) the problem falls into and whether it is possible to
specifyanystrategieslikeZincategories(e)to(g).

StrategyB(slightlylesscomprehensive)includesallofthestepsofStrategyAexcept7and8.

StrategyC(thesimpleststrategywhichisstillsystematicanduseful)includesonlysteps1,2,3,
6and9.

7.ReportingResultsofSensitivityAnalysis
It is common for written reports of sensitivity analyses in published papers to address only a
subsetoftheissuesonwhichtheSAcanprovideinformation.Ofcourseonemustbeselectivein
thereportinganddiscussionofresults,buttoooftendiscussionsofsensitivityanalysesdriftaway
fromthecentralissuebeinginvestigatedontointerestingbutrelativelyunimportantdetails.Inother
cases, SA results are presented without sufficient discussion of their consequences and
implications for the central issue. To avoid these traps, the following report structure is
recommendedasastandardminimum.

a. From the basecase model, or other information, what is the initial optimal
recommendationwhichistoformthestandardforcomparisonsintheSA?

b.Whichparametersmostaffecttheoptimalrecommendation?A tableofvaluesfor
a sensitivity index ranked according to their absolute value is recommended. If
appropriate, what are the breakeven levels of parameters for changes in the
recommendation?

c.Howdoestheoptimalrecommendationchangeiftheimportantparameters(from
(b))change?

d. What are the consequences of not following the optimal recommendation? For
example,howmuchlessprofitableareotherrecommendations?

e.Overall,whatlevelofconfidencecantherebethattherecommendationis in fact
optimal?

In addressing these issues, the space devoted to each need not necessarily be large, and the
relativeimportanceofeachwilldependontheparticularstudy.Point(a)isparticularlyimportantas
itensuresthatthediscussionoftheSAwillbewellfocussedandrelevant.Therecommendationto
state the "level of confidence" is not intended to provoke a formal probabilistic or statistical
statement,butatleastsomerelativelyinformalandsubjectivestatementofconfidenceshouldbe
made.Iftheconclusionissubjective,sayso.

Avoidthetrapofoverloadingthereportwiththeresultsofcategory(c).Asnotedabove,ahelpful
strategyinthisregardistodemonstratethatcertainparametershavelittleimpactontheimportant
decisionvariables,andthentoavoidreportingfurtherresultsfortheseparameters.

8.ConcludingComments
There is clearly much more to the use of a decision support model than finding a single optimal
solution. That solution should be viewed as the starting point for a wide ranging set of sensitivity
analysestoimprovethedecisionmaker'sknowledgeandunderstandingofthesystem'sbehaviour.

Evenwithoutundertakingtherelativelycomplexprocedureswhichexplicitlyinvolveprobabilitiesin
the sampling of scenarios or interpretation of results, sensitivity analysis is a powerful and
illuminating methodology. The simple approach to sensitivity analysis is easy to do, easy to
understand, easy to communicate, and applicable with any model. As a decision aid it is often
adequate despite its imperfections. Given its ease and transparency, the simple approach to SA
mayevenbetheabsolutebestmethodforthepurposeofpracticaldecisionmaking.
http://dpannell.fnas.uwa.edu.au/dpap971f.htm 14/16
3/15/2017 Sensitivityanalysis:strategies,methods,concepts,examples

Acknowledgments
Thepaperhasbeenimprovedbydetailedreviewercomments,forwhichIamgrateful.Ialsothank
JackKleijnenformakingmeawareofasetofliteratureIhadnotdiscovered.Muchoftheworkfor
this paper was done while I was on study leave at the Department of Agricultural Economics,
University of Saskatchewan, Canada and I thank the members of that Department for their
hospitality. Members of seminar audiences at the University of Western Australia and the
UniversityofMelbournealsomadehelpfulsuggestions.

References
Alexander,E.R.(1989).Sensitivityanalysisincomplexdecisionmodels,JournaloftheAmerican
PlanningAssociation55:323333.
Andres, T.H. (1996). Sampling methods and sensitivity analysis for large parameter sets, Journal
ofStatisticalComputationandSimulation(inpress).
Baird, B.F. (1989). Managerial Decisions Under Uncertainty, An Introduction to the Analysis of
DecisionMaking,Wiley,NewYork.
Bettonvil,B.andKleijnen,J.P.C.(1996).Searchingforimportantfactorsinsimulationmodelswith
manyfactors:sequentialbifurcation,EuropeanJournalofOperationalResearch(inpress).
Clemson, B., Tang, Y., Pyne, J. and Unal, R. (1995). Efficient methods for sensitivity analysis,
SystemDynamicsReview11:3149.
Eschenbach,T.G.andMcKeague,L.S.(1989).Expositiononusinggraphsforsensitivityanalysis,
TheEngineeringEconomist34:315333.
Eschenbach, T.G. and Gimpel, R.J. (1990). Stochastic sensitivity analysis, The Engineering
Economist35:305321.
Fiacco, A.V. (1983). Introduction to Sensitivity and Stability Analysis in Nonlinear Programming,
AcademicPress,NewYork.
French, S. (1992). Mathematical programming approaches to sensitivity calculations in decision
analysis,JournaloftheOperationalResearchSociety43:813819.
Hall, N. and Menz, K. (1985). Product supply elasticities for the Australian broadacre industries,
estimatedwithaprogrammingmodel,ReviewofMarketingandAgriculturalEconomics53:613.
Hamby, D.M. (1994). A review of techniques for parameter sensitivity analysis of environmental
models,EnvironmentalMonitoringandAssessment32:135154.
Hamby, D.M. (1995). A comparison of sensitivity analysis techniques, Health Physics 68: 195
204.
Hoffman, F.O. and Gardner, R.H. (1983). Evaluation of uncertainties in environmental radiological
assessmentmodels.In:J.E.TillandH.R.Meyer(eds.),RadiologicalAssessments:ATextbook
on Environmental Dose Assessment. US Nuclear Regulatory Commission, Washington D.C.,
Reportno.NUREG/CR3332,pp.11.111.55.
Kingwell, R.S. and Pannell, D.J. (eds.) (1987). MIDAS, A Bioeconomic Model of a Dryland Farm
System,Pudoc,Wageningen.
Kleijnen, J.P.C. (1992). Sensitivity analysis of simulation experiments: regression analysis and
statisticaldesign,MathematicsandComputersinSimulation34:297315.
Kleijnen, J.P.C. (1995a). Sensitivity analysis and optimization: design of experiments and case
studies, Proceedings of the 1995 Winter Simulation Conference (edited by C. Alexopoulos, K.
Kang,W.R.Lilegdon,D.Goldsman),pp.133140.
Kleijnen, J.P.C. (1995b). Sensitivity analysis and optimization of system dynamics models:
regressionanalysisandstatisticaldesignofexperiments,SystemDynamicsReview11:114.
Kleijnen, J.P.C. (1996). Fivestage procedure for the evaluation of simulation models through
statisticaltechniques,Proceedingsofthe1996WinterSimulationConference(inpress).
Lomas, K.J. and Eppel, H. (1992). Sensitivity analysis techniques for building thermal simulation
programs,EnergyandBuildings19:2144.
McKay, M.D. (1995). Evaluating prediction uncertainty. Los Alamos National Laboratory,
NUREG/CR6311(LA12915MS).
Morrison, D.A., Kingwell, R.S., Pannell, D.J. and Ewing, M.A. (1986). A mathematical
programmingmodelofacroplivestockfarmsystem,AgriculturalSystems20:243268.
Nordblom,T.,Pannell,D.J.,Christiansen,S.,Nersoyan,N.andBahhady,F.(1994).Fromweedto
wealth? Prospects for medic pastures in Mediterranean farming systems of northwest Syria.
AgriculturalEconomics11:2942.
Rios Insua, D. (1990). Sensitivity Analysis in MultiObjective Decision Making, Lecture Notes in
EconomicsandMathematicalSystemsNo347,SpringerVerlag,Berlin.

http://dpannell.fnas.uwa.edu.au/dpap971f.htm 15/16
3/15/2017 Sensitivityanalysis:strategies,methods,concepts,examples

SobieszczanskiSobieski, J. (1990). Sensitivity analysis and multidisciplinary optimization for

aircraftdesign:recentadvancesandresults,JournalofAircraft27:993.
Tzafestas, S.G., Theodorou, N. and Kanellakis, A. (1988). Recent advances in the stability
analysisofmultidimensionalsystems,InformationandDecisionTechnologies14:195211
Uyeno,D.(1992).MonteCarlosimulationonmicrocomputers,Simulation58:418423.

Appendix:Aconceptualdifficultywithoptimisationmodels
OnepotentialconceptualdifficultywiththeframeworkpresentedinSection3ariseswhenthistype
of SA is conducted with an optimisation model. A perceived benefit of SA is that it conveniently
allows assessment of the consequences of parameter uncertainty, even with a deterministic
model. However SA with a deterministic optimisation model most commonly generates only a
single optimal result for each combination of parameter values being tested. If, as is normal, the
value of the uncertain parameter will not be definitely known until after the strategy is fixed in
place, there is in fact a range of possible profit outcomes (a probability distribution of outcomes)
for each possible strategy. Thus if a standard SA approach is used to investigate parameter
uncertainty in a deterministic optimisation model, the resulting output will not be easy to relate to
the Bayesian decision theory framework outlined above it provides only a subset of the relevant
information. Note that this problem is unlikely to arise if a simulation model is used, since the
tendency with a simulation model is to generate a full set of SA results for each strategy under
consideration, providing more information about the probability distribution of outcomes for that
strategy.

Therearethreepossibleresponsestothisdifficultywithoptimisationmodels:

a.Dealwiththeparameteruncertaintybyexplicitlyrepresentingitwithinastochastic
model,ratherthanbyusingSAwithadeterministicmodel

b.Constraintheoptimisationmodeltoaparticularstrategyandgeneratesolutionsfor
that strategy for each combination of parameter values. This provides the
probabilitydistributionofoutcomesforthatstrategy.Repeattheprocessforeach
strategyofinterest.Inthisapproach,themodelisreallybeingusedforsimulation
rather than optimisation. However the optimisation capacity is still useful for
helpingselectwhichstrategiestosimulate.

c. Using subjective judgement and mindful of the correct decision theory approach,
estimate the posterior distributions based only on the single optimal result for
each scenario. While the quality of posterior distributions obtained in this way is
likely to be somewhat lower than those obtained by approaches (a) or (b), this
approach is computationally much easier. In practice, a set of single SA results
fromanoptimisationmodelcouldstillbeveryusefulifconsideredwithinthetype
of conceptual framework outlined earlier. An awareness of the inconsistency
between the SA results and the Bayesian decision theory framework should at
leasthelpthemodellerinterpretthesignificanceandimplicationsoftheresults.

Citation: Pannell, D.J. (1997). Sensitivity analysis of normative economic models: Theoretical
frameworkandpracticalstrategies,AgriculturalEconomics16:139152.

Thisversionofthepaperhasbeenmodifiedfromtheoriginaljournalarticletoreducetheemphasis
on economics (since the content is relevant to any discipline) and to simplify the section on
Bayesiandecisiontheory,movingpartofittotheAppendix.

DavidPannell'sWebPage

CopyrightDavidJ.Pannell,1997
Lastrevised:September29,2015.

http://dpannell.fnas.uwa.edu.au/dpap971f.htm 16/16