Eleven Critical Pitfalls in Simulation Modeling

by Averill M. Law, Ph.D.

The following material is a synopsis of some of the ideas presented in my three-day simulation short courses. Pitfall Number 1: Failure to have a well-defined set of objectives at the beginning of the simulation study. We recommend making a list of the specific questions that the model is to address and also the performance measures that will be used to evaluate the efficacy of various system configurations. Otherwise, it will be impossible to determine the appropriate level of model detail. Pitfall Number 2: Failure to communicate with the decision-maker (or the client) on a regular basis. This is essential to ensure that the correct problem is solved and to promote model credibility. There are many valid (technically sound) models that are not used in the decision-making process because they are not credible. Pitfall Number 3: Lack of knowledge of simulation methodology and also of probability and statistics. A significant percentage of the people involved in simulation modeling are only trained in how to use a particular simulation software package, which we feel is definitely not sufficient. Most experts in simulation modeling would agree that "programming" of the model represents only 25 to 50 percent of a sound simulation study. The simulation analyst must also be knowledgeable in simulation methodology (validating a model, selecting input probability distributions, designing and analyzing simulation experiments, etc.) and also probability and statistics (probability distributions, confidence intervals, etc.). Pitfall Number 4: Inappropriate level of model detail. A very common pitfall for beginning simulation analysts is to have an excessive level of model detail. We recommend starting "with a moderately detailed" model which is embellished as needed. The adequacy of a particular version of the model is determined in part by having the model reviewed by "subject-matter experts" and by the decisionmaker (or client). Modeling each aspect of the system will seldom be required to make effective decisions, and will also be infeasible due to time, money, or computer constraints.

Pitfall Number 5: Failure to collect good system data. If one is modeling an existing system, it is important to collect data on key system random variables. (For a manufacturing system, key random variables would probably include times to failure and times to repair for each machine.) Often this is not done because of project time constraints or because the simulation analyst does not realize that this is an important consideration. Pitfall Number 6: Belief that so-called easy-to-use simulation packages require a significantly lower level of technical competence. Some people believe that a so-called "easy-to-use" simulation package will make performing a simulation study a much easier task. This type of software can reduce the time to "program" a model for problems of modest complexity. However, for most realworld problems, programming in some form is required. Furthermore, the simulation modeler will still have to be concerned with formulating the problem, collecting and analyzing data, validating the model, modeling system randomness, designing and analyzing simulation experiments, and managing the overall simulation project. These activities require a significant amount of technical competence and experience. Pitfall Number 7: Blindly using simulation software without understanding its underlying assumptions. To facilitate ease of use, simulation-software vendors have added to their software powerful "macro" blocks (or modeling constructs) that model a significant part of a real-world system. However, these blocks are often not well documented, possibly resulting in the development of an invalid model. For example, one encounters many types of conveyors in practice, yet some simulation products only offer a few conveyor options. Pitfall Number 8: Misuse of animation. Animation is useful for communicating the essence of a simulation model to decisionmakers (who may not understand all of its technical details), for debugging simulation computer programs, and for suggesting improved operational procedures for a system. However, the efficacy of a particular system design should be decided by applying appropriate statistical procedures to carefully designed simulation experiments. Just because a "short" run of the animated simulation model seems okay, this does not mean that the model is either debugged or valid. Pitfall Number 9: Replacing a probability distribution by its mean. A common (but unfortunate) practice in simulation modeling is to represent a source of system randomness by the perceived mean value rather than its corresponding probability distribution. For example, consider a single-server queueing system where the mean interarrival time and mean service time are 1 minute and 0.99 minute, respectively. Suppose that the interarrival times and the service times actually each have an exponential distribution. Then the long-run average number in queue is

approximately 98. Suppose, on the other hand, that a simulation analyst takes every interarrival time to be a constant 1 minute and every service time to be a constant 0.99 minute. Then no customer ever waits in the queue! Thus, it is not sufficient to just get the mean correct we also have to represent variability in an appropriate way. Pitfall number 10: Using an inappropriate probability distribution. It is important to model each source of system randomness by an appropriate probability distribution. For example, many simulation practitioners represent the time to do some task by a normal distribution (a symmetric distribution). However, we have never seen a task-time data set that was actually normally distributed. In practice, most histograms have a longer right tail (positive skewness). Consider the single-server queueing system from Pitfall Number 9. If one models the service-time distribution by a symmetric distribution when, in fact, it is positively skewed, then the average number in queue may be significantly underestimated. Pitfall Number 11: Failure to perform a proper output-data analysis. A stochastic simulation model does not produce the true performance measures for the model it only produces statistical estimates of them. A simulation analyst must properly choose the simulation run length, the length of the warmup period (if one is appropriate), and the number of independent model replications (each using different random numbers). We recommend that confidence intervals be constructed for important performance measures. Note, however, that this can not be easily done using the data from one simulation run, since these data will not be independent (an assumption of classical statistics). It is also not possible to get legitimate standard deviation (or variance) estimates from one simulation run, yet a number of simulation software products provide these automatically.