1/8

CREDIBILITY IN EXPLANATORY MULTIPLE-CASE STUDIES A PRACTICAL APPROACH J. M. Vilas-Boas da Silva (1), (2), J. M. Kay (2) and J. A. Porto Pinho Dinis (3)
(1)

Escola Superior de Cincias e Tecnologia, Universidade Catlica Portuguesa Rua Dr. Mendes Pinheiro, 24, 3080 Figueira da Foz, Portugal
(2)

School of Manufacturing and Industrial Science, Cranfield University Cranfield, Bedford, MK43 OAL, United Kingdom Hoover Elctrica Portuguesa Rua D. Estefnia, 90A, 1000 Lisboa, Portugal
(3)

ABSTRACT This paper aims to present the broad conceptual design of a problem-solving research and to assure the credibility of its outcomes under a practitioners perspective. Therefore, the credibility of a research design for assessing the performance of different organisational solutions for an assembly line facing problems of broadening its product range is going to be the main issue of concern. Case Study is a well-known and frequently used research strategy, especially in ethnographic-like approaches. However, case study design is quite often mistreated, mainly when used for explanatory purposes. Therefore, our methodology is going to be based on the understanding and application of the theoretical background for multiplecase study design provided by Yin (1994). In particular, the classical aspects of the quality assurance of empirical social research, concerning the validity and reliability of design will be considered. The main contributions of this research are (i) the study of variety as a determinant of the control interface: push vs. pull; (ii) the discussion of the relevant performance measures for an assembly line; (iii) the practical application of Yins case study research methodology; (iv) the promotion of simulation techniques as a source of credible predicted data for both research and practice and, (v) the availability of a credible instrument to support decision making concerning the control interface. INTRODUCTION The main research problem addressed here is the impact of the introduction of a new line of products into an existing production system. The system is an assembly line fed by a continuous flow of materials and controlled under a push system paradigm. This increase in the variety is expected to create a disturbance in the operation of the manufacturing system that was designed to cope with a smaller range of products. The research challenge is to fully understand this disturbance, such that appropriate strategies can be chosen to compensate for its effects. This leads to a number of research questions, as follows: (i) Is the current alternative, the best way to organise the production system facing that disturbance? (ii) How far will the disturbance impact: just the assembly line, the production system or the plant itself? (iii) Which parameters are representative of the disturbance? (iv) How can we assess the

2/8

performance of potential alternative production systems facing the disturbance? (v) Can we be confident that the outcomes of the evaluating procedure are representative enough to provide guidance to decision-makers? The research strategy was selected considering the most adequate approach to address the stated research questions and the enquirys purpose (Robson, 1993). The enquiry mainly sought to explain the effect of increasing variety in the manufacturing system performance. There were also more variables of interest than data points because we had at most two production systems available to collect data. Despite each case being able to generate several data points (for different mixes), this setup suggested an in-depth study. Moreover, the boundaries for this empirical, non-controllable, contemporary phenomenon are not clearly evident (Yin, 1993; Yin, 1994). Two examples illustrating the last issue are, as follows: (i) Should the suppliers be included in the enquiry? And, (ii) should the unit of analysis be the organisation as whole, the plant, the production system or the assembly line? In summary, after the discussion, the purpose of the enquiry was considered explanatory and a case study research strategy was selected. The research question (v) raised the point about both the validity and the reliability of the research. This is a common flaw of case study research methodology. According to Yin (1993) the shortage of methodological guidance on how to undertake case study research, the excessive trust in individual interpretations as regards research validity and the soft acceptance of rival theories without questioning them are some of the most critical issues. Nevertheless, the resultant lack of credibility of the case study research strategy might be overcome through action (i) targeting an increase in the quality of the methodology; (ii) attempting valid inferences from events outside the laboratory and, (iii) positioning case study research within the framework of the traditional scientific method (Yin, 1993). Next paragraph will show how both careful and purposeful research design can help with these aspects. THE DESIGN OF THE RESEARCH The model construction Design is the second of the five phases of research (Yin, 1994). Its role is to link problem definition and therefore, the research questions, Explanatory purpose with the evidence to be collected and, with Independent Process Dependent variable interpretation and variable Predicted pattern conclusions (Yin, 1993; Criteria Dimensions of results Yin, 1994). In order to C1 D11...D1l Y11...Y1l start the design stage a Production theory is required (Yin, System Mix 1994), i.e., a prior Ci Di1...Dim Yi1...Yim Model explanation why a push phenomenon occurs the Cn Dn1...Dnn Yn1...Ynn pull way it does (Yin, 1993). This explanation is causal because it identifies cause-effect relationships Figure 1. The causal logic of the model between the performance

3/8

of the manufacturing system and the product variety portrayed by the product mix. Moreover, the theory should be able to predict a theoretical pattern of results for the several dimensions of the performance criteria (Figure 1). Furthermore, these outcomes should be measurable in order to allow hypotheses testing (Yin, 1993). A literature survey identified the push and the pull approaches as the major organisational paradigms of the manufacturing system (Figure 1). Therefore, these are the two rival theories that will be followed, in order to build up rival explanations. The formulation of rival hypotheses is a way that social scientists have to emulate the experimental method. Consequently, instead of the control of an infinite number of rival hypotheses enabled by randomisation, the explicitly specified rival hypotheses, which are the current disputation in the scientific community, are going to be addressed (Campbell in Yin, 1994). Furthermore, it was required to operationalise the theory in some way it could generate a theoretical pattern of outcomes. This will be done through the building up of two simulation models, one following the push logic and the other one the pull approach. Finally, the last step of the model construction, as depicted in Figure 1, concerns the definition of the criteria to assess performance and their related measurable dimensions. The exercise will start with a thorough literature survey. However, the results of this search have to be carefully analysed and weighted by the practitioners experience, in order to pick up the relevant dimensions matching the actual context, the manufacturing systems typology and the strategic options of the organisation. The definition of the strategy for data analysis Until this point both a theory and a rival theory were specified and models were defined to predict rival patterns of outcomes. This design, leading to mutually exclusive patterns, is stronger and, therefore, more credible than an alternative one that excluded the rival theory (Yin, 1994). The predicted mutually exclusive patterns will become the benchmark for comparison with actual data (Yin, 1993). This means that there must be at least two case studies (being one push and another pull) in order to test the two rival theories with the empirical evidence. Figures 2 and 3 summarise the practical aspects of the pattern matching logic using a non-equivalent dependent variables design, applied by Yin (1994; 1993) to case studies. This test will involve pattern-matching processes (Yin, 1994) between predicted (or theoretical) and empirical patterns. It follows that if the empirical push pattern matches its theoretical counterpart, it should not match the predicted pull pattern and vice-versa. If an unexpected match happens modification must be made to the theory, just as with experimental science (Yin, 1994). This will cause the review of the simulation models. A pattern match, to be considered a match should be sufficiently convincing to draw a conclusion, in order to be unequivocally accepted, through a visual comparison. Statistical tests are not adequate because there have not enough data points. The main aspects are that this lack of precision leads to interpretative discretion, which should make researchers cautious to assume very complex patterns (Yin, 1994). The purpose of an explanatory case study research, rather than the generalisation of the results from a sample to a population is to increase the understanding of theoretical propositions developed for a particular situation (Yin, 1993; Yin, 1994). The logic used to

4/8

link the several cases in a multiple-case study research should be a replication logic instead of sampling one. Then, the purpose is to test a theory, while explanation of a phenomenon (Yin, 1994), with empirical evidence (Yin, 1993). The credibility of the findings will increase with the number of replications (Yin, 1993). In order to achieve literal replication in a multiple-case study, two or more cases that confirm the same pattern of predictable outcomes should be found (Yin, 1994). Literal replication may help to rule out a threat to validity, if another case, where that particular threat does not take place, also confirms the predicted pattern. Figure 2 depicts graphically this issue. If however, there was only one case, a subset of the initial dependent variable would have to be identified in order to rule out the threat (Yin, 1994). The number of literal replications to be targeted decreases, if the difference between rival theories increases and also, if the required certainty decreases (Yin, 1994). Finally, theoretical replication is achieved when an empirical pattern of outcomes replicates a predicted one but it does not replicate the theoretical pattern generated by the rival theory, however, for predictable reasons. Figure 3 represents this situation schematically. It should be noticed that both the outcomes and measures are the same for the rival alternative propositions, which, in practice, were made operational by the push and the pull models. Of course the predicted patterns will be different (Yin, 1994). At last, the number of theoretical replications increases when the level of certainty, whether external conditions will produce different case study results, increases (Yin, 1994). Quality assurance of the research To avoid, as much as possible, equivocal evidence or biased views in findings and conclusions the four tests used to establish the quality of any empirical social research were accepted as valid for case studies (Yin, 1994). These are as follows: (i) construct validity; (ii) internal validity; (iii) external validity and, (iv) reliability (e.g., Robson, 1993; Lee, 1999). Construct validity The goal of construct validity is to establish the correct operational measures for the concepts being investigated (Yin, 1994). Its objects of interest are the measures, the instruments and the processes that operationalise the constructs of the research, in order to avoid subjective judgements (Yin, 1993; Yin, 1994). As it was already mentioned, the performance measures of the manufacturing system are coming from a literature survey that will be refined by the practitioners experience. Moreover, the main instrument to implement the theoretical approaches will be a commercial simulation tool that has been tested by the market for a number of years. The process will depend on the ability of the researchers to interpret the theoretical principles, construct a simulation model and review it, with the support of the empirical data, until it becomes acceptable. The main tactics to increase construct validity are (i) the use of multiple sources of evidence, (ii) to establish a chain of evidence and (iii) to have key informants reviewing the case report (Yin, 1994). Two plants of the industrial sponsor will be available for data collection and several people within each one will be addressed. Moreover, the plants are in different countries and they have different organisational systems: one, push, the other one, pull. In addition, the main representative of the industrial sponsor in the research has the job title of Plant Manager, which enables both high level contacts and the availability of the required means. So, we believe that both the conditions (i) and (iii) are fulfilled.

5/8

Finally, most care will be given to condition (ii) through the organisation and development of the case in such a way an external observer could follow the origin of some evidence from research questions to conclusions (Yin, 1994). Internal validity Internal validity is achieved through the specification of the unit of analysis, the development of a priori rival theories and, the collection and analysis of data to test these rivals (Yin, 1993). Moreover, pattern matching was described as the main technique used for analysis (Yin, 1994). The unit of analysis that was found more adequate was the plant, because some measures of the manufacturing systems performance take place at the plant level. However, as the assembly line is a major object of the analysis, it should be considered as an embedded subunit (Yin, 1994). On the other hand, the definition of the unit of analysis together with the development of the rival theories determines the limits of the data collection and analysis (Yin, 1994). Figure 2 shows how the pattern-matching technique should be applied. After the push model has been built up it is fed with the mixes that usually occur in the push case study (MixA1, , MixAo). For each of the mixes, the Internal Validity or Model Refining predicted pattern of Predicted Empirical results generated by the Main ? Push pattern of pattern of simulation model (YA1 , Match Case results results model Mix Mix ? ,YAo) should be (push) YA1 YA1mc compared with the empirical pattern Literal Replication => Stronger Internal Validity generated by the push case study (YA1mc , Predicted Empirical Other ? Push ,YAomc). The desirable pattern of pattern of Match Case results results results of the model Mix ? Mix (push) YK1 YK1oc comparison are either (i) the match of the Legend: mc - main case (push); oc - other push case patterns and the consequent claim of Figure 2. Internal validity and literal replication internal validity or, (ii) both the review and fine tuning of the model. Another results that can happen are not as desirable and the strategy to deal with them has to be postponed until their occurrence. Finally, a similar procedure should be followed for the pull model and respective case study.
A1 A1 K1 K1

External validity External validity is concerned with establishing the domain to which the findings of a study can be generalised. Case studys findings should be generalizable to theoretical propositions and not to populations (Yin, 1994) through analytic generalisation, just like experiments. Therefore, a theory would be tested through replications of the findings by empirical data. The empirical results would be considered stronger if two or more cases support the same theory but do not support an equally plausible rival theory (Yin, 1994), for predictable reasons. This is the basis for the discussion, which was formalised and graphically summarised in Figure 3, about the theoretical replication of the study, as follows:

6/8

1. A simple mix (MixA1) generated by the push case should originate different theoretical patterns when fed in the two rival simulation models (YA1 for the push model and YA1 for the pull model) for predictable reasons. If they do not differ enough, then reviewing the models is one possibility to progress towards theoretical replication. Besides that we should expect a Predicted Mix pattern of better performance Push ? results YZ1 model of the pull model, i.e., YA1 > YA1. Predicted ? pattern of Mix 2. A more complex Push results YA1 Secondary Main mix (MixZ1) model case case ? generated by the ? (pull) (push) pull case should Mix Predicted Pull originate different pattern of model results YA1 ? theoretical patterns when fed in the two Predicted Mix Pull rival simulation pattern of ? model results YZ1 models (YZ1 for the pull model and YZ1 for the push model) Figure 3. External validity and theoretical replication for predictable reasons. If they do not differ enough, then reviewing the models is one possibility to progress towards theoretical replication. Besides that we should expect a better performance of the pull model, i.e., YZ1 > YZ1. 3. At this point, several possible cross-combinations may happen. Even, if the conditions in paragraphs 1. and 2. are positively checked, cross-comparisons should still take place. For instance, we expect the push model to perform better for a simpler mix (YA1 > YZ1) and exactly the same for the pull model (YA1 > YZ1 ). We also expect that the pull model would perform better fed with a simpler mix (MixA1) than the push model, when this is fed with a more complex mix (YA1 > YZ1 ). However, we cannot predict if the push model fed with the simpler mix (MixA1) is going to perform better or worse than the pull model fed with the more complex mix (MixZ1). If all the other conditions were favourably checked it is believed that there is enough ground to accept whatever result comes out for this last condition. 4. Finally, two last questions might arise at this stage, as follows: 4.1. How probable is that theoretical replication might occur? At this point, the only thing that can be said is that the stronger the patterns, the more probable it is to get theoretical replication and, so, external validity for the models (Yin, 1993). 4.2. If only part of the predicted conditions verifies, can we still claim external validity? We may, because this issue cannot be treated as an algorithm. There is enough room for analysis and discussion, but only after the patterns of results come out.
Z1 A1 A1 Z1

Reliability Reliability is often considered as the possibility of repetition of a study or, of any of its stages (Yin, 1994). However, other authors prefer to call it credibility (Robson, 1993). In this paper, it will be assumed both a broader and more common meaning for credibility, which is the capacity of being believed. Therefore, this research will provide sufficient information on the methods used and on the justification for their use (Robson, 1993) in

Theoretical Replication => External Validity

7/8

order to make sure that, if another researcher followed the same procedures, he should arrive to the same findings and conclusions (Yin, 1994). The two tactics introduced to deal with reliability are the case study protocol and the case study database (Yin, 1994). In this research, a protocol will be developed to help the investigator to carry out the study. It will also contain the procedures and general rules to be followed in its use (Yin, 1994). In addition, a database will be created in order to make a clear separation between the evidentiary data and the research reporting. The aim is to be able to inspect the database that is going to lead to the study conclusions without incurring in any potential confusion with the reporting activity (Yin, 1994). CONCLUSIONS In the Introduction section the research problem, questions and strategy were presented. Moreover, the credibility of the outcomes was promised, based on careful and purposeful design of an explanatory case study research. The section, Design of the research, was subdivided into three subsections. In the subsection, The model construction, the research model was introduced as well as the supporting rival theories: the push and pull approaches to manufacturing organisation. In the subsection, The definition of the strategy for data analysis, a pattern matching analytic logic based on the non-equivalent dependent variables design, was presented. The core concepts concerning analytic generalisation were also introduced. These are the literal and the theoretical replication. In the Quality assurance subsection it was explained the way that the research design was committed to validity (construct, internal and external) and reliability. The quality of the research was defined as a function of these four tests. In summary, this design should give the most credible results that the current state of the art enables. Therefore, if it is successfully followed by the practitioner, who is already being involved in the research design as the second co-author of this paper, the decision-maker might increase its trust on the results of the simulation models. This means a smaller risk for the industrial sponsor on the choice of the most adequate production system to deal with the variety increase. As a further work, the strength of the models might be increased by getting more case studies and so, improving the validity of the research through the achievement of literal replication (Figure 2) and the expansion of theoretical replication (Figure 3). At first, this will become an easier exercise, if the prescribed tactics for helping to improve the repeatability of the study, even by other researchers, were followed. Secondly, there are more similar plants available within the industrial sponsor, which is a multi-site, multinational group. So, further cases are available if required. Moreover, external validity might still improve, if some of the cases are from other plants, external to the industrial sponsor group. Nevertheless, in the later case, more problems should be expected in the matching of the theoretical and empirical data patterns. Hence, the requirements for the research redesign have a much higher probability to arise. The presented research was mainly designed considering Yins recommendations. This might become a threat. Nevertheless, (i) Yins work is itself a result of the triangulation of several methodological sources; (ii) Dr. Yins background in experimental psychology introduces an extra factor against bias; (iii) since the publication of its first work, in the early eighties, there were no further advances in the methodology he proposed; (iv) current textbooks on social sciences methodology still refer to Yins work as the major reference, when approaching explanatory case study research (e.g., Lee, 1999; Robson,

8/8

1995; (v) there is the statement of Donald Campbell arguing for the uniqueness of this text and suggesting that it would become a standard in social sciences (in Yin, 1994). Some examples of other relevant threats to the proposed research design, to which special attention is required, are going to be listed in this paragraph, as follows: (i) In an embedded case study conclusions may focus on the subunit (the assembly line) failing to return to the unit of analysis (the plant) (Yin, 1994). (ii) The lack of strong patterns of data affects the internal validity of the study and especially the external validity. Patterns of results should be contrasting; otherwise conclusions are likely to be successfully challenged (Yin, 1993). (iii) A larger number of case studies imply more difficulty to match the empirical data with the established dimensions of measurement. (iv) Further complications to the replication logic can arise from trying other mixes, because mix complexity can vary in the range (MixA1, , MixAo), for the push case and, in the range (MixZ1, , MixZp), for the pull case. (v) Design is not everything in research. In order to be successful especial care is also required for data collection and, for modelling and reporting procedures. These are out of the scope of this paper and, therefore, were not addressed. The main contribution of this research to the Manufacturing Systems Analysis and Design (MSAD) body of knowledge is the study of variety as a determinant of the control interface: push versus pull. Another contribution to MSAD is the study and discussion of the relevant performance measures for an assembly line. The main contribution to the Methodology is the operationalisation and divulgation of Yins work targeting mainly explanatory case studies. Other contributions to methodology are (i) the improvement of the credibility of the case study research strategy and, (ii) the promotion of simulation techniques as a trustful source of data in explanatory case studies. The main contribution to Practice is to make available a credible instrument to help decision-making concerning the choice of the control interface. Therefore, the role of simulation models for helping to choose among practical solutions is reviewed. Finally, a potential contribution to Policy is the influence on the way MSAD policies are created, analysed and discussed. Therefore, the use and validation of theory is reviewed as a driver for action. REFERENCES Lee, T., (1999), Using qualitative research methods in organizational research, Sage Publications, Inc., USA. Robson, C., (1995), Real world research, Blackwell Publishers Inc., Great Britain. Stake, R., (1995), The art of case study research, Sage Publications, Inc., USA. Yin, R., (1993), Applications of case study research, Sage Publications, Inc., USA. Yin, R., (1994), Case study research: design and methods, 2nded. Sage Public., Inc., USA.