Rose Mann Vess Ey

Rosemann & Vessey/Improving the Relevance of IS Research
ISSUES AND OPINIONS
TOWARD IMPROVING THE RELEVANCE OF INFORMATION SYSTEMS RESEARCH TO PRACTICE: THE ROLE OF APPLICABILITY CHECKS1
By: Michael Rosemann Queensland University of Technology Brisbane, Queensland 4000 AUSTRALIA m.rosemann@qut.edu.au Iris Vessey The University of Queensland and Queensland University of Technology Brisbane, Queensland 4000 AUSTRALIA i.vessey@business.uq.edu.au or i.vessey@qut.edu.au
research life cycle, it leaves untouched the rigorous methods used to conduct the study, that is, it does not compromise traditional research models. The approach we propose is based on the analyses of three dimensions of relevance that are critical to practitioners attempts to internalize IS research findings (importance, accessibility, and suitability), and a comprehensive set of solutions that can be used to address them. Our analysis reveals that the most critical dimension for practice is the importance of the research to the needs of practice. The solution we propose to address that need is to conduct an applicability check on the research objects of interest. The applicability check forms an integral part of the research process, either prior to or following engagement in a typical research process. We present principles and criteria for the conduct and evaluation of an applicability check, which is primarily based on the focus group method, and secondarily on a modified nominal group technique. Keywords: Relevance, rigor, academic research, research process, applicability check, focus group method, modified nominal group technique
Abstract
This paper takes a first step in aiding researchers to improve the relevance of their research to practice. By proposing that Information Systems researchers conduct applicability checks with practitioners on the research objects (for example, theories, models, frameworks, processes, technical artifacts, or other theoretically based IS artifacts) they either produce or use in theory-focused research, our paper presents an actionable, systematic approach to evaluating, establishing, and further improving research relevance. Furthermore, because it is an approach that can be conducted as an additional step either at the beginning or the end of the traditional
Introduction
Information systems research relevance has long been an issue in the IS academic community. Research is relevant when it addresses the needs of one of a number of different stakeholders. Research could, for example, be relevant to other researchers, to research and development organizations, to boundary spanners such as consultants, to IS students, and
Carol Saunders was the accepting senior editor for this paper. Ola Henfridsson served as a reviewer. An additional reviewer chose to remain anonymous.
MIS Quarterly Vol. 32 No. 1, pp. 1-22/March 2008
even to society in general (Davenport and Markus 1999; Davison et al. 2004). Like much of the literature on rigor and relevance, we address relevance from the viewpoint of IS professionals (see, for example, Applegate and King 1999; Benbasat and Zmud 1999; Davenport and Markus 1999; Kavan 1998; Senn 1998). Relevance is essential for satisfying ISs practitioner constituents whom academia will educate as the next wave of IS professionals, as well as those who read reports of the research, disseminate the knowledge, and provide leadership to organizations on the effective management and utilization of information technologies (Kavan 1998). Without research outcomes relevant to practice, the very existence of a research discipline could be questioned because the discipline could well lack impact beyond its own (academic) community. Establishment of rigor in research, on the other hand, is essential for satisfying the traditional quality standards demanded of academic disciplines: IS academics need to conduct rigorous research to guarantee that it is of high quality in order to establish credibility, to publish in high quality journals, to attain tenure and promotion, and to compete for research funding (Applegate and King 1999; Dennis et al. 2006; Galliers 1994; Robey and Markus 1998). Rigor is manifested largely by using sound methodology (Benbasat and Weber 1996). The issue of IS research relevance is raised frequently within the IS community, most often in the context of the debate on rigor versus relevance. Recent commentaries appear in MIS Quarterly (Volume 23, Issue 1, 1999) and as part of specific research methodologies (for example, MIS Quarterly, Volume 28, Issue 3, 2004, with its focus on Action Research). Attention has also been paid to research relevance as an important issue in its own right, as in the special issue of the Information Resources Management Journal (Winter 1998), and a number of papers in Communications of the AIS (Volume 6, 2001). Note, also, that the 2007 European Conference on Information Systems was subtitled Relevant Rigour Rigorous Relevance, while the 2007 Australasian Conference on Information Systems has the theme The 3Rs: Research, Relevance and Rigour Coming of Age. Notwithstanding Benbasat and Webers (1996) statements to the contrary, it is not clear that the IS academic community has made much progress in recent years in placing more value on relevance. There appear to be three major issues. First, there are few incentives for researchers to conduct research that is relevant to practice due to the perception that it is not valued by the top IS research journals. While the majority of
researchers most likely will not change the focus of their research without changes to the incentive system, certain researchers do choose to conduct research that is designed to impact practice directly. A further group of researchers would likely conduct relevant research if they could be assured of maintaining rigor. While intellectual property arrangements could also be a disincentive to working with practice, emerging research funding models based on assessments of the research quality (rigor) and the research impact (on end users) of a research group (for example, the Australian Research Quality Framework) would increase the pressure to consider practitioners as relevant stakeholders in the research (DEST 2006). Second, there is a perception that it is impossible to attain both rigor and relevance in research. The IS academic community often views rigor and relevance as conflicting research objectives, that is, as the two extreme points of a continuum. Achieving one is viewed as necessarily compromising the other (see, for example, Davenport and Markus 1999). Hence the issue is perceived as that of rigor versus relevance. We argue, however, that it is possible to conduct research that is both rigorous and relevant (see, also, Benbasat and Zmud 1999; Fllman and Grnland 2002; Kock et al. 2002; Robey and Markus 1998; Senn 1998). Third, in addition to arguments internal to the IS academic community, there is limited demand on the part of practice for the outcomes of IS academic research. Evidence for this assertion can be found in the rather low number of practitioners who subscribe to IS journals or who attend IS conferences such as ICIS. The limited exposure of IS research to the practitioner community suggests that only a small portion of the body of academic IS knowledge is disseminated and evaluated for its potential relevance. As can be seen in the numerous citations referenced here, the IS community is concerned about the relevance of its research. The objective of this paper is to propose applicability checks as a way of allowing practitioners to provide feedback to the academic community on the research objects it produces or uses in theory-focused research. Applicability checks are evaluations by practice of the theories, models, frameworks, processes, technical artifacts, or other theoretically based IS artifacts that the academic community either uses or produces in its research. Applicability checks could be conducted on emerging IS research outcomes as well as on established research models, for example, models such as DeLone and McLeans IS success model (1992, 2003), the technology acceptance model
MIS Quarterly Vol. 32 No. 1/March 2008
(Davis 1989), and adaptive structuration theory (DeSanctis and Poole 1994; Poole and DeSanctis 1990). Further, an applicability check would improve future research by incorporating learnings into revisions to theories or models, as well as sensitizing the practitioner community to a body of research. Hence applicability checks would aid in improving the relevance of research conducted over time. It is important to emphasize that conducting an applicability check on research objects leaves intact the normal research process, thereby addressing relevance while still supporting a rigorous research process. Further, our suggestion to conduct applicability checks responds directly to the statement by Benbasat and Zmud (1999, p. 13) that the IS field does not possess the evidence with which to illustrate the impact of its research.This is an important question that academics should investigate.
addresses a real-world problem, and whether it is timely. Based on these criteria, we view research that is important as that which meets the needs of practice by addressing a realworld problem in a timely manner, and in such a way that it can act as the starting point for providing an eventual solution. Second, accessibility of research to practice encompasses whether the research is understandable, readable, and focuses on results rather than the research process (Klein et al. 2006). Substantial evidence suggests that practitioners often perceive research publications as being difficult to understand (see, for example, Benbasat and Zmud 1999; Kavan 1998; Kock et al. 2002; Robey and Markus 1998; Senn 1998). The research community has its own terminology and expends considerable effort reporting on methodological aspects (for example, significance tests, instrument validity, instrument reliability, etc.) that are often impossible for nonacademics to understand. Furthermore, academic papers refer to and build on related work, and often cannot be read in isolation from this body of knowledge. Third, applicability of research to practice encompasses whether the published article is complete, whether it provides guidance and/or direction, and whether it provides concrete recommendations (Klein et al. 2006). We believe, however, that even if practitioners believe that a research object is important, they may not be able to apply it. Many proposed models lack, for example, detailed insights into how their tenets can be achieved in practice (for example, appropriate top management support in critical success factor models). We therefore envisage research as going through a step prior to applicability, which we call suitability. If research is deemed to be important to practice and suitable for meeting the needs of practice, the research can be further elaborated to render it applicable to practice. Using the dimensions of importance, accessibility, and suitability, we thereby view research relevance as the degree to which practitioners can readily comprehend research as promising a solution potentially applicable to a problem existing in their organization.
On Research Relevance and Potential Solutions

While the dimensions that underlie research relevance are well established in the literature, potential solutions have received relatively little attention. Hence we present the dimensions of research relevance briefly and explore the potential solutions in more detail.
Dimensions of Research Relevance

Numerous publications have addressed the dimensions of research that render it relevant to the practitioner community. Perhaps the most visible is Benbasat and Zmuds (1999) characterization of relevance as an articles content, with dimensions of interesting, applicable, and current, and the articles style, with the dimension of accessible. More recently, Klein et al. (2006) used Benbasat and Zmuds characterization as the starting point for a comprehensive literature-based examination of relevance. This examination resulted in the following three dimensions of relevance: importance, accessibility, and applicability. We use Klein et al.s three dimensions of relevance as our starting point for characterizing research relevance and developing an approach to improve it. First, Klein et al.s view of the importance of research to practice encompasses whether the characteristic or process under consideration can be controlled within the organization, whether it focuses on a key management issue, whether it
Addressing the Relevance of Research to Practice

Figure 1 presents a comprehensive characterization of the ways in which research relevance can be improved based on a hierarchy of three perspectives. Each of these perspectives can impact positively the dimensions of relevance to practitioners, that is, importance, accessibility, and suitability.
Institutional Perspective
1. 2. Institutional governance approach Clinical approach
Project Governance Perspective

Forming alliances with 1. Companies 2. Research institutes 3. Professional societies 4. National and international funding bodies
Research Process Perspective

1. 2. 3. 4. 5. Identification of the research problem Theoretical development Research methodology Data analysis Communication of the findings
Figure 1. Solutions for Improving Research Relevance
The Institutional Perspective The institutional perspective, as the name suggests, refers to mechanisms that institutions put in place to provide an environment conducive to pursuing research that is relevant to practice. By institutions, we mean structures and mechanisms of social order and cooperation governing the behavior of two or more individuals. Institutions are identified with a social purpose and permanence, transcending individual human lives and intentions, and with the making and enforcing of rules governing cooperative human behavior (en.wikipedia.org/wiki/Institutions, accessed July 10, 2007). We can identify two major streams of thought based on the institutional perspective leading to increased research relevance: the institutional governance approach and the clinical approach. One example of the institutional governance approach is found in the 1996 report of the Leadership Task Force of the AACSB (American Assembly of Collegiate Schools of Business), which presented a number of strategies for dealing with issues surrounding the relevance of business research to practice. The major focus is on establishing a substantially broader approach to the conduct of academic research in an
attempt to make business school research relevant to practice. In Table 1, we present those aspects of the report that apply to IS/IT research, as presented in Saunders (1998). Other institutional-level approaches include the Research Assessment Exercise (RAE) in the United Kingdom and the Research Quality Framework (RQF) in Australia. The RAE, introduced in 1986, is a research funding model that takes into account both the quality of the research per se and the quality of the research in relation to practice. The RQF, which is due to be introduced into Australia in 2008, places a greater emphasis on research relevance (termed impact [DEST 2006]). The adoption of such frameworks increases the pressure to view practitioners as stakeholders in research that academics conduct. The clinical approach is based on Moodys (2000) analogy of IS to medicine. Moody argues that the applied discipline of IS should be patterned on the applied discipline of medicine.2 The medical discipline is characterized by a high level of inte-
Note that arguments made in the context of medicine are relevant to other applied disciplines, for example, architecture.
Table 1. AACSB Recommendations for Improving the Relevance of Business School Research, Adapted to IT Research (Based on Saunders 1998)
Develop Closer Links to Business and Technology Sabbaticals in corporations Industry-based projects for students Internships for junior faculty Business consulting Building partnerships and alliances with business groups Improve Faculty Skill Levels Re-evaluate tenure criteria Realign faculty reward processes Communicate standards for evaluation Revise Ph.D. Program Requirements Require at least a minimal level of business experience and managerial involvement as a requirement for admission or as a supplemental part of doctoral programs Adopt a strategy that interdisciplinary dissertations and studies of actual business practices are viewed positively within the dissertation process Develop business experience of students in doctoral programs Form Partnerships with Professional and Discipline-Based Organizations Individual schools, and the AACSB, should encourage disciplinary-based academic organizations to include practitioners in their annual meetings to help define new issues of which the membership would be aware
gration between research and practice that melds the two into a single community, perhaps via joint universityindustry appointments. In this instance, the researcher is also a practicing clinician. Further, the medical discipline does not distinguish between academic and practitioner journals and conferences. It places equal weight on the rigor and relevance of its research, and requires articles to be easily understandable to practitioners.3 The approaches presented here recommend several ways of improving the interaction between academia and business. If adopted, such approaches could result in major refocusing, and perhaps upheaval, not only among IS faculty, but also within universities. The Project Governance Perspective The project governance perspective reflects the influence of business on research projects via the formation of different kinds of research alliances. This influence appears to be operationalized almost exclusively via the funding of research projects that address the funding sources needs. These approaches tend to ensure that the research findings are at least relevant for the industry partner(s). As Table 2 shows, those alliances may be built at a number of different levels:
(1) specific companies, which provide various types of research funding; (2) research institutes, either university or nationally based, which attract a select body of sponsors who have first right to research findings in return for monetary support; (3) professional societies, which use a similar funding model to that used by research institutes; and (4) national or international funding bodies, which typically make cash contributions to collaborative research projects that can then be used to leverage corporate contributions. Table 2 also provides examples of alliances that are intended to be illustrative rather than comprehensive. Note that there are numerous regional differences within these types of research collaborations (see, also, Lyttinen 1999). The Project Research Process Because the research process perspective is totally under the control of the researcher, it can be used to address relevance directly. We identify a number of ways in which IS researchers may make their research more relevant to the IS practitioner community based on the research process. To organize these potential solutions, we introduce a typical research life cycle and characterize these approaches in terms of the phases involved: (1) identification of the problem; (2) theoretical development; (3) research methodology including data collection; (4) data analysis; and (5) communication of the findings. Figure 2 presents our model of the research life cycle, while Table 3 uses the model to present our analysis of suggested solutions.
See Introna and Whittaker (2004) for an in-depth analysis of why this approach was tried and ultimately discontinued at the MIS Quarterly.
Table 2. Examples of Types of Research Governance

Level Companies Research Institutes Examples Scholarships, chairs, projects funded by IT users, IT vendors, IS/IT consultants (Markus and Robey 1998) Cooperation with corporate research departments (e.g., IBM, Hewlett Packard, SAP, Infosys, Microsoft) University-based research centers with close industry relationships " Babson College: Process Management Research Center; Working Knowledge Research Center " MIT: Center for Information Systems Research " University of Arkansas: RFID Research Center " University of Memphis: Center for Supply Chain Management Nationally based research centers with close ties to industry " NICTA, Fraunhofer Gesellschaft Society for Information Management Advanced Practices Council
Professional Societies National and International Funding Bodies
National or international funding schemas that leverage the involvement of at least one industry partner with substantial cash and in-kind contributions in a research project European Framework Programme for Research and Technological Development Australian Research Council: Linkage Grants and Cooperative Research Centers
Relevance
Identification of the Research Problem Research Methodology Theoretical Development Data Analysis Communication of the Findings
Rigor
Time
Figure 2. The Research Life Cycle
Table 3. Approaches to Improving Research Relevance Based on the Research Life Cycle
Identification of the Research Problem Seek current practitioner issues Be aware of dominant issues; for example, as presented in the annual report, Critical Issues of Information Systems Management (CSC), and critical issues studies conducted by IS researchers (Senn 1998) Be aware of topics that are the focus of practitioner initiatives (Lyttinen 1999); for example, the SIM Advanced Practices Council grants (Senn 1998) Initiate, moderate, or participate in communities of practice (see, for example: www.bpm-roundtable.com) Conduct major issues studies using focus groups, interviews, surveys in order to formally derive an industry-driven research agenda Focus of the research Focus on needs of practitioner audience (Robey and Markus 1998) not only on academic interests Focus research on a theme (Senn 1998); however, Robey and Markus (1998) question the viability of such an approach, given rapid technological advances Avoid moving from issue to issue without producing principles, prescriptions and well formulated theories that create at best fragments of research not meaningful solutions (Senn 1998) Theoretical Development Adopt new models of research (Robey and Markus 1998) Applied theory research; using an established theory to address a problem in practice (industry-driven, not curiosity-driven) Evaluation research, which is not necessarily based in theory, involves examining a practical intervention using a well-established, rigorous research approach Policy research, which is commonly found in schools of political science and government, focuses on a broad area that requires resolution; such studies are rarely theoretically based Use useful logic and theory (Robey and Markus 1998) Embrace the role of non-deterministic theories; consider using contradictory logic (Robey and Markus 1998) Seek objective findings rather than ones that confirm assumptions and beliefs (Senn 1998) Ensure that you have action levers (Robey and Markus 1998) Research Methodology Form alliances with practitioners Practitioners will serve as critics of theory, research approaches, and findings (Senn 1998) Involve the practitioner directly in the research (Kavan 1998) Create win-win situations with the practitioner (see, for example, the SIM-APC model referenced above) (Kavan 1998; Senn 1998) Become sensitive to the nuances of corporate culture (Kavan 1998) Develop an appropriate solution process Use pilots, appropriate research methodologies (e.g., case studies, focus groups, action research), prototypes (Kavan 1998) Continue trend toward qualitative research (Kavan 1998) Ensure that the research (particularly data collection) involves sufficient numbers of appropriate practitioners (Senn 1998) Data Analysis Ensure you have credible evidence to support claims (Robey and Markus 1998) Describe evidence and the processes for generating it in ways that can be believed by practitioners (Robey and Markus 1998)
Table 3. Approaches to Improving Research Relevance Based on the Research Life Cycle (Continued)
Communication of the Findings Produce consumable research reports Use an accessible style tailored to the target audience (Benbasat and Zmud 1999; Kavan 1998; Kock et al. 2002; Robey and Markus 1998; Senn 1998) Produce reports that are shorter and more concise, more clearly written, and much better illustrated than those prepared for journals (Benbasat and Zmud 1999; Robey and Markus 1998; Senn 1998), have minimal statistical or mathematical detail, and few citations in the text (Markus and Robey 1998) Focus on results and their implications more than on process (Senn 1998) Use a top-down orientation (Kavan 1998) Use novel, critical, constructive story lines (Robey and Markus 1998) Ensure that you have useful implications for practice that include action levers for practitioners (Robey and Markus 1998) Produce multiple versions of research reports tailored for different audiences (Senn 1998) Expand publication outlets Choose more practitioner-oriented journals such Sloan Management Review, Communications of the ACM, Harvard Business Review, MIS Quarterly Executive, and journals that are distributed to the members of professional associations (Robey and Markus 1998; Fllman and Grnland 2002; Kock et al. 2002) Have trade journals carry abstracts of relevant academic research (Kavan 1998) Forge better partnerships between research outlets and the vendor community (Kavan 1998) Present a related body of research in a form suitable for practitioners; for example, an edited volume with original contributions on a particular theme; individual universities have also taken this approach (Robey and Markus 1998) Use traditional practice reports, management briefs, white papers, and the Internet (Senn 1998) Produce special issues on topics of interest to IS managers and hold themed conferences with both academics and practitioners (Kavan 1998) Use the Internet to shorten publication time (Robey and Markus 1998; Senn 1998); to distribute research findings to practitioners (Moody and Shanks 2000; Senn 1998); and to make data available (Mandviwalla and Gray 1998) Write a popular book (Robey and Markus 1998)
Note that Figure 2 presents the research life cycle phases on a continuum from rigor to relevance along the Y axis: those phases further from the origin are those that manifest greater research relevance. The two phases that have the most significant impact are those in which researchers seek to reflect the needs of practice. First, researchers may seek the help of practitioners in identifying research problems that are grounded in current practice and deal with important issues, rather than being driven by curiosity or opportunity. Second, researchers interested in establishing the relevance of their research communicate their research findings through accessible channels in a way that is easily understandable to practitioners. Between these two phases, there are several phases (theoretical development, research methodology, and data analysis) that may be largely invisible to practitioners.4 Hence these are the phases in which researchers should seek rigor.
It is immediately apparent from Table 3 that there is a plethora of recommendations for how to overcome the perceived problems with IS research from the viewpoint of its impact on practice in each of these phases. By far the greatest number is focused on communicating research findings. These recommendations take two forms: producing consumable research reports and expanding publication outlets. We also identified a number of suggestions for increasing research relevance in the identification of the research topic, theoretical development, and research methodology phases of the life cycle. Recommendations for increasing the relevance of data analysis were almost nonexistent, perhaps because data analysis is not controversial.
Improving Research Relevance Using Applicability Checks

In the sections that follow, we distinguish between the applicability check approach, which presents the generic solution
Note that collecting data from professionals does little to improve research relevance.
approach we are proposing, and the applicability check method, which presents a detailed process for conducting and evaluating applicability checks. In this section, we first motivate the use of applicability checks as an additional step at either the beginning or the end of a research life cycle. Given that applicability checks may not be appropriate in all circumstances, we then characterize the situations in which their use is appropriate.
Theoretical Foundations of Applicability Checks

The dimensions of research relevance and their potential solutions are the key to using applicability checks. From the perspective of the dimensions of research relevance, we sought to identify the dimension that needs to be satisfied for us to even concern ourselves with the relevance of the research conducted. It is our belief that the crucial conditions for producing relevant research are, first, whether the research is important to practice, and second, whether the research ultimately could be suitable for applying in practice. Accessibility, which refers largely to the presentation style rather than the substance of the research, is not critical to the research per se, though, of course, it is critical to communicating it. Further, suitability of the research to practice becomes relevant only when the research has already been deemed to be important to practitioners. Hence we focus here on the importance of research to practice. From the perspective of choosing a solution that evaluates the importance of research to practice, the institutional and project governance perspectives presented in Figure 1 are not under the control of the researcher, and do not play a large role in current IS research. We therefore consider solutions that the researcher can control and that can impact research today. These are solutions based in the research process perspective. Perusal of Figure 2 shows that our research life cycle does not address whether the research is important to practice. We propose addressing this dimension by evaluating the importance of the research object under investigation in a further step in the life cycle. A major advantage is that such an approach would leave untouched the traditional research life cycle, with its emphasis on rigor, thereby avoiding major discomfort for those researchers reluctant to seek relevance due to the perceived concomitant reduction in rigor. We therefore need to develop a way to determine the importance of research to practice.
In Figure 3, we show an applicability check occurring between two instantiations of the research life cycle. Such a check could therefore be conducted as (1) the final step in the research life cycle; (2) the first step in the life cycle; or (3) at the end of the first instantiation of the life cycle, leading into the second instantiation. The first possibility focuses on research just concluded. The second possibility is based on the existence of relevant prior research outcomes. The second and third possibilities both focus on the early phases of a research project, including the formulation of a new model, etc., in a complete research life cycle. In these instances, the applicability check would therefore lead to better research models, the development of more appropriate hypotheses, and so on. Our emphasis on evaluating research objects means that our primary focus in this context is on the theory development phase of the research life cycle. Hence, applicability checks per se may have only an incremental impact on the identification of the research problem, for example.
When to Conduct an Applicability Check

There are two situations to consider in establishing when to conduct applicability checks: (1) timing in the context of the extended research life cycle (process perspective) and (2) the types of research that lend themselves to evaluation via applicability checks (cross-sectional perspective).
Timing of an Applicability Check in the Extended Research Life Cycle

From a process perspective, there is a continuum of possibilities for when an applicability check that follows a traditional research life cycle can be conducted; note, therefore, that this consideration applies to possibilities one and three as presented in the discussion of the extended research life cycle in Figure 3. The continuum is determined by the level of maturity of the research object at the time the applicability check is conducted. For example, at one extreme, researchers could conduct applicability checks on research objects as soon as their findings are known, and therefore prior to publication, in which case there might be no publication to be shared with practitioners. At the other extreme, applicability checks could be conducted on mature IS models such as DeLone and McLeans (1992, 2003) IS success model. The timing of the applicability check has important implications for the relevance dimensions that researchers can evaluate. An applicability check conducted on mature, published research, can evaluate a research objects importance,
Relevance
Identification of the Research Problem Research Methodology Theoretical Development Communication of the Findings
Applicability Check
Identification of the Research Problem Communication of the Findings
Research Methodology Theoretical Development
Data Analysis
Data Analysis
Research Lifecycle 1 Rigor
Research Lifecycle 2
Time
Figure 3. The Extended Research Life Cycle
accessibility, and suitability to practitioners. On the other hand, lack of related publications when conducting an applicability check on a newly established research object would preclude the practitioner audience from evaluating the articles accessibility.
Boundaries of Applicability Checks

We now examine the types of research objects that are, or are not, suitable for examination using applicability checks. First, in suggesting that applicability checks should be conducted on research objects, we are making the implicit assumption that we are interested in examining theory-focused research and not research that is descriptive in nature or that addresses meta-issues within the IS community such as research on the IS academic discipline itself (see, for example, Vessey et al. 2002). Second, when the research objects are extremely complex (for example, deeply theoretical or mathematical in nature) or are not specified sufficiently well, they may not be appropriate to examine via applicability checks. Hence prior to conducting an applicability check, the research object may require prior elaboration by a research and development group or professional boundary spanners such as consultants. Third, applicability checks are unnecessary in studies that are, in some sense, sanctioned by practice. One such type of
study is action research, which involves the conduct of a realworld practitioner project by a team comprised of both researchers and practitioners (see, for example, Avison et al. 1999; Baskerville and Myers 2004; Davison et al. 2004; Kock et al. 2002). The role of the researchers is to apply and further develop theory that provides direction for the project. Hence, the symbiotic nature of the intervention results in continuous feedback between researchers and practitioners rendering an applicability check unnecessary. Fourth, we now consider the relevance of applicability checks in intensive and case study research. In seeking to understand a real-world phenomenon, research in behavioral IS typically produces or uses abstractions presented in the form of theories, models, frameworks, processes, or technical artifacts that describe that phenomenon. While this may be the traditional realm of hypothetico-deductive (positivist) research, it is also true of intensive research (Markus and Lee 1999), irrespective of whether its origins are positivist or interpretive in nature. Intensive studies engage the researcher in an on-going, indepth change process using methods such as ethnography, grounded theory, and participant observation, and thereby provide the researcher with ample opportunity to augment his/her confidence in what is to be reported. However, such studies most often also produce research objects, typically in the form of models (see, for example, Schultze and Orlikow-
10
ski 2004). Hence applicability checks could also be used to evaluate the research objects produced in these types of studies. We can apply a similar argument to case study research. In examining a real-world phenomenon using a traditional case study approach (for example, Yin 1994), engagement with practice usually ceases with the presentation of a case writeup that is assessed by the chief project liaison to establish the validity of the data. This data can then be used in academic pursuits such as theory development (see, for example, Eisenhardt 1989). However, the resulting theory often is not presented to the organization for evaluation of the theoretical model/constructs that result and their potential impact on the organization. We suggest, therefore, that applicability checks also be conducted on the objects resulting from traditional case study research.
that has a high level of maturity, participants would first present participants with published materials and formally assess these dimensions prior to conducting the applicability check proper. We indicate further such differences where necessary during the presentation of the method itself. Third, whether it is conducted at the beginning or the end of the research life cycle (see Figure 3), the applicability check method should leave the research object unchanged. Hence, action research, which takes place in tandem with a real-world intervention, is not an appropriate approach for conducting an applicability check. Fourth, an applicability check conducted following a research life cycle should be timely so that it does not increase the risk of contributing to outdated research outcomes.5 The applicability check should therefore simply be another step in the research life cycle rather than requiring a further research life cycle for its conduct. This suggests that research methods such as case studies, experiments, field experiments, and questionnaires are not appropriate for conducting an applicability check. Fifth, the requirement for timeliness further suggests that an applicability check that uses individuals as the unit of analysis may be too time-consuming. Hence a method that results in group feedback from a number of practitioners would be most appropriate. Further, group feedback has a number of advantages over individual feedback including the opportunity to react to and build on the responses of others (Stewart et al. 2007, pp. 46-47). Sixth, researchers should conduct the applicability check using relevant (and unbiased) practitioners as subjects. The interests of the subject group should therefore be appropriate to the focus of the research object under investigation; for example, CIOs are the appropriate subject group when the research object focuses on IS strategy.
The Applicability Check Method

In presenting a method for conducting applicability checks, we first need to specify the requirements of such a method. We then justify and present the specifications for our proposed solution approach.
Requirements for an Applicability Check Method

The nature and role of an applicability check suggests that it should meet a number of criteria. First, because practitioners are most often unfamiliar with academic research, it is important that group members are both aware of and understand the research being examined prior to taking part in the group discussion. Because it would not be possible to both inform the group members of the research under investigation and to conduct a meaningful intervention to meet the specified objectives, the method should include an initial exposure of the participants to the research object and its associated objectives and implications. Second, the requirement to familiarize practitioners with the research object under investigation requires the preparation of supporting materials. For applicability checks on research objects at all levels of maturity, researchers should prepare a set of materials describing the context, objectives, and expected utility of the research (no more than five pages in length). These may be the only materials presented to participants in an applicability check on a research object of low maturity. However, if researchers desire to evaluate the importance, accessibility, and suitability of a research object
The Proposed Applicability Check Solution

We first justify our selection of focus groups and a modified nominal group technique (NGT) as the foundation for our applicability check method. We then specify the method in a series of seven steps, as shown in Table 4. To ease understanding of the material that follows, we present a brief outline of those steps.
Further, note that, if appropriate, the conduct of an applicability check could lead to earlier commercialization of the research object than might otherwise have been the case.
11
(1) Planning the applicability check: The research object under consideration, objectives of the applicability check, and information needed are clearly specified. Further, the research question clearly identifies the research objective, target population, and specific issues to be addressed. (2) Selecting the person to conduct the check: The person conducting the check has in-depth knowledge of the research under investigation, as well as having significant social skills. (3) Ensuring that participants are familiar with the research object under examination: Participants in the check are provided with materials that introduce the research, the research object under investigation, and the implications of the object from the viewpoint of practice. Prior to the check proper, the person conducting the check ensures that each participant is sufficiently well informed to take part in the projected evaluation. (4) Designing the materials for conducting the check: The applicability check method adheres to wellestablished design criteria. The format of the pretested questions for the study, the sequence of their presentation, and an agenda that can fit into the time allotted are specified. (5) Establishing an appropriate environment for conducting the check: The person conducting the check creates an environment for running the check that is conducive to a fruitful interaction. (6) Conducting the check: The person conducting the check presents the agenda and the ground rules for conducting the check, then ensures that the check is conducted in a professional manner that results in unbiased input from all participants. (7) Analyzing the data: Procedures for analyzing qualitative data are used to analyze the data derived from the check. Multiple sources of evidence are used, data is coded, and a trail of evidence is provided from the raw data to the final outcomes.
Stewart et al. 2007). The requirement not met by focus groups is that for initial knowledge sharing between researcher and practitioner regarding the research object, its objectives, and implications. We propose that this phase be accomplished using the nominal group technique modified for use in combination with focus groups (Stewart et al. 2007, pp. 153-154). It would be implemented as a single step in the conduct of the applicability check method (Step 3), prior to the conduct of the focus group proper. The focus group method, which is a qualitative approach to behavioral science research, consists of group interviews that involve a small number of appropriate persons discussing the topics raised by a moderator who guides the interview process (Morgan 1998, p. 1). Focus groups have been widely used in a variety of disciplines such as marketing, clinical psychology, sociology and social psychology, and communication, among others (Stewart et al. 2007, pp. ix). Because participating group members will be selected IS practitioners with a common understanding of the issue addressed in the research, the groups will be quite homogeneous and the applicability check can therefore be more focused than the majority of typical focus groups (Stewart et al. 2007). The modified nominal group technique (NGT) consists of either interviews with non-colocated individuals and subsequent sharing of summaries of responses and ideas of other team members with the group (Stewart et al. 2007, pp. 153154), or face-to-face discussion followed by individual, private responses to the questions of interest (Delbecq et al. 1975). Because the modified NGT can be used without direct interaction among group members, we propose using it to aid focus group participants in gaining an initial understanding of the research under investigation prior to examining the issues in a face-to-face focus group setting. We also propose evaluating the familiarity of the participants selected with the research object under investigation prior to their participation in the check proper. Appendix A presents an initial example of using an applicability check to evaluate the relevance to practitioners of a popular IS research model, DeLone and McLeans IS success model (see, also, Rosemann and Vessey 2005). Our experiences with this exploratory study, which used focus groups alone, led us to introduce the modified NGT to allow applicability check participants to gain exposure to the research object of interest.
Requirements for the Applicability Check Solution

The focus group method meets all but one of the above criteria for an applicability check method. We therefore propose using focus groups as the foundation for conducting an applicability check (Krueger and Casey 2000; Morgan 1998;
Specifications for the Applicability Check Method

Our approach to developing a method for conducting applicability checks is analogous to that of Davison et al. (2004) in
12
Table 4. Specifications for the Applicability Check Method

1. Planning the Applicability Check 1a. Was the purpose of the research clearly defined in terms of the outcomes to be achieved and the information that is needed to examine those outcomes? 1b. Did the research question clearly identify the topic of the research, the population that is relevant to the question, and the specific issues of interest? 1c. Did the group consist of representative, unbiased members of the population under investigation? Selecting the Applicability Check Moderator 2a. Was the moderator a member of the research team? 2b. Did the moderator selected have a supportive leadership style? 2c. Did the moderator selected have empathy, an ability to listen, and to ask appropriate follow-up questions? 2d. Did the moderator selected ensure that s/he did not bias the responses of the participants? Ensuring Familiarity with the Research Under Investigation 3a. Did all group members receive copies of the research summary and associated questions? 3b. Did all group members share feedback on the research under investigation with the applicability check moderator? 3c. Were all group members sufficiently prepared to participate in the face-to-face part of the applicability check method? Designing the Applicability Check Interview Guide 4a. Did the interview questions relate directly to the stated purpose of the focus group? 4b. Were the questions largely open-ended (unstructured), unambiguous, and singular in purpose? 4c. Did the broader questions precede more probing questions? 4d. Did the more important questions appear near the beginning of the interview guide? 4e. Did the interview guide consist of less than 12 questions? 4f. Could the questions be addressed effectively within a two-hour period? 4g. Was the interview guide tested prior to use? Establishing the Applicability Check Environment 5a. Did the available facilities provide an environment appropriate to the conduct of an applicability check (for example, space, noise, white boards, recording, equipment, and so on)? 5b. Was there an initial meet-and-greet period prior to the formal focus group interaction? 5c. Were the less talkative group members, as observed during the meet-and-greet period, placed directly opposite the moderator and the more talkative members adjacent to the moderator? 5d. Were the names of individuals displayed on name tents? 5e. Did the focus group members sign human subjects consent forms / ethical clearance forms? 5f. Were the focus group members told that the discussion would be recorded and that all comments would remain confidential? Conducting the Applicability Check 6a. Did the moderator present the objectives of the group, the agenda for the discussion, and outline the ground rules to the group members? 6b. Did the moderator build rapport within the group by asking a factual question, for example, one related to the members place of employment? 6c. Did the moderator encourage all focus group members to share their thoughts? 6d. Did the moderator manage the time available in such a way that issues were covered effectively? 6e. Did the moderator probe for more specific information without influencing the response when s/he felt that the focus group member wished to elaborate further on a specific issue? 6f. Did the moderator deal effectively with any issues that arose during the focus group? Analyzing the Applicability Check Data 7a. Assuming multiple data analysis, was the analysis strategy documented, understood, and able to be clearly articulated by each member of the research team? 7b. Did the analysis procedure follow a sequential process involving field notes and recordings, oral summaries that verified key points that arose during the focus group, debriefing with the moderator team immediately following the group, and transcripts (if used)? 7c. Was there sufficient data to provide a trail of evidence, thereby ensuring that the analysis was verifiable (reliable)? 7d. Was the analysis completed following each focus group so that questions or the approach could be modified prior to the next focus group? 7e. Was the analysis team sufficiently skeptical with regard to the possibility of social desirability influences, groupthink, and the dominance of a focus group member or members? 7f. Was the reliability of the analysis established by using two coders to identify major themes and issues?
2.
3.
4.
5.
6.
7.
13
their development of a method for conducting canonical action research.6 We present our method as a set of principles that form the foundations of the method, each of which is further differentiated in terms of a set of criteria for conducting and evaluating each step. Hence these principles and criteria should be used both as guidelines by researchers using applicability checks to help establish in a rigorous manner the relevance of research to practitioners and as evaluation criteria by reviewers wishing to evaluate the conduct of an applicability check (see Markus and Lee 1999). Table 4 presents our method for conducting an applicability check based on a focus group method with the addition of a modified NGT to familiarize participants with the research object under investigation.
Using such approaches would complement the methods based on the research process that we use here. With respect to the applicability check method itself, the preliminary applicability check we conducted on DeLone and McLeans IS success model (see Appendix A) served as the first step in developing the method. The method we present here should now be evaluated by conducting checks on research objects in the context of on-going research. Such checks would consider every aspect of the check from issues as diverse as the accessibility or understandability of the materials used to support the check, the effectiveness of the moderator, the effectiveness of both the modified nominal group and focus group processes, among others. Further, it is possible that, as researchers gain experience with both the approach and the method itself, the applicability check method might be better differentiated to present methods specific to different types of research objects. Third, from the viewpoint of publication, the findings of applicability checks should be reported in traditional academic publications as well as in publications intended for practitioners (such as MIS Quarterly Executive). To further motivate researchers to focus on the need for research relevance, requirements for conducting applicability checks and other similar approaches to evaluating and/or improving research relevance could be incorporated into publication criteria, for example, by modifying both the editorial policies of journals and the evaluation forms completed by reviewers. Fourth, from the viewpoint of relationships between academia and industry, the increased awareness of IS research resulting from the conduct of applicability checks should ultimately encourage industry to increase its support for IS research. As well as further increasing the opportunity to improve the relevance of our research, it would raise the level of industry-supported research funding. Hence the conduct of applicability checks would, in this way, have a positive effect on one of the other potential solutions we highlighted earlier for achieving research relevance, the project governance perspective. The notion of applicability checks introduces new and potentially important avenues for research and, in addition to improving the relevance of research to practice, could become a distinct research area in its own right. From a disciplinary perspective, we also note that applicability checks are generalizable to the work of other academic communities that seek to be relevant to practice, rather than specifically to the IS academic community. Nonetheless, our primary hope is that the IS researchers embrace our suggestions regarding the conduct of applicability checks in their quest to improve the practical relevance of their research.
Implications
Using applicability checks as a way of improving research relevance has several implications. First, we envisage that researchers may need to expend considerable effort in recruiting appropriate applicability check participants. However, the fact that the knowledge gained from involvement in applicability checks would benefit practitioners by exposing them to new, potentially important research that could be of direct value to them, research that they would not otherwise have the opportunity of being exposed to for months or perhaps even years, should encourage participation. Further, recruitment would become much easier if the value of applicability checks were to become known to the practitioner community. A source of valuable knowledge for enrolling participants is the focus group literature, which places significant emphasis on the selection and recruitment process (see, for example, Krueger and Casey 2000; Morgan 1998; Stewart et al. 2007). Second, there are two implications for future research on applicability checks. With respect to the applicability check approach, we note that it is just one of a number of solutions that could be used to improve the relevance of research to the practitioner community. Future research could investigate other methods for improving research relevance, which, like applicability checks, are focused on the research process. Other approaches also might target different phases of the research life cycle rather than the theoretical development on which we focus here. Finally, institutional and project-level approaches could also be used to improve research relevance.
Note that the applicability check method could also be formalized using design science principles (Hevner et al. 2004).
14
Acknowledgments
The authors wish to thank Izak Benbasat and Lynne Markus, and the editor-in-chief, Carol Saunders, and the two reviewers for their insightful comments on earlier versions of this manuscript.
References
Applegate, L. M., and King, J. L. 1999. Rigor and Relevance: Careers on the Line, MIS Quarterly (23:1), pp. 17-18. Avison, D., Lau, F., Myers, M., and Nielsen, P. A. 1999. Action Research, Communications of the ACM (42:1), pp. 94-97. Baskerville, R., and Myers, M. D. 2004. Special Issue on Action Research in Information Systems: Making IS Research Relevant to Practice. Foreword, MIS Quarterly (28:3), pp. 329-335. Benbasat, I., and Zmud, R. W. 1999. Empirical Research in Information Systems: The Practice of Relevance, MIS Quarterly (23:1), pp. 3-16. Benbasat, I., and Weber, R. 1996. Rethinking Diversity in Information Systems Research, Information Systems Research (7:4), pp. 389-399. Cohen, J. 1960. A Coefficient of Agreement for Nominal Scales, Educational and Psychological Measurement (20), pp. 37-46. Davenport, T. H., and Markus, L. M. 1999. Rigor and Relevance Revisited: Response to Benbasat and Zmud, MIS Quarterly (23:1), pp. 19-23. Davis, F. D. 1989. Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology, MIS Quarterly (13:3), pp. 319-340. Davison, R. M, Martinsons, M.G., and Kock, N. 2004. Principles of Canonical Action Research, Information Systems Journal (14), pp. 65-86 Delbecq, A. L., Van de Ven, A. H., and Gustafson, D. H. 1975. Guidelines for Conducting NGT Meetings, in Group Techniques for Program Planning, A. L. Delbecq, A. H. Van der Ven, and D. H. Gustafson, (eds.), Glenview, IL: Scott, Foresman, 1975, pp. 40-66. DeLone, W. H., and McLean, E. R. 1992. Information Systems Success: The Quest for the Dependent Variable, Information Systems Research (3:1), pp. 60-95. DeLone, W. H., and McLean, E. R. 2003. The DeLone and McLean Model of Information Systems Success: A Ten-Year Update, Journal of Management Information Systems (19:4), pp. 9-30. Dennis, A. R., Valacich, J. S., Fuller, M. A., Schneider, C. 2006. Research Standards for Promotion and Tenure in Information Systems, MIS Quarterly (30:1), pp. 1-12. DeSanctis, G., and Poole, M. S. 1994. Capturing the Complexity in Advanced Technology Use: Adaptive Structuration Theory, Organization Science (5:2), pp. 121-147. DEST. 2006. Research Quality Framework. Assessing the Quality and Impact of Research in Australia: Research Impact, Canberra: Commonwealth of Australia.
Eisenhardt, K. M. 1989. Building Theories from Case Study Research, Academy of Management Review (14), pp. 532-550. Fllman, D., and Grnland, A. 2002. Rigor and Relevance Remodeled, in Proceedings of the 25th Information Systems Research Seminar in Scandinavia, K. Bdker, M. K. Pedersen, J. Nrbjerg, J. Simonsen, and M. T. Vendel (eds.), Bautajj, Denmark, August 10-13. Galliers, R. D. 1994. Relevance and Rigour in Information Systems Research: Some Personal Reflections on Issues Facing the Information Systems Research Community, in Business Process Re-Engineering, Proceedings of the IFIP TC8 Open Conference on Business Process Re-Engineering, B. C. Glasson, I. T. Hawryszkiewycz, B. A. Underwood, and R. A. Weber (eds.), Queensland, Australia, pp. 93-101. Hevner, A. R., March, S. T., Park, J., and Ram, S. 2004. Design Science in Information Systems Research, MIS Quarterly (28:1), pp. 75-105. Introna, L. D. and Whittaker, L. 2004. Journals, Truth and Politics: The Case of MIS Quarterly, in Information Systems Research: Relevant Theory and Informed Practice, B. Kaplan, D. P. Truex III, A. T. Wood-Harper, and J. I. DeGross (eds.), Boston: Kluwer Academic Publishers, pp. 103-120. Kavan, C. B. 1998. Profit through Knowledge: The Application of Academic Research to Information Technology Organizations, Information Resources Management Journal (11:1), Winter, pp. 17-22. Klein, G., Jiang, J. J., and Saunders, C. 2006. Leading the Horse to Water Communications of the AIS (18), pp. 259-274. Kock, N., Gray, P., Hoving, R., Klein, H., Myers, M., and Rockart, J. 2002. IS Research Relevance Revisited: Subtle Accomplishment, Unfulfilled Promise, or Serial Hypocrisy?, Communications of the AIS (8), pp. 330-346. Krueger, R. A. and Casey, M. A. 2000. Focus Groups: A Practical Guide for Applied Research (3rd ed.), Thousand Oaks, CA: Sage Publications. Lyttinen, K. 1999. Empirical Research in IS: On the Relevance of Practice in Thinking of IS Research, MIS Quarterly (23:1), pp. 25-28. Mandviwalla, M., and Gray, P. 1998. Is IS Research on GSS Relevant, Information Resources Management Journal (11:1), Winter, pp. 29-37. Markus, L. M., and Lee, A. S. 1999. Special Issue on Intensive Research Information Systems: Using Qualitative, Interpretive, and Case Methods to Study Information Technology Foreward, MIS Quarterly, (23:1), 1999, 35-38. Miles, M. B., and Huberman, A. M. 1994. Qualitative Data Analysis (2nd ed.), Thousand Oaks, CA: Sage Publications. Moody, D. L. 2000. Building Links between IS Research and Professional Practice: Improving the Relevance and Impact of IS Research, in Proceedings of the 20th International Conference on Information Systems, W. J. Orlikowski, S. Ang, P. Weill, H. C. Krcmar, and J. I. DeGross (eds.), Brisbane, Australia, December 10-13, pp. 351-360. Moody, D. L., and Shanks, G. 2000. Using On-Line Medical Knowledge to Support Clinical Decision Making: Towards
15
Evidence-Based Practice, in Proceedings of the IFIP Working Group 8.3 Conference on Decision Support through Knowledge Management, Stockholm, July. Morgan, D. L. 1998. The Focus Group Guidebook: Focus Group Kit 1, Thousand Oaks, CA: Sage Publications. Poole, M. S., and DeSanctis, G. 1990. Understanding the Use of Group Decision Support Systems: The Theory of Adaptive Structuration, in Organizations and Communication Technology, J. Fulk and C. Steinfeld (eds.), Newbury Park, CA: Sage Publications, pp. 173-193. Robey, D., and Markus, L. M. 1998. Beyond Rigor and Relevance: Producing Consumable Research about Information Systems, Information Resources Management Journal (11:1), Winter, pp. 7-15. Rosemann, M., and Vessey, I. 2005. Linking Theory and Practice: Performing a Reality Check on a Model of IS Success, in Proceedings of the 13th European Conference on Information Systems, D. Bartmann, F. Rajola, J. Kallinikos, D. Avison, R. Winter, P. Ein-Dor, J. Becker, F. Bodendorf, and C. Weinhardt (eds.), Regensburg, Germany, May 26-28. Saunders, C. 1998. The Role of Business in IT Research, Information Resources Management Journal (11:1), Winter, pp. 4-6. Schultze, U., and Orlikowski, W. 2004. A Practice Perspective on Technology-Mediated Network Relations: The Use of InternetBased Self-Serve Technologies, Information Systems Research (15:1), pp. 87-106. Sedera, D., Gable, G., and Chan. T. 2004. Measuring Enterprise Systems Success: The Importance of a Multiple Stakeholder Perspective, in Proceedings of the 12th European Conference on Information Systems, T. Leino, T. Saarinen, and S. Klein (eds.), Turku, Finland. Senn, J. 1998. The Challenge of Relating IS Research to Practice, Information Resources Management Journal (11:1), Winter, pp. 23-28. Stewart, D. W., Shamdasani, P. N., and Rook, D. W. 2007. Focus Groups: Theory and Practice. Thousand Oaks, CA: Sage Publications. Strauss, A., and Corbin, J. 1998. Basics of Qualitative Research Techniques and Procedures for Developing Grounded Theory (2nd ed.), Newbury Park, CA: Sage Publications.
Vessey, I., Ramesh, V., and Glass, R. L. 2002. Research in Information Systems: An Empirical Study of Diversity in the Discipline and Its Journals, Journal of Management Information Systems (19:2), pp. 129-174. Yin, R. K. 1994. Case Study Research: Design and Methods. Thousand Oaks, CA: Sage Publications, 1994.
About the Authors

Michael Rosemann is Professor of Information Systems and coleader of the Business Process Management Group at Queensland University of Technology, Brisbane. He received his MBA and Ph.D. in Information Systems from the University of Muenster, Germany. His main areas of interest are business process management, process modeling, enterprise systems, and ontologies. He has published more than 120 refereed papers including publications in journals such as Information Systems, IEEE Transactions on Knowledge and Data Engineering, European Journal of Information Systems, Decision Support Systems, and Information Systems Frontiers. Besides more than 40 journal publications, 70 conference publications, and 35 book chapters, Michael is the author or editor of five books. He is a member of the editorial boards of six journals and a member of the Australian Research Council College of Experts. Iris Vessey is Honorary Professor at the University of Queensland and Adjunct Professor at the Queensland University of Technology. She received her M.Sc., MBA, and Ph.D. in Management Information Systems from the University of Queensland, Australia. Her research interests focus on the evaluation of emerging information technologies, the management and organization of enterprise systems, and knowledge management strategies. She serves, or has served, as an associate editor at Information Systems Research, Journal of Database Management, Journal of Management Information Systems, MIS Quarterly, and Management Science, and serves on the executive boards of Information Systems Frontiers and International Journal of Information Systems and Management. During the first 8 years of its life, Iris served as secretary of the Association for Information System, as well as the International Conference on Information Systems following its merger with AIS. She is an inaugural Fellow of the AIS.
16
Appendix A
An Exploratory Applicability Check on DeLone and McLeans IS Success Model
We conducted an applicability check on DeLone and McLeans (1993, 2003) IS success model (see, also, Rosemann and Vessey 2005). This model addresses an issue crucial to business, is backed by significant research effort, and has been validated by further research following its initial formulation (DeLone and McLean 2003). Hence it is an effective model for assessing the reaction of the IS practitioner community to a significant body of mature IS academic research.
DeLone and McLeans Model of IS Success

DeLone and McLeans (1992) IS success model summarized and structured the extant literature on IS success into an overall model designed to capture the relevant aspects of the success of IS in practice, as well as the inter-relationships among them. The authors published a revised model in 2003 (see Figure A1). The second article, which examined just those studies on IS success that appeared following the publication of the first paper, reported that there are almost 300 articles in refereed journals that have referred to, and made use of, this IS success model. Hence, a substantial amount of the blood, sweat, and tears of IS research has been expended in trying to determine what constitutes IS success. We might therefore ask ourselves the question: Is DeLone and McLeans IS success model accessible to IS practitioners? If so, is it perceived as important for and applicable to the challenges of these IS practitioners?
The Applicability Check

The innovative nature of an applicability check required an exploratory approach to examine whether the approach was feasible. We chose to explore the use of focus groups in this first attempt at conducting an applicability check. This section details the conduct of that applicability check and presents the major findings.
Use of Focus Groups

The focus group participants we sought all had prior relationships with universities and therefore had a general appreciation for academic research. Because consultants have considerable influence on corporate IS activities, we performed our applicability check using IS consultants as well as corporate IS personnel. To increase the homogeneity of the focus groups, we conducted two separate sessions. The focus groups, which were conducted in Australia, lasted approximately 2.5 hours because the participants had to be exposed to the model under investigation as well as responding to the applicability check issues per se. The authors served as focus group moderators. We opened each focus group with the question: How do you measure the success of your most complex information systems? We then introduced DeLone and McLeans model to the group and answered any related questions. Finally, we compared the criteria identified in the focus group with DeLone and McLeans established academic model. The final question we addressed was: Would you consider using this model to measure IS success in your organization? If not, why not?
Data Analysis
We found that the directions taken by our two focus groups were quite sensitive to differences in the responsibilities of the representatives, both between and within groups. We therefore describe the findings for each of the groups separately and then compare them to DeLone and McLeans model. Focus Group I: Corporate Personnel. The three corporate IS personnel who formed the first focus group had quite diverse backgrounds, coming from the utility and finance industries, and from the public sector.
17
Information Quality Intention to Use System Quality Use Net Benefits
User Satisfaction Service Quality
Figure A1. DeLone and McLeans Revised Model of IS Success (2003)
The focus group member from the public sector was the manager of IT operations. Not surprisingly, therefore, his focus was on issues such as continuity measures and service desk response. He also believed that end-to-end project tracking from initiation through operation is an important success criterion. The focus group member from the finance industry was the organizations process manager. From a technical perspective, this participant viewed infrastructure up-time (availability) and service requests (help desk) as important criteria. From a business perspective, he believed that net business value both from a technical perspective (cost to keep a system running, including licensing) and a business perspective (developers meeting users expectations), as well as the business view of the performance of a system, should be assessed. The third focus group member was a Group Process Manager at a utility company. His first focus was on project success (on budget, on time) and on the benefits derived from projects 3 to 6 months following implementation. Important criteria for the latter were reduction in head count, throughput, time of process execution, as well as net benefits. He also stressed the fact that levels of system availability beyond the essential are a cost to the company. Other measures on which he focused included business operational support and service for operations, as well as service for all of the IT components of the technical infrastructure. Finally, he indicated that end-user satisfaction is a measure that his company monitors. Following sharing and discussion of the success criteria used in their three organizations, the three participants grouped their criteria and prepared a model that reflected their view of the success criteria. The group focused on achieving net business value or project benefits as the ultimate goal for the success of a system. User satisfaction was also perceived as a valid indicator. The conspicuous differences of this model from those of DeLone and McLean lie, however, in the emphasis on infrastructure, project, and operational issues, factors well beyond DeLone and McLeans focus on the success of a system. Operational effectiveness, measured using service level agreements (SLAs), was seen as another measure of system success. The infrastructure has an impact on the SLAs, as well as on the net business value of the project. Independent of these system-oriented criteria, criteria related to the execution of the IS project were identified as being important, including delivering on-time, on-budget, and in the requested quality. Focus Group II: Consultants. Two of the three consultants involved in the second focus group worked for large, global consulting companies. The third consultant, who focused on web-based solutions, formed part of a much smaller, global consulting firm. The group of consultants was surprisingly homogeneous in their comments on how they currently measure the success of information systems. Hence we present the findings for this group in terms of their major success criteria, which clustered as follows: Financial measures such as return on investment, net present value, shareholder value, cost savings. Productivity measures, such as reduction in headcount, etc. Comparisons of individually defined key performance indicators before and after the project.
18
Client satisfaction with the system; a related issue is the customer/partner/supplier happiness factor, as one participant phrased it. User-related measures of personal satisfaction such as user acceptance, user empowerment, and improved task skills. System-related measures such as accessible information and system responsiveness.
During the focus group, it became clear that the participants had difficulty differentiating between the success of an IT application as opposed to the success of the related project. It appears likely that this focus, which is evidenced in the statement below, is shaped by the consultants role in conducting implementation projects. We only engage in such implementation projects if they also provide our own organization with benefits. This includes not only financial benefits, but also a referencable client;7 and the consultants also have to improve their knowledge. It has to be a true win-win situation. A further suggested measure was the project-focused delivery upon project milestones. Comparison of IS Success Model with Models from Practice. What can practice learn from DeLone and McLeans model? First, probably the factor that was addressed least (and/or least well) by practice was intention to use/use, a factor that was all but absent among those of our six participants. One consultant referred to a related factor as workers support of the systems implemented; not trying to find a way around it. Nonetheless, the participants appeared intrigued with the notion and showed interest in potentially applying it in their corporations. However, they also saw potential issues in its measurement. Second, our participants did not refer to information quality in their initial assessment of success factors. When it was brought to their attention in DeLone and McLeans model, they commented that the effect of information quality can be compounded across a number of systems. Hence, the lack of attention to information quality appeared to be an oversight. Third, service quality was not used by our respondents in the way in which it was intended by DeLone and McLean, that is, as a measure of the service provided to those employees engaging in end-user computing. It does seem to us that systems and services are different, and distinct, entities and that their success should be evaluated using models designed specifically for each. Hence, we believe that the inclusion of service quality in DeLone and McLeans model is problematic. Next we present some of the factors elicited by our focus group participants that are missing from DeLone and McLeans model. First, both groups focused to a considerable extent on the project that delivered the systems under discussion, in terms of cost, time, and quality. While this is not surprising in the case of the consultants, the emphasis given to the project by corporate IS personnel was also substantial. These observations suggest that DeLone and McLeans model should be extended to include the influence of project quality on the ultimate success of the system. Second, both focus groups placed considerable emphasis on service delivery based on both systems and technology (the infrastructure), issues that are largely absent from DeLone and McLeans model. Again, this observation suggests that DeLone and McLeans model needs to be extended to include operational issues such as these.
Discussion
Here we evaluate the applicability check we conducted on DeLone and McLeans IS success model in terms of its performance on the dimensions of relevance: importance, accessibility, and suitability. This is followed by the implications of applicability checks and recommendations for future research. Discussion of Findings Based on Relevance Dimensions. With respect to importance, and therefore the fit, of the research to practice, all of the focus group participants perceived the model as important. Clearly, measuring the success of information systems is a timeless challenge, and one that has not been addressed satisfactorily to date. Most focus group participants confirmed the fit of factors such as information quality, system quality, operational service quality (viewed largely as the quality of the help desk), user satisfaction, and net benefits (sometimes using different terminology) to IS success before we exposed them to DeLone and McLeans IS success model. The notion of intention to use was
A referencable client is one that the company can use for promotion purposes.
19
new for all the practitioners and was viewed with interest. Furthermore, participants from both focus groups wanted a stronger recognition of the characteristics and success of the project leading to the implementation of the system. With respect to accessibility, none of the participants had heard about the model before or read any of the academic journals in which the model and its revisions had been published. It was satisfying, therefore, to observe that all participants could see the merit in DeLone and McLeans model; for example, they all requested the original papers. We also helped the practitioners involved to understand the model, which they perceived as largely intuitive. There was a certain challenge in both groups in understanding the semantics of the arrows linking the factors. While there might be issues in understanding the presentation of the related models, we do not see major issues in understanding their core. Nevertheless, we see lack of accessibility as the first critical hurdle in the transfer of DeLone and McLeans IS success model into practice. This is a concern because DeLone and McLeans model is one of the most widely discussed models in the IS community and related articles have been published in a wide variety of academic journals. With respect to suitability, some of the consultants were interested in using the model, but it is just an academic paper, as one participant phrased it, clearly distinguishing suitability from applicability. When the model is so successful in academia, why did nobody develop a tool based on this model that facilitates its application in practice? The consultants perceived as a major downside the fact that a potentially interesting and mature model could not be easily applied to their current challenges. The major limitation of our study is the preliminary nature of the applicability check conducted. Our objective was to provide initial feedback on the feasibility of conducting such checks and to illustrate the type of information that might be forthcoming. The methodology used for this initial test was therefore exploratory in nature. Implications for DeLone and McLeans IS Success Model. Based on our findings, we see potential for future research on DeLone and McLeans success model in several areas. First, further research needs to be conducted on the link between system and project success, a link identified by all participants and not addressed in DeLone and McLeans model. There was consensus that a system can be perceived as successful, but not the related project, and vice versa. This issue is, of course, more relevant in the early phases of the system life cycle. Nevertheless, we believe that the interrelationships between system and project success deserve further attention. Second, the feedback from the two types of focus group members (users and consultants) highlighted the different perceptions of these stakeholders. We see potential for more research into the development of a better understanding of the role of the stakeholder in evaluating IS success (see, for example, the work by Sedera et al. 2004). Third, the service quality dimension was ignored by our focus group participants based on lack of relevance to the issue at hand, that of, evaluating systems. Fourth, there is clearly an opportunity to derive automated solutions based on IS research. A solution for DeLone and McLeans success model could, for example, be made available via a simple web service, which would guide the user through an assessment of the success of their systems based on the model. Hence, such a solution would facilitate the immediate application of the model in practice.
Implications for an Applicability Check Method

We engaged in an exploratory investigation of the feasibility of conducting an applicability check on the relevance of DeLone and McLeans model of IS success to practice. In this regard, our research contributes to both the research methodology for conducting such a check and to illustrating the type of detailed information that an applicability check on the theories or models that are the subject of the check can provide. We learned from this experience that 1. 2. focus groups are an appropriate technique for conducting applicability checks, but that we needed to seek ways to use the time devoted to the group sessions more effectively. The two focus groups conducted here took 2.5 hours because we both introduced participants to the model and sought their responses to it during that time frame. We believed, following our experience here, that the group time could be shortened if the participants were exposed to the research object in an independent intervention prior to conducting the applicability check proper. Using this approach would also allow us to ensure that each participant fully understood the research object under investigation.
20
Appendix B
Details of Steps in the Applicability Check Method
We now present the seven steps of the applicability check method in more detail.
Step 1: Planning the Applicability Check

Effective planning is the key to conducting a successful applicability check. The objective of the applicability check must be clearly defined; that is, the research object under consideration must be clearly specified, as must the objectives of the applicability check, and the information that is needed to satisfy those objectives (1a). Further, the research question must clearly identify the objective, the target population, and the specific issues to be addressed (1b). Prior research suggests that focus groups are best conducted with 6 to 12 unbiased participants with interests in the type of research outcomes under investigation (Stewart et al. 2007, p. 58). The target audience would, for example, be users in the case of adoption or interface research, and CIOs in the case of research at the top organizational levels. Researchers should also keep in mind that both corporate IS professionals and consultants can offer legitimate, though varying, perspectives (see Rosemann and Vessey 2005). Further, three or four focus groups are necessary with each type of participant to reach saturation, that is, when no new information emerges (Krueger and Casey 2000). If saturation is not reached with three or four groups, additional groups need to be scheduled. Hence, this issue will result in a decision being made with regard to the number and composition of focus groups to be conducted (1c).
Step 2: Selecting the Applicability Check Moderator

Because of the focused, and specialized, nature of the research under investigation, the most appropriate moderator for the applicability check will most likely be a member of the research team (2a). The only situation in which this might not be true is when the applicability check of a mature research object is conducted by researchers who are not the authors of the research. Moderating focus groups requires specific social interaction skills. The moderator must balance the requirements of sensitivity and empathy, on the one hand, and objectivity and detachment, on the other (Stewart et al. 2007, p. 69). For example, the moderator must be supportive of the contributions of the group (2b), ask follow-up questions when appropriate (2c), and take care not to bias the responses given by participants either during the interaction or in summarizing the key points (2d).
Step 3: Ensuring Familiarity with the Research Under Investigation

This step is achieved via the modified NGT identified earlier as an essential part of the applicability check method. The moderator presents group members with a research summary and related materials that introduce the research, the research object under investigation, and the implications of the object from the viewpoint of practice (3a). Such materials also contain an initial set of questions so that the group members can actively engage with the materials. Questions are included in an attempt to ensure a certain level of preparedness for participation in the subsequent focus group. The moderator should schedule a follow-up phone call (or other mode of communication) at the time the materials are distributed, and just prior to the conduct of the focus group (3b) to ensure that the individual has sufficient understanding of the research to play a role in its evaluation (3c). Hence the moderator plays the role of boundary spanner in interpreting the research objects for the practitioners.
Step 4: Designing the Applicability Check Interview Guide

The applicability check method adheres to well-established criteria used in the design and format of questions for focus groups (4a, 4b), the sequence of their presentation (4c, 4d), as well as ensuring that the agenda is covered in the time allotted (4e, 4f) (see, for example, Krueger and Casey 2000; Morgan 1998; Stewart et al. 2007). Note, also, the need to pretest the focus group questions (4g).
Step 5: Establishing the Applicability Check Environment

It is important that an appropriate environment be prepared to facilitate group interaction in the face-to-face portion of the applicability check. First, appropriate facilities must be provided so that logistical issues do not constrain the conduct of the applicability check (5a). Second, the focus group members should be introduced in an informal setting prior to beginning the formal interaction (5b). This time should contribute
21
to making participants comfortable with the environment, as well as allowing the moderator to observe people who are likely to dominate, or shrink from contributing to, the group discussion. Third, the moderator should place group members so as to minimize the potential contribution of dominant members (adjacent) and maximize the potential contribution of those who are more reticent (opposite) (5c). Fourth, displaying participants names also facilitates the group interaction (5d). Fifth, the rules for obtaining ethical clearance should also be satisfied (5e). Finally, conditions for the focus group should be communicated to the group members; for example, participants should be told that their discussion will be recorded, that their comments will remain confidential, and that only the groups aggregated comments will be reported (5f).
Step 6: Conducting the Applicability Check

To ensure that participants are fully aware of the procedures to be followed, the moderator should commence the focus group by presenting the agenda for the discussion and outlining the ground rules to the participants (6a). Criteria 6b to 6f reflect the need for the moderator to encourage participation from each group member (6b and 6c) so as to ensure that there is no bias in the group findings, to address the issues at hand, and to ask probing questions when necessary to derive the information required (6e) (for some examples, see Klein et al. 2006), and to deal with any conflicts or issues that might arise (6f), all in the time set aside for the discussion (6d).
Step 7: Analyzing the Applicability Check Data

The criteria for conducting an appropriate analysis of the data obtained from the focus group are generic to the analysis of qualitative data (see, for example, Miles and Huberman 1994; Strauss and Corbin 1998). If multiple team members are involved in the data analysis, it is important that each clearly understands the strategy to be used in that analysis (7a). The steps in the analysis should be laid out formally and include the different types of data that might be collected: field notes, recordings, moderator summaries presenting key points, debriefing notes, and transcripts if used (7b). The basic aim is to ensure that a trail of evidence exists from what was shared in the focus group to the formal outputs of the group interaction (7c). If multiple focus groups are conducted, then the lessons learned in prior groups should be incorporated into the questions for subsequent groups (7d). It is also important for the analysis team to be aware of and actively seek evidence for such issues as social desirability bias, groupthink, and the contributions of dominant focus group members (7e). Finally, the reliability of formal coding should be assessed using multiple coders and adjustment made for chance agreements (7f) (see Cohen 1960).
22

Rose Mann Vess Ey

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Rose Mann Vess Ey

Uploaded by

Copyright:

Available Formats

Rosemann & Vessey/Improving the Relevance of IS Research

ISSUES AND OPINIONS

MIS Quarterly Vol. 32 No. 1, pp. 1-22/March 2008

Rosemann & Vessey/Improving the Relevance of IS Research

MIS Quarterly Vol. 32 No. 1/March 2008

Rosemann & Vessey/Improving the Relevance of IS Research

On Research Relevance and Potential Solutions

Dimensions of Research Relevance

Addressing the Relevance of Research to Practice

MIS Quarterly Vol. 32 No. 1/March 2008

Rosemann & Vessey/Improving the Relevance of IS Research

Project Governance Perspective

Research Process Perspective

Figure 1. Solutions for Improving Research Relevance

MIS Quarterly Vol. 32 No. 1/March 2008

Rosemann & Vessey/Improving the Relevance of IS Research

MIS Quarterly Vol. 32 No. 1/March 2008

Rosemann & Vessey/Improving the Relevance of IS Research

Table 2. Examples of Types of Research Governance

Professional Societies National and International Funding Bodies

Figure 2. The Research Life Cycle

MIS Quarterly Vol. 32 No. 1/March 2008

Rosemann & Vessey/Improving the Relevance of IS Research

MIS Quarterly Vol. 32 No. 1/March 2008

Rosemann & Vessey/Improving the Relevance of IS Research

Improving Research Relevance Using Applicability Checks

MIS Quarterly Vol. 32 No. 1/March 2008

Rosemann & Vessey/Improving the Relevance of IS Research

Theoretical Foundations of Applicability Checks

When to Conduct an Applicability Check

Timing of an Applicability Check in the Extended Research Life Cycle

MIS Quarterly Vol. 32 No. 1/March 2008

Rosemann & Vessey/Improving the Relevance of IS Research

Identification of the Research Problem Communication of the Findings

Research Methodology Theoretical Development

Research Lifecycle 1 Rigor

Figure 3. The Extended Research Life Cycle

Boundaries of Applicability Checks

MIS Quarterly Vol. 32 No. 1/March 2008

Rosemann & Vessey/Improving the Relevance of IS Research

The Applicability Check Method

Requirements for an Applicability Check Method

The Proposed Applicability Check Solution

MIS Quarterly Vol. 32 No. 1/March 2008

Rosemann & Vessey/Improving the Relevance of IS Research

Requirements for the Applicability Check Solution

Specifications for the Applicability Check Method

MIS Quarterly Vol. 32 No. 1/March 2008

Rosemann & Vessey/Improving the Relevance of IS Research

Table 4. Specifications for the Applicability Check Method

MIS Quarterly Vol. 32 No. 1/March 2008

Rosemann & Vessey/Improving the Relevance of IS Research

MIS Quarterly Vol. 32 No. 1/March 2008

Rosemann & Vessey/Improving the Relevance of IS Research

MIS Quarterly Vol. 32 No. 1/March 2008

Rosemann & Vessey/Improving the Relevance of IS Research

About the Authors

MIS Quarterly Vol. 32 No. 1/March 2008

Rosemann & Vessey/Improving the Relevance of IS Research

DeLone and McLeans Model of IS Success

The Applicability Check

Use of Focus Groups

MIS Quarterly Vol. 32 No. 1/March 2008

Rosemann & Vessey/Improving the Relevance of IS Research