TELEMEDICINE JOURNAL

Volume 2, Number 4, 1996

Mary Ann Liebert, Inc.

Health Systems Evaluation of Telemedicine:

A Staged Approach

HALLIE . DeCHANT, M.D., M.P.P.,* WALID G. TOHME, Ph.D.,+

SEONG K. MUN, Ph.D.,+ WENDELIN S. HAYES, D.O.,t and KEVIN A. SCHULMAN, M.D.*

ABSTRACT

Telemedicine promises greater access to health care of higher quality, potentially at lower
cost. The diverse applications of telemedicine technology developed to date have not been
evaluated systematically in terms of their ability to achieve these goals. Furthermore, the great
variety in telemedicine applications and the far-reaching consequences of new information
systems for health care delivery pose challenges to traditional methods of technology as¬
sessment. Methods appropriate for mature technologies may not be suitable for emerging
ones and, indeed, may risk stifling their development with premature negative conclusions.
The staged approach to technology assessment proposed here matches the analysis to the tech¬
nology's stage of development. It focuses on access, quality, and cost and considers the com¬
munication pathway employed in the telemedicine application. A staged approach to tech¬
nology assessment can inform and foster the development of new telemedicine technology
while allowing health care delivery systems to make rational decisions about adopting
telemedicine.

INTRODUCTION transform fundamentally the structure and

term "telemedicine" encompasses a
The wide range of communication technologies
that promise to improve access to medical care
and efficiency in its provision. It is increasingly
clear that we need to understand these tech¬
nologies and their impact on the health care
system better before we commit the substan¬
tial resources required to implement telemed¬
icine on a broader basis. Such an assessment
includes an understanding of technical capa¬
bilities, impact on patient care, and costs of
telemedicine relative to current practices.
Because telemedicine exemplifies the poten¬
tial of new information-processing systems to
process of medical care, adequate analysis of
telemedicine must go beyond the usual per¬
spectives of technology assessment. Technology
that can alter the entire paradigm of an enter¬
prise must be analyzed at a system level; that is,
by assessing the technology's effect on an entire
managed care organization or an integrated
health care delivery system. Thus, technology
assessment of telemedicine should take a broad
enough perspective to include the ways in
which modern information systems foster the
development of novel, efficient organizational
structures and processes of care delivery.
In this paper, we outline a staged approach
to technology assessment that can address both

"Clinical Economics Research Unit, Division of General Internal Medicine and tlmaging Science and Information
Systems Center, Department of Radiology, Georgetown University Medical Center, Washington, D.C.
303
304 DeCHANT et al.

the evaluation of individual telemedicine ap¬
plications and the potential of telemedicine to
instigate profound system change. This staged
approach also reduces the risk that premature
negative judgments will curtail the develop¬
ment of promising new technologies.
CHALLENGES OF TECHNOLOGY
ASSESSMENT
A comprehensive health technology assess¬
ment evaluates its object along dimensions of
technical capability, efficacy in clinical use, ef¬
fect on patient outcomes, and social priority.
Fuchs and Garber1 describe three stages of
technology assessment, which focus in turn on:
(1) technical characteristics; (2) efficacy (per¬
formance under optimal, controlled condi¬
tions), often in randomized clinical trials using
intermediate endpoints; and (3) clinical (pa¬
tient), social, and economic outcomes. Other
authors have recommended approaches that
follow the same broad outlines.2"6 Most evalu¬
ations of telemedicine technologies performed
to date fall into the first or second stages of
Fuchs and Garber's classification.
Assessment of emerging technologies poses
particular challenges and risks. Unfavorable
assessment of early prototypes of the tech¬
nologies may stifle development of more ef¬
fective versions.7 Conversely, failure to assess In the U.S., the Food and Drug Administration
technology as it develops may result in diffu¬
sion before its capabilities, limitations, and
costs are understood. Although the greater role
of managed care organizations in the health
care delivery system is changing the reim¬
bursement incentives for adoption of new tech¬
nology, market incentives in the past have
tended to favor early dissemination of tech¬
nology. Appropriate evaluation is especially
challenging in the case of information systems
sessment is not to limit the development of
technologies, but to curtail the diffusion of un-
proven orineffective ones. Technology assess¬
ment can be used to speed adoption of tech¬
nologies that improve the efficiency or
outcomes of the health care system.2 Finally,
technology assessment should be a core com¬
ponent of a business case study that examines
the investment in a technology and outlines the
costs and the outcomes to be expected from its
use. This type of assessment can be used to
make purchase or investment decisions or to
reevaluate those decisions after the technology
has been implemented to determine if it is
meeting the returns projected for the invest¬
ment. If the technology is not performing, an
application can be modified to help achieve ex¬
pected goals, further investment decisions can
be modified, or new strategies can be selected.
A STAGED APPROACH: TECHNOLOGY
ASSESSMENT IN PHARMACEUTICAL
DEVELOPMENT
Technology assessment has been applied ex¬
tensively to medical interventions such as
pharmaceutical products. Many countries, in¬
cluding the United States, have adopted ex¬
plicit guidelines for demonstrating the efficacy
of new drugs as a prerequisite for marketing.8
has developed a staged model of drug evalua¬
tion that moves from animal (preclinical) stud¬
ies, to pharmacokinetic studies in normal indi¬
viduals (Phase I), to dosing studies in diseased
individuals (Phase II), to comparisons against
standard therapy for diseased individuals
(Phase III).8 Other countries have also adopted
guidelines for evaluating the cost-effectiveness
of new drugs.9'10 The results of such analyses
are used to make decisions regarding price and

technology, which has diverse applications and reimbursement for new therapies.
potentially widespread effects. A staged approach to the evaluation of phar¬
Technology assessment can help guide maceutical products ensures safety and efficacy
rather than stunt the development of promis¬ at each stage before a medication can be dis¬
ing new technologies. First, it provides a frame¬ seminated to larger numbers of individuals.
work to help those who develop new tech¬ We propose an analogous set of stages for eval¬
nologies understand whether their system will uation of new technologies with potentially
address the needs of consumers and users. wide-ranging ramifications. In each stage of
Second, the focus of current technology as¬ this analysis, the evaluation is tailored to the
STAGED APPROACH TO EVALUATION 305

technology's state of development. The results method of technology assessment but extends
can be used to improve the technology before it in a way that captures telemedicine's poten¬
wider dissemination occurs. tial to produce system-wide change. In each
stage, the issue of interest is the effect of
telemedicine applications on the quality, ac¬
A STAGED APPROACH TO cessibility, or cost of medical care. In some
EVALUATION OF TELEMEDICINE stages, only individual components of each
endpoint are relevant; in other stages, the goal
One problem in assessing a telemedicine ap¬ is to integrate all three components into a
plication appropriately is that the impact it has global assessment of the technology. The par¬
on the health care delivery system may extend ties involved in the communication pathway
well beyond the original purpose for which it help to determine which endpoints are most
was designed. For example, by improving effi¬ salient at each stage.
ciency, telemedicine consultation may allow In what follows, we explore the operational-
care providers to spend more time with all pa¬ ization of the concepts "quality," "accessibil¬
tients, including those who do not require ity," and "cost" in the assessment of medical
telemedicine consultation. Many managed care technology.
organizations are focusing on the development Defining quality and then measuring it are
of information infrastructure and electronic complex tasks. Donabedian11 has proposed
medical records as a means of improving the that quality in medical care has two compo¬
efficiency of care, increasing the accountability nents: technical quality, judged relative to the
of care providers, and ensuring the quality of best in current knowledge and technology, and
care delivered to patients. Telemedicine appli¬ interpersonal quality, which uses individual
cations may play an important role within and social standards to judge performance in
these information systems, either as compo¬ such areas as privacy, confidentiality, informed
nents of the overall information infrastructure choice, and empathy. Brook12 has offered a
or as separate efforts designed to meet specific slightly different formulation of quality in
needs. medical care as having three components: ap¬
Given the potential of telemedicine to affect propriateness of care, technical competence,
several aspects of health care delivery simulta¬ and human dignity. Measurement of quality in
neously, we need to understand how it can be clinical applications of telemedicine therefore
used to improve the efficiency of health care requires attention to technical ability, the judg¬
delivery systems such as managed care orga¬ ment displayed in decisions to apply telemed¬
nizations, integrated delivery systems com¬ icine to a particular clinical situation, and its ef¬
prising a hospital and several outpatient cen¬ fect on the interpersonal aspects of the medical
ters, or systems that provide health care to encounter. The latter are best reflected in mea¬
military personnel. Technology should be sures of patient and provider satisfaction, al¬
viewed as a structural component of the sys¬ though these are also affected by the patient's
tem delivering care. Endpoints for a health sys¬ clinical outcome and therefore not entirely sep¬
tems assessment would be broad-based per¬ arable from technical ability and appropriate¬
formance measures, including efficiency in the ness.
delivery of care or outcomes of care for a fixed Accessibility has many possible meanings in
investment. the context of telemedicine, each of which may
Technology assessment should address both have different implications for cost and qual¬
the technical performance of specific applica¬ ity. Does the telemedicine application improve
tions and the overall effect of introducing them access to all types of providers? To specialists
into a health care delivery system. To accom¬ or primary care providers? If the latter, to pri¬

plish both tasks, we propose a staged approach mary care physicians or to nurse practitioners
in which the type of assessment that is appro¬ and physicians' assistants? Telemedicine back¬
priate varies with the stage of the technology's up that allows physician assistants or nurse
evolution. Our approach parallels the classic practitioners to practice in geographic areas
306
that have few or no physicians or that allows
physicians in such areas to obtain consultations
from specialists could improve access in both
time and space by reducing the need for travel.
Such increased access may improve the qual¬
ity of care but also increase the volume of ser¬
vices provided.
From a policy standpoint, in order to deter¬
mine whether a given improvement in access
is worthwhile, it is necessary to measure the
costs incurred, the costs avoided, and the net
effect on quality and access. These costs should
not be limited to the price paid to install
telemedicine systems but should include the
opportunity cost of other investments forgone
in favor of telemedicine. Gains in quality and
accessibility achieved for a given cost consti¬
tute measures of efficiency. Cosis of medical care
can be direct medical (the costs of medical ser¬
vices), direct nonmedicai (the costs of receiving
medical services such as the cost of trans¬
portation to a provider), indirect (the costs of
morbidity and mortality from a disease), or in¬
tangible (the cost of pain and suffering related
to the disease or treatment).13 Telemedicine can
affect any or all of these costs. The cost and cost-
effectiveness studies in the literature so far do
not address costs on this systematic basis and
often have a limited perspective, such as the
cost to a provider of setting up a telemedicine
application.
The outcome of primary importance in any
particular analysis vary depending both on
will
the perspective chosen and on the parties in¬
volved in the transfer of information. For a
managed care organization hiring specialist
physicians, the most important effect of a
telemedicine application that allows remote
consultation may be to improve the efficiency
of care. For patients, the primary consideration
for the same telemedicine project may be to im¬
prove access. Certain outcome domains will
also assume greater importance in particular
communication paths (physician to physician,
physician to allied health professional, or
health professional to patient). For a telemedi¬
cine application in which communication oc¬
curs between physicians (for example, consul¬
tation between a specialist and a generalist
physician), the primary concern in evaluation
may be the quality of care rather than access or
DeCHANT ET AL.
cost. On the other hand, telemedicine applica¬
tions that facilitate communication between
physicians and allied health professionals or
directly between physicians and patients may
be evaluated primarily on the basis of their ef¬
fect on access. Every analysis of a telemedicine
project should clearly identify both the path¬
way of communication and the analytic per¬
spective.
Our staged approach is outlined in Figure 1.
It incorporates an evaluation of each telemed¬
icine application along three axes: the stage ap¬
propriate to the technology's stage of devel¬
opment, the outcome domain(s) of interest,
and the communication pathway followed. To
motivate the description, we will refer
throughout to an example of technology
assessment of telemedicine applied to home
health care.
Utilization of home health care services has
increased in recent years as reimbursement
policies have favored earlier discharge of pa¬
tients from the hospital and expanded cover¬
age for home care.14 Between 1988 and 1994,
Medicare reimbursement for home health care
increased from $2 billion to $12.7. billion;
Medicare patients not enrolled in managed care
received 160 million home health care visits in
1993.14 Both for patients recently discharged
from the hospital and for those whose chronic
illness makes travel to physicians' offices diffi¬
cult, telemedicine has the potential to facilitate
physiologic monitoring, as well as communi¬
cation among physicians, allied health profes¬
sionals, patients, and families.15"17
A prototype application is the home telecare
project of the European prototype for
Integrated Care initiative to facilitate delivery
of health care and social services to elderly and
chronically ill individuals in the community.
The project has been implemented on a trial ba¬
sis in Northern Ireland and Spain.16 The home
telecare management system can monitor
physiologic variables such as blood pressure
and ECG tracings and transmit these data via
modem and public telephone lines to a central
workstation. A database contains information
about the patients' clinical history and relevant
emergency contacts or community resources.
When an alarm is triggered, either voluntarily
by patients or in response to abnormal physi-
STAGED APPROACH TO EVALUATION 307

Evaluation Method

Experimental
-
Stage I: Technical Efficacy
Individual Endpoints Assess accuracy, reliability

Specific Applications Stage II: Specific System Objectives

Assess single endpoints in domains of access, quality or cost
_

-
Stage III: System Analysis
Global Endpoints Assess global impact on access, quality and cost for system

System Effects Stage IV: External Validity

Assess global impact on access, quality and cost
_
in a different system
V
Observational

FIG. 1. Overview of staged approach to evaluation of telemedicine.

ologic data, the central workstation is dialed nology, radiology is well suited for evaluation,
automatically. An observer at the central work¬ because the plain radiograph can be considered
station contacts the patient and arranges ap¬ a gold standard for measurement of sensitivity
propriate management using information from and specificity (the true-positive rate and true-
the database. In what follows, we describe the negative rate). Nonetheless, only a few studies
evaluation that would be appropriate for this have used a blinded design to compare the ac¬
prototype at various stages of development curacy of radiologists' interpretation of digi¬
and dissemination. tally transmitted images with those of film ra¬
diographs.18'19,26 More typically, published
evaluations consist of "before and after" stud¬
Stage I: Technical Efficacy (Prove Capabilities of ies20'21 or case series.22,27
the Technology)
These early studies of telemedicine are nec¬
In the first stage, the role of technology as¬ essary before proceeding to evaluation of their
sessment is to evaluate the technical capabili¬ effects on patient care. They are analogous to
ties of the technology. This would define the Phase I studies of new drugs, which measure
technical efficacy of the technology in question: the doses needed to achieve specific blood con¬
its ability to transmit accurate information re¬ centrations but do not consider the effect of
liably. In order of increasing sophistication, those blood levels on the condition of interest.
studies of efficacy include case reports, case se¬ Accuracy and reliability form part of the "tech¬
ries, receiver operating characteristic (ROC) nical ability" definition of quality in medical
analyses, and comparisons of efficacy between care. Until a telemedicine application can per¬
telemedicine and existing practices. Most of the form its intended task accurately and reliably,
early studies in telemedicine are analyses of there is no need to investigate its effects on the
technical efficacy and clinical feasibility.18"25 quality, accessibility, or efficiency of medical
This is especially true for teleradiology, the care.
most extensively evaluated form of telemedi¬ When applied to the home telecare system,
cine to date. As a long-distance diagnostic tech¬ Stage I analysis would evaluate its technical
308
performance. The questions of concern would
be whether the telecare system performs reli¬
ably and delivers accurate information. The
analysis would measure the variance among
several measurements of the same variable
(replicability or precision), as well as the agree¬
ment between the mean test result and the true
value of the variable being measured (reliabil¬
ity)28 using randomized comparisons against a
gold standard, which will usually be conven¬
tional technology. These tests should be per¬
formed for the following functions: (Stage III).
Examples of Stage II studies from the litera¬
• Detection of vital signs (ECG, blood pres¬ ture include two proposals for cost and effi¬
sure) compared with standard ECG and ciency evaluations of picture archiving and
manual blood pressure management;
•
Monitoring and transmission of vital
signs, including appropriate triggering of
alarms;
• Accurate transmission of all diagnostic
data.
The technical capacity of the system should
also be measured in terms of its:
•
Ability to support interface requirements,
throughput, compatibility, and expand¬
ability;
•
Speed of information processing;
• Data security.
Having demonstrated that the home telecare
system can detect and transmit information
about the patient's condition accurately, the
next step is to assess the effects of such a sys¬
tem on quality (effect on functional outcomes
and interpersonal aspects of health care, ap¬
propriateness of use, and technical ability), ac¬
cessibility, and cost.
Stage II: Prove that Technology Can Meet
Specific System Objectives
The second stage evaluates the effect of a
technically sound new technology on broader
(but still narrowly defined) endpoints. As for
all health care applications, the domains of con¬
cern are quality, access, and cost. Because these
evaluations address technologies at an early
stage of development and diffusion, it is im¬
portant to be able to assess endpoints with rel¬
DeCHANT ET AL.
atively small sample sizes and within a rea¬
sonably short period of time. Therefore, even
though all three outcome domains are always
of interest, it may not be feasible to evaluate all
three domains simultaneously. A staged ap¬
proach could use small studies with limited
endpoints in single domains to refine certain
aspects of the technology before further devel¬
opment and dissemination. These preliminary
studies and product refinement should precede
the global assessment of the new technology
communications systems (PACS).29'30 One
study29 plans to measure the change in re¬
source utilization that follows introduction of
a PACS for the intensive care unit of a univer¬
sity hospital, while the other30 plans to mea¬
sure changes in the time taken to report radi¬
ology results before and after introduction of
PACS. Each study chooses a single endpoint
rather than an integrated assessment of the
technology's effect. In addition to technical ef¬
ficacy, another group of investigators31 devel¬
oped protocols to evaluate specific clinical end-
points (defined as intermediate endpoints such
as changes in therapeutic plans after telemedi¬
cine consultations) for different applications.
A Stage II analysis of the hypothetical home
telecare application would measure its effect in
a single domain, such as access to medical care
for patients who are chronically ill or who have
recently been discharged from the hospital.
This analysis could use either randomized or
quasiexperimental designs, although the for¬
mer would have greater internal validity. In a
controlled experiment, patients would be ran¬
domly assigned to experimental or control
groups, whereas in a quasiexperiment, patients
using home telecare might be compared with
historical controls.32 Accessibility would be
measured in terms of the time taken to initiate
treatment following changes in physiologic sta¬
tus or the number of physician contacts within
a given period. A study of the quality of home
telecare would have measures of functional
performance, quality life, mortality, read-
of
missions to the hospital, and patient satisfac¬
tion. A study of home telecare costs would
STAGED APPROACH TO EVALUATION 309

compare the cost of telemedicine with that of because they can obtain specialist consultation
conventional arrangements. more easily, access and satisfaction may im¬
prove for all patients. If greater ease of consul¬
tations causes physicians to request them more
Stage III: System Assessment (Global Evaluation often, would any improvement in patient out¬
Using Multiple Endpoints) comes that followed justify the cost of addi¬
In the third stage, technology assessment tional consultations? The goal of a systems as¬
should evaluate performance in all three do¬ sessment is to provide a summary evaluation
mains—quality, access, and cost—in order to for establishing priorities among alternatives.
understand the overall effect on the health care Because Stage III evaluations attempt to mea¬
delivery system. Stage III analyses should be sure an organization-wide effect, the unit of
conducted for mature applications. Hence, the analysis is the health care delivery system.
number of patients should be large enough to Randomized designs will be difficult to imple¬
permit simultaneous analysis of multiple end- ment except in organizations such as multistate
points across the domains of access, quality, and managed care systems that can randomly as¬
cost. Moreover, Stage III should address the im¬ sign large administrative units to telemedicine
pact of the telemedicine application on the or conventional practices.
health care delivery system, not just those com¬ A Stage III analysis of the hypothetical home
ponents directly involved in the application. telecare application would evaluate efficiency,
Because sample size (or power) calculations defined as the incremental improvement in
focus on a single measurement, investigators quality and accessibility at a given cost. A
will need to develop measurements for all three health care delivery system that serves chron¬
domains to ensure an adequate sample size for ically ill or recently discharged patients via
a Stage III evaluation. In the assessment of home telecare would compare its current per¬
home telecare, for example, the sample size for formance with historical precedents or a large
assessing an endpoint in access to care might administrative unit that uses conventional care.
exceed that for endpoints in quality and cost. The purpose of this type of analysis is to de¬
The sample size should then be based on that termine whether real-time information about
required for assessment of access. Detection of the vital signs of these patients and rapid ac¬
an effect may also be hindered by "noise" in¬ cess to their medical histories can be obtained
troduced by multiple processes within a health and also:
care system. However, this is a less significant

problem when the effect is large.
A comprehensive evaluation is necessary
when tradeoffs exist between individual do¬
mains. For example, telemedicine may improve
accessibility and quality but increase the cost
as well.33 Conversely, telemedicine may im¬
prove access and reduce costs but at the ex¬
1. Allows more individuals with illness of a
given severity to contact physicians more
quickly (access);
2. Results in higher rates of survival, better
functional outcomes, or greater satisfac¬
tion for patients who were chronically ill
or had recently been discharged from the

pense of lower patient and provider satisfac¬ hospital (quality);

tion compared with conventional face-to-face 3. Affects the cost of treating or transporting
care. Within a health care delivery system, patients with chronic illness or those who
telemedicine applications may affect personnel had been admitted to hospitals, compared
costs by permitting a given level of care to be with previous methods (cost).
provided by fewer physicians, while the at
same time necessitating expenditures for train¬
IV: Establishing External Validity
ing to implement the system. A comprehensive Stage Evaluation to Different Systems)
evaluation is also necessary because telemedi¬ (Extending
cine may affect the delivery of health care to The fourth stage of technology assessment ex¬
patients who do not use it directly. If physi¬ amines whether the overall evaluation of a tech¬
cians spend more time with all their patients nology in one system such as an academic med-
310
ical center and its affiliated clinics, applies to
other settings, such as managed care organiza¬
tions. This stage recognizes the unique require¬
ments and objectives of individual systems for
appropriate evaluations. The fourth stage is also
the most challenging and is rarely performed. In
fact, few technologies have been subjected even
to the sort of comprehensive evaluation de¬
scribed in Stage . The experimental design
concerns are the same as those in Stage III.
Systems that have multiple large administrative
units can randomly assign them to telemedicine
or conventional practice, while others must rely
on quasiexperimental designs.32
The questions of concern in Stage IV are sim¬
ilar to those in Stage III:
1. Does monitoring the physiological status
of chronically ill or recently discharged
patients via telemedicine allow physicians
to respond more quickly to significant
changes than do traditional arrange¬
ments? (access);
2. Do patient outcomes improve, in terms of
mortality, functional status, or satisfaction
with telemedicine? (quality);
3. How does the use of home telecare affect
the cost of care compared with standard
approaches? (cost).
APPLICATIONS OF THE STAGED
APPROACH
In summary, it is proposed that technology
assessments evaluate telemedicine along three
axes: the stage appropriate to the
application's
level of development, the communication path
in the application, and the domain(s) of inter¬
est. The three axes can be conceived of as the
edges of cube
a (Fig. 2). Thus, for Stage I ap¬
plications, the domain of interest will be tech¬
nical quality. For Stage II applications, quality,
access, or cost may be primary. Which domain
takes precedence will depend on the perspec¬
tive of the analysis (from the viewpoint of the
patient, the provider, or the payer) and on the
parties involved in the communication path¬
way. For Stages III and IV, a comprehensive
evaluation is indicated, with the most relevant
domain serving as the primary study endpoint.
DeCHANT ET AL.
Rather than combining the three outcome
domains (access, quality, and cost) into a sin¬
gle metric, which would necessitate weighting
them, it may be useful simply to report the out¬
come within each domain, in a manner analo¬
gous to the use of cost consequence analysis in
comparing the cost of alternative methods for
achieving a particular health outcome.9
The framework we propose could facilitate
assessment of existing telemedicine projects
and also guide reporting of future studies.
Authors could make it easier for readers to un¬
derstand the contribution of a new study by
describing the stage of the evaluation, the pri¬
mary communication path in the telemedicine
application under discussion, and the out¬
come domains considered. It may be helpful
to provide further detail within each of the cat¬
egories suggested. For example, the outcome
domains of interest in physician-to-physician
communications may differ if both parties are
specialists (quality > access) or if one is a gen-
eralist (access>quality). If the outcome do¬
mains are weighted, an explicit discussion of
weighting procedures should be included in
the analysis.
A staged approach to technology assessment
parallels and informs the evolution of emerg¬
ing technologies. Rather than performing a
Stage III analysis before the technology's abil¬
ity to achieve individual system objectives is
established, the staged approach indicates that
the technology should be modified to meet
baseline specifications before it is applied more
widely and before a more global judgment is
made about its adoption. Technology assess¬
ment can thus guide and foster the develop¬
ment of promising technologies rather than
compromise innovation. Our approach com¬
plements the efforts of researchers who have
proposed that clinical applications of telemed¬
icine be grouped into categories of like
processes to reduce the number of analyses
necessary.34"36
Technology assessment provides a rational
basis for systems' decisions about adopting
new technologies, whether those systems are
private organizations such as managed care
plans or public organizations such as Medicare
or military medical systems. However, these
adoption decisions are encountered only after
STAGED APPROACH TO EVALUATION
Any Health Professional
Patient
Communication
Pathways ~/Allied Health
Physician-
Professional
Physician-Physician
Access
Outcome
Domains Quality
Cost
12
Analytic Stage
FIG. 2. Dimensions of health
a product is marketed, ideally after Stage III in
the proposed model. In a broader policy sense,
technology assessment provides a basis for al¬
locating society's resources to particular health
care or other uses.
CONCLUSION
Telemedicine offers the possibility of elimi¬
nating distance as a barrier to medical care.
Whether improving access to care in this way
enhances the quality of medical care and the
efficiency of its delivery to a degree commen¬
surate with the cost of the technology remains
to be established. To evaluate this promising
technology systematically, we have suggested
a staged approach that incorporates consider¬
ations of access, quality, and cost in assessing
discrete applications of telemedicine and also
considers the implications of telemedicine for
the overall efficiency and quality of the health
care delivery system. By establishing a frame¬
work for evaluation at all stages of evolution,
this staged approach can inform and foster the
development of new telemedicine technolo¬
gies.
311
3 4
systems analysis of telemedicine.
ACKNOWLEDGMENTS
This work is sponsored in part by the
Department of the Army, Contract No. DAMD
17-94-V-4015. The content of the information
does not necessarily reflect the position or pol¬
icy of the U.S. Government, and no official en¬
dorsement should be inferred. Dr. DeChant
was supported by HRSA NRSA Training Grant
No. T32 PEI 0017 during the preparation of this
paper. The authors thank the editor and two
anonymous reviewers for their helpful com¬
ments.
REFERENCES
1. FuchsVR, Garber AM. The new technology assess¬
ment. Engl } Med 1990;323:673-677.
2. Laupacis A, Feeny D, Detsky AS, Tugwell PX. How
attractive does a new technology have to be to war¬
rant adoption and utilization? tentative guidelines for
using clinical and economic evaluations. Can Med
Assoc ] 1992;146:473^81.
3. Guyatt G, Drummond M, Feeny D, Tugwell P,
Stoddart G, Haynes RB, Bennett K, Labelle R.
Guidelines for the clinical and economic evaluation of
health care technologies. Soc Sci Med 1986;22:393-408.
4. Drummond MF, Stoddart GL, Torrance GW. Methods
312
for the economic evaluation of health care programmes.
Oxford: Oxford University Press, 1987.
5. Eisenberg JM, Schulman KA, Glick HA, Koffer H.
Pharmacoeconomics: economic evaluation of phar¬
maceuticals. In: Strom (ed): Pharmacoepidemiology,
ed 2. Chichester: John Wiley & Sons, 1994:469-194.
6. Task Force on Principles for Economic Analysis of
Health Care Technology. Economic analysis of health
care technology: a report on principles. Ann Intern
Med 1995;123:61-70.
7. Bashshur R. On the definition and evaluation of
telemedicine. Telemed J 1995;1:19-29.
8. Strom BL. What is pharmacoepidemiology In: Strom
BL (ed): Pharmacoeconomics, ed 2. Chichester: John
Wiley and Sons, 1994:3-14.
9. Canadian Coordinating Office for Health Technology
Assessment. Guidelines for economic evaluation of phar¬
maceuticals: Canada. Ottawa: The Office, 1994.
10. Commonwealth Department of Human Services and
Health. Guidelines for the pharmaceutical industry on prepa¬
ration of submissions to the Pharmaceutical Benefits Advisory
Committee: including major submissions involving economic
analyses. Canberra: Australian Government Publishing
Service, 1995.
11. Donabedian A. The epidemiology of quality. Inquiry
1985;22:282-292.
12. Brook RH. Quality of care: do we care? Ann Intern
Med 1991;115:486-190.
13. Eisenberg JM, Schulman KA, Glick H, Koffer .
Pharmacoeconomics: economic evaluation of pharma¬
ceuticals In: Strom BL (ed): Pharmacoeconomics, ed 2.
Chichester: John Wiley and Sons, 1994:469-193.
14. Welch HG, Wennberg DE, Welch WP. The use of
Medicare home health services. Engl } Med
1996;335:324-329.
15. Grigsby J, Kaehny MM, Sandberg EJ, Schlenker RE,
Shaughnessy PW. Effects and effectiveness of telemed¬
icine. Health Care Financing Review 1995;17:115-131.
16. Rodriguez MJ, Arreedondo MT, de Pozo F, Gomez EJ,
Martinez A, Dopico A. A home telecare management
system. / Telemed Telecare 1995;1:86-94.
17. Shultz EK, Bauman AA, Hayward AM, Holzman AR.
Improved care of patients with diabetes through
telecommunications. AnnNYAcad Sci 1992;670:141-145.
18. Ackerman SJ, Gitlin JN, Gayler RW, Flagle CD, Bryan
RN. Receiver operating characteristic analysis of frac¬
ture and pneumonia detection: comparison of laser-
digitized workstation images and conventional ana¬
log radiographs. Radiology 1993;186:263-268.
19. Scott WW, Rosenbaum JE, Ackerman SJ, Reichte RL,
Magid D, Weller JC, Gitlin JN. Subtle orthopedic frac¬
tures: teleradiology workstation versus film interpre¬
tation. Radiology 1993;187:811-815.
20. Slovis TL, Guzzardo-Dobson PR. The clinical useful¬
ness of teleradiology of neonates: expanded services
without expanded staff. Pediatr Radiol 1991; 21:333-335.
21. Goldberg MA, Rosenthal DJ, Chew FS, Blickman JG,
Miller SW, Mueller PR. New high-resolution telera¬
diology system: prospective study of diagnostic ac¬
curacy in 685 transmitted clinical cases. Radiology
1993;186:429-134.
DeCHANT ET AL.
22. Yamamoto LG. Wireless teleradiology and fax using
cellular phones and notebook PCs for instant access
to consultants. Am J Emerg Med 1995;13:184-187.
23. Ferrer-Roca O, Diaz-Cardama A, Pitti S, Ramos A,
Gomez E, Tejera A, Spencer J, Telemedicine in the
Canary Islands. Lancet 1995:345:1177.
24. Goldberg MA, Sharif HS, Rosenthal DI, Black-
Schaffer S, Flotte TJ, Colvin RB, Thrall JH. Making
global telemedicine practical and affordable: demon¬
strations from the Middle East. A]R Am } Roentgenol
1994;163:1495-1500.
25. Rafuse J. University of Alberta program links north¬
ern hepatitis patients with southern consultants. Can
Med Assoc J 1994;151:654-655.
26. Kagetsu NJ, Zulauf DRP, Ablow RC. Clinical trial of
digital teleradiology in the practice of emergency
room radiology. Radiology 1987;165:551-554.
27. Templeton AW, Dwyer SJ, Rosenthal SJ, Eckard DA,
Harrison LA, Cook LT. A dial-up digital teleradiol¬
ogy system: technical considerations and clinical ex¬
perience. AJR Am J Roentgenol 1991;157:1331-1336.
28. Institute of Medicine. Assessing medical technologies.
Washington, DC: National Academy Press, 1985.
29. Langlotz CP, Even-Shoshen O, Seshadri S, Brikman I,
Kishore S, Kundel H, Schwartz JS. A methodology for
the economic assessment of picture archiving and re¬
trievalsystems. / Digit Imaging 1995;8:95-102.
30. Bryan S, Weatherburn G, Keen J, Muris N, Buxton M.
Technology assessment of PACS: measurement and
analysis of radiologists' reporting times when re¬
porting from film. Proc SPIE 1994;2165:820-826.
31. Tohme WG, Hayes WS, Hailei D, Komo D, Pahira JJ,
Abemethy DR, Rennert W, Kuehl K, Hauser GJ, Mun
SK. Design and evaluation of a PC-based platform for
telemedicine applications. / Am Soc Inf Sci (in press).
32. Cook TD, Campbell DT. Quasi-experimentation: design
and analysis issues for field settings. Boston: Houghton-
Mifflin, 1979.
33. Bashshur RL. Telemedicine effects: Cost, quality and
access. J Med Syst 1995;19(2):81-91.
34. Grigsby J, Schlenker RE, Kaehny MM, Shaughnessy
PW, Sandberg EJ. Analytic framework for evaluation
of telemedicine. Telemed J 1995;1:31-38.
35. Grigsby J, Sandberg EJ, Kaehny MM, et al. Analysis of
expansion of access to care through use of telemedicine and
mobile health services: Report 2: Case studies and current
status of telemedicine. Denver: Center for Health Policy
Research, 1994.
36. Grigsby J, Kaehny MM, Schlenker RE, et al. Analysis
of expansion of access to care through use of telemedicine
and mobile health services: Report 4: Study summary and
recommendations for further research. Denver: Center for
Health Policy Research, 1994.
Address reprint requests to:
Kevin A. Schulman, M.D.
Clinical Economics Research Unit
Georgetown University Medical Center
2233 Wisconsin Ave. NW, Suite 440
Washington, D.C. 20007
This article has been cited by:

1. Lilian J. Beijer , Toni C.M. Rietveld , Marijn M.A. van Beers , Robert M.L. Slangen , Henk van den Heuvel , Bert J.M. de Swart
, Alexander C.H. Geurts . 2010. E-Learning-Based Speech Therapy: A Web Application for Speech TrainingE-Learning-Based
Speech Therapy: A Web Application for Speech Training. Telemedicine and e-Health 16:2, 177-180. [Abstract] [PDF] [PDF
Plus]
2. Micaela Brown , Nicola Shaw . 2008. Evaluation Practices of a Major Canadian Telehealth Provider: Lessons and Future
Directions for the FieldEvaluation Practices of a Major Canadian Telehealth Provider: Lessons and Future Directions for the
Field. Telemedicine and e-Health 14:8, 769-774. [Abstract] [PDF] [PDF Plus]
3. Rianne M.H.A. Huis in ’t Veld , Barbara C.H. Huijgen , Leendert Schaake , Hermie J. Hermens , Miriam M.R.
Vollenbroek-Hutten . 2008. A Staged Approach Evaluation of Remotely Supervised Myofeedback Treatment (RSMT) in
Women with Neck—Shoulder Pain Due to Computer WorkA Staged Approach Evaluation of Remotely Supervised Myofeedback
Treatment (RSMT) in Women with Neck—Shoulder Pain Due to Computer Work. Telemedicine and e-Health 14:6, 545-551.
[Abstract] [PDF] [PDF Plus]
4. Jim Grigsby , Angela G. Brega , Patricia A. Devore . 2005. The Evaluation of Telemedicine and Health Services ResearchThe
Evaluation of Telemedicine and Health Services Research. Telemedicine and e-Health 11:3, 317-328. [Citation] [PDF] [PDF
Plus]
5. Rashid Bashshur , Gary Shannon , Hasan Sapci . 2005. Telemedicine EvaluationTelemedicine Evaluation. Telemedicine and
e-Health 11:3, 296-316. [Abstract] [PDF] [PDF Plus]
6. James P. Marcin , Thomas S. Nesbitt , Stacey L. Cole , Robin M. Knuttel , Donald M. Hilty , Pamela T.
Prescott , Martha M. Daschbach . 2005. Changes in Diagnosis, Treatment, and Clinical Improvement Among Patients
Receiving Telemedicine ConsultationsChanges in Diagnosis, Treatment, and Clinical Improvement Among Patients Receiving
Telemedicine Consultations. Telemedicine and e-Health 11:1, 36-43. [Abstract] [PDF] [PDF Plus]
7. Barbara P. Yawn . 2000. Communication: Telemedicine: A New Framework for EvaluationCommunication: Telemedicine: A
New Framework for Evaluation. Telemedicine Journal 6:1, 55-61. [Abstract] [PDF] [PDF Plus]