Word Count:8,893

RUNNING HEAD: Psychotherapy and Controlled Clinical Trials

Validity of Controlled Clinical Trials of Psychotherapy:

Findings from the NIMH Collaborative Study of Depression

J. Stuart Ablon, Ph.D.1 and Enrico E. Jones, Ph.D.2

1Massachusetts General Hospital and Harvard Medical School

2University of California at Berkeley

American Journal of Psychiatry, 2002; 159:775-783

Correspondence concerning this article should be addressed to J. Stuart Ablon, Ph.D. at the
Massachusetts General Hospital, WAC 812, 15 Parkman Street, Boston, MA 02114 (voice 617724-5055; fax 617-726-7541). Electronic mail may be sent to sablon@partners.org.
The authors wish to thank the investigators of the NIMH TDCRP for their contributions to the
field in conducting this landmark study and for making the data available to the public for further
analyses. The authors also thank the Oversight Committee of the TDCRP for making transcripts
of the treatments available.
This work was supported by a NIH National Research Service Award (1F31MH11356), the
American Psychological Association Scott Mesh Honorary Scholarship, and grants from the
Humanities Department at the University of California, Berkeley. Special thanks are also
extended to the SLA Foundation for additional support.

Psychotherapy and Controlled Clinical Trials

Abstract
OBJECTIVE: This research extends a sequence of studies examining psychotherapy process in
the NIMH Treatment of Depression Collaborative Research Program (TDCRP). The present
study examines the following hypotheses: 1) there will be considerable overlap in treatment
process and technique in manualized therapies studied in a controlled clinical trial; and 2) that
intervention strategies that are common to the treatments are responsible for promoting patient
change. METHOD: Expert therapists developed two prototypes of ideal therapy process in
brief interpersonal (IPT) and cognitive-behavioral (CBT) therapies using the Psychotherapy
Process Q-set, an instrument designed to provide a standard language for describing treatment
processes. The prototypes were then applied to Q-sorts made by clinical judges of samples of
actual therapy hours from these two types of treatment conducted as part of the TDCRP. This
was done in order to determine the extent to which the process of these treatments conformed to
the ideal prototypes. RESULTS: Both IPT and CBT treatments adhered most strongly to the
prototype of ideal cognitive-behavioral therapy process. In addition, adherence to the CBT
prototype was most significantly correlated with positive outcome across the two treatments.
CONCLUSIONS: These findings suggest that the basic premise of controlled clinical trials, that
the interventions being compared represent separate and distinct treatments, may not have been
met in the TDCRP. Relying on brand names of therapy can be misleading. The implications of
these findings for using controlled clinical trials to study psychotherapy are discussed.

Psychotherapy and Controlled Clinical Trials

Validity of Controlled Clinical Trials of Psychotherapy:

Findings from the NIMH Collaborative Study of Depression
There is currently a sharp debate in the clinical research literature concerning what constitutes
the best method for empirically validating treatments. Some researchers have argued that it is
most important to determine the efficacy of a treatment (1), while others have suggested that the
notion of effectiveness is more important (2, 3). Efficacy studies emphasize the importance of
being able to draw a causal inference between the treatment provided and outcome (4). Such
studies rely on controlled clinical trial methodology where participants are randomly assigned to
different treatment conditions, which are closely controlled. It is then assumed that any change
that occurs differentially across treatment conditions can be attributed directly to the causal
effect of the treatment. The cost of being able to draw causal inferences lies in the fact that the
study must be so closely controlled that the treatments and the patients may no longer resemble
the reality of clinical practice (5, 6). Randomized clinical trials test a somewhat artificial
treatment in an artificially controlled setting with atypical patients, so the have little
generalizability to the real world of mental health care delivery.
Effectiveness studies, on the other hand, emphasize external validity and are less constrained
by research protocols. Participants in effectiveness studies are usually selected without using
exclusion criteria and they are free to choose whatever treatment they wish to receive. The
treatments are not standardized; that is to say, they are conducted as they would be in clinical
practice without necessarily having a set number of sessions or a treatment manual to guide
interventions. Treatments studied in this way are obviously closer to what is actually practiced
in the real world. However, without random assignment to treatment conditions, the causal

Psychotherapy and Controlled Clinical Trials

influence of the treatment cannot be assessed because the greater latitude for individual treatment
differences can confound treatment effects. Despite the fact that both efficacy and effectiveness
designs have significant shortcomings, controlled clinical trials have been embraced by the
scientific community and, therefore, have come to represent the state of the art research design
for empirically validating treatments.
Both efficacy and effectiveness studies of psychotherapy tend to focus on the outcome of
different interventions relative to each other. Less attention is typically paid to the link between
process and outcome. When a participant in a controlled clinical trial improves after undergoing
psychotherapy, it is assumed that the improvement was caused by the specific interventions that
were prescribed by a manual and monitored for adherence. However, this assumption relies
heavily on both the clinician's ability to apply certain techniques without using others, as well as
their close adherence to a particular treatment approach. It is impossible to say what factors were
associated with improvement after treatment unless the treatment process itself is studied.
A recent series of studies suggests that the basic assumption of controlled clinical trials of
psychotherapy can be questioned. We have shown that even in manualized treatments, elements
are borrowed from different treatment approaches and that these common techniques can be
among the active ingredients that are responsible for promoting positive patient change. We
demonstrated, for example, that brief psychodynamic treatments included a diverse set of
interventions, and that therapists, in addition to applying strategies considered to be
psychodynamic in nature, also applied to a considerable degree intervention techniques that are
usually associated with cognitive-behavioral approaches, e.g. examining faulty thinking and

Psychotherapy and Controlled Clinical Trials

irrational beliefs (7). In other words, there was significant overlap in how therapists conducted
treatment across theoretical models widely assumed to imply distinct intervention strategies.
Consistent with our own results, comparison studies conducted by other investigators have also
found extensive overlap across psychodynamic and cognitive-behavioral treatments conducted
when conducted by master therapists (8). In one well-studied treatment sample (9), several
investigators determined that cognitive-behavioral therapists occasionally used psychodynamic
strategies, and that it was these techniques that were responsible for promoting patient change
(10). This use of non-cognitive/behavioral techniques not prescribed by the manual probably
escaped the detection of adherence checks; nevertheless, they proved to be significant correlates
of patient change.
These provocative results raised serious questions about the conclusions typically drawn
from controlled clinical trials. The present study examines the following hypotheses: 1) there will
be considerable overlap in treatment process and technique in manualized therapies studied in a
controlled clinical trial; and 2) that intervention strategies that are common to the treatments are
responsible for promoting patient change. The NIMH Treatment of Depression Collaborative
Research Program (TDCRP) was chosen as the best dataset to attempt to further test these
hypotheses. It remains to date the most carefully conducted and methodologically sound,
randomized, controlled clinical trial comparing different forms of brief psychological therapies.
Participants from three sites in the TDCRP were randomly assigned to one of four brief
treatments for depression: interpersonal psychotherapy (IPT); cognitive-behavioral
psychotherapy (CBT); imipramine plus clinical management as a standard reference treatment

Psychotherapy and Controlled Clinical Trials

(IMI-CM); and pill placebo plus clinical management as a double-blind control group (PLACM). Patients improved in all treatment conditions, and the effect sizes of the psychotherapeutic
treatments in the TDCRP were found to be consistent with those reported in the literature (11,
12). When differences between the treatments outcomes were observed (13, 14), they were
inconsistent, not robust, and appeared only in specific areas. It is generally accepted that the
treatments in the TDCRP were effective and equivalent overall.
A further aim of the present study is to clarify the mechanisms by which patients in the
TDCRP improved as a result of these treatments. While both the IPT and CBT treatments were
shown to be generally effective (11), the specific active ingredients in these therapies have yet to
be identified. There have only been a few studies of the treatment process in IPT and CBT, and
these studies have focused on the therapeutic alliance (15) and therapist adherence (16). This
study extends the investigation of treatment processes in the TDCRP we recently reported (17).
In that study, verbatim transcripts of the treatment sessions were obtained. We then applied the
Psychotherapy Process Q-set (PQS) (18), a rating system designed to provide a standard
language for describing therapy process in a form suitable for quantitative analysis. A comparison
of therapist technique and the overall nature of interpersonal transactions in IPT and CBT
showed that there were both significant areas of overlap and difference in the process of the
treatments. More specifically, while there were some key differences in therapist activity level
and technique, there was a great deal of similarity in therapists' authoritative and supportive
stance, in the use of reassurance, and in offering of advice or counsel regarding alternative ways of
relating to others. In addition, patient in-session behaviors were quite similar. The present study

Psychotherapy and Controlled Clinical Trials

attempts to demonstrate whether and how the theories of therapy of IPT and CBT were
translated into actual practice. Expert therapists from these two theoretical orientations used the
PQS to generate templates or prototypes of an ideal treatment hour conducted from the
perspective of each of these schools of therapy. These prototypes are then applied to PQS
ratings of the verbatim transcripts, made by independent clinical judges, of IPT and CBT therapy
sessions. The extent to which the treatments conform to the expert-generated prototypes is
estimated quantitatively, and then correlated with outcome to determine whether the data
support the specific theories of change represented in these two treatments.
Overview of Methods
The methods of this study are presented in two separate sections. The first section describes
how a panel of experts develops prototypes of ideal sessions of IPT and CBT using an
instrument consisting of 100 items that describes the process of therapy empirically. The second
section describes how a group of independent observers rate the occurrence and salience of each
of those 100 items in actual sessions of IPT and CBT. This section then compares the ratings of
actual sessions to the prototypes and relates these findings to treatment outcome.
Development of Prototypes
Method
Responses by expert therapists of different orientations to a questionnaire form of the PQS
were used to develop prototypes of ideal treatments. Expert panels of cognitive-behavioral
(N=10) and interpersonal (N=11) therapists were comprised of leading theoreticians and
practitioners of each perspective. All of the expert therapists were highly experienced and

Psychotherapy and Controlled Clinical Trials

internationally recognized for their expertise. Each member of the expert panels was responsible
for training therapists in their orientation and most had published work concerning their approach
to psychotherapy. Many of the members of the expert panels were involved in the creation of
the treatment modality.
The instrument used to create the prototypes, the PQS, furnishes a language for describing
therapist-patient interaction in clinically relevant terms that can be analyzed quantitatively(18).
The PQS consists of 100 items that are tied to specific actions, behaviors, and statements. The
PQS has demonstrated both reliability and validity across the variety of studies and treatment
samples (10, 19, 20). Because the PQS is pantheoretical, it is possible to create prototypes of
different types of therapy using the same instrument, and then prototypes of different
therapeutic approaches can be directly compared. Each member of the panel of expert therapists
was asked simply to rate each of the 100 items of the Q-set questionnaire on a scale from 1 to 9,
according to how characteristic each item was of their understanding of an ideally conducted
course of therapy that adheres to the principles of their theoretical perspective. Each
questionnaire yielded one score for each of the 100 items of the Q-set. Each expert was also
asked to suggest additional items if they felt that the item set was not sufficient to capture
adequately the important aspects of their approach to treatment. There were no consistent
suggestions for additional items.
Results
Coefficient alpha reliabilities demonstrated that the level of agreement of the expert
therapists' ratings was high for the different orientations (Cognitive-Behavioral = .95;

Psychotherapy and Controlled Clinical Trials

Interpersonal = .96). A prototype of ideal cognitive-behavioral therapy had already been

developed for previous research (7). A new prototype of interpersonal therapy was then created
using the same small sample statistical method for studying points of view, sometimes called 'Qtechnique' (21-24). See (7) for a detailed description of method. The expert interpersonal
therapists' responses to the PQS were combined with expert psychodynamic and cognitivebehavioral therapists' ratings collected for previous research and subjected to a principal
components factor analysis. As would be expected, the factor analysis yielded three distinct
theoretical orientation factors with Eigenvalues above 1.0 after varimax rotation, which together
explained 70.9% of the variation in the correlations among the expert therapists. In other words,
the psychodynamic, interpersonal and cognitive-behavioral therapists had distinct points of view
about an ideal treatment process. Details of the psychodynamic and cognitive-behavioral
prototypes are described in (7). All of the expert interpersonal therapists' ratings had primary
factor loadings on the third factor that emerged. The average primary factor loading for the
interpersonal therapists was .79 (range .70 - .87).
Linear regressions were calculated to determine the contribution of each Q-item to the
interpersonal experts' factor. Factor scores represent the weighted sum of each Q-item. The items
with the highest factor scores are most defining of the factor, and the items with the lowest factor
scores are least defining of the factor. Table 1 is a copy of the 20 items with the highest factor
scores for the cognitive-behavioral technique factor that was developed for previous research.
Table 2 presents the 20 items with the highest factor scores for the new interpersonal technique
factor. In the interest of space, only those 20 items with the highest factor scores are displayed in

Psychotherapy and Controlled Clinical Trials

10

the tables even though the entire factor array of factor scores on all 100 items of the PQS was
included in the Q-prototypes. This is important to consider when evaluating the face validity of
the prototypes because items contributing significantly to a prototype may not be listed among
the 20 items with the highest factor loadings. For example, item #99 (which describes the
therapist challenging the patients view or core beliefs) received a high factor loading (.66)
indicating that the experts felt it was an important component of cognitive-behavioral therapy.
However, it does not appear in Table 1 since it was not among the 20 items with the highest
factor loadings. Because the prototypes comprise all 100 items of the PQS, they account for the
relative contribution of all items in the item set. Each prototype, therefore, contains all of the 100
items of the PQS (not only the ones that appear in the tables). No arbitrary cut-off points were
used so that all 100 items could be included in the prototype because it is important that the
description of an ideal treatment capture not only those items that are characteristic, but also
those that are uncharacteristic and not relevant. That is, a prototype should reflect what is
missing from an ideal treatment as well as what is prominent.
The reader will notice that Tables 1 and 2 share six items in common. That is, both the IPT
and CBT experts rated six of the same items as being among the 20 most defining aspects of their
respective treatments. These items have different factor loadings so that they each contribute to
the prototype to varying degrees. For example, item #45 (therapist adopts a supportive stance)
has a higher factor loading on the CBT scale, indicating that the experts judged the item to be
more characteristic of ideal CBT than ideal IPT.
----------------------

Psychotherapy and Controlled Clinical Trials

11

Insert Table 1 here

------------------------------------------Insert Table 2 here
---------------------Application of Prototypes
Method
The Q-prototypes of ideal interpersonal and cognitive-behavioral therapy were applied to
actual psychotherapy sessions from the TDCRP that had already been rated using the PQS in
order to determine how close the treatments are to the ideal prototypes.
The NIMH Treatment of Depression Collaborative Research Program (TDCRP):
Participants.
Participants were outpatients between the ages of 21 and 60 who met Research Diagnostic
Criteria (25) for a current episode of major depressive disorder, and who scored 14 or higher on a
revised 17-item Hamilton Rating Scale for Depression (26). Exclusion criteria included certain
other psychiatric disorders (bipolar and psychotic disorders), concurrent psychiatric treatment,
medical conditions that contraindicated use of imipramine, and the need for immediate
intervention (e.g., active suicide potential). Two hundred-fifty potential participants met these
criteria and were randomized into one of the four treatment conditions. The 239 participants who
entered treatment were primarily women (70%) and Caucasian (89%). The average age was 35.
Seventy-seven participants terminated prematurely (prior to completing at least 12 sessions of

Psychotherapy and Controlled Clinical Trials

12

therapy and 15 weeks of treatment). There were no significant differences among the
psychosocial treatments in the number of premature terminations (11). The remaining 162
participants are referred to as the "completer sample." The mean number of sessions in the
completer sample was 16.2 and the maximum 20.
Therapists and Treatments.
The IPT and CBT treatments in the TDCRP were carried out in accordance with detailed
manuals (27, 28). Ten therapists delivered IPT, and eight delivered CBT. The therapists, 71% of
whom were men, averaged over 11 years of clinical experience. They were all trained in the
treatment they provided, and tapes of sessions were reviewed to monitor adherence to treatment
protocols (12).
Assessment of Outcome.
Treatment outcome was measured from a variety of perspectives and with an array of
different measures. For example, the clinical evaluator-rated Hamilton Rating Scale for
Depression (HRSD) (26) and the client-rated Beck Depression Inventory (BDI) (29) measured
depressive symptoms; and the clinical evaluator-rated Global Assessment Scale (GAS) (30) and
the client-rated Hopkins Symptom Checklist HSCL-90) (31) measured overall functioning. The
participants also provided a 7-point Likert-type rating of their overall satisfaction with the
treatment they received. Information on the procedures and methods of the TDCRP beyond
what is presented here is available in Elkin et al (11, 32).
The Psychotherapy Process Q-Set (PQS).

Psychotherapy and Controlled Clinical Trials

13

The Psychotherapy Process Q-Set (PQS) is a 100 item instrument that furnishes a language
and rating procedure for the comprehensive description, in clinically relevant terms, of the
therapist-patient interaction in a form suitable for quantitative comparison and analysis (18).
Clinical judges read verbatim transcripts of an entire therapy hour and then sort the 100 items in
the Q-set on a continuum from least characteristic or negatively (category 1) to most
characteristic or salient (category 9). The middle pile (category 5) is used for items deemed either
neutral or irrelevant to the particular hour being rated. Each item contains a description of the two
opposite ends of the continuum along which the items are to be rated. It is important to note that
placement in the uncharacteristic direction does not signal that a particular behavior or experience
is irrelevant. On the contrary, an uncharacteristic ranking signals that the absence of the item is
meaningful and important to capture in the Q-sort the description. Most items have specific
instructions that provide examples of the distinction between uncharacteristic and neutral ratings.
For example, Q-item number 9 describes the therapist as "distant or aloof" when rated in the
characteristic range. However, when rated in the uncharacteristic range, the item indicates that the
therapist was "genuinely responsive or affectively involved" (the opposite of "distant or aloof").
Only if the item were irrelevant to the description of the hour would it be placed in the neutral
range. The number of cards sorted into each category of the Q-sort (from 5 at the extremes to 18
in the middle or neutral category) conforms to a normal distribution, requiring judges to make
multiple evaluations among items thereby avoiding halo effects and response sets (33). Unlike
most psychotherapy process measures, the PQS uses an entire hour as the unit of observation.
The items are tied to specific actions, behaviors, and statements. A detailed coding manual

Psychotherapy and Controlled Clinical Trials

14

provides the Q-items and their descriptions as well as operational examples. When rating, judges
are asked to take the position of a "generalized other" i.e. an observer who stands mid-way
between patient and therapist and who views the interaction from the outside. In placing each
item, judges are instructed to ask themselves: Is this attitude, behavior, or experience clearly
present (or absent)? If the evidence is not compelling, the judge is asked to search for specific
evidence of the extent to which it present or absent. Judges are asked to be as open-minded and
objective as possible.
The PQS was developed pantheoretically to assess therapist actions valued across therapies,
so it is especially useful for comparing the therapy process of different therapies (34, 35). The
PQS has been used to rate archived hours of both psychodynamic and cognitive-behavioral
therapies for which outcome measures are also available (10). It has also been used to rate clientcentered, Gestalt, and rational-emotive therapies (19). The PQS has demonstrated both reliability
and validity across the variety of studies and treatment samples (34). Interrater reliability, which
is calculated by correlating the Q-sorts of multiple raters across all 100 items of the PQS, has
been consistently satisfactory, with alpha coefficients ranging from .83 to .89 for two raters.
Reliability calculated at the individual Q-item level has also been consistently satisfactory,
ranging from .50 to .95, across several different samples. The instrument has also demonstrated
construct and discriminant validity. (10, 19, 34, 36).
Observer Q-sorts.
Verbatim transcripts of selected therapy sessions for each patient were rated according to a
forced normal distribution using the PQS. A pool of nine research-oriented psychologists and

Psychotherapy and Controlled Clinical Trials

15

masters-level graduate students in a clinical psychology doctoral program completed Q-ratings

for this study. The raters represented a range of theoretical perspectives, including
psychodynamic and cognitive-behavioral, although most were eclectic in their clinical
orientations. All raters were trained in the application of the Q-technique. Two verbatim
transcripts, a session early in treatment (session 4) and a session late in treatment (session 12)
were selected for each patient in the IPT and CBT samples. When transcripts of these sessions
were not available, the next closest session was selected. The length of treatments for these
patients ranged from 12 to 20 with an average length of 16.2 sessions. Transcripts were only
available from the TDCRP archive for two of the three sites that participated. Therefore,
transcripts were available for only 35 of the 47 participants who received IPT and 29 of the 37
participants who received CBT. All of the transcripts (N of treatment sessions = 128) were
randomized, and independent ratings were completed by at least two judges who were blind to
treatment type and session number. Average interrater reliability for Q-sorts achieved .82. There
were no significant differences in reliability between treatment modalities. When agreement was
below r = .50, the author of the PQS who is a licensed and practicing clinical psychologist (EJ)
was added as a third rater. Periodic calibration meetings were conducted to correct rater drift.
Statistical analyses comparing Q-ratings from sessions four and 12 revealed little change in
process over time. The independent Q-sorts (sessions 4 and 12) of the judges for each transcript,
therefore, were composited as they have been in previous studies using the PQS. The composited
Q-ratings were averaged across both sessions to obtain one score per Q-item for each patient.

Psychotherapy and Controlled Clinical Trials

16

The observer Q-sorts for each patient were correlated with the interpersonal and cognitivebehavioral therapy prototypes. That is, for each patient the composited Q-rating of each of the
100 items was correlated with the factor score for that item from the prototype. These
correlations with the Q-prototypes then were transformed into z-scores using the Fisher r to z
transformation. The z-scores represent the degree to which an hour of psychotherapy is
correlated with the Q-prototypes. The z-scores were then correlated with outcome to determine
whether those aspects of the therapy process that in theory should promote patient change are in
fact responsible for predicting positive outcome (Table 3). Partial correlations of the z-scores and
the outcome measures were calculated in order to control for pre-treatment scores.
Results
Figure 1 depicts the mean correlations of the observer Q-sorts and the Q-prototypes in CBT
and IPT. In both CBT and IPT the same pattern of correlations of the observer Q-sorts and the
prototypes emerged. The correlation between the observer Q-sorts and the cognitive-behavioral
prototype was significantly higher than the correlation between the observer Q-sorts and the
interpersonal prototype in both CBT and IPT (two tailed t-tests: t (28) = 11.92, p < .001; t (34)
= 5.39, p < .001, respectively). Both CBT and IPT adhered more to the cognitive-behavioral
therapy prototype than the interpersonal therapy prototype. In CBT, there was a high
correlation between the observer Q-sorts and the cognitive-behavioral prototype (M z-score =
.64, SD = .28) and only a small correlation between the observer Q-sorts and the interpersonal
prototype (M z-score = .18, SD = .1). In IPT, there again was a high correlation between the
observer Q-sorts and the cognitive-behavioral prototype (M z-score = .57, SD = .24) and a

Psychotherapy and Controlled Clinical Trials

17

moderate correlation between the observer Q-sorts and the interpersonal prototype (M z-score =
.39, SD = .11). The correlation between the observer Q-sorts and the interpersonal prototype
was significantly higher in IPT than CBT (two tailed t-test: t (62) = -8.24, p < .001). However,
there was no significant difference between the correlations of the observer Q-sorts and the
cognitive-behavioral prototype across CBT and IPT (two tailed t-test: t (62) = 0.97, NS). The
only significant difference between the two treatments was in adherence to interpersonal
prototype.
Table 3 displays the partial correlation coefficients of the adherence to the Q-prototypes in
both IPT and CBT and the outcome measures, controlling for pre-treatment scores. Positive
correlations reflect a favorable association with outcome. In the IPT treatment sample (N=35),
adherence to the cognitive-behavioral prototype was associated significantly with positive
outcome on five of the six outcome measures. Adherence to the interpersonal prototype was
associated significantly with positive outcome on three of the six outcome measures in IPT. In
the CBT treatment sample (N=29), adherence to the cognitive-behavioral prototype again was
associated significantly with positive outcome on all six of the outcome measures. Adherence to
the interpersonal prototype was associated significantly with positive outcome on two of the six
outcome measures. In summary, adherence to the cognitive-behavioral prototype predicted
positive outcome in both IPT and CBT in a consistent and robust manner. The effect sizes of the
correlations ranged from r = .36 - .53, which is particularly strong relative to other process outcome linkages found in the literature. Adherence to the interpersonal prototype also predicted
positive outcome in both IPT and CBT but in a less consistent and robust manner (effect sizes of

Psychotherapy and Controlled Clinical Trials

18

the correlations ranged from r = .29 - .48). It is unlikely that differences in the variability of
prototype scores across treatment conditions contributed to this pattern of correlations with
outcome. Levene tests for equality of variances revealed no significant differences in the variance
of IPT prototype adherence scores across IPT (variance = .01) and CBT (variance = .01), or in
the variance of CBT prototype adherence scores across IPT (variance = .06) and CBT (variance
= .08), f = .05, p = .83; f = 2.05, p = .16, respectively.
Discussion
Brand Names of Therapy can be Misleading
A remarkable finding of this study was that interpersonal therapists in the TDCRP fostered
as much of a cognitive-behavioral process as did the cognitive-behavioral therapists. In fact, the
interpersonal treatments adhered more to the prototype of CBT than the prototype of IPT.
According to the pattern of adherence to the prototypes, the difference between the process of
IPT and CBT in the TDCRP concerned the amount of interpersonal process that was fostered.
IPT included a more diverse set of therapeutic processes, consisting of significant amounts of
both cognitive-behavioral and interpersonal process. CBT, on the other hand, was more purely
cognitive-behavioral in emphasis. However, cognitive-behavioral process was most defining of
both IPT and CBT. Although the two treatments have been shown to be reliably distinguishable
despite containing some shared aspects (16), our current findings suggest that the overall process
of IPT is quite similar to CBT in practice, challenging the presumption of large differences across
these two brand names of therapies. Our previous research (17) had used a "zoom lens" to
examine process at a micro-analytic level and identified specific areas of difference as well as

Psychotherapy and Controlled Clinical Trials

19

similarity between the two treatments. The current study used a "wide-angle" lens to describe
overall process and found significant similarity between the treatments. Taken together, the
results of these two studies suggest that the similarities between the two therapies in the TDCRP
were more salient and defining of the overall process than the differences.
One hypothetical explanation for how the process of treatments with different brand names
and prescribed techniques can be so similar, concerns the theoretical language used by proponents
of different forms of psychotherapy. Clinicians from different orientations may use very
different terminology to describe psychological constructs and aspects of therapeutic process
that are actually quite similar. In this way, the labels of the treatments may appear quite
different. However, once theoretical jargon is stripped away, the treatments may share many
common processes. For example, a recent study asked experienced clinicians from different
theoretical backgrounds to read an interview with an unidentified psychologist and indicate his
orientation and the extent to which they agreed with his views (37). The psychologist was a wellknown behaviorist who was asked to describe, without using any jargon, how his orientation
would deal with the problem of agoraphobia clinically. Most respondents in the study
incorrectly judged the psychologist to be psychodynamic rather than a behaviorist. Furthermore,
without knowing the orientation of the psychologist being interviewed, the psychodynamically
oriented respondents agreed more with the behaviorist's viewpoints than did the behavioral
respondents! These provocative results suggest that there is considerable overlap between
different theoretical models on the question of how to work clinically to address psychological
problems and that much of this overlap is concealed by the use of theory-specific language.

Psychotherapy and Controlled Clinical Trials

20

Another potential explanation for the similarity between two therapies that are otherwise
easily distinguished from one another, concerns the contribution of the patient to the therapeutic
process. Most observations of distinctions between treatments tend to focus only on the
therapists' behaviors, interventions, and adherence to prescribed techniques in isolation from the
larger context in which they occur. However, the nature of therapeutic process is inherently
interactional. Previous reports on the TDCRP documenting the clear distinctions between IPT
and CBT focused on the therapists' actions and interventions (16). Our research suggests that
the patient's contribution to process in IPT and CBT was remarkably similar (17). That is,
patients in both brief treatments helped to foster a similar overall process. In addition, despite
specific distinguishable differences in therapist interventions and activity level, there were also
some similarities in IPT and CBT therapists' use of reassurance, support, and counsel (17). Thus,
the overlap in overall process of IPT and CBT may be the result of strong similarities in patients'
contribution to the process coupled with some significant areas of overlap in therapist behaviors.
Our programmatic series of studies went beyond treatment labels and techniques by studying
the actual process of different treatments and considering both the therapists' and the patients'
contribution to the process. The instrument that was used was specifically designed to avoid
biasing ratings through the use of theoretical jargon. The PQS manual and the Q-set items
themselves are not closely bound to theoretical concepts, but rather to notions of therapy
process. The influence of observers' theoretical predilections on their descriptions of the nature
of process is reduced within the flexible but stabilizing framework provided by the Q-set (34).
The Q-items are tied to concrete behavioral and linguistic cues that can be identified in recordings

Psychotherapy and Controlled Clinical Trials

21

of therapy hours rather than to theoretical jargon used by therapists during sessions. The Q-set
includes items describing therapist feelings and behaviors, patient feelings and behaviors, as well
as the interaction between therapist and patient.
Process Correlates of Outcome in Brief Therapy
This study also examined treatment processes relative to outcome. If interpersonal and
cognitive-behavioral therapy process can be quite similar, then it is not at all surprising that IPT
and CBT could achieve such strikingly similar outcomes in the TDCRP by most accounts (11).
Without examining the relationship between treatment process and outcome, it had been
concluded that the techniques of IPT and CBT were both quite different and yet equally
effective. Previous research had shown, however, that there were also key areas of overlap
between the two modalities (15-17, 38). Our results suggest that both IPT and CBT were
characterized by a similar amount of cognitive-behavioral process, and it was this shared
cognitive-behavioral process that was most robustly responsible for predicting positive outcome
in both IPT and CBT. While IPT contained significantly more interpersonal process than CBT,
this interpersonal process also contributed to positive outcome in both treatment modalities, as
evidenced by significant correlations between adherence to IPT and positive outcome across three
of the outcome measures in IPT and two of the outcome measures in CBT. Similarities in process
across treatments may help explain the lack of differential outcomes frequently observed in trials
of psychotherapy. What are often referred to as non-specific effects in the literature may actually
represent the specific effects of shared processes.

Psychotherapy and Controlled Clinical Trials

22

Our previous study of psychotherapy process in the TDCRP (17), which used individual Qitems as the unit of analysis, suggested more specifically how these treatments helped patients
with depressive symptoms. The majority of the individual Q-items that predicted positive
outcome in both IPT and CBT described the patient's contribution to process rather than the
therapists' actions and interventions. Thus, the similarities in the process of IPT and CBT that
corresponded better to the prototype of CBT than IPT and that were related to positive outcome
had more to do with the patients than the therapists. Results from our previous study showed
that the Q-items associated with possible change across both IPT and CBT described patients'
experience of a positive sense of self and an unambivalent, deferential, even idealized view of
their therapist. This constellation of Q-items captured the presence of a positive attachment to a
benevolent, supportive, and reassuring authority figure that apparently facilitated improvement
in both forms of brief treatment. Therapists in both IPT and CBT relied heavily on interpersonal
influence; they offered advice and guidance, suggested patients change their behavior, reassured
patients that they would feel better soon, and did not, as a rule, discuss features of their
interaction with the patient. In spite of their different theoretical orientations, our results suggest
that these two treatments share these common characteristics which are essentially supportive
rather than exploratory in nature and are associated with positive outcome at termination of
short-term treatment. The strong supportive elements of these two treatments have been
observed by other investigators as well (15, 16, 38).
Controlled Clinical Trials and Psychotherapy

Psychotherapy and Controlled Clinical Trials

23

The TDCRP represents the state of the art controlled clinical trial of psychotherapy where
therapists were extensively trained in treatments prescribed by manuals and routinely checked
for adherence (16). The controlled clinical trial paradigm enforces all of these experimental
controls on the therapy process in an effort to maintain pure and distinct treatments. With such
controls in place, improvement during the course of treatment is presumed to be the result of the
techniques described in the treatment manual. The results of this study and others suggest this
can be a flawed assumption because it may not be possible to fully control the conditions of
psychotherapy. The therapies in the TDCRP were not completely pure and distinct despite
raters abilities to reliably distinguish between the two treatments and each treatment having
passed adherence checks (16). Our results suggest that the two therapies were still characterized
by significant overlap. Adherence in the TDCRP had been deemed satisfactory in previous
analyses because IPT and CBT could be discriminated by each having had different profiles of
adherence and each scoring highest on its own scale meant to measure the specific features of
therapist behavior for that particular treatment (16). Using the Collaborative Study
Psychotherapy Rating Scale (CSPRS) (39), therapists in the TDCRP were said to show more
behaviors specific to their respective treatment than to other treatments (16). This same set of
analyses, however, also revealed that significant overlap and commonalities existed among the
two treatments. Both treatments contained an equivalent degree of what has been termed
common facilitative conditions. IPT therapists evidenced some behaviors on the CBT scale,
while CBT therapists evidenced even more behaviors on the IPT scale. These overlaps in
technique and process leave open the possibility that significant elements of treatment process

Psychotherapy and Controlled Clinical Trials

24

associated with other approaches may be present and related to treatment outcome. In addition,
adherence scales typically only assess therapist interventions in isolation from aspects of
interaction with the patient. Specific therapist interventions in isolation from the larger treatment
process rarely correlate significantly with outcome (40). While the therapists in the TDCRP may
have intervened more often in modality-specific ways, IPT and CBT shared some similar
conditions (15, 16). Our results suggest the two treatments were quite similar when overall
treatment process, including the patient's contribution, was considered. Similarities in the process
of IPT and CBT remain hidden from the view of outcome research, until revealed by careful
examination of the therapy process itself. If it is impossible to accurately and fully control the
process of psychotherapy, then the basic premise of a randomized, controlled clinical trial is not
valid. Thus, such studies can only reveal whether a treatment works and not how it works. One
cannot make any assumptions about what promotes therapeutic change in psychotherapy, even
in the best of controlled clinical trials, until the therapy process is examined.
This series of studies also suggests that drawing conclusions about what promotes patient
change in controlled clinical trials may be misleading even when comparing presumably pure and
distinct forms of therapy where therapists are able to adhere more closely to a particular
treatment approach. For example, it has been shown in previous research that even when
cognitive-behavioral therapies were shown to follow the cognitive-behavioral model closely, the
presence of minimal psychodynamic ingredients was shown to predict positive outcome (7, 10,
41). This finding was replicated in the TDCRP where even the minimal amount of interpersonal
process fostered in the cognitive-behavioral therapies also contributed significantly to positive

Psychotherapy and Controlled Clinical Trials

25

outcome. Therefore, it has been shown using multiple datasets and treatment modalities that,
even in small doses, techniques borrowed from another approach can be powerful predictors of
outcome (7).
The controlled clinical trial method was initially designed by medical science for use in studies
of medications. A physician administers a specific medication knowing it is the only medication
being administered in order to compare the results to a placebo condition. Unlike medication,
psychotherapy cannot be administered in such pure form and adherence is much more difficult to
measure. The controlled clinical trial methodology is effective in investigating medical
interventions for comparing psychotherapy to pharmacotherapy or their combination. It is
limited however, when imposed on psychotherapy alone which is an entirely different enterprise.
The reality of clinical practice is that therapists do not exclusively utilize certain techniques over
others. In clinical practice, therapists do not adhere strictly to a particular treatment. It is
therefore futile to attempt to force them to do so for the sake of research. Rather, therapists often
consider disparate intervention strategies to be compatible with their own theoretical orientation.
Limitations
Our findings must be considered in the context of the limitations of our study design and
methods. As in all studies, it is important to consider issues of replicability and generalizability.
Because this study relies on a relatively small number of subjects, it is possible that some bias
exists in the sample despite random assignment to treatment conditions. It is plausible, therefore,
that with a different set of therapists and a different set of patients, the process components of

Psychotherapy and Controlled Clinical Trials

26

IPT and CBT may be found to be distinct and have little overlap. Future process studies using
other psychotherapy samples are needed to address the issue of generalizability of these findings.
Future Directions
The external validity sacrificed in controlled clinical trials does not seem warranted if the
primary goal of being able to establish causal relationships cannot be attained due to the
methodological shortcomings described above. Many prominent psychotherapy researchers have
called for a shift in focus to the empirical validation of change processes rather than treatment
types (5, 42, 43). Psychotherapy research would profit from the study of change processes as
they occur naturalistically, rather than focusing on the empirical validation of brand names of
therapy, which are often misleading.
An important limitation in comparative treatment studies is their emphasis on treatment
procedures to the point of overlooking the importance of the role of the individual patient in a
treatment's outcome. It is assumed that if the patient sample is homogeneous in terms of
diagnosis (major depression in the TDCRP) the most important source of patient variance has
been controlled. Our studies (17) have demonstrated that patient in-session characteristics were
far more important correlates of outcome than treatment type. This finding is consistent with
those of other investigators (44) who have routinely found that patient qualities, including
personality characteristics, are far more robust predictors of outcome than are treatment
techniques. Naturalistic studies that assess relevant patient characteristics, and their interaction
with therapists and their intervention strategies, would be a fruitful complement to randomized
clinical trials in therapy research.

Psychotherapy and Controlled Clinical Trials

References

27

Psychotherapy and Controlled Clinical Trials

28

Table 1
Rank Ordering of Q-items by Factor Scores on Cognitive-Behavioral Prototype Factor
20 Most characteristic items of ideal cognitive-behavioral therapy
PQS #
38
30
4
85
17
45
23
31
69
27
80
86
37
73
57
88
72
95
28
48

Item description
There is discussion of specific activities or tasks for the P to
attempt outside of session.
Discussion centers on cognitive themes, i.e. about ideas or belief
systems.
P's treatment goals are discussed.
T encourages P to try new ways of behaving with others.
T actively exerts control over the interaction (e.g. structuring,
introducing new topics).
T adopts supportive stance.
Dialogue has a specific focus.
T asks for more information or elaboration.
P's current or recent life situation is emphasized in discussion.
T gives explicit advice and guidance.
T presents an experience or event in a different perspective.
T is confident or self-assured (vs. uncertain or defensive).
T behaves in a teacher-like (didactic) manner.
P is committed to work of therapy.
T explains rationale behind technique or approach to treatment.
P brings up significant issues and material.
P understand the nature of therapy and what is expected.
P feels helped.
T accurately perceives the therapeutic process.
T encourages independence of action or opinion in P.

Factor score
1.93
1.68
1.51
1.49
1.45
1.43
1.38
1.37
1.35
1.32
1.28
1.21
1.17
1.14
1.13
1.09
1.08
1.06
1.05
1.02

Note. Factor scores derived from expert cognitive-behavioral therapists' (N=10) ratings of the
Psychotherapy Process Q-set. PQS = Psychotherapy Process Q-set; T = therapist; P = patient.
A footnote should state that this table is reprinted from Ablon & Jones (1998), "How expert
clinicians prototypes of an ideal treatment correlate with outcome in psychodynamic and
cognitive-behavioral therapy", Psychotherapy Research, 71-83.

Psychotherapy and Controlled Clinical Trials

29

Table 2
Rank Ordering of Q-items by Factor Scores on IPT Prototype Factor
20 Most characteristic items of ideal interpersonal therapy (IPT)
PQS #
63
81
33
64
57
23
75
66
2
40
16
3
65
79
4
69
45
26
96
28

Item description
P's interpersonal relationships are a major theme.
T emphasizes P's feelings in order to help him/her experience them
more deeply.
P talks of feelings about being close to or needing someone.
Love or romantic relationships are a topic of discussion.
T explains rationale behind technique or approach to treatment.
Dialogue has a specific focus.
Termination of therapy is discussed.
T is directly reassuring.
T draws attention to P's non-verbal behavior.
T makes interpretations referring to actual people in P's life.
There is discussion of body functions, physical symptoms, or
health.
T's remarks are aimed at facilitating P's speech.
T clarifies, restates, or rephrases P's communication.
T comments on changes in P's mood or affect.
P's treatment goals are discussed.
P's current or recent life situation is emphasized in discussion.
T adopts supportive stance.
P experiences discomforting or troublesome (painful) affect.
There is discussion of scheduling of hours or fees.
T accurately perceives the therapeutic process.

Factor score
2.22
1.65
1.62
1.58
1.55
1.39
1.32
1.29
1.27
1.25
1.20
1.19
1.15
1.13
1.10
1.09
1.09
1.05
0.94
0.91

Note. Factor scores derived from expert IPT therapists' (N=11) ratings of the Psychotherapy Process
Q-set. PQS = Psychotherapy Process Q-set; T = therapist; P = patient.

Psychotherapy and Controlled Clinical Trials

30

Table 3
Q-Prototypes Correlated with Outcome in NIMH Interpersonal and Cognitive-Behavioral
Therapy Treatment Samples
Q-Prototypes
Interpersonal Prototype

Cognitive-Behavioral Prototype

Outcome measures
Interpersonal treatment sample (N = 35)
HRSD - 23
SAS Global Score
SCL-90-R
BDI
DAS
GAS

.32*
.27
.29*
.37*
.24
.24

.45**
.51**
.36*
.52**
.13
.47**

Cognitive-behavioral treatment sample (N = 29)

HRSD - 23
SAS Global Score
SCL-90-R
BDI
DAS
GAS

.26
.11
.30
.41*
.48**
.22

.49**
.38*
.36*
.53**
.46**
.45*

Note. A positive correlation reflects a favorable association with outcome. All Pearson correlations are
partial correlations controlling for patient pretreatment scores. HRSD-23 = 23-item Hamilton Rating
Scale for Depression; SAS Global Score = Global Score on the Modified Social Adjustment Scale; SCL90-R = Symptom Distress Checklist - Revised; BDI = Beck Depression Inventory; DAS =
Dysfunctional Attitudes Scale; GAS = Global Assessment Scale of the SADS-C.
* p .05
** p .01

Psychotherapy and Controlled Clinical Trials

31

CORRELATIONS WITH
Q-PROTOTYPES IN CBT AND IPT

Correlations with Q-Prototypes

0.8

Cognitive-Behavioral Prototype
Interpersonal Prototype

0.6

0.4

0.2

CBT
N=29

IPT
N=35

Row Numbers

Figure Caption

Figure 1. Mean correlations of observer Q-sorts and Q-prototypes in NIMH interpersonal and
cognitive-behavioral treatment samples. Correlations were transformed from Pearson correlation
coefficients into z-scores using the Fisher r to z transformation before averaging.

Psychotherapy and Controlled Clinical Trials

32

1. Jacobson NS and Christensen A: Studying the effectiveness of psychotherapy: How well can
clinical trials do the job? American Psychologist 1996; 51:1031-1039
2. Seligman MEP: The effectiveness of psychotherapy. American Psychologist 1995; 50:965974
3. Seligman M: Science as an ally of practice. American Psychologist 1996; 51:1072-1079
4. Hollon SD: The efficacy and effectiveness of psychotherapy relative to medications.
American Psychologist 1996; 51:1025-1030
5. Goldfried MR and Wolfe BE: Psychotherapy practice and research: Repairing a strained
alliance. American Psychologist 1996; 51:1007-1016
6. Goldfried MR and Wolfe BE: Toward a more clinically valid approach to therapy research. J
Consult Clin Psychol 1998; 66:143-150
7. Ablon JS and Jones EE: How expert clinicians' prototypes of an ideal treatment correlate with
outcome in psychodynamic and cognitive-behavioral therapy. Psychotherapy Research 1998;
8:71-83
8. Goldfried MR, Raue PJ and Castonguay LG: The therapeutic focus in significant sessions of
master therapists: A comparison of cognitive-behavioral and psychodynamic-interpersonal
interventions. J Consult Clin Psychol 1998; 66:803-810
9. Hollon SD, DeRubeis RJ, Evans MD, Wiemer MJ, Garvey MJ, Grove MW and Tuason VB:
Cognitive therapy and pharmacotherapy for depression: Singly and in combination. Arch Gen
Psychiatry 1992; 49:774-781
10. Jones EE and Pulos SM: Comparing the process in psychodynamic and cognitive-behavioral
therapies. J Consult Clin Psychol 1993; 61:306-316
11. Elkin I, Shea T, Watkins JT, Imber SD, Sotsky SM, Collins JF, Glass DR, Pilkonis PA,
Leber WR, Docherty JP, Fiester SJ and Parloff MB: National Institute of Mental Health
Treatment of Depression Collaborative Research Program: General effectiveness of treatments.
Arch Gen Psychiatry 1989; 46:971-982

Psychotherapy and Controlled Clinical Trials

33

12. Elkin I: The NIMH Treatment of Depression Collaborative Research Program: Where we
began and where we are now, in Handbook of psychotherapy and behavior change. Edited by
Bergin AE and Garfield SL. New York, Wiley, 1994
13. Elkin I, Gibbons R, Shea MT, Sotsky SM, Watkins JT, Pilkonis PA and Hedeker D: Initial
severity and differential treatment outcome in the National Institute of Mental Health Treatment
of Depression Collaborative Research Program. J Consult Clin Psychol 1995; 63:841-847
14. Ogles BM, Lambert MJ and Sawyer JD: Clinical significance of the National Institute of
Mental Health Treatment of Depression Collaborative Research Program data. J Consult Clin
Psychol 1995; 63:321-326
15. Krupnick J, Sotsky SM, Elkin I, Watkins J and Pilkonis PA: The role of therapeutic alliance
in psychotherapy and pharmacotherapy outcome: Findings in the National Institute of Mental
Health Treatment of Depression Collaborative Research Program. J Consult Clin Psychol 1996;
64:532-539
16. Hill CE, O'Grady KE and Elkin I: Applying the Collaborative Study Psychotherapy Rating
Scale to rate adherence in cognitive-behavior therapy, interpersonal therapy, and clinical
management. J Consult Clin Psychol 1992; 60:73-79
17. Ablon JS and Jones EE: Psychotherapy process in the NIMH Collaborative Study. J
Consult Clin Psychol 1999; 67:64-75
18. Jones EE: Therapeutic Action: A Guide to Psychoanalytic Therapy, Northvale, NJ, Jason
Aronson, 2000
19. Jones EE, Cumming JD and Horowitz MJ: Another look at the nonspecific hypothesis of
therapeutic effectiveness. J Consult Clin Psychol 1988; 56:48-55
20. Jones EE, Parke LA and Pulos S: How therapy is conducted in the private consulting room:
A multidimensional description of brief psychodynamic treatments. Psychotherapy Research
1992; 2:16-30

Psychotherapy and Controlled Clinical Trials

34

21. Cattell RB: Factor analysis; An introduction and manual for the psychologist and social
scientist, New York, Harper, 1952
22. Stephenson W: The study of behavior: Q-technique and its methodology, Chicago,
University of Chicago Press, 1953
23. Nunnally JC: Psychometric theory, New York, McGraw Hill, 1978
24. McKeown B and Thomas D: Q methodology, Newbury Park, CA, Sage Publications, Inc.,
1988
25. Spitzer RL, Endicott J and Robins E: Research Diagnostic Criteria (RDC) for a selected
group of functional disorders, 3rd, New York, Biometric Research, New York State Psychiatric
Institute, 1978
26. Hamilton M: Development of a rating scale for primary depressive illness. British Journal of
Social and Clinical Psychology 1967; 6:278-296
27. Beck AT, Rush AJ, Shaw BF and Emery G: Cognitive therapy of depression, New York,
Guilford Press, 1979
28. Klerman GL, Weissman MM, Rounsaville B and Chevron ES: Interpersonal psychotherapy
for depression, New York, Basic Books, 1984
29. Beck AT, Ward CH, Mendelson M, Mock JE and Erbaugh JK: An inventory for measuring
depression. Arch Gen Psychiatry 1961; 4:561-571
30. Endicott J, Spitzer RL, Fleiss JL and Cohen J: The Global Assessment Scale: A procedure
for measuring overall severity of psychiatric disturbance. Arch Gen Psychiatry 1976; 33:766-771
31. Derogatis LR, Lipman RS, Rickels K, Uhlenhuth EH and Covi L: The Hopkins Symptom
Checklist (HSCL): A self-report symptom inventory. Behavioral Science 1974; 19:1-15
32. Elkin I, Parloff MB, Hadley SW and Autry JH: National Institute of Mental Health
Treatment of Depression Collaborative Research Program: Background and research plan. Arch
Gen Psychiatry 1985; 42:305-316

Psychotherapy and Controlled Clinical Trials

35

33. Block J: The Q-sort method in personality assessment and psychiatric research, Springfield,
Ill., Charles C. Thomas, 1961
34. Jones EE, Hall S and Parke LA: The process of change: The Berkeley Psychotherapy
Research Group, in Psychotherapy research: an international review of programmatic studies.
Edited by Beutler L and Crago M. Washington, DC, American Psychological Association, 1991
35. Lambert MJ and Hill CE: Assessing psychotherapy outcomes and processes, in Handbook
of psychotherapy and behavior change. Edited by Bergin AE and Garfield SL. New York, Wiley,
1994
36. Jones EE, Krupnick JH and Kerig PK: Some gender effects in a brief psychotherapy.
Psychotherapy 1987; 24:336-352
37. Goldfried MR, Radin LB and Rachlin H: Theoretical jargon and the dynamics of
behaviorism. The Clinical Psychologist 1997; 50:5-12
38. Blatt SJ, Zuroff DC, Quinlan DM and Pilkonis PA: Interpersonal factors in brief treatment
of depression: Further analysis of the National Institute of Mental Health Treatment of
Depression Collaborative Research Program. J Consult Clin Psychol 1996; 64:162-171
39. Hollon SD, Evans MD, Auerbach A, DeRubeis RJ, Elkin I, Lowery A, Kriss M, Grive W,
Tuason VB and Piasecki J: Development of a system for rating therapies for depression:
Differentiating cognitive therapy, interpersonal psychotherapy, and clinical management
pharmacotherapy. Unpublished manuscript 1988;
40. Beutler LE and Crago M: Psychotherapy research: An international review of programmatic
studies, Washington, D.C., American Psychological Association, 1991
41. Castonguay LG, Goldfried MR, Wiser S, Raue PJ and Hayes AM: Predicting the effect of
cognitive therapy for depression: A study of unique and common factors. J Consult Clin Psychol
1996; 64:497-504
42. Howard KI, Moras K, Brill PB, Martinovich Z and Lutz W: Evaluation of psychotherapy:
Efficacy, effectiveness, and patient progress. American Psychologist 1996; 51:1059-1064

Psychotherapy and Controlled Clinical Trials

36

43. Garfield SL: Some comments on empirically supported treatments. J Consult Clin Psychol
1998; 66:121-125
44. Blatt SJ, Quinlan DM, Pilkonis PA and Shea TM: Impact of perfectionism and need for
approval on the brief treatment of depression: The National Institute of Mental Health
Treatment of Depression Collaborative Research Program revisited. J Consult Clin Psychol 1995;
63:125-132