1

SUSTAINING AND ADVANCING PERFORMANCE IMPROVEMENTS

ACHIEVED BY CREW RESOURCE MANAGEMENT TRAINING

Gary Grubb, J ohn C. Morey and Robert Simon
Crew Performance Group
Dynamics Research Corporation
Andover, Massachusetts 01810

ABSTRACT

The results of experience in applying team
coordination training reflect the differences in areas
of emphasis between organizations. For example, the
US Armys Aircrew Coordination Training (ACT)
program emphasized mission performance measures
and changes in behaviors while the US Air Force Air
Combat Command/US Air Forces Europe
(ACC/USAFE) Crew Resource Management (CRM)
program relied heavily on attitudinal measures and
student end of course critiques. Both program
evaluation methods and measurement approaches
revealed near-term and significant improvement in
mission effectiveness and flying safety due to initial
team training. Results of the Army ACT validation test
showed significant improvement in aircrews overall
mission effectiveness between pre- and post-training
evaluation scenarios. The results of Air Force CRM
attitudinal pre- and post-training surveys revealed
statistically significant positive shifts in student
attitudes towards CRM behaviors and skills. Post-
training trend analysis indicates that CRM attitudes
and skills, like technical flying skills, show
significant decay over time without refresher training,
day-to-day emphasis and evaluation.

INTRODUCTION

Crew or Cockpit Resource Management (CRM)
programs were instituted in the 1980s, first in
commercial aviation and later in military aviation, to
address adverse mishap rate trends that showed the
inability of many aviators to work well together in
periods of high stress or workload (Helmreich, Merritt,
& Wilhelm, 1999). Minor aircraft malfunctions were
resulting in fatal accidents with alarming regularity.
While aviators generally displayed excellent knowledge
and understanding of aircraft systems, operating
procedures, rules and regulations and other technical
information, they often displayed a glaring inability to
communicate effectively, distribute workload, maintain
or regain situational awareness and make sound
decisions. Military aviation took note of the success of
CRM in the civilian sector and instituted similar
training programs (Orlady & Foushee, 1987).

The U.S. Army implemented its version of
Aircrew Coordination Training (ACT) in 1994
(Department of the Army, 1992). During fiscal years
1984-1989, crew coordination failures identified by
the Army Safety Center directly contributed to the loss
of 147 lives and $292 million in aviation accident
costs. In 1990, Army leadership directed the Army
Aviation Center to focus its aviation training and
evaluation on crew performance. We worked closely
with Army aviation training, evaluation, and safety
personnel to develop, validate, and field an ACT
Exportable Training Package. J ust before initial
fielding, the Army estimated savings of nearly $30
million and 15 lives per year as a result of
implementing the ACT training and evaluation
program.

A joint NASA/Military Airlift Command
workshop in May 1986 provided the impetus to begin
CRM training in the U.S. Air Force (USAF),
beginning in the transport aircraft community. By the
early 1990s, CRM had made inroads into the fighter
community. The first fighter CRM programs were
oriented more towards research than operations and
were not widely embraced by experienced fighter
pilots. There was a sense among them that CRM was
just another safety course of little value to
experienced, risk taking fighter pilots. In 1995, this
cultural reality caused the USAF to introduce new
CRM ideas and behavioral-based training approaches.

CRM programs have been structured in various
ways and continue to evolve as the perspective of what
makes effective CRM training changes. Most
programs seem to include at least these basic elements:

! A discussion of the core behaviors or basic skill
sets that make up CRM. Each program structures
these core behaviors differently, but all contain
common elements.
! An examination of the applicability of CRM
behaviors in the real world. This typically takes
the form of one or more case studies of real-world
incidents or accidents and includes an analysis of
where or when proper CRM behaviors could have
been employed.


2
! Some type of role-playing or practice of CRM
behaviors in a simulated mission setting, i.e.,
line-oriented flight training (LOFT) or its
equivalent.
! Some form of assessment of the learning or
changes in attitudes and behaviors that have
taken place as a result of the training, and the
evaluation of the training by the students.

DIFFERENCES IN ORGANIZATINAL EMPHASIS

Methods of obtaining feedback from CRM
training include student responses to discussions,
case study questions, test questions, attitudinal
surveys, course critiques and evaluations, and
instructor observations and assessments. Nearly all of
these were used in the ACC/USAFE and Army
CRM/ACT programs. Our experience in applying team
coordination training reflects the differences in areas of
emphasis between the Army and Air Force programs.

For example, the Armys ACT program
emphasized mission performance measures and changes
in ACT behaviors while the Air Force CRM program
relied heavily on attitudinal measures and students end
of course critiques.

Behavioral and mission performance measures

Army ACT program performance methods and
measures included:

! ACT behaviors or Basic Qualities evaluated with
supporting behaviorally anchored rating scales
! Aircrew Training Manual task performance
! Mission performance of two flight simulator
scenarios similar in difficulty in terms of time
stress, navigational demands, quantity and
capabilities of simulated threat.

Attitudinal measures

Air Force CRM effectiveness measures focused
on changes in student attitudes towards CRM. Grubb,
Morey, and Simon (1999) showed that an attitude
survey instrument based on the Theory of Reasoned
Action provided assessment capabilities to detect
student attitude changes as a result of training.
Student attitudes provided diagnostic information and
insights into specific objectives of course curriculum,
command climate influences, and intent to perform
CRM behaviors in daily operations. Attitude measure
results were used to determine areas of emphasis for
program and courseware improvements such as
training program themes, course content, and training
frequency.
CRM TRAINING RESULTS

We worked closely with the Army Aviation Center
and Air Force Air Combat Command to implement
ACT/CRM training and evaluation methods and
measures.

Behavioral and mission performance results

Results of the Army ACT validation test showed
significant improvement in aircrews overall mission
effectiveness between pre- and post-training evaluation
scenarios.

! Overall mission effectiveness +20%
! Navigation accuracy +38%
! Mission objectives +44%

During the Army ACT validation test, all aircrews
made errors in completing the complex, tactical
simulator missions (Simon & Grubb, 1993). Pre- to
post-training results improved markedly for
navigation, instrument flight recovery, and mission-
threatening error performance measures.

! Course deviations 45%
! Arrive at correct landing zone +38%
! No early descent below minimums +32%
! Manage mission threatening error +27%

Like the mission effectiveness increases, results of
the Army ACT validation test showed significant
improvement in aircrew flying safety between pre- and
post-training evaluation scenarios.

! Unexpected weather recovery +25%
! Emergency detection +38%
! Number of aircraft crashes 43%

Instructor-evaluators rated crew performance of
the set of thirteen Army ACT behaviors (called Basic
Qualities) during each testbed mission. There was
improvement between the pre-training and post-
training evaluations in every Basic Quality.
Statistically significant improvements were reached on
12 of 13 Basic Qualities.

As a result of ACT training, Army aircrews
learned behavioral skills and team coordination
techniques to keep them focused on what is important
and to deal with emergencies and unforeseen problems
in such a way that they do not lose sight of mission
objectives.





3
Attitudinal Results

During every Air Force CRM training class,
students were presented with 24 attitude items
divided into pre- and post-training instruments of
eleven and thirteen items each. The Air Force results
revealed statistically significant positive shifts in
student attitudes towards CRM behaviors and skills.
Figure 1 shows the mean pre-training and post-training
attitude scores from3,600 students combined for all
platforms (showing the eight core behaviors of the
original USAF program).

30
32
34
36
38
40
42
44
G
ro
u
p
D
y
n
a
m
ic
s
S
itu
a
tio
n
a
l A
w
a
re
n
e
s
s
S
tre
s
s
A
w
a
re
n
e
s
s
R
is
k
M
a
n
a
g
e
m
e
n
t
H
u
m
a
n
P
e
rfo
rm
a
n
c
e
W
o
rk
lo
a
d
M
a
n
a
g
e
m
e
n
t
E
ffe
c
tiv
e
C
o
m
m
u
n
ic
a
tio
n
M
is
s
io
n
P
la
n
n
in
g
M
e
a
n
A
t
t
it
u
d
e
S
c
o
r
e
Pretraining
Posttraining

Figure 1. Mean pre- and post-training attitude scores
for CRM behaviors.

Program improvement was managed through
quarterly program reviews, which focused on
summary class results, and annual adjustments were
made to the program based on these reviews.
Analyses of attitude data over multiple quarters of
training indicated that the group dynamics (GD) and
situational awareness (SA) CRM behaviors required
special attention. Instructors received feedback after
each class from computerized post-training reports,
and were able to make rapid changes to their
facilitation techniques.

TREND ANALYIS FINDINGS

Army ACT implementation had an immediate
positive impact on operational safety. The aviation
Class A accident rate dropped from 1.75 per 100,000
flying hours (1993) to .75 (1996), a reduction of more
than 50 percent (Figure 2).
0.0
0.5
1.0
1.5
2.0
2.5
FY 93 FY 94 FY 95 FY 96 FY 97 FY 98 FY 99
A
C
C
I
D
E
N
T

R
A
T
E
/
1
0
0
,
0
0
0

F
L
Y
I
N
G

H
O
U
R
S
1.6
1.34
H
E
L
I
C
O
P
T
E
R
A
L
L

A
C
F
T
ACT
Started
1.64
1.9
A
L
L

A
C
F
T
H
E
L
I
C
O
P
T
E
R
1.77
2.0
A
L
L

A
C
F
T
H
E
L
I
C
O
P
T
E
R
0.83
0.9
A
L
L
H
E
L
O
.74
1.2
A
L
L
H
E
L
O
1.26
1.3
A
L
L

A
C
F
T
H
E
L
O
1.97
2.03
A
L
L

A
C
F
T
H
E
L
I
C
O
P
T
E
R

Figure 2. Army aviation Class A accident trend.

Unlike the ACC CRM program, Army ACT was
presented as one-time training without annual
continuation or sustainment training. Not unexpectedly,
the aviation accident rate increased when ACT was no
longer emphasized.

Temporary measures such as awareness videos,
assistance visits, safety newsletter articles, and a web-
based training support package have been ineffective
substitutes for focused ACT training. The director of
Army Safety recently stated, In fact, FY99 produced
Army aviations worst safety performance since
Desert Shield/Desert Storm.

To see whether the Air Force CRM program had
any demonstrable effect on mishaps, we asked the Air
Force Safety Agency to provide the fighter/attack
operations factor mishap rates in ACC and USAFE for
the period before and since CRM implementation. The
results are shown in Figure 3.












Figure 3. ACC/USAFE Fighter/Attack Mishap Rates

Both commands had been experiencing gradual
reductions in their human factors-related mishap rates,
even before they instituted CRM training. Although
ACCs Ops Factor rate became somewhat erratic after
the CRM program was introduced, so was its overall
Class A rate, with 1997 being a particularly bad year.

Human factors related mishaps have traditionally
accounted for nearly 80 percent of all mishaps (Flying
Safety, 2000; GAO, 1996). After introducing CRM
training in ACC and USAFE, human factors related
ACC and USAFE Fighter/Attack Mishap Rates
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
5.0
FY93 FY94 FY95 FY96 FY97 FY98 FY99
ACC CRM Training
USAFE CRM Training
ACC and USAFE Fighter/Attack Mishap Rates
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
5.0
FY93 FY94 FY95 FY96 FY97 FY98 FY99
ACC CRM Training
USAFE CRM Training


4
mishaps accounted for 20 percent less than the GAO
estimate: approximately 60 percent of the total Class
A mishaps each year. Surveys administered during
annual continuation training revealed an increased
emphasis on the importance of CRM by unit leaders
and aircrews alike. Aircrew members consistently
reported the positive effect of CRM training on flying
safety and mission effectiveness.

Initial implementation of crew resource
management training had a positive effect on US
Army and US Air Force flying operations. Like
technical flying skills, CRM attitudes and skills
decay over time without command emphasis and
evaluation. The Armys increase in Class A accident
rate when ACT was not emphasized is a good
example. Trend analysis of Air Force aircrew
member attitudes toward CRM revealed a 25 percent
decline in attitude towards CRM after 12-14 months
(Figure 4).
Figure 4. Attitude trend between training events.

INITIATIVES TO SUSTAIN PERFORMANCE
IMPROVEMENTS

The challenge is how to sustain and advance the
cultural and teamperformance improvements achieved
by initial CRM training. To this end, we recommend:

! Fully integrate CRM into the organizational
structure, command climate, rules, and regulations
that set the stage for daily flying operations.
! Institute ongoing CRM program evaluation and
sustainment activities to keep training realistic, and
relevant to operational missions and conditions.
! Correlate mishap investigation and mishap data
analysis to the CRM program structure (core
behaviors) so that mishap investigation data can be
used to target specific areas of the CRM training
programfor increased emphasis.
! Include instruction on strategies, tools, and
techniques that apply CRM behaviors and skills
to avoid, trap, or mitigate aircrew error.

REFERENCES

Department of the Air Force. (1998). Air
force instruction 11-290: Cockpit/crew resource
management training program. Washington, DC:
Author.

Department of the Army. (1992). Aircrew
coordination exportable training package (Vol. 1-3).
Fort Rucker, AL: U.S. Army Aviation Center.

Grubb, G., Morey, J ., & Simon, R. (1999).
Applications of the theory of reasoned action model of
attitude assessment in the air force CRM program.
Proceedings of the Tenth International Symposium on
Aviation Psychology, 298-301.

Helmreich, R., Merritt, A., & Wilhelm, J .
(1999). The evolution of crew resource management
training in commercial aviation. The International
J ournal of Aviation Psychology, 9 (1), 19-32.

Lacoste, Gene (1999, December). Improving
Aviation Safety Performance. Flightfax27 (12).1-3.

Orlady, H. W., & Foushee, H. C. (Eds.)
(1987). Cockpit resource management training:
Proceedings of NASA/MAC workshop (NASA CP
2455). Moffett Field, CA: NASA Ames Research
Center.

Simon, R., & Grubb, G. (1993). Validation of
crew coordination training and evaluation methods for
army aviation (Tech. Rep. No. E-785U). Wilmington,
MA: Dynamics Research Corporation.

USAF Class A Mishaps. US Air Force Flying
Safety Magazine, J an/Feb 2000, 47.

U.S. General Accounting Office. (1996).
Human Factors: Status of efforts to integrate research
on human factors into FAAs activities (GAO/RCED-
96-151, J une 27,1996). Washington, D.C.: Author






Research for this report was supported by contract
F44650-95-0002. The views, opinions, or findings
expressed are those of the authors and should not be
construed as an official position of the U.S.
Department of Defense or its agencies.
Attitude Trend Between Training Events
96 FT Post Training to 97 CTPreTraining
-100%
-75%
-50%
-25%
0%
25%
50%
75%
100%
125%
0 100 200 300 400 500 600
Days Between Courses
%

A
t
t
i
t
u
d
e

C
h
a
n
g
e
Student DataPoint Best FitCurve
Mean (96 FT Post) = 24.4
n = 352