ICAO - Human Factors

ICAO - Human Factors
Flight Safety Human Factors

The publications on this page are free and can be downloaded directly from this web site by clicking the langauage
icon next to the title (EN= English; FR=French; ES=Spanish)
Articles from the ICAO Journals

Table of Contents
Top Ten Recommended Introductory Reading List
of Free ICAO Material
Accident / Incident
Avionics
Crew Resource Management
Fatigue
Flight Operational Quality Assurance (FOQA), Line Operations Safety Audit (LOSA),
Normal Operations Safety Survey (NOSS), & Threat and Error Management (TEM)
Language
Maintenance
General Seminars
Training
Miscellaneous
Cross-Cultural Issues
Top Ten Recommended Introductory Reading List of Free ICAO Material
Article Language
Degani, A., & Wiener, E. (1995). Designing coherent flight-deck procedures for use in advanced technology
aircraft. ICAO Journal, 50(2), 23-25.
ICAO Secretariat (1996). Awareness grows of importance of human factors issues in aircraft maintenance and
inspection. ICAO Journal, 51(1), 19-21, 24.
ICAO Secretariat (2003). Effectiveness of security inspections depends on human proficiency. ICAO Journal,
58(1), 21-24.
MacBurnie, E. (1993). Human Factors in Aviation. ICAO Journal, 48(7), 7-26.
MacBurnie, E. (1996). Flight-Deck Automation: A pilot's perspective. ICAO Journal, 51(5), 5-30.
MacBurnie, E. (1999). Human Factors Today: managing human error. ICAO Journal, 51(8), 5-30.
Maurino, D. (2000). ICAO human factors programme expands scope beyond flight deck and ATC facility.
ICAO Journal, 55(1), 15-16, 29.
MacBurnie, E. (2002). The LOSA experience: safety audits on the flight deck. ICAO Journal, 57(4), 5-31.
MacBurnie, E. (2004). Aviation Language Proficiency. ICAO Journal, 59(1), 4-27.
Accident / Incident
http://www.icao.int/ANB/humanfactors/icaojournallist.htm (1 of 5)16-02-2007 22:36:23

Article Language
Australian Transport Safety Bureau (2005). Improvised approach has catastrophic consequences for Ilyushin
76 freighter. ICAO Journal, 60(1), 22-25, 30.
Bureau of Air Safety Investigation (1995). Australian accident report includes examination of organizational
and management failures, ICAO Journal, 50(7), 9-12, 24.
Matthews, R. (2004). Ramp accidents and incidents constitute a significant safety issue. ICAO Journal, 59
(3), 4-6, 25.
Maurino, D. (1993). Efforts to reduce CFIT accidents should address failures of the aviation system itself.
ICAO Journal, 48(4), 18-19.
Maurino. D. (1997). Many safety analysts perceive need to apply proactive safety approach to investigations.
ICAO Journal, 52(2), 21-22, 29.
Maurino, D. (1998). Full integration of human factors knowledge in investigation process would further
enhance safety. ICAO Journal, 53(3), 14-15, 25.
Maurino, D., Reason, J., Johnston, N., Lee, R. (1995). Exploring the role of the transportation system and
human factors in the crash of Flight 1363. ICAO Journal, 50(7), 14-17, 26.
Maurino, D., Reason, J., Johnston, N., Lee, R. (1995). Six years after the Dryden tragedy, many accident
investigation authorities have learned its lessons. ICAO Journal, 50(8), 20-25.
National Transportation Safety Board (2002). Report explains accident involving MD-82 overrun after
landing on a wet runway. ICAO Journal, 57(1), 6-10, 24-26.
Paries, J. (1994). Investigation probed root causes of CFIT accident involving a new-generation transport.
Transportation Safety Board of Canada (2000). Safety Board issues report on crews loss of control on go-
around attempt. ICAO Journal, 55(1), 18-20, 27-28.
Avionics
Article Language
Boorman, D. (2001). Todays electronic checklists reduce likelihood of crew errors and help prevent
mishaps. ICAO Journal, 56(1), 17-20, 36.
Clark, L. (1997). Avionics incorporating human-centered design improves pilot interface with automated
system. ICAO Journal, 52(4), 11-12.
Degani, A., & Wiener, E. (1995). Designing coherent flight-deck procedures for use in advanced technology
aircraft. ICAO Journal, 50(2), 23-25.
MacBurnie, E. (1996). Flight-Deck Automation: A pilot's perspective. ICAO Journal, 51(5), 5-30.
Maurino, D. (1998). ICAO supports proactive approach to managing human factors issues related to advanced
technology. ICAO Journal, 53(5), 17-18, 27, 29.
Newman, T. and Courtney, H. (1998). Standards for addressing human factors during aircraft certification
prove difficult to define. ICAO Journal, 53(3), 11-13, 24.
Shun, C. (2003). Ongoing research in Hong Kong has led to improved wind shear and turbulence alerts.
ICAO Journal, 58(2), 4-6, 28.
Spruston, D. (1998). A number of safety issues related to flight deck technology require our urgent attention.
Tarnowski, E. (1999). Understanding design philosophy can help pilots benefit from modern automated flight
systems. 54(9), 22-24, 29-30.
Veitengruber, J., & Rankin, W. (1995). Use of crew-centred design philosophy allows the introduction of new
capabilities and technology. ICAO Journal, 50(2), 20-22.
Crew Resource Management
Article Language
Svatek, N. (1990). Techniques in CRM training. ICAO Journal, 45(10), 12-13.

Taggart, W. (1987). CRM- a different approach to human factors training. ICAO Journal, 42(5), 13-16.
Weisman, G. (1991). A new partnership in CRM training. ICAO Journal, 46(10), 10-12.
Fatigue
Article Language
Chittick, J. (1998). Preferential scheduling for aircrew can help address problem of short-term accumulated
fatigue. ICAO Journal, 53(3), 16-17.
Graeber, R., Rosekind, M., Connell, L. & Dinges, D. (1990). Cockpit Napping. ICAO Journal, 45(10), 6-11.
NASA Ames Research Center. (1997). Crew fatigue research focusing on development and use of effective
countermeasures. ICAO Journal, 52(4), 20-22, 28.
Singh, J. (2003). Study of pilot alertness highlights feasibility of ultra long range flight operations. ICAO
Journal, 58(1), 14-15, 30.
Flight Operational Quality Assurance (FOQA),

Line Operations Safety Audit (LOSA),
Normal Operations Safety Survey (NOSS), &
Threat and Error Management (TEM)
Article Language
Proceedings of the Fourth ICAO-IATA LOSA and TEM Conference, Toulouse, France, 16-17 Novemeber
2006
Holtom, M. (2000). Properly managed FOQA programme represents an important safety tool for airlines.
ICAO Journal, 55(1), 7-11, 26-27.
Logan, T. (1999). Trend toward wider sharing of safety data is resisted by industry concerns, ICAO Journal,
54(1), 7-9, 27.
MacBurnie, E. (2002). The LOSA experience: safety audits on the flight deck. ICAO Journal, 57(4), 5-31.
Maurino, D. (1998). Human factors training would be enhanced by using data obtained from monitoring
normal operations. ICAO Journal, 53(1), 17-18, 23-24
Maurino, D. (2004). ICAO examining ways to monitor safety during normal ATS operations. ICAO Journal,
59(3), 14-16.
Proceedings of the First LOSA Week, Cathay City
Hong Kong, 12 to 14 March 2001
Proceedings of the First Threat and Error Management Workshop, San Salvador, El Salvador, 30 April to 1
May 2002
Proceedings of the Second LOSA Week, Panama City, Panama, 27 to 29 November 2001
Proceedings of the Third LOSA Week, Dubai United Arab Emirates, 14 t 16 October 2002
Proceedings of ICAO/ASPA Regional Seminar on Safety Reporting, Threat and Error Management (TEM)
and Cabin Safety, Mexico City, Mexico, 10 to 11 March 2004
US Airways (2003). Newly implemented line operations safety audit produces valuable data for air carrier.
ICAO Journal, 58(1), 11-12, 28-29.
First ICAO-LATA LOSA & TEM Conference Dublin, Ireland, 5 to 7 November 2003
Second ICAO-IATA LOSA & TEM Conference
Seattle, Washington, 3 to 4 November 2004
Language
Article Language

Douglas, D. (2004). Language tests can promote safer communications in international civil aviation
operations. ICAO Journal, 59(3), 17-18, 25-26.
MacBurnie, E. (2004). Aviation Language Proficiency. ICAO Journal, 59(1), 4-27.
Maintenance
Article Language
Australian Transport Safety Bureau (2004). Maintenance safety deficiencies highlight need for periodic
review of safety systems. ICAO Journal, 59(3), 9-12, 24-25.
ICAO Secretariat (1996). Awareness grows of importance of human factors issues in aircraft maintenance and
inspection. ICAO Journal, 51(1), 19-21, 24.
National Transportation Safety Board (2003). Poor maintenance practices led to crash of Alaska Airlines
Flight 261. ICAO Journal, 58(2), 19-23, 30.
Purdue Unviersity (2000). Survey results suggest need for more effective reporting of aircraft maintenance.
ICAO Journal, 55(1), 17, 28-29.
General Seminars
Article Language
Egypt sponsors regional seminar on safety and human factors. (1992). ICAO Journal, 47(12), 24.
Human factors in spotlight at second regional safety seminar. (1991). ICAO Journal, 46(12), .24.
Human Factors the focus of milestone Leningrad seminar. (1990) ICAO Journal, 45(5) 28.
Maurino, D. (1994). ICAO workshops gather important knowledge on perceptions of human factors training.
Stewart, J. (2001). Safety seminars planned for regions where maximum safety gains may be achieved. ICAO
Journal, 56(1), 21, 36-37
Training
Article Language
First Iberoamerican Conference on Safety and Training in Civil Aviation, Madrid, Spain 5 to 7
November 2002. (Spanish version only)
Johnston, A., & Maurino, D. (1990). Human Factors training for aviation personnel. ICAO Journal, 45(5), 16-
19.
Maurino, D. (1995). ICAO annex amendment introduces mandatory human factors training for airline flight
crews. ICAO Journal, 50(7), 13, 24-25.
Orlady, H. (1994). Airline pilot training programmes have undergone important and necessary changes in the
past decade. ICAO Journal, 49(3), 5-10.
Miscellaneous
Article Language
Courville, B. & Thisselin, J. (2004). Applying take-off thrust on unsuitable pavement surface may have
hidden dangers. ICAO Journal, 59(3), 7-8.
Edwards, C. (2000). Aircraft operators have built a generic hazard model for use in developing safety cases.
ICAO Journal, 55(1), 12-14, 27.

Foreman, P. (1998). Proposed free flight environment raises a number of pressing issues for the worlds
pilots. ICAO Journal, 53(5), 9-12, 27.
Gerstenfeld, A., & Stein, E. (1995). Simulation is a vital tool in the air traffic control human factors and
technical research, ICAO Journal, 50(3), 19-20.
ICAO Secretariat (2003). Effectiveness of security inspections depends on human proficiency. ICAO Journal,
58(1), 21-24.
ICAO Update (2003). Council President meets with authorities in France and Spain, addresses several events.
MacBurnie, E. (1999). Human Factors Today: managing human error. ICAO Journal, 51(8), 5-30.
Maurino, D. (1991). Education is key to ICAO's human factors programme. ICAO Journal, 45(10), 16-19.
Maurino, D. (1991). Management decisions have an impact on flight safety. ICAO Journal, 46(10), 6-9.
Maurino, D. (1998). ICAO supports proactive approach to managing human factors issues related to advanced
technology. ICAO Journal, 53(5), 17-18, 27, 29.
Maurino, D. (2000). ICAO human factors programme expands scope beyond flight deck and ATC facility.
ICAO Journal, 55(1), 15-16, 29.
Maurino, D. (2001). Amendment to PANS-OPS includes human factors related provisions for aircraft
operations. ICAO Journal, 56(1), 7-9, 34-36.
Mawdsley, D., & Maurino, M. (2005). Cabin safety can be enhanced through application of human factors
strategy. ICAO Journal, 60(1), 17-19, 30.
Pooley, E. (1999). Putting air safety management into practice demands a positive corporate safety culture,
Stewart, J. (1993). System approach to risk management focuses resources on most serious hazards. ICAO
Journal, 48(9), 12-13.
Cross-Cultural Issues
Article Language
Proceedings of the First ICAO Regional Seminar on Cross-Cultural Issues in Aviation Safety, Asia and Pacific
Region, Bangkok, Thailand, 12 to 14 August 1998
Proceedings of an ICAO/ASPA Regional Seminar on Cross-Cultural Issues in Aviation Safety, Mexico City,
Mexico, 5 to 6 March 2003

Awareness grows of importance of human factors
issues in aircraft maintenance and inspection
Until recently, the role of human factors in maintenance operations was not adequately addressed.
That is changing, because it is clear that human error in aircraft maintenance can have a serious
impact on flight safety.
nance procedure was "proscribed" (i.e

prohibited) in a service bulletin. The tech-
nician reading this concluded that the pro-
cedure was "prescribed" (i.e. defined, laid
T
RADITIONALLY, efforts to improve down) and proceeded to perform the for-
awareness of human factors in avia- bidden action. These types of problems
tion have been directed towards are becoming more prevalent now that air
flight crew performance and, to a lesser carrier aircraft are being manufactured all
extent, towards the performance of air traf- over the world. Sometimes the technical
fic controllers. Until recently, available lit- language of the manufacturer does not
erature showed little consideration of the translate easily into the technical language
human factors issues which could affect of the customer and the result can be
aircraft maintenance personnel who maintenance documentation that is diffi-
inspect and repair aircraft. There is at pre- cult to understand. Since so much mainte-
sent a growing awareness of the impor- nance information is written in English,
tance of human factors issues in aircraft there is a strong case to be made for use of
maintenance and inspection. The safety "simplified" English. Words that mean onc
and effectiveness of airline operations are thing to a certain reader should mean t h t
also becoming more directly related to the same thing to every other reader. For
performance of the people who inspect and example, a "door" should always be called the pylon attachment area. It is believed
service the aircraft fleets. Following is a a door. It should not be referred to as a that if this experience had been shared
review of key human factors issues in air- "hatch" or a "aanel." with other operators of similar aircraft, the
craft maintenance operations: communica- Communication with the aircraft manu- accident at Chicago might not have hap-
tion, training, hours of work, medical facturer, as well as between airlines, can be pened.
considerations and impact of the work crucial. If an operator, in maintaining its Before airlines are likely to engage in
environment. aircraft, discovers a problem that could such cooperation, however, they must be
degrade safety, then that problem should confident that such information will be
Importance of good communication be communicated to the manufacturer and used for accident prevention purposes only.
Communication is possibly the most to other operators of the same aircraft type. The use of such information to gain a mar-
important human factors issue in aircraft This is not always easy to do. Industry cost keting advantage over the reporting airline
maintenance. Without communication control measures and competitive pres- can only result in stifling all safety-related
among maintenance managers, manufac- sures may not place a premium on commu- interactions among operators.
turers, dispatchers, pilots, the public, the nication among airlines. However, civil avi-
government and others, safety standards ation authorities can play an important role Personnel training
would be diicult to maintain. by encouraging operators under their juris- Training methods for aircraft mainte-
It is most important that maintenance diction to interact frequently with one nance technicians vary throughout the
information be understandable to the ta-get another and the manufacturer of the air- world. In many States a common proce-
audience. The primary members of this craft they operate. dure is for a would-be technician to enrol in
audience are the inspectors and technicians A number of accidents could have been a relatively short-term course of training at
who undertake scheduled aircraft mainte- prevented if incident information from air- an aircraft maintenance technician training
nance and diagnose and repair aircraft mal- lines had been made known to the indus- centre. These centres provide training in
functions. New manuals, service bulletins, try. For example, the investigatio~lof the the skills required to pass examinations
job cards and other information to be used American Airlines DC-10 accident at Chica- given by the civil aviation authority (CAA)
by this audience should be tested before go in 1979 revealed that another airline had for the airframe and powerplant (A&P)
distribution to make sure that they will not been using the same unapproved engine technician's licence or certificate. In addi-
be misunderstood or misinterpreted. change procedures, in which the pylon and tion, it is possible in many States to obtain
Sometimes maintenance information is engine were removed and installed as a certification through an apprenticeship-
conveyed through a less-than-optimum unit rather than separately. Unlike Ameri- type programme whereby, over a period of
selection of words. Anecdotal evidence can, however, the other airline had discov- years, individuals learn their craft using on-
suggests a case where a certain mainte- ered that the procedure caused cracks in the-job training (OJT) methods.
and mphishted eI%@mkwskm 3t is
bp&t to provide &merive ehsmoom-
based training on u&d~lj$ng 6ystei prizlei-
pla. This is dBmIt to do wkde thejob
u~fi.11in maintaining generd &&on &-

waft+are not often. m d e d in m&n@in"tng
the airline Beet of cmn$kx, turbine-pm- with smdlw ~arriemand, in such
no p h r tdnfng in t r r i n g catres, In
the= GIBE%, the airlines are required to
provide prachlly all of the trzdniw.
Airline training &auld be a of &m-
mquire;d to maintain a fist of m o d m air-
cr&, mi&may indude taking maximal
j d a r or less tzqxs5-d g e m m 73%

tm?m.ef@ expected b th.ts$ming
a d dmm&ate'thi$ newlg tm@$re-$ how&
.edge g,e the m~skction0-f the t r & If~dl~
she and w m t l w t i o n of the t3paid.n~pro-

gmgum. However, mo& of the mmftuter-
bmd bstruction ccrsrendy Iln use wauld
damroam mdhod~.Certainly these new

tr8'id h light. wheduled mdntenanca Inkingkdumlogie~
would be expected ta
m p h 01d~sQIeCBI.
Usually M e or na remedial im&ucBan is
.aide mare Twm&

ICAO JOURNAL
agement should be aware of the hazards of conditions such as well lighted, comfort- situations. Task lighting needs a range of
such activities as repetitive inspection of able hangars for aircraft maintenance work, brightness, from 200 to 500 foot-candles,
identical items. A long history of research such is unlikely given the cost of building depending on the task. Affordable portable
shows that operator vigilance declines and operating these facilities at every air- lighting units which can be positioned near
rapidly on these tasks and error can follow. port served by airlines. Consequently, work areas or attached to adjacent struc-
Similarly, use of certain types of equipment much aircraft maintenance is performed tures for the performance of specific tasks
is associated with work error. Old-style under less-than-ideal conditions including are available in various sizes and intensities.
inspection devices rely heavily on techni- outdoor, night work in inclement weather. Outdoor, night-time activity demands
cians' skill in manipulating equipment and One of the most important work para- careful attention to lighting needs. A great
in detecting and interpreting subtle instru- meters in aircraft maintenance is lighting. deal of aircraft maintenance is performed
ment indications. Couple these difficulties It is very difficult to provide adequate light- under these conditions. There is an unfor-
with a fatigued technician, and the probabil- ing for all aspects of maintenance work, tunate tendency to rely on flashlights or
ity for error increases dramatically. including inspection and repair. Poor ambi- ambient lighting from open hangar doors
Shift supervisors need to be especially ent illumination of work areas was identi- for this work, because adequate portable
observant of technician fatigue and should fied as a significant deficiency during some lighting is either unavailable or time-con-
oversee and perform follow-up checks of accident investigations. suming lo obtain and set up. Management
tasks to discover any resulting errors. A great deal of lighting for specific tasks must be aware of the importance of provid-
Inspection during daylight hours of mainte- is provided by hand-held torches or flash- ing and requiring the use of adequate area
nance work accomplished the previous lights. The advantages of these lights are and task lighting. It is not a trivial issue.
night could also go a long way towards Adverse occurrences, resulting, at least
reducing the probability of an error.
Medical considerations
4 New training
partly, from lack of adequate lighting, are
often identified in many accident investiga-
tion reports.
Noise is another important factor affect-
Technician health and physical status technologies are being ing performance. Aircraft maintenance
can also influence work performance. Air-
craft maintenance and inspection activity is developed which may operations are usually intermittently noisy
sometimes physically demanding, especial- because of activities such as riveting,
ly on the maintenance technician who is complement or even machinery operation inside hangars, or
overweight, sick or poorly conditioned, and engine testing or run-up on ramps. Noise
could result in work being skipped, uncom- replace on-the-job can cause speech interference and can also
pleted or improperly performed. The need have health implications. Loud or intense
for good vision and sometimes for normal training and classroom noise tends to rcsult in heightened
colour vision is important as well. Older response of the human autonomic nervous
people frequently need vision correction in
methods. system. One of the results can be fatigue.
the form of glasses or contact lenses. Perhaps more important is the effect of
At present, there are no medical require- noise on hearing. Regular exposure to loud
ments for aircraft maintenance technicians. that they are portable and require no set- noise can result in permanent hearing loss.
As is the case with many people, techni- up time. Disadvantages include the lack of Lower-intensity noise can cause temporary
cians may not attend to visual deficiencies brightness and the fact that they usually hearing loss which can have safety implica-
on time, especially when we consider the encumber one hand, sometimes forcing tions in the workplace. Missed or misun-
fact that without periodic examinations, maintenance work or inspection activity to derstood communication resulting from
detection of gradual visual deficiency is dif- be performed with one hand only. noise interference or hearing loss can have
ficult until vision has deteriorated signifi- One frequently noted problem in several serious consequences.
cantly. Moreover, the technician may expe- observed maintenance hangars is poor Steps that can be taken by operators to
rience job insecurity and therefore avoid area lighting. Often hangar area lighting is deal with noise problen~sinclude control-
reporting failing eyesight. provided by ceiling-mounted units. These ling sources of noise by enclosing or insu-
Currently it is rare to find an operator or hard-to-reach units are frequently dust- or lating machinery, isolating noisy activities
administration that requires regular med- paint-coated, and burnt-out bulbs some- so that fewer people are exposed, provid-
ical screening of technicians to detect dis- times go unreplaced for a long time. In ing workers with hearing protection (and
orders that may impair their work perfor- addition, the number and placement of requiring its use), reducing engine run-up
mance. However, because of the increasing these units are sometimes insufficient to or testing to the minimum acceptable, and
correlation between aviation safety and the provide good area lighting conditions. Area measuring noise levels in work areas.
performance of maintenance technicians, it lighting in hangars should be at least in the Noise monitoring can identify where prob-
may be timely to consider implementing order of 100 to 150 foot-candles to provide lems exist, thereby enabling management
regular medical screening. adequate lighting. to take corrective actions. The serious con-
Maintenance and inspection tasks per- sequences of noise exposure should be
Work environment formed beneath aircraft structures and with- stressed so that workers see the need for
To understand human error in mainte- in confined spaces pose difficult lighting hearing protection and for controlling the
nance, it is essential to understand the problems. The structure shades work points level of noise wherever possible. Exposure
technician's responsibilities and to be from area lighting and, similarly, cramped to noise levels above 110 decibels should
familiar with the work environment in equipment compartmcnts will not be illumi- not exceed 12 minutes in an eight-hour
which maintenance is performed. nated by ambient hangar lighting. Special period, and continuous exposure to 85
While it is desirable to have ideal work task lighting should be provided for these continued on page 24
21
New panel to focus on
legal framework for GNSS Cost-benefit analysis
continued from page 15
The ICAO Council decided recently to establish a new panel
of legal and technical experts to evaluate a possible legal
By and large the productivity gains have not been retained by the
airlines over the long term since there has not been an improving
framework for the global navigation satellite system (GNSS).
Under terms of reference established by the Council, the
trend in financial performance. Although the impact of productiv-
panel of experts will consider the different types and forms of
ity improvements has been offset to some extent by the increases
in real input prices in the 1970s, most of the cost savings associ-
a long-term legal framework for GNSS, citing strengths and
ated with the full range of labour, fuel and aircraft productivity
weaknesses of various alternatives, and will also explore the
improvements have been passed on to the consumer in the form
possible need for a convention.
The panel will be composed of approximately 20 to 25
of lower fares and rates.
members nominated by Contracting States. Several interna-
This evidence suggests that the CNS/ATM systems may gen-
tional organizations have also been invited to nominate
erate reductions in fares and rates paid by passengers and ship-
pers of air freight. Lower fares and rates may eventually lead to a
observers 17
further round of benefits associated with an increase in travel
..................................................................................................................... generated by the price reductions.
MD-1I improvement programme Conclusion

continuedfrom page 9 Cost-benefit analysis of implementation of the CNS/ATM sys-
tems has been illustrated here by application of the net present
Passengers and crew will appreciate the ER's satellite commu-
value methodology in a single State in a process which involved
nications (satcom) system, which allows them to maintain voice,
many assumptions. The impact of alternative assumptions was
facsimile or personal computer connections on long over-water
investigated as part of an assessment of the financial risk.
flights. And airlines benefit from an additional revenue source
Because many assumptions undcrlic the cost-benefit evalua-
while enhancing service.
tions, it is not advisable to focus on a particular figure as repre-
Closely related to the satcom is the provision of future air nav-
senting the net economic impact of the implementation of the
igation (FANS) equipment for using the satellite-based communi-
CNS/ATM systems in a State However, consideration of the
cations, navigation and surveillance/air traffic management
range of possible cost-benefit results, and the probabilities or like-
(CNS/ATM) systems. This concept combines the elements of the
lihood of the associated combinations of assumptions, can indi-
aircraft navigation using satellite signals with ground-based air
cate whether investment in the new system would be likely to
traffic control (ATC) via a data link. Thc ER model will also fea-
yield a rate of return which should be sufficient to cover the
ture reduced vertical separation certification, which will allow
financial cost of funds required.
preferred routing on heavily congested routes such as over the
The economic impact of the new systems will differ among
North Atlantic.
States because of varying conditions and geographical features.
The MD-11ER provides operators with the ability to navigate
More important, the impact in any one State, and in a region, will
using the US. global positioning system (GPS) and to maintain a
depend on the success of international cooperation in the imple-
link with air traffic control using the satcom system. As FANS
mentation process. 0
develops, MD-11ER operators will benefit from more flexible nav-
igation, along with more timely traffic and weather information. .....................................................................................................................
These features can be retrofitted into previously delivered MD-
11s.
Human factors
McDonnell Douglas is committed to having the MD-11ER avail- continuedfrom page 21
able to customers by the end of the first quarter of 1996. decibel noise levels requires hearing protection. Both noise and
light levels can be easily measured with relatively inexpensive
hand-held meters. These are tasks that can be accomplished by
the operator's health or safety departments or by supervisors
INTERNATIONAL CIVIL AVIATION ORGANIZATION who have been trained in the use of this equipment.
Toxic materials in aircraft maintenance have become more preva-
Third Flight Safety and Human lent with the advent of more sophisticated aircraft that use compos-
ite materials in their structure or other hazardous substances such
Factors Global Symposium as tank sealants or structural bonding chemicals. Some non-
destructive evaluation methods such as X-rays are also potentially
hazardous. Employees should be informed of the hazards associat-
Aviation 2000: Integrating Human Factors Knowledge
ed with handling toxic materials. They should be instructed in prop
and Practice into Tomorrow's Aviation System er handling methods and provided with protective devices such as
protective clothing, rubber gloves and goggles.
9-1 2 April 1996
Although lhe above irilorrriatiori addresses rrlaintenance tech-
Sheraton Auckland Hotel and Towers, Auckland, New Zealand nicians' health and safety considerations, it has obvious implica-
No Registration Fee tions for aviation safety. It is evident that technicians whose per-
For information and registration contact: formance is impaired because of lack of health and personal safe-
Daniel Maurino or Haile Belai, InternationalCivil Aviation Organization, ty provisions will be more likely to commit errors affecting the
1000 Sherbrooke St. W., Montreal, Quebec, Canada H3A 2R2 overall safety of aircraft operation. This is of great concern
Telephone. (514) 286-6381 /286-6409 Facsimile: (5 14) 285-6759 because, as a general rule, the effects of human error in mainte-
nance are usually manifested in a time and location far displaced
from the maintenance shop. 0
ICAO JOURNAL
uman Factors
Magazine of the International Civil Aviation Organization
VOL. 48 - NO. 7 SEPTEMBER 1993
FEATURES
7 New approaches and commitments will be needed before we can resolve
the human factors problem in aviation.
8 CRM training has achieved notable success, but if the technique is to reach
its potential, there is a need for further dcvelopment of training curricula
and instructional methods.
10 The inexperienced pilot tends to accept readily the importance of human
factors as an integral aspect of pilot training.
14 Aer Lingus' innovative multi-crew training course integrates human factors
and technical training to prepare ab initio airline pilots for aircraft type
transition training.
18 In time it may be possible to optimize training by tailoring the instruction and
scenarios to recognize the great differences that exist between modern and
traditional cockpits.
20 We are only now beginning to understand what constitutes effective decision
making and what skills should be taught.
23 Conclusions from the report on the CFIT accident near Kathmandu on
31 July 1992.
DEPARTMENTS
27 ICAO Update
30 Posts Vacant
COVER
Human factors play a critical role in every aviation activity, from flight training to air-
line management. Statistics attribute about 75 per cent of aircraft accidents to lapses in
human performance, and ICAO has given the highest priority to increasing awareness
of the human factors considerations in all aviation disciplines.
Editor: Eric MacBumie Production Clerk: Denise Cooper-Altuve

Edztorzal Asszstant. Lyne Bertrand Deszgn Consultant' Rodolfo Bore110
THE OBJECTIVES of the Journal are to prov~dea conclse account of the actwltles of the IntemaDonal Cw11
A v ~ a t ~ oOrgan~zatlon
n and to feature a d d ~ t ~ o nmformatlon
al of Interest to Contractmg States and the
mntemabonal aeronaut~calworld. Reproduct~on~nwhole or m part of all uns~gnedmatenal 1s freely authonzcd.
For rlghts to reproduce s~gnedartxles, please wnte to the edltor. Published 10 times annually In Enghsh,
French and Span~sh
OPINIONSEXPRESSED in signed articles or in advertisements appearing in the ICAO Journal are the
author's or advertiser's opinions and do not necessarily rejkct those of ICAO. The mentlon of specific
companies or products m art~clesor advert~sementsdoes not imply that they are endorsed or recommended by
ICAO ln preference to others of a similar nature which are not mentioned or advert~sed.
NOTICE:ADVERTISERS have been adv~sedthat ~t1s preferable to publish advertisement text ~nthe language of
each edlbon. Those advertlsements which do not appear m the language of each ed~tlondo so at the lnslstence
of theu sponsors.
ADVERTISINGREPRESENTATIVE. Westem Med~aSales, Execuhve Office Pla~a,50 West Hillcrest Dr~ve,
S u ~ t e215, Thousand Oaks, CA 91360, U.S.A. Western U.S. and Paclfic Reg~onalRepresentative: He~de
Thorstenson. Telephone: 1805) 496-3500. Telex: 182640 WKVG. Fax: (805) 496-3522. All other reglons.
Western Medla Intemat~onal,800 Blvd Rent'-Lhesque West, Su~te2450, Montreal, Quebec, Canada H3B
4V7. Telephone: (514) 849-5892. Fax: (514) 393-3965.
OTHER CORRESPONDENCEshould be addressed to: The Editor, ICAO Joumal, Sulte 327, 1000 Sherbrooke
Street West, Montreal, Quebec, Canada H3A 2R2. Telephone. (514) 285-8222. Telex: 05-24513. Fax. (514)
288-4772. Publ~shedin Montreal (Canada). Second-class mail registration No. 1610. ISSN 0018 8778.
SUBSCRIPTIONS: US$20.00 per year (by surface), US$30.00 per year (by cur). For related communications,
D.M. Newman contact ICAO Document Sales U n ~at
t the above address, or telephone (514) 285-8026
Headed in the right direction
New approaches and commitments will be needed before we can resolve
the human factors problem in aviation.
5 AVIATION moves towards the Z l s t ten- ~ ~

.............. ..........................................
~ ~ ~ not ~always predictable,
. . with ~occasionally dire
A tury, one of our key objectives is the suc-

cessful application of human factors
knowledge in all aspects of the industry's - opera-
.
DR. PHILIPPE ROCHAT
ICAO SECRETARY GENERAL
..............................,,.........................................
results. Traditional approaches to personnel train-
ing and accident prevention must be re-evaluat-
ed. Second, we must improve the dialogue within
tions. the international civil aviation community. Today,
ICAO has initiated a sustained campaign t o increase the aware- more than ever, designers, regulators, trainers, safety investigators,
ness of the pervasiveness of human error in aviation. Today, the researchers and operational personnel must maintain an open and can-
aviation community can take advantage of employment selection cri- did exchange. Each needs t o know the solutions that the others are
teria that can predict successful on-the-job performance, and tech- exploring to improve aviation safety. The best engineering solution
nology gives us the potential, by way of highly automated may perhaps be in conflict with limitations inherent to humans. The
equipment, t o "engineer" the human error out of aviation. best training solution may not be applicable due t o constraints inher-
In spite of these commendable endeavours, statistics attribute ent t o design. Results of research may not be relevant t o the needs of
most accidents t o lapses in human performance, under the regret- an operational environment. Prevention lessons learned through the
tably recurring label of human error. I raise the question: is it possi- investigationof accidents may perhaps prove difficult to translate into
ble by way of education, training and new technology to improve action unless these are advanced in a meaningful context.
these statistics? Are we on the right track? The President of the Finally, as it relates t o aviation safety, we must think in collective
Council of ICAO, in opening the 29th Session of the ICAO Assembly rather than in individual terms. We must think in terms of system
last October, stated that ICAO attaches considerable importance to error rather than individual human error. This is nowhere more evi-
human factors. "While the safety record of civil aviation is highly dent than in the implementation of the ICAO CNSIATM systems.
commendable," he stated, "the fact remains that about 75 per cent Observing a systems approach to the design and implementation of
of all aviation accidents are due to human error." He further added, these systems, we achieve - again potentially - the synergistic
and in my own view this is the crux of the matter, "new skills, new combination of the best technology can produce and the best
approaches, new commitments are needed to resolve this particular humans can perform. If we do not, we may squander the significant
problem." system benefits. This must not be allowed t o happen.
I believe we are headed in the right direction. However, we will Attitudes towards human factors are changing. But change is sus-
make tangible progress only when we make the commitments and tainable only when it starts at the top. Change, as well as resources
adopt the approaches the dynamic international civil aviation system and safety, must be managed. Those who can best effect change are
demands. those who, by virtue of their positions, can make strategic decisions
Specifically, first we must recognize the challenge of increased tech- t o change direction and commit resources as necessary.
nology and automation, both in the interest of improved safety and Of course, t o implement management initiatives, those in charge
efficiency. The interactions between people and high technology are of the practical, hands-on implementation must possess the neces-
sary skills to achieve it. New challenges require new skills. New skills
can be acquired through training, but if new training approaches
are t o be developed, they must be preceded by a process of educa-
tion of the strategic decision makers in aviation. Aviation managers
must understand the concepts and challenges involved in these new
approaches t o safety.
One thing is clear: there is a disparate level of understanding
about aviation human factors in different regions of the world. We
are trying t o overcome this imbalance through the educational
efforts of the ICAO Flight Safety and Human Factors Programme,
which is designed to increase awareness of middle and senior man-
agers within the international aviation community about the impor-
tance of human factors in civil aviation.
ICAO is also using i t s Technical Cooperation Programme to pro-
vide assistance t o developing States in implementing human factors
The Second ICAO Flight Safety and Human Factors Global Sympo-
endeavours. The Trainair Programme, a major initiative recently
sium was held in Washington in April 1993. Shown at the sympo-
sium (I-r) are: Don Newman, Representative of the Unitedstates on
established by ICAO t o promote effectiveness of training within
the Council of ICAO; Dr. Philippe Rochat, ICAO Secretary General; international civil aviation, is an additional asset which will be used
Robert D. Cook, member of the ICAO Air Navigation Commission; to pursue human factors training and education. 0
Garland Castleberry,Associate Administrator for Aviation Standards,
This article is an adaptation of the opening address given by Dr. Rochat at the Second
U.S. Federal Aviation Administration; and Dr. William Fromme, Flight Safety and Human Factors Global Symposium. Copies of the symposium pro-
Director of the ICAO Air Navigation Bureau. ceedings (ICAO Circular 243) can be obtained from the ICAO Document Sales Unit.
SEPTEMBER 1993
CRM training has achieved notable success, but if the technique is to reach its potential,
there is a need for further development of training curricula and instructional methods.
................................................................................. tation of CRM will not be complete until fic controllers, mechanics, dispatchers,
ROBERTL. HELMREICH acceptance of its concepts is more uniform ground operations personnel, etc. Many of
THEUNIVERSITYOF TEXASAT AUSTIN among the organizations that use it. the observed problems in the aviation sys-
Just as CRM training has evolved over tem involve the interfaces among these
(UNITEDSTATES)
time, the research associated wtih it has groups (for example, miscommunication
................................................................................ changed in direction and focus. We are no and misunderstandings between pilots and
NSTRUCTION in the human factors of longer concerned with the basic question air traffic controllers). A number of new
crew coordination and communication, of whether human factors training has a courses incorporate these ideas, providing,
now known as crew resource manage- measurable impact, but rather with a new for example, specialized training for first
ment (CRM) training, has become world- set of questions that have arisen from the officers upgrading to captaincy. This cap-
wide in the last decade. Our research earlier investigations. taincy course concentrates on the interface
group has been studying the impact of It is important to recognize that most of the cockpit crew and especially the c a p
CRM throughout this period. Our data new CRM courses are very different from tain with other organizational elements, as
show that initial and recurrent training in early programmes that were derived from shown in Figwe 1. It also focuses on
these concepts, combined with practice in corporate management development train- improving specific behaviours by the cap-
line oriented flight training (LOFT), and ing. We are now encountering a '"third gen- tain that were discovered in our research to
continuing feedback and reinforcement, eration" of CRM programmes that focus on be the most important determinants of crew
lead to continuing improvement in crew specific behaviours and behavioural strate- effectiveness. These are the use of briefings
performance over time. gies and take a systems approach to as a means of building the team concept,
The notable successes of CRM training human factors. However, the success of communications and decision making, inter-
should not be interpreted as meaning that CRM does not mean that the technique personal skills, and leadership.
CRM training is now fully developed and has reached its potential. There is a need We have been studying the impact of
optimally effective. The data show large for further development of training curric- LOFT in a number of airlines and have
and significant diifcrcnccs bctwccn organi- ula and instructional methods. found that crews rate this training highly
zations in the impact of training and in the One of the most important insights for both human factors and technical pur-
levels of performance achieved. Even with- reflected in the new programmes is that poses. However, we have also observed
in organizations, large variability in perfor- CRM concepts and training should not be considerable unevenness in the quality of
mance continues to exist and recognizable limited to the flight deck. In the course of scenario designs, inadequate and incom-
subcultures that differ in human factors their work, crews interact with a number of plete briefings and debriefings, instructor
practices can be identified. The implemen- outside groups -flight attendants, air traf- focus on technical rather than human fac-
tors elements, and failure to simulate the
air traffic environment realistically. The
flaws in LOFT execution do not negate its
current usefulness; they only indicate how
much more can be achieved. An important
research question, especially for airlines
that lack extensive simulator facilities, is
whether we can achieve the same impact
in a training device as in a high fidelity sim-
ulator. We need also to understand the util-
ity for human factors training of more
limited simulations that do not encompass
a full mission.
As demonstrated in analyses of commu-
nications during accidents and in LOET
research, crews must simultaneously cope
with multiple tasks at the group level and
differ greatly in the demonstrated ability to
do this. Additional research is needed to
understand the concept of multi-tasking and
FIGURE '1. Captain interfaces during normal flight operatioh
to determine if CRM training can help
ICAO JOURNAL
ors training programmes in aviation." In Cockpit Rex dern~cPress
crews become more effective in handling ing CRM effective. We need to assess crew examining the aviation system. As we
multiple tasks. Another area of importance performance in the ability to reinforce become a global village and begin to see
is operations involving ultra-long flights that effective behaviour and to determine what further integration of crews from differing
require augmented crews, especially in areas need particular attention in training. cultures, we need to be sensitive to these
advanced-technology transports. Such While encouraging progress has been issues and to develop training strategies
extended teams raise issues of leadership, made in developing evaluation methodolo- that are sensitive to cultural differences.
shift changes, and the utilization of extra gies, much remains to be done to improve Consider, lor example, the fact that cul-
crew members in emergency situations. instructor and evaluator skills. tures differ greatly in relationships
Research into these issues could result in It is also crucial that we understand how between subordinates and superiors and in
new guidelines for operations and special- human behaviour is manifested in abnor- their individualistic versus collectivist ori-
ized new CRM training. mal circumstances such as those sur- entation. It seems likely that a better
CRM training has developed outside the rounding accidents and incidents. S. Pred- understanding of such cultural issues will
domain of traditional technical training and more, a specialist in human factors, has allow us to make training in leadership and
checking. This strategy has been highly refined a methodology for classifying and communications more effective in cultures
effective, but it has hindered the integra- coding crew verbal behaviour from cockpit that differ in these dimensions. Our
tion of technical and human factors train- voice recorder tapes and transcripts. Simi- research group has made cross-cultural
ing. Many participants conceive of CRM as larly, digital flight data recorders provide investigations a central part of its research.
something outside of and in addition to an objective record of control inputs and CRM provides an excellent example of
their technical training and evaluation. aircraft performance. These analyses, par- the interplay between basic and applied
CRM concepts need to be fully integrated ticularly of crews showing very effective research. Many of the findings that came
with all aspects of training and checking. responses to catastrophic mechanical fail- from basic research into attitudes and
One strategy for integration that is proving ures, demonstrate that CRM concepts group dynamics have been translated into
successful is to provide instructors and apply generally to extreme situations. How- specific practices in the aviation communi-
check airmen with advanced CRM train- ever, we also need a better understanding ty. We have reached a turning point in the
ing that concentrates on evaluating perfor- of human factors issues in all incidents that development of CRM where we need both
mance and debriefing and reinforcing occur in the system. No approach current- to broaden our scope and to build on the
effective behaviour. ly in use seems to be able to capture all solid base that has been established. It is
We now realize that technical expertise, of the relevant human factors compo- unfortunate that the severe, international
aptitude and training are not sufficient to nents. This should be a primary goal for economic crisis, particularly in aviation,
make an optimally effective aviator. Pilots research. poses a threat to further work. It is our
need strong interpersonal skills as well as The model of crew performance shown responsibility to disseminate the message
technical competence. Indeed, personality in Figure 2 emphasizes the multiple factors that the pay-offs from investments in this
factors may limit the effectiveness of CRM that influence the way groups behave and area will be great in terms of the safety and
training. We must devote energy to res- ultimately the outcome of each flight. A effectiveness of the aviation system.
earch into improving selection strategies if particularly important and often ignored
we are to optimize the aviation system, and element of the model is the influence of The author IS Dlrector of the Aerospace Crew Research
this research must consider the impact of organizational and national cultures on Project sponsored by the National Aeronautics and
new technology such as the "glass cockpit" crew behaviour. Some human factors Space Admrn~strat~on, Unlversrty of Texas and Federal
Avrat~onAdmrnistration The research reported here
on crew motivation and satisfaction. specialists stress the importance of under- was supported partly by NASA Ames Research Center
Evaluation is a critical element in mak- standing cross-cultural issues when and by the FAA.
SEPTEMBER 1993
chinq of human factors
to the ab initio student pilot
The inexperienced pilot tends to accept readily the importance of human
factors as an integral aspect ofpilot training.
tions, there is high-face validity in the school and flight instruction, which tends
Ross A. TELFER equal esteem given each of the course to consolidate the perception that theory is
components. Just as engineering con- inferior to practice. This gap can become a
tributes to aircraft systems, ergonomics chasm when a few intractable personalities
and psychology contribute to their efficient are operating to protect and advance each
operation. Because of the acceptance of a territory, when theory is presented in a
body of knowledge to be transmitted, and dislocated and uninspired manner, or when
F
OR TWO reasons, the teaching of because of this receptivity by the students, instruction is perceived only as preparation
human factors to ab initio pilots is teaching methods can vary widely. for a test. For trainees, a major source of
probably greeted without joy by fly- As is the case with all subjects of study integration is in the planning and presenta-
ing schools: first, because of trainees' lack for pilots, the link between the human fac- tion of thoughtful, individualized briefings
of familiarity with aviation operations, and tor and actual operations is vital. Ground and debriefings. The ability to present the
second, because it further extends training instruction needs to refer to examples big picture in a meaningful way is entirely
time and cost. However, ab initio status can from and implications for flying; flight dependent upon the individual instructor.
be a considerable advantage. It provides no instruction demonstrates applications dis- There is no place for patter here, and there
opportunity for trainees to develop precon- cussed in ground school. Instructors on are major implications for the ways in
ceptions, and this neutral attitude pro- the ground and in the air have the vital which we choose to test ab initio pilots.
motes receptivity. Human factors can be responsibility of continually demonstrating The ideal is for integration of all theoret-
given parity of status with other studies the integration or synthesis of studies so ical preparation with the experience of
and requirements. For example, the BSc. that their interrelationships are apparent. flight, as the opportunity arises. Excellent
(Aviation) course at the University of New- The involvement of flight instructors in examples of this are available in the vari-
castle (Australia) consists of four sections: ground school, or having the one instruc- ous pilot judgement training manuals pro-
aeronautical engineering (including en- tor for both facets of teaching, would also duced in the 1980s by Transport Canada,
gines and systems, avionics, design, matc- accclerate integration of the total curricu- the U S . Federal Aviation Administration
rials); aviation science (including meteorol- lum. From the outset, it is vital that the and Australia's Civil Aviation Authority.
ogy, navigation); aviation management pilot does not classify learning into the These show how instructors can introduce
(including aviation law and computer appli- rigidity of the technical and human group- learning opportunities before, during and
cations); and human factors (including ings which haunt airlines' efforts to inte- after flying lessons. Another structured
medicine, psychology and ergonomics). grate line oriented flight training (LOFT) approach is through problem-based learn-
Each of the four subject areas is designed and licensing requirements. ing, in which groups are given the task of
to link theory and practice. Each is also To be coldly realistic, however, it has to using their interactive skills and collective
intended to be integrated with fight train- be conceded that in ab initio pilot training, knowledge to solve problems which relate
ing and, ultimately, with the task of safely the integration of any of the subjects of directly to the occupation for which they
controlling an aircraft. study with actual flying is not accom- are being prepared, instead of studying a
Human factors is no different. It is con- plished very well. For a start, there is the variety of subjects (such as meteorology,
ceded, of course, that the extent to which unfortunate distinction between ground navigation, law and medicine and princi-
such integration occurs in ab initio pilot
training is contentious. This is a major
issue outside the bounds of the present
I Table I.Proposed ICAO human factors training curriculum
focus, but worthy of industry attention. It

is, however, ironic that we should have an
expectation that human factors, the most
recent addition to pilot training, be inte-
grated, but accept the tradition of ground
school/flight training dichotomy in both
instruction and testing. For the inexpert
and inexperienced pilot, there tends to be a
ready acceptance of the importance of
human factors as an integral aspect of pilot
training. Provided the examples and appli-
cations are drawn from operational situa-
ICAO JOURNAL
ples of flight). The rationale for this
p m c m is that the learner is more motivat-
ed to learn if the subject matter is drawn
from an actual aviation cxme study. This is
by no means a short a t to h u m fact~w
instruction, but the additional effort is
worth it, The instructor's t a s k include
devidng a suitable casestudy; undertaking
a task analysis t;a reveal all the factors
involved: clearly stating objecthe%prepar-
ing work sheets and sufficient resources to
provide the information needed; and pro-
visling appropriate assessment methods.
The Human Factors Training Curricu-
lum provided by ICAO [Cir~ular2271
includes details af the eight modules
shorn in T@Me1. The time allomtion ref-
lects the number of sub-topics (which
TWe I does not present) within each mod-
ule. To integrate such aprograame means
tW the topice and sub-topics need to be
linked with the appropriate stage of fligtot In-ffight sssasment of human factors, coupled with feedback to student and
training. Relevaace and planned progres- ground instructor, is one means of integrtrting humen fadom instrucfion with ab ini-
sion are essential?so that there k a natural tin ffight tminifig.
rather than an, artificial inclusion of the
human factors in airborne ixlshction. the entire instructionalprocess. differsGrun the ICAO programme in b t it
A simple approach m d d be to teach Such irmporknt topie$ can be suitably is intended fur a universi@ underg;r&duate
each module in turn. This uninspired emphasized by an instruct30nd deslgn be& programme over a total of three academic
instruction leaves tbe weighty responsilsil- \visu&ed as a spiml. This ia baaed m the years (six semesters). Many ab initio pro-
Ety for integration squarely on the indlvid- principle that topics are treated briefly @ grammes will not have the 1uxuy of such
ual leastequipped to handle i t the trainee. the broad sweep at the bottom of the spi- an ertended period in which to m l o r e
A preferable alkrn&ve is for the instruc- nJj with the most important ones revisikd concepb research hdings, a d a wide
tnr to extract from each lCA0 module the as the e r a 1 narrows but extends in depth. range lrrf applications. It would be adtcranta-
subtopics which relate to thevarious p h - Key pibt activities such a judgement and geous tca extend the tabk to include finer
es of ab inftio pilot training, preseating dabion-making would be introduced ini- details on the phaws of pilottraining (such
them fn a developments4 and interlinked tially Catthe baaze of the spiral) as a simple as the eight stages identified in T~bde,I ) ,
waF A suggested design k chronoJogicaId go/no-go decidon because ryf weather or 'PIowever, the principIes of ifiteggation and
mirrorbg the .widening experiencesof the recurrence of themes are the same.
trainee progresrsing through first solo,
mow-country, igstrufnent fight and, possi- If human factors Tachin$ sequence
bly* aerobatics or multi-crew operations.
1
Into this structum it is possible to envbage teaching is to 1 Optima& fafar human factors eduation
tobeintegfedwithflighttrai~ng,itfol-
where topics such as visud ilhsion, Hti- I
gue, workload, stress mancgement, deci-
tion making and judgment could appear
instructorsmust 1 lows that the choice of content must be
,idedbythenabureoftheflyinga~tivity.
Thus initially the focus of the study of
when a flight instructor and ground school
hstructor collaborate on the teasing
1 becommittedtotheir human&~rsistheIn~dud~~in~lo
purpose opemgons. This includes the p$y&ologiqtJ
programme They could sequence topics aspect@of safe solo operations and the reli-
for ground school tre~tmentand alw Ifey able processing of data for decision making
them to flight mercises occurring at that airwaft unwrviceability, then would return and psycho-motoractMties. Impo~tantly,it
time. in farmore complex ~ i r c u m s h n m(involv- dso includes the ability to evaluate and
At a h m a n factors workshop in Aus- ing several factom s t a h as pew pressure, monitor student perfarmde so as to re-
tralia in November 1@2, a group of experi- time, fuel, wgulatitm and perwnal compe- ognbe poor judgement, hulty dedsion
enced instructors and pilots dsfded on tencej later in trainin&. Then the approprh chains and attitudhd ha~ards.Whether
eight trdmmg phase9 into w h i d human ate sjolutian is by no means as apparent, one is motivated from within or aternally
factam topics could be integrated. The Such key activities may be present in virtu- (termed "locus of control? and the extent
eight phases were pre-first flighk pre-cir- ally aU training sorties. Aspiral curriculum to which one accepts command respond-
cuig circuits; advanced upper air w r k ; enablss mgar to&s to bs treated initially, bility comes in here; social psychd~gyand
emergencks; basic instrument flighe nav- then revisited for cmsolidation and elabe- group dynamics are not intraduced until
igation; and pawenger carrying, It is not ration. multi-crew operations cammence. Similar-
only a mawr of aDoe&.ng topics to phms, An example ofthe linking of human he- ly, bmic in&uctimal skills md howledge
however. Some topics (a&asjudgement) tars topics to devel~pingpilot expertise are dististinmished from more advanced
are more importat and tend to pervade and experience is shown in Tile 2. This applications, following the expansian of the
limited role of a student pilot to the respon-
sibilities of the commercial pilot. The rele-
vance of human factors is determined by
the training and development of the stu-
dent. If a human factors topic (e.g. the
glass cockpit or circadian disrhythmia)
does not have application at a particuIar
stage of training, it can be included in a
i Initial Assessment FLIGHT INSTRUCTOR
final stage of human factors instruction. l-Feedback to Ground School
This final treatment can anticipate other
FIGURE 1. Human factors assessment as a means of integration
possible circumstances and will provide a
link with in-service human factors training
courses provided by employers, whether it tive, however. In some instances, a com- ity of inducing change in participants.
be in general aviation or the airlines. bination of approaches will be appropriate.
It's a matter of professional judgement. Assessment as integration
Instructional method This judgement by instructors is An interesting possibility would be the
Essentially, instructors have two choices: derived from their personal ability, person- use of assessment as a means of integra-
economy class, traditional ground school ality and experience (the art of instruction) tion. If the ultimate test of human factors
lecturing with a set of student notes and a combined with a knowledge of the underly- teaching is the performance of the pilot,
multiple-choice test; or, ing theory. It follows that employers have then the logical assessor is the flight
doing the job properly. . . which means to allow a degree of autonomy for instruc- instructor rather than (or, as well as) the
integration of human factors topics. tors to use their personal background and ground school instructor. The flight
For students who have come directly style in presenting and structuring human instructor could follow a structured test
from a high school education, pedagogical factors experiences. In designing human sheet, with specific behaviours listed, and
methods are certainly possible. This is not factors instruction, it is efficient to utilize could grade the trainee.
only a matter of trainee age, but also one of the before (presage), during (process) and Feedback on performance would be
familiarity and expectation. For older, expe- after (product) phases of instruction for given by the flight instructor to the ground
rienced trainees, such methods are ineffec- maximum effect and the greatest probabil- instructor, who could then modify content
or method appropriately and await further
feedback. This model is shown in Figwe 1.
Table 2. Integrating human factors and pilot training
Learning human factors
The individualizationof learning is, per-
haps, an idealist's goal for human factors. If
it can be achieved, however, what more
appropriate subject can there be?
Students will find human factors differ-
ent in terms of the nature of the material to
be learned, the nature of the examinations
(typically multiple choice questions), and
attention; workload, stress, personality.
I the application of the knowledge, skills and
values to operations. In human factors
teaching, this may be no more than a ques-
tion of education (knowing why) or training
(knowing how). The latter is by far the eas-
ier to attain, and companies may settle for it
Human Performance on the grounds of economy. They are not
mutually exclusive, of course, and human
factors instruction can encompass both.
Aviation instruction Human factors is as much education
rn (knowledge and values or attitude) as train-

ing (skill), so it follows that the process of
instruction is as important as content. It is a
more effective instructional technique to
structure the situation so that the trainee
Instruction experiences disorientation rather than
rn merely reading about it. Suitable instruc-
tional design would start with an explana-
tion of disorientation in the classroom, a
planned sequence leading to disorientation
in actual flight, and a discussion of the expe-
rience and its causes in a debrieiing session.
That is integration of human factors.
ICAO JOURNAL
Andher teasion is between theory and
pmdce. While the asriation induntry
should be able to expect the same mid-
ance from research in instructional meth-
ods as it obtains from ergonomics or
engineering in aircraft deign, the teaching
of h u m n factors appears to suffer in corn
parison. Part of this is due to ignorance of
the application of human factors to aviation
practice, the vital links between h u m fac-
tors and flight safety. Instructor commit-
ment can help make this link. & key
mdable in trtaehing success is the eathusi-
wn of the instructor, If human factors
teaching is to s w e e d , instructom must be
committed to their purpose, Training the
trainer may need to precede human factors
training.
Rnally, it is relevant to discuss the imp&
cations of trdnee piIotd approachea to
human factors learning. There are three
predominant approaches to learning deep,
~iur%ce,and achieving,The deap approach
to learn,ing is intrindcaIly motivated, with a
dedre tu be competent in fie area of study.
To achieve deep understanding, learners
read widely and integrate their new knowl- negative effect of a surface approach to ing, especially if they are keen to achieve.
edge with their exi~tingknowledge base. learning, and a tendency for positive For em, procesa-focused methods such
The surface-orientedlearner, however, is effects to emergefrom the deep appraad. as p u p dimtsion, role plays and simula-
motivated by anxiety and the dedre to do The implications of t h e ~ results
e for the tion are more appropi4ate. Vicarious learn-
the minimal amount of work required to integration of human factors into ab hitio ing thraugh approximations of reality
p a l the subject. 9urface-oriented strate- pilot training for airlines apply to toe (such as scenarios, accident investigation
gies include rots learning and reproduc- method of imtrzldon and the mailability reports, case studie~,"war stories" or crit-
tion af material prwided in courae notes or of resources. It is apparent that natural ical inddentsj are other means which get
manuds. The third approach, achieving, is ariosity and intrinsic motiva~onneed to quite claw to actual experience. In brief$
concerned with ego enhancement and be ut3zed as much powible. Reference theq the choice of method depend8 upon
organsng the time, source and place of to actual examples from aviation practice is the extent to which btructors and stu-
learning. a valuable teaching technique vehich dents wan1 surface or deep learning.
The relationships betveen approaches dependsupon thorough preparation where Intermtingl~~ deep approaches are those
to learning and performance in aviation the content (such as hypogar for example] which will promote greatest integration of
have now been examined frr several differ- extends beyond thc? immediate experience human factors. The deep learner requires
entpopulatioas as part of an ongoing proof the i n s ~ t o rR. videotape of actual pilot a wide range of learning resources: ex-
ject at the University of Mewcastle. An performance in a hypobarii: chamber is the per% an staff willing to have both formal
early study e m b e d approachea to learn- next best thing t~the experience itself, and and informaJ. dfseussion~books; videos;
ing (md their relationships with learning infinitely preferable to s h p l y hearing or charts: models; components; a facilitative
outcomesj in a wrnple of 62 commercial reading about the elfects of hypda. mvironment; appropriate Budget from the
pilot trainees, Data were gathered on Casestudies of incidents and accidents company; encouragement from company
approaches to learning and on individual relating to human bctors topics are valu- executivmii,and so en. The deep learner
performance in each of the nine ground able resources, as me co~kpitrecordings also requires the time and enc~uragernent
school topics (such as aerodynamics, nav- or their transcri@ions, Lectures, assigx- to use the&elearning reaurces to advan-
igation or flight plannind and the time it m a t s , reading*or group projects such as we. Instmetore who seek the minimum
took trainees to fly solo. accident md Incident arydp5iis and discus- performance nmessary to me& lkensing
The most pronninent finding was the ion, have appkation. The purpose and requirements will be rate.ariented, pre-
conai&ently significant negative relailon- result of each differs, howwar. senting notes to be learned and trial ques-
ship bemeen ground school scar& and The extent to which the hasas of schonl tions to be prac;tised"The aim urrill be ta
the surface approah to learning. Ah i m o teaching (pedwogy) or those of teaching pass a test, not to understand and apply
pilots wbo reparted adoptbg a surface adults Candragogy] provide theunderlying human factors knowledge, Introduciag
approa~hto learning wored lower on all principles of instruetion is determined by human factorsWpim needs pkming-, inte-
measures of ground school learning than the pmportioa of intrinsically-motivated, grating it into E i h t training requirm care-
those who adopted a lms surEace-oriented mature individuals who are tjelf-managed ful preparation and expert exeation. 17
approach. Trainees adopting a deep and have acquired a rwervoir of experi-
approach to learning aksD went solo earlier. ence upon which to draw, lkese are the Dr. Telfer is Head of the Department of Aviation a t the
This study demonsfmted the generally students who will awomplish deep learn- Univeoity af Newcastle.
Aer Lingus' innovative multi-crew training course integrates human factors and
technical training to prepare ab initio airline pilots for aircraft type transition training.
costs. These alone justify the Aer Lingus Classroom training

multi-crew training course, though the Basic technical and operational jnforma-
qualitative and operational benefits are also tion are first introduced in classroom lec-
substantial. tures and discussions conducted by an
There is an emphasis on group discus- experienced pilot. This training takes into
sions and experiential learning techniques, consideration the pilot's perspective, and
T
HE BASIC objective of the multi- including various didferent types of simula- anecdotes are used to emphasize practical
crew training course at Aer Lingus is tion. Trainees are allocated to two-member issues. There is an applied focus to this
to establish from the very beginning "crews" throughout their training (includ- training and it is specifically aimed at the
certain fundamental aspects of airline oper- ing ground school) and cooperative learn- development of effective operational under-
ations and modern cockpit management. By ing is encouraged. Crew pairings are standings and competencies, rather than
this means, Aer Lingus seeks to make a changed periodically. The basic objective an "end of course" examination.
clean break with the individualistic ethos of all generic training exercises is to pro- During the classroom phase, trainees
which characterizes most ab initio training. vide basic human factors skills, based upon are exposed to training in the use of air-
Practical aspects of crew coordination, com- the full operational integration of relevant craft public address systems, as well as
munications, and cooperative cockpit man- technical and human factors components. integrated emergency procedures training
agement are emphasized throughout. Simulation methods include written simu- in the company of experienced cabin crew.
Various methods are used to achieve this, lations, low fidelity computer-generated Practical training here includes actual fire-
some of which are reviewed here. simulations, generic flight management fighting in teams of two in a simulated
The course has a number of key training system (FMS) simulations and convention- aircraft cabin. There is also a general intro-
objectives, including the early integration al "full mission" simulator training. duction to the human factors consid-
of technical and human factors fundamen- In accordance with ICAO Annex I erations underlying standard operating
tals. The training is not aircraft type-specif- (Personnel Licensing) stipulations, some procedures (SOPs). This is followed by a
ic. Instead, "generic" or generalized, 25 hours of training in basic human factors detailed introduction to the operational
training has been designed specially for knowledge is provided during initial ab ini- application of specific Aer Lingus SOPs
the course, which comprises approximate- tio training. This is completed prior to and standard call-outs. In association with
ly three weeks of ground school and 12 starting the multi-crew training course and these discussions there is a review of key
four-hour simulator sessions. While this is provides a human factors knowledge base- aspects of crew resource management
a considerable training investment, there line. The multi-crew training course itself (CRM). Where possible, practical exercis-
are very considerable savings in later (air- emphasizes practical and applied human es involving crew management and com-
craft type-specific) training activities and factors skills. munications skills are embedded within
each training activity.
Low-fidelity simulations
The first integrated introduction to tech-
nical, operational and human factors issues
in airline operations involves the use of
"written"simu1ations. These are used to
generate questions and classroom discus-
sion. For example, one written simulation
integrates decision making in the light of
Aer Lingus operational policy with respect
to weather, along with the interpretation of
actual meteorological information and the
relevant navigation and approach charts.
During this simulation the relevant crew
briefings for cockpit and cabin personnel
are practised in a classroom setting.
A series of short low fidelity computer-
generated simulations (using Microsoft
Flight Simulator 4) are used to introduce
IGAO JOURNAL
ilppli~ratimof SOPS
&g the taMq
noise abatement, appma~hand lading
phams of QihL The last of them simula-
practical wpe& af CRll/r md cxrckpit m-

agemmt tmhnique%are explicitly ink*
d u & d There w-e otkw applied training
a&vitiee, hduding a wries of gene&
FMS laifo if om^ here again the emphasis
is on the Megrated applimtian of SOPS
atld stmdaad Gd~-0utS5 mmbined with the
coo use ofthe EMS aad d relevant
dam*The trahdnginvolves hquenk t%qm

sare ta short training scenarios (15 to 25
minuteis] ~ & C F than l o w Tndep&*t&n-
=a, rather tbm the place -Pv6k:reit is fXmt

acquired. &ly &nul&ar traiaing is dwat-
ed ta Bade aircraft handling and nmm-
flee, inducting fill apprimdaa of alI SCPs
retard b &omd .on a mrnmrkr to tot%&&

imtant aco~t$s
to the devant video &a
d m debrief@, By u&ng aspmid hand-
held d&m, the video record cm be
ment is that if we want pilots to perform as Training and checklist including real time liaison with cabin crew
a crew - as team members - we should design philosophy and passengers;
train them as a crew throughout. To fully Following is a description of the ratio- there should be positive reinforcement
optimize such training, all operational activ- nale underlying one specific multi-crew of good operating prastices throughout:
ities and crew decision-making behaviour training course simulator training tech- and,
must be grounded in operationallyrealistic nique: the use of generic checklists. training requirements should not cause
trainiig exercises. Technical drills and air- It was necessary to develop a generic any temporary suspension of operational
craft handling are thus viewed as being a airline "normal checklist" when designing "reality."
close second to CRM and multi-crew cock- the multi-crew course simulator training These basic design principles were
pit management skills, since the former syllabus. S i c e half of the simulator train- applied when developing the checklist. In
vary from aircraft type to aircraft type. In ing sessions are devoted to the operational addition, the accident record suggests that
this approach, aircraft handling and the management of non-normal events, a suit- certain important CRM principles are not
application of technical drills are treated as able '"on-normal checklist" was also always understood or actioned effectively
activities which are embedded within the required. It was felt that a generic non-nor- by pilots. It is, for instance, assumed that:
pilot's broader operationalunderstandings ma1 checklist which emphasized human aircraft control is formally allocated to
and competencies. factors considerations would offer unique one of the pilots;
During the early training design phase of training opportnnities. The following * situational awareness and fault diagnosis
the Aer Lingus mult.i-crew training course, points were considered when designing are jointly evaluated;
it was therefore decided that the simulator this checklist: drills are executed on the correct system
training of preference would expose trai- it should facilitate successful operational or engine;
nees to a representative series of opera- management of non-normal events by the there is an appropriate division of task3
tional events. By combining the common crew; between crew-members;
operational aspects and characteristics of human factors principles and crew co- the crew will promptly and effectively
normal and non-normal events, as consid- ordination issues should explicitly figure in address all operational consequences of
ered across various aircraft types, a eom- the checklist design and content; the failure;
prehensive and representative series of all communications generated by non- appropriate communications and coordi-
generic training scenarios was designed. normal events should be addressed, nation with the passenger cabin will occur.
Dublin-based Aer Lingus has introduced a multi-crew training course to emphasize practical aspects of crew coordination, com-
munications and cooperative cockpit management. (Photo: Boeing Commercial Airplane Group)
16 ICAO JOURNAL
The training intention behind the Aer
Gngw non-normal operations chedclist is
to structure, m d guide, trainee responges
and learnfng activitie~in light of thew
human perfomwce consideratima Rather
than t d trainees about the hnportaace of
a r t a h CBM behavioum, such as commu-
nications, the checklist is used to form*
and drive the desired codgit management
behavioum. Video debriefing is sub&-
qumtly used to provide feedback on a&d
cockpit perfmmance,
One selected exampb from the Aer
Lingus generic non-normal operatiom
checMi&is provided in Fig@%1.This Uut+
trates the generic "Sptems Mallunction"
&ecklist which is used in the event of a
failure or ma~hnctionof any aircraft @s-
tem, &light instrumnt. Proper see of the
ehecldist alao ensures that the relevant
cockpit acthities are mrbaliiaed and active-
ly coordinated between crew manbeus.
Conwquently it also serves as an aid to Aer Lingus uses low-fide!w smuwrron ro nerp rramecps ream now ro apply sranaard
effective crew dedsian making.
~pemfjngp~oredutes,checkIEsts. basic CRM skills and airline operating procedures.
The boxed items are "memory items*
to be called out md directed by the flying hwe an operatitad impact ti~n3.This meam that normal flight plan-
pilot. Su%sequently the non-flying pilot R ~ / ~ P E R A TC~bONNg A m L? ning, and all operatima1 documentatim,
reads the checklist Item aloud to confum Fuel Stab$: OptionsAvailable? are consofidakd into the actual simulator
that the check has been satkfa~torily PJon-standard Airc& Configuration mining enviroment h further mems that
complebd. There is only one technical Required? many of the individualtraining topies -for
"dean item on this checHtt -name- * P E H Q W C E DEGIEAD~T~o~? mample:,ele&cs and hydraulic failures -
ly to switch off any relevant system Aicrafk? arefimt encollfltered in a realigtic Bne can-
switches. Memwy items on the cheddht Rmmy: Lplngthand W&a Canditiom? text, where bath the techrricai problem1
sexk to ensure qproprhte c m coordina- C~MMUNICA~OMS REQUIRED: and the red tlme opemtionalmanagemmt
tion, communimtions and decision mak- Cabin Crw, of that problem, are matedas an integrat-
ing.It will be noted that the items on this Air Tr&k bntrok ed whole for trajaiag purp~-ses.
generic check1"it will have continuing Pawngem;
relevance after multi-crew training has Colrrgm, Concludon
been completed. The multi-crew course son-normal Aer LingusbuIti-crew murse training
Under tEre "Operational Conse~uence$ checklist provides trdnees with a concise has been sufficientlys u c c e far ~ the~ dr-
. . .Edua-te" heading, a nutinbe of supple- inb-oduction to the structured use of non- line to d e s i g ~a d m i h generic trainiug
mentary headings were recently added to normal &ems M m a mnlti-crew opw- course for command upgrade candidates.
p r m p t md assist the crew as they asswa ational enviromenL The checklist &elf TZle decision to conduct pre-command sim-
the operational impact of the relevant pmb- facilites the management of a compe+ ulator training d g generics &mulationis
lem, These rehrience the main areas in hensive q e c t r m of operationalproblem clealy a atrow endorsement of .the per-
wuch tlon-normal events most frequently d& simulator trtxhhg. These reprwen- ceived success and d u e of the entire
tative cockpit rnawgement issues all arise multi-crew training initiative. Trairxing
in a full mi&onn,real We, mulkrew oper- which empharii~eeoperationally realistic
I SYSTEMS MALFUNCTION 11 ational environment, with 'rpacks;en@mp
and "cabin crew" fully integrated in@ the
tasks - embeddecf within a fuRnfl-misf&tn
en\Pironment -offem a very effective
training situation. T h i ~
repesents a rich method of e f f e d ~ e l pintegrating h m a n
human factam anand CRM learningenviroa- factors.and*t skills,
ment for neu. pilob, The le~wazlearnedby Am Lingus is that
the development of multi-crew cockpit
T&mespt;cificsirnulatar training management skills through geaedc simu-
Given the gxepmtion pravided by the lati011 sessian~has a much wider a p p b -
multi-crevkaining c o m a it has been pou tion than the er;dini-ng of ab initio pilats.
sible to infroduce highly innovative type- Geneh simulation aho has the ~ignikant,
specMc @asitionsyllabi for gradtlathg the added benefit that it is mtremely coat-
course's ab iinitio trainee&. Here traditional effeclive,
sthtirat61- training p r d w IzaS been large-
$ ignored* with most training sessions Capt. J o h m t a n h head o f Training D e v e l o p m ~ n at
t
designed aound line operational sirnula- &r Lmgus
Cockpit resource management and flight
training for the advanced-technology cockpit
In time it may be possible to optimize training by tailoring the instruction and scenarios to
recognize the great differences that exist between modern and traditional cockpits.
three-pilot flight deck. nology cockpit. These lessons emphasize

To the credit of the industry, by the mid- not hardware and systems, but crew coor-
dle of the decade the problems were under dination demanded by automation. For the
control, and failure rates of crews transi- sake of economy, airlines wish to keep
tioning into aircraft equipped with the recurrent LOFT scenarios of a generic
glass cockpit were no greater than those (model-independent) nature. Transition
I
N JANUARY 1981 a group of leading usually found in conventional aircraft. The training creates the opportunity to tailor a
authorities in the field of pilot training industry could begin to prepare for the LOFT programme to the technology of the
met at the National Aeronautics and rapid acquisition of new aircraft, route cockpit.
Space Administration (NASA) Arnes expansion, mixed fleets of derivative
Research Center to discuss the fairly models and equipment differences within Philosophical questions
recent concept of line oriented flight train- the same models, extended range twin- The rapid development of cockpit
ing &OFT). This meeting, which would engine operations (ETOPS) of two-pilot air- automation, and the lack of operational
later be regarded as a milestone in pilot craft over the ocean, and a tidal wave of doctrine by which it could be governed, led
training, came coincidentally at a turning newly hired pilots, many with less flying some airlines to formulate a "philosophy of
point in the history of cockpit automation. experience than those previously obtained. automation." In 1990 Delta Air Lines adopt-
Just a few months earlier the first McDon- By the end of this century the glasscockpit ed a one-page automation philosophy that
nell Douglas MD-80 (originally DC-9-80) aircraft will be not the oddballs of the would guide its approach to equipment
aircraft had left Long Beach to join the worldwide fleet, but the mainstay. R.L. acquisition, training, and operational doc-
fleets of regional carriers. The MD-80 Helmreich summarized the situation in a trine for the carrier's rapidly growing fleet
brought to the short- and medium-haul car- paper presented to a 1991 symposium on of modern aircraft. Delta also implemented
rier a level of cockpit sophistication previ- aviation pyschology: a new training course which is required of
ously found only in wide-body aircraft. Boe- all pilots transitioning to glass-cockpit air-
The impact of cockpit automation presents a
ing's 767s were only months behind, craft for the first time. It precedes ground
number of challenges for future research.
propelling the aviation industry into an school and is model-independent. The pri-
While we know that crews are behaving dif-
even higher level of flight deck technology, mary goals of the new course are to
ferently in advanced technology and stan-
the era of the "glass cockpit." explain Delta's philosophy of automation
dard aircraft, we do not yet know whether
It is significant to note that at that his- and to relieve some of the anxieties and
these differences are reflective of training or
toric meeting there was no discussion of misconceptions that pilots often bring to
of characteristics of particular automation
the implications of the rapidly escalating ground schools for advanced technology
philosophies and aircraft designs. Most
sophistication of cockpit automation for aircraft.
LOFT scenarios being used are "generic"in
pilot training or for LOFT. Even such eso-
the sense of not being based on characteris-
teric topics as training for engine-out ferry Automation and crew coordination
tics and capabilities of advanced technology
flights were covered at this meeting, but Training experts in the airline industry
aircraft.
the impact that automation would have on and in government have generally assum-
pilot training was yet to be recognized. LOFT scenarios can be designed that ed that CRM training programmes are
The participants at that meeting proba- are ATC-intensive. For example, one air- essentially model-independent: identical
bly could not have imagined what lay line, America West, has produced an inter- training was delivered to all pilots at a
ahead in the next decade of pilot training. esting scenario in which a line of given airline, regardless of the type of air-
In the years to follow, airline training thunderstorms tempts the crew to deviate craft they were or would be flying. The effi-
departments throughout the world were to from course toward a military operations cacy of this can no longer be accepted as
experience the birth pangs of the glass area (MOA). In order to avoid penetrating true without proof, as there is mounting
cockpit. They would learn that a Boeing the MOA, assistance from ATC is required. evidence that crew coordination and com-
767 is not simply a 727 with some extra Realism is taken to the ultimate when a munication in the glass-cockpit aircraft is
boxes, and they would struggle with high- flight of fighters appears in the pilots' view. qualitatively different than in the tradition-
er failure rates in transition training than Several carriers have begun to insert al cockpit. If this proves to be the case,
they had never seen before. Not only were into their cockpit resource management then, at the least, modules of CRM pro-
the world's airlines facing an industrial rev- (CRM) and recurrent training programmes should be devoted to crew
olution in the cockpit, but they were simul- grammes modules that demand the crew resource management in the advanced-
taneously witnessing the beginning of the coordination and workload management technology aircraft. Such a move would be
end of the era of the flight engineer and the required to successfully fly the high-tech- logical, and consistent with one of the
ICAO JOURNAL
guidelines pmpoml for GM training by
human factors e x p e r k "Customize the
training to reflect the nature and needs of
the orgznization." To these words could be
added, "and the generic type of cockpit
tmhnology as well."
At this h e there b no solid cxperhm-
tal evidence that these presumed differ-
ences actually exist, let alone that they are
worthy of, or addressable by, a tailored
CRM p r a g ~ m m for~ advanced-k&nolo.gy
aircmft. The q&o& study of B ~ e i n g757
crews points in this direction, but couldnot
by itself be taken as sufficient reason to
launch a CRM programme in mtomatioa
Implications for IhFT

Little has been written about the impli-
cations of the high-technology cockpit for
the design and conduct of LOFT, although
it could be argued that the modern cockpit
may require tailored CRM programmes or
modules.
Utilization of LOFT techniques may
appear in transition training, where the
issue may be forced by the needs to adapt
to a new cockpit. Automation-specific sce-
narios will be adopted more slowly in
recurrent training, where economic con-
siderations continue to dictate model-inde-
pendent scenarios.
The high-technology cockpit, with its Evidence is mounting thar crew coordination and communications in the glass cock-
strengths and weaknesses, offers novel pit may be qualitatively different than in the traditional cockpit. (Photo:Airbus Industrie)
opportunities for scenario design. For
example, in traditional aircraft it has been remainder of the flight. of augmented crews (one or two additional
necessary to induce abnormal conditions Thus automation provides the LOFT pilots) into a two-pilot cockpit provides new
(e.g. system failures) in order to elevate the designer with the opportunity to build sce- challenges for crew coordiiation.
workload and stress on the crew in a realis- narios that will directly address the prob-
tic manner, in order to create a situation lems and opportunities of two-pilot crews Summary
which demands crew coordination. Thus working in the advanced-technologycock- As we learn more about the crew coordi-
their proficiency at coping with these condi- pit, particularly those involving the aircraft nation and communication requirements of
tions can be evaluated, as well as preserved automation/ATC interface as it exists the glass cockpits, and sharpen our ability
on video tape for selfevaluation. The mod- today. The LOFT constructed for these air- to provide effective CRM and line oriented
ern cockpit has enough normal, built-in craft can and should exercise those pecu- flight training, we may be able to optimize
stressors to do the job, particularly in the liar characteristics of the modern cockpits, training by tailoring the instruction and
area of air traffic control instructions. and can easily create situations that scenarios to recognize the great differ-
The glass cockpit presents new opportu- demand that one practise CRM principles. ences that exist between modern and tradi-
nities for scenario design that do not tional cockpits. According to some experts,
require emergencies per se, just difficult Implications for the future the use of LOFT-type training has changed
problems at the human-automation inter- In a few years the glass-cockpit aircraft the texture of airline training. The author
face. Any pilot of a glass-cockpit aircraft will dominate the world's fleets; more believes that with the development of new
will be happy to tell you what they are. advanced models with new automatic fea- cockpit technologies, that texture will con-
tures are on the way. Furthermore, for rea- tinue to change, and possibly in ways that
Automation-rich LQFT scenarios sons external to cockpit technology, these are not easily predicted at this time.
The resourceful LOFT designer has aircraft likely will be operated under more
new tools to work with, and these clearly stressful conditions than we have known in
meet the criteria of realism and line validi- the past. Airlines will face new demands to This article is a condensation of a chapter by the author
in Cockpit Resource Management, by E. L. W~ener,8. G.
ty. One highly experienced LOFT designer meet market conditions, adapt the modern Kanki, and R. L. Helmreich, published in 1993 by
remarked to the author that the best LOFT aircraft to the ATC system (instead of the Academic Press. The author's work has been supported
he had ever constructed contained a single other way around, as it should be), and In part by a research grant from NASA's Ames Research
Center t o the University of M~ami,jointly funded by
failure: the autopilot dropped off early in absorb new hires who may be quite inexpe- NASA and the Federal Aviation Admmistration. The
the flight and could not be restored. This rienced compared to those that the airlines opimons expressed here are those of the author, and
failure produced implications for the have previously attracted. The introduction not of any organization.
SEPTEMBER 1993
Lessons from research on expert
decision making on the flight deck
We are only now beginniag to understand what constitutes effectiue
decision making and what skills should be taught.
portation Safety Board (NTSB). There we offered, what dimensions are relevant for
JUDITH ORASANU find cases of crews that flew into thunder- comparing models, and what we want in a
NASA AMESRESEARCH CENTER storms and encountered wind shear; decid- car. The task is a matter of deciding which
ed to reject a take-off after they were off car will best satisfy our needs. In contrast,
(UNITEDSTATES)
the ground; decided to land after retracting decisions in the cockpit are not ends in
the landing gear in preparation for a go- themselves - they are the means by
around; took-off with snow and ice on the which crews achieve their larger goal,
D
ECISION MAKING is an essential wings; or decided to fly on to their destina- namely, to deliver passengers and aircraft
component of a captain's expertise. tion on battery power rather than return to safely to their destination. Several features
The captain is responsible for mak- the airport from which they had just depart- characterize decision tasks in the cockpit
ing the hard decisions: choosing where to ed. NTSB reports from 1983 to 1987 impli- (as well as in other complex domains such
divert after a system malfunctions, when cate crew judgement and decision making as nuclear power, military command and
fuel is short and weather is deteriorating; in 47 per cent of the fatal accidents. control, medicine, and fire-fighting) and
determining how to cope with a passenger's Decision making is a component of i d u e n c e the nature of the decision
medical emergency; evaluating whether to most crew resource management (CRM) process:
take-off with a placarded system given past training courses given by major airlines Dynamic conditions. Conditions change
experience with the projected weather and (see US. Federal Aviation Administration over time, making the situation unstable
traffic at the destination. While the captain Advisory Circular No. 120-51, 1993). How- and unpredictable (e.g. weather or some
has ultimate responsibility for the decision, ever, little scientific research has been avail- system malfunctions).
the entire crew in the cockpit, in the cabin, able until recently to support that training. Ill-structured problems. It may not be
and on the ground can provide information We are only now beginning to understand clear to the crew what the problem is, what
and suggestions that contribute to a good what constitutes effective decision making options are available, or what would consti-
and safe decision. The captain's judgement and what skills should be taught. tute the best solution.
is most critical when conditions are Decision making in the cockpit is unlike High-risk. Decision errors, especially
ambiguous and no clear guidance is pro- decision making in several other situations. during abnormal and emergency condi-
vided in manuals, checklists, or company Most often we thiik of decision making as tions, can have severe consequences.
policy. choosing between options A, B, and C, Time pressure. Certain decisions must be
The significance of poor decisions is evi- such as when we buy a new car. In the case made very quickly. The aviation environ-
dent in reports from the US. National Trans- of car-buying we know what models are ment is unforgiving: correct decisions
made too late can be fatal.
Competinggoals. While safety is usually
paramount, economic considerations put
pressure on crews to save fuel and to be on
time. In addition, crews must conform to
government regulations and company
rules. Sometimes safety is pitted against
these other very real goals.
Multiple participants. Several partici-
pants (cockpit crew, cabin crew, company
dispatch, maintenance, ATC) may bring
different perspectives, knowledge, and
goals to a problem. Conversely, multiple
participants can monitor and evaluate each
other, reducing the likelihood that impor-
tant information is overlooked or that
faulty plans are adopted.
Expertise. While pilots may not be expert
decision makers, they bring vast knowl-
edge and experience to the decisions they
make.
Decision problems in the cockpit do not
ICAO JOURNAL
come neatly packaged, with Eh6 ~ptlans>
0 ~ 1 smd m & a I ~ &dmrly @&ci;Eied,
Before a deWm can be made, the crew
muat re~wbe that a &uakion edsts &at
fizxpiresattention, The nnzltlrre afthb @rob-
1m m ~ &be d&a&ned, ifs @ev&ly
a=~wd, md optkms wmkked. In Len-
WL de&ion making in dynamfoemiron-
men& &ni two a j a r compatmt.9:
itnation m e m e a t m d choice erf a
caw= of wtis1a.
Situatiosmmsment involve@intwpt-t-
ing the CUES that ~ignda problem and
jud&8 the levels ~f risk and .time prm- fxf~ msider relev& apfierm
ewe. $om$ d d & o n e re&v a &st rm- A mroIlary o'f good s b t i s n asestmmt
ponse, SIP& as r e j w t i ~a t h - o f f or L rmlbhg JNhgt Woxmticm is nmded in
dwidhg ts, go around, Others alllaw time! order to m&e a gaod deci&m ?%enseek-
mtkas and to gather ing it Emrf u&g it I-rr the w 12
4 r2fafl-rooJsiner
on. Some decidone r hydraulic fa8w in
maditi;onB thdtkiggef the crew must
a p r ~ d e rd ~ a &d&&%d n ~ &Xi- rewgnbxithe c o m q u w s of 1;Inehydrm-
some bvolue ckim from among lie fa3h-g: mmqal gwr and alterna&flap
~ptionz+~ mMpre4ulm prior- axteadon pr0r:ducs u-erequi i&
t4n
pro%lemsdvtngbewum ~ m the
x Irsnfinp
lutions ar options &&, be&a&e& SQa
Where do@ mgerthe mtw into this go-5trmlrdf~ mde&&le, rn&w=W a
picaawe? Studies af eq&-tsh domaim mn~lderatiofi.T h e mfisQ&ints &de
ranghi@&omp h y s b aad dhws tr)college f&&on 3 md d@w~ what
dm@dansmd me,di&ne show that mpm- taw a mod @olution,
60a Mutwm haw we p e k d v i ~ ahd B e @ d b of w h t h afwt
~ or a a ~ r ~
P-nd ta grablem*In g$m3-& ~ e . d 3 e : dehiexate tkWm mud be ma&, more
problem is that modern aircraft are so reli-
able that most pilots have little real experi-
ence coping with abnormal and emergency
1. Recognize and interpret cues I situations, and hence few learning opportu-
risk (present and future) nities. While this a situation for which we
time available are all thankful, it means that other oppor-
tunities must be found to allow crews to
practise the needed skills. Line oriented
flight training (LOFT) provides such an
opportunity, but these sessions offer limit-
ed time for handling diverse problems. Use
of low-fidelity simulators could be used to
exercise component skills. Classroom time
often is devoted to reviewing videotaped
reenactments of accidents and discussing
ideal solutions. However, discussion of
how you should respond is not quite the
same as doing it. Ideally, discussion of deci-
sion strategies would be complemented by
real-time practice.
The research community has not yet
discovered a "silver bullet" for training or
Gding that will guarantee that every deci-
sion is the best possible one. At this point we
effective captains manage the situation in Errors that result in poor decisions are can only recommend processes that assure
ways that allow them to make effective the converse of effective decision strate- that the crew understands the situation
decisions. A primary way to do this is to gies just described. NTSB accident reports and what would constitute an appropriate
plan for contingencies. They anticipate suggest that serious outcomes result from response. A few general recommendations
what might happen and set "triggers" for poor situation assessment, mainly when can be made:
themselves. Then, if and when conditions crews fail to appreciate the risk inherent in Understand what the problem is before
deteriorate, they are prepared rather than a situation. Crews that get into trouble uni- acting.
surprised. Effective crews shift their think- formly seem to suffer from severe cases of Assess the risk and time factors.
ing from a high workload, time-pressured "get-there-itis." Many accidents occur fol- Match your response strategy to the fea-
phase to a lower pressured one. lowing schedule delays or at the end of tures of the situation (e.g. fast vs. thought-
Similarly, when a complex problem re- long trips, when the crew is eager to get ful response).
quires evaluation of several options, more home. Crews exhibit a strong tendency to Set up contingency plans whenever pos-
effective captains arrange the situation so assume that ambiious cues fall within the sible.
they have thinking "space" to sort through range of normal, rather than being seen as Consider the implications- the nonob
the possibilities. They can do this by potentially dangerous, and carry on as if vious future consequences -before decid-
offloading tasks, including responsibility nothing is wrong until the consequences ing on a course of action.
for flying the plane. Or the captain may buy overtake them, Manage workload to allow time for deci-
time by requesting a holding pattern or Discrepant information often is exp- sion making when time and fuel permit.
vectors. This allows time to gather and lained (or wished) away, especially if it is Create a shared problem model by com-
evaluate information before committing to ambiguous. Consider the case of a Boeing municating with the crew (cockpit, cabin,
a course of action. Effective time and work- 737 on takeoff roll with snow and ice on its and ground). Be sure aIl understand what
load management strategies not only wings and engine sounds that suggested the problem is, what the plan is, and who is
reflect awareness of the demands of the less than full take-off power, despite the doing what.
problem; they also reflect sensitivity to engine pressure ratio (EPR) readings. Expertise comes with experience. By
one's own cognitive limitations, especially When the first officer brought the dis- practising the above in relatively benign
under high-stress conditions. crepant cues to the captain's attention, the situations, when workload is low, crews
An integral part of decision making in captain repeatedly affirmed that every- can get in the habit of thinking of these
general and task management in particular thing was OK. The captain was using only strategies. Experience may be a great
is communication. Effective captains do the pieces of information that fit his model teacher, but some lessons exact a high
not function as 'lone rangers." When faced of normalcy and ignored those cues that cost. The aviation industry is trying to pro-
with problem situations, they share their indicated a problem with engine power. vide opportunities to learn at low risk and
concerns, make clear what they want to Realistic assessment of the risk in a situa- low cost. How to do so is the challenge - to
accomplish, and invite participation by tion, facing the possibility of a worst-case our collective expertise.
other crew members. They also listen to scenario, and preparing for it before the sit-
the suggestions offered by other crew uation begins to unravel are probably the
members. This interaction allows the crew only strategies that assure safe decisions. Aeronautla and Space Admlnlstratlon'sAmes Research
to build a shared model that assures they Given the number of hours that flight Center. The opmions expressed In this article are the
are all working to solve the same problem crews accumulate, we might expect them author's and should not be construed as officlal p o k y
and facilitates contributions by all. all to be "expert" decision makers. The of any government agency.
ICAO JOURNAL
Conclusions from report on
CFIT accident near Kathmandu
The aircraft's flight data and cockpit voice recorders were instrumental
in providing information needed to determine the causes of the accident.
SYNOPSIS
The flight was conducting the Sierra
..."*,*.CI...l**1.*~~I.*.Ii...h.'.i*.*~*'i1.,**. .... .............
*...*a *.I..I I.,..,....,I '
I..,...,
VHF omnidirectional radio rangeldistance

DISSEMINATING INFORMATION ON CFIT OCCURRENCES
measuring equipment (VOFUDME) app- One type of accident that continues to take a heavy toll on international civil aviation is
roach t o runway 02 at Tribhuvan Interna- controlled flight into terrain (CFIT). In its 131st session (late 1992), the ICAO Air
tional Airport, Kathmandu, in instrument Navigation Commission agreed that CFIT was a critical flight safety problem and that
weather conditions. A flap fault occurred urgent action was required by ICAO. Accordingly, the Commission requested that the
while the flight was on the approach; this Secretariat develop a comprehensiveprogramme, one specifically aimed at taking preven-
caused the crew t o ask for a clearance tive actions on CFIT.
back t o Calcutta, a decision that was in
Even though there was initial success in curbing the number of CFIT oceurr :s fol
keeping with both Company and perfor-
mance requirements, which necessitate
lowing the development and implementation of ICAO provisions for the ground proximity
the use of full flaps for the steep final warning system (GPWS), these types of occurrences have increased in numbers over the
approach. Shortly (21 seconds) after mak- past severalyears. (See "ICAO initiates comprehensive programme to prevent occurrence
ing this request, at a distance of approxi- of CFIT accidents," ICAO Journal, January/February 1993.)
mately 12 nautical miles from the Kath- To provide timely dissemination of critical flight safety information, extracts of findings
mandu VOR, the flap fault was rectified from official investigations of significant CFIT occurrences will be published in the Jour-
by retracting and then reselecting the nal from time to time. This month the Journal presents the findings, causes and o
flaps. The crew determined that i t was
tional recommendations contained in the report of the Commission for the Accident
not possible t o continue the straight-in
tigation of T G 3 l l (Thai Airways International Airbus Industrie A31@304,HSTID, near
approach, due t o the steep descent angles
required and the position of the aircraft. Kathmandu, Nepal on 31July 1992).
The crew stated to the control tower that Readers should note that the Commission cident 1 tigation of TG3X1 was
they wished t o start their approach again established by the Government of Nepal for the purpose of advancmg aviation safety. It is
and requested a left turn back t o the not the function of the Commission to assign blame or determine civil or criminal liability
Romeo fix, which is 41 nautical miles Readers who would like to obtain the accident report should contact the Government
south south-west (202 radial) of the Kath- of Nepal. Write to: Ministry of Tourism & Civil Aviation, Department of Civil Aviation
mandu VOR. The controller, in the non-
Babar Mahal, Kathmandu, Nepal
radar environment, responded by clearing
the flight t o make the Sierra approach,
'................ 1..
I
which starts at the 202 radial and 16 nau-
tical miles from the VOR. The crew maintain an altitude of 11,500 feet and that the crew was in the process of
response t o the clearance was t o report was t o "proceed t o Romeo" and contact inserting "Romeo" and other related
that, at the moment, they could not land the area control centre (ACC) controller. navigational information into the flight
and t o ask again for a left turn back t o The flight, commencing a descent while management system (FMS), but were
Romeo t o start their approach again. in the turn, completed a 360-degree experiencing difficulties.
After further dialogue with the con- turn, momentarily rolling out on head- The flight continued towards the
troller, which included requests for a left ings of 045 and 340 degrees, and again north on a heading of 025 degrees and
turn, the crew unilaterally initiated a proceeded toward the north on a head- then, at about 16 nautical miles north,
right turn from the aircraft's 025 degree ing of 025 degrees magnetic. When the the heading was altered t o the left t o
heading and commenced a climb from flight was about five nautical miles 005 degrees. Slightly over one minute
an altitude of 10,500 feet t o flight level south-west of the Kathmandu VOR, the later, the ground proximity warning sys-
180, when the flight was about 7 nauti- crew contacted the area control centre tem (GPWS) sounded the warning "ter-
cal miles south of the Kathmandu VOR. and stated that the aircraft was "head- rain, terrain" followed by "whoop,
The crew reported t o the tower con- ing 025" and they wished t o proceed t o whoop, pull-up"; the aural warning con-
troller that the flight was climbing and Romeo t o start their approach again; tinued until impact approximately 16
the controller replied by instructing the adding they had "technical problems seconds later. Engine thrust was increas-
crew t o report at 16 nautical miles for concerned with the flight." It was again ing and "level change" had been
the Sierra approach. During the turn, established that the flight was t o pro- announced in the cockpit, just before the
there was more discussion between the ceed t o Romeo and the crew agreed t o impact occurred at the 11,500-foot level
tower controller and the flight, where it "report over Romeo." It was determined of a 16,000-foot peak; the accident site
was established that the aircraft was t o from the cockpit voice recorder (CVR) was located on the 015 radial (north-
SEPTEMBER 1993
north east) at 23.3 nautical miles from Sierra approach, however the clearance aircraft was five miles north of the
the Kathmandu VOR. All on board, 99 did not include the Romeo fix or a direc- Kathmandu VOR.
passengers and 14 crew members, lost tion of turn. 23. Because of mountainous terrain,
their lives, and the aircraft was 13. The flight crew continued t o ask the published safe altitude within 25
destroyed. for a clearance t o Romeo, specifying a nautical miles north of Kathmandu air-
left-turn direction, but they did not port is flight level 210, which was
CONCLUSIONS receive a clearance satisfactory t o them approximately 9,500 feet higher than the
Findings nor did they initially receive any other altitude of the aircraft.
1. The Sierra VOWDME approach t o further instructions. 24. The company en-route chart
Katmandu Airport requires steep aircraft 14. The crew of TG311 did not per- showed only a track of 021 degrees mag-
descent angles. ceive that the flight had a valid clearance netic for the airway joining Romeo and
2. Company procedures and aerody- for a new Sierra approach, but under- the Kathmandu VOR, whereas the track
namic performance considerations requ- stood that they were t o continue their t o Romeo is 202 degrees magnetic.
ire that full slats and flaps configuration present approach. 25. The very high frequencyldirection
be achieved by the 13 DME point for the 15. After four requests for a left turn finding (VHFIDF) equipment, which pro-
Sierra VOWDME approach t o Kathmandu and a review of the minimum obstruc- vides an indication of bearing of an air-
Airport. tion clearance altitude (MOCA), the crew craft from near the centre of the airport,
3. The visibility at the airport was of TG311 initiated a climbing right turn does not utilize the area control centre
below the operator's limit of 3,000 from an altitude of 10,500 feet, intend- radio frequencies and has only one indi-
metres as the approach was commenced; ing t o climb t o an altitude of FL180 cator, which is located in the control
continuing the approach with such visi- (above the minimum off-route altitude tower.
bility, until reaching the outer marker, (MORA) of 17,200 feet). 26. When the aircraft was north of
was permitted by the Company's Flight the airport, the flight was communicat-
Operations Manual (FOM). ing with the area control centre and thus
4. The flight profile was proceeding I The crew's response to its bearing from the airport was not indi-
normally until a slatiflap selection of 151 cated on the VHFIDF equipment.
15 was attempted, at which time a flap the GPWS warning was 27. When requesting the aircraft's
fault occurred. position, the area control centre and
5. The flap fault prevented extension
not in accordance tower controllers only asked for distance
of the flaps beyond 15 degrees because
of actuation of the screwjack torque lim-
with the manufacturer's I from the VOR, but not radial informa-
tion, thus the aircraft position was not
iter. procedures. determined.
6. The screwjack torque limiter actua- 28. When transmitting the aircraft's
tion was likely caused by increased position the crew of TG311 gave only dis-
screwjack system friction, aggravated by 16. The tower controller cleared the tance from the VOR (DME), not the radi-
the momentary extension of the right- flight t o descend back t o 11,500 feet, al, and thus the flight's geographical
wing spoilers. which caused the crew t o stop their location was never passed t o the ACC or
7. Despite the absence o f checklist ascent and commence a descent back t o tower controllers.
guidance, the crew was able t o recover 11,500 feet. 29. A heading report by TG311 o t
the normal operation of the flaps by 17. The tower controller did not ini- "025" was likely not heard by the ACC
retracting the slatslflaps t o 1510, in accor- tially indicate whether or not TG311 controller.
dance with the operator's Aircraft Oper- could proceed t o the Romeo fix. 30. It was not possible or appropriate
ating Manual (AOM) procedure. 18. While the aircraft was in the right for the controllers t o provide heading
8. Once the flap fault was rectified, 11 turn, while nearing the 202 radial of the vectors t o TG311 or any other aircraft.
minutes prior t o impact, the crew decided Kathmandu VOR, the tower controller 31. At least some of the crew's efforts
that the flight could continue t o cleared the flight t o the Romeo fix and t o input the Romeo fix and the Simara
Kathmandu, but the aircraft was too high instructed the flight t o contact the ACC non-directional beacon (NDB) on the
and too close t o the airport t o achieve the controller. FMS appeared t o be successful, but the
required approach profile and the straight- 19. The crew continued the right turn crew did not accept the information for
in approach could not be continued. and travelled toward the north-north unknown reasons; the aircraft was not
9. TG311 made an unusual request for east, which was opposite t o the direction turned t o the south-south west toward
a clearance t o the Romeo fix, specifying t o the Romeo fix. the Romeo fix.
a direction of turn, t o join the Sierra 20. At some point in the flight from 32. The captain likely assessed that
approach t o carry out another approach. the latter portions of the 360-degree turn the copilot was having difficulty
10. In the plan view of the operator's t o the right, the crew became unaware inputting data into the FMS.
approach chart, Romeo was misleadingly of where the flight was proceeding. 33. The crew's interpretation o f the
depicted as the start of the Sierra 21. The ACC controller issued another FMS navigation data appeared t o be a
approach. valid clearance t o Romeo; the clearance problem at certain times, commencing
11. The crew may have requested a was acknowledged by TG311. near the end of the 360-degree turn
clearance t o Romeo because of i t s depic- 22. Neither the ACC controller nor the until the impact.
tion on the operator's approach chart. TG311 captain succeeded in communicat- 34. The crew's use of the FMS for nav-
12. The tower controller issued a valid ing that the flight's progress was not in igation was uncoordinated and may have
clearance t o TG311 t o carry out the accordance with its clearance when the led t o confusing system outputs, thus
ICAO JOURNAL
reducing the crew's ability t o conduct 48. There is no certification require- crew when confronted with the discon-
effective navigation problem solving. ment t o provide compass cardinal head- tinued approach.
35. It is likely that the copilot realized ing letters. 50. The search, following the accident,
that the aircraft was in a potentially dan- 49. There was no indication that the was hampered by the expectation that
gerous flight situation approximately 30 crew had received simulator training for the aircraft was operating south of the
seconds before the terrain impact. Kathmandu, even though Kathmandu is airport, by weather difficulties at the
36. The copilot communicated his con- identified by the operator as an airport time of the search, and by an absence of
cern in a mitigated manner. with special operational considerations, immediate witness information.
37. The intent of the communication which led t o increased workload for the 51. The flight crew was certified and
may not have been understood by the
captain, perhaps because of the mitigat-
ed style of communication chosen by the
copilot or because the captain misinter- EPPESEN KATHMANDU, NEPAL
112.3 0 TRIBHWAN INTL
preted the comment, or possibly for both *ATIS
Approach Control through Tower
of these reasons.
38. The GPWS provided an excessive 'KATHMANDU Tower 118.1
terrain closure rate (Mode 2A)warning.
*Ground 121.9
Alt Set: MB Trans level: FL 150 MSA -.- - --
39. The crew's response t o the GPWS Rwy Elev: 148 MB Trans alt: 13500'(9187') Apt. Elev 4390'
I
warning was not in accordance with the 9-
NX* *
Is
@Pilots are requested to acknowledge
manufacturer's procedures. ike reception of ATlS broadcast t(1
~ ~ N c o n c e r n eon
d first contact.
40. The operator's procedures for
responding t o GPWS did not provide suf-
ficient guidance t o the crew.
41. The captain assessed the GPWS 7106
n 0
warning as false. %>
42. Because of the topography near

the accident site and aircraft perfor-
mance limitations, impact with terrain
likely could not have been avoided even
if the crew had reacted instantly t o the BETWEEN D9.OAND D6.0 USE
LS LCTR FOR TRACK GUIDANCE
GPWS warning. 7~1,
c,
43. It was unlikely that any cockpit

navigation displays had failed, but the
possibility that one FMS ceased working,
because of simultaneous programming,
cannot be eliminated.
py
44. The aircraft navigation systems AUTHORIZED. ACFT MUST MAIN-
TAIN -1 (6187') UNTIL
were operating sufficiently t o allow (L gO
$r
ESTABLISHEDI N B ~ U N D ~ N
02z9
[KTM R-202) AND CLEARED BY
effective navigation. ATC FOR FINAL APPROACH. I
-'#
45. During the approach, the crew's KIT ESTABLISHED INBOUND uD13.0
workload was increased because of com- 3 N 022" LKTM R-2021CAN EXPECT
munication difficulties between them

and the air traffic control agencies and
with the other aircraft on approach, due
t o the radio clarity, language difficulties
and the use of non-standard phraseology.
46. From the first contact with the
ACC controller after the 360-degree turn
until the copilot mentioned the north
direction, the TG311 crew never dis-
cussed the aircraft's flight path.
47. The electronic flight instrument
ISSED APPROACH:
limb STRAIGHT AHEAD to VOR.continue climboutboundonVOR R 4 2 2 to~g:jhen
IGHT turn onto 4 DME Arc, intercept and follow inbound on R-106 to VOR. Then proceedvia
CIIX
~~1.43
system (EFIS) and radio magnetic indica- rtbound VOR R-291 to crossWHlSKEY at or above 9500' (5187') and ioin holding pattern.
tor (RMI) compass displays do not con- STRAIGHT-IN LANDING R
y, - 2 CIRCLE-TO-LAND
tain the letters "N", "S", "E" or "W" t o
show cardinal headings, which might
have provided directional cues t o prompt
the crew.
MDNHJ~~0 (8077
Q@W!VIS!Bl,
I
MDA(HJCEI1-VLS
t NIGHT
Jeppesen chart for the Sierra approach

to Kathmandu, Nepal. (A copy of the
operator's approach chart was not avail-
able for publication.) ANGES: Marker added.
I
@ JEPPESEN SANDERSON, INC., 1985, 1993. ALL RIGHTS RESEW
SEPTEMBER 1993
qualified for the flight in accordance Aviation (DGCA) of Nepal.
with existing regulations. rew appeared to be
52. The air traffic controllers receive Operational Recommendations
training t o the standards required by working in an The aircraft's GPWS provided a warn-
ICAO; no individual licences or ratings
are issued by Nepal. uncoordinated manner 1 ing Of terrain, but too late t o avoid
impact. The Commission recognizes that
when operating the FMS. 1

53. The ACC controller on duty, com- the accident site was on the side of a
municating with the flight was a trainee very steep mountain face and that the
with nine months experience. GPWS provided warning in accordance
54. The controller supervising the with i t s design. ICAO and the GPWS
trainee controller was also fulfilling con- igation duties. manufacturers are currently studying the
trol duties. Contributing factors were: the mis- use of the GPWS and the system itself.
55. The aircraft was certified, equip- leading depiction of Romeo on the oper- The Commission was provided with con-
ped, and maintained in accordance with ator's approach chart used by the flight siderable information regarding the acci-
existing regulations and approved proce- crew; a flap fault, although corrected, dent because of the performance of the
dures. required that the initial approach be dis- flight data recorder (FDR) and the CVR.
56. There was no evidence of any air- continued; and radio communication dif- Information from this accident will be
frame failure prior t o or during the flight. ficulties between t h TG311
~ crew and useful in seeking improvements to both
57. The mass and centre and gravity of the air traffic controllers that stemmed GPWS equipment and procedures related
the aircraft were within prescribed limits. from language difficulties and ineffec- t o i t s use, not just for the A310 but for
tive discussion of apparent unresolved all aircraft requiring GPWS.
Cause problems. During the investigation of the acci-
The probable causes of the accident dent, the Commission noted that at sev-
were: TG311 flight crew's management SAFETY ACTION eral times during the flight the crew
of the aircraft flight path wherein the Interim Recommendations of the appeared t o be working in an uncoordi-
flight proceeded in a northerly direction Commission nated manner when aperating the FMS.
which was opposite to the cleared point On 20 October 1992, the Commission As noted during the investigation, the
Romeo t o the south; ineffective radio issued interim recommendations in order possibility exists that technical problems
communication between the area con- t o address some of the safety issues can be introduced as a result of both
trol centre controller and the TG311 revealed as a result of the investigation pilots inputting data into the FMS.
flight crew which allowed the flight t o up to that time. Letters were sent t o the Although no direct evidence was found
continue in the wrong direction, in that Department of Aviation CDOA) Thailand, that technical problems were created by
the TG311 crew never provided the air- the Directeur General de I'Aviation Civile the crew's action, the potential for such
craft's VOR radial when stating DME and (DGAC) of France, the Administrator of problems is real. These considerations
the controller never solicited this infor- the Federal Aviation Administration are not unique t o the A310 alone, but
mation and thus the aircraft's position (FAA) of the United States of America, are equally applicable t o other FMS-
was not transmitted at any time; and the Certification Director of the Euro- equipped aircraft. It is recognized that
ineffective cockpit crew coordination by pean Joint Airworthiness Administration crew training emphasizes the require-
the TG311 crew in conducting flight nav- (JAA) and the Director General of Civil ment for crew coordination when using
the FMS, but the events of this accident
reinforce the importance of this aspect
REPORTED CFIT ACCIDENTS 1 of training.
Accordingly, the Commission makes
The following CFIT accidents involving civil airliners were reported to ICAO d the following recommendations (to be
1992 and part of 1993. In some cases ICAO did not receive complete reports on coordinated by the DGCA of Nepal):
currences. GPWS manufacturers and ICAO be
A CFIT accident is one in which the aircraft is controllable but flies into obs made aware of the operational and tech-
o r terrain without the awareness of the crew. nical information related t o GPWS per-
3 January 1992 24 March 1992 6 January 1993 formance in this accident.
Beech 1900 Boeing 707-300 de Havilland DHC-8 Manufacturers of flight management
Gabriels, NY Athens, Greece Paris, France
%$@"
3.: .> systems be made aware of the operational
United States and technical aspects of this accident.
8 June 1992 13 January 1993 @&$ tm
20 January 1992 Beech 99 Airliner Embraer 100 Bandeirante $ti The operators and training centres be
Airbus A320 Fort McClellan. AL. Sellafreld. Cumbria +.I encouraged to continue to emphasize, t o
Strasbourg, France United States United Kingdom Q
if
,I operators of aircraft equipped with FMS,
the importance of coordinated use of
9 February 1992 31 July 1992 25 May 1993
Convair 640 Airbus A310 Metro I1 the FMS. 17
Kafountine, Senegal Kathmandu, Nepal Santa Fe, NM
United States
15 February 1992 28 September 1992 The operational safety recommendations contained in
McDonnell Douglas DC-863 Airbus A300 the report of the Commission fox the AecCdent
Kathmandu, Nepal Investigation of TG3ll are currently under review by
tbe ICAO Secretariat as part of it6 programme to
address the CFiT issue.
ICAO JOURNAL
J O U R N A L
V O L U M E 54, N O . 5 JUNE 1 9 9 9
HUMAN FACTORS TODAY:

Managing human error
- - -
ICAO providing the tools and leadership needed
for enhancing safety worldwide
Safety has been bolstered through recent efforts to heighten awareness of human factors, but a fill
commitment on the part of States and the industry is necessary before the aviation community can
achieve the highest level of safety possible.
knowledge about human factors. The practical since it must deal with real
DR.ASSADKOTAITE reason the community must respond to problems in a real world. Through this
ICAO COUNCIL this need is, of course, to ensure that programme, ICAO has provided the avi-
civil aviation continues to achieve its ulti- ation community with the means and
tools to anticipate human error and con-
H
UMAN FACTORS is a critical mate goal: the safe and efficient trans-
aspect of aviation safety, one portation of passengers and goods. tain its negative consequences in the
that ICAO began to address - Before the aviation community can operational environment. Furthermore,
primarily by sensitizing the aviation apply this knowledge with any success, ICAO's efforts are aimed at the system
industry to this new dimension of avia- however, there are two fundamental req- -not the individual.
tion safety - almost a decade ago. uisites that must be met. Firstly, the avi- It is important to have a broad, sys-
ICAO convened the first in a series of ation industry must ensure that temic approach to safety and human fac-
global symposia on flight safety and human-technology interaction remains tors because the requirement for
human factors in 1990. From the begin- human-centred. ICAO adopted a philos- integration goes well beyond the consid-
ning, when the first event was held in a ophy of human-centred automation as eration of isolated human-technology
city known then as Leningrad, there was early as 1991 and has made this one of interface issues. Human performance
a conviction that international aviation the pillars of its CNS/ATM systems con- takes place in an operational context,
could make enormous progress in cept. A philosophy of human-centred and human factors knowledge must
improving safety through the application automation is the only safeguard against therefore be applied to operational sys-
of human factors knowledge. the development of uncooperative tems. It is because of this that the activi-
That first symposium was a turning human-technology interfaces with the ties of the ICAO flight safety and human
point and set the stage for follow-up potential for safety breakdowns. factors programme must be integrated
meetings in the United States in 1993, in The second requisite deals with inte- with two closely connected, major sys-
New Zealand in 1996 and, finally, in Chile gration. The industry has designed temic safety initiatives.
in 1999. There have been encouraging excellent technology that has undoubt- The first of these initiatives, the global
developments since 1990, but it must be edly contributed to improvements in aviation safety plan (GASP), was devel-
emphasized that we still have challenges safety, but we need to take another look oped by the ICAO Air Navigation
to pursue: nine years after the Leningrad at the operational contexts into which Commission in 1997 and subsequently
Symposium, human error remains a sig- these technologies are deployed. In approved by the ICAO Council and
nificant safety concern. other words, human capabilities and Ern- endorsed by the ICAO Assembly. GASP
The purpose of the worldwide sym- itations should be taken into account was designed to coordinate and provide a
posia and 10 regional seminars which when defining the blueprint for the sys- common direction to the efforts of
were held in the past decade was to tem and before any system can enter into Contracting States and the aviation
increase the awareness of States, indus- operation. ICAO has taken the initiative industry to the extent possible in safety
try and organizations in all ICAO in this challenge by developing human matters. It is a tool that allows ICAO to
regions about the importance of human factors standards that consider human focus resources and set priorities, giving
factors. The ongoing implementation of performance in present and future oper- emphasis to those activities that will con-
the ICAO communications, navigation, ational environments, and the aviation tribute the most to enhancing safety. It
surveillance and air traffic management community is strongly encouraged to fol- should come has no surprise, therefore,
(CNS/ATM) systems concept has intro- low these standards. that the flight safety and human factors
duced new challenges, and also new pos- The ICAO flight safety and human fac- programme is among the six major activ-
sibilities for human error, thus creating tors programme is safety-oriented and ities that comprise the plan. However,
yet another opportunity to apply our vast operationally relevant. Moreover, it is before GASP can have much effect, it is
ICAO JOURNAL
crucial to achieve the highest
degree of global cooperation in
implementing the major ele-
ments. ICAO can provide the
framework for GASP to function,
but it will ultimately be the con-
certed effort of the aviation com-
munity, States and industry - a
global cooperation - that will
largely determine its success.
This worldwide cooperation
underlies another major sys-
temic safety initiative: the ICAO
universal safety oversight audit
programme. This is a pro-
gramme of regular, mandatory,
systematic and harmonized safe-
ty audits carried out by ICAO
in all Contracting States. The
information obtained from these
audits will allow States, given
reasonable time, to remedy defi- A deeper understanding of human error holds the key to continued improvement in the air
ciencies in safety oversight safety record.
responsibilities. Mechanisms
have been developed that will which flight takes place. On the one States engage in far-reaching coopera-
enhance transparency and increase dis- hand, the physical environment, with tion on safety problems related to the
closure of results, as well as eventually extreme temperatures and pressures, influence of human factors, and directed
expand the programme to all areas rele- makes unsupported human life impossi- ICAO to develop and introduce material
vant to civil aviation safety. There is no ble. In addition, speeds allowing ultra on various aspects of this issue. The goal
better example of global cooperation in long-haul, trans-meridian operations in now, as then, is to further improve safety
action. short periods of time require careful in aviation by making States and the
Profound social and political forces consideration of basic human perform- industry more aware of, and responsive
are reshaping our world on a daily basis, ance issues such as jet-lag and circadian to, the importance of human factors in
inexorably leading towards globalization. disrythmia. On the other hand, the civil aviation operations. This is accom-
Solutions to safety problems lie in the socio-economic environment, with mar- plished through the provision of practi-
cooperation that exists among countries, ket demands that require aviation organ- cal human factors material and measures
and among countries and the industry at izations to attempt to produce "more developed on the basis of experience.
large. All this needs to be supported by with less" in order to remain economi- The full understanding of human
a corresponding level of global coordi- cally viable, generates inevitable bur- error holds the key to continued success
nation in all aspects of civil aviation with dens upon those who operate, maintain in providing people everywhere with the
relevance to safety. Safety, after all, is not and control the system. safest means of transportation ever cre-
a national, regional or even continental One major obstacle remains to make ated. Humans, it must be said, can never
issue, but an issue on a global scale. As aviation's safety record all but perfect: outperform the system that bounds
a global body, ICAO can and will assume human error. Human error is a safety them. In no small way, safety practition-
its leadership role through various ini- obstacle not just in the cockpit but in ers in all disciplines of aviation are the
tiatives, but States -through their reg- every process surrounding flight opera- players who will define further strate-
ulatory bodies - and the international tions, from designing and manufacturing gies for understanding human error,
aviation community at large must be aircraft and navigation equipment to the making it possible to achieve our safety
fully committed to the attainment of the commercial decisions that affect daily objectives. 0
highest level of safety. operations.
The safety of civil aviation is particu- The objective today remains un- This article is an adaptation of the opening address
given by 1C40 Council President Dr. Assad Kotaite at the
larly notable when one considers the changed from 1986, when the ICAO 4th Global Fhght Safety and Human FactorsSymposium
extraordinarily hostile environment in Assembly proposed that Contracting held in Santiago, Chile from 12 t o 15 April 1999.
JUNE 1999
CRM training primary line of defence against
threats to flight safety, including human error
................................................................................. erable evidence has also been accu- recently defined CRM as "instructional
L. HELMREICH
ROBERT mulated to show the effectiveness of strategies that seek to improve team-
UNIVERSINOF TEXAS CRM training in changing behaviours work in the cockpit." While effective
(UNITEDSTATES) and attitudes. teamwork is clearly important, it is not
................................................................................. The situation is not, however, entirely the primary goal of CRM training.
S
INCE THE END of the 1980s the positive. There are four areas of con- The following description is a more
concept of crew resource manage- tention. Firstly, CRM has been faulted for accurate representation of the latest CRM
ment (CRM) has spread world- failing to eliminate pilot error and acci- programmes: crew resource management
wide, has come under some criticism, dents. Secondly,there is confusion among is a subset of the human factors disci-
and is now finding its place in our safety some members of the research and oper- pline which is concerned with human-
culture by focusing on error and threat ational communities about its definition. machine and human-human interfaces
management. CRM training is mandated Also, it is claimed by some critics that and the integration of human operators
for pilots in all ICAO Contracting States, CRM issues are too subjective to be eval- and maintenance personnel within an
and airlines of some countries of the Eure uated fairly, thus subjecting pilots to the overall system. CRM consists of the
pean Union will be required to evaluate risk of discrimination. Finally, there is application of human factors knowledge
individual proficiency in CRM-related confusion about the role of CRM pro- to the special case of flight crews and
behaviours. CRM training is also being grammes within organizations. their interactions with each other, with
extended to other personnel in the avia- Has CRM failed? The contention that other groups, and with the technology in
tion system, including flight attendants, it has not achieved its mission because the system. Broadly defined, CRM con-
dispatchers and maintenance technicians. accidents still occur is based on a pro- sists of the effective utilization of all
As far as its application is concerned, found misunderstanding about human available human, informational, and
CRM is certainly a success story: consid- capabilities and limitations. By their very equipment resources toward the goal of
nature, humans are prone to errors safe and efficient flight. More specifical-
and no amount of training, howev- ly, it is the active process employed by
er sophisticated or intensive, can crew members to identify existing and
change human nature. Humans potential threats and to develop, commu-
will make errors, and accidents nicate, and implement plans and actions
and incidents will occur in complex to avoid or mitigate perceived threats.
systems. CRM also supports the avoidance, man-
The definition of CRM is misun- agement, and mitigation of human
derstood. Although CRM pro- errors. The secondary benefits of effec-
grammes were clearly rooted in tive programmes are improved morale
efforts to reduce accidents caused and enhanced efficiency of operations.
by "pilot error", comprehension of Many are of the view that CRM issues
the programme goals over the are subjective and cannot be evaluated.
years has faded, perhaps in part Fear of inequitable evaluation is certain-
because of the extension of the ly a legitimate concern of pilots whose
training concept to flight atten- livelihoods may be threatened. However,
FlgUre I. A model ot threat ana error manage- dank and other personnel. An indi- contemporary CRM programmes focus
The model features external threats Iin cation of this misunderstanding on specific and well-defined behaviours.
red boxes), internal threats (blue box), CRM
behaviours (in dark green boxes) and outcomes can been seen in the work of scien- Those behaviours chosen for evaluation
(light green and black boxes). tists in the United States who in the Europcan Union (which are close-
ICAO JOURNAL
ly related to the "behaviourial markers"
defined by the University of Texas
research group) are objective and
observable.
What is the role of CRM in the organ-
ization? Some have argued that CRM
should ultimately disappear as it be-
comes fully integrated into technical
training. With hindsight, however, most
experts now realize that crew resource
management is, and should remain, a sep
arate aspect of training. Crew resource
management falls within the area bridg-
ing the safety, flight training and flight
operations departments, and training in
crew resource management is ongoing
and driven by objective data reflecting Flight operations are part of a complex system that is heavily influenced by national,
operational issues. CRM is not a one- organizational and professional cultures.
time intervention, but rather a critical
and continuing component of a safety tures characterized by high power dis- attitudes and policies about human error,
culture. tance show great concern for the group the openness of communications between
and harmonious relationships, and def- management and flight crew, and the level
Culture: the missing element erence to leaders. Another relevant of trust between flight crew and senior
Early CRM programmes and investi- dimension concerns the attitude toward management. Organizational culture also
gations of human error in accidents rules. Those high on this attribute influences norms regarding adherence to
viewed the cockpit as an isolated uni- believe that rules should not be broken, rcgulations and SOPs. Of great impor-
verse. With growing sophistication, it is that written procedures are needed for tance, the organizational culture deter-
now understood that flight operations all situations, and that strict time limits mines the level of commitment to safety
are part of a complex system that is should be observed. This dimension has and the strength of a safety culture.
heavily influenced by cultures. There are proved to be one on which there are Professional culture. Many professions
three intersecting cultures that surround large and highly significant cultural dif- exhibit strong cultures and develop their
every flight crew - national, organiza- ferences. At the high end, many Asian own norms and values along with recog-
tional, and professional. One expert has cultures are rule-oriented, while the nizable physical characteristics such as
defined culture very aptly as "the soft- United States and United Engdom are uniforms or badges. In aviation, the pos-
ware of the mind." More technically, cul- examples of the other end of the contin- itive aspects of the professional culture
ture consists of the shared norms, values uum, demonstrating much lower con- are shown in strong motivation to do
and practices associated with a nation, cern for rules and written procedures. well and a high level of professional
organization or profession. Some facets Looking at the cockpit environment, pride. There is also a negative compo-
of national culture may influence behav- national culture influences how junior nent that is manifested in a sense of per-
iour in the cockpit. crew members relate to senior pilots, sonal invulnerability. University of Texas
National culture. Two related dimen- including their willingness to speak up researchers have found that the majori-
sions of national culture have particular with critical information. It is demon- ty of pilots of all nations agree that their
relevance for aviation: individualism ver- strated in the way information is shared. decision-making is as good in emergen-
sus collectivism, and "power distance" Culture influences adherence to stan- cies as it is in normal situations, that
(i,e. the remoteness of those in society dard operating procedures (SOPs). It their performance is not affected by per-
having the authority to make major poli- has also been found, unexpectedly, that sonal problems, and that they do not
cy decisions; in a culture with low power culture is strongly associated with a per- make more errors when highly stressed.
distance, for example, decision makers ception of automation and with attitudes While the positive aspects of profession-
are generally accessible). about the appropriate use of automation. al culture undoubtedly contribute to avi-
Those from individualistic cultures Organizational culture. Organizations ation's excellent safety record, the sense
that also have low power distance tend to can function within a national culture or of invulnerability can lead to risk taking,
focus on the self, autonomy, and person- can extend across national boundaries. An failure to rely on fellow crew members,
al gain, while people from collectivist cul- organization's culture demonstrates its and errors.
JUNE 1999
CRM programmes built on data behaviours that have been associated with nal errors, the crews themselves may
CRM programmes provide a primary accidents and incidents (and that form the err and, again, CRM behaviours stand as
line of defence against the threats to basis for contemporary CRM training). the last line of defence. If the defences
safety that abound in the aviation system Data from LOSA provide a valid picture of are successful, error is managed and
and against human error and its conse- system operations that can guide organi- there is recovery to a safe flight. If the
quences. Today's CRM training is based zational strategy in safety, operations and defences are breached they may result
on accurate data about the strengths and training. A particular strength of LOSA is in additional error or an accident or inci-
weaknesses of an organization. Building that the process identifies examples of dent. In our three-airline database, 72
on detailed knowledge of current safety superior performance that can be rein- per cent of all flights faced one or more
issues, organizations can take appropri- forced and used as models for training. threats to safety, with the number of
ate remedial actions. Data collected in LOSA threats on a flight ranging from zero to
There are five criti- Type of incident A c are proactive and can 11.The average was two threats to safety
cal sources of data 01 be used immediately to on each flight. The most common
which CRM program- prevent adverse events. threats encountered were challenging
mes can be built, eacl University of Texas re- terrain (on 58 per cent of flights),
source illuminating a Figure 2. Threat and error rate at searchers have partici- adverse weather (28 per cent), aircraft
different aspect of flight three airlines ( A, B and t). pated in eight audits with malfunctions (15 per cent), unusual ATC
operations. Data can be a total of nearly 4,000 commands (11 per cent), external
obtained from formal evaluations of per- flights observed. The data from the errors, including ATC, maintenance,
formance in training and on the line, as three most recent audits, which include ground handlers, etc. (8 per cent), and
well as incident reports from non-puni- threat recognition and error manage- operational pressures (8 per cent).
tive systems that encourage open comment, are described below. These three Threats to safety arose most frequently
munication and surveys of flight crew LOSA projects were conducted both in in the descent, approach and landing
perceptions of safety and human factors. the United States and in international phases (40 per cent).
Quick access recorders provide informa- operations and involved three air carri-
tion on the conduct of the flight. It ers, including two based in the United A model of crew-based error
should be remembered, however, that States. Errors made within the cockpit have
while these data provide a reliable indi- received the most attention from safety
cation of what occurred, they do not indi- A model of threat and error investigations, and crew error has been
cate why it happened. Line operations Data are most valuable when they fit implicated in around two-thirds of air
safety audits (LOSA) are also an impor- within a theoretical or conceptual frame- ~ r a s h e s Our
. ~ analyses of human error
tant source of information. work. Our research group has developed have led us to reclassify and redefine
The nature and value of LOSA. Line a general model of threat and error in error in the aviation context. Opera-
operations safety audits are programmes aviation (see Figure 1, page 6). tionally, flight crew error is defined as
that use expert observers to collect data As the model indicates, risk comes from action or inaction that leads to deviation
about crew behaviour and situations on both expected and unexpected threats. fiom crew or organizational intentions or
normal flights. They are conducted under Expected threats include such factors as expectations. This definition classifies
strict non-jeopardy conditions, meaning terrain, predicted weather, and airport con- five types of error:
that no crew member is at risk for ob- ditions. Those that are unexpected include intentional noncompliance or viola-
served acti0ns.l Observers code threats air traffic control (ATC) commands, sys- tions of SOPS or regulations (examples
--
to safety and how they are addressed, tem malfunctions, and operational pres- include omitting required briefings or
errors and their management, and specific sures. Risk can also be increased by checklists) ;
errors made outside procedural, in which the intention is
the cockpit by, for correct but the execution flawed;
Intentional Procedural cornmunica,ion Proficiency Or;rnal example, A X , main- communication ervon that occur when
Type Of fleet noncornpllance
- tenance and dispatch. information is incorrectly transmitted or
External threats are interpreted (examples include incorrect
countered by the defen- readback to ATC or communicating
ces provided by CRM wrong course to the other pilot);
lxhvbum Wkn suc- groficiency enom that indicate insuffi-
cessful, these lead to cient knowledge or lack of stick and rud-
Figure 3. Percentage of error types within different types of fleets
at one airline. Included are data related to two types of advanced- a safe flight. der skills; and
technology flight decks and two types of conventional aircraft. In addition to exter- operational decision errors in which
ICAO JOURNAL
CREW RESOURCE U A N A G E M E
crews make a discretionary decision that The model of threat and error illus- and 3. (A consequential error is one
unnecessarily increasesrisk (examplesin- trated in Figure 1 aids analysis of all resulting in an additional error or an
clude extreme manoeuvres on approach, aspects of error, response, and outcome. undesired aircraft state.) This finding
choosing to fly into adverse weather, or The failure or success of defences such indicates the importance of airlines'
over-relianceon automation). as CRM behaviours can also be evaluat- determining the status of their own oper-
Three types of responses to these ed. Errors thus classified can be used not ations rather than assuming that their
errors have been identified. In the first only to guide organizational response but organization conforms to an industry
instance, the error is detected by the as scenarios for training, either in class- standard. Variability can result from dif-
crew and managed before it becomes room or line-oriented flight training ferences in the operating environment,
consequential or leads to additional error. (LOFT). as well as from differing organizational
In the second case, the error is cultures and subcultures. Of particu-
detected, but the crew's action or lar interest is the range of variables
inaction leads to a negative outcome. Phase Of flight Errors errors which became consequential, termi-
The last type of response is, in fad, a nating in an undesired state.
failure to respond. The crew does ( Certainly, there is differential risk
not react either because the error is at the three organizations which par-
undetected or because the crew ticipated in the study. The three air-
chooses to ignore it. Figure 4. Percentage of errors and consequential lines share some common factors,
errors experienced by three airlines.
Definition and classification of but the data presented here are
errors and responses are based on the shown primarily to indicate the type of
observable process, without considera- LOSA results information that can be obtained and uti-
tion of the outcome. There are three pos- Examination of the aggregate data lized for safety.
sible outcomes: in the first case, it is from the first three LOSAs in which Phase of flight is also strongly associ-
inconsequential, meaning that the error error was measured is instructive. ated with the occurrence of errors and
has no effect on the safe completion of Errors were committed by 73 per cent of their consequences, as shown in Figure 4.
the flight. This is the model outcome, the crews observed. The number of Consistent with analyses of worldwide
one that is illustrative of the robust errors on a flight ranged from zero to 14, approach and landing accidents, the
nature of the aviation system. In another with an average of two errors per flight. highest percentage of errors (and the
case, the error places the aircraft in a The most frequent error type identified highest percentage of consequential
condition that increases risk (undesired by the study was intentional non-compli- [serious] errors) occurs during this
state). This outcome includes incorrect ance (or violation), followed by those of a phase of flight. Clearly, special attention
vertical or lateral navigation, unstable procedural nature. The high percentage should be directed toward enhancing
approaches, low fuel state, and hard or of errors in the procedural non-compli- performance in the approach and land-
otherwise improper landings. A landing ance category is alarming (this finding is ing phase.
on the wrong runway or at the wrong air- described in greater detail below). Specific errors. Although a wide array
port would fall under this category. In Procedural errors doubtless have mul- of error classifications was observed,
addition, the response to error, as noted tiple causes. They can reflect the inherent some major problem areas emerged.
above, can result in another error that limitations of humans in accomplishing Earlier audits pointed to the use of auto-
again initiates the cycle of response. difficult tasks, often under high work- mated systems as an industry-wideprob
High risk situations can be mitigated load conditions, or they may be an indi- lem. Consistent with these findings, the
by crew action; for example, recognition cation that the procedures themselves most frequent classification of error in
of an unstable approach can trigger a are sub-optimal. Of all the errors LOSA involved the operation of automat-
decision to initiate a go-around. Response observed, 18 per cent were corrected ed systems (mode control panel and
to an undesired state may end in another before they could have any conse- flight management computer). Errors
error or, in the worst case, the result is quences, 5 per cent resulted in a negative included wrong settings, wrong modes
an accident or incident. outcome despite detection by the crew, and failure to verify settings, along with
and 77 per cent elicited no response. numerous others. Overall, these
1. In practice, members of the University of Texas Of particular significance was the fact accounted for 31 per cent of all errors.
project have trained observers from participating air- that there were very large differences in The second highest classification was
lines, and also serve as observers. Their presence
across all organizations allows the research team to threat, error, and the percentage of checklist errors (24 per cent) such as
make valid cross-airline comparisons. errors that became consequential non-standard terminology, procedural
2. Early investigations tended to focus on the crew as
the sole causal factor. Today, of course, it is known between fleets within one airline, and errors, performance from memory, and
that almost all accidents are system-related. between airlines, as shown in Figures 2 failure to use required challenge and
JUNE 1999
response methods. The third highest ization of operations cannot be achieved ior crew members, many current prob-
category, accounting for 13 per cent of with idiosyncratic adherence to proce- lems seem to be associated with weak
errors, consisted of sterile cockpit viola- dures. There is also compelling evi- leadership and the abdication of author-
tions. Fourth highest, at 8 per cent, dence that safety threats are associated ity. While the importance of the identi-
were ATC-related crew errors such as with violations. For example, a Flight fied markers is not surprising, the
missed calls, omitted information, and Safety Foundation (FSF) analysis of results do provide important validation
acceptance of instructions that increase global approach and landing accidents of the importance of CRM-related
risk unnecessarily (i.e. "slam d u n k found that more than 40 per cent of behaviours.
approaches). The fifth highest category these accidents involved violations of
(5 per cent) consisted of briefing errors, SOPS. Also, an analysis of LOSA data Data, CRM and safety culture
failure to conduct required brietings or indicates that those who commit inten- The analysis of data from a variety of
leaving out required information. The tional non-compliance errors are more sources (training evaluations, incident
remainder of errors fell into a variety of likely to commit other types of errors. It reports, surveys, and LOSA) aids organ-
categories. can be concluded that violations are izations in the diagnosis and under-
associated with greater risk in opera- standing of their culture and its
Violations matter subcultures. Without an under-
The research group was dis- standing of their own cultures,
mayed by the high proportion CRM not a one-time intervention.,
is organizations cannot mount
s
-a.
* "7
of intentional non-compliance effective programmes to opti-
errors found in the data. Several but rather'a critical and continuing combonent rnize them. Data on how crews
points regarding these violations deal with threats to safety and
should be considered. First, as avoid and manage error help
noted above, there were very organizations develop and main-
large differences between airlines and tions, and further analyses may give us tain a safety culture. LOSA data, in par-
between fleets within airlines. Hence, greater insight into the nature of this ticular, are of enormous value because
one cannot generalize from these data relationship. they are proactive and allow organiza-
about the general frequency of violations to take appropriate action before
tions in the global aviation system. This CRM as a countermeasure accidents and incidents occur. Proactive
point is further emphasized by the fact One of the most informative aspects interventions are a defining characteris-
that the three carriers included in the of LOSA data is the ability to link threat tic of an effective safety culture.
study all came from countries that recognition and error management with Data also iden* critical areas for
scored very low on commitment to the specific behaviourial markers that ongoing CRM training. However, as noted
rules. It would be incorrect to assume form the core of crew resource manage- above, CRM is not a universal panacea for
that pilots from other cultures would be ment. These emerge very clearly in safety problems in the aviation system.
equally cavalier in disregarding formal observer ratings and provide a clear Accidents and incidents almost always
rules. On the other hand, the universal portrait of the actions taken by effective have multiple roots and many cannot be
pilot belief in personal invulnerability crews. Those who deal proactively with changed by training alone. Organizations
may foster a disregard for rules. The safety threats and error management nurturing a safety culture must also deal
fact that many rules are broken does not exhibit several behaviours. They brief with those issues identified by LOSA and
imply that pilots have a death wish or one another on known safety threats; other data sources. Interventions may
feel contempt for formal requirements. ask questions and speak up; constantly include making revisions to procedures,
One must also consider the possibility re-evaluate their decisions; communi-
continued on baae 29
that the proliferation of regulations may cate operational plans clearly; prepare
have created a contradictory, unwieldy and plan for safety threats; distribute the Robert L. Helmreich is a Professor at the University of
Texas at Austin and leader of the University of Texas
and inefficient operating environment workload and tasks; and exercise vigi- Team Research Project, an ongoing study of crew re-
that invites violations. lance. Captains show leadership. source management issues. The research reported on
here, focusing on the management of human error,
Although many violations may be Leadership is an overarching behav- commenced in late 1997.
committed with the good intention of iour that governs interaction on the The research described in this article was conducted
with financial support from the US. Federal Aviation
increasing operational efficiency, organ- flight deck. Although much of the atten- Administration and was made poss~bleby the partici-
izations cannot and should not tolerate tion in early CRM programmes was their patlon of three airlines and the full cooperation of
personnel, who also assisted with data analysis
disregard for established procedures. directed toward overcoming the effects and development of conceptual models.
previous article by Prof. Helmreich, with focus on
There are several compelling reasons of autocratic captains who failed to solic- theAeffect of national culture on flight crew behaviour,
for this. One is, of course, that standard- it or accept critical information from jun- appeared in October 1996 (pp. 14-16).
ICAO JOURNAL
Tools developed by aircraft builder designed
to help airlines manage human error
Boeing's latest h u m a n factors analysis tool, to be released to the industry later this year o n a no-cost
basis, focuses o n flight crew compliance with procedures.
occurrence, we can never be certain it today. So why are errors often blamed on
will not occur. Consequently, we must negligence or incompetence without
enhance our ability as an industry to looking more broadly at the system and
manage error in order to mitigate its the way it supports (or doesn't support)
consequences and to learn what system- human performance? Even when more
H
ELP for efforts to resolve human atic factors contribute to its occurrence. serious incidents and accidents occur, it
factors issues is available horn is rare to see a thorough human factors
aircraft manufacturers, and Boeing Error management tools analysis conducted.
in particular has identified several ways Aviation has traditionally relied on If the aviation industry is to make the
in which a manufacturer can play a s u p selection, training, licensing and detailed human performance gains necessary for
portive role. For example, Boeing is work- written procedures to assure saiety. dramatic reductions in the accident rate,
ing to develop and provide pilot training While these are important barriers to it needs more extensive and reliable
aids and to supply tools for managing human error, this emphasis ignores the feedback on how humans interact with
human error and mechanisms for feed- very real contributions that design, envi- technology in the real world. The indus-
back, and is also conducting human fac- ronment, and other factors make to try needs to foster further development
tors research. The company supports human performance. An over-reliance of human factors tools, databases and
initiatives in the human factors area by on discipline to make the system work support policies across all sectors of the
ICAO and the International Air Trans- well characterizes many government industry, not just for flight crews. Of
port Association (IATA). authorities as well as air carriers. The course, the biggest challenge will be the
Many readers will be familiar with the phrase "blame and train" probably best political and legal frameworks needed to
training aid designed to prevent con- describes the predominant attitude to- encourage honest reporting when
trolled flight into terrain (CFIT) acci- wards those who err and are caught. As human error occurs.
dents; developed by a task force formed a result, human performance issues are In 1991 Boeing initiated an effort to
by ICAO and the Flight Safety Foun- often not given the systematic level of shift the focus of accident analysis away
dation (FSF), the training aid was pro- analysis they deserve in order to prevent from primary cause to the development
duced by Boeing. A more recent joint their future occurrence. of accident prevention strategies. This
effort that involved manufacturers has Yet, there has always been an implicit was accomplished by reviewing and
culminated in the development of the assumption that the trained pilot or analysing commercial jet aeroplane acci-
upset recovery training aid. These re- mechanic can always be counted on to dents over a 10-year period (1982-91),
cent examples of training aids bear con- remain sufficiently
siderable similarity to the development flexible and creative
of the wind shear avoidance training aid to fill the gaps in the
in the mid-1980s. system to maintain
A manufacturer can help in capitaliz- safe performance.
ing on human factors to achieve a signif- Given the often unpre-
icant reduction in the accident rate. This dictable nature of the
is accomplished by developing and pro- aviation operating
- .pilot =
considerations; (C) non-flyinq
environment, there is adherence to procedure; (D) .embedded
viding methods and tools to manage
20
human error more effectively. Over the no doubt that this piloting skills; (E) design improvement;
past several years, our human factors uniquely human abili- (F) ca~ainlinstructor~ilot exercise of
lo
authority; and (G) maintenance or
specialists have been focusing on the ty has been a major inspection action.
fact that human error is inevitable. While factor in making avia- Boeing data
all of us can do our best to prevent its tion as safe as it is
JUNE 1999
HUMAN FACTORS
an accident or serious incident. Un- ture of the U.S. Air Transport Associa- the flight crew is not subject to punish-
fortunately, the latter approach is rarely tion (ATA) Human Factors Committee, ment or disciplinary action unless they
exercised in incident investigations, industry involvement has been expand- were deliberately neglectful or acted in a
resulting in a mostly negative view of ed to include a multi-cultural team in reckless manner. Within this framework,
flight crew performance when proce- order to adapt PEAT into an incident the crew's professionalinput is sought to
dures aren't followed. Together we con- analysis tool that meets global needs. improve the overall understanding of
sistently fail to consider the number of The industry team consisted of eight air- what happened and to gain insight as to
accidents that may have been prevented lines from the United States, Europe, why it happened. PEAT provides the
because the crew did deviate from the
procedures.
PEAT is similar in design to MEDA
and likewise assumes that there are rea-
sons the pilot either failed to follow the
procedure or made an error in following
it -that is, the error was not malicious.
This allows the analyst to interview the
pilot involved and document the error
and the reasons behind it. Both tools
take advantage of what we know about
the cognitive or decision-makingaspects
of procedure adherence, and they offer
an inventory of the types of procedural
errors that might occur and the factors
that can influence human use of proce-
dures.
Development of PEAT For some time now, human factors

Boeing first tried to devel-
op a better understanding c -
I specialists have been focusing
methodology for guiding the collection
of this input along with other relevant
non-adherence with proce- facts and data.
dures by returning to the on the fact that human error is inevitable. In contrast to the wide variability in
accidents that had been current airline investigation methods,
analysed for its accident pre- rn =-
vention strategies study, and applying an and Asia, working together with Boeing
PEAT provides consistency in applica-
tion and results. The PEAT form,
analysis that focused on the cognitive and the International Federation of Air designed to be used by a trained safety
factors that could be responsible. Line Pilots' Associations (IFAZPA) . The officer, can facilitate the investigation of
Despite repeated attempts to apply the team participated in an eight-month field specific types of incidents, i.e. those
analysis across various accidents, how- validation using preliminary paper ver- involving non-adherence to procedures.
ever, Boeing finally concluded that the sions of PEAT to investigate their own As such, it addresses all the pertinent
disparity in accident report quality, their incidents involving significant non-adher- analysis elements. The Boeing-industry
inconsistency in addressing human fac- ence to procedures, and to adapt PEAT team found that by asking such ques-
tors issues, and the inability to interview to better meet their requirements. tions they obtained information that sub-
crew members made it impossible to The software version of PEAT has stantially expanded their ability to
achieve a reliable result. been designed to facilitate a paradigm understand the incident.
Boeing then turned its attention to shift in how incident investigation is con- The adoption of the PEAT philosophy
serious operational incidents. As a ducted. PEAT is based on a philosophy by the safety officer also facilitates the
result, PEAT has developed into a struc- which acknowledges that professional shift of the investigation focus away from
tured, in-depth analytic tool based on a flight crews rarely fail to comply with a what happened and who is responsible to
cognitive approach. It is designed to procedure intentionally, especially if it is why it happened by focusing on the key
facilitate incident investigations and to likely to result in an increased safety contributing factors. As stated above,
aid in the development of countermea- risk. It therefore requires the airline to 1. Federal Aviation Administration, Human Factom
sures. While the initial accident-oriented explicitly adopt a non-jeopardy approach Team Report on the InterfacesBetween Flight Crews and
Modern Flight Deck Systems, Washington, D.C., 1996.
effort was accomplished within the struc- to incident investigation. In other words,
JUNE 1999
flight crews rarely make procedural ent types of information on similar or usually asked (thus expanding the scope
errors intentionally; however, there are related incidents and offer an opportuni- of the investigation); encourages ques-
circumstances and factors that affect ty to spot potential risk areas. tions no one dared to ask before; and
crew decisions and can contribute to Finally, PEAT provides a mechanism helps move investigators away from the
such errors. Therefore, the desired for Ieedback and data sharing. PEAT "blame and train" mindset. Feedback
change in crew behaviour can only be facilitates the communication of relevant also indicated that PEAT is flexible
accomplished by objectively addressing information to various departments, both enough to support airline-unique needs.
why the incident occurred. internal and external, to the airline
A model of human information pro- organization. For example, if an investi- PEAT implementation
cessing, depicted in Figwe 2, empha- gation reveals the need for improvement Boeing Flight Technical Services will
-
sizes this point. Crew actions are the in the area of procedural development, be responsible for worldwide industry
consequence of complex mental opera- the relevant information can be readily implementation of PEAT starting in
tions that are characteristic of human shared with both the flight standards and 1999. Effective adoption and application
cognition and that are clearly influenced trainjig departments. If maintenance has of the PEAT process and software
by available information and the sur- been identified as a cantributing factor, requires hands-on training. Training and
rounding environment, includ- implementation of PEAT at cus-
ing airline policies and culture 7 tomer airlines will be coordinated
as well as regional culture. 1 1 through the customer service
As pointed out by the FAA, representatives for each carrier.
some procedural deviations In this manner, operators can
have produced desirable out- reserve specific training slots
comes for safety. Therefore, it tensive and reliable feedback on how humans throughout the year. As part of
is important to obtain a bal- I the implementation plan, airline
anced perspective on flight nteract with technoloo senior management will be pro-
crew adherence to procedures. vided with an overview of the
PEAT's structure enables op- PEAT philosophy, process, re-
erators to do just that. Its format is struc- the pertinent information may be shared quired organizational support, and a
tured so that it can be used to help internally with airline maintenance, and model for successful airline implementa-
understand what contributed to a flight externally with the manufacturer. tion. To facilitate crew cooperation, air-
crew's correct decision regarding inten- A manufacturer needs to know when lines may also want to include pilot
tional deviation. This type of information a crew interface design may have con- representatives in such briefings. This
may eventually prove valuable in training tributed to a procedural deviation. PEAT will be followed by the training of safety
and in modifying existing standard oper- will enable Boeing to improve future officers for effective application of the
ating procedures. product design by furthering our under- PEAT process, and the training of ana-
By implementing more effective data standing of such critical incidents, PEAT lysts in the use of the PEAT software
collection and consistent analysis over also can foster the data sharing of "best database and analysis capabilities.
time, PEAT can make incident error practices" among operators, should air-
trends more visible. This trend informa- lines wish to share this information.The Conclusion
tion can provide more obvious opportu- software has built-in security features Today's air transportation system is
nities for early intervention both within designed to provide strict control over very safe, and many safety professionals
the airline and potentially across the the storage of data, access to data and have played a role in making that hap-
industry. This is also one reason why the nature of data shared. While airlines pen. However, there is still a significant
Boeing has sought to enable PEAT are encouraged to share PEAT data amount of work ahead as the industry
results to be readily integrated with among themselves and with Roeing,
those from less serious incidents. PEAT such sharing is not required.
analytic outcomes can be readily entered Perhaps the industry team's com- Dr. Graeber is Chief Engineer, Human Factors, a t
Boeing Commercial Airplanes, where he is responsible
into industry safety bases existing today ments after the field evaluation provide for managing Boeing's human factors activities in air-
which are typically used to track inci- the most concise summary from the craft design, product development, safety analysis and
accident investigation. Dr. Graeber serves on several
dents that do not require a formal inves- user's viewpoint about PEAT's value to industry committees including the JAA Human Factors
tigation, and which are often reported by an operator. According to the team mem- Steering Group, the lATA Human Factors Task Force
and the Flight Safety Foundation's lcarus Committee.
crews themselves. Thus, PEAT can be bers, PEAT legitimizes the depth of the This article is an adaptation of a paper presented
by Dr. Graeber t o the 4th Global Flight Safety and
used in conjunction with other available inquiry; provides a systematic approach Human Factors Symposium in Santiago. Chile from 12
industry safety tools to compare differ- to the investigation; raises questions not to 15 April 1999.
ICAO JOURNAL
New concept in human factors training develops
controllers' skills at efficient teamwork
Eurocontrol has developed a human factors programme which provides training in teamwork for
ATC staff: The course, adaptable to suit the experience and cultures of different countries, parallels
similar training which has been provided to flight crews and other airline personnel for some time.
services in the European Civil Aviation controller may have no further perform-
EUROPEAN ORGAN~SAT~ON FOR THE Conference (ECAC) area.' Increase in ance evaluations. Since explicit training
SAFETY OF AIR NAVIGATION traffic capacity, together with improved in the principles of teamwork is rarely
safety and efficiency, remain the main provided, the development of teamwork-
related skills depends largely on the
0
VER the last 20 years, airlines drivers for the entire EATCHIP or
have been increasingly success- EATMP work programme. nature of the controller's team. Some
ful in implementing the idea of The EATMP human factors integra- European countries have individual shift
enhanced teamwork. Many airlines tion programme concentrates on the rosters which do not allow for fixed-team
around the world apply the principles of development of harmonized and inte- structures and staff work in temporary
crew resource management (CRM) to grated human factors methodology for teams based on the random allocation of
pilots and other operational airline staff, current and future air traffic manage- work positions, similar to the crew com-
and it is perhaps surprising that "con- ment. The main objective of this work is positions in airlines.
troller resource management" did not to develop and apply human factors prin- For the purpose of developing a con-
develop in conjunction with this concept. ciples and methods for maximizing cept for team resource management, it
Although a great deal of effort and human performance and making the seemed necessary to concentrate more
expertise is devoted to training individu- best use of evolving technology. on teamwork processes than on team
als in the technical skills necessary for air structure and to use a more generic def-
traffic control (ATC), little has been done Teamwork versus teams inition of teams. Thus a team in ATS can
to train these individuals to function as The increasing complexity of ATC be defined as a group of two or more per-
team members. Incidents and accidents tasks requires use of a more structured sons who interact dynamically and inter-
in which inadequate teamwork has been approach to ensure that air traffic con- dependently within assigned specific
shown to be a factor indicate that more trollers have an opportunity to develop roles, functions and responsibilities.
attention needs to be focused on this the attitudes, knowledge and skills that Team members have to adapt continu-
important area, and the adoption of the promote efficient teamwork. However, ously to each other to ensure the estab-
title "team resource management" has in the ATC context, the attempt to define lishment of a safe, orderly and expedi-
been introduced to reflect the importance a team can be rather difficult. The term tious flow of air traffic.
of teamwork to the safe and efficient con- team is used in many European coun-
duct of air traffic services (ATS) . tries to describe the control staff or Feasibility study and
This article describes the back- groups who work together in one ATC concept development
ground, experience and future plans for unit (e.g. aerodrome, approach or area In July 1994 a study group was creat-
team resource management from the control). The definitions of team range ed to investigate the possibility of a team
perspective of the European air traffic from controllers working in the same resource management (TRM) pro-
management programme (EATMP),for- sector to controllers working in different gramme in the European Civil Aviation
merly known as the European air traffic sectors but belonging to one particular Conference (ECAC). The study group
control harmonization and integration shift or watch. Controllers usually reviewed the relevant CRM/TRM publi-
programme (EATCHIP). become members of a team after licens- cations, conducted a survey of team-
EATCHIP commenced in 1990 in ing and validation and may stay in the work-related ATC incidents, and distrib-
response to commercial pressure. The same team for the rest of their careers.
objective was to find communal answers Teams are often self-organized struc- 1. Currently ECAC has 37 member States of which
28 are also Eurocontrol member States. In the next
to the development and implementation tures, with specitic rules and roles. few years, it is expected that the remaining States
of improvements to the air navigation Apart from ratings, once validated, a will also become members of Eurocontrol.
JUNE 1999
uted a questionnaire to determine the Germany, Italy, Romania, Switzerland day-to-day examples of the controller's
attitude of controllers to teamwork in and the United Kingdom. Most of these work in the form of videos, incident
ATC. A TRM training survey was also countries ran TRM prototype courses in material and discussion. The prototype
undertaken. It identified current team 1998. In the meantime, the task force set course also includes a set of evaluation
training activities inside and outside the up national working groups to promote materials which will allow a common
ECAC area. TRM and to prepare the next phases assessment of the possible changes in
The results of these studies showed within their respective States. This course content, methods and opinions,
that failures in teamwork contribute to included presentations and discussions and allows an evaluation regarding the
incidents and often have a negative with management, operational staff and changes in attitudes towards teamwork.
effect on the performance of controllers, staff associations to foster organization- ECAC States were invited to partici-
clearly indicating the need for a TRM wide awareness of TRM ideas. Some pate in the customization and testing of
programme. States had already identified controllers the TRM material. Most of the partici-
In February 1995 a TRM task force who would become TRM facilitators. pating States required external support
was established for an initial 12-month However, the most important part of this in adapting the course; some States con-
period. The principal objective of the task phase was the communal development tracted companies with experience in
force was to produce guidelines for the of a prototype TRM course. crew resource management, while oth-
development and implementation of team ers were assisted by Eurocontrol. Many
resource management. A secondary Prototype course valuable lessons were learned while
objective was to foster awareness of the From the outset the idea was to start facilitating TRM course customization
benefits of teamwork in ATS, and to pro- with the harmonization and integration and facilitator training in Austria, Portu-
duce a draft syllabus on which TRM train- procedure as early as possible and devel- gal, Ireland, Denmark and at the Euro-
ing courses could be modelled. The task op a generic prototype TRM course on control Upper Area Control Centre at
which national organiza- Maastricht, Netherlands.
tions could build their A four-step method (described below)
own tailored courses. was chosen to guarantee efficient cus-
Active management support and a carefully Harmonization in this tomization and local "ownership" of the
respect assumed that 80 TRM material, and to achieve harmo-
prepared information campaign exert a critica per cent of the course nization.
would consist of com- Human Factors is - independent of
influence on oersonneli attitude to TRM. mon material and ins- culture - often seen as a rather "fuzzy"
tructions for the facilita- subject, and it was realized that a struc-
tors, and the remaining tured approach would certainly help to
force included ECAC State representa- 20 per cent would provide sufficient scope overcome this perception. There was
tives from Austria, France, Germany, for States to be able to adapt it to their also a benefit in applying a facilitation
Switzerland, the United Kingdom, Euro- needs and include their national exam- technique: the technique would work as
pean Organisation for the Safety of Air ples, exercises and cultural influences. a role model to show that facilitation of
Navigation (Eurocontrol) Headquarters, The course was prepared by a con- complex discussions is quite possible
the Eurocontrol Institute of Air Naviga- sultant with a proven record of develop (discussions were needed to help the
tion Services, and the International Fed- ing CRM courses for airlines. A proto- participants understand the relevance of
eration of Air Traffic Controllers' Associ- type course based on the TRM the different topics). A standard method
ations (IFATCA) - a mixed team of guidelines was delivered in September would also enable a comparison of cus-
active controllers, training staff and 1997.This development was undertaken tomization in different countries to be
human factors experts. In February 1996, in close cooperation with a second TRM made. Last, but not least, time would be
the task force completed its mandate and task force and members of the national gained. Initial custornization of all eight
the TRM guidelines were published. TRM working groups to ensure a highly modules took about four days, after
After the approval of the guidelines specific ATC product. which participants needed one more
the TRM task force received a new man- The course is a three-day facilitation week to finalize the product.
date. This work was to prepare and coor- programme for operational controllers The method was applied at least one
dinate the introduction, testing and eval- and covers the areas of teamwork, team month after an initial presentation of the
uation phases of team resource roles, communication, situational aware- TRM concept and the outline of the pro-
management. Meanwhile more States ness, decision making and stress. These totype to management and operational
joined the task force, which today com- rather academic topics have been inte- controllers. In this presentation the
prises members from Austria, France, grated in a highly practical way, using importance of the formation of a national
ICAO JOURNAL
This a%cls was w r i m By a?.Wlanfrad Barbarina,
Michiel Woldring and Dr. Anne Isaac, sll of the Ajr
TeaMe Man@ement Hmwn Resource8 Unit ar the
European Orp&&x foq t h ~RafQ of /air
Navlptbn ~EurqmroU,Bruss&&.
This a@.tMeis an adaptatioh Uf pr$wntad
by Eut~mtmlt.o the 4% GiabaY Wight LWy a8d
Human b&m %$nqmdvrnat san$iilg%Chile fm 11
.tp t 5 A@J. 19%.
JUNE 1999
Airport operators and regulators need to more
completely address human factors issues
Certain aspects of aviation security have been neglected in favour of expensive technological solutions.
This imbalance can be corrected by evaluating security operations from a holistic perspective.
supervision of people is the critical of selection, the objective is to recruit

element in any security system. Part of those people who are most likely to per-
this inadequacy is related to f i e uneven form well on the job. There are several
funding levels for technology versus factors to consider including an individ-
ual's capacity for dealing with stress, a
W
ITHIN the recent past it would human factors.
seem that, in North America at Human factors issues in the field of high workload, and interaction with dif-
least, the emphasis on resour- aviation security can be more completely ferent kinds of people while using state-
ce allocation in the field of aviation secu- addressed by applying a framework that of-the-arttechnology properly. Given the
rity has been placed on the development identifies principal dimensions along high turnover rate characteristic of most
and deployment of new technologies to which to expend available resources. security operations, the selection process
address all security requirements. This This article describes a human factors receives short shrift because of the pres-
approach has been adopted despite a his- framework that involving four principal sure to meet operational requirements
torical perspective that clearly demon- axes that must be considered. on a continuous basis.
strates that ignoring human factors Among the four dimensions are the Training of personnel usually includes
issues will lead to gaping holes in the human operator, technology, organiza- two components: classroom instruction,
technological solutions, no matter how tional and operational environments, and an activity that is increasingly displaced
infallible these systems may appear. certification of personnel and technolo- by computer-based training (CBT)
A compelling example of this trend gy. Attempting to achieve a more com- which may not be equivalent to previous
was reported recently in the Air Safety plete perspective on the human factors training; and on-the-job training. On-the-
Week newsletter of 21 December 1998by issues related to aviation security job training may be quite variable and its
a former head of U.S. Federal Aviation requires activity on at least these four relation to the initial training may not
Administration (FAA) security during the dimensions. always be optimal or obvious.
mid-1980s. Since the loss of Pan Am 103 The first dimension of a human factors Yet another component is the assess-
over Lockerbie, Scotland in December fkamework for aviation security is related ment necessary to determine whether
1988,Air Safety Week reported, the FAA to the human operator mandated to use a the training provided is appropriate and
has spent more than US. $307 million on particular technology deployed for secu- of sufficient depth and breadth to maxi-
aviation security, primarily on new tech- rity purposes. The focus here is on per- mize the probability that each individual
nologies related to explosive detection sonnel selection and training. In the case operator will perform to the desired
systems. Over the same period, how- level. A related element requires the
ever, only a minuscule amount, evaluation of personnel in achieving
approximately US. $5 million, has some predetermined and objective
been spent on addressing human fac- level of performance proficiency that
tors issues in aviation security. needs to be validated in the field.
Consequently,precious research and This required level of performance
development resources have been needs to be achievable through the
concentrated within a narrow and training provided.
limited scope. It is also necessary to objectively
The former head of FAA security, evaluate the cognitive strategies used
B.H. Vincent, asserted that the by security personnel. Although all
commitment to human resources operators may reach the minimal
performance standard required,
and the upgrading of The selection, training and supervisron or securi-
systems was at an inadequate level tv * .Dersonne/ is considered a critical element in
some individuals may demonstrate
and that the selection, training and any security system. significantly higher levels of profi-
18 ICAO JOURNAL
ciency possibly as a result of a different experience with computers; sibly leading to reduced turnover, allow-
strategy. It may be possible to feed these the sampling method required of ing airport authorities to maintain high-
successful strategies into the selection operators (this is especially critical in the ly skilled, trained and motivated staff for
and training processes. development of explosive detection sys- longer periods. This, in turn, results in a
The second dimension of the human tems); and high level of security.
factors framework is related to the tech- standardizingthe ergonomic elements Related elements include team issues.
nological developments required. This of the technology. Some security personnel, for example,
dimension, too, is concerned with the ele- One example of such standardization cannot work with other team members for
ments of training, but in this instance the would be to specify that all X-ray scan- various, possibly cultural reasons, thereby
focus is on the technology required to ners have: a threat image projection sys- undermining system performance to
provide computer-based training and how tem to allow regulators and security possibly dangerous levels. It may be ben-
this can be adequately delivered without providers to objectively evaluate detec- eficial to apply team resource manage-
actually practising on the security equip- tion performance. Such a system allows ment training (an article on team resource
ment to be used on the job. Also needed the virtual inclusion of a threat image management appears on page 15).
is development of a computer-basedtest into that of the scan- An element also
to assess different CBT packages, ned baggage image. found to be critical in
Another important component is relat- This example is cited other operational con-
ed to screener assist technology. This here not because of texts is increased
technology is available primarily a perceived need to awareness of the im-
because it can be readily programmed standardize this par- portance of shift man-
into software. However, only a few ticular technology, agement, Staff should
options may actually be used by opera- but because it repre- be assigned s h i s in
tors and it is not known whether auto- sents a good exam- uch a way that the
mated detection technology actually ple of where a global a p a c t of operator
improves system performance levels. design standard would fatigue is minimized.
Any potentid improvements will be criti- allow monitoring of Other elements reflect
cally dependent on the level of automat- performance levels which standards are
A framework for addressing human
ed "hits," false alarms, nuisance alarms over time and across factors issues related to a security
in place and how they
and the normalized difference between regions. Such exten- operation is comprised of four axes. are enforced, and how
hits and false alarm rate. This is because sive monitoringwould these standards mesh
the high number of false or nuisance lead to intervention when unacceptable into the policies, procedures and pro-
alarms can lead to a lowered confidence variances occur, Standards allow com- cesses in place within a particular orga-
in the technology. ConverseIy, a high parison, and the perfect place to apply nizational environment.
number of hits can lead to a false level of standards may be within the technologi- It is also important to pay close atten-
confidence and exclusive reliance on the cal elements. tion to the elements related to the opera-
technology. Only high hit rates com- The third dimension to the human fac- tional environment in order to avoid
bined with extremely low false alarm tars framework is one that does not have undoing what may have been tremen-
rates may actually provide the necessary as high a profile as the two described dous improvements elsewhere in the
conditions for improvement in system above. It delineates elements that com- system. Not enough has been done with
performance that still requires objective prise the organizational and operational respect to this aspect of human factors to
demonstration. environments. Elements include the be able to answer all the questions that
Also of importance is the ergonomic physical aspects of the working environ- have arisen except to acknowledge that
design of technology. Ergonomic consid- ment (e.g. noise, light, dust, tempera- more action is required in this area.
erations include but are not limited to: ture). Although personnel may be The fourth dimension of humanfactors for
the displays used in presenting infor- accurately selected, properly trained and aviation security comprises elements with
mation; equipped with the latest technology, opti- respect to critically important certification
the visual properties of the image; mal performance may not be possible in
- - co%tinued S -.--
T an $age 27
p u
the controls and menus required to certain physical environments. Dr. Vinceht i s a Senior Ergonomist at the Trans-
access the necessary image or informa- Training and field performance stan- portation Development Centre, Montreal, where he is
tion, including their layout; dards also need to be evaluated, and the currently responsible for multimodal human factors
research and development for Transport Canada.
how the interface is to be developed information utilized to provide job incen- This article IS an adaptation of a presentation given
by Dr. Vincent t o the ath Global Fl~ghtSafety and
to take into account that some operators tives. Increased motivation can be Human Factors Symposium in Sahtiago. Chile from 12
using the equipment will have had no achieved through improved wages, pos- to 15 April 1999.
JUNE 1999
Shift in safety paradigm is key to future
success in reducing air accidents
Safety approaches have evolved over the years, fiom a focus on individual performance to an emphasis
on systemic flaws. Yet,fiGrther evolution in safety strategy will be needed before the aviation community
can address the challenges of the coming years.
nology cockpit design. However, a 1996 oriented performance. This means, in

JEAN P A R I ~ S human factors report by the U.S. Fed- effect, that systems govern their own
DEDALE
S.A. eral Aviation Administration (FAA) and safety. The famous "Swiss cheese" model
(FRANCE) the European Joint Aviation Authorities postulated by James Reason indicates
UAA) was an indication of what to that the organizational environment in
T
EN years ago, a review of the expect from the evolution of the airwor- which personnel work is determined by
international standards and rec- thiness certification process. management decisions and processes
ommended practices as formu- During the past decade, there has that also determine the extent to which
lated in the ICAO annexes would not been a vast flow of ideas and exchanges the work context fosters or deters errors
have yielded much focus on human fac- of experience and perspectives between and violations (Figure 1).
tors. ICAO Annex 1, which deals with academics, regulators and safety practi- As for automation, several concerns
personnel licensing, was the only annex tioners. A consensus has emerged that have emerged. One is that automation is so
that included human factors training safety depends on teams more than indi- efficient and reliable it can lead to compla-
requirements. Annex 13, which address- viduals, and that team synergy can be cency, challengingflight crews' situational
es investigation and reporting on inci- learned. It is also acknowledged that awareness and eroding basic operating
dents and accidents, had an appendix operators are not autonomous (i.e. they skills which are required when automated
that described the standard accident work within the context of a larger sys- systems fail. However, the results of an
report format, but the only human fac- tem), and that systems govern their own Airbus Industrie study indicate that the
tors element related to individual train- safety. Operators are seen as risk man- introduction of new technologies has been
ing and medical aspects. agers, culture is viewed as a possible associated with safety improvements since
A similar review today would show issue, and errors themselves do not the 1950s, and that this relationship has
that crew resource management (CRM) cause accidents (instead it is the negative continued with the latest generation of
has been introduced into Annex 6, the consequences of errors that can lead to highly automated aircraft (see Figure 2).
accident report format has been amend- accidents or incidents). Lastly, automa- The notion that errors do not cause
ed to include references to organization- tion is neither a "silver bullet" nor a devil. accidents may be more provocative. The
al aspects of accidents, the revised Individual behaviour is determined by tendency is to believe that errors cause
ADREP 2000 version of the ICAO acci- many factors including the crew selec- accidents because most accident scenar-
denthncident database includes provi- tion process, training philosophy, cockpit ios include human error. But the psy-
sions for coding CRM issues and root procedures, cockpit design and working chological process of error, which is
causes. In addition, 14 digests address- conditions. Together with corporate, pro- essentially a deviation from the opera-
ing human factors applications to various fessional and even ethnographic cul- tor's intention, often is confused with the
domains have been produced, and the tures, these factors determine individual unexpected and, in some cases, unsafe
proceedings of various regional semi- behaviour and therefore govern safety- effect of erroneous actions within a sys-
nars and global symposia have been tem. The system's resistance to or
published. tolerance of errors is a more critical
As far as aircraft design is con- safety parameter than the number
cerned, neither Annex 8 (Airworthiness and the nature of errors.
of Aircraft) nor the two worldwide A second pivotal factor is the abili-
regulation references, the European ty of the error producers, the front-
JAR 25 and the U.S. FAR 25, have line personnel, to detect and evaluate
been changed. They remain inade- Figure 1. "Systems" govern their own safety. the potential consequences of their
quate for human factors in high-tech- (based on I. Reason's model) errors in order to correct or mitigate
20 ICAO JOURNAL
begin by definfndgits safety go&, 'Ih
god m y be ta hwe hwer accidents
regardlea of grawth or it muXd be fie
ellminationof amidtats mtirels if su& a
gaal irz wen podble. The hdustry must
alm 4e:brmin~how to keep pace w3th
the growth rjf &%ternmmpl&tyi, &nd
mt End n myto make an already safe
I Normative safety
'Total Quality Management Safety
Safety results from comprehe
and procedures
Good professionals adhere to
Good procedures and good pr
make a safe world
Ecological Safety
'Safety Through Adaptahon'
Av~at~on operat~onscannot be speclfled In thelr
ent~rety(In part because they Include humans)
Devlat~onsfrom nommal operation are both a
potentla1 threat to safety and a necess~tyfor
adaptation
Human operators are (up to now) the only
~telhgentcomponent of the system
JUNE 1999
with these challenges if it persists in its and to global interactions (the interna- acddent per two or three million flights,
current approach. This is because all tional aviation system). as currently displayed by the best oper-
safety strategies have an apogee, an Large and complex systems do not ators, the potential for improvement is so
asymptotic levd of safety they are unable obey linear causality. Rather, they are the limited that the normative approach will
to surpass. For example, the imposition scene of circular causality, with strong not achieve the desired outcome.
of rules and punishment can improve feedback and feedforward effects which Negative collateral effects, especially
safety when applied to an anarchical envi- can augment or inhibit the consequences those caused by human risk manage-
ronment, but that strategy peaks when of an action. Intrinsically stable, they ment, would no longer be of second
concealed errors and violations prevent a resist forces with a huge inertia or plas- order compared to the benefits of the
ticity and effect change by normative strategy. This means that the
leverage. However, they also normative and ecological approaches
can involve interactions that must be considered complementary.
are not obviously evident and Intelligent rather than blind compli-
which may trigger divergent ance to safety regulations must be reha-
processes and ripple effects. bilitated as a key safety issue. The
The functioning and the fail- boundary between what we "need to
ure modes of larger, more know" and what is "nice-to-know'knd
interconnected systems chal- the philosophical difference between
Fi$ure 2, Accident trends for different generations of lenge traditional h e a r models a&erence and interpretation must be re-
aircraft 1959-96. (Airbus industrie data) and necessitate the use of thought. Training will have to give oper-
more sophisticated systems ators enough knowledge far them to
better understanding of failure. Doing thimking. understand what they do and why they
more of what is already being done no Secondly, there is the growing com- do it at a relevant operational level.
longer yields better results, because neg- plexity of the system model. This initially Interface and automation design will
ative collateral effects reach the same was the SHEL model in which a system have to respect naturalistic human deci-
order of magnitude, was a set of interactions between main sion-making and risk perception mecha-
A shift in the safety paradigm, the fun- components. Then there was the afore- nisms, showing the risk and the margins
damental rules and principles that are mentioned Swiss cheese model which rather than hiding them.
believed to be the definition of and the linked organizationalstructure and hier- More generally, the front-line opera-
conditions for safety, is the key to the archy with real-time safety. These were tional realities of risk management also
future. A safety paradigm is seen in the followed by a tight coupling of systems must be accounted for and properly
principles underlying accident and inci- to their environment and the effects of understood. Current preventive strate-
dent investigation, in design philosophy opaque coupling between apparently gies cannot be proactive since they are
and concepts, in training philosophies, independent components of the system. based on feedback which is appropriated
and in the role of procedures, punish- That evolution represents a shift in the by safety specialists, investigators, ana-
ment of violators and the like. metaphor of systemic models from lysts and the like - all using engineer-
Until the early 1980s, individual pilot machines or technological solutions to ing design logic. Feedback should be
errors or violations were a key issue. natural or ecological solutions. The ecol- understood as a component of, and a
The CRM era followed, stressing crew ogy metaphor suggests that since sys- contributor to, organizational learning.
synergy. The 1990s have been the systems are not totally stable, they are Rather than an external loop independ-
temic era, acknowledging the organiza- adaptable at the local, real-time level of ent from operations, it should be inte-
tional dimensions of safety. A careful individuals as well as at a global, collec- grated into daily production schemes.
review of this evohtion highlights two tive long-term level. The machine Scheduling should allow for debriefing
main directions in the way thinking has metaphor implies a normative approach - - continued
- an page 28
changed. to safety, ~ H k b ag reduction of variety Jean Paribs IS a founding member and Managing
D~rectorof D&daleSA, a consultancy that focuses on
One is that the segment of the system and a process which is inher- the human factors dimensions of safety in the aviation,
that has to be under surveillance for ently reactive. The ecology metaphor nuclear and maritme fields. Mr. Parks is a former air
acddent investigator, and has been a member of the
safety reasons has continuously expand- imp1ies a and learning ICAO Flight Safety and Human Factors Study Group
ed. mom individual interactions, it has t0 safety. since its ~nceptionin 1988, He holds a commercial pilot
l~censewith instrument, multi-engine, turboprop and
evolved through local interactions (one The normative has been the instructor ratings, and a helicopter private pilot license.
team, including cockpit crew, then cabin keystone of safety improvements for at This article is an adaptation of a paper presented
by Mr. P a r k to the 4th Global Flight Safety and
crew, ground staff and air traffic control) least the past O' years but it is nearing Human Factors Symposium held in Santrago, Chile
to organizational interactions (an airline) its apogee. Beyond a safety level of one from 12 to 15 ~pril1999.
ICAO JOURNAL
Human factors programme introduced in parallel
with CNSIATM systems implementation
Chile has moved decisively to establish a plan for applying human factors knowledge throughout its
aviation system. A key impetus has been a national project to introduce satellite-based technologies.
by the analysis took the form of a resist- cepts. For this purpose, a structured
ance to change from management who course has been acquired with simple
were concerned about the scale of equip- training aids which can be adapted to all
ment investment required by the new types of audiences.
technology and the necessity to divert No concept can be implemented effec-
u
resources for human factors training. tively without solid support from senior
NTIL fairly recently, human fac- Another challenge identified by the management, and for this reason the
tors was a discipline left to a few human factors analysis was the capacity of DGCA has conducted an awareness pro-
initiates who, although they had the world aviation community to produce gramme at the executive level of the
some knowledge of it, could not really be seamless CNS/ATM systems given the organization.
called experts. This was particularly evi- socio-economic, cultural and development The Escuela Tecnica Aeronautica (ETA),
dent as the concept of crew resource differencesthat exist among countries. the DGCA's institute for training opera-
management (CRM) evolved. Chile's analysis indicated that the tional professionals and technicians, has
With CRM, air carriers began to move greatest difficulty in implementing a inaugurated a 42-hour human factors
ahead with their own training progam- human factors programme arose from course in the final semester of its pro-
mes and, soon after,the subject came under the yap between theoretical knowledge gramme. This is supplemented by CRM
greater scrutiny as ICAO developed hu- and the practical application of that training using an air traffic control
man factors standards and recommend- knowledge. Clearly, it is hard to account (ATC) simulator.
ed practices that States could incorporate for human factors when introducing new A human factors course for non-oper-
into their regulaFory regimes. ICAO also technology, and Chile's Directorate ational personnel will begin in 2000 and
began to disseminate valuable informa- General of Civil Aviation (DGCA) has is scheduled to continue for five years.
tion in the form of a series of digests on designed an aggressive plan to ensure This course is designed to reach person-
different aspects of human factors. integration. nel who are neither senior management
The principal challenge for many The objective of the plan is to incorpo- or ETA graduates. A current trial pro-
administrations has been how to define rate human factors concepts as a manage- gramme provides practice and gives
the perceived need without a sound ment and flight safety tool throughout instructors a better command of the sub-
knowledge base. In Chile, this challenge the aviation system by 2010. Several ject. Also, a two-year plan has been
has been met in part by developing a DGCA personnel have been assigned for developed with the objective of inform-
matrix analysis that focused on threats training in the operation of new tech- ing all DGCA clerical officers about
to safety, opportunities, weaknesses and nologies. To be provided by the U.S. human factors.
strengths. This analysis has been used Federal Aviation Administration (FAA), A sub-programme is targeted at
to develop a human factors programme the training focuses on flight inspection, instructors certified by the DGCA to
in parallel with the country's implemen- design of instrument approaches based work in the private sector with airlines,
tation of communications, navigation, on the global navigation satellite system flying clubs and schools which provide
surveillance and air traffic management (GNSS), and wide area augmentation training for cabin crew, mechanics, dis-
(CNS/ATM) systems using satellite tech- system (WAAS) engineering and opera- patchers and other aviation personnel.
nology. The human factors programme tions. In addition, a CNS/ATM systems continued on page 28
has been kept as simple as possible - the manual for use by air traffic controllers Mr. Makr~novIS Ch~efo f the DGCA Flight Safety
goal being a paradigm shift which would and flight crews has been developed. Department and a Colonel (Ret.) In the Ch~leanAir
Force, where he has served i n a number o f safety-relat-
encourage acceptance of the human fac- The goal is to set up a cadre of highly ed posts.
tors discipline at all regulatory and oper- motivated professionals who will act Th~sart~cleIS an adaptation o f a paper presented
by Mr. Makrlnov at the 4th Global F l ~ g h Safety
t and
ational levels of Chile's aviation system. more as facilitators than as instructors in Human Factors Syrnpos~umat Sant~ago,Ch~lefrom 12
The main obstacle to safety identified disseminating awareness of the new con- t o 15 Aprd 1999
JUNE 1999
Human error a warning flag for regularors and managers,
Council President tells safety conference
Despite an impressive safety record, with a relatively low fatal the resources, investigate failures of the system and take
accident rate that has remained practically unchanged for remedial action, Dr. Kotaite explained.
over two decades, a further reduction in the rate of fatal air- "Human error should become a warning flag for regulators
craft accidents is possible as the industry moves into the and managers, a possible symptom that individual workers
21st century, ICAO Council President Dr. Assad Kotaite have been unable to achieve the system goals because of
informed participants at a safety conference in London in difficult working environments, flaws in policies and proce-
early June. dures, inadequate allocation of resources, or other deficien-
Innovative technology as well as human factors endeav- cies in the architecture of the system," Dr. Kotaite informed
ours have already contributed significantly to the aviation the participants - representatives of numerous govern-
safety record, but the most important safety strategy today ments, manufacturers and airlines.
centres on the relationship between management and safety, The importance of the role of management is reflected in
Dr. Kotaite said in opening the conference, organized by the current safety efforts, with three of the six key activities which
Royal Aeronautical Society, on 3 June. comprise the ICAO global aviation safety plan (GASP) orient-
The Council President stressed the role of management in ed on management.
enhancing safety. "Simply put," he stated, "managers play a The universal safety oversight audit programme, the flight
fundamental role in defining and sustaining the safety culture safety and human factors programme, and the controlled
of their organizations." flight into terrain (CFIT) programme, the Council President
Regulators and airline management define the environ- indicated, all emphasize organizational and managerial solu-
ment within which operational personnel conduct their tasks: tions to safety issues (see also "ICAO providing tools and
it is they who determine the policies and procedures, allocate leadership needed for enhancing safety worldwide", page 4).
The CFlT programme, Dr. Kotaite observed by way of
example, "moves away from the simplistic notion that CFlT
accidents are simply caused by substandard human perfor-
mance . . . CFlT accidents are truly systemic accidents." For
that reason, a substantial part of the ICAO message con-
cerning CFlT is directed at decision makers in the aviation
system.
The Council President concluded his address by observing
that current and future safety challenges will require innova-
tive approaches, not versions of old solutions that were suc-
cessful in the past. He noted that a rapidly changing world
demands adaptation.
"Human error is nothing new," Dr. Kotaite pointed out, "but
the massive introduction of advanced technology in contem-
porary aviation and its resulting interface with people brings
new challenges . . . requiring new solutions." 0
Y2K initiatives have already achieved

substantial progress, UN meeting told
ICAO presented a status report on Y2K readiness at the Unit-
ed Nations on 23 June 1999 which reaffirmed that safety
remains the top priority of the world aviation community.
The ICAO report, which also highlighted the Y2K initiatives
of the International Air Transport Association (IATA) and the
Airports Council International (ACI), pointed to substantial
progress in all critical areas of air transport and in the devel-
opment of contingency plans. Contracting States of ICAO are
responsible for the safety of their airspace, airline and airport
operations, and the provision of air navigation services.
Aircraft manufacturers and airlines have already indicated
that aircraft will continue to operate normally through the year
2000 date change and during other date changes which are
ICAO JOURNAL
considered critical to computer systems. While airports have The 265 member airlines of IATA have collectively spent an
developed Y2K contingency plans, travellers may experience estimated U.S. $2.3 billion to resolve Y2K problems. IATA has
some inconveniences during the Y2K date change as air- pursued an active campaign to increase industry awareness
ports in most cases are dependent on community services and monitor the progress of Y2K programmes at 2,000 air-
and utilities to ensure smooth operations. ports and over 180 air traffic service providers. IATA field
Another area of concerted activity is the provision of air work involves close cooperation with ICAO on regional con-
navigation services, which use a significant number of essen- tingency planning efforts, and with ACI in promoting local
tial computer-based systems. coordination between airports and airlines.
The ICAO Y2K action plan focuses on disseminating infor- ACI has provided its 1,350 member airports with specific
mation about the Y2K problem, raising the level of awareness assessment tools, such as a standard action checklist and a
within the international civil aviation industry, assessing the list of systems which may require scrutiny, as well as a refer-
progress of States in addressing the Y2K problem, and sup- ence definition of Y2K compliance.
porting States' efforts as well as those of air transport organi- During the remainder of 1999 and the early months of 2000,
zations. The plan encourages the development of national con- ICAO, IATA and ACI will continue to coordinate their respective
tingency plans while working through planning groups toward Y2K programmes in information gathering and dissemination,
the development of contingency plans at the regional level. site visits, problem solving and contingency planning, the UN
A report on the state of Y2K readiness will be available to meeting of national Y2K coordinators was informed. 0
ICAO Contracting States in August, when contingency plans
are expected to be finalized. The status report will be based
on information ICAO had received from States by 1 July. In
the meantime, information on Y2K compliance is being dis-
seminated to States as soon as it becomes available so that
national and regional contingency plans and other Y2K activ-
ities can be updated. ICAO will operate a global coordination
unit during the Y2K transition period, which will assist with
implementation of regional plans.
Year 2000 programmes at IATA and ACI, implemented in
cooperation with ICAO, have ensured the widest possible
scope of activity by the international aviation industry.
JUNE 1999
ICAO Council appointment A second source of information comes from the use of a ques-
Victor P. Kuranov has been appointed tionnaire concerned with attitudes and behaviours, which can be
Representative of the Russian Federa- administered before and after the TRM course. Often a more
tion on the Council of ICAO. His robust method of evaluating the changes in these attitudes and
appointment took effect on 14 May behaviours is to administer a third questionnaire, identical to the
1999. second, some four to six months after the TRM course.
A Doctor of Technical Sciences spe- A third and more rigorous evaluation comes from the corre-
cializing in navigation and air traffic lation of these attitudinal changes with observation or interviews
V k t o r E ! KuranOv management, Mr. Kuranov has, over the
(Russian Federation) of the same personnel to gauge meaningful behaviourial
course of his career, received training at
changes. From this methodology, measurable positive changes
a number of technical institutes in his
in interaction should be present following the course. Lastly, the
country, including graduate training at the State Civil Aviation
Scientific Research lnstitute in Moscow. His professional ultimate validation should be found in a correlation between the
background includes several years of service with the "Aeron- TRM programme and a decrease in the frequency of incidents
avigatsiya" State Scientific Research lnstitute in Moscow, within the system. The latter two methodologies are highly com-
most recently as its Executive Director. plex and take considerable time to apply. It is for this reason that
Mr. Kuranov has participated in the development of vari- the evaluation phase of the Eurocontrol TRM programme used
ous technical regulatory documents governing the former only the course evaluation and the monitoring of attitudinal and
Union of Soviet Socialist Republics and the Russian Federa- behaviourial changes as an assessment of its effectiveness.
tion, as well as in negotiations with other States related to the Safety questionnaire. The development of the air traffic con-
international civil aviation activities of his country. He has trol safety questionnaire (ATCSQ) was based on the work under-
also participated regularly in sessions of the ICAO Assembly
taken previously in developing flight crew resource manage-
and in the work of various ICAO technical bodies, including
ment and, in the medical field, operation room management. The
the ICAO Special Committee on Future Air Navigation Sys-
tems (FANS Phases I and II), the CNSIATM Implementation
TRM programme clearly defines seven areas of concern in its
Advisory Group (ALLPIRG) and the Global Navigation Satel- prototype course. Within these areas, items were identified and
lite Systems Panel (GNSSP). 0 became the basis of the ATCSQ.
The questionnaire consists of four main sections. The first
section concerns attitudes towards the quality of training, work-
Team resource management ing conditions and documentation, and the last section concerns
continued fiom page 17 demographic information.The second and third sections contain
is culturally dependent. Some ATC organizations have designed the main evaluative information and are concerned with atti-
specific courses for different ATC functions. tudes and the responses associated with behaviours.
Evaluation process. The objective of team resource man- The results of the evaluation of this questionnaire show that
agement is the use of all available resources (people, informa- the safety questionnaire is a robust, reliable instrument and is
tion and equipment), in order to achieve the safe and efficient informative for the purpose for which it was designed. A few
movement of air traffic. This objective is obviously ambitious questions indicated some bias and will therefore be reworded for
and, as in the case of its flight deck counterpart - crew future use. In terms of the results concerning the attitudes
resource management - there will be difficulties in the mea- towards the professional training and working environment, the
surement of its effectiveness. However, to ignore the challenge questionnaire clearly indicated acceptable satisfaction with most
of this evaluation would be foolish. of the areas of the TRM course, but some aspects concerned
Programme evaluation should not only provide information with operations and safety manuals, handling of emergency traf-
on the effects of the TRM course but should also provide direc- fic and feedback in daily operations, were not as positive.
tion for further initiatives. The most basic type of information The results were based on a small response sample (30 sub-
comes from participants' evaluations, usually collected by ques- jects) and could give a rather biased picture of the impact of the
tionnaire at the end of the TRM course. Positive reactions to the TRM course. The results did, nevertheless, illustrate a signifi-
course provide necessary but not sufficient evidence of impact. cant change within some of the subject domains. Most of these
While a positive reaction to the course is not sufficient in itself changes were in the anticipated direction; that is, the course
to indicate a positive s h i i in attitudes, a negative reaction to the had changed the attitudes of participants in favour of better and
course is an almost certain indication that positive behaviour more cooperative teamwork and more sympathetic team roles.
change is not going to occur. Overall the results showed that the questionnaire is sensitive
to changes in attitude and, with a larger sample and strict adher-
ence to data gathering, a more meaningful database could be
established in the future. This will not only strengthen the use of
such a questionnaire but will help individual countries to cus-
tomize their individual needs for TRM applications.
The outlook. After the assessment and evaluation of the testing
phase, the task force will issue recommendationsfor further irnple-
mentation of the TRM concept. This will include advanced imple-
ICAO JOURNAL
mentation plans for team resource management in all 37 ECAC
States and its incorporation into the selection, training (including
upgrade and refresher training) and licensing of ATC staff.
The initial implementation of the TRM concept is likely to go
through an evolutionary process, just as CRM did when it was
introduced. The target population will be extended to other
operational staff in air traffic services and air traffic manage-
ment. Other areas, such as incident and accident investigation,
also need to consider teamwork-related aspects in their evalua-
tions and reports.
The success of the development and introduction of automa-
tion in ATM will partly depend on enhancing and supporting the
teamwork culture, both in the control centre and on the flight
deck. The day when controllers and pilots will be trained in com-
bined CRM/TRM courses may not be far off. 0

attempts to discover ways to make this system even safer. The
industry is accepting that challenge, both worldwide and
regionally.
Addressing the most serious safety problems - CFIT,
approach-and-landing and loss-of-controlaccidents - can pro- Human factors related to these elements need to be consid-
vide high leverage for accident prevention, and there are many ered along the certification axis because all of the axes need to
interventions available for use today, including those focusing provide for a seamless stream that contributes to maximizing
on human factors. security levels while keeping the human operator in the fore-
Although safety practitioners represent a very small portion front of any actions.
of the worldwide flight operations community, each individual The accompanying figure (see page 19) is a representation
can play a large role within his or her airline and geographic of the four axes that form a framework for addressing human
region. It's up to each to do his part and to carefully consider factors issues in the field of aviation security. Two of these axes,
and implement the appropriate safety interventions if the indus-
concerned with the human operator and with technology, have
try is to positively contribute to the worldwide safety story.
a user focus. It is here that almost all of the resources have
been expended. The other two axes, concentrating on the orga-
nizational and operational environments and on certification,
Security operations
have a mandate focus. These dimensions seem to have been
continuedfrompage 19
issues. It is necessary to take into account the process by which neglected with respect to human factors.
individuals and organizations are certified. What standards are The human operator and the organizational and operational
required, and are they in place? A third element concerns re- environment are both realitybased dimensions, while technology
certification.What period is optimal, and when is it necessary to and certification are objective-based.The first two dimensions
d e c e r an~ individual or organization? concern personnel selection, training, turnover, motivation and
I * Airline Pilot Qualification Program Certification after 10 months CPL/IR

More than 60% of Crossair pilots have trained at HORIZON = IS0 9001 certified Please contact: . I
-
-- -
-
-
HOTSwissO
FlightN
Academy Ltd
~ckerstrasse4, CH-8180 Bulach/Zurich
phone i l l - l B 6 2 0'7 07, fax 1 - 1 6 2 02 1
http://www.horizon-sfa.ch, info@horizon-sfa.ch E
Z
$ I
Where flying c a r e e r s t a k e off
JUNE 1999
wages, while the latter two are concerned with the objectives
that technology needs to meet and the certification of such
Safety strategy
technology and operators. It would be problematic to ignore as well as individual and collective communication, and debrief-
any of these human factors aspects. ing tools and practices should be developed.
Summary and conclusion. Human factors has been rec- It has been said that safety is not a static victory, but a
ognized for several decades as a primary factor in aircraft cock- dynamic control of a permanent hazard. From this perspective,
pit design. Recently, awareness of the importance of human normative safety reaches its apogee when large, complex and
factors in all areas where humans interact with technology has tightly coupled systems are involved. Procedures and regula-
grown. This is certainly the case in aviation security, where the tions are needed, but they are not the ultimate solution as they
deployment of increasingly sophisticated technology heightens will not allow the aviation community to meet the challenges
the demands on the human operator. of the coming years. As long as humans are a part of the sys-
Demands on security personnel may have a negative impact tem, they are both a source of potential hazard and a form of
on the performance of the system (i.e. human operator plus protection.
technology). The objective of the system is for the human Human knowledge and adaptability are critical. And because
observer to interact with the technology and render a correct perception and intelligence are involved, causation processes
decision in, for example, accepting or rejecting luggage for cannot be a linear chain of independent links.
loading on board aircraft. Progress towards an even safer system implies organiza-
A framework that allows systematic evaluation of human fac- tional learning. Also, ironically, it means that errors, deviations,
tors research and interventions could contribute to minimizing incidents and accidents must be considered as a fundamental
system error and optimizing system performance. The frame- source of knowledge in the drive for increased safety. This
work is designed to allow airport operators and regulators to could well be the challenge of the next 10 years. 17
focus their efforts on human factors issues that may negatively
affect the performance of the aviation security system.
It is to our detriment if we focus our resources along only Human factors programme
one or two axes of activity. The application of a human factors continued from Page 23
perspective has revealed at least four dimensions where we The DGCA plans to require that candidates for license
need to apply resources in order to improve and maximize the renewal demonstrate a knowledge of human factors. This
level of security. It may be that the relatively neglected axes requirement will take effect in 2000 and continue until 2008, by
may provide for greater efficiencies in performance. Although which time all licenses then in force will have been subject to
the framework described here was developed with aviation renewal at least twice (the longest term of validity of a certifi-
security in mind, the various elements may be of potential use in cate held by technical aeronautical personnel is four years).
evaluating other areas that require human factors initiatives. The objective is to promote human factors awareness through-
out the DGCA and the industry it serves and supervises.
The human factors programme introduced in Chile is close-
ly tied to the plan to introduce the CNS/ATM systems. Chile's
CNS/ATM project has been progressing steadily since
December 1996, when the DGCA and the U.S. Trade and
Development Agency agreed to develop a feasibility study and
define a CNS/ATM systems architecture for Chile. Some of
these elements are now being incorporated into a national
implementation plan, which is in its final revision stage.
With expertise provided by the U.S. FAA, Chile has been
developing an air navigation system based on the global posi-
tioning system (GPS), a component of GNSS. In March 1998,
the two agencies agreed to facilitate training of DGCA personnel
in subjects related to satellite-based navigation. The FAA also
supplied three WAAS reference stations to the DGCA (they are
installed at Santiago, Balmaceda and Antofagasta). In
December 1998, it provided a Boeing 727 for landing demon-
strations at Santiago International Airport. The B-727 complet-
ed a series of Category I approaches using signals generated
by the WAAS stations and corrected by the FAA Technical
Center in Atlantic City. While the acceptable standard of accu-
racy is 7.6 metres, the Santiago trial revealed an accuracy as
precise as three metres.
To better evaluate WAAS technology, the DGCA has pur-
ICAO JOURNAL
chased two reference stations to supplement the three on loan.
The five stations are currently being used to transmit signals
through the Inmarsat I11 geostationary satellite, reducing GPS-
induced error to 7.6 metres (from 300 m).
1 POSTS VACANT
The DGCA has also acquired software which will allow Accounting Officer, Accounting Services Section, Finance
users to predict satellite availability along their routes, pro- Branch, Office of the Secretary General, Headquarters
viding better flight planning. To be installed at 23 facilities - Montreal. Ref. PC 99/21/P-4.
from Arica, near the northern border with Peru, all the way An advanced level post at Headquarters, Montreal. Essential
to Antarctica - the software enables satellite data to be pre- qualifications and experience: University degree or equivalent
sented to flight crews in a graphic format that can be under- academic qualifications in financial management, preferably
stood readily. with professional certification as Chartered Accountant or
Importantly, all these advances in introducing new technol- Certified Public Accountant; at least ten years' financial or
ogy in Chile's aviation system have been made in step with the accounting experience some of which with a national govern-
development of an effective human factors programme. While ment or international organization; experience in the utilization
of modern financial software packages; ability to develop and
the Chilean model may not be ideal for all countries, it strives to
maintain computerized financial information systems; experi-
adapt human factors to Latin American idiosyncracies and may
ence in the implementation of financial management/account-
be of interest to other administrations which need to incorpo- ing software in a client/server environment; experience in
rate similar training in their aviation systems. 17 preparing financial reports and statements; ability to analyse
and suggest solutions to accounting and financial problems;
experience in managing, planning and coordinating the work of
CRM study subordinate staff; ability to prepare clear, concise and accurate
continued fiom page 10 reports and correspondence; initiative, judgement, thorough-
changes in the nature and scope of technical training, schedul- ness and the ability to maintain harmonious working relation-
ing changes, revised rostering practices, and the establishment ships in a multinational environment. Desirable qualifications
or enhancement of a safety department. and experience: Knowledge of UN or ICAO functions and its
There are basic steps that every organization needs to follow organization.
to establish a proactive safety culture that is guided by the best Command of one of the following languages of the
Organization (Arabic, Chinese, English, French, Russian,
possible operational data. These include:
Spanish) is essential and a good working knowledge of one of
a non-punitive policy toward error;
the others is desirable.
commitment to taking action to reduce error- inducing con-
ditions;
data that show the nature and types of errors occurring;
training in error avoidance and management strategies for
crews;
training in evaluating and reinforcing threat recognition and
error; Aviation Organization is a specialized agency of the Unitec
management for instructors and evaluators; and Nations. Headquartered in Montreal, ICAO develops inter
national air transport standards and regulations and serve
fostering of trust. as the medium for cooperation in all fields of civil aviatio~
Trust is a critical element of a safety culture, since it is the among its 185 Contracting States.
lubricant that enables free communication. It is gained by
demonstrating a non-punitive attitude toward error (but not
deliberate violations) and showing in practice that safety con-
cerns are addressed. Data collection to support the safety cul-
ture must be ongoing and findings must be widely
disseminated. (One airline had its LOSA report bound, and
placed copies in every aircraft as well as every base for crews
Web S~te
to peruse.) http.//www lcao ~n
CRM training must make clear the penultimate goals of Sitatex: PAREUYA Facs~mile:011-5115-750974
threat recognition and error management. Ancillary benefits, REGIONAL OFF1 Cable ICAOREP, Pans
Facsrmle: 011-33-1-46418500
such as improved teamwork and morale, are important but not Asla and Pactfic 0 Telephone 011-33-1-46418585
Bangkok, Thatland
the driving force. Finally, instructors and check airmen need S~tatexBKKCAYA
special training in both evaluating and reinforcing the concepts Cable ICAOREP, Bangkok
Telex TH87969 ICAOBKK TH
and in relating them to specific behaviours. Facsmle 011-662-537-8199
Telephone 011-662-537-8189
If all of the needed steps are followed and management's
credibility is established, a true safety culture will emerge and
the contribution of CRM to safety will be recognized. 17
JUNE 1999
PROCEEDINGS OF THE FIRST LOSA WEEK
CATHAY CITY
HONG KONG
12 TO 14 MARCH 2001
Table of contents
Foreword
Objectives and agenda
Introduction
The Line Operations Safety Audit (LOSA)
Presentations
An introduction to LOSA
LOSA and Safety Culture
The phases of LOSA
Making the safety process better
The Air New Zealand 1998 LOSA
LOSA: a natural extension?
Short Course Observer Training

LOSA observation forms
Threat management worksheet
Error management worksheet
Crew interview document
LOSA error codebook
Threat and error management exercises
Threat and error management training
Conclusions
List of participants
FOREWORD
Safety of the civil aviation system is the major objective of the International Civil Aviation
Organization. Considerable progress has been made, but additional improvements are needed and
can be achieved. It has long been known that less than optimum human performance underlies the
vast majority of aviation accidents and incidents, indicating that any advance in this field can be
expected to have a significant impact on the improvement of aviation safety.
This was recognized by the ICAO Assembly, which in 1986 adopted Resolution A26-9 on Flight
Safety and Human Factors. As a follow-up to the Assembly Resolution, the Air Navigation
Commission formulated the following objective for the task:
To improve safety in aviation by making States more aware and responsive to the importance
of human factors in civil aviation operations through the provision of practical human factors
material and measures developed on the basis of experience in States, and by developing and
recommending appropriate amendments to existing materials in Annexes and other documents
with regard to the role of human factors in the present and future operational environments.
Special emphasis will be directed to the Human Factors issues that may influence the design,
transition and in-service use of the [future] ICAO CNS/ATM systems.
In November 1999, the Air Navigation Commission approved a second-phase Plan of Action on
the ICAOs Flight Safety and Human Factors Programme. The collection and analysis of Human
Factors safety data was discussed under the objective of continued integration of Human Factors
considerations into the aviation environment. Human Factors data accessed through existing
reporting schemes originate in accident/serious incident investigation reports, or incident
reporting systems. Such data record unsuccessful human performance (i.e., unmitigated
operational errors). To develop countermeasures to human error it is essential to expand the field
of observation, and access Human Factors data from normal operations. Examples of industry
attempts to capture such data are self-reporting schemes such as the Aviation Safety Action
Programme (ASAP), and flight data acquisition and analysis programmes such as the Flight
Operations Quality Assurance (FOQA) Programme. The Line Operations Safety Audit (LOSA)
was considered a natural extension of these attempts, since it permits recording successful human
performance (i.e., mitigated operational errors), and it leads to more complete conclusions to
develop countermeasures to human error.
LOSA started as a research project funded by FAA and developed by the University of Texas
Human Factors Research Project. It was first presented at the ICAO Fourth Global Flight Safety
and Human Factors Symposium held in Chile, in April 1999. Following the Symposium, ICAO
acquired an in-depth knowledge of the project, and this led to the development of a link between
the ICAO Flight Safety & Human Factors Programme and LOSA. A letter from the Secretary
General of ICAO was sent to the FAA Administrator in February 2000, informing FAA of
ICAOs commitment and support to LOSA, thus formalizing the link. The objective in linking the
FS&HF Programme and LOSA was to come up with an operational deliverable for States and
airlines, within a reasonable period of time.
Progress has been faster than expected, and the accelerated acceptance of LOSA observed during
2000 would suggest a short-term exponential growth of LOSA within the airline community. In
addition to a multiple-fold increase in the number of airlines practising LOSA, an ICAO LOSA
Manual was completed in November 2000, and is expected to be ready for publishing in
November 2001. The manual is a comprehensive source of information for States and airlines. It
was written observing a users-guide style, and it provides all the necessary background as well
as information about the basic tools in LOSA.
If the growth of LOSA materialises, two alternatives are being considered as milestones in the
continuation of the project: either a summit conference at ICAO Headquarters during 2002,
with a view to discuss broad-scale implementation issues; or dedicating the Fifth Global Flight
Safety and Human Factors Symposium, planned for 2003, as a LOSA implementation meeting.
The long-term objective is to give LOSA away, by providing States and airlines with the
necessary tools to implement and manage LOSA or a LOSA-like data collection system on their
own. LOSA ties with widely-implemented electronic data acquisition systems, as well as incident
reporting systems, so it should naturally flow into existing schemes within the airline industry.
With this in mind, ICAO organized, with the generous sponsorship of Cathay Pacific and with the
cooperation of the University of Texas Human Factors Research Project and Continental Airlines,
a three-day meeting on the Line Operational Safety Audit (LOSA) in Hong Kong, on 12-14
March 2001. The meeting was held at the Cathay Pacific complex in Hong Kong, known as
Cathay City. The first day included an introduction to LOSA, followed by a full-day LOSA
observer training. Participants had the opportunity to observe an abbreviated, short training
course, and thus form a complete picture of LOSA. The last day included Continentals threat and
error management training, followed by a strategy session.
Fifty-three participants representing fourteen airlines within the Asia Pacific region, the Hong
Kong Civil Aviation Department, the Civil Aviation Safety Authority (Australia), the Australian
Defence Force, Airbus, Boeing, the Association of Asia Pacific Airlines and IFALPA attended
the meeting.
The event was opened by Captain Henry Craig, Manger Training Boeing, Cathay Pacific, in
representation of Captain Rick Fry, General Manager Flying. The working language of the
meeting was English.
_______________
Objectives and Agenda
To provide a forum to introduce LOSA to the broadest possible airline audience within the
Asia/Pacific region;
To conduct LOSA observer training specific for Cathay, while allowing participant airlines to
observe Cathay-specific training to grasp a better understanding of LOSA;
To provide the basics of threat and error management training;
To discuss possibilities, constraints and resources required for the implementation of LOSA
at the airline-specific level;
To discuss the potential cooperation among airlines within the Asia/Pacific region in sharing
resources and technology as needed to implement LOSA; and
To discuss the potential contribution of non-airline organizations, including civil aviation

authorities, aircraft manufacturers and professional associations, to the worldwide
implementation of LOSA.
Monday, 12 March
1000 - 1200 Coordination meeting

LOSA Week Steering Committee
1300 - 1630 Welcoming remarks

Cathay Pacific
An introduction to LOSA
D. Maurino - ICAO
R. Helmreich - UT
J. Klinect - UT
B. Tesmer - Continental
C. Kriechbaum - Air New Zealand
S. Henderson - Ansett
1630 - 1700 Q&A session
Tuesday, 13 March
0900 - 1700 LOSA observer training sessions
J. Klinect - UT
Wednesday, 14 March
0900 - 1300 Threat and error management training

D. Gunther - Continental
1400 - 1700 LOSA implementation strategy discussion

All participants
1700 Conclusions and wrap up

LOSA Week Steering Committee
_______________
INTRODUCTION training interventions can be re-shaped and
reinforced based on successful performance,
This report introduces a programme for the that is to say, positive feedback. This is
management of human error in aviation indeed a first in aviation, since the industry
operations. The programme, known as has traditionally collected information on
LOSA, or Line Operations Safety Audit, is failed human performance, such as in
proposed as a critical organizational strategy accidents and incidents. Data collected
aimed developing countermeasures to through LOSA is proactive and can be
operational errors. LOSA is an immediately used to prevent adverse events.
organizational tool to identify threats to
LOSA is a mature concept, but its
safety, minimize the risks such threats may
implementation is still in a developing stage.
generate, and implement measures to
Although initially aimed at flight deck
manage human error in operational contexts.
operations because of a question of priorities
LOSA allows airlines to diagnose their
vis--vis limitations in resources, there is no
levels of resilience to systemic threats,
reason why the methodology could not be
operational risks and front-line personnel
applied to other aviation operational sectors,
errors, thus providing a principled, data-
including air traffic control, maintenance,
driven approach to prioritize and implement
cabin crew and dispatch. The initial research
actions to enhance safety.
and project definition was a joint endeavour
between the University of Texas at
LOSA is a programme that uses expert and Austin/Human Factors Research Project and
highly trained observers to collect data about Continental Airlines, with funding provided
flight crew behaviour and situational factors by the Federal Aviation Administration
on normal flights. The audits are conducted (FAA). In 1999, ICAO endorsed LOSA as
under strict non-jeopardy condition; the primary tool to develop countermeasures
therefore, flight crews are not at risk for to human error in aviation operations,
observed actions and errors. The observers developed an operational partnership with
record and code potential threats to safety the University of Texas and Continental
observed and how the threats are addressed Airlines, and made LOSA the central focus
during the flight. They record and code the of its Flight Safety and Human Factors
errors such threats generate, and how flight Programme for the period 2000-2004.
crews manage these errors. Lastly, observers
As of January 2000, the funded research
record and code specific behaviours that
project had conducted eight audits with a
have been known to be associated with
total of nearly 4,000 flights observed. These
accidents and incidents.
audits were conducted both within the
LOSA is linked to with Crew Resource United States and in international operations
Management (CRM) training. Since CRM is and involved two U.S. and one non-U.S.
essentially error management training for operator. The number of operators joining
operational personnel, data from LOSA LOSA has since January 2000 constantly
forms the basis for contemporary CRM increased and continues to increase, and
training re-focus and design. Data from includes major international operators from
LOSA also provides a real-time picture of different parts of the world and diverse
system operations that can guide cultures.
organizational strategies in regard to safety,
The role of ICAO in the LOSA programme
training and operations. A particular strength
is that of an enabling partner. ICAO will
of LOSA is that it identifies examples of
increase awareness of the international civil
superior performance that can be reinforced
aviation about the importance of LOSA, will
and used as models for training. In this way,
facilitate research in order to collect data as
necessary, and will act as cultural mediator
in the unavoidably sensitive aspects of data
collection, by aiming to integrate multi-
cultural observation teams. In line with these
objectives, this report is a first attempt at the
exchange information, and therefore to
increase awareness within international civil
aviation about LOSA. The report is a
generic introduction to the concept of
LOSA, the tools employed, the methodology
underlying the programme, and the potential
remedial actions to be undertaken based on
the data collected. A very important caveat
must be introduced at this point: the report is
not intended to convert readers in instant
expert observers and/or LOSA auditors. In
fact, it is strongly warned that LOSA should
not be attempted without formal training of
observers on the methodology, for the
following reasons.
Firstly, the forms presented in the report are

for practical illustration purposes
exclusively, since they are periodically
amended as a function of experience and
feedback from continuing audits. Secondly,
formal training in the methodology, in the
use of tools, and most important, in the
handling of the highly sensitive data
collected by the audits, is absolutely
essential. Thirdly, the proper structuring of
the data obtained from the audits is of
paramount importance. While LOSA
remains in its development stage, it is highly
desirable that LOSA training be coordinated
through the programmes founding partners.
As the methodology evolves and reaches
maturity, and broader industry partnerships
are developed, LOSA will be available
without restrictions to the total of the
international civil aviation community.
______________
THE LINE OPERATIONS SAFETY AUDIT (LOSA)
Robert L. Helmreich1, James R. Klinect, John A. Wilhelm & J. Bryan Sexton

University of Texas Human Factors Research Project
Department of Psychology
The University of Texas at Austin
Austin Texas
ABSTRACT 2. Incident reports. These are useful because

they provide insights into a far larger database of
The sources of data available to monitor system saves and near misses. They suffer, however,
operations and the safety culture in an airline are because of their voluntary nature and the resultant
described. In particular, the type of information fact that the actual baseline of occurence of
provided by the Line Operations Safety Audit (LOSA) various categories of events is unknown. Despite
methodology developed by the University of Texas efforts to assure pilots and other aviation
Human Factors Research Project is described. A model personnel of the non-jeopardy nature of reports
of Threat and Error Management derived from LOSA under initiatives such as the Aviation Safety
data is reviewed. The historical development of LOSA Action Programs (ASAP: FAA 2000), these
and its uses are presented along with representative programs do not elicit complete reporting. This is
data. The proactive uses of LOSA data for research, both because of embarassment at acknowledging
organizational safety initiatives, and curriculum error and because of suspicion about being
development are described. sanctioned. Nevertheless, programs such as ASAP
do provide invaluable information and allow
UNDERSTANDING SYSTEM SAFETY organizations to take needed safety action prior to
serious accidents and incidents.
An understanding of flight safety can only be gained
from valid, empirical data about normal operations. 3. Line Checks. Although required by civil
There are several sources of such data, each aviation regulators in most countries, line checks
incomplete. However, in combination they can provide generally lack in diagnosticity, espcially in the
a good understanding of the strengths and weaknesses United States, where grading is generally on a
of operations. Aside from proficiency checks of pass-fail basis and in many organizations fewer
technical competence, usually conducted in the than 1% are deemed unsatisfactory. As a result,
simulator, sources of data include: there is minimal diagnosticity obtained at
significant cost (one major airline cites a cost of
1. Accident investigation. Exhaustive analyses $1,000 per line check with no utility in obtained
of factors surrounding accidents have been a information). Pilots are certainly displaying their
primary source of safety information in aviation. best, not necessarily their normative behavior
However, accidents are infrequent events that during a line check.
usually reflect the concatenation of rare factors as
eloquently described by James Reason (1990) in 4. Flight Data Recorder monitoring (Flight
his classic Swiss cheese model. As a result, such Operational Quality Assurance: FOQA, FAA,
investigations probably do not uncover normative, 2001). Especially in new generation aircraft, it has
unsafe operational practices. become almost routine to utilize flight recorder
data to monitor exceedences in performance of the
aircraft and to use these data for safety analysis,
1
Research supporting this paper was supported by Federal Aviation Administration Grants 99-G-004, Robert Helmreich, Principal Investigator.
We wish to thank Captain Bruce Tesmer for his invaluable collaboration in all aspects of LOSA development and for suggesting the name.
without jeopardy to flight crews. While FOQA inspector riding on the jumpseat. The fact that
data provide essential information about what numerous instances of procedural and regulatory
happens in terms of deviations from organizational violations are observed attests to the achievement of
expectations, the data do not provide any insights trust with those observed.
into why the deviations occurred.
At a more macro level, the interview, survey, and
5. Normal Flight Monitoring LOSA. The observations provide both objective and subjective data
Line Operations Safety Audit (LOSA) was on strengths and weaknesses associated with
developed by the University of Texas Human professional and organizational culture, the National
Factors Research Project in conjunction with Airspace System, aircraft design, especially issues
major airlines in the United States as a means of related to automation, and the level of support
collecting normative data on crew performance provided to crews by ground operations, maintenance,
during line flights. The focus of this paper is to and dispatch.
describe the current state of LOSA and the
application of LOSA data for research, LOSA History
organizational safety initiatives, and training.
LOSA was first described here in 1999 at the 10th Thirteen LOSAs have been completed or are in
International Symposium (Helmreich, Klinect, and progress at this time. The initial five involved only
Wilhelm, 1999; Klinect, Wilhelm & Helmreich, assessment of crew performance on the CRM-related
1999), but its scope and acceptance have changed behavioral markers, technical proficiency, and overall
significantly and this will serve as an update as crew effectiveness. The significant shift to include
well as a discussion of the multiple uses of data recording of threats and errors and their avoidance and
collected. management was intitated in collaboration with
Captain Bruce Tesmer of Continental Airlines. At the
DEFINITION, SCOPE, and GOALS OF LOSA same time, the addition of the survey and/or interview
as an integral part of the data collection was finalized.
LOSA consists of a family of methodologies applied Eight LOSAs with the threat and error orientation have
to normal flight operations to assess their strengths and been completed in U.S. major and regional carriers and
weaknesses. At the heart of LOSA is the non-jeopardy, major international airlines.
systematic assessment from the jumpseat of operational
threats and cockpit crew errors and their management. Growing Support for LOSA
Tabulation of threats and errors is augmented by
assessment of CRM-related behaviors associated with Daniel Maurino of the International Civil Aviation
effective and ineffective flightdeck management Organization (ICAO) has been a strong supporter of
(behavioral markers: Helmreich & Merritt, 1998; normal process monitoring as represented by LOSA.
Helmreich, Wilhelm, Klinect, & Merritt, in press). In LOSA has been presented at meetings of ICAO, the
the future, plans are being developed to link LOSA International Air Transport Association, the Air
observations with FOQA data while still preserving the Transport Association, and the ICARUS Committee of
essential, non-jeopardy nature of the methodology. the Flight Safety Foundation. ICAO has named LOSA
its primary human factors initiative for 2000-2005 and
Current LOSA practice combines the observational is issuing a LOSA Handbook this year. Costa Periera,
data with structured interviews of crewmembers Secretary General of ICAO sent a formal letter to Jane
regarding safety issues and/or a survey of attitudes Garvey, FAA Administrator in June of 200 regarding
regarding safety practices, safety and organizational LOSA:
culture, and cockpit management using a specialized
version of the University of Texas Flightdeck [LOSA] acquires direct, first-hand data on
Management Attitudes Questionnaire (see Helmreich the successful recovery from errors by flight
& Merritt, 1998 for application of the FMAQ across crews during normal line flights. [It] is
organizations and national cultures). aimed at collecting data on successful
human performance; and this is indeed a
The key to obtaining useful data is the credible first in our industry, since aviation has
assurance to pilots that the observations are without traditionally collected data on failed human
jeopardy to them. With this trust a picture of flight performance, such as an accident or incident
operations quite different from that obtained by a investigation.
check airman conducting a line check or an FAA
LOSA concepts were presented at a regional human capacity, external stressors, poor group dynamics, and
factors conference in Mexico Citiy in February, 2001. cultural influences. Errors were noted in 64% of flights
An Asian LOSA Summit involving the major carriers observed. (4) manage aircraft deviations, which are
in the region was held in March, 2001, hosted by defined in the model as undesired aircraft states (for
Cathay Pacific in Hong Kong. This will be followed by example, wrong configurations, speed, heading, etc.).
a European LOSA Congress and a Middle Eastern Undesired aircraft states were found in 32% of flights.
meeting hosted by Emirates in Dubai. The goals of These safety tasks are shown in Figure 1.
these meeting are to demonstrate the methodology and
to discuss how to make LOSA widely available to
organizations while preserving the scientific integrity
of data collected and maintenance of a useful Error Avoidance
international LOSA database.
In discussions with senior FAA officials, the Threat Management
potential extensions of LOSA methodology have been Safety

Tasks
discussed. These include adapting parts of the
Error Management
methodology as a tool for FAA Air Carrier Inspectors
to use during enroute evaluations and applying it to
evaluate performance in Line Oriented Evaluations Undesired Aircraft State
Management
(LOE) as part of the Advanced Qualification Program
(AQP). Obviously, under jeopardy conditions such as a
route check by an FAA inspector, the observed
behavior is likely to be quite different from those Figure 1. Safety Tasks .
obtained during LOSA. It is only the classificatory
methodology that may be usefully adapted to the The distinction among safety tasks is important
formal evaluation setting. because different CRM practices (which can be defined
as threat and error countermeasures) are differentially
PLACING LOSA IN CONTEXT: THE THREAT associated with effective accomplishment of each of
AND ERROR MANAGEMENT MODEL the four tasks. When the cockpit crew has put an
aircraft is in an undesired state (for example, at the
The complex data developed through LOSA are best wrong speed or altitude), the primary task is recovery
understood in a model of threat and error in the from the undesired state. From LOSA data we have
aviation system that reflects not only external threats validated the importance of observable CRM skills in
and errors (for example, operational errors by air accomplishing these tasks. The necessary skills fall
traffic controllers, but also enables probing for latent into four groups: (1) Team Climate; (2) Planning; (3)
threats residing in the organization or system such as Task Execution; and (4) Review and Modify. Not
organizational and professional cultures, rostering surprisingly, team climate behaviors such as active
practices, design factors, etc. We presented the model leadership and establishing a team environment are
at the 10th Symposium (Helmreich Klinect, & cricial for all four safety tasks. Planning, in contrast, is
Wilhelm, 1999). Discussion of the components of the most related to error avoidance and threat management.
model is found in Helmreich, Wilhelm, Klinect, & Task Execution behaviors such as monitoring and
Merritt (in press). workload management are central to error
management. Review and Modify countermeasures,
For safe operations, in addition to the the technical which include evaluaton of plans, inquiry and
task of flying, crews must accomplish four safety tasks: assertiveness, are most relevant for threat management
(1) use proactive strategies to avoid committing errors. and undesired aircraft state management.
(2) manage operational complexity, which translates
into threat management. Across all operations The Types and Distribution of Crew Error
observed, threats were recorded in 79% of flights,
including not only environmental factors such as In developing the model of error management we
terrain, weather, and equipment malfunctions but also found empirically that all of the observed errors could
errors external to the cockpit (for example, by ATC). be classified into five types. These are:
(3) manage crew errors. Error is an inevitable result of 1. Procedural errors where crews are trying to
human limitations such as fatigue and other follow procedures but execute them
physiological factors, limited memory and processing
incorrectly (for example, making incorrect U.S. pilots are least accepting of the importance of
entries in the FMC). adhering to rules (Helmreich & Merritt (1998). The
2. Communication errors in which information is fact that such a low percentage of these violations were
improperly or incompletely communicated, consequential might tempt one to dismiss them as
witheld , or misunderstood. trivial, rather like driving a couple of miles over the
3. Proficiency errors where tasks are improperly speed limit on a deserted superhighway. However,
executed because of a lack of skill or additional analyses led to a different conclusion. We
knowledge. split the database into those crews that had at least one
4. Decision errors involving situations not violation and those that had none. We then compared
covered by procedure or regulation in which the number of other (non-violation) errors committed
crews take actions that unnecessarily increase by each group and the percentage of these other errors
risk. that resulted in undesired aircraft states or an additional
5. Intentional non-compliance when crews error. The results showed that crews with a violation
knowingly violate company policy or were almost twice as likely to make other types of
regulations. errors and that these other errors were twice as likely to
be consequential. Thus crews who violate put the
The distribution of errors across airlines completing organization at substantially greater risk, suggesting
a threat and error LOSA is shown in Figure 2, which that while an effective safety culture adopts a non-
gives the percentage of each error type and the punitive attitude towards inadvertent error, violations
percentage of each type which are consequential. By should not be tolerated.
consequential, we mean errors which result in an
undesired aircraft state. The Risky Phase
Non-compliance
Phase of Flight Threats Errors
Procedural
Pre-Departure / Taxi 30% 25%
Communication
Takeoff / Climb 22% 22%
Cruise 10% 10%
Proficiency
Descent / Approach / Land 36% 40%
Decision
0 20 40 60 80 1 00 Taxi / Park 2% 3%
% consequential % of all Errors
Figure 2. LOSA errors and their consequences Figure 3. Incidence of threat and error by phase of
flight
The highest percentage of errors involve intentional
non-compliance or violations, but only 6% of these led Threats and errors were also examined by phase of
to an undesired aircraft state. In contrast, only 5% of flight, with results that strongly confirmed the Flight
errors reflected a lack of proficiency, but more than Safety Foundation Approach and Landing Accident
60% of these were consequential. Similarly, decision Reduction initiative based on the high incidence of
errors were infrequent, but more than half of those that fatal accidents during this phase of flight (Flight Safety
occurred were consequential.2 Foundation, 1998). Our data, shown in Figure 3 on the
previous page, indicate that the highest number of both
Violations Matter threats and errors occur during descent, approach, and
landing. These findings in our initial LOSAs led to a
The high percentage of procedural non-compliance special focus on this aspect of flight operations that we
was surprising, although it is consistent with our data have called The Blue Box as shown in Figure 4. We
showing that of those surveyed in over 20 countries, have started to collect additional data to understand
better the dynamics that occur from the pre-descent
2 briefing through flare and touchdown.
Diferences between these numbers and those reported in earlier
publications are because we are continuously updated the database.
There results remain highly stable across organizations.
represents a start towards understanding the reasons
why crews fail to comply with procedures.
Safety initiatives. LOSA data provide organizations

with concrete data on line operations. Continental
Airlines, for example, has used the data to address
Brief TOD procedures and to provide guidance for crews
18000 ft.
4 regarding high threat operations. The data can also be

10000 ft.
Slow and
integrated with information from quick access
Configure Stabilized
FAF/OM
Approach
Bottom
recorders (FOQA: FAA 1999) to provide greater
Lines
insight into areas of risk. FOQA data were used in our
Flare / Touchdown most recent LOSA to pick airports that represent high
levels of operational threat.
Training and curriculum development. LOSA data

Figure 4. The Blue Box provide insights into areas in need of special training
(for example, one airline initiated new training on
Examining Blue Box data reveals that more captain leadership because this was an area identified
automation and decision errors occur duting this phase. as weak). Another important characteristic of LOSA
This is also the time when threats (including errors) data is the identification of superior performance.
associated with ATC are most difficult to manage. 28% Examples of outstanding behavior (instead of failures)
of errors occuring in the Blue Box resulted in can provide powerful learning.
undesired aircraft states, with the most frequent being
incorrect configurations, vertical deviations, and speed Continental Airlines under the direction of Captain
too high. Blue Box data provide guidance for Donald Gunther (Gunther, in press) has developed a
organizational actions to manage threat and error. new CRM Threat and Error Management course that is
being given to all pilots. The course is build directly on
USING LOSA DATA data from LOSA and, as a result, has high credibility
with pilots.
LOSA data have three major uses for research, for
organizational safety initiatives, and for the REFERENCES
development of training curricula.
Federal Aviation Administration. (2000). Aviation
Research. LOSA data are invaluable for research as Safety Action Programs. Washington, DC. Advisory
they capture the behavior of professionals performing Circular 120-66A.
challenging work in the real world. We have only
scratched the surface of the data for research as most Federal Aviation Administration. (2000). Flight
energy has been addressed to refining the methdology Operational Quality Assurance (FOQA) Notice of
and establishing the database. Several findings, Proposed Rulemaking. Washington, D.C.
however, have already emerged that are of both
theoretical and practical interest. Hines (1998), for Flight Safety Foundation: (1998). Killers in
example, found that under conditions of operational Aviation. Flight Safety Digest. Arlington, Va. Author.
complexity crew performance was significantly better
with the captain serving as pilot not flying while under Gunther, D. (in press). Threat and Error
more benign conditions it made no difference. Bryan Management Training. In Proceedings of the Eleventh
Sexton (Sexton & Klinect, in press) has found that International Symposium on Aviation Psychology, The
crews with more positive (and congruent) safety Ohio State University.
attitudes commit errors that are more likely to be
inconsequential, trap more errors, are less likely to Helmreich, R.L., Klinect, J.R., & Wilhelm, J.A.
commit sequences of errors, and less likely to make (1999). Models of threat, error, and CRM in flight
unstable approaches. This linkage between attitudes operations. In Proceedings of the Tenth International
and behavior is important both practically and in Symposium on Aviation Psychology (pp. 677-682).
addressing an old controversy in psychology about the Columbus, OH: The Ohio State University.
linkage between attitudes and behavior. This also
Helmreich, R.L., & Merritt, A.C. (1998). Culture at
work in aviation and medicine: National,
organizational, and professional influences. Aldershot,
U.K.: Ashgate
Helmreich, R.L., Wilhelm, J.A., Klinect, J.R., &

Merritt, A.C. (in press). Culture, error and Crew
Resource Management. In E. Salas, C.A. Bowers, & E.
Edens (Eds.), Applying resource management in
organizations: A guide for training professionals.
Princeton, NJ: Erlbaum.
Hines, W.E. (1998). Teams and Technology: Flight

Crew Performance in Standard and Automated
Aircraft. Unpublished doctoral dissertation. The
University of Texas at Austin.
Klinect, J.R., Wilhelm, J.A., & Helmreich, R.L.

(1999). Threat and error management: Data from line
operations safety audits. In Proceedings of the Tenth
International Symposium on Aviation Psychology (pp.
683-688). Columbus, OH: The Ohio State University.
Reason, J. (1990). Human error. New York:

Cambridge University Press.
Sexton, J.B. & Klinect, J.R. (in press).The link

between safety attitudes and observed performance in
flight operations. In Proceedings of the Eleventh
International Symposium on Aviation Psychology, The
Ohio State University.
PRESENTATIONS
An Introduction to
LOSA:
The Perspective of
ICAO
Captain Dan Maurino
Flight Safety and Human Factors - ICAO
First LOSA Week
Cathay City, Hong Kong
12-14 March 2001
The Ultra-Safe System
Fragile systems
Individual risk management
10-3 Accident investigation
Safe systems
Design
Regulations & procedures
10-5 Reporting systems
Ultra safe systems

Uncertain accident prediction
Reporting systems inconsistent
10-6 Political management
Common Sense Approach
ISO 9000/TQM safety
z Safety results from specifications
(comprehensive rules & procedures)
z Good professionals adhere to rules &
procedures
z Good professionals and good rules &
procedures ensure safety
The Pentium Illusion
GIGO
WYGIWYA
Data Does Not Mean Information
What About Human Error?
Design & manufacture
Management & supervision
Training & maintenance
Stakeholders
Operational
personnel
Errors reside
in the person
From Clear to Opaque Causality
Direct & linear Circular & random
Broadening the Scope

Stakeholders
Operational
personnel
Errors reside
in the context
Unanticipated Disturbances
Operational
contexts
Warning! cannot be
To err is
human
entirely
Insert
pre-
carefully specified
Monitoring Normal Practice
Stakeholders
Operational
Errors do not personnel
cause accidents
Operational Behaviours
Accomplishing the systems goals
Safety Production
Processes & Outcomes
Error:
Causes and
consequences
are not linear
in their
magnitude
Training Behaviours
Accomplishing training goals
Safety Production
Error: Once in A Million Departures
Flaps Checklist Unheeded

omitted failure warning
Error Deviation Amplification Degradation/

breakdown
Error: Quite Frequently
Flaps Checklist Effective

omitted works warning
LOSA
Error Deviation Amplification Normal

operation
Where Experts Dare
Crisis Operation Design
Expert operating Normative

zone operating zone
A Principled Approach
Consequential
Proficiency 69%
Operational Decision 51%
Communications 11%
Procedural
Procedura 23%
Violations 2%
0 10 20 30 40 50 60 70
Percentage of Errors
Violations and Open Systems
Accident Production
Higher objectives
Incident
Regulations
Violations Safe &
Risk efficient Technology
system People
performance
Training
Max System Output

The Role of the Regulator
Think about the spirit rather than the
letter of the law
Legislation
Decoration
Litigation
Aviation: a geriatric and

saturated system facing
significant threats
Getting the Best for the Buck
Intervention Systemic approach
efficiency Safety culture
Training
Error Management
Error Resistance
Regulations Flexible regulation
Engineering solutions Adaptive

technology
10-2 10-3 10-4 10-5 10-6 10-7

Risk of accident
The Ultra-Safe System: Epilogue
z Aviation cannot be entirely specified
Disturbances will occur; humans will
inevitably make errors
z Basic normative framework
z Flexible & adaptive implementation
z Deviation management
Danger: loss of control of the

deviation management process rather
that deviations themselves
The Line Operations Safety Audit
(LOSA) and Safety Culture
Robert L. Helmreich
The University of Texas Human Factors Research Project

LOSA Summit
Cathay Pacific
Hong Kong
Human Factors
Research Project
Data for a Safety Culture
How do airlines monitor safety?
Accident investigation
Incident reports
Data slanted to events resulting from system and flight crew failures
Line checks
Data show crew proficiency and procedural knowledge
Flight Data Recorders FOQA
Data show what happened in terms of flight parameters
Observing normal flights Line Operations Safety Audit (LOSA)
Gives data on why things happen and how they are managed
Provides a more realistic baseline of safety
Line Operations Safety Audit (LOSA)
LOSA Jump seat observations of flight crew performance
during regular scheduled flights
Observers unobtrusive collecting data not participating in flight
Team of observers from different backgrounds

Line pilots / Union representatives
Check airmen
Safety and Training pilots
UT observers
All data are DE-IDENTIFIED and CONFIDENTIAL

Purpose of LOSA
Data provide a baseline of safety pertaining to:
1. Crew performance strengths and weaknesses
Proficiency
Decision-Making
CRM skills
Procedural compliance
2. System performance strengths and weaknesses

Culture
Airspace System airports and navigational Aids
Aircraft design / automation
Standards / Training / Safety / Maintenance
Crew support ATC, Cabin, Ground, and Dispatch
LOSA History 1992 - 2001
Threat and Error

Flight Management LOSA Management LOSA
Delta domestic Continental Latin America
Delta international Continental Express
TWA Gulfstream Express
American Air New Zealand
Continental Air Micronesia
USAir Continental
Delta
USAir
Cathay Pacific
Components of LOSA
Part 1. Flight crew survey

Part 2. Flight crew interview
Part 3. Descent and approach
Part 4. CRM Countermeasures
Part 5. Threat and error management
LOSA Data: Flight Crew
For each flight segment, observers collect data on:
General Flight Crew Flight Crew Performance

Information Behavioral markers
Demographics
Crew errors and violations
Attitudes / perceptions (FMAQ)
Undesired aircraft states
Safety interview comments
Technical data for approaches
Type and stability
Flight Description
Threat management
Observer narrative
Error management
Overt threats
Operational complexity Undesired state
management
The Safety Bottom Line
To fly safely, crews must
1 Avoid committing errors Error Avoidance
2 Manage operational complexity Threat Management

Safety
Bottom Line
3 Manage their own errors Error Management
4 Manage aircraft deviations Undesired Aircraft

State Management
Error Avoidance
Safety
Bottom Line
Error Avoidance
Complete error avoidance is impossible errors are
inevitable
LOSA data from 429 flight segments at five airlines

64% of the flights observed had at least one error
Must look for roots of error to strengthen system defenses

Human Limitations Lead to Error
Limited memory capacity

Limited processing capacity
multi-tasking capability
Limits imposed by stressors
tunnel vision
Limits imposed by fatigue and other physiological
factors
Poor group dynamics
Cultural influences
Threat Management

Safety
Bottom Line
Threat Management
Threats - Originate outside the flight crews influence but require active
management to prevent them from becoming consequential to safety
Adverse weather Operational time pressures
Terrain Non-normal operations
Traffic ATC command events / errors
Airport conditions Cabin events / errors
A/C malfunctions MX evens / errors
Automation events Dispatch events / errors
Communication events Ground crew events / errors
LOSA - 79% of the flights observed had at least one threat

Error Management

Safety
Bottom Line
Flight Crew Error Types
1. Intentional Noncompliance violations
ex) Performing a checklist from memory
2. Procedural Followed procedures but wrong execution

ex) Wrong altitude setting dialed into the MCP
3. Communication Missing information or misinterpretation

ex) Miscommunication with ATC
4. Proficiency Error due to a lack of knowledge

ex) Lack of knowledge with automation
5. Decision Discretionary decision that unnecessarily increased

risk
ex) Unnecessary navigation through adverse weather
Error Frequencies and Outcomes
Violations
Procedural
Communication
Proficiency
Decision
0 20 40 60 80 100
Percent Consequential Percent Frequency
The Importance of Violations
Airlines cannot allow violations to normalize
Why?
Violations cultivate complacency and a disregard of rules
Crews that commit at least one intentional noncompliance error are

twice as likely to:
Commit unintentional errors (Procedural, Communication ..)
Commit consequential errors that lead to additional errors or

undesired aircraft states
Error Management Results
Error Responses Most errors are undetected
64% Undetected
31% Detected and effectively managed
5% Detected and mismanaged
Error Outcomes Most errors are inconsequential

Inconsequential 77%
Additional Errors 6% (Error Chains)
Undesired Aircraft States 17%
Undesired Aircraft State Management

Safety
Bottom Line
4 Manage aircraft deviations Undesired Aircraft

State Management
Undesired Aircraft States Resulting
from Threat and Error
Aircraft deviations at edge of the safety envelope
Incorrect aircraft
Unstable approach
configurations
Long or hard landing
Vertical deviations -
altitude Wrong taxiway / runway
Lateral deviations - Wrong airport
heading Wrong Country
Speed deviations
LOSA 32% of the flights had an undesired aircraft state

The Riskiest Phase
Phase of Flight Threats by Phase Errors by Phase
Cruise 10% 10%
Descent / Approach / Land 36% 40%
Taxi / Park 2% 3%
Descent / approach / land phase also contains the most

variability in crew performance and consequential errors
Safety Culture:
Survey Data and Error Management
Human Factors
Research Project
Flt Ops listens
My safety sugg acted upon
Mgmt not compromise safety

Safety
Culture
Encouraged to report unsafe
I know safety channels
Chief Pilot availability
FMAQ survey items form index of Safety Culture

Pilot Perceptions of Safety Culture
# Undesired States per Segment

1.2
1.1
1 1.13
0.9
0.8
0.7
Scale 0-100
0.6
0.8
0.5
0.4
0.3
0.2
0.1
0 0.19
Low Middle High
Pilot Perceptions of Safety Culture
% Errors that were Trapped

100
90
80
70
60
Scale 0-100
50
40
57.5%
30
20
10 23.6%
0 14.3%
Low Middle High
A Model of Threat and Error Management
Human Factors
Research Project
Purpose and Uses
The Threat and Error Management model was derived from
LOSA data and guides further refinement.
The model is being used by airlines as a framework for
analysis of incident and ASAP data
Critical Definition
Latent Threats are aspects of the system that predispose
the commission of errors or can lead to undesired aircraft
states
ATC practices
Scheduling practices that result in fatigue
Organizational, national, professional culture
etc
Latent System Organizational Professional
Threats
Expected Unexpected Events External
External Error
Error
Unexpected Events External Error
External Events and Risks and
and Risks
Threats and Risks
Risks
Violations - Communication
Procedural - Proficiency
Operational Decision
Threat Recognition Error Management

CRM and Error Avoidance
Behaviors
Behaviors Behaviors
Undesired Additional
Outcomes Inconsequential Error
Aircraft State
CRM Undesired State

Behaviors Management
Behaviors
Recovery to Additional
Safe
Final Flight Safe Flight Error
Outcome
Incident
Accident
LOSA and CRM
Human Factors
Research Project
Twenty Years of Change
Since 1981, CRM has evolved steadily from early courses that
were derived from corporate management and focused on
individual styles and skills
By 1998, the focus of the fifth generation had shifted to error

management
In 2001, sixth generation has become Threat and Error

Management
CRM skills defined as threat and error countermeasures
CRM
Problem Little data supporting CRM curriculum content
How do flightcrews detect and manage error in normal operations?
What are effective and ineffective error management strategies?
What magnitude does CRM play in error management?
LOSA - provides critical data
Continental Airlines use of LOSA data

Recognized need for revisiting basic CRM due to decay in acceptance
1996 Error Management (5th generation CRM) all pilots
2001 Threat and Error Management (6th Generation CRM) all pilots
An Expanded Definition of CRM
Sixth Generation CRM
Error Avoidance
Threat Management
CRM Skills
Error Management
Undesired Aircraft State

Management
Characteristics of Threat and Error
Management CRM
Based on data from airlines own experience
LOSA, FOQA, ASAP
Allows crews to assess threats and their management
Examines effective and ineffective error management
Operational focus avoids games and irrelevant activities
Gives crews specific threat and error countermeasures

LOSA and 6th Generation CRM
LOSA validates the role CRM plays in safety
Error Avoidance
Threat Management
Error Management
Undesired Aircraft State Management
Four groups of countermeasures (CRM skills) show up in LOSA

reflecting both superior and below standard crew performance
1. Team building
2. Planning
3. Execution
4. Review / Modify
Using LOSA Error Data for CRM Design
and Organizational Interventions
Violations - suggest poor procedures, weak captain leadership and/or a

culture of non-compliance
Procedural errors - may indicate poor workload management and/or poor
procedures
Communications errors - may reflect inadequate CRM (monitoring and
challenging) or complacency
Proficiency errors - suggest pressures to train and/or need for higher

standards and better monitoring by check airmen
Decision errors - may indicate need for more CRM training on expert
decision making and risk assessment
When LOSA Succeeds
Top management recognizes the need for data
Resources are provided for the process
Commitment to use the data for change
Pilots support the process

When LOSA is Inappropriate
LOSA will not work in a culture of blame

and punish
LOSA should not be undertaken without the

cooperation of the regulatory authority
LOSA Meetings 2000-2001
Australian Aviation Psychology Symposium,

Melbourne,
November, 2001
International Aviation Psychology Symposium
Ohio State University, March
Asian LOSA Summit, Hong Kong, March
ICAO Human Factors, Sta Cruz, Bolivia, April
European LOSA Summit, Zurich, TBD
Middle Easter LOSA Summit, Dubai, TBD
Latin American LOSA Summit, Panama, TBD
Summary
LOSA has proved to be a valuable safety

initiative
LOSA facilitates CRM curriculum

development
LOSA is still under development

It will evolve just as CRM has
University of Texas
Human Factors
Research Project
www.psy.utexas.edu/psy/helmreich/nasaut.htm
The Phases of LOSA
James Klinect
LOSA Week, Cathay Pacific, Hong Kong
March 12-14
Phases of LOSA
I. Planning
II. Observer Training / Recalibration
III. Observations
IV. Data Entry / Cleaning / Analysis
V. Final Report
VI. Communication of Results
Phase I
Planning
Planning Committee
First - Form a LOSA steering committee with all
major stakeholders represented
Flight operations
Safety
Training
Standards
Union
Aim is to give everyone ownership of the project

Planning and Education
Large part of the planning phase is education
Education to the line pilots - letters

Introduction letter three months before the project
Reminder letter on the week of project initiation
All letters should be signed by management and union
representatives
All forms of communication must emphasize:

LOSA is a non-jeopardy collection of safety data
All data are de-identified and confidential
Planning and Commitment
After education, need to get commitment from
regulatory, management, and union parties
Regulatory - Commitment to brief and maintain

the integrity of a non-jeopardy process
Management Commitment to use the data and

make changes if necessary
Union Commitment to support LOSA and offer

a quality check to the process
The Shotgun LOSA Approach
After education and gaining commitments, need to
establish LOSA goals
Shotgun LOSA it is only a start

Number of observations is dependent upon airline size
Want to observe as many different crews as possible
Provides a first baseline for threat and error management
For fleet comparisons, 30-50 observations per fleet

is enough to make confident conclusions
Observer Team
Composition of the observer team
Diversity is key all observers are biased to something
Line check airmen make sure most of their

observations are done outside of their own fleet
When selecting observers, think about someone

that will take the time do a good job
Many observations takes 30 min. to an hour to write up
Phase II
Observer Training / Recalibration
Observer Training
Two solid days
Typical schedule
Introduction to LOSA
How to be a fly on the wall observer
Observation form training and calibration
Wrap-up with logistics
End of training should mark the beginning of

LOSA observations
Dont allow lag time between training and observations
Recalibration
Recalibration is critical for data quality assurance
What is recalibration?
After 3 to 5 observations, time is set aside to go over
observer questions or concerns
Can be done individually or in a group
Once recalibration is complete, the observer is set

free to conduct the rest of their observations
Phase III
Observations
Observations
Scheduling - Make sure the observer has a game
plan for their observations
Example)
Observe 20 segments
10 long haul 777 and 10 regional A320
Make one out and back to SFO (LOSA highlighted airport)
Encourage observers to complete their forms

electronically allows for e-mail and copies
Instruct observers to send all of their observations

to a trusted site to maintain confidentiality
Phase IV
Data Entry / Cleaning / Analysis
Data Entry
All data are entered into Microsoft Access and
analyzed with a statistical software package (SPSS)
Have a process in place to organize data entry

Assign each returned form with a unique ID number
Keep a log of the ID numbers for effective tracking

and monitoring of LOSA
Also include identifiers such as aircraft type and city pairs
Data Cleaning
What is data cleaning?
Process of examining all threats, errors, and undesired
aircraft states to ensure correctness and coding
accuracy
Data cleaning takes time

Most recent data cleaning for 372 observations
180 pages of narratives (8 pt font, single-spaced)
1100 threats
730 errors
120 undesired aircraft states
Roundtables
The best way to clean is to divide and conquer
One person to go over the narratives to extract
unlogged threats, errors, and undesired aircraft states
Another person covers the accuracy of threats
Another person on errors and undesired aircraft states
Bring all problems to Roundtables for discussion

Keep the group small LOSA planning committee
The Final Roundtable jointly validates the

dataset for analysis
Data Analysis: Most General
Observation form data
Demographics overview
Threat exposure with management rates
Errors committed with management rates
Undesired aircraft states with management rates
Above/Below Good Threat and Error Countermeasure ratings
Crew interview data

Breakdown by fleet and code comments by theme
Survey data
Number of respondents who agreed / disagreed with each item
Phase V
Final Report
Final Report
The final report provides the most general
diagnostic data of strengths and weaknesses
Limitation The final report only scratches the

surface on why things occurred
Many answers to the why questions requires a

coding analysis of the narratives hard work
The focus of future research hopes to ease this process
Phase VI
Communication of Results
Communication of Results
Executive briefing
Give top-level management the first look
Have recommendations ready on how to use the data to
incorporate change if necessary
Fleet manager debriefing

Provide an in-depth analysis that are fleet specific
Good place to form committees for change
Communication to the line

Provide information about the results and how they are
being to used to improve the system
LOSA
Making The SafetyProcess
Better
Line Operations Safety Audits

Improving The Safety Margin
What has your company done to improve your safety
margin in the last five years?
Manuals Updated ATC Arr/App/Dep Help revise
Training Enhanced
Airport Issues Resolve
Equipment Upgraded
Maintenance Increased Staff
New Procedures - Added
Ground Towing Super-Tug
Old Procedures Out
Checklists Modified New Flight Sim Buy & Install
New Aircraft Purchased Flight Following Upgrade
Dispatch Improved WX, Aircraft and ground
NOTAMS, Flt Plan communications equipment
New systems Windshear, Web-site Install Info for crews
(E)GPWS Continuing Safety Program -
Old aircraft - Retired Develop
Why Are These Safety
Improvements Required?
Because Things
CHANGE!
NOT ETOPS CERTIFIED
Flies Good, but the visual needs work.
Instruments on the first blind flight
There was no FMS
CHANGES in an organization happen:
1). Through its PEOPLE
or
2). Through a TRAUMA (Accident)
Either Way, Change Will

Happen!
SYSTEM CHANGES
Affect Your Line Operations
Things are happening, on the line, that increase
your risk of a major accident
- and -
You do not know about them!
Why are things happening?
System Changes
Create Threats (Overt & Latent)
Stimulate Crew Errors
Why dont we know about these
things
Change limits your Big Picture
Crews are operating unsupervised
Crews do not report they fear punishment
Crews normal performance is different from
Check ride performance
Management has difficulty screening out
valid reported crew concerns from over-
reported crew complaints
Why we need know about these
things
These unknown things (error chains &

undesired aircraft states) - are the
precursors to accidents / incidents
LOSA provides data that identify these

accident/incident precursors
Line Operations Safety Audit
(LOSA)
Providing Data
To Better Manage CHANGES
LOSA - Gives You:
Safety data on what pilots really do
Data on what good crews do right
Buy-in by employees
Proactive approach
A plan for continuous improvement

Employee Buy-In & Impact on
Safety
Observations show reality
Previous encountered events
Crews believe in management when

management makes improvements
Results From Being Proactive
With LOSA
Results:
Better policies & procedures
Better crew compliance
Fewer unstable approaches
Better risk assessment of operations at all
levels
Targets for safety resources
A Continuous Improvement Plan
LOSA - A Focus Safety Tool
LOSA A Baseline Safety Tool
LOSA A Rotating Safety Tool
LOSA A Safety Change Measurement Tool

Past LOSAs
1996 836 segments (Entire System)
1997 COEX
1998 Latin American
1999 Pacific
2000 Entire System

LOSA 2000
There were five parts to LOSA 2000:

Part 1. Flight Crew Survey
Part 2. Flight Crew Interview
Part 3. Stabilized Approaches
Part 4. CRM Counter-Measures
Part 5. Threat and Error
Survey Data
Organizational Climate
Safety Culture
Training/Checking
Working Together
Stress
Organizational Climate items
Pilot morale is
high
I'm proud to work

for CO
I like my job
Organizational Climate at CO Across Time
100
90
80
70 82%
60 72%
Scale 0-100
50 63%
40
30
20
10
0
1996 1999 2000

CO CO CO
Organizational Climate Across Airlines

100
90
80
70
60
Scale 0-100
50
40
30
20
10
0
a1 1 2 3 4 000 5
Asi
A A e 1 A er 1 A A
US US ro p US A m US CO
2 US
Eu S
CO Safety Culture over time
Safety Culture Items
100
90
80
Pilots trust senior mgmt at CO 70
S cale 0-100
60
50
68%
40 53% 57%
30
Mgmt never compromises safety for profits
20
10
0
1996 1999 2000
Mgmt - Flt Ops listen & care about our concerns
Safety Culture Across Airlines

100
My safety suggestions would be acted upon 90
80
S cale 0-100 70
60
50
I'm encouraged to report unsafe conditions 40
30
20
10
0
Senior mgmt at CO is doing a good job
er 1 1 er 2 2 er 1 er 3 00 0
Am pe pe Am O2
N E u ro N Am E u ro S N Am C
Satisfaction with Training and Checking
Ground School
Checklists
Quality of initial training
Sim-based training
Flight Stds & Training overall
Fairness of checking
Improvement in Teamwork and Cooperation since 1996
Ramp Personnel CO 1996

CO 2000
Scheduling
Gate Agents
Flight Attendants
Dispatch
Maintenance
Other cockpit crew
High Satisfaction with the Quaility of Teamwork and Cooperation

Realistic Appraisal of Stress Items
I am more likely to make errors in emergencies
Personal problems affect my performance
I let others know when I'm becoming overloaded
I am less effective when stressed/fatigued
Situation awareness suffers when I am fatigued

Interview Topics
Safety concerns: Other topics
Operations Training
ATC Communications
Dispatch Stable Approaches
Checklists
SOP
Interview Responses By Category
350 320
300
262
N um ber Of R esponses
250
200 173
156
150
102
100
50
11
0
ATC and Checklists Flight Dispatch SOP Other Areas
Airports Operations
Response Rates - ATC & Airports:
By Fleet Type
2.50
2.20
2.00
1.76
Responses per Crew
1.50 1.41
1.33 1.30
1.00
0.50
0.00
B737 B757 B777 DC-10 MD-80

Responses by Percent
0%
2%
4%
6%
8%
10%
12%
14%
16%
18%
20%
Ground Delay
9%
Flow/Reroute/Hold
6%
Clnc/Chgs/Rwy swap
8%
Comm/ Language
18%
Slamdunk/Spacing
16%
Arrival/Dept.Procedures
19%
Rwy / Twy
5%
Congstn/Danger
14%
ATC & Airports - Response Categories
Misc
5%
Airports With Serious Concerns : Traffic, Procedures,
Airport Layout (52 of 120)
14 13
12
10 9
8
8
6
6 5
4 3 3
2
2 1 1 1
0
SNA DCA EWR LAX IAH ATL, SFO, STL, GDL, MDW, UIO,
LGA, CLE, JAX, PDV, CDG, TGU,
SAN LAS Rome AUS, CDC, SJO,
MXP MEX Milan
Non-Conforming (Unstable)
Approaches
LOS A: No n-Co nfo rming Appro ac he s
Outs ide Re quire d
1000 ft o r 500 ft Windo w
-49%
-59% -59%
1996 - ALL 1998 - L.A. 1999 - P ACIFIC 2000 - ALL

NCA: Type Of Approach
80%
80%
60%
40%
13%
20% 7%
0%
Visual Non-Prec Prec
NCA: Pilot Flying
100%
80% 47% 53%
60%
40%
20%
0%
Capt FO
NCA: Number Of Parameters Out-Of-Limits
@ The Window - Per Approach
47%
50%
40%
30% 20% 20%
20% 13%
10% 0 0
0%
ONE TWO THREE FOUR FIVE SIX
NCA: Parameters Not Met @

The Window
73%
80%
60% 47%
40%
40%
33%
20% 13%
0%
Speed Vert Dev Unspool Vert/Spd Lat Dev
NCAs Affected By ATC (47%)
47%
50%
40% 27%
30% 20%
20%
10%
0%
App Chng Rwy Chng Slam Dnk
NCA - Crew Response At The Window
Force a Ldg Go - Around

Other Targeted Successes
Checklist protocol errors
Coordination with AQP
Captain Leadership performance
Intentional Non-Compliance errors
FOQA validation/coordination
Validated our change process
THE AIR NEW ZEALAND 1998
LINE ORIENTATED SAFETY
AUDIT
Captain Chris Kriechbaum

CRM Programme Manager (NZ)
US
600 pilots, 1600 cabin crew

All Boeing fleets, 7 x 747, 13 x 767, 16 x 737
97 sectors audited
No QAR data
WHY?
Pro-active approach to safety

Needed facts not opinions
Needed to establish where our programme sat
with the rest of the international community
Down under inferiority complex??
THE EXPERIENCE
Enjoyable
Management commitment initially a challenge
Dont underestimate the administrative

support required
THE RESULTS
Generally extremely positive
Automation issues were a dominant theme
Recommendations for procedure changes
Culture issues
THE BENEFITS
Established a baseline
Elevated CRM profile with in the Company
Credibility in International Aviation Community
Removed the scatter gun effect of training
Combined Training
Targeted Error Management refocus
Updated Command Course
SOP changes
Just Culture
Value for money
Justification of Capital and other Expenditure.
WHERE TO FROM HERE ?
A continuous
STANDARDISATION
cycle of
improvement
TRAINING SAFETY
L.O.S.A.
O.O.R.s
Q.A.R.s
LOSA
A Natural Extension?
Simon Henderson
Ansett Australia
Operations and Standards
nt
c ie
ffi
e
nd
a
f e
S a
Operations and Standards
Assumption
Operations
Real world Check
performance is Data
accurately reflected
by check
performance
Tension and Conflict
Collected Data
Training Need
Tension and Conflict
Collected Data
nt
e
em
ag
an
M
i sk
R
Training Need
Traditional Data From?
Inc k
ide ec
n ts Ch
Operations
Safety
Observe Examine
Data
Reports
? ?
Additional Data From?
Inc k
ide ec
n ts Ch
Operations
Safety
Observe Examine
Data
Reports
Aud
it
Q AR
? ? LO
SA
Data Management
What do you need to do?

Evaluate the training system ability to deliver the product
needed
Capture compliance related information
Capture/measure individual standards
Capture/measure the reliability of the evaluators in the
training system
Additional Data From?
Inc k
ide ec
n ts Ch
Operations
Safety
Observe Examine
Data
Reports
Aud
it
AR
Q LO
RISK SA
MANAGEMENT
Risk Management
Crew Capability
Availability
and Demand
Risk
Likelihood and Consequence
Risk Management
From LOSA database. Likelihood based on

Threat
LOSA observations, QAR, etc. Likelihood
Consequences based on SME input, LOSA

Threat
consequentials, Safety etc. Consequence
Capacity based on Situation. From

Crew
Standards database, LOSA, etc. Capacity
Demand based on the Threat. From

Crew
Standards Database, incident review etc. Demand
Match Data to the Training
Need
Collected Data
nt
c ie
ffi
e
nd
a
f e
S a
Training Need
Captain Simon Henderson
Flight Training Project Manager
Ansett Australia
Simon_Henderson@ansett.com.au
SHORT COURSE
OBSERVER TRAINING
Short Course Observer Training
LOSA Week, Cathay Pacific, Hong Kong
March 12-14, 2001
The University of Texas Human Factors

Research Project
Reasons for a Short Course
Not really training it is more informational
Provides you with an in-depth understanding on the type

of data collected through a LOSA
An opportunity to exchange ideas to improve the current

process
Lets keep it interactive ask questions anytime

Conducting a LOSA Observation
Observation Tools / Instruments
1. LOSA Observation Rating Form 9.0
Demographics
Narratives
Threat and error countermeasures (CRM behavioral markers)
Descent / Approach / Land technical data
Threat and error management
Crew interview
2. LOSA Code Book 9.0
3. Pocket Notepad and Quick Reference Card

Observation Guidelines
Goal #1: - Strive to be a fly on the wall observer
Main issue when does an observer speak up?
Think of yourself as a jump seat rider on another airline
Avoid filling out the observation form in the cockpit take notes
Dont attempt to debrief unless asked
Keep all data de-identified and confidential
If observers fail to be flies on the wall LOSA will fail

Observation Time
For flights under three hours, stay in the cockpit
For flights over three hours, only observe:

Predeparture / Taxi-out
Takeoff / Climb
45 minutes after TOC (Good time to conduct the interview)
45 minutes before TOD
Descent / Approach / Land
Taxi-in
The Captain Brief
Ask for permission to ride and observe
Stress that data is de-identified and confidential
Ask if you can take notes on a notepad
Determine the best time to conduct the crew interview
Set up a schedule for time away from the cockpit
Inform the Lead Flight Attendant

LOSA Observation Form
Walk-Through
Demographics
(Page 1)
Demographics Top Three
1. Observer ID stay consistent
2. Dont forget the observation number - start with #1
3. Collect demographics as at the start of the crew interview

Narratives and Threat and Error
Countermeasures
(Pages 2-7)
Narratives Top Three
1. Narratives should provide a context - what and how
1. What did the crew do well?
2. What did the crew do poorly?
3. How did the crew perform when confronted with threats, errors,
or other significant event?
2. Keep the narrative concise and simple
3. When in doubt, write in the narrative

Countermeasures Top Three
1. Only rate with confidence - Dont have to rate everything
2. Another way to think about the ratings..

1 = Poor Major contributor to failing a check ride
2 = Marginal Would require a debrief
3 = Good Standard performance
4 = Outstanding Training tape quality
3. If you give a 1,2, or 4 rating, justify it in the narrative

Technical Worksheet
(Page 4)
The Most Dangerous Phase
Data from the Descent / Approach / Land Phase
Most accidents worldwide (ALAR)
Most threats
Most mismanaged threats
Most crew errors
Most mismanaged crew errors
Most variability in threat and error countermeasure ratings

The Blue Box
Brief TOD
Transition
4 Altitude
10000 ft.
Slow and
Approach
FAF/OM Bottom
Lines
Flare / Touchdown
Descent/Approach/Land Top Three
1. Fill out the technical data worksheet first and use The
Blue Box landmarks in your narrative
2. If the approach was unstable, laundry list the

contributing factors
3. We defer tell us what is going on in The Blue Box

Threat and Error Management
Threats
Threat Induced
Inconsequential Threat Management Incident or
Accident
Error
Crew Error
Responses
Error Outcome
Error Induced
Incident or
Accident
Threat Management Worksheet
(Page 8)
Threats Defined
Threat Events or errors that originate outside the influence of the
flight crew but require their attention to maintain safety
1. Environmental Threats 4. Crew Support Threats

Adverse WX Dispatch events / errors
Terrain Ground events / errors
Airport Conditions MX events / errors
Heavy traffic / TCAS events
5. Operational Threats
Time Pressures
2. ATC Threats
Irregular Operations
Command events / errors
Radio congestion / poor
Language difficulties reception
3. Aircraft Threats 6. Cabin Threats

Malfunctions Cabin events / FA errors
Automation events
Threat Management Top Three
1. If a threat induced a crew error, enter NO for
effectively managed
2. See more than five threats? staple another worksheet

and renumber
3. When in doubt, make sure you include answers the

following somewhere on the worksheet
1. What was the threat?
2. How did the crew respond to the threat?
3. What was the outcome of the threat?
Error Management Worksheet
(Pages 9-11)
Threats
Threat Induced
Accident
Error
Crew Error
Responses
Error Outcome
Error Induced
Incident or
Accident
Crew Error Types
Threats
Threat Induced
Accident
Intentional
Noncompliance
Procedural
Communication
Proficiency
Decision
Crew Error
Responses
Error Induced
Incident or
Error Outcome Accident
Error Types Defined
1. Intentional Noncompliance intentional deviations from SOP
Ex) Performing a checklist from memory
2. Procedural Followed procedures with incorrect execution

Ex) Wrong attitude setting dialed

Ex) Miscommunication with ATC
4. Proficiency Error due to the lack of knowledge or skill

Ex) Lack of knowledge with automation
5. Decision Crew decision unbounded by procedures that

unnecessarily increased risk
Ex) Unnecessary navigation through adverse weather
START
Was the error associated with a: YES
- Miscommunication Communication Error
- Misinterpretation
- Failure to communicate Go to Page 5
pertinent information
NO

- Lack of knowledge Proficiency Error
- Lack of stick and rudder
proficiency Go to Page 5
NO
Was the error associated with a:
- Decision that increased risk in YES Operational Decision Error
which there were no written
procedures to follow Go to Page 6
NO
Was the error associated with a: Intentional Noncompliance

- Intention not to follow written YES
Error
regulations or company
procedures Go to Page 2
NO
Procedural Error
Go to Page 3 and 4
Crew Error Responses
Threats
Threat Induced
Accident
Error
Detected & Action

Detected & Ignored
Undetected
Error Induced
Error Outcome Incident or
Accident
Crew Error Responses Defined
Detected and Action (Trap or Exacerbate)
Error is detected
Crew did something to manage or mismanage the error
Detected and Ignored (Fail to respond)

Error is detected or willingly committed
Crew intentionally elected not to manage the error
Undetected (Fail to respond)

Error is undetected
No error management takes place
Error Outcomes
Threats
Threat Induced
Accident
Error
Error Management
Undesired Aircraft
State
Undesired Aircraft Error Induced

State Management Incident or
Accident
Undesired Aircraft States Defined
Undesired aircraft states A state in which the aircraft is placed in
a compromised situation that increases risk to safety
Lateral deviation Unstable approach
Vertical deviation Abrupt aircraft control
Speed too high Long landing
Speed too low Firm landing
Incorrect aircraft configurations Wrong taxiway / ramp
Flight controls
Wrong runway
Systems
Fuel Runway incursion
Automation
Undesired Aircraft State Responses
Threats
Threat Induced
Accident
Error
Crew Error
Responses
Undesired Aircraft
State
Detected & Action Error Induced

Detected & Ignore Incident or
Undetected Accident
Threats
Threat Induced
Accident
Crew Error
Crew Error
Responses
Undesired Aircraft
State
Crew Undesired Error Induced

Aircraft State Incident or
Responses Accident
Error Management Top Three
1. Dont spend too much time on the codes it is not an
exact science
2. See more than five errors? staple another worksheet

and renumber

1. What was the crew error?
2. How did the crew respond to the error?
3. What was the outcome of the error?
LOSA Observation Form
EXAMPLE
Observer Information
Observer ID (Employee number) 3059
Observation Number #1
Crew Observation Number

(e.g., 1 of 2 indicates segment one for a crew that you observed across two segments) 1 Of 1
Flight Demographics
City Pairs (e.g., PIT-CLT) PIT - LAX
A/C Type (e.g., 737-300) B-757
Pilot flying (Check one) CA FO X
Time from Pushback to Gate Arrival (Hours:Minutes) 4:55

Late Departure?
(Yes or No) Yes How late?
(Hours:Minutes)
Crew Demographics
CA FO SO/FE Relief 1 Relief 2
Base PIT PIT
Years experience for all airlines 35 5
Years in position for this A/C 7 1 month
Years in automated A/C
(FMC with VNAV and LNAV) 12 1 month
First LEG the crew has EVER flown together
Crew Familiarity First DAY the crew has EVER flown together
(Check one)
Crew has flown together before X
The University of Texas Human Factors Research Project - LOSA Observation Form Version 9.0 1
Predeparture / Taxi-Out
Your narrative should provide a context. What did the crew do well? What did the crew do poorly? How did the crew perform
Narrative when confronted with threats, crew errors, and significant events? Also, be sure to justify your behavioral ratings.
The CA established a great team climate positive with open communication. However, he seemed to be in a rush
and not very detail oriented. The FO, who was relatively new to the A/C, tried to keep up but fell behind at
times. The CA did not help the cause by interrupting the FO with casual conversation (marginal workload
management).
All checklists were rushed and poorly executed. The CA was also lax verifying paperwork. This sub-par behavior
contributed to an undetected error - the FO failed to set his airspeed bugs for T/O (poor monitor/cross-
check). The Before Takeoff Checklist should have caught the error, but the crew unintentionally skipped over
that item. During the takeoff roll, the FO noticed the error and said, Missed that one.
The Captains brief was interactive but not very thorough (marginal SOP briefing). He failed to note the
closure of the final 2000 of their departing runway (28R) due to construction. Taxiways B7 and B8 at the end of
the runway were also out. The crew was marked poor in contingency management because there were no plans
in place on how to deal with this threat in the case of a rejected takeoff. Lucky it was a long runway.
1 2 3 4
Poor Marginal Good Outstanding
Observed performance had safety Observed performance was barely Observed performance was Observed performance was truly
implications adequate effective noteworthy
Planning Behavioral Markers Rating
2
The required briefing was interactive and - Concise, not rushed, and met SOP requirements
SOP BRIEFING
operationally thorough - Bottom lines were established
3
Operational plans and decisions were - Shared understanding about plans - Everybody on
PLANS STATED
communicated and acknowledged the same page
3
Roles and responsibilities were defined for - Workload assignments were communicated and
WORKLOAD ASSIGNMENT
normal and non-normal situations acknowledged
1
CONTINGENCY Crew members developed effective strategies to - Threats and their consequences were anticipated
MANAGEMENT manage threats to safety - Used all available resources to manage threats
Execution Behavioral Markers Rating
1
Crew members actively monitored and cross- - Aircraft position, settings, and crew actions were
MONITOR / CROSS-CHECK
checked systems and other crew members verified
2
WORKLOAD Operational tasks were prioritized and properly - Avoided task fixation
MANAGEMENT managed to handle primary flight duties - Did not allow work overload
3
Crew members remained alert of the
VIGILANCE - Crew members maintained situational awareness
environment and position of the aircraft
- Automation setup was briefed to other members
AUTOMATION Automation was properly managed to balance
- Effective recovery techniques from automation
MANAGEMENT situational and/or workload requirements
anomalies
Review / Modify Behavioral Markers Rating

Existing plans were reviewed and modified when - Crew decisions and actions were openly analyzed
EVALUATION OF PLANS
necessary to make sure the existing plan was the best plan
3
Crew members asked questions to investigate - Crew members not afraid to express a lack of
INQUIRY
and/or clarify current plans of action knowledge - Nothing taken for granted attitude
Crew members stated critical information and/or
ASSERTIVENESS - Crew members spoke up without hesitation
solutions with appropriate persistence
Takeoff / Climb
Normal takeoff besides one error. As the crew started to clean up the aircraft, the FO called flaps up before
the flap retraction speed. The CA trapped the error and did not retract the flaps until the proper speed.
After passing 10000 all the way up to the TOC, the CA and FO failed to cross-verify multiple altitude changes.
There was no intention on part of the CA to verify. In addition, since it happened multiple times, the observer
coded it as an intentional noncompliance.
1 2 3 4
1
3
2
anomalies

EVALUATION OF PLANS
INQUIRY
Cruise
Routine no comments
Descent / Approach / Land Technical Worksheet
Descent (Above 10,000 ft.)

1 Was the approach briefed before the TOD? (Yes / No) Yes
2 Did the crew begin the descent before or at the FMS TOD? (Yes / No) Yes
Did the aircraft get significantly above/below the FMS or

3
standard path? (Yes / No) No
If Yes, explain in the narrative the cause and whether the
crew tried to regain the path.
Approach and Land (Below 10,000 ft.)

Instrument backup on Yes X
Visual X visual approach?
(Check One) No
4 Approach flown?
(Check one) Type of precision
Precision
approach
Type of nonprecision
Nonprecision
approach
5 Approach: Hand flown or Automation flown? Hand-flown
Did the aircraft get significantly above/below a desirable

6
descent path? (Yes / No) Yes
If Yes, explain in the narrative the cause and whether the
crew tried to regain the path.
Close to or at minimum maneuvering

speed
During flap extension, flaps
X
Close to or at the maximum flap
7 were generally extended: extension speed
(Check one)
Above maximum flap extension speed
(If this happens, be sure to describe in the narrative)
8 Weather (Check One) VMC X IMC
Stabilized Approach Parameters 1500 AFE 1000 AFE 500 AFE

Target airspeed between 5 and +15 Yes Yes Yes
Vertical speed 1000 fpm Yes Yes Yes

9
Engines spooled Yes Yes Yes
Landing configuration
Yes Yes Yes
(Final flaps / gear down)
On proper flight path (G/S and localizer) Yes Yes Yes
Descent / Approach / Land The Blue Box
Briefing TOD
Transition
4
Altitude
10000 ft.
Slow and configure
FAF / OM Stabilized Approach

Bottom Lines
Flare / Touchdown
Think blue box. Describe significant events from the TOD to landing using the picture above to define landmarks. Talk about
Narrative how the crew performed when confronted with threats and crew errors. Also, be sure to justify your behavioral ratings.
Briefing to TOD The CA and FO did a nice job with the approach brief, which was completed by the TOD. Much
better than their takeoff brief. They expected runway 25L from the Civet Arrival for a straight-in visual
approach. Jepp charts were out, contingencies talked about, and everything was by the book. The FO asked a
lot of questions and the CA was patient and helpful. Nicely done!
10000 to slowing and configuring ATC cleared the crew to 25L, but at 8000, ATC changed us to the Mitts
Arrival for runway 24R due to a slow moving A/C on 25L. The CA changed the arrival and approach in the FMC
and tuned the radios. As soon as everything was clean, ATC called back and told the crew they could either land
on 25L or 24R at their discretion. Since time was a factor, the crew discussed and decided to stick with the
approach into 24R. The crew was flexible and the CA did a nice job assigning workload. He directed the FO fly
the plane while he checked everything over one more time.
The crew was also better monitors and cross checkers. However, their execution of checklists was still a little
sloppy late and rushed.
The crew did a nice job staying vigilant with heavy traffic in the area used ATC and TCAS effectively.
Bottom lines to Flare / Touchdown The approach was stable, but the FO let the airplane slip left, which
resulted in landing left of centerline. Since the FO was new to this aircraft (1 month flying time), the observer
chalked it up to a lack of stick and rudder proficiency. .
Taxi-in The crew did a great job navigating taxiways and crossing the active 24L runway. Good vigilance and
teamwork.
1 2 3 4
Planning Behavioral Markers Rating
4
The required briefing was interactive and - Concise, not rushed, and met SOP requirements
SOP BRIEFING
operationally thorough - Bottom lines were established
4
Operational plans and decisions were - Shared understanding about plans - Everybody on
PLANS STATED
communicated and acknowledged the same page
4
Roles and responsibilities were defined for - Workload assignments were communicated and
WORKLOAD ASSIGNMENT
normal and non-normal situations acknowledged
3
CONTINGENCY Crew members developed effective strategies to - Threats and their consequences were anticipated
MANAGEMENT manage threats to safety - Used all available resources to manage threats
2
3
3
3
anomalies
4
EVALUATION OF PLANS
3
INQUIRY
Overall Flight
Narrative This narrative should include your overall impressions of the crew.
Overall, the crew did a marginal job with planning and review/modify plans during predeparture. However, during
the descent/approach/land phase, it was excellent. Their execution behaviors were marginal to good for the
entire flight.
While the takeoff brief was marginal, the CA made an outstanding approach brief. Open communication was not
a problem. Good flow of information when the flights complexity increased with the late runway change. They
really stepped it up.
The big knock against this crew involved checklists, cross verifications, and all monitoring in general. They were
a little too complacent during low workload periods (e.g., No altitude verifications during climb). The CA set a
poor example in this regard.
During predeparture, the CA introduced an unnecessary element of being rushed, which compromised workload
management. However, his decisiveness and coordination in the descent/approach/land phase kept his leadership
from being marked marginal.
1 2 3 4
Overall Behavioral Markers Rating

4
COMMUNICATION Environment for open communication was - Good cross talk flow of information was fluid,
ENVIRONMENT established and maintained clear, and direct
3
Captain showed leadership and coordinated flight - In command, decisive, and encouraged crew
LEADERSHIP
deck activities participation
Yes Rating
Did you observe a flight attendant briefing on No
the first leg of the pairing? (Check one)
X
No opportunity to
observe
CA FO
Contribution to Crew Effectiveness 2 3
Rating
3
Overall Crew Effectiveness
Threats Events or errors that originate outside the influence of the flightcrew but require
active crew management to maintain safety.
Threat Description Threat Management
Phase of
Threat ID
Flight
Effectively
Threat 1 Predepart/Taxi managed?
Describe the threat How did the crew manage or mismanage the threat?
Code 2 Takeoff/Climb
3 Cruise (Yes / No)
4 Des/App/Land
5 Taxi-in
Threat mismanaged CA failed to talk about the construction

Runway and taxiway construction on
T1 4 1 No and closures in his brief. No plans in place in the event of a
their departing runway (final 2000)
RTO.
Late ATC runway change - changed
Threat managed CA reprogrammed the FMC, handled the
T2 runway to 24R from 25L due to a slow 50 4 Yes radios, and placed emphasis on the FO to fly the aircraft.
moving aircraft on 25L
ATC called back and told the crew that Threat managed CA asked for the FOs preference. They
T3 it was at their discretion to land on 24R 50 4 Yes mutually decided to continue the approach into 24R because it
or 25L was already in the box.
Threat managed The crew closely monitored the traffic

T4 Heavy congestion going into LAX 3 4 Yes with the help of ATC and TCAS.
T __
Threat Codes
Departure / Arrival Threats Operational Threats Cabin Threats Crew Support Threats
1 Adverse weather / turbulence / IMC 30 Operational time pressure delays, 40 Cabin event / distraction / interruption 80 MX event
2 Terrain OTP, late arriving pilot or aircraft 41 Flight attendant error 81 MX error
3 Traffic Air or ground congestion, TCAS warnings 31 Missed approach 82 Ground handling event
4 Airport construction, signage, ground conditions 32 Flight diversion ATC Threats 83 Ground crew error
5 TCAS RA/TA 33 Unfamiliar airport 50 ATC command challenging clearances, late changes 84 Dispatch/ paperwork event
34 Other non-normal operation events 51 ATC error 85 Dispatch / paperwork error
Aircraft Threats max gross wt. T/O, rejected T/O 52 ATC language difficulty 86 Crew scheduling event
20 Aircraft malfunction 53 ATC non-standard phraseology 87 Manuals / charts incomplete /
21 Automation event or anomaly 54 ATC radio congestion incorrect
22 Communication event - radios, ATIS, ACARS 55 Similar call signs
99 Other Threats
Error Description Error Response / Outcome
Phase of Error Type
Error Crew Error Error
Error ID
flight
1 Intentional Code Who Who Response Outcome
Describe the crew error and associated 1 Predepart/Taxi Noncompliance
committed detected
undesired aircraft states 2 Takeoff/Climb 2 Procedural Use 1 Detected & Action 1 Inconsequential
3 Cruise 3 Communication Code the error? the error? 2 Detected & Ignored 2 Undesired state
4 Des/App/Land 4 Proficiency Book 3 Undetected 3 Additional error
5 Taxi-in 5 Decision
E1 FO failed to set his airspeed bugs. 1 2 211 2 7 3 3
Error Management Undesired Aircraft State

Associated Crew Undesired Undesired
Error ID
with a Undesired State Response Aircraft State

Who
threat? Aircraft Outcome
How did the crew manage or mismanage the error? detected
State 1 Detected & Action
the state? 2 Detected & Ignored
(If Yes, enter Code 1 Inconsequential
Threat ID) 3 Undetected 2 Additional error
E1 No Error chain to E2
Who Committed /
Undesired Aircraft State Codes
Detected Codes
Flightcrew Other people Configuration States Aircraft Handling States All Phases Approach / Landing States
1 CA 8 ATC 1 Incorrect A/C configuration - flight controls, brakes, 40 Vertical deviation 80 Deviation above G/S or FMS path
2 FO 9 Flight attendant thrust reversers, landing gear- 41 Lateral deviation 81 Deviation below G/S or FMS path
3 SO / FE 10 Dispatch 2 Incorrect A/C configuration systems (fuel,
4 Relief Officer 11 Ground electrical, hydraulics, pneumatics, air- 42 Unnecessary WX penetration 82 Unstable approach
5 Jumpseat Rider 12 MX conditioning, pressurization, instrumentation) 43 Unauthorized airspace penetration 83 Continued landing - unstable approach
3 Incorrect A/C configuration automation
6 All crew Aircraft 4 Incorrect A/C configuration - engines 44 Speed too high 84 Firm landing
members 20 Aircraft 45 Speed too low 85 Floated landing
systems Ground States 86 Landing off C/L
7 Nobody 20 Proceeding towards wrong runway 46 Abrupt aircraft control (attitude) 87 Long landing outside TDZ
21 Runway incursion 47 Excessive banking
99 Other 48 Operation outside A/C limitations
22 Proceeding towards wrong taxiway / ramp 99 Other Undesired States
23 Taxiway / ramp incursion
24 Wrong gate
Phase of Error Type
Error ID
flight Who
1 Intentional Code Who Response Outcome
Describe the crew error and associated 1 Predepart/Taxi Noncompliance detected
committed
undesired aircraft states 2 Takeoff/Climb 2 Procedural Use the 1 Detected & Action 1 Inconsequential
3 Cruise 3 Communication Code the error? 2 Detected & Ignored 2 Undesired state
error?
5 Taxi 5 Decision
In running the Before Takeoff Checklist,

E2 1 2 200 2 7 3 1
the FO skipped the takeoff data item.
FO called flaps up prior to the flap

E3 2 2 299 2 1 1 1
retraction speed.

Error ID
with a Undesired Who State Response Aircraft State

threat? Aircraft detected Outcome
How did the crew manage or mismanage the error?
State the 1 Detected & Action
(If Yes, enter Code state? 2 Detected & Ignored 1 Inconsequential
Errors mismanaged The bug error should have been caught with the
Before Takeoff Checklist, but the FO unintentionally skipped that
E2 No item. All checklists during this phase were poorly executed. The FO
caught the error during the takeoff roll.
Error managed CA saw that the aircraft was not at the proper
E3 No speed and waited to retract the flaps. Good monitoring in this case.
Phase of Error Type
Error ID
flight Who
1 Intentional Code Who Response Outcome
Describe the crew error and associated 1 Predepart/Taxi Noncompliance detected
committed
3 Cruise 3 Communication Code the error? 2 Detected & Ignored 2 Undesired state
error?
5 Taxi 5 Decision
CA and FO failed to verify multiple

E4 2 1 140 1 6 2 1
altitude changes.
FO, who was new to the aircraft, let it slip

a little to the left during the final
E5 4 4 402 2 6 1 2
approach. Resulted in landing left of the
centerline.

Error ID

E4 No No error management intentional error
Error mismanaged FO tried to correct but still landed left of the

E5 No centerline. Approach was stable and they made the first high-speed 86 6 1 1
taxiway. The CA did not verbalize the deviation during the approach.
52
LOSA Crew Interview
1) Training
a) Is there a difference in how you were trained, and how things really go in line
operations?
b) If so, why?
2) Standardization
a) How standardized are other crews that you fly with?
b) If there is a lack of standardization, what do you think is the reason(s) for
procedural non-compliance?
3) Automation
a) What are the biggest automation gotchas for this airplane?
4) Overall safety improvements concerns and suggestions for improvement

a) Flight Ops
b) Dispatch
c) Airports and ATC
d) SOPs
LOSA Error Code Book 9.0
The University of Texas Human Factors Research Project
Klinect, J. R, Wilhelm, J. A., & Helmreich, R. L

Error Type Decision Tree
START
- Miscommunication YES
Communication Error
- Misinterpretation
NO

- Lack of knowledge YES Proficiency Error
- Lack of stick and rudder
proficiency Go to Page 5
NO

- Decision that increased risk in YES Operational Decision Error
procedures to follow Go to Page 6
NO

Intentional
- Intention not to follow written YES Noncompliance Error
procedures
Go to Page 2
NO
Procedural Error
Go to Page 3 and 4
University of Texas Human Factors Research Project Version 9.0 10/1//00 1

Intentional Noncompliance Error Codes
Sterile Cockpit Errors 142 Failure to cross-verify altimeter settings
100 Sterile cockpit violation
Hard Warning Errors
Callout Errors 160 Failure to respond to GPWS warnings
104 Omitted takeoff callouts (i.e., V-speeds) 161 Failure to respond to TCAS warnings
105 Omitted climb or descent callouts 162 Failure to respond to overspeed warning
106 Omitted approach callouts
Briefing Errors
Crew to ATC Errors 170 Omitted takeoff briefing
109 Altitude deviation without ATC clearance 171 Omitted approach briefing
110 Course or heading deviation without ATC clearance 172 Omitted flight attendant briefing (only for the first
(deviation more than 20 degrees) flight of a trip or crew change)
111 Use of nonstandard ATC phraseology 173 Omitted engine-out briefing
112 Omitted position report to ATC
113 Omitted non-radar environment report to ATC 179 Intentional failure to arm spoilers
114 Omitted call signs to ATC
Approach Errors
Checklist Errors 180 Failure to execute a go-around after passing
120 Checklist performed from memory procedural bottom lines of an unstable approach
121 Completed checklist not called complete 181 Speed deviation without ATC clearance
122 Checklist not performed to completion 183 Intentionally flying below the G/S
123 Use of nonstandard checklist protocol (i.e., use of 184 PF makes own flight control settings
nonstandard responses)
124 Omitted checklist
125 Self-performed checklist no challenge or response Automation and Instrument Setting Errors
126 Omitted abnormal checklist 185 PF makes own MCP changes
127 Self initiated checklist not called for by PF 186 PF makes own FMC changes
128 Self initiated checklist not called for by CA 187 Failure to set altitude alerter
129 Checklist performed late or at wrong time 189 Setting altimeters before the transition altitude
190 Using equipment placarded inoperative
Cross-Verification Errors
140 Failure to cross-verify MCP / altitude alerter Other Noncompliance Errors
changes 195 Taxi-in or out without a wing walker
196 A/C operation with unresolved MEL item
141 Failure to cross-verify FMC/CDU changes before 199 Other noncompliance errors not listed in the
execution code book

Procedural Error Codes
Checklist Errors 245 Wrong MCP navigation select setting
200 Missed checklist item (NAV/GPS/ILS/VOR switch)
201 Wrong checklist performed 246 PF makes own MCP changes
202 Checklist performed late or at the wrong time 247 Wrong MCP setting on the auto-throttle switch
203 Forgot to call for checklist
206 Wrong response to a challenge on a checklist (i.e., item Flight Management Computer / Control Display Unit
not checked that was responded to as checked) Errors
207 Completed checklist not called complete 249 Failure to cross-verify FMC/CDU changes / position
209 Omitted checklist 250 Wrong waypoint / route settings entered into the FMC
233 Omitted abnormal checklist 251 Failure to execute a FMC mode when needed
252 Wrong mode executed in the FMC
Primary Instrument or Panel Errors 253 Wrong mode left engaged in the FMC
210 Wrong altimeter settings 254 Wrong present position entered into the FMC
211 Wrong bug settings (i.e., airspeed or altimeter) 255 Wrong weights / balance calcs entered into the FMC
212 Failure to set altitude alerter 256 Wrong speed setting entered into the FMC
213 Failure to cross-verify altimeter settings 257 PF makes own FMC changes
214 Failure to cross-verify altitude alerter 258 Wrong FMC format for input
205 Wrong approach selected in the FMC
Lever and Switch Errors 204 Other wrong CDU entries / settings
215 Failure to extend the flaps on schedule
216 Failure to retract the flaps on schedule 259 Wrong nav radio frequency
217 Wrong display switch setting
218 Failure to leave thrust reversers extended Radio Errors
219 Failure to lower the landing gear on schedule 260 Wrong ATIS frequency dialed
220 Failure to bring up the landing gear on schedule 261 Wrong ATC frequency dialed
221 Failure to extend the speed brakes on landing 262 Wrong squawk
222 Failure to retract the speed brakes
223 Failure to engage thrust reversers on landing Documentation Errors
224 Failure to retract thrust reversers after landing 263 Wrong ATIS information recorded
225 Failure to turn on the landing lights 264 Wrong runway information recorded
226 Wrong fuel switch setting 265 Wrong V-speeds recorded
227 Failure to turn on TCAS 266 Wrong weights and balance information recorded
228 Failure to turn on the fasten seat belt sign 267 Wrong fuel information recorded
229 Failure to arm spoilers 268 Missed items on the documentation (flight plan,
230 Failure to turn on the A/C packs (no pressurization) NOTAMS, or dispatch release)
231 Wrong panel setup for an engine start 269 Misinterpreted items on the documentation (flight
278 Wrong power settings for T/O plan, NOTAMS, or dispatch release)
279 Wrong autobrake setting 270 Wrong time calculated in the flight plan
232 Other incorrect switch or lever settings 271 Wrong clearance recorded
Mode Control Panel Errors

234 Failure to cross-verify MCP / altitude alerter changes Procedural Errors Continued
235 Wrong MCP altitude setting dialed
236 Wrong MCP vertical speed setting dialed on Next Page
237 Wrong MCP speed setting dialed
238 Wrong MCP course setting dialed
239 Wrong MCP heading setting dialed
240 Wrong setting on the MCP autopilot or FD switch
241 Wrong MCP mode executed
242 Wrong MCP mode left engaged
243 Manual control while a MCP mode is engaged
244 Failure to execute a MCP mode when needed

Callout Errors Hard Warning Errors
275 Omitted takeoff callouts (i.e., V-speeds) 293 Failure to respond to GPWS warnings
276 Omitted climb or descent callouts 294 Failure to respond to TCAS warnings
277 Omitted approach callouts
Job Sequence Errors Briefing Errors

280 Executing the correct job procedures out of sequence 272 Incomplete flight attendant briefing
273 Incomplete cruise briefing
Handling Errors 274 Incomplete approach briefing
281 Unintentional lateral deviation 295 Omitted takeoff briefing
282 Unintentional vertical deviation 296 Omitted approach briefing
286 Unintentional speed deviation 297 Omitted flight attendant briefing
298 Omitted engine-out briefing
Ground Navigation Errors
283 Attempting or actually turning down the wrong runway Other Procedural Errors
284 Attempting or actually turning down the wrong ramp / 299 Other procedural errors not listed in the code
taxiway / gate book
287 Attempting or actually lining up for the incorrect runway
288 Attempting or actually lining up off C/L
289 Failure to execute a go-around after passing procedural
bottom lines of an unstable approach
290 Missed runway
291 Missed taxiway
292 Missed gate

Communication Error Codes
Crew to ATC Errors 320 Wrong taxiway communicated
300 Wrong readbacks or callbacks to ATC 321 Wrong runway communicated
301 Missed ATC calls 322 Wrong takeoff callouts communicated
302 Omitted call signs to ATC 323 Wrong climb and descent callouts communicated
303 Failure to give readbacks or callbacks to ATC 324 Wrong approach callouts communicated
305 Omitted position report to ATC 325 Wrong gate assignment communicated
306 Omitted non-radar environment report to ATC 335 Crew miscommunication that lead to a
307 Misinterpretation of ATC instructions misinterpretation
309 Crew omitted ATC call 336 Wrong engine out procedures stated
310 Missed instruction to hold short
Other Communication Errors
350 Misinterpretation of ATIS
399 Other communication errors not listed in the
Crew to Crew Errors code book
319 Wrong airport communicated
Proficiency Error Codes

400 Lack of systems knowledge
401 Lack of automation knowledge
402 Lack of stick and rudder proficiency
403 Lack of knowledge to properly contact ATC
404 Lack of procedural knowledge
405 Lack of weather knowledge
406 Lack of knowledge of standard ATC phraseology
407 Lack of knowledge to contact company (i.e., gate assignments)
499 Other knowledge or proficiency based errors not listed in the code book

Operational Decision Error Codes
Descent and Approach Errors
500 Failure to execute a go-around before reaching procedural bottom-lines
501 Unnecessary low maneuver on approach
502 Approach deviation (lateral or vertical) by choice
503 Decision to start the descent late
520 Operating at the edge of the performance envelope (no buffer for error)
Navigation Errors
510 Navigation through known bad weather that unnecessarily increased risk (i.e., thunderstorms or wind shear)
512 Decision to navigate to the wrong assigned altitude
513 Decision to navigate on the incorrect heading or course
514 Decision to navigate without ground clearance
521 Speed too high for operating environment
ATC Errors
530 Accepting instructions from ATC that unnecessarily increased risk
531 Making a request to ATC that unnecessarily increased risk
532 Failure to verify ATC instructions
533 Altitude deviation without ATC notification
534 Course or heading deviation without ATC clearance
535 Accepting a visual in nonvisual conditions
Crew Interaction Errors

540 Non-essential conversation at inappropriate times
Automation Errors
550 FMC over-reliance used at inappropriate times
551 FMC under-reliance not used when needed
552 Heads down FMC operation
553 Discretionary omission of FMC data (e.g., winds)
Instrument Errors
560 Lack of weather radar use
Checklist Errors
570 Failure to complete a checklist in a timely manner (i.e., after takeoff checklist)
Paperwork Errors
590 Failure to cross-verify documentation or paperwork
Other Operational Decision Errors

599 Other operational decision errors not listed in the code book

Threat and Error Management Worksheet Codes
Threat Codes
20 Aircraft malfunction 53 ATC non-standard phraseology 87 Manuals / charts incomplete /
99 Other Threats
Who Committed /
Detected Codes
4 Relief Officer 11 Ground electrical, hydraulics, pneumatics, air-conditioning, 42 Unnecessary WX penetration 82 Unstable approach
5 Jumpseat Rider 12 MX pressurization, instrumentation) 43 Unauthorized airspace penetration 83 Continued landing - unstable approach
systems 86 Landing off C/L
7 Nobody
Ground States 46 Abrupt aircraft control (attitude) 87 Long landing outside TDZ
20 Proceeding towards wrong runway
47 Excessive banking
99 Other 21 Runway incursion
48 Operation outside A/C limitations
24 Wrong gate

THREAT AND ERROR MANAGEMENT
EXERCISES
Threat and Error Management Exercises
Version 9.0
Klinect, J.R., Wilhelm, J.A., & Helmreich, R.L.
The University of Texas Human Factors Research Project1
March 7th, 2001
1 This research was supported by the Federal Aviation Administration Grant Number 99-G-004, Robert L. Helmreich, PhD, Principal Investigator.
Threat and Error Management Worksheet Codes
Threat Codes
20 Aircraft malfunctions 53 ATC non-standard phraseology 87 Manuals / charts incomplete /
99 Other Threats
Who Committed /
Detected Codes
4 Relief Officer 11 Ground electrical, hydraulics, pneumatics, air- 42 Unnecessary WX penetration 82 Unstable approach
5 Jumpseat Rider 12 MX conditioning, pressurization, instrumentation) 43 Unauthorized airspace penetration 83 Continued landing - unstable approach
systems Ground States 86 Landing off C/L
7 Nobody 20 Proceeding towards wrong runway 46 Abrupt aircraft control (attitude) 87 Long landing outside TDZ
21 Runway incursion 47 Excessive banking
99 Other 48 Operation outside A/C limitations
24 Wrong gate
The University of Texas Human Factors Research Project Version 9.0 2

Exercise One
Predeparture / Taxi-out The Captain requested an extra 3000 pounds of fuel for possible weather at the destination. After ground confirmed the fueling, the
First Officer alerted the Captain that they were 2000 pounds off. The Captain radioed back to ground and the remaining 2000 pounds was loaded.
Exercise Two
Predeparture / Taxi-out - While taxiing to the assigned runway, the First Officer performed the entire Taxi Checklist from memory. The Captain noticed and
ignored it. In the end, everything was set correctly.
Exercise Three
Takeoff / Climb - During climb and cleared to FL270, the First Officer heard the clearance but dialed 230 instead of 270 in the MCP. The Captain caught the
error on cross-verification.
Descent / Approach / Land During the approach, the crew had a hard time understanding ATC accents. At one point, the First Officer misunderstood a
controller and thought they had cleared them direct to the MAMAS waypoint when in actuality, they were only cleared to the closer OREGON waypoint. Before the
First Officer started to reprogram the FMC, the Captain stated he had heard something else. They called ATC back and verified the proper route.
Exercise Four
Descent / Approach / Land During a 30-degree bank on a visual approach, the First Officer allowed the aircraft to get 15 knots below minimum maneuvering
speed. The low speed had to be pointed out by the observer. The FO said thanks and immediately increased the speed. The observer noted that this was most
likely due to a lack of stick and rudder proficiency.
Exercise Five
Takeoff / Climb Before T/O, the Captain looked at the radar and noticed a red asymmetrical cell passing over the airport. In order to get to the destination
before the imposed curfew, the crew elected to takeoff when they could have waited a few minutes for it to pass over. The takeoff was a little rough but they got
through it with no significant problems. They arrived at the destination15 minutes early.
Exercise Six
Predeparture / Taxi-out and Takeoff During the after start flow, the FO forgot to turn on the packs to pressurize the aircraft. He would have caught it with the
After Start Checklist, but he skipped that item. At 8000 feet into climb, both pilots noticed that the plane was not pressurized. The First Officer promptly corrected
the error.
Exercise Seven
Takeoff / Climb During climb, the crew discovered that a fuel pump low pressure light was illuminated. All crewmembers were heads down working the problem
when ATC instructed them climb to FL350. The First Officer (PNF) read back FL350 but failed to make the change in the MCP. The crew also failed to point and
shoot the altitude change in the MCP. After a few moments, ATC noticed that the aircraft remained at FL270 and called the crew to correct the problem.
Exercise Eight
Descent / Approach / Land On the downwind leg of a visual approach at 6000ft., ATC asked them if they could make an immediate turn for the outer marker
and maintain 180 knots as they crossed over the marker. If they could, they would be number two for landing. They were already early, but the Captain accepted
the challenge without asking the First Officer (PF). The First Officer was noticeably reluctant at first, but he said nothing, made the close-in turn, and extended the
speed brakes to begin the approach. The FO was really pushed but he crossed over the marker on speed. At 1800, the Captain noticed that the speed brakes
were left extended. The First Officer corrected the problem and continued a stabilized approach.
Graduation Exercise
Descent / Approach / Land At 1500 feet during a precision approach in IMC weather, the Captain noticed he was more than a dot high on the glide slope. The
Captain corrected but he got a vertical speed call from the First Officer. At 1000, they had a sink rate of approximately 1500 fpm (Stabilized approach criteria
states vertical speed 1000 fpm at 1000 for a precision approach). The FO asked the Captain if he wanted to execute a go-around but he ignored him and
continued the approach. The result was an unstable approach with a long landing outside the touchdown zone.

Exercise One
Phase of
Threat ID
Flight Effectively
Threat 1 Predepart/Taxi
managed?
3 Cruise (Yes / No)
4 Des/App/Land
5 Taxi-in
T __
T __
Crew Error Description Crew Error Response / Outcome

Phase of Error Type
Error ID
Flight
Code Who Response Outcome
1 Intentional Who
Describe the crew error and associated 1 Predepart/Taxi detected
Noncompliance committed
the error?
3 Cruise 3 Communication Code error? 2 Detected & Ignored 2 Undesired state
E __
E __
Crew Error Management Undesired Aircraft State

Error ID

E __
E __

Exercise Two
Phase of
Threat ID
Flight Effectively
managed?
3 Cruise (Yes / No)
4 Des/App/Land
5 Taxi-in
T __

Phase of Error Type
Error ID
Flight
1 Intentional Who
the error?
E __
E __

Error ID

E __
E __

Exercise Three
Phase of
Threat ID
Flight Effectively
managed?
3 Cruise (Yes / No)
4 Des/App/Land
5 Taxi-in
T __

Phase of Error Type
Error ID
Flight
1 Intentional Who
the error?
E __
E __

Error ID

E __
E __

Exercise Four
Phase of
Threat ID
Flight Effectively
managed?
3 Cruise (Yes / No)
4 Des/App/Land
5 Taxi-in
T __

Phase of Error Type
Error ID
Flight
1 Intentional Who
the error?
E __
E __

Error ID

E __
E __

Exercise Five
Phase of
Threat ID
Flight Effectively
managed?
3 Cruise (Yes / No)
4 Des/App/Land
5 Taxi-in
T __
T __

Phase of Error Type
Error ID
Flight
1 Intentional Who
the error?
E __
E __

Error ID

E __
E __

Exercise Six
Phase of
Threat ID
Flight Effectively
managed?
3 Cruise (Yes / No)
4 Des/App/Land
5 Taxi-in
T __

Phase of Error Type
Error ID
Flight
1 Intentional Who
the error?
E __
E __

Error ID

E __
E __

Exercise Seven
Phase of
Threat ID
Flight Effectively
managed?
3 Cruise (Yes / No)
4 Des/App/Land
5 Taxi-in
T __

Phase of Error Type
Error ID
Flight
1 Intentional Who
the error?
E __
E __

Error ID

E __
E __

Exercise Eight
Phase of
Threat ID
Flight Effectively
managed?
3 Cruise (Yes / No)
4 Des/App/Land
5 Taxi-in
T __

Phase of Error Type
Error ID
Flight
1 Intentional Who
the error?
E __
E __
Crew Error Management Crew Undesired Aircraft State

Error ID

E __
E __

Graduation Exercise
Phase of
Threat ID
Flight Effectively
managed?
3 Cruise (Yes / No)
4 Des/App/Land
5 Taxi-in
T __

Phase of Error Type
Error ID
Flight
1 Intentional Who
the error?
E __
E __

Error ID

E __
E __

Threat and Error Management Worksheet Exercise Solutions

Solution: Exercise One
Phase of
Threat ID
Flight Effectively
managed?
3 Cruise (Yes / No)
4 Des/App/Land
5 Taxi-in
1 Threat managed Crew asked ground to load an

T1 WX at the destination Adverse 1 Yes
WX
extra 3000 lbs.
83
Incorrect fueling Threat managed FO caught the ground crew
T2 Ground 1 Yes
(2000 lbs. short) Crew error on a cross-check.
Error

Phase of Error Type
Error ID
Flight
1 Intentional Who
the error?
E __ NO ERRORS
Error ID

E __ NO ERRORS

Solution: Exercise Two
Phase of
Threat ID
Flight Effectively
managed?
3 Cruise (Yes / No)
4 Des/App/Land
5 Taxi-in
T __ NO THREATS

Phase of Error Type
Error ID
Flight
1 Intentional Who
the error?
120
FO ran the Taxi Checklist from 2 1
E1 1 1 Checklist 2 1
memory from FO CA
memory
E __

Error ID

No management intentional noncompliance error was

E1 No
detected and ignored by the Captain.
E __

Solution: Exercise Three
Phase of Flight
Threat ID
Effectively
Threat 1 Predepart/Taxi managed?
Describe the threat 2 Takeoff/Climb How did the crew manage or mismanage the threat?
Code 3 Cruise (Yes / No)
4 Des/App/Land
5 Taxi-in
Difficulty understanding ATC 52

T1 4 No Threat mismanaged Misinterpretation by FO.
accents ATC language

Phase of Error Type
Crew Error Error
Error ID
Flight
Error Code Who Response Outcome
1 Intentional Who
Describe the crew error and 1 Predepart/Taxi detected
associated undesired aircraft states 2 Takeoff/Climb 2 Procedural Use the 1 Detected & Action 1 Inconsequential
Code Book the error?
3 Cruise 3 Communication error? 2 Detected & Ignored 2 Undesired state
4 Des/App/Land 4 Proficiency 3 Undetected 3 Additional error
FO dialed an incorrect MCP 235 2 1

E1 2 2 Wrong MCP 1 1
altitude altitude dialed
FO CA
307 2 1
E2 FO misinterpreted a clearance 4 3 Misinterpret
FO CA
1 1
ATC instruct
Error ID

E1 No Error managed CA caught the error on a cross-check.
Error managed Captain trapped the error by staying in

E2 Yes T1
the ATC communication loop good monitoring.

Solution: Exercise Four
Phase of
Threat ID
Flight Effectively
managed?
3 Cruise (Yes / No)
4 Des/App/Land
5 Taxi-in
T __ NO THREATS

Phase of Error Type
Crew Error Error
Error ID
Flight Error Who Response Outcome

1 Intentional Code Who
undesired aircraft states 2 Takeoff/Climb 2 Procedural the 1 Detected & Action 1 Inconsequential
Use the error?
3 Cruise 3 Communication Code Book error? 2 Detected & Ignored 2 Undesired state
FO exhibited a lack of stick and 402

rudder proficiency Failed to Lack of 2 7
E1 4 4 stick and 3 2
maintain speed during a 30- rudder
FO Nobody
degree bank turn. proficiency
E __

Error ID

No management Both the error and undesired aircraft 45 5

E1 No Jumpseat 3 1
state went undetected poor vigilance. Speed too low
Rider
E __

Solution: Exercise Five
Threat ID
Phase of Effectively
Threat Flight managed?
Code
1 Predepart/Taxi (Yes / No)
2 Takeoff/Climb
Time pressure did not want to 30

T1 Op 1 No Threat mismanaged Crew got in a hurry
miss the curfew in DCA Pressure
1 Threat mismanaged Crew unnecessarily flew

T2 Red asymmetrical weather cell Adverse 2 No
WX
through the adverse WX

Phase of Error Type
Error ID
Flight
1 Intentional Who
the error?
510
Poor decision to fly through WX Decision
E1 when they could have waited a few 2 5 to fly 6 6 2 2
into All Crew All Crew
minutes for it to pass over. adverse
WX
Error ID

Error mismanaged Crew knew they made a risky decision. 42 6

E1 Yes T1 Penetration 2 1
They wanted to avoid further delay. adverse WX
All Crew

Solution: Exercise Six
Threat ID
Code
2 Takeoff/Climb
T __ NO THREATS

Phase of Error Type
Error ID
Flight
1 Intentional Who
the error?
230
FO forgot to turn on the packs 2 7
E1 1 2 Fail to 3 3
during his flows turn on FO Nobody
packs
200 5
Crew missed the pack item on the 7
E2 1 2 Missed
All 3 2
Before Start Checklist checklist
crewmembers
Nobody
item
Error ID
with a Who State Response Aircraft State

Undesired
threat? detected Outcome
How did the crew manage or mismanage the error? Aircraft
the 1 Detected & Action
State Code
(If Yes, enter state? 2 Detected & Ignored 1 Inconsequential
E1 No Error Chain E2
2
Errors mismanaged Simple miss on the checklist 6
E2 No Incorrect A/C 1 1
became consequential. Resulted in an unpressurized A/C config - All Crew
pressurization

Solution: Exercise Seven
Threat ID
Code
2 Takeoff/Climb
20 Threat mismanaged Crew got distracted from

T1 Fuel pump low pressure light A/C 2 No
malfunction
their flight duties

Phase of Error Type
Crew Error Error
Error ID

undesired aircraft states 2 Takeoff/Climb 2 Procedural the 1 Detected & Action 1 Inconsequential
Use the error?
235 2 7
E1 Failed to dial MCP altitude 2 2 Wrong MCP
FO Nobody
3 3
altitude
Failure to perform altitude 234 6 7

E2 2 2 Failure to 3 2
awareness procedures cross-verify
All Crew Nobody

Error ID

E1 Yes T1 Error chain E2
Errors mismanaged Crew devoted their attention to the 40 8

E2 Yes T1 Vertical 3 1
fuel pump problem with no one left to fly the airplane deviation
ATC

Solution: Exercise Eight
Threat ID
Code
4 Des/App/Land (Yes / No)
ATC challenged the crew to 50 Threat mismanaged CA accepted the clearance

T1 ATC 4 No
accept a tough-to-meet clearance command
without discussing it with the FO (PF)

Phase of Error Type
Crew Error Error
Error ID

Describe the crew error and 1 Predepart/Taxi detected
associated undesired aircraft states 2 Takeoff/Climb 2 Procedural the 1 Detected & Action 1 Inconsequential
Use the error?
Decision to accept a risky ATC 530 1 2

E1 4 5 Accept risky 2 3
clearance ATC instruct
CA FO
222
FO left the speed brakes 2 1
E2 4 2 Failure to 1 1
extended at 1800 retract speed FO CA
brakes
Error ID

Error mismanaged CA accepted a clearance that

E1 Yes T1 unnecessarily pushed the FO. In response, the FO said
nothing and made the close-in turn
E2 Yes T1 Error managed CA caught the extended speed brakes

Solution: Graduation Exercise
Threat ID
Code
1 Threat mismanaged Partially contributed to the

T1 IMC WX Adverse 4 No
WX
unstable approach

Crew Error Error
Error ID
Phase of Error Type Error Who Response Outcome

Flight Code Who
Describe the crew error and detected
committed
associated undesired aircraft states 1 Intentional the 1 Detected & Action 1 Inconsequential
Noncompliance Use the error?
4 Des/App/Land Code Book error? 2 Detected & Ignored 2 Undesired state
2 Procedural 3 Undetected 3 Additional error
282
CA got more than a dot high on 1 1
E1 4 2 Unintentional 1 2
the glideslope vertical CA CA
deviation
180
Crew intentionally did not Failure to 6 6
E2 4 1 execute a 2 2
execute a go-around missed
All Crew All Crew
approach
Crew Undesired Undesired
Error ID
Undesired Who State Response Aircraft State

Associated
Aircraft detected Outcome
with a How did the crew manage or mismanage the error?
threat?
Code state? 2 Detected & Ignored 1 Inconsequential
3 Undetected 2 Additional error
Error mismanaged CA tried to correct the glideslope

82 6
E1 Yes T1 deviation at 1500 but ended up with a 1500 fpm sink rate Unstable
All Crew
1 2
at 1000. Unstable approach. approach
Error mismanaged Crew failed to execute a go-around 88 6

E2 Yes T1 1 1
and ended up landing long outside the TDZ. Long landing All Crew

THREAT AND ERROR MANAGEMENT
TRAINING
THREAT and ERROR MANAGEMENT TRAINING
Captain Don Gunther, Continental Airlines

Manager Human Factors
Captain Bruce Tesmer, Continental Airlines
Safety, Manager Crew Performance
ABSTRACT on the realization that all crew errors cannot, and will
not be avoided. Therefore, crews must use all means to
Line Operational Safety Audit (LOSA) data have three successfully resolve the error(s) to reduce or eliminate
major uses for research, for organizational safety the consequences.
initiatives, and for the development of training .
curricula1. The development and format of an Error T&EM
Management (EM) course is reviewed and how the ERROR MANAGEMENT

data from a follow on LOSA confirms the effect of the
training. From the follow-on LOSA, data is then used
to develop the new Threat & Error Management
(T&EM) course.
1996 LOSA
In 1996 Continental Airlines conducted a system-wide

Line Operation Safety Audit (LOSA). LOSA is a non- 3/4/01 HF Team 12
Human Factors Team3/4/01 Dec 2000
jeopardy observation of line crews during normal
operations. LOSA is designed to identify overt and
latent threats, crew errors, and aggregate crew The course introduced how SOPs, checklists, etc.
performance on how threats/errors are managed (use of (Resist) are important guards against possible errors
counter-measures). LOSA also identifies the links and how the use of CRM counter-measure skills
between crew errors, threats and other errors, as they (Resolve) can be practiced and improved to more
become the precursors to accidents/incidents. It is far effectively manage error. Additionally, all Check
more important to use LOSA data to affect system Airmen were trained in identifying crew error and to
changes than policing individual crews or pilots. Based reward effective error management when it was
on the 1996 LOSA results, Continental focused on present and not just focus on a crews lack of success
crew error and the countermeasures necessary to avoid, in trying to avoid all error. The CRM markers and
trap and mitigate errors before they became error management were also imbedded in training
consequential. Working with the University of Texas courses where crews are held accountable not only for
Human Factors Research Project Team, Continental their technical skills but also their skills at using CRM
developed an Error Management course that differed markers as effective error counter-measures.
from previous CRM programs.
The operations safety change process that began with
Error Management LOSA 1996 was about to complete a full cycle: 1) use
LOSA to collect error data in line operations, 2). Flight
Starting in 1997, all Continental pilots began attending Standards & Training use LOSA analysis to make
a new CRM course named Error Management. changes, 3). Line pilots receive training on the changes,
Pilots gained an understanding of human (pilot) error, 4). Conduct a follow-on LOSA to measure the
error avoidance, error-trapping and mitigating the effectiveness of the changes. It has always been
consequence of error. A major focus of the course was difficult to collect hard data on the value of CRM
courses and their effect on crew performance.
However, the system-wide LOSA 2000, conducted
1 after completion of the Error Management Course,
System Safety and Threat and Error Management: The
Line Operational Safety Audit (LOSA). provided measurement against the base line data
11th International Symposium on Aviation Psychology collected in LOSA 1996.
March 7, 2001 Columbus, OH
Robert L. Helmreich, James R. Klinect, John A. Wilhelm & J.
Bryan Sexton
successfully and reduce crew errors associated with
LOSA 2000 these threats.
The Line Operational Safety Audit (LOSA) 2000, Threat & Error T&EM
when compared to the 1996 LOSA, showed that the Management

CAL pilots not only accepted the principals of Error MAINTANCE PROBLEM
WEATHER ALTERNATE
LOW VISIBILITY TAXI
DE - ICING
Management but had incorporated them into their

everyday operation on the line. LOSA 2000 showed a WRONG TURN
sizeable improvement in the areas of checklist usage, a ALTITUDE DEVIATION
70% reduction in non-conforming (not meeting

stabilized approach criteria) approaches and an
increase in overall crew performance. While
conducting LOSA 2000 observations, the observers
saw another area of crew performance that needed to 3/8/01 HF Team 36
Dec 2000
be addressed in addition to Error Management. The Human Factors Team3/8/01
new area of crew performance concerned threat

recognition and threat management. The LOSA 2000 Threats = Red Flags: In reviewing several accidents
data and analysis concerning threats led to the and incidents over the last several years, it became
development of the current 2001 Threat and Error apparent that where there were several threats not
Management CRM course. properly managed. There were also crew errors that
together, played a significant role in the mishap.
Threat & Error Management Threats must be identified and assessed as Red Flag
warnings. When crews successfully recognize and
Based on the premise that crew errors occur on normal acknowledge threats as red flags, they are in a better
daily flights, a question was formed from pilot position to manage that threat so it becomes
discussions during the Error Management training. The inconsequential. Accident/incident crews typically do
question is: what defines a normal flight? The not recognize all the threats, or their severity, and
answer decided upon, defined a normal flight as one accumulate red flags, which invites crew error. Crews
having no threats. This would be considered a pristine are most vulnerable when they acquire several threats,
flight, requiring no crew effort to change anything from (red flags) and have employed no strategies to manage
the plan, through the execution, of flying from them. Accumulation of red flags places the crew at the
departure to destination. While this is unusual, it does edge or corner of the operating envelope where time
happen. That being the case, the LOSA 2000 team and options are limited. To effectively manage threats,
defined a threat as anything that requires a they must be identified, then assessed, and then
crewmembers time/attention/action above and beyond countered. Identification of threats comes through
the tasks of a pristine flight. LOSA 2000 observers many system alerting methods i.e. aircraft system
were trained to observe the threats crews faced and alerting lights, bells, horns, voice, and devices such as
how they were managed. These were external threats GPWS, TCAS, winds hear etc. Those devices may also
(weather, maintenance, passenger problems, show the best course of action to counter the threat.
operational pressures, distractions/interruptions, ATC However, there are not systems and devices to detect,
errors /language/communications problems, etc.) that assess, and counter all threats. Effective use of CRM
were not crew errors but came from external sources counter-measures provides primary threat detection and
and increased the potential for error, if not managed management for threats not having system alerts, and
properly. What surfaced during these observations provides backup for those that do. Abundant examples
were the strategies good pilots use to effectively exist showing that failure to employ CRM counter-
manage threats. These strategies, pro-active in nature, measures was the last option for detecting a threat that
were sometimes personal techniques that pilots have ended in a CFIT accident with the GPWS warning
developed over time in order to effectively operate in sounding for 30 seconds or more with no corrective
todays complex environment. Other strategies that are action taken by the crew .One of the primary counter-
common and used routinely have developed into measures to managing threats effectively is Flight Deck
procedures (SOPs). An example would be the use of a Leadership. The attributes of Flight Deck Leadership
pre-departure briefing to review a power loss on take- (setting the example, planning ahead, initiative,
off. This briefing prepares the crew to make a better fostering communications, etc.) are the foundation for
decision, should a loss of power occur when risk is effective strategies to manage the threats crews face
high and decision time is minimal. Crews that every day.
effectively use strategies manage external threats
. Effective Red Flag (threat) management reduces
the complexity of the operating environment,
decreasing the potential of crew error. Strategies,
whether personal or SOP, need to be employed
consistently so threats can be more easily recognized
and managed. Interactive communications, vigilance,
monitoring & challenging are full time strategies the
assist crews in identifying Red Flags. These
countermeasures arent developed at the time of the
event but are developed and deployed pro-actively. In
academic terms it means, Get your stuff together
before the stuff hits the fan, or words to that effect.
The environment we operate in has only become more
complex over the last few years and will continue to
present our pilots with an increased number of
challenges. As a pilot group we must raise the bar
and accept that outstanding performance today will
only be standard in the future. We must improve
ourselves to reach that new above standard
performance level. Threat & Error Management
training is one of the means pilots can use to reach an
improved level of performance that will enable them to
deal with the increased challenges of maintaining a
safe operation. The idea behind all CRM courses is to
define the best practices in applying threat and error
management counter-measures to reduce or eliminate
the consequence of threats and errors, which are the
precursors of accidents and incidents. Safer operations
can be had by imbedding the best practices of our
pilots into our training and everyday operation. Pilots
learn many of their positive traits from the sharing of
ideas and experiences with their peers and then
applying them to their own operational philosophy.
The Threat & Error Management course is a means
by which the pilots can prepare for the future and be
part of the team successfully building a threat and error
management culture.
IMPLEMENTATION STRATEGY SESSION
115
Conclusions experience, and taking into consideration the

typical size, fleet composition and route
In the last session of the Second LOSA Week, structure of Latin American airlines
participants engaged in an open forum discussion (approximately 40 aircraft and 500 pilots), it
with the following objectives: was estimated that about 100 observation
segments would be necessary to ensure
To discuss possibilities, constraints and statistical validity. Based upon this, the figure
resources required for the implementation of of (US) $50 000/60 000 was provided. It is
LOSA at the airline-specific level; exceedingly important to introduce the strong
caveat that such figure was a rough estimate,
C To discuss the potential cooperation among and may substantially differ in specific airline
airlines within the Ibero-American community cases. The case of a mini-LOSA was
in sharing resources and technology as needed discussed. In such case, costs would
to implement LOSA; significantly vary from the figure provided.
C To discuss the potential integration of an 3. An issue specific to the region was discussed
Ibero-American LOSA steering committee; at large. Some airlines in Latin America are
and relatively small, in some cases less than 10
aircraft and 100 pilots. In such cases,
C To discuss the potential contribution of non- maintaining confidentiality and objectivity of
airline organizations, including civil aviation the observations may be difficult if not
authorities, aircraft manufacturers and impossible. As solution to this problem, the
professional associations, to the worldwide possibility of inter-carrier cooperation was
implementation of LOSA. considered. Smaller airlines could pool
resources, and observers from airlines other
that the one being observed could conduct the
The main points debated and conclusions actual observation. The exchange of observers
arrived to thereafter follow. de-personalizes and thus removes subjectivity
from the observations. The data could be then
1. Questions about language and culture were sent to the central data processing centre at
raised early in the discussions. LOSA is the University of Texas. In this way, by
essentially a data acquisition tool, and some pooling observations which would only be
participants felt that, within Latin contexts, identifiable by the University of Texas, a
the means to obtain data and, most important, volume of data large enough could be
the management of the information such data obtained, while ensuring confidentiality.
generates, may substantially differ from the
way data is acquired and information 4. In line with the perceived need to pool
managed in Anglo-Saxon contexts. Extreme resources to overcome financial constraints,
perceptions were expressed, including a view the contribution of regional associations, such
that Latin systems are designed in such a way as AITAL, was considered. However, in
that they can only function by violating the regard to LOSA, the Pan American Aviation
norms. On the other hand, other participants Safety Team (PAAST) was considered a more
felt that rather than allowing culture to stand appropriate vehicle. One of the mandates of
in the way of LOSA, culture should be used to PAAST is to identify and facilitate the
facilitate LOSA, by leveraging upon the implementation of safety tools in Latin
strengths of Latin culture while trying to America. Thus, supporting and facilitating the
neutralise potential weaknesses regarding implementation of LOSA within Latin
information management. America would perfectly fit within this
mandate. Nevertheless, regardless of the
2. The costs of implementing LOSA were participation of regional associations, there
discussed. Based upon existing industry was consensus within the group that a joint
116
effort by a pool of airlines within the region, but the methodology to implement it, LOSA,
with proper initial support, could provide all is not equally so. This, however, is not the
the resources necessary to deploy and conduct case with TREM. Participants felt that TREM
LOSA, with the exception of data should immediately be initiated within the
management. It was felt that data management Ibero-American airline community; and that
should be conducted by the University of the right way to proceed is to plant the seed of
Texas. While the idea of engaging local Latin LOSA though TREM. Therefore, it was
American universities in the LOSA project agreed that the organization of a TREM
was briefly entertained, it was discarded since training the trainers workshop in the region
no participant could identify any university in should be an immediate priority. In this way,
Latin America that conducts Human Factors airlines could count on a core group of TREM
research. course designers/facilitators to move forward
in the implementation of TREM and thus
5. The contribution of the Colegio Profesional pave the way to LOSA.
de Aviadores of Spain was considered.
Representatives of Spanish airlines strongly 10. In conclusion, the following points summarise
felt that LOSA could first be introduced the discussion as well as immediate avenues
within the Spanish context through the of action:
participation of the Colegio.
C There was agreement as to the
6. The participation of pilots professional relevance of the concept of normal
associations was considered essential. operations monitoring to aviation
Without unions support, LOSA was simply safety;
considered not viable.
C LOSA could be applied within the
7. It was considered very important that the Latin American context, although its
concept of normal operations monitoring methodology might need some
becomes an ICAO standard. However, many specific adaptation;
participants felt that it was equally important
to initiate and sustain an aggressive campaign, C The Colegio Profesional de Pilotos
within each airline and country in Latin Aviadores seems to be the most
America, to lobby for the concept. Such appropriate organization to launch
campaign should be a personal endeavour by LOSA in Spain;
all those individuals who believe in the
potential of LOSA as a safety tool. C TREM is immediately applicable
within the Ibero-American airline
8. Participants expressed regret that, community and is considered the
unfortunately, very few civil aviation vehicle to introduce LOSA in Latin
authorities from the region attended the America; and
meeting. The few civil aviation authorities
present at the meeting endorsed LOSA C ICAO should organize, in a very near
without reservations. future and as a matter of urgent
priority, a training the trainers
9. The discussion then moved on towards Threat workshop for Latin American airlines
and Error Management Training (TREM), in order to qualify TREM course
presented as a re-conceptualization of CRM. designers and facilitators.
There was absolute consensus of opinion that
TREM is the vehicle to introduce LOSA
within the Ibero-American airline community.
The line of reasoning followed was that the _______________
concept of normal operations monitoring is
valid and acceptable across the community,
LIST OF PARTICIPANTS
SURNAME GIVEN NAMES AIRLINE COMPANY ADDRESS
Anca Jose Ansett Ansett Australia, Building 186, Cnr, Service & Grants Rds. Melbourne Airport 3045, Australia joey_anca@ansett.com.au
Antoni Eugene Callistus SIA Singapore Changi Airport, Singapore 819643 SIN-T2-01A geneam@pacific.net.sg
Ballantyne Thomas Orient Aviation Orient Aviation, P.O Box 2109, Carlingford Court, Carlingford NSW 2118, Australia tomball@ozemail.com.au
Bent John GECAT GE Capital Aviation Training (HK) 6/F, Flight Training Centre, Cathay Pacific City, 10 Scenic Road, Hong Kong john.bent@gecat.com
Burdekin Susan Gay ADF Sue Burdenkin Bsc CPL, 25 Goldner Circuit, Melba ACT 2615, Australia rob.sue@dynamite.com.au
Chou Jyh Shyan China Airlines China Airlines, No.3 Alley 123, Lane 405, Tung Hwa N.Rd. Taipei, Taiwan, R.O.C. jyh-hsyan_chou@email.china-airlines.com
Duan Frank (Weiping) China Northen Airlines China Northern Airlines, 3-1 Xiao He Yan Rd. Sheyang 110043 China frankduan_99@sina.com
Edkins Graham Derek Qantas Qantas Airways Ltd, Flight Safety Department MA9 Qantas Jet Base, 203 Coward St. Mascot 2020 Australia gedkins@qantas.com.au
Evers Owen James China Airlines China Airlines, No.3 Alley 123, Lane 405, Tung Hwa N.Rd. Taipei, Taiwan, R.O.C. o-james_evers@email.china-airlines.com
Fabresse Anne Airbus Airbus Industrie B28 Building, 5 Rue Gabriel Clerc, BP33, S1707 Blagnac Cedex, France
Ghandour Malik Nassib China Airlines China Airlines - San Francisco International Airport mghandour@earthlink.net
Greenfield Peter William Emirates Emirates, Operations Centre (FC 256) P.O. Box 92, Dubai - U.A.E. peter.greenfield@emirates.com
Greeves Brian IFALPA IFALPA, M10 Scenic Villas, 26 Scenic Villa Drive, Pok Fu Lam, Hong Kong bgreeve@attglobal.net
Guang Ren Su Xiamen Xiamen Airlines, 22 Dailiao Road, Xiamen, China
Gunther Donald L. Continental Continental Airlines frigate1@ix.netcom.com
Hankins Frank Moris Boeing Boeing China Inc. Tower A, 16/F, Pacific Century Place No.2A, Worker's Stadium Rd North Chaoyang District, Beijing, 10027 frank.m.hankins@boeing.com
Harris Dayle Miguel EVA EVA Air, 15th Floor, Flight Safety Dept., c/o Danny Ho, 376 Hsin-nan Rd., Section 1, Luchu, Taoyuan Hsien, Taiwan dale.harris@attglodoal.net
Hawke Mike Qantas Qantas Airways Ltd, Qantas Centre Building C/3, 203 Coward Street, Mascot NSW 2020 Australia mjhawke@optushome.com.au
Helmreich Robert U of T The University of Texas Crew Research Project, 1609 Shoal Creek Blvd., Austin, TX78701-1022, USA helmreich@psy.utexas.edu
Henderson Simon Thornton Ansett Ansett Australia, Building 186, Cnr, Service & Grants Rds. Melbourne Airport 3045, Australia henders@melbpc.org.au
Hori Hiroto ANA ANA, 3-3-2, Haneda Airport Ota-ku, Tokyo 144-0041, Japan
Hsu Chung-cheng Steven EVA EVA Air, 15th Floor, Flight Safety Dept., c/o Danny Ho, 376 Hsin-nan Rd., Section 1, Luchu, Taoyuan Hsien, Taiwan Stevenhsu@mail.evaair.com.tw
Jennings Herschel Todd EVA EVA Air, 15th Floor, Flight Safety Dept., c/o Danny Ho, 376 Hsin-nan Rd., Section 1, Luchu, Taoyuan Hsien, Taiwan todd@jennings.net
Jeremica Vernon Benedict Boeing The Boeing Company, P.O. Box 3707 MC14-HA, Seattle, WA 98124-2207 vernon.b.jeremica@boeing.com
Jian Li China Northen Airlines China Northen Airlines, DongTa Airport, Shenyang China
Jingchen Tang Xiamen Xiamen Airlines, 22 Dailiao Road, Xiamen, China
Johnson Robert Boeing Boeing China Inc. Tower A, 16/F, Pacific Century Place, No.2A, Worker's Stadium Road, North Chaoyang District, Beijing, 100027 robert.e.johnson8@boeing.com
Keith Leroy Allen AAPA AAPA, 9th Flor, Kompleks Antarabangsa Jalan Sultan Ismail 50250 Kuala Lumpur, Malaysia lakeith@aapa.org.my
Klinect James U of T The University of Texas Crew Research Project, 1609 Shoal Creek Blvd., Austin, TX78701-1022, USA klinect@mail.utexas.edu
Kriechbaum Christopher Hugh Air NZ Air New Zealand Ltd, Private Bag 92007, Aukland 1, new Zealand chris.kriechbaum@airnz.co.nz
Kuei Cherng China Airlines China Airlines, Taipei C.K.S. International Airport kuei744@m4.is.net.tw
Lima Frederico EVA EVA Air, 15th Floor, Flight Safety Dept., c/o Danny Ho, 376 Hsin-nan Rd., Section 1, Luchu, Taoyuan Hsien, Taiwan fredericolima@attglobal.net
Markson Greg Cathay Pacific Rm. 5.1, FTC, Cathay Pacific City, 8 Scenic Rd., Lantau, Hong Kong greg_markson@cathaypacific.com
Maurino Daniel ICAO 999 University St., montreal, Quebec H3C 5H7, Canada Cmaurino@icao.int
Morii Tamaki JAL Japan Airlines, West Passenger Terminal 3-2, Haneda Airport 3 chome, Ota-ku, Tokyo 144-0041 tamaki.morri@jal.co.jp
Peacock Gerard Rene SIA Singapore Airlines Ltd, SIA Training Centre Flt. Ops Div., Flt. Crew Training, SIN STC 03-C, Airmail Transit Centre. PO BOX 501 Singapore 918101 gerard_peacock@singaporeair.com.sg
Quay Chew Eng SIA Singapore Airlines Ltd, SIA Training Centre Flt. Ops Div., Flt. Crew Training, SIN STC 03-C, Airmail Transit Centre. PO BOX 501 Singapore 918101 CE_Quay@singaporeair.com.sg
Radzi Azmi Bin Mohd Malaysian Malaysian Airlines arhfam@tm.net.my
Rodgers Michael CASA CASA, Baillieu House, 71 Northbourne Ave PO Box 2005, Canberra ACT 2601 rodgers_m@casa.gov.au
Snelgar Robin Anthony SIA Singapore Airlines Ltd, SIA Training Centre Flt. Ops Div., Flt. Crew Training, SIN STC 03-C, Airmail Transit Centre. PO BOX 501 Singapore 918101
Soh Tommy Tian Seng SIA Singapore Airlines Ltd, SIA Training Centre Flt. Ops Div., Flt. Crew Training, SIN STC 03-C, Airmail Transit Centre. PO BOX 501 Singapore 918101 tommysoh@magix.com.sg
Liangtian REN Xiamen Xiamen Airlines, 22 Dailiao Road, Xiamen, China
Sydiongco Jim EVA EVA Air, 15th Floor, Flight Safety Dept., c/o Danny Ho, 376 Hsin-nan Rd., Section 1, Luchu, Taoyuan Hsien, Taiwan
Tai Chi Hung Mandarin Mandarin Airlines, Shun Shan Airport, Taipei, Taiwan, R.O.C.
Tan Charlie SIA Singapore Airlines Ltd, SIA Training Centre Flt. Ops Div., Flt. Crew Training, SIN STC 03-C, Airmail Transit Centre. PO BOX 501 Singapore 918101 charlie_tan@singaporeair.com.sg
Tesmer Bruce Continental Continental Airlines Btesme@coair.com
Topfer Ian Qantas Qantas Airways Ltd, Qantas Centre Building C/2, 203 Coward Street, Mascot 2020 Australia itopfer@qantas.com.au
Tsay Chiou Yueh China Airlines China Airlines, No.3 Alley 123, Lane 405, Tung Hwa N.Rd. Taipei, Taiwan, R.O.C. chiou-yueh_tsay@email.china-airlines.com
Xiaogang Peng Xiamen Xiamen Airlines, 22 Dailiao Rd., Xiamen, China
Yeh William EVA EVA Air, 15th Floor, Flight Safety Dept., c/o Danny Ho, 376 Hsin-nan Rd., Section 1, Luchu, Taoyuan Hsien, Taiwan
Ying Yuan (Ms.) China Eastern Airlines China Eastern Airlines, Shanxi Branch, Flight Operations Dept.,Tai Yu Rd. 32 030031, Taiyuan Shanxi, China
Zheng Han China Eastern Airlines China Eastern Airlines, Shanxi Branch, Flight Operations Dept.,Tai Yu Rd. 32 030031, Taiyuan Shanxi, China
PROCEEDINGS OF THE FIRST ICAO-IATA
LOSA & TEM CONFERENCE
DUBLIN
IRELAND
5 TO 7 NOVEMBER 2003
TABLE OF CONTENTS
Agenda ................................................................................................................................... (ii)
The Line Operations Safety Audit (LOSA) ................................................................................. 1
Presentations
Improving Safety in an Ultra Safe System

$ Monitoring Normal Operations: the Perspective of ICAO ........................................... 11
$ Collecting Safety Data from Normal Operations: the Perspective of IATA ................ 13
The Safety Change Process

$ Managing Threat and Errors in Airline Operations ..................................................... 17
$ The Ten Operating Characteristics of Line Operations Safety Audits ........................ 25
$ The Safety Change Process Following Line Operations Safety Audits .................... 26
Collecting Safety Data from Normal Operations

$ Cathay Pacific Line Operations Safety Audit: Training for Safety .............................. 32
$ Air New Zealand: The experience of a Arepeat@ LOSA ................................................... 35
$ The Braathens experience ........................................................................................ 38
$ The Futura experience .............................................................................................. 43
$ Collecting Safety Data from in Service Occurrences and Learning from them .......... 47
$ Boeing Safety Approach: The Value of Collaboration ................................................ 54
$ Learning about automation from Line Operations Safety Audits ................................ 60
Workshops
Training Instructors on Threat and Error Management (TEM) ................................................. 75
Training Flight Crew on Threat and Error Management (TEM)

$ TEM General .............................................................................................................. 84
$ TEM: Improving Flight Crew Monitoring and Cross-Checking .................................. 92
$ TEM: Automation and Security .................................................................................. 99
List of participants .................................................................................................................. 115

Agenda
Tuesday, 04 November
1900 2100 Registration, collection of badges and reception
Wednesday, 05 November
0800 0830 Late registration and collection of badges
Opening & welcome
0830 Conference Chairman

Captain Denis Coughlan
Director of Training
Aer Lingus
Opening remarks
0835 0855 Mr. Eamonn Brennan, Chief Executive of the Irish Aviation Authority
Mr. Willie Welsh, Chief Executive - Aer Lingus
Ms. Jill Sladen, Manager Safety - IATA
Capt. Dan Maurino, Coordinator, Flight Safety and Human Factors - ICAO
Improving Safety in an Ultra Safe System
0900 0920 Monitoring Normal Operations: the Perspective of ICAO

Capt. Dan Maurino - ICAO
0920 - 0940 Collecting Safety Data from Normal Operations: the Perspective of IATA
Ms. Jill Sladen - IATA
0940 1020 Managing Threat and Errors in Airline Operations

Prof. Robert Helmreich - UT
1020 1050 Coffee-break
1050 - 1120 The Ten Operating Characteristics of Line Operations Safety Audits
Mr. James Klinect UT/TLC
1120- 1150 The Safety Change Process Following Line Operations Safety Audits
Capt. Don Gunther - Continental
Collecting Safety Data from Normal Operations
1150 1220 Cathay Pacific Line Operations Safety Audit: Training for Safety
Capt. Henry Craig Cathay Pacific
1220 1235 Q&A session
1235 - 1345 Lunch
1345 1415 Air New Zealand: the experience of a repeat LOSA

Capt. Chris Kriechbaum Air New Zealand
1415 1445 The Braathens experience

Capt. Morten Ydalus Braathens
1445 - 1515 The Futura experience

Capt. Robert Aran - Futura
1515 1545 Coffee-break
1545 1615 Collecting Safety Data from In Service Occurences and Learning from Them
Mr. Jean-Jacques Speyer - Airbus
1615 1645 Boeing Safety Approach: The Value of Collaboration

Dr. Curt Graeber Boeing
1645 - 1715 Learning about automation from Line Operations Safety Audits
Capt. Carlos Arroyo Landero - IFALPA
1715 1730 Q&A session
Thursday, 06 November
Workshops
Training Instructors on Threat and Error Management (TEM)
0900 - 1215 Mr. James Klinect & Capt. Pat Murray - UT/TLC
1215 1330 Lunch break

Training Flight Crew on Threat and Error Management (TEM)
1330 1700 WORKSHOP
TEM: General
Mike Bombala - IATA
TEM: Improving flight crew monitoring and cross-checking

Capt. Robert Sumwalt US Airways/ALPA
Friday, 07 November
0900 1230 TEM: Automation and Security

Capt. Don Gunther - Continental
1230 Conference conclusions and wrap up

Ten Years of Change Crew Resource Management 1989-1999
Robert L. Helmreich1
University of Texas Team Research Project
Since the end of the 1980s Crew The definition of CRM is

Resource Management (CRM) has grown misunderstood. Although CRM programmes
globally, has come under some attack, and is were clearly rooted in efforts to reduce pilot
finding its place as an essential component of a error accidents, over the years understanding of
safety culture by focusing on error and threat the goals of programmes has faded, perhaps in
management. CRM training is now mandated for part because of the extension of the training to
pilots of all member states of ICAO and airlines flight attendants and other personnel. An
in member states of the European Union will be indication of this misunderstanding can been
required to evaluate individual proficiency in seen in the work of scientists in the United States
CRM-related behaviours. CRM training is also who recently defined CRM as Instructional
being extended to other components of the strategies that seek to improve teamwork in the
aviation system, including flight attendants, cockpit. While effective teamwork is clearly
flight dispatchers, and maintenance personnel. In important, it is not the primary goal of CRM
terms of its application, CRM is certainly a training. The following is a more accurate
success story. Considerable evidence has also representation of the latest, 1999 model CRM
been accumulated to show the effectiveness of programmes: CRM is a subset of the Human
CRM training in changing behaviours and Factors discipline, which is concerned with
attitudes (e.g., Helmreich & Foushee, 1993). human-machine and human-human interfaces,
and the integration of human operators and
The situation, however, is not entirely
maintainers within an overall system. CRM
positive. There are three areas of contention. 1)
consists of the application of human factors
CRM has been attacked for failing to eliminate
knowledge to the special case of flight crews and
pilot error and accidents; 2) There is confusion
their interactions with each other, with other
among some members of the research and
groups, and with the technology in the system.
operational communities about the definition of
Broadly defined, CRM consists of the effective
CRM; 3) It is claimed by some critics that CRM
utilization of all available human, informational,
issues are too subjective to be evaluated fairly,
and equipment resources toward the goal of safe
thus subjecting pilots to the risk of
and efficient flight. More specifically, it is the
discrimination; and 4) There is confusion about
active process employed by crewmembers to
the role of CRM programmes within
identify existing and potential threats and to
organisations. Before turning to the substance of
develop, communicate, and implement plans and
the discussion, it is necessary to address each of
actions to avoid or mitigate perceived threats.
these issues.
CRM also supports the avoidance, management,
Has CRM has failed in its mission? and mitigation of human errors. The secondary
The contention that CRM has failed its mission benefits of effective CRM programmes are
because accidents still occur is based on a improved morale and enhanced efficiency of
profound misunderstanding of human operations.
capabilities and limitations. By their very nature,
CRM issues are subjective and
humans are inevitably prone to errors and no
cannot be evaluated. Fear of inequitable
training, however sophisticated or intensive, can
evaluation is certainly a legitimate concern of
change human nature. Humans will make errors
pilots whose livelihoods may be threatened.
and accidents and incidents will occur in
However, contemporary CRM programmes
complex systems.
focus on specific and well-defined behaviours.
Those behaviours chosen for evaluation in the
1
Research reported here was conducted with the support of FAA Grants 92-G-017 and 99-G-004, Robert Helmreich, Principal
Investigator. Thanks are due to all of the participating airlines and their personnel who made the project possible. Particular credit is
due Captain Sharon Jones, James Klinect, Captain Bruce Tesmer, and John Wilhelm for data analysis and development of the
conceptual models.
European Union (which are closely related to the and other members of the British Empire define
Behavioral Markers defined by our research the other end of the continuum with much lower
group) are objective and observable. concern for rules and written procedures.
What is the role of CRM in the Looking at the cockpit environment,
organisation? Some have argued that CRM national culture influences how juniors relate to
should ultimately disappear as it becomes fully seniors, including their willingness to speak up
integrated into technical training. We once with critical information. It is demonstrated in
supported this notion, but with hindsight we now the way information is shared. Through Rules
realise that it is and should be a separate aspect and Order, culture influences adherence to SOPs.
of training. CRM falls at the interface between We have also found, unexpectedly, that culture is
safety departments, flight training, and flight strongly associated with liking for automation
operations. CRM programmes represent ongoing and attitudes about the appropriate use of
training driven by objective data reflecting automation.
operational issues. CRM is not a one-time
Organisational culture. Organisations
intervention, but rather a critical and continuing
can function within a national culture or can
component of a safety culture.
extend across national boundaries. An
Cultures the Missing Element organisations culture demonstrates its attitudes
and policies about human error, the openness of
Early CRM programmes and
communications between management and
investigations of human error in accidents
flightcrew, and the level of trust between
viewed the cockpit as an isolated universe. With
flightcrew and senior management.
growing sophistication, we now understand that
Organisational culture also influences norms
flight operations are part of a complex system
regarding adherence to regulations and SOPs. Of
that is heavily influenced by cultures. There are
great importance, the organisational culture
three intersecting cultures that surround every
determines the level of commitment to safety and
flight crew national, organisational, and
the strength of a safety culture.
professional (Helmreich & Merritt, 1998).
Hofstede (1980) defines culture very aptly as Professional culture. Many
the software of the mind. More technically, professions such as aviation have strong cultures
culture consists of the shared norms, values, and and develop their own norms and values along
practices associated with a nation, organisation, with recognisable physical characteristics such as
or profession. We shall not be concerned with all uniforms or badges. The positive aspects of the
the facets of national culture, but those aspects professional culture are shown in strong
that may influence behaviour in the cockpit. motivation to do well and a in a high level of
professional pride. There is also a negative
National culture. Two related
component that is manifested in a sense of
dimension of national culture identified by
personal invulnerability. In our research we have
Hofstede (1980) have particular relevance for
found that the majority of pilots of all nations
aviation, Individualism-Collectivism and Power
agree that their decision making is a good in
Distance (PD). Those from individualistic, low
emergencies as normal situations, that their
PD cultures tend to focus on the self, autonomy,
performance is not effected by personal
and personal gain while those from collectivist,
problems, that they do not make more errors
high PD cultures show great concern for the
under high stress, and that they can leave behind
group and harmonious relationships and
personal problems. While the positive aspects of
deference to leaders. Another relevant dimension
professional culture undoubtedly contribute to
has been labelled Rules and Order (Helmreich &
aviations splendid safety record, the macho
Merritt, 1998). Those high on this attribute
attitude of invulnerability can lead to risk taking,
believe that rules should not be broken, that
failure to rely on fellow crewmembers, and error.
written procedures are needed for all situations,
and that strict time limits should be observed. CRM as Defence Built on Data
This dimension, which is conceptually similar to
CRM programmes provide a primary
Hofstedes Uncertainty Avoidance has proved to
line of defence against the threats to safety that
be one on which there are large and highly
abound in the aviation system and against human
significant cultural differences. At the high end,
error and its consequences. Todays CRM
Taiwan and many Asian culture are most rule
training is based on accurate data about the
oriented while the United States, Great Britain,
strengths and weaknesses of an organisation. threat and error in aviation that is shown below
Building on detailed knowledge of current safety in Figure 1.
issues, organisations can take appropriate
remedial actions, which will include topics in External Threats Expected Unexpected
CRM. There are five critical sources of data, Events and Risks Events and Risks
External Error
each of which illuminates a different aspect of

flight operations. They are: 1) Data from formal
evaluations of performance in training and on the
line; 2) Incident reports from non-punitive Internal Threats Crew-Based
systems that encourage open communication; 3) Errors
Surveys of flightcrew perceptions of safety and

human factors; 4) Quick Access Recorders
Threat Recognition Error Detection
(QAR) that provide information on the conduct CRM Behaviors and Error Avoidance
Behaviors
and Response
Behaviors
of flight. (It should be noted that these data
provide a reliable indication of what happens but
not why things happen; and 5) Line Operations
Safety Audits (LOSA). I will focus on what we Outcomes
Additional
Recovery to
have learned from LOSA. A Safe Flight
A Safe Flight Error
The nature and value of LOSA. Line

Operations Safety audits are programmes that Incidents / Accident
use expert observers to collect data about crew
behaviour and situational factors on normal
flights. They are conducted under strict non-
jeopardy conditions, meaning that no crews are
at risk for observed actions.2 Observers code Figure 1. A model of threat and error in aviation
observed threats to safety and how they are As the model indicates, risk comes from both
addressed, errors and their management, and expected and unexpected threats. Expected
specific behaviours that have been associated threats include such factors as terrain, predicted
with accidents and incidents (and that form the weather, and airport conditions while those
basis for contemporary CRM training). Data unexpected include ATC commands, system
from LOSA provide a valid picture of system malfunctions, and operational pressures. Risk
operations that can guide organisational strategy can also be increased by errors made outside the
in safety, operations, and training. A particular cockpit, for example, by ATC, maintenance, and
strength of LOSA is that the process identifies dispatch. External threats are countered by the
examples of superior performance that can be defences provided by CRM behaviours. When
reinforced and used as models for training. Data successful, these lead to a safe flight. In addition
collected in LOSA are proactive and can be used to external errors, the crew themselves may err
immediately to prevent adverse events. The and again CRM behaviours stand as the last line
University of Texas project has participated in of defence. If the defences are successful, error is
eight audits with a total of nearly 4,000 flights managed and there is recovery to a safe flight, If
observed. In this paper, data from the three most the defences are breached, they may result in
recent audits, which include threat recognition additional error or an accident or incident. In our
and error management, are discussed. These three airline database, 72% of all flights faced
three LOSA projects were conducted both in the one or more threats, with a range of 0 to 11. The
U.S. and in international operations and involved average was two threats per flight. The most
two U.S. and one non-U.S. carrier. common threats encountered were difficult
A Model of Threat and Error terrain on 58% of flights, adverse weather on
28%, aircraft malfunctions on 15%, unusual
Data are most valuable when they fit within a ATC commands on 11%, external errors
theoretical or conceptual framework. Our (including ATC, maintenance, ground handlers,
research group has developed a general model of etc.) on 8%, and operational pressures on 8%.
Threats occurred most frequently in the descent,
2
In practice, members of the University of Texas project approach and landing phase (40%).
have trained observers from participating airlines and also
serve as observers. Their presence across all organisations
allows us to make valid cross-airline comparisons.
A Model of Crew-Based Error country would be classified as an undesired state.
3) Additional error the response to error, as
Errors made within the cockpit have
we have noted, can result in an additional error
received the most attention from safety
that again initiates the cycle of response.
investigations and crew error has been
implicated in around two-thirds of air crashes Undesired states can be mitigated by
Helmreich & Foushee (1993).3 Our analyses of crew action, for example, recognition of an
error have led us to reclassify and redefine error unstable approach and a decision to initiate a go-
in the aviation context. Operationally, flightcrew around). Response to the undesired state may
error is defined as action or inaction that leads to also result in another error. Finally, in the worst
deviation from crew or organisational intentions case, the result may be an accident or incident.
or expectations. Our definition classifies five
The model aids analysis of all aspects of
types of error: 1) Intentional noncompliance
error, response, and outcome. The failure or
errors or violations of SOPs or regulations.
success of defences such as CRM behaviours can
Examples include omitting required briefings or
also be evaluated. Errors thus classified can be
checklists; 2) Procedural errors in which the
used not only to guide organisational response
intention is correct but the execution flawed; 3)
but also as scenarios for training, either in
Communication errors that occur when
classroom or LOFT. The full error management
information is incorrectly transmitted or
model is shown graphically in Figure 2.
interpreted. Examples include incorrect readback
to ATC or communicating wrong course to the Intentional Noncompliance
other pilot; 4) Proficiency errors that indicate a Procedural
Error
lack of knowledge or stick and rudder skill; and Types Communication
Proficiency
5) Operational decision errors in which crews
Operational Decision
make a discretionary decision that unnecessarily
increases risk. Examples include extreme
manoeuvres on approach, choosing to fly into Trap
Error Responses Exacerbate
adverse weather, or over-reliance on automation. Fail to Respond
Response to error and error

outcomes. Three responses to crew error are
Inconsequential Undesired State Additional Error
identified: 1) Trap the error is detected and
managed before it becomes consequential or
leads to additional error; 2) Exacerbate the
Undesired
error is detected but the crews action or inaction State Mitigate Additional Error
leads to a negative outcome; 3) Fail to respond Responses
the crew fails to react to the error either Crew-Based

Accident/Incident
because it is undetected or is ignored.
Definition and classification of errors
and responses are based on the observable
process without consideration of the outcome. Figure 2. A model of flightcrew error.
There are three possible outcomes: 1)
Inconsequential the error has no effect on the
safe completion of the flight. This is the modal
outcome, a fact that illustrative of the robust P r o f ic ie n c y
nature of the aviation system; 2) Undesired O p e r a tio n a l
state the error results in the aircraft being in a D e c is io n
C o m m u n ic a t io n s
condition that increases risk. This includes
incorrect vertical or lateral navigation, unstable P ro c e d u ra l
approaches, low fuel state, and hard or otherwise V io la t io n s
improper landings. A landing on the wrong 0 10 20 30 40 50 60
runway, at the wrong airport, or in the wrong P e rc e n ta g e o f E rro rs
3
Early investigations tended to focus on the crew as the sole
causal factor. Today, of course, we realize that almost all Figure 3. Distribution of error types
accidents are system accidents as discussed by Helmreich &
Foushee (1993) and Reason (1998)
Error Results from LOSA
Examination of the aggregate data from A B C
the first three LOSAs in which error was
T h re a ts p e r s e g m e n t 3 .3 2 .5 .4
measured is instructive. Errors were committed
by 73% of the crews observed. The range of E rro rs p e r se g m e n t .8 6 1 .9 2 .5
errors on any flight was from zero to fourteen, % c o n s e q u e n t ia l 18% 25% 7%

with an average of two per flight. As Figure 3
illustrates, the most frequent type of error in the
data from the three airline study was intentional
non-compliance (or violation) followed by
procedural. The high percentage of procedural
non-compliance is alarming and will be Figure 4. Threats and errors in three airlines.
discussed later. Procedural errors doubtless have
multiple causes. They can reflect the inherent
limitations of humans accomplishing difficult
tasks, often under high workload conditions or I n t e n tio n a l O p e ra tio n a l
P ro c e d u r a l C o m m u n ic a t io n P ro fic ie n c y
N o n c o m p lia n c e D e c is io n
they may be an indication that procedures A ll
themselves are sub-optimal. Of all the errors F le e ts 33% 39% 12% 9% 7%
A d v te c h
observed, 18% were trapped, 5% were 1 40% 31% 15% 8% 7%
exacerbated, and 77% elicited no response. A d v te c h
2 30% 44% 19% 4% 4%
Errors differ by types in whether they become Conv1
3 c re w 17% 55% 9% 11% 9%
consequential. While proficiency and operational
Conv2
decision errors are least common, they are more 3 c re w 53% 20% 10% 10% 8%
likely to become consequential.

Of particular significance was the fact
that there were very large differences in threat, Figure 5. Percentages of error types within fleets
error, and percent becoming consequential in one airline.
between fleets within airlines and between
airlines as shown in Figures 4 and 5. By Phase of flight is also strongly
consequential, we mean resulting in an additional associated with the occurrence of errors and their
error or an undesired aircraft state. This finding consequences, as shown in Figure 6. Consistent
indicates the importance of airlines determining with analyses of world wide approach ad landing
the status of their own operations rather than accidents (Khatwa & Helmreich, 1998), the
assuming that their organisation conforms to highest percentage of errors occurs during this
some industry standard. Variability can result phase of flight and the highest percentage
from differences in the operating environment as becoming consequential. Clearly, special
well as from differing organisational cultures and attention should be directed toward enhancing
subcultures. Note in particular the range of performance in this phase.
variables which became consequential,
terminating in an undesired state. There is P h a s e o f F lig h t E rro rs C o n s e q u e n tia l
certainly differential risk in the three P re flig h t/T a xi 23% 8%
organisations. There are some common factors in
T a k e o ff/C lim b 24% 12%
the three airlines, but the data that follow are
shown primarily to show the type of information C ru is e 12% 15%
that can be obrained and its utility for safety. D e s c e n t/A p p ro a c h / 41% 23%
L a n d in g
Figure 6. Percent of errors and their

consequences by phase of flight.
Specific errors. Although a wide array unwieldy, and inefficient operating environment
of error classifications was observed, some major that invites violations (Reason, 1997).
problem areas emerged. Earlier audits pointed to
Although many violations may be
the appropriate use of automated systems as an
committed with the good intention of increasing
industry-wide problem (Helmreich, Hines, &
operational efficiency, organisations cannot and
Wilhelm, 1997; Sherman, Helmreich, & Merritt,
should not tolerate disregard for established
1997). Consistent with these findings, the most
procedures. There are several compelling reasons
frequent classification of error in LOSA involved
for this. One is, of course, that standardisation of
the operation of automated systems (mode
operations cannot be achieved with idiosyncratic
control panel and flight management computer.
adherence to procedures. There is also
Errors included wrong settings, wrong modes,
compelling evidence for the threat to safety
and failure to verify settings, along with
associated with violations. First, a Flight Safety
numerous others. Overall, these accounted for
Foundation analysis of global approach and
31% of all errors. The second highest
landing accidents found that more than 40%
classification (24%) was checklist errors such as
involved violations of SOPs (Khatwa &
non-standard terminology, procedural errors,
Helmreich, 1998). Second, analysis of LOSA
performance from memory, and failure to use
data indicate that those who commit intentional
required challenge and response methods. The
non-compliance errors are more likely to commit
third highest category consisted of sterile cockpit
other types of errors. Thus, it can be concluded
violations, accounting for 13%. Fourth highest at
that violations are associated with greater risk in
8% were ATC-related crew errors such as missed
operations. Further analyses may give us greater
calls, omitted information, and accepting
insight into the nature of this relationship.
instructions that unnecessarily increase risk (i.e.,
slam dunk approaches). The fifth highest CRM as Countermeasures
category (5%) consisted of briefing errors,
failure to conduct required briefings or leaving One of the most informative aspects of
out required information. The remainder of LOSA data is the ability to link threat
recognition and error management with the
errors fell into a variety of categories.
specific behavioural markers that form the core
Violations Matter of CRM. These emerge very clearly in observer
ratings provide a clear portrait of the actions
We were dismayed by the high
taken by effective crews. Those who deal
proportion of intentional non-compliance errors
proactively with threat and error exhibit the
found in the data. Several points regarding these
following behaviours:
violations should be considered. First, as we
have noted, there were very large differences Active Captain leadership
between airlines and between fleets within Briefing known threats
airlines. Hence, one cannot generalise from these Asking questions,
data about the general frequency of violations in speaking up
the global aviation system. This point is further Decisions made and
emphasised by the fact that the three carriers reviewed
included in the study all came from countries Operational plans clearly
that scored very low on commitment to rules, as communicated
measured by our Rules and Order scale Preparing/planning for
(Helmreich & Merritt, 1998). It would be threats
incorrect to assume that pilots from other
Distributing workload and
cultures, especially those high on in adherence to
tasks
procedures would be equally cavalier in
Preparation and planning -
disregarding formal rules. On the other hand, the
staying ahead of the curve
universal pilot belief in personal invulnerability
may foster a disregard for rules. The fact that Vigilance --monitoring
many rules are broken does not imply that and challenging
violating pilots have a death wish or have
contempt for formal requirements. One must also Leadership is an overarching behaviour that
consider the possibility that the proliferation of governs interaction on the flightdeck. Although
regulations may have created a contradictory, much of the attention in early CRM programmes
was directed toward overcoming the effects of
autocratic captains who fail to solicit or accept Data that show the nature
critical information from junior crewmembers, and types of errors
many current problems seem to be associated occurring
with weak leadership and the abdication of
authority. While the importance of the identified Training in error
markers is not surprising, the results do provide avoidance and
important validation of the importance of CRM- management strategies for
related behaviours. crews
Training in evaluating and
reinforcing threat
Data, CRM and Safety Culture recognition and error
The analysis of data from a variety of management for
sources (training evaluations, incident reports, instructors and evaluators
surveys, and LOSA) aid organisations in the Trust is a critical element of a safety culture,
diagnosis and understanding of their culture and since it is the lubricant that enables free
its subcultures. Without an understanding of its communication. It is gained by demonstrating a
own cultures, organisations cannot mount non-punitive attitude toward error (but not
effective programmes to optimise them. Data on violations) and showing in practice that safety
how crews deal with threat and avoid and concerns are addressed. Data collection to
manage error help organisations develop and support the safety culture must be ongoing and
maintain a safety culture. LOSA data, in findings must be widely disseminated.4 CRM
particular, are of enormous value because they training must make clear the penultimate goals of
are proactive and allow organisations to take threat recognition and error management.
appropriate action before accidents and incidents Ancillary benefits such as improved teamwork
occur. Proactive interventions are a defining and morale are splendid, but not the driving
characteristic of an effective safety culture. force. Finally, instructors and check airmen need
Data also identify critical areas for ongoing special training in both evaluating and
CRM training. However, as noted earlier, CRM reinforcing the concepts and in relating them to
is not a universal panacea for safety problems in specific behaviours.
the aviation system. Accidents and incidents If all of the needed steps are followed
almost always have multiple roots and many and managements credibility is established, a
cannot be changed by training alone. true safety culture will emerge and the
Organisations nurturing a safety culture must contribution of CRM to safety will be
also deal with those issues identified by LOSA recognised.
and other data sources, interventions may include
revising procedures, changing the nature and References
scope of technical training, changing scheduling Helmreich, R.L., & Foushee, H.C. (1993). Why
and rostering practices, establishing or enhancing Crew Resource Management? Empirical
a safety department, and a variety of other and theoretical bases of human factors
actions. training in aviation. In E. Wiener, B.
There are basic steps that every Kanki, & R. Helmreich (Eds.), Cockpit
organisation needs to follow to establish a Resource Management (pp. 3-45). San
proactive safety culture that is guided by the best Diego, CA: Academic Press.
possible data on its operations. These include the Helmreich, R.L., Hines, W.E., & Wilhelm, J.A.
following: (1996). Crew performance in advanced
Trust technology aircraft: Observations in 4
airlines. The University of Texas
A non-punitive policy Aerospace Crew Research Project
toward error Technical Report 96-8.
Commitment to taking Helmreich, R.L., & Merritt, A.C. (1998). Culture
action to reduce error- at work in aviation and medicine: National,
inducing conditions
4
One airline had its LOSA report bound and placed copies in
every aircraft as well as every base for crews to peruse.
organizational, and professional influences.
Aldershot, U.K.: Ashgate.
Helmreich, R.L., Merritt, A.C., & Wilhelm, J.A.
(in press). Error and resource management
across organizational, professional, and
national cultures. In E. Salas, C.A. Bowers,
& E. Edens (Eds.), Applying resource
management in organizations: A guide for
training professionals. Princeton, NJ:
Erlbaum.
Helmreich, R.L., Merritt, A.C., & Wilhelm, J.A.
(1999). The evolution of Crew Resource
Management in commercial aviation.
International Journal of Aviation
Psychology, 9, 19-32.
Hofstede, G. (1980). Cultures Consequences:
International Differences in Work-Related
Values. Beverley Hills, CA: Sage.
Khatwa, R. & Helmreich, R. (1998). Analysis of
critical factors during approach and landing
in accidents and normal flight. Flight Safety
Digest. 17, 1-256.
Reason, J. (1997). Managing the Risks of
Organizational Accidents. Aldershot, U.K.:
Ashgate.
Sherman, P.J., Helmreich, R.L., & Merritt, A.C.
(1997). National culture and flightdeck
automation: Results of a multination
survey. International Journal of Aviation
Psychology, 7, 311-329.
University of Texas Crew Research Project
Website:
www.psy.utexas.edu/psy/helmreich/nasaut.htm
PRESENTATIONS
Monitoring
Normal Line Operations:
A Progress Report
Captain Dan Maurino
Flight Safety and Human Factors - ICAO
First IATA/ICAO LOSA/TEM Conference
Dublin, Ireland
5 - 7 November 2003
When the Book Goes Out of the Window
Baseline
performance
Baseline performance S A
Practical
System
LO
Drift
design Operational
deployment Operational
performance
Threat & Error Management Model (Doc 9803)
Threats
Threat Induced
Accident
Crew Error
Crew Error
Responses
Undesired Aircraft
State

Responses Accident
The Expanding Role of the TEM Model
TEM Model as
9Training tool (Numerous airlines & IATA)
9Safety management tool (IATA/SAC)
9Research tool (Boeing/Madrid)
9Licensing tool (ICAO)
TEM - Licensing Tool (ICAO FCLT/P)
Competencies SKAs Performance criteria
Follow SOPs
Threat Management Perform Briefing
CRM skills State plans
Assign workload
TEM Contingency planning
Error Management principles Monitor/Cross-check
Manage automation
Undesired Aircraft
State Management

2003 Milestones
The LOSA experience in FUTURA
9First LOSA in a Latin airline
9First non -TLC LOSA
The LOSA experience in Braathens
9First European LOSA
The LOSA experience in Aeromexico
9First TLC LOSA in a Latin airline
Air New Zealand LOSA
9First repeat LOSA other than Continental
2003 A Most Significant Milestone
The Expansion of LOSA to ATM
Eleventh Air Navigation Conference (ANConf/11)
9Develop guidelines to monitor normal ATS operations based
on LOSA
Normal Operations Safety Survey (NOSS)
Project Team
Kick-off workshop Friday afternoon
9ASA, CANSO, Airways NZ, Eurocontrol, IATA, ICAO,
IFATCA, LVNL, NASA/UT, NATS, NavCanada, UK CAA
ICAO NOSS world-wide conference last quarter 2004
Clarifying the Interfaces
ICAO, UT and TLC
Concept: the need to monitor normal operations on a
systematic basis (normal operations monitoring)
Tool: LOSA So far, the only existing tool
UT: research component of the concept
TLC: implementation of the tool
ICAO strongly endorses the concept, and supports LOSA
as the only existing tools
9ICAO will support any other tool to be developed based on
the TEM Model and the Ten Operating Characteristics
A Peacekeeper Perspective
LOSA, CRM and NOTECHS
LOSA: a tool to capture systemic data
NOTECHS: a CRM evaluation template
LOSA: cognition & context
NOTECHS/CRM: behaviours & stereotypes
CRM: just one piece of the infinitely broader
information picture developed from LOSA
9A situation, generated by all the wrong reasons, that
must be put behind
What the Future Holds
ICAO fully endorses and supports normal ops. monitoring
9 Enabling partner with all organizations and aviation domains
9 IATA IFALPA - US ALPA IFATCA Boeing - Bombardier
9 LOSA/TEM Conferences
9 ICAO LOSA Manual (Doc 9803)
9 June 2002 edition of the ICAO Journal
9 Amendment No 1 ICAO HF Training Manual (Doc 9683)
Amendment to Doc 9683 to include NOSS/TEM in ATS
Recommendation and subsequent standard [in Annex 6] within
the context of SMS
Collecting Safety Data
from Normal Operations:
The Perspective of IATA

Jill Sladen-Pilon
Manager, Safety
1
2
Introduction to IATA
273 Airline Members from 143

countries accounting for 98% of
total international traffic
3
Leading in Safety
Lead the global airline commitment to achieve a continuous
improvement in safety.
Safety Infrastructure
Auditing Safety
Cargo Safety Data

Safety Management
Cabin Safety
Safety Training
4
STEADES
Safety Data
Management
ASR LOSA
IOSA FDM
Data
Essentials
Analysis
Core STEADES 5
STEADES
Safety Data
Management
Safety Boards
Manufacturers ICAO
Communicator
ATSPs
Risk-based Reports CAAs
Smaller Airlines
ASR LOSA
Manager
FDM Global
IOSA Individual
Data
Essentials
Analysis
Core STEADES SMSS
6
Safety Bulletin
Regular pulses of safety
intelligence
Threats to operational
safety
Released monthly, 3rd
edition in production
Now available in English,
French and Spanish
7
1. IATAs position on LOSA
IATA endorses LOSA as a tool for

normal operations monitoring and
toward understanding and
reducing operational errors, when
it incorporates the 10 essential
operating characteristics.
8
2. Communication
IATA is part of the LOSA advisory

board
communicate lessons learned
9
3. TEM as a Training Tool
CRM/Threat and Error Management-NEW
Aviation Training and Development Institute

KNOWLEDGE. EXPERIENCE. NETWORKING. SKILLS. RESULTS
10
IATA Safety Manual
1. Safety Management Systems for Airlines

2. IATA Safety Management Support System
3. Risk Management
4. Threat & Error Management
5. Flight Data Management and Analysis
6. Emergency Response Planning
7. Safety Report
11
4. IATA Safety Report 2002
Accident Analysis
Colour printed
CD-ROM Including:
Supporting Documents
Safety Toolkit
CEO Brief
Related web links
12
IATA Data Classifications
ENVIRONMENTAL
E1 Meteorology
E2 ATC/conflicting traffic
TECHNICAL E4 Birds/Foreign Object
Damage
T2 Engine failure, malfunction
T11 System failure
ORGANISATIONAL
HUMAN Factors
O2 Inadequate SOPs
13
5. TEM as an analytical tool
Safety Report
our current classifications employ this
philosophy
Considering LOSA/TEM coding of threats
and errors
Incident Review Meetings
reporting of incidents using a threat and
error framework
14
Threat and Error
Management
Analytical Toolkit: Template
15
Threat and Error Management (TEM)
Threats
Threat Mngmt
Errors
Error
Mngmt
16
Outcome
Introduction to Event
XYZ Airways BA-100 aircraft, scheduled
flight ABC to CDE. Takeoff in fair
weather
No. 2 engine failed ~10 minutes (60
miles) after takeoff, air turnback &
single engine approach to ABC
Crash occurred ~1 minute after single
engine go-around was ordered, 2km
lateral from end of runway
Fire destroyed airplane; 36 fatal / 1
survivor
17
Threats
1. Engine failure during climb

2. Problems with compass/
pressurization
3. Night engine out approach
4. Engine out go-around
18
Threat Management
1. Engine failure during 1. Managed - Engine
climb shutdown according to
SOP
2. Problems with 2. Mismanaged - Working
compass/ pressurization problem
pressurization during engine out approach
3. Night engine out 3. Mismanaged - Low altitude
approach outbound on procedure/Off
course at minimums
4. Engine out go- 4. Mismanaged - Pitched up
around to stick-shaker/ Speed well
below Vref 19
Errors
1. Control of pressurization
2. Low on instrument approach
3. Off course on ILS
4. Pitch control on engine out go-
around
20
Error Management
1. Control of 1. Mismanaged - Let distraction
pressurization carry over to instrument
2. Low on instrument approach
approach 2. Mismanaged - FO questioned
application of SOPs/ FO
3. Off course ILS callouts
3. Mismanaged - At minimums
mile off course
4. Pitch control of
engine out go- 4. Mismanaged - Initial pull-up
around put airplane in slow
speed/low altitude/ engine out
position
21
Prevention Strategies
Training should include the

management of distractions
during engine failure situations
Engine out training at night
during low visibility
22
Benefits
Clear presentation of the facts
Identifying Threats and Errors
leads to recognising effective
countermeasures and prevention
strategies.
23
Unlock the value of your data
24
LOSA: Line Operations Safety
Audit: History and Status
Robert L. Helmreich, PhD, FRAeS
Human Factors Research Project

Dublin
November 5, 2003
Data to Isolate Safety Issues
Accident investigation
Limited, non-representative sample
Incident reports (CHIRP, ASRS and ASAP)
Data slanted to events resulting from system and flight crew failures
UT HF developing data category system with AA & CO
Formal checkrides (Line and Proficiency)
Data show crew capability and procedural knowledge
Flight Data Recorders QAR (FOQA)
Data show what happened in terms of flight parameters
Non-jeopardy observation of normal flights-LOSA
Give data on why things happen and how they are managed
Provides realistic baseline of safety data
Is proactive
LOSA
Jump seat observations of flight crew performance during
regular scheduled flights
Observers unobtrusive collecting data not

participating in flight
Team of observers from different backgrounds
Line pilots / Union representatives
Check airmen
Safety and Training pilots
Credible outside observers (for reliability check)
All data are DE-IDENTIFIED and CONFIDENTIAL

Scientific Background
LOSA uses systematic observation of behavior
Systematic observation is a validated
methodology using observers trained and
calibrated to high reliability
Systematic observation has been employed in
scientific studies of crew performance in
demanding environments
For example, Aquanauts in undersea habitats during
Project SEALAB and Project Tektite
History of LOSA
Formal LOSA developed at request of Delta
Airlines to validate operational impact of
Human Factors (CRM) training
The focus of initial LOSA was systematic

assessment of CRM-related crew behaviors
In 1997, collaboration with Continental

Airlines to expand LOSA to include threats
and errors and their management
Purpose of LOSA
Provide valid empirical data on:
1. Crew performance strengths and
weaknesses
Proficiency
Decision-Making
CRM skills
Procedural compliance
2. System performance strengths and
weaknesses
Culture
Airspace System airports and navigational Aids
Standards / Training / Safety / Maintenance
Crew support ATC, Cabin, Ground, and Dispatch
LOSA (1997-2003)
Continental Latin EVA Air
America Uni Air (Taiwan)
Continental Express Frontier
Air New Zealand QANTAS
Air Micronesia Singapore
Continental Silk Air
Braathens
Delta
Alaska
USAirways
Air New Zealand (2nd)
Cathay Pacific
LOSA Components
Part 1. Threat prevalence & mgt
Part 2. Error prevalence & mgt
Part 3. CRM Countermeasures
Behavioural markers
Part 4. Flight crew interview
Part 5. Flight crew survey
LOSA Data: Flight Crew
For each flight segment, observers collect data on:
General Flight Crew Information Flight Crew Performance
Demographics Behavioral markers
Attitudes / perceptions / Crew errors and
safety culture (FMAQ) violations
Safety interview
Undesired aircraft states
comments
Technical data for
Flight Description
approaches
Observer narrative
Type and stability
Overt threats
Threat management
Operational complexity
Error management
Undesired state
management
Threat
Threat Categories
Threats - Originate outside the flight crews influence but require active
management to prevent them from becoming consequential to safety
Environmental Threats Airline Threats

Adverse weather Operational time pressure
ATC events / errors Cabin events / errors
Terrain MX events / errors
Traffic A/C malfunctions / MELs
Airport conditions Ground / Ramp events /
errors
Dispatch events / errors
Ground crew events / errors
Latent Threats
Aspects of the system that predispose the
commission of errors or can lead to undesired
aircraft states
ATC practices
Organizational, national, professional culture
Aircraft characteristics
Qualification standards
Regulatory practices and oversight
Flawed procedures
Scheduling and rostering practices
Threat Prevalence
Last 10 LOSAs
1835 flights with 7576 threats
4.1 threats per flight average
98% of flight segments had one or more threats

Range across airlines 94% to 100%
2/3 were environmental threats; 1/3 were airline threats

44% of environmental threats occur in descent / approach / land
72% of airline threats occur in predeparture
Most prevalent threats

Adverse weather 26% of all threats
ATC clearances/late changes - 21% of all threats
Threat Management
Last 10 LOSAs
960 mismanaged threats in 1835 flights
Average 1 mismanaged threat every 2 flights
87% of threats effectively managed
Environmental and airline threats managed equally well
Highest mismanagement rates
17% of ATC threats were mismanaged
15% of Aircraft malfunctions /MELs threats were mismanaged
Error
Error Avoidance
Complete error avoidance is impossible

errors are inevitable
80%+ of all flights observed had one or
more errors
Must look for sources of error to strengthen
system defenses
Broad Error Categories
Intentional Noncompliance violations

ex) Performing a checklist from memory
Procedural Followed procedures but wrong
execution
ex) Wrong altitude setting dialed into the MCP
Communication Missing information or
misinterpretation within cockpit
ex) Miscommunication by crew with ATC
Decision Discretionary decision that unnecessarily
increases risk
ex) Unnecessary navigation through adverse weather
Decision Error
Choice increasing risk in a situation with

multiple courses of action possible
time available to evaluate alternatives
no discussion of consequences of alternate

courses of action
no formal procedures to follow
Error Prevalence
Last 10 LOSAs
1835 flights with 5172 errors = 2.8 errors per
flight on average
82% of flight segments had one or more errors
32% of all errors are intentional noncompliance
errors
Error Response
25% Detected and action taken
30% Detected and ignored

Almost all intentional noncompliance
45% Fail to detect

Red Flag: improve monitoring and cross-checking
Error Outcomes
71% Inconsequential
21% Undesired aircraft state
8% Additional error
1484 mismanaged errors in 1835 flights = 0.8

mismanaged errors per flight on average
77% of intentional noncompliance errors were

inconsequential
Are procedures in need of review?
A compromised situation placing the flight at increased
risk
Speed too high Long landing no go
Speed too low around
Incorrect aircraft configuration Firm landing

Flight controls
Systems Forced landing
Fuel Wrong taxiway, ramp,
Automation
runway, country
Runway incursion
Error/Outcome Summary
Procedural
Communication
Decision
Non-Compliance
0 20 40 60 80 100
% of errors % consequential
7% of errors involved a lack of technical proficiency
Phase of Flight Effects
Threats by Errors by
Phase of Flight
Phase Phase
Cruise 8% 6%
Descent / Approach /
Land
33% 44%
Taxi / Park 4% 4%
Descent / approach / land phase contains the most variability

in crew performance and most consequential errors
Threat and Error
Is a high level of threat associated with
more errors?
There are slightly more unintentional errors
when there are many threats, but the
relationship is weak
There is no relationship between threats and
intentional non-compliance
Profile of Airline X
Average from 4 LOSA
Major Statistics Airline X Archive Comparison
Airlines
Average number of
threats per flight 4.3 4.3
% of threats that were
mismanaged 8% 15%
Average number of
errors per flight 3.0 2.5
% of errors that were
mismanaged 18% 31%
Airline X
Average from 4 LOSA
Major Statistics Airline X Archive Comparison
Airlines
Average number of
mismanaged errors per 56 75
100 flights
Average number of
undesired states per 42 53
100 flights
Average number of
mismanaged undesired 6 8
states per 100 flights
Myths from the Days of Jurassic
Jets
Myth 1: The glass cockpit will eliminate
human error on the flight deck
Reality: Automation is the second largest
source of pilot error
Myths
Myth 2: Dog and duck automation

Myths
Myth 2: Dog and Duck automation will

prevail
3 individuals in crew
Pilot, dog, duck
Pilot reaches for controls
Duck quacks
Dog wakes and bites pilot
Reality: the human is still in the loop

FMAQ Safety Scale Items
My suggestions about safety would be
acted upon if I expressed them to
management
Management will never compromise safety

concerns for profitability
I am encouraged by my supervisors and

coworkers to report any unsafe conditions
High scores on the FMAQ Safety
Scale are associated with fewer
violations of SOPs and fewer
Undesired Aircraft States
University of Texas
Model (UT-TEMM)
The Threat and Error Management model was derived
from LOSA data and guides further refinement.
The model is being used by airlines as a framework

for analysis of incident and ASAP data
IATA is using it as the framework for analysis of

worldwide accidents and incidents
Threat and Error Management Model
Threats:
Latent and Overt
Threat M anagement
Errors
Inconsequential Error M anagement Incident / Accident
Undesired Aircraft
State
Undesired Aircraft
State M anagement
Applying LOSA Data
LOSA data have two primary uses:
1. Assessing system safety
2. Identifying issues for action
LOSA database has data on airports, aircraft, crew
experience, organizational and professional culture
Providing airlines with feedback on their own
operations
Observers provide valid record of what crews do on the
line
Show areas of strength as well as those needing
improvement
Data help airlines prioritize and evaluate safety efforts
CRM training
Using Error Data for
Organizational Interventions
Violations - suggest poor procedures, weak captain
leadership and/or a culture of non-compliance
Procedural errors - may indicate poor workload

management and/or poor procedures
Communications errors - may reflect inadequate CRM

(monitoring and challenging) or complacency
Decision errors - may indicate need for more CRM

training on expert decision making and risk assessment
LOSA and CRM
CRM has evolved through 6 generations
LOSA data provide airlines with critical

topics for 6th generation CRM with its
focus on threat and error management
Emerging Problems with CRM
Not accepted by all (boomerangs)
Purpose not understood -- to help us get

along better
Lacking a universal rationale that can be

accepted in every culture and organization
Slippage over time need for refresher

Fifth Generation CRM :
Error Management 1996-
Focus on managing human error
Training in limitations of human
performance
Universal nature of human error
use of incident & accident data to illustrate
Continuation of earlier generation training

topics under error management framework
Continental Airlines used LOSA data
to develop Threat and Error
Management CRM training
the 6th Generation
Sixth Generation CRM:
1999 -
Fifth generation was useful, but created
resistance among pilots who did not like
the idea that their task was managing
their own errors
LOSA documents not only errors but also
the nature and prevalence of threats in
the operating environment shows
superior performance as well as problems
Training Issues in Threat
and Error Management
Human limitations as sources of error
The nature of error and error management
Culture and communications
Expert decision-making
Training in using specific behaviors and procedures as
countermeasures against threat and error
Briefings
Inquiry
Sharing mental models
Conflict resolution
Fatigue and alertness management
Analysis of positive and negative events and accidents
Significant LOSA Events
LOSA Weeks
Hong Kong
Panama
Dubai
Dublin
ICAO Journal May 2002 features LOSA

ICAO LOSA Handbook 2002
Recognition and Extension of LOSA
Flight Safety Foundation/Aviation Week and Space

Technology Laurels, 2001
DOSA - Dispatch LOSA (with Continental)
University of Texas Center of Excellence in Patient
Safety adopting LOSA methodology for
systematic observational research into operating
rooms and emergency rooms
The LOSA Collaborative
An umbrella research organization led by James Klinect
Collaborative functions
1) Implementing LOSA and maintaining the integrity of the
process and database by certifying observers and data
2) Supporting research efforts of the University of Texas
3) Offering user interface data to manufacturers to enhance
design process and de-identified system data to industry
and regulators
4) Coordinating a support group of participating LOSA airlines
to exchange information on data use and oversight for the
process
The University of Texas
www.psy.utexas.edu/HumanFactors
ICAO / IATA LOSA & TEM Conference
November 5-7, 2003 Dublin, Ireland
The Ten Operating

Characteristics of LOSA
James Klinect
The University of Texas / The LOSA Collaborative
The University of Texas Human

Factors Research Project
LOSA is not a:
Regulatory compliance audit
New form of line checks
Next generation CRM training course
LOSA is similar to a cholesterol check

Snapshot of system performance strengths and weaknesses
LOSA is defined by having 10 operating characteristics

Supported by ICAO, IATA, US ALPA, IFALPA, The University of Texas and
The LOSA Collaborative
LOSA Operating Characteristics
1. Jumpseat observations during normal operations
2. Anonymous and confidential data collection
3. Voluntary crew participation
4. Joint management / union sponsorship
5. Trusted and trained observers

LOSA Operating Characteristics
6. Safety-targeted data collection form
7. Trusted data collection site
8. Data cleaning roundtables
9. Data-derived targets for enhancement
10. Results feedback to line pilots

Why LOSA Operating Characteristics?
Regulator Check Airman Nobody

Angel Natural
Performance
- Pilot trust in LOSA + Performance
- LOSA value to airline +
LOSA Observer
LOSA operating characteristics enhance:

Pilot trust in LOSA
Probability of observing as close to natural performance as possible
LOSA value to the airline
Summary
LOSA is similar to a cholesterol test
Snapshot of performance strengths and weaknesses
LOSA operating characteristics define the project by

enhancing pilot trust and insuring value to the airline
For LOSA to be successful, airlines must:

Embrace LOSA as a learning project
Be prepared for positive or negative results
Have a defined safety change process to implement data-driven
safety solutions next presentation by Don Gunther
Questions?

www.psy.utexas.edu/humanfactors
www.losacollaborative.org
Following Line Operations
Safety Audits (LOSA)
Captain Don Gunther

Director Human Factors & Safety
Continental Airlines
The Continental Airlines
Safety Change & Training
Development Program
Process For A Greater Level Of Safety

Are We Really Safe?
Safer than in the past?
Not as Safe as we need to be?
How Do We Know?
No ones been Killed in our operations.

We havent had a resent Major Accident.
SO WHAT!
The Wake-up
Any Carrier can suffer a major accident at any time. Were
only as safe as our last Accident.
We must continue to investigate accidents so they are not

repeated. But, we must also reduce accident precursors to
eliminate future accidents.
Remember, Things are happening in everyday, normal flight

operations and -- we dont know about them!
All flights are exposed to risk. The only sure way to avoid the
risk of an accident is to not fly at all.
Because Flying IS our Business:
System Threats must be identified and

reduced/eliminated.
Crew Errors must be avoided and managed.
A Safety Measurement System must be used to

identify Targets for improving the normal
flight operations safety margin.
In The Past - Safety Changes:
Were made in response to
Accidents/incidents
FAR changes
FAA directives
NTSB accident investigations.
Were made, based on the experience & intuition of Flight

Operations Managers.
Were not based on any current operational data concerning

accident/incident precursors, because there was no data
available.
Were successful in achieving a very low accident rate.

Todays Safety Change Program
Discovers Targets using safety data and analysis concerning
accident PRECURSORS.
Discovers the What, Why, How associated with safety

events.
Todays safety data tools include:

Flight Operations Quality Assurance (FOQA)
Aviation Safety Action Program (ASAP)
Continental Airlines Safety Information System (CASIS)
Advanced Qualification Program (AQP)
Provides Flight Operations Managers additional insight

required to improve upon the current accident rate.
Why LOSA, ASAP FOQA, etc?
The essence of a good flight data analysis and

reporting system is that it should be confidential and
non-punitive. The concept is that it is better to know
about a potential problem - so that it can be analyzed
and the underlying reasons corrected in order to
prevent its reoccurrence before it leads to something
more serious - than to punish those that might have
made an error, etc
-Flight Safety Foundation
November 2002
Is Data All We Need?
No.
The new explosion in data collection has

brought forward the New Data Wave.
Is Data All We Need?
No. The new explosion in data collection

has brought forward the New Data Wave.
The data must be mined and analyzed to

identify the Safety Targets
The Safety Change Program then takes the

targets and translates them into Changes.
Data Driven - Safety Change &
Training Development Program
4Measure (with LOSA,
LOSA FOQA, ASAP) to obtain
targets
4Detailed analysis of targeted issues
4List of potential changes for improvement
4Risk analysis and prioritization of changes
4Selection and funding of changes
4Implementation of changes (training development &
actual training)
4Time for changes to stabilize while training is
completed.
4Re-measure
What Types Of Changes Are Made?
Changes in operational philosophy, policies and

procedures.
Changes to the System, the aircraft, hardware,

software
Changes to Threat & Error Management Strategies

and Countermeasures. (Humanware)
The Safety Change &
Training Development Program
Change itself is a safety threat.
An explanation of why the change was made

improves acceptance.
Change is more readily accepted when there are

data to support the change.
When change data & information precede the

change, training to proficiency is achieved sooner.
Training Development
Is meant to be proactive.
Needs to be based on valid data.
May require thinking outside the box.
Must be human centered and tested.
Needs to be a continuous process.

Use of LOSA, FOQA, ASAP
Data
in the Process For
A Greater Level Of Safety
The Continental Airlines Safety
Change & Training Development
Program
Safety Changes
(FOQA)-
improved arrivals into Tel Aviv, Mexico City using
FMS
Pitch Report to counter tail-strikes
Information flow to Maintenance and Engineering (flap
issues, data base issues, predictive W/S performance)
(ASAP)-
Change in altitude restriction for arrivals into EWR
Improved Pre-departure Clearance format
Joint FAA/Industry effort to work ATC issues at
targeted airports
LOSA
Line Observation Safety Audit
LOSA 1996
Error Managements
early focus was:
Managing crew error

Error Management
RESIST
HARDWARE & SOFTWARE THAT
EXISTS BEFORE THE HUMAN ENTERS
RESISTANCE
HARDWARE & SOFTWARE THAT EXISTS BEFORE
THE HUMAN ENTERS
GPWS SOPs
TCAS CHECKLISTS
TRAINING AUTOMATION
MANUALS ATC
Error Management
RESIST
RESOLVE
WHAT THE HUMAN BRINGS TO THE SYSTEM
RESOLVE
PROFICIENCY DECISION MAKING

VIGILANCE EXPERIENCE
ASSERTIVENESS LEADERSHIP
MONITORING & SIT. ASSESSMENT
CHALLENGING CHECKLIST DISCIPLINE
NASA Guidelines
Positively delegate flying and monitoring duties
Monitoring is as important as flying
Flying pilot does not become involved with

secondary tasks
When conflict arises-resolve with outside source
When in doubt-must express!

MONITORING & CHALLENGING
Monitor
Take
Challenging Action
steps
Express
your
view
Error Resolved
Monitoring &
Challenging
Monitor
Take
Action
The Heart and Sole of

Express
Threat & Error
your view
Management
Error Resolved
The Process
1st year: LOSA and data analysis followed
by course development
2nd & 3rd years: Training course for all
crewmembers, Check Airmen training and
imbedding of EM into courseware, etc.
4th year: Preparation for next LOSA and
targeting areas to be measured and new
areas to be emphasized.
Line Observation Safety Audit
LOSA 2000
LOSA data shows another area of
vulnerability...
THREATS
THREATS
Influences that can lead to crew error
Passenger events
Distractions ATC
Cabin Crew
Terrain
Weather
Maintenance Similar call sign
Ground Crew Time pressures
Flight
Heavy traffic
diversion
System malfunction
Unfamiliar airport
Automation event Missed approach

THREATS
Threats
Do not equal errors
Increase error potential
Threats = Red Flags

Strategies
Industry, corporate and personal
countermeasures crews use to manage
threats
Will reduce the number of hard errors
Will improve the error management process by
increasing the awareness of potential errors and
the proper countermeasures to be used
Threat and Error
Management
The Process
1st year: LOSA and data analysis followed
by course development
2nd & 3rd years: Training course for all
crewmembers, Check Airmen training and
imbedding of TEM into courseware, etc.
4th year: Preparation for next LOSA and
targeting areas to be emphasized.
TEM
the challenge
How to improve Threat

identification
Get it on the RADAR
Weather
Time Pressure Unfamiliar Airport
System Malfunction Terrain
Automation Event
Distraction Radio Congestion
Diversion
Missed
Cabin Approach
Events
STRATEGIES
to counter THREATS
HARDWARE / SOFTWARE TOOLS

Checklists, FARS, SOPS, Aircraft Warnings, Etc.
TOOLS YOU BRING TO THE FLIGHT
Resolve to use SOPS, Crew

Briefings, Communications
Etc.
Side 1
Cathay Pacific
LOSA
Capt. Henry Craig
Cathay Pacific
Training for Safety

Aiming for Excellence
CX LOSA Cathay Pacific Airways 2003
Side 2
Why LOSA at Cathay?

Proactive not Reactive
Wanted a health check!
Conducted a home-grown LOSA in 1998
Other data doesnt tell the full story
A culture where good enough isnt good enough
A sound track record, doesnt guarantee a safe future
Significant changes had already taken place
CX LOSA Cathay Pacific Airways 2003 LOSA.2

Side 3
Why LOSA at Cathay?

Proactive not reactive
Crew Demographic Changes
Recruitment (F/Os Cadets; S/Os)
Experience levels changing
(especially F/Os and Relief Pilots-in-Charge)
F/Os: less sectors; shallower background; younger

Side 4
Why LOSA at Cathay?

Crew demographic changes
Aircraft Changes
Fleet size doubled (in just over 10 years)
Predominately analogue to all EFIS
Predominately 3-man cockpit to 2-man cockpit
ULR & Crew Complement & Automation
= less manual flying
Side 5
Why LOSA at Cathay?

Crew demographic changes
Prophesy
Aircraft changes Hearsay
Opinion
Snapshot through Measurement Guesswork
Already data-rich (QAR; Pilot Training Reports;
CAR; CRP; ASR etc.)
Establish an accurate baseline in Line Operations
Ensure change is data/process driven - not PHOG
(Ian Brooks)
LOSA is data with a human face - it closes the gap

Side 6
Considerations
Find a LOSA Champion
Dont underestimate -
amount of hard work required
amount of co-ordination and co-operation required
number of roadblocks to overcome
number of stakeholders who want ownership
Establish agreement with all stakeholders:

the accountants; the pilots; the schedulers; the cabin
crew; the outports

Side 7
The Steps
Be patient!
Fact-finding mission to LOSA Collaborative
Count the Cost / Appoint a Program Manager
Liaise with LOSA Collaborative (listen well)
work out a schedule
appoint & train auditors
Produce a written agreement between Company and Pilots
Union
Communicate (NTC; Crews News; email; Union magazine;
letter to ISM; pilot forums / C&T meetings)
Be flexible!
Side 8
LOSA Flights June 2001

Aircraft type Observations
A330/340 regional 55
B777 51
A330/340 augmented 28 *
B747- 400 26 *
Total: 160
(* less than targeted)

Side 9
Cathay LOSA Report (March 2002)
Threat Results
Error Results
Undesired Aircraft State Results
Threat & Error Countermeasure Results
Crew Interview Comments
Results measured against the LOSA archive

(5 other anonymous airlines)

Side 10
An event or error outside the
Threats crews influence, that requires
crew management and attention
(LOSA Report)
Common Threats:
Adverse Weather
e.g. Thunderstorms, Windshear etc.
ATC
e.g. Difficult clearances, language, errors etc.
Aircraft Malfunctions
e.g. Systems, engine overtemps & vibration,
brake overheats, APU, cabin etc
Others
e.g. Cabin distractions, ground & ramp problems,
ground maintenance etc.

Side 11
Undesired The aircraft is placed in a

compromising situation
Aircraft States that reduces safety
margins (LOSA Report)
U.A.S.
2/3 of these were linked to a threat
1/4 were linked to weather (June in SE Asia!)
Aircraft Handling
Speed; vertical & lateral deviations; weather penetration
Descent/Approach/Landing
Unstable approach; continued landing; long/short landings
Incorrect Configuration
Systems; flight controls; automation; or engine
More examples
Late configuration; wrong heading/taxiway; no anti-ice; flap overspeed

Side 12
Moving On
LOSA produced no major surprises
(command leadership / no intentional non-compliance / cabin threats)
Verifiable data supporting Targets for Enhancement

(took time - up to 12 months - to digest the report)
Required deeper analysis to achieve implementation

(LOSA project manager to produce specific recommendations)
Posted abbreviated LOSA report on FOP Website

(Full hardcopy report available for all aircrew to read)
LOFT with Threat and Error concepts and components

(briefing / debriefing built on LOSA results & TEM principles)

Side 13
Moving On
Cathay considered LOSA as one of a number
of safety-measurement inputs.
LOSA is not a stand-alone program.
FDAP (bi-monthly analysis and reports)

Undertook ALARS (Flight Safety Foundation) Toolkit
evaluation
Flightcrew Training Reports (under constant review)

Side 14
Moving On
We used LOSA to validate data from other sources
- saw similar indicators elsewhere
Adherence to SOPs
Workload management Automation management
Monitor / cross- check Outdated briefings
ATC threats (exhaustive and by rote)
Side 15
Moving On
LOSA together with other data

has driven some changes:
New stabilised approach criteria
New SOP for Flap and Gear selection
New take-off and approach briefings

(efficient and relevant name the threat!)
New standardised approach and landing callouts
Introduction of Threat & Error Management program

Side 16
Looking Ahead
Life is lived forward but understood backward.
Kierkegaard
Everyone wants hindsight in advance!
LOSA is
ensuring safety comes first
in Cathay...
LOSA results, integrated with a TEM culture, give us a chance

to see some things before they happen.

AIR NEW ZEALANDs LOSA
PROGRAMME
Captain Chris Kriechbaum Captain Julian Alai

CRM Programme Manager LOSA Project Manager
LOSA Programme
1998
2001 - aborted
2003
Draft report stage
Final Report December 2003
Dublin LOSA Week

Air NZ - our vitals then and now!!
660 pilots, 1700 cabin crew - a small airline!
Two pilot union groups
All Boeing fleets, 7 x 747, 13 x 767, 16 x 737
Mix of glass (744 & 767) and non glass (742 & 732)
Now all glass
Long ( 14hrs max) & Short (.5 hrs min.) haul mix
Sectors audited - 97/1998, 200/2003
No QAR data - then or now
A 320 Fleet implementation imminent
Dublin LOSA Week
Business Rationale behind the 1998
LOSA
An observation of crew in the real environment
Pro-active approach to safety.
Required facts not opinions.
Wanted an international comparison of our CRM
programme.
Needed a threat and an error bench mark.
Needed a safety snapshot from a crew
perspective
Down under inferiority complex??
Dublin LOSA Week
1998 Data
The main errors included:
Failure to cross-verify
52% of flights had at MCP/FMC/CDU settings
least one error Omitted or incomplete F/A
78 separate errors briefings
recorded on a total of Incorrect settings / frequencies
91 flights selected
Between Zero and 5 Incomplete cockpit briefings
errors per flight
Checklist omitted, incomplete, or
Average of 0.86
done from memory
errors on every flight
Consequential error Failure to set altitude alerter
rate high Missed ATC calls or incorrect
read back
Dublin LOSA Week
1998 RESULTS
CRM behavioural markers generally positive
Consequential error rate, the incentive for change
Automation issues were a dominant theme
Recommendations for procedure changes
Culture issues
Dublin LOSA Week

1998 CHANGES
9Change in the Pilot/Management culture
- Just Culture Introduction
9Skill improvement in error management
- TEM Training
9Leadership training for Captains
- Enhanced Command Course
9Improve F/A briefings
- Combined EP training
9 Terrain & Altitude awareness training
- Improved Situational Awareness Modules
9Verification of FMC/MCP settings
- SOP Changes
Dublin LOSA Week
THE BENEFITS
Established a baseline
Elevated CRM profile with in the Company
Credibility in International Aviation Community
Removed the scatter gun effect of training
Combined Training
Targeted Error Management refocus
Updated Command Course
SOP changes
Just Culture
Value for money
Justification of Capital and other Expenditure.
Dublin LOSA Week

Motivation behind the 2001 LOSA
Ansett Australia purchase & amalgamation of

both Flight Operations.
Base-line measurement before merging.
Measure the success of operational changes
since the 1998 audit.
Aborted, due to Ansetts being put into
receivership.
Dublin LOSA Week

THE 2001 EXPERIENCE
Selection of observers
- Involved union input
- Training Managers matrix
Set schedule for sectors
Dublin LOSA Week

Business Rationale behind the 2003
LOSA
Make a comparison of the operational safety
health of the operations system with that obtained
in 1998
Base-line before the A320 implementation.
Attempt to measure our risk profile
Provide further guidance towards future
improvements.
Timing - a window of opportunity before the new
A320 fleet.
Dublin LOSA Week
Convincing the Bean-counters!
THE BUSINESS CASE

Financial scrutiny this time round was intense.
Long term approach was adopted, especially in
terms of potential financial benefits.
Change in ANZ Operations management
- Politics due to external pressures, was higher
- Bigger promises made
- Final sign-off was only 10 days before start
LOSA Project Manager appointed - Julian Alai
Dublin LOSA Week
THE 2003 EXPERIENCE
Observer selection
- Involved union input
- Refined selection matrix
- Mix of internal and external observers
- Pilot and non current pilots
In depth observer training, including a
NZCAA representative.
Dublin LOSA Week

LESSON LEARNED
Use of non pilots - both good and bad in this.

Time frame was too short - created pressures
Too much publicity?? - reject rate higher than
anticipated
Methodology in 98 was significantly different
than in 2003.
Set schedule for sectors - consider a more
flexible method
Comparisons 98 to 03, were difficult to make.
Significantly more information to work with.
Dublin LOSA Week

WHERE TO FROM HERE?
Consideration of integrating LOSA outputs

with other safety data
Mini audits planned
Possible joint audits with other airlines
Pooling of observers
Keep all observers current
Dublin LOSA Week

Contact Details
Captain Julian Alai

julian.alai@airnz.co.nz
Captain Chris Kriechbaum

chris.kriechbaum@airnz.co.nz
Dublin LOSA Week

Dublin 5 November 2003
Capt. Morten Ydalus/Capt. Hvard Vestgren
Braathens Flight Crew Dept.
Brief Overview
S2
Why LOSA in Braathens?
S3
Need to know facts
Establish status of crew performance:
Proficiency
Desicionsmaking
CRM skills
Loyalty to procedures
Establish status in relation to system performance:

Culture
Airspace systems airports / navaids
Procedures
Flt. Standards / Training / maintnance
Crew support: ATC, Cabin; Ground, Flight Watch
S4
Turbulent years
Braathens Went through some very

difficult years with almost bankruptcy,
large personel cutbacks, and a
takeover by the traditional competitor
What do these facts mean for
Operational quality and safety?
S5
LOSA
S6
Preparation phase
Must set up proper organization

Planning is essential
Positive relations with the Union
Selection of observers
Diverse group
Info to Pilots
LOSA Newsletters, hard copy and on Intranet
Observer training
Very important and intense
Logistical issues
Tickets, Id cards
S7
174 Observations Jan - Feb
Winter Operations
All observers used Laptops

New data program from TLC
Still each observation took 2-3 hours to record
Secure transfer of data direct to TLC
Worked well
Web survey answered by appr. 1/3 of
the pilot group
S8
Data Cleaning in Austin
Lot of work
Union present
Very educational
Got a feel for all the data before The Report
S9
Report
Very Thorough
Lot of data
TLC good presentation to management
S10
After Report
S11
Threats
Passenger events
Distractions ATC
Cabin Crew
Terrain
Weather
Maintenance Similar call sign
Flight
Heavy traffic
diversion
System malfunction
Unfamiliar airport
Automation event Missed approach
S12
Results - Threats
4.7 threats pr flight, very high relative to other LOSA

Archive airlines
50% of these were due to Adverse Weather, 20%
due to Airport and Ground Conditions (Winter
operations)
Braathens has one of the best Threat management indexes ever
observed by the LOSA Collaborative. Braathens Crews are excellent
Enviromental Threat Managers.
Very few threats observed from Dispatch and Cabin
Crew
Aircraft malfunctions / MEL items
Some ATC threats
S13
Errors
HUMANS MAKES MISTAKES!
Examples of errors:
Crew forgot to select autobrake RTO
Premature level off
Not adhering to ATC clearance
S14
Results - Errors
Rate of proficiency errors was very low, suggesting
no apparent knowledge or skills deficit
Procedural drift, the inevitable adaption of
procedures to better suit the temporal contraints of
the local enviroment.
Half the errors logged were intentional
noncompliance errors, 75% had no consequential
outcome
25% of the intentional non-compliance errors led to
new errors or an Undesired Aircraft state
80% of all Aircraft handling errors were mismanaged
S15
Results - Procedures
Descent/approach/land contains most threats and errors
Rank does not appear to affect crew performance
Threat managment slightly better when Captain flying
No effect of rank on error occurence or management
Braathens crews were very experienced, crews with least
experience made fewer errors.
Delays had no effect on crewperformance, rather opposite
Web survey identified several automation traps and areas of
confusion between 737 NG and Classic
Web survey identified a wish for more training in Turbulence and
Extreme WX conditions
Intentional Non-Compliance:
OPS, FMA Callouts, MA construction, decend wind entries
Unstable Approaches
Deviations from defenition caused by, terrain GP/PAPI transitions, short rwys,
landing techniques / procedures
Aircraft malfunctions / MEL items
S16
Sensitive data
Info to pilot group

Pilot meetings
Intranett
Info on TEM courses
Difficult with printed info (sensitive
information we do not want to have
lying around)
Top Management reluctant to release
data except to working groups
What if press gets hold of it?
S17
Work groups
S18
Work Groups
Flight Operations / Procedures
New / modified flight procedures
New Flight Manuals (electronic)
Stabilised approach project
z New data module on Line Checks, recording approach
parameters
z FOQA collaboration/FOQA issues
Technical
Mel issues
CDL issues
Automation issues
TEM
S19
TEM in Braathens
S20
CRM/Leadership
TEM course from 1. Dec 2003

1 day all flight crew
CCM on PGT
Developed with help of Don Gunther
Using actual LOSA flights as cases
S21
Safety Change Process
S22
Braathens Safety Change Process
1. Statistical
7. Install 8.Re- analysis
changes Measure
6. Approve Safety 2. Change

changes authority &
Change responsibility
5. Risk
assessment / 3. Process
priorities strategies
4. In-depth
analysis
S23
10 Operating Characteristics
S24
Ten Operating Characteristics
(Doc 9803)
1. Jump seat observations during normal flights

2. Voluntary crew participation
3. Confidential, safety-minded data collection
4. Joint airline-pilot sponsorship
5. Targeted observation instrument
6. Trusted, trained, standardized observers
7. Trusted data collection site
8. Data cleaning round tables
9. Data-derived targets for enhancement
10. Feedback of results to line pilots
S25
Did we comply?
On the 9 first completely

On results to Pilots, not as well as we
would have liked, but we are still
working on it
S26
Future
S27
Thank You!
S28
LOSA FUTURA
LOSA
experience within
FUTURA
First ICAO-IATA LOSA &TEM Conference
Dublin, Ireland
November 2003
Cptn
Cptn.. Robert
Robert Aran i Escuer
Escuer
Responsible
Responsible for
for CRM
CRM && Human
Human Factors
Factors
Compaa
Compaa Hispano-Irlandesa de
Hispano-Irlandesa de Aviacin
Aviacin (FUTURA)
(FUTURA)
Gabinete de Seguridad Unidad CRM y FFHH

Due to its own nature LOSA would never be
applicable to organizations with pathologic
LOSA FUTURA
characteristics, being barely efficient in those
considered bureaucratic for their reaction
capacity is extremely slow..
1
Three lessons learned by FUTURA
Those organizations whose activity is only
focused on the survival of its project due to
LOSA FUTURA
the lack of resources or because their goal is
short-term profitability or because they are
based on labor disputes, latent or active, are
not the ideal ground for LOSA.
2
In spite of certain weaknesses, LOSA is the
most powerful proactive safety tool ever
LOSA FUTURA
designed to sustain safe operations at the
highest possible level of an organization
3
LOSA FUTURA
The required cultural change
Management compromise with Human Factors.

Survey conducted to determine the Organizational
Status.
Adaptation of CRM Training to JAR requirements.
FOQA program implementation.
Flight and Cabin crews joint CRM practices.
MRM implementation.
LOSA implementation.
Introduction of TEM workshops and practices.

LOSA FUTURA
LOSA Program Chronology
Initial approach to the LOSA concept (UT papers)
LOSA Week held in Panama
First TEM Workshop held in San Salvador
Signature of LOSA Protocol by Pilot Trade Unions

Training Course for LOSA Observers
Creation of the LOSA Committee in Futura
Creation of the LDB (LOSA Data Base)
LDB Training for Observers.
LOSA publicity on the Intranet and mailing to all
pilots.
Starting of the Observation.
LOSA FUTURA
Three key points for

the implementation
of LOSA
within Futura
LOSA FUTURA
1. Signature of LOSA Protocol
LOSA REF -UCRM-P01
PROTOCOL PAGE 1
LINE OPERATIONS SAFETY AUDIT
1. INTRODUCTION:
This Protocol, whose signature by the representatives of the

different Pilot Trade Unions will validate the implementation
of LOSA in the Company and introduces the bases of a
program to manage human error in aerial operations. That
program hereinafter called LOSA (line Operations Safety
Audit) is proposed by the Safety and Security Office as the
basis for the development of measures to counteract
operative errors.

LOSA FUTURA
2. Creation of LOSA Data Base
Some Search Criteria in Futuras LDB
Non-stabilized approaches, according to: Airport and Number of flights

per Airport.
Errors, according to Airport, phase of Flight , type of Error and code.
Errors Committed and Detected according to: Who makes the Error,
Who detects the Error, Airport and phase of Flight.
Flight
Flight Threats, according to Airport, phase of Flight, kind of Threat and
Code.
Undesired Aircraft States, according to:
Airport, phase of Flight, Undesired Aircraft State type and code.
Comparison of Errors, Flight Threats and U-A-S, according to the
Airport.
Behavioral markers, according to the phase of Flight.
General Crew Efficiency
0 2 0
Non-Stabilized Approaches
LOSA FUTURA
0 0 0 MAN 1 0 0 0
0 0 0 MXP 1 0 0 0
0 0 0 NWI 2 0 0 0
8 ORK 3 0 0 0
7 0 0 0
SNN 1 0 0 0
6 0 0 0
5 TMP 1 0 0 0
N Flights
4
Errors 500
3
N Flights Errors 500Errors 1000 Errors 1500 Errors 1000
2 7 0 0 0
6 0 0 0 Errors 1500
1 8 1 0 0
0 3 0 0 0
ACE
8 AGP5 LPA 2 PMI 0TFS FUE DUB
5 0 0 0
Airport
6 0 1 0
Supuesto ficticio. DEMO BDL-FUA

LOSA FUTURA
Error Code (Todas)
Number of Errors and Type according
Suma de Errors to the phase of Flight
35
30
25 Error Type
Intentional / no compliance
20
Procedural
15 Communication
Decision
10 Proficiency
0
Predepart / Taxi Takeoff / Climb Cruise Des / App / Land Taxi-in
Phase of flight
Supuesto ficticio. DEMO BDL

LOSA FUTURA
3. Observers Training
It is essential a high level of standardization as well as

the maximum knowledge of:
Standard Operational Procedures (SOPs)
The Aircraft Type
The philosophy and functioning of ICAO Doc 9803

LOSA FUTURA
LOSA adaptation
Doc 9803 AN/761 by ICAO contains the basic documents
for its implementation, however each organization should
adapt LOSA depending on :
The Aircraft Type
The Observation Purpose

LOSA FUTURA
LOSA
as an Integral Tool for
Safety

LOSA FUTURA
Datos Generales de la Observacin LOSA-FUTURA
Febrero - Marzo 2003
N Vuelos
N Crews
% Crews
Vuelos Largos
Medios
Cortos
0 10 20 30 40 50 60 70 80

More than 400 LOSA data obtained during the last LOSA
observation, have proved its utility to
LOSA FUTURA
Eliminate, modify or develop SOPs and checklists items

Redesign training modules for:
-Dispatchers
-Mechanics
-Cabin crew
-Pilots
Learn how crews manage threats
Learn why specific events took place
Learn how crews manage errors
Design TEM training based on operational data
LOSA FUTURA
The future of LOSA within Futura
JOINT WORK
FOQA LOSA
Error Code (Todas)
Suma de Errors
35
30
25 Error Type
Intentional / no compliance
20
Procedural
15 Communication
Decision
10
Proficiency
5
0
Predepart / Taxi Takeoff / Climb Cruise Des / App / Land Taxi-in
Gabinete de Seguridad Unidad CRM y FFHH Phase of flight

LOSA FUTURA
After LOSA observations, Futura will
implement the Organizational
corrective actions
but, what to do with all these

data of Systemic nature?

LOSA FUTURA
Systemic Nature Threats
ATC
Slots
Aeropuerto
Callsigns
Total
0 5 10 15 20 25 30 35
Porcentaje N de Eventos
LOSA FUTURA
FUTURA
Safety Cabinet
CRM & Human Factors Unit
unidadcrm@futura-aer.com

ICAO-IATA LOSA & TEM Conference Dublin,Ireland, 5 to 7November 2003 Collecting Safety Data from In Service Occurrences and Learning
from them
l Where did we learn from?
4 Goldmines of information if put to good business practice

Presented by l What did we deliver ?
4 Briefing Notes
4 FCOM / QRH enhancements
4 Operational and Human Factors Studies
J-J Speyer
Director Operational Evaluation,
Human Factors and Communication l What did we learn ?
4 Lessons -learned in design, operations, training
AIRBUS 2003 S.A.S. All rights reserved. Confidential and proprietary document.
Collecting Safety Data from In Service 4 Assessing Robustness of Operational & Training Assumptions
Prevention Strategies and Safety Awareness Information
Occurrences and Learning from them 4
l Where are we going ?
by J-J Speyer and Michel Trmaud, Airbus Customer Services 4 Threat and Error Management
4 Continued Flight Safety Enhancement

Collecting Safety Data from In Service Occurrences and Learning from them ICAO-IATA LOSA & TEM Conference
Collecting Safety Data from In Service Occurrences And learning from them through proper analysis
Goldmines of information if put to good business practice Need for Analytical Methods and Tools ( GAIN WG B )
COLLECT
COLLECT
l Identify reports with similar characteristics,
ANALYZE
ANALYZE
1 l Extract information from reports in a structured way,
FORMULATE AND
FORMULATE AND
IMPLEMENT
IMPLEMENT
l Derive patterns and trends in large amounts of data,
MONITOR
MONITOR l Identify important , contextual and event sequence data,
4DATA and TEXT MINING METHODS,

From Role of Analytical Methods and Tools ( GAIN )
4Air Safety and Human Factors Reports, FOQA Data,
Collecting Safety Data from In Service Occurrences and Learning from them ICAO-IATA LOSA & TEM Conference Collecting Safety Data from In Service Occurrences and Learning from them ICAO-IATA LOSA & TEM Conference
Need for Multiple Reporting Sources Need for Analysis Workflow
Understand the
Understand the Facts
Facts
ISOs What ?? Why
What Why ??
Operational Review Applicable

Applicable
Review
Support Feedback
Standards
Standards
Events
Training
Feedback Identify Operational
Identify Operational
and Human
and Human Factors
Factors
FOM Feedback ASRS,

LOMS / AIRS / LOAS
BASIS SIE, Identify
Identify
Lessons-learned
Lessons-learned
Confidential
Reports
Problems Statements Develop
Develop
Contributing Factors Prevention Strategies
Other
Recommendations Events
Develop Safety
Develop Safety
Awareness Tools
Awareness Tools
Page 1
Need to reach Informed Decision Making Need to look Beyond Reported Events : Prevention
l Events categorization : Passenger aircraft, At 37 000 ft

Avionics Smoke warning + Cargo Compartment Fire
4Event consequences Heavy smoke in cockpit

Crew sequence of actions ?
4Event descriptors : Freighter aircraft, In final approach
System of function affected Avionics Smoke warning + Cargo Compartment Fire

Heavy smoke in cockpit
Condition descriptor No crew action, landing continued
runway vacated and aircraft evacuated
l Operational and human factors markers

4Subtly intertwined and embedded into one another
l Assessing robustness of :
4Operational assumptions Reported Potential
Event Event
4Training assumptions ( Prior - Precursor )
Back to Analysis Workflow Operational and Human Factors Analysis
Understand the
Understand the Facts
What ?? Why
Facts
Why ??
l Tailored to Airbus needs as aircraft manufacturer :
What
4 Identify recurring factors

Review Applicable
Review Applicable
Standards
Standards 4 Identify recurring patterns
4 Identify lessons -learned
and Human
and Human Factors
Factors
2 4 Develop prevention strategies
Identify
Identify
Lessons-learned
4 Support lessons-learned process
( design, procedures, training )
Lessons-learned
Develop
Develop
Prevention Strategies l Markers optimised for compatibility with :
4 ICAO ADREP classification ( with translation tables )
Develop Safety
Develop Safety
Awareness Tools
Tools
Awareness 4 BASIS,AIRS, IATA STEADES, BASI SIAM ( keywords )
Using Databases for Safety Data Management Involved Domains and Factors
Consequences Sub Consequences Descriptors Descriptors Narrative Situation Recognition Procedures Crew Performance Environment&
level 1 level 2 DENOM Diagnosis Circumstances PROBLEM PROCEDURE WEATHER CHALLENGED
MISSION GOAROUND ATC ATM Go-around due stabilised approach criteria not met. unstabilized at approach gate inadequateATCinstruction
SITUATION RECOGNITION / DIAGNOSIS
RECOGNITION
PROCEDURE(S)
CONDITIONS AIRCRAFT OPERATIONAL
MS
IC NONE APU OTHER APUnotshutdownuntilpassingFL220 Situation
complex combination of proc's
workloadmanagement
NATPChecklists In-flightfailure
CREW
FLIGHTPATH
CRM OPERATIONAL SYSTEMS
CONDITION/
DESIGN
ERGONOMICS
ASSUMPTIONS
OR
workloadmanagement WARNINGS VISIBILITY ACTIONS FACTORS ENVIRONMENT RUNWAY
Diagnosis CREW
MISSION GOAROUND L/G NO EXTAND Go-aroundfrom1000ftduecrewdistraction. late problem recognition unstabilized at approach gate incorrect configuration by crew & COCKPIT & EXECUTION CONDITION CONFIGURATION SAFETY
DIAGNOSIS
NATPSterileCockpit EFFECTS CONTENT PRINCIPLES
FLT PATH PATHDEV
FLT PATH PATHDEV

HANDLING
Consequences
WEATHER
LOADING
LIGHTNING
Tookoffbeforereceivingfinalloadsheet
Altitude deviation of 600ft due flight crew distraction.

late problem recognition
condition not detected Procedures

unchallengedcrewerror
workloadmanagement
operationalstress
pilot/controller communications
COCKPIT /
CREW HMI PROCEDURE PROCEDURE PROCEDURE
NARRATIVES
ECAM warnings? No detection of No procedure? Unsuccessful? Hand flying? Navigation Crew team skills? Effectoffatigue Heavy rain? Dispatch under Automation Challenged
thunderstorm ALERTS CABIN
condition? error? on : MEL? surprise? operational
DIAGNOSIS ASPECTS TYPE ACCESS CONTENTS
ATC SEPARATIONLOSS L/G
Descriptors
NO EXTAND Late lowering of gear due to distractions
Crew Performance approach fast crewpersonalfactors
inadequateseparation
Local warnings?
EFFECTS
Incorrect
Low visibility
of procedure?
Incorrect? Use of
automation? Altitude
Flightcrew
communications?
recognition?
decision-making?
Turbulence?
In-flight failure? Inexplicit
(procedural)
assumption?
MS
IC NONE FUEL IMBALANCE Fuelimbalanceduecrewdistraction. late problem recognition workloadmanagement cabin medical emergency identification deviation? action-taking? warning?
FLT PATH ALTDEV AUTO MODES 300ft altitude deviation due crew distraction. automationnotunderstood
Environment altitudedeviation inadequateATCinstruction
Aural warnings? /assessmentof
condition? Incorrect?
Incomplete?
Use of systems?
Excessive pitch
Planning?
Workload?
Windshear?
Incorrect
configurationby Inexplicit
Challenged
training
assumption?
Othercockpit Coordination of Crosswind?
Circumstances attitude? flightcrew? guidance?

effects? Incomplete? procedures? Undue action on Briefings?
Go-aroundfrom1000ftduetobecomingunstableavoidingflapspeed detected inadvertant action incorrect configuration by crew anycontrol?
MISSION GOAROUND AUTO AP DISC exceedanceafterA/Pdisconnectedwhenflapselectedinerror approach fast
Effectofmultiple Ambiguous? Other aspects /
CREW PERFORMANCE
Excessive bank
Monitoring?
ATC pressure? Tailwind? Other aspects / Inadvertent
Challenged
design safety
angle? factors? interference
simultaneous factors? Operational principle?
Other traffic? with
LIMITATIONS OVRSPD HANDLING STAR Momentary exceedance of bird speed limit due crew distraction late problem recognition inadequatepreparation/planning warnings? error? Contaminated
Unstabilized systems
Links between Back-up? runway?
approach : o r Otherchallenged
procedures? Interruption?
FLT PATH LATDEV HANDLING LOC Fly through localiser due to crew distraction late problem recognition inadequateprioritysetting incorrect configuration by crew Other aspects /
factors?
Actionslip? High ?
Fast ?
Decision-making? AIRCRAFT
Slippery runway?
automation? aspects ?
ATC SEPARATIONLOSS L/G NO EXTAND Gearnotselecteddownat2000ft,butdownat1500ft delayedaction low level windshear Other aspects / Distraction?
inadequateATCinstruction
factors? CREW
Non adherence to Low ? ATTITUDE
ATC COMLOSS AUTO A/THR Reverse idle thrust not selected during landing roll. Autothrust left engaged condition not detected unchallengedcrewerror inadequateATCinstruction PROCEDURE CREW SOPs? Slow ? THREATLossof Visual
CREW
Difficult use of
duringlandingrollduetodistractiononR/T ERROR lateral offset ? situational Airport aspects /

AND
illusions? anycontrol?
EXECUTION ACTIONS Experience MANAGEMENT

awareness? factors?
COORDINATION
FLT PATH SWERVE AUTO A/THR 5kt flap speed excedence due crew distraction condition not detected complex combination of proc's workloadmanagement inadequateATCinstruction
MANAGEMENT
factor? Other aspects /
factors? Flight crew / Navaids aspects
FLIGHT PATH
Other aspects /
factors?
Other aspects /
factors?
g:
Skill factor?
cabincrew
coordination?
/factors? CONTROL
in
FLT PATH ALTDEV ALT SET Incorrect altimeter setting after map shift late problem recognition inadequatepreparation/planning navaid factors
rn ad
Navigation charts
:
Other aspects / Other aspects /
ATC COMLOSS ALT SET Crew missed setting standard altimeter setting when cleared to flight level. late problem recognition complex combination of proc's unchallengedcrewerror
n ginpartlcomply understood re
/ information
Wa
factors? factors? aspects / factors?
i ad
ENVIRONMENT & CIRCUMSTANCES
rn o
t t
Altitude bust of 450ft during departure from LGW. ete condition degradedsituationalawareness incorrect configuration by crew
re
FLT PATH ALTDEV NAV MAPSHIFT Other aspects /
W a500feet. t o complex combination of proc's p

factors?
em
ALT SEPARATIONLOSS FLTCRL CONFIG Late landing flap selected due crew distraction. Stable by late problem recognition degradedsituationalawareness inadequateATCinstruction
p noiste t
at
AIRCRAFT
elatemproblemrecognition
CREWFLT CREWAFFECTED WN
I DOW CRACK Loud whistling on flight deck during climb and descent. intra-cockpitcommunications
CREW t
WEATHER /
t t OPERATIONAL RUNWAY SYSTEMS
o ORGANIZATIONAL
o n
ATC COMLOSS HANDLING UNSTABAPP High energy approach due distraction, aircraft stable by 600ft. workloadmanagement inadequateATCinstruction
t a
ENVIRONMENTAL
ENVIRONMENT CONDITIONS CONDITION / FACTORS FACTORS
no D
ATC SEPARATIONLOSS L/G NO EXTAND Gear not down til 880ft, due distraction. late problem recognition workloadmanagement inadequateseparation
CONDITIONS
CONFIG
Do
1013 not set during climb. Rectified during level off. incomplete condition NATPChecklists pilot/controller communications
ATC COMLOSS ALT SET notunderstood
FLT PATH TAXIEXCURS FMS RESET Aircraft left paved surface whilst taxing due crew distraction. condition not detected degradedsituationalawareness unfavorable runway assigment
NATPSOP's
Page 2
Collecting Safety Data from In Service Occurrences and Learning
Operational and Human Factors Markers ( excerpt ) from them
CREW PERFORMANCE

AIRCRAFT
CREW ATTITUDE
PROCEDURE CREW
ERROR
THREAT
AND
CREW l What did we deliver ?
EXECUTION ACTIONS MANAGEMENT COORDINATION
MANAGEMENT FLIGHT PATH
4 Briefing Notes
CONTROL
Incorrect Detected 4 FCOM / QRH enhancements

Correct and Intra-cockpit
Use of Inadvertent TBD Overspeed
Complete Flight Guidance Action Communications 4 Operational and Human Factors Studies
Incorrect / Incorrect Undetected Gross

Cockpit / Cabin l What did we learn ?
Unwarranted Use of Inadvertent TBD Navigation
Communications
Deviation Systems Action Error 4 Lessons -learned in design, operations, training
4 Assessing Robustness of Operational & Training Assumptions
Detected
Inadequate Navigation Cultural
Delayed Action TBD
Hand-flying
Slip
Deviation Factors 4 Prevention Strategies and Safety Awareness Information
l Where are we going ?

Undetected Inadequate
Untimely Altitude
Incomplete Action TBD Preparation / 4 Threat and Error Management
Action Deviation
Slip Planning

Note : Excerpt from actual list of factors
Airbus / FSF ALAR Briefing Notes Airbus / FSF ALAR Briefing Notes
l Present each topic in a global context
l Recall the relevant operational standards,

flying techniques and best practices
l List the factors that may lead flight crews to deviate

from relevant standards ( awareness )
3
l Provide or suggest company prevention strategies and
personal lines-of-defense ( prevention or correction )
l Provide a summary of key points ( training tips )
l Cross-reference related ICAO recommendations

and JAR / FAR regulations
FCOM / QRH Enhancements ( examples ) Operational and Human Factors Studies performed
l New criteria ( cues ) for the recognition of fuel leaks l In-service occurrences ( general overview )
l Smoke / smell procedures
l Runway excursions and overruns
l Recognition of engine tail pipe fire versus engine
compartment fire ( ENG FIRE ) l Non
Non--adherence to published procedures
l Flight with unreliable airspeed l Unstabilized approaches

l Tailstrike prevention and bounce recovery
l Situations beyond the scope of published procedures
l SOPs enhancement for approach-and- landing accidents
reduction ( ALAR ) l Situational awareness
l New QRH procedure for response to stick-shaker l Fatigue and alertness

activation on takeoff ( WBs only )
l Distractions and interruptions
Page 3
Non--Adherence to Procedures involve several sub categories
Non Screening Reports relative to Non
Non--Adherence to Procedures
From complex HFs to basic sediments to start the cycle
of good business practice feedback l What is absence of recognition other than trap into
l No perception of relevant information (input), lack of Rigor ?
DESIGN
l Misperception of information (pattern matching), l How can recognition issue lead
self
:to Procedural
Adherence? in it
l Procedural Design (input,interpretation), end n
as a
TRAINING
l Procedural Experience/ Training (long !
tupidterm memory),
l How canbprocedural
s u
t no
t
( OK
)
subtlety lead to correct
ple S t o r
Adherence? s )
S
l Decision Making Heuristics
ep it
im(decision making), Fac e n e s O K
an war (
: Ke OPERATIONS Huml What al A
KISS
l CRM (awareness & attention management), )
tionis rushed eaction other Kthan premature nc (O
itua
s SDecision igila ns
ue &
VMaking? ptio )
l Cultural Aspects (influencing factors),
a t ig I n terru (TBD
F &
l Personality Aspects/Attitudes (influencing factors), lsWhatraisctundue ions interpretation other than biased
INFLUENCING FACTORS s Di
Decision
st
l Er
Making?
a
rors
l Situational Factors (influencing factors), d u r
oce
s Pr
Recognition issue leading to Non Adherence Screening Reports relative to Situational Awareness
EVENT RECOGNITION RECOGNITION PROCEDURE REMARKS
WARNING CREW DIAGNOSIS
?
EXECUTION Processed Incident Topics (8 families)
ce
ENGTAILPIPEFIRE NO ENG FIRE WARNING ENG TAIL PIPE FIRE E N G F I R E D R I L L A P P L I E D he
ren l Crew Awareness Messages and Systems,
E N G T A I L P I P E F I R E N O T A P P L Id
lA
A310 TRIGGERED REPORTEDBYATC ED
d ura l On-Ground Procedures and Maneuvers,

e
roc F O L L O W E D l Door Warnings,
ENG FLAMEOUT DUE UNDETECTED FUELOUTRTANK QRH PROCEDURE NOT FUEL FEED
TOFUELSTARVATION AUTOFEEDFAULT LOWLEVEL FROMOUTRTANK
D I S R E G A R D E D A S S U S P E C T E DP
A 300 - 600
S P U R I O Ud
S
to FOLLOWINGFAULT
4 l Engines,
lea l Flight Path Control,
As
ue
THROTTLE LEVER THROTTLE LEVER NOT AT IDLE
is A B O U T 1 0 F E E T B A C K
IRCRAFT PUSHED TRACTOR NON COMPLIANCE WITH 6 EVENTS SO FAR
NOT AT IDLE NOT SET AT IDLE n SOPs FOR SETTING THR l Fuel System Management,
Ao
M E S S A G E C R tE i
ni
DURING MES TED LVR TO IDLE BEFORE MES
og l Navigation and Communications,

A320
n rec l Thrust Management and RTO,

UNWARRANTEDIFSD
ca DECREASINGOILPRESSURE MISUNDERSTANDING UNWARRANTEDENGINE TYPICALCASEOF
BASED ON DECREASING
w BUT NO LOW OIL PRESSURE OF OIL PRESSURE AND LOP SHUTDOWN HURRIEDREACTION
OILPRESSURE
A320
Ho WARNING INDICATIONS CLIMB PHASE
Incidents relative to Crew Awareness Messages and Systems Incidents relative to Crew Awareness Messages and Systems
OPS SA Level 1 SA Level 2 SA Level 3
A/C EVENT EVENTNARRATIVE OPERATIONAL ANALYSIS PRACTICAL OUTCOME from LESSONS LEARNED
CODE 1 PERCEPTION COMPREHENSION PROJECTION
A321 FUEL CLOG ENG 2 FUEL FILTER CLOG in cruise with Crew Awareness message that Maintenance Not difficult to detect
ENG 2 FUEL FILTER CLOG crew awareness
message activation
Action is Due.
Recommendations
P12 hinge around decision-making
and expectation of system response,
A319 IDG IDG 1 LOSS FOLLOWING DISPATCH WITH IDG 2 Crew Awareness message, no other message as
INOP. A i r c r a f t d i s p a t c h w i t h I D G 2 I N O P a n d A P U long as APU GEN is operative.Need to confirm that Not difficult to detect
GEN ON as per MEL followed by IDG 1 ( GEN 1 ) the MMEL repair category for the IDG is consistent
failure with the current system reliability (IDG MTBUR) P12
A320 SMOKE AFT CARGO COMPT SMOKE + AFT LAV SMOKE + LAV SMOKE is a Crew Awareness message.
OIL SMELL IN CABIN Not difficult to detect Specific training related to the new SMOKE/AVNCS SMOKE procedure was
A330 EGPWS
IFTB and Root cause traced to APU oil leakage
SPURIOUS EGPWS WARNING The crew perception of inability to accelerate and

Lessons Learned
P12 fertilize: planned when this procedure was revised.
low rate of climb was confirmed to be the result of
, Not difficult to detect
ells
continued flight on the back side of the power curve
Design (ECAM WG for A380 Crew Awareness Messages)

P12
A319
m RTO DUE TO FORWARD CARGO DOOR WARNING ECAM warnings (crew awareness message) and
,S
flight phase inhibitions (ECAM warning inhibited
during phases 1, 4, 5, 7, 8 ,10) are the same for all Not difficult to detect OIT only since no specific operational recommendation.
WS
doors.
s,
P12
, Documentation (FCOM,QRH),
GP her ger
) e t
Correct monitoring and data FCOM and QRH references for Volcanic Ash Encounter recalled. General
a rig
A320 VOLC ASH VOLCANIC ASH ENCOUNTER AFTER TAKEOFF
,( E resulting in both both front windshields cracked
T observation Volcanic Ash Awareness information in OPSCONF proceedings.
W
AM ent ion OpsNotRecommendations (OIT/FOT)
P12
A320 GPWS GPWS MODE 2 WARNING DURING INITIAL Aircraft vectored below the Sector Safety Altitude
ent
Basic airmanship to ensure proper approach speed when near the ground.
difficult to detect
EC em
APPROACH but still above 4100 ft SSA.The aircraft speed was
Att
P12
ncl
too high with regard to the altitude.
om
A321 OIL ENG BEARING 4 OIL SYS - HI PRESS CREW Crew Awareness messages do not call for the
f r o I n d AWARENESS MESSAGE ACTIVATION. flight crew to refer to the FCOM but referring it is
Training
Not difficult to detect Overreliance on default values Smoke
(Smoke/Avionics Procedure)
Overprojection of current trends The ECAM WG agreed that expanded info is needed for each Crew
et
Precautionary diversion initiated in cruise.A/C standard practice.
s sa
Awareness message on A380.
age
redispatched under MEL pending trouble shooting
P12 N23 N32
ativ nes
andcorrectiveaction.
ess rel are

A320 WINDOW WINDOW ANTI ICE ECAM MESSAGE + SPARKLES. This sparkling event is considered as a "window
ANTI ICE L WINDOW ECAM Crew Awareness arcing" event.Event crew elected to select the Not difficult to detect Lack of/ poor mental models Overprojection of current trends
M e s w
message activation.Sparkles observed underneath WINDOW HEAT to OFF, in the absence of ECAM and
P12 (absence of ECAM and (IFTB to avoid further window
ag
left sliding window (extensible cord found chaffed FCOM guidance, and to perform an IFTB considering
wA
FCOM guidance) cracking)
ess
and damaged by conduit tube) the presence of sparkles (and possibly the further
potential fo
Cre
A320 HYD HYD - G RSVR LO AIR PRESS + G RSVR LO LVL + Apparently, lack of crew actions in response to the N21 N32
M
Y RSVR OVHT - RUNWAY EXCURSION initial HYD G LO LVL ECAM warning.
Correct Perception Lack of/ poor mental model Overprojection of current trends Review our general recommendations regarding the response to red and
ral
During landing, crew selected the A/SKID & NW Apparently, lack of RTO decision following the amber warnings (i.e., immediate versus possibly-differed action)
STRG switch to OFF after activation of both activation of an amber warning in phase 2 leading to P14 N21 N32
e n e reversers.Directional control difficulties experienced

andRunwayexcursion.
NATP. Review our recommendations for rejected takeoff (response to ECAM
warnings and cockpit effects below 100 kt versus above
G
A320 SMELL BURNING SMELL FROM SEAT ELECTRONIC Burning smell from SEB is an "identified type-of- Not difficult to detect Overreliance on default values Overprojection of current trends
B O X .D u r i n g c r u i s e b u r n i n g s m e l l d e t e c t e d a r o u n d event".In most such events no cabin crew action is
seat rows 1-3 left but no smoke. Precautionary taken in flight until maintenance action is performed P12 N23 N32
diversion. Maintenance found no evidence of burn (affected SEB remains powered during the
wires or damaged harnesses. diversion)
Page 4
Screening Reports relative to Fatigue & Alertness As catalyzed by Operational and Human Factors
l Some incident -types were reportedly caused / influenced l Situation Recognition / Diagnosis issues pertaining to e.g. :
under pressure of fatigue & alertness / vigilance : 4 Condition not understood, inadvertant system interference,
late problem recognition,
l Procedural Issues pertaining to e.g. :
4Air Traffic Management, ATC Separation Issues
4 NATP/SOPs, Entry into QRH unclear, Situations beyond Published
4FlightPath Issues ( swerves/deviations, altitude deviations, Procedures, Action Steps Unclear, Deviations from Procedures,
speed drops, over-speed exceedances, unstable approaches) l Crew Performance issues mainly pertaining to e.g. :
4 Degraded Situational Awareness, Distraction Management, Workload
4Mission Emergencies, go-arounds & diversions, Management, Flight Monitoring, Tactical Decision-Making, unstabilized
In Flight Turn-backs & Shutdowns, at approach gate, inadequate hand-flying,
l Environment and Circumstances mainly pertaining to e.g. :
4 Pilot/Controller Communications, Inadequate ATC Instruction,
Inadequate Separation, Turbulence, Windshear , Nav Charts Factors,
In-Flight or Hidden Failure, Runway Incursion/ Obstruction, Incorrect
Configuration, Operational Stress, Currency , Spatial Disorientation,
Screening Reports relative to Distractions and Interruptions Distraction caused by / resulting in the specific safety events
l Air Traffic Control A/C communication & separation losses,
l Cabin & Flight crew being affected or incapacitated,

ss ey.
e Ody l Flight Path Control (alt bust / dev,path dev,lat dev,vert dev,
n in th
f d is tractio
o flare,taxi/rwy inc/exc,swerves,spd loss/drops,stall,),
effects x. rivers
.
th e
tating a
devas ears with w speak to
bus d 5 l Non respect of Limitations (overspeeds ,flap limit speeds),
ribed lock their not to
r desc b sked
Home posed to gers are a l Threat for CFIT/ALAR (stabilization criteria not met,
p ro s e n
And pas
days, unstable/rushed/fast/slow approaches,),
Nowa
l Go-around/IFTB/Delayed starts/diversions,
l Loss of System Functions,
Themselves caused by / resulting in specific safety issues Themselves caused by / resulting in specific safety issues
l Flight Control Difficulties (flaps not set,speed-brakes,gear l Ergonomics (plate/map holders,cockpit displays,sun visors),
not extended/retracted,delayed,configuration warnings),
l Noises (engines,ATC buzzer,headset chatter,radio altitude
l Difficulties with automation (auto throttle,mode callout,warning,intercom,interference,mobiles,loudspeaker),
management,),
l MEL/SOPs,checklists,procedural difficulties (omission,
l Systems management (APU not shut down,parking item not completed / skipped / forgotten ),
brakes not set,anti-ice not selected,hot start),
l Language confusion/difficulties,call sign misunderstandings,
l Fuel management (pump selections, limitations,
l Passenger / cabin distraction,cabin secure reports,
quantities,imbalances,),
l Hazard linked with kites,birds,wildlife,dogs,

l Cabin pressurization difficulties,
l Airport infrastructure (lighting,marking,),
l Altimeter barometer setting,
l Burning smell/smoke,
l TCAS/GPWS alerts,
Page 5
Themselves caused by / resulting in specific safety issues from them
l Navigation issues (FMS data introduction and databases,
flying through localizer,flying past radio beacon,)
l What did we deliver ?
l Weather issues (lightning,fog,haze,hail,turbulence,wake
turbulence,C/Bs,wind-shear) and avoidance thereof, 4 Briefing Notes
l Technical system failures or system failures (preset altitude 4 FCOM / QRH enhancements
not acquired,descend through selected altitude), 4 Operational and Human Factors Studies
l Load-sheet figures not obtained/not introduced, suspicious l What did we learn ?

items,fuel quantities,LMCs not communicated), 4 Lessons -learned in design, operations, training
l Fatigue and Alertness issues, 4 Assessing Robustness of Operational & Training Assumptions
4 Prevention Strategies and Safety Awareness Information
l Crew Resource Management issues,
l Concurrent Task Management issues, l Where are we going ?
l As catalyzed by Operational and Human Factors, 4 Threat and Error Management

Lessons-learned in design, operations, training Assessing Robustness of Operational Assumptions
Typically,in the spirit of the OEAP, Lessons Learned fertilized: l Challenging our operational model of human
performance :
l Design,
4 ECAM WG for future standards and A380, 4Standard Operating Procedures - SOPs
( e.g., task sharing, callouts )
4 A380 DSSs,
4Rules for abnormal and emergency conditions
l Documentation,
4 FCOM,QRH,MMEL,FCOM Bulletins,
6 ( e.g., ECAM / QRH philosophy )
4Threat-related prevention strategies :
l Operational Procedures and Recommendations,
RTO, Windshear , CFIT, ALAs, Turbulence,
4 /FOTs,
Wake Turbulence, Volcanic Ash Encounter,
4 SOPs, Loss of Control, Runway Incursions, Midair Collision
l Content of Dissemination Processes, Avoidance, Altitude Deviation / Flight Level Bust,
4 OLMs, OPSConfs, Hangar Flying, Other weather threats and environmental hazards
l Training, such as fatigue,
Assessing Robustness of Training Assumptions Prevention and Recovery Strategies : Manage distractions
l Recognize and Identify the disruption,

l Challenging our training model of human
performance : l Re
Re--establish Situational Awareness :
4Basic elements of airmanship 4Identify
: WHAT WAS I DOING?
4Ask: WHERE WAS I INTERRUPTED?
4CRM best practices ( e.g., mutual backup, cross-
check, decision-making, ) 4Decide / Act :
WHAT DECISION OR ACTION to get back on track?

4HF best practices ( e.g., situational awareness,
pilot/controller communications, )
l The following decision-
decision-making process to be applied :
4Operations Golden Rules 4Prioritize : FLY,NAVIGATE,COMMUNICATE and MANAGE!

4Training items (e.g., briefings, simulator drills, IOE, ) 4Plan : TAKE TIME TO MAKE TIME!
4Verify : THAT POSTPONED ACTIONS WERE FINALIZED!
Flying on line as trained
l Verbalize to mark interruptions and protect what you are doing,
Page 6
Prevention and Recovery Strategies : Manage Fatigue & Alertness Prevention and Recovery Strategies : Monitoring Skills
l How does your body manifest being tired? l Traditional CRM has excelled to challenge or speak up when
something seems unsafe,
4To challenge one must detect which requires effective monitoring,
The buffer zone
l What effect does being tired or being less alert have on your
monitoring behavior ? Mental Resources
to control Warning Signals:
l Traditionally monitoring has not been held as aYoure
Situation primarylosingtask.
Awareness control!
4PNF duties typically include handling radio comms, operating
D d a l e 1996- 2000
Loss of
l How does diminished alertness affect your performance in gear and flaps, keeping a flight log. control
flight ?
In control
4Monitoring is typically not listed, giving the impression it is a
Mental Resources
secondary to other tasks, to control
Actions
l How did you cope with poor alertness in the past ?
ACRM - 2 0 0 0 Module 2 Situation Control- V4 u# 1 5/19

Prevention and Recovery Strategies : Think & Trap Errors from them
l Training for improved monitoring in the simulator requiring l Where did we learn from?
instructor patience. 4 Goldmines of information if put to good business practice
l What did we deliver ?

E should catch
l Students Perceive Non-timely detected
their own mistakes.
Situation
Mitigate 4 Briefing Notes
errorsSto be better debriefed.
Awareness 4 FCOM / QRH enhancements
S Comprehend Trap 7 4 Operational and Human Factors Studies
l LOSAAfound that crews failed to detect many errors noted by
Threat l What did we learn ?
cockpit
I observers.
Management Project Avoid 4 Lessons -learned in design, operations, training
4 Assessing Robustness of Operational & Training Assumptions

l Improving & sharpening
ESSAI : Enhanced crew
Safety through monitoring
S ituation skills inwith
Awareness Integration ESSAI
training by:
(EEC program)
4 Prevention Strategies and Safety Awareness Information
4Minimising loss of Situation Awareness (SA) l Where are we going ?
4Increasing Threat Management (TM) skills 4 Threat and Error Management

TEM in the frame of Continued Flight Safety Enhancement Change Process : Continued Flight Safety Enhancement
Threats in the context

FOM Tools, Methods
and Training
Threat Management Safety Awareness

Information / Publications
FOM
Feedback
Errors
Prevention Strategies Events
and Awareness Tools and/or Observations
Error
Management
Contributing Factors Consolidation

and with TEM
Lessons- learned
Operational and Airbus

Human Factors Analysis Operators

Page 7
Thank you
Page 8
The Boeing Safety Approach:
The Value of Collaboration
R. Curtis Graeber, Ph.D.

Boeing Commercial Airplanes
Seattle, Washington, USA
First ICAO-IATA LOSA & TEM Conference

Dublin, Ireland
5-7 November 2003
Todays Topics
Boeings Safety Approach

Boeing-TLC Collaboration
Boeings use of LOSA data
Some observations
Next Steps
W016J.2
Boeings Safety Approach
Be proactive
Investigate accidents and incidents
Rely on experience of our customers
Continuously improve Airplane Life Cycle
process
Seek Global Involvement
Airlines
Manufacturers
Regulatory agencies
Safety and flight crew organizations
W016J.3
Because the Aviation System is Complex,
All Parts of Industry Must Work Together
~800 airlines ~200 languages
1,350 major
airports
Over 200 countries
150,000+ Over 14,000 airplanes

flight crew (Western built)
Boeings Safety Approach: HF Research
Rely on data to drive safety related decisions
Conduct and support internal safety research
W016J.6
747-400 Capt Eye Fixations
Collaborate globally on external research
NASA/FAA
NLR
Madrid BRTC
Universities (Illinois, Ohio State, Cranfield, Massey, etc.)
University of Texas HF Project/The LOSA Collaborative (TLC)
Create and disseminate safety tools and

processes
Flight Operations Support Personnel (FOSP)
Procedural Event Analysis Tool (PEAT)
Maintenance Error Decision Aid (MEDA)
Line Operations Safety Audits (LOSA)
W016J.8
Threat & Error Management (TEM) Model
Threats
Threat Induced
Accident
Crew Error
Crew Error
Responses
Undesired
Aircraft State

Responses Accident
W016J.9
TEM Can Guide Human Factors Research
Boeing Research & Technology Center,
Madrid, Spain
Applied cross-cultural research

TEM framework for comparability and consistency
Program aimed at capturing the contextual factors
that affect performance
Formal agreement between BRTC and the
Spanish Professional Pilots College
Support from SEPLA/Spain and ASPA/Mexico
W016J.10
TLC Boeing Collaboration Agreement
A mutually beneficial collaboration:
Agreement signed May 2002
TLC has provided Boeing 1,100 de-identified LOSA

observations
Represents a mutually valuable research partnership
Boeing is using the data to help TLC to improve LOSA
TLC can provide Boeing customers with a two-way

channel of communication between them and Boeing
W016J.11
TLC
Gains manufacturers perspective and expertise in
promoting global aviation safety
Intellectual scientific and expert technical support
Detailed design knowledge
Ability to compare results across other databases
Increased opportunity to benefit all airlines
Economic and Manpower Resources
Boeing
Airlines
W016J.12
TLC .......................................... Value 9

Boeing
Access to information about crew errors during
normal operations
Insights about how crews recover from errors
Invaluable, objective insight into the design and
operation of our products
Synergistic research partnership
Airlines
W016J.13
Experienced-Based Lifetime Safety Cycle
New
Technology
Laboratory Flight
Previous SR&Os Testing Testing
DR&O Design
FARs, reviews
JARs Analyses
Analyses
Customer Testing
requirements Delivery
Boeing
design and
Design Produce
Validate
and SA
In-Service
LO
regulatory Operation
requirements certify
Service
bulletins,
etc. DATA
300,000
telexes per
Lessons BCA In-Service Safety Process
year
Learned
Potential
Data
Accident
Issue
Analysis
Accident
Issue Classification
Corrective Action
Management
85,000
Problem Modification Test Data
Incident
Investigation
COSP Bulletin
Issue Database
accidents,
Data Board EIBs*
Resolution Review requirements PASC incidents to

Reports Board In-Service
Events
Functional
Safety
Councils
SRBs*
SRP
date
Safety Review
Board Members
Potential
Supplier
Flight Test Safety Issues
Factory CSD
Lead Airline Process Involvement
Continuous feedback of information COSP filter

FAA * EIB = Engineering Investigation Board
SRB = Safety Review Board 7-23-02 SPS-032
BCA In-Service Safety Process

To Affect Safety, Enhancements Must Improve
the Operation of Current Production Airplanes
25000
New Designs
20000
Number of
airplanes at 15000 Total Airplanes Produced
years end After 1998
10000
5000 Retained Fleet
Out of Production Models (as of 1998)

0
1998 2000 2005 2010 2015
3/25/98 VIS-2-7W
TLC ......................................... Value 9

Boeing ..................................... Value 9
Airlines
Availability of manufacturers perspective
Invaluable research partnership
Convergent Validity
Safety Related Products
Airplane design
Safety guidelines
Training guidance
Procedure evaluation and development
W016J.16
TLC ......................................... Value 9

Boeing ..................................... Value 9
Airlines .................................... Value 9
The Value of Collaboration is

INVALUABLE!
W016J.17
Improving the Human Interface
Types of Possible Data Sources:
Accidents
Incidents (ASAP)
FOQA
LOSA
W016J.18
Accident Data
Example: NTSB, AIB, etc.

Can provide extraordinary detail about single
events
Very dependent upon the expertise and
processes of the investigating authority
Poor comparability, and usually insufficient
number, worldwide to understand HF issues
Can be high value data source when applicable
W016J.19
Incident Data
Example: ASRS, CHIRP, ASAP
Key factors frequently identified
Incidents can support precursor search
Extremely high value data source to corroborate
other data
Level of detail varies
Participant recollection can be suspect
Hypothesized factors may not be available
W016J.20
FOQA Data
Proprietary and closely protected by operators

Very detailed objective data
Uniquely suited for multi-flight statistical analysis
Requires information about context to understand
why
High value but limited in scope
W016J.21
LOSA Data
Vary in value
Facts & events = highest value
Background = puts facts & events in context,
high value
Evaluation of flight crew = some value
Can sometimes support rate of occurrence
analysis
Greater accuracy than anecdotal analysis
Very high value data source
W016J.22
LOSA data
Provides more than single flight context

Can provide better detail than incident reports
Value of narrative portion could be substantially
improved with application of reliable data-mining
techniques
Offers potential to include focused observations
on issues of interest
W016J.23
Boeings Use of LOSA Data
Search for threats and errors related to our

fleet
Apply and improve latest data mining tools to
Enable efficient in-depth analysis of
narratives
Further refine techniques and tune the tools
Develop capability to correlate results with
other databases and gain convergent validity
W016J.24
Boeings Use of LOSA Data
Methodology
Quantitative analysis
Text classification of narratives (coding)
Correlation analysis, including other databases
Manual analysis is unavoidable
W016J.25
Boeing Use of LOSA Data
Findings-to-date
Quantitative analysis
Field data is very rich but would benefit from more
accurate/standard recording
Current numbers from the data are sufficient for
simple frequency studies
More observations are needed to find trends and
patterns
Narrative analysis
Unstable approaches
Runway/taxi incursions
Quantitative Analysis
An Example
AIRPLANE TYPE-MODEL % IN DATASET % WITH THREAT/ERROR LINKS
727 4.5
737-200 6.5
737-300 3.2 6.5
737-800 7.5
747-400 10.5 58.1
757 7.1 3.2
767-200 .6
767-300 5.3
767-300ER .9 9.7
767 4.3
777 8.2 9.7
DC10 6.2 6.5
MD11 2.2
MD11 Freighter 3.9
MD80 10.0 3.2
MD88 12.1 3.2
MD90 6.9
W016J.27
Boeing Use of LOSA Data
Findings-to-date (cont.)
Text classification (coding)
Richness of text makes classification difficult
A machine can correctly code with 60-70% accuracy,
so can a human, but the machine is much faster
Humans under-assign / Machines over-assign codes
Humans and machines together can do a very good
job at coding LOSA narratives
Correlation analysis
Correlating within and across databases can reveal
important relationships within LOSA data and with
other safety data
W016J.28
Correlation and Convergence
Operations
SURVEYS ASAP
BASIS
Mitigated Threats
Trapped Errors ASRS
Unmitigated Threats
LOSA Consequential Errors
Incidents, Accidents
FDM PEAT
Lessons Learned
?????
THREATS & DEFENSES
Normal Abnormal
Operations
Operations SURVEYS
ASAP
BASIS
LOSA ASRS
PEAT
FDM
THREATS & DEFENSES

Unmitigated Threats
Mitigated Threats Lessons Learned Consequential Errors
Trapped Errors Operations Incidents, Accidents
Design
Training
Global Safety Outreach
Next Steps
Summarize data findings per airplane model

Obtain more LOSA data
Analyze possible design, training, and
procedure implications for existing and future
models
Initiate convergence study
W016J.31
Conclusions
LOSA provides a unique opportunity to improve
global aviation safety
Boeing endorses and will continue to support
improvements in LOSA
By cooperating, we can greatly enhance LOSAs
potential for improving safety
Consistent implementation is key
Will learn what we dont know
Offers potential sharing of lessons learned
Safety tools must find common ground to better
leverage our lessons learned
W016J.32
Thank You!
W016J.33
Learning about
automation from LOSA
Capt. Carlos Arroyo-Landero

What is IFALPA?
4 The International Federation of
Air Line Pilots Associations
4 Represents more than 100,000 pilots worldwide
4 93 member associations
4 12 Standing Committees (8 technical/4
professional)
4 Permanent observer status at the ICAO Air
Navigation Commission
Dublin, Ireland
ICAO-IATA LOSA & TEM
CONFERENCE
Capt. Carlos Arroyo-Landero / IFALPA November 5th, 2003
IFALPAs Official Position
4 IFALPA is presently in the process of writing and

reviewing policy on auditing systems.
4 IFALPA regards LOSA as a very promising auditing
system which can be beneficial to both pilots and
airline management, when properly implemented.
4 Most important to IFALPA is that an auditing system
such as LOSA is completely confidential and shall
never be used to punish pilots. Participation shall
always be on voluntary basis.
4 Further criteria for LOSA and other auditing systems
are being developed.
Dublin, Ireland
CONFERENCE
IFALPA+LOSA
Simplified Version
4 IFALPA SUPPORTS LOSA AND IS
DEVELOPING POLICY BASED ON THE
HOW TO SET UP A LOSA PROGRAM (by
Robert Sumwalt III) + other inputs
4 IFALPA SUPPORTS THE USE OF LOSA

DATA IN ASSISTING IN AIRCRAFT
DESIGN
Dublin, Ireland
CONFERENCE
IFALPAs HUMAN PERFORMANCE
(HUPER) COMMITTEE, CONSIDERS
LOSA A VERY IMPORTANT SAFETY
INITATIVE
Dublin, Ireland
CONFERENCE
LOSA and the LINE PILOT
Dublin, Ireland
CONFERENCE
The customers
4 The direct beneficiary of this ops audit is the
line pilot
4 this audit will help us to identify problems so that
we can fix them for you, which makes the system
safer for all us to work in
4 The ultimate beneficiaries are the
airline and its customers
4 anytime we can improve our product, our
customers benefit + we are safer/more efficient
Dublin, Ireland
CONFERENCE
The Magic of Culture
4 Is aviation technologically driven?
A simple device: propulsion, wings, control

surfaces: thats all
Social values and negotiation determining
performance, economy, schedules dictates
engineered content, technology.
Culture in the left-hand seat
Technological Acceleration
4 Moores Law: doubling of microchip capacity every

18 months
4 Similar trajectories in horsepower, materials
4 Technological diffusion: an arithmetic process
4 Increasing gap
4 Consequences in a large-scale system
An heterogeneous system
Increasing
gap
Technological growth
Technological
diffusion
t
The Safety Case
S L E
Mismatches at the [cultural] interfaces =

breeding grounds for human error
Dublin, Ireland
CONFERENCE
Automation culture:
Interlocking elements
Just Designing User

culture culture culture
Dublin, Ireland
CONFERENCE
Traditional Nesting of Cultures
PILOT ORGANIZATIONAL
CULTURE
PILOT CULTURE
NATIONAL CULTURE
Dublin, Ireland
CONFERENCE
Risk Management
4 Risk Assessment
Understand hazards
Estimate likelihood
Understand consequences
4 Risk Reduction
Eliminate hazards
Reduce likelihood
Minimize consequences
Dublin, Ireland
CONFERENCE
Human Error
4 It has become fashionable to claim that human error
is implicated in 80-90% of all major accidents. While
probable close to the truth, this statement adds very
little to our understanding of how and why
organizational accidents happen. In the first place, it
could hardly be otherwise, given the range of human
involvement in hazardous systems.
4 Second, the term human error conveys the

impression that all unsafe acts can be lumped into a
single category. But errors take different forms, have
different psychological origins, occur in different parts
of the system and require different methods of
management.
James Reason
Dublin, Ireland
CONFERENCE
What Makes Decisions Difficult?
Sources of Difficulty
Captains
60
Percent of Responses
First Officers
50
40
30
20
10
0
Safety Conflicts Metacognitive Personal/Professional
Dublin, Ireland
CONFERENCE
Risk Perception Drives Decisions
Half Empty
Focus = negative: :
Weather approaching Half Full

Windshear likely Focus = positive:
Avoid risk -> Windshear diminishing
Change plan Accept mitigated risk->
Action: Delay Continue with mod. plan
departure until Action: Review takeoff
weather improves windshear procedures,
Adjust T/O configuration
Dublin, Ireland
CONFERENCE
Culture and Pilot Risk
Management?
4 How are risks
perceived?
4 What are risk attitudes?
4 Quick decisions valued?
4 What role do ALL crew
members play?
4 What goals does
company emphasize
and reward?
4 Uncertainty avoidance?
Dublin, Ireland
CONFERENCE
A plane is a plane, a pilot is a pilot
Its True: There is a professional aviation culture,
an approved and accepted way of doing things.
It includes:
4 How to select
& and train
pilots
4 How to fly a
plane
4 How to look
good in a
uniform
Dublin, Ireland
CONFERENCE
It takes two to tango.. when
using automation
4 But whos
leading?
4 Whos following?
4 And what do you
do if you dont
know the steps??
Dublin, Ireland
CONFERENCE
Prediction
There is a significant correlation between
automation events, and the extent to which
a pilot is technologically involved.
The modern aviation dance step..

1. The closer the fit, the
easier it is to follow
2. The worse the fit, the
more difficult it is to
even recognize the
steps.
Dublin, Ireland
CONFERENCE
Dancing at the Automation Interface:
Four Possibilities
ASSIMILATION INTEGRATION
One size fits all
Follow the Modify to suit
leader conditions
Just Economics?
COSMETIC MARGIN-
COMPLIANCE ALIZATION
Not so easy to Dont even

follow, but lets pretend - not so recognize the steps
successful
Flight Deck Design Philosophies
4 Use new technologies and functional
capabilities only when:
4 They result in clear and distinct
operational or efficiency advantages,
and there is no adverse effect to the
crew-centered interface
No where else does a product last 30
years that has such high risks in
development and such high capital
expenditure!
We must understand pilot actions and
be guided by appropriate philosophies
in design
Airplane Design Role
4 Rely on data to drive safety related decisions
4 Conduct and support internal safety research
Dublin, Ireland
CONFERENCE
747-400 Capt Eye Fixations
Dublin, Ireland
CONFERENCE
Flight Deck Design Philosophies
4 Design systems to be error-tolerant

4 The hierarchy of design alternatives is: simplicity,
redundancy, and automation
Dublin, Ireland
CONFERENCE
Human-Centered Design
Process
Evaluate and
Philosophy Test Designs:
Of Operations: Walkthroughs
Performance Models Performance Tests
Flight Deck Operations Usability Tests
QRH Role Design Reviews
New
Understand Pilots Develop QRH Develop QRH
And Operators: Design Candidate and
Data Collection/Analysis Requirements Designs Design
Guide
Identify
Performance Apply
Issues: Human Factors
Data Collection/Analysis Principles
Dublin, Ireland
CONFERENCE
General Findings
4 The Boeing QRH needs improvement in the
following areas:
Checklist indexing, selection, and organization
Navigation aids and place-keeping
Checklist step design
Technical and performance data design
Operational consequences
Supporting Rationale
Simplified use of English Language
Dublin, Ireland
CONFERENCE
Factors affecting communication
English level
% of pilots with poor level
of proficiency in English,
according to age
Source Amalberti & Racca, 1988
45
40
35
30
25
20
15
10
5
0
<30 30-35 36-39 40-45 45-49 >50
Dublin, Ireland
CONFERENCE
Why is Language Proficiency
Important?
4 There is a need for effective communication in the aviation
system
between airplane and crew
between manufacturer and crew
between airline and crew
between flight crew members
between the flight crew and the cabin crew
between the flight crew and other airline employees outside
of the airplane.
between the flight crew and ATC
4 There are many opportunities for misunderstandings (barriers to
effective communication)
4 English is the language of aviation
Dublin, Ireland
CONFERENCE
Do We Have Any Answers?
4 Yesnot perfect but a good point of departure.
4 Rely on the science of training
9 Findings
9 Principles
9 Strategies
4 Leverage what we know about
9 Teamwork
9 CRM training
9 LOFT
9 Culture (so far)
9 Learning
9 Expertise
Dublin, Ireland
CONFERENCE
Power Distance
4 Expectations and acceptance of uneven distribution of
power
4 Influences planning speed and information flow
4 High Power Distance: Interactions are hierarchical
4 Low Power Distance: Interactions are (more) egalitarian
Dublin, Ireland
CONFERENCE
Adapting to Mismatches:
Conceptual Modeling
4 Procedures and equipment designed by and for one

national group are routinely adopted by other nations
4 Conceptual modeling captures complex systems to

simulate the impact of cognitive differences
4 Incorporating cognition differences improves conceptual

modeling of command decision processes.
Dublin, Ireland
CONFERENCE
Current and Future Directions
4 Identify cultural vulnerabilities of aviation
demands
4 Explore additional aviation roles, i.e. ATC
personnel
4 Learn about less studied regions like South
America
4 Detect safety related cognitive differences
4 Accommodate cognitive differences with training
and design
Dublin, Ireland
CONFERENCE
Frequent Questions and Concerns
4 There are many contributions to variability.

What about personality? Professional culture?
4 Research literature is based on lab work, on
culturally insensitive tools, and on old data. Is
this a reasonable base?
4 Every person is unique. Are we stereotyping?
4 What about the immediate dynamics?
4 Is it ethically justifiable to treat different groups
differently?
Dublin, Ireland
CONFERENCE
Culture :
a question of
Knowledge?
Knowledge :
a question of Culture?
Airbus is in constant interaction with Pilots
ICAO-IATA LOSA & Dublin, Ireland

TEM CONFERENCE Capt. Carlos Arroyo-Landero / IFALPA
November 5th, 2003
Setting Training Standards for all
A319/A320/A321 A340
A330
Dublin, Ireland
CONFERENCE
Pilot population Background
Last aircraft
Light / fighter
Eastern aircraft
Old 3 crew members
Airbus
16,8%
33,5%
16,2% Glasscockpit
28,6%
Old 2 crew members
Dublin, Ireland
CONFERENCE
Identifying Pilot Background & Entry Level
Airbus Airlines
Operational philosophy Airline specifics
Documentation Pilot background
Procedures Authorities
Safety Safety
Authorities Cost
Training
Training philosophy
Course content definition
Teaching techniques
Evaluation
Instructors training
Training media
Dublin, Ireland
CONFERENCE
Standardizing on Operational Rules
4 10 high level design rules have led to these Ops Rules :
Fly these aircraft like any other aircraft.

Fly, navigate, communicate - in that order.
One head up at all times.
Cross check the accuracy of the FMS.
Know your FMA at all times.
When things dont go as expected - take over.
Use proper level of automation for the task.
Practice task sharing and back up each other.
And the Largest Focus on the safety part ...
Relating theory to operation: the most difficult game
in town as said Professor James Reason
Situation Perceive Mitigate

Awareness
Comprehend Trap
Threat
Management Project Avoid
Automation can affect every piece of the system!

It goes to the heart of knowledge and awareness.
Dublin, Ireland
CONFERENCE
IS THERE SOMETHING THAT
COMPUTERS CANT DO?
4 Computers manage symbols however...they just
process their physical form, not their meaning. (Varela)
4 In consequence, Computers cannot handle events
that require knowing the meaning of a situation.
4 The knowledge of the meaning is required especially
in unforeseen situations.
4 For that reason, we can conclude that computers are
not a good solution to manage unforeseen situations.
Dublin, Ireland
CONFERENCE
Is opacity good?
4 On information systems, every level of design is
independent from the ones preceding it, maintaining
its own logic.
4 Therefore, Specialists can dedicate themselves to
their specific fields with out having to worry about
the upper or lower levels of design other than their
own in the same system.
4 This means that the same specialist can work with
information systems similar to those of HIS LEVEL
OF KNOWLEDGE even if they are radically different
from the levels he may have below.
4 Note: In Aviation, this is the foundation on which we base multi-
qualification.
Dublin, Ireland
CONFERENCE
FROM A PILOTS POINT OF
VIEW
Dublin, Ireland
CONFERENCE
VIEW
4 Pilots dont have to become computer geeks
4 But have to know how to work the system,
and the data that it uses to operate
4 To get there, system design has to go farther
than just ergonomics or automation
4 It is imperative to develop new systems in a
way that are easy to comprehend
4 This is the real challenge for system
designers: TRANSPARENCY
Dublin, Ireland
CONFERENCE
VIEW
4 Inside cockpits based on mechanical or
electromechanical designs, the pilot can easily
relate his own activity with the design of the
system. The system offers information that the
pilot has to integrate in his mind, based on his
knowledge of the function.
4 Just as a driver can improve his level of
ability if he has mechanical knowledge that can
give more logic to his performance.
Dublin, Ireland
CONFERENCE
VIEW
4 Inside information technology based cockpits, pilots are
instructed to react to the information received from the
system...but they have no knowledge of the internal
Logic of the system. The system does not give out the
Data, but the INTEGRATED INFORMATION THAT THE
PILOT NEEDS...according to the systems designer
judgment.
4 To comprehend the analysis of the system is far
more difficult, given the complexity of the interactions
taking place in its interior, and that are not explained.
Dublin, Ireland
CONFERENCE
VIEW
4 In consequence, a deterioration of the pilots

role can be produced in new generation
airplanes.
Starting from a conceptual knowledge, we
can end up practicing predetermined skills
and routines.
Dublin, Ireland
CONFERENCE
SOME IMPORTANT
ASPECTS OF
INFORMATION
TECHNOLOGY IN
AVIATION
Dublin, Ireland
CONFERENCE
1. Information technology has played a key
role in the improvement of the following
points:
1. The number of Pilots in the cockpit has been

reduced.
2. Ergonomics have been improved and the number
of indicators reduced.
3. More efficient designs can be fabricated, even if
they are less stable, given that information
technology virtually stops any mistake on its
operation.
4. Work load during the critical phases of flight is
also reduced.
5. Some human mistakes are reduced.
Dublin, Ireland
CONFERENCE
2. Independency between the levels of design
on Information Systems also allows:
1. Reduction on training costs
2. Crew Multi-Qualification.
Facing this scenario..What is the

problem?
Dublin, Ireland
CONFERENCE
SOME CONCLUSIONS
4 The emphasis in technology leads the pilots so
that they will have an image on how the system
works.
4 However, that image is the product of the system,
not the way on which it actually works internally.
4 Meaning that the pilot does not know how the
system operates internally
4 Making it very difficult to diagnose or fix a
problem
4 So, the role of the pilot as an alternative resource
of the system is limited.
Dublin, Ireland
CONFERENCE
FOLLOWING THE COMPASS
The information systems shouldnt be

designed within the limits of the
available information technology, but
taking into consideration the peoples
ability to operate that information
system.
Dublin, Ireland
CONFERENCE
LOSA FOR DESIGN

November 5th, 2003
STATISTICS
4 Statistics are
like a bikini,
what they
show is
interesting,
what they hide
is vital!
Dublin, Ireland
CONFERENCE
LOSA observations
4 20% of LOSA flights have observable

automation errors.
4 25% of automation errors have contributing
threats mostly ATC threats.
4 46% of automation errors are committed
during descent / approach / land, 22% /
takeoff / climb, and 22% during predeparture /
taxi-out
Source: University of Texas
Dublin, Ireland
CONFERENCE
LOSA observations
4 Most frequent automation errors observed:

1. 15% of all automation errors observed are:
wrong MCP altitude dialed
2. 11% wrong waypoint / route entered into FMC
3. 8% wrong MCP mode engaged
Note.- These errors use Boeing terminology but also include Airbus equivalent.
4. 61% of automation errors are inconsequential,

27% are linked to undesired aircraft states and
12% are linked to additional errors.
Source: University of Texas
Dublin, Ireland
CONFERENCE
Drawbacks of Automation
Complacency or Over-reliance
4 Automation is so efficient and

reliable that it can induce
complacency
4 Monitoring a system that runs
almost perfectly is boring
4 Such reliability tends to
transform active monitoring into
passive monitoring
4 Pilots tend to check that the
automation behaves as intended
instead of flying the aircraft!
Dublin, Ireland
CONFERENCE
Difficulties in Programming
4 Programming tends to prematurely determine the
representation a pilot has of the next phases of the flight
4 Consequences of errors shift into the future
4 Data-bases: aids that can turn into traps
Difficulties in Programming
The TG 311 A310-304 crash near

Kathmandu, Nepal, 1992.
A flap fault occurred on the approach.
As no reply came from the ATC, the crew

initiated a right turn and a climb from 10500 ft to
FL180.
The crew completed a 360 turn instead of a 180

turn.
The flight continued toward the north.
During the 6 minutes the crew attempted to find

Romeo in the FMS data-base and visualize it on
the ND.
11/06/2001 14
Dublin, Ireland
CONFERENCE
Coping with complexity,
regulations and change of
context

November 5th, 2003
Modelling the Natural Safety
Control of situation
Too many
questions
System and situation unexplored NON controllable
Area
Complexity
Performance
NON controllable
Area routines MAXIMUM
PERFORMANCE
Late error
detection
routines Cognitive
surprises
routines
Uncomfortable
area NON controllable
No sensation Area
Too
routines many
tasks
piling up
Too many errors Work intensity
Workload
Dublin, Ireland
CONFERENCE
Performance and technology
induced problems
Performance
Comprehension
Management
New
Back to
solutions?
manual?
LOSS OF CONTROL
Workload Management
Dublin, Ireland
CONFERENCE
Where do we go from here?
Dublin, Ireland
CONFERENCE
FDAP (FOQA)
Dublin, Ireland
CONFERENCE
HEAD UP DISPLAYS
Dublin, Ireland
CONFERENCE
Dublin, Ireland
CONFERENCE
HEAD UP DISPLAYS
Dublin, Ireland
CONFERENCE
Dublin, Ireland
CONFERENCE
HUDS+ 3D PFD/ND
Prevent
runway
incursion!
Dublin, Ireland
CONFERENCE
Dublin, Ireland
CONFERENCE
THANK YOU !
WORKSHOPS
ICAO / IATA LOSA & TEM Conference
November 5-7, 2003 Dublin, Ireland
Training Instructors on Threat and

Error Management (TEM):
LOSA Observer Training Overview
James Klinect and Patrick Murray
The University of Texas / The LOSA Collaborative
The University of Texas Human

Factors Research Project
Presentation Goals
Provide an overview of the LOSA observer training
process
Brief introduction to how threat and error management is

observed and recorded during a LOSA
Skeleton observer training outline for airlines conducting

LOSA without third party assistance
LOSA Observer Training
Length: Five days
2 days training
2 days to complete two observations and write-up
1 day recalibration
Airline is encouraged to invite other stakeholders to

observe the training to enhance organizational buy-in
Regulators
Managers
Line checkers
Training Overview Outline
Confidentiality / Voluntary Consent
LOSA Observation Form Overview

Demographics
Narratives
CRM Performance Markers (Threat and Error Countermeasures)
Descent / Approach / Land Technical Worksheet

Model Introduction
Threat and Error Management Worksheets Overview
Exercises
Crew Interview
Confidentiality / Voluntary Consent
Confidentiality
Goal #1: - Ask observers to be a fly on the wall
Main issue when does an observer speak up?
Think of yourself as a jump seat rider on another airline
Avoid filling out the observation form in the cockpit take notes
When asked to be debriefed, just say everything looked fine
Keep all data de-identified and confidential
LOSA will fail if observers fail to be flies on the wall

Voluntary Consent
Two stages of voluntary consent
1. Union endorsement
2. Captain briefing
Captain briefing
Ask for permission to ride and observe
Stress that data is de-identified and confidential
Ask if you can take notes on a notepad
Determine the best time to conduct the crew interview
Set up a schedule for time away from the cockpit
Inform the Lead Flight Attendant

LOSA Observation
Form Overview
Observation Tools / Instruments
1. LOSA Observation Form

Demographics
Narratives
Threat and error countermeasures (CRM performance markers)
Descent / Approach / Land technical data
Crew interview
2. LOSA Code Book
3. Pocket Notepad
The Four LOSA Observer Tasks
1. Write a narrative on:

What the crew did well as risk managers?
What the crew did poorly as risk managers?
How the crew performed when confronted with threats or errors?
2. Log all threats, how they were managed, and outcomes
3. Log all errors, how they were managed, and outcomes
4. Assess CRM performance and its relation to threat and

error management
Demographics
(Page 1)
Demographics Top Three
1. Observer ID stay consistent
2. Dont forget the observation number - start with #1
3. Collect demographics at the end of the crew interview

Narratives and CRM Performance
(Pages 2-7)
Narratives Top Five
1. Narratives should provide a context - what and how

1. What did the crew do well as risk managers?
2. What did the crew do poorly as risk managers?
3. How did the crew manage threats, errors, or other significant event?
2. Keep the narrative concise and simple
3. Record things that no other data source can record

4. Youre the expert - state assumptions make conclusions
5. When in doubt, write in the narrative

CRM Performance Top Three
1. Only rate with confidence - Dont have to rate everything
2. Another way to think about the ratings..

1 = Poor Major contributor to failing a check ride
2 = Marginal Would require a debrief
3 = Good Standard performance
4 = Outstanding Training tape quality
3. If you give a 1,2, or 4 rating, justify it in the narrative

Technical Worksheet
(Page 4)
The Most Dangerous Phase
Data from the Descent / Approach / Land Phase
Most accidents worldwide (ALAR)
Most threats
Most mismanaged threats
Most crew errors
Most mismanaged crew errors
Most variability in CRM performance

The Blue Box
Brief TOD
Transition
4 Altitude
10000 ft.
Slow and
Approach
FAF/OM Bottom
Lines
Flare / Touchdown
Descent/Approach/Land
Top Three
1. Fill out the technical data worksheet first and use The
Blue Box landmarks in your narrative
2. If the approach was unstable, laundry list the

contributing factors
3. We defer tell us what is going on in The Blue Box

Threats
Threat Induced
Accident
Error
Crew Error
Responses
Error Outcome
Error Induced
Incident or
Accident
(Page 8)
Threats Defined
Threat Events or errors that originate outside the influence of the
flight crew but require their attention to maintain safety
1. Environmental Threats 4. Crew Support Threats

Adverse WX Dispatch events / errors
Terrain Ground events / errors
Airport Conditions MX events / errors
Heavy traffic / TCAS events
5. Operational Threats
Time Pressures
2. ATC Threats
Irregular Operations
Command events / errors
Radio congestion / poor
Language difficulties reception
3. Aircraft Threats 6. Cabin Threats

Malfunctions Cabin events / FA errors
Automation events
Threat Management Top Three
1. If a threat induced a crew error, enter NO for effectively
managed
2. See more than five threats? staple another worksheet

and renumber

1. What was the threat?
2. How did the crew respond to the threat?
3. What was the outcome of the threat?

(Pages 9-11)
Threats
Threat Induced
Accident
Error
Crew Error
Responses
Error Outcome
Error Induced
Incident or
Accident
LOSA Crew Error Defined
Definition: Observable deviation from organizational or
crew expectations
Errors are recorded at the crew level

Usually involve an aircraft handling issue, communication
problem or deviation from SOP
Observers cannot capture all form of crew error

Cannot get in the pilots head for cognitive errors
Result: LOSA provides underestimates of what actually

occurs in normal operations
Crew Error Types
Threats
Threat Induced
Accident
Intentional
Noncompliance
Procedural
Communication
Decision
Crew Error
Responses
Error Induced
Incident or
Error Outcome Accident
Error Types Defined
1. Intentional Noncompliance intentional deviations from SOP

Ex) Performing a checklist from memory
2. Procedural Followed procedures with incorrect execution

Ex) Wrong attitude setting dialed

4. Decision Crew decision unbounded by procedures that

Ex) Unnecessary navigation through adverse weather
START
- Miscommunication Communication Error
- Misinterpretation
NO
Decision Error
- Decision that increased risk in Go to Page 6
procedures to follow Qualifiers for Decision Errors
1 Did the crew have options?
2 Did the crew discuss options before

making a decision?
NO
3 Did the crew have or buy time to
discuss options?
Was the error associated with a: Intentional Noncompliance

- Intention not to follow written YES
Error
procedures Go to Page 2
NO Procedural Error
Go to Page 3 and 4
A Word About Decision Errors
Definition - No written procedures to follow and the crew
Qualifiers
1. Did the crew have options?
2. Did they crew discuss their options before making a decision?
3. Did the crew have or buy time to discuss options?
If the answer is NO to any of the qualifiers, then think

about a decision error
Error Responses and Outcomes
Threats
Threat Induced
Accident
Error
Crew Error
Responses
Error Outcome
Error Induced
Incident or
Accident
Error Responses
1. Detected & Action
1. Error is detected
2. Crew did something to manage or mismanage the error
2. Ignored
1. Error is detected or willingly committed
2. Crew intentionally elected not to manage the error
3. Undetected
Error is undetected
No error management takes place
Error Outcome
1. Inconsequential
Error was effectively managed or had no further risk after
commission
2. Additional Error
Error was mismanaged to link to an additional error
3. Undesired Aircraft State

Error was mismanaged to produce a aircraft state that
increased risk
Undesired aircraft states A state in which the aircraft is placed in
a compromised situation that increases risk to safety

Speed too high
Long landing
Speed too low
Incorrect aircraft configurations Firm landing
Flight controls Wrong taxiway / ramp
Systems
Wrong runway
Fuel
Automation Runway incursion
Undesired Aircraft State Cues
Usually caused by a combination of handling and procedural errors
Typically have a length of time that passes without an error being

detected
Crews usually detect a state first rather than the error or errors that
contributed to the state
Incorrect aircraft configurations

Typically one step away from more serious undesired aircraft states
Error Management Top Three
1. Dont spend too much time on the codes it is not an
exact science
2. See more than five errors? staple another worksheet

and renumber

1. What was the crew error?
2. How did the crew respond to the error?
3. What was the outcome of the error?

Threat and Error
Management Exercises
Exercise One
Predeparture / Taxi-out The Captain requested an
extra 3000 pounds of fuel for possible weather at the
destination. After ground confirmed the fueling, the First
Officer alerted the Captain that they were 2000 pounds
off. The Captain radioed back to ground and the
remaining 2000 pounds was loaded.

Threat ID Phase of
Flight Effectively
Threat managed?
Describe the threat 1 Predepart/Taxi How did the crew manage or mismanage the threat?
3 Cruise (Yes / No)
4 Des/App/Land
5 Taxi-in
1 Threat managed Crew asked ground to load an extra

T1 WX at the destination Adverse 1 Yes
WX
3000 lbs.
83
Incorrect fueling Threat managed FO caught the ground crew error on a
T2 Ground 1 Yes
(2000 lbs. short) Crew cross-check.
Error

Phase of Error Type
Was the
Error ID
flight Error Crew Error Error

error 1 Intentional Code Who Who Response Outcome
Describe the crew error and associated 1 Predepart/Taxi proficiency Noncompliance committed detected
undesired aircraft states based? 2 Procedural Use
2 Takeoff/Climb
3 Communication the error? the error? 1 Detected 1 Inconsequential
3 Cruise Code 2 Ignored 2 Undesired state
4 Des/App/Land Yes or No 4 No code Book 3 Undetected 3 Additional error
E NO ERRORS

Error ID

Undesired
the 1 Detected
State Code
(If Yes, enter state? 2 Ignored 1 Inconsequential
E NO ERRORS
Exercise Two
Predeparture / Taxi-out - While taxiing to the assigned
runway, the First Officer performed the entire Taxi
Checklist from memory. The Captain noticed and
ignored it. In the end, everything was set correctly.

Threat ID Phase of
Flight Effectively
Threat managed?
3 Cruise (Yes / No)
4 Des/App/Land
5 Taxi-in
T __ NO THREATS

Phase of Error Type
Was the
Error ID

error 1 Intentional Code Who Who Response Outcome
Describe the crew error and associated 1 Predepart/Taxi proficiency Noncompliance committed detected
undesired aircraft states based? 2 Procedural Use
2 Takeoff/Climb
3 Communication the error? the error? 1 Detected 1 Inconsequential
3 Cruise Code 2 Ignored 2 Undesired state
4 Des/App/Land Yes or No 4 No code Book 3 Undetected 3 Additional error
120
FO ran the Taxi Checklist from 2 1
E1 1 No 1 Checklis 2 1
memory t from FO CA
memory

Error ID

Undesired
the 1 Detected
State Code
(If Yes, enter state? 2 Ignored 1 Inconsequential
No management intentional noncompliance error was

E1 No
detected and ignored by the Captain.
E
Exercise Three
Descent / Approach / Land During a 30-degree bank
on a visual approach, the First Officer allowed the
aircraft to get 15 knots below minimum maneuvering
speed. The low speed had to be pointed out by the
observer. The FO said thanks and immediately
increased the speed. The observer noted that this was
most likely due to a lack of stick and rudder proficiency.

Threat ID Phase of
Flight Effectively
Threat managed?
3 Cruise (Yes / No)
4 Des/App/Land
5 Taxi-in
T __ NO THREATS

Phase of Error Type
Was the
Error ID

error Who Response Outcome
Describe the crew error and 1 Predepart/Taxi proficiency Noncompliance detected
committed
associated undesired aircraft states 2 Takeoff/Climb based? 2 Procedural the 1 Detected 1 Inconsequential
3 Communication Use the error?
3 Cruise
Code Book error? 2 Ignored 2 Undesired state
4 Des/App/Land Yes or No 4 No code 3 Undetected 3 Additional error
FO exhibited a lack of stick

286
and rudder proficiency 2 7
E1 4 Yes 2 Unintentional 3 2
Failed to maintain speed speed FO Nobody
deviation
during a 30-degree bank turn.

Error ID

Undesired
the 1 Detected
State Code 2 Ignored
(If Yes, enter state? 1 Inconsequential
No management Both the error and undesired aircraft state 45 5

E1 No Jumpseat 3 1
went undetected poor vigilance. Speed too low
Rider
E
Exercise Four
Takeoff / Climb During climb, the crew discovered that

a fuel pump low pressure light was illuminated. All
crewmembers were working the problem when ATC
instructed them climb to FL350. The First Officer (PNF)
read back FL350 but failed to make the change in the
MCP. The crew also failed to cross-verify the altitude
change in the MCP. After a few moments, ATC noticed
that the aircraft remained at FL270 and called the crew
to correct the problem.
Threat ID
Code
2 Takeoff/Climb
20 Threat mismanaged Crew got distracted from their

T1 Fuel pump low pressure light A/C 2 No
malfunction
flight duties

Phase of Error Type
Was the
Error ID

Describe the crew error and error 1 Intentional Who Who Response Outcome
proficiency Code
associated undesired aircraft 1 Predepart/Taxi Noncompliance committed detected
states 2 Takeoff/Climb based? 2 Procedural
the error? the error? 1 Detected 1 Inconsequential
3 Communication Use
3 Cruise 2 Ignored 2 Undesired state
4 No code Code Book
4 Des/App/Land Yes or No 3 Undetected 3 Additional error
235 2 7
E1 Failed to dial MCP altitude 2 No 2 Wrong MCP
FO Nobody
3 3
altitude
Failure to perform altitude 234 6 7

E2 2 No 2 Failure to 3 2
awareness procedures cross-verify
All Crew Nobody

Error ID

Undesired
the 1 Detected
E1 Yes T1 Error chain E2
Errors mismanaged Crew devoted their attention to the fuel 40 8

E2 Yes T1 Vertical 3 1
pump problem with no one left flying the airplane deviation
ATC
Exercise Five
Descent / Approach / Land On the downwind leg of a visual approach at
6000ft., ATC asked them if they could make an immediate turn for the outer
marker and maintain 180 knots as they crossed over the marker. If they
could, they would be number two for landing. They were already early, but
the Captain accepted the challenge without asking the First Officer (PF).
The First Officer was noticeably reluctant at first, but he said nothing, made
the close-in turn, and extended the speed brakes to begin the approach.
The FO was really pushed but he crossed over the marker on speed. At
1800, the Captain noticed that the speed brakes were left extended. The
First Officer corrected the problem and continued a stabilized approach.

Threat ID Phase of Effectively
Code
ATC challenged the crew to 50 Threat mismanaged CA accepted the clearance without
T1 ATC 4 No
accept a tough-to-meet clearance command
discussing it with the FO (PF)

Phase of Error Type
Was the
Error ID
flight Crew Error Error

Describe the crew error and error 1 Intentional Error Code Who Who Response Outcome
associated undesired aircraft 1 Predepart/Taxi proficiency Noncompliance committed detected
states 2 Takeoff/Climb based? 2 Procedural Use
the error? the error? 1 Detected 1 Inconsequential
3 Cruise 3 Communication Code Book 2 Ignored 2 Undesired state
4 Des/App/Land Yes or No 4 No code 3 Undetected 3 Additional error
Decision to accept a risky 530 1 2

E1 4 No 5 Accept risky 2 3
ATC clearance ATC instruct
CA FO
222
FO left the speed brakes 2 1
E2 4 NO 2 Failure to 1 1
extended at 1800 retract speed FO CA
brakes
Error ID

Undesired
the 1 Detected
Error mismanaged CA accepted a clearance that

E1 Yes T1 unnecessarily pushed the FO. In response, the FO said
nothing and made the close-in turn
E2 Yes T1 Error managed CA caught the extended speed brakes
Crew Interview
Crew Interview Top Three
1. Start by giving the crew members the right to decline
answering any question
2. Make it a conversation may not get to all questions
3. Ask the demographics at the end of the interview
4. Make it safety-oriented dont get into too much

management bashing
Questions?

www.psy.utexas.edu/humanfactors
www.losacollaborative.org
Threat and Error
Management
ICAO and IATA, LOSA and TEM Conference

4 - 7 November, 2003
Dublin, Ireland
Captain Michael Bombala

This course is developed with support from the
IATA Human Factors Working Group and the
Line Operations Safety Audit (LOSA) Advisory
Board.
Air Line Pilots Association( ALPA) and

International (IFALPA), Continental Airlines, The
International Civil Aviation Organisation (ICAO),
The LOSA Collaborative, University of Texas
and US Airways have generously contributed to
the development of this course.
How are Aircrew made safe?
Proficiency
Technical knowledge
Physical and mental well being
CRM
Evolution of CRM
NASA research inspired this
response to non-mechanical
accidents
Goal to reduce Human Error

Problems with past CRM
Adaptation of business models
Confusing
Industry has evolved and matured
Diluted
Separation of technical and CRM
training
Forgot the primary objective of CRM
Reduce Human Error
Error

Why do we make errors?
Inevitable result of human limitations such as
- fatigue and other physiological factors,

- limited memory and processing capacity
- external stressors
- poor group dynamics
- cultural influences
Why do we make errors?
Lack of experience
Rushed
Distractions
Stress
Crews make mistakes several times during
each flight, most of which are unimportant
However it can be beneficial to recognize and
learn from errors, since it will help manage
your resources better during the next flight
Error
Actions or inactions by the crew that lead
to deviations from organizational or flight
crew intentions or expectations.
Errors in the operational context tend to

reduce the margin of safety and increase
the probability of incidents or accidents.
Error environments
Increasing workload
Undo time pressure
Fatigue
Procedural non-compliance
Poor crew coordination
Interruptions/distractions
Crew Invulnerability
Perception
Pilots perceive themselves as unbreakable
under pressure, that they can handle all
problems
Reality
Pilots are affected by various factors which
influence their ability to perform, their
personal limitations
Fifth Generation CRM
Cannot totally eliminate error
Avoid errors being made
Manage errors by trapping or mitigating
their consequences
Error Management
Threat

The Real World
What is a normal flight?
What hazards do aircrew have to deal

with on the line?
Runway incursions
Both pilots have taxi charts available

Both pilots monitor taxi clearance
Captain will verbalize any hold short
instructions, FO to request confirmation
from Captain if not received
Threats by Phase of Flight
Phase of Flight Threats by Phase
Pre-Departure / Taxi 30%
Takeoff / Climb 22%
Cruise 10%
Descent / Approach 36%

/ Land
Taxi / Park 2%
Latent Threats
Factors that affect our ability to operate safely
without our prior knowledge of their
consequences.
People unwillingly create conditions for crews to
commit error by negligence in;
- design
- manufacture
- regulations
- procedures
Latent Threats
ATC practices
Scheduling practices that result in fatigue
Organizational, national, professional
culture
Aircraft characteristics
Qualification standards
Regulatory practices
What is threat?
Are situations external to your own
working environment that must be managed
during normal, everyday flights.
Why is it a problem?
Such events increase the operational
complexity of flight and pose a safety risk to
the flight at some level increase error
potential
How do we respond to threat?
CRM countermeasures
Our Goal
Need to refocus CRM towards error

reduction and threat recognition
Adaptation of existing skills towards

countermeasures in the real world
Sixth Generation of CRM
Focuses on CRM as set of
countermeasures against threat and error
Avoidance of threat and error
Undesired aircraft state management

TEM
What do we have to do to fly safely
from A to B?
Avoid committing errors
and reduce threat
Manage operational complexity

Safe
= Flight
Manage our own errors
Manage aircraft deviations

Areas of Vulnerability
Within 1,000 ft of level off
Cruise / Descent
transition
Transition Alt.
Descent, approach
10,000 ft and landing
Taxi-out Cruise Taxi-in

Takeoff Descent
Climb Approach
Pre-Departure Land Taxi / Park
Taxi
Errors by Phase of Flight
Phase of Flight Errors by Mismanaged
Phase Errors
Cruise 6% 17%
Descent / Approach / 44% 35%

Land
Taxi / Park 4% 24%
Blue Box
Most crew errors (LOSA)
Most consequential crew errors (LOSA)
Most accidents worldwide
CFIT
Errors dealing with automated systems
represent the largest source of error
Types of Error
Intentional Noncompliance violations
Ex) Checklist from memory
Procedural followed procedures with incorrect execution
Ex) Wrong altitude setting dialed
Communication Missing information or misinterpretation
Proficiency Lack of knowledge or skill
Ex) Lack of knowledge with automation
Decision Crew decision unbounded by procedures that
Ex) Unnecessary navigation through adverse wx
Aircraft deviations at edge of the safety envelope
Incorrect aircraft configurations

Vertical / lateral deviations
Speed too high / low
Door Slides not armed
Path/glide slope deviations
Unstable approach
Long landing
Wrong taxiway/runway
Runway incursion
Countermeasures

CRM Skill Groups
Four groupings of CRM skills are
threat and error countermeasures
Team building and climate

Planning
Execution
Review / modify
Team Climate
How do you create an atmosphere for
constructive TEM?
develop and maintain a good

communication environment
effective leadership
Important in all aspects of safety ( CRM/TEM)

Planning
Preparation in dealing with threat and
avoiding error by using;
- Good Briefings
- Stating Plans
- Workload assignment
- Contingency management
Planning for Known Threats
How do you plan for known threats?
Diagnose- identify, knowledge, memory,
problem solving skills
Generate solutions find alternatives,
hard to do under stress
Assess risks predict consequences and
success rate
Execution
Application of countermeasures to threat
and error
- Monitor / Crosscheck
- Workload Management
- Vigilance
- Automation Management
Distractions
Identify the interruption
Ask what was I doing before...

being interrupted
Decide what action to take to get back on track

Vigilance
Is being alert to the situation, not only what
is happening, but what may happen
Anticipate problems
Gives you the ability to think about options
without being under the stress of an
emergency
Particular attention must be devoted to
altitude and course changes
Review / Modify Countermeasures
Use against unexpected threat or when
aircraft is in an undesired state.
Evaluation of plans review and modify

plans when necessary
Inquiry ask questions to clarify
Assertiveness state critical information
with persistence
Decision Making
for an Unexpected Threat
Perceive situation
- gather and process information
- vigilant alert to situation
- knowledge- variation from the norm
What is the risk?
Is it time critical?
Select a course of action
Avoidance
Organization
SMS
Must look for sources of threat and error to
strengthen system defenses
Develop and maintain a positive safety culture
Open communication
Collect data on threat and error areas
High levels of training and proficiency
SOPs and Checklists
Avoidance
You
Maintaining your health
Maintain high level of training and proficiency
Following SOPs
Proper use of Checklists
Minimizing distractions
Planning ahead
Open two-way communication
Maintaining situational awareness
How to Manage Threat and Error
Avoidance
Threat Management
Countermeasures
Error Management

Management
Countermeasure Skills
Team Climate
Planning
Task Execution
Review and Modify
Avoidance
Threat
Threat Management
Crew Error
Error Management
Undesired Aircraft
State
Undesired
Aircraft State
Management Model
Incident
How does TEM fit into a
Safety Management System?
TEM supports SMS by bringing in hard

data
Linkage between Safety and CRM /TEM
Safety change process
TEM can be used for a focus group at an
airline
More Realistic Training
Focus on Threat and Error Recognition

Provide countermeasure skills to crews
Feedback-help management understand
line operation
Tools to Improve Safety
SOPs and Checklists
- strategies put forth by the company
Training - Technical proficiency

- Countermeasure (CRM) skills
TEM model as an analytical tool of incidents

LOSA
First ICAO-IATA LOSA and TEM
Improving Flight-Crew Monitoring Skills

Captain Robert Sumwalt
Chairman
Human Factors and Training Group
Air Line Pilots Association, International
Introduction
z Each crewmember must carefully monitor the

aircrafts flight path and systems, and actively cross-
check the actions of each other.
z Effective monitoring and cross-checking can literally
be the last line of defense
When this layer of defense is absent the error
may go undetected, leading to
adverse safety consequences
2
Good monitoring is important
z By better monitoring and

cross-checking, a
crewmember will be more
likely to catch an error or
unsafe act
z This detection may break a
chain of events leading to an
accident scenario
3
Why improve Monitoring: Accident Data
z Inadequate crew monitoring or challenging was a factor in 84% of

37 crew-caused air carrier accidents reviewed in a NTSB safety
study.
76% of the monitoring/challenging errors involved failure
to catch something that was causal to the accident
17% of the monitoring/challenging errors were failure to
catch something that contributed to the accidents cause
z Poor monitoring was a factor in 63% of the ALA accidents
reviewed by the FSF ALAR working groups.
z 50% of the CFIT accidents reviewed by ICAO to support the FSF
CFIT efforts involved poor monitoring.
4
Why improve Monitoring: Incident Data
z Researchers examined 200

incident reports submitted to NASA
ASRS
What ASRS Data
Tell About Inadequate z They found evidence that
Flight Crew Monitoring inadequate monitoring can lead to
adverse safety consequences
Altitude deviations
CFTT
Stall
Loss of aircraft control
Course/heading/track deviations
5
Why improve monitoring: LOSA Data
z Roughly 42% of observed errors in the University of

Texas LOSA archive were undetected by flight crew.
z In a recent LOSA, 19% of errors could have been
eliminated by more effective crew monitoring and
cross-checking.
z In that same LOSA, 69% of undesired states could
have been eliminated by more effective monitoring.
6
US Airways is actively working to improve pilot
monitoring
7
Underlying factors associated with poor monitoring
1. Until now, the industry has not made monitoring a

primary task.
When listing PNF duties, we often list duties such as
handling radio communications, keeping flight logs
and operating gear and flaps.
Monitoring is not one of the duties primarily listed,
but rather it seems to be treated as a secondary
task, or not addressed at all.
8
2. Effective monitoring is not easy and intuitive

requires a skill and discipline
9
3. There is somewhat of a monitoring paradox that

works against effective monitoring.
Serious errors do not occur frequently which can lead
to boredom and complacency
A low-probability, high-criticality error

is exactly the one that must be
caught and corrected.
10
4. Although traditional CRM courses have generally

improved the ability of crewmembers to challenge
others when a situation appears unsafe or
unwise
many of these courses provide little or no explicit

guidance on how to improve monitoring.
11
5. We seem satisfied that we cannot improve

monitoring, and simply explain it as, Humans just
are not good monitors.
While it may be true that humans are not naturally
good monitors, crew monitoring performance can be
significantly improved through policy changes and
crewmember training.
12
ASRS Monitoring Study
z This study was a good resource

for helping us get stared and
What ASRS Data providing data
Tell About Inadequate z The objectives of the study were
Flight Crew Monitoring to identify factors that contribute
to monitoring errors, and
z Offer operationally- oriented
recommendations to
increase awareness of
this subject
improve crew monitoring
13
Number of monitoring errors
0
10
20
30
40
50
60
taxi-out
takeoff
climb
cruise
ASRS Monitoring Study:
14
crs/des trans.
descent
holding
approach
landing
taxi-in
Flight Phase where Monitoring Errors Occurred
Initiation
ASRS Monitoring Study:
Number of tasks crew was doing shortly before or during
initiation of monitoring error *
Reports 89%
140
*As reported
in 200 reviewed
120 ASRS reports
100
80
40%
60
40
20
0
0 1 2 or more
15
ASRS study significant findings
z 76 percent of monitoring errors occurred when

aircraft was climbing, descending or on approach
(vertical flight phase)
z 30 percent of the reports indicated that pilots were

programming the FMS shortly before or during the
monitoring error
16
An Effective Approach to Improve Monitoring
Developing well thought-out SOPs

Training monitoring skills
Practicing those skills

AC 120-71A Standard Operating Procedures
z Revised in February 2003, this AC contains template

SOPs that can be adopted by air carriers to improve
monitoring.
19
Developing SOPs
z Change title of Pilot-Not-Flying (PNF) to Pilot

Monitoring (PM)
Describeswhat the pilot should be doing (monitoring)

versus what he/she is not doing (not flying)
20
Developing SOPs
Monitoring Responsibility
The PF will monitor/control the aircraft, regardless of
the level of automation employed.
The PM will monitor the aircraft and actions of the PF.
Monitoring is a primary responsibility

of each pilot.
21
Developing SOPs
z Both pilots will have taxi charts

available, when necessary
z Both pilots will monitor taxi
clearance
z Captain will verbalize to First
Officer any hold short
instructions
FO will request confirmation
if not received
22
Developing SOPs
z When approaching an entrance to an active runway,

both pilots will ensure the hold short or crossing
clearance is complied-with before continuing with non-
monitoring tasks (checklists, company radio calls, etc.)
23
Developing SOPs
z During high workload, FMS inputs will be made by

PM, upon the request of PF.
High workload examples
below 10,000 feet
within 1000 feet of level off or Transition Altitude.
24
Developing SOPs
z Perform non-essential duties/activities during lowest

workload periods (e.g., cruise altitude or level flight)
z When able, brief anticipated approach prior to top-of-
descent
z PF will brief PM where or when delayed climb or
descent will begin
z During the last 1000 feet of altitude change, both pilots
will focus on making sure the aircraft levels at the
assigned altitude
z US Airways eliminated 10,000 ft announcements and
specified that the pre-arrival announcement be
conducted just after leaving cruise altitude.
25

First, we must change our approach to monitoring.

Instructors must [teach and] insist that the non-flying
crewmember monitors the flier effectively.
Good monitoring skills are not inherent in pilots as
they progress in their careers. Therefore, effective
monitoring techniques must be trained and
rewarded.
- Captain Frank J. Tullo

Aviation Week and Space Technology
May 21, 2001
27
z NTSB safety study states that simulator training

provides a good opportunity to teach and practice
monitoring and crosschecking.
NTSB Safety Study of Crew-caused Accidents
28
z Starting from day 1 of training, ensure all monitoring/

crosschecking SOPs are followed.
z Discuss how barriers are cut in half with one pilot out
of the loop.
z Train workload management so at least one pilot is
always monitoring during low workload and both pilots
are monitoring as much as possible during high
workload.
z Acknowledge good monitoring.
Introduction of occasional subtle failures in
simulator training, such as failure of automation to
level-off at proper altitude
29

Practicing monitoring skills
z Know and comply with SOPs

z Pilots must actively monitor the aircraft.
z This means that they must mentally fly the aircraft,
even when the autopilot or other pilot is flying.
Monitor the flight instruments just as you would when
hand flying.
If the aircraft (or other pilot) is not doing what it is
supposed to do, actions should be taken to rectify the
situation.
31
z In approximately one-third of the cases studied by

researchers, pilots failed to monitor errors, often
because they had planned their own workload poorly
and were doing something else at a critical time.
Jentsch, Martin, Bowers (1997)
z Threat and Error Management with a focus on
monitoring and cross-checking is a good way to teach
pilots better workload prioritization skills.
32
z Pilots should recognize those flight phases where

poor monitoring can be most problematic.
z Strategically plan workload to maximize monitoring
during those areas of vulnerability (AOV)
Examples of non-monitoring tasks that should be
conducted during lower AOV include stowing charts,
programming the FMS, getting ATIS, accomplishing
approach briefing, PA announcements, non-essential
conversation, etc.
33
Areas of Vulnerability
Descent,
Within 1000 ft
of level-off Approach
and Landing
Transition alt Cruise-Descent
Transition, or
anytime you are
10,000 ft
anticipating a clearance
Taxi-in
Taxi-out
34
z By pre-briefing the approach in low workload periods,

greater attention can be devoted to monitoring/cross-
checking during descent.
z In fact, LOSA data show that crews who briefed the
approach after Top-Of-Descent (TOD) committed 1.6
times more errors during the descent/ approach/land
flight phase than crews who briefed prior to TOD.
35
z One way of assessing your current monitoring ability

is to ask: How often do I miss making the 1,000 to
level-off altitude callout?
When this callout is missed, chances are that you are
not actively monitoring the aircraft.
36
Paradigm shift
z It must become accepted

that monitoring is a core
skill, just as it is currently
accepted that a good pilot
must posses good stick
and rudder and effective
communicational skills.
37
Summary
z Inadequate flight crew monitoring has been

cited by a number of sources as a problem
for aviation safety.
z While it is true that humans are not naturally
good monitors, crew monitoring performance
can be significantly improved through policy
changes, training and by pilots following an
active monitoring concept.
38
The challenge
z Take this concept back home with you and
implement a program at your carrier to
improve monitoring
39
If I had been watching the instruments,
I could have prevented the accident."
- FO after being involved in fatal

CFIT accident
40
HCAT
HUMAN
CENTERED
AUTOMATION
TRAINING
The Agenda
Introductions
Nature of Automation
Continental's Automation Policy

Threats Whats it doing now?
Strategies to effectively manage automation
Case study AA 965 Cali
Nov 2003
Introductions
Name:
Aircraft & Position: From To
Rate the level of automation flown:
1--------------------------------------------------10
Expectations ? Concerns !!
Nov 2003
To ERR Is Human
To REALLY screw up,

you need a computer!
Nov 2003
Air Inter- Survival in the Sky
Nov 2003
Intended Expectations
Reduced workload and fatigue

Fewer errors
Enhanced SA
Increased efficiency
Enhanced safety
Nov 2003
Reality: Reports from pilots who use it!
Workload Increased, Decreased, More mental
Errors More, Fewer, Harder to catch
SA Degraded, Enhanced
Efficiency Decreased, Increased
Safety Compromised, Enhanced
Automation is a different kind of tool
Nov 2003
Nov 2003
What challenging environments do
crews encounter operating
automated aircraft?
Nov 2003
Non - Radar / High Terrain
Environment
Long Duty Periods and Fatigue Display Differences
ETOPS / LRN
International Ops
Area and Special Airport
Qualifications
Nov 2003
Automation in Flight Operations
The Third Crewmember
FMS
Fast/accurate computations
Contingency planning
Enhances crew SA
Creates time for:
Planning /problem solving
Decision making
Monitoring - challenging -verification
It is not meant to challenge a crewmembers role or
responsibilities, but rather to
Compliment a Crews Strengths
Judgment
Situation assessment
Decision making ability
How are you going to know what

has been entered in the Flight
Management System (FMS)?
How Many Ways Are There?
To climb the airplane?
To navigate laterally?
To descend the airplane?
Nov 2003
Control Display Unit (CDU)
Two Questions in Mind Prior to Executing
Exec Any Change:
Obtain confirmation
1. What do I expect the before EXECuting
EXEC
airplane to do now? any change
2. How do I verify it?

EXEC Verbalize
Verify
Monitor
Nov 2003
Long term changes
Flight Mode Annunciator
(FMA)
Nov 2003
Mode Awareness Strategies
Autothrottle Pitch Roll Autopilot F/D
Anticipating Automatic Mode Changes

VNAV PTH to VNAV SPD when energy state is
high
VNAV PTH to ALT HOLD during nonprecision
approach or STAR
Nov 2003
EFIS - Map Display
Nov 2003
Continental Airlines
Automation Policy
Nov 2003
General Automation Policy
Pilots shall be proficient in all capabilities of their aircraft

including the automated systems. Continental Airlines
policy is to fly the aircraft using the highest level of
automation, consistent with the requirement to maintain
basic flying skills.
Pilots should realize the more complex the situation, the

higher the threat level. As threats increase, automation
usage when properly applied, will improve overall
performance and safety. Pilots are authorized to choose an
appropriate level of automation consistent with a changing
flight environment.
Nov 2003
Levels of Automation:
Autopilot/Auto-throttle with LNAV, VNAV & flight

Highest guidance
Lowest Hand Flown without flight guidance
There are many variations between the highest and lowest

levels of automation.
Select the level that optimizes situational awareness while
reducing pilot workload.
Hand flying to maintain proficiency should only be
accomplished in low threat environments.
Nov 2003
Specific Automation Policy
When the autopilot is on, the PF will normally manipulate

the MCP and the CDU, and the PM will verify. When the
autopilot is off, the PF will call for all changes to the MCP
and the CDU. The PM will make the input and the PF will
verify. The crew should brief and clearly understand their
respective duties.
Verbalization between crewmembers is extremely

important for flight deck situational awareness. Many
threats and errors can be countered by effective
communication. Pilots shall verbalize, verify and
monitor in the following manner:
Nov 2003
Specific Automation Policy
(continued)
Prior to executing any changes in the CDU, the pilot

making entries should verbalize the change(s). Both pilots
should verify the change(s) and monitor for expected
aircraft performance.
With any mode changes to the MCP, the PF should

verbalize the change(s). Both pilots should verify the
change(s) using the FMA and monitor for expected aircraft
performance.
When selecting the Autopilot and/or Auto throttle on or

off, the PF should verbalize the change. Both pilots should
verify the change and monitor for expected aircraft
performance.
Nov 2003
Automation Confusion or Frustration ?
Go up a level? Ill give ya 42 levels
Na, go down a level! of automation!
RNP approaches?
hand fly?
Before you
Just pull your head out of your a_________!

utomation
Nov 2003
Levels of Automation
18 Levels of Automation
I Hand Flown; No AT Raw Data
II Hand Flown; No AT Flt Guidance MCP
III Hand Flown AT; Raw Data; MCP Speed
IV Hand Flown; AT Flt Guidance - LNAV, VNAV
V Hand Flown AT Flt Guidance - LVL Chg, VS, VOR/IOC, Approach, MCP
VI AP (CWS, Pitch, Cmd Roll) No AT Flt Guidance - Hdg. VOR/LOC, Approach
VII AP (CWS, Pitch, & Roll) No AT Raw Data
VIII AP (CWS Roll, Cmd Pitch) No AT Flt Guidance - LVL Chg, V/S
IX AP (CWS Roll, Cmd Roll) AT Flt Guidance - Hdg. VOR/LOC, Approach, MCP Speed
X AP (CWS Roll, Cmd Pitch) AT Flt Guidance - Hdg. VOR/LOC, LVL Chg. V/S, Alt hold; MCP speed
XI AP (CWS Pitch, Cmd Roll) AT Flt Guidance - LNAV; MCP Speed
XII AP (CWS Roll, Cmd Pitch) AT Flt Guidance - VNAV; MCP Speed
XIII AP (Cmd Pitch & Roll) AT Flt Guidance - MCP - Non FMS
IV AP (Cmd Pitch & Roll) AT Flt Guidance - LNAV, VNAV
XV AP (Cmd Pitch & Roll) No AT Flt Guidance - Lvl Chg, V/S, VOR LOC, Approach, Alt Hold
XVI AP (Cmd Pitch & Roll) No AT Flt Guidance
XVII AP (Cmd Pitch & Roll) No AT Flt Guidance - FMS LNAV, Lvl Chg, V/S, etc. Alt Hold
XVIII AP (Cmd Pitch & Roll) AT Flt Guidance - FMS VNAV, Hdg, VOR/LOC, Approach
Nov 2003
Quiz to follow!
Guideline for Changing Levels
If overloaded or confused ------
Some situations--------------------- Go UP?
Nov 2003
Nov 2003
Hardware & Software that

exists before the human
enters What the human brings to the
system
`
Nov 2003
Critical Times
What do you consider a critical phase of flight?
Vertical phases of flight (especially
with 1000 feet of an altitude)
Approaching waypoints
Not established on a route of flight
Below 10,000 feet
Nov 2003
Automation Threats
Complacency
Distractions
High workload
Heads down at critical times
FMS Dumb and dutiful
Mode changes
Automation surprise
Display differences
Loss of basic airmanship skills
Threat Complacency
Assuming the automation is programmed

correctly
Over reliance on the automation
Failure to monitor / verify
Failure to use charts
Strategies ?
Nov 2003
Threat Distractions
Monitoring Errors by Phase of Flight
(Data based on 170 ASRS reports)
76% of reported monitoring errors occurred in

some mode of vertical flight
ri ng
o
Taxi-in
Landing
o ni t
v e M
t i
Approach
ff ec
e
Holding Pattern
Descent
Cruise Descent
In
Cruise
Climb
Takeoff
Taxi-out Strategies?
0 10 20 30 40 50 60
Number of error
Nov 2003 events
Threat High Workload
Workload Management
Conventional
Nov 2003
Threat High Workload
Workload Management
Automated vs. Conventional
Automated
Conventional
Strategies?
Nov 2003
Threat Heads Down
(Out of the Loop)
at Critical Times
FMC programming
The paperwork shuffle
Company In range and MX calls
Getting the ATIS
Arrival PA
Workload Management (Late brief/checklist)
How do these influence the
monitoring process?
Strategies?
Nov 2003
Threat FMS Dumb and Dutiful
Accepts data as long as its in correct format

Cannot differentiate misspelled fixes
(Ex. MSY vs. MYS)
Strategies?
Verbalize Verify - Monitor
Nov 2003
Threat Mode Changes
Mode Changing Errors
184 ASRS Reports 1990-1994
74% Vertical Nav 26% Lateral Nav
Data Base
Crew Coordination
Programming Error
Errors
Auto Sys Fail
Lack Understanding
Mode Transition
Unknown
0 20 40 60 80 100
Number of events
Nov 2003
Threat Automation Surprise
What is it doing?
What happened?
What is it going to do next?
How did we get this ------ up?
Strategies?
Verbalize - Verify - Monitor
Nov 2003
Threat
Loss of Basic Airmanship Skills
Failure to backup automation descent
planning
Failure to use en route / arrival charts
Loss of chart knowledge or ability to locate
information
Strategies
Practice hand flying - Practice in low threat environment
Use 3 to 1 rule to back up descent
Use the enroute charts
Routinely brief MEAs, MSA, MOCA, etc.
Nov 2003
Automation Threats
Complacency
Distractions
High workload
Heads down at critical times
Dumb and Dutiful
Mode changes
Automation surprise
Display differences
Loss of basic airmanship skills
Any of these can lead to a CFIT accident
Another
Threat That May Lead To CFIT
Complex instrument procedures
Nov 2003
Strategies to Prevent CFIT?
Enhance Situational Awareness
Start briefing early during low
workload
Review and use enroute/arrival

charts
Brief MSA, MEA, MOCA, etc.
Contingency planning What if?
Proper use raw data as a backup

RNP RNAV/ Constant Rate
Approaches
Will Eliminate dive and drive

approaches
ILS-Like guidance will replace
18 different kinds of approaches
Key initiative in reducing fatal
accidents by 80% by 2007
RNP RNAV Approach
Strategies?
Practicing RNAV approaches in good
weather low stress environment
Using the QRH to setup and brief the
approach
Watch for the Errors:
VNAV unused / not selected during descent
Path unexecuted
Unable to engage VNAV at lower altitude
Late checklist
Nov 2003
An Aviators Nightmare
Unmanaged Threats
Unmanaged Automation
error
The Consequences - Tragic
Nov 2003
Case Study: AA 965 Cali
December 20, 1995
2142 local time
Experienced, good B757 crew
Capt. had been there 13 times, FOs first
Night with no significant weather (VMC)
Non radar environment
Late departure
Long day ?
Nov 2003
AA 965 - Cali
What threats did this crew encounter?
What errors contributed to this accident?
What role did automation play in this accident?
What strategies could be used to prevent a CFIT

encounter with automation or without?
Nov 2003
Transcript
Cali had no approach radar at the time (T / E).
Cabin crew rest issues being discussed prior to

descent (T / E). why?
FO expresses concern about getting down for

landing to the north on runway 1. (Hint & Hope).
No requirement (at time) for an approach

briefing when weather VFR. (T / E).
Nov 2003
Crash
VOR
Tulua
site
Rosa
NDB
Nov 2003
RWY 1
Nov 2003
Transcript
1 = Captain, 2= First Officer
HOT 1, HOT 2, Crewmember hot microphone
RDO 1, RDO 2, Crewmember radio transmission
CAM 1, CAM 2, Cockpit area microphone
BOG, Radio transmission from Bogot Approach
APR, Radio transmission from CALI approach control
Nov 2003
Nov 2003
Nov 2003
RWY 1
Nov 2003
These CDU displays were retrieved from a circuit card from one of
the Flight Management Computers. Data was retained in non-
volatile memory.
Nov 2003
2134:59 Cleared to Cali VOR,
Top of Page 14 descend and maintain 15,000
AA 965 is 52 miles from AP

19 nm or approx. 3 min from ULQ
Middle Right Page 15 2136:31 wind is calm are you able

approach RW 19
Aircraft is:
1 min (5-6 NM) North of ULQ
Distance Tulua
Descending through 19,000
(ULQ) to Cali 37 -39 NM from the AP
32 nm
Top right Page 17 2137:42 all right Roza one to one nine,
twenty one miles, ah five thousand feet
Capt enters R in FMC and executes
Aircraft starts turn to left
Identify the THREATS and ERRORS

How did the use of automation contribute to this
accident?
What automation strategies might have
prevented this accident?
Fight complacency
Clearly defined roles PF / PM
Briefing and communication
Management workload
Utilize raw data
Limit being heads down at critical times
Keep basic flight skills practiced
Nov 2003
Verbalize, Verify, Monitor
Hardware & Software that

exists before the human
enters What the human brings to the
system
`
Nov 2003
Will:
Reduce workload and fatigue
Result in fewer errors
Enhance SA
Increase efficiency
Enhance safety
It takes a commitment on your part
Nov 2003
THREAT & ERROR MANAGEMENT
& SECURITY
RESIST
RESOLVE
Security Changes
While many of these initiatives may

provide enhanced security protection
for crew and passengers alike, it is
important that the human performance
implications of any changes be
carefully considered.
RESIST
RESOLVE
THREATS
Influences that can lead to crew error
Terrorist events
Distractions Fatigue
Cabin Crew
Communication Weapons
Screening
Pax disturbance Assessment of

Threat level
FAMs Time pressures
Flight
LEOs
diversion
System malfunction
Medical emergency
Locked Door Pax overreaction

Threat: Locked cockpit doors & crew
isolation
isolation
Communication during a disturbance
isolation
Communication during a Medical Emergency

isolation
Communication with Federal Air Marshals

FEDERAL AIR
MARSHAL

THREATS
Influences that can lead to FAM error
Passenger events
Distractions Fatigue
Familiar w/ Crew
Other LEOs
Weather
Maintenance Medical Emergencies
Flight
Being Identified
diversion
System malfunction
Unfamiliar airport
Gate Agents Drunks

STRATEGY:
Elements of Briefing
Shared mental model = Outline for Teamwork
Who should be there?

What is briefed?
What can you omit?
GOAL
PLAN
INTENT
isolation
Communication with Federal Air Marshals
COMPLACENCY
Civil Aviation Still a Target
Transportation Security Intelligence Service TSIS

What Have You Seen?
What STRATEGIES Will You Develop?

Summary
Use Threat & Error Management in Security Training to:
Identify a list of human performance threats, as a result

of the current security changes, and develop
STRATEGIES to mitigate these THREATS.
Focus on helping our crews better manage the risks

associated with these changes.
Fight SECURITY COMPLACENCY!

LIST OF PARTICIPANTS
FIRST ICAO-IATA LOSA & TEM CONFERENCE
List of participants
Country Name Organization Phone Fax e-mail

Australia 1) Dr. Matthew Thomas University of South +61 438 808 808 +61 8 8302 0021 matthew.thomas@unisa.cds.au
Australia
2) Mr. Marcus Knauer Airservices Australia 61 7 3866 3698 61 7 3866 3326 marcus.knauer@airservicesaustra
om
Austria 3) Capt. Wolfgang Mueller Aviation Forum +43 2236 49449 +43 2236 49449 office@aviationforum.com
4) Capt. Peter W. Beer Lauda Air +43 699 1117 3356 +43 223 624 700 beerp@attglobal.net
5) Mag. Beinhauer Lauda Air +43 17000 0 peter.hoffer@laudaair.com
6) Capt. Hans Haerting Lauda Air +43 1 7000 78101 hans.haerting@laudair.com
7) Dr. Alois Farthofer Aviation Psychologist 43 7237 5067 a.farthofer@aon.at
Belgium 8) Fabien Lemoine TNT Airways S.A. 32 4 239 35 02 32 4 239 35 09 Fabien.Lemoine@tnt.com
9) Vanreyten Norbert TNT Airways S.A.
Canada 10) Mr. Greg Down NAV Canada 604 775 9600 604 775 9657 downg@navcanada.ca
11) Dr. Victor Ujimoto University of 519 836 1806 519 837 9567 vjuimoto@uoguelph.ca
Western Ontario
Chile 12) Mr. Andres Gonzalez LanChile 562 565 2129 562 565 2151 agonzalezc@lanchile.cl
Colombia 13) Capt. Hugo E. Blanco Avianca, S.A. 413 8793 413 8793 apsanchez@summa.aero
Barragn
14) Mr. Luis J. Avila Gerlein Avianca, S.A. 413 8793 413 8793 apsanchez@summa.aero
Czech Republic 15) Capt. Jaroslav Hejkal CAA +420 2 2011 1990 hejkal@caa.cz
16) Capt. Miroslav Chalupnicek Czech Airlines 4202 2011 3169 4202 2011 3169 miroslav.chalupnicek@csa.cz
17) Mr. Ladislav Mika CAA +420 9722 1212 +420 9722 31032 ladislav.mika@mdcr.cz
Denmark 18) Mr. Soren H. Nielsen CAA +45 361 86279 SORN@SLV.DK
19) Mr. Finn Mikkelsen Dancopter +45 761 21 450 +45 761 21 459 svs@dancopter.dk
Finland 20) Capt. Heikki Tolvanen Finnair 358 40 520 4330 heikki.tolvanen@finnair.com
21) Capt. Pekka Erkama Finnair 358 40 554 6843 pekka.erkama@finnair.com
France 22 ) Capt. John Scully Airbus 33 5 61 83 33 561 93 2254 john.scully@airbus.com
23) Mr. Stephane Deharvengt DGAC 33 1 58 09 6687 33 1 58 09 4513 stephane.deharvengt@aviation-
civile.gouv.fr
24) Capt. Bertrand de Courville Air France bedecourville@airfrance.fr
25) Capt. Gilles Laurent Air France gilaurent@airfrance.fr
Germany 26) Mr. Richard Lenz Lufthansa 49 69 696 91260 49 69 696 6352 RICHARD.LENZ@DLH.DE
27) Capt. Lutz Becker German ALPA 49 6102 3700 49 6102 370298 werner@ucockpit.de
28) Capt. Hartmut Fabisch Intercockpit 49 69 696 96512 49 60 696 76407 HFH@intercockpit.com
29) Capt. Volker D. Link CAA 49 6142 946180 49 6142 946159 volker.link@lba.de
30) Mr. Rolf Monning CAA 4915311 2355 510 49 15311 2355 rolf.monning@lba.de
742
Ireland 31) Capt. Ray Conway Ryanair 8121 395 8121 213 reillys@ryanair.com
32) Capt. Jack Killoch Cityjet 353 1 870 0100 353 1 870 0185 jack.killoch@cityjet.com
33) Capt. Brian Tyrrell Cityjet 353 1 870 0100 353 1 870 0185 brian.tyrrell@cityjet.com
34) Mr. Mick Hipwell Cityjet 353 1 870 0121 353 1 870 125 mick.hipwell@citijet.com
35) Mr. Mick OConnor Cityjet 353 1 870 0193 353 1 870 0205 michael.oconnor@citijet.com
36) Capt. Ben Bowles Air Contractors 353 1 812 1964 353 1 812 0964 B.Bowles@aircontractors.com
(Ireland) Ltd.
37) Capt. Con Murphy Air Contractors 353 1 812 1964 353 1 812 0964 C.Murphy@aircontractors.com
(Ireland) Ltd.
38) Capt. Ted Murphy Jetmagic
39) Capt. Henry Donohoe Aer Lingus
40) Capt. Denis Coughlan Aer Lingus
41) Capt. Jimmy Courtney Aer Lingus
42) Capt. Conor Rock Aer Lingus
43) Capt. Tom Salter Aer Lingus
44) Capt. Davina Pratt Aer Lingus
45)Capt. Colm Wynne Aer Lingus
46) Capt. Paul Soffe Aer Lingus
47) Capt. Brian Hickey Aer Lingus
48) F/O David Sleith Aer Lingus
49) Capt. Edward Smith Aer Lingus
50) Capt. Alan Fitzpatrick Aer Lingus
51) F/O Hans Munnelly Aer Lingus
52) Capt. Frank Condon Aer Lingus
53) Capt. Mick Mulligan Aer Lingus
54) Capt. Brendan Bruton Aer Lingus
55) F/O Paul Mc Lernon Aer Lingus
56) Capt. Jeff Brownlow Aer Lingus
57) Capt. Con Foley Aer Lingus
58) Capt. Simon Croghan Aer Lingus
59) Capt. Enda Byrne Aer Lingus
60) Capt. Mike Murphy Aer Lingus
61) F/O Stephen OReilly Aer Lingus
62) Capt. Jimmy Devlin Aer Lingus
63) F/O Jim Corby Aer Lingus
64) Tom Curran Aer Lingus
65) Capt. Crios OCinneide Irish CAA
66) Dr. Sam Cromie Aerospace 605 1053 671 2006 sdcromie@tdc.ie
Psychology Research
67) Mr. Daniele Baranzini Aerospace 605 1053 671 2006 baranzid@tdc.ie
Psychology Research
68) Dr. Nick McDonald Aerospace 605 1053 671 2006 nmcdonald@tdc.ie
Psychology Research
69) Mr. Paul Liston Aerospace 605 1053 671 2006 listonp@tcd.ie
Psychology Research
70) Ms. Marie Ward Aerospace 605 1053 671 2006 warda@tcd.ie
Psychology Research
71) Julie Garland Aer Arann 814 1060 814 5260 mark.alder@aerarann.com
72) Capt. Dave Sheppard Aer Arann 814 1060 814 5260 mark.alder@aerarann.com
73) Capt. Mark Alder Aer Arann 814 1060 814 5260 mark.alder@aerarann.com
74) Capt. Conor Nolan IALPA
75) Capt. Joe Elliot IALPA
76) Mr. Pat Feely GECAT
77) Capt. Neil Johnston ICAO Consultant
Italy 78) Capt. Claudio D. Caceres Volare Group 39 335 783 4601 39 445 8000 47 c.d.caceres@libero.it
79) Capt. Bruno del Monte Azzurra Air 39 0331 285 801 39 0331 285 5872 bruno.delmonte@azzurraair.it
80) Capt. Francesco Castaldi Sviluppo Aeronautico 39 045 594 541 FCAST@TIN.IT
81) Eng. E. Deodati Ente Nazionale 390644 596275 390644 596271
Aviazone Civile
Japan 82) Capt. Kimihiro Yamada ANA +81 03 5757 3683 +81 03 5757 5406 kim.yamada@ana.co.jp
83) Mr. Yutaka Tanno JAL 3 5756 3482 3 5756 3576 yutaka.tanno@jal.com
84) Capt. Shigteru Shugyo JAL 3 5756 3111 3 5756 3523 shigeru.shugyo@jal.com
85) Capt. Satoru Nakamura Japan Air System 81 3 5756 8186 81 3 5756 8854 hiroshi.tsukagoshi@jal.com
Co., Ltd.
86) Mr. Shiro Tachibana Japan Air System 81 3 5756 8186 81 3 5756 8854 hiroshi.tsukagoshi@jal.com
Co., Ltd.
Kenya 87) Capt. Geoff Price Regional Air +254 20 605730 +254 20 500845 gprice@airkenya.com
Lantau Hong Kong 88) Capt. Andy Kumaria Dragonair 852 3193 3310 852 3193 8843 andy.kumaria@dragonair.com
Lithuania 89) Capt. Vilmantas Mazonas Lithuanian Airlines +370 525 25544 +370 521 668 28 v.mazonas@lal.lt
Luxembourg 90) Capt. F. Lucien Friob Luxair +352 4798 4500 +352 4798 4599 lucien.friob@luxair.lu
Malaysia 91) Capt. Missman Leham Malaysia Airlines 603 852 52960 603 85253 104 missman@mas.com.my
Malawi 92) Capt. Kalero Micandamire Air Malawi 265 1692 245 2651 692 325 opsdirector@malawi.net
Malta 93) Capt. Michael Obrien Department of Civil 356 212 22 938 356 212 39 278 michael.obrien@gov.mt
Aviation
Mexico 94 ) Capt. Manuel Bustamante AVIACSA 55 5701 7842 aviacsafe@aviacsa.com.mx
Morocco 95) Capt. Abdallah Aboulkabila Royal Air Maroc 212 22 2663 212 22 2597 mobed@royalairmaroc.com
96) Capt. Driss El Fahli Royal Air Maroc 212 22 2663 212 22 2597 delfahli@royalairmaroc.com
Netherlands 97) Capt. B.P. Vandenborn Marduc & Flight 31 251 374958 marduc@hetnet.nl
Safety
98) Capt. Hans Sypkens VNV, Dutch ALPA +31 204 498 585 +31 204 449 8588 hans.sypkens@wxs.nl
99) Capt. Bart Peters VNV, Dutch ALPA +31 204 498 585 +31 204 449 8588 secr@vnv-dalpa.nl
100) Mr. Charlie Govaarts ATC the Netherlands 31 20 406 603 31 20 406 3608 c.govaarts@lvhl.nl
New Zealand 101) Capt. Julian Alai Air New Zealand 64 9 256 4384 64 9 256 3797 julian.alai@airnz.co.nz
102) Capt. Tim Allen CAA 64 4 560 9422 64 4 560 9452 allent@caa.govt.nz
103) Capt. Chris Kriechbaum Air New Zealand 64 9 255 8384 64 9 2563 797 chris.kriechbaum@airnz.co.nz
Norway 104) Capt. Haavard Vestgren Braathens +47 482 00591 +47 648 26594
105) Capt. Jarle Gimmestad Braathens
106) Capt.Gudmund Taraldsen CAA +47 2331 7925 +47 2331 17995 tar@caa.no
107) Capt. Oddbjorn Jensen CAA +47 2331 17919 +47 2331 17995 oje@caa.no
108) Mr. Bjorn Johansen Wideroes Airlines 47 755 13500 47 755 13581
ASA
109) Mr. Kenneth Lauritzen Wideroes Airlines 47 755 13500 47 755 13581
ASA
Poland 110) Mr. Andrzej Winieswski Civil Aviation Office 4822 630 1531 4822 630 1787
111) Mr. Waldemar Krolikowski Civil Aviation Office 4822 630 1544 4822 630 1787 wkrolikowski@ulc.gov.po
112) Mr. Zbigniwe Slusarek LOT Polish Airlines 4822 606 8685 4822 606 7935 z.slusarek@lot.pl
113) Capt. Wieslaw Jedynak LOT Polish Airlines 4822 606 7326 4822 606 7920 w.jedynak@lot.pl
Romania 114) Mr. Mihaela Georgescu CAA 4021 204 1571 4021 204 1572 mihaela.georgescu@caa.ro
115) Mr. Razvan Prunean Inspectorate of Civil 4 021 223 3079 4 021 3124 791 aaib@mt.ro
Aviation
Singapore 116) Mr. Jose M. Anca Singapore Air 65 654 98028 65 67874109 Joey_Anca@singaporeair.com.sg
117) Mr. Hock Yew Khoo Singapore Air HY_Khoo@singaporeair.com.sg
South Africa 118) Capt. Levin Scully South African 27 978 5830 27 978 2965 RikaAlbarn@flysaa.com
Airways
Spain 119) Capt. Francisco Candela Air Nostrum 96 196 0343 96 196 0328 fcandela@airnostrum.es
120) Ms. Ana Diez Air Nostrum 34 96 196 0253 34 96 196 0315 adiez@airnostrum.es
121) Mr. Alfonso Barba Aena 971 449 740 971 745 134 abarba@aena.es
122) Mr. Fernando Martnez Spanair 971 74 5020 971 74 5226 martinez@spanair.es
Ruz-Ayucar
123) Mr. Ignacio Mrquez- Spanair 971 745 020 971 745 5326 martinez@spanair.es
Horaga
124) Capt. Juan Nunez Air Europa +34 971 178 382 +34 971 178 398 crm-ffhh@air-europa.com
125) Jos Costa Air Europa +34 971 178 382 +34 971 178 398 crm-ffhh@air-europa.com
126) Capt. Juan Mauri de Vera COPAC 34 91 324 5031 34 91 324 5033 copac@copac.es
127) Capt. Cristina Perez COPAC 34 91 324 5031 34 91 324 5033 copac@copac.es
Cottrell
128) Mr. Rafael Bejarano Iberia +34 91 587 3698 +34 329 30 39 rbejarano@iberia.es
129) Ms. Carmen Linares Iberia +34 91 587 3698 +34 91 329 3039 clinares@iberia.es
Miquez
130) Mr. Rafael Ciudad Iberia 34 91 587 4178 jmponz@iberia.es
Sweden 131) Bo Johansson Swedish Safety 46 11 192009 bo.ken.johansson@ifv.se
Aviation Authority
Switzerland 132) Capt. Timothy Crowch Mayday Ltd. 41 52 317 3624 41 52 317 3622 maydaysafety@attglobal.net
133) Mr. Gallus A. Bammert Swiss Aviation 41 1 564 59 56 gallus.bammert@swiss-aviation-
Training training.com
134) Mr. Renato Breda Swiss Aviation 41 1 564 5956 renato.breda@swiss-aviation-
Training training.com
135) Ms. Sibylle Da Pra Swiss International 41 793 710 158 41 1564 4026 prsy@swiss.com
Airlines
Taiwan 136) Capt. Harry Holling EVA Air +886 3351 6352 +886 3351 0025 harry.who@msa.hinet.net
137) Capt. Jim Sydiongco EVA Air
138) Ms. Judy Tsay China Airlines +886 2 271 23 141 +886 2 251 46 923 mdprincess@email.china-
ext. 6540 airlines.clom
139) Chien-Hung (Taylor) Pai China Airlines +886 2 271 23 141 +886 2 251 46 923 md11fo@ms75.hinet.net
140) Jin-kook Choi Asiana Airlines jkchoi2@flyasiana.com
141) Ms. Son Tae-ja Asiana Airlines
Tunisia 142) Mr. Adel Bouajina CAA 216 71 700 ext 3160 216 71 704 927 Bouajina.adel@Email.ati.tu
United Kingdom 143) Capt. Graham Gray UK CAA 44 1293 828230 44 1293 82404 grahamgray@org.caa.co.uk
144) Ms. Fiona Merritt UK CAA 44 1293 573 485 44 1293 573 984 fiona.merritt@org.caa.co.uk
145) Dr. Edmund Hughes Maritime& 44 23 80 397 872 44 23 80 329 251 edmund_hughes@mega.gov.uk
Coastguard Agency
146) Mr. Norman Macleod Kittyhawk Training 44 1480 810 526 44 1480 811 426 norman@turboteams.com
Technology
147) Mr. Ron Elder UK CAA 44 1293 573 079 44 1293 573 974 ron.elder@org.caa.co.uk
United States 148) Dr. Sherry Chappell Delta Air Lines 404 773 8820 404 773 0643 sherry.chappell@delta.com
149) Capt. Ralph E. Hicks Delta Air Lines 404 715 6816 404 715 2680 ralph.hicks@delta.com
150) Mr. Chuck Schramek Delta Air Lines 404 715 1112 404 715 1853 chuck.schramek@delta.com
151) Mr. Steve Hill Delta Air Lines 404 715 1112 404 715 1853 steve.hill@delta.com
152) Mr. Jack Rubino United Airlines 303 780 5645 303 780 3770 jack.rubino@ual.com
153) Capt. Mike Taylor FEDEX 901 397 9820 901 397 9597 mdtaylor@fedex.com
154) Chris Henry UT/Aerospace
Research
Boeing 155) Dr. Juergen Hoermann Boeing Hans.J.Hoermann@boeing.com
EUROCONTROL 156) Dr. Ian R. Patterson EUROCONTROL 322 729 3515 322 729 9149
157) Mr. Eoin McInerney EUROCONTROL 352 436 0611 352 438 669 eoin.mcinerney@eurocontrol.int
158) Mr. Marc Deboeck EUROCONTROL 322 729 9139 322 729 4787 marc.deboeck@eurocontrol.int
159) Dominique Van Damme EUROCONTROL
160) Mr. Giancarlo Ferrara EUROCONTROL 322 729 3712 322 724 9082 giancarlo.ferrara@eurocontrol.int
IFATCA 161) Mr. Bert Ruitenberg IFATCA 514 866 7040 514 866 7612 office@ifatca.org
HFWG 162) Capt. Mohammed Aziz Middle East Airlines
163) Ms. Martine Lacoste Air France
164) Mr. Jerry Allen Delta
PRESENTERS
Capt. Dan Maurino ICAO
Ms. Jill Sladen IATA
Capt. Mike Bombala IATA
Prof. Robert Helmreich UT
Mr. James Klinect UT/TLC
Capt. Pat Murray UT/TLC
Capt. Don Gunther Continental
Capt. Henry Craig Cathay Pacific
Capt. Morten Ydalus Braathens
Capt. Robert Aran Futura
Mr. Jean-Jacques Speyer Airbus
Dr. Curt Graeber Boeing
Capt. Carlos Arroyo Landero IFALPA
Capt. Robert Sumwalt US Airways/ALPA
Capt. Chirs Kriechbaum Air New Zealand
G:\HF\WP\LOSA\Dublin\Proceedings\list of participants.wpd
Development of am aviation safety case is essential as it focuses a company's top management
and staff on the real risks that need to be managed and ensures that every reasonable effort
is taken to provide safe operations.
The introduction of a safety case offers a an operation and addresses only the
CLIFFORDJ. EDWARDS company's senior management the major hazards, such as the potential for
SHELLAIRCRAFT opportunity to identify the major safety fatal accidents, which are critical to the
(UNITEDKINGDOM) risks. Based on this knowledge, a com- company's well being. Although a com-
pany's board can establish controls that pany's safety case is subordinate to its
AFETY improvements have been
S
reduce the likelihood of such risks caus- safety management system, they should
achieved over the years through ing an accident. interact so that each safety case assures
numerous developments, includ- The commitment and organization control of its hazards. The safety man-
ing better aircraft design, redundant sys- that assures continuing safe operations agement system and the safety case are
tems, improved working practices and is achieved through the introduction of linked in many ways, primarily through
the introduction of quality assurance a safety manag~mentsystem. A safety the hazard registers, with the safety
programmes, to name just a few. management system must be led by top management system's hazard register as
Despite all that has been accom- management and must address all the master list of all hazards.
plished, experts predict an increase in the aspects of the business that have the The key steps in developing a safety
potential to cause harm. case require that a corporate safety man-
HAZARDIDENTIFICATION AND CATEGORIZATION
The structured approach agement system exists or at least is
l. viat ti on taken to identify, assess and being developed. The safety case draws
I Qase control the hazards is known on corporate safety objectives and policy,
I
as hazard management, a which must make safety an explicit pri-
Workplace
process that results in the ority, at least equal to any other business
Procedures development of a hazard reg- imperative. Based on corporate deci-
- (defined in SMS)
ister. Throughout 1999, Shell sions as to what safety level is to be man-
Figure
Aircraft worked with a num- aged, hazards are identified and risks
number of aircraft accidents as worldwide ber of airlines and other operators to assessed and controlled. Management
air traffic continues its steady growth in build a generic hazard register (Figwe must also develop and maintain a sup-
the years ahead. Unless significant 1)that can be tailored to any operator, portive culture that is "just" and "learn-
changes are made to improve the nearly enabling resources to be focused on the ing." In aviation, this cultural change
flat accident rate, by 2010 there could be areas of greatestrisk. An efficient way to requires a willingness to learn from haz-
an average of one airline accident per manage this process is the safety case. ards and threats as well as from acci-
week. Left unchecked, this level of accidents and incidents. At the same time,
dents would alarm the public and could Developing the safety case management must deal sensitively with
place many aircraft operators in financial A company's safety management sys- those responsible, unless reckless or
difficulty or even out of business. For that tem, which is defined as a systematic deliberate behaviour warrants discipli-
reason alone, the cost of enhancing safety and explicit approach to managing risk, nary action. It is essential that training
systems is easily justified. is largely a loss control management provide all staff with an understanding of
To further decrease the accident rate, system. It defines how the company safety management and the extent of the
safety management needs to be per- intends to manage safety as an integral corporation's commitment to safe opera-
ceived by senior management - espe- part of its overall business. A safety man- tions.
cially a company" chief executive officer agement system addresses all aspects of A safety case is the "systematic and
and board of directors - as an essential safety in the operation and should deal structured demonstration by a company
business requirement and not an activi- with all levels of risk. By comparison, a to provide assurance, through compre-
ty to be addressed only by subordinates. safety case focuses on specific parts of hensive evidence and argument, that the
ICAO JOURNAL
Hazards
Fuel People
Jnairworthy Aircrafl
BOWTIEMJILW~B
urw alone we mt enough
w they win bs &mr;nvent-
ed if th& purpose ia not
Figure 3.
Potential consequence of the ~nc~dent Increas~ngprobab~l~ty
I
a d , in that it has th& ptitm&l B r harm

thrcqgh it6 hhrent mmgp Tf the &-
craft is n ~ aminMxed
t in sr, wntrolled
~taka hamdom emnt may o w , and
theri&re meamrw m r-ed pre-
rm'tthtkra &n&m &am rnr~rt?iniq~ The
hhi4s fiat.crew &don7inaccorh~d:
with prwedww and a h ~ ~ wifX ~ 4 b ~
rg&ot-e~;pmtting equilibrium. lf t h e e
meaanr+s -the d r d will ikdy ad-
fer a ~ o n s q m m .
The initialtat& caf the m h h o p e wm
to Mm@ ham& and list thew c s an
entry psht. hseW&g az hWwd a& %mb
Wf~g with the patmthl to emme h&
endbled pmasripmb to tOqdentBy b a r d s
md co&m they hsd energy whi& mdd
be relewdwd mt3e h m . T h ~ p r o a m
tba contitln& idmt@ing prtMM flight
md gmmd lmmds2iflddbgh&m.
The g x x ~ r nof a h
locaVom esuld mrW &mt ~ontsoh
w rwovery mak;tlrm.
The.mrk&ops mmed on to Idmfiy
Appropriate data from different airlines should be pooled *incident investigation often addressed effect rather than
to support arguments for improvements that need to be made cause and therefore denied the company the chance to learn.
by manufacturers, ATC, regulators and airports. Finally, The hazard modelling workshops were carried out over
broadband satellite communications would allow cost-effec- eight months in 1999 with pilots and engineers from eight air-
tive transmission and analysis of FOQA data in real time. lines and five helicopter operators. These workshops pro-
duced two generic hazard models, one each for f i e d - and
rotary-wing application. Nineteen generic hazards were iden-
tified. Each of the hazardous events was discussed at length,
continued from page 6 and control methodologies defined. It became clear that the
knowledge of the investigative process. The aviation investiga- means of controlling a hazard varied depending on whether
tion specialist spends much of his time with the investigator in the aircraft was in flight, undergoing maintenance or moving
charge and the various investigative groups. By understanding on the ground. In all, four fixed-wing and six rotary-wing loca-
what is occurring in the investigation and coordinating the fac- tions were defined. To aid with generation of bow-tie models
tual information which the investigator in charge will release for each hazardous event, generic threat and threat control
to the public, the aviation investigation specialist can brief fam- lists were assembled. These included descriptions for each
ily members with up-to-date information on a daily basis. It is threat and the source where the threat or control would be
important to emphasize that the victims and their family mem- relevant. These generic models can be adopted for any air-
bers must receive the same factual information as the media. craft operation. The generic hazard model is now being trans-
The development of any family assistance programme lated into the field by a number of operators who are
requires a partnership among many organizations and a com- customizing it to specific operations.
mitment to the shared goal of assisting victims and their fami- Conclusion. Development of a safety case involves signifi-
ly members. This is especially true for air carriers, which have cant effort by aircraft operators. However, projected growth
a fundamental responsibility to victims and their families fol- in the number of accidents is unacceptable. Current efforts
lowing an aviation disaster. It has been demonstrated in the are somewhat piecemeal and are not reducing the accident
past three years that by planning, coordinating and communi- rate. A positive, integrated approach with support structures is
cating with one another, positive changes can take place in the required to improve the situation. To make further progress
way the aviation industry and government respond to the will require changes in corporate culture, including manage-
needs of aviation accident victims and their families. 0 ment's approach to safety. Some would argue that the indus-
try is over-regulated,but this viewpoint is insupportable when
the costs of human life and corporate liability are taken into
Generic hazard model consideration. 0
continued fiom page 14
decisions about which to address, and when.
Safety improvements. Many of the improvements identi- Accident report
fied could be made without much effort or cost. Even so, continued fiom page 20
some additional controls were identified that would have real a flight service station specialist, as opposed to a tower con-
costs. The prime findings of the process were that: troller, at the Fredericton airport at the time of the arrival of
*management reviews must be more active to ensure that the flight was not material to this occurrence.
intended improvements take place; Safety action taken. Procedures outlined in Air Canada's
*safety competence and accountability are often ill-defined or Aircraft Technical Bulletin and Bombardier's All Operator
missing in the organization, in particular the ability to trace Message, issued following this occurrence, will reduce the
safety accountabilityfrom the CEO down; possibility of ice accumulation on the CL-65 aircraft.
training in non-flying/technical areas was lacking, especially Nevertheless, there is still a risk that while an aircraft is oper-
when staff are promoted to management with significant ating below 400 feet AGL, ice could accumulate to an extent
changes in skill and knowledge requirements; that aircraft performance would be materially affected with-
*there was a significant amount of work being done with the out the pilots being aware that they had entered icing condi-
best of intentions but without regard to procedural require- tions, or that ice had accumulated. If the amber ICE light
ments; were not inhibited below 400 feet, however, an extra safe-
*use of procedures, notably in engineering, was not system- guard would be in place to alert pilots to the presence of ice.
atic and often not assessed by supervision or audit; In a TSB Aviation Safety Advisory to Transport Canada on 9
*workplace monitoring and supervision practices were inad- April 1999, it was suggested that Transport Canada consider
equate; taking action to remove the inhibition of the ICE light below
*processes to manage change were ineffectual; 400 feet.
*audit processes were frequently inadequate; In February 1998 the TSB, in an Aviation Safety Advisory,
*human factors were not well addressed, with shortfalls in suggested that Transport Canada consider eliminating the
training and/or application of the principles; and ELT carriage exemption for turbojet aircraft. In response,

ICAO - Human Factors

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

ICAO - Human Factors

Uploaded by

Copyright:

Available Formats

ICAO - Human Factors

Flight Safety Human Factors

Articles from the ICAO Journals

Top Ten Recommended Introductory Reading List of Free ICAO Material

http://www.icao.int/ANB/humanfactors/icaojournallist.htm (1 of 5)16-02-2007 22:36:23

Crew Resource Management

http://www.icao.int/ANB/humanfactors/icaojournallist.htm (2 of 5)16-02-2007 22:36:23

Flight Operational Quality Assurance (FOQA),

http://www.icao.int/ANB/humanfactors/icaojournallist.htm (3 of 5)16-02-2007 22:36:23

http://www.icao.int/ANB/humanfactors/icaojournallist.htm (4 of 5)16-02-2007 22:36:23

http://www.icao.int/ANB/humanfactors/icaojournallist.htm (5 of 5)16-02-2007 22:36:23

nance procedure was "proscribed" (i.e

u~fi.11in maintaining generd &&on &-

j d a r or less tzqxs5-d g e m m 73%

she and w m t l w t i o n of the t3paid.n~pro-

damroam mdhod~.Certainly these new

Usually M e or na remedial im&ucBan is

.aide mare Twm&

MD-1I improvement programme Conclusion

VOL. 48 - NO. 7 SEPTEMBER 1993

Editor: Eric MacBumie Production Clerk: Denise Cooper-Altuve

5 AVIATION moves towards the Z l s t ten- ~ ~

A tury, one of our key objectives is the suc-

focus, but worthy of industry attention. It

rn (knowledge and values or attitude) as train-

costs. These alone justify the Aer Lingus Classroom training

practical wpe& af CRll/r md cxrckpit m-

dam*The trahdnginvolves hquenk t%qm

=a, rather tbm the place -Pv6k:reit is fXmt

retard b &omd .on a mrnmrkr to tot%&&

three-pilot flight deck. nology cockpit. These lessons emphasize

Implications for IhFT

VHF omnidirectional radio rangeldistance

42. Because of the topography near

43. It was unlikely that any cockpit

munication difficulties between them

Jeppesen chart for the Sierra approach

when operating the FMS. 1

HUMAN FACTORS TODAY:

Development of PEAT For some time now, human factors

supervision of people is the critical of selection, the objective is to recruit

nology cockpit design. However, a 1996 oriented performance. This means, in

Y2K initiatives have already achieved

continued from page 14

I * Airline Pilot Qualification Program Certification after 10 months CPL/IR

Objectives and agenda

The Line Operations Safety Audit (LOSA)

Short Course Observer Training

Threat and error management training

To provide the basics of threat and error management training;

To discuss the potential contribution of non-airline organizations, including civil aviation

1000 - 1200 Coordination meeting

1300 - 1630 Welcoming remarks

1630 - 1700 Q&A session

0900 - 1700 LOSA observer training sessions

0900 - 1300 Threat and error management training

1400 - 1700 LOSA implementation strategy discussion

1700 Conclusions and wrap up

Firstly, the forms presented in the report are

Robert L. Helmreich1, James R. Klinect, John A. Wilhelm & J. Bryan Sexton

ABSTRACT 2. Incident reports. These are useful because

international LOSA database.

In discussions with senior FAA officials, the Threat Management

potential extensions of LOSA methodology have been Safety