LOSA: Line Operations Safety Audit: History and Status

Robert L. Helmreich, PhD, FRAeS

Human Factors Research Project The University of Texas at Austin
Darling Harbour December 1, 2003

LOSA

Jump seat observations of flight crew performance during regular scheduled flights

Observers unobtrusive collecting data not participating in flight

Team of observers from different backgrounds

Line pilots / Union representatives Check airmen Safety and Training pilots Credible outside observers (for reliability check)

All data are DE-IDENTIFIED and CONFIDENTIAL

Scientific Background

LOSA uses systematic observation of behavior Systematic observation is a validated methodology using observers trained and calibrated to high reliability Systematic observation has been employed in scientific studies of crew performance in demanding environments

For example, Aquanauts in undersea habitats during Project SEALAB and Project Tektite

History of LOSA

Formal LOSA developed at request of Delta Airlines to validate operational impact of Human Factors (CRM) training The focus of initial LOSA was systematic assessment of CRM-related crew behaviors In 1997, collaboration with Continental Airlines to expand LOSA to include threats and errors and their management

Purpose of LOSA

Provide valid empirical data on: 1. Crew performance strengths and

weaknesses

Proficiency Decision-Making CRM skills Procedural compliance Threat and error management

2.

weaknesses

System performance strengths and

Culture Airspace System airports and navigational Aids Aircraft design / automation Standards / Training / Safety / Maintenance Crew support ATC, Cabin, Ground, and Dispatch

Threat and Error Management LOSA (1997-2003)

Continental Latin America Continental Express Air New Zealand Air Micronesia Continental Delta USAirways Cathay Pacific

EVA Air Uni Air (Taiwan) Frontier QANTAS Singapore Silk Air Braathens Alaska Air New Zealand (2nd)

LOSA Components

Part 1. Threat prevalence & mgt Part 2. Error prevalence & mgt Part 3. CRM Countermeasures Part 4. Flight crew interview Part 5. Flight crew survey

LOSA Data: Flight Crew

For each flight segment, observers collect data on:
General Flight Crew Information

Flight Crew Performance

Demographics Attitudes / perceptions / safety culture (FMAQ) Safety interview comments Observer narrative Overt threats

Behavioral markers Crew errors and violations Undesired aircraft states

Flight Description

Technical data for approaches

Type and stability

Operational complexity

Threat management Error management Undesired state management

Threat Categories
Threats - Originate outside the flight crews influence but require active management to prevent them from becoming consequential to safety

Environmental Threats

Airline Threats

Adverse weather ATC events / errors Terrain Traffic Airport conditions

Operational time pressure Cabin events / errors MX events / errors A/C malfunctions / MELs Ground / Ramp events / errors Dispatch events / errors Ground crew events / errors

Latent Threats

Aspects of the system that predispose the commission of errors or can lead to undesired aircraft states

ATC practices Organizational, national, professional culture Aircraft characteristics Qualification standards Regulatory practices and oversight Flawed procedures Scheduling and rostering practices

Threat Prevalence
Last 10 LOSAs

1835 flights with 7576 threats

4.1 threats per flight average

98% of flight segments had one or more threats

Range across airlines 94% to 100%

2/3 were environmental threats; 1/3 were airline threats

44% of environmental threats occur in descent / approach / land 72% of airline threats occur in predeparture

Most prevalent threats

Adverse weather 26% of all threats ATC clearances/late changes - 21% of all threats

Threat Management
Last 10 LOSAs

960 mismanaged threats in 1835 flights

Average 1 mismanaged threat every 2 flights Range across airlines 79% to 92%

87% of threats effectively managed

Environmental and airline threats managed equally well Highest mismanagement rates

17% of ATC threats were mismanaged 15% of Aircraft malfunctions /MELs threats were mismanaged

Error Avoidance

Complete error avoidance is impossible errors are inevitable 80%+ of all flights observed had one or more errors

Must look for sources of error to strengthen system defenses

Broad Error Categories

Intentional Noncompliance violations
ex) Performing a checklist from memory

Procedural Followed procedures but wrong

execution
ex) Wrong altitude setting dialed into the MCP

Communication Missing information or

misinterpretation within cockpit
ex) Miscommunication by crew with ATC

Decision Discretionary decision that unnecessarily

increases risk
ex) Unnecessary navigation through adverse weather

Decision Error
Choice increasing risk in a situation with
multiple courses of action possible time available to evaluate alternatives no discussion of consequences of alternate courses of action no formal procedures to follow

Error Prevalence
Last 10 LOSAs 1835 flights with 5172 errors = 2.8 errors per flight on average 82% of flight segments had one or more errors

Range across airlines 70% to 94%

32% of all errors are intentional noncompliance errors

Error Response

25% Detected and action taken 30% Detected and ignored 45% Fail to detect

Red Flag: improve monitoring and cross-checking

Error Outcomes

71% Inconsequential 21% Undesired aircraft state 8% Additional error

1484 mismanaged errors in 1835 flights = 0.8 mismanaged errors per flight on average
77% of intentional noncompliance errors were inconsequential

Are procedures in need of review?

Undesired Aircraft State

A compromised situation placing the flight at increased risk

Lateral deviation
Vertical deviation Speed too high Speed too low Incorrect aircraft configuration Flight controls Systems Fuel Automation

Unstable approach
Abrupt aircraft control Long landing no go around

Firm landing Wrong taxiway, ramp, runway, country

Runway incursion

Error/Outcome Summary
Procedural

Communication

Decision

Non-Compliance 0 20 40 60 80 100

% of errors

% consequential

7% of errors involved a lack of technical proficiency

Phase of Flight Effects

Phase of Flight
Pre-Departure / Taxi Takeoff / Climb Cruise Descent / Approach / Land

Threats by Phase 40% 15% 8% 33%

Errors by Phase 26% 20% 6% 44%

Taxi / Park

4%

4%

Descent / approach / land phase contains the most variability in crew performance and consequential errors

Threat and Error

Is a high level of threat associated with more errors?

There are slightly more unintentional errors when there are many threats, but the relationship is weak There is no relationship between threats and intentional non-compliance

Profile of Airline X
Major Statistics Average number of threats per flight Airline X

Average from 4 LOSA Archive Comparison Airlines

4.3 8% 3.0

4.3 15% 2.5

% of threats that were mismanaged

Average number of errors per flight

% of errors that were mismanaged

18%

31%

Airline X
Major Statistics Average number of mismanaged errors per 100 flights Average number of undesired states per 100 flights Average number of mismanaged undesired states per 100 flights Airline X Average from 4 LOSA Archive Comparison Airlines

56
42 6

75
53 8

Myths from the Days of Jurassic Jets

Myth 1: The glass cockpit will eliminate human error on the flight deck

Reality: Automation is the second largest source of pilot error

Myths

Myth 2: Dog and Duck automation will prevail

3 individuals in crew Pilot, dog, duck

Pilot reaches for controls Duck quacks Dog wakes and bites pilot

Reality: the human is still in the loop

University of Texas Threat and Error Management Model (UT-TEMM)

The Threat and Error Management model was derived from LOSA data and guides further refinement. The model is being used by airlines as a framework for analysis of incident and ASAP data IATA is using it as the framework for analysis of worldwide accidents and incidents

Threat and Error Management Model

Threats: Latent and Overt

Threat M anagement

Errors

Inconsequential

Error M anagement

Incident / Accident

Undesired Aircraft State

Undesired Aircraft State M anagement

Applying LOSA Data

LOSA data have three primary uses: 1. Identifying issues for action 2. Assessing system safety

LOSA database has data on airports, aircraft, crew experience, organizational and professional culture Providing airlines with feedback on their own operations Observers provide valid record of what crews do on the line Show areas of strength as well as those needing improvement Data help airlines prioritize and evaluate safety efforts

CRM training

3.

International database for human factors research

Using Error Data for Organizational Interventions

Violations - suggest poor procedures, weak captain leadership and/or a culture of non-compliance Procedural errors - may indicate poor workload management and/or poor procedures Communications errors - may reflect inadequate CRM (monitoring and challenging) or complacency Decision errors - may indicate need for more CRM training on expert decision making and risk assessment

Predicting LOSA Performance from Survey (FMAQ) Data

Forming Safety Culture Scale

Pilots observed in LOSA also completed a survey with items from the UT Flight Management Attitudes Questionnaire Items related to organizational culture regarding safety were summed to form a Safety Culture Scale

Safety Culture Scale Items

.69

The managers in Flight Operations listen to us and care about our concerns. My suggestions about safety would be acted upon if I expressed them to management. Management will never compromise . safety concerns for profitability I am encouraged by to report any unsafe conditions I observe. I know the proper channels to report my safety concerns. I am satisfied with Chief Pilot and Assistant Chief Pilot availability.

.70

.62

Safety Culture

.65

.42

.50

Safety Indices
% Crews with an Undesired State
100 90 80 70 60 50 40 30 20 10 0

67%

Scale 0-100

54% 14%
Base Rate

Low Safety Culture Crews

Avg Safety Culture Crews

High Safety Culture Crews

LOSA and CRM

Twenty Years of Change

Since 1981, CRM has evolved steadily from early courses that were derived from corporate management and focused on individual styles and skills By 1998, the focus of the fifth generation had shifted to error management In 2001, sixth generation has become Threat and Error Management

CRM skills defined as threat and error countermeasures

Continental Airlines used LOSA data to develop Threat and Error Management CRM training the 6th Generation

Threat and Error Management 1999

Fifth generation was useful, but created resistance among pilots who did not like the idea that their task was managing their own errors LOSA documents not only errors but also the nature and prevalence of threats in the operating environment shows superior performance as well as problems

Training Issues in Threat and Error Management

Human limitations as sources of error The nature of error and error management Culture and communications Expert decision-making Training in using specific behaviors and procedures as countermeasures against threat and error

Briefings Inquiry Sharing mental models Conflict resolution Fatigue and alertness management

Analysis of positive and negative events and accidents

Recognition and Extension of LOSA

Flight Safety Foundation/Aviation Week and Space Technology Laurels, 2001 DOSA - Dispatch LOSA (with Continental) University of Texas Center of Excellence in Patient Safety adopting LOSA methodology for systematic observational research into operating rooms and emergency rooms

The LOSA Collaborative

An umbrella research organization led by James Klinect Collaborative functions
1)
2)

3)

4)

Implementing LOSA and maintaining the integrity of the process and database by certifying observers and data Supporting research efforts of the University of Texas Human Factors Research Project Offering user interface data to manufacturers to enhance design process and de-identified system data to industry and regulators Coordinating a support group of participating LOSA airlines to exchange information on data use and oversight for the process

The University of Texas Human Factors Research Project

www.psy.utexas.edu/HumanFactors