Methodology and initial analysis results for development of non-invasive and hybrid driver drowsiness detection systems

Eugene Zilberg Compumedics Medical Innovation Pty Ltd 30-40 Flockhart St Abbotsford 3067 Australia ezilberg@compumedics.com.au Zheng Ming Xu Compumedics Medical Innovation Pty Ltd 30-40 Flockhart St Abbotsford 3067 Australia MingXu@compumedics.com.au David Burton Compumedics Medical Innovation Pty Ltd 30-40 Flockhart St Abbotsford 3067 Australia dburton@compumedics.com.au Murad Karrar Compumedics Medical Innovation Pty Ltd 30-40 Flockhart St Abbotsford 3067 Australia muradkarrar@compumedics.com.au Saroj Lal University of Technology Sydney (UTS) Broadway NSW 2007 Australia Sara.Lal@uts.edu.au
It has been shown that a number of physiological indicators are associated with the increase in drowsiness [3] and can respectively be used for detecting drowsiness. The reliable indicator appears to be the spectral content of electroencephalogam (EEG) [7,8,9,13], blink rate [14], individual blink parameters [4], and degree of eye closure [6] as well as other physiological parameters. These advances contributed to development of a number of prototype drowsiness detection devices however integration of these systems into vehicles is still significantly impeded by relative invasiveness and inability to accurately respond to the continuum of driving behaviours and different sources of signal noise and artefacts. Compumedics proposed use of non-invasive piezofilm movement sensors that can be incorporated into car seat, seat belt and steering wheel [2]. These sensors are potentially capable of recording patterns of drivers movements, breathing and even heart rate that could be used for identifying the level of drowsiness. Another aspect of Compumedics patented technology includes integration of different kinds of signal analysis including morphological processing of EEG and eye movement patterns that was successfully used for automatic analysis of sleep recordings [11].

Abstract
Application of piezofilm movement sensors integrated into the car seat, seat belt and steering wheel was proposed for development of a non-invasive and hybrid systems for detecting driver drowsiness. A car simulator study was designed to collect physiological data for validation of this technology. Methodology for analysis of physiological data, independent assessment of driver drowsiness and development of drowsiness detection algorithm by means of sequential fitting and selection of regression models is presented. Statistical analysis shows that during the episodes of transitions to dangerous levels of drowsiness movement variations recorded by the seat sensors are decreasing. This finding indicates that the piezofilm movement sensors could be used as noninvasive devices for detecting the level of drowsiness on their own or in combination with other physiological signals.

1. Introduction
Driver drowsiness is an important factor in the motor vehicle accidents [3,5]. It was demonstrated that driving performance deteriorates with increased drowsiness with resulting crashes constituting more than 20% of all vehicle accidents [12].

The 2nd International Conference on Wireless Broadband and Ultra Wideband Communications (AusWireless 2007) 0-7695-2842-2/07 $25.00 2007

This article describes the study conducted as a part of the ARC grant with University of Technology Sydney to explore possibilities offered by this technology, explains the data analysis methodology and presents initial analysis results focusing on investigation of association between transition to a state of drowsiness and patterns of seat movement signals.

2. Methods
2.1. Study description
A car simulator study was conducted at the Monash University Accident Research Centre (MUARC). 60 non-professional fully-licensed drivers aged between 20 and 60 years old were recruited for the study. The standard health exclusion criteria were applied. The driving experiment lasted up to 3 hours in the afternoon preceded by 15 minutes adaptation interval. Both day and night scenarios were relatively monotonous with realistic scenery and repetition period of at least 20 minutes. The recorded physiological signals included 10 EEG channels (C4, O2, Fp1, Fp2, T7, T8, P7, P8 referenced to the respective opposite A1 or A2 corresponding to international 10-20 system), chin EMG, ECG, EOG, eye lid movement sensor (Respironics REMview), thoracic respiratory band, 8 steering wheel pressure signals and multiple piezofilm car seat and steering wheel movement sensor signals. The movement sensors included 7 sensors on the steering wheel, 5 on the back of the seat and 5 on the bottom section of the seat. The Compumedics Siesta portable sleep diagnostic recording system was used for physiological data recording. The information for the assessment of the drowsiness level included 4 video signals (two images of the drivers face, one of the steering wheel and one of the driving scenery shown in Figure 1), estimate of the percentage of eyelid closure over the pupil and time (PERCLOS) [6] from the FaceLAB system as well as a number of driving performance parameters.

All eye (including eyelid) movements were visually detected for EOG, Fp1/2 and eye lid movement sensor data. For all scored eye movement events their morphology was classified and parameters of individual segments were estimated. For the EEG data the different types of analysis were employed including standard spectral analysis, spectral analysis after exclusion of eye movements and artifacts and morphological analysis similar to that used in automatic scoring of sleep stages.

Figure 1. Example of the video images

2.3. Selection of drowsiness rating scale

While the estimate of PERCLOS as a measure for drowsiness was recorded in the study, it was sensible to apply a drowsiness rating scale based on observation of the video images of the drivers face, steering wheel and the road scenery. The rationale for this approach was that the observer ratings of the videotaped segments of drivers at various stages of drowsiness were found to be reliable and consistent [15] and subsequently video analysis was recommended and used in a number of studies as an independent variable for assessment of fatigue [1,9]. The obvious advantage of PERCLOS is that drowsiness rating process could be automated however multiple video images and availability of a trained observer enable arguably more reliable method of assessing drowsiness. The developed observer drowsiness rating scale included 5 levels similar to [15] from alert to extremely drowsy and was based on observing a number of indicators including eye blinks, eye lid movements, degree and duration of eye closures, direction and focus of eye gaze, patterns of facial, body movements, yawning etc. Guidelines for determining the drowsiness states are presented in the Table 1. The drowsiness was estimated for 10 s intervals. From the description of the drowsiness levels it appears that an effective drowsiness countermeasure should be able to reliably detect at least Level 3 that was called significant drowsiness.

2.2. Preprocessing of physiological data

Prior to conducting exploratory and statistical analysis the physiological data were preprocessed as follows. For the seat movement sensor data the peak to peak values were calculated over 2 s intervals with 1 s overlap. For the steering wheel movement and pressure sensor data the peak to peak values were calculated over 1 s intervals with 0.5 s overlap.

The 2nd International Conference on Wireless Broadband and Ultra Wideband Communications (AusWireless 2007) 0-7695-2842-2/07 $25.00 2007

Table 1. Observer drowsiness scale based on the video analysis

Drowsiness Level Drowsiness State Video image indicators Normal fast eye blinks, often reasonably regular; Apparent focus on driving with occasional fast sideway glances; Normal facial tone; Occasional body movements. Increase in duration of eye blinks, and possible increase in the rate of eye blinks; Increase in duration and frequency of sideway glances; Appearance of glazed-eye look and abrupt irregular movements rubbing face/eyes, moving restlessly on the seat. Occasional yawning. Occasional disruption of eye focus; Significant increase in the eye blink duration; Disappearance of eye blink patterns observed during the alert state; Reduction on the degree of eye opening; Occasional disappearance of facial tone; Episodes without any body movements. Discernible episodes of almost complete eye closure, eyes are never fully open; Significant disruption of eye focus; Periods without any body movements (longer than for level 2) and facial tone followed by abrupt large body movements. Significant increase in duration of the eye closure episodes; Longer durations of episodes with no body movements and sometimes followed by large isolated correction movements

Alert

Slightly drowsy

seat and steering wheel, maximum durations of eye lid movements and percentages of EEG power in various combinations of frequency bands. Some of these plots for different subjects are presented for illustrative purposes in Figure 2. These examples indicate that there is potential for establishing associations between increase in drowsiness level and reduction in the peak to peak values of car seat piezofilm movement signal, increase in the eye lid movement duration and increase in the EEG alpha band relative power. It is important to state that the associations were not so evident for all participants.

2.5. Identification of episodes of transitions to significant drowsiness

While an effective drowsiness detection algorithm should operate appropriately across all drowsiness levels and be robust to artifacts the starting point in the algorithm development could be identification of a combination of physiological parameters that are sensitive to a transition to a drowsiness state that is perceived to be dangerous for driving. An obvious consideration for an appropriate combination of parameters is that it should not be likely to be associated with other patterns of drowsiness level variations rather than transition to drowsiness. The appropriate dangerous level of drowsiness was selected as significantly drowsy based on appearance of eye closure episodes. At this stage of analysis 115 episodes of transition to significant drowsiness were identified for 22 participants. An example of selected episodes of transition to drowsiness for one of the studies is presented in Figure 3 with the transition episodes highlighted.

Moderately drowsy

Significantly drowsy

Extremely drowsy

2.6. Statistical analysis of association between transition to significant drowsiness and reduction in the movement sensor signal variation at the back of the seat
Exploratory analysis suggested that movement sensors on the car seat had more prominent variations than the movement and pressure sensors at the steering wheel. Also the sensors located on the back of the seat appeared to be most responsive as demonstrated in Figure 2a. Therefore the initial statistical analysis was focusing on capability of those sensors to detect transitions to drowsiness. The peak to peak variations of the sensor signals were analysed for the episodes of transition to significant drowsiness extended by 30 seconds forward to include at least the initial part of drowsiness episode.

2.4. Exploratory analysis of associations between observer drowsiness rating and physiological indicators
To substantiate selection of appropriate hypotheses and statistical models the exploratory analyses of potential associations between the observer drowsiness rating and various physiological indicators were conducted. These analyses included calculations for every 10 s observer rating interval of average peak to peak values of the piezofilm movement sensors on the

The 2nd International Conference on Wireless Broadband and Ultra Wideband Communications (AusWireless 2007) 0-7695-2842-2/07 $25.00 2007

(a)
4 2 obs. rating 0 0 1000 2000 3000 4000 5000 6000 0 0 1000 2000 3000 4000 4 2 obs. rating 5000 6000

0.1 back 1 0.05 0

0.05 s eat 1

1000

2000

3000

4000

5000

6000

1000

2000

3000

4000

5000

6000

0.05 back 2

0.04 s eat 2 0.02

1000

2000

3000

4000

5000

6000

1000

2000

3000

4000

5000

6000

0.1 back 3 0.05 0

0.1 s eat 3 0.05 0

1000

2000

3000

4000

5000

6000

1000

2000

3000

4000

5000

6000

0.1 back 4 0.05 0

0.1 s eat 4 0.05 0

0 x 10

1000
-3

2000

3000

4000

5000

6000

0 x 10

1000
-5

2000

3000

4000

5000

6000

4 2 0

10 back 5 8 6

s eat 5

1000

2000

3000

4000

5000

6000

1000

2000

3000

4000

5000

6000

(b)
5 4 3 2 1 0 0 1000 2000 3000 Time, s 4000 5000 6000 ey e mov ement duration, s observ er rating

The total durations of analysed episodes varied from 1.5 to 6 minutes. Across those intervals the peak to peak variations were averaged over 30 seconds. The time direction was inverted so all analysed episodes started from significant drowsiness and ended with a less drowsy state with variable durations of those states depending on the timing of a preceding episode of significant drowsiness. The hypothesis of association between transition to significant drowsiness and reduction in the movement sensor signal variations was tested by means of fitting linear regression models with time as a covariate. These models are fitted for all data as well as for individual subjects and movement sensors. Potential correlation between measurements taken at different time points was accounted for by means of fitting the general estimating equations (GEE) regression models [10]. Stata statistical software was used for fitting those regression models.

5 4 3 2 1 0 0 1000 2000 3000 4000

3. Results
5000 6000

(c)
Video Score 5 4 3 2 1 0 -1 0 500 1000 1500 2000 2500 3000 3500

3.1. Statistical analysis of association between transition to significant drowsiness and reduction in the movement sensor signal variations on the back of the seat
Typical patterns of change in the averaged peak to peak values of the movement signal with increase in the time interval before the transition to significant drowsiness are presented in Figure 4. The presented values are normalised with respect to the value at the beginning of significant drowsiness. The pattern of increase was found to be quite dominant being present at 85 out of 115 episodes.
4 3 2 1 4 3 2 1 2.5 2 0 2 4 6 8 10 12 s eat 2 linear regress ion 0 2 4 6 8 10 12 s eat 1 linear regress ion

percentage
1.5

0.5

500

1000

1500

2000

2500

3000

3500

Figure 2. Example of likely association between increase in the observer drowsiness rating and (a) reduction in the peak to peak values of some of the seat movement signals, (b) increase in the eye movement duration, and (c) and increase in the relative power of the EEG alpha band.

Observer drowsiness rating with transitions to significant drowsiness 5 4.5 4 3.5

1.5 1 1.5 1 0 2 4 6 8

s eat 3 linear regres s ion 10 12

s eat 4 linear regress ion 0.5 0 2 4 6 8 10 12

3 2.5

2 s eat 5 linear regress ion

1.5

2 1.5 1 0.5 0 -0.5 0 100 200 300 Time, s 400 500 600
1 0 2 4 6 8 10 number of 30 s ec onds interval prior to s iginificant drowsines s 12

Figure 3. Example of selection of episodes of transition to drowsiness

Figure 4. Example of change in the averaged peak to peak values of the movement signal with increase in the time interval before the transition to significant drowsiness

The 2nd International Conference on Wireless Broadband and Ultra Wideband Communications (AusWireless 2007) 0-7695-2842-2/07 $25.00 2007

The effect of increase in the seat movement magnitude with the time interval prior to a transition to significant drowsiness was investigated by means of fitting regression models with the interval number as a covariate. There are three possibilities for correlation between analysed data between measurements at different time positions within the same transition episode for the given movement sensor, between measurements of different sensors at the same time and across all measurements for the given subject. Potential correlation was taken into account by means of fitting the general estimating equations (GEE) regression models with different correlation matrix models. The results of fitting these models for all combined data are presented in Table 2 (a). It is evident that there is statistically significant increase of movement magnitude with time for all considered correlation structures as well as for a simple linear regression. Taking correlation into account increased the standard error of the effect estimate. The results of fitting regression models for the first 10 subjects are presented in Table 2(b). The exchangeable correlation structure was used and correlation between all measurements within the transition (called episode in the Table) and only between the measurements for the given sensor within the transition (called sensor x episode) was considered. For the linear regression model 19 out of 22 subjects were found to have increase in the movement magnitude with time and for 12 of these subjects the association was significant at 5% significance level. For the GEE model with correlation between sensors and time positions 16 out of 18 subjects were found to have increase in the movement magnitude with time and for 9 of these subjects the association was significant. Finally for the GEE model with correlation between time positions 20 out of 22 subjects were found to have increase in the movement magnitude and for 11 of these subjects the association was significant. The results of fitting regression models for five(5) individual sensors are presented in Table 2(c). The exchangeable correlation structure was used and correlation between all measurements for the subject and only between the measurements within the given transition (called episode) was considered. For all models the increase in the movement magnitude with time was found statistically significant.

the back of the car seat. This finding can be considered as the first step in deriving the accurate and reliable algorithm for detection of driver drowsiness. The logic of the algorithm development can be viewed as a sequence of fitting the appropriate statistical models while determining suitable methods of processing different physiological indicator signals, combining those parameters in an optimum way and expanding the temporal scope of these models in the process. The first step would comprise investigation of significance of time course of changes in functions of individual physiological signals during the episodes of transitions to the dangerous drowsiness states. The signals and respective processing methods that are found to have statistically significant variations over the transition episodes could be selected as potential candidates for being the algorithm components. The second step of algorithm development would comprise determination of the combinations of individual drowsiness measures that are most strongly associated with the odds (or log odds) of a state of dangerous drowsiness or a number of different drowsiness stages based on the observations from the episodes of transitions to drowsiness. Finally a number of selected combinations of individual drowsiness measures could be validated across the complete set of recorded observations including determination of sensitivity and specificity as well as ROC curves. The important aspects of development of the drowsiness detection system are its practicality, robustness and non-invasiveness. While the discussed approach to the algorithm development is capable of integrating and comparing different combinations of physiological measures those that are minimally invasive will be given priority. This consideration was the reason behind the focus on analysing properties of the seat movement sensors as presented in this study.

5. References
[1] S.M. Belz, G.S. Robinson, and J.G. Casali, Temporal separation and self-rating of alertness as indicators of driver fatigue in commercial motor vehicle operators, Hum Factors, 2004, 46(1), pp. 154-169. [2] Burton, D., Vigilance Monitoring system, in A.P. Office (ed.)no AU 200022710B2, Australia, 2004. [3] S.K. Lal and A. Craig, A critical review of the psychophysiology of driver fatigue, Biol Psychol, 2001, 55(3), pp. 173-194. [4] P.P. Caffier, U. Erdmann, and P. Ullsperger, Experimental evaluation of eye-blink parameters as a drowsiness measure, Eur J Appl Physiol, 2003, 89(3-4), pp. 319-325.

4. Discussion
It was established in this study that during the transitions to significant drowsiness states there is statistically significant reduction in a measure of variation of the piezofilm movement sensors located in

The 2nd International Conference on Wireless Broadband and Ultra Wideband Communications (AusWireless 2007) 0-7695-2842-2/07 $25.00 2007

Table 2. Change in the averaged peak to peak movement sensor signal per 30 seconds prior to the transition to significant drowsiness for different correlation assumptions for (a) the combined data, (b) the individual participants, and (c) the individual movement sensors. (a)
Regression model Linear regression GEE, independence, sensor x episode GEE, independence, episode GEE, independence, subject GEE, exchangeable, sensor x episode GEE, exchangeable, episode GEE, exchangeable, subject GEE, autoregressive, sensor x episode GEE, unstructured, sensor x episode Regression coefficient 0.047 ([0.036; 0.058], p < 0.001) 0.047 ([0.032; 0.062], p < 0.001) 0.047 ([0.024; 0.069], p < 0.001) 0.047 ([0.027; 0.067], p < 0.001) 0.044 ([0.031; 0.057], p < 0.001) 0.044 ([0.026; 0.062], p < 0.001) 0.046 ([0.029; 0.063], p < 0.001) 0.051 ([0.036; 0.065], p < 0.001) 0.039 ([0.024; 0.055], p < 0.001)

Linear regression 1 0.038 ([0.026;0.050] p < 0.001) 0.065 ([0.038;0.092] p < 0.001) 0.051 ([0.030;0.072] p < 0.001) 0.036 ([0.021;0.052] p < 0.001) 0.043 ([0.007;0.080] p < 0.001)

GEE, exchangeable, episode 0.035 ([0.019;0.052] p < 0.001) 0.055 ([0.023;0.087] p < 0.001) 0.045 ([0.022;0.069] p < 0.001) 0.033 ([0.013;0.053] p = 0.001) 0.058 ([0.005;0.111] p < 0.001)

GEE, exchangeable, sensor x episode 0.037 ([0.020;0.053] p < 0.001) 0.057 ([0.023;0.091] p = 0.001) 0.046 ([0.024;0.069] p < 0.001) 0.034 ([0.018;0.051] p < 0.001) 0.044 ([0.001;0.089] p = 0.046)

(b)
Linear regression 1 2 3 4 5 6 7 8 9 10 0.014 ([-0.011; 0.038], p = 0.267) -0.002 ([-0.028; 0.024], p = 0.855) 0.041 ([0.011; 0.072], p = 0.008) 0.000 ( [-0.010; 0.009], p = 0.945) 0.041 ( [0.021; 0.060], p < 0.001) 0.063 ([0.013; 0.113], p = 0.021) 0.085 ( [0.057; 0.112], p < 0.001) 0.038 ([0.015; 0.062], p = 0.001) 0.229 ([0.142; 0.315], p < 0.001) 0.016 ([-0.029; 0.061], p = 0.495) GEE, exchangeable, episode 0.003 ([-0.028;0.035], p = 0.842) 0.000 ([-0.017;0.015], p = 0.932) 0.035 ([0.002; 0.068], p = 0.039) -0.002 ([-0.013;0.009], p = 0.739) 0.047 ([0.005; 0.089], p = 0.027) 0.076 ([0.006; 0.147], p = 0.034) 0.084 ([0.024; 0.144], p = 0.006) 0.049 ([-0.004;0.101], p = 0.070) 0.178 ([0.012; 0.344], p = 0.036) 0.022 ([-0.039;0.083], p = 0.473) GEE, exchangeable sensor x episode 0.003 ([-0.015; 0.022], p = 0.711) -0.002 ([-0.018; 0.013], p = 0.785) 0.025 ([-0.005; 0.055], p = 0.101) 0.000 ([-0.014; 0.015], p = 0.982) 0.047 ( [0.024; 0.00], p < 0.001) 0.074 ( [0.023; 0.126], p = 0.005) 0.086 ([0.043; 0.128], p < 0.001) 0.046 ([0.012; 0.080], p = 0.008) 0.180 ([-0.017; 0.377], p = 0.074) 0.020 ([-0.025; 0.066], p = 0.381)

[5] J. Connor, P. Norton, S. Ameratunga, E. Robinson, L. Civil, R. Dunn, J. Bailey, and R. Jackson, Driver sleepiness and risk of serious injury to car occupants: population based case control study. Br. Med. J. 2002, 324 (7346), pp. 1125. [6] D. F. Dinges, and Grace, R. PERCLOS: A valid psychophysiological measure of alertness as assessed by psychomotor vigilance, Federal Highway Administ., Office of Motor Carriers. Report No. FHWA-MCRT-98-006, 1998. [7] T. Jung, S. Makeig, M. Stensmo, and T. J. Sejnowski, Estimating alertness from EEG power spectrum. IEEE Trans. Biomed. Eng. 1997, 44(1), pp. 60-69 [8] S.K. Lal and A. Craig, Driver fatigue: electroencephalography and psychological assessment, Psychophysiology, 2002, 39(3), pp. 313-321. [9] S.K. Lal and A. Craig, Reproducibility of the spectral components of the electroencephalogram during driver fatigue, Int J Psychophysiol, 2005, 55(2), pp. 137-143. [10] K.Y. Liang and S. L. Zeger, Regression analysis for correlated data, Ann Review of Public Health 1993, 14, pp.43-68. [11] Rochford, P.D., Ruehland, W., Cherchward, T. and Pierce, R. J. Evaluation of automated versus manual scoring of polysomnographs on sleep disordered breathing. Australian Sleep Association Meeting, 2006. [12] F. Sagberg, Road accidents caused by drivers falling asleep, Accident Analy Prevention, 1999, 31(6), pp. 639649. [13] Santamaria, J. and Chiappa, K.H. The EEG of Drowsiness, Demos Publications, New York, 1987. [14] J. A. Stern, D. Boyer, and D. Schroeder, Blink rate: A possible measure of fatigue, Human Factors, 1994, 36(2), pp. 285-297. [15] W.W. Wierwille, and L.A. Ellsworth, Evaluation of driver drowsiness by trained raters, Accident Analysis & Prevention, 1994, 26(5), pp. 571-581.

(c)

The 2nd International Conference on Wireless Broadband and Ultra Wideband Communications (AusWireless 2007) 0-7695-2842-2/07 $25.00 2007