Abstract
Advanced driver-assist systems (ADAS) enable drivers to relinquish operational control of the vehicle to automation for part of the total drive. While these features are engaged, drivers have an increased risk of losing awareness of their environment. Current ADAS broadly utilizes hands-on-the-wheel or eyes-on-the-road driver supervision strategies to continually monitor steering-wheel torque and drivers’ head and eye positions to ensure driver attention. The current work examines the effect of hands-on-the-wheel and eyes-on-the-road driver supervision strategies on change detection, mind wandering, and gaze behavior in a low-fidelity semi-autonomous driving task.
Keywords
Introduction
Advanced driver-assist systems (ADAS), often a combination of lane assist, autosteer, speed control, automatic emergency braking, etc., have enhanced driving by relieving drivers of part of the driving task for part of the total drive. These ADAS are classified as SAE Level 2 automation as they require drivers to monitor the system at all times (SAE, 2016), even while the ADAS features are engaged. To ensure that drivers maintain their attention on the road, the National Transportation Safety Board recommends that ADAS be used in conjunction with driver-state monitoring systems as a safety-critical functionality (NTSB, 2021).
Popular driver monitoring systems require drivers to keep their hands on the wheel or eyes-on-the road while ADAS is engaged. Tesla, for example, requires drivers to have their hands on the wheel while Autopilot is engaged. The vehicle senses torque on the steering wheel as a proxy for driver engagement. If the vehicle does not detect optimal torque, it sends auditory and visual signals alerting drivers to direct their attention to the driving task (Tesla, Inc., 2021). Recently, Tesla also implemented a cabin camera to measure driver inattentiveness (Tesla, Inc., 2021). Cadillac employs head and eye tracking to ensure drivers have their eyes on the road while their Super Cruise feature is engaged. Like Tesla, Super Cruise combines auditory and visual alerts to re-engage drivers with the system (Cadillac, 2020). While the use of ADAS has been shown to reduce situation awareness and increase off-road glances (Hungund, Pai & Pradhan, 2021), the effectiveness of the hands-on-the-wheel and eyes-on-the-road driver supervision strategies on driver attention has not been well documented in the literature.
Driver attention is typically assessed using a combination of subjective measures, driving behavior, and eye tracking. Subjective measures often include SAGAT (Endsley, 1988), SART (Taylor, 1990), and NASA-TLX (Hart & Staveland, 1988) to assess situational awareness or workload. Measures of semi-autonomous driving behavior include takeover quality and response time, post-takeover lane position, vehicle speed, SD of lane deviation, and brake pedal force (Fisher et al., 2020). Eye tracking metrics involve heads-up time, off-road glances, pupil diameter, SD of fixation position, length and duration of fixations, etc. (Liang et al., 2021). Although less commonly used in the driving literature, gaze entropy has previously been utilized in aviation and nuclear power plants as a measure of distribution of fixations across specified areas of interest (AOI). (Lanini-Maggi et.al., 2021, Lee et.al., 2021). There are three commonly used measures of entropy. Stationary gaze entropy (SGE) is a metric of distribution of fixations across AOIs. Higher SGE implies that viewers' fixations were equally distributed across all AOIs (Lanini-Maggie et al., 2021). Dwell time entropy (DTE) is similar to SGE but assesses the amount of time in each AOI. Higher entropy indicates an equal distribution of time spent looking at each AOI (Lee et.al., 2021). Finally, gaze transition entropy (GTE) considers the transition of viewers' gaze between AOIs. Lower GTE implies more consistent scan patterns, while higher GTE suggests less predictable ones (Lanini-Maggi et.al., 2021; Krejtz et al., 2015). Combined, these entropy metrics aid in understanding the distribution of attention across AOIs and visual scanning behavior that can complement other commonly used eye-tracking metrics.
The current study served two purposes. First, we compared hands-on-the-wheel and eyes-on-the-road driver supervision strategies during a 15-minute low-fidelity semi-autonomous driving task. We assessed change detection performance, mind wandering, and visual scanning behavior. The second aim was determining which eye-tracking measures were sensitive to change detection and mind-wandering behavior differences.
Methods
Participant & Design
Eighty participants (20 women and 3 non-binary) were recruited from the psychology subject pool at a midwestern university. Participants were 18-27 years old (M = 19.73 and SD = 1.56) and had valid driver’s licenses. The experiment employed a between-subjects design. Participants were randomly assigned to one of the two driver supervision strategies: (a) hands-on-the-wheel or (b) eyes-on-the-road, with 40 in each group. T-tests revealed no significant differences between the two driver supervision strategies for gender, age, years of driving experience, and use of ADAS features.
Task
Participants completed a 15-minute, low-fidelity semi-autonomous driving task. Participants were seated in front of a monitor with a steering wheel and foot pedal. During the simulated drive, participants viewed a dashcam video taken from the driver’s perspective during a highway drive. A change detection task was embedded approximately halfway through the drive. As the vehicle passed out from an underpass, there was a naturally occurring flash of light, and the video was mirrored, effectively giving the appearance of driving on the left-hand side of the road. The road signs during this segment had a backward text. After 22 seconds, the video was mirrored again (Fig 1). Mind wandering was probed thrice using chimes at 4 mins, 8:13 mins, and 11:25 mins (Fig 1). At approximately 15 mins, a beep sounded, signaling the driver to take over by placing their hands on the wheel and tapping the foot pedal. This ended the drive.

Experimental Task.
Materials
Stimuli
The stimulus was a video (Wind Walk Travel Videos, 2020) taken from the driver’s perspective of a highway drive along I-110 connecting San Pedro to downtown Los Angeles in California. The video was presented with a resolution of 1920x1080 pixels.
Tobii Pro Eye tracker
The Tobii Pro X3-120 remote desktop-mounted eye tracker collected data during this study. The eye tracker, consisting of near-infrared light illuminators and sensors, has a sampling rate of 120 Hz. In addition, the Tobii I-VT fixation filter was applied at data export with default settings.
Desktop Set-up
Stimuli were presented on a 24-inch monitor mounted with the remote eye tracker. A Logitech G920 driving force steering wheel was mounted on the table before the monitor. A USB single-switch foot pedal was used as a brake pedal.
Mind wandering prompts
Thought probes assessed mind wandering (Robinson, Miller & Unsworth, 2019). Upon hearing a chime, participants were asked to speak the number associated with one of the three options that best represented their thoughts at the time of the chime: (1) I am focused on the driving task, (2) My mind is wandering, (3) My mind is blank. A label indicating these response options was provided just below the monitor.
Post Drive Questionnaire
The 11-question post-drive questionnaire was adapted from SAGAT (Endsley, 1988) and assessed level 1 SA. Additionally, two questions specifically targeted the change detection portion of the drive. Following the protocol of Simons and Chabris (1999), participants were first asked if they noticed anything unusual during their drive. Then, a follow-up multiple-response question asked whether participants noticed a simulation glitch, their car switching to the other side of the road, speeding vehicles, backward road signs, an accident, and/or a fire at the side of the road. The responses accepted as correct detections included the simulation glitch, their car switching to the other side of the road, and the backward road signs.
Procedure
Upon obtaining informed consent, participants were randomly assigned to one of the two driver supervision strategies. They then completed a practice session to get acquainted with the driver supervision strategy, different sounds, and tasks involved in the experiment. Participants in the hands-on-the-wheel supervision strategy were instructed to keep their hands on the steering wheel at all times; those in the eyes-on-the-road condition were asked to keep their eyes on the road, with their hands in a neutral position resting on their thighs. Participants in both conditions were instructed to take control of the vehicle upon hearing a beep by placing their hands on the steering wheel and pressing the foot pedal. Following the practice session, eye-tracker calibration took place, and the drive began. At the end of the driving task, participants completed a post-drive questionnaire. The entire session lasted at most 35 minutes.
Eye Tracking Analysis
Eye movement data were collected for the entirety of the drive time (i.e., 15 mins). We analyzed five eye tracking metrics, including mean saccade amplitude, mean fixation duration, stationary gaze entropy, dwell time entropy, and gaze transition entropy. The visual environment was divided into nine equal areas of interest (AOI) minus the sky to facilitate the entropy analysis. Each AOI was approximately 642x270 pixels. The location of fixations, the number of fixations, and the fixation duration within each AOI were used to calculate entropy, as described below.
Entropy Analysis
SGE is obtained by applying Shannon’s entropy equation to the probability of fixations in each AOI (Eq. 1). Similarly, DTE utilizes the probability of time spent in each AOI.
where
Gaze transition entropy (GTE), known as Markov’s entropy, utilizes Markov’s chain model and considers the eye movement pattern between AOIs. GTE estimates the uncertainty in the following fixation location, given the current fixation location. The following equation gives the first-order Markov transitions for fixations (Krejtz, et al., 2015):
Where
For interpretation, relative entropies are obtained by dividing the actual entropy of the system by the maximum entropy
Results
All statistical analysis was performed using R version 1.1.456. This paper focuses specifically on the drive's change detection (CD) segment, comprised of the three epochs shown in red in Figure 1. Epoch 1 included the 22 seconds prior to the change. Epoch 2 included the 22 seconds during which the driving scene was reversed. Note that the mind-wandering prompt sounded immediately after the conclusion of Epoch 2 to assess drivers’ perceptions of their own focus of attention during the change detection epoch. Epoch 3 included the 22 seconds directly following this mind-wandering prompt.
Self-report measures
Change Detection Analysis (Fig 2A)
Participants in the hands-on-the-wheel condition (58.1%) were numerically more likely to report detecting the change in driving environment compared to those in the eyes-on-the-road condition (41.9%); however, a chi-square test of independence using Yates’ continuity correction revealed that this difference was not significant χ 2 (1, n = 79) = 0.69, p = 0.405 with an effect size of Cramer’s V = 0.119.

Self-report measures.
Mind Wandering Analysis (Fig 2B)
The mind wandering prompt required participants to report one out of the following three options – (1) I am focused on the driving task, (2) My mind is wandering, (3) My mind is blank. Due to a small number (N = 4) of participants reporting option 3, it was combined with option 2. A chi-square test of independence revealed that the proportion of participants who reported mind wandering during the change detection segment was significantly higher in the eyes-on-the-road condition (65.5%) compared to the hands-on-the wheel condition (34.5%); χ 2 (1, n = 79) = 3.81, p = 0.05 with an effect size of Cramer’s V = 0.246.
A follow-up chi-square test of independence analyzed the relationship between CD and mind-wandering (Fig2C). Participants who failed to detect the change were more likely to report mind wandering (79.3%) than those who detected the change (50%), χ 2 (1, n = 79) = 5.44, p = 0.02 with an effect size of Cramer’s V = 0.289. Framed differently, participants who reported being on-task were equally likely to report detecting the change, while only 15% of those who reported mind wandering reported detecting the change.
Eye Tracking Measures
A 2x3 mixed factorial ANOVA assessed the effects of condition as a between-subjects factor and epoch as a within-subjects factor on all eye tracking measures. Results are presented in Table 1. Cohen’s f is reported for effect sizes. Across all measures, there was no significant effect of condition nor significant interactions between condition or epoch.
ANOVA results for the effect of Condition.
Mean saccade amplitude and Mean Fixation Duration (Fig3A-B)
The epoch had a significant main effect on both mean saccade amplitude and mean fixation duration (Table 1.) Post-hoc analysis using Tukey’s HSD revealed that the mean saccade amplitude significantly reduced from the pre-CD epoch (M = 2.93, SD = 1.58 degrees) to the CD epoch (M = 2.43, SD = 1 degree). Similarly, the mean fixation duration was significantly reduced from the pre-CD epoch (M = 316.89, SD = 154.03 ms) to the CD epoch (M = 238, SD = 80.63 ms).

Eye Tracking Metrics for Condition.
Entropy Measures (Fig3C-E)
While DTE did not significantly differ between epochs, SGE and GTE did. Post-hoc analysis using Tukey’s HSD indicated no differences between the epochs for SGE. However, GTE significantly increased from the CD epoch (M = 0.02, SD = 0.12 bits) to the post-CD epoch (M = 0.09, SD = 0.2 bits). The increase in GTE indicates that the scan patterns were significantly more chaotic in the post-CD epoch compared to the change detection epoch.
The significant epoch effects for saccade amplitude, fixation duration, and the two entropy measures likely reflect a response to the visual environment changes accompanying the CD manipulation.
Change Detection Analysis (Fig4A-E)
To examine the effect of noticing the change on eye tracking measures, we performed a series of 2x2 mixed factorial ANOVAs with CD (people that reported detecting the change compared to those that didn’t) as a between-subjects factor and epoch (pre-CD and CD epoch) as a within-subjects factor. Results are presented in Table 2. Eye tracking metrics did not differ between those who noticed the change and those who did not.

Eye Tracking Metrics for Change Detection.
ANOVA results for the effect of Change Detection.
Mind Wandering Analysis (Fig 5A-E)
Likewise, to examine the effect of mind wandering on eye tracking measures, we performed a series of 2x2 mixed factorial ANOVAs with mind wandering (people who reported mind wandering compared to those that reported being on task) as a between-subjects factor and epoch (change detection and post-change epoch) as a within-subjects factor. Results are presented in Table 3. There was no effect of mind wandering across all measures. Effects of epoch on mean saccade amplitude and mean fixation duration are consistent with those reported as part of the condition analysis in Table 1.

Eye Tracking Metrics for Mind Wandering.
ANOVA results for the effect of Mind Wandering.
Mean Saccade Amplitude and mean fixation duration (Fig5A-B)
The mean saccade amplitude significantly increased from the CD epoch (M = 2.43, SD = 1 degree) to the post-CD epoch (M = 2.86, SD = .17 degrees). Similarly, the mean fixation duration significantly increased from the CD epoch (M = 238, SD = 80.63 ms) to the post-CD epoch (M = 257.51, SD = 88.52 ms). The increase in these two metrics indicates that participants likely returned to passively monitoring the driving environment during the post-CD epoch.
Entropy Measures (Fig5C-E)
SGE significantly decreased from the CD epoch (M = 0.31, SD = 0.12 bits) to the post-CD epoch (M = 0.28, SD = 0.13 bits). There was a main effect of epoch on GTE. However, it was qualified by an interaction with mind wandering. Among those who reported mind wandering, there was no difference in GTE between the CD and the post-CD epoch. GTE significantly increased in the post-CD epoch for those that reported being on task, indicating more chaotic and less predictable scan patterns t(56.58) = 2.77, p = 0.007.
Discussion
ADAS technology requires drivers to monitor their environment actively, changing their role to primarily supervisory. ADAS, however, has previously been shown to reduce driver situation awareness and decrease their attention to the roadway. This, in turn, may negatively affect their ability to resume operational control of the vehicle when required safely. This study compared two common driver supervision strategies, hands-on-the-wheel, and eyes-on-the-road, in a low-fidelity semi-autonomous driving task.
Results showed a slight preference for the hands-on-the-wheel driver supervision strategy. While those in the hands-on-the-wheel condition were less likely to report mind wandering during the CD segment, they were also numerically more likely to detect the change, although not significantly so. In fact, neither supervision strategy strongly supported CD behavior. Across both conditions, only 39% of participants noticed when the driving environment was mirrored for 22 seconds of the drive.
Participants assigned to the eyes-on-the-road supervision strategy were significantly more likely to mind wander. Additionally, those that reported being on task were equally likely to detect the change. Of those who reported mind wandering, only 20% of participants detected the change. Notably, despite the dramatic differences in CD performance when mind wandering, these differences did not emerge in the eye tracking metrics. While these metrics did not produce apparent candidates for predicting change detection or mind wandering, it is worth noting that those who detected the change had a numerically higher SGE and DTE, meaning they were scanning the visual environment more than those that did not detect the change. Although not reported here, we analyzed the standard deviation of horizontal and vertical fixation position and percent dwell time in the periphery. These metrics also did not vary as a function of the two driver supervision strategies, CD performance and MW behavior. Future work should explore the utility of these metrics for assessing driver engagement.
While our results point to some potential benefits for hands-on-the-wheel strategy over the other, we did not examine takeover quality and response time or driving performance following the takeover request. Additionally, the CD task in this experiment was not naturalistic as it involved a manipulation that could not happen in the real world. The lack of interaction and feedback from the simulation may also be responsible for promoting MW behavior. In our future work, we plan to repeat the current study using a CD manipulation tied to the information about the status of the automation. Additionally, we plan to translate this study into a medium-fidelity driving simulator to examine the effect of the hands-on-the-wheel and eyes-on-the-road driver supervision strategies on takeover readiness, performance, and quality.
