With a Little Help From My Car: Sharing Automated Vehicle’s Situation Awareness Reduces Driver-Initiated Automation Disengagement Without Delaying Takeover Response Time

What to Share: Vehicle Situation Awareness

In partially automated driving, both driver and automation share responsibility for driving and monitoring. Maintaining appropriate SA throughout each trip is crucial. SA is defined as “the perception of elements in the environment within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future” (Endsley, 1995, p. 36). For a proper use of driving automation, drivers need to assess system capabilities and boundaries, thus react accordingly such as calibrating trust based on system capabilities, maintaining readiness to respond when intervention is requested, and executing smooth transitions between automated and manual control. For this reason, drivers need information about what the AV system senses and intends, such as planned trajectory, lane changes, and speed adjustments (i.e., vehicle’s SA). The vehicle, on the other hand, needs to detect how the driver is monitoring the environment over time (i.e., drivers’ SA) to estimate the driver’s state (Figure 1). This model in Figure 1 guides our approach to sharing vehicle’s situation awareness (VSA) with the driver. While this model illustrates the complete bidirectional SA framework in human–automation interaction, this study focuses specifically on one direction of this relationship: how sharing vehicle SA with drivers affects their behavior. Following Endsley’s three levels of SA, we identified corresponding VSA elements that support each level: (a) perception—information about objects the vehicle detects, (b) comprehension—the vehicle’s understanding of the situation with confidence level, and (c) projection—the vehicle’s and other road users’ predicted trajectory and movement intentions. By sharing these elements, we aim to improve the driver’s SA of both the environment and the automation system. For the driver, this additional level of SA (i.e., SA of driving automation in Figure 1) can make the driver’s SA more complex, but effectively sharing VSA information with the driver can support their interaction with the system, especially assist successful TOCs in complex situations (Lee et al., 2022).

OPEN IN VIEWER

Figure 1.

During driving a partially automated vehicle, both the driver and the vehicle maintain and update situation awareness (SA). The driver needs SA of both the environment and the driving automation, and the vehicle needs SA of both the environment and the driver through driver monitoring (modified from Lee et al., 2023).

Research has explored sharing automation information in both SAE L2 and L3 systems. Colley et al. (2021) found that adding detailed information during take-over requests did not improve drivers’ SA, suggesting that more information does not necessarily enhance understanding during transitions. Conversely, Swain et al. (2024) showed that displaying the vehicle’s planned maneuvers and perceived objects improved perceived usefulness and reduced stress in SAE L3 systems. However, a gap remains in understanding its effects on driver behavior in complex urban environments. Despite different driver responsibilities between SAE L2 and L3 systems, effective communication of vehicle perception and intentions is crucial across automation levels. Research has focused primarily on highways, neglecting urban driving’s unique challenges where driver understanding is most needed, especially during TOCs in complex settings such as intersections and parking zones (Du et al., 2024; Stapel et al., 2022).

How to Share: Augmented Reality Head-Up Display

The design and function of the Human-Machine Interface (HMI) is crucial to support drivers’ SA (Carsten & Martens, 2019). Equally important is aligning information needs with effective HMIs, considering their characteristics. Displaying VSA information to the driver in complex urban environments can be challenging in terms of driver workload and distraction. In urban environments, drivers already experience increased visual and cognitive workload due to dense and mixed traffic, signage, and complex road layouts. Additional VSA information could potentially contribute to visual clutter, divided attention, and information overload if not carefully designed. Research shows that poorly implemented information displays can increase driver’s off-road glance time and delay critical responses (Koo et al., 2015). Augmented reality head-up displays (AR HUDs) have characteristics to mitigate the challenges. AR HUDs use a windshield (or a transparent screen) as a display where users can see the actual environment and augmented digital contents simultaneously. This allows drivers receive information without glancing away from the road and provides contextual information. Research shows well-designed AR HUDs can reduce workload compared to dashboard displays (Bengler et al., 2015; Feierle et al., 2019; Israel et al., 2010), decrease visual distraction (Campbell et al., 2016; Damböck et al., 2012), and improve response by directing attention forward (Feierle et al., 2019; Rusch et al., 2013). These studies presented navigation cues, hazard alerts, and vehicle dynamics information. For example, Feierle et al. (2019) used path markers and collision object highlighting, while Rusch et al. (2013) highlighted roadside hazards to direct driver attention.

Research Questions

We designed AR HUDs to share VSA with drivers and tested their effect on driver-automation interaction in complex urban environments. We selected AR HUDs as the display medium based on their ability to provide spatially registered information that maps directly to the real-world environment visible through the windshield. This conformal display characteristic is particularly advantageous in complex urban environments where drivers must integrate automation information with dynamic visual scenes containing multiple road users, intersections, and potential conflicts. We hypothesized that VSA information helps calibrate driver reliance on automation and facilitates TOCs. We focused on system-to-driver TOCs, classifying them as: (a) “takeover” (system-initiated) or (b) “driver-initiated override” (driver-initiated), following our previous taxonomy (J. Lee, Lee, et al., 2022) that aligns with existing research (Colley et al., 2021; Lu et al., 2016; Maggi et al., 2022). We further categorized takeovers as “partial” (accelerator/steering activation) or “full” (brake activation). Partial takeover occurs when drivers temporarily provide input to either longitudinal control (acceleration) or lateral control (steering) while the automation continues to handle the other aspect and returning to full SAE Level 2 automation when input ceased. Full takeover occurs when drivers press the brake pedal, resulting in complete manual control. In this paper, we considered driver-initiated overrides as “unnecessary” when they occurred before the automation system reached its capability limits, as in this study, the conflict-causing event had not yet developed into imminent threats or clear behavioral patterns requiring immediate intervention. From a trust calibration perspective, our goal was to promote appropriate reliance on automation when it is capable while maintaining readiness to respond when needed.

We developed three research questions (RQ) to examine how sharing VSA information affects driver–automation interaction. First, sharing VSA information should help drivers better understand system capabilities and boundaries, which could improve trust calibration and increase appropriate reliance on automation when the system is capable (RQ1). Second, while sharing VSA information may benefit driver understanding, it is important to verify that additional information does not impose cognitive load that could delay drivers’ response when intervention is needed (RQ2). Third, when drivers have multiple response options (partial vs. full takeover), VSA information may affect not only when drivers respond but also how they respond in terms of takeover type and response quality (RQ3). The RQs are listed below.

RQ 1. Does sharing VSA information reduce unnecessary driver-initiated overrides in urban environments?

To answer these questions, we designed a set of AR HUDs varying levels of VSA information, developed two different urban driving scenarios, and tested the effect of sharing VSA through the AR HUD on drivers’ TOC behaviors in the urban environment using a driving simulator. We measured driver reliance through frequency of driver-initiated overrides. We assumed fewer overrides indicate appropriate reliance when automation reliability is high and the system is capable. We also evaluated response time to takeover requests and analyzed takeover patterns (partial vs. full) and response magnitude (i.e., braking intensity) to understand how VSA information impacts takeover behaviors.

Participants

Sixty-five participants were recruited, but three were excluded due to technical issues, leaving data from 62 participants (32 females; mean age = 41.7, SD = 14.9, age range = 21–64 years). Participants were recruited from a midsized Midwest town, had valid U.S. driver’s licenses (≥2 years), were at least 20 years old, and in good health. Participants had varying levels of prior experience with automated driving features, as this was not used as a screening criterion. They were compensated $100 per hour for approximately 1 h 45 min of participation.

Apparatus

We used a General Motors Research Driving Simulator, providing a 360° × 30° roadway view. The simulated environment was a four-lane urban street (speed limit 25 mph). Some segments of the road allowed for SAE Level 2 automated driving (i.e., the vehicle handles both longitudinal and lateral controls) by the simulated vehicle, while other segments required the vehicle to issue a takeover request. The simulated vehicle represented a hypothetical SAE Level 2 automated driving system capable of pertaining in urban environments. The simulated vehicle had a temporal control sharing function, allowing the driver to steer or accelerate without fully disengaging from the automation. Additionally, the vehicle automatically resumed SAE Level 2 automated driving when driver input ceased. A Smart Eye Pro remote eye tracker with two dashboard-mounted cameras (sampling frequency: 60 Hz) recorded drivers’ visual attention throughout the drive.

Experiment Design

The study used a mixed design with AR HUD condition (Baseline, Box, Telltale, Chevron, and Dynamic ODD) as a between-subjects factor and driving situation (parking lane and intersection) as a within-subjects factor. Participants were randomly assigned to an AR HUD condition and completed two variations of each driving situation, resulting in four test trials per participant.

AR HUD Conditions

We identified three types of VSA information: (a) perception (objects detected by the vehicle), (b) comprehension (vehicle confidence in understanding the situation), and (c) projection (predicted trajectory of ego and other road users). We conceptualized that the driving automation operates through continuous cycles of situation awareness-decision-action in real-time. Since this complete SA process is difficult to fully visualize moment-to-moment, we applied the SA framework to decide which aspects of VSA information could be selectively highlighted on the AR HUD.

Based on previous research, we selected specific visual features to convey different aspects of VSA (see Figure 2): box highlighting for perception (Feierle et al., 2019; Wintersberger et al., 2017), telltale lines for other vehicle trajectories (Ghori et al., 2018; Lindemann et al., 2018), chevron paths for ego vehicle trajectory (Feierle et al., 2019; Pfannmüller et al., 2015), and color-coded confidence indicators (Beller et al., 2013; Kunze et al., 2019). In our implementation, perception information shows what the automation detects in the environment through object highlighting, projection information displays the automation’s planned trajectory or predicted movements of other road users, and comprehension information represents the system’s assessment of environmental complexity and its capability to manage that complexity in the current driving situation. Unlike traditional sequential SA processing, our VSA components were not strictly hierarchical, with comprehension information displayed through confidence-coded trajectory elements rather than as separate environmental comprehension indicators.

OPEN IN VIEWER

Figure 2.

Examples of the four AR HUD conditions and VSA elements shared with the driver. VSA elements that are unique for each condition are highlighted with red dashed lines (note that the dashed red lines are added only for the manuscript readers and participants did not see them in the experiment).

The AR HUD conditions (Table 1) were manipulated additively to observe incremental effects of each VSA element, with one exception: the Telltale and Chevron conditions both provided perception and projection cues but differed in trajectory orientation (i.e., other vehicles vs. ego vehicle). The Dynamic ODD condition included all the features of the other conditions and the AV’s confidence level indicated by three colors for the chevron cues: green (AV maintained full control), yellow (temporarily decreased confidence but still capable, requiring increased driver attention), and red (AV could not maintain control, requiring immediate intervention). We chose this discrete representation to balance information richness with ease of comprehension in dynamic driving situations, though some research has explored more granular representations of uncertainty (e.g., Kunze et al., 2019).

OPEN IN VIEWER

Table 1.

AR HUD Conditions and VSA Elements Shared by Each Condition. The Baseline Condition Did Not Display Any VSA Cues.

	Perception Cue	Projection Cue	Comprehension Cue
Baseline
Box	Perceived objects on the road
Telltale	Perceived objects on the road	Projection of other vehicles’ trajectory
Chevron	Perceived objects on the road	Projection of ego vehicle’s trajectory
Dynamic ODD	Perceived objects on the road	Projection of ego vehicle’s trajectory	AV confidence level for driving

Driving Situations

We designed four urban driving scenarios, some involving takeover requests and others not, based on previous research (encountering a cyclist, a car merging from a parking lane, a car entering an intersection, and a pedestrian crossing; Feierle et al., 2019; Mahajan et al., 2021; Samuel et al., 2016). This paper analyzed two scenarios (parking lane, Figure 3; intersection, Figure 4), both involving takeover requests. These two scenarios were selected to examine how different urban driving context affect driver–automation interaction. The parking lane scenario features moving objects and potential lateral conflicts, while the intersection scenario involves stopped vehicles and potential path conflicts with crossing traffic. Importantly, in both scenarios, while conflict-causing vehicles (V1) may have been visible to drivers at distances where driver-initiated overrides typically occurred, the threats were not yet imminent (parking lane) or the vehicle intentions clear enough (intersection) to warrant intervention, supporting our classification of early overrides as unnecessary given the automation’s continued capability at those moments. The other two scenarios (cyclist and pedestrian crossing) did not involve takeover requests and are discussed elsewhere (J. Lee et al., 2023).

OPEN IN VIEWER

Figure 3.

Illustration of the parking lane situation. The ego vehicle (green car) entered a street parking zone where it issued a takeover request. Drivers needed to take control and avoid colliding with a car (V1) pulling out from a parking spot on the right side.

OPEN IN VIEWER

Figure 4.

Illustration of the intersection situation. A vehicle (V1) traveling southbound approached the intersection, stopped at a red light to make a left turn. The ego vehicle (green car) traveling from left to right issued a takeover request as it approached the intersection because the system couldn’t predict V1’s potential movement.

The driving automation had a staged warning system. It issued either a “red” alert or a two-step “amber” alert followed by a “red” alert, depending on the system’s predictability and proximity to its limit (see Figure 5). In the parking lane situation, only a red alert was issued, whereas in the intersection situation, an amber alert was initially issued, followed by a red alert if the driver had not taken control. If drivers did not respond to the takeover request, the simulated vehicle would slow down and come to a stop if required.

OPEN IN VIEWER

Figure 5.

Variations of driving automation icon in the instrument panel and steering light colors. Each sample represents one of four conditions: “red alert,” “amber alert,” “automation on,” and “automation off,” from top to bottom. Both red and amber alerts displayed the message “TAKE CONTROL OF YOUR VEHICLE,” but the takeover icon’s color and steering light color varied between the two alert types.

Procedure

Participants signed informed consent forms and completed demographic surveys before a practice drive in the simulator. Eye tracking calibration used a nine-point procedure before the main drive. Participants received instructions about the AV’s capabilities as “The automated vehicle (AV) … can steer, throttle, and brake by itself and respond to most of the events on the road (e.g., making an unprotected left turn, changing lanes, and avoiding a collision with a road user). However, the AV is not perfect, and it still requires you to monitor its behavior for cases when it can’t handle a road event. In such cases, the AV will alarm you to take vehicle control (takeover request, TOR).” During the practice drive, participants received hands-on training with manual driving, automation activation, and temporal control sharing by practicing steering, acceleration, and brake inputs, learning that automation would maintain partial control if they temporarily took control of one aspect and return to full SAE Level 2 automation when input ceased. Before the main drive, experimenters provided comprehensive reminders about automation activation, temporal control sharing, and takeover request signals. Participants assigned to AR HUD conditions also received instruction about their specific display elements. After driving, participants completed postexperiment surveys and debriefing.

Data Reduction

The AV state variable recorded drivers’ automation use in three levels: (a) SAE Level 2 (both lateral and longitudinal control), (b) Level 1 (either lateral or longitudinal), and (c) Level 0 (manual driving). Disengagement from Level 2 without a takeover request was categorized as a driver-initiated “override,” while disengagement after a takeover request was a system-initiated “takeover.” Takeovers were further classified as “full” (pressing brake pedal, changing to Level 0) or “partial” (steering or accelerating, changing to Level 1).

Response time was measured from the first alert (either the amber alert or the red alert) to the driver’s control input. For full takeovers, response magnitude (i.e., how hard drivers brake in response to takeover requests) was calculated using the maximum brake velocity within 1.5 s of the driver’s response. This measure and its cutoff were selected after comparing several alternatives, as it provided sufficient sensitivity to distinguish between different braking responses. We assumed that a lower maximum brake velocity indicates softer braking and better performance, reflecting improved controllability, as the tested scenarios require a gradual response with anticipation, and sudden braking is not typically required.

Drivers’ glance behavior was analyzed using the eye tracking data to validate their visual attention patterns. The percentage of glance time to the forward roadway was calculated for the overall drive and for specific time windows around takeover events (10 s before and after).

Analytic Methods

To examine drivers’ takeover behavior, we applied three analyses (Figure 6). In Analysis 1, a mixed-effect logistic regression model predicted TOC types (i.e., likelihood of driver-initiated override), with AR HUD condition as a fixed effect and subject as a random effect. Analysis 2 used a linear mixed-effect model to predict takeover response time, with AR HUD condition and situation as fixed effects and subject as a random effect. For both models, we reported marginal R² (variance explained by the fixed factor alone), and conditional R² (variance explained by the fixed and random factors) as effect sizes. Our analytical approach focused on comparing each experimental condition to the baseline condition, following standard practice for categorical regression where one level serves as the reference. This approach allows for interpretation of each VSA combination’s effectiveness relative to having no VSA information, which is particularly appropriate given our additive experimental design where VSA elements were systematically built up. To provide comprehensive analysis, we also conducted post-hoc pairwise comparisons between all conditions using estimated marginal means, presented both with and without Tukey adjustment for multiple comparisons.

OPEN IN VIEWER

Figure 6.

Analysis plan with dependent measures and statistical models applied. Each analysis was designed to answer corresponding research questions (Analysis 1-RQ 1, Analysis 2-RQ 2, and Analysis 3-RQ 3).

Analysis 3 applied a two-part mixed-effect model for takeover type (full vs. partial) and response magnitude (maximum brake velocity). This approach appropriately handles mixed discrete and continuous responses (J. D. Lee et al., 2021). The first part used binary logistic regression to predict probability of full takeover, while the second part used truncated regression to predict brake pedal press magnitude for full takeovers only. We tested four model variants with different random effects structures: (a) a model including only an intercept for the random effect of participants (allowing individual differences in intercepts), (b) a model including both intercept and slope associated with the AR HUD condition for the random effect (allowing individual differences in intercept and slopes), (c) a version of (a) with the interaction between the AR HUD condition and situation, and (d) a version of (b) with the interaction between the AR HUD condition and situation. The Bayesian estimation approach provides 95% credible intervals rather than traditional p-values. Credible intervals represent the range of parameter values that contain the true value with 95% probability based on the observed data. We considered effects statistically significant when 95% credible intervals did not include zero, which roughly corresponds to p < .05 in frequentist statistics.

All analyses used R software with “lme4” (Bates et al., 2015) for the mixed-effects models, “brms” (Bürkner, 2017) for the two-part models, and “tidyverse” (Wickham et al., 2019) for data manipulation and visualization. We used a .05 alpha level for statistical significance.

RQ1: Effects on Likelihood of Driver-Initiated Overrides

Drivers could disengage automation anytime despite encouragement to use it. We identified three control transfer types: driver-initiated override (driver takes control from automation without takeover request), system-initiated takeover (drivers takes control from automation per request from automation), and driver-initiated handover (driver gives control to automation). Our analysis focused on the first two types where drivers received control.

Figure 7 shows driver-initiated overrides were less frequent than system-initiated takeovers across all conditions. These categories are mutually exclusive within each scenario, as all scenarios led to a takeover request if drivers hadn’t already overridden the automation.

OPEN IN VIEWER

Figure 7.

Percentage of driver-initiated override was smaller in the Chevron and Dynamic ODD conditions compared to the Baseline. While the full sample size was 52 per condition (13 participants x 4 trials), incomplete trials due to technical issues were excluded, resulting in unbalanced samples across conditions.

The Chevron and Dynamic ODD conditions had the lowest percentage of driver-initiated overrides (9% and 2%, respectively), indicating that drivers in these conditions tended to rely on the driving automation until prompted to take over by the system. The results of the mixed-effect logistic regression model indicated that both the Chevron and Dynamic ODD conditions prompted fewer driver-initiated overrides compared to the Baseline condition (p < .05 for the Chevron condition and p < .01 for the Dynamic ODD condition; conditional R² = 0.34, and marginal R² = 0.28).

To further examine the relationships between all AR HUD conditions, we conducted post-hoc pairwise comparisons using estimated marginal means (Table 2). With Tukey adjustment for multiple comparisons, only the DynamicODD condition differed significantly from both Baseline and Box conditions (both p < .05), representing the most conservative interpretation of condition differences. Without multiple comparison adjustment, additional significant differences emerged: Baseline vs. Chevron (p < .05), Box vs. Chevron (p < .05), and Telltale vs. DynamicODD (p < .05). Both analytical approaches support the conclusion that VSA combinations including ego vehicle trajectory information (Chevron and DynamicODD) are most effective in reducing driver-initiated overrides, with DynamicODD showing the most robust effects.

OPEN IN VIEWER

Table 2.

Post-Hoc Pairwise Comparisons Between AR HUD Conditions for Driver-Initiated Override Likelihood With and Without Multiple Comparison Adjustment.

AR HUD Condition		Estimate	SE	z-Ratio	Adjusted p	Nonadjusted p
Baseline	Box	−0.09	0.51	−0.17	0.99	0.87
	Telltale	0.46	0.53	0.87	0.91	0.39
	Chevron	1.52	0.67	2.27	0.16	0.02*
	DynamicODD	3.03	1.1	2.77	0.04*	0.01**
Box	Telltale	0.55	0.55	1	0.85	0.32
	Chevron	1.6	0.68	2.36	0.12	0.02*
	DynamicODD	3.12	1.1	2.83	0.04*	0.01*
Telltale	Chevron	1.05	0.69	1.54	0.54	0.12
	DynamicODD	2.57	1.11	2.33	0.14	0.02*
Chevron	DynamicODD	1.51	1.17	1.29	0.69	0.2

RQ2: Effects on Response Time to Takeover Requests

We then examined the effect of the AR HUD conditions on drivers’ response time to takeover requests. We applied a linear mixed-effect model and found that the main effect of situation was statistically significant (p < .001, conditional R² = 0.59, and marginal R² = 0.34). Specifically, drivers’ response time in the parking lane situation was faster compared to the intersection situation. However, we did not find a significant difference in drivers’ response time across the AR HUD conditions (Figure 8).

OPEN IN VIEWER

Figure 8.

Response time to the takeover request did not differ across AR HUD conditions, but the parking lane situation was associated with faster response time compared to the intersection situation. The error bars represent 95% confidence intervals.

RQ3: Effects on Takeover Types and Response Magnitude

The driving automation system allowed for both full takeover (taking control of both longitudinal and lateral movement) and partial takeover (taking control of either longitudinal or lateral movement). Figure 9 compares the percentages of full takeover and partial takeover responses across the AR HUD conditions, revealing that full takeover was the primary way drivers responded to the takeover request.

OPEN IN VIEWER

Figure 9.

Percentage of partial takeover was higher in the Dynamic ODD condition, but this difference was not statistically significant. Note that sample sizes are smaller than in Figure 7, as this figure includes only system-initiated takeovers. Participants could either override or takeover, leading to unequal sample sizes across conditions.

To examine the likelihood of full takeover (initiated by pressing the brake pedal) and response magnitude (measured by maximum brake velocity within 1.5 s from takeover response), four versions of the two-part mixed model were fitted. The widely applicable information criterion (WAIC) was calculated to compared the models (Vehtari et al., 2017), where smaller values indicate a better model fit. Four models’ WAIC values were: 1334.1 (intercept model), 1330.1 (slope model), 1326.5 (interaction model based on the intercept model), and 1325.4 (interaction model based on the slope model). Based on the WAIC values, we selected the two best models (i.e., the two interaction models) and presented their outcomes in Figure 10. The results indicate that the likelihood of full takeover and braking intensity as a response to takeover requests varied across AR HUD conditions and situations. The parking lane situation was associated with lower probability of full-takeover responses and softer braking response compared to the intersection situation. The Box condition did not lead to a difference in the full-takeover probability compared to the Baseline condition but was associated with a softer braking response compared to the Baseline condition. An interaction effect was found between the Dynamic ODD condition and the situation, indicating harder braking response with the Dynamic ODD condition under the parking lane situation. To aid in the interpretation of the model output, Figure 11 shows the response magnitude patterns across the AR HUD conditions and driving situations.

OPEN IN VIEWER

Figure 10.

Model estimates and 95^th credible intervals for response type and magnitude. Left column (“Response”) indicates whether the driver fully took over by pressing the brake pedal and right column (“Magnitude”) indicate how hard the driver press the brake pedal. All estimates for AR HUD conditions are relative to the Baseline condition, and estimates for the parking lane situation are relative to the intersection situation. Filled points indicate where 95% credible intervals do not include zero, which roughly corresponds to p < .05 in frequentist statistics.

OPEN IN VIEWER

Figure 11.

Illustration of response magnitude patterns across AR HUD condition and driving situations. The Box condition and parking lane situation was associated with softer braking press; the Dynamic ODD condition under the parking lane situation was associated with harder braking press. The error bars represent 95% confidence intervals.

To further analyze the data, we applied a shorter time window of 10 s before and after takeover to assess if there was any difference in drivers’ visual attention to the forward roadway near TOCs. While the results showed consistently high attention to the forward roadway across all conditions (mean = 91%), there was no significant difference in the percentage of glance time to the forward roadway between these two time-windows and across the AR HUD conditions.