Abstract
Extended exposure to reliable automation may lead to overreliance as evidenced by poor responses to auto-mation errors. Individual differences in trust may also influence responses. We investigated how these factors affect response to automation errors in a driving simulator study comprised of stop-controlled and uncon-trolled intersections. Drivers experienced reliable vehicle automation during six drives where they indicated if they felt the automation was going too slow or too fast by pressing the accelerator or brake pedal. Engage-ment via pedal presses did not affect the automation but offered an objective measure of trust in automation. In the final drive, an error occurred where the vehicle failed to stop at a stop-controlled intersection. Drivers’ response to the error was inferred from brake presses. Mixture models showed bimodal response times and revealed that drivers with high trust were less likely to respond to automation errors than drivers with low trust.
Keywords
Introduction
In partially automated vehicles, drivers are expected to monitor the surrounding environment, resume manual control when the vehicle reaches operational design domain (ODD) limits, and remain vigilant to automation errors (SAE, 2021). However, vigilance requires mental effort (Bainbridge, 1983; Warm et al., 2008), and sustained visual attention while moni-toring the roadway can be compromised by several factors (J. Lee et al., 2019). For example, driving for an extended period using automated systems in a monotonous environment can di-minish the useful field of view (Roge et al., 2002). People be-come less vigilant when experiencing constantly reliable auto-mation (Parasuraman et al., 1993). Also, drivers can decide to engage in non-driving relevant tasks that can undermine the perception of crucial information and a decrease in situation awareness (Casner & Schooler, 2015). Such effects are not lim-ited to vehicle automation and have also been documented in many studies that have explored responses to automation errors between constantly reliable and variably reliable automation. These studies show error detection was worse in groups with constantly reliable automation compared to groups that experi-ence automation with variable reliability (Molloy & Parasuraman, 1996; Parasuraman et al., 1993). Overall, experience with highly reliable automation undermines automation error detec-tion and response, which ultimately reduces driving safety (Greenlee et al., 2022).
Understanding the factors that contribute to overreliance can help mitigate it. One such factor is trust in automation. Trust has been identified as a factor that influences automation use (J. D. Lee & See, 2004). More specifically, overtrust in automated systems can lead to misuse. Overtrust is defined as “poor cali-bration in which trust exceeds the systems capabilities” and misuse is defined as an “overreliance on automation” (J. D. Lee & See, 2004; Parasuraman & Riley, 1997). In previous studies, this effect of trust on automated system reliance has been demonstrated by promoting trust in some drivers and low-ering trust in others (Körber et al., 2018). Results show that in a safety-critical situation, the trust-promoted group took longer to respond than the trust-lowered group. In this present study, we extend this research by exposing drivers to reliable driving automation followed by an automation error, where the vehicle fails to stop at a stop-controlled intersection. The effect of trust on drivers’ responses to automation errors is examined. Exist-ing studies generally examine trust variation using subjective trust ratings or introduce trust variation as a between-subject variable (Kohn et al., 2021). This research uses objective trust measures and implements mixture models to account for trust-related variability in responses to the automation error.
Methods
Participants
Twenty-four people (16 female, 8 male; aged between 25 and 55, M = 29, SD = 5) participated in the study. Participants were drivers from the Madison, WI area. Inclusion criteria in-cluded possession of a valid driver’s license for at least 2 years. The study lasted approximately 2 hours per participant and driv-ers were compensated US $30/hr. This study was approved by the Education and Social/Behavioral Science Institutional Re-view Board at the University of Wisconsin – Madison.
Apparatus
A fixed-based simulator (NADS Minisim™) was used for the study. The driving scenario was a four-lane suburban street. This was visible via three 43 x 25.4-inch monitors with the cen-ter monitor placed 4.5 feet from the driver, producing a 135o field of view.
Experimental Design and Independent Variables
A 4 (intersection type) x 3 (automation style) within-sub-ject design was implemented. A replicated Latin square design was used to counterbalance the conditions.
Intersection type
There are four intersection types: two intersections with stop signs and two without stop signs. These were differentiated by the presence of cross-path traffic which drivers could see but did not interact with.
Automation style
Drivers experienced three automated driving styles: conservative, moderate, and aggressive. All au-tomation styles detect and brake at stop-controlled intersections and were capable of longitudinal and lateral control of the ve-hicle. Figure 1 shows how the speed profiles distinguish the three automated driving styles. Pilot study data from the manual driving scenarios guided the development of the automated driving style. The aggressive, moderate, and conservative styles were determined using the 15th, 50th, and 85th percentile of driv-ers’ manual driving data such as mean deceleration, mean ac-celeration, distance to the stop line when the speed first goes below 1 mph during the approach to stop-controlled intersec-tions and stop duration at stop-controlled intersections (Domeyer et al., 2019; Kamaraj et al., 2023; J. D. Lee et al., 2021). Table 1 shows the values that differentiate each driving style. Note that, during the error event the automation did not detect a stop sign and failed to brake but maintained longitudi-nal and lateral control.
Speed vs distance to stop line for the conservative, moderate, and aggressive driving styles. Styles are differentiated by the difference in the initiation of braking and stopping distance from the stop line.
Variables differentiating the automation’s conservative, moderate, and aggressive driving style behavior at stop-controlled intersections.
Dependent Variables
Dependent measures examined here include subjective and objective trust in automation and response to automation errors. Subjective trust is assessed via questionnaires while objective trust is inferred from the driver’s pedal inputs while monitoring the automation. Prior research suggests that pedal inputs can measure trust (J. D. Lee et al., 2021). This research uses objec-tive trust in automation to predict drivers’ responses to automa-tion errors. Subjective ratings of trust in automation (Muir & Moray, 1996) were used for post-hoc descriptive data analysis.
Trust in automation
Trust in the automation was assessed subjectively via surveys at the end of each automated drive. In addition, drivers were asked to press the brake pedal if they felt the automation was driving too fast and to press the gas pedal if they felt the automation was going too slow. The time drivers press the accelerator and brake pedals is used as an objective trust measure. More time pressing the pedals indicates lower trust in automation and vice versa. The time spent pressing the pedals was estimated using all non-zero values of pedal presses.
Response to automation error
The error event in this study was a failure of the automation to stop at a stop-controlled intersection. During this event, drivers were expected to press the brake pedal. Note that the most conservative driving style begins decelerating 963 ft before the stop line. We use this point as the stimulus onset, i.e., the first observable evidence of an automation error. Brake pedal presses recorded after this point indicate response onset, and indicate the time taken to respond to the automation error (Engström et al., 2022). The response time is the difference between the response and stimulus onset. Fast responders braked closer to the 963 ft mark and slow re-sponders braked closer to the stop line. Drivers who responded after the stop line were labeled late responders and drivers who responded to the automation error before the stop line were la-beled early responders. For drivers who did not respond to the error event (n = 2), the average response time of the late re-sponders was used in place of the non-response data point.
Procedure
Once drivers arrived at the study site, inclusion criteria (valid driver’s license, age, driving experience) were verified. Drivers were briefed about the study's purpose as well as poten-tial risks and benefits before obtaining informed consent. They then completed pre-study questionnaires and drove practice drives to acquaint them with the simulator. Data collection commenced with two manual drives to further allow familiarity with the simulator, three automated drives where the automa-tion performed reliably (each automation style experienced twice consecutively), and a final drive automated drive where the automation fails at a stop-controlled intersection. Wellness questionnaires were administered after each drive to monitor for simulator sickness. The driving style for the final error event was assigned such that drivers were equally distributed across the three styles (i.e., eight drivers experienced the error in the conservative, eight in the moderate, and eight in the aggressive driving style). During automated driving, drivers were informed that the vehicle is a fully self-driving car that will center itself on the lane and brake and stop at intersections. They were asked to imagine that they were test driving different automated vehi-cles. The gas and brake pedal were inactive during automated driving. Drivers were asked to monitor the vehicle automation and press the brake pedal if they thought the automation was going too fast and the accelerator pedal if they thought the au-tomation was going too slow. After all the drives were com-pleted, drivers were interviewed about their experience with the automation, debriefed, and paid for their time.
Data Processing and Analysis
The tidyverse R package was used for data wrangling and visualization. Preliminary analysis of the data revealed possible heterogeneity of the response time to the automation error, vio-lating the assumption of a single underlying distribution for the data. To address this, regression mixture models were used to analyze the data, implemented via the flexmix R package (Grun & Leisch, 2007; Tan & Mueller, 2016).
Mixture models
Behavior data such as those obtained from driving behavior often exhibit heterogeneity (Park et al., 2010). For instance, if we assume that driver behavior may vary based on factors such as age or gender. When fitting regression mod-els to these data, the models are likely to vary across these dif-ferent groups of data. At times, the factors influencing the het-erogeneity in the data are unidentified or unobservable. For ex-ample, risk-seeking vs. risk-averse driving behavior may influ-ence the response to certain stimuli but may not be observed. In such cases, mixture models help model the probability of sub-groups belonging to unobserved groups in the data. This study applied regression mixture models to identify sub-groups in the data. These models assume that sub-groups are defined by dif-ferent regression models. The method implemented here uses the Expectation-Maximization (EM) algorithm to find the max-imum likelihood estimates for each subgroup. First, members of the group are randomly assigned to each subgroup and max-imum likelihood estimates are found for each subgroup. Group members that are more likely to fit in another group are resorted and then maximum likelihood estimates are recalculated. This step is repeated to identify the maximum likelihood partition.
Results
Influence of Trust in Automation on Response to Automa-tion Errors
Figure 2 shows the relationship between each pair of vari-ables for all drivers (brake pedal press time, gas pedal press time, and response time to error). The distributions of these var-iables are shown along the diagonal and the Pearson correla-tions are shown on the top right. During debrief interviews, one of the drivers reported noticing an automation error; however, this driver did not respond by pressing the brake pedal within the intersection bounds. As a result, this data point was ex-cluded from the analysis as it was not possible to determine the response time without the pedal press input. Thus, 23 data points—one for each driver—were used to examine the associ-ation between pedal press-based trust and the responses to the automation error.
Scatterplot (bottom left) and Pearson correlation (top right) of the pairs of brake pedal press time, accelerator pedal press time, and response time. The distribution of each variable is shown along the diagonal.
Given the bimodal distribution of the data, mixture models were estimated for two clusters. The first model used the brake pedal-based trust as the covariate and the second used the ac-celerator pedal-based trust as the covariate. Table 2 summarizes the mixture model fit. For model 1, Cluster 1 had a ratio of 0.53 which indicates that 15 points had a non-zero likelihood of be-ing in that cluster, and of those 15 points, 53% (i.e., cluster size = 8) were best fit by that cluster. Cluster 2 had a ratio of 0.68 which indicates that 22 points had a non-zero likelihood of being in that cluster, and of those 68% (i.e., cluster size = 15) were best fit by that cluster. For model 2, Cluster 1 had a ratio of 0.45 i.e., 11 points had a non-zero likelihood of being in that cluster, and of those 11 points, 45% (i.e., cluster size = 5) were best fit by that cluster. Cluster 2 had a ratio of 0.78 i.e., 23 points had a non-zero likelihood of being in that cluster, and of those 78% (i.e., cluster size = 18) were best fit by that cluster.
Mixture model summary for k = 2 clusters for models examining the effect of pedal press-based trust on response to automation error.
Following cluster identification, each cluster was qualita-tively assessed for features that distinguished them. The most distinguishing feature of each cluster was whether the drivers were early or late responders. Cluster 1 consists of mostly late responders (five late responders and three early responders) and Cluster 2 consists of only early responders (fifteen early re-sponders). Mixture models were fit to the data of each cluster. For model 1 (response time ~ brake pedal press time, see Figure 3), Cluster 1 and Cluster 2 fitted a linear model to predict the response time with the brake pedal presses. For Cluster 1, which consists mostly of the drivers who responded to the automation error after the stop line, the model’s explanatory power is sub-stantial (R2 = 0.92). The model’s intercept, corresponding to the brake pedal press = 0, is at 18.59 (95% CI [17.22, 19.96], t(6) = 26.54, p < .001). Within this model, the effect of the brake pedal press is statistically significant and negative (β = -0.91, 95% CI [-1.12, -0.70], t(6) = -8.54, p <.001; Std. β = -0.96, 95% CI [-1.18, -0.74]) indicating that drivers who spent less time engag-ing with the brake pedal took longer to respond to the automa-tion error. For Cluster 2, which consists of drivers who re-sponded to automation error before the stop line, the model’s explanatory power is very weak (R2 = .00001). The model’s in-tercept, corresponding to brake pedal press = 0, is at 10.17 (95% CI [8.90, 11.44], t(13) = 15.65, p < .001). Within this model, the effect of the brake pedal press is not statistically significant (β = .0009, 95% CI [-0.14, 0.14], t(13) = 0.01, p = 0.989; Std. β = 0.003, 95% CI [-0.54, 0.55]).
Response time to automation errors versus trust indicated by brake pedal interaction for each cluster identified via mixture modeling.
For model 2 (response time ~ accelerator pedal press time, see Figure 4), Cluster 1 and Cluster 2 fitted a linear model to predict the response time with the gas pedal presses. In Cluster 1, which consists of the drivers who responded to the automa-tion error before the stop line, the model’s explanatory power is very weak (R2 = .004). The model’s intercept, corresponding to the gas pedal press = 0, is at 18.78 (95% CI [16.51, 21.05], t(3) = 16.22, p < .001). Within this model, the effect of the gas pedal press is not statistically significant (β = -0.005, 95% CI [-0.10, 0.09], t(3) = -0.11, p = 0.910; Std. β = -0.06, 95% CI [-1.19, 1.06]). For Cluster 2, which consists of all drivers who re-sponded to the automation error before the stop line, the model’s explanatory power is very weak (R2 = .0007). The model’s intercept, corresponding to gas pedal press = 0, is at 9.99 (95% CI [8.50, 11.48], t(16) = 13.16, p < .001). With this model, the effect of the accelerator pedal press is not statisti-cally significant (β = 0.005, 95% CI [-0.08, 0.10], t(16) = 0.11, p = 0.911; Std. β = 0.03, 95% CI [-0.46, 0.52]).
Response time to automation errors versus trust indicated by gas pe-dal interaction for each cluster identified via mixture modeling.
The two clusters identified from model 1 (response time ~ brake pedal press time) and model 2 (response time ~ accelera-tor pedal press time) show that cluster 1 from both models con-sisted primarily of late responders (see responses before and af-ter the stop line in Figure 3 and Figure 4). Model 1 – cluster 1 indicates that the effect of the brake pedal pressing time was negatively associated with response time. This indicates that drivers who spent more time pressing the brake pedal in prior drives were quicker to respond to the automation error whereas those that spent less time pressing the brake pedal were slower to respond to the automation error. Note that we assume that more pedal pressing indicates low trust in the automation and less pedal pressing indicates higher trust in the automation. Thus, it follows that late responders are those that have higher subjective trust ratings in reliable automation and early re-sponders are those that have lower trust in reliable automation. We verify this assumption by comparing the mean subjective trust in reliable automation across the early and late responders. One driver was removed as an outlier from the group of early responders. A Welch Two Sample t-test testing the difference of the mean subjective trust in reliable automation for the early and late responders (MEarly responders = 1.68, MLate responders = 2.42) shows that the effect is negative, statistically significant, and large (difference = -0.74, 95% CI [-1.31, -0.16], t(10.91) = -2.80, p = 0.017; Cohen's d = -1.70, 95% CI [-3.05, -0.29]). This supports the assumption that drivers who responded late to the automation error had more trust in the automation.
Discussion
The goal of this research was to examine the effect of trust on drivers’ responses to automation errors. Mixture models identified two clusters based on the drivers’ engagement with reliable automation via brake and gas pedal presses and re-sponse to automation errors. Generalized regression models were applied to each cluster that was identified via mixture modeling. In the models that examined the effect of brake pe-dal-based trust on drivers’ response to the automation error, one cluster that comprised early responders to automation errors showed no significant effect of trust on response time (see Fig-ure 3 Cluster 2). In the other cluster (see Figure 3 Cluster 1) which consisted primarily of late responders, more brake pedal presses were associated with quicker responses to automation errors. Both models that examined the effect of gas pedal-based trust showed no significant effect of trust on the response time to automation error. Results also indicated that late responders had higher subjective trust ratings. Thus, the assumption that the late responders are generally high trusters was verified.
This study reinforces previous findings that automation monitoring alone does not indicate appropriate reliance and ex-tended experience with seemingly perfect automation can result in overtrust and automation-induced complacency (Parasuraman et al., 1993; Wickens et al., 2015). We also demonstrate how trust dynamics vary across drivers. Specifically, while all participants experienced the same automation, their responses to the automation differed (Li et al., 2023). Mixture models helped reveal that one group of drivers became complacent fail-ing to respond to automation errors promptly. Consistent with this pattern, subjective ratings of trust of these drivers show that these drivers trusted the reliable automation more. These results are consistent with studies that have found differences in trust dynamics across people that lead to bimodal distributions (Bhat et al., 2022; Liu et al., 2021). A focus on strategies to calibrate trust in drivers with high trust in automation could prove useful in managing their reliance on automation. For instance, drivers with high levels of trust may benefit from strategies that in-crease engagement with automation, such as collaborative driv-ing and feedback from the vehicle.
In addition to developing more collaborative driving de-signs, this study also shows that long-term continuous engage-ment with automation through pedal presses or other behavioral indicators could be used to estimate and predict driver response to automation errors. Such state monitoring goes beyond as-sessing drivers’ momentary readiness to take control based on whether their eyes are on the road and hands are on the wheel. This research suggests that driver states over longer time hori-zons may be related to when and how often they intervene.
Limitations of the study relate to how simulation and ex-perimental settings translate to vehicles on the road. Simulation may encourage different behaviors due to reduced risk com-pared to on-road driving. In addition, because this study does not include a production system, with specific vehicle perfor-mance and driver feedback designs, the results should be aug-mented data from drivers in actual vehicles.
Conclusions
Automation-induced complacency and overreliance can undermine drivers’ responses to unexpected errors. We showed a bimodal distribution of trust can emerge when people use re-liable automation, and no engagement is encouraged by the ve-hicle. Strategies to support driver engagement in vulnerable groups of drivers may help reduce the effects of automation-induced complacency. Future work will focus on comparing strategies to support driver engagement and trust calibration.
Footnotes
Acknowledgements
This study was funded by Toyota Collaborative Safety Re-search Center (CSRC) and a grant from the National Science Foundation (Award number 1739869). We thank the members of the University of Wisconsin-Madison, Cognitive Systems Laboratory for their insightful discussions and comments.
