How Can the Trust-Change Direction be Measured and Identified During Takeover Transitions in Conditionally Automated Driving? Using Physiological Responses and Takeover-Related Factors

Abstract

Objective

This paper proposes an objective method to measure and identify trust-change directions during takeover transitions (TTs) in conditionally automated vehicles (AVs).

Background

Takeover requests (TORs) will be recurring events in conditionally automated driving that could undermine trust, and then lead to inappropriate reliance on conditionally AVs, such as misuse and disuse.

Method

34 drivers engaged in the non-driving-related task were involved in a sequence of takeover events in a driving simulator. The relationships and effects between drivers’ physiological responses, takeover-related factors, and trust-change directions during TTs were explored by the combination of an unsupervised learning algorithm and statistical analyses. Furthermore, different typical machine learning methods were applied to establish recognition models of trust-change directions during TTs based on takeover-related factors and physiological parameters.

Result

Combining the change values in the subjective trust rating and monitoring behavior before and after takeover can reliably measure trust-change directions during TTs. The statistical analysis results showed that physiological parameters (i.e., skin conductance and heart rate) during TTs are negatively linked with the trust-change directions. And drivers were more likely to increase trust during TTs when they were in longer TOR lead time, with more takeover frequencies, and dealing with the stationary vehicle scenario. More importantly, the F1-score of the random forest (RF) model is nearly 77.3%.

Conclusion

The features investigated and the RF model developed can identify trust-change directions during TTs accurately.

Application

Those findings can provide additional support for developing trust monitoring systems to mitigate both drivers’ overtrust and undertrust in conditionally AVs.

Keywords

trust-change directions conditionally automated driving takeover transitions physiological parameters random forest

Introduction

Automated driving technology has the potential to increase traffic efficiency (Teoh and Kidd, 2017), safety, and fuel efficiency (Fagnant and Kockelman, 2015). However, we still have a long way toward fully autonomous driving being widespread. Besides, vehicles with conditionally automated driving systems such as traffic jam pilots have been developed. According to the vehicle automation level by the Society of Automotive Engineers (SAE) International, the role of drivers may change from manual operators to disengaged supervisors in the SAE Level 3 automated driving (SAE, 2018). Several new challenges related to human factors have been raised, such as trust and acceptance of users, especially for situations in which drivers are asked to resume control of the vehicle due to system limitations or failures (Abe et al., 2018). Trust is one of the important challenges that impacts the public acceptance and use of AVs (Choi and Ji, 2015).

Trust in automation is defined as “the attitude that an agent will help achieve an individual’s goals in a situation characterized by uncertainty and vulnerability” (Lee and See, 2004). To systematically understand factors that influence trust in automation, Hoff and Bashir (2015) proposed a three-layered trust model, including dispositional, situational, and learned trust. Dispositional trust denotes an individual’s overall tendency to trust automation, independent of context or specific systems. Situational trust describes trust as a combination of the external environment and operator states. Learned trust is established by the operator’s evaluations of a system based on historical experience or the current interaction. The takeover transition, as the main component of the interaction of the driver with the SAE level 3 automated driving system, has a potential impact on situational trust and learned trust (Körber et al., 2018c; Jin et al., 2020). When trust in AVs decreases during the TTs, drivers may avoid using the system and return to manual control, thereby reducing the safety benefits and other goals of automated driving pursued (Seet et al., 2020). Thus, it is essential for exploring the effects and relationships between the trust-change direction and takeover-related factors or physiological parameters, and to establish a model identifying the changes in trust during TTs, which can provide additional support for the designing of the vehicle-driver interface and trust monitoring system to reduce the trust erosion due to TORs.

Factors Influencing Driver Trust

Previous studies have examined the factors that affect the trust of drivers, such as systems transparency, training, driving style, and perceived risk, (Hergeth et al., 2017; Ekman et al., 2019; Li et al., 2019; Zhang et al., 2019; Kraus et al., 2020; Azevedo-Sa et al., 2021) and provided some interesting findings. For instance, the “defensive” automated driving style was perceived as more trustworthy than the “aggressive” driving style (Ekman et al., 2019). Li et al. (2019) and Zhang et al. (2019) found a significant negative relationship between perceived safety risk and driver trust. Besides, a few researchers paid attention to the effects of takeover-related factors on trust in AVs, such as the TOR lead time and takeover frequencies (i.e., the number of takeovers).

The TOR lead time is defined as the time available between one TOR and the system limit (Gold et al., 2018b). It is considered to affect the system-perception reliability and drivers’ takeover safety and then impact trust. For example, untimely TOR lead time represents a risky takeover situation, which reflects the unreliability of the system and then can reduce trust (Lee and See, 2004). Jin et al. (2020) examined the effect of the TOR lead time (i.e., 7 s and 10 s) on driver trust, showing that when drivers experienced the shorter TOR lead time, they had a lower subjective trust rating. Besides, driver trust would be related to not only the TOR lead time but also the takeover frequencies. Hergeth et al. (2017) and Körber et al. (2018c) suggested that the mean of the pre-and post-takeover trust rating increased with takeover frequencies.

Trust Measurements in Automated Vehicles

To understand how trust impacts the acceptance of AVs, reliable metrics are needed to measure trust. Multidimensional questionnaires containing sub-dimensions (e.g., trust, perceived risks) were used to measure driver trust (Jian et al.,2000; Körber, 2018a; Holthausen, 2020). Furthermore, to reduce the frustration of participants facing repeated measurements, other researchers used a single-dimensional questionnaire during the experiment (Hergeth et al., 2016). However, self-reported trust ratings cannot provide continuous measurements, and therefore cannot estimate real-time changes in trust, and also are intrusive, which may be difficult to use in applied settings (Hergeth et al., 2016; Walker et al., 2019). To continuously estimate driver trust in real-time, the objective measurements of trust such as monitoring behaviors, skin conductance, and heart rate have been explored in many studies.

Previous empirical evidence demonstrated that driver trust is closely linked to monitoring behaviors (Hergeth et al., 2016; Körber et al., 2018b; Walker et al., 2019). For example, Hergeth et al. (2016) found a higher subjective trust rating associated with a lower monitoring behavior. Körber et al. (2018b) reported that the trust promotion group induced by the initial information paid less attention to roads or dashboards. Besides, many scholars have explored relatively novel methods to measure trust, such as skin conductance (SC) and heart rate (HR) (Waytz et al., 2014; Khawaji et al.,2015; Morris et al., 2017; Wintersberger et al., 2017; Gupta et al., 2019; Petersen et al., 2019; Perello-March et al., 2021). However, attempts at using these measures to capture trust in automation are still mixed results. For example, drivers showed lower self-reported trust ratings and higher SC in risky driving situations (Morris et al., 2017). Humans have more trust and weakened HR increases during a vehicular accident when they drive an anthropomorphism AV (Waytz et al., 2014). However, Gupta et al. (2019) and Perello-March et al. (2021) could not report any significant effects between trust and the SC. Petersen et al. (2019) also found a disconnection between trust and HR. Furthermore, Kohn et al. (2021) comprehensively reviewed trust measurements in existing empirical works, suggesting that the monitoring behavior is a more established measure than other physiological measures (e.g., the SC and HR), since it can provide a stable correlation of trust and is less disruptive to the original task.

Prediction Models for Trust in Automated Vehicles

To calibrate the levels of driver trust, it is also vital to develop a trust prediction model for AVs. There is some research on the predictive model of driver trust during the non-takeover period in AVs (Azevedo-Sa et al., 2020; Ayoub et al., 2021; Liu et al., 2021). For instance, Azevedo-Sa et al. (2020) established the trust model in AVs based on interactive experiences (e.g., system malfunction types and system usage time) and drivers’ behavior (e.g., non-driving-related tasks performance) by the Kalman filter approach, which can estimate driver trust during the non-takeover process. Liu et al. (2021) proposed a customized trust model by clustering humans based on their trust dynamics to improve trust prediction performance during the non-takeover.

Research Motivations and Main Contributions

Considering that self-reported data might always have limitations due to individual and situational biases, which may not accurately reflect driver trust (McBride et al., 2010; Rosenman et al., 2011; Petersen et al., 2019). Yu et al. (2021) proposed a combination of self-reported trust ratings and system usage frequencies to measure driver trust in advanced driver assistance systems. Drivers, as disengaged supervisors in AVs, will almost always use the system for a short period or longer. It may be difficult to obtain a vailed trust measurement through system usage frequencies and apply it to AVs. In SAE level 3 automation, drivers always have difficulty negotiating TTs safely because they are out of the control loop (Du et al., 2020a), which can undermine trust and in turn affect drivers’ reliance and use of AVs. Thus, we need to find reliable indicators to measure the change values of driver trust during TTs. However, the measurement of trust in AVs still relied on subjective evaluation as the ground truth. To measure trust in AVs accurately, a hybrid method of combining subjective ratings with the objective behavior indicator is introduced into our study. Lee and See (2004) concluded that users’ reliance on automation depends to a certain extent on trust. That is, when trust reduces, users may decrease their reliance on automation and increase the monitoring behavior. Thus, it is reasonable to believe that when drivers reduce trust during TTs, after completing takeover missions and reactivating the autopilot mode, they will pay more attention to the road or the dashboard. Moreover, as noted previously, the monitoring behavior can provide a more stable correlation of trust than the other physiological measures such as the SC and HR. Thus, we will combine the change values of the subjective ratings and the monitoring behavior before and after takeovers to measure the trust-change directions during TTs.

In addition, previous studies have explored the effects of potential factors on driver trust, such as systems transparency, training, TOR lead time, and takeover frequencies. However, the influence of takeover scenario types on driver trust is rarely mentioned. Vogelpohl et al. (2018) found that scenario types can affect the takeover criticality, while perceived risks can influence driver trust (Azevedo-Sa et al., 2021). Thus, we speculate that the takeover scenario may affect the changes in driver trust. Critical or unexpected TORs could be considered rather risky situations (Maule & Hockey, 1993). In these cases, drivers would feel safety risk, which might lead to a sharp drop in trust, inducing the increasement in stress, and then their physiological parameters would change accordingly (Mühl et al., 2020). If driver trust is not improved timely, misoperations might occur due to the distrust of the system, and then increase the risk of accidents.

Besides, existing studies have examined the relationships between physiological responses (e.g., the SC and HR) and trust to predict driver trust in AVs. Nevertheless, as mentioned above, these results are still mixed conclusions that need to be further explored. Also, as far as we knew, the connection between physiological parameters (i.e., SC and HR) and the trust-change directions during TTs is rarely considered because the relationships among the trust-change direction during TTs, psychological parameters, and takeover-related factors could be complex. Surely, exploring their associations can be beneficial to understand changes in driver trust due to altering the driving situation, providing guidance for the development of a trust monitoring system to calibrate trust, and ultimately, decreasing the risk of driving accidents.

Finally, to mitigate the trust miscalibration issues such as undertrust and overtrust, a few researchers established the trust model focused on estimating driver trust during non-takeover processes, which, however, cannot estimate the trust-change direction during TTs. The takeover missions will be recurring events that could undermine the trust and acceptance of drivers, and, therefore, the successful introduction of AVs. It is vital to establish a model identifying the trust-change direction during TTs to foster better driver-vehicle trust. To the best of our knowledge, there is no research on the model of the trust-change direction during TTs. As aforementioned, stress-mediated the SC and HR may also change when driver trust is altered. Besides, the trust-change direction during TTs is closely linked with takeover-related factors (i.e., the TOR lead time and takeover frequencies), and scenario types may affect the changes in driver trust during TTs.

Thus, we will comprehensively examine the relationships and effects among physiological parameters (i.e., the SC and HR), takeover-related factors (i.e., TOR lead time, takeover frequencies and scenario types), and the trust-change direction during TTs. Furthermore, we will establish an identifying model of the trust-change direction during the takeover period using the random forest, based on the SC, HR, and takeover-related factors. The main contributions of this study are summarized in brief as follows:

1) A hybrid method of combining the changes in the subjective rating and objective behavior indicator (i.e., the monitoring behavior) before and after takeovers is proposed to measure the trust-change direction during TTs.

2) The effects and relationships between takeover-related factors, drivers’ physiological responses, and the trust-change direction during TTs are explored and analyzed.

3) A model for identifying the trust-change direction during TTs is developed based on the driver’s multimodal physiological data and takeover-related factors.

Method

Participants

A total number of 34 graduate students participated in the experiment (mean age = 23.8; standard deviation [SD] = 3.9; 3 females and 31 males). Each participant received a cash bonus of 100 yuan. All of participants had a valid driver’s license and normal or corrected-normal vision. Participants had their driver’s licenses for at least 1 year. This study was approved by the Institutional Review Board in the college of mechanical and vehicle engineering at Hunan University.

Apparatus

The experiment was conducted in a static driving simulator. The virtual world was projected on three primary displays (size 27 inches). The simulation of driving scenarios was programmed in Prescan software 8.6.0. The different driving parameters (e.g., ego speed, lead speed, acceleration, braking) were displayed in the graphical user interface (GUI) (Figure 1a). The vehicle was programmed to simulate the SAE L3 automation (SAE, 2018), which can handle the lateral and longitudinal control. The automation could be toggled via the first red button on the right side of the steering wheel (Figure 1b) and was also shut off by steering or braking input. The TOR signal was emitted by the Arduino Uno (Figure 1c). All questionnaires were collected online with two mobile phones.

As shown in Figure 1, the driver was equipped with psychophysiological sensors. Gaze behaviors were recorded using SMI glass with a tracking frequency of 60 Hz (Figure 1d). The SC and electrocardiogram (ECG) were measured with a system (MP150, Biopac Systems Inc, CA) with a sampling rate of 200 Hz (Figure 1e, Figure 1f).

The non-driving-related task (NDRT) in the study was a visual Surrogate Reference Task (SuRT) (ISO 14198, 2012), which was presented on a tablet (Figure 1). In this task, participants were instructed to select, by tapping on the larger circle (diameter 47 pixels) among 49 distractor circles (diameter 40 pixels). The placement of the display needed participants to reallocate their visual focus completely away from the road environment if they wanted to engage in the task. The reason for utilizing the SuRT with manual input was to simulate the hands-off wheel and the eyes-off the road condition.

Experimental Design

Information Material

We used a 2 $\times$ 3 within-subject design in our study. The factors were two TOR lead time conditions (4 s vs. 8 s) and three scenario types (i.e., roadwork, stationary vehicle and dynamic truck). The experimental procedure is depicted in Figure 2.

At the beginning of the experiment, participants watched an approximately one-minute introductory video about an automated vehicle from YouTube.com (https://www.youtube.com/watch?v=V76J9Ij3cRI). In the video, the main functions and sensors in conditionally AVs were explained. Subjects also saw a non-critical takeover situation caused by the missing lane lines. While the second part of the introductory information was presented in text form. It is described that the AV system has a few limitation situations requiring driver’s manual intervention, which are illustrated in Table 1 (Gold et al., 2018a; Wandtner et al., 2018; Schwarz et al., 2019; Körber et al., 2018c). Besides, participants read the text descriptions: when the AV encounters a limitation situation, it will release a buzzer sound in advance and require the driver’s manual intervention at this time. The third part of the information material was the introductory driving taking nearly 5 minutes. First, they were prompted to steer and brake manually to be familiar with driving in the driving simulator. Then, they experienced one TOR with no surrounding traffic and no obstacle during the driving session of 2 minutes. And thereby, no actual intervention was required. Participants were instructed to regain manual control.

TABLE 1:

The System’s Limitation Information

Initial Information About the Automated Driving	Text Descriptions	Events Occurred During Driving
Potential problems at sharp bends	√	No
Potential problems with missing lane lines	√	No
Potential problems with a temporary change in lane lines, such as roadworks	√	Yes
Potential problems with static obstacles, such as broken-down vehicles	√	Yes
Potential problems with sensor failure	√	Yes

Scenarios

Three takeover scenarios were designed based on the above system’s limitations. In the stationary vehicle scenario, one TOR was triggered by the broken-down vehicle directly in front of the same lane (Gold et al., 2018a, Figure 3a). For the roadwork scenario, one TOR was triggered by a temporary change in lane lines. An information board with words (WORK ZONE AHEAD), roadwork sign, and turn left ahead sign directly in front of the same lane was used as a static obstacle (Körber et al., 2018c, Figure 3b). In the dynamic truck scenario, the temporary sensor failure of the AV resulted in its longitudinal control failing and triggered one TOR (Gold et al., 2018a, Figure 3c). The slow leading truck (60 km/h) right in front of the same lane was regarded as a dynamic obstacle.

A leading vehicle with 100 km/h obscured these situations to increase the criticality of the takeover situations. With nearly a TTC of 5 s or 9 s, the leading vehicle suddenly swerved to the left lane, allowing the subjects a view of the obstacles at 4 s or 8 s TTC (Figure 3). When an auditory TOR in form of buzzing (2700 Hz, 74 dB) was emitted at 4 s (44.44 m for the dynamic obstacle, 111.12 m for the static obstacle) or 8 s TTC (88.88 m for the dynamic obstacle, 222.24 m for the static obstacle), requesting drivers manually regain control, the automated driving system was shut down for drivers to conveniently take over control of the vehicle. Each participant experienced six takeover situations altogether, which were formed by the arrangement and combination of two different TOR lead time levels and three takeover scenarios. To minimize the ordering effect, the sequence of the takeover scenarios across participants was counterbalanced via Latin Square.

Non-Driving-Related Task Surrogate Reference Task

In each takeover event, the NDRT appeared every interval 120 seconds, and 2 or 3 times in total, each time lasting 60 seconds. One of the presentations lasting 30 seconds was interrupted by a TOR. In the next 45 seconds, drivers coped with takeover missions and observed the road environment without engaging in NDRT. Immediately after, the autopilot system was reactivated, and the NDRT was presented for another 30 seconds (Figure 2). Besides, to reduce the predictability of takeover situations, the driving time before a TOR (ranging from 2.5 to 6.5 min) was manipulated by performing at most 3 NDRT periods.

The experiment had a gamification aspect to it. Participants competed for monetary bonuses to increase the real-world consequences of the simulated experiment. Participants got two points for completing an NDRT correctly. Ten points were deducted if the driven vehicle left the lane boundary. Besides, a penalty of 50 points was given if a crash with another obstacle occurred in the lane. The scoring mechanism encouraged subjects to rely on automation, and they could focus on engaging the NDRT. After the experiment, the participant with the highest score was rewarded cash bonuses of 300 yuan.

Instructions and Experiment Track

The vehicle drove in the right lane at 100 km/h on a two-lane freeway. The headway was set to 1.5 s. Automated lane changing or overtaking was not implemented. Participants were told to drive in the right lane and were only allowed to change lanes in case of emergency. After passing the obstacle, participants were instructed to come back to the right lane. And then, drivers reactivated the autopilot mode by themselves or were prompted by the experimenter 45 s after the TOR was issued. Participants were asked to safely drive at all times. Furthermore, participants must perform the NDRT whenever the experimenter required the participant to engage in the NDRT.

Questionnaires

In this study, a single-dimensional questionnaire of trust was used to reduce the frustration of the subjects in face of repeated measurements during the trial. In reference to earlier studies, the subjects were asked to slide the slider to rate their trust levels on a scale from 0 to 100 (“On a scale from 0 to 100, how much do you trust the system?”) (Hergeth et al., 2016). A single-item trust rating was both filled at the beginning and end of each NDRT (Figure 2).

To assess the criticality of takeover situations, an 11-point scale proposed by Naujoks et al. (2015) was used. The scales ranged from 0 (imperceptible) to 10 (uncontrollable) with the anchor categories: imperceptible (0), harmless (1–3), unpleasant (4–6), dangerous (7–9), and uncontrollable (10). The criticality ratings of takeover situations were filled out at the end of the NDRT after a TOR.

Procedure

After participants have been welcomed and signed an informed consent form, participants filled out an online questionnaire on demographic data, watched the introductory video, and received introductory information in text form. The eye-tracker was worn and calibrated. Next, two SC electrodes were attached to participants’ distal phalanges to obtain the SC signal (Figure. 4a; Scerbo et al., 1992). Three electrodes were placed on the body in the lead II configuration (Figure. 4b) to record the ECG (Dupre et al., 2005). The participants received several instructions, and the introductory drive started. And then, the participants were invited to exercise the NDRT. If the participants said that they were comfortable with engaging in the NDRT, they were asked to sit quietly in the driver’s seat for 5 minutes to collect the physiological resting baseline. The experimental drive started. The experiment duration lasted nearly 1.5 hours.

Feature Generation

In this subsection, we first preprocessed the physiological data to remove the noise and then standardized it. Finally, the feature extraction of the preprocessed data was implemented.

The SC signal was downsampled to 25 Hz and smoothed by a median smoothing filter of 1-second width for eliminating the movement artifact (Lascurain-Aguirrebeña et al., 2019). The phasic component of the SC was extracted using a 0.05 Hz high-pass filter. A bandpass filtered 0.5–35 Hz was used to clean the ECG signal noise (Howells et al., 2014). Heart rate was extracted by the auto-threshold detect method after removing the baseline. As shown in Figure 1, we defined two areas of interest (AOI) to assess participants’ fixations: (1) the road part which included the middle part of the driving simulator screen and two side rearview mirrors (size 500*550 mm); (2) the tablet (size 248*179 mm), on which the NDRT was presented. Outcome variables were monitoring frequency (i.e., percentage of fixations in the road AOI) and monitoring ratio (i.e., percentage of time spent viewing the road AOI).

To eliminate the individual differences in the SC and HR, it was standardized based on max-min normalization (Dawson et al., 2016). The maximum refers to the driver’s maximum during the takeover transition and the minimum refers to the subject’s minimum during the resting baseline. Meanwhile, gaze behaviors and subjective trust ratings across participants were normalized by the Z-score (Bernison et al., 2007). The pre-processing of the SC and HR was conducted by Biopac Acqknowledge software (Biopac systems Inc, CA). The pre-processing of gaze behaviors was performed in MATLAB. As shown in Table 2, we calculated the statistical measures of the above-mentioned physiological parameters using the time window of the takeover process. A total of 17 physiological parameters was obtained. The feature abbreviation adopts the name_max/mean/med, etc.

TABLE 2:

Descriptions of Produced Features (HR = heart Rate; SC = Skin conductance; min = minimum; max = maximum; standard deviation = SD; Median = med)

Features (Type^a)	Unit	Category	Explanations
SC/Phasic SC (con)	Microsiemens	SC	Max, SD, med, and mean of the SC; △SC: Max change in SC (relative to the initial value of a takeover stage); max of the phasic SC;
SC/Phasic SC (con)	(Microsiemens) ^2/Hz		Max of SC power spectral density; max of phasic SC power spectral density
First-order difference indices of SC (con)	Microsiemens/s		Max, SD, med, and mean of the SC for the first-order difference
HR indices (con)	Beat per minute	HR	Max, mean, SD of the HR
Monitoring behaviors (con)	Millimeter	Gaze behaviors	Monitoring frequency; monitoring ratio
Scenario types (cat)		Takeover-related factors	Stationary vehicle (encoded as 1); roadwork (encoded as 2); dynamic truck (encoded as 3)
TOR lead time (cat)	Seconds		4 s or 8 s lead time
Takeover frequencies (con)			First, second, third, fourth, fifth, and sixth takeover

Type ^a: con = continuous variable; cat = categorical variable.

Data Description

The takeover process stage started with a “TOR” and ended when drivers negotiated the takeover mission and reactivated the AV system (see Figure 5). We showed the physiological data during the takeover transition and the monitoring behaviors before and after takeover in this subsection.

Data from three participants were excluded because of the malfunctions of the physiological sensors. The physiological parameter distributions (i.e., SC and HR) of 31 subjects during TTs are shown in Figure 6a and b, respectively. In general, drivers’ SC and HR distribution are relatively scattered, which do not follow normality through viewing their QQ plots.

Two drivers’ fixation time distributions before and after takeovers are shown in Figure 7. Driver’s fixation time of road after experiencing TORs has changed. The changing trend may highly depend on the external driving environment. For example, the fixations on road for subject 5 were unchanged after experiencing the first TOR (Figure 7(a)). However, after experiencing the second TOR, this driver’s fixations on road significantly increased.

Ground Truth and Statistical Analyses

We assume that driver trust remains unchanged within 1 minute when the AVs work in a normal state. If a driver experiences one TOR or a system error, the trust level may change suddenly, as shown in Figure 8. To measure drivers’ trust responses to TORs accurately, we combined the changes in the subjective trust rating and monitoring behavior before and after takeovers to obtain the trust-change direction during TTs by a commonly unsupervised learning method: the K-means clustering (Guo and Fang, 2013).

A GLMM was implemented by the SPSS version 25 to examine the effects and relationships between takeover-related factors, drivers’ physiological responses, and the trust-change direction during TTs (i.e., trust increasement or trust reduction). Besides, the bootstrapping method (Hayes, 2012) was used to test the mediating effect of the takeover criticality on the relationship between scenario types and the trust-change directions.

Model Development and Evaluation

Drivers’ physiological parameters and takeover-related factors were used to establish an identification model of the trust-change directions during TTs through the random forest (RF) approach. The model was trained with the RF algorithm for the following reasons. First, the RF as an ensemble method is insensitive to overfitting and provides high accuracy results (Yu et al., 2019b). Second, it can still maintain good prediction accuracy, even if the many input data are missing or noisy (Yu et al., 2019a). Finally, it can also give us the importance of features and make the model interpretable. As illustrated in Table 3, four other widely used machine learning methods mentioned were applied for comparisons, including support vector machine (SVM), decision tree (DT), k-nearest neighbors (KNN), and Naive Bayes (NB).

TABLE 3:

Machine Learning Algorithm and Training Process

Machine Learning Methods	Techniques	Hyper-Parameters
Random forest (RF)	Train the standard decision tree algorithm with a set of bootstrapping samples, that is, randomly select training samples with replacement. Repeat many times to form a forest (Breiman, 2001).	Tree number, tree depth
Support vector machine (SVM)	Project the data to low-dimensional space and find the maximum soft margin that separates the classes (Noble, 2006)	Kernel, Kernel coefficient regularization parameter
Decision tree (DT)	Recursive selection of optimal features. Using one or several decision rules to perform the classification (Quinlan, 1986)	Tree depth
K-nearest neighbors (KNN)	Calculate Euclidean distance between unlabeled and labeled points to find K-nearest neighbors. Use classification rules to determine the unlabeled points (Keller et al.,1985)	K
Naive Bayes (NB)	Use maximum likelihood estimation to estimate prior probability and likelihood (Rish, 2001)	/

As mentioned above, each driver experienced six takeover situations, so a total of 186 samples are available (31 subjects). Considering the challenge of human behavior data collection, a 10-fold nested cross-validation method was used to train models and compare test results (Lee et al., 2013; Du et al., 2020b). The 9-fold training set was used to tune the hyper-parameters with the inner loop and generate classifiers. The model evaluation was based on the remaining 1-fold testing set. With 10-fold cross-validation, all the data samples in the dataset only appeared once in the test dataset. The average F1-score and accuracy of the 10-folds were used to evaluate the model performance.

Results

The result section mainly includes four parts. Section 3.1 demonstrated the clustering result of the trust-change direction during TTs. Section 3.2 showed the effects and relationships between takeover-related factors, physiological responses, and the trust-change direction during takeover missions. Furthermore, the mediating effect analysis of the takeover criticality on the trust-change direction was also shown in section 3.3. Finally, section 3.4 presented the result of the identification model for the trust-change direction.

K-Means Clustering

Consistent with previous studies (Hergeth et al., 2016; Körber et al., 2018b), the subjective trust rating is also significantly correlated with the monitoring behaviors (i.e., monitoring ratio and the monitoring frequency) in this study, with correlation coefficients of −0.35 and −0.39. To mitigate the potential bias of the self-reported trust rating, we used a hybrid measurement that combined the differences in the subjective rating and monitoring behavior before and after takeovers to better represent the changes in driver trust during TTs. The Spearman correlation analysis showed that the correlation coefficient between the monitoring ratio and the monitoring frequency is 0.90 (p < 0.001), indicating multicollinearity between the two features. Thus, only the monitoring frequency was selected as the objective measurement of driver trust since it is stronger correlated with trust.

The proposed trust measurement way and only the subjective rating way both used the K-means method to obtain the ground truth of the trust-change directions during TTs. Among all numbers of clusters varying from two to nine, two clusters had the largest average silhouette coefficient with 0.45 and 0.75 in the two measurement ways, and therefore the trust-change direction was classified into two groups (i.e., the trust-increasing and trust-decreasing groups). For the proposed measurement way, as shown in Figure 9a, the trust-increasing group showed a decreasing trend in monitoring frequency (Mean = −1.92%, SD = 3.11%) and nearly remained unchanged in subjective ratings (Mean = 0.34, SD = 6.88). The trust-decreasing group had an increasing trend in monitoring frequency (Mean = 3.76%, SD = 4.16%) and a reducing trend in self-reported trust ratings (Mean = −12.20, SD = 10.73). For the subjective measurement way, the changes in the self-reported trust rating were similar to the result of our proposed clustering way (Figure 9b). It was almost unchanged (Mean = 0.32, SD = 5.18) in the trust-increasing group, and a drop (Mean = −15.59, SD = 10.03) in the trust-decreasing group.

Compared to the subjective clustering result, the samples of trust decreasing were increased by 22 (Table 4). It was indicated that, after integrating subjective data and the monitoring frequency, the classification result of the trust-change directions did alter.

TABLE 4:

Sample Distributions of the Trust-Change Directions During TTs in Two Clustering Ways

Trust-Change Directions	Subjective and Objective	Subjective	Sample Sizes for Changed Direction Change
Trust-Change Directions	Sample Sizes	Sample Sizes	Sample Sizes for Changed Direction Change
Trust decreasing	103	81	29
Trust increasing	83	105	7

To explore whether individual and situational biases of the subjective ratings can be reduced by adding the monitoring frequency into self-reported trust ratings. First, the relations among the trust-change directions, physiological parameters, and takeover-related factors were examined by the Spearman analysis. The physiological responses and takeover-related factors with a P-value below 0.05 are illustrated in Table 5. The statistical significances of their relationship were the same in the two measurement ways. However, compared to the baseline method, the correlation coefficients between the trust-change directions and the HR_SD, TOR lead time, and takeover frequencies, in our proposed way, were increased by 11.1%, 31.0%, and 43.8%, implying that the hybrid method can improve the relations among them.

TABLE 5:

Correlation Analysis Results Between the Trust-Change Directions and Physiological Parameters, Takeover-Related Factors in Two Clustering Ways (N = 186)

Dependent Variable	Features Abbreviation	Correlation Coefficients	Features Abbreviation	Correlation Coefficients	Features Abbreviation	Correlation Coefficients
DV1	SC_max	−0.22**	SC_mean	−0.16*	SC_med	−0.17*
	HR_SD	−0.20**
	ST	−0.21**	TORLT	0.38***	TF	0.23**
DV2	SC_max	−0.22**	SC_mean	−0.16*	SC_med	−0.17*
	HR_SD	−0.18*
	ST	−0.21**	TORLT	0.29***	TF	0.16*

Note: The feature abbreviation adopts the name _max/mean/median, etc.; DV1 = Dependent variable of the proposed way; DV2 = Dependent variable of the subjective clustering way; Scenario types = ST; TOR lead time = TORLT; Takeover frequencies = TF. The following symbol for all the figures and tables applicable were used: *p < 0.05; **p < 0.01; ***p < 0.001.

And then, the samples with alteration in the direction of trust-change were further analyzed by crosstab to examine the distribution difference of takeover-related factors (i.e., scenario types and TOR lead time) in the trust-increasing and trust-decreasing groups. As shown in Figure 10, the crosstab analysis result illustrated that the TOR lead time of 4 s emerged 18 times in the trust-decreasing group, accounting for 62.1% of the group, which was higher than the TOR lead time of 8 s (37.9%). It is consistent with the results of previous studies that the TOR lead time is positively associated with driver trust.

In summary, these results may be provided with suggestive pieces of evidence, indicating that a combination of the subjective rating and objective measure (i.e., the monitoring frequency) can improve the classification results of the trust-change directions.

Besides, drivers’ physiological responses to different TOR lead time conditions (i.e., 4 s and 8 s) were plotted in Figure 11. Both the SC_med and HR_SD of 4 s TOR lead time were higher than those of 8 s TOR lead time (p < 0.1, p < 0.05).

Mixed Model Analysis

To increase the stability of the GLMM, physiological features were divided into three fragments using the quartile, representing the three levels of low (first quartile), medium (second–third quartile), and high (fourth quartile), respectively. And then, all variables in Table 5 are regarded as input variables for the GLMM, using the stepwise method to remove the insignificant variables and obtain the final model.

Table 6 presents the GLMM estimation results of the effects and relationships between takeover-related factors, drivers’ physiological responses, and the trust-change directions during TTs, including estimates, standard errors, and t-value (F (8,177) = 10.43, p < 0.001). The table indicates that takeover-related factors (i.e., scenario types, TOR lead time, and takeover frequencies) have significant effects on the trust-change directions. Meanwhile, there are significant relationships between the trust-change directions and physiological parameters (i.e., SC, HR) during TTs. The regression coefficient for the GLMM indicates the magnitude of each variable on the possibility of the response: a positive number represents that the driver is more likely to increase trust during the takeover transition, and a negative number represents that the driver is more likely to decrease trust.

TABLE 6:

Estimation Result of the GLMM

Parameters	Estimate	95% CI			SE	t-value		Odds Ratio	P-Value
Intercept	0.32	−0.95	1.58		0.64	0.49		1.38
Takeover-related factors
Dynamic truck	−1.69	−2.51	−0.87		0.42	−4.05		0.18	***
Roadwork	−1.59	−2.46	−0.71		0.45	−3.57		0.20	***
Stationary vehicle	_^a	_^a	_^a		_^a	_^a		_^a	_^a
8 s TOR lead time	2.02	1.36	2.69		0.34	6.00		7.56	***
4 s TOR lead time	_^a	_^a	_^a		_^a	_^a		_^a	_^a
Takeover frequencies	0.31	0.14	0.47		0.08	3.66		1.36	***
Physiological parameters
Medium SC_med	−1.12	−1.91	−0.34		0.40	−2.83		0.33	**
Low SC_med	_^a	_^a	_^a		_^a	_^a		_^a	_^a
High HR_SD	−1.53	−2.63	−0.44		0.56	−2.76		0.22	**
Medium HR_SD	−1.17	−1.96	−0.37		0.40	−2.89		0.31	**
Low HR_SD	_^a	_^a	_^a		_^a	_^a		_^a	_^a
Goodness-of-fit of the model
df	AIC			BIC			Log-likelihood
8	890.17			896.46			−886.46

Note: Standard error = SE; Confidence interval = CI.

^aReference category.

Results of Takeover-Related Factors

The significant main effect of the scenario types on the trust-change directions during TTs was found through the GLMM analysis (F (2,177) = 11.19, p < 0.001). The estimated marginal mean (EMM) of the likelihood of trust elevation during TTs is 72.7%, 35.3%, and 33.0% in the stationary vehicle, the roadwork, and the dynamic truck scenarios (Figure 12a). What is more, compared to the reference category, the odds of trust increasing during TT are reduced by 80% and 82% in the roadwork and dynamic truck scenarios (Table 6). It was implied that drivers were more likely to increase trust when dealing with the takeover situation of the stationary vehicle.

The GLMM revealed that the TOR lead time significantly affects the trust-change directions (F (1,181) = 36.05, p < 0.001). Figure 12b shows that the EMM of the likelihood of increased trust during the takeover transition is 24.5% and 71.1% in the TOR lead time of 4 s and 8 s, and their difference is 46.6% (p < 0.001). As demonstrated in Table 6, compared with 4 s TOR lead time, the odds of the trust elevation during the TTs are increased by 7.56 times in 8 s TOR lead time. It was indicated that the TOR lead time is a good predictor variable for trust-change directions during TTs.

Furthermore, as shown in Table 6, the regression coefficient of takeover frequencies is a positive value, indicating that the probability of increased trust during TTs is elevated with the increment of the takeover frequencies. Specifically, each increment in the number of takeovers increases the odds of trust elevation by 36% during the takeover transition. Any interactions were not found between the takeover-related factors.

Results of Physiological Parameters

For the skin conductance, only a significant relationship between the SC_median and trust-change directions was found through the GLMM analysis (F (2,177) = 4.29, p < 0.05). Figure 13a shows that the EMM of the likelihood of increased trust is 60.1% and 32.9% in the low and the medium SC_median. The difference in the EMM of the likelihood of increased trust in the low SC_median is 27.2%, compared with the medium SC_median ( p < 0.01). In addition, when drivers were in the medium SC_median, the odds of increased trust were reduced by 67% compared with drivers in the low SC_median. The interaction effects of the SC were not found (Table 6).

Besides, the GLMM analysis result showed that the trust-change direction is significantly related to the driver’s HR (F (2,177) = 5.00, p < 0.01). As demonstrated in Figure 13b, the EMM of the likelihood of increased trust is 68.7% in the low HR_SD and is significantly higher than 40.7% and category, the odds of trust elevation are reduced by 78% and 69% in the medium and high HR_SD (Table 6).

The Mediating Effect of the Takeover Criticality

To confirm whether the influence of takeover scenario types (i.e., stationary vehicle, dynamic truck, and roadwork) on the trust-change directions could be implemented by the mediating effect of the takeover criticality. As shown in Figure 14, after controlling for the TOR lead time and takeover frequencies, the bootstrapping method (Hayes, 2012) was used to test the mediating effect of takeover criticality, which assesses the significance through the range of the bootstrap confidence interval. If the range includes zero, the relationship is not significant.

As shown in Table 7, taking the stationary vehicle scenario as the reference group, the 95% confidence intervals of the roadwork and dynamic truck scenarios relative to the reference group are [−0.456, −0.004] and [−0.698, −0.060], respectively, implying that their relative mediation effects are both significant. Specifically, compared with the reference group, the takeover criticality ratings were increased by 0.473 and 0.882 when drivers experienced the roadwork and dynamic truck scenarios, and then the odds of their trust elevation during TTs correspondingly decreased. Besides, their relatively direct effects are both significant (p < 0.01, p < 0.05), showing that the takeover criticality partially mediated the influence of the roadwork and dynamic truck scenarios on the trust-change directions, compared to the reference group.

TABLE 7:

The Mediating Effect of Scenario Types on the Trust-Change Directions via the Takeover Criticality

IV	M	DV	CF	Effect of IV on M(a)	Effect of M on DV(b)	Direct Effects (c’)	Bootstrapping Results in Indirect Effects
IV	M	DV	CF	Effect of IV on M(a)	Effect of M on DV(b)	Direct Effects (c’)	(a*b)	a*b (95% CI)
RW	TC	TCD	TORT; TF	0.473*	−0.350*	−1.132**	−0.166^d	(−0.456, −0.004)
DTK	TC	TCD	TORT; TF	0.882***	−0.350*	−1.038*	−0.309^d	(−0.698, −0.060)

Note: IV = independent Variable; M = mediating variable; DV = dependent variable; CF = control factors; roadwork = RW; dynamic truck = DTK; takeover criticality = TC; trust-change directions = TCD; TOR lead times = TORT; takeover frequencies = TF; confidence interval = CI; d = Significant mediation effect.

Recognition Models of the Trust-Change Direction During Takeover Transitions

Based on physiological features and takeover-related factors in Table 6, the identification models for the direction of trust-change during TTs were established by the RF method and four other machine learning approaches, which were implemented using the packages of the sklearn library. To improve the robustness of machine learning results, the 10-fold cross-validation was run 50 times (i.e., 50 different random seeds) for every machine learning method (Du et al., 2020b). Besides, Wilcoxon’s Signed-Rank test was used to compare the performance of five machine learning models and different feature subset models (Japkowicz and Shah, 2011). Since we compared the performance of the RF model to the other four machine learning models, and the full feature model to other subset feature models-resulting in four and two statistical tests, respectively, the Bonferroni correction was used to counteract the increased probability of Type Ⅰ, which adjusted the significance level to $α =$ 0.0125 and $α =$ 0.025.

Model Performance Comparisons

Table 8 shows the comparison results of the accuracy and F1-score of the random forest with the other four models by Wilcoxon’s Signed-Rank test. It was indicated that the accuracy, F1-score, and stability (i.e., the SD of accuracy and F1-scores) of the RF model both outperformed the other four models. Besides, Figure 15 shows the receiver operating characteristic (ROC) curves of the RF and the other four machine learning algorithms. The ROC curve of the RF model is above and to the left of the SVM, KNN, and DT models at most of the thresholds. However, there was no difference between the ROC curve of the RF and NB model. Comprehensive considering the accuracy, F1-scores, model stability, and ROC curve, the RF model may be more suitable for establishing the model of the trust-change direction during TTs.

TABLE 8:

The Recognition Accuracy and F1-Score of Machine Learning Approaches and their Comparisons to the Random Forest Model (10-Fold)

Algorithm	Accuracy			F1-Scores
Algorithm	Mean	SD	T-test (*p < 0.0125)	Mean	SD	p-value (*p < 0.0125)
RF	0.737	0.010	-	0.773	0.008	-
NB	0.730	0.012	P = 0.0008*	0.761	0.011	P = 0.0000*
SVM	0.716	0.016	P = 0.0000*	0.750	0.016	P = 0.0000*
KNN	0.695	0.013	P = 0.0000*	0.7316	0.011	P = 0.0000*
DT	0.667	0.024	P = 0.0000*	0.711	0.025	P = 0.0000*

The Confusion Matrix and Feature Importance

Figure 16a demonstrates the confusion matrix of the RF model. The recall is 80.6% and the precision is 74.1%, showing that the model has better completeness compared to the exactness. By randomly permuting one feature in the out-of-bag data (i.e., one-third of the samples are not in the bootstrap data), and measuring how much this arrangement reduces the accuracy of the model. We represent the feature importance through the proportion of accuracy decline.

Figure 16b illustrates the estimated result of all variables’ feature importance using the out-of-bag estimate method. The most important is the TOR lead time followed by the takeover frequencies. Besides, the scenario types and the standard deviation of the HR are also important features for identifying the trust-change directions during TTs with the feature importance value of 0.06 and 0.05. Finally, we found that the median of the SC also contributed to the model performance.

Besides, the identification accuracy of the RF model at different takeover scenario types is also shown in Figure 17, which is 62.9%, 74.2%, and 83.9% in the roadwork, stationary vehicle, and dynamic truck scenarios.

Effects of Features on the Random Forest Model Result

The accuracy and F1-score for the RF algorithm using the full feature set were higher than using other feature subsets (Figure 18 and Table 9). Specifically, if only the features of physiological parameters (i.e., SC_med and HR_SD) were used, the prediction accuracy and F1-score were 0.608 and 0.670. If only takeover-related factors (i.e., TOR lead time, scenario types and takeover frequencies) were used, the prediction accuracy and F1-score were 0.696 and 0.736. This suggested that combining the environmental aspects with physiological features reflecting the driver’s internal mental state is necessary for the establishment of a high-performance model. The mean accuracy and F1-score of the model using takeover-related factors and physiological parameters as input features increased to 0.737 and 0.773.

TABLE 9:

Random Forest Recognition Accuracy and F1-Score With Different Feature Subsets and their Comparisons to the Full Feature Model

Feature Subsets	Accuracy			F1-Scores
Feature Subsets	Mean	SD	T-test (*p < 0.025)	Mean	SD	T-test (*p < 0.025)
All	0.737	0.010	-	0.773	0.008	-
Takeover-related	0.696	0.012	P = 0.000*	0.736	0.012	P = 0.000*
Physiological	0.608	0.013	P = 0.000*	0.670	0.012	P = 0.000*

Discussion

This paper aimed to propose an objective method to measure and identify the trust-change direction during TTs in conditionally AV, mainly including three main aspects: (1) combined the change values in the monitoring behavior and subjective ratings before and after takeovers to estimate the trust-change direction during TTs; (2) explored the relations among takeover-related factors (i.e., the TOR lead time, takeover frequencies, and scenario types), drivers’ physiological responses (i.e., the SC and HR), and trust-change directions during TTs; and (3) developed an RF model to identify the trust-change direction during TTs, which is based on the multimodal physiological signals and takeover-related factors.

Discussion in Trust Measurements

Previous studies used mostly subjective ratings to measure trust in automation, but these self-reported trust ratings have some limitations: (1) they cannot provide continuous information for real-time estimation of driver trust (Walker et al., 2019) and (2) they always have biases from different drivers and driving situations (Rosenman et al., 2011). To alleviate the problem of relying only on self-reported ratings that may not accurately reflect the changes in driver trust during TTs, the differences in the monitoring behavior and subjective rating before and after the takeovers were combined to measure and classify the trust-change direction. The trust-change direction (i.e., trust increment and reduction) during TTs was obtained by the K-means clustering algorithm. Our results showed that integrating subjective ratings and monitoring behaviors can change the classification of the trust-change direction and achieve more objective and reliable trust-change measurements. This is consistent with the result of a study on trust measurement for advanced driver assistance systems (Yu et al., 2021), and they found that a combination of self-reported trust ratings and system usage frequencies can more accurately measure driver trust in this system.

Discussion in Statistical Analyses

Takeover-Related Factors

As expected, the analysis result of the GLMM showed that drivers were more likely to increase trust during TTs when they experienced a longer TOR lead time level. The result might be explained that the longer TOR lead time can reduce the difficulty and risk of takeover missions, and then affect situational trust (Hoff and Bashir, 2015). This result is consistent with the previous study (Jin et al., 2020). We also observed that familiarity with the takeover increased the odds of trust elevation during TTs. It may be associated with experiencing multiple takeover missions that can help drivers understand the purpose and process of the automated driving system, and then affect the learned trust of drivers (Hoff and Bashir, 2015). This matches with the results of research by Hergeth et al. (2017) and Körber et al. (2018c).

The GLMM analysis also revealed that drivers were more likely to decrease trust when dealing with roadwork and dynamic scenario than the stationary vehicle situation. Furthermore, the mediating effect analysis indicated that the takeover criticality rating increased when drivers experienced the roadwork and dynamic truck scenario, compared with the stationary vehicle situation, and then the odds of their trust elevation during TTs correspondingly were decreased. Consistent with previous studies, the roadwork and dynamic truck scenarios had a higher takeover criticality rating than the stationary vehicle situation (Vogelpohl et al., 2018; Yi et al., 2022), and then drivers were more likely to reduce trust ratings in critical situations (Morris et al., 2017; Li et al., 2019; Azevedo-Sa et al., 2021).

Physiological Parameters

For the SC, it was found that drivers were more likely to increase trust during TTs in the low SC_med, compared to the medium SC_med through the GLMM analysis. This is consistent with the research results of Khawaji et al. (2015) and Morris et al. (2017), which reported that trust increased with the reduction of the SC. However, we found that when drivers were in the low and high SC_med, there was no significant difference in their odds of trust increasement during TTs. One possible reason for this result is that the high SC level is only associated with emotional arousal and cannot distinguish emotional valences such as excitement or stress (Lang et al., 1993). Drivers may be excited or happy that they can safely handle the takeover mission, which may increase driver trust during TTs. This in turn may obscure the negative correlation between the SC and trust-change directions. Besides, the GLMM analysis result also suggested that the trust-change direction is negatively correlated with the standard deviation of the HR during TTs. This may be attributed to the increased stress on drivers resulting from trust reduction during TTs (Mühl et al., 2020), which then leads to the acceleration of stress-mediated HR (Healey and Picard, 2005; Rastgoo et al., 2018).

Discussion in the Trust-Change Direction Model During Takeover Transitions

Compared with the other four popular machine learning approaches (e.g., SVM, DT, and KNN), the performance of the RF method seems to outperform the other classification methods in identifying the trust-change direction during TTs. This is lined with prior studies on recognizing driver trust in ADAS and predicting the takeover performance (Du et al., 2020b; Yu et al., 2021). The RF approach also shows its supremacy in identifying the trust-change direction during TTs. This may be associated with the advantages of bagging and bootstrapping techniques, which reduce the effects of overfitting and improve generalization.

Our results showed that the model performance with the full set of features is improved compared to partial features (i.e., physiological data only and takeover-related data only). One possible reason for this result is that the data from different modalities can reflect the same change of trust and each modality carries trust-related information. Besides, there is some trust-related information across the modalities that can provide complementary information to improve the model performance of the trust-change directions. For example, full physiological data reflect the internal mental state of drivers and interactions with driving environments, such as stress, which may be related to the change in driver trust (Morris et al., 2017; Mühl et al., 2020). Thus, fusing multimodal data can improve the identified performance of the model. Using vehicle sensors and advanced wearable technology, the above features can be recorded in a minimally invasive way to identify the trust-change directions during TTs in conditionally AVs.

Limitations and Future Work

Several limitations of this study need consideration and should be addressed in future research. First, as only the influences of the TOR lead time and takeover scenario type on driver trust were examined, the study merely covered a small number of possible takeover situations when drivers were engaged with a visual Surrogate Reference Task in low traffic density. Further study could be extended to exploring the impacts of these two factors on driver trust when they are performing different types of NDRT (e.g., reading and proofreading text) or encountering different levels of traffic density (e.g., low and high traffic densities). Second, the study was conducted with a fixed-base driving simulator in a controlled laboratory, and the research samples comprised mainly young males. The physiological signals of drivers may be different when they are of diverse ages and drive in a real-world environment, as the physiological signals collected are affected by the combination of some factors, such as temperature, and age (Gavazzeni et al., 2008; Doberenz et al., 2011). Future studies can recruit diverse drivers in a natural driving environment to increase the generalization of models. Finally, this study used the statistics of the physiological signals of drivers during the takeover period as the model input and did not consider the sequence dependence among the time-series data. Future studies could try to use long-short-term memory (LSTM) or a combination of convolutional neural networks and LSTM to identify the trust-change directions during TTs.

Application Potential

Those findings of the takeover-related factors revealed two layers of variability in driver-vehicle trust (situational trust and learned trust), which may provide additional support for the concept design of the driver-vehicle interface in conditionally AV. And the model of the trust-change directions may provide implications for developing trust monitoring systems in SAE level 3 AVs to mitigate both overtrust and undertrust. For example, if the model detects that the driver loses trust after experiencing TTs, the SAE level 3 AVs can foster better driver-vehicle trust by proactively communicating (Collet & Musicant, 2019). Besides, the physiological parameters can be applied to not only identify the trust-change direction during TTs but also monitor the internal mental state of drivers by receiving continuous and real-time physiological signals. Based on the monitoring result, a mild onboard environment can be offered to drivers when they need to relieve the sub-optimal mental state caused by the takeover transition to improve the driver’s experience (Chung et al., 2019).

Conclusions

The study provides a unique perspective for objectively and reliably measuring and identifying the trust-change directions during TTs. A combination of the change in the monitoring behavior and subjective trust rating before and after takeovers can reliably estimate the changes of trust during TTs in conditionally AVs, which may help to reduce the biases of using only subjective evaluations.

Besides, the present study has examined the effects and relationships between takeover-related factors, drivers’ physiological responses, and trust-change directions during TTs. Drivers are more likely to increase trust during the takeover transition when they are in the longer TOR lead time condition, with more takeover frequencies and dealing with the stationary vehicle scenario. The change in driver trust during the takeover process has negative correlations with the standard deviation of the HR and the median of the SC.

More importantly, a recognition model of the trust-change direction during TTs is established by the RF algorithm, based on the driver’s multimodal physiological data (i.e., the SC and HR) and takeover-related factors (i.e., TOR lead time, takeover frequencies, and scenario types). The identification performance of the RF classifier has an F1-score of 77.3% and an accuracy of 73.7%.

KEY POINTS

Combining the changes in the subjective trust rating and monitoring frequency before and after takeovers can objectively and reliably measure the trust-change direction during takeover transitions.

The influence of scenario types on the trust-change direction during takeover transitions can be implemented by the mediating effect of the takeover criticality.

The present results indicate negative connections between drivers’ physiological parameters (i.e., the SC and HR) and the trust-change direction during takeover transitions.

The study finds, that fusing driver’s physiological data and takeover-related factors (i.e., TOR lead time, takeover frequencies, and scenario types), can achieve a relatively accurate classification for the trust-change direction during TTs, with an F1-scores of 77.3%.

Footnotes

Acknowledgments

This work was supported by the National Natural Science Foundation of China under grant number 51975194, the Natural Science Foundation of Hunan Province under grant number 2021JJ30121, and the State Key Laboratory of Automotive Safety and Energy under Project No. KFZ2203. We sincerely thank constructive comments and valuable suggestions from anonymous reviewers.

Author Contributions

Binlin Yi: Methodology, Data curation, Formal analysis, Writing original draft. Haotian Cao: Writing—review & editing. Xiaolin Song: Conceptualization, Project administration, Funding acquisition. Jianqiang Wang: Writing—review & editing. Song Zhao: Writing—review & editing. Wenfeng Guo: Visualization, Investigation. Dongpu Cao: Software, Validation.

Ethical Approval

This study was approved by the Institutional Review Board in the college of mechanical and vehicle engineering at Hunan University.

Binlin Yi received his Master’s degree in Mechanical Engineering at the Hunan University. He is currently a Ph.D. candidate in Mechanical Engineering from the Hunan University.

Haotian Cao received a PhD degree in Mechanical Engineering from the Hunan University. He is currently a Research Associate in Hunan University.

Xiaolin Song received the PhD degree in mechanical engineering from the Hunan University. She is consecutively a Professor with Hunan University.

Jianqiang Wang received the PhD degree in vehicle application engineering from Jilin University. He is currently a Professor with Tsinghua University.

Song Zhao received the M.Eng. degree in global automotive manufacturing engineering from the University of Michigan-Ann Arbor. He is currently pursuing the PhD degree from University of Waterloo.

Wenfeng Gou received the B.E. degree in design and manufacture of machinery and automation from Hunan University. He is currently a Ph.D. candidate in Mechanical Engineering from the Hunan University.

Dongpu Cao received a PhD degree from Concordia University. He is currently an Associate Professor with University of Waterloo.

References

Abe

Sato

Itoh

(2018). Driver trust in automated driving systems: The case of overtaking and passing. IEEE Transactions on Human-Machine Systems, 48(1), 85–94. https://10.1109/THMS.2017.2781619.

Ayoub

Yang

X. J.

Zhou

(2021). Modeling dispositional and initial learned trust in automated vehicles with predictability and explainability. Transportation research part F: traffic psychology and behaviour, 77, 102–116. https://doi.org/10.1016/j.trf.2020.12.015

Azevedo-Sa

Jayaraman

S. K.

Esterwood

C. T.

Yang

X. J.

Robert

L. P.

Tilbury

D. M.

(2020). Real-time estimation of drivers’ trust in automated driving systems. International Journal of Social Robotics, 13, 1–17. https://doi.org/10.1007/s12369-020-00694-1.

Azevedo-Sa

Zhao

Esterwood

Yang

X. J.

Tilbury

D. M.

Robert

L. P.

Jr (2021). How internal and external risks affect the relationships between trust and driver behavior in automated driving systems. Transportation research part C: emerging technologies, 123, 102973. https://doi.org/10.1016/j.trc.2021.102973

Breiman

(2001). Random forests. Machine learning, 45(1), 5–32. https://doi.org/10.1023/A:1010933404324

Bernison

G. G.

Quigley

K. S.

Lozano

(2007). Cardiovascular psychophysiology. In Cacioppo

J. T.

Tassinary

L. G.

Berntson

G. G.

(Eds.) Handbook of Psychophysiology (pp. 182–210). Cambridge: Cambridge University Press.

Choi

J. K.

Y. G.

(2015). Investigating the importance of trust on adopting an autonomous vehicle. International Journal of Human-Computer Interaction,31(10), 692–702. https://doi.org/10.1080/10447318.2015.1070549

Chung

W. Y.

Chong

T. W.

Lee

B. G.

(2019). Methods to detect and reduce driver stress: a review. International journal of automotive technology, 20(5), 1051–1063. https://doi.org/10.1007/s12239-019-0099-3

Collet

Musicant

(2019). Associating vehicles automation with drivers functional state assessment systems: A challenge for road safety in the future. Frontiers in human neuroscience, Neurosci, 13, 131. https://doi.org/10.3389/fnhum.2019.00131

10.

Dawson

Schell

Filion

(2016). The Electrodermal System. In Cacioppo

Tassinary

Berntson

(Eds.) Handbook of Psychophysiology (pp. 217–243). Cambridge: Cambridge University Press. https://doi.org/10.1017/9781107415782.010

11.

Doberenz

Roth

W. T.

Wollburg

Maslowski

N. I.

Kim

(2011). Methodological considerations in ambulatory skin conductance monitoring. International Journal of Psychophysiology, 80(2), 87–95. https://doi.org/10.1016/j.trf.2020.12.015

12.

Yang

X. J.

Zhou

(2020a). Psychophysiological responses to takeover requests in conditionally automated driving. Accident Analysis & Prevention, 148, 105804. https://doi.org/10.1016/j.aap.2020.105804

13.

Zhou

Pulver

E. M.

Tilbury

D. M.

Robert

L. P.

Pradhan

A. K.

Yang

X. J.

(2020b). Predicting driver takeover performance in conditionally automated driving. Accident Analysis & Prevention, 148, 105748. https://doi.org/10.1016/j.aap.2020.105748

14.

Dupre

Vincent

Iaizzo

P.A.

(2005). Basic ECG Theory, Recordings, and Interpretation. In: Iaizzo

P.A.

(eds) Handbook of Cardiac Anatomy, Physiology, and Devices. Humana Press. https://doi.org/10.1007/978-1-59259-835-9_15

15.

Ekman

Johansson

Bligård

L. O.

Karlsson

Strömberg

(2019). Exploring automated vehicle driving styles as a source of trust information. Transportation research part F: traffic psychology and behaviour,65, 268–279. https://doi.org/10.1016/j.trf.2019.07.026

16.

Fagnant

D. J.

Kockelman

(2015). Preparing a nation for autonomous vehicles: opportunities, barriers and policy recommendations. Transportation Research Part A: Policy and Practice, 77, 167–181. https://doi.org/10.1016/j.tra.2015.04.003

17.

Gavazzeni

Wiens

Fischer

(2008). Age effects to negative arousal differ for self‐report and electrodermal activity. Psychophysiology. 45(1), 148–151. https://doi.org/10.1111/j.1469-8986.2007.00596.x

18.

Gold

Körber

Lechner

Bengler

(2016). Taking over control from highly automated vehicles in complex traffic situations: the role of traffic density. Human Factors, 58(4),642–652. https://doi.org/10.1177/0018720816634226

19.

Gold

Naujoks

Radlmayr

Bellem

Jarosch

(2018a). Testing scenarios for human factors research in level 3 automated vehicles. Advances in Intelligent Systems and Computing, 597, 551–559. https://link.springer.com/chapter/10.1007/978-3-319-60441-1_54

20.

Gold

Happee

Bengler

(2018b). Modeling take-over performance in level 3 conditionally automated vehicles. Accident Analysis & Prevention, 116, 3–13. https://doi.org/10.1016/j.aap.2017.11.009

21.

Guo

Fang

(2013). Individual driver risk assessment using naturalistic driving data. Accident Analysis & Prevention, 61, 3–9. https://doi.org/10.1016/j.aap.2012.06.014

22.

Gupta

Hajika

Pai

Y. S.

Duenser

Lochner

Billinghurst

(2019). In ai we trust: Investigating the relationship between biosignals, trust and cognitive load in vr. Proceedings of the 25th ACM Symposium on Virtual Reality Software and Technology, 1–10. https://doi.org/10.1145/3359996.3364276

23.

Hayes

A. F.

(2012). PROCESS: A versatile computational tool for observed variable mediation, moderation, and conditional process modeling. http://www.afhayes.com/public/process2012.pdf.

24.

Healey

J. A.

Picard

R. W.

(2005). Detecting stress during real-world driving tasks using physiological sensors. IEEE Transactions on intelligent transportation systems, 6(2), 156–166. https://doi.org/10.1109/TITS.2005.848368

25.

Hergeth

Lorenz

Krems

J. F.

(2017). Prior familiarization with takeover requests affects drivers’ takeover performance and automation trust. Human Factors, 59(3), 457–470. https://doi.org/10.1177/0018720816678714

26.

Hergeth

Lorenz

Vilimek

Krems

J. F.

(2016). Keep your scanners peeled: Gaze behavior as a measure of automation trust during highly automated driving. Human Factors,58(3), 509–519. https://doi.org/10.1177/0018720815625744

27.

Hoff

K. A.

Bashir

(2015). Trust in automation: Integrating empirical evidence on factors that influence trust. Human Factors,57(3), 407–434. https://doi.org/10.1177/0018720814547570

28.

Holthausen

B. E.

Wintersberger

Walker

B. N.

Riener

(2020). Situational Trust Scale for Automated Driving (STS-AD): Development and Initial Validation. In Proceedings of the 12th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, 40–47. https://doi.org/10.1145/3409120.3410637

29.

Howells

F. M.

Rauch

H. L.

Ives-Deliperi

V. L.

Horn

N. R.

Stein

D. J.

(2014). Mindfulness based cognitive therapy may improve emotional processing in bipolar disorder: pilot ERP and HRV study. Metabolic brain disease, 29(2), 367–375. https://doi.org/10.1007/s11011-013-9462-7

30.

Japkowicz

Shah

(2011). Evaluating learning algorithms: a classification perspective. Cambridge: Cambridge University Press. https://doi.org/10.1017/CBO9780511921803

31.

Jian

J. Y.

Bisantz

A. M.

Drury

C. G.

(2000). Foundations for an empirically determined scale of trust in automated systems. International journal of cognitive ergonomics, 4(1), 53–71. https://doi.org/10.1207/S15327566IJCE0401_04

32.

Jin

Chen

Shi

(2020). How Driving Experience Affect Trust in Automation from Level 3 Automated Vehicles? An Experimental Analysis. Proceedings of the IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), 1–6. https://doi.org/10.1109/ITSC45102.2020.9294247

33.

Keller

J. M.

Gray

M. R.

Givens

J. A.

(1985). A fuzzy k-nearest neighbor algorithm. IEEE transactions on systems, man, and cybernetics, 15(4), 580–585. https://doi.org/10.1109/TSMC.1985.6313426.

34.

Khawaji

Zhou

Chen

Marcus

(2015). Using galvanic skin response (GSR) to measure trust and cognitive load in the text-chat environment. Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems, 1989-1994. https://doi.org/10.1145/2702613.2732766

35.

Kohn

S. C.

De Visser

E. J.

Wiese

Lee

Y. C.

Shaw

T. H.

(2021). Measurement of Trust in Automation: A Narrative Review and Reference Guide. Frontiers in psychology, 12, 604977. https://doi.org/10.3389/fpsyg.2021.604977.

36.

Körber

(2018a). Theoretical considerations and development of a questionnaire to measure trust in automation. Bagnara

Tartaglia

Albolino

Alexander

Fujita

(Eds) Proceedings of the 20th Congress of the International Ergonomics Association, 13–30. https://doi.org/10.1007/978-3-319-96074-6_2

37.

Körber

Baseler

Bengler

(2018b). Introduction matters: Manipulating trust in automation and reliance in automated driving. Applied ergonomics, 66, 18–31. https://doi.org/10.1016/j.apergo.2017.07.006

38.

Körber

Prasch

Bengler

(2018c). Why do I have to drive now? Post hoc explanations of takeover requests. Human Factors, 60(3), 305–323. https://doi.org/10.1177/0018720817747730

39.

Kraus

Scholz

Stiegemeier

Baumann

(2020). The more you know: trust dynamics and calibration in highly automated driving and the effects of take-overs, system malfunction, and system transparency. Human Factors, 62(5), 718–736. https://doi.org/10.1177/0018720819853

40.

Lang

P. J.

Greenwald

M. K.

Bradley

M. M.

Hamm

A. O.

(1993). Looking at pictures: Affective, facial, visceral, and behavioral reactions. Psychophysiology, 30(3), 261–273. https://doi.org/10.1111/j.1469-8986.1993.tb03352.x

41.

Lascurain-Aguirrebeña

Newham

D. J.

Galindez-Ibarbengoetxea

Casado-Zumeta

Lertxundi

Critchley

D. J.

(2019). Association between sympathoexcitatory changes and symptomatic improvement following cervical mobilisations in participants with neck pain. A double blind placebo controlled trial. Musculoskelet. Musculoskeletal Science and Practice, 42, 90–97. https://doi.org/10.1016/j.msksp.2019.05.001

42.

Lee

J. D.

See

K. A.

(2004). Trust in automation: Designing for appropriate reliance. Human Factors, 46(1), 50–80. https://doi.org/10.1518/hfes.46.1.50_30392

43.

Lee

J. J.

Knox

Baumann

Breazeal

DeSteno

(2013). Computationally modeling interpersonal trust.Frontiers in psychology, 4, 893. https://doi.org/10.3389/fpsyg.2013.00893

44.

Holthausen

B. E.

Stuck

R. E.

Walker

B. N.

(2019). No risk no trust: Investigating perceived risk in highly automated driving. Proceedings of the 11th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, 177–185. https://doi.org/10.1145/3342197.3344525

45.

Liu

Akash

Misu

(2021). Clustering Human Trust Dynamics for Customized Real-time Prediction. Proceedings of 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), 1705–1712. https://doi.org/10.1109/ITSC48978.2021.9565016

46.

Maule

A. J.

Hockey

G. R. J.

(1993). State, Stress, and Time Pressure. In: Svenson

Maule

A.J.

(eds) Time Pressure and Stress in Human Judgment and Decision Making. Boston: Springer. https://doi.org/10.1007/978-1-4757-6846-6_6

47.

McBride

S. E.

Rogers

W. A.

Fisk

A. D.

(2010). Do Younger and Older Adults Differentially Depend on an Automated System? Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 54(2),175–179. https://doi.org/10.1177/154193121005400208

48.

Morris

D. M.

Erno

J. M.

Pilcher

J. J.

(2017). Electrodermal Response and Automation Trust during Simulated Self-Driving Car Use. Proceedings of the Human Factors and Ergonomics Society Annual Meeting,61(1), 1759–1762. https://doi.org/10.1177/1541931213601921

49.

Mühl

Strauch

Grabmaier

Reithinger

Huckauf

Baumann

(2020). Get ready for being chauffeured: Passenger’s preferences and trust while being driven by human and automation. Human Factors, 62(8), 1322–1338. https://doi.org/10.1177/0018720819872893

50.

Naujoks

Purucker

Neukum

Wolter

Steiger

(2015), Controllability of partially automated driving functions–does it matter whether drivers are allowed to take their hands off the steering wheel? Transportation research part F: traffic psychology and behaviour, 35, 185–198. https://doi.org/10.1016/j.trf.2015.10.022

51.

Noble

W. S.

(2006). What is a support vector machine? Nature biotechnology, 24(12), 1565–1567. https://doi.org/10.1038/nbt1206-1565

52.

Perello-March

J. R.

Burns

C. G.

Woodman

Elliott

M. T.

Birrell

S. A.

(2021). Driver state monitoring: Manipulating reliability expectations in simulated automated driving scenarios. IEEE Transactions on Intelligent Transportation Systems. https://doi.org/10.1109/TITS.2021.3050518

53.

Petersen

Robert

Yang

Tilbury

(2019). Situational awareness, driver’s trust in automated driving systems and secondary task performance. SAE International Journal of Connected and Autonomous Vehicles, Forthcoming. http://dx.doi.org/10.2139/ssrn.3345543

54.

Quinlan

J. R.

(1986). Induction of decision trees. Machine learning, 1(1), 81–106. https://doi.org/10.1007/BF00116251

55.

Rastgoo

M. N.

Nakisa

Rakotonirainy

Chandran

Tjondronegoro

(2018). A critical review of proactive detection of driver stress levels based on multimodal measurements. ACM Computing Surveys, 51(5), 1–35. https://doi.org/10.1145/3186585

56.

Rish

(2001). An empirical study of the naive bayes classifier. IJCAI 2001 Workshop on Empirical Methods in Artificial Intelligence. 3 (22), 41–46.

57.

Rosenman

Tennekoon

Hill

L. G.

(2011). Measuring bias in self-reported data. International Journal of Behavioural and Healthcare Research, 2(4), 320–332. https://doi.org/10.1504/IJBHR.2011.043414

58.

Scerbo

A. S.

Freedman

L. W.

Raine

Dawson

M. E.

Venables

P. H.

(1992). A major effect of recording site on measurement of electrodermal activity. Psychophysiology, 29(2), 241–246.https://doi.org/10.1111/j.1469-8986.1992.tb01693.x

59.

Schwarz

Gaspar

Brown

(2019). The effect of reliability on drivers’ trust and behavior in conditional automation. Cognition, Technology & Work, 21(1), 41–54. https://doi.org/10.1007/s10111-018-0522-y

60.

Seet

Harvy

Bose

Dragomir

Bezerianos

Thakor

(2020). Differential Impact of Autonomous Vehicle Malfunctions on Human Trust. IEEE Transactions on Intelligent Transportation Systems. https://doi.org/10.1109/TITS.2020.3013278

61.

Society of Automotive Engineers (2018). Taxonomy and Definitions for terms related to on-road motor vehicle automated driving systems for on-road motor vehicles(J3016). SAE International.

62.

Teoh

E. R.

Kidd

D. G.

(2017). Rage against the machine? Google's self-driving cars versus human drivers. Journal of safety research, 63, 57–60. https://doi.org/10.1016/j.jsr.2017.08.008

63.

Vogelpohl

Kühn

Hummel

Gehlert

Vollrath

(2018). Transitioning to manual driving requires additional time after automation deactivation. Transportation research part F: traffic psychology and behaviour, 55, 464–482. https://doi.org/10.1016/j.trf.2018.03.019

64.

Walker

Wang

Martens

M. H.

Verwey

W. B.

(2019). Gaze behaviour and electrodermal activity: Objective measures of drivers’ trust in automated vehicles. Transportation research part F: traffic psychology and behaviour, 64, 401–412. https://doi.org/10.1016/j.trf.2019.05.021

65.

Wandtner

Schömig

Schmidt

(2018). Secondary task engagement and disengagement in the context of highly automated driving. Transportation research part F: traffic psychology and behaviour, 58, 253–263. https://doi.org/10.1016/j.trf.2018.06.001

66.

Waytz

Heafner

Epley

(2014). The mind in the machine: Anthropomorphism increases trust in an autonomous vehicle. Journal of Experimental Social Psychology, 52, 113–117. https://doi.org/10.1016/j.jesp.2014.01.005

67.

Wintersberger

von Sawitzky

Frison

A. K.

Riener

(2017). Traffic augmentation as a means to increase trust in automated driving systems. Proceedings of the 12th biannual conference on italian sigchi chapter, 1–7. https://doi.org/10.1145/3125571.3125600

68.

Cao

Song

Zhao

Guo

(2022). How to identify the take-over criticality in conditionally automated driving? An examination using drivers’ physiological parameters and situational factors. Transportation research part F: traffic psychology and behaviour, 85, 161–178. https://doi.org/10.1016/j.trf.2021.12.007

69.

Bao

Feng

Sayer

(2019a). Examination and prediction of drivers’ reaction when provided with V2I communication-based intersection maneuver strategies. Transportation research part C: emerging technologies, 106, 17–28. https://doi.org/10.1016/j.trc.2019.07.007

70.

Chen

Bao

(2019b). Quantifying visual road environment to establish a speeding prediction model: an examination using naturalistic driving data. Accident Analysis & Prevention, 129, 289–298. https://doi.org/10.1016/j.aap.2019.05.011

71.

Bao

Zhang

Sullivan

Flannagan

(2021). Measurement and prediction of driver trust in automated vehicle technologies: An application of hand position transition probability matrix. Transportation research part C: emerging technologies, 124, 102957. https://doi.org/10.1016/j.trc.2020.102957

72.

Zhang

Tao

Zhang

Lin

Zhang

(2019). The roles of initial trust and perceived risk in public’s acceptance of automated vehicles. Transportation research part C: emerging technologies, 98, 207–220. https://doi.org/10.1016/j.trc.2018.11.018