Understanding Human-Agent Teaming: A Mixed-Methods Examination of Perceptions and Interactions

Participants

Our study included 161 participants (99 male, 61 female, 1 NA) with ages ranging from 18 to 44 (M = 21.59, SD = 3.93) who were recruited through various channels such as personal networks, distributing flyers across a university campus, emails, LinkedIn posts, online forums, and snowball sampling methods. Participants were either compensated with a $25.00 Amazon gift card or extra credit for approved university courses.

Qualitative Analysis Procedure

The data were analyzed using a thematic analysis approach (Braun & Clarke, 2023) and included four steps. First, the data were carefully cleaned by thoroughly reviewing the, organizing, and de-identifying participant responses. In cases where responses were missing, partial data were retained and used in the analysis. Second, using an inductive thematic analysis (Braun & Clarke, 2023), codes were generated from participants’ responses, independently identified by three researchers with regular consensus meetings. During the consensus meetings, a fresh spreadsheet was used to add the identified codes and they were tallied with frequency counts. Third, the codes were integrated into overarching themesq (see Table 1). For example, codes that identified emotions (e.g., annoyed, disappointed) were merged into the theme Emotional Response. To maintain traceability, participant IDs and code locations were added to the spreadsheet. During this process, the raw data were reviewed to ensure the context in which a response was captured was accounted for. Lastly, three researchers met and comprehensively reviewed the themes for consistency, and the themes were accepted. The process involved reviewing each theme, its description, and the context of the raw data one final time to ensure wuane were reflecting the participant responses accurately, such as reflecting both semantic meaning and latent meaning (Braun & Clarke, 2023). For example, a response such as, “I tried to look for the problem and fix it,” indicated a general behavioral change in approach Performance/Behavioral Change - General change) with an undertone of taking responsibility to “fix” the agent’s mistake (Blame/Responsibility – Self).

OPEN IN VIEWER

Table 1.

Overarching Themes from Qualitative Responses.

Theme	Description	Example
Emotional response	An emotional response, such as annoyed or betrayed	“I felt slightly betrayed by the teammate”
Performance/behavioral change	A change in their behavior, such as motivating the agent to be more careful, decreasing reliance, or a general change	“I only used it when I had to from then on. I felt it was bringing us all down as a team.”
Blame/responsibility	Described an entity (self, technological limits, or agent) for having responsibility for the mistake	“Thought I should’ve motivated them to do SAR more slowly.”
Expectations	Expected the violation	“It did not shock that much.”
Trust	Lost trust after the mistake(s)	“I felt like I can’t trust the automated system in preform the task.!!”
Wrong agent	The participant indicated the wrong agent as responsible for the violation	N/A

Qualitative Findings

The recurring themes identified are in Table 1. In response to violations, we found Emotional Response was the most prevalent theme overall, with 86 occurrences across all conditions. Of all instances of themes, emotional response had a 50% frequency of occurrence. Emotional responses included various emotions such as annoyed, disappointment, surprise, and upset. For example, one participant stated, “I felt a little frustrated especially when it made the same mistake twice.” Interestingly, conditions in which individual agents repaired had more emotional responses recorded than conditions in which all members or none repaired. Notably, the condition in which the violating agent repaired had fewer participants feeling “Surprised” than other conditions. The findings suggest that observable emotions such as frustration and disappointment underscore the importance of addressing emotional impacts in designing agent behaviors. The reduction of surprise when the violating agent performed the repair indicates a shared expectation among the agent and participant, which could have implications for enhancing trust by setting expectations in human-agent interactions.

In addition to their emotional responses, participants also reported their behavioral responses in which they indicated a change in their behavior in response to the mistake or the consequences of the mistake. We recorded 25 instances of performance/behavioral changes in total. Participants reported becoming more alert or cautious following the agent’s mistake, indicating an adjustment in their behavior to mitigate potential risks. For example, a participant stated that their reliance on the violating agent decreased, stating, “I only used it when I had to from then on. I felt it was bringing us all down as a team.” Such findings indicate a behavioral adaptation when the team or human participant experiences a violation, suggesting that participants perceive and response to violations as potential risks to performance. These findings highlight the notion that a noticeable change in the participants’ behavior with regards to agent tasking and communication, may indicate a change in trust among the human agent team, and may be a useful behavioral indicator to monitor.

Participants assigned blame or responsibility to entities for the mistake for 10% of recorded themes. Participants sometimes attributed the mistake to the agent (e.g., “I felt they should have been more careful and must try avoiding the same mistake twice”), to themselves (e.g., “Thought I should’ve motivated them to do [search and rescue] SAR more slowly.”), or to the human builders of the technology (e.g., “I also knew that it was not entirely its fault, for it was built by humans.”). Moreover, participants suggested they expected the violation in 5 instances across all conditions. For example, one participant wrote, “Also the fact that the same accident that happened was also in the tutorial, so it did not shock that much.” Moreover, we recorded only 10 instances of participants explicitly mentioning a loss of trust. These findings indicate that this is a nuanced attribution process, one in which participants recognize the broader network of working in a HAT, and it in turn may influence how they perceive accountability in human-agent teams and interactions. The results suggest that it may be beneficial to assess how best to mitigate or rectify a violation across all dimensions of the human agent team, whether that be from the agent itself directly, the manufacturer, or on a broader level.

In response to repairs, we found that participants’ perception of a repair occurring changed based on the agent who repaired. When the violating agent enacted a repair, roughly 1/3 of participants (8/28) did not perceive this as a repair. For participants in the condition in which the full team repairs, only one participant did not perceive this as a repair. Another interesting finding was that participants in which there was no repair occasionally answered yes to the previously mentioned qualitative question and cited themselves and behavioral changes they pushed onto the agents as an act of repair. The findings suggest that the recognition of a repair event in HATs depends heavily on the agent or team member enacting it. Showcasing, contrary to considerable research, the agent who violates may not be the most ideal candidate to perform the repair, and it is, in fact, more influential when all members of the team contribute to a repair.

Exploratory Quantitative Results

We conducted further exploratory analyses by combining survey measures with qualitative themes, aligning with recommendations for mixed methods integration (Creswell & Creswell, 2018). Specifically, we converted qualitative themes into dummy-coded quantitative variables to examine potential associations with individual differences, such as personality traits (Extraversion, Agreeableness, Conscientiousness, Neuroticism, and Openness; International Personality Item Pool, IPIP; Donnellan et al., 2006), domain experience, propensity to trust automation (Jessup et al., 2019), and perfect automation schema (Merritt, 2015), or potentially confounding factors such as caffeine amount consumed. The only significant correlation between individual differences and qualitative themes was between Agreeableness and the theme of expecting a violation (r = −.256, p = .002). Though weak, the negative correlation between Agreeableness and expectations suggests that more agreeable individuals may not expect violations. This concept is supported by previous literature, which states what when individuals are more agreeable, trust in automation is higher, (Hanna & Richards, 2015). The weak negative correlation between agreeableness and expectations also indicates that agreeable-leaning personalities may yield delayed, or less severe reactions when a trust violation occurs. In practicality, this concept may point to the fact that certain interventions for repairing trust may be customizable depending on the personality type of the user.

The themes showed strong positive correlations with each other (ranging from r = .865 to .982). The strong, positive correlations among the themes may suggest that participants experience a trust violation holistically, affecting multiple facets such as emotions, cognitive appraisals of the situation, and behaviors.

Moreover, we investigated the relationships between themes and the overall trust dissolution in the violating agent. Trust dissolution was calculated by subtracting the survey after the second violation from the survey before either violation and dividing by 10. There was a significant negative correlation between the theme of trust loss and trust dissolution in the violating agent (r = −.229, p = .006). The results indicate that as expressions of trust loss increase, trust in the violating agent deteriorates further, for example, as the qualitative data mentioned trust loss more, quantitative trust scores changes. This finding emphasizes the importance of addressing trust in reducing the erosion of trust over time. Additionally, we conducted a hierarchical regression consisting of controls (IPIP, caffeine consumption, domain experience, propensity to trust, perfect automation schema) and the qualitative themes. This model was not statistically significant (p = .059), but explained 67.5% of the variance in the dissolution of trust in the violating agent. While this should be interpreted cautiously due to the lack of statistical significance, the variance accounted for is large.

Lastly, an analysis of variance (ANOVA) was conducted to examine the factors influencing the dissolution of trust in the violating agent. The assumption of equal variances was met for trust dissolution across groups, as Levene’s Test of Equality of Error Variances was non-significant, F(4,137) = 0.989, p = .416. The overall model was significant, F(41, 100) = 1.671, p = .020, explaining approximately 40.7% of the variance in the dissolution of trust (R² = .407). The adjusted R² = .163 indicated a more modest fit when accounting for the number of predictors. Significant main effects were observed for the themes Performance/Behavioral Change and Trust Loss. Changes in performance behavior significantly predicted the dissolution of trust F(1,100) = 4.734, p = .032, which indicates that participants who discussed performance change experienced greater trust dissolution than those who did not. Trust Loss had a significant impact on the trust dissolution, F(1,100) = 7.182, p = .009. This suggests that those who discussed trust loss experienced a different pattern of trust dissolution than those who did not mention losing trust in the violating agent.

A significant three-way interaction was found among conditions, Emotional Response, and Performance/Behavioral Change, F(4,100) - 3.262, p = .015. This suggests that the effect of performance behavioral changes on trust dissolution depends on both the emotional response and the specific condition. Lastly, there was a significant interaction between Performance/Behavioral Change and Blame/Responsibility, F(1,100) = 3.754, p = .005. Interestingly, when examining the same factors’ influence as above on the overall slope of trust restoration in the violating agent, the overall model was not significant, F(1,139) = 1.129, p = .301.