Assessing Interactional Competence Through Generative AI: Comparing Large Language Models as AI Interlocutors in the Paired Oral Discussion Test

Introduction

A challenge in large-scale, high-stakes oral assessments is to validly assess oral communication. Interactional competence (IC), or the ability to communicate effectively with others, represents an essential component of second language (L2) oral communication that should be assessed accordingly (Roever & Kasper, 2018). For assessment purposes, IC is defined here as an individual’s ability to contribute to shared understanding through oral interaction (Ockey & Chukharev-Hudilainen, 2021), allowing for the separation of individual test-taker performance from the co-constructed nature of interaction. IC has been conceptualized as a multidimensional construct consisting of various observable features that serve as indicators of IC ability and can be organized into dimensions such as topic management, interactional management, and interactive listening (Vo, 2019), although the specific features and their observability may vary across contexts (Galaczi, 2014; He & Dai, 2006).

Traditionally, IC has been assessed through paired or group tests, which provide the necessary context and input to elicit IC features (Ockey & Chukharev-Hudilainen, 2021). Unlike oral proficiency interviews, where the examiner leads the conversation, paired tasks place test takers in a symmetrical relationship, eliciting a broader range of interactional features more representative of authentic academic peer interaction (Brooks, 2009; Galaczi & Taylor, 2018). However, the inclusion of interlocutors may increase the challenge of judging test takers’ individual IC ability, as test takers’ performances can be affected by their interlocutors. It is difficult to isolate one test taker’s ability from another due to the variability inherent in human partners (May, 2009), which can introduce construct-irrelevant variance to the test (Galaczi & Taylor, 2018). Previous studies have identified interlocutor variability as including differences in personality traits (Ockey, 2009) and proficiency levels (Davis, 2009).

To address this challenge, some researchers have used Spoken Dialogue Systems (SDS) as speech partners instead of humans. An SDS is a computer system that can engage in oral interactions with test takers (Timpe-Laughlin et al., 2017). For example, Ockey and Chukharev-Hudilainen (2021) developed an SDS for paired discussion tasks by combining speech recognition, keyword detection, and text-to-speech (TTS) synthesis. Compared with human partners, the SDS condition yielded higher score dependability, likely due to its standardized interaction and consistent opportunities to demonstrate IC features. Notably, the system’s intentional communication breakdowns increased the reliability of IC assessment by eliciting certain IC features that may not readily surface in spontaneous interaction, such as meaning negotiation and repair, particularly in human–human interaction where test takers may otherwise mask their lack of understanding (e.g., by nodding along) (Ockey & Chukharev-Hudilainen, 2021; Ockey et al., 2023). However, while the SDS minimized variability, its rule-based inference relying on keyword detection and predetermined response pathways limited responsiveness to test-taker input, resulting in less natural and authentic discourse. This underscores the tension between achieving standardization and maintaining authenticity in IC assessment.

Recent advancements in generative artificial intelligence (GenAI) offer promising solutions. Large language models (LLMs), such as OpenAI’s GPT and Anthropic’s Claude series, can generate coherent, context-appropriate responses in real time rather than selecting from predetermined templates. Advanced speech synthesis technologies have the potential to produce realistic speech with diverse speech varieties at controlled intelligibility levels. Building on these advances, the present study developed an LLM-driven SDS called Generative Interactional Competence Elicitor (GICE) to examine how such technologies can support standardized yet authentic IC assessment. In particular, certain IC features may be inherently less common in spontaneous interaction when communication breakdowns or conflicting opinions are absent (He & Dai, 2006). GICE incorporates intentional communication breakdowns as an elicitation strategy, similar to ICE (Ockey & Chukharev-Hudilainen, 2021), to improve score dependability for IC while structuring the interaction to reflect key characteristics of the paired discussion task—maintaining an opposing stance and simulating peer-like interaction—thereby creating conditions for sustained negotiation and interactional demands necessary to demonstrate a broader range of IC features.

A critical design decision in developing GICE involved selecting an appropriate LLM to serve as the AI partner, as different models may create distinct interactional conditions that affect test validity. One study in cognitive science, Mayor et al. (2025), found distinct conversational behaviors between GPT-4 and Claude 3.5—such as differences in turn length and alignment—suggesting that model selection may meaningfully impact test-taker interactions. Yet systematic comparisons of LLMs for IC assessment purposes remain limited. Furthermore, from a partner-modeling perspective, speakers adapt their interactional behavior based on perceived partner capacity (Fischer, 2016); in human–computer interaction, reduced interpersonal language has been observed when computer partners are perceived as having limited interpersonal skills (Shechtman & Horowitz, 2003). Since test takers’ perceptions of the AI partner’s capacity, such as its perceived authenticity and interactiveness, may influence their interactional engagement, understanding these perceptions is relevant to evaluating model suitability for IC assessment.

This study therefore provides a direct comparison of GPT-4o and Claude 3.5 Sonnet as AI partners in paired discussion tasks, investigating their effectiveness for IC assessment within the context of a local university English Placement Test of Oral Communication (EPT OC). Two research questions guided this research.

Research Questions

Methods

Results

System Performance

Table 1 presents the comparison of system performance between GPT-4o and Claude 3.5 Sonnet. Claude outperformed GPT across all three evaluation criteria. For breakdown activation, GPT achieved only 33% success in creating deliberate communication breakdowns, failing to trigger the nonsense strategy in 11 of 12 cases. Claude successfully activated all breakdown strategies across all test takers (100%). For stance maintenance, GPT frequently shifted positions or sought common ground (41.7% success), whereas Claude maintained topic and opposition (91.7%). For persona adherence, while both models implemented natural speech features (e.g., fillers, hesitations), only Claude somewhat successfully incorporated peer-like anecdotes, often with emotional expressions (50% vs. 0%).

OPEN IN VIEWER

Table 1.

Model Comparison by Evaluation Criterion.

Criteria	GPT-4o (n = 12)	Claude 3.5 Sonnet (n = 12)
Breakdown activation	33%	100%
Stance maintenance	41.67%	91.67%
Persona adherence	0%	50%

IC Demonstrations by the Test Takers

Discourse analysis revealed distinct IC feature distributions across the three dimensions and two LLM conditions (see Table 2 and Figure 1). Topic management was the most frequently observed dimension for both models. All four features appeared in most test takers’ interactions, except for Participant 8, who struggled with developing topics in the Claude condition (their first attempt). However, while all test takers demonstrated the ability to reply in an appropriate time (Table 2), five test takers occasionally failed to do so with GPT (16 total instances), often missing turns while formulating responses, compared to two test takers with Claude (6 total instances; 5 by Participant 8). Initiating topics, though least frequent, was elicited at least once for both conditions as participants were encouraged to start the conversation.

OPEN IN VIEWER

Table 2.

Overall Frequency of IC Features in Test-Taker Responses (GPT-4o vs. Claude 3.5).

Feature	n a		Total count b		% Per dimension
	GPT	Claude	GPT	Claude	GPT	Claude
[TM] Replying in an appropriate time	12	12	66	66	35.68	39.76
[TM] Connecting topics	12	12	52	44	28.11	26.51
[TM] Developing topics	12	11	50	41	27.03	24.70
[TM] Initiating topics	12	12	17	15	9.19	9.04
[TM] Total			185	166	100	100
[IM] Recognizing a position	12	12	54	63	43.55	47.37
[IM] Self-correction	12	12	59	49	47.58	36.84
[IM] Clarification response	11	12	9	15	7.26	11.28
[IM] Clarification request	1	3	1	3	0.81	2.26
[IM] Checking comprehension	1	3	1	3	0.81	2.26
[IM] Total			124	133	100	100
[IL] Confirming comprehension	10	7	17	14	70.83	56.00
[IL] Questioning for opinion/information	3	7	5	8	20.83	32.00
[IL] Prompting	2	3	2	3	8.33	12.00
[IL] Total			24	25	100	100

Note. [TM]: topic management; [IM] interactional management; [IL]: interactive listening.

a

Number of test takers who demonstrated a particular IC feature.

b

Total frequency of occurrences across all test takers.

OPEN IN VIEWER

Figure 1.

Interactional competence features identified in participants’ interactions across the GPT-4o and Claude 3.5 Sonnet conditions (n = 12).

Interactional management showed more variation. Recognizing a position, self-correction, and clarification response were well-demonstrated in both conditions. Despite all test takers demonstrating position recognition, five occasionally failed to maintain or clearly express their stance with GPT (16 instances), compared with only one in the Claude condition (6 instances). The following excerpt illustrates this pattern, showing how GPT sought common ground and the test taker agreed rather than maintaining the assigned opposing position (see Supplemental Appendix C for full illustrative excerpts):

Participant 2

AI (GPT-4o):

What if there was a balance between exams and projects, so students could benefit from both methods of assessment?

TT:

Yeah, I do agree with that.

Clarification requests and checking comprehension were rare. GPT generated nonsense only once, which led to a clarification request from one test taker. In contrast, Claude produced nonsense across all test takers. Only three test takers requested clarification, while seven did not address the nonsense, often avoiding it altogether. In two cases, the conversation ended before the test taker could respond. The following excerpt illustrates this avoidance pattern:

Participant 9

AI (Claude Sonnet 3.5):

[Nonsense strategy activated] I think the zorbilation of mipflunks can be misleading for real learning assessment, so we need to consider how zorbilation of mipflunks affects student evaluation.

TT:

That is true, but after college, most of people go out in the real world, so learning all the theories or equations or something, that cannot be like ideal . . .

Interactive listening was the least observable dimension in both conditions. Questioning for opinion/information and prompting were particularly limited.

Discussion

The results indicated that Claude 3.5 Sonnet outperformed GPT-4o across all evaluation criteria, demonstrating broader and more consistent IC feature elicitation. Of particular note, GICE was designed to introduce nonsense words to elicit clarification requests. In the GPT condition, such requests were nearly absent (1 of 12 participants), largely due to GPT’s failure to activate this breakdown strategy. In the Claude condition, only three instances of clarification requests were observed, while seven test takers avoided responding to the AI’s nonsense utterances. This variation may reflect differences in IC ability, as clarification constitutes active meaning negotiation while avoidance bypasses repair opportunities. These findings underscore the importance of creating deliberate communication breakdowns, as certain features like clarification requests may not naturally surface in spontaneous interaction (He & Dai, 2006).

GPT-4o’s weaker performance compared with Claude 3.5, particularly in activating breakdown strategies, may reflect underlying differences in how the models balance helpfulness with instruction compliance. GPT-4o, trained through reinforcement learning from human feedback (RLHF), is optimized to produce responses that align with human-preference judgments favoring a coherent, helpful conversation (OpenAI, 2024). In contrast, Claude 3.5’s principle-based training (Bai et al., 2022) appears to result in stronger adherence to instructions even when doing so violates conversational norms, such as producing nonsense in this study. External benchmark evaluations showed that Claude 3.5 demonstrated higher consistency than GPT-4o in following complex, multi-turn instructions—particularly when retrieving and acting upon dispersed directives across extended dialogues (Epstein et al., 2024). As a result, when researchers or test developers desire to target particular language features, it may be more appropriate to use Claude than GPT.

Furthermore, several test takers in the GPT condition occasionally failed to demonstrate timing appropriateness and stance recognition, which may be explained by two factors: GPT’s inconsistent stance maintenance and its discourse style. Rather than consistently defending an oppositional position, GPT frequently diverted topics, sought common ground, or asked for mutual solutions, reflecting weaker stance maintenance (41.7% vs. 91.7% for Claude). This behavior likely increased cognitive load, requiring additional processing that led to mistimed responses. Moreover, as reflected in test takers’ perceptions, GPT’s tendency to dominate and lead the conversation may have reduced opportunities for test takers to clearly articulate and defend their assigned stance, contributing to failures in stance recognition. While one might argue that this increased difficulty could help discriminate between higher- and lower-ability test takers, such behavior instead reflects GPT’s limited capacity to function as a peer interlocutor within the EPT task design. The same test takers succeeded with Claude, indicating that these failures stemmed from AI performance rather than test-taker ability. By failing to maintain the oppositional role required by the EPT, GPT inadvertently created conditions that obscured rather than revealed test takers’ IC abilities.

Four IC features were less frequently elicited across both conditions: checking comprehension, confirming comprehension, prompting, and questioning for opinion/information. For the comprehension-related features, test takers’ awareness that they were interacting with an AI rather than a human may have reduced their perceived need to verify understanding. In addition, backchanneling to demonstrate interactive listening may have felt unnatural when the AI partner provided no backchannels due to technical constraints. The limited use of inquiry skills (questioning and prompting) suggests that participants either lacked these abilities or that the AI partners’ underdeveloped arguments—providing counterclaims without fully elaborating their stances—reduced opportunities for deeper engagement. These findings indicate that further system prompt refinement and additional IC elicitation strategies will be necessary to consistently elicit these less-represented features.

The thematic analysis revealed that LLM-driven SDS produced more natural interactions than rule-based systems through their generative capabilities, which enabled more fluid turn-taking. While both models implemented natural speech features that contributed to a peer-like impression, Claude’s higher persona adherence score (50% vs. GPT’s 0%) was reflected in test takers’ perceptions: Claude was perceived as more authentic and student-like, particularly because of its emotional expressions and personal anecdotes. By contrast, GPT’s more conversation-leading style was perceived as less authentic and more AI-like, as test takers noted it “generated some speaking” rather than engaging as a genuine peer. However, the nonsense strategy may involve a trade-off between IC elicitation and maintaining persona adherence, as such utterances may feel less natural and less characteristic of human–human interaction.

Authenticity should be examined in relation to its impact on test validity rather than assumed to be inherently beneficial (Lewkowicz, 2000). The findings point to a key design challenge for LLM-based L2 oral communication assessments: balancing LLM capabilities to ensure authentic yet reliable IC elicitation. Purely generative systems may overly assist test takers through their high inferencing capabilities, potentially allowing them to mask the very skills the assessment aims to measure. For instance, LLMs may infer meaning from unclear utterances and respond smoothly without any communication breakdown, thereby preventing repair sequences from occurring. Moreover, LLMs are less transparent and potentially less robust than rule-based systems, as their responses are not fully predictable. To address these concerns, this study introduced structured communication breakdowns to reveal test-taker strategies, an approach found to be critical for ensuring consistent IC opportunities across test administrations. Drawing on these findings, Claude 3.5 Sonnet (with plans to transition to Claude Sonnet 4) was selected for implementation in GICE.

In developing similar LLM-driven SDSs, several practical considerations should guide design and model selection. First, the purpose and context of the assessment should determine what is prioritized. For large-scale, high-stakes assessment, this study prioritized standardization of model behavior over conversational smoothness to ensure consistent IC elicitation opportunities. Second, task specifications and the target construct should inform system prompt design. In this study, the AI partner was required to maintain an opposing stance and function as a plausible peer interlocutor rather than an examiner in an interview task type. Finally, model performance should be evaluated against construct-relevant criteria, as demonstrated in this study, with system prompts iteratively refined to optimize alignment with the assessment’s goals.

Several limitations should be acknowledged. First, as a small pilot study, the findings have limited generalizability and should be validated with larger, more diverse samples representing different first-language and cultural backgrounds. Second, this study excluded rater scoring analysis; future research should examine how LLM-driven interactions may impact IC evaluation and rater perceptions. Third, the current system did not include body language such as facial expressions and gestures. Given growing evidence that body language constitutes an important component of IC (Vo, 2019), future research should incorporate visual features and examine their impact on both test takers’ IC demonstrations and their perceptions of AI partners.

Conclusion

This study compared GPT-4o to Claude 3.5 Sonnet to inform model selection in the development of GICE, an LLM-driven SDS designed to elicit IC in paired discussion tasks in English placement testing. The findings revealed that different LLMs create distinct interactional conditions, influencing both IC feature elicitation and test takers’ perceptions of the AI partners. These results provide preliminary evidence that model selection can meaningfully impact AI-mediated speaking assessments.

As technology and LLMs advance rapidly with new versions released regularly, the findings underscore the need to redirect focus toward the construct being measured over technological novelty. A careful balance must be maintained between authenticity and validity. The results suggest that establishing clear evaluation criteria for LLMs—carefully grounded in construct relevance and test purpose—and applying iterative prompt engineering to optimize performance for specific assessment contexts are critical when using LLM-driven SDS.

Supplemental Material