Abstract
The use of large language models (LLMs) in language assessment, particularly in spoken dialogue systems (SDSs) for assessing speaking, remains at an early stage. This study explored the use of an LLM-driven SDS to assess second language speaking ability. Using a within-participant design, we compared the paired discussion performance of 30 participants interacting with a self-built LLM-driven SDS (E-Talk) versus a human interlocutor, focusing on the interlocutor effect on test scores and fine-grained linguistic and interactional features. Results did not yield substantive differences in oral performance across the two conditions, although the interactions with the LLM-driven SDS displayed slightly lower scores for pronunciation and language use, slower speech rate, and reduced lexical diversity alongside increased syntactic complexity, and greater initiative in introducing new ideas, prompting responses, and guiding discussions toward negotiation, coupled with weaker interactive listening. These findings suggest that LLM-driven SDSs can serve as a potential, usable interlocutor in dialogic speaking assessments, eliciting key aspects of speaking ability. That said, further refinement is needed to better capture interactive listening and collaborative meaning construction, highlighting the importance of interpreting SDS-based performance relative to its specific interactional affordances.
Keywords
Introduction
Paired and group oral tasks have been widely adopted in second language (L2) speaking assessment for their potential to elicit authentic interaction and co-constructed meaning (Galaczi & Taylor, 2018; Plough et al., 2018). However, concerns over practicality (Gokturk & Chukharev, 2024), interlocutor effects (Artunç & Hart, 2019; Shin, 2020), and scoring consistency (Van Moere, 2006; Youn & Chen, 2021) have prompted growing interest in automated virtual interlocutors as a potential alternative. These virtual interlocutors are powered by spoken dialogue systems (SDSs), that is, intelligent programs that engage users in natural conversation through integrated components such as automatic speech recognition (ASR), natural language processing (NLP), and text-to-speech (TTS) synthesis. SDSs have supported a wide range of speaking tasks, from highly constrained formats like read-aloud to more spontaneous tasks like role plays and paired discussions. The release of ChatGPT has further accelerated interest in large language model (LLM)-based dialogue systems, positioning them as a potential “game changer” in language education and assessment (Chapelle, 2025).
Despite these promising developments, research on the use of SDSs, particularly advanced LLM-driven SDSs, as speaking partners in assessment contexts is still in its infancy. Earlier studies on rule-based SDSs have shown their potential to provide standardized interactional settings and elicit a range of speaking constructs (e.g., Gokturk & Chukharev, 2023, 2024; Ockey & Chukharev, 2021). However, these systems are often limited by robotic voice quality, unnatural turn-taking, and a restricted ability to capture features of interactional competence (IC), with many lacking non-verbal cues essential for natural communication (Timpe-Laughlin et al., 2022). These limitations raise concerns about interactional authenticity and construct coverage. More recent studies applying LLM-driven SDSs to speaking assessment (e.g., Eguchi et al., 2025; Karatay & Xu, 2025; Timpe-Laughlin et al., 2025) have shown that such systems can effectively elicit key IC features, offering new possibilities for more naturalistic and adaptive task delivery. However, despite these advances, little is known about whether and the extent to which SDS-delivered tasks elicit similar oral performances as human interlocutor tasks, especially in the domains of speech fluency, linguistic complexity, and interactional quality. Evidence gained in these investigations can inform to what extent SDS-delivered tasks support valid representations of the target speaking construct (Eguchi et al., 2025). To address these gaps, the present study compares the use of a multimodal, LLM-driven SDS (E-Talk, featuring a lip-synced digital avatar interface) versus a human interlocutor to deliver paired discussion tasks in terms of scores test-takers receive and language features of their speaking performances.
Literature Review
The Construct of Paired Speaking Tasks in L2 Speaking Assessment
The construct of paired speaking tasks in L2 speaking assessment is grounded in theories of communicative language ability (Bachman & Palmer, 2010), which view speaking as a multidimensional communicative ability involving the interplay of linguistic knowledge, strategic competence, and contextual appropriateness. When applied to paired tasks, this construct highlights the inherently interactive and co-constructed nature of communication. In such tasks, speakers draw on both language-related resources and interactional resources to jointly manage the discourse, for example, through turn-taking, topic development, responsiveness, and interactive listening (Galaczi, 2014; Lam, 2021; Roever & Ikeda, 2022). This perspective reflects an integrated view of speaking as simultaneously cognitive-linguistic and social-interactional in nature.
Within this broader construct, four dimensions are particularly relevant to paired speaking assessment: language use, fluency, pronunciation, and IC. Language use generally concerns the range, appropriateness, and control of grammatical and lexical resources, which can be further characterized into syntactic complexity and lexical richness (Lu, 2010; Ortega, 2003; Read & Read, 2000). Fluency, in a narrow sense, refers to the temporal aspects of speech delivery, often reflected in speech rate, pausing behavior, and repair processes (Skehan, 2003). Pronunciation concerns the extent to which speech is intelligible and comprehensible, as influenced by the use of segmental and suprasegmental features (Suzukida & Saito, 2022). These three dimensions have long been central to speaking assessment and align with the complexity, accuracy, and fluency (CAF) framework for examining L2 performance in second language acquisition (Housen & Kuiken, 2009).
Alongside these individual dimensions, IC has been increasingly recognized as a defining component of paired speaking tasks (e.g., ; Lam et al., 2023; Roever & Kasper, 2018). IC is broadly concerned with the ability to co-construct purposeful, contextually appropriate, and mutually responsive interaction (Galaczi & Taylor, 2018). Features commonly associated with IC include turn management, topic development, interactive listening, breakdown repair, and non-verbal behaviors (Galaczi & Taylor, 2018; Lam et al., 2023; May et al., 2020). Its conceptualization, however, remains contested, particularly regarding the extent to which IC resides within the individual or emerges through the reciprocal coordination of turns, actions, and shared understanding between interlocutors (McNamara & Roever, 2006). Its inherently emergent and jointly negotiated nature makes it more challenging to assess consistently than the other dimensions, a challenge further compounded by the increasing complexity of technology-enhanced interactional environments. Nevertheless, empirical studies have shown systematic variation in interactional features across proficiency levels, supporting IC as a meaningful dimension of speaking assessment (Galaczi, 2014; Roever & Ikeda, 2022).
Together, the theoretical foundations of communicative language ability, the CAF tradition in examining L2 performance, and the growing body of empirical research on IC in paired speaking tasks provide a basis for conceptualizing L2 speaking as an integrated, socio-cognitive construct.
Research on SDS in L2 Speaking Assessment
To assess L2 speaking, SDSs have been applied and studied as the virtual partner in dialogic speaking tasks, including interviews (e.g., Forsyth et al., 2019; Saeki et al., 2024), paired discussions (e.g., Gokturk & Chukharev, 2024; Karatay & Xu, 2025; Ockey & Chukharev, 2021; Ockey et al., 2023), and role plays (e.g., Dombi et al., 2022, 2024; Eguchi et al., 2025; Karatay, 2022; Litman et al., 2016; Saeki et al., 2024; Timpe-Laughlin & Dombi, 2020; Timpe-Laughlin et al., 2022, 2023, 2025).
Early studies have demonstrated the potential of using SDSs to engage in conversations with human users, although this line of research was largely limited to small-scale, proof-of-concept systems. For example, Evanini et al. (2014) piloted a trialogue-based SDS with young English learners, reporting high task completion rates and positive user feedback. Similarly, Litman et al. (2016) explored SDS-delivered conversational tasks linked to the Common European Framework of Reference for Languages (CEFR) levels and noted key challenges: limited language output when the SDS worked well, and unnatural interactions when it did not. These early exploratory studies laid the groundwork for later research by illustrating the potential and the limitations of SDS-delivered speaking assessments.
Building on these early explorations, more recent research has shifted toward systematically examining the capacity of SDSs to elicit speech samples suitable for assessing broader speaking constructs. Current studies can be broadly categorized into three major strands. The first strand focuses on the linguistic dimensions of speaking ability, with studies employing automated or manual discourse analysis to examine features such as pronunciation, fluency measures (e.g., speech rate, pausing), lexical diversity, and grammatical complexity. Findings demonstrate that SDS-delivered tasks can elicit speech samples that differentiate proficiency levels across these linguistic features (e.g., Gokturk, 2020), supporting the potential of SDS as a tool for measuring the linguistic aspects of L2 speaking ability. The second strand centers on IC, where researchers investigate micro-level interactional features, such as topic development, turn management, interactive listening, and repair management. Studies have shown that many key IC features, including appropriateness of responses, initiating/developing/connecting topics, checking/confirming comprehension, self-correcting mistakes, etc., can be consistently observed in participants’ speech (e.g., Eguchi et al., 2025; Gokturk & Chukharev, 2024; Karatay & Xu, 2025; Ockey et al., 2023; Timpe-Laughlin et al., 2025), suggesting that SDS can provide standardized yet sufficiently interactive conditions for eliciting IC-related performance. The third strand examines specific functional aspects of speaking, particularly speech acts like requests. Using multimodal SDS frameworks (e.g., Timpe-Laughlin et al., 2022), this line of research analyzes the realization of requests in terms of directness, strategy use, and the employment of modifiers. Notably, distinct patterns of L2 request behavior in SDS-delivered communication have been observed, differing from those commonly found in face-to-face interaction (e.g., Timpe-Laughlin & Dombi, 2020).
Comparability between an SDS and a Human Interlocutor
While prior research has shown that SDSs can elicit key linguistic, interactional, and functional aspects of L2 speaking ability, evidence has also pointed to differences in how these features are realized compared to face-to-face interactions (e.g., Timpe-Laughlin & Dombi, 2020). This raises important questions about the variations in oral performance across human- and SDS-delivered tasks, particularly in terms of test scores and specific language features.
One of the most consistent findings in SDS-delivered speaking assessments is that test-takers tend to receive lower scores when interacting with SDS than with a human interlocutor. Ockey and Chukharev (2021), for example, examined the psychometric properties of a rule-based SDS, the Interactional Competence Elicitor (ICE), by comparing holistic ratings between SDS- and human-delivered conditions. While no differences were observed in grammar and vocabulary, test-takers received lower ratings in fluency, pronunciation, and particularly IC in the SDS condition, with IC showing the largest effect size and greater score variance in a subsequent G-study. Although raters considered ICE an acceptable alternative, they nevertheless expressed a preference for human interlocutors. Building on this work, Ockey et al. (2023) focused specifically on IC ratings and found, through frequency analyses and Many-Facet Rasch Modeling, that more IC features were observable and ratable in the SDS condition. However, human-delivered interactions continued to receive higher ratings on most IC dimensions. Taken together, these findings suggest that while SDS-delivered tasks may produce opportunities for displaying IC, human interlocutors tend to elicit more favorable ratings, potentially reflecting rater expectations or socially mediated evaluation rather than differences in observable performance alone.
Regarding oral responses elicited through paired speaking tasks in both face-to-face and human–machine formats, research indicates that they share some features with face-to-face interaction but do not fully replicate it. Forsyth et al. (2019) found only slight differences in response quantity and complexity, with human interaction yielding longer responses and slightly more complex language, supporting the feasibility of automated tasks as an alternative for speaking assessment. Similarly, Timpe-Laughlin et al. (2022) observed similar levels of syntactic complexity, lexical variety, and fluency between SDS and human interlocutor conditions in request-based tasks, but noted that SDS interactions featured longer turns, fewer speaker changes, and more direct, transactional language. Follow-up analyses (Dombi et al., 2024; Timpe-Laughlin et al., 2023) revealed that SDS-mediated speech contained fewer indirect strategies, supportive moves, or socially motivated behaviors, and that multiple functions (e.g., greetings, topic initiation) were often compressed into a single turn, highlighting a more functional and less socially nuanced discourse style. While SDS tasks can elicit a range of linguistic features, they may fall short in capturing the social-pragmatic dimensions typical of human interaction.
Notably, all these earlier studies employed rule-based SDSs with limited conversational flexibility. Most of these findings suffered from significant technological limitations, including robotic speech, rigid scripting, unnatural turn-taking, and an inability to respond adaptively to spontaneous input. Such limitations pose threats to interactional authenticity and often constrain opportunities for test-takers to demonstrate the full range of their speaking ability (Forsyth et al., 2019; Gokturk & Chukharev, 2024). More recently, this line of research has begun to shift with the introduction of LLM-driven SDSs in L2 speaking assessment. Karatay and Xu (2025) and Timpe-Laughlin et al. (2025) demonstrate that LLM-driven systems can effectively elicit IC features and distinguish between proficiency levels. Extending this emerging evidence base, Eguchi et al. (2025) directly compared IC-relevant practices in LLM-driven SDS-delivered and human tutor-delivered role-plays, focusing on preference organization, sequence expansion, and turn-taking. They suggested that SDS-delivered role plays afforded learners a narrower repertoire of turn-taking practices, resulting in a constrained IC construct, one that diverges from, yet is not entirely distinct from, the construct elicited by human-interactive tasks. Building on this insight, recent conversation-analytic work by Choi and Oh (2026) argues that, rather than evaluating AI interlocutors against human conversational norms, systems such as ChatGPT should be conceptualized as categorically distinct but interactionally usable interlocutors.
With the emergence of LLM-driven SDSs that enable more flexible and adaptive interaction, it remains unknown whether previously observed score differences between human-interactive and SDS-delivered speaking tasks persist. More importantly, it is unclear how interactional and linguistic features are differentially enacted and co-constructed across the two task conditions during task performance. Thus, a comparative approach is needed to examine how LLM-driven SDSs afford distinct interactional practices and how these task-condition-specific patterns relate to score interpretation.
The Present Study
The present study compares test-takers’ interactions with a self-built, LLM-driven SDS versus a human interlocutor, examining the interlocutor effect on test scores, linguistic measures, and interactional features. By analyzing patterns of convergence and divergence, this study aims to provide empirical insights into how virtual interlocutors function in paired speaking assessments and the extent to which they can effectively capture key dimensions of speaking ability. The study is guided by the following research questions (RQs):
Methodology
Participants
Test-Takers
A total of 31 participants took part in the study. One participant was subsequently excluded due to technical issues with the audio recording, resulting in a final sample of 30 participants for analysis. Of these, seventeen were undergraduate students, four were master’s students, and nine were doctoral students. All were adult English L2 learners (age M = 24.5, SD = 3.17; 9 males, 21 females) randomly recruited from a tertiary university in China. Table 1 summarizes the participants’ English proficiency as indicated by standardized test scores and their approximate alignment with China’s Standards of English Language Ability (CSE). Additionally, their academic disciplines were different, and none of them were English majors. Most of them (n = 23, 77%) claimed to have prior experience of using LLM-driven tools such as ChatGPT, Doubao, Kimi, etc., mainly for translation, writing, vocabulary, and reading. Participation was entirely voluntary. Before data collection, they were all informed of the study purpose and procedures, provided written consent, and received financial compensation for their time.
Participants’ English Proficiency and Approximate CSE Levels.
Note. CET = College English Test (see Zheng & Cheng, 2008, for a test review of CET); CSE = China’s Standards of English Language Ability. IELTS = International English Language Testing System. English Gaokao refers to the national college entrance examination administered by the National Education Examinations Authority (n.d.) in China. CSE levels were inferred based on prior alignment studies (Dunlea et al., 2019; Jin et al., 2022).
Raters
Two PhD candidates in applied linguistics participated as raters in the study. Both had extensive rating experience for the paired discussion tasks on the College English Test (CET), a nationwide high-stakes English proficiency examination in China. They received a 3-hr face-to-face training session before the rating process. In cases of any rating disagreement, the second author served as a third rater to facilitate consensus among the raters (see the Rating section for details).
Coders
A PhD candidate and a master’s student in applied linguistics worked together as coders for IC coding. Prior to coding, they completed several rounds of trial coding training to reach consensus on their understanding of the IC coding scheme. The training focused on finalizing the feature exemplars and completing coding practices.
The Human Interlocutor
To control for interlocutor variability, a single human interlocutor was recruited to participate in all paired discussion tasks. She was a female L1 Chinese speaker and an applied linguistics master’s student with advanced English proficiency, as evidenced by an “Excellent” rating on the Test for English Majors-Band 8 (TEM-8), the highest level in China’s national English proficiency test for English majors. Prior to data collection, the second author explained the study purpose and provided the interlocutor with task instructions that were similar to the task prompts used in the E-Talk system (see Supplementary Material 1 and note that all supplementary material is available on the Open Science Framework [OSF]; Min et al., 2026). Specifically, she was required to play the role of a peer student and respond naturally during the discussion. A trial session was also conducted in which she practiced all the tasks with the second author.
Instruments
The E-Talk System
The E-Talk is a self-built multimodal platform for English learning and assessment. The assessment module adopts a modular architecture integrating ASR, a domain-specific LLM built on Qwen2-72B (Guanzhi, n.d.), and a lip-synced digital avatar (see Figure 1).
The E-talk system architecture.
The interaction begins when the user initiates a speaking turn by clicking to record and submit their speech via the mobile interface. The submitted speech is transcribed by the ASR module. 1 The text is then screened by a sensitive words detection component to filter inappropriate content. If sensitive words are detected, users are prompted to rephrase. The filtered text is converted into a prompt for the LLM, which generates a contextually relevant text-based response based on standardized, zero-shot prompts aligned with the speaking tasks (see Supplementary Material 1 for LLM prompt design and implementation details). This response is managed by the dialogue management module, which maintains conversation flow and updates the dialogue history. Finally, the generated text is synthesized into video via the digital avatar and returned to the user through the mobile interface, creating a seamless loop of interactive communication. The ASR module uses whisper-large-v3-600 (1.5G) for state-of-the-art transcription. The text sensitivity detection module, powered by Alibaba Cloud (n.d.), screens for six content types: violence, contraband, sexuality, profanity, pull-in traffic, and regional discrimination. Figure 2 illustrates the user interface used during the tasks.
The user interface of E-talk.
The Paired Discussion Tasks
We adapted a group discussion task from the speaking subtest of a high-stakes English proficiency exam for non-English-major undergraduates at a Chinese university (see a related study from Min et al., 2022). This task type reflects the communicative demands of paired academic discussion in a university setting, where students are expected to exchange ideas, negotiate solutions, and sustain purposeful interaction, closely aligning with the target language use domain underpinning the integrated construct of L2 speaking adopted in this study. The subtest includes an individual monologue, a group discussion, and a question-and-answer task. The group discussion task (two to four students) served as the basis for the present study. To ensure sufficient oral data, we selected two topics per task condition (E-talk and human tasks) from the test bank and added one practice topic to familiarize participants with E-Talk. In total, five parallel tasks were prepared, covering topics such as group projects, time management, campus transportation selection, course selection, and academic performance, all closely related to students’ academic and campus life. Participants were required to discuss and find the best solutions, with either a human or E-Talk acting as a peer interlocutor. Each task took around 8 min. Figure 3 provides an example task instruction.
Example paired discussion task instruction.
Rating Scale
The analytical rating scale was adapted from Ockey and Chukharev (2021), whose four-subconstruct framework – language use, fluency, pronunciation, and IC – captures both individual linguistic production dimensions and co-constructed interactional demands, aligning with the integrated construct of L2 speaking adopted in this study. The scale has also been applied in SDS-based speaking assessment research (e.g., Gokturk, 2020), providing an empirically informed basis for comparing performance across human-delivered and SDS-delivered conditions. The IC domain was expanded into more specific subdomains informed by May et al.’s (2020) IC checklist to ensure consistency between IC rating and coding. Each subconstruct was rated on a 1–5 scale (total score = 20) to allow for greater differentiation of speaking ability. Detailed descriptors for each score level are provided in Supplementary Material 2.
Linguistic Feature Extraction
To examine test-takers’ oral performance at a fine-grained level, automated tools were used to extract features across three linguistic domains informed by the CAF tradition (e.g., Koizumi & In’nami, 2024; Lu, 2012). Fluency features were extracted using the Praat script “Syllable Nuclei v2” (De Jong & Wempe, 2009), which automatically detects syllables and pauses from audio recordings, yielding seven features related to speech speed and pausing. Lexical richness and syntactic complexity were assessed using the Lexical Complexity Analyzer (LCA; Lu, 2012) and the L2 Syntactic Complexity Analyzer (L2SCA; Lu, 2010), yielding 25 and 14 features, respectively. Linguistic accuracy was not analyzed due to the current limitations of automated tools for error detection in spontaneous speech and the resource-intensive nature of manual annotation. A complete list of the 46 linguistic features is provided in Supplementary Material 3.
IC Coding Scheme
Interactional features were coded using May et al.’s (2020) IC checklist. Although originally developed as a learning-oriented feedback tool, the checklist offers an empirically grounded taxonomy of observable interactional behaviors derived from thematic analysis of examiner reports on the Cambridge B2 First paired task, a format comparable to the decision-making task used in this study. The checklist, designed for B1C1 learners, also aligns with our test-taker population. It defines 9 macro themes and over 25 pairs of positive/negative micro features. We coded only the presence of positive features, aligning with our focus on interactional strengths and ensuring greater reliability. Features involving body language and subjective judgments (e.g., “interacting confidently and naturally”) were excluded due to transcription limitations and coding consistency. The final coding scheme with example transcripts is provided in Supplementary Material 4.
Procedures
Collecting Speech Samples and the Questionnaire
Before testing, all participants completed an online background questionnaire and signed a consent form. Each participant completed four discussion tasks, two with the E-Talk and two with a human interlocutor, with task condition order alternated across participants (human–human first vs. E-Talk first). Although full topic counterbalancing was not feasible due to sample size, each of the five topics was completed by 12 participants across the two task conditions. The E-Talk tasks were administered via an iPhone 12, while the human tasks were conducted face-to-face. All tasks were completed individually in a quiet room, with the participant and the interlocutor present, and the second author supervising the procedure. To maintain a peer discussion scenario, participants were not informed in advance that the human interlocutor was pre-arranged. A practice task was provided before the E-Talk task to familiarize participants with the system. Each discussion was preceded by 2 min of preparation and was fully audio- and video-recorded. In total, the study collected approximately 960 min of spoken data across 120 performances (30 participants × 2 task conditions × 2 topics). Supplementary Material 5 provides conversation analysis (CA)-based example transcripts from the same test taker under both task conditions as illustrative speech samples.
Rating
The same two experienced raters scored all performances using the same rating rubric across both task conditions. To ensure consistent application of the rating rubric, particularly for IC, a 3-hr face-to-face training session was conducted. The two raters independently scored four sample performances per task condition, covering a full range of scores (1–5) as pre-rated by the second author, with discrepancies discussed and resolved through consensus to calibrate rating standards. Ratings were conducted on audio-recorded performances over a 2-week period to minimize potential halo effects. Inconsistent cases were first separated into two batches and re-rated by the same rater with a 1-week interval. Where discrepancies remained unresolved after re-rating, the third rater independently scored the same set of cases and final scores were then determined through discussion among all three raters until consensus was reached. Both intra-rater and inter-rater reliability were estimated using intraclass correlation coefficients (ICCs), calculated via the R package irr (Gamer et al., 2019) based on a single-measure, absolute-agreement, two-way mixed-effects model. Following Koo and Li’s (2016) guideline, where ICC values above 0.75 indicate good reliability, results showed strong inter-rater agreement for both E-Talk (average ICC = 0.90) and human interactions (average ICC = 0.86), as well as strong intra-rater reliability for Rater 1 (average ICC = 0.86) and Rater 2 (average ICC = 0.85). Detailed ICC results and explanations across raters, tasks, and rating dimensions are provided in Supplementary Material 6.
Data Processing
All oral data were processed for automated analysis using NLP tools, focusing exclusively on test-takers’ speech output. Audio recordings were transcribed verbatim via iFlyrec (n.d.), retaining repetitions, false starts, and repairs. These transcripts were then manually segmented into words and sentences with reference to the original audio, following criteria from Foster et al. (2000) and Hwang et al. (2020). The analysis was based on the analysis of speech unit (AS-unit) framework (Foster et al., 2000), in which sentence boundaries were determined by pause duration, content shift, and pitch contour. An AS-unit is defined as a single speaker’s utterance consisting of an independent clause together with any subordinate clauses associated with it (Foster et al., 2000). Clauses preceding a falling pitch, followed by a ⩾500-ms pause and topic shift, were treated as separate sentences. Interrupted or scaffolded speech was marked as a single AS-unit. Disfluencies were removed, including repetitions, self-corrections, false starts, and partial words, following Foster et al.’s (2000) classification. Textual noise, such as extraneous punctuation, was also checked. For fluency analysis, individual conversational turns were first merged into a single file per participant/task, after which syllable- and pause-related features were extracted following De Jong and Wempe’s (2009) protocol. Additional processing details are provided in Supplementary Material 7.
IC Coding
Two coders annotated 25% of the raw transcripts per condition through an iterative process involving scheme refinement, trial coding, and reliability checks. Coding was conducted at the level of speaker turns, with each turn treated as a meaningful interactional unit with continuous reference to the original audio recordings. Three codes, (a) “Take the initiative or show willingness to start,” (b) “Give partner time to express and formulate their ideas,” and (c) “Take turns to speak and share the floor with partner,” were excluded due to task design (The participants were required to start first thus excluding code a) or consistent participant behavior (Participants were talking turn by turn without interrupting their partner thus excluding codes b and c). The process continued until inter-coder agreement exceeded 90% (Mackey & Gass, 2016), after which the second coder independently completed the remaining data. Intra-coder agreement assessed one month later showed high consistency (85% for E-Talk task, 84.85% for human task).
Data Analyses
To compare participants’ performance in ratings (RQ1) and automated linguistic and coded interactional features (RQ2), separate (generalized) linear mixed-effects models (LMMs) with dummy-coded predictors were fitted to account for repeated observations from the same participants and multiple sources of variability in the assessment design.
For score differences (RQ1), task condition was included as a fixed effect to estimate the average difference in speaking scores between the two interaction conditions. As we only had two trained raters, they were also specified as a fixed effect to control for potential rater severity differences. Participants were modeled with random intercepts and random slopes for task condition. The random intercept for topic was removed due to negligible variance and singular fit issues (see Supplementary Material 8 for full model specifications).
For linguistic features measures (RQ2), candidate automated linguistic features were first screened through correlation analyses with holistic scores, following Cai and Yan (2024). Highly intercorrelated features or those representing the same construct were reduced by retaining the feature with the strongest correlation, yielding 12 features for analysis (see Supplementary Material 3). Each feature was examined using LMMs with task condition as the fixed effect and participant and topic as random intercepts. Random slopes for task condition were excluded due to convergence or singularity issues.
Interactional features (RQ2) were analyzed using generalized linear mixed-effects models (GLMMs) to accommodate their frequency-based distributions: binomial models for binary features and negative binomial models for count features. For negative binomial models, phonation time (log-transformed) served as an offset to control for interaction length. Task condition was included as a fixed effect, while participant and topic were specified as random intercepts. Effects were reported as odds ratios (OR) for binomial models and incidence rate ratios (IRR) for negative binomial models.
To control the false discovery rate (FDR) from multiple comparisons, p values were adjusted using the Benjamini–Hochberg procedure (Benjamini & Hochberg, 1995), with a significance threshold of α = 0.05. Model residuals were inspected visually (Q–Q plots) and statistically, with |z| > 3.29 for skewness/kurtosis indicating non‑normality (Kim, 2013). Analyses were performed in Jamovi (Version 2.6.2.0) (2024) for LMMs and in R using glmmTMB package (McGillycuddy et al., 2025) for GLMMs.
Results
RQ1: Participants’ Overall Speaking Performance in E-Talk and Human Interactions
Table 2 summarizes the descriptive statistics for each subconstruct and the total score across the two task conditions. Normality of model residuals assessed through the z value of skewness and kurtosis was within acceptable ranges, providing support for the approximate normality of the residuals. Results show a consistent lower score trend in participants’ performance while they talk with E-Talk.
Descriptive Statistics of Speaking Ratings Across Task Conditions.
Note. IC = interactional competence; M = mean; SD = standard deviation. z values of skewness and kurtosis were computed for model residuals to assess normality.
To further examine the task-condition effects, LMMs were fitted with task condition and rater as fixed effects, and participants were modeled with random intercepts and random slopes for task condition. Table 3 summarizes the fixed effect of task condition on test scores. For fluency, the task-condition effect did not reach statistical significance (β = −.16, p = .09). Similarly, IC scores did not differ significantly by task conditions (β = −.11, p = .28). In contrast, statistically significant task-condition effects were observed for language use and pronunciation. For language use, participants scored an estimated 0.21 points lower in the E-Talk condition than in the human task condition (β = −.21, p = .04). A similar pattern was found for pronunciation, with a negative task-condition effect (β = −.25, p = .02), indicating modest advantages for the human task condition in these analytic dimensions. At the holistic level, the task-condition effect on total speaking scores did not remain statistically significant after FDR correction (β = −.73, p = .053).
Fixed Effects of Task Condition on Speaking Ratings.
Note. IC = interactional competence; SE = standard error; CI = confidence interval; 0 = human task condition, 1 = machine task condition. The p values were adjusted using the FDR correction method.
RQ2: Participants’ Linguistic and Interactional Features in E-Talk and Human Interaction
Linguistic Features
Table 4 presents the descriptive statistics for the linguistic feature measures across the two task conditions, covering fluency, lexical richness, and syntactic complexity. Overall, fluency measures were consistently higher in the human interaction, with participants producing more syllables (M = 854.63, SD = 229.53) and speaking at a faster rate (M = 3.46, SD = 0.72) than in E-Talk interaction (M = 689.82, SD = 264.81; M = 2.95, SD = 0.61). In contrast, the lexical richness measures showed broadly similar mean values and standard deviations across the two conditions. For syntactic complexity, participants tended to produce longer and more syntactically complex sentences in E-Talk interaction, as suggested by the higher mean length of sentence (M = 13.01, SD = 2.71) and sentence complexity ratio (M = 1.68, SD = .33). However, coordination at the T-unit level was similar across the two conditions. While z values of skewness and kurtosis indicated departures from normality for speech rate, articulation rate, and coordinate phrases per T-unit, inspection of Q–Q plots of the model residuals revealed no substantial deviations from normality. Given the robustness of LMMs to minor violations of the normality assumption, no corrective procedures were applied.
Descriptive Statistics of Fluency, Lexical Richness, and Syntactic Complexity Across Task Conditions.
Note. M = mean; SD = standard deviation. z values of skewness and kurtosis were computed for model residuals to assess normality.
To examine the extent to which linguistic features differed between the two task conditions, each feature was tested using a separate LMM with task condition as the fixed effect and participant and topic as random intercepts. Table 5 presents the linguistic features that have statistically significant task-condition effects. Results for all 12 features are provided in Supplementary Material 9. Overall, task-condition effects were observed for six linguistic features across fluency, lexical richness, and syntactic complexity.
Fixed Effects of Task Condition on Fluency, Lexical Richness, and Syntactic Complexity.
Note. SE = standard error; CI = confidence interval; Task condition coded as 0 = human; 1 = machine. The p-values were adjusted using the FDR correction method.
Significant differences were observed for two fluency measures. Participants produced substantially more syllables (β = −164.82, p < .001) and exhibited a higher speech rate (β = −.50, p < .001) in human interactions. No significant task condition difference was found for articulation rate (p = .28).
Two of the six lexical richness indices differed significantly, reflecting reduced lexical variation in the E-Talk condition. While lexical density (LD) was higher with E-Talk (β = .02, p < .001), lexical diversity as indexed by root type-token-ratio (RTTR) was lower (β = −.32, p < .001). No significant task condition differences were found for Lexical Sophistication-II (LS-II), Number of Different Words (expected sequence 50) (NDW-ES50), corrected verb variation-I (CVVI), or adverb variation (AdvV).
Two measures showed significantly greater syntactic complexity in the E-Talk condition. Specifically, participants produced a longer mean length of sentence (MLS) (β = 1.38, p < .001) and a higher sentence complexity ratio (C/S) (β = .17, p < .001). The task-condition effect for coordinate phrases per T-unit (CP/T) was not significant (p = .64).
Interactional Features
Table 6 provides the descriptive statistics of the interactional features across task conditions. Overall, frequencies of interactional features varied considerably, with some behaviors occurring regularly while others were infrequent or absent in many interactions. Across both conditions, several features related to idea contribution and development occurred relatively frequently. For example, Feature 11 (ideas elaboration and justification) showed comparatively high mean frequencies in both task conditions. Similarly, features such as Feature 3 (new idea initiation), Feature 5 (task-relevant idea contribution), and Feature 10 (question-based invitation) were found regularly across interactions, with a maximum frequency of nine occurrences. In contrast, several features, including Feature 13 (interaction steering), Feature 14 (joint decision negotiation), Feature 17 (view acknowledgment and compromise), and Feature 18 (partner support), showed very low mean frequencies, indicating that these behaviors occurred only sporadically across task conditions.
Descriptive Statistics of Interactional Features Across Task Conditions.
Note. N = number of participants; M = mean; SD = standard deviation. Min and Max represent the minimum and maximum frequencies of interactional features observed within a single interaction.
To address whether these interactional features differ between the two conditions, each feature was examined using a separate GLMM, with task condition as the fixed effect and participant and topic as random intercepts. Table 7 summarizes the interactional features with statistically significant task-condition effects, while full results for all features are reported in Supplementary Material 10. Selected excerpts from the interaction data are provided below to illustrate key quantitative patterns.
Fixed Effects of Task Condition on Interactional Features.
Note. IRR = incidence rate ratio, SE = standard error; 0 = human task condition, 1 = machine task condition. The p values were adjusted using the FDR correction method.
Participants were more likely to introduce new ideas after completing the discussion of the current one with E-Talk (β = .62, p = .01) and to use appropriate language to shift from one idea to another (β = .58, p = .02). Excerpt 1 illustrates this pattern, in which Participant 7 acknowledges E-Talk’s idea (“Yes, it’s much easier”) before explicitly shifting the discussion (“And another question is. . .”) to introduce a new concern regarding the potential exaggeration of self-reported contributions.
Excerpt 1 P2: Participant 7; Task: Struggling with group project (average holistic score = 15.25 CSE = 6)
In terms of responding to a partner, polite disagreement with justification also occurred more often in E-Talk interactions (β = .50, p = .03). Excerpt 2 demonstrates this behavior, in which Participant 30 expresses mild disagreement (“Not really.”) in response to E-Talk’s proposed solution, followed by an explicit justification citing the large number of scooters on campus and the resulting difficulty of charging. E-Talk’s neutral repair move (“I apologize for the confusion”) and invitation to elaborate (“Could you please provide more details. . .”) appear to lower the social and affective cost of disagreement, encouraging the participant to expand on their position.
Excerpt 2 P30: Participant 30; Task: Struggling with campus transportation (average holistic score = 13.25, CSE = 6)
Regarding interaction maintenance, actively inviting the partner by asking questions was substantially more frequent in E-Talk interactions, with the expected frequency being more than twice that observed in human interactions (β = .82, p < .001). Mixed patterns were observed for features related to negotiation toward a common decision. While beginning to negotiate toward an outcome at an appropriate time occurred markedly more often in E-Talk interactions (β = 1.93, p < .001), working toward a decision by synthesizing and evaluating points was less frequent in E-Talk interactions than in human interactions (β = −1, p < .001). As shown in Excerpt 3, Participant 2 takes an active stance in driving the discussion toward a conclusion, explicitly guiding E-Talk to wrap up while maintaining their leading role through directed questions, reflecting an awareness of their increased responsibility in building consensus with a machine partner.
Excerpt 3 P2: Participant 2; Task: Struggling with campus transportation (average holistic score = 11.8, CSE = 7)
Excerpt 4 captures the contrasting pattern observed in human interactions, in which Participant 2 engages in collaborative negotiation by aligning with the interlocutor’s suggestions, expanding on them, and jointly refining the emerging conclusion through mutual co-construction rather than being explicitly directed by either interlocutor.
Excerpt 4 P2: Participant 2; HI: Human interlocutor; Task: Struggling with group project (average holistic score = 13.5, CSE = 7)
Finally, listener support behaviors (e.g., back-channeling) were far less common with E-Talk interlocutors, with the expected frequency reduced to near zero relative to human interactions (β = −4.61, p < .001).
Discussion
Motivated by the growing interest in LLM-driven SDSs in L2 speaking assessment and the need to capture the distinctive performance patterns associated with SDS-delivered interaction, the present study examined how speaking performance is elicited when interacting with E-Talk, a self-built LLM-driven SDS, in comparison with a human interlocutor. Specifically, it investigated learners’ speaking scores across the two interactional contexts (RQ1), drawing on both holistic ratings and analytic dimensions, including fluency, language use, pronunciation, and IC. Building on this, the study further explored how fine-grained linguistic and interactional features are realized in SDS-delivered interaction (RQ2), using automated measures of fluency, lexical richness, and syntactic complexity, alongside micro-level coding of interactional behaviors. Given the exploratory nature of the study and the relatively small sample size, the findings reported here warrant cautious interpretation and are intended to offer initial empirical evidence that informs future large-scale investigations, rather than definitive conclusions about the validity of SDS-based speaking assessments.
RQ1: Comparison of Participants’ Speaking Scores in E-Talk and Human Interactions
Overall, participants tended to receive higher scores in the human condition, with statistically significant differences observed in pronunciation and language use, whereas no significant differences were found in fluency or IC. These results are consistent with prior studies (e.g., Ockey & Chukharev, 2021; Ockey et al., 2023), though some nuanced differences emerged.
Scores in the SDS condition were generally lower, particularly for pronunciation and language use, echoing prior research reporting higher scores in human interactions (e.g., Ockey & Chukharev, 2021). At the same time, whereas earlier work did not observe significant differences in language use across task conditions, the present study found significantly lower language use scores in the SDS condition. One possible explanation is that participants may have paid closer attention to clarity and accuracy when interacting with humans, possibly due to heightened concerns about comprehensibility and social impression. In contrast, when interacting with an automated interlocutor, they may have perceived less social pressure, leading to slightly reduced control over these features. This aligns with Timpe-Laughlin et al. (2023), who found human interaction to be more socially engaging than human–machine exchanges. With respect to the lower pronunciation scores in the SDS condition, it likely concerns the increased salience of pronunciation-related issues for raters. In human interaction, mutual understanding can be achieved through multiple interactional resources beyond speech alone, including non-verbal cues and back-channeling, which allow participants to signal attention and comprehension. This is further supported by the significantly lower levels of interactive listening behaviors observed in the SDS condition. In contrast, interaction with the SDS relied almost exclusively on the audio channel to establish mutual comprehension. Consequently, instances of misunderstanding made pronunciation clarity and intelligibility more visible to raters, potentially contributing to lower pronunciation scores in the SDS condition (see also Supplementary Material 5, for example transcripts).
No significant main effect of task condition on fluency ratings was found. While some earlier studies have reported slightly lower fluency scores in SDS-delivered tasks, these effects were minimal and were attributed to occasional system interruptions that could disrupt speech continuity (e.g., Ockey & Chukharev, 2021). In the present study, due to the turn-based, user-controlled recording and submission mechanism of E-Talk, overlapping speech and system-initiated interruptions were unlikely and therefore rare in the SDS condition. This click-based turn-taking mechanism allows participants more planning time and the opportunity to express their ideas without time pressure. Although learners sometimes hesitated or paused, reflected in slower speech rates but similar articulation rates (discussed below), these behaviors likely represent context-appropriate planning rather than the lack of speech fluency. This contrasts with other voice-activated SDSs (e.g., Choi & Oh, 2026; Eguchi et al., 2025), where recurrent overlaps and interruptions were observed. In those systems, the silence-based turn-transition mechanism likely treated some pauses as turn-yielding cues, resulting in early or overlapping turn entries relative to the interlocutor’s pause. Collectively, these findings underscore the need to consider system design when interpreting fluency performance in SDS contexts. Variations in turn‑management protocols, timing control, and response triggers may shape how fluency is both realized and perceived by raters. In light of this, stakeholders should critically evaluate the interactional affordances of different systems, recognizing that design features may influence observed performance.
Interestingly, no main effect of task condition was found in IC scores. Previous research has suggested lower IC scores when test-takers interacted with machine partners (e.g., Ockey & Chukharev, 2021). One plausible explanation relates to the evolving nature of SDSs. The LLM-driven SDSs allow for interactive behaviors, such as responding, clarifying, and negotiating meaning. The generated varied and responsive turns may have created a fluid interactional environment, making the dialogue less dependent on test-takers to initiate specific interactive moves, i.e., behaviors that raters often associate with strong IC (Timpe-Laughlin et al., 2025). These capabilities may have reduced the contrast between human–human and human–machine interactions in terms of observable opportunities for demonstrating IC. This is partially supported by the analysis of interactional features, which did not detect statistically significant task-condition effects for approximately 67% of the coded features. In addition, IC is a multidimensional construct encompassing responding, developing ideas, and demonstrating interactive listening. Collapsing these dimensions into a single composite score may have obscured subtle differences in interactional performance across task conditions. Taken together, the absence of a significant difference in IC scores should be interpreted with caution. The finding may reflect both the increasingly interactive affordances of LLM‑driven SDSs, which reduce observable contrasts in interactional behaviors, and potential limitations in how IC was operationalized and aggregated into a single score. Without direct evidence of raters’ interpretive processes, it remains difficult to disentangle these influences. Future research incorporating rater questionnaires or interviews would help clarify how raters perceive and evaluate interactional behaviors when comparing human and machine interlocutors.
RQ2: Comparison of Participants’ Linguistic and Interactional Features in E-Talk and the Human Interactions
Linguistic Features
In general, participants produced less fluent, less lexically diverse, but more syntactically complex speech with E-Talk than with a human interlocutor, highlighting the influence of communicative context on spontaneous language production. These findings partially aligned with previous studies on role-plays (Dombi et al., 2024; Timpe-Laughlin et al., 2022) while also illustrating the unique affordances of LLM-driven SDS for L2 speaking assessment.
First, the fluency measures revealed a main task-condition effect, particularly in the amount of speech produced and temporal distribution. Participants in the SDS condition produced fewer syllables and a lower speech rate. However, articulation rate showed little evidence of a task-condition effect, indicating that the reduced speech rate may be more closely related to longer pauses than to slower articulation. This pattern likely reflects the click-and-submit interactional process discussed earlier, which led to delayed speech onset when interacting with E-Talk. In contrast, the human interlocutor offered richer interactional resources, such as collaborative completion and back-channeling, which helped to minimize extended pauses (see Supplementary Material 5, for illustrative CA‑based transcripts). From a rater’s perspective, however, fluency scores did not differ significantly across task conditions, suggesting that the observed pauses might not be strongly penalized as disfluency. These findings highlight the need to interpret fluency measures in SDS tasks within their specific interactional context, where a lower speech rate may reflect adaptation to the system’s limited responsiveness rather than an underlying deficiency in fluency. Future research may examine raters’ scoring processes through interview or think-aloud methods to better understand how such behaviors are perceived and evaluated.
In contrast, syntactic complexity and lexical density were higher in the SDS condition, with increased C/S, MLS, and LD. These findings suggest that the SDS context elicited a more information-dense and structurally elaborate speaking style. Specifically, higher C/S and MLS in the SDS condition suggest that participants tended to combine multiple T-units into more complex clausal structures, indicating an effort to “package” ideas and convey nuanced or logically connected content. This pattern is complemented by higher LD, which reflects a denser use of content words. In the absence of interactive cues (e.g., back-channeling), participants may have compensated by condensing multiple ideas within single utterances to minimize potential miscommunication. Thus, reduced speech rate and increased complexity should be viewed as complementary adaptations to SDS-specific constraints, reflecting a shift toward efficiency and clarity per utterance. Similar trends were reported in role-play studies (Dombi et al., 2024; Timpe-Laughlin et al., 2022), which elicited longer monologic turns and fewer exchanges, reinforcing their transactional nature. For assessment purposes, this underscores the need to consider how SDS features may influence the construct being measured, potentially favoring information delivery over interactional fluency.
Lastly, despite the increased syntactic complexity and LD in SDS exchanges, participants used less diverse vocabulary, as indicated by the reduced RTTR. This contrasts with Timpe-Laughlin et al. (2022), who reported that lexical variety indices showed little discrepancy between task conditions. The reduced diversity may indicate greater reliance on familiar or high-frequency vocabulary to maintain clarity and efficiency in SDS interactions. This is in line with the lower language use scores and suggests that SDS tasks might impact participants’ employment of their lexical and syntactic resources. While reduced lexical diversity does not necessarily indicate a limitation in learners’ vocabulary knowledge, it does highlight the need to carefully interpret lexical diversity measures in technology-delivered tasks, as task design and interactional conditions may shape language production in ways that do not fully reflect learners’ underlying lexical competence. Additionally, we highlight the concept of “robot-recipient design” (Tuncer et al., 2024), which refers to speakers’ systematic adaptations toward a machine interlocutor. Prior studies have documented such adaptations in human–machine interaction, including shifts in lexical choice, syntactic complexity, interactional style, and discourse organization (Choi & Oh, 2026; Dombi et al., 2022). Distinctive patterns of language use may thus reflect how speakers adjust to the system. Future research could analyze the SDS’s language in depth to explore its potential role in shaping speakers’ recipient design.
Interactional Features
Among the 21 micro-interactional features, seven differed significantly between conditions, reflecting variations in contributing ideas, responding to a partner, maintaining interaction, negotiating decisions, and interactive listening. For the remaining features, the analyses showed little evidence of task-condition effects. This overall pattern suggests that LLM-driven SDSs can elicit a broad range of IC constructs, aligning with findings from Gokturk and Chukharev (2024), Ockey and Chukharev (2021), and Ockey et al. (2023). Nonetheless, the observed differences indicate that certain IC aspects remain sensitive to task conditions, consistent with recent findings (e.g., Eguchi et al., 2025).
Participants were more likely to introduce new ideas and use appropriate language to shift ideas when interacting with E-Talk. This pattern may reflect a stronger task completion orientation in the SDS condition, with participants choosing to move the discussion forward by explicitly marking transitions and introducing additional considerations to ensure adequate coverage of options. A similar orientation to progressivity has been reported in Choi and Oh (2026), where the participant strategically let go of minor points to move the interaction forward.
Polite disagreement with justification occurred more frequently in E-Talk interactions, which may reflect the reduced social and affective risk associated with disagreeing with a machine interlocutor. The LLM’s neutral and non-evaluative stance may create a psychologically safer environment for expressing disagreement, even when followed by explicit justification. This reduction in interactional anxiety may allow for a more authentic demonstration of learners’ underlying language ability, as affective factors that are arguably construct-irrelevant are less likely to interfere with linguistic performance.
The substantially higher frequency of inviting the interlocutor by asking questions in E-Talk interactions may reflect differences in turn-taking mechanics. Unlike human interlocutors, SDSs rarely interrupt, prompting participants to take more initiative in advancing the conversation by contributing ideas sequentially and asking explicit questions to maintain flow and avoid breakdowns (Karatay & Xu, 2025). This tendency may be particularly salient in E-Talk, where the availability of moment-by-moment turn-by-turn feedback is more limited, increasing the need for participants to manage topic progression explicitly. Prior research suggests that turn-taking in SDS-delivered role plays differs strikingly from human interaction, as SDS offer less support for turn management and fewer interactional resources, reflecting their “reduced responsiveness and limited temporal sensitivity” (Eguchi et al., 2025, p. 29). Consequently, some turn-taking behaviors in SDS tasks may reflect system conditions rather than participants’ turn-management ability alone. Such adaptive behavior may also reflect participants’ familiarity with LLM-based tools, as many reported prior experiences using them. While this familiarity may introduce construct-irrelevant variance, it also raises the possibility that it constitutes part of the construct itself, insofar as it reflects participants’ artificial intelligence (AI) literacy. Given the increasing integration of LLMs into daily life, it may be both difficult and unnecessary to exclude such familiarity, lending support to calls for construct expansion to include effective communication with AI interlocutors (Xi, 2023).
Participants showed discrepancies in how they negotiated toward a common decision across task conditions. In the E-Talk setting, they more frequently initiated negotiation at appropriate points toward an outcome, whereas in human interactions, they more often worked toward a decision by synthesizing points raised in the discussion. This divergence may reflect the system’s predictable responsiveness, which encouraged participants to rely on the system to conclude the discussion, as well as the absence of relational concerns such as politeness, leading to more direct negotiation styles. Participants also appeared aware of their “increased role” in building consensus with a machine partner, taking a more active stance (Dombi et al., 2022, p. 8), in contrast to the step-by-step co-construction and mutual alignment more characteristic of human interactions.
Finally, the higher frequency of listener support and interest displays in the human condition highlights a key limitation of SDS-delivered tasks in capturing interactive listening. Despite limited inclusion in existing rubrics, interactive listening is increasingly recognized as a valid indicator of IC, supported by its link to proficiency (e.g., Galaczi, 2014; Lam et al., 2023; Roever & Kasper, 2018). However, SDS tasks pose challenges for assessing this construct, raising concerns about construct underrepresentation (Gokturk & Chukharev, 2024; Timpe-Laughlin et al., 2022). This may stem from the absence of social cues and non-verbal signals, like back-channeling, nodding, or eye contact, which are integral to human interaction but less relevant in human–machine exchanges, as also noted by Karatay and Xu (2025) and Eguchi et al. (2025). For example, E-Talk currently uses only a single mechanical cue (a click) to initiate a turn, reducing test-takers’ opportunities to determine appropriate turn-taking based on verbal and non-verbal cues. Consequently, the LLM-driven SDS task appears to represent a narrower construct of turn-taking-related IC.
Beyond the significant differences discussed above, several features showed near-zero frequencies in E-Talk interactions while occurring occasionally in human interactions (i.e., interaction steering, joint decision negotiation, view acknowledgment and compromise, and partner support). Although these differences did not reach statistical significance, the directional pattern suggests that certain interactional behaviors may be less readily elicited in SDS-delivered communication, potentially reflecting the inherent constraints of human–machine interaction on naturalistic social behaviors. These patterns warrant further investigation with larger samples.
Conclusion
This study compared human–machine and human–human paired discussion tasks across ratings, linguistic features, and interactional behaviors. While the two task types shared broadly similar performance profiles, differences highlight the unique affordances of SDS-delivered tasks. Participants received slightly lower scores in pronunciation and language use, while IC and fluency remained similar. Linguistically, SDS interactions featured reduced speech rate and lexical diversity but increased syntactic complexity. Interactionally, SDS tasks prompted greater initiative, including proposing new ideas, expressing disagreement with justification, asking questions, and advancing negotiation, whereas human interactions supported more collaborative decisions and richer listener feedback.
These findings offer important insights into the appropriate use of LLM-driven SDSs in L2 speaking assessment. Although the distinct interactional affordances of SDSs echo prior concerns regarding construct underrepresentation (e.g., Gokturk & Chukharev, 2024; Timpe-Laughlin et al., 2022), the observed differences in performance appear to reflect task-dependent adaptations to interacting with an AI system rather than uniform declines in speaking ability. This distinction stresses the importance of interpreting SDS-based performance in relation to the specific interactional affordances through which interaction is realized. A central validity concern, as raised by Jang and Sawaki (2025), is whether performance in such tasks “truly reflects language ability rather than a learner’s ability to navigate AI-generated outputs” (p. 362). The present findings suggest that this concern may be more productively addressed by conceptualizing LLM-driven SDSs as a distinct but standardized interlocutor type, rather than as a deficient approximation of human interaction. At the same time, the differences observed across LLM-driven SDSs indicate that not all AI tools are equivalent for assessment purposes (Huang & Yan, 2025). This heterogeneity points to the need for further research examining how specific interaction mechanisms, including turn-taking mechanics, interface design, and potential communicative pressure, shape task performance and interactional behavior, and how these features relate to construct validity considerations. Establishing an evaluative framework for AI-based dialogue systems will be essential for guiding principled tool selection and implementation, ensuring that technological innovation supports, rather than compromises, the construct validity of L2 speaking assessment.
The present study has several limitations. First, the analysis primarily focused on rating and analyzing test-takers’ oral responses, without examining stakeholders’ perceptions of using SDS for L2 speaking assessment. As test-takers constitute a key stakeholder group in speaking assessment, future research should incorporate questionnaire surveys or in-depth interviews to explore their task experiences and attitudes. Similarly, raters’ perceptions also warrant investigation, particularly to determine whether potential awareness of task conditions may have influenced their scoring judgments. Second, although statistical modeling was employed to identify performance differences across conditions, the study did not conduct fine-grained discourse analysis of the interactions. Consequently, the analysis was limited in its ability to capture how interactional moves unfolded sequentially or how test-takers oriented to prior turns in real time. Future research could integrate quantitative analyses with detailed discourse-analytic approaches to provide a richer account of turn-by-turn interactional dynamics in both human–machine and human–human contexts. Third, the relatively small sample size limits the generalizability of the findings. Although the within-subject design and FDR correction were employed to enhance statistical rigor, future studies could involve larger samples of both test-takers and raters to broaden the empirical base for understanding the distinctive affordances and limitations of SDS-delivered paired speaking assessments. Finally, although E-Talk employed a multimodal interface with a lip-synced avatar, the present study relied solely on audio recordings and did not examine non-verbal aspects of interaction. Nor was response latency systematically measured. Future research could incorporate analyses of non-verbal cues and response latency to investigate how multimodal and technical factors shape turn-taking, interactional behaviors, and rater perceptions in SDS-delivered speaking tasks.
Footnotes
Author Contributions
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by the Zhejiang Provincial Planning Office of Philosophy and Social Science (26QNYC003ZD).
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Disclosure of the use of artificial intelligence (AI)
Generative AI was used during the manuscript preparation and revision stages. The authors used OpenAI’s ChatGPT (accessed between 2025 and 2026) as a language and editorial support tool. Specifically, ChatGPT was used for language checking, formatting, and reviewing the clarity of selected passages, including the presentation of findings, figure captions, and responses to editorial queries. The tool was not used to design the study, generate test materials, collect data, conduct statistical analyses, code interactional data, produce results, or make final methodological or interpretive decisions. Any suggestions generated by the tool were critically reviewed and, where appropriate, revised by the authors, who take full responsibility for the final content of the manuscript.