Effects of ChatGPT-Based Human–Computer Dialogic Interaction Programming Activities on Student Engagement

Student Engagement in Programming Activities

Student engagement refers to the energy and effort students dedicate to their learning activities, observable through behavioral, cognitive, and emotional indicators (Bond, 2020). Maintaining active and sustained engagement is essential in programming education (Bile, 2022). Existing studies have shown that when students’ engagement in learning increases, they will acquire higher levels of computational thinking skills in programming learning (Li, Sun, et al., 2024). Students who exhibit strong engagement in their programming activities are more likely to employ comprehensive learning strategies, which is conducive to the gradual development and mastery of computational thinking skills (Lee, Wu, et al., 2024). Programming education currently faces numerous challenges, including the monotony and abstraction of programming languages, varying student backgrounds and abilities, a lack of problem-solving and critical thinking skills, weak connections between theoretical and practical learning, and low motivation and interest in programming (Yildiz Durak, 2018, 2020). To address these issues, empirical studies have explored methods such as gamified education (Smiderle et al., 2020), adaptive immediate feedback (Marwan et al., 2020), augmented reality technology (Theodoropoulos & Lepouras, 2021), and pair programming (Xu & Correia, 2024a, 2024b). These strategies have been shown to positively affect student engagement while fostering computational, critical, and problem-solving skills (Scherer et al., 2020). The effectiveness of these approaches in enhancing student engagement ultimately stems from their ability to encourage interaction with learning content, tools, environments, or peers. Research indicates that effective interactions during programming, such as direct dialogues, code reviews, or collaborative problem-solving, promote active participation in programming tasks (Gharbaoui et al., 2024). Therefore, integrating dialogic interaction into programming activities is pivotal for enhancing student engagement.

Dialogic Interaction Promoting Student Engagement

Dialogic interaction in programming activities aims to support students in developing a deep understanding of programming languages while enhancing higher-order cognitive skills and problem-solving abilities (Prasad et al., 2022). This objective is achieved through collaborative tasks such as writing code, executing programs, debugging, solving problems, and providing peer feedback. Among dialogic interaction strategies, pair programming has become one of the most widely used approaches in programming education. In pair programming, two students share a single computer to collaboratively construct knowledge and solve problems. One student writes the code, and the other reviews and guides the process by offering design and logic suggestions (Xu et al., 2023). These roles are periodically rotated, ensuring active participation and shared learning for both students. Compared with solo programming, pair programming has been shown to improve students’ programming knowledge, enhance learning outcomes (Xu & Correia, 2024a, 2024b), and promote the development of computational thinking (Zhou et al., 2024). Additionally, it fosters a positive dialogic interaction process characterized by cognitive load distribution, complementary skill integration, shared knowledge resources, and emotional support, all of which enhance student engagement and learning performance (Tan et al., 2024). However, Krizsan and Lambic (2024) highlighted challenges in pair programming because of individual differences in programming skills, personality traits, and learning styles. Considerable disparities in skill levels can lead to frustration for less proficient students or boredom for more advanced ones, adversely affecting learning efficiency and motivation. Moreover, students may have differing needs regarding the execution of pair programming, such as role allocation, task management, and communication strategies. In cases where one participant fails to provide sufficient support or feedback, the benefits of pair programming may not be fully realized (Bowman et al., 2021). These limitations highlight the need for alternative approaches that can provide personalized support and equitable learning experiences, particularly for students with varying skill levels. Recent advancements in artificial intelligence (AI) have introduced new possibilities for dialogic interaction, with tools like ChatGPT offering a promising solution to address these challenges.

ChatGPT-Based Dialogic Interaction

ChatGPT distinguishes itself from traditional programming environments and tools as an artificial intelligence language model designed to interact with users through natural language. Research has indicated that ChatGPT can be effectively integrated into programming courses, yielding cognitive and non-cognitive benefits (Gong et al., 2024). Cognitively, ChatGPT provides detailed and guidance on programming concepts and practices, facilitating students’ understanding of programming languages (Jošt et al., 2024). Additionally, it delivers immediate feedback, aiding students in evaluating their comprehension. This functionality supports essential components of self-regulated learning, including planning, goal setting, self-monitoring, and self-assessment (Hartley et al., 2024). From a non-cognitive perspective, ChatGPT enhances student engagement by offering dialogic interaction during programming activities (Qureshi, 2023). On the one hand, its ability to simulate human-like dialogue creates a more interactive and engaging learning experience, fostering students’ interest and motivation in programming education (Kosar et al., 2024). On the other hand, as a non-judgmental tool, ChatGPT encourages exploration and error-making, helping students learn through trial and error while reducing pressure and increasing their sense of achievement (Ma et al., 2024). Unlike pair programming, ChatGPT compensates for the knowledge and skill limitations of human partners (Bian et al., 2024). Furthermore, it can adapt to individual personality differences and learning paces, maintaining balance in human-computer dialogue (Bansal et al., 2024). However, the accuracy of ChatGPT in programming tasks is affected by factors such as task complexity, programming languages, problem descriptions, questioning approaches, and program size (Bucaioni et al., 2024; Rahman & Watanobe, 2023; Sun, Boudouaia, Zhu et al., 2024). For example, ChatGPT may produce syntactically correct but logically flawed code, especially for complex or ambiguous tasks. Such errors require students to possess sufficient foundational knowledge to detect and correct them, which may be challenging for novices (Popovici, 2024). Additionally, students’ artificial intelligence (AI) literacy and their cognitive understanding of ChatGPT can limit the effectiveness of their dialogic interactions and engagement in programming activities (Jing et al., 2024). For example, the frequent use of code generation tools can lead to surface-level engagement, where students focus more on patching AI outputs than on understanding the underlying principles (Bringula, 2024). Therefore, it is essential to design scientifically grounded and pedagogically sound ChatGPT-based programming activities that maximize the benefits of AI-driven dialogic interaction while minimizing potential drawbacks.

Additionally, we conducted a search in multiple academic databases, including Web of Science, Scopus, and Google Scholar, using keyword combinations: “ChatGPT” AND (“K-12” OR “high school” OR “secondary education”) and “ChatGPT” AND (“programming education” OR “computational thinking”). We screened studies related to the application of ChatGPT in K-12 or high school programming education, excluding those that only involved higher education or non-educational fields. A total of 115 studies on the application of ChatGPT in K-12 or high school education were identified, of which 52 focused on K-12 or high school programming education. Moreover, we found that most studies concentrated on the technical functions and potential applications of ChatGPT, while in-depth empirical research on its impact on student learning behaviors, cognition, and emotions in K-12 or high school education remains scarce. Therefore, although the application of ChatGPT in education is gradually increasing, research in K-12 or high school environments is still limited, and more empirical studies are urgently needed to fill this gap. To bridge this research-practice gap, this study investigates the specific effects of ChatGPT-based activities on students’ behavioral, cognitive, and emotional engagement. The study is guided by the following research questions:

RQ1: What are the differences in the effects on student behavioral engagement between ChatGPT-based activities and pair programming activities?

Participants

This study employed a quasi-experimental design to compare student engagement between those participating in ChatGPT-based activities (experimental group) and those involved in pair programming activities (control group). Apart from the intervention tool, the teachers, teaching methods, and learning activities were identical for both groups. The study was conducted in a Python programming course at a middle school in China, with 109 students voluntarily participating. The experimental group included 53 students, whereas the control group comprised 56 students. Before the experiment began, a comparison of characteristics was conducted between the experimental and control groups, including gender, age, programming experience, and academic performance. The gender distribution was relatively balanced between the experimental group (23 males and 30 females) and the control group (29 males and 27 females), with no significant differences observed (χ² = 0.56, p = .45). The average ages of the experimental and control groups were 15.2 years (SD = 0.8) and 15.3 years (SD = 0.7), respectively, with no significant differences (t = 0.72, p = .47). A survey revealed no significant differences in programming experience between the two groups (Experimental group: M = 1.8 years, SD = 0.9; Control group: M = 1.7 years, SD = 0.8; t = 0.65, p = .52). Additionally, the average grades of the two groups in mathematics and information technology courses were compared, with no significant differences found (Mathematics: t = 0.89, p = .38; Information Technology: t = 0.76, p = .45). Furthermore, all participants were given a pretest on computational thinking skills and Python programming ability. The results showed no significant differences between the experimental and control groups, with average scores of 72.5 (SD = 8.3) and 73.1 (SD = 7.9), respectively (t = 0.34, p = .73). Before the study, students were told to record their behavior and that the data would be used for research purposes only. All screen recordings were anonymous to protect student privacy.

Experimental Process

Figure 1 illustrates the design and implementation process of the study, which is divided into three stages. In the first stage, teachers provide lectures on the fundamental concepts and foundational knowledge of Python programming. This stage, which lasted for 5 weeks with two one-hour sessions per week (totaling 10 hours), includes Python programming fundamentals (control structures, functions, methods, etc.) and data structures (lists, stacks, queues, etc.). To ensure students were well-prepared for subsequent tasks involving the greedy algorithm, the lectures also incorporated practical examples and exercises that gradually increased in complexity. For instance, students were introduced to the concept of max-heap as a foundational data structure, which later supported their understanding of the greedy algorithm in the card-splitting problem. For students in the experimental group, researchers demonstrate how to interact with ChatGPT for assistance, such as finding solutions, generating programming examples, and optimizing programs. Simultaneously, students in the control group received training on pair programming collaboration techniques. For example, students were taught how to better negotiate and alternate roles during programming tasks to ensure balanced participation and shared learning opportunities. They were also taught how to address potential issues that may arise in pair programming, such as unequal participation and communication barriers, thereby creating a supportive and productive learning environment.

OPEN IN VIEWER

Figure 1.

Experimental procedure.

In the second stage, researchers thoroughly considered the existing cognitive levels and programming experience of high school students. To ensure that students could smoothly comprehend and apply the greedy algorithm, the teacher first explained the working principles and applicable scenarios of the greedy algorithm in detail through practical cases and step-by-step derivations, ensuring that students could grasp the basic concepts and advantages. Subsequently, we designed a phased learning activity that breaks down complex programming tasks into multiple manageable sub-tasks, including decomposition, pattern recognition, abstraction, algorithm design, and evaluation. The aim is to allow students to progressively build an understanding of the entire algorithm as they complete each sub-task. At each stage, students were engaged in independent practice, exploring and applying the learned algorithms through hands-on operations. The teacher provides appropriate guidance and support throughout the process.

In the third stage, the experimental and control group students were required to complete a programming task based on the problem of splitting the cards evenly within the greedy algorithm framework with identical objectives and time constraints. Both groups are required to engage in five key programming activities: decomposition, pattern recognition, abstraction, algorithm design, and evaluation.

Pair Programming Activity

Students in the control group participated in the programming activities using the pair programming method. Previous research has shown that both male - male and male - female pairs in programming learning can significantly boost students’ computational thinking skills and programming self – efficacy (Wei et al., 2021). Since the control group had a relatively balanced gender ratio (29 males and 27 females), to maximize the advantages of pair learning, we adopted a male - female pairing strategy, with the remaining two males forming a pair. Based on the Python programming pretest scores, male and female students were ranked separately from highest to lowest. First, the highest-scoring male was paired with the lowest-scoring male within the male group. Subsequently, a cross-pairing strategy was adopted, pairing high-performing males with low-performing females or high-performing females with low-performing males. However, we did not assign fixed roles within the pairs. Instead, students were allowed to negotiate and decide their respective roles during the programming activities, aiming to promote natural interaction and collaboration between students while giving them more autonomy to explore different modes of cooperation. Figure 2 provides an example of students engaged in pair programming activities.

OPEN IN VIEWER

Figure 2.

Example of pair programming activities.

In the decomposition activity, students were required to identify the objectives and constraints of the fair distribution of playing cards problem and divide it into multiple sub-problems to be addressed. In the pattern recognition activity, students analyzed the distribution patterns of the cards, attempting to determine the demand value for each pile. They considered how to adjust quantities through movements between adjacent piles, thereby recognizing operational patterns within the problem. In the abstraction activity, students defined variables to represent the quantity, state, and average of each card pile, describing how movement operations affect pile states using mathematical expressions. Additionally, they were encouraged to create flexible, reusable functions. During the algorithm design activity, students developed a Python program to solve the problem of evenly splitting the cards using the greedy algorithm approach. Finally, in the evaluation activity, after completing their designs, students verified whether their program produced correct results. They also assessed whether the algorithm addressed special cases, such as scenarios where the number of cards in a pile became negative. Students were further encouraged to optimize the algorithm’s performance based on their original design.

ChatGPT-Based Activities

Unlike the control group, students in the experimental group engaged in human–computer dialogic interaction programming activities using ChatGPT. At each stage of the activity, students interacted with ChatGPT to receive programming assistance. Figure 2 illustrates a typical interaction, where ChatGPT provides feedback on a student’s algorithm design.

In the decomposition activity, students identified key information in the problem with ChatGPT’s assistance, breaking it into smaller, more manageable sub-problems. Based on ChatGPT’s feedback, students adjusted the granularity of their decomposition to achieve the most effective results. In the pattern recognition activity, students analyzed similarities in scenarios of the greedy algorithm provided by ChatGPT, identifying patterns and rules relevant to solving the card-splitting problem. They then applied these identified patterns to the solution using ChatGPT’s suggestions. During the abstraction activity, students defined variables to represent the quantity, state, average, and other attributes of each card pile, describing how movement operations altered the piles’ states using mathematical expressions. With ChatGPT’s guidance, they abstracted identified patterns and rules into functional modules, generating functions based on the rules and constraints of the card-splitting problem. In the algorithm design activity, students collaborated with ChatGPT to develop a Python program using the greedy algorithm to solve the card-splitting problem. They effectively utilized ChatGPT’s features, including code generation, completion, hints, error detection, and repair. Students were also encouraged to explore alternative solutions to the problem. In the evaluation activity, students verified whether their program produced correct results and considered whether the algorithm addressed special cases, such as situations where the number of cards in a pile became negative. They entered their complete code into ChatGPT to evaluate its correctness, efficiency, readability, and other qualities. Based on ChatGPT’s feedback, students refined and optimized their algorithm’s performance. Figure 3 illustrates an example of students engaging in human–computer dialogic interaction programming activities using ChatGPT.

OPEN IN VIEWER

Figure 3.

Example of a programming activity within a ChatGPT conversational interaction.

In the experiment process, to mitigate the risk of students over-relying on ChatGPT, which could potentially affect their knowledge retention, several measures were implemented. First, researchers explicitly informed students that ChatGPT’s responses might contain logical flaws, syntax errors, or deviations from the task objectives (e.g., generating redundant code or straying from the core logic of the greedy algorithm). Students were required to verify the correctness of ChatGPT’s outputs through questioning, testing, and logical reasoning. Second, both the experimental and control groups received programming task sheets organized in the form of question chains, which aimed to shift students’ focus from “obtaining answers” to “understanding the process” through metacognitive reflection. Furthermore, the task sheets for the experimental group included staged questioning templates and an error reflection module, requiring students to document and analyze the erroneous answer generated by ChatGPT as well as the process of correcting it.

Data Collection and Instruments

Previous research indicates that students’ enhanced behavioral, cognitive, and emotional engagement is respectively reflected in reduced off - task behaviors, increased time spent on advanced cognitive activities, and active participation in programming tasks (Wong & Liem, 2022). Given that assessing computational thinking is a critical component of programming education research (Tikva & Tambouris, 2021), and several validated assessment frameworks have been developed to measure multiple dimensions of computational thinking (Kilic et al., 2021; Lafuente Martinez et al., 2022; Tsai et al., 2021). This study employs three methods for data collection and constructs analysis frameworks based on computational thinking to evaluate learners’ engagement.

First, during the third stage of the activity, the interactions between the experimental group students and ChatGPT, as well as the programming behaviors of both the experimental and control group students on the computer, were recorded using screen recording software. Additionally, the interactions of the control group students were documented through the built-in computer cameras. These recordings captured actions such as code writing, debugging, modifications, and other behavioral data. Click-stream analysis, a technique used to analyze user interaction behaviors on digital interfaces (Yürüm et al., 2023), was applied to these recordings to reflect students’ operational sequences and learning pathways. The behavioral data were then analyzed using the five core computational thinking skills proposed by Anderson (2016) as the coding framework. Each element was measured through observable behaviors processes during programming activities, as show in Table 1. To provide a more comprehensive analysis of students’ engagement, we expanded the behavioral coding framework to include non-learning-related behaviors, such as irrelevant operations and irrelevant thinking. This supplementary analysis enabled a deeper examination of students’ overall behavioral performance in programming activities and allowed for a more nuanced evaluation of the effectiveness of their engagement in programming learning. Independent coders labeled behaviors in the recorded videos using the behavioral coding framework, with markings made every 30 seconds. Both coders were provided with detailed coding guidelines and examples to minimize subjective bias. A consistency test of the coding results yielded a Kappa coefficient of 0.895, indicating a strong level of coding reliability.

OPEN IN VIEWER

Table 1.

Learning Behavior Coding Framework.

Demotion	Index	Coding	Behavior description	Examples
Learning behavior	Decomposition	DC	Students demonstrated decomposition by breaking down the complex problem of evenly splitting cards into smaller, manageable sub-problems	Students break the problem down into “calculate the average of each pile of cards” and “determine the number of cards to move.”
	Pattern recognition	PR	Students engaged in pattern recognition by analyzing recurring rules in card distribution	Students note that “cards can only be moved to adjacent piles” and discuss how to use this rule
	Abstraction	AS	Students created abstract representations of the problem by filtering out unnecessary details	Students define the variable pile_states = [5,3,7] to represent the number of cards in each pile
	Algorithm design	AD	Students developed a systematic solution and implemented the greedy algorithm in Python	Students write code: while max(pile_states) > average:move_cards()
	Evaluation	EV	Students assessed the correctness and efficiency of their algorithms	Students test the code and discover that “the program crashes when the number of cards is negative,” then modify the code to handle this situation
Irrelevant behavior	Unrelated thinking	UT	Students suddenly start discussing topics unrelated to the task during programming, or exhibit obvious signs of distraction	Students suddenly start discussing topics unrelated to the task, such as “last night’s football game.”
	Unrelated operation	UO	Students frequently switch windows during programming, browsing websites unrelated to the task, or using applications unrelated to the task	Students turn on social media to browse news or videos unrelated to their studies

Second, we recorded the conversations and interactions between students (control group) and between students and ChatGPT (experimental group) using audio recording devices or screen recording software. These recordings were subsequently transcribed into textual data and combined with the textual data from the learning reports submitted by the students, collectively providing insights into cognitive changes. The learning reports include analyses of programming tasks, identified challenges, solutions, self-reflections, and learning summaries. We used the computational thinking assessment framework proposed by Brennan and Resnick (2012) as the basis for coding students’ cognitive engagement, modifying it to align with the research objectives. The revised framework retains three primary indicators: computational concepts, computational practices, and computational perspectives, as detailed in Table 2. Computational concepts encompass the programming concepts students engage with, which are transferable to other programming or non-programming contexts. These include variables, operations, functions, conditions, and loops, among others. Computational practices refer to the skills and behavioral patterns students develop during programming. These practices go beyond content acquisition, emphasizing the learning process itself. In this study, computational practices focus on problem analysis, solution design, process development, and evaluation optimization. Computational perspectives describe how individuals perceive and position themselves within computational activities and how they express themselves and interact with the world through computational tools and methods. This study categorizes computational perspectives into three sub-indicators: expressive thinking, inquisitive awareness, and skeptical attitude (Peracaula-Bosch & González-Martínez, 2024). Coders labeled each sentence according to the cognitive coding framework. After completing the coding process, a consistency test yielded a Kappa coefficient of 0.873, indicating good reliability of the coding results.

OPEN IN VIEWER

Table 2.

Cognitive Coding Framework.

Primary index	Secondary index	Code	Cognitive description	Example
Computational concepts	Variables	V	Defining and modifying variables of diverse data types	Student defines variable: Average = sum(pile_states)/len(pile_state) to calculate the average of cards
	Operations	O	Applying operators to execute specified computations	Students use the “+” and “-” operators to calculate the number of card moves
	Functions	F	Creating and utilizing distinct functions for modular programming	Students define the function: move_cards() to handle the movement logic of the cards
	Conditions	C	Implementing conditional statements, including “if” and “if-else” constructs	Students write code: if pile_states[i]>average:move_cards()
	Loops	L	Employing iterative control structures such as “while” and “for” loops	Students write code: while max(pile_states) > average:move_cards()
Computational practices	Problem analysis	PA	Investigating discrepancies by analyzing input datasets and results	Students analyzed the input data and found that “some stacks had significantly higher card counts than average.”
	Solution design	SD	Developing, refining, and sharing solutions using cognitive mapping techniques	Students draw a flowchart to show the logic of the card movement
	Process development	PD	Adopting incremental and iterative approaches, emphasizing modularity, reuse, and thorough testing and debugging in coding	Students first code to realize the essential functions and then gradually refine and debug the code
	Evaluation optimization	EO	Evaluating algorithm performance and comparing metrics to optimize outcomes	Students test the code and find that “the algorithm is inefficient in some cases,” then optimize the code to reduce the number of cycles
Computational perspectives	Thinking expression	TE	Expressing individual problem-solving strategies while recognizing and fostering creativity	Students describe their solution in natural language: “We achieve balance by gradually adjusting the number of cards in each pile.”
	Follower-up awareness	FA	Exploring computational phenomena, principles, methodologies, and foundational logic in depth	Students discuss “why the greedy algorithm is applicable to this problem” and attempted to understand the logic behind it
	Critical attitude	CA	Performing critical analysis of issues, datasets, and solutions while avoiding superficial or preconceived conclusions	Students question the code generated by ChatGPT: “This solution doesn’t handle negative numbers. We need to improve it.”

Third, we collected data on students’ emotional engagement in programming learning using a questionnaire survey. The scale for emotional engagement in learning, developed by Wang and Fredricks (2016), was adapted to fit the study’s objectives. The adjusted scale, presented in Table 3, evaluates four affective dimensions: motivation, self-efficacy, interest, and anxiety. To ensure the validity of the questionnaire, we conducted a pretest to analyze its reliability and structural integrity before formal distribution. The sample for the pretest included 112 students (54 males and 58 females) from the same school who did not participate in the formal experiment. The students were mainly aged between 15 and 17 years old and had a basic understanding of Python programming, but they had not participated in any programming activities assisted by ChatGPT. To ensure the structural integrity of the emotional engagement survey scale, exploratory factor analysis (EFA) was conducted, detailed in Table 3. The EFA results revealed a clear four-factor structure aligned with the hypothesized dimensions of motivation, self-efficacy, interest, and anxiety. All items exhibited strong loadings on their respective factors, which were derived from principal component analysis with varimax rotation. The four factors cumulatively accounted for 62.65% of the total variance, with individual contributions as follows: motivation (18.2%), self-efficacy (16.8%), interest (15.4%), and anxiety (12.3%). Dawson (2017) considers a value ranging from 0.5 to 0.9 as acceptable, so the total variance explained value of approximately 60% is acceptable. Each subscale demonstrated high reliability, with Cronbach’s α values ranging from 0.78 (anxiety) to 0.87 (self-efficacy).

OPEN IN VIEWER

Table 3.

Emotional Engagement Survey Scale.

Dimension	Item	Factor loading	Cronbach’s α
Motivation	My curiosity drives my desire to learn programming	0.76	0.83
	Exploring new concepts in programming motivates me to continue learning	0.82
	My academic grades are the primary reason I study programming	0.68
Self-efficacy	I Feel confident and positive during the programming learning process	0.74	0.87
	I Am eager to tackle challenging tasks in programming classes	0.79
	I Trust my ability to solve difficult programming problems	0.71
Interest	I Enjoy discovering new aspects of programming	0.85	0.81
	I Find programming learning unengaging	0.63
	I Lack enthusiasm for attending programming classes	0.59
Anxiety	I Feel apprehensive when learning new programming concepts	0.72	0.78
	I Frequently experience anxiety during programming activities	0.81
	I Often feel frustrated while learning programming	0.77

Data Analysis

For data analysis, this study utilized lag sequence analysis, cognitive network analysis, and statistical data analysis methods.

First, lag sequence analysis was employed to examine the probability of a behavior occurring after a specific preceding behavior, identifying statistically significant relationships between behaviors (Lei et al., 2024). GSEQ (Generalized Sequential Querier) is a tool for sequence analysis, widely used in lag sequential analysis (Li, Jiang, et al., 2024). In this study, the labeled behavioral sequences were imported into GSEQ to evaluate the state of behavioral engagement through the frequency and sequence transitions of students learning behaviors.

Second, cognitive network analysis was used to visualize the relationships between cognitive elements of individuals or groups, forming cognitive network models that describe cognitive performance when solving complex problems (An & Zhang, 2024). This method was applied to analyze the state and changes in students’ cognitive engagement during programming activities. Initially, the textual data from students’ dialogues and reports were segmented into sentences to ensure accurate labeling and verification. The coded data from the control group (Group A) and experimental group (Group B) were then imported into the ENA Webkit, an important tool for epistemic network analysis, which enables researchers to analyze and visualize the structure and patterns of connections between cognitive elements (Su et al., 2024).

Last, to validate the data from the emotional engagement survey scale, reliability and validity tests were conducted using IBM SPSS 26.0. The Cronbach’s α coefficient was 0.835, and the KMO value was 0.846, confirming good reliability and validity of the data. Descriptive statistics, including mean and standard deviation, were calculated for each dimension. Difference analyses were then conducted to assess the significance of differences in emotional engagement between the two groups.

Results for RQ1 (Behavioral Engagement)

Table 4 presents the learning behavior frequencies and transition sequences of students in the control and experimental groups. The five activity processes are observed in the programming activities of both groups, though the proportions of each activity differ. Algorithm design (AD) accounts for the highest frequency in the learning activities of both the control group (25.7%) and the experimental group (24.1%). Abstraction (AS) represents the smallest proportion for the control group (10.3%), whereas evaluation (EV) is the least frequent activity for the experimental group (11.3%). Regarding unrelated behaviors, the control group exhibits higher levels of unrelated mental (UT, 7.4%) and unrelated behavioral (UO, 8.0%) activities compared with the experimental group. These findings suggest that ChatGPT-based activities may enhance students’ behavioral engagement.

OPEN IN VIEWER

Table 4.

Student Programming Behavior Frequency Statistics.

Group	Behavior code
	Learning behavior					Irrelevant behavior		Total
	DC	PR	AS	AD	EV	UT	UO
Control	488 (18.0%)	410 (15.2%)	277 (10.3%)	695 (25.7%)	417 (15.4%)	200 (7.4%)	217 (8.0%)	2704
Experimental	445 (17.1%)	595 (22.9%)	415 (16.0%)	627 (24.1%)	293 (11.3%)	176 (6.8%)	49 (1.8%)	2600

Seven behavior codes generate 49 possible behavior sequences. Using the residual values calculated by the GSEQ software, sequences with values greater than 1.96 were regarded as significant. Ultimately, 25 significant sequences were identified for the control group and 23 for the experimental group. Figure 4 provides a behavior transformation plot based on these significant behavior sequences.

OPEN IN VIEWER

Figure 4.

Diagram of learning behavior sequence transition. (a) Control group (b) Experimental group.

Figure 4 consists of nodes representing learning behaviors and directed arrows indicating significant transitions between behaviors. The numbers on the arrows represent the adjusted residuals (Z-values), which quantify the significance of each transition (Z >1.96, p < .05). Thicker arrows denote higher Z-values, reflecting stronger sequential dependencies. An independent two-sample t test, assuming unequal variances, showed a statistically significant difference in the “Learning Behavior → Irrelevant Behavior” transition between the control group (mean = 410, SD = 50, N = 27) and the experimental group (mean = 228, SD = 40, N = 25) at the α = 0.05 level (t (50.00) = −10.00, p = .00, Cohen’s d = −3.06). Figure 4(a) and (b) display the programming behaviors of the control and experimental groups, respectively. A typical learning pathway in the control group begins with Decomposition (DC), transitions to Pattern Recognition (PR) (Z = 6.27), followed by Abstraction (AS) (Z = 6.54), Algorithm Design (AD) (Z = 6.85), and finally Evaluation (EV) (Z = 9.58). This sequence (DC→PR→AS→AD→EV) aligns with the problem-solving process in programming. However, significant transitions to Unrelated Thinking (UT) and Unrelated Operations (UO) during Algorithm Design (AD→UT: Z = 7.62; AD→UO: Z = 7.09) indicate students in the control group frequently engage in cognitive activities and operational behaviors unrelated to problem-solving during the algorithm design phase. These include browsing irrelevant information and performing unnecessary interface operations, indicating a lack of focus and engagement in programming behaviors. Contrast with control group, students using ChatGPT exhibited fewer transitions to off-task behaviors. Instead, they showed stronger focus on iterative refinement, such as a significant circular effect was observed between evaluation (EV) and algorithm design (AS) (AD→EV: Z = 9.22; EV→AD: Z = 6.85). However, the integration of ChatGPT introduced some new unrelated behaviors. For example, students were more likely to copy and paste answers or code directly (AD→UO: Z = 4.07; EV→UO: Z = 3.33), or switched between tasks prematurely, moving on to other activities before completing the current one (PR→AD: Z = 4.46). Despite this fact, the study also found that students gradually learned to ask questions more effectively and verify answers through multiple interactions with ChatGPT. This iterative feedback process helps students to progressively deepen their understanding of the problem, even though there may be some dependency in the initial stage.

Results for RQ2 (Cognitive Engagement)

Figure 5 shows the cognitive networks of students in the control and experimental groups. In Figure 5(a), along the X axis, a two-sample t test assuming unequal variance showed control group (mean = −0.22, SD = 0.13, N = 27 was statistically significantly different at the alpha = 0.05 level from experimental group (mean = 0.23, SD = 0.05, N = 25; t (34.49) = 16.94, p = .00, Cohen’s d = 4.57). Along the Y axis, a two-sample t test assuming unequal variance showed control group (mean = 0.00, SD = 0.27, N = 27 was not statistically significantly different at the alpha = 0.05 level from experimental group (mean = 0, SD = 0.25, N = 25; t (50.00) = 0.00, p = 1.00, Cohen’s d = 0). The difference in programming activity strategies is the main reason for the significant difference between the two groups on the X-axis. The lack of significant difference on the Y-axis may be due to the programming tasks having a smaller impact on the cognitive dimension represented by the Y-axis for both groups. In Figure 5(b) and 5(c), nodes (such as PD, FA, FO, etc.) represent different cognitive elements, with larger nodes indicating that the cognitive element occupies a more important position in the students’ thought processes. The thickness of the lines between nodes represents the strength of the association between different cognitive elements. For example, the thicker line between “Process Development” (PD) and “Problem Analysis” (PA) for students in the experimental group suggests that they are more inclined to closely integrate problem analysis with the development of solutions during programming tasks. These differences in network structure further support the advantages of students in the experimental group in terms of cognitive integration and problem-solving abilities.

OPEN IN VIEWER

Figure 5.

Student cognitive network graph. (a) Distribution of cognitive network centroids for control and experimental groups (b) Control group (c) Experimental group.

Figure 6 overlays the cognitive network structures of the control and experimental groups, detailed in Table 5.

OPEN IN VIEWER

Figure 6.

Cognitive network superimposition diagram of control and experimental classes.

OPEN IN VIEWER

Table 5.

Comparative of Cognitive Network Structures Between Experimental and Control Groups.

Category	Control group	Experimental group
Cognitive connections (edge weight>=0.05)	3 connections	8 connections
Key cognitive nodes	Conditions (C)	Solution design (SD)
	Loops (L)	Process development (PD)
	Operations (O)	Thought expression (TE)
	Problem analysis (PA)	Evaluation optimization (EO)
	Process development (PD)	Critical attitude (CA)
	Evaluation optimization (EO)	Functions (F)
		Operations (O)
		Conditions (C)
Strongest associations (edge weight>=0.05)	Conditions ↔ evaluation optimization (C-EO)	Solution design ↔ operation (SD-O)
	Loops ↔ evaluation optimization (L-eo)	Solution design ↔ Thought expression (SD-TE)
	Evaluation optimization ↔ process development (EO-PD)	Solution design ↔ process development (SD-PD)
		Process development ↔ Functions (PD-F)
		Process development ↔ Thought expression (PD-TE)
		Thought expression ↔ operation (TE-O)
		Critical attitude ↔ evaluation optimization (CA- EO)
		Conditions ↔ Functions (C-F)
Computational dimensions	Primarily two dimensions	All three dimensions
	- Concepts	- Concepts (e.g., operations)
	- Practices (Minimal focus on perspectives)	- Practices (e.g., evaluation optimization)

The experimental group exhibits a significantly greater number of cognitive connections than the control group, with a focus on nodes such as solution design (SD), process development (PD), thought expression (TE), and evaluation optimization (EO). Stronger links are evident between key elements such as solution design and operation (SD-O), solution design and thought expression (SD-TE), process development and solution design (SD-PD), and process development and functions (PD-F). The cognitive elements in the experimental group encompass broader areas, integrating the three dimensions of computational concepts, practices, and perspectives. In contrast, the cognitive nodes of the control group are primarily concentrated around computational concepts and practices, such as conditions (C), loops (L), problem analysis (PA), and process development (PD), with minimal attention to computational perspectives. These findings suggest that ChatGPT-based activities broaden students’ programming cognitive dimensions, fostering more comprehensive engagement with computational concepts, practices, and perspectives.

Results for RQ3 (Emotional Engagement)

Table 6 presents the results of the student emotional engagement survey analysis. The Shapiro-Wilk test indicated that the data were approximately normally distributed (p > .05), and the Levene’s test confirmed that the assumption of homogeneity of variance was met (p > .05). According to the independent samples t test, the experimental group had a mean score of 33.17 with a standard deviation of 5.78, while the control group had a mean score of 28.28 with a standard deviation of 7.35, indicating a statistically significant difference between the two groups (p < .05). Further analysis of the scores across the dimensions of emotional engagement revealed significant differences in learning interest, self-efficacy, and anxiety, whereas no significant differences were observed in the motivation dimension.

OPEN IN VIEWER

Table 6.

Results of Differential Analysis.

Measurement dimension	Group	Mean	Standard deviation	F	Sig	Cohen’s d
Interest	Control	9.80	2.46	1.23	.000	2.25
	Experimental	14.51	1.61
Self-efficacy	Control	5.57	1.41	.125	.000	1.92
	Experimental	8.24	1.37
Motivation	Control	8.09	2.51	.047	.767	0.06
	Experimental	8.24	2.39
Anxiety	Control	6.00	2.01	.789	.000	−1.20
	Experimental	3.29	2.50

The results indicate that students in the experimental group exhibited significantly higher emotional engagement compared with those in the pair programming group. Specifically, ChatGPT-based activities increased students’ interest and self-efficacy while reducing their anxiety. However, no significant differences were observed in the motivation dimension, suggesting that these activities did not have a notable effect on students’ motivation to learn programming.

This study explored the effect of ChatGPT-based activities on student engagement by comparing behavioral, cognitive, and emotional engagement between students participating in these activities and those engaged in pair programming. The key findings are as follows.

First, students in the experimental group demonstrated a significant reduction in irrelevant behaviors during programming learning activities. They interacted most frequently with ChatGPT during the decomposition, algorithm design, and evaluation phases. This result aligns with those of Ghimire and Edwards (2024), who argue that ChatGPT positively influences student engagement in programming activities, particularly by assisting with debugging and conceptual understanding conceptual issues. The embodiment of students’ self-regulated learning process using ChatGPT during the activity is a primary factor contributing to this outcome. ChatGPT not only provides immediate feedback but also assists students in monitoring and adjusting their learning process through its dynamic scaffolding function (Chen & Hou, 2024). On the one hand, ChatGPT provides instant responses whenever students encounter challenges or seek feedback on solutions, enabling them to quickly identify and correct mistakes. This immediate feedback mechanism sustains students’ engagement in learning behaviors (Zhong & Zhan, 2024). By contrast, students in the control group had limited programming experience and were unlikely to assume a ChatGPT-like role within their pairs. Even the higher-performing students lacked the expertise to provide the level of instant, accurate feedback that ChatGPT offers. For instance, during the card-splitting task, control group students often spent significant time negotiating solutions or debugging errors, whereas experimental group students could rely on ChatGPT for immediate guidance. This difference highlights the unique advantages of ChatGPT-based activities, particularly for novice learners. On the other hand, lower behavioral engagement in pair programming can be attributed to the complexity and abstraction of programming content, which often exceeds students’ current knowledge and skill levels. This mismatch leads to an increase in off-target operational behaviors during learning activities (Groothuijsen et al., 2024). However, for students in the experimental group, ChatGPT serves as a learning scaffold. Specifically, through dialogic interactions, ChatGPT assesses students’ current knowledge and skills, identifies gaps relative to their learning objectives, and constructs scaffolding within their “zone of proximal development” (Yang et al., 2024). This approach alleviates the imbalance between task difficulty and skill development, optimizing the learning gradient. As a result, it minimizes learning-unrelated behaviors triggered by cognitive overload or task difficulty.

Second, the study found that students engaged in ChatGPT-based activities not only thoroughly explored computational concepts and practices but also expanded their cognition into the dimension of computational perspectives. It signifies a transformation in students from being mere technical operators to becoming computational thinking practitioners equipped with critical thinking and creative expression. Analysis of screen recordings from the control group revealed that students often relied on syntax guides or code templates without adapting them to the problem context. This aligns with findings from existing research, students in pair programming activities often focus on mechanically memorizing the syntax and semantics of programming languages, aiming to produce patterned code directly (Zhao et al., 2022). However, due to fragmented and disorganized knowledge, combined with limited programming skills, these students face significant cognitive challenges when attempting to encode knowledge and solve complex programming tasks. As a result, their cognitive engagement remains restricted to computational concepts and practices. As noted by Sun, Liu et al. (2024), beginners often experience high cognitive load when learning programming independently, stemming from insufficient knowledge of syntax and the absence of a systematic knowledge framework. By contrast, in ChatGPT-based activities, ChatGPT plays a facilitative role by generating programming examples and constructing programming frameworks. It provides partial solutions tailored to the target scenario, requiring students to focus on completing the remaining problem-solving tasks. Engaging deeply with ChatGPT allows students to identify and address problems more efficiently, thereby reducing cognitive load. When cognitive overload is alleviated, students can allocate more cognitive resources to deep learning and complex problem-solving. Additionally, the study encountered instances where ChatGPT produced code snippets that were syntactically correct but logically flawed. For example, during the card-splitting task, ChatGPT occasionally failed to handle edge cases such as negative card counts. This consistent with the findings of Haindl and Weinberger (2024), ChatGPT often generates programming resources that are limited, incomplete, or inaccurate. To address these issues, students are advised to adopt measures such as iterative questioning and query optimization, as well as critical evaluation and error correction. These approach compelling students to exert additional effort to refine solutions and verify the accuracy and applicability of responses. This iterative engagement not only promotes deeper cognitive processes but also cultivates students’ metacognitive abilities, prompting them to reflect on and adjust their own cognitive processes (Zhou et al., 2023).

Third, the study found that ChatGPT-based activities can greatly enhance students’ interest and self-efficacy while alleviating anxiety during the learning process. However, these activities do not have a significant positive effect on students’ motivation to learn programming. This finding contrasts with Yilmaz & Karaoglan Yilmaz, (2023), who reported that ChatGPT-assisted programming instruction enhances self-efficacy and learning motivation. The discrepancies may be attributed to the heterogeneity of student populations and the multidimensional nature of motivation. On the one hand, substantial heterogeneity exists among student groups regarding their acceptance of ChatGPT, influenced by their attitudes toward new technology and their ability to effectively use it (Chang et al., 2024). High school students, compared with college students, may hold more conservative attitudes toward new technology or lack proficiency in its application, limiting their willingness and motivation to use ChatGPT for programming learning. On the other hand, learning motivation is multidimensional, encompassing intrinsic factors (such as interest and curiosity) and extrinsic factors (such as rewards and grades). Even if students’ interest and self-efficacy are enhanced, their overall learning motivation may not increase if extrinsic motivational factors remain unchanged or if additional barriers exist (Lo et al., 2024).

While ChatGPT-based activities have demonstrated significant advantages in enhancing student engagement, it is crucial to acknowledge the unique benefits of traditional pair programming, particularly in fostering social interaction, communication, and collaborative problem-solving skills, which are essential for students’ holistic development (Hopcan et al., 2022). However, the reliance on ChatGPT for immediate feedback and guidance may reduce the necessity for peer collaboration, potentially limiting students’ opportunities to develop interpersonal and collaborative competencies. This raises concerns about whether such activities could inadvertently diminish students’ ability to cultivate critical social and teamwork skills. On the other hand, it is important to recognize that while students in the control group, as novices, may lack the expertise to provide the same level of scaffolding support as ChatGPT, pair programming has been widely proven to enhance learning through peer discussion, knowledge co-construction, emotional support, and sustained motivation. These collaborative interactions not only reduce feelings of isolation and anxiety but also indirectly promote learning persistence (Liu & Li, 2024). In comparison, ChatGPT may not fully replicate the emotional connection and creative ideation that naturally arise during human interaction. However, its core strength lies in its ability to simulate an expert role, providing professional guidance through precise problem diagnosis and structured knowledge delivery (Park & Kim, 2025). This characteristic also explains why ChatGPT-based activities outperform traditional pair programming in enhancing student engagement. Therefore, educators should consider combining ChatGPT-based activities with pair programming activities to ensure that students benefit from the strengths of both approaches.

Implications

For the theoretical contributions, ChatGPT’s human–computer dialogic interaction enables a more student-centric approach to personalized learning (Bettayeb et al., 2024). This study integrates human–computer dialogic interaction based on ChatGPT into programming activities to examine its effects on learning behavior, cognition, and emotional engagement, thereby deepening the understanding of the efficacy of ChatGPT and similar conversational programming tools in programming education. The findings demonstrate that ChatGPT-based activities greatly enhance student engagement in programming learning. ChatGPT-based activities differ fundamentally from traditional pair programming by leveraging unique advantages such as immediate feedback, timely prompts, and dynamic scaffolding (Huesca et al., 2024). These features address key limitations of pair programming, particularly in novice–novice pairs where students may lack the expertise to effectively guide each other. Additionally, the study indicates that ChatGPT’s emotional feedback and programming resources effectively mitigate negative emotions such as anxiety and boost students’ interest and self-confidence in programming. These findings contribute to a deeper understanding of students’ needs during programming activities, particularly by providing insights into the types of support and feedback students expect from dialogic interactions.

For the practical contributions, ChatGPT-based activities may help educators better understand the utility of ChatGPT in programming education. This understanding could also motivate educators to design more effective dialogic interaction programming activities that ensure students engage in programming learning through high-quality interactions. To promote student engagement in programming learning, educators can adopt the following strategies: (1) Utilize ChatGPT to dynamically construct programming learning scaffolds that adaptively adjust task difficulty based on students’ knowledge and skill levels. These scaffolds should be removed progressively through ongoing assessments of students’ growing knowledge and skills to prevent habitual dependency. Implementing an explicit fade-out design strategy is an effective approach (Liao et al., 2024). Teachers can begin by assigning relatively basic tasks and allowing students to receive help and guidance through ChatGPT-based dialogic interaction. Subsequently, students can be tasked with independently completing a similar problem, encouraging them to generalize and transfer the knowledge and skills learned from previous tasks, thereby demonstrating their ability to independently solve similar problems. (2) Allow students to integrate isolated programming elements into knowledge schemas using programming application examples and frameworks provided by ChatGPT, thereby reducing cognitive load. However, students must clearly articulate their problems and needs beforehand. If students fail to specify the expected behaviors or functions of their code, ChatGPT should prompt them to describe the desired outcomes or behaviors explicitly before providing substantive assistance (Denny et al., 2024). (3) Provide clear usage guidelines and share successful cases of using ChatGPT to enhance students’ understanding of the tool’s value. Guide students on how to pose effective questions to ChatGPT and assess the quality of its answers (Al-Abdullatif & Alsubaie, 2024). Encourage students to build a positive interactive relationship with ChatGPT by engaging actively in its use and evaluation. Additionally, regularly collect feedback and suggestions from students about their experiences with ChatGPT to continuously refine and adjust teaching strategies.

Limitations and Future Directions

Several limitations should be considered when interpreting the findings and implications of this study. First, although this study controlled for baseline characteristics (e.g., gender, age, prior programming experience) and covariates (e.g., academic performance, programming ability) that influence forming pairs and using ChatGPT, other studies have shown that students’ learning styles and motivation may also have potential effects on their performance (Strzelecki et al., 2024). Future research could implement dynamic role rotation mechanisms based on real-time performance or preferences, and integrate personalized ChatGPT use pathways that are consistent with students’ cognitive levels and creative tendencies. Second, although selecting a single school and a specific programming language was necessary to control for potential confounding variables, thereby allowing for a more precise isolation of the effects of ChatGPT-based activities on student engagement, the findings of this study align with prior research on the benefits of dialogic interaction and AI-assisted learning in programming education (Fan et al., 2025; Yun et al., 2025). This suggests that the observed effects may be applicable to other contexts. Therefore, future research could replicate this study in diverse educational settings, including different student populations, and multiple programming languages. Additionally, longitudinal studies could explore how the effects of ChatGPT-based activities evolve over time as students gain more programming experience and knowledge. Third, despite the multiple effective measures we adopted in the experimental design (such as critical thinking training, error warning mechanisms, structured question templates, and interaction constraints, and problem chain-driven task design) to address potential biases in ChatGPT’s responses, ethical concerns related to AI reliance, and to prevent students from mechanically applying ChatGPT-generated solutions, existing studies have shown that ChatGPT with a guiding mechanism may be more effective (Lee, Chen et al., 2024). Future research could introduce it to provide students with step-by-step guidance through phased prompts and dynamically adjust the granularity of these prompts based on students’ real-time performance. Furthermore, to further explore the effectiveness of AI-assisted coding tools, future research could conduct cross-tool comparisons. For example, experiments could be designed to compare ChatGPT, GitHub Copilot, and Code.org in terms of knowledge retention, engagement, and cognitive development. This would help leverage their complementary strengths to develop hybrid models, such as using ChatGPT for conceptual guidance and Copilot for code generation.