Assessing College Student Engagement: Development and Validation of the Student Course Engagement Scale

Abstract

Student engagement appears to be associated with high-quality learning outcomes. Thus, it is important to measure whether students are engaging effectively in class. This article describes the development and validation of a measure of student engagement named the Student Course Engagement Scale (SCES). Exploratory and confirmatory analyses were conducted on two independent samples (n = 543 and n = 893, respectively) of Taiwanese college students. The resultant 20-item SCES demonstrated good reliability and factorial validity as well as being correlated positively with other engagement measures. The results also confirm the measurement invariance of the five-factors SCES across different gender groups. This instrument provides a practical and valuable tool for assessing course engagement among Chinese college students.

Keywords

student engagement validation Chinese college students measurement invariance

Introduction

There is a strong need for a reliable measure to assess whether students engage effectively in their courses. Student engagement has been found to positively correlate with a wide range of student outcomes, including academic performance, school adjustment, college achievement, and persistence rates (Dotterer & Lowe, 2011; Handelsman, Briggs, Sullivan, & Towler, 2005; Kahu, 2013; Motti-Stefanidi, Masten, & Asendorpf, 2015; Reyes, Brackett, Rivers, White, & Salovey, 2012; Van Ryzin, 2011; Wonglorsaichon, Wongwanich, & Wiratchai, 2014). In particular, academic course engagement appears to lead to great course achievement, and such achievement also leads to continued or increased engagement in future courses (Svanum & Bigatti, 2009). The instructors’ role is crucial as the level of student engagement depends partly on the supportive quality of the classroom environment (Reeve, 2006). Knowing the students’ degree of engagement should thus provide useful information to instructors so that they can work with individual students or modify the classroom atmosphere to enhance engagement (Handelsman et al., 2005).

Recently, there has been a dramatic increase in the number of colleges and universities in China and Taiwan (Hu, Ching, & Chao, 2012; Yin & Wang, 2016). At the same time, there has been a noticeable decline in overall education quality among these colleges and universities, and many researchers have started to express concerns about the quality of teaching and learning of these institutions (Hu et al., 2012; Liu & Rosa, 2008). As the importance of student engagement is becoming recognized, there is growing interest in understanding and assessing student engagement in Chinese populations.

However, the majority of research has focused on identifying those factors that are related to student engagement (e.g., Chang, 2015, 2012; Lee & Sun, 2010; Sun, 2014), whereas research concerned with deriving measures for assessing the extent of student engagement has been limited. The existing measures all have shortcomings for application to Chinese-speaking populations. Some were designed for Western populations such as the National Survey of Student Engagement (NSSE; 2000), some were translated from other versions such as the Student Course Engagement Questionnaire (SCEQ; Chou, 2008; Handelsman et al., 2005), and others are constrained by weaknesses (discussed later) such as the Learning Engagement Scale for College Students (LESCS; Lin & Huang, 2012). Given the importance of student engagement and the current lack of an appropriate tool, it is imperative to develop a specific, reliable measure to assess Chinese college students’ engagement in their courses.

As student engagement is typically considered to comprise a variety of indicators rather than just course-related activities, this makes it difficult or impossible to develop comprehensive instruments for measuring engagement in college courses (Svanum & Bigatti, 2009). Therefore, the present study focused specifically on student engagement in learning of college academic courses. Our aim was to develop an instrument, named the Student Course Engagement Scale (SCES), for assessing the extent of Chinese college students’ engagement in their course learning and to test the psychometric properties of this instrument.

Researchers have proposed a number of different approaches to characterize student engagement. Chapman (2003) described student engagement as students’ willingness to participate in routine school activities. Krause and Coates (2008) considered student engagement as the quality of effort students devote to educationally purposeful activities. Axelson and Flick (2011) identified student engagement as how involved or interested students appear to be in their academic tasks and how connected they are to their learning environment (including peers, professors, and the institution). Bulger, Mayer, Almeroth, and Blau (2008) broadly defined student engagement as an integrated involvement including interest, effort, motivation, and the time spent by the student on particular learning tasks. From the above, it appears student engagement encompasses both physical and psychological dedication. In the current study, we defined student engagement as the amount of physical and psychological energy a student dedicates to the course (Kazmi, 2010).

Recent research has reported that student engagement is a multidimensional construct (e.g., Christenson, Reschly, & Wylie, 2012; Fredericks, Blumenfeld, & Paris, 2004; Handelsman et al., 2005). In a review of the literature, Chapman (2003) considered student engagement as students’ cognitive investment in, active participation in, and emotional engagement with specific learning tasks. Similarly, Fredericks et al. (2004) suggested student engagement is a multifaceted construct that includes behavioral, emotional, and cognitive components: Behavioral engagement refers to the extent to which students actively respond to the learning tasks presented (e.g., participation in school); emotional engagement refers to the extent of affective attitudes or emotional reactions to teachers, classmates, and academics presented (e.g., reactions to school environment); and cognitive engagement refers to the extent to which students attend to and expend mental effort in the learning tasks encountered (e.g., self-regulation, learning goals, investment in learning). We concur with the viewpoint of Fredericks et al. that student engagement is a metaconstruct that includes behavioral, emotional, and cognitive components that capture how students behave, feel, and think when interacting with their learning environment.

Currently, there are few available measures for assessing student engagement. In Western countries, the best-known survey instrument is the NSSE (2000). The NSSE addresses various aspects of the college experience, including how students spend their time at university and the intellectual, personal, and social gains students realize from their studies. It is used in colleges and universities across the United States and Canada to gather data self-reported by students about their college experience (Lutz & Culver, 2010). However, as Handelsman et al. (2005) noted, the NSSE assesses students’ overall perceptions—it focuses on active learning and other school experiences but does not focus on individual courses.

A similar national survey is the Australian Survey of Student Engagement (AUSSE), which has been used in Australian and New Zealand institutions (Coates, 2007). Students are asked to report the time they devote to various learning activities, and students’ educational outcomes (including development of cognitive and socioemotional capability, academic achievement, departure intention, and overall satisfaction) are then measured. This survey instrument is based on the NSSE, but its questions were reconceived to ensure relevance to Australian and New Zealand higher education contexts (Luzeckyj, Schmidt, & Reynolds, 2012).

In addition to these two national surveys, Handelsman et al. (2005) developed the SCEQ, a more specific measure of college student course engagement. Research utilizing this measure found it to be internally consistent and supported a multidimensional construct of student engagement (Handelsman et al., 2005). Results with the SCEQ revealed four dimensions of college student engagement that were distinct and reliable: skills engagement, participation/interaction engagement, emotional engagement, and performance engagement (Handelsman et al., 2005). Appleton, Christenson, Kim, and Reschly (2006) developed the Student Engagement Instrument (SEI), a survey designed to measure self-perceptions of cognitive and affective engagement. Subsequent research found evidence of the internal consistency, factor structure, convergent and divergent validity, and measurement invariance for the SEI for various student populations (Betts, Appleton, Reschly, Christenson, & Huebner, 2010; Lovelace, Reschly, Appleton, & Lutz, 2014; Reschly, Betts, & Appleton, 2014).

Although the above-mentioned measures provide important information regarding student engagement, they were all designed for Western populations. Instruments for measuring student engagement for Chinese populations are few and imperfect (Chou, 2008; Lin & Huang, 2012). Chou (2008) examined the appropriateness of the SCEQ in Taiwan. The results of Chou differed from those of Handelsman et al. (2005) in terms of factors as in addition to the original four factors, a new factor (named attitude engagement) was found. The reliability and validity of SCEQ scores were demonstrated, and Chou concluded that the instrument could be applied in Taiwan. However, compared with a Chinese-language instrument designed for students in Taiwan and China, the SCEQ is likely to have biases due to translation and cultural differences.

Another scale, the Scale of Technology University Students’ English Learning Engagement, was recently developed by Yang, Wu, and Chou (2014) based on the research of Fredericks et al. (2004). This scale was developed for application to Chinese college students and consists of three components: behavioral, emotional, and cognitive. Although this scale was found to have good reliability and validity, its application is quite limited as it was specifically constructed for one course—English.

Another Chinese-language instrument worth noting is the LESCS (Lin & Huang, 2012). After reviewing the relevant literature, Lin and Huang (2012) developed the LESCS for assessing the degree of Taiwanese college students’ engagement in learning. The scale includes five factors—skills, emotion, performance, interaction, and attitude engagement—and was reported to not only have good model fit but also have good reliability and validity. The results for this five-factor model of student engagement were consistent with the findings of Fredericks et al. (2004) and Chou (2008) and extended the theory of Handelsman et al. (2005). Yet, although the LESCS is local, valid, and not restricted to a specific course, it was still inadequate for our study as it was not designed for assessing course-level engagement. Instead, the LESCS assesses student engagement from overall experience in learning tasks (example items: I always participate in school activities such as workshop or speech, I am proud of being a member of my school, I always talk to my teachers about my career planning, I am interested in school extracurricular activities, etc.). Moreover, as the measurement invariance of the LESCS was not reported in Lin and Huang (2012) or elsewhere, one cannot be sure that the construct of engagement is being assessed similarly in each group. This constrains the application of groups’ comparisons as measurement invariance is a prerequisite for meaningful comparisons across groups (e.g., ethnic, socioeconomic, or gender groups).

Studies by Chou (2008) and Lin and Huang (2012) have identified the presence of five distinct forms of engagement in classroom settings: skills engagement, emotional engagement, performance engagement, interaction engagement, and attitude engagement. The first factor, skills engagement, represents student engagement through practicing skills (e.g., taking notes in class). This construct is similar to the behavioral engagement of Fredericks et al. (2004) that refers to the extent students make active response to the learning tasks. The second factor, emotional engagement, represents student engagement through emotional involvement with the school activities (e.g., desiring to learn the material). This construct is similar to the emotional engagement of Fredericks et al. that refers to the extent of affective attitudes to learning environment. As for the other three factors, performance engagement, interaction engagement, and attitude engagement, these represent other aspects of student engagement: performance on the learning tasks (e.g., doing well on tests), participation in school and interactions with instructors and other students (e.g., participating in group discussions), and contribution to the learning activity (e.g., consistent attendance). These constructs are similar to the cognitive engagement of Fredericks et al. that refers to the extent to which students are attending to and expending mental effort in the learning tasks. To sum up, the five-factor model of student engagement that emerged from Chou and Lin and Huang is consistent with the engagement theory of Fredericks et al., suggesting that this model can be adopted to assess Chinese college students’ engagement in their learning tasks.

In addition, student engagement research typically explores how engagement varies between gender groups (e.g., Johnson, Crosnoe, & Elder, 2001; Martin, 2007). However, few studies have examined the measurement invariance of the instrument when making group comparisons (Martin, 2007). Nonetheless, if the instrument assessing student engagement operates differently across gender groups, it is inappropriate to compare levels of engagement by gender (Glanville & Wildhagen, 2007). Therefore, when validating a measure, it is important to confirm that the factor model of student engagement demonstrates measurement invariance across gender groups.

As noted above, there is need for an appropriate instrument of student course-level engagement for Chinese student population. To address this need, we designed a two-phase study to develop and validate such an instrument. In the first phase, based on the findings of Chou (2008) and Lin and Huang (2012) with Taiwanese students, we constructed a new five-facet instrument, named the Student Course Engagement Scale (SCES), for assessing the degree of Chinese college students’ course-level engagement and explored the factor structure. In the second phase, we examined the psychometric properties of the instrument including reliability, validity, and measurement invariance. In sum, the purposes of the study were (a) to construct a course engagement scale for Chinese college students; (b) to assess factor structure, validity, and reliability of this scale via exploratory factor analysis (EFA) and confirmatory factor analysis (CFA); and (c) to examine the measurement invariance of the scale across gender groups.

Method

This study involved two phases. The first phase was to generate an initial pool of items to assess the extent of college students’ engagement in their academic learning. An EFA was conducted to examine the factor structure of the scale items, and Cronbach’s alpha coefficient was calculated to examine the internal consistency of the constructed scale. The second phase was to test the factor structure of the scale obtained from EFA through the other independent sample. In this stage, apart from the Cronbach’s alpha coefficients, the criterion-related validity was also examined, and furthermore, the invariance of all parameters (including factor loadings and item intercepts) in the model between male and female groups was tested. Finally, the item results were checked against the theoretical framework for student engagement.

Scale Construction

A research team of assistant students and two academic colleagues participated in the item construction process. To generate a pool of items, we first reviewed previous research relative to the constructs under investigation (e.g., Chou, 2008; Handelsman et al., 2005; Lin & Huang, 2012). This initial stage of instrument development hypothesized that five components of individual engagement could provide a structure for the development of items that would measure student involvement in academic learning. As to the number of items, on one hand, we intended to have a brief instrument to measure student course engagement; on the other hand, the references scales including the modified SCEQ and LESCS scales both have five factors and nearly 30 items. Therefore, a pool of 30 items was constructed to capture facets of learning engagement arising from five factors (six items each). “Skills engagement” refers to the learning strategy or practicing techniques student adopts to perform well in the class (e.g., I can apply the learned knowledge or skills to do homework). “Emotional engagement” refers to the affective states or responses of students when interacting with class material or learning environment (e.g., I always desire to learn in class). “Performance engagement” refers to student’s achievement in the class (e.g., I always do well on the tests). “Interaction engagement” refers to student’s participation in class and having mutual influences with instructors and others (e.g., I am seldom blamed by teacher). “Attitude engagement” refers to the student’s position on the undertaken course (e.g., I seldom skip class).

For enhancing content validity, two counseling psychologists with expertise in teaching as well as scale construction and a group of undergraduate psychology students were invited to evaluate the draft. Based on the feedback of experts and students, two items were deleted due to redundancy and several statements were reworded for clarity. Furthermore, the resulting scale with 28 items was piloted on a sample of 90 college students, and the item-total correlation was calculated for each item. Following Kline’s (1993) suggestion, three items with an item-total correlation less than .3 were to be excluded. This procedure resulted in the preliminary SCES of 25 items covering the five hypothesized components.

Participants and Procedure

Participants in this study included college students from six colleges or universities in Central Taiwan. After the scale was administered, the entire data set was scrutinized to detect and delete those with missing or invalid responses (they were less than 4%). There were two independent samples for the two phases of the study. The Phase 1 sample (n = 543), for the EFA, consisted of 285 (53%) females and 258 (47%) males with ages ranging from 20 to 25 (M = 21.69, SD = 2.35). Class levels included 153 freshmen (28%), 249 sophomores (46%), 88 juniors (16%), and 53 seniors (10%). There were 27 students with parents having up to 9 years of education (5%), 138 with parents completing 12 years of education (25%), and 378 with parents with tertiary education (70%). The Phase 2 sample (n = 893), for the CFA and multigroup invariance test, consisted of 444 (50%) females and 449 (50%) males with ages ranging from 20 to 26 (M = 21.23, SD = 2.57). Class levels included 287 freshmen (32%), 363 sophomores (41%), 138 juniors (15%), and 105 seniors (12%). There were 36 students with parents having up to 9 years of education (4%), 250 with parents completing 12 years of education (28%), and 607 with parents with tertiary education (68%).

Participants were recruited from classes in the introduction to psychology, accounting, computer science, physical education, and linguistics at six colleges and universities in Central Taiwan. Participants were informed about the purpose of the study and completed consent forms. Participation was voluntary (participants were not offered any course credit), and questionnaire responses were kept confidential. It took about 20 min for participants to complete the questionnaires. Students who chose not to participate in this research were permitted to leave the classroom. The participation rate for the two samples was 75.2% (543 of 722) and 80.7% (893 of 1,107), respectively.

Measures

Participants were asked to provide demographic information including age, gender, grade level, parental education, and major in college. Participants in the first sample only completed the preliminary SCES, but participants in the second sample completed both the preliminary SCES and LESCS (Lin & Huang, 2012).

The preliminary SCES

The preliminary SCES, a self-report questionnaire consisting of 25 items, was used to assess the degree of college students’ engagement. In addition to the SCES, students were asked to rate the course from which they were recruited for this study. Items were rated on a 5-point Likert-type scale from 1 (never true) to 5 (always true). Higher scores indicate a student has high degree engagement in the course. The preliminary SCES was constructed and administered in Chinese.

LESCS

The 20-item LESCS (Lin & Huang, 2012) was used to examine criterion-related validity. The scale tapped students’ engagement from five facets: Skills (four items), Emotion (five items), Performance (four items), Attitude (four items), and Interaction (three items). Students rated each item based on the degree to which they agreed with each statement using a 5-point Likert-type scale, ranging from 1 (never true) to 5 (always true). The higher the score, the better the degree of student engagement. Lin and Huang reported that the LESCS scores have good reliability, validity, and stability. In the current study, the Cronbach’s alpha coefficient for each subscale was .84, .80, .83, .82, and .80, respectively.

Statistical Analyses

An EFA was conducted on the first sample using maximum likelihood extraction with promax rotation in SPSS 20. The Kaiser–Meyer–Olkin (KMO) measure of sampling adequacy and Bartlett’s test of sphericity were used to determine whether the data sets were factorable. Item and factor retention decisions were guided by the following criteria: Items should preferably weigh greater than .4 on the relevant factor and less than .4 on all other factors, parallel analysis, eigenvalue > 1, scree plot was examined, conceptual clarity, and interpretability. Cronbach’s alpha coefficient was calculated to determine the internal consistency of the scale scores.

The derived factor structure from the EFA was examined using CFA in LISREL 8.70. Taking into account the ordered categorical data, robust maximum likelihood was used as a method of estimation using a polychoric correlations matrix (Joreskog, 2004). While assessing model fit, as the Satorra–Bentler chi-square (SB χ²) is sensitive to the sample size, the application other indices of fit has been suggested (Joreskog & Sorbom, 1996). These indices included the goodness-of-fit index (GFI ≥ .90), the nonnormed fit index (NNFI ≥ .90), the root mean square error of approximation (RMSEA ≤ .08), the standardized root mean square residuals (SRMR ≤ .08), and the comparative fit index (CFI ≥ .90). Evaluating the criterion-related validity of the SCES was based on the Pearson correlations between the subscale scores of SCES and LESCS.

Measurement invariance across groups was investigated using multiple group CFAs in LISREL 8.70. Given the ordered categorical data, the method used was robust maximum likelihood estimation, based on the polychoric correlation matrix (Joreskog, 2004). Following Sass (2011) and Treloar, Martens, and McCarthy (2014), a two-step approach to testing measurement invariance was used. First, a baseline model was established where all item intercepts and factor loadings were free to vary across gender groups with reasonable fit to the data. Second, to test whether all item intercepts and factor loadings were the same across gender groups, this constrained model was established by constraining all factor loadings and intercepts as being equal across both groups. Differences in SB χ² and CFI were analyzed to compare the models. A nonsignificant difference in SB χ² and a ΔCFI of less than or equal to .01 between models indicates support for measurement invariance (Cheung & Rensvold, 2002; Hu & Bentler, 1998).

Results

EFA

An EFA was conducted on the 25-item preliminary SCES for the first sample (n = 543). Before performing this analysis, the adequacy of the data set was assessed to ensure that factor analysis was appropriate. The measure of sample adequacy (Bartlett’s statistic) was 7,122.070 (p < .001) with KMO = .912, indicating that factor analysis was appropriate. The number of factors to retain was based on a combination of methods stated earlier. Table 1 shows the factor loadings for the results from EFA. The preliminary analysis showed that five candidate items did not have significant loadings and were discarded. We then carried out a second EFA with the remaining 20 items (KMO = .907, Bartlett’s statistic = 6,101.322). Based on the parallel analysis and the scree plot, five factors were extracted with all factor loadings greater than .4.

Table 1.

Items and Factor Loadings From the Initial EFA.

Item	Factor
Item	1	2	3	4	5
18. I can look over class notes between classes to make sure I understand the material.	.901	−.112	.018	−.031	.049
19. I can take good notes of the curriculum.	.879	.041	−.047	−.069	−.041
17. I always do my best to catch the importance of the curriculum.	.780	.036	−.006	−.048	.046
20. I can apply the learned knowledge or techniques to do homework.	.674	.019	.058	.075	−.004
16. I always put forth effort into the curriculum.	.649	.025	.073	.024	.065
11. It is important to obtain knowledge and develop techniques.	−.067	.918	−.109	.010	.006
12. Everyone should do his or her best to receive more education.	.030	.873	−.160	.052	−.018
14. Everything learned in class is of value to me.	.029	.557	.227	−.139	.025
13. It is important to learn in school.	.165	.512	.059	.002	−.113
10. I always desire to learn in class.	−.145	.506	.277	.030	.072
6. I am seldom blamed by teacher.	.308	.398	−.090	.190	−.093
15. Teacher always talks to me.	.059	.372	.153	−.163	.266
9. I feel I am important in my class.	−.024	.267	.205	.049	.150
25. I am good at the curriculum.	.027	−.039	.805	.066	−.035
23. I always do well on the tests.	−.106	.028	.757	.076	−.060
22. I am confident that I can learn and do well in the curriculum.	.060	−.086	.723	−.021	.062
24. I do my best to get a good grade in the curriculum.	.182	−.030	.653	.011	−.048
21. I choose to give up or do easy part while encountering great difficulties in the curriculum.	−.019	.057	.188	−.176	.152
3. I seldom cut the class.	.035	−.051	−.019	.886	−.019
1. I go to class every day unless I am sick.	−.006	.068	.032	.791	−.008
2. I am seldom late for school.	−.099	−.023	.057	.739	.104
8. I always complete the assignment on time.	.329	.058	−.014	.395	−.029
5. I always participate actively in small-group discussions.	.068	−.048	−.107	−.025	.974
4. I am glad to answer questions teacher proposed in class.	−.031	−.018	−.013	.181	.791
7. I always take the initiative in asking questions when I don’t understand the instructor.	.003	.027	.154	.004	.680

Note. The bold-faced items indicate they did not have significant loadings. EFA = exploratory factor analysis.

Table 2 presents factor loadings and other important information for the retained items (items have been renumbered in sequence). The EFA resulted in five factors. The first factor consisted of five items and was labeled “skills engagement.” Items in the factor reflect the learning strategy student adopted for succeeding in the academic learning (e.g., I can take good notes of the curriculum). The second factor consisted of five items and was labeled “emotional engagement.” Items in this factor reflect the emotional involvement with the class material (e.g., It is important to obtain knowledge and develop skills). The third factor consisted of four items and was labeled “performance engagement.” Items in the factor reflect students’ performance on the learning activities (e.g., I always do well on the tests). The fourth factor consisted of three items and was labeled “interaction engagement.” Items in the factor reflect the interactions of student with teachers and classmates (e.g., I always participate actively in small-group discussions). The last factor consisted of three items and was labeled “attitude engagement.” Items in this factor reflect the student’s attitude to the undertaken curriculum (e.g., I seldom cut the class). The resulting five-factor model accounted for 70% of the variation.

Table 2.

Factor Loadings, Reliability, Communalities (h²), Factor Correlation, and Descriptive Data for the SCES.

Item	Factor					h ²	M	SD
Item	Skills	Emotion	Performance	Interaction	Attitude	h ²	M	SD
X1 (18)	.893	−.092	.000	.035	−.005	.74	3.57	0.83
X2 (19)	.870	.056	−.058	−.050	−.042	.69	3.60	0.80
X3 (17)	.765	.060	−.007	.036	−.031	.63	3.62	0.81
X4 (20)	.668	.018	.047	−.010	.096	.57	3.57	0.78
X5 (16)	.640	.032	.070	.060	.035	.56	3.46	0.78
X6 (11)	−.044	.925	−.096	.008	.008	.75	3.59	0.85
X7 (12)	.052	.852	−.127	−.007	.037	.70	3.70	0.83
X8 (14)	.056	.494	.232	.038	−.118	.42	3.66	0.87
X9 (10)	−.114	.463	.281	.095	.012	.43	3.45	0.92
X10 (13)	.195	.454	.062	−.103	.027	.34	3.59	0.85
X11 (25)	.009	−.024	.828	−.027	.035	.67	3.11	0.90
X12 (23)	−.108	.043	.751	−.041	.043	.50	3.12	0.90
X13 (22)	.050	−.067	.706	.083	−.046	.55	3.06	0.91
X14 (24)	.163	−.030	.671	−.040	−.003	.54	3.24	0.85
X15 (5)	.057	−.022	−.095	.970	−.060	.81	2.95	1.00
X16 (4)	−.033	−.001	−.003	.794	.143	.73	2.94	0.97
X17 (7)	−.003	.033	.169	.692	−.037	.65	2.88	1.02
X18 (3)	.062	−.044	−.037	−.047	.912	.79	3.49	1.10
X19 (1)	.026	.076	.029	−.016	.761	.67	3.58	1.09
X20 (2)	−.072	−.011	.047	.090	.725	.57	3.37	1.07
Eigenvalues	8.09	1.91	1.56	1.34	1.15
% of variance	40.44	9.56	7.77	6.72	5.74
M	3.56	3.60	3.13	2.93	3.48
SD	0.67	0.66	0.73	0.90	0.96
Coefficient α (total score = .92)	.89	.81	.83	.88	.85
Factor correlation matrix
Skills	—
Emotion	.57	—
Performance	.63	.51	—
Interaction	.38	.35	.56	—
Attitude	.53	.51	.52	.43	—

Note. Numbers in parentheses represent the initial number of items. Pattern coefficients with values of .40 or greater are in bold. SCES = Student Course Engagement Scale.

All correlations are significant (p < .01).

Furthermore, as presented in Table 2, the Cronbach’s alpha coefficients for the individual subscales ranged from .81 to .89 and .92 for the total score, providing support for internal consistency. The intercorrelations among the individual SCES subscales ranged from .35 to .63. These results echo the factor analytic findings suggesting that the various subscales represent overlapping, yet somewhat distinct, aspects of student’s learning engagement.

CFA

We next tested whether the factor structure of the 20-item SCES was reproducible in the second sample (n = 893). Multiple fit indices as described earlier were employed to evaluate the fitness of the factor structure as derived through EFA. Results of the CFA suggested that the overall fit of model was good: χ² = 1,303.31, df = 160, p = .00, GFI = .90 (≥.90), NNFI = .96 (≥.90), RMSEA = .078 (≤.08), SRMR = .045 (≤.08), CFI = .96 (≥.90). Also, the factor loadings of items were presented in Table 3; all the t values of items showed statistical significance at the .05 level (21.35 < t < 34.63, omitted).

Table 3.

Descriptive Statistics and Factor Loadings of CFA (n = 893).

Item	M	SD	Factor loading
X1	3.53	0.86	.91
X2	3.55	0.82	.86
X3	3.59	0.85	.81
X4	3.55	0.81	.74
X5	3.43	0.82	.83
X6	3.56	0.83	.80
X7	3.68	0.82	.75
X8	3.63	0.86	.75
X9	3.45	0.93	.74
X10	3.60	0.83	.67
X11	3.13	0.90	.85
X12	3.09	0.90	.76
X13	3.04	0.89	.77
X14	3.17	0.89	.78
X15	2.96	1.02	.89
X16	2.93	0.98	.90
X17	2.88	1.06	.84
X18	3.44	1.10	.85
X19	3.56	1.07	.80
X20	3.32	1.06	.80

Note. CFA = confirmatory factor analysis.

The alpha coefficient of the overall scale was .92, and ranged from .82 to .90 for the five subscales, thus suggesting the SCES scores had good internal consistency. Furthermore, for examining the criterion-related validity of SCES, its relationship with another learning engagement measure, the LESCS, was assessed. Correlations of the SCES subscales to the criteria are shown in Table 4. As expected, the SCES scores were positively correlated with LESCS.

Table 4.

The Correlation Between LESCS and SCES.

	SCES
	Factors	Skills	Emotion	Performance	Interaction	Attitude	Total scale
LESCS	Skills	.65	.56	.50	.58	.57	.70
	Emotion	.53	.73	.61	.49	.50	.74
	Performance	.48	.68	.69	.57	.60	.73
	Interaction	.56	.50	.55	.74	.61	.76
	Attitude	.49	.53	.56	.63	.70	.75
	Total scale	.70	.75	.72	.75	.75	.83

Note. LESCS = Learning Engagement Scale for College Students; SCES = Student Course Engagement Scale.

All correlations are significant (p < .001).

Invariance Testing for Gender

Following the CFA discussed above, a multigroup analysis was performed to test the measurement invariance of the SCES across gender. Table 5 presents a summary of goodness-of-fit statistics for the assessment of gender invariance. As shown in Table 5, fit indices supported the adequacy of the five-factor model in groups with different gender. As the unconstrained model (where all item intercepts and factor loadings are relaxed) represented an acceptable fit to the data, that model was used as a baseline model for further invariance tests. The constrained model was then compared with the baseline model to determine if there were differences in the χ² values. The result showed that the difference between the two models was not significant (Δχ² = 25.13, Δdf = 30, p > .05). The ΔCFI was less than .01, also supporting the result. Thus, the measurement invariance of the five-factors SCES across different gender groups was confirmed.

Table 5.

Fit Statistics for the Multigroup Confirmatory Factor Analysis.

	χ²	df	GFI	NNFI	RMSEA	SRMR	CFI
Male (n = 449)	846.69	160	.90	.95	.075	.051	.96
Female (n = 444)	911.50	160	.88	.93	.076	.056	.95
Configural invariance (baseline model)	1,758.19	320	.88	.95	.073	.057	.95
Equal factor loadings and intercepts	1,783.32	350	.88	.95	.073	.052	.95

Note. GFI = goodness-of-fit index; NNFI = nonnormed fit index; RMSEA = root mean square error of approximation; SRMR = standardized root mean square residuals; CFI = comparative fit index.

Check of Theoretical Framework for Student Engagement

To provide more empirical evidence that the five subscales of the SCES align with the behavioral, cognitive, and emotional factors proposed by Fredericks et al. (2004), a CFA was conducted on the SCES with the second sample. The method for performing the CFA is similar to the preceding analyses. A three-factor model comprised of 20 items was examined. Items from the Intersection, Performance, and Attitude subscales were used to measure the construct of cognitive engagement. Items from the Skill subscale were used to measure the construct of behavioral engagement. Finally, items from the Emotion subscale were used to measure the construct of emotional engagement.

Multiple fit indices described earlier were used to evaluate the fitness of the factor structure. Results of the CFA suggested that the overall fit of the model was good: χ² = 3,423.76, df = 167, p = .00, GFI = .88 (slightly less than .90), NNFI = .90 (≥.90), RMSEA = .08 (≤.08), SRMR = .079(≤.08), CFI = .91 (≥.90). As expected, these scales align with the behavioral, cognitive, and emotional components of student engagement proposed by Fredericks et al. (2004).

Discussion

The purpose of this study was to develop a self-report inventory, the SCES, for assessing the extent Chinese students engage in their college courses and to investigate the reliability, validity, and factor structure of this new instrument. Based on a theory of student engagement and findings from relevant research on Taiwanese students, the study identified items leading to the construction of a 20-item inventory, consisting of five facets: skills, emotion, performance, interaction, and attitude. Results from two studies indicated that SCES and its subscales provide reliable and valid scores. It assessed student engagement arising from five components of engagement. The positive correlations between the SCES scores and the LESCS (Lin & Huang, 2012) support SCES’s criterion-related validity. The internal consistency was also supported by high Cronbach’s alpha coefficients. Furthermore, measurement invariance was established across gender (male vs. female). Most importantly, the five subscales of the SCES were found to align with the behavioral, cognitive, and emotional factors in student engagement proposed by Fredericks et al. (2004). Therefore, this research provides empirical evidence of the usefulness of the SCES in assessing course-level engagement of Chinese college students.

The initial scale items were generated based in part on earlier findings of Chou (2008), Handelsman et al. (2005), and Lin and Huang (2012). Five factors were derived from the EFA, and the CFA was then performed to determine whether the hypothesized model fit the data. The CFA results confirmed the five factors of the SCES. Thus, after analyses via EFA and CFA, a five-factor model of student engagement was supported consistent with a multidimensional construct of student engagement (Christenson et al., 2012; Fredericks et al., 2004; Handelsman et al., 2005). The emergence of the five factors was consistent with previous findings on student engagement with samples of Taiwanese students (Chou, 2008; Lin & Huang, 2012).

Results also suggested that the five factors of the SCES align with the tripartite model of behavioral, emotional, and cognitive engagement described by Fredericks et al. (2004). This is perhaps not surprising as the theoretical framework proposed by Fredericks et al. explicitly guided our instrument development. Specifically, the combination of Intersection, Performance, and Attitude subscales in the SCES corresponds to the construct of cognitive engagement; the Skill subscale in the SCES corresponds to the construct of behavioral engagement; and the Emotion subscale in the SCES corresponds to the construct of emotional engagement. In summary, our findings are consistent with the viewpoint of Fredericks et al. that student engagement is a metaconstruct that includes behavioral, emotional, and cognitive engagement.

It is worthwhile to contrast this new instrument (SCES) with the LESCS. They are both valid and brief instruments specifically designed for Chinese students. However, there are several important differences. First, the LESCS is a survey for overall experience in learning tasks, rather than being aimed specifically at academic courses learning. The overall experience in school may comprise various activities (e.g., time or task management skills, involvement in enriching educational experiences, or feelings of being supported by learning communities). In contrast, the SCES was meticulously designed to provide a brief instrument focused especially on student engagement in college course. Second, the issue of measurement invariance across gender for LESCS was not reported by Lin and Huang (2012), so it is not appropriate to directly use the LESCS for gender group comparisons. However, the evidence of measurement invariance across gender was noted for the SCES, so it can be meaningfully used to compare differences in student engagement between males and females. Therefore, we recommend that researchers and educators adopt the SCES for assessing the extent of Chinese student engagement in their college courses.

The results of the study have some implications for educators and future research. As Handelsman et al. (2005) asserted, getting students to be more engaged in the courses is one of the approaches that improves the learning environment and optimizes outcomes. The findings of the current study indicate that we can measure student engagement based on specific aspects including student’s skills, emotion, performance, interaction, and attitude in course learning. In addition to providing an easily administered and comprehensive snapshot of students’ engagement, the SCES assesses engagement at the microlevel by focusing engagement within a course rather than on overall active learning or other educational experiences. The information collected by the SCES may be especially useful for faculty members to consider when proposing interventions that aim to enhance engagement and improve student outcomes in the classroom.

Students with different majors may or may not differ significantly regarding the engagement in their course learning (Zhao, Carini, & Kuh, 2005). To answer this question, more investigation is needed to compare the SCES across college majors. In addition, there may be other variables (e.g., psychological variables or socioeconomic status) that significantly impact student engagement. Future research may help to identify additional factors or mediators that are associated with higher levels of engagement. Finally, further tests of associations with relevant outcomes (e.g., teacher-rated student classroom participation, rate of assignment completion, final course grade, etc.) are needed to reinforce the predictive power of the SCES.

This study has some limitations. As the SCES was deliberately restricted in its focus, it was not comprehensive in its assessment of student engagement. For example, it does not measure engagement arising school activities outside of the classroom such as extracurricular activities or school clubs. Researchers who are interested in a more global view of school engagement should start with a much broader item pool than was used in this research. Furthermore, in this study, information was gathered based solely on self-reports; it would be advantageous to include alternative approaches. For example, one could obtain information via face-to-face interviews or by surveying instructors (Harris, 2008). Another limitation is the restricted sample of participants. First, there were more freshmen and sophomores in our sample than juniors and seniors, and this may have hidden student engagement differences among grades. Second, all of our participants were from Taiwan, so it was not possible to explore differences in the social or cultural environments. Given the large numbers of Chinese students from China and Hong Kong, the generalization of the current findings to these groups might also be limited. Thus, replication of this study on other populations (e.g., on Chinese students in Mainland China or Hong Kong or on uniform samples across grade levels) would help to further evaluate the factor structure and psychometric quality of the scale and provide further evidence of the instrument’s validity and reliability.

On the basis of the foregoing discussion, we conclude that the SCES is a valid and reliable measure for assessing course-level school engagement of Chinese college students. Compared with other scales, this scale is unique as it focuses on college courses and is designed for Chinese college students. Considering the lack of scales to specifically assess student course engagement in Chinese population, it is hoped that the SCES will become a useful tool for educational research as well as an aid to teachers. The SCES may help to further explore the nature of student engagement, especially for those interested in investigating the factors related to student engagement among Chinese student population.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Appleton

J. J.

Christenson

S. L.

Kim

Reschly

(2006). Measuring cognitive and psychological engagement: Validation of the Student Engagement Instrument. Journal of School Psychology, 44, 427-445.

Axelson

Flick

(2011). Defining student engagement. Change: The Magazine of Higher Learning, 43, 38-43.

Betts

J. E.

Appleton

J. J.

Reschly

A. L.

Christenson

S. L.

Huebner

E. S.

(2010). A study of the factorial invariance of the Student Engagement Instrument (SEI): Results from middle and high school students. School Psychology Quarterly, 25, 84-93.

Bulger

M. E.

Mayer

R. E.

Almeroth

K. C.

Blau

S. D.

(2008). Measuring learner engagement in computer-equipped college classrooms. Journal of Educational Multimedia and Hypermedia, 17, 129-143.

Chang

C. C.

(2015). Analysis of learning involvement of junior high students by grade level and gender. Taiwan Education Review, 4, 143-146.

Chang

D. F.

(2012). Exploring the theory and praxis of college student engagement. Evaluation in Higher Education, 8, 41-62.

Chapman

(2003). Alternative approaches to assessing student engagement rates. Practical Assessment, Research & Evaluation, 8. Retrieved from http://pareonline.net/getvn.asp?v=8&n=13

Cheung

G. W.

Rensvold

R. B.

(2002). Evaluating goodness-of-fit indexes for testing MI. Structural Equation Modeling, 9, 235-255.

Chou

R. T. C.

(2008). To construct college students’ course engagement questionnaire. Journal of Taiwan intelligent Technologies and Applied Statistics, 6, 173-188.

10.

Christenson

S. L.

Reschly

A. L.

Wylie

(2012). Handbook of research on student engagement. New York, NY: Springer Science.

11.

Coates

(2007). A model of online and general campus-based student engagement. Assessment & Evaluation in Higher Education, 32, 121-141.

12.

Dotterer

A. M.

Lowe

(2011). Classroom context, school engagement, and academic achievement in early adolescence. Journal of Youth and Adolescence, 40, 1649-1660.

13.

Fredericks

J. A.

Blumenfeld

P. C.

Paris

A. H.

(2004). School engagement: Potential of the concept, state of the evidence. Review of Educational Research, 74, 59-109.

14.

Glanville

J. L.

Wildhagen

(2007). The measurement of school engagement: Assessing dimensionality and measurement invariance across race and ethnicity. Educational and Psychological Measurement, 67, 1019-1041.

15.

Handelsman

M. M.

Briggs

W. L.

Sullivan

Towler

(2005). A measure of college student course engagement. Journal of Educational Research, 98, 184-191.

16.

Harris

L. R.

(2008). A phenomenographic investigation of teacher conceptions of student engagement in learning. The Australian Educational Researcher, 35, 57-79.

17.

Bentler

(1998). Fit indices in covariance structure modeling: Sensitivity to underparameterized model misspecification. Psychological Methods, 10, 424-453.

18.

Y. L.

Ching

G. S.

Chao

P. C.

(2012). Taiwan student engagement model: Conceptual framework and overview of psychometric properties. International Journal of Research Studies in Education, 1, 69-90.

19.

Johnson

M. K.

Crosnoe

Elder

(2001). Students’ attachment and academic engagement: The role of the race and ethnicity. Sociology of Education, 74, 318-340.

20.

Joreskog

K. G.

(2004). Structural equation modeling with ordinal variables using LISREL (2nd ed.). Chicago, IL: Scientific Software International.

21.

Joreskog

K. G.

Sorbom

(1996). LISREL 8. User’s reference guide. Chicago, IL: Scientific Software International.

22.

Kahu

E. R.

(2013). Framing student engagement in higher education. Studies in Higher Education, 38, 758-773.

23.

Kazmi

(2010). Sleep walking through undergrad: Using student engagement as an institutional alarm clock. College Quarterly, 13, 1-15.

24.

Kline

(1993). The handbook of psychological testing. London, England: Routledge.

25.

Krause

K. L.

Coates

(2008). Students’ engagement in first year university. Assessment & Evaluation in Higher Education, 33, 493-505.

26.

Lee

Y. M.

Sun

S. H.

(2010). The relationship between autonomous motivation of course-taking and learning engagement on college students. Journal of Research in Education Sciences, 55, 155-182.

27.

Lin

S. H.

Huang

Y. C.

(2012). Development of learning engagement scale for college students. Psychological Testing, 59, 373-396.

28.

Liu

Rosa

M. J.

(2008). Quality assessment of undergraduate education in China: A policy analysis. Higher Education Management and Policy, 20, 79-96.

29.

Lovelace

M. D.

Reschly

A. L.

Appleton

J. J.

Lutz

M. E.

(2014). Concurrent and predictive validity of the student engagement instrument. Journal of Psychoeducational Assessment, 32, 509-520.

30.

Lutz

M. E.

Culver

(2010). The National Survey of Student Engagement: A university-level analysis. Tertiary Education and Management, 16, 35-44.

31.

Luzeckyj

Schmidt

Reynolds

L. C.

(2012). Using Australian Survey of Student Engagement data to stimulate discussion in a First Year Community of Practice. In Mortimer

, ed. 15th International First Year in Higher Education Conference. Brisbane, QLD: QUT Events. 2012 FYHE Conference. Brisbane, QLD. June 2012, pp. 1-5.

32.

Martin

A. J.

(2007). Examining a multidimensional model of student motivation and engagement using a construct validation approach. British Journal of Educational Psychology, 77, 413-440.

33.

Motti-Stefanidi

Masten

Asendorpf

J. B.

(2015). School engagement trajectories of immigrant youth: Risks and longitudinal interplay with academic success. International Journal of Behavioral Development, 39, 32-42.

34.

National Survey of Student Engagement. (2000). The NSSE report: National benchmarks of effective educational practice. Bloomington: Indiana University Center for Postsecondary Research and Planning.

35.

Reeve

(2006). Teachers as facilitators: What autonomy-supportive teachers do and why their students benefit. The Elementary School Journal, 106, 225-236.

36.

Reschly

A. L.

Betts

Appleton

J. J.

(2014). An examination of the validity of two measures of student engagement. International Journal of School & Educational Psychology, 2, 106-114.

37.

Reyes

M. R.

Brackett

M. A.

Rivers

S. E.

White

Salovey

(2012). Classroom emotional climate, student engagement, and academic achievement. Journal of Educational Psychology, 104, 700-712.

38.

Sass

D. A.

(2011). Testing measurement invariance and comparing latent factor means within a confirmatory factor analysis framework. Journal of Psychoeducational Assessment, 29, 347-363.

39.

Sun

J. C.-Y.

(2014). Influence of polling technologies on student engagement: An analysis of student motivation, academic performance, and brainwave data. Computers & Education, 72, 80-89.

40.

Svanum

Bigatti

S. M.

(2009). Academic course engagement during one semester forecasts college success: Engaged students are more likely to earn a degree, do it faster and do it better. Journal of College Student Development, 50, 120-132.

41.

Treloar

H. R.

Martens

M. P.

McCarthy

D. M.

(2014). Testing measurement invariance of the Protective Behavioral Strategies Scale in college men and women. Psychological Assessment, 26, 307-313.

42.

Van Ryzin

M. J

. (2011). Protective factors at school: Reciprocal effects among adolescents’ perceptions of the school environment, engagement in learning, and hope. Journal of Youth and Adolescence, 40, 1568-1580.

43.

Wonglorsaichon

Wongwanich

Wiratchai

(2014). The influence of students school engagement on learning achievement: A structural equation modeling analysis. Social and Behavioral Sciences, 116, 1748-1755.

44.

Yang

C. H.

M. L.

Chou

P. P.

(2014). A study of technology university students’ English learning engagement scale construction. Bulletin of Meiho University, 33, 1-16.

45.

Yin

Wang

(2016). Undergraduate students’ motivation and engagement in China: An exploratory study. Assessment & Evaluation in Higher Education, 41, 601-621.

46.

Zhao

C. M.

Carini

R. M.

Kuh

G. D.

(2005). Searching for the peach blossom Shangri-La: Student engagement of men and women SMET majors. Review of Higher Education, 28, 503-525.