(Re)Measuring Community College Student Engagement: Testing a Seven-Factor CCSSE Model

Theoretical Dimensions

While such framing broadly articulates that “engagement is about two elements: what the student does and what the institution does” (Wolf-Wendel et al., 2009, p. 413), this dynamic exchange is not always reflected in theoretical or empirical efforts approaching community college student engagement. Such a critique has been carefully presented by Hatch (2017) who has expanded on this interaction to consider community college student engagement, in both terminology and practice, as sociocultural in nature. As Hatch (2017) writes,

Engagement is not something a student does or experiences but rather . . . a lived reality that is co-constructed by students along with their peers, faculty members, and others, who all interact within colleges in a simultaneous specific and broad context. (p. 6)

Through this lens, community college student engagement is as much, if not more, about the community college and its organizational actors as it is about the student themselves.

This sociocultural approach finds further support in a recent book chapter by Tillapaugh (2019). Building on and synthesizing critical perspectives (e.g., Duran & Jones, 2019; Stewart, 2017), Tillapaugh problematizes the concept of engagement by considering the narratives, perspectives, and reasoning of those who do not get involved. In this presentation, Tillapaugh specifically includes consideration of contexts—the historical, intellectual, emotional, and political environments which directly and indirectly animate student experiences—and their role in understanding student involvement and engagement (see also Duran & Jones, 2019). Elaborating on this point, Tillapaugh (2019) further notes that there needs to be “acknowledgement that one context can be experienced and perceived in a multitude of ways by multiple students” (p. 200). This is an important consideration not only for theoretical work on engagement but also for measurement efforts and their subsequent interpretations.

Measurement Dimensions

In addition, several studies have been critical of the CCSSE and its construct validity, accurate measurement of student engagement, psychometric properties, and evidence of a factor structure that comports with theoretical and practical realities (Hatch, 2012; Nora et al., 2011; Roman et al., 2010). These concerns about the CCSSE suggest that additional research on the instrument and its operation is necessary to provide evidence for or against its value as a valid and appropriate measure of a more sociocultural understanding of community college student engagement. Hence, this study returns to using an original theoretical driver of this instrument, Chickering and Gamson’s (1987) “Seven Principles for Good Practice in Undergraduate Education,” to motivate, create, and test a seven-factor confirmatory model using items from the CCSSE.

The choice to pursue this avenue of inquiry was made for two primary reasons, which collectively comprise the rationale for our efforts. First, given the widespread use of the CCSSE, we identified a need to reconsider these guiding principles and their ability to inform strategic leaders within community college environments. As McCormick et al. (2013) describe in their historical account of engagement theory and measurement, “The [seven] principles emphasize the responsibility of leaders and educators to ensure that students engage routinely in high levels of effective educational practice” (p. 54). Thus, we see inherent advantages in returning to this original presentation to further consider the CCSSE in light of supporting sociocultural approaches to understanding student engagement, which is a contemporary perspective that focuses not exclusively on the student but instead on the dynamic exchanges between the student and the community college.

Second, we wanted to investigate the extent to which CCSSE data reflected the perceptions of the collegiate environment, as originally reported by the students themselves with respect to theorized learning practices. As will be further discussed in the literature review, current reporting structures are based on five benchmarks which draw on Chickering and Gamson (1987) as well as other theoretical perspectives (see McCormick et al., 2013). These benchmarks include the following: (a) active and collaborative learning, (b) student effort, (c) academic challenge, (d) student–faculty interaction, and (e) support for learners (Marti, 2005). As Pike (2013) notes in his discussion of measuring engagement via the NSSE, benchmarks were designed to “represent clusters of good educational practices and to provide a starting point for examining specific aspects of student engagement” rather than “represent underlying psychological constructs” (p. 163). While we understand, as Hatch (2017) observes in citing these efforts, that such “benchmarks were designed not as psychometric scales but as a heuristic measure for engendering conversations about related practices by practitioners” (p. 7), the survey items were designed to capture student behaviors and perceptions. As McCormick et al. (2013) write, “Because beliefs and attitudes are antecedents to behavior, perceptions of the campus environment are a critical piece in assessing a student’s receptivity to learning” (p. 57). Thus, although the benchmarks themselves are heuristics, the items themselves reflect students’ beliefs, practices, and perceptions of the community college learning environment.

Integrating Theory and Measurement

Addressing this tension in light of renewed theoretical energies may in turn open new possibilities for interpretation. From a more traditional or student-centric view, benchmarks logically follow as they provide community college stakeholders with a snapshot of student involvement during a given academic year relative to other institutions. Taken from a contextualized sociocultural vantage such as that considered by Hatch (2017) and Tillapaugh (2019), however, measuring engagement demands consideration of theoretically derived psychometric properties beyond heuristic benchmarks. Put simply: measures of engagement which explicitly link theory to items and items to constructs may ultimately be more accurate indicators of engagement than heuristic benchmarks.

To provide an example of this distinction, numerous questions on the CCSSE ask a student to indicate the extent to which a college emphasizes certain educational practices or forms of involvement. Specifically, we believe that questions investigating forms of support (e.g., “support to thrive socially” and “support you need to help you succeed at this college”) can be considered as reflecting underlying psychological impressions beyond what benchmarking would indicate. We contend, again from our contemporary interpretation of engagement, that such questions thus benefit from consideration of psychometric properties (e.g., construct reliability and validity) to more fully account for individual response patterns beyond those which can be reported by benchmarks. Such psychometric approaches also may provide the same or, perhaps, even greater practitioner benefit, allowing for an understanding of which set(s) of student perceptions (as evidenced by higher loadings) are most directly informing specific perceptions of a given practice or set of practices.

Student Engagement

Broadly considered, studies of student engagement have shown that certain institutional and individual practices generally lead to better outcomes for students, subsequently increasing retention and completion (Kimbark et al., 2017; Webber et al., 2013). However, because student engagement encompasses many nuances, including both behavioral and psychological components, the large variety of definitions has caused confusion as each provides a different focus or lens on this important topic (Angell, 2009; Kahu, 2013; Trowler, 2010). Kahu (2013) provided a helpful summary by stating that “student engagement was seen as an evolving construct that captures a range of institutional practices and student behaviours [sic] related to student satisfaction and achievement, including time on task, social and academic integration, and teaching practices” (p. 759).

Engagement, though, must be carefully considered in empirical contexts for both its quantity and quality. With respect to quantity, the more practices that exist, the more students will, at least theoretically, become engaged; however, engagement itself might not be associated with higher graduation rates. As Johnson and Stage (2018) found in their recent multi-institutional study, “engagement experienced from these practices alone was not necessarily an indicator of likely college completion” (p. 24) and could not be connected to institutional outcomes at large. With respect to quality, high-impact practices certainly vary greatly by institution. Kuh and Kinzie (2018), in their response to Johnson and Stage, point out that the depth and robustness of the high-impact practice is just as important as having the practice alone. As such, continued research is necessary to probe the importance of student engagement on the attainment of meaningful credentials.

Furthermore, it is vital to consider the extent to which such discussions of practices usefully map on to community college students with their unique student populations and resource constraints (see Wyner, 2014). As research in this area has demonstrated, student demographics (Gibson & Slate, 2010), the activities students choose to engage in (Saenz et al., 2011), and the unique institutional cultures of individual community college settings (Dudley et al., 2015) can influence student engagement and its resultant effects.

Measuring Student Engagement

As increasing numbers of institutions focus on student engagement, researchers in turn have questioned how to measure this phenomenon. One of the most popular measures for student engagement, utilizing the student involvement theory of Astin (1984) and the seven principles of good practice by Chickering and Gamson (1987), is the NSSE, which was developed in 1999 by researchers at the University of Indiana. The focus of this survey is on students attending 4-year, baccalaureate institutions representing “student behaviors that are highly correlated with many desirable learning and personal development outcomes” (Kuh, 2002, p. 2). The needs of and ways with which students connect with their higher education institution can vary significantly depending on gender (Bronkema & Bowman, 2018), age (Heagney & Benson, 2017; Timms et al., 2018), or socioeconomic status or financial situation (Pearce & Down, 2011).

Seeing a need for a similar survey for students at 2-year institutions, researchers at the University of Texas–Austin developed the CCSSE in 2001 (Center for Community College Student Engagement [CCSSE], 2017; McClenney, 2007). Since its initial design, it has undergone several revisions, but has remained largely unchanged since 2005. This instrument operationalized theoretical underpinnings from the work of Astin (1984) and Chickering and Gamson (1987) along with Pace’s (1984) research on student effort.

Marti (2005) has provided the clearest explanation of the formation of the CCSSE, particularly in describing its psychometric properties and the factors and constructs that underlie the instrument. An exploratory factor analysis (EFA) using 2003 survey data was used to solely inform a CFA, a type of structural equation model where items are grouped in factors based on relevant theory and research a priori, or before the model is tested. Although no information regarding the estimation method or the intricacies of the model were provided, the tested CFA was found to be significant with a latent variable model underlying the instrument. Such analysis was offered as evidence for a nine-factor model (Marti, 2005, 2009), employing only 39 of the more than 120 survey items (including demographics and self-reported academic data) that are part of the CCSSE instrument.

Marti’s (2005) initial CFA provided evidence for a nine-factor model with five benchmarks being offered to guide practice. These benchmarks were established by a technical advisory panel who “reviewed the CFA results and then assigned items to benchmarks, taking into account the results of factor analysis and reliability tests—and also applying expert judgment” (Marti, 2005, p. 14). Notably, the developers of the CCSSE have indicated that the instrument is a holistic measure of engagement without a latent underlying factor structure—hence the ability to selectively add and subtract items as desired. They offered that the purpose of doing a CFA was to understand relationships and not to confirm or deny a singular factor structure. Marti (2005) explicitly stated that the “survey was not designed to measure a set of latent constructs defined a priori” (p. 13). In addition, Marti (2005) noted the limitation of the instrument is that the items themselves were not designed to load on specific factors in accordance with any specific theory. He goes on to say that “confirmatory factor analysis models assume orthogonality, and it is evident that the dimensions of student engagement are non-orthogonal” (p. 13).

Understanding this presentation, we note two important features relevant to our current study. First, we observe that while the items themselves were not designed to measure specific theoretically derived constructs—as would be the case with the traditional use of CFA—the development of the CCSSE instrument was not atheoretical (McCormick et al., 2013). Instead, as is reflected in the benchmarks, the items and associated constructs draw heavily on extant theoretical models, the primary of which is Chickering and Gamson’s (1987) principles (e.g., student–faculty interaction). Second, benchmarks have inherent trade-offs that must be considered. On the positive side, they provide clear indicators and points of comparison that can direct resource allocation and guide practice. On the potentially adverse side, however, benchmarks may not be nuanced enough to truly capture the full dynamics of student experiences, especially given the wide diversity of engagement patterns existing among contemporary community college students. With specific attention to the critical considerations offered by Tillapaugh (2019), we believe it is an appropriate and worthwhile use of CCSSE data to directly test its efficacy using Chickering and Gamson’s (1987) practices and underlying student perceptions of their appearance. This is not to undo benchmarking but to suggest an alternative approach which more fully captures sociocultural dynamics of engagement using theoretically supported items in the CCSSE.

Evaluating the CCSSE

In the spirit of free exchange and accuracy, other researchers have used the CCSSE instrument to provide additional evidence for further considering the CCSSE and its use. Mandarino and Mattern (2010) used a five-factor model similar to the final benchmarks of the CCSSE and found that the predictive validity of the instrument was sub-optimal. Angell (2009) identified and tested a four-factor structure using the entire 120-question instrument. Roman et al. (2010) looked at three of the five benchmarks—the so-called Retention Index—and found little meaningful evidence of a connection between the Index of the CCSSE and actual retention data. It is hard to directly compare these studies as some utilized sound theoretical reasoning while others utilized exploratory techniques to coalesce around a particular factor. However, the variety of results raises questions regarding the reliability and validity of the instrument despite the author’s evidence of convergent and predictive validity.

Perhaps, the most direct and in-depth challenge to research emerging from the instrument developers was provided by Nora et al. (2011) in their study investigating the elements of the internal factor structure of the CCSSE. Their study examined what items loaded onto latent factors and then compared these constructs with the five-factor benchmarks created in the CCSSE. Their findings found a different grouping of items on each of the factors than what was reported by the instrument developers. As such, Nora et al. (2011) stated that the “items in the CCSSE do not realistically capture a sense of student engagement and that scales provided by the benchmarks do not predict various student outcomes” (p. 119). In response, McCormick and McClenney (2012), part of the initial design team of the NSSE and CCSSE, responded to these findings by questioning the “unspecified procedure” that was used to evaluate the five factors and the methodology that Nora et al. used in their study (p. 328). Primarily, the authors were concerned with Nora et al.’s exclusion of some factors that may have made their factor analysis closer to the original nine-factor model found by Marti (2009).

It is clear the researchers on both sides of this argument have strong feelings regarding the use and interpretation of the CCSSE. Nora et al. (2011) sum it up best by saying:

At the core of these arguments is the validity and ability of each instrument to truly capture those constructs purportedly designed to provide different measures of student engagement and to meet assumptions of validity and conceptual soundness. While arguments and counterarguments have been made about the survey instruments’ intents, most specifically that they are not intended as a true measure of student engagement, reality points to its use by higher education institutions as a scorecard for an institution’s ability to engage its students. (p. 108)

Nora et al. (2011) also leave open the possibility that additional research should continue to evaluate the instrument, particularly using robust modeling techniques, such as CFA, that can identify factor structures within the instrument and rigorously evaluate the fit of such models.

To that end, this research study seeks to identify a model where the factor structure is determined a priori and is solely based on theoretical underpinnings. Can an alternative organization of the CCSSE items, grounded in an original theoretical motivation and not repackaged into thematic benchmarks, provide a useful and practical model to conceptualize contemporary understandings of student engagement for community college professionals? We believe it can. In the next sections, we detail our method and results.

Sample

The analysis sample consists of 1,076 responses, with 524 students from a 2012 sampling and 552 students from a 2013 sample. Both years’ responses were drawn from two rural community colleges in southern and western Virginia that took the CCSSE test within the specific sampling procedures required by the CCSSE (2017). Despite the differences in years, the survey instrument used during each sampling was the exact same, allowing the scores to be integrated. Listwise deletion, whereby all responses with missing data (excluding demographic questions) are removed from the sample, was performed to avoid imputation techniques that might bias the measurement analysis (Tabachnick & Fidell, 2013). A subsequent analysis revealed no systematic bias present within the deleted cases. After listwise deletion, a final sample size of 905 students was retained. Even with listwise deletion, this study uses more than the recommended number of cases for its parameter estimation (see Kline, 2011). Table 1 provides all available descriptive characteristics.

OPEN IN VIEWER

Table 1.

Descriptive Statistics for CCSSE Sample (N = 905).

Variables	Frequency	%
Gender
Female	490	54.1
Male	408	45.1
Missing	7	0.8
Age (years)
18–19	321	35.5
20–21	250	27.6
22–24	115	12.7
25–29	80	8.8
30–49	109	12.0
50+	23	2.5
Missing	7	0.8
Race/ethnicity
White	790	87.3
Hispanic	40	4.4
African American	26	2.9
Asian	13	1.4
Other	25	2.8
Missing	11	1.2
Native English speaker
Yes	844	93.3
No	56	6.2
Missing	5	0.6
International student
Yes	23	2.5
No	875	96.7
Missing	7	0.8
Married
Yes	124	13.7
No	775	85.6
Missing	6	0.7
Enrollment
Part-time	700	77.3
Full-time	199	22.0
Missing	6	0.7

Note. CCSSE = Community College Survey of Student Engagement.

Instrument

The CCSSE consists of well over 120 items. For this study, 37 items were included that theoretically fit within Chickering and Gamson’s (1987) theoretical approach, the seven principles. Many of the items are the same as were used in either Marti’s (2009) nine-factor analysis or CCSSE’s five benchmarks. This includes the items dealing with having conversations with others different than yourself that were removed prior to the five benchmarks, and the addition of other items that were not included in either of the previous models. Other items, like “used the internet or instant messaging to work on an assignment,” while included in CCSSE’s benchmarks, did not appropriately fit into any of Chickering and Gamson’s seven principles and were not included in our model. Estimating in this theoretically derived fashion allowed us to interpret constructs along dimensions proposed by Chickering and Gamson. This included a construct reflecting respect for diverse talents and ways of knowing central to our critical sociocultural approach in defining and operationalizing community college student engagement.

Model

Theoretical and published research is the basis of this study. As such, a CFA is an appropriate testing method (Kline, 2011). As this model was developed using Chickering and Gamson’s (1987) seven principles, we are only testing the efficacy of mapping existing CCSSE items onto this theory. Alternative models are not necessary in this instance as we are not attempting to achieve optimal model fit via model comparison utilizing the exact same set of items (e.g., testing a six- vs. a seven- vs. an eight-factor model). All models were tested using LISREL 8.80 (Jöreskog & Sörbom, 2006).

Estimation Method

When using a CFA, the estimation method chosen should be unbiased and efficient, providing consistent parameter estimates (Finney & DiStefano, 2013). Because of the ordered categorical nature of the CCSSE, normal theory estimators like generalized least squares (GLS) and maximum likelihood (ML) are not appropriate. Both GLS and ML assume a multivariate normal distribution with continuous data and use a covariate matrix to estimate the model. Using ML or GLS with categorical data can lead to a variety of problems including biased chi-square values, standard errors, and parameter estimates (T. A. Brown, 2006; Finney & DiStefano, 2013).

Instead, the models tested in this study used a polychoric correlation matrix instead of a Pearson correlation or covariate matrix with the observed variances of the indicators not analyzed. Polychoric correlations assume that normally distributed, unobserved continuous variables underlie the given observed variables (Savalei, 2014). To provide the best parameter estimates, robust diagonally weighted least squares (rDWLS) was used to estimate the parameters of the models. This estimation method has been shown to be more asymptotically efficient and have smaller sampling variability than ML or weighted least squares (WLS) resulting in more accurate chi-square values and parameter estimates. For the weight matrix in the estimator, rDWLS uses the asymptotic covariance matrix of the correlations, but the inversion occurs only on the diagonal instead of the full matrix. This allows for adjustment to the standard errors of the parameter estimates since they are biased. In addition, the chi-square test statistic and the fit indices are adjusted to correct for the bias using rDWLS (Finney & DiStefano, 2013).

Assessing Model-Data Fit

Before estimating the confirmatory factor analyses, it is necessary to select appropriate global fit indices. Even with categorical data, much of the discussion around fit indices is the same as if it were continuous because the chosen estimator, rDWLS, adjusts the parameters using scaling techniques. Thus, the chi-square goodness-of-fit significance test, an exact test statistic of model-data fit, is appropriate to use. Non-significant chi-square values indicate that the model does not fit significantly worse than a just-identified model.

Hu and Bentler (1998) advocate for reporting one absolute and one incremental fit index in addition to the chi-square statistic. Yu and Muthén (2002) found that for categorical data that has a large sample size, the root mean square error of approximation (RMSEA) and comparative fit index (CFI) are best suited for absolute and incremental indices, respectively. RMSEA estimates lack of fit of the population data to the model due to model misspecification, not sampling error. It is the amount of model misfit per degree of freedom and the closer to zero, the better the fit. The CFI, measuring incremental fit, compares the hypothesized model with a null model with a value of 1.0 indicating that the lack of fit is completely from the null model. As Hu and Bentler (1999) did for normal theory estimators, Yu and Muthén (2002) provided suggested cutoff values for each of the indices when data are non-normal or categorical in nature. For the RMSEA, strong scores are below 0.05, whereas scores above 0.95 for the CFI are desirable. Both the RMSEA and CFI perform well with large sample sizes above 200. Thus, considering multiple global fit indices, like the chi-square, RMSEA, and CFI, instead of relying on a single fit index (Hu & Bentler, 1998; Marsh et al., 2004) provides a more holistic picture of global model-data fit. Although global indices are helpful, they can mask smaller, localized areas of model-data misfit. These localized areas can be analyzed from the residuals estimated between the observed and the model-implied correlations. Higher absolute values on these polychoric correlations, generally greater than |.15| or |.20|, indicate areas of local misfit. It is also important to calculate reliability and construct replicability. This statistical approach evaluates how well a set of items represents a latent variable, or in our case, seven latent variables. For this study, the H statistic, developed by Hancock and Mueller (2001), can be helpful because we are interested in the characteristics of the factor itself as opposed to respondents’ scores on each factor. As the statistic approaches 1, the magnitude of the factor loadings increases. Hancock and Mueller (2001) recommend that optimal reliability values for a construct should be above .70.

Limitations

The primary limitation with this research is concerned with limited access to usable data sets. While the sample size was appropriate to run the model, the generalizability of the data might not be strong considering only two institutions from Virginia were part of the data set. Given the high level of community college student’s demographic heterogeneity on a national level, future evaluations should include a larger data set from a more diverse representation of the country to evaluate potential models.

In addition, given our theoretical justification, this model was selected for examining the relationships between items and constructs. Alternative theoretical assumptions or beliefs could certainly lead to alternative model specifications. Considering the challenges associated with this instrument in our initial presentation, particularly related to the thematic benchmarks, we would encourage other scholars to leverage this limitation into their own future research considerations.

Linking theory to practice, we recognize that truly capturing the engagement dynamic between students and institutions quantitatively is itself a challenge. In this study, we have suggested reconsidering student-level CCSSE in light of critical sociological perspectives of engagement in hopes that this reinterpretation can improve institutional practice through providing additional robust measures. Efforts seeking to overcome this limitation could collect data from students as well as institutions, potentially using our scales to inform such efforts (see also Hatch, 2017).