Preliminary Validation of the SCM in a Sample of Early Adolescent Public School Children

Abstract

The school climate measure (SCM) has demonstrated robust psychometrics in regionally diverse samples of high school–aged adolescents, but remains untested among early adolescents. Confirmatory factor analysis was used to establish construct validity and measurement indices of the SCM using a sample of early adolescents from public schools located in Central Appalachia (n = 1,128). In addition, known-groups validity analyzed each SCM domain against self-reported academic achievement and school connection. Analyses confirmed all 10 SCM domains fit the data well with strong internal consistency and factor loadings. Known-groups analyses suggest students who reported higher academic achievement and school connection demonstrated higher perceptions of school climate. Findings provide evidence that extends the use of the SCM to early adolescents and may support school-based policy.

Keywords

middle school setting/environment

In the United States, middle school is often a transitional period where early adolescents adjust to more autonomy and responsibility in preparation for high school. Moreover, early adolescents are in a delicate emotional and neurological state where learned behaviors track into adulthood (Spengler, Roberts, Lüdtke, Martin, & Brunner, 2016). For example, students who have a positive school experience and graduate from high school are more likely to abstain from tobacco use, maintain a healthy weight, and positively manage stress (Zimmerman & Woolf, 2014). In addition, a positive school experience often helps students prepare for postsecondary education and better occupational opportunities (Mann et al., 2018).

Public schools in the United States serve an estimated 90% of the adolescent population (National Center Education Statistics, 2016). Research suggests early adolescents are prone to developing negative behavioral patterns when exposed to bullying, violence, drugs, and unhealthy stress (Sigurdson, Undheim, Wallander, Lydersen, & Sund, 2015). As a result, students’ academic performance may suffer and lead to a loss of interest in school, mental health issues, and delinquent behaviors (Strøm, Thoresen, Wentzel-Larsen, & Dyb, 2013). Schools, therefore, are well-suited to provide preventive health services and education during key stages of human development (Eccles & Roeser, 2010).

Beneficial behavioral patterns learned at school might serve as future social assets. For instance, students may flourish when they perceive school as a safe and protected environment (Sweeney & Von Hagen, 2015). Of particular importance to perceived school safety, and thus a positive learning environment, are student–teacher relationships and school connectedness (Anderson, 1982). Students who report more positive engagement with their teachers report significantly reduced violence, feeling safe at school, and increased grade point averages (Zullig et al., 2014). Moreover, students who report greater attachment to their school have demonstrated improved classroom behavior, attendance, and grades (Blum, 2005). Therefore, it is essential that schools promote a positive social–emotional setting to foster ideal educational and health-related outcomes to support student success (Bradshaw, Koth, Thornton, & Leaf, 2009).

School climate may be defined as “. . . patterns of people’s experiences of school-life and reflects norms, goals, values, relationships, teaching and learning practices, and organizational structures” (Cohen, McCabe, Michelli, & Pickeral, 2009, p. 182). Evidence suggests students that attend schools with a positive school climate engage in less disruptive behavior (Gregory et al., 2010), have greater academic motivation and achievement (Thapa, Cohen, Higgins-D’Alessandro, & Guffey, 2013), and elevated well-being (Suldo, Thalji-Raitano, Hasemeyer, Gelley, & Hoy, 2013) compared with students from schools with a negative school climate. Therefore, investigating school climate is important because student health and learning may be influenced by a school’s physical, social, and emotional environment (Michael, Merlo, Basch, Wentzel, & Wechsler, 2015).

A student’s connection or “happiness” with school has been described as a representation of social bonding with the broader institution (Thorlindsson & Bernburg, 2006). Research suggests school connection is an important indicator of adolescent mental and emotional health, which may affect academic success (Suldo, Riley, & Shaffer, 2006), and school-based wellness (Tian, Chen, & Huebner, 2013). This is noteworthy because students who bond with their school tend to have higher perceptions of school climate and are more likely to do well in school (Huebner, Gilman, Reschly, & Hall, 2009). However, the research base that explains the relationship between school climate, school connection, and academic achievement among early adolescents is still relatively scarce (Wang & Degol, 2016).

With the passage of Every Student Succeeds Act (ESSA), states and communities are given the freedom to tailor and assess public schools to optimize student achievement (Senate Committee on Health Education Labor and Pensions, 2016). Although annual federal accountability guidelines require schools to report standardized metrics for reading and mathematics, ESSA recommends states and school districts include at least one measure of school quality and safety. One emergent instrument that may fulfill ESSA’s recommendations is the school climate measure (SCM, Zullig et al., 2015), which evaluates perceptions of school climate at the student level. The SCM is a 42-item multidimensional instrument measuring 10 domains of school climate

The SCM has demonstrated robust estimates of reliability and validity across multiple, diverse samples of high school–aged adolescents (Zullig et al., 2015; Zullig, Koopman, Patton, & Ubbes, 2010). The SCM’s robust psychometric evidence, ease of use, cost of administration, and broad applicability gave rise to its inclusion as the only measure of school climate in the PhenX Toolkit (see Hamilton et al., 2011, for a review), which was funded by the National Human Genome Research Institute to compose a core set of high-quality, well-established, low-burden measures intended for use in large-scale studies. However, a current limitation of the SCM is an evaluation of its psychometric properties among early adolescents has not been undertaken.

Such an investigation extends the potential generalizability and applicability of the SCM to multiple age groups of adolescents to support ESSA’s success. In addition, this may increase our understanding of contextually based school climate characteristics among early adolescents. For example, although it is plausible to think there may be similar school climate characteristics between early adolescents and high school–aged adolescents, research suggests the developmental and learning needs between these two groups may be different (Scott, Hirn, & Alter, 2014). This idea is illustrated in a study by De Pedro, Gilreath, and Berkowitz (2016), where their findings suggest students in higher grades (e.g., 11th and 12th) reported lower perceptions of school climate (i.e., stronger feelings toward disengagement from school). This suggests that as students transition from middle to high school, their perception of a positive school climate tends to diminish overtime (Wang & Dishion, 2012). This idea seems practical as middle schools tend to be smaller (Bedard & Do, 2005) and retain more of the elementary school focus on student–teacher relationships and developmental appropriateness (see Levin & Mee, 2016, as an example) because middle school students are less independent than high school students. Conversely, high schools tend to be larger (Henry, 2008), so students assume greater levels of autonomy, which may contribute to diminished perceptions of school climate (Pianta & Allen, 2008).

This preliminary study addressed these literary gaps by first exploring the internal consistency and construct validity of the SCM among a sample of early adolescents from Central Appalachia. A second aim of this study assessed the SCM against academic achievement and school connection. Specifically, it was hypothesized that the SCM would (a) demonstrate acceptable observed reliability and validity estimates in this sample of early adolescents and (b) positive perceptions of school climate would be significantly associated with higher academic performance and connection with school.

Method

Participants

As part of a perennial community health promotion project designed to strengthen protective and reduce risk factors among young adolescents, this study used baseline data from three public middle schools located in the Central Appalachian region in the United States. Students from the region represent a spectrum of diverse characteristics from families living in severe isolation/poverty to modest privilege/affluence (Appalachian Regional Commission, 2017).

A total of 1,358 students enrolled in sixth, seventh, and eighth grades were surveyed using cross-sectional clustered sampling techniques. The aggregated sample consisted of 1,154 respondents from all schools (response rate = 82.4%). Data cleaning procedures removed a total of 26 participants due to unreliable responses resulting in a final sample of 1,128 observations (response rate = 80.7%). The sample was evenly distributed between female (50.5%) and male (49.5%). Most students reported their race as White (female 82.2%, male 79.2%) and academic achievement as mostly As/Bs (female 74.0%, male 67.3%). Additional sample characteristics may be found in Table 1.

Table 1.

Sample Characteristics (n = 1,128).

Variable	Male		Female
Variable	n	%	n	%
Biological sex	555	49.5	567	50.5
Race
White	434	79.2	457	82.2
All other	114	20.8	99	17.8
Age (in years)
11	77	14.2	107	19.0
12	155	28.1	167	29.7
13	227	41.2	210	37.3
14+	91	16.5	79	14.0
Grade in school
Sixth	140	25.4	156	27.7
Seventh	203	36.8	209	37.1
Eighth	209	37.9	199	32.3
GPA
Mostly As/Bs	313	67.3	368	74.0
Mostly Cs	112	24.1	100	20.1
Mostly Ds/Fs	40	8.6	29	5.8
Maternal education
College graduate	143	28.2	141	27.0
High school graduate	79	15.6	113	21.6
Less than high school	21	4.1	34	6.5
Does not know	265	52.2	235	44.9

Note. Missing values have been omitted from table calculations. GPA = grade point average.

Measures

Dependent variable

The SCM

The SCM contains 42 items assessing 10 domains: positive student–teacher relationships (eight items), order and discipline (six items), opportunities for student engagement (six items), school physical environment (four items), academic support (four items), parental involvement (three items), school connectedness (four items), perceived exclusion/privilege (three items), school social environment (two items), and academic satisfaction (two items). Survey participants indicated how much they agree or disagree with the item statements on a 5-point Likert-type scale from “strongly disagree” (coded 1) to “strongly agree” (coded 5). Items were then combined into domains (subscales) and centered on the mean for sensitivity and interpretation (Kelley, Evans, Lowman, & Lykes, 2017) with higher scores indicative of more positive perceptions of school climate. For additional details on SCM constructs and a priori item validation, see Zullig et al., 2015; Zullig et al., 2010 for a review.

Independent variables

Academic achievement

To capture student achievement, self-reported grades in Mathematics and English were acquired using two items under a single question “What were your FINAL grades LAST year?” with selection options A (coded 1) through F (coded 5). The two grade variables were then combined to form a single grade variable then reversed categorically into high achievement (coded 2) represented by mostly As/Bs, Cs being left unchanged as a midpoint (coded 1) for interpretation, and low achievement (coded 0) represented by mostly Ds/Fs.

School connection

Five items assessed school connection (Thorlindsson & Bernburg, 2006) using a 5-point Likert-type scale with response options ranging from “applies almost always to me” (coded 1) to “applies almost never to me” (coded 5). Three items asked students about their studies, “I find the school studies pointless,” “I am bored with the studies,” and “I feel I do not put enough effort into studies”; and two items ask students about their school, “I want to quit school” and “I want to switch schools.” Higher scores indicating more connection with school.

For the known-groups analysis, scale items were separated into five variables. Three variables were related to students’ connection with their studies and two with their connection with school. Each variable was then collapsed into three ordered categories by combining response options 1 and 2 into “always” (coded 0), 3 as “sometimes” (coded 1), 4 and 5 into “never” (coded 2). Students in the “never” group are classified to demonstrate higher connection with school.

Covariates

Maternal education

Maternal education has been shown to influence student outcomes and was chosen as a proxy for student socioeconomic status (Prickett & Augustine, 2015). Maternal education was obtained by asking students about their mother’s educational attainment with the question “What is the highest level of schooling your mother has completed?” Responses were “college graduate” (coded 3), “high school graduate” (coded 2), “less than high school” (coded 1), and “I don’t know” (coded 0).

Biological sex

Differences between females and males has been noted throughout the school-based literature and was chosen to account for the potential covariance among biological sexes (Marcenaro–Gutierrez, Lopez–Agudo, & Ropero-García, 2017). Biological sex was assessed by asking respondents “Are you a boy or girl?” Girl was (coded 1) and boy was (coded 0).

Procedures

Prior to data collection, passive consent (Chartier et al., 2008) was acquired by sending a letter home to caregivers in addition to several opportunities to opt their child out of the study, including returning a signed letter to the school, emailing the principal investigators, or calling either the school or principal investigators (parental opt-out rate < 1%). During the passive consent period, each school was assigned a supervising contact agent (trained data collector) to provide oversight during the day of the data collection.

Data were collected using either a paper-and-pencil or web-based questionnaire, which has been shown to have little difference in quality response rates among adolescents (Wyrick & Bond, 2011). Participation was voluntary and made available to all students without capturing direct identifiers (i.e., names). Students were free to answer all or part of the survey and could opt out of participation at any time. In total, the survey instrument contained 262 items and averaged 30 to 45 min to complete. Schools and supervising contact agents were provided a small dollar amount as an incentive for their time and participation. For additional details on data collection procedures, see Kristjansson, Sigfusson, Sigfusdottir, and Allegrante (2013). In this study, specific variables were selected from the data to address study aims and associated hypotheses. All aspects of the data collection were approved by West Virginia University’s Institutional Review Board (IRB, protocol 1406345394A001).

Data analysis

Descriptive statistics

All analyses were performed in SAS 9.4® (SAS Institute, 2014). Frequencies were used to analyze the discrete (categorical) variables of biological sex, race, age, classification (grade), and maternal education. Any major outliers found were considered for deletion if correction was not an option.

Observed reliability

Cronbach’s (1951) alpha was used to determine observed reliability of the SCM domains and if any instrument items may cause factor suppression. Alpha coefficients (α) are reported in Table 2.

Table 2.

School Climate Measure (SCM) Domains, Reliability Coefficients, and Factor Loadings.

Domain/factor	% Variance	α	Factor Loadings
1. Positive student–teacher relationships (eight items)	17.2	.92	0.71	0.71	0.76	0.74	0.67	0.83	0.85	0.86
2. Order, safety, and discipline (six items)	16.6	.88	0.76	0.72	0.81	0.84	0.66	0.70
3. Opportunities for student engagement (six items)	9.1	.87	0.77	0.76	0.75	0.76	0.72	0.66
4. School physical environment (four items)	6.6	.93	0.84	0.90	0.86	0.90
5. Academic support (four items)	10.7	.85	0.72	0.79	0.76	0.80
6. Parental involvement (three items)	6.2	.79	0.68	0.75	0.82
7. School connectedness (four items)	6.0	.85	0.76	0.76	0.84	0.73
8. Perceived exclusion/privilege (three items)	5.6	.89	0.81	0.90	0.83
9. School social environment (two items)	4.9	.87	0.86	0.88
10. Academic satisfaction (two items)	2.2	.78	0.82	0.77

Note. Cronbach’s alpha (α) were used for observed interitem reliability. % variance explained are from weighted communality estimates.

Construct validity

Confirmatory factor analysis was performed on all 10 SCM domains to determine instrument performance and model fit. As model fit is determined by multiple indices, standard root mean residual (SRMR) and root mean square error of approximation (RMSEA) were calculated with values < .05 viewed as desirable. In addition, to indicate incremental fit, the comparative fit index (CFI) and Tucker–Lewis index (TLI) were also calculated with values > .95 viewed as desirable (Hu & Bentler, 1999). In addition, an interscale correlation matrix (see Table 3) was produced as a means of evaluating discriminant validity.

Table 3.

SCM Interscale Correlations.

SCM Domain	Scale 1	Scale 2	Scale 3	Scale 4	Scale 5	Scale 6	Scale 7	Scale 8	Scale 9	Scale 10
Scale 1: Positive student–teacher relationships	—
Scale 2: Order, safety, and discipline	.68**	—
Scale 3: Opportunities for student engagement	.69**	.70**	—
Scale 4: School physical environment	.50**	.56**	.56**	—
Scale 5: Academic support	.59**	.55**	.58**	.49**	—
Scale 6: Parental involvement	.44**	.38**	.38**	.28**	.37**	—
Scale 7: School connectedness	.60**	.57**	.55**	.54**	.49**	.48**	—
Scale 8: Perceived exclusion/privilege	.24**	.27**	.32**	.20**	.18**	.08*	.28**	—
Scale 9: School social environment	.55**	.49**	.52**	.49**	.51**	.37**	.51**	.22**	—
Scale 10: Academic satisfaction	.46**	.38**	.42**	.36**	.51**	.35**	.53**	.16**	.40**	—

Note. SCM = school climate measure.

p ⩽ .05. **p < .01.

Known-groups validity

To support our measurement model and construct validity, a series of ANCOVAs were used to explored “known-groups” among the 10 SCM domains and the selected academic performance and school connection items theorized to be associated with school climate while controlling for maternal education and biological sex. In addition, to examine unique difference (post hoc comparisons) among variable levels by computing least squares means, a Tukey–Kramer adjustment set at an alpha level of .05 was performed. Therefore, for these analyses, it was hypothesized that students who reported lower perceptions of school climate would also report significantly lower academic performance and lower perceptions of school connection when compared with those who reported higher perceptions of school climate. For the collapsed independent variables, effect sizes were calculated from the post hoc comparisons to determine the magnitude of practical significance by reporting the f effect size index for multiple means (Cohen’s f). Cohen’s f ranges of .10, .25, and .40 indicate small, medium, and large effects, respectively. When interpreting the f statistic for multiple mean comparisons, small effect sizes (≤ .10) are generally not considered practically important, whereas medium and large effects (> .10) are believed to be important (Cohen, 1988).

Results

Using the PROC CALIS procedure, the “mod” statement, a Wald’s test, and Lagrange multiplier indices (SAS Institute, 2014), we further tested and scrutinized factor loadings to ensure they were robust (.40 or greater) and not complex (i.e., not load on more than one factor). All factor loadings were appropriate, so modification models were not necessary. Table 2 reports the results for the confirmatory factor analysis depicting the factor loadings, internal consistency estimates (α) and communality estimates of each SCM domain. Factor loadings ranged from .66 to .90 suggesting the saturation of items fit well within latent constructs. Overall, the factor model fit the data well, χ² = 2132.5(774), p ⩽ .0001, CFI = .95; TLI = .94; RMSEA = .03, 90% CI = [.03, .04], SRMR = .03. This being the first exploration of the SCM among a sample of early adolescents with inclusion of all 10 domains, the fit statistics are consistent with prior research among high school students (Zullig et al., 2015).

Factorial invariance testing was also performed to ensure the confirmatory factor analysis measurement model functioned equally across groups (Dimitrov, 2010). The invariance model fit reasonably well for males, χ² = 1,571.76(774), p < .0001; CFI = .94; TLI = .94; RMSEA = .040; 90% CI = [.040, .046], and females, χ² = 1,671.09(774), p < .0001; CFI = .94; TLI = .94; RMSEA = .045; 90% CI = [.042, .048]. Chi-square difference, standardized parameter estimates, and residuals were nearly equivalent across groups, thus suggesting that the model is invariant across biological sex.

Interscale correlations for the SCM domains are displayed in Table 3. Correlation coefficients ranged from r = .70 (p < .0001) between order and discipline and opportunities for student engagement to r = .08 (p = .0082) between perceived exclusion/privilege and maternal involvement. The variability in strength suggests that although interscales are significantly correlated, there is discriminability between the domains.

ANCOVA models were all found to be statistically significant with F values ranging from F(16, 818) = 4.34, p < .0001 (perceived exclusion/privilege) to F(16, 811) = 20.13, p < .0001 (academic support). This suggests students’ perceptions of school climate among each SCM domain demonstrated associations with academic achievement and school connection in the hypothesized direction. When students reported positive views among SCM domains, academic performance and school connection also moved in the desired direction, even after controlling for maternal education and biological sex.

Least-squared means and standard deviations for all 10 SCM domains, academic achievement, and school connection are reported in Table 4. Post hoc comparisons using a Tukey–Kramer adjustment were used to assess the unique mean differences between levels of academic achievement and school connection. In addition, comparisons between means for levels of academic achievement, school connection, and each corresponding SCM domain along with effect sizes are reported in Table 4. As anticipated, not all SCM domains demonstrated statistical significance between levels of academic achievement and school connection. Moreover, each ANCOVA showed linear declines in SCM domain mean scores compared with the referent group between levels of academic achievement and school connection. Effect sizes ranged from small (f = .02) between school physical environment and “wanting to quit school” to large (f = .50) between school connectedness and “being bored” with studies.

Table 4.

Known-Group Analysis With 10 SCM Constructs, Grades, and School Affect.

Variables	Student–Teacher relationshipsn = 842M (SD)	Order and Disciplinen = 843M (SD)	Opportunities for Engagementn = 836M (SD)	School Physical Environmentn = 829M (SD)	Academic Supportn = 828M (SD)	Parental Involvementn = 833M (SD)	School Connectednessn = 834M (SD)	Perceived Exclusionn = 835M (SD)	School Social Environmentn = 832M (SD)	Academic Satisfactionn = 829M (SD)
Grades
Mostly As/Bs	25.7 (7.6)*	21.3 (5.5)*	21.4 (5.4)	15.1 (3.7)	7.6 (1.6)**	9.0 (3.3)*	10.9 (4.4)	9.3 (3.5)	6.8 (2.1)	5.8 (2.4)*
Mostly Cs	24.5 (8.4)	20.6 (6.6)	21.0 (6.0)	20.6 (6.6)	6.7 (2.0)**	8.3 (3.5)*	10.8 (4.3)	9.3 (3.5)	6.7 (2.3)	5.1 (2.3)*
Mostly Ds/Fs^†	22.2 (8.9)f = .26	19.3 (6.3)f = .20	19.8 (5.9)f = .17	14.5 (3.8)f = .11	6.1 (2.3)f = .43	7.2 (3.0)f = .31	10.4 (4.1)f = .08	8.6 (3.6)f = .15	6.6 (2.5)f = .06	4.7 (2.4)f = .25
I find school studies pointless
Never	24.5 (7.3)	20.9 (5.2)	21.5 (4.8)*	15.5 (3.3)*	6.9 (1.6)	8.6 (3.3)*	11.1 (4.2)*	9.5 (3.5)	6.9 (1.9)*	5.3 (2.3)
Sometimes	23.4 (7.0)	19.7 (5.3)	19.9 (5.3)	14.1 (3.8)	6.5 (1.7)	7.8 (3.2)	10.0 (3.5)	8.7 (3.2)	6.4 (1.9)	4.9 (2.1)
Always^†	24.5 (9.1)f = .11	20.5 (6.9)f = .14	20.8 (6.6)f = .20	15.0 (4.3)f = .26	7.0 (2.3)f = .16	8.0 (3.6)f = .18	11.0 (4.6)f = .19	8.9 (3.8)f = .17	6.7 (2.6)f = .17	5.3 (2.6)f = .14
I am bored with my studies
Never	26.7 (6.9)**	22.1 (5.1)**	22.0 (4.7)**	15.3 (3.3)*	7.0 (1.6)*	8.7 (3.3)^*	12.4 (4.2)**	9.8 (3.5)*	7.3 (1.8)**	5.9 (2.2)**
Sometimes	23.9 (7.1)*	20.4 (5.1)*	20.8 (5.2)*	15.0 (3.3)	6.9 (1.6)	8.2 (3.1)	10.6 (3.5)*	8.7 (3.3)	6.7 (1.9)	5.3 (2.2)*
Always^†	21.8 (8.3)f = .39	18.6 (6.2)f = .36	19.5 (6.0)f = .28	14.3 (4.2)f = .14	6.5 (2.2)f = .16	7.6 (3.6)f = .20	9.1 (4.0)f = .50	8.7 (3.6)f = .25	6.2 (2.4)f = .32	4.4 (2.4)f = .38
I do not put enough effort into studies
Never	24.3 (7.5)	20.8 (5.5)	21.0 (5.2)	14.9 (3.5)	7.2 (1.5)*	8.1 (3.4)	10.6 (4.4)	9.2 (3.6)	6.7 (2.0)	5.5 (2.3)*
Sometimes	23.8 (7.7)	20.4 (5.7)	20.9 (5.4)	15.0 (3.4)	6.7 (1.9)	8.0 (3.2)	10.6 (3.9)	9.0 (3.3)	6.6 (2.2)	4.8 (2.2)
Always^†	24.3 (8.7)f = .05	19.9 (6.3)f = .08	20.4 (6.3)f = .06	14.6 (4.3)f = .07	6.5 (2.3)f = .24	8.3 (3.6)f = .05	10.9 (4.3)f = .03	9.0 (3.6)f = .05	6.8 (2.4)f = .07	5.3 (2.6)f = .23
I want to quit school
Never	25.6 (7.4)*	21.0 (5.5)	21.3 (5.1)	15.3 (3.5)	7.2 (1.6)*	8.1 (3.3)	11.0 (4.2)	8.9 (3.5)	6.7 (2.0)	5.3 (2.4)
Sometimes	23.5 (7.6)	20.0 (5.4)	20.7 (5.7)	15.1 (3.5)	6.6 (1.8)	8.1 (3.4)	10.7 (3.9)	8.8 (3.2)	6.7 (2.3)	5.0 (2.0)
Always^†	23.3 (9.2)f = .21	20.1 (6.8)f = .12	20.3 (6.7)f = .10	14.2 (4.4)f = .16	6.6 (2.6)f = .27	8.3 (3.8)f = .04	10.4 (4.6)f = .08	9.5 (4.0)f = .11	6.7 (2.6)f = .02	5.2 (2.6)f = .10
I want to change schools
Never	24.9 (7.5)*	20.8 (5.5)	21.4 (5.1)*	15.2 (3.5)	7.0 (1.6)	8.5 (3.4)	11.0 (4.2)	9.1 (3.5)	7.0 (2.0)*	5.4 (2.4)
Sometimes	25.0 (7.4)*	20.3 (5.5)	21.1 (5.3)	14.4 (3.6)	6.8 (1.9)	8.1 (2.7)	10.8 (3.8)	9.4 (3.4)	6.8 (2.2)	5.0 (2.3)
Always^†	22.4 (8.9)f = .20	19.9 (6.6)f = .08	19.7 (6.6)f = .17	15.0 (4.5)f = .15	6.6 (2.5)f = .11	7.9 (3.8)f = .12	10.2 (4.5)f = .10	8.7 (3.9)f = .11	6.3 (2.5)f = .16	5.2 (2.4)f = .12

Note. SCM = school climate measure.

†

Referent group for pairwise comparisons, Cohen’s f = .10 small, .25 medium, and .40 large (Cohen, 1988). *p ⩽ .05. **p < .01.

Among school connectedness, students who reported never feeling bored with their studies reported significantly higher school connectedness, M = 12.4, SD = 4.2, t(2) = 4.66, p < .0001, compared with students who reported sometimes feeling bored, M = 10.6, SD = 3.5, t(2) = 3.54, p = .0012, and students who reported always feeling bored, M = 9.1, SD = 4.0, t(2) = 7.73, p < .0001, with their studies (Cohen’s f = .50). This suggests students who reported more connection to school were significantly more likely to report enjoyment in school (i.e., not feeling bored) toward their studies. Among school connection categories, medium to large effect sizes were detected between “being bored with studies” and seven of the 10 SCM domains.

The magnitude of effect for academic achievement demonstrated medium-to-large effect sizes for four of the 10 SCM domains with academic support (SCM 5) revealing the most robust effects. Results displayed a linear trend between students who reported mostly As/Bs (M = 7.6, SD = 1.6) compared with students who earned mostly Cs, M = 6.7, SD = 2.0, t(2) = 6.21, p < .0001, and mostly Ds/Fs, M = 6.1, SD = 2.3, t(2) = 6.58, p < .0001. Furthermore, students who earned mostly Cs were also significantly different, t(2) = 2.41, p = .0425, than students who indicated earning mostly Ds/Fs (Cohen’s f = .43), suggesting students who reported greater academic success also reported significantly greater academic support.

Discussion

The purpose of this study was to explore the efficacy of the SCM among public school early adolescents as an extension from previous literature among high school–aged adolescents (Zullig et al., 2014). As accountability measures evolve in educational practice, versatile, low burden measures of school climate, such as the SCM, may be useful to educational leaders, administrators, school psychologists, and counselors in assessing, tracking, and improving nonacademic factors related to important student outcomes as required by, for instance, the ESSA.

In the current study, psychometric analyses suggest the data fit the measurement model well. In addition, measurement models between males and females also fit the data well, with invariance tests reporting no difference between biological sexes. This finding suggests the use of the SCM may not require alternation to accommodate for contextual characteristics such as biological sex. However, additional replication research is needed to confirm this finding.

Interscale correlations suggest the SCM domains are related but distinguishable from one another, further supporting the multidimensionality of the SCM. Finally, known-groups validity results suggest expected and practically important associations between the SCM domains, school connection, and academic achievement. The SCM operated in consistent and predictable ways with these groups of students known to face increased academic adversity. In sum, students who reported lower academic achievement and less school connection reported significantly lower perceptions of school climate, even after controlling for maternal education and biological sex.

Positive student–teacher relationships followed by order and safety were the most pronounced domains in the measurement model. Our findings are relatively consistent with other school climate literature (see Wang & Degol, 2016, for a review), which suggests the above domains are important to school climate. It is well known that teachers using high-quality classroom management practices tend to be more influential during the learning process (National Research Council, 2005). In addition, when schools provide equitable disciplinary action and appropriate expectations of behavior, students tend to perform better (Blum, 2005).

Interestingly, a large effect size was observed between groups for academic performance where effect sizes for the school connection variables were small to medium. Although somewhat speculative, this may suggest that academic support and school connection are not directly related or perhaps are mediated by a confounding variable not accounted for in our analysis. It is also possible that a student may believe they understand how to do their homework and feel confident in doing so, but still find the assignment “boring” or “pointless.” Although additional research is needed to explore these possibilities, if academic performance and school connection are independent of one another, there may be other factors associated with student perceptions of academic tasks regardless of support.

Notably, student perceptions of being “bored with their studies” was significantly and negatively associated with all 10 SCM domains, of which seven domains had medium-to-large effect sizes. To the best of the authors’ knowledge, this is the first time this variable has been tested using the SCM. Although current results highlight potential directions for research, our findings support the inclusion of school connection as part of school climate studies. In addition, our findings suggest the prioritization of appropriate academic relevance, engagement, rigor, and fun for early adolescents may be important to students and promote a positive school climate. Therefore, understanding how students perceive a connection with their academic studies and school (positive or negative) and its relationships to school climate may present useful information for school-based interventions and policy.

Perhaps unique to our findings, our conceptualization of school connectedness was strongly associated with students who reported being “bored with their studies,” and to a lesser extent, students’ who reported their school studies as “pointless,” but not with “putting effort into their studies,” “wanting to quit school,” or “change schools.” This was somewhat surprising as school connectedness has been shown to support students in multiple facets of the learning environment (Blum, 2005) and that it alone may be a representation of all other ecologic functions of the school environment and to the greater community. However, this was not the case in our model. This result may be an indication of other SCM domains accounting for some of the known components of school connectedness. For example, positive student–teacher relationships and order and discipline are often incorporated in other measures of school connectedness, whereas the SCM measures these domains separately.

Limitations

First, despite encouraging reliability and validity evidence related to the SCM in this predominantly White sample of public school early adolescents in Central Appalachia, results may not be generalizable to the larger Appalachian region or to early adolescents in the United States. In addition, the racial homogeneity of the sample may further limit the external validity of study findings. Although the SCM has been tested in a racially diverse sample of high school–aged students (see Zullig et al., 2014, for a review), studies with heterogenetic samples are needed. Second, although results replicate findings with high school–aged students (Zullig et al., 2015; Zullig et al., 2014; Zullig et al., 2010), research with early adolescents in other regions of the United States is needed to replicate the findings reported in this preliminary study. Third, data collection relied on cross-sectional observations. Similar studies may benefit from using prospective designs that add temporal complexity and direction to school climate models. By better understanding changes in the perceptions of school climate, interventions that support and promote a positive school climate may present practical implications toward student outcomes. Fourth, due to the relative simplicity of our model and statistical methods, we are unable to rule out confounding variables and correlated error estimates. Complementary studies using advanced statistical modeling, while keeping parsimony in mind, may support further refinement of the SCM validity and affect on student outcomes. Fifth, observations were self-reported and subject to threats from bias. However, our response rates for this sample were more than adequate, indicating that statistical biases in the data was likely minimal. Finally, it is possible some statistical inflation of students’ academic achievement may have been present due to self-report in our models. Thus, additional studies using objective measures of academic achievement are needed to further evaluate the generalizability of the SCM to other samples of early adolescents.

Implications for Practice

This preliminary study supports the validity and reliability of the SCM in a sample of public school early adolescents in Central Appalachia for the first time. Robust estimates of internal consistency reliability and construct validity through confirmatory factor analysis suggest excellent model fit for the 10 SCM domains and invariant by biological sex among this sample. Furthermore, findings between this sample and those for high school–aged adolescents in research using the SCM (Zullig et al., 2015; Zullig et al., 2014; Zullig et al., 2010) are similar, suggesting developmental differences between early adolescents and high school–aged adolescents may not be that different, at least as it pertains to assessing school climate with the SCM.

This study also demonstrated significant associations among self-reported academic achievement, school connection, and the SCM domains. Effect sizes from these associations ranged from modest to large, demonstrating the multidimensionality of the SCM for early adolescents and that it is related to important academic outcomes and motivations, and specifically boredom with studies and self-reported grades. These combined findings stress the importance of multiple contextual influences at school and further highlight that student achievement is much more than a product of classroom instruction. Given that high achievement also involves positive interpersonal relationships, order within the school, and students’ motivation to learn (Hoy & Hannum, 2016), this research underscores the use of the SCM as a potential tool for meeting ESSA requirements for early adolescents.

Thus, as schools’ search for strategies to satisfy accountability reporting requirements, the SCM may be a useful and versatile instrument to inform school-wide decision making related to programmatic school processes and policy. For example, the Arizona Department of Education (AZDOE) included several SCM domains (positive student/teacher relationships, order and discipline, school physical environment, and academic support) as part of its safe and supportive schools (S3) grant (see Zullig et al., 2014) to work with schools in identifying needs, setting goals and objectives, and tracking progress toward meeting those goals. The selection of four domains by AZDOE for their grant efforts also reinforces the multidimensional nature and versatility of the SCM.

Lastly, our findings provide preliminary evidence to support the importance of measuring school climate as one of many factors related to academic achievement and school connection, which may be integrated as part of a comprehensive plan to help improve schools attain desired student and school outcomes.

Footnotes

Acknowledgements

This work was supported by Sisters Health Foundation in Parkersburg, West Virginia; ICE Collaborative 2014-2015. The authors would like to extend our gratitude to the funder for their generous support.

Authors’ Note

The opinions expressed are those of the authors and do not represent the views of Sisters Health Foundation in Parkersburg, West Virginia.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Shay M. Daily

Megan L. Smith

References

Anderson

C. S.

(1982). The search for school climate: A review of the research. Review of Educational Research, 52, 368–420.

Appalachian Regional Commission. (2017). Person below poverty level, 2011-2015. Retrieved from https://www.arc.gov/reports/custom_report.asp?REPORT_ID=70

Bedard

(2005). Are middle schools more effective? The impact of school structure on student outcomes. Journal of Human Resources, 40, 660–682.

Blum

R. W.

(2005). A case for school connectedness. Educational Leadership, 62(7), 16–19.

Bradshaw

C. P.

Koth

C. W.

Thornton

L. A.

Leaf

P. J.

(2009). Altering school climate through school-wide positive behavioral interventions and supports: Findings from a group-randomized effectiveness trial. Prevention Science, 10, 100–115. doi:10.1007/s11121-008-0114-9

Chartier

Stoep

A. V.

McCauley

Herting

J. R.

Tracy

Lymp

(2008). Passive versus active parental permission: Implications for the ability of school-based depression screening to reach youth at risk. Journal of School Health, 78, 157–164. doi:10.1111/j.1746-1561.2007.00278.x

Cohen

(1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.

Cohen

McCabe

Michelli

N. M.

Pickeral

(2009). School climate: Research, policy, practice, and teacher education. Teachers College Record, 111, 180–213.

Cronbach

L. J.

(1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297–334.

10.

De Pedro

K. T.

Gilreath

Berkowitz

. (2016). A latent class analysis of school climate among middle and high school students in California public schools. Children and Youth Services Review, 63(2), 10–15.

11.

Dimitrov

D. M.

(2010). Testing for factorial invariance in the context of construct validation. Measurement and Evaluation in Counseling and Development, 43, 121–149. doi:10.1177/0748175610373459

12.

Eccles

J. S.

Roeser

R. W.

(2010). An ecological view of schools and development. In Meece

J. L.

Eccles

J. S.

(Eds.), Handbook of research on schools, schooling, and human development (pp. 6–17). New York, NY: Routledge.

13.

Gregory

Cornell

Fan

Sheras

Shih

T.-H.

Huang

(2010). Authoritative School discipline: High school practices associated with lower bullying and victimization. Journal of Educational Psychology, 102, 483–496.

14.

Hamilton

C. M.

Strader

L. C.

Pratt

J. G.

Maiese

Hendershot

Kwok

R. K.

. . . Haines

(2011). The PhenX Toolkit: Get the most from your measures. American Journal of Epidemiology, 174, 253–260.

15.

Henry

D. B.

(2008). Changing classroom social settings through attention to norms. In Yoshikawa

Shinn

(Eds.), Toward positive youth development: Transforming schools and community programs (pp. 40–57). Oxford, NY: Oxford University Press.

16.

Hoy

W. K.

Hannum

J. W.

(2016). Middle school climate: An empirical assessment of organizational health and student achievement. Educational Administration Quarterly, 33, 290–311.

17.

L. T.

Bentler

P. M.

(1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1–55. doi:10.1080/10705519909540118

18.

Huebner

E. S.

Gilman

Reschly

Hall

(2009). Positive schools. In Lopez

S. J.

Snyder

C. R.

(Eds.), Oxford handbook of positive psychology (2nd ed., pp. 561–569). Oxford, NY: Oxford University Press.

19.

Kelley

Evans

M. D.

Lowman

Lykes

(2017). Group-mean-centering independent variables in multi-level models is dangerous. Quality & Quantity, 51, 261–283.

20.

Kristjansson

A. L.

Sigfusson

Sigfusdottir

I. D.

Allegrante

J. P.

(2013). Data collection procedures for school-based surveys among adolescents: The Youth in Europe Study. Journal of School Health, 83, 662–667. doi:10.1111/josh.12079

21.

Levin

D. M.

Mee

(2016, March). Achieving the vision of this we believe: How middle schools can meet the essential attributes of middle level education. Retrieved from https://www.amle.org/BrowsebyTopic/WhatsNew/WNDet/TabId/270/ArtMID/888/ArticleID/615/Achieving-the-Vision-of-This-We-Believe.aspx

22.

Mann

M. J.

Kristjansson

A. L.

Smith

M. L.

Daily

S. M.

Thomas

Murray

(2018). From tactics to strategy: Creating and sustaining social conditions that demand and deliver effective school health programs. Journal of School Health, 88, 333–336. doi:10.1111/josh.12614

23.

Marcenaro–Gutierrez

Lopez–Agudo

L. A.

Ropero-García

M. A.

(2017). Gender differences in adolescents’ academic achievement. Young, 26, 250–270. doi::10.1177/1103308817715163

24.

Michael

S. L.

Merlo

C. L.

Basch

C. E.

Wentzel

K. R.

Wechsler

(2015). Critical connections: Health and academics. Journal of School Health, 85, 740–758.

25.

National Center Education Statistics. (2016). Enrollment trends. Digest of Education Statistics, 2015 (NCES 2016-014). Retrieved from https://nces.ed.gov/fastfacts/display.asp?id=65

26.

National Research Council. (2005). How students learn: History, mathematics, and science in the classroom. Washington, DC: The National Academies Press.

27.

Pianta

R. C.

Allen

J. P.

(2008). Building capacity for positive youth development in secondary school classrooms: Changing teachers’ interactions with students. In Shinn

Yoshikawa

(Eds.), Toward positive youth development: Transforming schools and community programs (Vol. 1, pp. 21–40). New York, NY: Oxford University Press.

28.

Prickett

K. C.

Augustine

J. M.

(2015). Maternal education and investments in children’s health. Journal of Marriage and Family, 78, 7–25. doi:10.1111/jomf.12253

29.

SAS Institute. (2014). Base SAS 9.4 procedures guide: Statistical procedures. Cary, NC: Author.

30.

Scott

T. M.

Hirn

R. G.

Alter

P. J.

(2014). Teacher instruction as a predictor for student engagement and disruptive behaviors. Preventing School Failure, 58, 193–200.

31.

Senate Committee on Health Education Labor and Pensions. (2016, June 29). The every child achieves act of 2015. Retrieved from https://www.help.senate.gov/imo/media/The_Every_Child_Achieves_Act_of_2015–summary.pdf

32.

Sigurdson

J. F.

Undheim

A. M.

Wallander

J. L.

Lydersen

Sund

A. M.

(2015). The long-term effects of being bullied or a bully in adolescence on externalizing and internalizing mental health problems in adulthood. Child and Adolescent Psychiatry and Mental Health, 9, Article 42..

33.

Spengler

Roberts

B. W.

Lüdtke

Martin

Brunner

(2016). Student characteristics and behaviours in childhood predict self-reported health in middle adulthood. European Journal of Personality, 30, 456–466.

34.

Strøm

I. F.

Thoresen

Wentzel-Larsen

Dyb

(2013). Violence, bullying and academic achievement: A study of 15-year-old adolescents and their school environment. Child Abuse & Neglect, 37, 243–251.

35.

Suldo

S. M.

Riley

K. N.

Shaffer

E. J.

(2006). Academic correlates of children and adolescents’ life satisfaction. School Psychology International, 27, 567–582.

36.

Suldo

S. M.

Thalji-Raitano

Hasemeyer

Gelley

C. D.

Hoy

(2013). Understanding middle school students life satisfaction: Does school climate matter? Journal of Applied Research and Quality of Life, 8, 169–182.

37.

Sweeney

S. M.

Von Hagen

L. A.

(2015). Middle school students’ perceptions of safety: A mixed-methods study. Journal of School Health, 85, 688–696.

38.

Thapa

Cohen

Guffey

Higgins-D’Alessandro

(2013). A Review of School Climate Research. Review of Educational Research. 83(3), 357–385. doi:10.3102/0034654313483907

39.

Thorlindsson

Bernburg

J. G.

(2006). Peer groups and substance use: Examining the direct and indirect effect of leisure activity. Adolescence, 41, 321–339.

40.

Tian

Chen

Huebner

E. S.

(2013). The longitudinal relationships between basic psychological needs satisfaction at school and school-related subjective well-being in adolescents. Social Indicators Research, 119, 353–372. doi:10.1007/s11205-013-0495-4

41.

Wang

M. T.

Degol

J. L.

(2016). School climate: A review of the construct, measurement, and impact on student outcomes. Educational Psychology Review, 28, 315–352. doi:10.1007/s10648-015-9319-1

42.

Wang

M. T.

Dishion

T. J.

(2012). The trajectories of adolescents’ perceptions of school climate, deviant peer affiliation, and behavioral problems during the middle school years. Journal of Research on Adolescence, 22, 40–53. doi:10.1111/j.1532-7795.2011.00763.x

43.

Wyrick

D. L.

Bond

(2011). Reducing sensitive survey response bias in research on adolescents: A comparison of web-based and paper-and-pencil administration. American Journal of Health Promotion, 25, 349–352.

44.

Zimmerman

Woolf

S. H.

(2014, June 5). Understanding the relationship between education and health. Paper presented at the Roundtable on Population Health Improvement, Washington, DC.

45.

Zullig

K. J.

Collins

Ghani

Hunter

A. A.

Patton

J. M.

Huebner

E. S.

Zhang

(2015). Preliminary development of a revised version of the school climate measure. Psychological Assessment, 27, 1072–1081. doi:10.1037/pas0000070

46.

Zullig

K. J.

Collins

Ghani

Patton

J. M.

Huebner

E. S.

Ajamie

(2014). Psychometric support of the school climate measure in a large, diverse sample of adolescents: A replication and extension. Journal of School Health, 84, 82–90. doi:10.1111/josh.12124

47.

Zullig

K. J.

Koopman

T. M.

Patton

J. M.

Ubbes

V. A.

(2010). School climate: Historical review, instrument development, and school assessment. Journal of Psychoeducational Assessment, 28, 139–152.