Abstract
Despite its popularity, the factorial structure of the Psychological Sense of School Membership (PSSM) Questionnaire remains controversial. The current study set out to clarify its structure across different cultural contexts. The study was carried out among 1,928 adolescents aged 11 to 20 years in the Netherlands, Kenya, Indonesia, and Spain. Item-based measurement models with a single factor did not show a good fit. A bifactor method (defining a method factor for negatively worded items) did not show a good fit either. However, when items were combined per target to be rated (school, student, teacher, other people, and self), measurement invariance of a one-factorial model was found. Internal consistencies observed across the cultural groups studied were high. In addition, an evaluation of functional equivalence indicated that the relationship between PSSM and life satisfaction was invariant across countries. Implications for assessment and future directions are discussed.
School is an important developmental context for adolescents, providing not only an opportunity for academic development but also for social and emotional development (Minuchin & Shapiro, 1983). The sense of belonging, attachment, and acceptance within the school context is especially salient for the majority of the adolescents. School belonging has been variously defined, with recent reviews indicating that the key components include affiliation to school, an individual’s perception of fitting in and being and accepted by others in the school. Goodenow (1993) defines sense of school belonging as the extent to which a student feels personally accepted, respected, included, and supported by others within the school social environment. Studies indicate that adolescents’ sense of school belonging is related to a wide array of outcomes including academic engagement, academic achievement, motivation, psychological well-being, mental health, and the existence of problem behavior (Anderman, 2002; Goodenow & Grady, 1993; Hagborg, 1994; Sánchez, Colón, & Esparza, 2005; Shochet, Homel, Cockshaw, & Montgomery, 2008). Given this salience, it is important to understand how best to measure the construct.
One of the most widely used measures of this construct is the Psychological Sense of School Membership (PSSM; Goodenow, 1993). The measure was developed “to assess the adolescent’s perceived belonging or psychological membership in the school environment” (Goodenow, 1993, p. 79), and has been applied in various contexts. Despite its popularity, the factorial structure of the PSSM remains controversial. In its original format, the PSSM was conceptualized as a unidimensional measure with one underlying factor (You, Ritchey, Furlong, Shochet, & Boman, 2011). However, various scholars report different structures ranging from two to three factors, with some having to exclude items to identify a clear structure (Shochet, Smith, Furlong, & Homel, 2011; Togari, Sato, Yamazaki, & Otemari, 2011). Table 1 presents a brief overview of studies investigating factorial structure of PSSM. Given the ambiguity in the structure of the PSSM and the growing importance of understanding PSSM, there is an urgent need to further evaluate its factorial structure (Shochet et al., 2011). Furthermore, given the increased interest among educationalists and policy makers to understand factors influencing educational outcomes in different cultural settings, it is essential to examine the extent to which the PSSM can be used across cultural contexts. The current study set out to test different theoretical and corresponding measurements models of the PSSM across cultures as well as the similarity of its link with well-being (i.e., functional equivalence).
Summary of Studies Looking at Factorial Structure of PSSM.
Note. PSSM = Psychological Sense of School Membership; CFA = confirmatory factor analysis.
There are very few studies on student–school relationships (i.e., belonging, connectedness, and attachment) that have been carried out in non-Western settings such as Kenya and Indonesia. However, we do expect that the construct will function similarly across different cultural contexts for two reasons. First, the few studies on school belonging or school connectedness in non-Western context such as Africa, Asia, and Oceania show that belonging and connectedness are associated with good psychological outcomes (Abubakar, van de Vijver, Mazrui, Murugami, & Arasa, 2014; Karcher & Lee, 2002; Qinmei, 2006), similar to what has been observed in Western contexts. In addition, there is both extensive theoretical work and empirical evidence that shows that the need to belong is a universal human need (Baumeister & Leary, 1995). In the current study, where we sample from four different cultural contexts, we would expect that PSSM functions in a similar manner at the construct level.
Theoretical and Measurement Models
The discussion on the factorial composition of the PSSM combines both theoretical and method considerations. Belongingness is assessed as a unidimensional construct in various studies using the PSSM (You et al., 2011). A salient, though potentially problematic methodological aspect of the PSSM is its use of both positively and negatively worded items. The use of positively and negatively worded items was adopted to reduce response bias; however, negatively worded items tend to be scored by participants in a manner that is not opposite to the positively worded items, which leads to shared variance of the items induced by their directionality. This shared variance of items that is restricted to items that are keyed in the same direction can create spurious, independent factors of positively and negatively worded items (for further details, see Hankins, 2008; Marsh, 1996; Spector, Van Katwyk, Brannick, & Chen, 1997). This phenomenon has now been acknowledged as contributing to method artifacts being labeled as substantive factors. Using confirmatory factor analysis, one can test for potential wording effects by creating a method factor that accounts for the shared variance of the negatively worded items (bifactor method). A recent study has evaluated the potential role of wording effects on the factorial structure of PSSM (Ye & Wallace, 2014). The study observed that, after removing some poorly performing items and modeling the effects of wording, PSSM had three substantive factors. This allows for the clarification of the structure without the potential confounds of the item wording. The literature presents several instances of a three-factor model (e.g., Togari et al., 2011; You et al., 2011). To evaluate each of these possibilities, we ran four models: unifactorial, unifactorial with a method factor, three-factor model with a method factor, and two-factor model (Cheung & Hui, 2003). An evaluation of these factor models points to various issues. First, several of these models use an abbreviated scale, as the analysis required some items to be removed. In addition, the model by Togari et al. (2011) produces factors that are based on items with the same “target” person in the item; for instance, items focusing on the teacher cluster together and items focusing on students tend to cluster together. Consequently, we also evaluated a model based on parcels that were created taking into consideration who the target/context is in each item (see Figure 1 for the five clusters created and Table 2 for the items belonging in each of the clusters).
The standardized coefficients from the confirmatory model on the structure of PSSM based on rated targets as parcels.
Overview of Items Comprising Each Parcel.
Method
Sample and Procedure
The study was carried out as part of a larger study on mental health and well-being among adolescents (secondary school students) in four countries: Netherlands, Kenya, Indonesia, and Spain. The samples were selected randomly from their school populations. Within schools, we used stratified random selection of classrooms to recruit students. As we were interested in the broad range of students in the school population, we got students from the highest and lowest grades within schools. All students in the selected classrooms were invited to take part in the study. In each of these countries, questionnaires were administered in group sessions. Table 3 provides sample characteristics and the language of administrations for each country. These measures were forward translated to the target languages and independently back translated to English. When items needed further clarifications during the back translations stage, a meeting to harmonize the translations was carried out between the authors.
Sample Demographics by Country.
Measures
PSSM scale
It is an 18-item scale that measures adolescents’ perceived sense of belonging in the school setting (Goodenow, 1993). Students responded to items such as “People at this school are friendly to me,” using a 5-point Likert-type format ranging from 1 (not at all true) to 5 (completely true).
The Brief Multidimensional Students’ Life Satisfaction Scale (BMSLSS; Huebner, Seligson, Valois, & Suldo, 2006) was applied as a measure of positive psychological well-being. The measure includes six items, five of them focusing on specific domains (family, friends, school, self, and living environment) and one item focusing on global well-being. A sample item is “I would describe my satisfaction with my family life as . . .” These items are scored on a 7-point Likert-type scale ranging from 1 (terrible) to 7 (delighted). Cronbach’s alpha was .78 in Indonesia, .66 in Kenya, .80 in the Netherlands, and .77 in Spain.
Analytic Approach
We carried out a series of multigroup confirmatory factor analyses using AMOS 18. The general guideline is that a nonsignificant chi-square reflects an acceptable fit to the data (Hu & Bentler, 1999). However, given the sensitivity of the chi-square statistic to sample size, we did not consider this in the current study. In addition, values greater than .95 for the Tucker–Lewis index (TLI) and comparative fit index (CFI) are considered to reflect an excellent fit while values between .95 and .90 are considered indicative of an acceptable fit. In a multigroup analysis, the change in CFI is an important indicator for evaluating the suitability of hierarchically nested models; a CFI change of less than .010 is taken to be supportive of the more restrictive model. The root mean square error of approximation (RMSEA) is also reported as it has been shown to be sensitive to model misspecification; values less than .06 are considered indicative of a good fit while those between .06 and .08 are considered indicative of an acceptable model. We specifically avoided testing the adequacy of the model in each country separately as our study question specifically focused on the cross-cultural applicability of the scale.
Results
We used multigroup confirmatory factor analysis to test the models. The first model tested was the single factor model as originally conceptualized by the test developer. The model showed a poor fit to the data, even for the configural invariance (unconstrained) model, χ2(516, N = 1,941) = 4.093, p < .001, χ2/df = 3.799, TLI = .768, CFI = .804, and RMSEA = .038. A second model investigated the potential impact of negative wording by modeling a latent variable that partials the shared variance among negatively worded items; although slightly better, the model still showed a poor fit to the data: χ2(496, N = 1,941) = 1434.30, p < .001, χ2/df = 2.893, TLI = .834, CFI = .873, and RMSEA = .031. A third model investigating the proposed two-factor model that distinguishes between positively and negatively worded items was also evaluated. Again, the model did not show a good fit to the data: χ2(528, N = 1,941) = 1645.97, p < .001, χ2/df = 3.11, TLI = .82, CFI = .84, and RMSEA = .033. A fourth model investigating the proposed three-factor model with a method factor (Ye & Wallace, 2014) was also evaluated but was found to have a negative variance. Hence, this inadmissible result was not further considered.
A previously unaccounted moderator in the PSSM items is the target of the items: Some refer to persons whereas other items refer to institutions. The last model that was tested clustered items that made a split in these targets. Based on this categorization of targets, five clusters (parcels) were created (see Figure 1). This model showed an excellent fit to the data with the unconstrained model indicating the following fit: χ2(12, N = 1,941) = 20.20, p < .001, χ2/df = 1.68, TLI = .98, CFI = .99, and RMSEA = .016. An inspection of the fit indices indicates that when all criteria are taken together, the best fit would be the measurement weights model (given the change in CFI and Akaike information criterion [AIC]) although the statistics of the measurement intercepts model still showed an acceptable fit (see Table 4). These results suggest that PSSM is best conceptualized as a unidimensional measure in which items sharing a target of belongingness (such as items referring to other students) should be taken together as subscales. The PSSM showed an adequate alpha across all countries. Cronbach’s alpha was .82 in Indonesia, .73 in Kenya, .84 in the Netherlands, and .86 in Spain.
Invariance Models and Goodness-of-Fit Indexes of the Multigroup Analysis.
Note. RMSEA = root mean square error of approximation; TLI = Tucker–Lewis index; CFI = comparative fit index.
As an evaluation of functional equivalence, we examined the relationship between the latent structure of PSSM and life satisfaction in a multigroup analysis. Our results indicated that the relationship between sense of belonging and life satisfaction was similar across contexts; measurement weights: χ2(47, N = 1,941) = 125.06, p < .001, χ2/df = 2.66, TLI = .967, CFI = .973, ΔCFI = .006, and RMSEA = .029.
Discussion
Consistent with earlier reports, we observed that a unidimensional model, as originally conceptualized in the development of the PSSM, does not adequately explain the factor structure of the measure across cultural contexts. Contrary to our expectations, the problems with the factorial structure could not be fixed by fitting a two-factor model or by modeling the effects of negative wording in a method factor. Most of the recent evaluations of the factorial structure in other psychological scales with both positive and negative items, such as the General Health Questionnaire (Hankins, 2008) and Rosenberg Self-Esteem Scale (Marsh, 1996), have pointed to the impact of negative wording. Our analyses strongly suggest that finding the expected factor structure is jeopardized by the failure to take commonalities of items with a common target person into account. Some items refer to the extent to which peers treat the student well; other items refer to experiences with teachers while still other items refer generally to adults in the school context. The differences in reference point may lead to shared error variance among the items with a common reference target by focusing on a specific person across all these items; this dependence is never considered in analyses of the instrument. The plausibility of our interpretation is supported by results from earlier studies. For instance, in an investigation of the factor structure of the PSSM in Japan (Togari et al., 2011), three factors were identified. These factors were labeled acceptance by teachers, acceptance by students, and belonging; however, five items had to be excluded from the analysis. Two of the factors were based on a common reference target. In addition, in the literature related to school connectedness, it has been observed that in evaluating school connectedness, it is better to have subscales related to different targets in the school context (Lohmeier & Lee, 2011; Sass, Castro-Villarreal, McWhirter, McWhirter, & Karcher, 2011). For instance, Sass and associates evaluate connectedness to different target groups including peers, teacher, self-in-the present, self-in-the-future, and school. Similarly, Lohmeier and Lee, when developing the School Connectedness Scale, included items specifically targeting three sources of connectedness: school, teacher/adult, and peers. Our solution to cluster items retains the theoretically assumed unidimensionality of school belongingness, but it separates the items based on the contexts/targets.
Implications of Our Results
Our evaluation of PSSM across cultural contexts has three implications. First, the PSSM is best seen as a one-dimensional measure that has a problematic structure at item level and has adequate psychometric properties to warrant its use as a measure of school belongingness in different cultural contexts. Second, when conducting cultural comparisons, the measure is adequate if one clusters the items before carrying out a confirmatory factor analysis. Third, based on our results and those from other studies, it seems that there may be a need to refine the scale further to ensure clarity in measurement, especially if the measure is to be used across cultural contexts. These refinements may need to be carried out at an item level to ensure any ambiguity is removed; for instance, the items asking about “people” in the school contexts may potentially be confusing as some may include adults while other students may rate based on relationship with their peers. A refinement on the exact target group may contribute toward ensuring the measure is interpreted in a similar manner by the respondents.
Limitations of Our Study
Our evaluation of invariance issues is statistical. Therefore, there may be other explanations at linguistic and semantic level that we did not address. Furthermore, as we did not carry out cognitive interviewing, there may be subtle differences in interpretation of items across the contexts of our study that we currently cannot report about. Future studies that use mixed-methods approaches may go a long way in elucidating the sources of errors and contribute to the refinement of the scale.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.