Design and analysis of a scale to measure pro-environmental behaviours among adolescents / Diseño y análisis de una escala de conductas proambientales para adolescentes

Measurement of pro-environmental behaviour among adolescents

The literature review conducted shows several approaches to the study of pro-environmental behaviour in children and adolescents, albeit with important limitations. For example, several studies have sought to evaluate the parental effect, generating ad hoc instruments that assume that behaviours can be deployed by adults, children and adolescents alike (Collado et al., 2019; Grønhøj & Thøgersen, 2017), thus limiting a characterization designed specifically for this population. Other studies have generated instruments for particular contexts (Meinhold & Malkus, 2005; Naim et al., 2019) or oriented towards an evolutionary dimension, such as moral development (Krettenauer, 2017) or sustainability (Fraijo Sing et al., 2012).

Some frequently used instruments are designed to measure attitudes and not necessarily behaviours (Musito Ferrer et al., 2020). In general, most of these instruments developed ad hoc for these studies only report general psychometric properties such as averages, standard deviations or Cronbach’s alpha coefficients, or do not even report these (Collado et al., 2019; Meinhold & Malkus, 2005). In this context, Lange and Dewitte (2019) identify the General Ecological Behaviour-GEB (Kaiser, 1998; Kaiser & Wilson, 2004) scale as one of the most widely used and robust instruments, due to the thoroughness of the psychometric analyses applied. The GEB scale has been adapted to different cultures (Kaiser & Wilson, 2000) and to provide further evidence on its structure (Kaiser & Wilson, 2004). There is also a version of the scale adapted for children (Evans et al., 2007) and for adolescents, GEB-a (Kaiser et al., 2007).

The present study

Considering the above, the study presented in this article sought to generate a relevant instrument that adequately characterizes the experience of the Chilean adolescent population with respect to their pro-environmental behaviours for use in a research context. The first step was to review different instruments that have measured pro-environmental behaviour, deciding to take as our basis the version of the General Ecological Behaviour scale devised for adolescents (GEB-a; Kaiser et al., 2007).

GEB-a (Kaiser et al., 2007) is a scale that measures pro-environmental behaviour from the perspective of the underlying attitude to behaviour, allowing for some control over the influence of socio-economic variables (Moser & Kleinhückelkotten, 2018). Indeed, Otto et al. (2016) note that in Chile there is an important crossover between saving money and certain pro-environmental actions at medium-low and low socioeconomic levels, since sustainable means of transport, energy and water savings and product reuse are more economical.

We began compiling evidence of validation using the 40 original items of the GEB-a. However, the items presented a decontextualization in the pro-environmental behaviours included, in terms of their cultural, temporal and evolutionary relevance. This led to the design of a new instrument, based on established standards for educational and psychological testing (American Psychological Association [APA] et al., 2018). In accordance with these standards, in the different stages of the study, we collected evidence of validity based on the content of the test, on the response processes, on the internal structure and based on relationships with other variables, in addition to estimating the reliability obtained in each of the stages.

Methodology

The design of this instrument is part of a larger study that investigates pro-environmental and prosocial behaviour among Chilean adolescents (Berger & Andaur, 2022). This study applied a mixed methodology and considered three stages as described in the procedures section.

Participants

Teenagers between 12 and 18 years of age from different schools in the Metropolitan Region of Chile took part in the research. The pilot application used convenience sampling via social media, giving a sample of 181 adolescents (55.6% female, M = 15.7, SD = 1.61). The second stage considered two successive applications of the revised instrument with n = 753 (aged 11 to 20, M = 13.38, SD = 1.62; 407 female, 320 male and 26 prefer not to say) and 529 adolescents (aged between 11 and 21, M = 13.54, SD = 1.59; 269 female, 237 male and 23 prefer not to say) contacted through schools. In parallel, we collected evidence of response processes through two cognitive interviews with adolescents, selected intentionally. Finally, the third stage considered a sample of 585 students (aged between 11 and 19, M = 14.07, SD = 1.78; 315 male, 243 female and 27 who do not report their sex) belonging to 11 educational establishments. This sample was split in order to perform confirmatory factor analysis, which in the first sample explored the behaviour of the new items (n = 285) so as to confirm in the second sample (n = 295) the final model proposed from the analyses carried out with the first sample.

Procedure

The first stage of the study entailed building an initial version of the instrument based on the GEB-a, considering expert judgement and the existing literature, thus providing evidence of validity based on the content of the test. This version was applied through a pilot test to a sample of adolescents, examining the first evidence of validity evidence based on the internal structure. We then made modifications based on the pilot analysis and conducted cognitive interviews to gather evidence of response process validity, and so adjust the instrument. The second stage considered the application of the instrument in two successive samples, exploring new evidence of internal structure validity. Finally, the third stage considered a new reformulation of the instrument, introducing new items and modifying the response space, performing confirmatory analyses to evaluate these changes. In the latter case, the sample was divided into two to evaluate the behaviour of the new items with the first half and then choose a model to eventually confirm said model with the second half of the sample. This last piece of information provided new evidence of internal structure, as well as evidence of relationships with other variables, which we will describe later. Table 1 shows the evidence of validation collected in each stage.

OPEN IN VIEWER

Table 1.

Stages of the study and validity evidence associated with each stage.

χ2 model	Stage 1	Stage 2	Stage 3
Content validity evidence	X		X
Response process validity evidence	X	X
Internal structure validity evidence	X	X	X
Validity evidence based on relationships with other variables			X

In all stages, we were very careful regarding the ethical aspects of the study, requesting authorization from the schools (when the participants were contacted through them), obtaining informed parental consent and asking the participants for informed consent. The study was reviewed and approved by the Scientific Ethics Committee on Social Sciences, Arts and Humanities of the Pontificia Universidad Católica de Chile. In the early stages, data were collected online through the Google Forms platform given the context of the COVID-19 pandemic. In the last stages, data were collected online and via physical questionnaires answered using paper and pencil according to the preferences of each school. In both cases, members of the research team who had been trained for this purpose gathered data collectively and synchronously. A psychologist member of the team who was experienced in this procedure carried out the interviews in person.

Instruments

Data analysis

To analyse cognitive interviews, we followed the process of information reduction (Miller et al., 2014), which summarizes, compares and concludes in relation to the main aspects that hindered the response processes of the participants.

To analyse the evidence of internal structure, we performed generalized ordinal confirmatory factor analyses (parametric) using four samples. We decided to treat the data as ordinals, following the scales proposed by Stevens (1946), considering that the scales used in the instrument only allow these categories to be ordered (typical of an ordinal scale), and it could not be argued that the distance between one category and another would be the same (typical of an interval or ratio category).

In the first sample (pilot), by carrying out various confirmatory analyses, we investigated the behaviour of the items, and then confirmed this proposal in the two samples used in the second stage of the process. In the third stage, when new items were being tested, the sample was randomly divided into two to perform further confirmatory analyses that allowed us to investigate the operation of the new items, and then to confirm the resulting model.

In the confirmatory factor analyses, we used the WLSMV estimator, which is robust for ordinal data. To evaluate the fit of the confirmatory models, we used the fit indicators established by Hooper et al. (2008). The Chi-Squared absolute fit index, which is sensitive to large sample sizes, tends to be significant, so we must also consider other fit indicators. In this case, we applied the following indicators according to the criteria established by Hooper et al. (2008). A CFI of .95 would indicate a good fit, with a fit of .90 deemed to be adequate. For the SRMR and RMSEA indicators, values between .05 and .08 would be acceptable, while a good fit would be given by values below .05. Moreover, as Hoe (2008) points out, a ratio of less than 3 between Chi-Squared and Degrees of Freedom could be interpreted as a reasonably good Chi-Squared indicator.

In addition, in the different stages, we carried out a partial credit analysis within the framework of the item response theory models (Masters, 1982), evaluating the difficulties of the items and the fit of each item to the established model. We used a single parameter model, leaving the factor or slope loadings fixed and leaving the difficulties of the items free.

To analyse the evidence of validity in relation to other variables, we considered the correlation with the instrument of connection with nature, where the hypothesis that the scores of both instruments would converge was supported. The correlation between the latent variables yielded by a joint model was considered an adequate estimation, since it contemplates both instruments from the latent variables measurement model and is better able to model the error.

Finally, to estimate the reliability of the instrument, we used indicators that estimate reliability from the estimated scores in the latent variable. In these cases, the reliability index is calculated in relation to the location of respondents in the latent variable rather than in the traditional way that considers raw scores (Wilson, 2005). Specifically, we used the Separation Reliability indicator, which is interpreted in a similar way to Cronbach’s alpha, an indicator that represents a proportion of the variability associated with the attribute being evaluated. In order to determine which values are considered appropriate, as indicated by the standards for educational and psychological tests (APA et al., 2018), this aspect was determined according to the expected use of the instrument, where the greater the consequences, the greater the accuracy/reliability required of the instrument. In this case, the proposed use is to characterize variability in pro-environmental behaviour and make inferences at the aggregate level in order to compare average levels of different groups in research contexts.

We performed exploratory, confirmatory, correlation and partial credit analyses using Mplus 8 version 8.8 (Muthén & Muthén, 2017) and R 4.0.2.

Results

The results are presented based on the different sources of evidence specified by the standards for educational and psychological tests (APA et al., 2018), considering that evidence was collected in the different stages of the study.

Content validity evidence

In the first instance, based on the GEB-a scale, we defined the content of the instrument based on expert judgement and a literature review. The experts consulted in stage one and stage three of the study were experts in the topic of environmental care (and members of the research team), as well as experts in psychometry.

In the first stage, we observed difficulties with items on energy conservation. The original instrument included behaviours that did not apply to the Chilean context (e.g., ‘In winter, I turn the heating up to wear short sleeve clothing’ or ‘In winter, I turn the heating off when I leave my room for more than an hour’). Likewise, some expected behaviours in other contexts are only applicable to subgroups in the Chilean context given their socioeconomic status, for example, due to access to travel and hotels.

Notwithstanding this first definition of contents, once the instrument was applied, new aspects were identified that had to be adjusted, again consulting with experts to adjust the instrument before the third and final stage. The final instrument has eight items from the original scale. Fifteen of them were modified following our consultation of experts and the survey of response processes, as detailed in the following section, and 11 items were constructed to replace the eliminated items. We ensured that all the areas covered by the original instrument were represented in the final version. Energy conservation and mobility had the fewest items, given that teenagers have little autonomy in relation to these areas.

In addition, it is important to consider that the original version of GEB-a was devised 14 years ago. Therefore, we must take into account certain changes in technology and habits when making such adjustments.

Response process validity evidence

The main aspects that emerged in the cognitive interviews conducted in the first stage of the study were the different understandings that adolescents had about the same item, because they referred to obsolete devices (e.g., stereo) or words that lead to different understandings (e.g., ‘biodegradable packaging’, which some teenagers understood as returnable packaging and others as biodegradable carrier bags). These items were updated and refined to ensure that all adolescents could refer to the same action.

Furthermore, it also became clear that adolescents have less agency with regard to behaviours that depend more on the decisions of their parents than on themselves. For example, for the item ‘Unplug electrical appliances (TV) when I am not using them’, younger teenagers commented that they were not allowed to unplug devices because of their age or because they are connected to the internet, so we chose to modify that item with elements that adolescents did have access to unplugging, such as chargers. There were also items that involved resources not managed by adolescents, for example, ‘I contribute to environmental organizations’, which could be interpreted as contributing money or time, so the two types of responses were differentiated. Finally, there were different response processes according to socio-economic level in relation to the availability of recycling points (e.g., ‘I take glass to the recycling centre’, replaced by ‘I separate glass bottles out for recycling’), the use of heating, or transport options (e.g., in a sprawling city like Santiago, it is sometimes not feasible to cycle to places since the distances involved are too great, thereby invalidating the item ‘I prefer cycling to places’).

Internal structure validity evidence

With the pilot sample, we carried out confirmatory factor analyses, initially contrasting one- and six-factor models, following previous study procedures that suggested a one-factor solution since the six-factor model did not significantly improve the fit to the model (Kaiser & Wilson, 2004; Kaiser et al., 2007). In the pilot sample, the six-factor model failed to converge adequately given the high correlation between three factors. So we contrasted three- and four-factor models, in which one factor presented low reliability. In terms of interpretation and parsimony, the one-factor solution was the most appropriate. After this first analysis, we eliminated items that presented low factor loadings or a behaviour that did not fit the partial credit model performed, taking care to maintain coverage in terms of content. Despite this, the areas of energy conservation and mobility were not adequately represented, so in the third and last stage of the study, we incorporated new items and modified others to adjust the scale to the theoretically expected structure of dimensions.

In stage one, the one-factor model from the pilot version had 21 items, with the following fit indicators: χ²(189) = 266,685, p ⩽ .001, with a ratio of χ²/df = 1.4 between the Chi-Squared value and its degrees of freedom, which indicates a good fit despite the fact that the original Chi-Squared statistic indicated a poor fit of the model. The model also presented good fits in the following indicators: CFI = .97, RMSEA = .05 and SRMR = .08. However, since this sample was limited (N = 181), we decided to evaluate in a second stage (including two larger successive samples) a larger model of 24 items including some that were modified following their suggested elimination during the pilot stage. We evaluated this 24-item model with a new sample (n = 753) and had to exclude two items from the final model chosen because of their low factor loadings (‘When I brush my teeth, I leave the water running until I finish’ and ‘At my parties, I ask my parents to avoid using disposable plates and cutlery’) and another because it hindered the convergence of the model (‘I prefer products with biodegradable packaging’). The indicators of the final 21-item model were: χ²(189) = 820,965, p ⩽ .001, with a poor fit, even considering the ratio between the Chi-Squared statistic and the degrees of freedom, which is χ²/df = 4.3. It also did not present good fit indicators for the following: CFI = .87, RMSEA = .07 and SRMR = .09. Based on these results, we made further modifications to the items based on the evidence of the response process and experts, and carried out new analyses with a new sample (N = 529). This analysis confirmed the 21 items, improving the fit indicators in relation to the previous sample, although they still did not achieve a good fit: χ²(170) = 712,974, p ⩽ .001, χ²/df = 4.2, CFI = .91, RMSEA = .078 and SRMR = .085.

Finally, considering that there was still room for improvement in the instrument in relation to its internal structure and content coverage, we incorporated new items and modified others. In addition, based on the opinion of psychometric experts, we modified the response space, improving the interpretability of the items, including dichotomous and frequency items (‘never’, ‘a few times a year’, ‘one to three times a month’ and ‘one to three times a week’, ‘every or almost every day’). We also incorporated a third response scale (‘never’, ‘rarely’, ‘most of the time’ and ‘always’). We split a new sample to perform confirmatory factor analyses with the first half to investigate the operation of the new items (making various adjustments to the model with the same sample based on the observed empirical results, n = 285) and conducting a confirmatory factor analysis with the second half of the sample without adjustments (n = 295).

The final model had 34 items (see Supplementary Material). Since there were different response spaces, we created a two-factor model that maintains the assumption of one-dimensionality of the main factor of interest but controls for possible method effects associated with the use of dichotomous and polytomous response formats. The fit indicators of this model in the exploratory sample (n = 285) were: χ²(493) = 835,993, p ⩽ .001, with an adequate ratio of χ²/df = 1.7. It also achieved good indicators for CFI = .94, RMSEA = .049 and SRMR = .08. The fit indicators for the confirmatory sample (N = 295) were: χ²(493) = 799,190, p ⩽ .001, with a ratio χ²/df = 1.6, which makes for a good fit when making the correction in the estimation given the high sensitivity of Chi-Squared to samples greater than 200. In addition, it obtained the following indicators: CFI = .95, RMSEA = .046 and SRMR = .076, presenting good CFI and RMSEA fit indicators and adequate SRMR fit (Table 2).

OPEN IN VIEWER

Table 2.

Model fit indices.

Model	χ2	df	χ2/df	CFI	RMSEA	SRMR
Stage 1: Sample 1 Model with 21 items	266.685***	189	1.4	.97	.048	.088
Stage 2: Sample 2 Model with 21 items	820.965***	189	4.3	.87	.067	.089
Stage 2: Sample 3 Model with 21 items	712.974***	170	4.2	.91	.078	.085
Stage 3: Exploratory sample of 34 items	835.993***	493	1.7	.94	.049	.08
Stage 3: Confirmatory sample of 34 items	799.190***	493	1.6	.95	.046	.076

Note: χ² = chi-squared; df = degrees of freedom; CFI = comparative fit index; RMSEA = root mean square error of approximation; SRMR = standardized root mean square residual; ***p < .001

In this last sample, we also carried out a partial credit analysis. Figure 1 details some aspects of this analysis through a Wright Map (Torres Irribarra & Freund, 2020).

OPEN IN VIEWER

Figure 1.

Wright Map: empirical map of items emerging from the partial credit model.

Each point marked on the right side of the items represents the point in the latent variable where the previous response categories are no longer more likely and the next category becomes more likely. They are the most informative points for an item and allow us to identify the difficulties associated with each category of response for an item. On the left side, the distribution of people along the latent variable is mapped out. In general terms, there is adequate coverage of the items for the measured population, which is consistent with the high levels of reliability reported below. We also find items with different levels of difficulty, allowing us to report both high and low levels for the latent variable.

Some examples of polytomous items considered easier refer to behaviours such as switching off the TV when it is not in use (item 17), reusing shopping bags (item 24) and unplugging electronic devices (e.g., mobile phone, computer or tablet chargers) when not in use (item 34). Some examples of dichotomous items considered very difficult refer to behaviours such as participating in marches or movements that support causes in favour of the environment (item 1) and enrolling in lectures, workshops or environmental classes to be better informed (item 3). Some of the more difficult polytomous items refer to behaviours such as talking to friends about issues related to the environment (item 21) and letting someone know that they have taken an action that harms or damages the environment (item 33).

Validity evidence based on relationships with other variables

We correlated the scores obtained in the last version of the instrument with the Disposition to Connect with Nature scale. We expected to see a positive and therefore convergent correlation, based on both the conceptual model and previous evidence (Neaman et al., 2021). We evaluated a joint model ¹ between the final version of the pro-environmental behaviour instrument and an adjusted model of connection with nature. The correlation between both latent variables was r = .61 (p ⩽ .001).

Reliability analysis

We used the Separation Reliability indicator, which was .86 in the first sample and .81 and .87 in the second stage for the first and second samples, respectively, all indicators of high levels of reliability depending on the proposed uses.

The final version of the instrument had an estimate of .91 using the separation reliability estimator in the exploratory subsample and .90 in the confirmatory subsample, once again showing good levels of reliability depending on the proposed uses for this instrument, which in this case is to characterize variability in pro-environmental behaviour and make inferences at the aggregate level in order to compare average levels of different groups in research contexts.

Discussion

There is broad consensus within academia, public policies and international organizations that developing pro-environmental attitudes and behaviours in the population, and specifically in children and adolescents, is an unquestionable priority. We must understand what we mean by sustainability (Berger & Andaur, 2022) and the processes through which sustainability can be developed (Mónus, 2022). This requires relevant instruments, appropriate to the socio-cultural context and to the population in which they are applied (Markle, 2013). However, when reviewing the current literature, there is an absence of instruments, especially for adolescents and in the Latin American context. There is broad dynamism in what constitutes pro-environmental behaviour. Thus, public policies, environmental education strategies and measurement instruments may become obsolete quickly, requiring constant review and updating. Moreover, since pro-environmental behaviours are largely dependent on the sociocultural context, the instruments developed must be culturally adjusted and adapted to specific populations.

Pro-environmental behaviours in a specific sociocultural and geographical context are specific and relevant to a specific population. For the purposes of the instrument developed in this study, we see that the socio-economic conditions and the public and/or private availability of pro-environmental alternatives (recycling centres, transport) determine possible behaviours, including the use of more efficient heating systems, recycling and cycling as a means of transport, among others (see, e.g., Araya-Córdova et al., 2021). Moreover, given the great socio-economic inequality in Chilean society (and consequently in educational and parental practices; see, e.g., Ulloa-Vidal et al., 2017), a relevant instrument in some socio-economic contexts may not be relevant in others.

A second aspect relates to autonomy and the areas in which adolescents can make decisions, for example with regard to home practices. Adolescent behaviours are complex and have multiple determinants, depending not only on their attitudes or orientations but also on the beliefs and practices of their families, formal educational models and the public offer focused on this population. So, strategies aimed at developing these behaviours must consider these and other possible elements in order to be effective.

Pro-environmental behaviours are deployed in different dimensions, such as energy conservation, mobility and transport, waste reduction, recycling, consumption and vicarious behaviours towards conservation (Kaiser et al., 2007), among others. While an instrument can capture this dimensionality, the analysis of the instrument developed in this study delivered a one-dimensional solution. Possibly, these dimensions are more easily distinguishable in the adult population, considering the relevance of the sociocultural context and the lower autonomy of adolescents mentioned previously. Although the instrument does not allow us to identify these dimensions separately, in its design and review of the items to be included, we took care to ensure balance and representation for each of them.

Another important aspect to discuss in developing an instrument like this is the response format. The instrument described in this study, following the proposed GEB-a scale, considered the inclusion of items with polytomous and dichotomous response formats. This variety in the format seems adequate considering that the partial credit analysis identified items of varying difficulty.

Limitations of the study and future research pathways

The study presented here provided extensive evidence based on the internal structure of the instrument, reaffirming previous hypotheses of single dimensionality. However, there was little evidence of response processes, which should be explored in greater depth in future research. In addition, the construct was adequately represented by items related to recycling, avoiding waste, vicarious behaviour and reduction of consumption, but less well represented in areas such as energy consumption and mobility. Some of the new items tested in these areas did not behave adequately, requiring future qualitative surveys of types of items that could provide better coverage in these areas. This way, the instrument could achieve greater robustness. It could also be interesting to consider the possibility of integrating new areas and emerging dimensions, such as digital civic participation, which can significantly influence pro-environmental behaviours (Liu et al., 2021). Another limitation is the level of generalization of the results. Although broad data were taken that considered 11 schools from different socio-economic levels, it was not an intentional sample, representative at the country level, which remains a future challenge. It is also important to consider that the instrument is based on self-reporting, which can bias the type of information that respondents provide. Finally, future research could explore smaller versions of the instrument that are invariant to the one proposed in this article.

Despite these limitations, the instrument developed in this research makes a significant contribution to the field of study, since it is theoretically oriented and provides evidence of its structure and reliability. Having a relevant instrument offers an important advance in the promotion of pro-environmental behaviours among the adolescent population, thus favouring the development of societies that care for and about the environment, integrating and developing sustainable lifestyles.