Evaluating Longitudinal Invariance in Dimensions of Mental Health Across Adolescence: An Analysis of the Social Behavior Questionnaire

Abstract

Measurement invariance over time (longitudinal invariance) is a core but seldom-tested assumption of many longitudinal studies on adolescent psychosocial development. In this study, we evaluated the longitudinal invariance of a brief measure of adolescent mental health: the Social Behavior Questionnaire (SBQ). The SBQ was administered to participants of the Zurich Project on the Social Development of Children and Youths in up to four waves spanning ages 11 to 17. Using a confirmatory factor analysis approach, metric invariance held for all constructs, but there were some violations of scalar and strict invariance. Overall, intercepts tended to increase over time while residual variances decreased. This suggests that participants may become more willing or able to identify and report on certain behaviors over time. The noninvariance was not practically significant in magnitude, except for the Anxiety dimension where artifactual increases over development would be liable to occur if invariance is not appropriately modeled. Overall, results support the utility of the SBQ as an omnibus measure of psychosocial health across adolescence.

Keywords

longitudinal invariance Social Behavior Questionnaire adolescence mental health

A key goal of longitudinal research in adolescence is to illuminate processes of development. Usually this is achieved through the examination of stability and change in emotional, psychological, and behavioral traits over several years. The validity of conclusions drawn from such research relies on the availability of comparable trait estimates over development. This in turn requires at least partial longitudinal invariance, namely, that a subset of items capture the same construct on the same measurement scale over time (Edwards & Wirth, 2012; Widaman, Ferrer, & Conger, 2010). Measurement invariance for items can be violated for a variety of reasons, especially in periods of substantial developmental change such as adolescence. When this happens and it is not appropriately modeled, apparent changes over time could be due to changes in the way a construct is measured, true changes could be masked, or an entirely different construct could be measured at different time points (e.g., Edwards & Wirth, 2009).

Core constructs in adolescent psychosocial development research include conduct issues, depression, anxiety, attention deficit hyperactivity disorder (ADHD), and prosocial behavior. A sizeable body of research has sought to characterize development of these dimensions in adolescence. This includes studies of average developmental trajectories, developmental trajectory subtypes, developmental interrelations, and studies identifying predictors and outcomes of developmental trajectories (e.g., Carlo, Crockett, Randall, & Roesch, 2007; Crocetti, Klimstra, Keijsers, Hale, & Meeus, 2009; Dekker et al., 2007; Luengo Kanacri, Pastorelli, Eisenberg, Zuffianò, & Caprara, 2013; Marmorstein, 2009; Martino, Ellickson, Klein, McCaffrey, & Edelen, 2008; Murray, Eisner, & Ribeaud, 2016b; Murray, Obsuth, Eisner, & Ribeaud, 2017; Murray, Obsuth, Zirk-Sadowski, Ribeaud, & Eisner, 2016; Nantel-Vivier et al., 2009; Van Oort, Greaves-Lord, Verhulst, Ormel, & Huizink, 2009). Studies of this kind have, for example, suggested that across adolescence there are general decreases in antisocial behavior, ADHD symptoms, anxiety, depression, and prosociality, with a possible rebound in late adolescence for the latter. However, these average trends are in the context of considerable variation across individuals with, for example, growth mixture analyses, often revealing a nontrivial subgroup for whom levels of these dimensions increase across development (e.g., Crocetti et al., 2009). Studies in this research area have also delineated pathways by which these dimensions may influence one another. Conduct problems may, for example, put an individual at risk of academic and social difficulties that in turn increase the risk of anxiety and depression (e.g., van Lier et al., 2012).

The above-mentioned studies and other longitudinal studies of adolescent psychosocial development rely on trait estimates that can be validly compared across development; however, the array and pace of changes that occur during adolescence make this a challenge. Over development, constructs may change in nature, particular indicators may lose their developmental appropriateness, or samples may reach a floor or ceiling on items that previously discriminated well between different trait levels (e.g., Edwards & Wirth, 2012). Early adolescence, for example, sees a spike in certain types of antisocial behavior such as rule-breaking and noncompliance with authority. For most individuals, these behaviors decline in late adolescence and adulthood (e.g., Barker et al., 2007). As such, endorsing an item measuring one of these behaviors would tend to index a greater degree of underlying severity in late adolescence than the same behavior endorsed in early adolescence. Similarly, the attention deficit versus hyperactivity/impulsivity symptoms of ADHD have previously shown evidence of differential developmental trajectories. The former has shown a potential curvilinear trajectory while the latter show a more definite and steady decline over development (e.g., Murray, Obsuth, et al., 2016). Over the course of adolescence this could, for example, lead to increasing bifurcation of a previously unitary ADHD construct into separate inattention and hyperactivity/impulsivity constructs. More generally, items referring to symptoms or behaviors within the context of typical early adolescent social environments and activities (e.g., school) may be less relevant by late adolescence.

When items cannot be considered comparable across time, valid inferences about changes in the underlying trait rely on identifying and modeling the nature and extent of noncomparability. “Comparability” comes in different degrees and types that can be modeled and tested statistically. A useful framework for doing so is that of longitudinal measurement invariance within confirmatory factor analysis (CFA; e.g., see Millsap & Cham, 2012). Longitudinal measurement invariance is when expected observed score distributions given trait levels are independent of the wave at which the measure was administered. Longitudinal invariance using CFA tests a slightly weaker version of this, namely, that the expected mean and variance of observed score distributions given latent trait levels are independent of measurement wave. In this framework, a measure may show (from weakest to strongest) no invariance, configural invariance, metric invariance, scalar invariance, or residual invariance.

Configural invariance means that only the pattern of factor loadings for a measure is the same across time. Metric invariance is where both factor loading patterns and factor loading magnitudes are equal across time. When metric invariance holds, comparisons of factor variances and covariances are supported, for example, in cross-lagged and autoregressive panel models. When metric invariance does not hold, an inventory may not be measuring the same construct across time. Scalar invariance is when factor patterns, loadings, and intercepts are equal across time. When scalar invariance holds, inferences about factor mean differences over time are supported. This is relevant when, for example, fitting second-order growth curve models. Finally, residual (or strict) invariance is when factor patterns, loadings, intercepts, and item residual variances are equal over time. When this holds, differences in means and variances of the observed scores can be attributed to differences on the underlying latent factors. Only in this case are inferences about stability and change in a trait over time based on observed scores (e.g., sum scores) supported. Where any level of the above-described invariance assu mptions are not met, it is often possible to nonetheless obtain valid comparisons of latent constructs across time. To do so requires that at least some items are longitudinally invariant, giving “partial invariance.” Partial invariance often suffices provided that the noninvariance in the remaining items is explicitly modeled (e.g., Edwards & Wirth, 2012).

Surprisingly few studies of adolescent development report longitudinal invariance analyses either in their own right or to support other longitudinal analyses. Some studies have suggested that high levels of invariance across adolescence can be achieved for specific measures of mental health dimensions, such as ADHD, depression, anxiety, and conduct problems (e.g., Leopold et al., 2016; Motl, Dishman, Birnbaum, & Lytle, 2005; Sterba et al., 2010; Verhoeven, Sawyer, & Spence, 2013) while others have identified noninvariance (Mathyssek et al., 2013; Sterba et al., 2010). In this study, we sought to build on this limited evidence base and evaluate longitudinal invariance in a brief measure of five dimensions of psychosocial functioning across development: the Social Behavior Questionnaire (SBQ; Tremblay et al., 1991).

The SBQ shares origins with the Strengths and Difficulties Questionnaire (Goodman, 1997). Its first reported use was in a study by Tremblay et al. (1991) where two preexisting scales were combined: 28 items from the Preschool Behavior Questionnaire (Behar & Stringfield, 1974), itself an adaptation of the Children’s Behavior Questionnaire (Rutter, 1967), and 10 items from the Prosocial Behavior Questionnaire (Weir & Duveen, 1981). Though originally developed for young children, it has since been applied in older children and adolescent samples. It has been adopted in several large-scale studies internationally, where variants have been administered in English, French, and German. For example, items of the SBQ were integrated in the behavior evaluations in the Canadian National Longitudinal Survey of Children and Youth (Sprott, Jenkins, & Doob, 2000), the Nuremberg-Erlangen Prevention and Development Study (e.g., Lösel & Stemmler, 2012), the Zurich Project on the Social Development of Children and Youths (z-proso; Eisner & Ribeaud, 2007), and the Quebec Longitudinal Study of Kindergarten Children (QLSKC; Rouquette et al., 2014). It is used in self, parent, and teacher report form.

In spite of the large number of published studies using the SBQ items, there have been only a small number of previous dedicated studies of its psychometric properties. The few that have been conducted have generally supported the factorial validity, criterion validity, and reliability of the SBQ items (e.g., Murray, Eisner, & Ribeaud, 2017; Murray, Eisner, & Ribeaud, 2016a; Tremblay et al., 1991; Tremblay, Vitaro, Gagnon, Piché, & Royer, 1992). Murray et al. (2017), for example, analyzed the ranges of measurement for which the teacher-reported SBQ items reliably captured the dimensions of prosociality, externalizing, ADHD, and internalizing in z-proso at ages 7 through to 15. The latter three traits were operationalized using bifactor measurement models where the specific dimensions of ADHD were inattention and hyperactivity/impulsivity, the specific dimensions of internalizing were anxiety and depression, and the specific dimensions of externalizing were physical aggression, oppositional defiant disorder/conduct disorder, and reactive aggression. They found that SBQ items could reliably capture a wide range of trait levels for the general factors across ages 7 to 15 years. Murray, Obsuth, et al. (2017) reported a factor analysis of all self-reported SBQ items together with the newly developed Violent Ideations Scale (Murray et al., 2016a) in the z-proso sample when the participants were aged 17 years. All items from the SBQ loaded on the intended factors (prosociality, internalizing, ADHD, reactive aggression, and proactive aggression). In contrast to the SBQ reliability study by Murray et al. (2017), this study did not support a distinction between anxiety and depression; however, Murray, Obsuth, et al. (2017) cited concerns about overfactoring, and it is possible that extracting further factors would have yielded separate anxiety and depression factors. Arguably, understanding the longitudinal measurement properties of an instrument is an important step in its validation and in construct validation more broadly (e.g., Edwards & Wirth, 2009); however, this has not been tested for any version of the SBQ across any developmental period. In this study we, therefore, evaluate longitudinal measurement invariance of the SBQ in the z-proso sample.

Method

Participants

Data came from the z-proso study. This is a longitudinal cohort study of youth, focusing on the development of prosocial and antisocial behaviors. The study began in 2004 when the participants were aged 7 years and entering school. Participants were selected according to a school-level stratified random sampling procedure that took into account school size and location. All children entering the first grade at selected schools in that year were invited to participate. The study included separate child and parent intervention components in the early waves; however, as there was little evidence that they had any substantive short- or long-term effects, it is usually judged reasonable to treat the data as observational (e.g., Averdijk, Zirk-Sadowski, Ribeaud, & Eisner, 2016; Malti, Ribeaud, & Eisner, 2011). Comprehensive accounts of recruitment, participant characteristics, attrition, and assessment procedures can be found in previous publications (e.g., Eisner & Ribeaud, 2007) and on the z-proso website (http://www.cru.ethz.ch/en/projects/z-proso.html). The current study focusses on the latter four measurement waves when the majority of participants were aged 11, 13, 15, and 17 years, respectively. Across these waves, 1,523 (51% male) participants contributed data, representing 91% of the original target sample.

Measures: Prosociality, Anxiety, Depression, ADHD, and Aggression

Prosociality, anxiety, depression, ADHD, and Aggression were measured with the SBQ. Z-proso includes some adaptations of the SBQ, mainly additional aggression items reflecting the fact that a major research theme of the study is antisocial behavior development. In addition, whereas the original SBQ was on a 3-point response scale, the version administered in z-proso offers respondents a 5-point scale. In our longitudinal invariance analyses, we focus on items that were common across all measurement waves to simplify the interpretation of any noninvariance identified. The majority of these items were part of the original SBQ reported in Tremblay et al. (1991). One anxiety item, two depression items, and two aggression items were added in z-proso to maintain developmental appropriateness in the adolescent period. Overall, the SBQ administered in z-proso included eight items measuring prosociality that were common across all measurement waves (age 11, 13, 15, and 17), four measuring depression, and four measuring anxiety. In addition, four items measuring ADHD and eight measuring aggression (four each for the subtypes of proactive and reactive aggression) were administered at the latter three measurement waves (age 13, 15, and 17) only. Thus, we analyze a slightly different set of items as compared to the original SBQ items developed for preschoolers. Abbreviated item contents are provided in the Results tables. All items are measured on a 5-point Likert-type scale from never to very often. Apart from the anxiety and depression items, which referred to the frequency of behavior/symptoms in the past month, all items referred to frequency of behavior/symptoms in the past year. All were administered in paper-and-pencil format in German: the official language of the study location.

Statistical Procedure: Longitudinal Factorial Invariance

Longitudinal factorial invariance was assessed within a CFA framework. We treated items as continuous because it is usually reasonable to treat ordered-categorical items with at least five response options as continuous (e.g., Rhemtulla, Brosseau-Liard, & Savalei, 2012). Benefits of doing so include better accounting for missingness (because full information maximum likelihood estimation can be employed), a larger simulation evidence base to draw on to guide model selection based on model fits (e.g., Chen, 2007; Cheung & Rensvold, 2002; Meade, Johnson, & Braddy, 2008), and the availability of information theoretic criteria to further guide model selection (e.g., Raftery, 1995). Furthermore, Sass, Schmitt, and Marsh (2014) presented simulation study results suggesting that fit comparisons for invariance testing using weighted least squares means and variances estimation do not perform well.

Comparisons across a series of increasingly constrained models provided information on the level of invariance that could be achieved for each trait. We report χ² difference tests for information but note that these are likely to be overly sensitive to minor misspecifications given our large sample size (e.g., Meade et al., 2008). As such, for model selection, we relied primarily on the approximate fit indexes of comparative fit index (CFI), Tucker–Lewis index (TLI), and root mean square error of approximation (RMSEA), which are less influenced by sample size and model complexity than the χ² difference test. In particular, we use the cutoff criteria defined by Chen (2007), developed using a simulation study of different types of invariance at different sample sizes. According to these criteria, metric invariance would hold in our sample when the CFI decreased by less than .010, RMSEA increased by less than .015, and standardized root mean square residual (SRMR) increased by less than .030 with the addition of metric invariance constraints. Scalar invariance would hold if CFI decreased by less than .010, RMSEA increased by less than .015, and SRMR increased by less than .010 with the addition of scalar invariance constraints. Residual (or strict) invariance would hold if CFI decreased by less than .010, RMSEA increased by less than .015, and SRMR increased by less than .010 with the addition of scalar invariance constraints. Noting that these criteria did not have power to detect minor violations of invariance, Meade et al. (2008) suggest an increase in CFI of >.002 should be used to detect noninvariance; however, given the large number of comparisons to be conducted overall, we elected to use Chen’s (2007) more conservative criteria to avoid the detection of a large number of trivially small violations of invariance.

To test configural, metric, scalar, and residual invariance, the following series of increasingly strict models were fit to test invariance. First, in the configural model, patterns of loadings were fixed equal over time but loadings and intercepts were free to vary. Scaling and identification were achieved by fixing the mean and variance of the latent factor at baseline to 0 and 1, respectively, and by fixing the loading and threshold of a reference item to equality over all time points. This method assumes that the reference item is invariant over time. Choosing a reference item that is not invariant can lead to the appearance of noninvariance in truly invariant items and/or to the appearance of invariance in truly noninvariant items (e.g., Yoon & Millsap, 2007). However, there was neither prior empirical evidence nor strong theoretical rationale on which to base the selection of reference variable for the SBQ. As such, we provisionally selected the first item in each subscale to be the reference variable and then checked that the invariance constraints with these items were not associated with large modification indices at either the metric or scalar invariance model stages. If they were, we allowed these constraints to be freed and instead thereafter relied on other items to act as reference variables. Residual covariances between the same items measured over time were also estimated.

Configural invariance was judged to hold when the configural model had RMSEA < .08, SRMR < .08, and TLI and CFI > .95 (e.g., Hu & Bentler, 1999; Schermelleh-Engel, Moosbrugger, & Muller, 2003). In the metric model, equality constraints over time were added on the remaining first-order loadings. In the scalar model, equality constraints over time were added on the remaining intercepts. In the strict invariance model, residual variances were fixed equal over time. If at any stage invariance was not supported, modification indices and expected parameter changes were used to identify specific constraints that did not hold. These were iteratively released, retesting invariance with the release of each individual constraint until either partial invariance held or there were only two items left with invariance constraints at that level. In this latter case, invariance was judged not to hold at that level. No further constraints were added to items that did not show invariance at a lower level, for example, scalar constraints were not placed on items that did not show metric invariance.

Finally, the practical consequences of any noninvariance identified were investigated. To do this, we compared two estimates. The first was a linear slope factor mean from a growth curve models fit using a model that assumed full invariance (all loadings, intercepts, and residual variance assumed equal over time). The second was the same parameter from a model that assumes the highest level of invariance that was actually attained. The difference provides a quantification of the bias that could be expected to result from the noninvariance, if not appropriately modeled.

Results

Overview

Invariance was not supported at any level according to the χ² difference test; however, as discussed above, this test is likely not appropriate for our large sample size as it will identify even trivially small misspecifications as statistically significant. Partially invariant models were achieved in all cases according to the fit criteria of Chen (2007). The practical significance of the noninvariance identified was overall small: Unstandardized slope estimates from partially invariant versus (falsely assumed) fully invariant models generally differed only at the second decimal place. The one exception was anxiety, for which slopes were overestimated by more than 50% when longitudinal invariance was incorrectly assumed.

Prosociality

Fits for the prosociality models are provided in Table 1. The configural model was a single-factor model. Both configural and metric invariance held, but scalar invariance did not. Iterative release of equality constraints on intercepts in the following sequence resulted in a partially invariant model (M2a in Table 2): Item 48 at age 11, Item 47 at age 17, Item 41 at age 17, and Item 48 at age 13. Adding residual invariance constraints, fit statistics suggested further noninvariance. Based on modification indices and expected parameter changes, constraints on the item residual variances for Item 47 at age 11 and then Item 41 at age 11 were released. At this point, partial residual invariance was judged to hold. Parameters for this final model (M3a) are provided in Table 2. These show that for Item 48, the intercept increased over time. In addition, the residual variances for Items 41 and 47 were larger at age 11 than at subsequent measurement points. The linear slope factor mean from a latent growth curve model fit using the final measurement model developed in these analyses was −0.290. The model assuming full measurement invariance across development yielded a corresponding estimate of −0.209.

Table 1.

Model Fits for Prosociality Invariance Models.

Model	Model description	χ²	df	p	CFI	TLI	RMSEA	SRMR	AIC	BIC
Model fits
M0	Configural	885.052	410	<.001	.965	.958	.028	.033	106087.155	106886.421
M1	Metric	926.658	431	<.001	.963	.958	.027	.036	106086.761	106774.129
M2	Scalar	1556.853	452	<.001	.918	.910	.040	.043	106674.956	107250.428
M2a	Scalar partial	1089.311	448	<.001	.953	.948	.031	.037	106215.415	106812.200
M3	Strict (partial)	1255.434	468	<.001	.942	.938	.033	.054	106341.538	106831.754
M3a	Strict partial	1168.715	466	<.001	.948	.945	.031	.047	106258.818	106759.691
Model fit differences
M1-M0	Metric-configural	41.606	21	.005	−.002	0	−.001	.003	−0.394	−112.292
M2-M1	Scalar-metric	630.195	21	<.001	−.045	−.048	.013	.007	588.195	476.299
M2a-M1	Scalar partial-metric	162.653	17	<.001	−.01	−.01	.004	.001	128.654	38.071
M3-M2a	Strict (partial)-scalar partial	166.123	20	<.001	−.011	−.01	.002	.017	126.123	19.554
M3a-M2a	Strict partial-scalar partial	79.404	18	<.001	−.005	−.003	0	.01	43.403	−52.509

Note. df = degrees of freedom; CFI = comparative fit index; TLI = Tucker–Lewis index; RMSEA = root mean square error of approximation; SRMR = standardized root mean square residual; AIC = Akaike information criterion; BIC = Bayesian information criterion. Final model indicated in boldface.

Table 2.

Parameters for Most Invariant Prosociality Model.

Item number and content	Loadings				Intercepts				Residual variances
Item number and content	Age 11	Age 13	Age 15	Age 17	Age 11	Age 13	Age 15	Age 17	Age 11	Age 13	Age 15	Age 17
41: Help clear up	0.423	0.423	0.423	0.423	3.335	3.335	3.335	3.558	0.987	0.793	0.793	0.702
48: Understand feelings	0.515	0.515	0.515	0.515	3.621	3.917	4.143	4.143	0.888	0.741	0.563	0.563
49: Share	0.445	0.445	0.445	0.445	3.855	3.855	3.855	3.855	0.696	0.696	0.696	0.696
42: Settle dispute	0.627	0.627	0.627	0.627	3.413	3.413	3.413	3.413	0.796	0.796	0.796	0.796
40: Sympathy	0.719	0.719	0.719	0.719	4.016	4.016	4.016	4.016	0.487	0.487	0.487	0.487
43: Help injured	0.696	0.696	0.696	0.696	4.042	4.042	4.042	4.042	0.470	0.470	0.470	0.470
46: Comfort	0.841	0.841	0.841	0.841	4.032	4.032	4.032	4.032	0.415	0.415	0.415	0.415
47: Listen to others	0.472	0.472	0.472	0.472	3.615	3.615	3.615	3.930	1.211	0.800	0.800	0.649
Factor means	Ages 11 = 0, 13 = −0.368, 15 = −0.336, 17 = −0.242
Factor variances	Ages 11 = 1, 13 = 1.086, 15 = 0.923, 17 = 0.919

Note. Noninvariant parameters are indicated in boldface.

Anxiety

Fits for the anxiety models are provided in Table 3. The configural model was a single-factor model. Both configural and metric invariance held. Scalar invariance did not hold, but partial invariance was achieved with the iterative release of scalar constraints on Item 1 measured at age 11 and then the same item measured at age 13. No further noninvariance was identified with the addition of residual invariance constraints on the scalar invariant items. Partial residual invariance was, therefore, judged to hold. Parameters for this model are provided in Table 4. These show that the intercept for Item 4 increased across ages 11, 13, and 15. The linear slope factor mean from a latent growth curve model fit using the final measurement model developed in these analyses was 0.257. The model assuming full measurement invariance across development yielded a corresponding estimate of 0.544.

Table 3.

Model Fits for Anxiety Invariance Models.

Model	Model description	χ²	df	p	CFI	TLI	RMSEA	SRMR	AIC	BIC
Model fits
M0	Configural	212.222	74	<.001	.979	.966	.035	.029	57861.782	58277.4
M1	Metric	247.734	83	<.001	.975	.963	.036	.034	57879.294	58246.956
M2	Scalar	541.796	92	<.001	.931	.91	.057	.057	58155.355	58475.061
M2a	Scalar partial	278.714	90	<.001	.971	.961	.037	.036	57896.273	58226.636
M3	Strict (partial)	306.053	100	<.001	.968	.962	.037	.044	57903.613	58180.692
Model fit differences
M1-M0	Metric-configural	35.512	9	<.001	−.004	−.003	.001	.005	17.512	−30.444
M2-M1	Scalar-metric	148.807	9	<.001	−.044	−.053	.021	.023	276.061	228.105
M2a-M1	Scalar partial-metric	30.98	7	<.001	−.004	−.002	.001	.002	16.979	−20.32
M3-M2a	Strict (partial)-scalar partial	27.339	10	.002	−.003	.001	0	.008	7.34	−45.944

Table 4.

Parameters for Most Invariant Anxiety Model.

Item number and content	Loadings				Intercepts				Residual variances
Item number and content	Age 11	Age 13	Age 15	Age 17	Age 11	Age 13	Age 15	Age 17	Age 11	Age 13	Age 15	Age 17
01: Crying	0.662	0.662	0.662	0.662	2.034	2.034	2.034	2.034	0.573	0.573	0.573	0.573
03: Fear	0.574	0.574	0.574	0.574	1.637	1.637	1.637	1.637	0.478	0.478	0.478	0.478
04: Worry	0.668	0.668	0.668	0.668	2.070	2.322	2.637	2.637	0.748	0.793	0.815	0.815
62: Sleepless	0.504	0.504	0.504	0.504	2.463	2.463	2.463	2.463	1.274	1.274	1.274	1.274
Factor means	Ages 11 = 0, 13 = 0.052, 15 = 0.162, 17 = 0.276
Factor variances	Ages 11 = 1, 13 = 1.207, 15 = 1.494, 17 = 1.699

Note. Noninvariant parameters are indicated in boldface.

Depression

Fits for the depression models are provided in Table 5. The configural model was a single-factor model and showed good fit. Both configural and metric invariance held, but scalar invariance did not. The scalar invariance constraint on Item 63 at age 13 was removed to achieve partial scalar invariance. Partial residual invariance was achieved with the addition of residual invariance constraints to all remaining items. The parameters from this model are provided in Table 6. Item 4 had a larger intercept at age 13. The linear slope factor mean from a latent growth curve model fit using the final measurement model developed in these analyses was 0.716. The model assuming full measurement invariance across development yielded a corresponding estimate of 0.725.

Table 5.

Model Fits for Depression Invariance Models.

Model	Model description	χ²	df	p	CFI	TLI	RMSEA	SRMR	AIC	BIC
Model fits
M0	Configural	109.904	74	.0043	.994	.991	.018	.022	56587.043	57002.661
M1	Metric	139.798	83	<.001	.991	.987	.021	.027	56598.937	56966.599
M2	Scalar	276.202	92	<.001	.971	.962	.036	.035	56717.341	57037.047
M2a	Scalar partial	197.280	91	<.001	.983	.978	.028	.029	56640.419	56965.454
M3	Strict (partial)	245.175	102	<.001	.977	.974	.030	.035	56666.314	56932.736
Model fit differences
M1-M0	Metric-configural	29.894	9	<.001	−.003	−.004	.003	.005	11.894	−36.062
M2-M1	Scalar-metric	136.404	9	<.001	−.02	−.025	.015	.008	118.404	70.448
M2a-M1	Scalar partial-metric	57.482	8	<.001	−.008	−.009	.007	.002	41.482	−1.145
M3-M2a	Strict (partial)-scalar partial	47.895	11	<.001	−.006	−.004	.002	.006	25.895	−32.718

Table 6.

Parameters for Most Invariant Depression Model.

Item number and content	Loadings				Intercepts				Residual variances
Item number and content	Age 11	Age 13	Age 15	Age 17	Age 11	Age 13	Age 15	Age 17	Age 11	Age 13	Age 15	Age 17
05: Sad no reason	0.595	0.595	0.595	0.595	1.731	1.731	1.731	1.731	0.829	0.829	0.829	0.829
08: Unhappy	0.68	0.68	0.68	0.68	2.051	2.051	2.051	2.051	0.516	0.516	0.516	0.516
63: Bored	0.269	0.269	0.269	0.269	2.634	2.870	2.634	2.634	0.833	0.867	0.833	0.833
64: Feel alone	0.721	0.721	0.721	0.721	1.704	1.704	1.704	1.704	0.468	0.468	0.468	0.468
Factor means	Ages 11 = 0, 13 = 0.223, 15 = 0.575, 17 = 0.728
Factor variances	Ages 11 = 1, 13 = 1.337, 15 = 1.657, 17 = 1.827

Note. Noninvariant parameter is indicated in boldface.

Attention Deficit Hyperactivity Disorder

Fits for the ADHD models are provided in Table 7. The configural model for ADHD was a single-factor model. Configural, metric, scalar, and residual invariance all held. Parameter estimates from the residual invariance model are provided in Table 8. The slope factor mean for a linear growth curve model using this fully invariant ADHD model was 0.114.

Table 7.

Model Fits for Attention Deficit Hyperactivity Disorder Invariance Models.

Model	Model description	χ²	df	p	CFI	TLI	RMSEA	SRMR	AIC	BIC
Model fits
M0	Configural	187.202	39	<.001	.969	.948	.051	.029	43308.221	43578.614
M1	Metric	215.756	45	<.001	.964	.948	.051	.036	43324.775	43563.357
M2	Scalar	251.771	51	<.001	.958	.946	.052	.037	43348.790	43555.561
M3	Strict	302.348	59	<.001	.949	.943	.053	.045	43383.367	43547.724
Model fit differences
M1-M0	Metric-configural	28.554	6	<.001	−.005	0	0	.007	16.554	−15.257
M2-M1	Scalar-metric	36.015	6	<.001	−.006	−.002	.001	.001	24.015	−7.796
M3-M2	Strict-scalar	50.577	8	<.001	−.009	−.003	.001	.008	34.577	−7.837

Table 8.

Parameters for Most Invariant Attention Deficit Hyperactivity Disorder Model.

Item number and content	Loadings			Intercepts			Residual variances
Item number and content	Age 13	Age 15	Age 17	Age 13	Age 15	Age 17	Age 13	Age 15	Age 17
12: Restless	0.729	0.729	0.729	2.761	2.761	2.761	0.582	0.582	0.582
13: Concentration	0.726	0.726	0.726	2.348	2.348	2.348	0.512	0.512	0.512
16: Inattention	0.652	0.652	0.652	2.684	2.684	2.684	0.67	0.67	0.67
17: Hectic/fidgety	0.539	0.539	0.539	2.741	2.741	2.741	0.739	0.739	0.739
Factor means	Ages 13 = 0, 15 = 0.105, 17 = 0.304
Factor variances	Ages 13 = 1, 15 = 1.046, 17 = 1.126

Aggression

Fits for the aggression models are provided in Table 9. The configural model was a first-order oblique model with first-order factors: proactive aggression and reactive aggression. The configural model showed reasonable fit, and configural invariance was judged to hold. Metric but not scalar invariance held. Release of the intercept constraint on Item 61 at age 17 was necessary to achieve partial scalar invariance. To achieve partial residual invariance, it was necessary to release the constraints on the residual variance of Item 37 at age 17 and then on Item 72 at age 13. Parameter estimates for this model are provided in Table 10. These show that the intercept for Item 61 was lower at age 17 while the residual variance for Items 37 and 72 was larger at earlier waves. The linear slope factor mean from a latent growth curve model fit using the final measurement model developed for reactive aggression was −0.041 and for proactive aggression was −0.210. The models assuming full measurement invariance across development for these constructs yielded corresponding estimates of −0.084 for reactive aggression and −0.202 for proactive aggression.

Table 9.

Model Fits for Aggression Invariance Models.

Model	Model description	χ²	df	p	CFI	TLI	RMSEA	SRMR	AIC	BIC
Model fits
M0	Configural	1161.785	213	<.001	.919	.895	.055	.054	70507.736	71096.239
M1	Metric	1189.546	225	<.001	.917	.899	.054	.055	70511.498	71036.378
M2	Scalar	1400.582	237	<.001	.900	.884	.058	.056	70698.534	71159.792
M2a	Scalar partial	1273.555	236	<.001	.911	.896	.054	.055	70573.506	71040.066
M3	Strict (partial)	1464.645	251	<.001	.896	.886	.057	.060	70734.597	71121.630
M3a	Strict partial	1382.150	249	<.001	.903	.892	.055	.057	70656.101	71053.738
Model fit differences
M1-M0	Metric-configural	27.761	12	.006	−.002	.004	−.001	.001	3.762	−59.861
M2-M1	Scalar-metric	211.036	12	<.001	−.017	−.015	.004	.001	187.036	123.414
M2a-M1	Scalar partial-metric	84.009	11	<.001	−.006	−.003	0	0	62.008	3.688
M3-M2a	Strict (partial)-scalar partial	191.09	15	<.001	−.015	−.01	.003	.005	161.091	81.564
M3a-M2a	Strict partial-scalar partial	108.595	13	<.001	−.008	−.004	.001	.002	82.595	13.672

Table 10.

Parameters for Most Invariant Aggression Model.

Item number and content	Loadings			Intercepts			Residual variances
Item number and content	Age 13	Age 15	Age 17	Age 13	Age 15	Age 17	Age 13	Age 15	Age 17
Proactive aggression
37: Threaten	0.368	0.368	0.368	1.235	1.235	1.235	0.230	0.230	0.156
51: Bossing	0.503	0.503	0.503	1.740	1.740	1.740	0.500	0.500	0.500
52: Forcing	0.688	0.688	0.688	1.579	1.579	1.579	0.295	0.295	0.295
61: Humiliate	0.632	0.632	0.632	1.891	1.891	1.621	0.551	0.551	0.373
Reactive aggression
53: When teased	0.407	0.407	0.407	3.000	3.000	3.000	1.033	1.033	1.033
54: When something taken	0.657	0.657	0.657	1.620	1.620	1.620	0.295	0.295	0.295
55: When not getting something	0.374	0.374	0.374	1.797	1.797	1.797	0.630	0.630	0.630
72: When insulted	0.766	0.766	0.766	1.705	1.705	1.705	0.335	0.203	0.203
Factor means	Proactive aggression: Ages 13 = 0, 15= −0.05, 17 = −0.104
Factor means	Reactive aggression: Ages 13 = 0, 15= −0.213, 17 = −0.445
Factor variances	Proactive aggression: Ages 13 = 1, 15= 0.937, 17 = 0.742
Factor variances	Reactive aggression: Ages 13 = 1, 15 = 0.832, 17 = 0.742.

Note. Noninvariant parameters are indicated in boldface.

Discussion

In the current study, we examined the important but seldom-tested assumption of longitudinal factorial invariance for five dimensions of adolescent psychosocial functioning measured by the SBQ. We found that items largely functioned equivalently across waves, supporting their use in longitudinal analyses. Where noninvariance was identified, the most consistent pattern was larger residual variances at earlier time points. This suggests that the adolescents may have become better able to report on the presence of certain behaviors or symptoms over time. The potential bias arising from falsely assuming invariance with these data (e.g., by using sum scores) is not likely to be substantial, except for developmental analyses involving anxiety.

Longitudinal invariance analyses are rarely reported in studies of adolescent development. This is in spite of the fact that the majority of conclusions drawn about psychosocial development from longitudinal data rely on the assumption that at least metric invariance holds. Furthermore, the selection of measures for longitudinal studies seldom explicitly considers the degree of comparability of measures over the relevant phases of development as a criterion. In this study, we use this evaluated longitudinal invariance for five dimensions of adolescent mental health measured by the SBQ. All subscales showed at least metric invariance which is consistent with the idea that they measure the same constructs over adolescence. This makes the SBQ a good candidate for use as a brief omnibus measure of adolescent psychosocial development.

Scalar invariance was violated for some items across the SBQ subscales. There was a general tendency here for item intercepts to increase over measurement waves. As such, for a given latent trait level, expected item scores were higher at later time points. The implication of this is that a researcher using observed scores (e.g., sum score for a subscale) would see an artifactual increase (or an attenuated decrease) in levels of the traits measured with these items over adolescence. However, as so few items were affected, it is likely that unbiased comparisons of levels over time could be made using a partial measurement invariance model in which the scalar invariance constraints that did not hold were not imposed (Byrne, Shavelson, & Muthén, 1989; see also Ferrer, Balluerka, & Widaman, 2008). Comparisons of latent growth curve models from models appropriately modeling noninvariance versus models that assumed full invariance revealed substantial discrepancies only for anxiety. For anxiety, the positive slope over time was estimated as 52% larger when assuming invariance because the model attributed an increase in an item intercept to an increase in factor means across time.

A possible explanation for the trend toward increasing intercepts over time is that adolescents become more attuned to the behaviors and symptoms about which the items ask over time. This could be a function of the repeated administration of the questionnaires whereby participants are cumulatively primed to detect certain symptoms and behaviors. It could also be a function of increasing capacity to identify and report on these same symptoms and behaviors arising from maturity. An alternative explanation would be that participants become more comfortable disclosing information about their negative behaviors and symptoms over time as they build trust with the study over measurement waves; however, the fact that the same trend was seen in prosociality—a trait with positive connotations—calls this interpretation into question. The one item that deviated from this pattern was an item measuring tendency to experience boredom, an indicator of depression. This item had a larger intercept at age 13 than at other ages. One possibility is that this reflects a normative elevation of boredom specifically around this age (Spaeth, Weichold, & Silbereisen, 2015).

There were also some violations of residual invariance. In general, residual variances were larger at age 11, suggesting that measurement error decreased over time. This is likely to reflect increased familiarity with the questionnaire on repeated administrations and an increase in the reliability of self-reports that comes with maturity. However, the fact that these violations were few and generally of a small magnitude suggests that by age 11 self-reports are not substantially less reliable than those at older ages.

There are implications of these findings both for users of the z-proso data set specifically and researchers of adolescent development more generally. Users of the data set should ideally adopt the measurement models for the anxiety, depression, prosociality, and aggression outlined in the Results section, especially for anxiety. For certain models with a high degree of computational complexity, such as those involving a large number of latent interactions, using factor scores estimated from the measurement models would represent a more practical solution. In these cases, the researcher should confirm that the determinacy of the factor scores is adequate, for example, >.90 (Gorsuch, 1983). For ADHD, which showed full invariance over development, using sum scores over latent variable measurement models would not be expected to introduce substantial bias. However, there would nonetheless be other benefits to using a latent variable measurement model, such as greater power due to the disattenuation for unreliability and the ability to test model fit and identify misspecifications.

While these results provide validity support for the SBQ in showing that most items remain comparable across development, users of the SBQ in other studies should independently investigate invariance and develop appropriate measurement models to take account of any violations identified. There are no guarantees that invariance results would generalize across studies because the methodological differences between studies using the SBQ and the different languages and cultural contexts in which it is administered could influence longitudinal invariance. For example, the SBQ is administered on a 3- rather than 5-point response format in some studies. Comparisons of invariance results across different studies using the SBQ would help home in on the features generating the noninvariance.

More generally, our study illustrates that in spite of the myriad changes occurring across adolescence, it is possible to construct measures that capture dimensions such as anxiety, depression, ADHD, aggression, and prosociality in a comparable way across development. This is critical to robust research into developmental trends in these dimensions. Any study seeking to make inferences about changes in variances, covariances, and means (or statistics derived from these) should follow a procedure similar to that outlined in our Method and Results section, in order to build appropriate measurement models to account for noninvariance prior to testing their substantive hypotheses. In some cases, it may be necessary to simultaneously investigate invariance with respect to other variables. For example, tests of differential subgroup developmental trajectories (e.g., sex differences in development) would call for investigations of invariance by both subgroup and measurement wave.

The fact that the SBQ showed high levels of invariance over time likely reflects the fact that the wording of items is relatively general and not tied to specific contexts (e.g., school) or activities (e.g., schoolwork). When planning longitudinal studies, including a core set of “context-free” or “developmentally neutral” items that could be expected to show invariance over development may be crucial to ultimately obtaining comparable trait estimates over longer time spans. This is not to discourage the inclusion of items that are specific to some developmental periods; omitting these may result in crucial manifestations of traits being missed. For these items to be included in studies of change over time, however, they need to be anchored via developmentally invariant items (e.g., Edwards & Wirth, 2009). The SBQ items may, for example, serve as useful anchors when developing new measures for longitudinal studies or administered alongside more comprehensive measures of the dimensions it measures.

Finally, it is important to consider the limitations of the approach of the current article. First, factorial invariance does not guarantee that a measure captures the same psychological phenomenon over time; it is possible for metric invariance to hold, for example, when the psychological process underlying responses is changing (e.g., Widaman, Little, Geary, & Cormier, 1992). Second, we selected fit criteria for detecting invariance to protect against making Type 1 errors. However, this also increases the likelihood of missing minor violations of invariance. Third, we focused on items that were identical over multiple waves. As such, the measures may not have been representative of the traits during particular time periods. Fourth, although overall tests of invariance at the scale level tend to be relatively robust, this is not necessarily true at the item level where, for example, iterative release of constraints based on modification indices or the selection of a noninvariant reference variable can result in misidentifying the location of noninvariance (e.g., Johnson, Meade, & DuVernet, 2009). As such, our item-level inferences should be regarded with more caution. Finally, we did not have any a priori hypotheses regarding which items should be noninvariant and in what way (e.g., whether intercepts should increase over development). As such, our interpretations of the noninvariance were entirely post hoc. Future studies may benefit from deriving specific a priori hypotheses about noninvariance across time from developmental theories of the traits being analyzed.

Conclusions

Although adolescence is a time of considerable change, it is possible to identify items that show comparable measurement properties across this period: a prerequisite for supporting valid inferences about development. Where measurement invariance was violated, it tended to be the case that residual variances were larger at earlier waves while intercepts were larger at later waves. In the SBQ, only the anxiety subscale showed practically significant levels of noninvariance.

Footnotes

Acknowledgements

We are grateful to the children, parents, and teachers who provided data for the z-proso study and the research assistants involved in its collection.

Authors’ Note

Ingrid Obsuth is now affiliated to University of Edinburgh, Edinburgh, UK and Denis Ribeaud is now affiliated to University of Zurich, Zurich, Switzerland.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research is supported by the Jacobs Foundation (Grant 2010-888) and the Swiss National Science Foundation (Grants 100013_116829 & 100014_132124) is also gratefully acknowledged.

References

Averdijk

Zirk-Sadowski

Ribeaud

Eisner

(2016). Long-term effects of two childhood psychosocial interventions on adolescent delinquency, substance use, and antisocial behavior: A cluster randomized controlled trial. Journal of Experimental Criminology, 12, 21-47.

Barker

E. D.

Séguin

J. R.

White

H. R.

Bates

M. E.

Lacourse

Carbonneau

Tremblay

R. E.

(2007). Developmental trajectories of male physical violence and theft: Relations to neurocognitive performance. Archives of General Psychiatry, 64, 592-599.

Behar

Stringfield

(1974). Manual for the Preschool Behavior Questionnaire. Durham, NC: Lenore Behar.

Byrne

B. M.

Shavelson

R. J.

Muthén

(1989). Testing for the equivalence of factor covariance and mean structures: The issue of partial measurement invariance. Psychological Bulletin, 105, 456-466.

Carlo

Crockett

L. J.

Randall

B. A.

Roesch

S. C.

(2007). A latent growth curve analysis of prosocial behavior among rural adolescents. Journal of Research on Adolescence, 17, 301-324.

Chen

F. F.

(2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling, 14, 464-504.

Cheung

G. W.

Rensvold

R. B.

(2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling, 9, 233-255.

Crocetti

Klimstra

Keijsers

Hale

W. W.

III Meeus

(2009). Anxiety trajectories and identity development in adolescence: A five-wave longitudinal study. Journal of Youth and Adolescence, 38, 839-849.

Dekker

M. C.

Ferdinand

R. F.

Van Lang

N. D.

Bongers

I. L.

Van Der Ende

Verhulst

F. C.

(2007). Developmental trajectories of depressive symptoms from early childhood to late adolescence: Gender differences and adult outcome. Journal of Child Psychology and Psychiatry, 48, 657-666.

10.

Edwards

M. C.

Wirth

R. J.

(2009). Measurement and the study of change. Research in Human Development, 6, 74-96.

11.

Edwards

M. C.

Wirth

R. J.

(2012). Valid measurement without factorial invariance: A longitudinal example. In Harring

J. R.

Hancock

G. R.

(Eds.), Advances in longitudinal methods in the social and behavioral sciences (pp. 289-311). Thousand Oaks, CA: Sage.

12.

Eisner

Ribeaud

(2007). Conducting a criminological survey in a culturally diverse context lessons from the Zurich Project on the Social Development of Children. European Journal of Criminology, 4, 271-298.

13.

Ferrer

Balluerka

Widaman

K. F.

(2008). Factorial invariance and the specification of second-order latent growth models. Methodology, 4, 22-36.

14.

Goodman

(1997). The Strengths and Difficulties Questionnaire: A research note. Journal of Child Psychology and Psychiatry, 38, 581-586.

15.

Gorsuch

R. L.

(1983). Factor analysis (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.

16.

L. T.

Bentler

P. M.

(1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55.

17.

Johnson

E. C.

Meade

A. W.

DuVernet

A. M.

(2009). The role of referent indicators in tests of measurement invariance. Structural Equation Modeling, 16, 642-657.

18.

Leopold

D. R.

Christopher

M. E.

Burns

G. L.

Becker

S. P.

Olson

R. K.

Willcutt

E. G.

(2016). Attention-deficit/hyperactivity disorder and sluggish cognitive tempo throughout childhood: Temporal invariance and stability from preschool through ninth grade. Journal of Child Psychology and Psychiatry, 57, 1066-1074.

19.

Lösel

Stemmler

(2012). Preventing child behavior problems in the Erlangen-Nuremberg Development and Prevention Study: Results from preschool to secondary school age. International Journal of Conflict and Violence, 6, 214-224.

20.

Luengo Kanacri

B. P.

Pastorelli

Eisenberg

Zuffianò

Caprara

G. V

. (2013). The development of prosociality from adolescence to early adulthood: The role of effortful control. Journal of Personality, 81, 302-312.

21.

Malti

Ribeaud

Eisner

M. P.

(2011). The effectiveness of two universal preventive interventions in reducing children’s externalizing behavior: A cluster randomized controlled trial. Journal of Clinical Child & Adolescent Psychology, 40, 677-692.

22.

Marmorstein

N. R.

(2009). Longitudinal associations between alcohol problems and depressive symptoms: Early adolescence through early adulthood. Alcoholism, 33, 49-59.

23.

Martino

S. C.

Ellickson

P. L.

Klein

D. J.

McCaffrey

Edelen

M. O.

(2008). Multiple trajectories of physical aggression among adolescent boys and girls. Aggressive Behavior, 34, 61-75.

24.

Mathyssek

C. M.

Olino

T. M.

Hartman

C. A.

Ormel

Verhulst

F. C.

Van Oort

F. V.

(2013). Does the Revised Child Anxiety and Depression Scale (RCADS) measure anxiety symptoms consistently across adolescence? The TRAILS study. International Journal of Methods in Psychiatric Research, 22, 27-35.

25.

Meade

A. W.

Johnson

E. C.

Braddy

P. W.

(2008). Power and sensitivity of alternative fit indices in tests of measurement invariance. Journal of Applied Psychology, 93, 568-592.

26.

Millsap

R. E.

Cham

(2012). Investigating factorial invariance in longitudinal data. In Laursen

Little

T. D.

Card

N. A.

(Eds.), Handbook of development research methods (pp. 265-278). New York, NY: Guildford.

27.

Motl

R. W.

Dishman

R. K.

Birnbaum

A. S.

Lytle

L. A.

(2005). Longitudinal invariance of the Center for Epidemiologic Studies-Depression Scale among girls and boys in middle school. Educational and Psychological Measurement, 65, 90-108.

28.

Murray

A. L.

Eisner

Ribeaud

(2016a). Development and validation of a brief measure of violent thoughts: The Violent Ideations Scale (VIS). Assessment. Advance online publication. doi:10.1177/1073191116667213

29.

Murray

A. L.

Eisner

Ribeaud

(2016b). The development of the general factor of psychopathology “p-factor” through childhood and adolescence. Journal of Abnormal Child Psychology, 44, 1573-1586.

30.

Murray

A. L.

Eisner

Ribeaud

(2017). Can the Social Behavior Questionnaire help meet the need for dimensional, transdiagnostic measures of childhood and adolescent psychopathology? European Journal of Psychological Assessment.

31.

Murray

A. L.

Obsuth

Eisner

Ribeaud

(2017). Identifying early markers of later onset ADHD symptoms. Journal of Attention Disorders. Advance online publication. 10.1177/1087054717705202

32.

Murray

A. L.

Obsuth

Zirk-Sadowski

Ribeaud

Eisner

(2016). Developmental relations between ADHD symptoms and reactive versus proactive aggression across childhood and adolescence. Journal of Attention Disorders. Advance online publication. doi:10.1177/1087054716666323

33.

Nantel-Vivier

Kokko

Caprara

G. V.

Pastorelli

Gerbino

M. G.

Paciello

. . . Tremblay

R. E.

(2009). Prosocial development from childhood to adolescence: A multi-informant perspective with Canadian and Italian longitudinal studies. Journal of Child Psychology and Psychiatry, 50, 590-598.

34.

Raftery

A. E.

(1995). Bayesian model selection in social research. Sociological Methodology, 25, 111-163.

35.

Rhemtulla

Brosseau-Liard

P. É.

Savalei

(2012). When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions. Psychological Methods, 17, 354-373.

36.

Rouquette

Côté

S. M.

Pryor

L. E.

Carbonneau

Vitaro

Tremblay

R. E.

(2014). Cohort profile: The Quebec longitudinal study of kindergarten children (QLSKC). International Journal of Epidemiology, 43, 23-33.

37.

Rutter

(1967). A children’s behaviour questionnaire for completion by teachers: preliminary findings. Journal of Child Psychology and Psychiatry, 8, 1-11.

38.

Sass

D. A.

Schmitt

T. A.

Marsh

H. W.

(2014). Evaluating model fit with ordered categorical data within a measurement invariance framework: A comparison of estimators. Structural Equation Modeling, 21, 167-180.

39.

Schermelleh-Engel

Moosbrugger

Muller

(2003). Evaluating the fit of structural equation models: Tests of significance and descriptive goodness-of-fit measures. Methods of Psychological Research, 8, 23-74.

40.

Spaeth

Weichold

Silbereisen

R. K.

(2015). The development of leisure boredom in early adolescence: Predictors and longitudinal associations with delinquency and depression. Developmental Psychology, 51, 1380-1394.

41.

Sprott

J. B.

Jenkins

J. M.

Doob

A. N.

(2000). National Longitudinal Survey of Children and Youth (Report prepared for the Applied Research Branch, Strategic Policy, Human Resources Development Canada). Gatinea, Quebec, Canada: Human Resources Development Canada.

42.

Sterba

S. K.

Copeland

Egger

H. L.

Jane Costello

Erkanli

Angold

(2010). Longitudinal dimensionality of adolescent psychopathology: Testing the differentiation hypothesis. Journal of Child Psychology and Psychiatry, 51, 871-884.

43.

Tremblay

R. E.

Loeber

Gagnon

Charlebois

Larivee

LeBlanc

(1991). Disruptive boys with stable and unstable high fighting behavior patterns during junior elementary school. Journal of Abnormal Child Psychology, 19, 285-300.

44.

Tremblay

R. E.

Vitaro

Gagnon

Piché

Royer

(1992). A prosocial scale for the Preschool Behaviour Questionnaire: Concurrent and predictive correlates. International Journal of Behavioral Development, 15, 227-245

45.

van Lier

P. A.

Vitaro

Barker

E. D.

Brendgen

Tremblay

R. E.

Boivin

. (2012). Peer victimization, poor academic achievement, and the link between childhood externalizing and internalizing problems. Child Development, 83, 1775-1788.

46.

Van Oort

F. V. A.

Greaves-Lord

Verhulst

F. C.

Ormel

Huizink

A. C

. (2009). The developmental course of anxiety symptoms during adolescence: The TRAILS study. Journal of Child Psychology and Psychiatry, 50, 1209-1217.

47.

Verhoeven

Sawyer

M. G.

Spence

S. H.

(2013). The factorial invariance of the CES-D during adolescence: Are symptom profiles for depression stable across gender and time? Journal of Adolescence, 36, 181-190.

48.

Weir

Duveen

(1981). Further development and validation of the prosocial behaviour questionnaire for use by teachers. Journal of Child Psychology and Psychiatry, 22, 357-374.

49.

Widaman

K. F.

Ferrer

Conger

R. D.

(2010). Factorial invariance within longitudinal structural equation models: Measuring the same construct across time. Child Development Perspectives, 4, 10-18.

50.

Widaman

K. F.

Little

T. D.

Geary

D. C.

Cormier

(1992). Individual differences in the development of skill in mental addition: Internal and external validation of chronometric models. Learning and Individual Differences, 4, 167-213.

51.

Yoon

Millsap

R. E.

(2007). Detecting violations of factorial invariance using data-based specification searches: A Monte Carlo study. Structural Equation Modeling, 14, 435-463.