Abstract
The Child’s Challenging Behavior Scale, Version 2 (CCBS–2; Bourke-Taylor, Law, Howie, & Pallant, 2010; Bourke-Taylor, Pallant, & Law, 2014) is a short scale that was designed to measure a mother’s rating of the prevalence of challenging behaviors exhibited by her child that compromised self-reported maternal mental health. Using a mixed-methods instrument design model, the authors of the CCBS–2 identified challenging behaviors from mothers’ perspectives and developed the scale. The core construct measured by the CCBS–2 comprises behaviors that mothers themselves identify as challenging and that are associated with compromising their own mental health and caregiving capacity.
Items on the CCBS–2 were configured after expert consultation and an initial qualitative study (N = 8) with mothers and professionals (Bourke-Taylor, Howie, & Law, 2010). Subsequent phone interviews with concurrent survey data collection then occurred with mothers of children with a disability (N = 152) to evaluate the instrument’s psychometric properties (Bourke-Taylor, Law, et al., 2010). The purpose of the CCBS–2 is to identify families in which a child’s behavior and the parental experience of the behavior are indeed a problem for the family and therefore identifies the family for appropriate further assessment and services. The CCBS–2 has nine items consisting of statements about behaviors alongside a 4-item response set (ranging from strongly agree to strongly disagree) for mothers to complete about their own child (Table 1). There are multiple advantages to using a short parent-report scale: It is a cost-effective and time-efficient way to detect or identify higher risk child–parent dyads; eliminates the potentially difficult task of observing or eliciting less-than-cooperative behavior outside of the child’s usual environments and routine; provides a family-centered and collaborative tool with which to commence a discussion between parent and professional; and may help a professional to decide what action to take, ranging from offering education, providing reassurance, or deciding whether a more detailed assessment should be conducted (Glascoe, 2013).
Sample Items From the CCBS–2
Note. CCBS–2 = Child’s Challenging Behavior Scale, Version 2. Items were scored on the following continuum: 1 = strongly agree; 2 = agree; 3 = disagree; 4 = strongly disagree.
Reverse scored.
The scale is unique in that it provides a snapshot of the maternal view of behavior. Past evaluations of the CCBS–2 have demonstrated that mothers with mental health conditions and reduced self-reported mental health also reported a higher prevalence of challenging behavior exhibited by their child (i.e., higher CCBS–2 scores) than mothers without mental health conditions (Bourke-Taylor, Law, et al., 2010). Moreover, when mothers reported poorer mental health they also reported higher prevalence of challenging behavior (higher CCBS–2 scores) when they had a child with a disability or a typically developing younger child (Bourke-Taylor, Law, et al., 2010; Bourke-Taylor, Pallant, & Cordier, 2017).
Past research has indicated that many mothers with mental health conditions or higher stress levels are more likely than other mothers to notice and report difficult or challenging behavior in their child, regardless of the child’s age (Baker et al., 2003; Totsika, Hastings, Emerson, Lancaster, & Berridge, 2011) and whether the child has a disability (Bennett et al., 2012; Emerson, Einfeld, & Stancliffe, 2011) or is typically developing (Bolton et al., 2003; Kim-Cohen, Moffitt, Taylor, Pawlby, & Caspi, 2005). Most behaviors described in the CCBS–2 are externalizing behaviors such as aggression, violence, oppositional and disruptive behavior, resistance to routine, rejection of other adults as carers, and a state of being persistently unhappy or discontent (see Table 1 for items).
Although external behaviors are often the result of an internal emotional state, the CCBS–2 focuses on externalizing behaviors because these items were identified by mothers as most strongly compromising their mental health and caregiving capacity. The scale is potentially useful for clinicians to identify both women and children in need of intervention to reduce maternal stress; improve maternal mental health; and address challenging, difficult, or disruptive behavior displayed by the child. Moreover, past research has suggested that maternal identification of challenging, externalizing behavior at age 3 is predictive of similar behavior at age 5 (Kerr, Lunkenheimer, & Olson, 2007).
It seems reasonable to suggest that mothers can identify challenging behaviors that may have an impact outside of the home. Compared with previous research about behaviors that compromise performance in school, externalizing behaviors make socializing with other children more difficult (Arim et al., 2015; Langeveld, Gundersen, & Svartdal, 2012; Sturaro, van Lier, Cuijpers, & Koot, 2011), can compromise academic performance (Metsäpelto et al., 2015; van Lier et al., 2012; Zimmermann, Schütte, Taskinen, & Köller, 2013), and can negatively influence teacher–student relationships (Silver, Measelle, Armstrong, & Essex, 2005). Therefore, the CCBS–2 should correlate well with other known, gold standard child behavior scales that detect behavioral difficulties.
The COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN; Mokkink et al., 2010a, 2010b) is a useful guide to both plan and report the psychometric evaluations of instruments used in clinical practice and research. The COSMIN framework guided the ongoing evaluation of the psychometric properties of the CCBS–2 and was applied in the current study. Previous evaluations of the CCBS–2 have investigated reliability status, including a single-factor structure and good internal reliability with a population of children with a disability (Bourke-Taylor et al., 2014) and without a disability (Bourke-Taylor et al., 2017). Validity, including content and face validity during development, has been investigated, and construct validity has been evaluated with a sample of mothers of children with a disability with and without a mental health condition (Bourke-Taylor, Law, et al., 2010) and mothers of typically developing children (Bourke-Taylor et al., 2017).
Construct validity was also measured through hypothesis testing with the first version of the CCBS (Bourke-Taylor, Law, et al., 2010). Investigation of the relationship between challenging behavior and maternal mental health revealed moderate correlations, and there were significant differences in CCBS scores between groups of mothers with and without a mental health condition. Moreover, the presence of challenging behavior was associated with reduced family cohesion and maternal empowerment among mothers of children with a disability. Preliminary investigations of the challenging-behavior construct revealed a correlation with psychosocial problems exhibited by the child, a lack of cooperation when taking medication, and significantly higher CCBS scores in children diagnosed with autism or an additional childhood psychiatric disorder (Bourke-Taylor, Law, et al., 2010).
To date, the criterion validity of the CCBS–2 has not been investigated. The COSMIN taxonomy identifies criterion validity as an important psychometric property because similarity (convergence) and difference (divergence) with the gold standard tool can determine the extent to which the new tool is an adequate reflection of the proven construct (Mokkink et al., 2010b). By comparing a new tool with a well-evaluated, widely used one, an important benchmark of equivalency may be reached. Gold standard and widely used tools that measure childhood behaviors, including difficult or challenging behavior, are the Child Behavior Checklist (CBCL; Achenbach & Rescorla, 2000, 2001) and the Strengths and Difficulties Questionnaire (SDQ; Goodman, 1997). The COSMIN stipulates that investigations of criterion validity require identification of a gold standard and generation of hypotheses before research is conducted. COSMIN identifies correlational analysis as the most appropriate statistic when one is using cross-sectional methods and comparing instruments with continuous variables (Mokkink et al., 2010a; Streiner, Norman, & Cairney, 2014).
The purpose of this research was to extend validity of the CCBS–2 and complete investigations of its criterion validity by comparing maternal scoring of the CCBS–2, CBCL, and SDQ with typically developing children ages 3–9 yr. We addressed two hypotheses in this study:
Hypothesis 1: When rated by mothers of typically developing children ages 3–9 yr, CCBS–2 scores will support convergent validity and correlate more than moderately (ρ > .3) and positively with the age-related CBCL externalizing-related subscales and the overall Externalizing score and will have weaker correlations with internalizing-related subscales and the overall Internalizing score.
Hypothesis 2: When rated by mothers of typically developing children ages 3–9 yr, CCBS–2 scores will support convergent validity and correlate more than moderately (ρ > .3) and positively with the SDQ externalizing-related subscales and the overall Externalizing score and have weaker correlations with internalizing-related subscales and the overall Internalizing score and support divergent validity by correlating moderately and negatively with the Prosocial subscale.
Method
A mail survey design was used to recruit mothers of children ages 3–9 yr across Australia. Before recruitment, the study was approved by the Monash University Human Ethics Committee during data collection and by the Australian Catholic University Human Ethics Committee during analysis and write-up.
Participants and Recruitment
Inclusion criteria required that the mother was the primary carer of a typically developing child age 3–9 yr, 11 mo, a resident of Australia, and able to complete the survey in English. See Table 2 for participant details. Other data related to maternal mental health and to establish certainty that the children described were typically developing were collected and analyzed during this project and were published elsewhere (Bourke-Taylor et al., 2017).
Characteristics of the Typically Developing Australian Mother–Child Sample
Note. M = mean; SD = standard deviation.
Mothers were recruited over 8 mo by means of websites, child-related magazines, newspaper columns, and snowball sampling. All mothers self-selected in response to a notice about “behaviors of young Australian children.” Interested participants phoned the primary researcher. Once verbal consent was received, mothers were sent the survey and provided written informed consent along with the returned survey. Names were not recorded after the questionnaires were received and data were entered. In this article, we report on the anonymous mail survey, which included demographic questions, the CCBS–2, the CBCL younger and older child versions, and the SDQ.
Instruments
Child’s Challenging Behavior Scale, Version 2.
In addition to the properties already described in this article, internal consistency for the CCBS–2 has been determined to be very good (Cronbach’s αs have been reported to range from .77 to .84 in previous studies), with no statistically significant gender variations (Bourke-Taylor, Law, et al., 2010; Bourke-Taylor et al., 2017). Classical test theory established one factor, and Rasch analysis determined that a 9-item scale with a 4-item response scale created the most sound instrument (Bourke-Taylor et al., 2014). Moreover, construct validity has also been established, with significant differences found in CCBS–2 scores between groups of children according to age (p = .008), presence of a disability (p < .001), and school entry (p = .018).
Child Behavior Checklist, Younger and Older Child Versions.
Both the younger and the older versions of the CBCL were selected because the ages of children in this study ranged from 3 to 9 yr, crossing both instruments: the CBCL for 1.5- to 5-yr-olds and the CBCL for 6- to 18-yr-olds. Versions of the CBCL have been validated on Australian children younger than age 10 yr (Bayer et al., 2012).
Parent versions were used in this study because they described child behavioral or emotional problems over the past 2 mo (CBCL 1.5–5 yr) or 6 mo (CBCL 6–18 yr; Achenbach & Ruffle, 2000). Both forms contain a 3-point scale—0 = not true, 1 = somewhat or sometimes true, and 2 = very true or often true—for rating specific problems. For both instruments, normative data are used to identify scores outside the typical range for the appropriate age group and are represented in the form of T scores (Achenbach & Ruffle, 2000). This section describes the derived scores pertaining to hypothesis testing and subsequent data analysis to determine criterion validity.
The CBCL (1.5–5) contains 99 items with an additional item to include problem behaviors not previously identified. The CBCL (1.5–5) is a parent-rated scale that evaluates problem behaviors in the home setting in the previous 2 mo with the opportunity for extended responses under some items. Items are scored and plotted onto seven syndrome scales (see Table 3), for the Aggressive Behavior, Emotionally Reactive, Attention Problems, Anxious/Depressed, Sleep Problems, Somatic Complaints, and Withdrawn subscales (Achenbach & Rescorla, 2000). An Internalizing score for the CBCL (1.5–5) is calculated using the Emotionally Reactive, Anxious/Depressed, Somatic Complaints, and Withdrawn subscales; Externalizing scores are calculated using the Attention Problems and Aggressive Behavior subscales. Sound psychometric properties have also been established for the CBCL (1.5–5), including test–retest, reliability and content, criterion (Achenbach & Rescorla, 2000), and construct validity with typically developing children (Bayer et al., 2012). The CBCL has good internal consistency and construct validity related to children with autism spectrum disorder (Pandolfi, Magyar, & Dill, 2009) and predictive validity between Externalizing scores and childhood psychiatric issues (Keenan & Wakschlag, 2000).
Comparison of the CCBS–2, the Strengths and Difficulties Questionnaire, and the Child Behavior Checklist
Note. CCBS–2 = Child’s Challenging Behavior Scale, Version 2.
p < .05. **p < .01.
The CBCL (6–18) contains 113 items that are grouped into eight subscales: Anxious/Depressed, Withdrawn/Depressed, Somatic Complaints, Social Problems, Thought Problems, Attention Problems, Rule-Breaking Behavior, and Aggressive Behavior (Achenbach & Rescorla, 2001). Internalizing and Externalizing overall scores are specifically calculated (see the “Data Collection, Management, and Analysis” section). The CBCL (6–18) has excellent reliability (test–retest, r = .94), adequate internal consistency (Cronbach’s αs = .63–.79), cross-informant agreement (parents, r = .76), and stability over 12 and 24 mo (rs = .70 and .61, respectively; Achenbach & Rescorla, 2001). The assessment’s validity has also been established, with support for discriminant and convergent validity (Gomez, Vance, & Gomez, 2014), as well as content and criterion validity (Achenbach & Rescorla, 2001). Scoring can also be completed using the six scales oriented to the Diagnostic and Statistical Manual of Mental Disorders (4th ed., DSM–IV; American Psychiatric Association, 1994), which have good reliability, internal consistency (Cronbach’s αs = .71–.89), and convergent validity (Nakamura, Ebesutani, Bernstein, & Chorpita, 2009).
Strengths and Difficulties Questionnaire.
In this study, both the SDQ and the CBCL were used as gold standard comparators because they measure a construct similar to that of the CCBS–2, but the SDQ also provides a Prosocial Behavior subscale. As described in Hypothesis 2, criterion validity would involve divergent validity by correlating moderately and negatively with the Prosocial subscale. Also as mentioned in Hypothesis 2, we expected that the CCBS–2 would correlate with the SDQ’s subscales.
The parent-rated SDQ is a 25-item screening tool designed to highlight child behaviors that are regarded as a strength (prosocial behavior) and behaviors identified as involving difficulties (peer problems, conduct, emotion, hyperactivity; Goodman, 1997). The assessment has versions for the child, parent, and teacher in relation to children ages 4–8 yr. The parent-rated SDQ has sound psychometric properties reported for Australian children (Gomez, 2014; Hawes & Dadds, 2004; Mathai, Anderson, & Bourne, 2004). Moderate concurrent validity (rs = .39–.56) has been established between DSM–IV diagnosis by a clinical team and an SDQ-generated diagnosis (Mathai et al., 2004). Moderate to strong internal consistency (Cronbach’s αs = .59–.80) and stability (rs = .63–.77) were also demonstrated in a population of Australian children ages 4–9 across all subscales (Hawes & Dadds, 2004).
The current study extended use of the SDQ to children age 3 yr. Convergent validity has been established between the parent-rated and child-rated SDQ, with discriminant validity recognized between all subscales (correlations <.85), excluding Conduct Problems and Hyperactivity. A British study also reported cross-sectional predictive validity between the SDQ and the CBCL with an overall correlation of .87 (Goodman & Scott, 1999). Criterion validity has been established between mothers’ scores of their child’s behavior as rated using the SDQ and the CBCL (Goodman & Scott, 1999).
Data Collection, Management, and Analysis
We used IBM SPSS Statistics (Version 22; IBM Corp., Armonk, NY) for all data entry and management. Descriptive statistics (means, standard deviations, and frequencies) were generated for all scale items and for scale scores. Few items were missing, with a range of 0.5%–1% for the CBCL and SDQ questionnaire items. The CCBS–2 was normally distributed, although the CBCL older and younger versions and the SDQ were not. The SDQ and CBCL subscale scores were derived through specific summation of scores per manual directions. Further summary scores were computed according to published directions and transformed to SDQ and CBCL Internalizing and Externalizing scores.
Hypotheses 1 and 2 were investigated through the use of nonparametric correlation statistics between the SDQ subscales as well as through derived internalizing and externalizing scores and between the subscales of the CBCL younger and older and derived internalizing and externalizing scores. According to Cohen (1988), correlation coefficients in the order of .10 are weak, those of .30 are moderate, and those of .50 are strong in terms of magnitude.
Results
Data were collected from every Australian state. The final sample consisted of 337 mother–child dyads with typically developing children ranging in age from 3 to 9 yr (see Table 2). Children were mainly from families that had mothers who were largely university educated (72%), were two-parent families (93%), and had employed mothers (61%).
We calculated correlations between the CCBS–2 and the SDQ subscales and derived Internalizing and Externalizing scores. We also calculated correlations between the CCBS–2 and the younger version subscales and the CCBS–2 and the CBCL older version subscales. Internalizing and Externalizing scores were calculated for the CBCL younger and older versions and were also correlated with the CCBS–2. See Table 3 for results of the correlation analyses.
Hypothesis 1
As expected, the convergent validity of the CCBS–2 was supported with moderate or stronger correlations with both the CBCL younger and older version subscales and the Externalizing scores. As predicted, weaker correlations were calculated for the Internalizing scores and related subscales for both versions of the CBCL.
Hypothesis 2
As expected, the convergent and divergent validity of the CCBS–2 was supported with moderate or stronger positive correlations with the SDQ Externalizing score and related subscales. Weaker (though moderate) correlations were calculated for the SDQ Internalizing score and two subscales (Emotions and Internalizing). The Peer Problems subscale was only weakly correlated. As predicted, the CCBS–2 scores and the SDQ Prosocial subscale had moderate negative correlations. Moreover, the CCBS–2 correlated weakly or moderately with all additional CBCL subscales that do not contribute to the Internalizing and Externalizing subscales (i.e., Emotional Reactivity, Attention Problems, Sleep Problems, Social Problems, and Thought Problems).
Discussion
This study investigated the criterion validity of the CCBS–2 through a comparison with two gold standard assessments—the CBCL younger and older versions and the SDQ—using a sample of Australian children ages 3–9 yr as scored by their mothers. Criterion validity was supported given that the CCBS–2 correlated with the CBCL younger version, with the strongest correlations evident between externalizing behaviors and the Externalizing and Aggressive subscales. Criterion validity was also supported, with the strongest correlations evident with the CBCL older version Rule-Breaking, Aggressive, and Overall Externalizing Behaviors subscales. Criterion validity was further supported by strong correlations between the CCBS–2 and SDQ externalizing-related subscales and derived externalizing scores. An inverse moderate correlation with the SDQ Prosocial subscale also supported criterion validity and indicated that children who scored higher for challenging behaviors were also likely to have scored low for positive prosocial behaviors.
Past evaluations and the current findings suggest that the importance of this tool is twofold. First, the CCBS–2 may identify parents in need of assistance related to parenting or caring for their child when challenging behavior is present. Second, it may help distinguish young children who require further evaluation and possibly intervention. With regard to the identification of parents (particularly mothers), the impact of managing a child’s challenging behaviors, coupled with childhood disability, on maternal mental health cannot be underestimated. Substantial research supports the associations between caring and mental health among mothers of children with disabilities (Bourke-Taylor, Howie, Law, & Pallant, 2012; Etherington, McDougall, DeWit, & Wright, 2015). Because mothers with mental health conditions are more likely than mothers without mental health conditions to notice and report challenging behavior in their children (Bourke-Taylor, Law, et al., 2010), garnering a mother’s perspective of her child’s behaviors may help identify women whose mental health merits further investigation by clinicians. In other words, the CCBS–2 could be used not only to guide child-focused interventions but also to identify women who could benefit from advice or interventions to reduce parenting-related maternal stress.
Limitations
Although our sample size was adequate for assessing criterion validity according to COSMIN, this study was conducted with a relatively small number of Australian women. Tertiary-educated mothers were likely overrepresented in the sample compared with the population of Australia, and annual household income was inflated in this sample at more than twice the national average (Australian Bureau of Statistics, 2008, 2013). Socioeconomic status has been inversely correlated with childhood problem behaviors (Boyle & Lipman, 2002), and parental education levels can predict externalizing behaviors in adolescence, particularly in boys (Sourander & Helstelä, 2005). As such, replications of the study with much larger and broader population samples are required to allow the development of norms for clinical use. Similarly, all children sampled in this study were typically developing. If the CCBS–2 is to be used to identify children at risk of behavior problems, then data from a broader sample of children will also need to be collected to establish cutoffs for clinical use. Moreover, the SDQ has not been validated with 3-yr-old Australian children (28% of this sample), and thus the results should be interpreted with caution. Future research might validate the SDQ with younger children and replicate the study excluding 3-yr-olds.
Implications for Occupational Therapy Practice
The CCBS–2 has been developed by occupational therapists. Given all the psychometric development and evaluation to date, the CCBS–2 has several clinical and research implications for occupational therapy practitioners:
The CCBS–2 can be used as a clinical tool to initiate conversations between parents and occupational therapy practitioners about behaviors exhibited by children with a disability.
The CCBS–2 can be used as an initial assessment tool for occupational therapy practitioners to initiate referrals to psychology, family, or mental health and well-being services.
It can help identify at-risk families who have a need to be given priority consideration for services when wait lists or access to services is an issue.
It could be used as an initial assessment before initiating more detailed occupational therapy assessment of attention, behavior, sensory functioning, cognition, language, or social or academic skills or assessment of self-care or play skills.
Conclusions and Future Directions
This study demonstrated significant, moderate to strong correlations between the CCBS–2 and other measures of childhood behavior problems, thus supporting criterion validity compared with the SDQ and CBCL. The findings suggest that the CCBS–2 is an easily administered and scored screening tool of the parent’s perspective of childhood behaviors, which identifies children in need of more complete evaluation for problematic behaviors. Sensitivity and specificity are important aspects of scale functioning in screeners, and further investigation of these properties of the CCBS–2 is warranted to establish positive and negative predictive values. The instrument may also have some clinical utility as an outcome measure in the monitoring of intervention effectiveness; as such, there is a need to evaluate the responsiveness of the CCBS–2 in the future, along with the evaluation of the measure’s sensitivity to change in longitudinal and interventional studies.
Footnotes
Acknowledgments
This research was funded by a Monash University Early Career grant. We thank occupational therapists Aislinn Lalor, Rebecca Stephan, and Sarah Grzegorczyn for assistance with this project.