The Factor Structure of Child Behavior Checklist Scores With Elementary School Students Referred to Counseling Within Low-Income Communities

Abstract

The Child Behavior Checklist (CBCL) is one of the most frequently used assessments of social, emotional, and behavioral functioning; however, previous research has noted inconsistency in the factor structure and items included on the Child Behavior Checklist for Ages 6 to 18 Years (CBCL/6-18) when tested with diverse samples of client populations. Thus, the purpose of our investigation was to examine the factor structure of CBCL/6-18 scores (N = 459) with diverse American children referred to receive school-based mental health counseling enrolled in five Title I elementary schools in the Southeastern United States. We performed confirmatory factor analysis (CFA) and principal component analysis (PCA) on CBCL/6-18 scores to examine the factor structure and internal consistency reliability of the data. Results demonstrated an inadequate fit for model and further data analyses resulted in a three-factor, 32-item model (41.40% of the variance explained). Implications of the findings support a new conceptual framework of the CBCL/6-18 to provide a more parsimonious model when working with diverse populations, specifically children from low-income families.

Keywords

Child Behavior Checklist factorial analysis school-based mental health counseling interventions

The prevalence of psychological disorders among adolescents is increasing, with about half of all diagnosed mental illnesses beginning in childhood (National Institute of Mental Health, 2016). Furthermore, children from low-income families (i.e., receiving financial services from the government, enrolled in Title I schools) are at an increased risk of acquiring a mental health disorder, as approximately 64% of children living in low-income homes have a diagnosable psychiatric disorder and 23% receive appropriate mental health services (Stewart et al., 2013). One potential explanation for the prevalence of mental health disorders in children living in low-income homes may be that they are at an increased risk to be exposed to traumatic events throughout their childhood (e.g., community violence, neglect), triggered by food insecurity, episodes of homelessness, parental substance abuse, parental unemployment, marital discord, parental mental illness, and parental incarceration (Osofsky et al., 2015). Furthermore, children having ethnic minority backgrounds are at an increased risk of experiencing a traumatic event, with 61% of African American/Black and 51% of Latino/Hispanic children experiencing at least one adverse childhood experience in comparison with 40% of Caucasian/White and 21% of Asian American children (Blodgett & Lanigan, 2018; Bruce, & Waelde, 2008). Due to the rapid emotional and physiological development of children, exposure to trauma and childhood maltreatment at an early age may have serious psychological, behavioral, and academic consequences (Osofsky et al., 2015). Due to the abovementioned risk factors, children living in low-income communities and of ethnic minority backgrounds may experience high rates of mental health issues, which go often untreated due to barriers in receiving appropriate psychological services (e.g., limited transportation, lack of funds, stigma; Lambie et al., 2019; Solomon et al., 2016), and these mental health disorders often persist into adulthood.

Although mental health disorders within children are becoming more prevalent, it is difficult to identify and monitor mental health symptomology in children due to various developmental changes (Kariuki et al., 2016). Thus, several childhood mental health assessments have been developed to assist therapeutic interventions to mitigate potential mental health disorders (e.g., depression; Kariuki et al., 2016; Osofsky et al., 2015). Assessments based on parents or caregivers’ self-report of their child’s mental health symptomology provide a significant measurement tool (Al-Hendawi et al., 2016). Specifically, caregivers’ perception of their children’s mental health functioning through rating scales (a) complements other sources of information such as direct observation; (b) provides access to contexts not available to other professionals working with the child (e.g., clinicians or educators); (c) serves as cost-effective collected sources of data; and (d) offers a dimensional approach to classifying patterns of behaviors (Al-Hendawi et al., 2016). However, researchers have questioned the evidence of reliability and validity of these assessment instruments when measuring ethnic minority children and children from low-income families, calling for further examination of these measures (Tyson et al., 2011).

One of the most used assessments to measure children’s social and emotional problems is the Child Behavior Checklist for Ages 6 to 18 Years (CBCL/6-18; Achenbach & Rescorla, 2001). The CBCL/6-18 is a part of the Achenbach System of Empirically Based Assessments (ASEBA), noted as one of the most used collection of psychological questionnaires by educators, mental health professionals, and behavioral specialists in identifying child syndromes (Gomez & Vance, 2014). The CBCL/6-18 included a normative sample of 80% Caucasian children and more than 80% maternal informants, posing criticism of evidence of construct validity for the scores with ethnically diverse populations (Berubé & Achenbach, 2010). In contrast, the norm-referenced scales of the CBCL/6-18 were developed through factor analysis of data gathered from the general pediatric population of children, providing a better representation of child symptomology throughout the general population (Berubé & Achenbach, 2010).

The CBCL/6-18 was created to measure children’s functioning and behavioral and emotional problem areas to monitor treatment effectiveness (Achenbach & Rescorla, 2001). The revised CBCL (Achenbach & Rescorla, 2001) gathers caregivers’ self-report of their child’s internalizing, externalizing, and total problem behavior within the preceding 6-month period (CBCL/6-18; Al-Hendawi et al., 2016). The CBCL/6-18 has been implemented in various settings, including academic, juvenile justice, and clinical locations (Moruzzi et al., 2010). The revised CBCL/6-18 has been translated into approximately 80 languages or dialects and has been employed in more than 30 societies of children from different ethnic backgrounds, including samples from, China, Germany, Norway, Turkey, and the United States (Al-Hendawi et al., 2016; Kariuki et al., 2016).

The CBCL/6-18 is used in research and in clinical settings; however, research examining the evidence of validity and reliability of the CBCL/6-18 identifies inconsistent results (Gomez & Vance, 2014). The developers originally purported the CBCL/6-18 as a multidimensional model that was constructed as a 120-item assessment, given on a scale of 0 (not true), 1 (somewhat or sometimes true), or 2 (very true or often true). The factor structure of the CBCL/6-18 results provided eight behavioral and emotional problem areas described as syndrome scales, including (a) Anxiety/Depression, (b) Withdrawal/Depression, (c) Somatic Complaints, (d) Social Problems, (e) Thought Problems, (f) Attention Problems, (g) Rule-Breaking Behavior, and (h) Aggressive Behavior (Achenbach & Rescorla, 2001). As a result of the diverse factor structures identified for CBCL/6-18 scores, the interpretation of CBCL scores to identify child syndromes should be done with caution (Gomez & Vance, 2014). In addition, some researchers have removed CBCL/6-18 items to achieve a stronger factor structure with better fit indices (Pandolfi et al., 2012). For example, Al-Hendawi and colleagues (2016) tested the factor structure of the CBCL/6-18 scores and replaced items that were shown to be ineffective with six new problem items in a sample of children in Qatar (N = 1,360). In addition, the result of a confirmatory factor analysis (CFA) identified that 10 out of the 96 items did not load highly on factors with loadings less than 0.5 and the CBCL exhibited poor discriminant validity in the domain of Social Problem Syndromes, as six out of nine factors had an intercorrelation value that was higher than .85. Thus, Al-Hendawi and colleagues (2016) reported that the CBCL/6-18 may not be culturally sensitive in Arabic populations due to redundancy of symptoms items associated with both internalizing and externalizing problem scores. Specifically, the items on the CBCL/6-18 may not be functioning as intended in diverse ethnic groups (Al-Hendawi et al., 2016).

Although the CBCL/6-18 syndrome scales are not identical to diagnosable clinical disorders, they do align to diagnostic criterion. Specifically, Gomez and Vance (2014) examined the factor structure of CBCL/6-18 scores with a sample of clinically referred children (N = 720) in accordance to the relationships with anxiety disorders, depressive disorders, attention and hyperactive disorders (ADHD), and oppositional defiant disorders (ODD). The factor mixture modeling (FMM) results were associated positively with only anxiety disorders, implying a lack of support for the external validity of the CBCL/6-18. Therefore, the researchers concluded that the CBCL/6-18 was not capable of differentiating children with respect to depression, ADHD, or ODD, despite having subscales that consist of Anxiety/Depression, Attention Problems, Rule-Breaking Behavior, or Aggressive Behavior (Gomez & Vance, 2014). As a result, the CBCL/6-18 suggests limitations for facilitating clinical diagnosis and judgment, supporting caution in the interpretation of the assessment results (Gomez & Vance, 2014).

Although significant research exists using the CBCL/6-18, “relatively few reports have been published that describe how various American ethnic groups (e.g., Black/African American, Hispanic/Latino American) differ with respect to scores on the CBCL” (Armsden et al., 2000, p. 62). Moreover, few reports have been published on the factor structure of the CBCL/6-18 scores with American samples, whereas many have been tested with samples outside of the United States (Dedrick et al., 2008; Dumenci et al., 2004). In an attempt to address the limited research examining the factor structure of CBCL/6-18 scores in the United States, Tyson and colleagues (2011) examined the cross-ethnic measurement equivalence of the CBCL/6-18 with a sample of adopted youth in the United States who had special needs (African American children: n = 295; Caucasian children: n = 615). Results from a CFA identified that the CBCL/6-18 data for the full sample did not have an acceptable model root mean square error of approximation (RMSEA) fit criteria (0.90; Byrne, 2016; Kline, 2015). Thus, a multigroup confirmatory factor analysis (MGCFA) was justified, and researchers removed specific items on the CBCL/6-18 to examine if potential ethnic group effects would emerge. The results of the MGCFA revealed that none of the equivalent models fit the data well across either group, goodness-of-fit index (GFI) ranged from 0.69 to 0.79, and the comparative fit index (CFI) ranged from 0.68 to 0.76. Tyson and colleagues’ results offer evidence that the construct validity of the CBCL/6-18 scores did not meet the test of measurement equivalence across the two groups of adopted youths, concluding that the CBCL/6-18 may not provide accurate estimates of mental health symptomology across subgroups, specifically with children from diverse cultural backgrounds.

Mano and colleagues (2009) also examined the factor structure of CBCL/6-18 scores within a sample of ethnic minority youth in the United States (African American adolescent sample, N = 145) and their CFA results indicated a poor factor model fit (RMSEA = 0.13). In addition, the results identified lower internal consistency reliability for the CBCL/6-18 scores as compared with the norm sample (Achenbach & Rescorla, 2001), with Cronbach’s alpha coefficients in the range of α = .65–.88. Mano and colleagues’ findings align with Tyson and colleagues’ (2011) result, as these researchers reported that the robustness of the CBCL/6-18 may be questionable with minority samples and that some of the factors measured by the CBCL/6-18 may not have the same meaning across cultural groups.

Due to the inconsistency in the factor structure and items included on the CBCL/6-18 when tested with diverse samples of client populations, the need for additional research testing evidence of validity for CBCL/6-18 scores with diverse ethnic samples is apparent (Tyson et al., 2011). In addition, because of the frequency of the use of the CBCL/6-18 in the educational and mental health setting by practitioners (e.g., educators and mental health clinicians), further research examining the factorial structure of the CBCL/6-18 with diverse samples (e.g., ethnic minority and economically disadvantaged children) is warranted. Furthermore, based on previous literature, we hypothesize that the factor structure of the CBCL/6-18 will have an inadequate fit with diverse American children. Thus, the purpose of our investigation was to examine the factor structure of CBCL/6-18 scores (N = 459) with diverse American children referred to receive school-based mental health counseling enrolled in five Title I elementary schools in the Southeastern United States. Title I elementary schools receive additional funding to support students living in low-income communities, and thus the sample included children from low-income homes (Jacob, 2007). Therefore, the following research questions guided our investigation:

Research Question 1: What is the factor structure of the CBCL/6-18 data with a referred sample of ethnically diverse elementary-aged students receiving school-based mental health counseling from low-income communities?

Research Question 2: What is the internal consistency reliability of the CBCL/6-18 scores with an ethnically diverse referred sample of elementary-aged students receiving school-based mental health counseling from low-income communities?

Method

Procedure

Before data collection, we obtained approval from the university’s institutional review board (IRB). The data collected were part of two larger studies exclusive of one another located within the southeastern region of the United States. Both studies provided school-based mental health counseling services at no cost to elementary-aged children in five Title I elementary schools. School personnel identified participants in the studies (e.g., administrators and school counselors) as exhibiting problematic behaviors, and parents or legal guardians provided consent at the outset of the investigation. In addition, parents or legal guardians were able to refer their children to receive services. Data collection for our study was part of the assessment measures used to monitor participating children’s outcomes and evaluate treatment efficacy. Therefore, we did not recruit participants for this study exclusively, but rather purposively sampled from archival records of the two more extensive studies. Specifically, the data amassed for our investigation included children receiving school-based mental health counseling interventions at Title I elementary schools whose parent or legal guardians completed a CBCL/6-18 between January 2014 and May 2018.

Participants

An aggregate of 471 children participated in the study; however, data cleaning procedures and inclusion criteria resulted in the removal of data from 12 participants (N = 459). Based on the cumulative percentage of the general student population at the five data collection sites, most of the participants (93.5%) were from a low socioeconomic background and received free or reduced price lunch. The participants (males, n = 241 [52.7%]; females, n = 216 [47.3%]) represented in the study were elementary school students presenting with externalizing (e.g., physical aggression, disobeying rules) or internalizing (e.g., anxiety, depression) problems. The parents or legal guardians completing the CBCL/6-18 mostly identified as female (n = 305; 66.6%) and reported relationship to the participant as a biological parent (n = 398; 87.9%).

Table 1 presents participants’ full demographic data, representing a diverse pool of participants. The mean age of the participants was 7.51 (SD = 1.43; range = 6–11) across varying grade levels: kindergarten (n = 48; 10.6%), first grade (n = 83; 18.3%), second grade (n = 92; 20.3%), third grade (n = 96; 21.2%), fourth grade (n = 74; 16.3%), and fifth grade (n = 60; 13.2%). In addition, participants represented diverse racial and ethnic backgrounds, with the majority identifying as either Hispanic/Latinx (n = 149; 32.5%), Black/African (n = 139; 30.3%), or Caucasian/White (n = 132; 28.8%) (percentages do not total 100 due to rounding).

Table 1.

Participants’ Demographic Characteristics.

Characteristic	n	Total percentage
Gender
Female	216	47.3
Male	241	52.7
Grade level
Kindergarten	48	10.6
First grade	83	18.3
Second grade	92	20.3
Third grade	96	21.2
Fourth grade	74	16.3
Fifth grade	60	13.2
Racial background
Asian	3	0.7
Black/African American	139	30.3
Caucasian/White	132	28.8
Hispanic/Latinx	149	32.5
Multiracial	32	7.0
Native American	3	0.7
Parent/caregiver gender
Female	305	66.6
Male	153	33.4
Parent/caregiver relationship
Biological parent	398	87.9
Step-parent	4	0.9
Grandparent	13	2.9
Adoptive parent	10	2.2
Foster parent	3	0.7
Other	25	5.5

Note. N = 459 (percentages do not total 100 due to rounding or missing values). There were missing data from the demographics, therefore some of the characteristics may not equal 100% of the participants.

Data Instrumentation: CBCL/6-18

The CBCL/6-18 is a 120-item standardized measure designed to assess behavioral issues in children ages 6 through 18 (Achenbach & Rescorla, 2001). The parent/caregiver who spends the most time with the child completed the CBCL/6-18 using a dichotomized three-point response scale of 0 (not true of the child), 1 (somewhat or sometimes true), or 2 (very true or often true). The CBCL/6-18 items characterize common issues or syndromes in children across eight subscales: (a) Anxious/Depressed (13 items; e.g., “Cries a lot”), (b) Withdrawn/Depressed (eight items; e.g., “There is very little he or she enjoys”), (c) Somatic Complaints (11 items; e.g., “Feels dizzy or lightheaded”), (d) Social Problems (11 items; e.g., “Clings on adults or too dependent”), (e) Thought Problems (15 items; e.g., “Can’t get his or her mind of certain thoughts: obsessions”), (f) Attention Problems (10 items; e.g., “Can’t sit still, restless, or hyperactive”), (g) Rule-Breaking Behavior (17 items; e.g., “Doesn’t seem to feel guilty after misbehaving”), and (h) Aggressive Behavior (18 items; e.g., “Argues a lot”). In addition to the 103 items from the eight subscales, the CBCL/6-18 produces two overarching domains: Internalizing Problem scores (Anxious/Depressed, Withdrawn/Depressed, and Somatic Complaints) and Externalizing Problem scores (Rule-Breaking Behavior and Aggressive Behavior). The CBCL/6-18 further encompasses a Total Problem score produced by the sum of the 103 items from the eight subscales as well as 16 items on “other problems” and up to one item added by the parent/caregiver denoting an identified behavior not covered on the CBCL/6-18. Thus, Achenbach and Rescorla (2001) purported the CBCL/6-18 as a second-order factor model with the first order represented by the eight correlated subscales and the second-order factors of Internalizing and Externalizing problems.

The CBCL/6-18 scores generate raw scores; however, for interpretation, professionals using the CBCL/6-18 must convert scores to T scores using Achenbach and Rescorla’s (2001) normative sample for the CBCL/6-18. For the eight subscales, scores of 70 or greater indicate a clinical range, scores between 65 and 69 are in the borderline range, and scores less than 65 are in the normal range. Similarly, for Internalizing, Externalizing, and Total Problems, the cutoff point for the normal range is a T score less than 60, borderline from 60 to 63, and 64 or greater for a clinical range. Overall, the evidence of reliability of the CBCL/6-18 scores is good with an average Cronbach’s alpha of .80 (range = .63–.97), a test–retest reliability of .88 (range = .8–.94), and an interrater reliability of .73 (range = .57–.88) (Achenbach & Rescorla, 2001). However, as noted, evidence of construct validity for CBCL/6-18 scores is inconsistent (Tyson et al., 2011).

Data Cleaning and Screening

Before conducting analyses, we checked assumptions of normality and assessed the prevalence of missing data using Statistical Package for Social Sciences (SPSS; Windows Version 25.0). During the preliminary data check, we assessed the missingness of data by examining the mean differences of the 118 items using Little’s (1988) missing completely at random (MCAR) test in SPSS. The results revealed a nonsignificant chi-square (χ² = 7,855.09, df = 7,122, p = .96) and a small percentage of data (<0.02%) were missing; therefore, we determined data MCAR and ignorable (e.g., <5%; Kline, 2011; Osborne, 2013). To manage missing values, we used a full information maximum likelihood (FIML) approach in Analysis of Moment Structures (AMOS; Version 24.0). Enders (2010) noted FIML to be the most pragmatic missing data estimation approach as it does not replace or impute missing values, but instead estimates population parameters that would best produce the estimates from the sample data. Accordingly, cases were preserved, and our sample size met the recommended minimum size of 250 cases or more (Schumaker & Lomax, 2016).

To examine normality of the data, we checked outliers using the Mahalanobis D² distance at p < .001 (Kline, 2011). As a result, we deemed three cases erroneous and removed them from the dataset, resulting in a total sample size of 459. We found that our data had violated Mardia’s coefficient of multivariate kurtosis, resulting in multivariate nonnormality. To rectify multivariate normality, an assumption of CFA (Tabachnick & Fidell, 2013), we used parameters within our data analyses (see the “Data Analysis” section) robust to violations of normality.

Data Analysis

We examined scores from 118 items on the CBCL/6-18 for this study. Both items 56h and 113 were excluded from data analysis as they were open-ended items. After data screening and preliminary analyses, we conducted CFA in two stages using AMOS (Version 24.0) to determine the best fitting model. To maximize the likelihood of sampling the observed correlation matrix and mitigate threats of nonnormality, we employed a maximum likelihood estimation (Tabachnick & Fidell, 2013). We covaried all latent variables, constrained error term weights to one per default, and constricted one regression path weight to one within each latent variable to adhere to AMOS requirements (Byrne, 2016).

In the first CFA phase, we tested each of the eight syndrome subscales separately to determine whether the relationships among item scores for each syndrome subscale were accounted for by a single latent factor and could be interpreted exclusive of the CBCL/6-18 total score. Next, we analyzed the first order of the CBCL/6-18 by testing the eight syndrome subscales as an oblique model. In the second CFA phase, we examined the multidimensionality of the CBCL/6-18 using three models. The first model (Model 1) analyzed the CBCL/6-18 as a two-dimensional model represented by Internalizing and Externalizing Problem scores from their five syndrome subscales (Internalizing [Anxious/Depressed, Withdrawn/Depressed, Somatic Complaints] and Externalizing [Rule-Breaking Behaviors, Aggressive Behavior]). Model 2 was the original 118-item, hierarchical model purported by the developers (Achenbach & Rescorla, 2001). Model 2 contained a first order of eight syndrome subscales, with a second order represented by Internalizing and Externalizing Problems with the Other Problem items and Social Problems, Thought Problems, and Attention Problems syndrome subscales loading onto the third-order CBCL/6-18 total scores. Last, after a severe reduction of items, we tested Model 3, a reduced 71-item hierarchical model with the same multidimensional factor structure as Model 2.

CFA results were evaluated using multiple fit statistics to assess the adequacy of overall model fit. To compensate for the inherent biases of the chi-square statistic, including its sensitivity to sample size (Fan et al., 1999), we evaluated the chi-square test of model fit (χ²; Hu & Bentler, 1999), the CFI (Bentler, 1990), the Tucker–Lewis index (TLI; Tucker & Lewis, 1973), and the RMSEA with 90% confidence interval (CI; Steiger, 1990). We established cutoff criteria for fit indices based on Hu and Bentler’s (1999) recommended criteria for covariance structural equation modeling (SEM). Hence, the model is acceptable if the chi-square is not significant and the ratio of chi-square to degrees of freedom (χ²/df) is less than or equal to two. The CFI and TLI demonstrate a satisfactory fit when they exceed 0.90 and an appropriate fit when they exceed 0.95. Last, the RMSEA suggests a good fit when it is less than 0.05 and a respectable fit between 0.05 and 0.08 (Hu & Bentler, 1999).

Subsequently, based on the results of the CFA, we elected to conduct a principal component analysis (PCA; Pallant, 2016) in SPSS (Version 25.0) to identify plausible models of the CBCL/6-18 with these data due to many low factor loadings. Previous research supports our decision; Kline (2015) recommended that researchers use dimension reduction techniques (e.g., exploratory factor analysis [EFA], PCA, etc.) when a CFA indicates low factor loadings, indicating that the current data may not fit the original number of factors as suggested. A PCA approach is more robust to concerns of “factor indeterminacy” found in EFA and provides a simpler empirical summary of the dataset (Stevens, 1996, p. 363; Tabachnick & Fidell, 2013). Accordingly, we conducted a PCA with oblimin rotation in SPSS using the 95th percentile to investigate the factor structure of the CBCL/6-18 (Fabrigar et al., 1999).

To determine the number of factors to extract, scree plots (Cattell, 1966) and parallel analysis (PA; Horn, 1965) were used to generate simulated eigenvalues based on random data (N = 1,000) to our dataset eigenvalues. In comparison with more conventional statistical indices such as a Kaiser eigenvalue greater than 1 criterion (Kaiser, 1960), scholars support PA as a superior method to extract significant factors (Hair et al., 2019; Henson & Roberts, 2006). Item performance and retention was assessed using the following criteria: (a) factor coefficients > 0.40 were considered significant and retained (Hair et al., 2019), (b) items cross-loading onto more than one factor >0.40 were discarded (Tabachnick & Fidell, 2013), and (c) communalities < 0.50 were considering negligible and did not contribute significantly to variance (Hair et al., 2019).

Results

CFA

The first series of CFAs tested the CBCL/6-18 eight syndrome subscales and the first-order eight-factor model. The Withdrawn/Depressed syndrome subscale was the only model that fit the data and demonstrated a satisfactory fit; χ²(20) = 44.01, p = .002; χ²/df = 2.20; CFI = 0.95; TLI = 0.92; RMSEA = 0.05 [0.03–0.07], whereas the other seven subscales and the first-order eight-factor model also failed to fit the data as evidenced by poor fit indices displayed in Table 2. In the second phase, the multidimensional CBCL/6-18 scores were tested. Both Model 1 and Model 2 indicated an inadequate fit with the only fit index within cutoff criteria (0.05–0.08) being the RMSEA values at 0.07 and 0.06 (see Table 2).

Table 2.

CFA Model Fit Indices.

Measurement Models	χ²	df	p	χ²/df	CFI	TLI	RMSEA	90% CI
Anxious/Depressed	231.84	65.00	.00	3.57	0.71	0.66	0.08	[0.07, 0.09]
Withdrawn/Depressed	44.01	20.00	.00	2.20	0.95	0.92	0.05	[0.03, 0.07]
Somatic Complaints	146.99	44.00	.00	3.34	0.76	0.70	0.07	[0.06, 0.08]
Thought Problems	339.21	90.00	.00	3.77	0.62	0.56	0.08	[0.07, 0.09]
Attention Problems	151.70	35.00	.00	4.33	0.83	0.78	0.09	[0.07, 0.10]
Social Problems	254.95	44.00	.00	5.79	0.68	0.60	0.10	[0.09, 0.11]
Rule-Breaking Behaviors	383.80	118.00	.00	3.25	0.59	0.53	0.07	[0.06, 0.08]
Aggressive Behavior	698.99	135.00	.00	5.18	0.77	0.74	0.10	[0.09, 0.10]
First-Order Eight-Factor	12,282.32	5,122.00	.00	2.40	0.52	0.50	0.06	[0.05, 0.06]
Model 1	6,296.29	2,138.00	.00	2.95	0.59	0.57	0.07	[0.06, 0.07]
Model 2	18,047.89	6,773.00	.00	2.67	0.46	0.48	0.06	[0.06, 0.06]
Model 3	7,846.10	2,544.00	.00	3.08	0.61	0.60	0.07	[0.07, 0.07]

Note. CFA = confirmatory factor analysis; df = degrees of freedom; p = probability of chi-square; χ²/df = relative chi-square; CFI = comparative fit index; TLI = Tucker–Lewis index; RMSEA = root mean square of approximation; CI = confidence interval.

Given the poor model fits, we examined the outer loadings of each item to determine the overall appropriateness of the CBCL/6-18 items to our data (Costello & Osborne, 2005; Tabachnick & Fidell, 2013). We retained items with outer loadings 0.40 or greater and discarded those less than (Hair et al., 2019). Subsequently, after a severe reduction of items (n = 47) with low outer loadings, a 71-item hierarchical model was tested with the same factor structure as Model 2. The results of Model 3 also demonstrated a poor fit with all fit indices except for the RMSEA, thus overall failing to meet adequate criteria (Hu & Bentler, 1999). However, an assessment of outer loadings from Model 3 indicated 19 items with outer loading values below the 0.4 threshold. Therefore, we elected to conduct a PCA to explore the factors of the CBCL/6-18 with this population given that the CFA results indicated these data do not fit the original theorized factor structure (Kline, 2015).

PCA

A PCA in SPSS was employed to test the underlying dimensions of the 118 items of the CBCL/6-18 by transforming them into smaller sets of linear combinations, also known as factors or principal components (Pallant, 2016). In this investigation, we specifically selected PCA (as opposed to EFA) given that the factor structures of the eight syndrome scales failed to support the data and thus potentially the latent constructs purported in the original CBCL/6-18 measurement model. Thus, to reduce the number of variables and explore plausible factors in the CBCL/6-18 while accounting for the most amount of variance, we performed PCA (Tabachnick & Fidell, 2013). We first examined the Kaiser–Meyer–Olkin (KMO) value (0.835) and Bartlett’s test of Sphericity (p < .001) and determined the appropriateness of the data. Initial internal consistency reliability estimates (coefficient alpha) for scores on the original CBCL/6-18 model were all in the acceptable range (cf. Henson, 2001): Anxious/Depressed (α = .74), Withdrawn/Depressed (α = .77), Somatic Complaints (α = .71), Social Problems (α = .73), Thought Problems (α = .74), Attention Problems (α = .79), Rule-Breaking Behavior (α = .71), and Aggressive Behavior (α = .89).

Next, results of the PA (Horn, 1965) indicated that the CBCL/6-18 scores best represented a nine-factor solution. Therefore, we found in the initial PCA a nine-factor solution that explained 37.19% of the variance in the correlation matrix. Through a continual process of rerunning a series of PCAs, items were eliminated that produced factor pattern/structure coefficients less than 0.40, resulting in 52 item deletions.

As the factor structure can easily change after item deletions, a series of factor analyses were employed to refine the model. In all iterations, we used PCA, rechecked the number of factors via PA, examined PA eigenvalues, and inspected the scree plot. For a more parsimonious model, we elected to delete items that did not load onto a factor at greater than 0.4, with substantial cross-loadings after the initial solution or low communalities (<0.5), resulting in a total of 86 item deletions. Therefore, the final solution (see Table 3) resulted in a three-factor, 32-item model (see Supplemental Figure S1), explaining 41.40% of the variance. In addition, the coefficient alpha reliability for each of the 32-item CBCL three factors was as follows: Factor 1 (Externalizing Behaviors; n = 18), .91; Factor 2 (Internalizing Behaviors; n = 8), .79; and Factor 3 (Other Problem Behaviors; n = 6), .68.

Table 3.

Final Factor Structure of the CBCL/6-18.

Item	Factor
Item	1	2	3
3: Argues a lot	0.63
8: Can’t concentrate, can’t pay attention for long	0.55
10: Can’t sit still, restless, or hyperactive	0.63
16: Cruelty, bullying, or meanness to others	0.62
19: Demands a lot of attention	0.62
20: Destroys his or her own things	0.65
21: Destroys things belonging to his or her family or others	0.69
22: Disobedient at home	0.76
23: Disobedient at school	0.64
25: Doesn’t get along with other kids	0.59
26: Doesn’t seem to feel guilty after misbehaving	0.60
28: Breaks rules at home, school, or elsewhere	0.74
37: Gets in many fights	0.61
41: Impulsive or acts without thinking	0.70
57: Physically attacks people	0.56
78: Inattentive or easily distracted	0.54
95: Temper tantrums or hot temper	0.65
97: Threatens people	0.58
42: Would rather be alone than with others		0.65
65: Refuses to talk		0.62
71: Self-conscious or easily embarrassed		0.59
75: Too shy or timid		0.70
88: Sulks a lot		0.57
103: Unhappy, sad, or depressed		0.65
111: Withdrawn, doesn’t get involved with others		0.62
112: Worries		0.66
67: Runs away from home			0.52
73: Sexual problems			0.61
96: Thinks about sex too much			0.56
99: Smokes, chews, or sniffs tobacco			0.56
101: Truancy, skips school			0.72
105: Uses drugs for nonmedical purposes (don’t include alcohol or tobacco)			0.69

Note. CBCL/6-18 = Child Behavior Checklist for Ages 6 to 18 Years.

Discussion

Our study aimed to examine the factor structure and internal consistency reliability CBCL/6-18 scores in assessing behavioral/emotional problems in Title I elementary schools, representing a diverse pool of participants. The first series of CFA tested the first order of the CBCL/6-18 as a diagnostic approach to its eight subscales. Results demonstrated an inadequate fit for the individual syndrome subscales with the exception of the Withdrawn/Depressed. In addition, the first-order eight-factor CBCL/6-18 model failed to fit the data. Thus, given that the CBCL/6-18 provides cutoff scores (i.e., clinical and borderline) for the eight syndrome scales (in addition to total score; Achenbach & Rescorla, 2001) and the factor structures failed, these data warrant caution in the use and interpretation of the subscale scores for this population.

During the second phase, we tested a series of hierarchical models of the CBCL/6-18 scores. We analyzed a two-dimensional model represented by the Internalizing and Externalizing Problem scores from the five syndrome subscales for the first model, indicating a poor fit. The second model contained a first order of eight syndrome subscales, with a second order represented by Internalizing and Externalizing with the Other Problem items and Social Problems, Thought Problems, and Attention Problems syndrome subscales loading onto the third-order CBCL/6-18 total scores. Overall, the results identified an inadequate fit for these CBCL/6-18 data (the only fit index within the cutoff criteria was the RMSEA). Furthermore, we examined the outer loadings of each item to determine the overall appropriateness of the CBCL/6-18 items to our data. In Model 3, CBCL/6-18 items (n = 47) were reduced while keeping the same factor structure as Model 2, demonstrating a poor model fit.

Next, we conducted a PCA to determine the number of items to retain and factors of the CBCL/6-18 score, resulting in a revised CBCL/6-18 three-factor, 32-item model. The large number of item deletions was a result of having low factor pattern/structure coefficients. The 32 items retained were from the following syndrome subscales: Social Problems (one item), Attention Problems (two items), Rule-Breaking Behavior (three items), and Aggressive Behavior (13 items). The first factor, “Externalizing Behaviors,” consisted of items from the following syndrome subscales: Aggressive Behavior (n = 11), Attention Problems (n = 4), Social Problems (n = 1), and Rule-Breaking Behaviors (n = 2). The second factor was labeled “Externalizing Behaviors,” as the majority of these items (n = 13 out of 18) fell into the Externalizing Behaviors category, providing perhaps a better understanding of what these items are trying to capture from the parents’ report. The second factor, “Internalizing Behaviors,” consisted of items from the following syndrome subscales: Withdrawn/Depressed (n = 5), Anxious/Depressed (n = 2), and Aggressive Behavior (n = 1). The second factor was labeled “Internalizing Behaviors” because all items in Factor 2 are captured in the internalizing subscale except for one. However, this item (e.g., Item 88: sulks) that was originally a part of the Aggressive Behavior syndrome subscale appears to fit the essence of the depressed nature of Factor 2. All five items on Factor 3 fell into the original syndrome subscale, Rule-Breaking Behavior, and thus were categorized as “Other Problem Behavior.”

The internal consistency reliability estimates for the CBCL/6-18 scores were found to be within the acceptable range, except for Factor 3. Many of the items appear to be in consideration of older children/adolescents (i.e., sex problems, thinks about sex, uses drugs). Given that the current sample contains elementary-only children, the internal consistency may not be as accurate. This result aligns with Tyson and colleagues’ (2011) findings of acceptable range of the internal consistency reliability. Factors loadings and fit statistics for the eight-syndrome CBCL/6-18 structure were not found to fit the data with these participants. Our results were similar to Tyson and colleagues’ findings, identifying that evidence of construct validity for CBCL/6-18 scores was not met for the test of measurement equivalence across two groups of adopted youth, concluding that the CBCL/6-18 may not provide accurate estimates of mental health symptomology across subgroups, especially with children. Furthermore, our findings align with Mano and colleagues’ (2009) results indicating a poor model fit for ethnic minority youth in the Unites States. Mano and colleagues’ CFA analyses indicated that the CBCL/6-18 provided a poor fit to the observed data (African American adolescents). Mano and colleagues’ (2009) findings identified that a two-factor model with double loadings for Social Problems and Thought Problems provided better fit statistics; however, their item deletion was not as large as the one in this study. Gomez and Vance (2014) concluded that a two-class, two-factor model provided the most support after conducting a CFA of the CBCL/6-18 using a sample of clinic-referred children but did not use any children outside of a clinical setting. Our findings align with other research identifying limited evidence of factorial structure of the CBCL/6-18 scores, supporting a new conceptual framework of the CBCL/6-18 to provide a more parsimonious model when working with diverse populations. Nevertheless, our findings resulted in the reduction of a large number of CBCL/6-18 items, whereas other researchers did not remove as many items (Mano et al., 2009). The analysis revealed that it is necessary to delete multiple items to ensure overall appropriate fit with our data involving children from low-income communities. Furthermore, we utilized PA (Horn, 1965) to support our results, offering a unique contribution for our study.

Limitations and Future Recommendations

There is a need for research investigating the factor structure of CBCL/6-18 scores with diverse samples of children, in particular children from low-income communities. As with all research, our investigation had limitations. The factor structure results we identified were unique and caution should be taken in interpreting CBCL/6-18. A limitation within our study is related to sampling as our participants were not randomly selected from children attending a Title I elementary school. School personnel and/or parents or guardians referred participants to the study. Furthermore, a large portion of the sample was referred and included in the study due to exhibiting externalizing problems within the clinical or borderline range. Despite the noted limitations, the results support a conceptual framework of the CBCL/6-18 with fewer items to provide a more stringent model when working with diverse populations, specifically children from low-income families. Future research with the CBCL/6-18 should employ random sampling to increase generalizability of the findings. Although the majority of the participants were from an ethnic minority sample, 30% of the participants were White/Caucasian. Thus, the White/Caucasian portion of the sample is noted as a limitation in this study. In addition, future research may examine additional variables to add to be more inclusive of symptomology of diverse samples. For example, measurement invariance is warranted to test differences within samples such as the demographic factor of race/ethnicity.

There is a need to measure children’s behavioral issues within diverse samples; however, evidence of the validity of CBCL/6-18 scores is questionable. Therefore, our investigation further examined the factor structure of CBCL/6-18 scores with a diverse sample of elementary school students, identifying a poor model fit. The factor structure of the CBCL/6-18 scores using PCA and PA (Horn, 1965) was explored, resulting in a 32-item model with three factors, explaining 41.40% of the variance. As a result, practitioners should interpret CBCL scores with caution when using the instrument with children from diverse backgrounds (i.e., ethnic minority, low-income families). Furthermore, future research is needed to further investigate the results of our 32-item, three-factor CBCL/6-18 measurement model to ensure that appropriate assessment measures are employed when working with children from diverse environments.

Supplemental Material

Figures – Supplemental material for The Factor Structure of Child Behavior Checklist Scores With Elementary School Students Referred to Counseling Within Low-Income Communities

Supplemental material, Figures for The Factor Structure of Child Behavior Checklist Scores With Elementary School Students Referred to Counseling Within Low-Income Communities by Saundra M. Tabet, Mary K. Perleoni, Dalena Dillman Taylor, Viki P. Kelchner and Glenn W. Lambie in Assessment for Effective Intervention

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Saundra M. Tabet

Supplemental Material

Supplemental material is available for this article online at

References

Achenbach

Rescorla

(2001). Manual for the ASEBA school-age forms and profile. Department of Psychiatry, University of Vermont.

Al-Hendawi

Keller

Cloninger

(2016). A psychometric analysis of the Child Behavior Checklist for elementary school children in Qatar. Assessment for Effective Intervention, 41(4), 220–229.

Armsden

Pecora

Payne

Sykawiecz

(2000). An intake profile of youth placed in long-term foster care using the Child Behavior Checklist/4-18. Journal of Emotional and Behavioral Disorder, 8(1), 49–64.

Bentler

P. M.

(1990). Comparative fit indexes in structural models. Psychological Bulletin, 107(2), 238–246.

Berubé

R. L.

Achenbach

T. M.

(2010). Bibliography of published studies using the Achenbach System of Empirically Based Assessment: 2006 edition. Research Center for Children, Youth, & Families, University of Vermont.

Blodgett

Lanigan

J. D.

(2018). The association between adverse childhood experience (ACE) and school success in elementary school children. School Psychology Quarterly, 33(1), 137–146.

Bruce

E. J.

Waelde

L. C.

(2008). Relationships of ethnicity, ethnic identity, and trauma symptoms to delinquency. Journal of Loss and Trauma, 13(5), 395–405.

Byrne

(2016). Structural equation modeling with AMOS: Basic concepts, applications, and programming (3rd ed.). New York, NY: Routledge.

Cattell

R. B.

(1966). The scree test for the number of factors. Multivariate Behavioral Research, 1(2), 245. doi:10.1207/s15327906mbr0102_10

10.

Costello

Osborne

(2005). Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis. Practical Assessment, Research, and Evaluation, 10(7), 1-9.

11.

Dedrick

R. F.

Tan

T. X.

Marfo

(2008). Factor structure of the Child Behavior Checklist/6-18 in a sample of girls adopted from China. Psychological Assessment, 20(1), 70–75. https://doi.org/10.1037/1040-3590.20.1.70

12.

Dumenci

Erol

Achenbach

T. M.

Simsek

(2004). Measurement structure of the Turkish translation of the Child Behavior Checklist using confirmatory factor analytic approaches to validation of syndromal constructs. Journal of Abnormal Psychology, 32(3), 335–340.

13.

Enders

C. K.

(2010). Applied missing data analysis. Guilford Press.

14.

Fabrigar

L. R.

Wegener

D. T.

MacCallum

R. C.

Strahan

E. J.

(1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4(3), 272-299. doi:10.1037/1082-989X.4.3.272

15.

Fan

Thompson

Wang

(1999). Effects of sample size, estimation methods, and model specification on structural equation modeling fit indexes. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 56–83.

16.

Gomez

Vance

(2014). Confirmatory factor analysis, latent profile analysis, and factor mixture modeling of the syndromes of the Child Behavior Checklist and Teacher Report Form. Psychological Assessment, 26(4), 1307–1316. https://doi.org/10.1037/a0037431

17.

Hair

J. F.

Black

W. C.

Babin

B. J.

Anderson

R. E.

(2019). Multivariate data analysis (8th ed.). Upper Saddle River, NJ: Pearson.

18.

Henson

R. K.

(2001). Understanding internal consistency reliability estimates: A conceptual primer on coefficient alpha. Measurement and Evaluation in Counseling and Development, 34(3), 177-189.

19.

Henson

R. K.

Roberts

J. K.

(2006). Use of exploratory factor analysis in published research. Educational and Psychological Measurement, 66(3), 393–416.

20.

Horn

J. L.

(1965). A rationale and test for the number of factors in factor analysis. Psychometrica, 30, 179–185.

21.

L. T.

Bentler

P. M.

(1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6, 1–55.

22.

Jacob

B. A.

(2007). The challenges of staffing urban schools with effective teachers. The Future of Children, 17(1), 129–153. https://doi.org/10.1353/foc.2007.0005

23.

Kaiser

H. F.

(1960). Application of electronic computers to factor analysis. Educational & Psychological Measurement, 20(1), 141–151.

24.

Kariuki

S. M.

Abubakar

Murray

Stein

Newton

C. C.

(2016). Evaluation of psychometric properties and factorial structure of the pre-school Child Behavior Checklist at the Kenyan Coast. Child and Adolescent Psychiatry and Mental Health, 10. https://doi.org/10.1186/s13034-015-0089-9

25.

Kline

R. B.

(2015). Principles and practice of structural equation modeling (4th ed.). New York, NY: Guilford Press.

26.

Lambie

G. W.

Solomon

Joe

J. R.

Kelchner

V. P.

Perleoni

M. K.

(2019). A school-based mental health counseling intervention with students in Title I elementary schools. Children & Schools, 41, 161–168. https://doi.org/10.1093/cs/cdz011

27.

Little

R. J.

(1988). A test of missing completely at random for multivariate data with missing values. Journal of the American Statistical Association, 83(404), 1198–1202.

28.

Mano

K. E. J.

Davies

W. H.

Klein-Tasman

B. P.

Adesso

V. J.

(2009). Measurement equivalence of the Child Behavior Checklist among parents of African American adolescents. Journal of Child and Family Studies, 18(5), 606–620.

29.

Moruzzi

Pesenti-Gritti

Brescianini

Salemi

Battaglia

Ogliari

(2010). Clumsiness and psychopathology: Causation or shared etiology? A twin study with the CBCL 6–18 questionnaire in a general school-age population sample. Human Movement Science, 29(2), 326–338. https://doi.org/10.1016/j.humov.2010.01.005

30.

National Institute of Mental Health. (2016). Child and adolescent mental health. https://www.nimh.nih.gov/health/topics/child-and-adolescent-mental-health/index.shtml

31.

Osborne

J. W.

(2013). Best practices in data cleaning. Thousand Oaks, CA: Sage.

32.

Osofsky

Kronenberg

Bocknek

Hansel

T. C.

(2015). Longitudinal impact of attachment-related risk and exposure to trauma among young children after Hurricane Katrina. Child & Youth Care Forum, 44(4), 493–510.

33.

Pallant

(2016). SPSS survival manual: A step by step guide to data analysis using IBM SPSS (6th ed.). McGraw Hill.

34.

Pandolfi

Magyar

C. I.

Dill

C. A.

(2012). An initial psychometric evaluation of the CBCL 6-18 in a sample of youth with autism spectrum disorders. Research in Autism Spectrum Disorders, 6(1), 96–108.

35.

Schumacker

R. E.

Lomax

R. G

. (2016). A beginner’s guide to structural equation modeling. New York, NY: Routledge.

36.

Solomon

Bierbrauer

Harris

Belser

Wheeler

Crunk

Lambie

W. G.

(2016). Experiences of caregivers and students in an elementary school-based mental health counseling practicum program. Journal of Child and Adolescent Counseling, 2(2), 125–142. https://doi.org/10.1080/23727810.2016.1228771

37.

Steiger

J. H.

(1990). Structural model evaluation and modification: An internal estimation approach. Multivariate Behavioral Research, 21, 309–331.

38.

Stevens

(1996). Applied multivariate statistics for social sciences (3rd ed.). Lawrence Erlbaum.

39.

Stewart

S. L.

Leschied

den Dunnen

Zalmanowitz

Baiden

(2013). Treating mental health disorders for children in child welfare care: Evaluating the outcome literature. Child & Youth Care Forum, 42(2), 131–154.

40.

Tabachnick

B. G.

Fidell

L. S.

(2013). Using multivariate statistics (6th ed.). Pearson.

41.

Tucker

L. R.

Lewis

(1973). A reliability coefficient for maximum likelihood factor analysis. Psychometrika, 38, 1–10.

42.

Tyson

E. H.

Teasley

Ryan

(2011). Using the Child Behavior Checklist with African American and Caucasian American adopted youth. Journal of Emotional and Behavioral Disorders, 19(1), 17–26.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.24 MB