Effectiveness of a Trauma-Informed Care Initiative in a State Child Welfare System: A Randomized Study

Abstract

Trauma-informed care (TIC) initiatives in state child welfare agencies are receiving more attention, but little empirical evidence exists as to their efficacy. The purpose of this study was to assess changes in self-reported practices and perceptions of child welfare staff involved in a multifaceted, statewide TIC intervention. Ten child welfare offices were matched and randomized to an early or delayed cohort. Staff were surveyed at Time 1 prior to any intervention, Time 2 postintervention for Cohort 1, and Time 3 postintervention for Cohort 2. The survey covered six domains: trauma screening, case planning, mental health and family involvement, progress monitoring, collaboration, and perceptions of the state’s overall system performance. Linear mixed modeling assessed the effect of the intervention. Cohort by time interaction was significant for three intervention targets. We demonstrate, using a rigorous study design, the mixed results of a multimodal intervention to improve trauma-informed attitudes, practices, and system performance. TIC initiatives must account for complex, dynamic contextual factors.

Keywords

trauma informed child welfare services child PTSD/trauma dissemination/implementation program evaluation

It is well-documented that youth involved in the child welfare system have significantly higher rates of trauma exposure than their counterparts in the general population. Eighty-five percent or more of youth involved with child welfare, and over 90% of those served by juvenile justice, report exposure to trauma (Miller, Green, Fettes, & Aarons, 2011; Rosenberg et al., 2014). Traumas that bring children into the child welfare system include physical or sexual abuse, neglect, and witnessing domestic violence. In addition, child welfare specific experiences, such as removal from their home, being in foster care, or placement disruptions, are often experienced as traumatic. The adverse effects of trauma are well-known; childhood trauma negatively affects brain development, cognitive and academic functioning, and social and emotional development (Nemeroff et al., 2006; Shonkoff, 2012) and increases risk of psychiatric disorders in childhood, adolescence, and adulthood (Felitti et al., 1998; McLaughlin et al., 2012). Youth in child welfare and juvenile justice have disproportionately more mental health problems; 50% of youth in these systems fall in the clinical range for mental health problems, a rate that is 2.5 times higher than the general population (Burns et al., 2004; Casanueva, Smith, Dolan, Ringeisen, & Horne, 2012; Horwitz et al., 2012; Kolko et al., 2010; Landsverk, Burns, Stambaugh, & Rolls-Reutz, 2006). Moreover, children in foster care consume around 29% of total Medicaid spending on behavioral health services (US$8,100 annual per child; Pires, Grimes, Allen, Gilmer, & Mahadevan, 2013).

Given the high prevalence of trauma exposure in child protection and juvenile justice populations and significant costs associated, it is incumbent upon child serving systems, including child welfare, to care for these children in ways that prevent further exposure and mitigate its potentially negative effects. In the past 20 years, there has been burgeoning interest in initiatives at the organizational and system level to increase awareness and responsiveness to the impact of trauma. These are often referred to as “trauma-informed care” initiatives or “TIC” (e.g., Harris & Fallot, 2001; The National Child Traumatic Stress Network, 2015; Substance Abuse and Mental Health Services Administration, 2014). Although these initiatives make intuitive sense and at face value seem to be worthwhile and important undertakings, significant questions remain as to their definition and conceptualization, measurement, impact, and cost effectiveness. As Hanson and Lang (2016) point out, despite a proliferation of TIC initiatives across child serving systems, TIC still lacks clear and standard definitions and consensus on components and measures. Solid evidence of improved system level or child outcomes as a function of TIC is also lacking.

While questions still abound, preliminary steps to better define and study TIC are underway. Hanson and Lang (2016) mapped the most common definitions of TIC in the literature and identified three primary domains: (1) workforce development (training, awareness, and secondary traumatic stress), (2) trauma-focused services (evidence-based practices and standardized screening and assessment practices), and (3) organizational environment and practices (collaboration, service coordination, safe physical environment, written policies, and defined leadership).

A few studies have evaluated outcomes of TIC at the level of child welfare staff. Two have shown greater overall system readiness and capacity to being more trauma-informed following a statewide TIC intervention (Bartlett et al., 2016; Lang, Campbell, Shanley, Crusto, & Connell, 2016). Others have demonstrated improved trauma-informed knowledge, attitudes, and skills as a function of staff training (Conners-Burrow et al., 2013; Fitzgerald et al., 2015; Kerns et al., 2016; Kramer, Sigel, Conners-Burrow, Savary, & Tempel, 2013). One study found increased collaboration across child welfare and mental health and preliminary evidence for improved referrals and child outcomes for those receiving evidence-based treatment services (Bartlett et al., 2016).

Despite these promising early findings, these studies highlight a multitude of challenges associated with implementation of TIC initiatives within state child welfare agencies, particularly surrounding trauma screening (Kerns et al., 2016; Lang, 2017). Moreover, all of the above studies have significant limitations. None have utilized study designs that incorporate control conditions or randomization methods, thereby limiting conclusions that may be drawn. Most offer only preliminary findings prior to completion of multiyear initiatives or offer findings of only one element of TIC (e.g., workforce training). Few of these studies have evaluated practice change at the individual level (e.g., frequency of screening, trauma-informed case planning). Moreover, no prior studies have included trauma-informed system change targeted at both child protection and juvenile justice services (JJS).

The purpose of our study is to rigorously examine, using a randomized, matched-pairs, crossover design, whether a 5-year, multipronged, statewide TIC initiative in a child welfare agency changed trauma-informed attitudes, skills, and behaviors, and perceptions of system performance related to trauma among child welfare staff. We also aim to provide recommendations to promote future rigorous research related to trauma-informed interventions.

Method

Our study was conducted as part of The New Hampshire Partners for Change Project, a 5-year initiative funded by the Department of Health and Human Services Administration for Children and Families, Children’s Bureau from 2012 to 2017, and one of 20 grantees nationwide tasked with implementing TIC in their state or tribal child welfare systems. The grant was awarded to Dartmouth College, in partnership with the NH Division for Children, Youth and Families (DCYF). The overall aim was to improve the social–emotional well-being and developmentally appropriate functioning of children and families served by NH DCYF through TIC strategies. Prior to the start of this Project, Child Protective Services (CPS) and JJS were merged into one State Division (DCYF). Because they shared common leadership, and there was strong interest in unifying values and practices within these two services, we decided to implement our TIC activities in both child protective and JJS systems.

The specific objectives of the Partners for Change initiative included the following: (1) installation of universal screening for trauma exposure, posttraumatic symptoms, and well-being needs of all adjudicated children and youth in child protective and JJS, (2) data-driven case planning informed by trauma screening results, (3) enhanced progress monitoring through rescreening and increased coordination between child welfare and mental health sectors, (4) increased trauma-focused competencies among child welfare staff, (5) increased collaboration between child welfare and community-based behavioral health services, (6) psychotropic medication monitoring, (7) use of evidence-based trauma treatments by mental health providers, and (8) service array realignment strategies. We implemented the project statewide across 11 DCYF district offices (DOs) and in the associated mental health agencies that serve these DOs. In this article, we focus on the activities and outcomes specific to the child welfare system and Objectives 1–5 to assess the degree to which the intervention changed TIC attitudes, skills, behaviors, and perceptions of DCYF staff.

We used a randomized, matched-pairs, crossover design to deliver and rigorously evaluate the intervention, and ensure that all DOs received the intervention. Figure 1 provides an overview of the intervention, study design, data collection points, and time line. Dartmouth’s Committee for the Protection of Human Subjects approved the intervention and study protocols.

Figure 1.

Intervention and study design.

Intervention Description

The project was guided by a core leadership team comprised of high-level DCYF administrators, including the agency director, the Dartmouth project team (principal investigator, coinvestigator, project coordinator), and the evaluation team. The leadership team met monthly throughout the life of the project to plan and guide all activities, remove barriers to implementation, review and change policies to support new practices, advise on evaluation activities (e.g., timing of surveys), and discuss evaluation findings. The project was introduced to all DCYF field staff by the agency director, field administrators, and DO supervisors.

In the first year of the project, we conducted a comprehensive needs and readiness assessment, which included interviews and focus groups with a diverse mix of stakeholders. Stakeholders included children, families, advocacy groups, and leaders and staff from all sectors of state child welfare, juvenile justice, and community behavioral health services. We used the needs assessment and preproject (2011–2012) results from a statewide administration of the Chadwick Trauma System Readiness Tool (TSRT) to create the initial version of the intervention and measures. We then pilot-tested the intervention activities and related evaluation methods (e.g., training evaluation tools; survey) in one DO to assess the feasibility and utility of the processes and measures. We modified the intervention and measures based on the results of the pilot.

Following pilot testing, we matched the remaining 10 DOs based on case mix, size, and geography (e.g., rural vs. urban) into five pairs. We then randomly assigned each DO in each pair to either an Early Intervention Group (Cohort 1) or Late Intervention Group (Cohort 2). Prior to the start of intervention, we collected baseline data (Time 1) from the 10 DOs. We then implemented the intervention in Cohort 1 DOs from October 2014 to June 2015. Based on Cohort 1 evaluation results, we made a few adjustments to the intervention (e.g., training activities) and implemented the intervention with Cohort 2 DOs between October 2015 and June 2016.

Our target audience for the intervention was all DCYF CPS and JJS staff and supervisors in the 10 DOs. Figure 2 outlines the training elements provided to each DO with the intervention, which consisted of the following primary components: (1) installation and implementation of a new, web-based “Mental Health Screening Tool” (MHST) to assess trauma exposure, post-traumatic stress symptoms and overall child well-being; (2) 3 monthly training workshops focused on basic principles of TIC and its application to child protection and juvenile justice practice, and training and practice in using the MHST; (3) embedded consultative support within each DO for ½- to 1-day per week for 3 months following training to provide guidance to staff in screening implementation, interpreting MHST results, and communicating results to families and behavioral health providers; (4) identification of and advanced training to one to three staff champions (“trauma specialists”) in each DO to support and maintain TIC practices; and (5) subcommittee work to review and implement supportive system-level processes and policies.

Figure 2.

Training elements per Division for Children, Youth and Families district office.

Prior to launching the training series in each DO, the project coordinator met with DO leaders to introduce the intervention and training objectives and to determine optimal scheduling for trainings. This included attending to staffing and caseloads at each DO and identifying any local issues that needed to be considered during implementation. Based on these meetings, trainings and other DO-based activities were scheduled at each of the five DOs in each cohort in a staggered approach but within the allotted intervention period for each cohort (see Figure 1).

During the posttraining consultation period, each DO identified one to three CPS or JJS direct service staff to become trauma specialists. Trauma specialists were tasked with providing ongoing local support for staff around implementation of trauma-informed practices, with particular attention to implementation and interpretation of screening. The group of trauma specialists met monthly to develop skills and build processes to provide local support to other staff in their DOs in screening, case planning, and progress monitoring practices. Trauma specialists also received training in a range of additional topics identified by the group and project team (e.g., complex trauma, secondary traumatic stress, and principles of evidence-based mental health treatments for trauma/attachment) based on issues affecting staff in the field.

In addition to training and posttraining consultation, multiple subcommittees were created to change system-level processes and policies related to TIC practices and policies. Project staff worked with DCYF administrators and field staff to establish formal protocols and policies for integrating the new screening, case planning, and progress monitoring practices within the larger operational structure of the child welfare system. By Year 5, these subcommittees were wrapped into one trauma steering committee. In addition, the leadership of ongoing trauma-informed child welfare practices (including sustained use of the MHST) was fully migrated to DCYF.

Measures

We selected six TIC domains to measure in this study: (1) trauma screening, (2) case planning, (3) referrals for trauma-focused treatment and involving families in meeting behavioral health needs of child, (4) progress monitoring, (5) collaboration between DCYF staff and mental health providers, and (6) system-level TIC practices. The first five domains focused primarily on skills and behaviors of child welfare staff. The sixth domain focused on staff perceptions of the state’s child serving system performance on core TIC practices (e.g., screening, referrals, trauma-focused treatment provision, and progress monitoring).

No single instrument existed to measure these six domains at the start of this study particularly in relation to behaviors and skills. Given this, we developed a new survey based on a comprehensive review of TIC-related literature, input from content experts, existing scales, and our baseline needs assessment. We adapted one existing scale and paid particular attention to items from a previously constructed measure to include in the survey. The adapted scale came from Gittell’s Relational Coordination Scale (Gittell, 2002; Gittell et al., 2000; Gittell, Seidner, & Wimbush, 2010; Havens, Vasey, Gittell, & Lin, 2010) to measure collaboration and coordination between child welfare and mental health. Items from the TSRT (https://ctisp.org/) provided additional guidance for us in terms of TIC practices to include.

We pilot-tested the survey with seven staff in the pilot DO to assess the clarity, comprehension, and relevance of the items. The final survey included items related to each of the six domains noted above and demographic information. Table 1 provides a brief description of the items in each domain, the number of items included, and an example for each. All the items were rated on a 5-point Likert-type scale. Demographic variables included job role (JPP or CPS), percentage of time in a typical month working directly with children and families, and years in current position at DCYF. We also included a measure of workplace stress because literature suggests that stress in the workplace can impact engagement and satisfaction among child welfare staff (Boyas, Wind, & Ruiz, 2015; Johnco, Salloum, Olson, & Edwards, 2014; Kim & Kao, 2014). The demographic variables and workplace stress measures were included as controls in our models to account for additional chance imbalances between the two cohorts and to statistically control for the levels of workplace stress.

Table 1.

Trauma-Informed Care Survey Domains.

Scale Label	Scale Description (Sample Item)	Cronbach’s α
Scale Label	Scale Description (Sample Item)	T1	T2	T3
Trauma screening (4 items)	Frequency and proficiency of screening for current or past trauma, abuse and/or neglect, and impacts to child development (“Thinking about your current work with children, youth and families at DCYF, how often do you screen for impacts that grief and loss may have had on child’s development?”)	.83	.83	.81
Case planning (3 items)	Frequency of case planning and communication of screening results to other providers (“When first screening children and youth for trauma, abuse and/or neglect, how often do you use screening results to develop the child/youth’s case plan?”)	.83	.89	.78
Family involvement (6 items)	Frequency and proficiency related to making referrals for trauma-informed services, and involving birth, foster, relative, and/or adoptive family(ies) in meeting behavioral health needs (“Thinking about your work with children, youth and families, please rate your current level of proficiency or skill with making referrals for trauma-informed mental health services for children?”)	.71	.73	.82
Progress monitoring (3 items)	Frequency of rescreening, updating case plan, and communication with mental health provider about child’s progress and ongoing behavioral health needs (“For children receiving ongoing services from DCYF, how often do you make updates to the child’s case plan?”)	.69	.69	.68
Collaboration (9 items)	Attitudes related to shared vision, information-sharing, mutual respect and coordination of efforts between child welfare and mental health providers (Agreement scale; “DCYF and mental health providers work together to promote child well-being.”)	.88	.86	.86
System performance (7 items)	Attitudes about the state’s overall child-serving system in carrying out a range of trauma-informed practices (Quality rating scale; “How is the System currently doing at providing evidence-informed treatments for children/youth with past or present trauma?”)	.90	.90	.84
Work stress (5 items)	Frequency of feelings of stress or positive challenge by working with DCYF children and families; frequency of having adequate support from coworkers and supervisors, and from being able to manage workloads. (“During my work with children and families, I feel emotionally drained”)	.77	.83	.79

Note. DCYF = Division for Children, Youth and Families.

We demonstrated the psychometric properties of the survey items and domains in several stages. First, we performed several initial exploratory factor analyses using data from Time 1. Based on these results, we conducted a final exploratory factor analysis on the items that comprised the study variables. All of the items loaded primarily on their theoretical factors and only a few items displayed high cross-factor loadings or had communalities that were lower than .50. With few exceptions, the results supported our hypothesized factor structure. As such, we created scales to use in the mixed effects models. To establish the reliability of our scales, we conducted Cronbach’s α on the resulting scales and across the three waves of data collection. Table 1 provides information about the six scales, including a description of the content, representative items, and scale reliability. Cronbach’s αs reflected adequate reliability indices that remain consistent across the three data collection points.

Study Sample and Procedures

Our survey sample included all CPS and JJS staff targeted for the intervention. We obtained staff names, e-mail addresses, and location at each time point from DCYF’s central office and compared them to training attendance lists to capture all potential respondents. At each administration, we asked DCYF leaders to introduce the survey and its purpose and encourage all staff to respond. We used Qualtrics (Provo, UT) to administer the survey online over a 6-week period at each time point (see Figure 1) and sent individual survey links followed by four reminders. The survey provided an explanation of the project, survey aims, and confidentiality assurances. Consent was presumed by participation, and no incentives were used due to state regulations prohibiting employees from receiving nonsalary money or gifts.

Missing Data Imputation

Patterns of missing data across the variables were typical for studies conducted over extended periods with nonresponse rates at the start of the study (Wave 1) averaging 8% and increasing to an average of 41% at Wave 2, and 57% by the end of the study (Wave 3). Participants, on average, were missing responses on 31% of study variables (n = 8 of 25). Across the study, the amount of missing data was similar between cohorts (F = 2.43, p = .1214), between levels of respondent’s years in current position (F = 2.50, p = .0628), and time spent with families (F = 0.92, p = .4528). Across the six outcome variables, missing data patterns were comparable and the increases in missing responses over time occurred at similar rates.

To retain the information from study participants who responded at Wave 1, we conducted imputation for missing data using an expectation–maximization algorithm (Dempster, Laird, & Rubin, 1977; Schafer, 1997). We created 50 imputed data sets including the items for each construct and cohort membership as auxiliary correlates in the imputation procedure to improve missing data accuracy. The results were pooled over the 50 imputations according to recommendations by Rubin (1987).

Analyses

Descriptive statistics and frequencies for demographic characteristics by cohort were examined using χ² tests and correlated means t test. Pearson product moment and point-biserial correlations were conducted for both cohorts at Time 1 and Time 3. Linear mixed models (LMM) were used to determine the effect of the intervention on each of the six outcome variables. We conducted all analyses using SAS Version 9.4 for Windows (SAS Institute Inc., 2013) and generated all graphs using Excel™.

The models included two fixed factors: cohort (early vs. late intervention) and time (three survey time points) and an interaction between cohort and time. Each statistical model also included the control factors of job role, time spent directly with clients, years in position, workplace stress, and a missing data index to account for the effect of imputation. Participants were nested within their matched pair to account for the correlated errors associated with clustering. The control variables and person-level intercepts were included in the model as random effects.

We identified four post hoc contrasts of interest between the adjusted mean scores of the outcome variables and used the significance tests generated from Proc Mianalyze (SAS Institute Inc., 2013) in conjunction with the Holm–Bonferroni method (Holm, 1979) to maintain an overall familywise Type I error rate of 0.05 (two-tailed). The four contrasts were as follows: between cohorts preintervention at Time 1 to show the cohort equivalence on the measures, between Cohort 1 postintervention and Cohort 2 preintervention at Time 2 to show the cross-cohort effect of the intervention, and within each cohort pre- and postintervention to show the treatment effect. For Cohort 1 (early intervention), this contrast was between the first two time points and for Cohort 2 (late intervention), the contrast was between the last two time points.

Results

Survey Respondents

Of the 372 DCYF staff at the 10 intervention DOs, 191 (51.3%) responded to the survey at Time 1 (preintervention). There were no statistically significant differences in response rates between respondents and nonrespondents at Time 1 on cohort membership, work location, or between matched pairs. Because the focus of this analysis is on the intervention targeted to staff practices with children and families, we only included responses from the 157 individuals who reported working directly with children, youth, and families in any part of their job. An additional 12 individuals were excluded from the analysis due to missing responses on all of the survey items included in this study. Our final data set included responses from 145 participants completing Time 1 surveys, 77 who were in the early intervention cohort and 68 who were in the late intervention cohort. We followed these 145 participants across the duration of the study and imputed data where it was missing at Time 2 and Time 3.

At Time 1, the cohorts were similar across job category, years in current position, time spent with clients, and job description/roles (see Table 2), but respondents in Cohort 2 reported greater workplace stress than respondents in Cohort 1. Over 60% of the participants in both cohorts worked within CPS and another 30% worked within JJS which is consistent with the overall staff distribution at DCYF. Most respondents had worked in their current positions 4 or more years, and over 60% spent more than half of their time working directly with clients in a given month. Respondents were largely in assessment, family service, and probation and parole officer roles; the three main staff groups targeted with the intervention. See Table 2 for participant characteristics.

Table 2.

Respondent Characteristics.

Respondent Characteristics	Cohort 1 (n = 77)		Cohort 2 (n = 68)		Comparison
Respondent Characteristics	%	n	%	n	Comparison
Job category					χ²(2) = 0.10	p = .95
District, division, bureau, or department leadership	5.56	4	4.62	3
Child protection services	63.89	46	63.08	41
Juvenile justice services	30.56	22	32.31	21
Missing		5		3
Years in current position					χ²(3) = 2.61	p = .46
Less than a year	17.46	11	20.69	12
1–3 Years	17.46	11	27.59	16
4–9 Years	33.33	21	24.14	14
More than 10 years	31.75	20	27.59	16
Missing		14		10
Time spent with clients in a month					χ²(4) = 3.90	p = .42
0–10%	1.3	1	7.35	5
11–25%	9.09	7	5.88	4
26–50%	22.08	17	20.59	14
51–75%	27.27	21	29.41	20
76–100%	40.26	31	36.76	25
Missing		0		0
Job description					χ²(5) = 2.15	p = .83
Assessment	28.77	21	23.08	15
JJ probation and parole officer	27.4	20	29.23	19
Family service	20.55	15	26.15	17
Permanency/resource	6.85	5	9.23	6
Supervisor/field administrator	9.59	7	9.23	6
Other	6.85	5	3.08	2
Missing		4		3
	Mean		Mean		Comparison^a
Workplace stress	2.36		2.63		t = −2.40	p = .02

^a Based linear mixed model of the Workplace Stress Scale using Time 1 data.

Correlations

Correlations among the six scales used in this study and selected demographic measures are presented in Table 3. At Time 1, most of the scales were moderately intercorrelated, suggesting that the behaviors and attitudes associated with TIC are part of a broader grouping of TIC skills and knowledge. Attitudes about collaboration and system performance were highly related (r = .60, p < .001). Case planning practices were positively, though only modestly, related to frequency of progress monitoring (r = .38, p < .001) and trauma screening (r = .35, p < .001). The relationship between MH referral/family involvement practices and attitudes toward system performance was somewhat weaker but still significant (r = .33, p < .001). Progress monitoring was positively linked to MH referral/family involvement practices (r = .32, p < .001). Trauma screening practices were modestly associated with more positive attitudes toward system performance (r = .32, p < .001) and higher MH referral/family involvement practices (r = .33, p < .001). At Time 3 the correlations were slightly lower between the scales, although the moderate associations between study variables persisted.

Table 3.

Correlations Between Demographic Variables and Scales at Time 1.

Scale	1	2	3	4	5	6	7	8	9	10
1. Case planning
2. Collaboration	.178*
3. Progress monitoring	.377***	.253**
4. System performance	.333***	.597***	.247**
5. Family involvement	.325***	.127	.316***	.205*
6. Trauma screening	.353***	.069	.219**	.324***	.334***
7. Work-related stress	.042	.054	−.061	.119	.112	.106
8. Years in current position	.053	−.009	.081	−.126	−.020	−.015	−.080
9. Time with clients	.151^†	.088	.052	.064	.089	.152^†	−.052	.027
10. Job in juvenile justice	.157^†	.046	.209*	−.059	−.135	.044	−.309**	.331***	.164^†
11. Job in child protection services	−.065	−.043	−.157^†	.092	.184*	.016	.364***	−.368***	−.040	−.894***

Note. N = 145.

^†p < .10. *p < .05. **p < .01. ***p < .001.

Table 4.

Correlations Between Demographic Variables and Scales at Time 3.

Scale	1	2	3	4	5	6	7	8	9	10
1. Case planning
2. Collaboration	.068
3. Progress monitoring	.293**	.161
4. System performance	.116	.472***	.203*
5. Family involvement	.194†	.105	.229*	.077
6. Trauma screening	.254*	−.019	.270**	.076	.294**
7. Work-related stress	.021	.045	−.015	.123	.000	−.032
8. Years in current position	.027	−.003	.158†	−.083	.083	.003	−.127
9. Time with clients	.056	−.017	.084	.058	−.015	.101	−.025	.027
10. Job in juvenile justice	.133	.0300	.325***	.069	.010	.193*	−.194†	.331***	.164†
11. Job in child protection services	−.067	−.051	−.206*	−.061	.049	−.099	.249*	−.368***	−.040	−.894***

Note. N = 145.

†p < .10. *p < .05. **p < .01. ***p < .001.

Linear Mixed Modeling

We used linear mixed modeling (LMM) to examine the effect of the intervention on the six outcome variables. Because our interest was in the treatment by time interaction and the model covariates were included as controls, we only present the adjusted means for the two cohorts across the three time points, along with the contrasts that reached statistical significance after the Holm–Bonferroni adjustment (see Table 5). Full model results are available from the first author.

Table 5.

Post Hoc Comparisons of Scale Averages (Least Squares Means) by Cohort and Time (Wave).

Scale	Cohort 1			Cohort 2
Scale	T1	T2	T3	T1	T2	T3
Collaboration	3.31	3.25	3.40	3.46	3.41	3.50
Family involvement	3.57	3.60	3.40	3.67	3.42	3.27
Progress monitoring	2.34	2.30	2.24	2.39	2.22	2.18
Case planning	3.48	3.56^a	3.60	3.45	2.72^ab	3.65^b
Trauma screening	3.52	3.67^a	3.35	3.63	3.04^ab	3.52^b
System performance	3.45	3.45^a	3.36	3.22	3.05^ab	3.34^b

Note. N = 145. In given row, averages with the same letters differ significantly at least p < .05 after application of Holm–Bonferroni method to control Type I error rate.

Results of the LMM showed significant differences for three of the six outcome variables (see Table 1): initial case planning and communication, trauma screening, and ratings of overall system performance (see Figures 3 –5). The patterns of adjusted means of these variables for the two cohorts were similar.

Figure 3.

Average Case Planning scale score by cohort and time.

Figure 4.

Average Trauma Screening scale score by cohort and time.

Figure 5.

Average System Performance scale score by cohort and time.

The impact of the TIC intervention on trauma-informed initial case planning and communication was mixed. This scale included 3 items asking staff how often they use trauma screening results to develop case plans and how often they communicate screening results to a child’s MH or other clinical providers. Preintervention, the reported frequency of these case planning practices was similar between the cohorts (t = 0.13, p = .896).

Postintervention (Time 2), Cohort 1 staff reported average frequency of case planning similar to their preintervention levels ( $\bar{x}$ = 3.48 vs. 3.56, t = −0.52, p = .601) but significantly higher than the frequency of case planning by Cohort 2 staff ( $\bar{x}$ = 3.56 vs. 2.72, t = 3.60, p = .0003). The intervention had a greater effect on case planning practices of respondents in Cohort 2, with significant increases in average frequency ratings from 2.72 to 3.65 (t = −5.23, p < .0001).

The pattern of adjusted means of the Trauma Screening Scale shows that the effect of the intervention on frequency of and skills in trauma screening was similar to that of case planning. At Time 1, trauma screening practices were similar between the cohorts (t = −0.61, p = .543). At Time 2, reported levels of trauma screening increased slightly in Cohort 1 offices, but not significantly (t = −1.09, p = .276), yet they were significantly higher than screening levels reported by staff in Cohort 2 (t = 3.09, p = .0021). After participating in the intervention, Cohort 2 trauma screening frequency and skills increased significantly between Time 2 and Time 3 (t = −2.61, p = .0099).

Perceptions of DCYF’s TIC system performance were similar between the cohorts at Time 1 (t = 1.87, p = .0615). This scale included ratings of the system’s performance on seven TIC practices, such as timeliness of trauma-focused treatments. For Cohort 1, perceptions did not change postintervention (t = 0.06, ns). In Cohort 2, perceptions of system performance dropped between Time 1 and Time 2 and increased significantly at Time 3 (t = −2.61, p = .0097).

Discussion

We demonstrate, using a rigorous study design, the mixed results of a multimodal, multiyear intervention to improve trauma-informed attitudes, practices, skills, and system performance. While we saw significant differences in the TIC domains of self-reported screening, case planning, and overall system performance between Cohorts 1 and 2 at Time 2, the differences result from a decrease in TIC practices for Cohort 2, rather than an increase by Cohort 1. By Time 3, Cohort 2 improved significantly in trauma screening, initial case planning, and perceptions of system performance while Cohort 1 remained steady.

Although we cannot be certain, we hypothesize that Cohort 1 DOs may have been able to maintain TIC practices in a challenging environment because of the intervention. The opioid crisis was just beginning to build momentum in the year that Cohort 1 received the intervention. It may be that the TIC intervention, which provided an infusion of resources, training, and consultation services, buffered Cohort 1 staff from the higher demands on the system. Without this infusion, Cohort 1 may have responded to the crisis similar to Cohort 2.

For Cohort 2, the intervention improved attitudes and behaviors for trauma screening, case planning, and TIC system performance at Time 3. While Cohort 2 was receiving the intervention, the child welfare system was being stretched even more. Perhaps staff in Cohort 2 DOs were particularly receptive to a TIC approach and the additional support provided via the project given the continued opioid crisis and increased numbers of children entering the child welfare system.

In our assessment, the fact that we did not find a more significant effect from this multifaceted, and fairly costly, intervention raises some questions about the effectiveness of comprehensive TIC interventions. Our intervention is among a small number of other large TIC studies in child welfare settings (Bartlett et al., 2016; Kerns et al., 2016; Lang et al., 2016) that targeted activities across all three primary domains of TIC as identified by Hanson and Lang (2016). Our study focused on comprehensive TIC attitudes and practice changes, whereas other studies have focused on changes in system readiness (Lang et al., 2016), or just one domain of TIC (e.g., screening; Kerns et al., 2016). Nevertheless, our findings are consistent with the mixed findings of prior studies and underscore the complexities and challenges around implementation and evaluation of large-scale TIC interventions in child welfare settings.

Our study’s unique contribution is our randomized, cohort design that adequately addressed missing data and revealed the mixed intervention effects over time in groups exposed to the same contextual conditions. For example, if we had not had a control group and were only interpreting Cohort 1 findings, we would have concluded that the intervention had no effect. Similarly, if we were only able to evaluate Cohort 2 postintervention, we would have concluded that the intervention was successful. By comparing cohorts, we see that the findings are not so clear-cut. With this design, we gained a more accurate and nuanced understanding of the effects of a comprehensive TIC intervention in the complex context of child welfare service systems.

Given the strain and reductions in budget that NH and most child welfare state systems have been under for a number of years (Shoenberg, 2009), and the difficulty of making change in complex social service systems (Greenhalgh, Robert, MacFarlane, Bate, & Kyriakidou, 2004), we also contend that our results are not surprising. In the course of our study (2012–2017), we saw significant stressors on NH’s system, including the ongoing opioid crisis, a 36% increase in the number of children entering foster care, high staff turnover, severe workforce shortages, and a cascade of top leadership changes including the governor, the commissioner of HHS, and the director of our Child Welfare Agency. Consequently, it often felt that we were attempting to implement and study an intervention in a system under siege.

The stressors we documented are not unique to NH and are in fact common across other child welfare systems. This suggests that other comprehensive TIC interventions, in other child welfare systems, may have similar mixed results. Does this mean that these types of interventions should not be attempted in the current climate of child welfare? Or that the interventions were underresourced given system challenges and could be more effective, provided more resources were available? A more critical question may be how do we best reduce the impacts of trauma on the children and families served by current child welfare systems while advocating for much needed attention to the severe challenges these systems face?

One important direction future research must take is to better understand which TIC intervention components or specific activities are most effective. For example, it may be preferable to focus predominantly on specific targeted activities such as screening children for trauma and behavioral health symptoms to identify needs and increase access to services rather than targeting more general practices that are not trauma specific (e.g., collaboration). Our findings suggest this may be possible since we saw better outcomes for screening and case planning compared to less trauma specific outcomes such as cross system collaboration. However, comprehensive, multipronged TIC interventions were developed in part due to evidence that demonstrated difficulty obtaining practice change in one narrow area (Department of Health and Human Services, 2012). In other words, general TIC training may be needed to support specific practice changes, but the balance of how much of each is needed is not known.

Despite our mixed results, our staff and leader participants strongly supported adopting a “trauma lens.” They clearly perceived a need to identify and mitigate the effects of trauma. However, TIC interventions may need to be redefined, more targeted, or expanded in new ways given the current climate of competing demands and scarce resources. We wish that there was the political will to fully support child welfare systems and serve vulnerable children and families. Until that point, we must identify approaches and processes that best assist child welfare systems to address trauma with the resources available to them.

Limitations

At the time we developed the instruments utilized in this study, there were few existing instruments which measured outcomes associated with the goals of our project. Given this, we worked hard to identify the key constructs and establish validity evidence for our scales (cf. American Educational Research Association, American Psychological Association, National Council on Measurement in Education, 2014). We established content validity through extensive review of the literature, feedback from experts, and input from stakeholder interviews and pilot participants. Factor analyses confirmed the conceptual framework and provided validity evidence based on the scales’ internal structure. Furthermore, the high reliability indices over three waves of data collection indicate our scales provided robust and precise measurement of the constructs over time. Future work should focus on extending the validity evidence of our scales, including convergent and criterion-related validity.

A related limitation is that most of our data are self-report, and we were unable to establish reliable, objective measures in the course of the project, such as screening rates or referrals. This was primarily due to the state’s poor data systems, which are improving but were not accurate enough to establish, for instance, the population-eligible denominator to use for calculating rates of screens completed. Even so, we used the same measures across both cohorts and three time points which allowed us to systematically examine the same outcomes.

Another limitation of the study is the relatively small sample size and missing data. While we can’t definitively explain the cause of the missing data, we surmise it is due to a myriad of reasons, such as heavy workloads. We were able to reduce the impact of missing data with imputation, which is widely accepted and viewed as more valid than excluding cases with missing data and complete case analysis. Since the results using imputed data were effectively the same as those without imputation, we are confident in the validity of our findings.

Our study examined the intervention as a whole comparing two cohorts which limits our ability to examine the effectiveness of different TIC components. We believe that such a dismantling study is warranted and would be an important next step for future research.

Finally, while we believe that our results can be generalized to other states in light of the commonality of our contextual stressors, we also recognize limitations based on the size and demographics of NH, a small, mostly rural state, with a fairly homogenous White Caucasian ethnic and racial demographic. The small size of NH allowed us to have regular involvement of senior level child welfare administrators in decision-making and messaging for both the implementation and evaluation of our intervention. Moreover, the small number of DOs and centralized nature of our system allowed for more uniformity in TIC activities across DOs. It is unknown if this TIC intervention would be feasible in a larger, more urban state with a diverse ethnic and racial demographic and a larger or decentralized child welfare system.

Conclusion

While TIC within state child welfare and juvenile justice systems has an intuitive and strong appeal, it is costly and requires significant buy in and sustained commitment from overburdened service systems to implement. We posit that additional research is necessary to identify whether certain domains of TIC might be more effective than others, particularly those that are theoretically specific to trauma and have measurable, objective child and family outcomes. Future studies would ideally utilize similarly rigorous designs to continue to advance the field’s understanding of how best to deliver and measure interventions in the complex, and important work, of enhancing child welfare systems.

Footnotes

Acknowledgments

The authors wish to acknowledge the New Hampshire Division for Children, Youth and Families, Eileen Mullen, Catherine Meister, Lorraine Bartlett, Cathleen Yackley, Rebecca Parton, and all of the children, youth, families, provider organizations, and staff who participated.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by grant #90C01099 from the Administration for Children and Families.

References

American Educational Research Association, American Psychological Association, National Council on Measurement in Education. (2014). Standards for Educational and Psychological Testing. Washington, DC: American Educational Research Association.

Bartlett

J. D.

Barto

Griffin

J. L.

Fraser

J. G.

Hodgdon

Bodian

(2016). Trauma-informed care in the Massachusetts child trauma project. Child Maltreatment, 21, 101–112. doi:10.1177/1077559515615700

Boyas

J. F.

Wind

L. H.

Ruiz

(2015). Exploring patterns of employee psychosocial outcomes among child welfare workers. Children and Youth Services Review, 52, 174–183. doi:10.1016/j.childyouth.2014.11.002

Burns

B. J.

Phillips

S. D.

Wagner

H. R.

Barth

R. P.

Kolko

D. J.

Campbell

Landsverk

(2004). Mental health need and access to mental health services by youths involved with child welfare: A national survey. Journal of the American Academy of Child & Adolescent Psychiatry, 43, 960–970. doi:10.1097/01.chi.0000127590.95585.65

Casanueva

C. W. E.

Smith

Dolan

Ringeisen

Horne

(2012). NSCAW II wave 2 report: Child well-being (2012-38). Retrieved from Washington, DC: U.S. Department of Health & Human Services.

Conners-Burrow

N. A.

Kramer

T. L.

Sigel

B. A.

Helpenstill

Sievers

McKelvey

(2013). Trauma-informed care training in a child welfare system: Moving it to the front line. Children and Youth Services Review, 35, 1830–1835. doi:10.1016/j.childyouth.2013.08.013

Dempster

A. P.

Laird

N. M.

Rubin

D. B.

(1977). Maximum likelihood from incomplete data via the EM Algorithm. Journal of the Royal Statistical Society. Series B (Methodological), 39, 1–38.

Department of Health and Human Services, Administration on Children, Youth and Families. (2012). Initiative to improve access to needs-driven, evidence-based/evidence-informed mental and behavioral health services in child welfare. HHS-2012-ACF-ACYF-CO-0279. Washington, DC: FOA from DHHS Administration for Children and Families.

Felitti

V. J.

Anda

R. F.

Nordenberg

Williamson

D. F.

Spitz

A. M.

Edwards

… Marks

J. S.

(1998). Relationship of childhood abuse and household dysfunction to many of the leading causes of death in adults. The Adverse Childhood Experiences (ACE) Study. The American Journal of Preventive Medicine, 14, 245–258.

10.

Fitzgerald

M. M.

Torres

M. M.

Shipman

Gorrono

Kerns

S. E.

Dorsey

(2015). Child welfare caseworkers as brokers of mental health services: A pilot evaluation of Project Focus Colorado. Child Maltreatment, 20, 37–49. doi:10.1177/1077559514562448

11.

Gittell

J. H.

(2002). Coordinating mechanisms in care provider groups: Relational coordination as a mediator and input uncertainty as a moderator of performance effects. Management Science, 48, 1408–1426. doi:10.1287/mnsc.48.11.1408.268

12.

Gittell

J. H.

Fairfield

K. M.

Bierbaum

Head

Jackson

Kelly

… Zuckerman

(2000). Impact of relational coordination on quality of care, postoperative pain and functioning, and length of stay: A nine-hospital study of surgical patients. Med Care, 38, 807–819.

13.

Gittell

J. H.

Seidner

Wimbush

(2010). A relational model of how high-performance work systems work. Organization Science, 21, 490–506. doi:10.1287/orsc.1090.0446

14.

Greenhalgh

Robert

MacFarlane

Bate

Kyriakidou

(2004). Diffusion of innovations in service organizations: Systematic review and recommendations. Milbank Quarterly, 82, 581–629.

15.

Hanson

R. F.

Lang

(2016). A critical look at trauma-informed care among agencies and systems serving maltreated youth and their families. Child Maltreatment. doi:10.1177/1077559516635274

16.

Harris

Fallot

R. D.

(2001). Envisioning a trauma-informed service system: A vital paradigm shift. New Directions for Mental Health Services, 2001, 3–22. doi:10.1002/yd.23320018903

17.

Havens

D. S.

Vasey

Gittell

J. H.

Lin

W. T.

(2010). Relational coordination among nurses and other providers: Impact on the quality of patient care. Journal of Nursing Management, 18, 926–937. doi:10.1111/j.1365-2834.2010.01138.x

18.

Holm

(1979). A Simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6, 65–70.

19.

Horwitz

S. M.

Hurlburt

M. S.

Goldhaber-Fiebert

J. D.

Heneghan

A. M.

Zhang

Rolls-Reutz

… Stein

R. E. K.

(2012). Mental health services use by children investigated by child welfare agencies. Pediatrics, 130, 861–869. doi:10.1542/peds.2012-1330

20.

Johnco

Salloum

Olson

K. R.

Edwards

L. M.

(2014). Child welfare workers’ perspectives on contributing factors to retention and turnover: Recommendations for improvement. Children and Youth Services Review, 47, 397–407. doi:10.1016/j.childyouth.2014.10.016

21.

Kerns

S. E.

Pullmann

M. D.

Negrete

Uomoto

J. A.

Berliner

Shogren

… Putnam

(2016). Development and implementation of a child welfare workforce strategy to build a trauma-informed system of support for foster care. Child Maltreatment. doi:10.1177/1077559516633307

22.

Kim

Kao

(2014). A meta-analysis of turnover intention predictors among U.S. child welfare workers. Children and Youth Services Review, 47, 214–223. doi:10.1016/j.childyouth.2014.09.015

23.

Kolko

D. J.

Hurlburt

M. S.

Zhang

Barth

R. P.

Leslie

L. K.

Burns

B. J.

(2010). Posttraumatic stress symptoms in children and adolescents referred for child welfare investigation. A national sample of in-home and out-of-home care. Child Maltreatment, 15, 48–63. doi:10.1177/1077559509337892

24.

Kramer

T. L.

Sigel

B. A.

Conners-Burrow

N. A.

Savary

P. E.

Tempel

(2013). A statewide introduction of trauma-informed care in a child welfare system. Children and Youth Services Review, 35, 19–24. doi:10.1016/j.childyouth.2012.10.014

25.

Landsverk

Burns

Stambaugh

L. F.

Rolls-Reutz

(2006). Mental health care for children and adolescents in foster care: Review of research literature. Retrieved from http://www.casey.org/Resources/Publications/pdf/MentalHealthCareChildren.pdf

26.

Lang

J. M.

(2017). Trauma screening in child welfare: Lessons learned from five states. Journal of child & adolescent trauma, 10, 405–416. doi:10.1007/s40653-017-0155-y

27.

Lang

J. M.

Campbell

Shanley

Crusto

C. A.

Connell

C. M.

(2016). Building capacity for trauma-informed care in the child welfare system: Initial results of a statewide implementation. Child Maltreatment. doi:10.1177/1077559516635273

28.

McLaughlin

K. A.

Greif Green

Gruber

M. J.

Sampson

N. A.

Zaslavsky

A. M.

Kessler

R. C.

(2012). Childhood adversities and first onset of psychiatric disorders in a national sample of US adolescents. Archives of General Psychiatry, 69, 1151–1160. doi:10.1001/archgenpsychiatry.2011.2277

29.

Miller

E. A.

Green

A. E.

Fettes

D. L.

Aarons

G. A.

(2011). Prevalence of maltreatment among youths in public sectors of care. Child Maltreatment, 16, 196–204. doi:10.1177/1077559511415091

30.

The National Child Traumatic Stress Network. (2015). Creating trauma-informed systems. Retrieved from hhtp://www.nctsn.org/resources/topics/creating-trauma-informed-systems

31.

Nemeroff

C. B.

Bremner

J. D.

Foa

E. B.

Mayberg

H. S.

North

C. S.

Stein

M. B.

(2006). Posttraumatic stress disorder: A state-of-the-science review. Journal of Psychiatric Research, 40, 1–21. doi:10.1016/j.jpsychires.2005.07.005

32.

Pires

Grimes

Allen

Gilmer

Mahadevan

(2013, 12). Identifying opportunities to improve children's behavioral health care: An analysis of medicaid utilization and expenditures. Faces of Medicaid Data Brief. Retrieved from https://www.chcs.org/media/Identifying-Opportunities-to-Improve-Childrens-Behavioral-Health-Care2.pdf

33.

Rosenberg

H. J.

Vance

J. E.

Rosenberg

S. D.

Wolford

G. L.

Ashley

S. W.

Howard

M. L.

(2014). Trauma exposure, psychiatric disorders, and resiliency in juvenile-justice-involved youth. Psychological Trauma: Theory, Research, Practice, and Policy, 6, 430–437. doi:10.1037/a0033199

34.

Rubin

(1987). Multiple imputation for nonresponse in surveys. New York, NY: Wiley.

35.

SAS Institute Inc. (2013). SAS/STAT 13.1 User’s Guide. Cary, NC: Author.

36.

Schafer

J. L.

(1997). Analysis of Incomplete Data. London, England: Chapman and Hall.

37.

Shoenberg

(2009, 10 15). Layoffs cut into human services—Largest department hit the hardest. Concord Monitor. Retrieved from http://www.concordmonitor.com

38.

Shonkoff

J. P.

(2012). Leveraging the biology of adversity to address the roots of disparities in health and development. Proceedings of the National Academy of Sciences U S A, 109, 17302–17307. doi:10.1073/pnas.1121259109S

39.

Substance Abuse and Mental Health Services Administration. (2014). SAMHSA’s Concept of Trauma and Guidance for a Trauma-Informed Approach. HHS Publication No. (SMA) 14-4884. Rockville, MD: Substance Abuse and Mental Health Services Administration.