Engagement in Trauma-Specific CBT for Youth Post-9/11

Abstract

Treatment participation was examined among youth enrolled in an evaluation of cognitive behavioral therapy (CBT) for trauma following the 9/11 World Trade Center disaster. Staff at nine agencies serving a predominantly low-income, ethnically diverse population were trained to deliver CBT and structured engagement strategies. A total of 445 youth ages 5 to 19 were eligible for CBT, and 417 (94%) received at least one treatment session. Pretreatment and treatment show rates and overall dose were examined. Treatment participation rates were higher than those typically reported in community studies of children’s mental health services. Regression analyses indicated variability across sites in treatment show rates with the highest rates at sites where services were delivered in schools. However, sites, demographic factors, and trauma symptoms accounted for a small amount of variance in treatment participation overall. The study suggests that structured engagement strategies, linked to evidence-based treatments, may improve treatment participation for youth.

Keywords

engagement trauma treatment mental health youth

The attacks on the World Trade Center (WTC) Towers on September 11, 2001 had a substantial mental health impact on children in the New York City area. In the aftermath of the attacks, a large-scale epidemiological study of more than 8,000 students in New York City public schools documented elevated prevalence rates of mental health disorders, including increased rates of posttraumatic stress disorder (PTSD; 10.5%), generalized anxiety (12.3%), agoraphobia (15%), separation anxiety (12.3%), depression symptoms (8.4%), and conduct disorder (10.9%; Hoven et al., 2002). Furthermore, more than two thirds of students surveyed indicated having exposure to traumatic events prior to 9/11. Despite these elevated rates of mental disorders, only 19% of students overall and a third of youth with trauma exposure had sought or received any services from a mental health professional inside or outside of school (Hoven et al., 2002). These rates of mental health service use are consistent with the extant research on children’s mental health service use. This article is a correlational analysis of the efforts of nine large New York City child mental health agencies using structured engagement strategies to recruit and engage youth and their parents to participate in treatment as part the largest evidence-based practice (EBP) implementation study of evidence-based cognitive behavioral therapy (CBT) for youth trauma.

In general, research has consistently shown that most youth who require mental health treatment services do not receive them (Burns et al., 1995; Kataoka, Zhang, & Wells, 2002; Kazdin, 1993; Ringel & Sturm, 2001; Stiffman, Earls, Robins, & Jung, 1998; U.S. Department of Health and Human Services [USDHHS], 1999, 2000, 2001). Kataoka and colleagues (2002) estimated that only 21% of children in need of mental health services actually receive them. Even when parents initiate services, community-based mental health programs often report “no-show” rates for first-time intake appointments exceeding 50% (Lerman & Pottick, 1995). Moreover, estimates suggest that between one-half and three-fourths of children with mental health needs either do not engage in treatment or drop out of care prematurely (Kazdin, 1993). The situation is even bleaker for poor, urban minority children. Low-income, urban communities have the highest rates of child mental health needs (as high as 40%; Tolan & Henry, 1996) and the lowest rates of mental health service use (Griffin, Cicchetti, & Leaf, 1993; McKay, Pennington, Lynn, & McCadam, 2001). A recent study by McKay, Lynn, and Bannon (2005) documented that although 72% of youth referred for treatment to an urban community-based mental health clinic received at least one clinic appointment, only 9% of youth were in treatment after 12 weeks. Barriers to mental health service use among low-income, multiethnic, urban children and adolescents is well documented (USDHHS, 2001) and includes concrete, contextual (i.e., community or agency level) as well as psychological obstacles. Literature reviews suggest that lack of time, transportation, lack of child care, presence of community violence, stigma, and long wait-lists are formidable barriers for many families (McKay & Bannon, 2004).

In the field of childhood trauma specifically, service use tends to be low in spite of well-documented effects of traumatic events on child mental health (see Gurwitch, Sitterle, Young, & Pfefferbaum, 2002 for review). Following the Oklahoma City bombing in 1995, only 6.8% of the total sample of middle and high school students reported having contact with counseling support (Gurwitch et al., 2002). Universal screening of children in Oklahoma City revealed that a large percentage of children who manifested posttraumatic stress (PTS) symptoms had never had contact with a mental health professional (Gurwitch et al., 2002). No correlation was found between PTS symptoms and help-seeking for children with the highest levels of PTS, suggesting that children with the greatest need for mental health services were not receiving treatment (Pfefferbaum et al., 2003). Among parents surveyed in Manhattan 2 months following the 9/11 attacks, only 22% indicated their children received counseling related to the disaster (Stuber et al., 2002). Children who were male, had parents with symptoms of PTSD, and had at least one sibling living at home were more likely to receive counseling. Postdisaster service use was also predicted by manifestations of severe or very severe PTS; however, a large proportion of children with severe or very severe PTS reactions received no services after the attacks (Fairbrother, Stuber, Galea, Fleischman, & Pfefferbaum, 2003). In sum, the available research suggests that children exposed to traumatic events have poor access to mental health services in general, much less access to specialty mental health care or evidence-based trauma treatment.

A burgeoning evidence base on effective services reviewed in numerous reports (Burns & Hoagwood, 2004; Hoagwood, Burns, Kiser, Ringeisen, & Schoenwald, 2001; USDHHS, 1999; Weisz, Hawley, Pilkonis, Woody, & Follette, 2000), has combined with a public health urgency to accelerate implementation of evidence-based treatments (EBTs) in community settings (see public policy reports “Bridging Science and Service” and “Translating Behavioral Science Into Action”; National Institute of Mental Health [NIMH], 1999, 2000). Consequently, integrating empirical knowledge into real-world practice settings rapidly while increasing access and use of services, particularly in underserved communities, is one of the most significant public health issues in the field of children’s mental health services (USDHHS, 1999, 2000, 2001).

Numerous studies have identified family-level barriers to and predictors of treatment participation including demographic characteristics such as race, ethnicity, socioeconomic status, and gender (Kataoka et al., 2002); insufficient insurance coverage (Kataoka et al., 2002); and other obstacles such client beliefs and distance to services (Chow, Jaffee, & Snowden, 2003; Fortney, Rost, Zhang, & Warren, 1999; Hines-Martin, Brown-Piper, Kim, & Malone, 2003). Research suggests that urban minority families living in poverty are confronted with additional obstacles to help-seeking such as lack of knowledge about services (McKay & Bannon, 2004), stigma, distrust of professionals, and lack of social support (Hoagwood et al., 2001; Owens et al., 2002; McKay & Bannon, 2004; Tolan, Hanish, McKay, & Dickey, 2002). The plethora of obstacles that children and families face in accessing effective services, even in those rare circumstances when they are available, suggest a strong need to pay greater attention to methods for engaging and retaining children and adolescents in community-based mental health treatments. However, attempts to engage children and their parents into mental health services appear to be relatively cursory in real-world settings. Research suggests that strategies to engage parents and youth into mental health services typically involve making a circumscribed number of attempts to contact the families before new referrals are dropped (Coatsworth, Santisteban, McBride, & Szapocznik, 2001). In an early study of systematic engagement strategies, Szapocznik and his colleagues (1988) conducted informal surveys and described “engagement as usual,” generally, as therapists’ attempts to empathize and be supportive of callers’ attempts to seek services. Increased rates of treatment participation clearly require more systematic and rigorous approaches to engage children and their parents into services.

Fortunately, ongoing explorations by researchers to improve engagement and prevent premature termination have yielded promising interventions. A growing body of evidence has shown that systematic and intensive telephone and first interview engagement interventions implemented with youth and their families can increase service use substantially (McKay, McCadam, & Gonzalez, 1996; McKay, Nudelman, McCadam, & Gonzalez, 1996; Szapocznik et al., 1988). The success of these engagement interventions is likely the result of establishing a positive initial working alliance with the caregiver and developing strategies to facilitate the attendance of all family members at the intake appointment (Szapocznik et al., 1988). McKay and colleagues have over a decade of studies on telephone and first interview structured engagement strategies within urban child mental health settings (McKay, McCadam, et al., 1996; McKay, Nudelman, et al., 1996; McKay, Stoewe, McCadam, & Gonzales, 1998). Results have indicated that application of these strategies improves attendance rates by 30% at initial clinic appointments and increases families’ length of stay in services.

Engagement interventions applied during the first phone contact or first interview by clinic staff or clinicians are generally brief. Specific components of the engagement interviews include clarifying the need for therapy from the client’s perspective, promoting caregiver investment in treatment, eliciting potential obstacles to attendance, and engaging in problem solving to address potential barriers to attendance. These techniques can be flexibly applied by clinic staff and clinicians mainly during the early phases of the treatment continuum. For example, they can be used by clinic staff during the first phone contact to increase initial engagement in pretreatment visits for intake and assessment or by clinicians on first interviews with parents or children to establish a strong working alliance for therapy.

The present study is a correlational analysis of treatment participation of youth and caregivers recruited into an open trial of CBT for traumatized children and adolescents following the 9/11 disaster at the WTC. Given the known challenges of engaging families in mental health services in highly stressed urban settings as well as the particular psychological barriers posed by trauma victimization, clinic staff at participating sites were trained on structured engagement strategies (McKay et al., 1998). Importantly, this engagement training augmented intensive training and consultation on two evidence-based CBTs for trauma: one for children (Cohen, Mannarino, & Deblinger, 2002) and one for adolescents (Layne, Saltzman, Pynoos, & Steinberg, 2002). A thorough description of the rationale for integrating engagement training with evidence-based CBT for trauma is described in more detail elsewhere (Child and Adolescent Trauma Treatments and Services [CATS] Consortium, 2007). The focus in this study is on the youth recruited into the active treatment arm of the study—those who were eligible to receive the evidence-based CBT for trauma. Although the study involved training and supervision of staff on CBT and engagement strategies, it was an implementation feasibility evaluation, not a controlled clinical trial. Consequently, exclusion criteria were minimal, and all of the CBT treatments and engagement strategies were implemented in real-world practice settings with real-world clinicians, clinic staff, and their supervisors. Gold standard checks for fidelity via audiotaped or videotaped interactions were not possible, although providers systematically tracked treatment attendance. Thus, we were able to assess child and adolescent mental health need and monitor treatment participation and outcomes under conditions that closely approximated those experienced in routine care.

The primary purpose of this study was to examine levels and predictors of treatment participation across nine sites that participated in the open trial. We defined treatment participation as show rates for pretreatment (e.g., intake and evaluation) and treatment visits and overall “dose” or number of treatment sessions. Show rates were defined as the ratio of pretreatment and treatment visits attended to the total number of appointments made to reflect the rate of attendance in services. We differentiated among these three indicators of treatment participation (i.e., pretreatment, treatment visits, and dose) because each represents a distinct form of engagement in or compliance with treatment (Daley & Zuckoff, 1999; Staudt, 2007). Missed pretreatment visits can be an indicator of low child or family motivation for help, low need for services, or the existence of clinic-level barriers such as heavy paperwork burdens or long waits. Missed treatment visits can reflect problems in the therapeutic relationship, ongoing motivation for change, or can reflect simple anxieties or avoidance of addressing key therapeutic goals. Overall, dose is a strong indicator of the degree to which therapeutic targets or components of treatment are delivered. It should be noted that our examination of treatment participation did not include other critical aspects of participation such as adherence to treatment protocols or completion of homework assignments.

Most treatment outcome studies focus on dose alone, but the exploration of different types of treatment participation along the service continuum is important for understanding inefficiencies in service delivery. Our goal was to explore engagement in such a way as to identify challenges and facilitators to engagement in the context of an open trial of EBTs under highly naturalistic conditions in “real-world” practice settings, including school-based and outpatient clinics across ethnically diverse communities. Thus, another purpose was to examine differences in treatment participation across sites relative to key demographic factors and mental health symptomatology in the belief that the location in which services are offered and delivered can have a great impact in service participation (Annunziato et al., 2008; Atkins et al., 2006; Catron & Weiss, 1994).

Method

Participants

After the 9/11 terrorist attack of the WTC, the New York State Office of Mental Health (NYSOMH) received funding from the Substance Abuse and Mental Health Administration to implement and evaluate evidence-based trauma treatments for children and adolescents affected by the WTC disaster. CBT was the treatment approach of choice in the project. Through a competitive proposal process, nine agencies were selected to participate in the project, which came to be known as the CATS Consortium. The CATS sites represented community, academically affiliated, and medical institutions providing community-based mental health services. Services were provided in a variety of service settings, including outpatient and school-based mental health clinics licensed by NYSOMH. Table 1 presents the demographic characteristics of the sample across the nine sites. Overall, the sample was largely non-White (89%), but sites differed on race/ethnicity as well. At all but one (Site 3), Hispanics represented the highest percentage of youth in treatment, and four sites (Sites 5, 6, 7, and 9) were overwhelmingly Hispanic. Sites also differed in terms of poverty status. Not surprisingly, the same three sites with large percentages of Hispanic/Latino participants were also largely low income. Two of these program sites (Sites 5 and 6) served the same largely low-income Dominican neighborhood, and the third (Site 7) served a low-income neighborhood of Hispanics from mixed nationalities.

Table 1.

Demographic Characteristics and Trauma Symptom Scores by Sites

CATS Site	n	Age M (SD)	% Female	% WH	% AA	% HL	% OT	% <15K^a
Site 1	65	14.91 (2.14)	71	12	22	34	32	18
Site 2	33	11.24 (3.22)	58	6	36	36	21	48
Site 3	40	11.02 (3.05)	65	50	15	20	15	23
Site 4	52	11.87 (3.89)	64	23	12	39	27	34
Site 5	21	8.76 (1.34)	52	0	5	86	10	80
Site 6	60	12.05 (3.22)	42	0	2	97	2	83
Site 7	95	9.60 (1.41)	41	3	0	88	8	72
Site 8	51	11.55 (3.19)	65	4	41	43	12	47
Site 9	28	11.82 (2.71)	82	0	21	64	14	68
Total	445	11.54 (3.22)	57	11	15	59	16	52

Note: CATS = Child and Adolescent Trauma Treatments and Services; WH = Whites; AA = African American; HL = Hispanic/Latino; OT = Other Ethnicity; <15K = household incomes below US$15,000/year.

Due to missing data percentage is based on n = 386.

Across the nine sites, 445 children and adolescents were assessed eligible for the evidence-based CBT treatments tested in the CATS project. A more detailed description of the study sample has been described elsewhere (CATS Consortium, 2007). Outreach for study participants included presentations to disaster-specific trauma service providers and nondisaster-related mental health treatment and service providers (e.g., Project Liberty, child welfare agencies, victim services providers, and district attorneys offices) by site as well as project staff on behalf of the consortium. Participating sites also recruited children and youth through referrals from within their own agencies and the surrounding community. At intake, parents were asked to complete a demographic survey that included questions regarding family structure, race/ethnicity of parents and children, socioeconomic status (household income, education, occupation), and child’s age and gender. The sample was predominantly female (58%), the mean age of participants was 11.5 (SD = 3.2) years. Overall, the majority of the sample was Latino/Hispanic (62%) and a large percentage (45%) was low income with total household incomes of less than US$15,000 per year.

Procedure

If children or adolescents screened positive, participants and their parents were brought in for consent, intake, and assessment. Engagement strategies were implemented during the intake and assessment process. Treatment assignment was based on a cutoff score for trauma symptom severity on the University of California, Los Angeles (UCLA) PTSD Reaction Index for DSM-IV (PTSD-RI; Pynoos, Rodriguez, Steinberg, Stuber, & Frederick, 1998). To receive the evidence-based CBT (i.e., trauma-specific CBT), participants had to report WTC disaster exposure and significant trauma symptomatology, defined as a score of 25 or greater on the PTSD-RI. Children or adolescents who scored between 17 and 24 on the measure of PTSD were provided CBT skills training. As the CBT skills group included a range of different service modalities and was not necessarily delivered by project-trained clinicians, we were unable to collect pretreatment and treatment participation data for this group. Thus, our sample for this study included only the 445 children and youth who screened eligible for trauma-specific CBT.

All participants were treated in accordance with ethical standards of research and participated with the active consent of parents and youth assent. Client engagement and retention in treatment was documented from the point of first phone or in-person contact with participants. Thus, consent to track participation data was obtained in-person or over the phone on first contact by clinicians or site staff conducting intake or responding to inquiries. This first level of consent was obtained to ensure that sites were formally documenting pretreatment contacts with children and families, including the attempts made to arrange for formal intake and assessment. During these initial contacts, sites were expected to use the evidence-based engagement strategies, while screening for whether formal assessment for eligibility into the project was warranted. The screening process was left up to each site and varied. For example, some sites used the WTC exposure questionnaire and some used broad trauma screening questions to determine that formal assessment would be conducted. Once a determination was made to conduct a formal assessment, separate parent consent and youth assent were obtained to complete the full baseline battery and to determine eligibility for CBT treatment. Consent forms and procedures were approved by the NYSOMH Institutional Review Board and the institutional review boards of each of the nine agencies.

Measures

PTS symptoms

The UCLA PTSD-RI (Pynoos et al., 1998) is a widely used self-report measure of PTSD for youth ages 7 to 19 that conforms to the Diagnostic and Statistical Manual of Mental Disorders (4th ed.; DSM-IV; American Psychiatric Association, 1994) criteria. There are separate versions for children and adolescents. The PTSD-RI includes a trauma events inventory (13 items); items to assess the level threat or danger of the traumatic event (A1 criterion) and the subjective evaluation of the event (A2 criterion), and a frequency-rating scale of DSM-IV PTSD symptoms for reexperiencing (Criterion B), avoidance (Criterion C), and hyperarousal (Criterion D). A total symptoms severity score of 38 on the PTSD-RI has been found in previous research to indicate high sensitivity and specificity for a diagnosis of full PTSD (Rodriguez, Steinberg, Saltzman, & Pynoos, 2001a, 2001b). The scale developers suggested defining severity by the total symptom severity score on the PTSD-RI as follows: low = 17 to 24, moderate = 25 to 37, and severe = 38 or greater. In the present study, Cronbach’s alpha for the items that make up the overall severity score were .83 for the adolescent version and .74 for the child version.

Treatment engagement and retention

We calculated three following variables to characterize engagement and retention: (a) pretreatment show rate was the ratio of the number of intake and assessment visits made over the total number of pretreatment visits scheduled, (b) treatment show rate was the ratio of the number of treatment appointments made over the total number of treatment visits scheduled, and (c) dose was the number of treatment visits made. CATS study procedures included clinic staff documenting treatment engagement in two ways: contact logs and case completion forms developed specifically for this project. The contact logs were kept by clinic staff as a running record of the contacts made with or on behalf of participants. Clinicians and staff involved in intake, assessment, and treatment at each of the sites were instructed to note contacts related to pretreatment efforts (e.g., intake and assessment) and treatment sessions, collateral contacts and parent contacts during the course of treatment, with instructions for documenting visits that were scheduled and not kept (“no shows”), either missed or canceled. The case completion form provided a second source of information about the total number of treatment sessions, which were used to validate the contact logs.

Interventions and Treatments

Structured engagement strategies

Each of the sites was asked to form an “engagement team” of key staff including site study coordinators, clinicians, receptionists, or research assistants. The engagement team attended training, implemented engagement interviews, and attended ongoing supervision and consultation sessions. Engagement teams were trained in empirically supported strategies used in prior research to increase parent and youth engagement in mental health treatment (e.g., McKay, Nudelman, et al., 1996). The training consisted of an 8-hr intensive workshop striving to enhance provider understanding of barriers to child mental health as they relate to child, family, community, and system levels. Following the training, the CATS research team along with engagement trainers followed up with each provider agency through on-site consultation visits to discuss, and, if necessary, refine engagement plans.

Members of the engagement team were trained to conduct engagement interviews with parents or youth on first contact by phone and during the first interview. The goal of engagement interviewing is for the provider to form a collaborative working relationship with primarily adult caregivers and often with the youth. This overarching goal is accomplished by focusing the interviewer on four key areas: (a) clarifying needs, (b) maximizing caregiver investment and efficacy, (c) identifying potential obstacles to attendance, and (d) addressing barriers to engagement. The first goal, clarifying the need for mental health treatment, involves exploring the current concerns of the parent and obtaining detailed information on the presenting problem and the current mental health needs of the child. In addition to defining the current mental heath need, it was important to assess parents’ willingness and ability to schedule and attend the initial intake appointment. For example, the worker might inquire about the parents’ relationships with their child’s teachers as well as the parents’ past experience with therapy and with other professionals (e.g., previous teachers). These inquiries would address negative attitudes with helping professionals that might have developed over time. Finally, clarifying parents’ needs also involves matching parent and child needs with what the clinic can realistically provide. In the CATS project, this meant clarifying with parents the use of CBT as the treatment of choice as well as describing any other services offered by the agencies.

The second goal of the first contact, maximizing the caregivers’ investment and efficacy in relation to help-seeking, involves promoting the caregivers’ belief that treatment can be helpful. For parents who have had poor past experiences with mental health services, this task may involve emphasizing the family’s existing strengths and praising their decision for seeking treatment. Finally, the interviewer seeks to identify and address potential barriers to treatment participation. During the telephone contact, workers remind the parent about their first appointment by reviewing the time and date, clarifying exactly how the caregiver plans to travel to the appointment and providing directions, and identifying any potential obstacles (i.e., child care, lack of car fare). Potential obstacles to treatment attendance are addressed by intake staff or clinicians during this first contact.

Trauma treatments

Two evidence-based trauma-focused CBTs were selected for implementation: (a) trauma-focused CBT (TF-CBT; Cohen et al., 2002) and (b) trauma and grief component therapy for adolescents [Trauma and Grief Component Therapy (TGCT) Layne et al., 2002]. For a discussion of why these were selected see CATS Consortium (2007). Aside from their differences, the two trauma-specific CBTs (TF-CBT and TGCT) share many components, including psychoeducation, affect regulation, relaxation, cognitive restructuring, gradual exposure techniques, and homework assignments. Both manualized treatments outline a 16- to 20-week course of treatment. The suggested “mechanism of action” for these trauma treatments is the “trauma narrative” or gradual exposure, a process whereby youth are guided through their trauma experience to create a type of narrative or visual representation along with associated thoughts and feelings. Gradual exposure helps children process their experiences and become desensitized to them, with the goal of reducing distress, functional impairments, and “cognitive distortions” or “unhelpful thoughts.”

Statistical Analyses

Hierarchical regression analysis was the data analytic strategy used to examine predictors of treatment participation. In our analysis, we wanted to examine the demographic and clinical predictors of treatment participation, adjusting for site differences, in the belief that understanding demographic variables and trauma symptomatology would reflect and explain differences across sites, the challenges of engagement or both. For these reasons, hierarchical multiple regression was chosen as the most effective analytic approach for exploring the relative contribution of each individual variable not only on various types of treatment participation but also on the overall contribution of groups of variables (e.g., site, demographics, and symptoms). Therefore, parallel regression analyses were conducted with pretreatment show rates, treatment show rates, and dose as the criterion variables and sites entered in the first step, demographic variables (gender, age, race/ethnicity, and household income) entered in the second step, and baseline trauma symptomatology entered in the third step. Dummy codes were created for gender with females coded as the reference group, for ethnicity with Hispanics as the reference group, and for income with children from families below 15K per year as the reference group. Hispanics were used as the reference group because they represented the largest ethnic group in the sample. Sites were dummy coded with Site 7 used as the reference group, because that site had the highest study recruitment (n = 95). Trauma symptomatology was defined as the total severity score on the PTSD-RI. In the absence of studies that examine the relationship between symptoms and treatment participation, it is plausible that trauma symptoms could have either a positive or negative relationship to treatment attendance. On one hand, greater symptomatology may prompt help-seeking on the part of children and parents. On the other hand, avoidance is a criterion of PTSD diagnosis and children exposed to traumatic events, and their parents are sometimes reluctant to participate in treatment for fears of retraumatization. All multiple regression analyses were conducted using SPSS 17.0 software.

Results

Descriptive Statistics

The means and standard deviations for engagement variables and trauma symptomatology across the nine sites are provided in Table 2. We also provide the settings in which services were delivered, defined broadly as whether they were delivered in outpatient settings or school-based settings. Overall, of the 445 children and adolescents assessed eligible for CATS trauma treatment, the average PTSD Total Severity scores was 36 at baseline. This level of trauma symptomatology is high and clinically significant in that it approaches a score of 38, which prior research with the PTSD-RI indicates high sensitivity and specificity for a probable diagnosis of full PTSD (Rodriguez et al., 2001a, 2001b). There was high variability across the sites in trauma symptom scores. Trauma symptomatology across eight of the nine sites was within the moderate range (25–37), which was expected given the design of the study. However, at one site (Site 1) trauma symptomatology was in the severe range (M = 40.23). It should be noted that Site 1 also had the highest average age for children (M = 14.91), suggesting that they served primarily adolescents.

Table 2.

Means and Standard Deviations of Engagement Variables by CATS Program Site

	Symptom severity		Pretreatment rates^a		Treatment rates^b		Dose
Sites	n	M (SD)	n	M (SD)	n	M (SD)	n	M (SD)	Service setting
Site 1	65	40.23 (10.01)	61	0.80 (0.23)	54	0.73 (0.20)	60	8.82 (6.56)	OP, SB
Site 2	33	37.51 (9.72)	30	0.85 (0.21)	29	0.70 (0.29)	32	12.03 (5.39)	OP
Site 3	40	36.76 (9.14)	35	0.86 (0.24)	37	0.74 (0.24)	39	11.18 (6.73)	OP, SB
Site 4	52	37.33 (9.63)	34	0.89 (0.17)	39	0.71 (0.18)	43	12.07 (6.07)	OP, SB
Site 5	21	32.62 (8.41)	11	0.90 (0.19)	12	0.95 (0.06)	12	11.25 (4.99)	SB
Site 6	60	36.62 (8.79)	55	0.85 (0.20)	52	0.70 (0.17)	58	11.59 (6.76)	OP
Site 7	95	35.00 (9.37)	75	0.93 (0.16)	86	0.87 (0.16)	95	12.64 (6.47)	SB
Site 8	51	35.20 (8.68)	50	0.80 (0.23)	46	0.62 (0.21)	50	10.76 (7.25)	OP
Site 9	28	36.54 (8.40)	28	0.91 (0.17)	28	0.80 (0.19)	28	14.25 (6.46)	OP, SB
Total	445	36.61 (9.36)	379	0.84 (0.24)	383	0.75 (0.21)	417	11.54 (6.59)

Note: CATS = Child and Adolescent Trauma Treatments and Services; OP = outpatient mental health services; SB = school-based mental health services.

Ratio of phone or personal contacts made to contacts attempted.

Ratio of total treatment visits made to total visits made, missed, and cancelled.

Nearly 95% (417) of participants had contact logs or case completion forms that provided information on overall dose. However, for calculation of show rates, cases were excluded from analysis if there was no detail provided on the contact log for either the pretreatment or treatment visits. Excluding cases with no detailed information reported on the contact logs, we were able to identify 379 (85%) participants with valid pretreatment show rates, 383 (86%) with valid treatment show rates, and 417 (84%) with valid data on dose. Overall, treatment show rates were lower than pretreatment show rates. Across the nine sites, there was high variability in show rates and overall dose. Pretreatment show rates ranged from 80% to 93%, treatment show rates ranged from 62% to 95%, and dose ranged from an average 8.82 (Site 1) to 14.25 (Site 9) sessions per child. Site 7 had the highest pretreatment show rates, the second highest treatment show rate, and the highest mean dose levels of the nine sites in the study. These numbers were likely elevated because Site 7 delivered a majority of its services in school-based mental health clinics, which was also true of Site 5, the site with the highest treatment show rate. Site 7 was used as the reference site because of its highest rates of treatment participation and high overall recruitment of participants.

Figure 1 represents the amount of treatment received by youth in the sample. These rates were calculated with the total sample of children who were recruited eligible for CBT for trauma (n = 445) as the denominator, rather than on only the youth with treatment data available (n = 417), to avoid overestimating treatment participation. As shown in the first bar of the graph, of the 445 children and adolescents recruited into the active CBT treatment arm, 394 (89%) received some treatment, with a range from 1 to 37 sessions. The remaining 11% of children and adolescents did not receive any treatment despite being assessed eligible for CATS. Chi-square analysis and t tests showed that there were no significant differences between those with at least 1 treatment session and the remainder of the sample (n = 51) on demographic characteristics or baseline trauma symptomatology. The remaining bars show the percentages of youth who received varying doses of treatment. The ranges of dose are grouped by clinically significant amounts of treatments. One to 7 sessions of treatment is considered suboptimal based on meta-analyses of treatment outcome research that suggests that 8 sessions is the point at which 50% of clients begin to show improvement among adult patients (Howard, Kopta, Krause, & Olinsky, 1986). Eight to 11 sessions represent a minimally acceptable dose of treatment. Twelve to 20 sessions represent an optimal course of treatment based on the manual content and guidelines of the treatment developers for the therapies delivered in our project (Cohen et al., 2002; Layne et al., 2002). The highest range (21–37) indicates youth who exceeded the upper limits of CBT treatment, which is generally considered short term and time limited. Overall, two-thirds of children and adolescents in CATS received at least the minimally acceptable dose of treatment or more.

Figure 1.

Children and adolescents with a score of 25 or above on the PTSD-RI and WTC exposure were eligible for CATS CBT for trauma, and those below the cutoff were assigned to treatment as usual

Correlations between each predictor and the outcome variables as well as the zero-order correlations among the outcome variables are summarized in Table 3. Demographic characteristics and trauma symptomatology were weakly but significantly correlated to treatment participation. Age was negatively correlated with treatment participation, suggesting that older children were less likely to attend treatment sessions on a consistent basis. Other significant correlates of treatment participation included African American ethnicity, Hispanic ethnicity, and trauma symptomatology. African American ethnicity was associated with lower treatment show rates, Hispanic ethnicity was associated with greater dose, and trauma symptomatology was negatively associated with pretreatment show rates and dose. Finally, indicators of treatment participation were moderately correlated with each other. The highest correlations were between pretreatment and treatment show rates and dose, suggesting that children with fewer missed appointments—before and during treatment—also tended to have more treatment sessions overall.

Table 3.

Correlations Between Predictor Variables and Outcome Variables

Variable	PT show	T show	Dose
1. Gender	.02	.04	−.08
2. Age	−.12*	−.18**	−.11*
3. White	.04	.07	.04
4. African American	−.07	−.16**	−.08
5. Hispanic	−.03	.08	.10*
6. Other	−.00	.01	−.09
7. Income	.08	.03	.06
8. PTSD total	−.13*	−.06	−.12*
9. PT show		.30**	.35**
10. T show			.48**
11. Dose

Note: PTSD = posttraumatic stress disorder; PT show = pretreatment show rates; T show = treatment show rates; Dose = session count; Income = household income; PTSD Total = overall trauma symptoms.

p < .05. **p < .01. ***p < .001.

Three parallel hierarchical multiple regression analyses were conducted to identify predictors of pretreatment and treatment show rates and dose. Sites were entered in the first step, demographic variables (gender, race, ethnicity, and income) were entered in the second step, and overall trauma severity score was entered in the final step. Results of the regression analysis are summarized in Table 4. Only the standardized coefficients (β) for the third step in the model with all the predictors entered simultaneously are shown. The reported R² and ΔR² are for each step in the regression. Results suggest that for treatment show rates, sites accounted for the largest amount of variance explained in the model, R² = .20, F(8, 332) = 10.57, p < .001, whereas demographics, ΔR² = .03, F(6, 326) = 1.72, p = .12, and trauma symptomatology added little and no variance explained, respectively. Note that coefficients for specific sites are all negative, indicating that these had significantly lower treatment show rates in comparison with Site 7, the reference site. Most of the sites were negatively associated with treatment show rates, with the exception of Sites 5 and 9. Of note, Sites 5, 7, and 9 all delivered a significant proportion of their services in school-based mental health programs. Site 8, the site with the highest negative coefficient for treatment show rate, provided primarily outpatient services. Taken together these results suggest that treatment show rates in schools were superior to treatment show rates delivered in outpatient clinics.

Table 4.

Predictors of Treatment Participation: Summary of Results From Hierarchical Multiple Regression

	Pretreatment show			Treatment show			Dose
	β	R ²	ΔR²	β	R ²	ΔR²	β	R ²	ΔR²
Step 1: Site		.04	.04		.20	.20***		.04	.04
Site 1	−.24**			−.24**			−.16*
Site 2	−.08			−.22**			−.04
Site 3	−.24***			−.27**			−.12
Site 4	−.21**			−.27**			−.06
Site 5	−.00			.07			−.07
Site 6	−.06			−.30***			−.05
Site 8	−.16			−.41***			−.13
Site 9	−.01			−.09			.05
Step 2: Demographics		.09	.05*		.22	.02		.08	.04*
Gender	−.07			−.02			−.15**
Age	−.04			−.03			−.06
White	.06			.11			.02
African American	−.02			.00			.00
Other Ethnicity	.05			−.01			−.03
Income	.22**			.07			.12*
Step 3: Symptoms		.10	.01		.22	.00		.08	.01
PTSD severity	−.09			.00			−.10

Note: PTSD = posttraumatic stress disorder. Site 7 was used as the reference site, and Hispanics were used as the reference ethnic group.

p < .05. **p < .01. ***p < .001.

The amount of variance explained by site was low for both pretreatment show rates, R² = .04, F(8, 320) = 1.86, p = .07, and dose, R² = .04, F(8, 355) = 1.70, p = .10, however, specific demographic variables contributed significant variance for both sets of predictor variables. For pretreatment show rates and dose, demographics added 5%, ΔR² = .05, F(6, 314) = 2.53, p = .02, and 4%, ΔR² = .04, F(6, 349) = 2.73, p = .01, of variance, respectively. Household income was positively associated with pretreatment show rates and dose but not treatment show rates, suggesting that children and adolescents from households with greater income tended to be more likely to miss fewer pretreatment sessions and attend more treatment sessions overall compared with those from low-income households. In addition, gender was negatively associated with dose, suggesting that males had lower overall dose than females.

Overall, the amount of variance explained by the regression models was low, with R² ranging from 4% to 22%; however, the effect sizes for amount of ΔR² ranged from .04 to .08, suggesting moderate effects for the addition of demographic variables and trauma symptom over site variables. In a separate analysis, demographic variables and trauma severity were entered in the first step, and sites were entered in the second step to determine the significance of the change in R² over and above demographic and trauma symptom scores. Sites accounted for a significant increase in variance explained for pretreatment and treatment show rates but not dose. The R² increased from .06 to .23 for treatment show rate and from .06 to .11 for pretreatment show rates.

Discussion

As efforts to implement more effective EBTs into routine care settings become widespread, examining the participation of clients into specialized care becomes of paramount importance. This study examined the impact of linking empirically supported engagement strategies to implementation of effective evidence-based trauma treatment (i.e., trauma-specific CBT). As part of the largest EBP implementation study on children’s trauma treatment, the CATS project examined treatment participation when structured engagement approaches were introduced into real-world agencies whose staff had been trained to provide evidence-based trauma treatments for children and adolescents following the 9/11 WTC disaster. Three types of treatment participation were examined: pretreatment show rates, treatment show rates, and dose. We also examined the relationship of demographic variables, including gender, age, ethnicity and household income, and baseline trauma severity scores to these various indicators of treatment participation. Although high rates of treatment participation were documented, regression models of key predictors of treatment participation explained little variance in the various indicators of treatment participation.

Overall, we found treatment participation to be high compared with data from prior studies in community settings that documented no show rates as high as 50% and average number of treatment sessions as low as three visits (Kazdin, 1993; Lerman & Pottick, 1995; McKay et al., 2005; Miller, Southam-Gerow, & Allin, 2008). Findings from this study demonstrated that a majority of children and youth in CATS remained in treatment for a substantial amount of time, essentially reversing the trend seen in studies of mental health services use, where rates of retention drop by almost 75% after only 3 sessions (McKay et al., 2005). In contrast, more than 85% of children and adolescents referred and assessed for CATS treatment received some treatment. More than two thirds of those who received treatment attended at least 8 treatment sessions, and nearly half (49%) received 12 or more sessions. Pretreatment show rates were more than 80%, treatment show rates were 75%, and the average number of sessions attended by participants was almost 12. We also found moderately high correlations among the various indicators of treatment participation, with the highest correlations between pretreatment and treatment show rates and dose. These data suggest that efforts to get clients to pretreatment visits and decrease no shows can increase overall dose of treatment. Overall, this study provides preliminary support that structured engagement strategies, in conjunction with the use of an EBT, can enhance the likelihood that youth would receive an adequate dose of trauma-specific CBT.

In addition, we used hierarchical multiple regression analyses to explore the relationship of predictors of treatment participation, including sites, demographic characteristics, and trauma symptomatology to pretreatment and treatment show rates and overall dose. These predictors accounted for only a small amount of variance in the indicators of treatment participation. Sites were significant predictors of treatment show rates, but they only accounted for 20% of the variance. Five sites demonstrated lower treatment show rates compared with our reference site. However, the models explained 10% or less of the variance for pretreatment show rates and dose. It is noteworthy that the reference site and the only sites nearly on par for high treatment show rates with the reference site delivered services largely in school-based mental health programs. The sites that had poorer treatment show rates were the sites that delivered their services primarily in outpatient care. These findings support prior research that demonstrates that school-based mental health services increase accessibility and use of services over clinic-based services, especially for low-income disadvantaged children (Atkins et al., 2006; Catron & Weiss, 1994). Delivering services in the schools can increase treatment show rates because children and youth are easily available for therapy sessions. CBT for trauma is particularly suitable and effective for delivery in schools because CBT can largely be delivered individually with much less disturbance in school routines, unlike psychosocial treatments and schoolwide prevention efforts for other behavioral disorders (e.g., disruptive behavior disorders) that often require more involvement with parents, teachers, or groups of children (Dane & Schneider, 1998).

In the regression models, demographic variables added a small but significant amount of variance explained for pretreatment show rates and dose. Household income was positively associated with pretreatment show rates and dose, which would be expected given that greater financial stability and higher socioeconomic status afford families the resources necessary to overcome many of the typical obstacles to treatment participation, such as transportation and child care. With regard to dose, gender was negatively associated with number of treatment sessions, with boys having fewer treatment sessions than girls. This contradicts findings from other disaster services research, which suggests that boys are more likely than girls to receive counseling (Stuber et al., 2002); however, little data are available about gender differences in treatment participation beyond treatment entry. We also examined the impact of trauma symptoms on treatment participation and found that higher symptomatology was not a significant predictor of treatment participation rates. These findings appear counterintuitive and contradict those of previous children’s mental health services research that indicates that greater need, defined by child’s clinical severity, is a primary and positive predictor of services use (Miller et al., 2008).

The fact that the predictors we included in the model explained only a very small amount of variance suggests three possibilities. First, it is likely that unmeasured variables other than those examined in the present study better explain treatment participation. It should be noted that much of the literature on mental health service use focuses on obstacles to treatment participation such as demographic characteristics (e.g., Kataoka et al., 2002), and in the present study, treatment participation rates were high and many of the common obstacles to treatment participation such as inadequate insurance were irrelevant. There is relatively little research on facilitators of treatment participation, and it is possible that the factors that predict poor participation are not the same ones that predict high participation. Second, in the absence of other variables that significantly account for a greater amount of variance in treatment participation, it is possible that the efforts of the sites to engage children and families into treatment helped to increase treatment participation. The fact that provider sites differed in treatment participation and that sites explained the greatest—albeit small overall—amount of variance in our models suggests that we must better understand the agency or organizational factors that most contribute to treatment participation (Glisson et al., 2008). Finally, our results suggest that services delivered in the schools resulted in superior treatment participation rates, which represents a major step forward and a potential blueprint for improving children’s mental health services in general. More than a decade ago the Great Smoky Mountains Study of Youth documented the low access to specialty mental health services (e.g., services delivered in formal clinical care) in children’s mental health services and the dominant role that schools played in mental health care (Burns et al., 1995). Our research suggests that high-quality mental health care in the form of EBTs delivered by specialty mental health care providers in schools can lead to greater treatment participation.

Despite these findings, the study had a number of notable limitations that suggest the findings must be interpreted cautiously. First, the study was hindered by the lack of an appropriate community comparison group, without which it is difficult to determine whether these rates of engagement were a function of chance alone. However, the rates of treatment participation in this study far exceeded those of reported in general community samples (Miller et al., 2008). Second, the study had no formal mechanism to assess fidelity to the engagement strategies, significantly limiting the extent to which we can attribute increased rates of retention to the engagement training. The availability of data to document fidelity of program implementation threatens the internal validity of outcome studies and particularly in instances where differences across program sites are found (Dane & Schneider, 1998). In the present study, fidelity data could have shed light on the differences we found across sites. Third, the study findings were limited by missing treatment participation data (11%). Although we found no evidence of systematic bias between the 394 youth who received treatment and untreated youth, there is no way of knowing whether missing data produced biases from other unexplored variables. Finally, this research was correlational, precluding any statements of causality. Thus, while sites, school-based services, and demographic characteristics are associated with treatment show rates in the present study, future research must focus on experimental designs to determine a causal link between these factors and treatment participation.

Our findings of higher than expected rates of treatment participation subsequent to training and consultation on engagement strategies may have been a result of several factors. First, the training included intensive consultation tailored to individual sites’ challenges; this individualization may have contributed to the high rates of treatment participation. In addition, the context of the delivery of CATS was unique in that it was implemented at a time when there was heightened attention to the mental health needs of children and adolescents. Many public and private efforts were undertaken to increase public education and awareness about the effects of trauma and the mental health needs of adults and youth. It is possible that the greater attention to mental health needs served to destigmatize mental health service use and contribute to an increased acceptability of mental health services in general. In addition, linking empirically supported engagement strategies with access to evidence-based clinical services seems highly likely to be an effective combination of practices (McKay et al., 2004). Factors such as therapeutic alliance have been linked to treatment participation (Kazdin, 1993), and the enhancements of clinical care afforded by the combination of practices (i.e., structured engagement and EBT) in CATS may have enhanced alliance.

Footnotes

Acknowledgements

The authors appreciate the contributions of the Peer Mentoring Group at Columbia University’s Division of Mental Health Services and Policy Research, including Mary Cavaleri, Erum Nadeem, Alissa Gleacher, Cathryn Galanter, and Jennifer Wisdom.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research and/or authorship of this article.

References

American Psychiatric Association. (1994). Diagnostic and statistical manual of mental disorders (4th ed.). Washington, DC: Author.

Annunziato

R. A.

Rubinstein

Sheikh

Maurer

Cotter

McKay

M. M.

. . . Shemesh

(2008). Site matters: Winning the hearts and minds of patients in a cardiology clinic. Psychosomatics: Journal of Consultation Liaison Psychiatry, 48, 386–391.

Atkins

M. S.

Frazier

S. L.

Birman

Adil

J. A.

Jackson

Graczyk

P. A.

. . . McKay

M. M.

(2006). School-based mental health services for children living in high poverty urban communities. Administration and Policy in Mental Health and Mental Health Services Research, 33, 146–159.

Burns

B. J.

Costello

E. J.

Angold

Tweed

Stangl

Farmer

Erkanli

(1995). Children’s mental health service use across service sectors. Health Affairs, 14, 147–159.

Burns

B. J.

Hoagwood

K. E.

(2004). Evidence-based practice, Part 1: Research update. Child and Adolescent Psychiatric Clinics of North America, 13, xi–xiii.

Catron

Weiss

(1994). The Vanderbilt school-based counseling program: An interagency, primary-care model of mental health services. Journal of Emotional and Behavioral Disorders, 2, 247–253.

CATS Consortium. (2007). Implementing CBT for traumatized children and adolescents after September 11: Lessons learned from the Child and Adolescent Trauma Treatments and Services (CATS) Project. Journal of Clinical Child and Adolescent Psychology, 36, 581–592.

Chow

J. C.

Jaffee

Snowden

(2003). Racial/ethnic disparities in the use of mental health services in poverty areas. American Journal of Public Health, 93, 792–797.

Coatsworth

J. D.

Santisteban

D. A.

McBride

C. K.

Szapocznik

(2001). Brief Strategic family therapy versus community control: Engagement, retention and an exploration of the moderating role of adolescent symptom severity. Family Process, 40, 313–332.

10.

Cohen

J. A.

Mannarino

A. P.

Deblinger

(2002). Child and parent trauma-focused cognitive behavioral therapy: Treatment manual. Unpublished manuscript.

11.

Daley

D. D.

Zuckoff

(1999). Improving treatment compliance: Counseling and systems strategies for substance abuse and dual disorders. Center City, MN: Hazelden.

12.

Dane

A. V.

Schneider

B. H.

(1998). Program integrity in primary and early secondary prevention: Are implementation effects out of control? Clinical Psychology Review, 18, 23–45.

13.

Fairbrother

Stuber

Galea

Fleischman

A. R.

Pfefferbaum

(2003). Posttraumatic stress reactions in New York City children after the September 11, 2001 terrorist attacks. Ambulatory Pediatrics, 3, 304–311.

14.

Fortney

Rost

Zhang

Warren

(1999). The impact of geographic accessibility on the intensity and quality of depression treatment. Medical Care, 37(9), 884–893.

15.

Glisson

Landsverk

Schoenwald

Kelleher

Hoagwood

K. E.

Mayberg

. . . The Research Network on Youth Mental Health. (2008). Assessing the organizational social context (OSC) of mental health services: Implications for research and practice. Administration and Policy in Mental Health, 35, 98–113.

16.

Griffin

J. A.

Cicchetti

Leaf

P. J.

(1993). Characteristics of youths identified from a psychiatric case register as first-time users of services. Hospital & Community Psychiatry, 44, 62–65.

17.

Gurwitch

R. H.

Sitterle

K. A.

Young

B. H.

Pfefferbaum

(2002). The aftermath of terrorism. In La Greca

A. M.

Silverman

W. K.

Vernberg

E. M.

Roberts

M. C.

(Eds.), Helping children cope with disasters and terrorism (pp. 327–357). Washington, DC: American Psychological Association.

18.

Hines-Martin

Brown-Piper

Sanggil

Malone

(2003). Enabling factors of mental health service use among African Americans. Archives of Psychiatric Nursing, 17, 197–204.

19.

Hoagwood

Burns

B. J.

Kiser

Ringeisen

Schoenwald

S. K.

(2001). Evidence-based practice in child and adolescent mental health services. Psychiatric Services, 52, 1179–1189.

20.

Hoven

C. W.

Duarte

C. S.

Lucas

C. P.

Mandell

D. J.

Rosen

(2002). The effects of the world trade center attack on New York City public school students-initial report to the New York City Board of Education. New York, NY: Applied Research and Consulting, LLC & Columbia University Mailman School of Public Health & New York State Psychiatric Institute.

21.

Howard

K. J.

Kopta

S. M.

Krause

M. S.

Olinsky

D. E.

(1986). The dose-effect relationship in psychotherapy. American Psychologist, 41, 159–164.

22.

Kataoka

S. H.

Zhang

Wells

K. B.

(2002). Unmet need for mental health care among U.S. children: Variation by ethnicity and insurance status. American Journal of Psychiatry, 159, 1548–1555.

23.

Kazdin

A. E.

(1993). Premature termination from treatment among children referred for antisocial behavior. Journal of Child Psychology and Psychiatry, 31, 415–425.

24.

Layne

C. M.

Saltzman

W. R.

Pynoos

R. S.

Steinberg

A. M.

(2002). Trauma and grief component therapy. New York: New York State Office of Mental Health.

25.

Lerman

Pottick

(1995). The parent’s perspective: Delinquency, aggression and mental health. Switzerland: Harwood Academy.

26.

McKay

M. M.

Bannon

(2004). Engaging families in child mental health services. Child and Adolescent Psychiatric Clinics of North America, 13, 905–921.

27.

McKay

M. M.

Hibbert

Hoagwood

Rodriguez

Murray

Legerski

Fernandez

(2004). Integrating evidence-based engagement interventions into “real world” child mental health settings. Brief Treatment and Crisis Intervention, 4, 177–186.

28.

McKay

M. M.

Lynn

C. J.

Bannon

W. M.

(2005). Understanding inner-city child mental health need and trauma exposure: Implications for preparing urban service providers. American Journal of Orthopsychiatry, 75, 201–210.

29.

McKay

M. M.

McCadam

Gonzales

(1996). Addressing the barriers to mental health services for inner-city children and their caretakers. Community Mental Health Journal, 32, 353–361.

30.

McKay

M. M.

Nudelman

McCadam

Gonzales

(1996). Evaluating a social work engagement approach to involving inner-city children and their families in mental health care. Research on Social Work Practice, 6, 462–472.

31.

McKay

M. M.

Pennington

Lynn

C. J.

McCadam

(2001). Understanding urban child mental health service use: Two studies of child, family, and environmental correlates. Journal of Behavioral Health Services & Research, 28, 475–483.

32.

McKay

M. M.

Stoewe

McCadam

Gonzales

(1998). Increasing access to child mental health services for urban children and their caregivers. Health & Social Work, 23, 9–15.

33.

Miller

L. M.

Southam-Gerow

M. A.

Allin

R. B.

(2008). Who stays in treatment? Child and family predictors of youth client retention in a public mental health agency. Child & Youth Care Forum, 37, 153–170.

34.

National Institute of Mental Health. (1999). Bridging science and service: A report by the National Advisory Mental Health Council’s Clinical Treatment and Services Research Workgroup. Rockville, MD: National Institutes of Health, National Institute of Mental Health.

35.

National Institute of Mental Health. (2000). Translating behavioral science into action: Report of the National Advisory Mental Health Council Behavioral Science Workgroup. Rockville, MD: National Institutes of Health, National Institute of Mental Health.

36.

Pfefferbaum

North

C. S.

Doughty

D. E.

Gurwitch

R. H.

Fullerton

C. S.

Kyula

(2003). Posttraumatic stress and functional impairment in Kenyan children following the 1998 American embassy bombing. American Journal of Orthopsychiatry, 73, 133–140.

37.

Pynoos

Rodriguez

Steinberg

Stuber

Frederick

(1998). UCLA PTSD Index for DSM-IV. Los Angeles: UCLA Trauma Psychiatry Services.

38.

Ringel

J. S.

Sturm

(2001). National estimates of mental health utilization and expenditures for children in 1998. Journal of Behavioral Health Research, 28, 319–333.

39.

Rodriguez

Steinberg

A. S.

Saltzman

W. S.

Pynoos

R. S.

(2001a, December). PTSD Index: Psychometric analyses of the adolescent version. Symposium conducted at the Annual Meeting of International Society for Traumatic Stress Studies, New Orleans, LA.

40.

Rodriguez

Steinberg

A. S.

Saltzman

W. S.

Pynoos

R. S.

(2001b, December). PTSD Index: Psychometric analyses of the child and parent versions. Symposium conducted at the Annual Meeting of the International Society for Traumatic Stress Studies, New Orleans, LA.

41.

Staudt

(2007). Treatment engagement with caregivers of at-risk children: Gaps in research and conceptualization. Journal of Child and Family Studies, 16, 183–196.

42.

Stiffman

A. R.

Earls

Robins

L. N.

Jung

K. G.

(1998). Problems and help seeking in high-risk adolescent patients of health clinics. Journal of Adolescent Health Care, 9, 305–309.

43.

Stuber

Fairbrother

Galea

Pfefferbaum

Wilson-Genderson

Vlahov

(2002). Determinants of counseling for children in Manhattan after the September 11 attacks. Psychiatric Services, 53, 815–822.

44.

Szapocznik

Perez-Vidal

Brickman

A. L.

Foote

F. H.

Santisteban

Hervis

Kurtines

W. M.

(1988). Engaging adolescent drug abusers and their families in treatment: A strategic structural systems approach. Journal of Consulting and Clinical Psychology, 56, 552–557.

45.

Tolan

P. H.

Henry

(1996). Patterns of psychopathology among urban poor children: Comorbidity and aggression effects. Journal of Consulting and Clinical Psychology, 64, 1094–1099.

46.

Tolan

P. H.

Hanish

L. D.

McKay

M. M.

Dickey

M. H.

(2002). Evaluating process in child and family interventions: Aggression prevention as an example. Journal of Family Psychology, 16, 220–236.

47.

U.S. Department of Health and Human Services. (1999). Mental health: A report of the surgeon general. Rockville, MD: U.S. Department of Health and Human Services, Substance Abuse and Mental Health Services Administration, Center for Mental Health Services, National Institutes of Health, National Institute of Mental Health.

48.

U.S. Department of Health and Human Services. (2000). Report of the surgeon general’s conference on children’s mental health: A national action agenda. Rockville, MD: U.S. Department of Health and Human Services, Substance Abuse and Mental Health Services Administration, Center for Mental Health Services, National Institutes of Health, National Institute of Mental Health.

49.

U.S. Department of Health and Human Services. (2001). Youth violence: A report of the surgeon general. Rockville, MD: U.S. Department of Health and Human Services, Substance Abuse and Mental Health Services Administration, Center for Mental Health Services, National Institutes of Health, National Institute of Mental Health.

50.

Weisz

J. R.

Hawley

K. M.

Pilkonis

P. A.

Woody

S. R.

Follette

W. C.

(2000). Stressing the (other) three R’s in the search for empirically supported treatments: Review procedures, research quality, relevance to practice and the public interest. Clinical Psychology: Science and Practice, 7, 243–258.