Abstract
Summary
While previous studies have offered insight into evidence-based practices that are effective in promoting safety and well-being, the underlying contextual implementation conditions that influence these outcomes in child welfare agencies are less understood. To address this gap, this study relied on organizational survey data collected from child welfare workers and supervisors during the process of implementing an evidence-based practice—the Positive Parenting Program—and merged those data with data gathered by the Parents' Assessment of Protective Factors survey.
Findings
Results showed a significant increase between baseline and termination of services in social connections and overall protective factors index scores. Parents who received services from organizations inclusive of supportive leadership experienced greater increases in social connections. Lower protective factors index scores were associated with workers’ perception that evidence-based practices were required; however, scores increased as supportive leadership increased. Parents who identified as African American experienced lower protective factors index scores compared to parents in other racial categories. Parents who were engaged with workers who felt evidence-based practices were appealing were likely to experience greater increases in protective factors index scores.
Applications
Intended client outcomes are more likely to be achieved when agency leaders (1) provide workers support to learn and use evidence-based practices and (2) rely on methods to increase evidence-based practice appeal rather than mandate implementation. Future research is needed to (1) understand why the implementation contexts or Positive Parenting Program itself are not as effective for African Americans as they are for oteher racial/ethnic groups and (2) validate findings in other agency contexts and with other evidence-based practices.
In 2017, the public child welfare (CW) system in the United States was inundated with 2.4 million reports of abuse and neglect. Of those reports, nearly 675,000 have been substantiated (U.S. Department of Health and Human Services (UDHHS), 2019a). While CW agencies are dedicated to promoting safety, permanency, and well-being, agency leaders and caseworkers have had some degree of flexibility in terms of what services to provide to achieve these outcomes and which one to prioritize over others (Garcia & DeNard, 2017; Jordan & Connelly, 2016). In recent years, there has been a push to pay special attention to child well-being, given that chronic exposure to abuse and neglect contributes to poor developmental and psychosocial outcomes (Courtney & Dworsky, 2006; Fratto, 2016; Garcia et al., 2017; Herrenkohl et al., 2012; Zeanah & Humphreys, 2018). To that end, there has been increased pressure and demand to demonstrate services achieve intended outcomes (i.e. safety, permanency, and well-being). CW leaders and scholars have cultivated and sustained research–practice partnerships to examine whether evidence-supported parenting interventions (ESPIs) improve parenting behaviors and capacities and, in turn, reduce risk for out-of-home placement. For example, in a 2019 study, Garcia et al. identified six evidence-based treatments regarded as “well-supported by research evidence” and categorized as having “medium” or “high” relevance to CW populations by the California Evidence-Based Clearinghouse. These ESPIs include Parent–Child Interaction Therapy (PCIT), Trauma-Focused Cognitive Behavioral Therapy (TF-CBT), Level 4 TripleP (Pathways Positive Parenting Program), and Multi-Systemic Therapy (MST). While Incredible Years and Multidimensional Family Therapy met the selection criteria for their effectiveness and relevance, they did not, like the aforementioned ESPIs, meet a “diversity” threshold of at least 25% of the study sample identifying as African American and/or Latinx.
With ESPIs identified, Garcia et al. (2019) interviewed scholars and leaders who had experience (1) examining whether these ESPIs achieve intended outcomes and (2) implementing ESPIs in CW agencies. Scholars reaffirmed intended outcomes were achieved, including but not limited to the following: improved parent-child relationships (PCIT); fewer child externalizing behaviors (PCIT and MST); improved school attendance (MST); fewer mental health symptoms and disruptive behaviors and less substance abuse (MST); less juvenile justice involvement (MST); reduction in PTSD, depression symptoms, and total behavior problems (TF-CBT); and less disruptive behavior (TripleP). Moreover, parents showed greater improvements in parenting practices (PCIT), parenting efficacy (TripleP), and decreases in parental anger (TripleP) and parental stress (PCIT).
While these studies offered insight into which programs are effective and relevant in a CW services context, they did not connect implementation contexts to outcomes. In other words, the underlying contextual conditions that increase the likelihood that these programs were implemented to fidelity and contributed to achieving intended outcomes are less understood. Without clear understanding of these pathways, it will be increasingly difficult to replicate procedures or offer guidance to CW workers and leaders to validate outcomes in other contexts. The unresolved underlying question herein is how context is operationalized. To help answer this question, scholars have proposed several implementation science frameworks, including the Practical, Robust Implementation and Sustainability Model (PRISM) (Feldstein & Glasgow, 2008) to assess program maintenance and outreach in health care contexts, the Exploration Preparation Implementation and Sustainment (EPIS) framework to guide implementation efforts in CW and mental health agencies (Aarons et al., 2011), and the Consolidated Framework for Implementation Research (Damschroder et al., 2009), to help researchers enhance theory and validate existing factors that are hypothesized to influence implementation outcomes. After Albers et al. (2017) completed an exhaustive review of implementation frameworks, they pointed out that all of them highlighted that the implementation “ecology” or implementation context (e.g. organizational capacity, culture, climate, leadership, and positive attitudes toward evidence-based practices (EBPs)) must be optimized. However, they asserted that many of the frameworks are atheoretical, not supported by rigorous research designs, and that some of the frameworks are universal (e.g. EPIS, PRISM) while others are geared toward specific treatments. In that regard, more effort should be devoted to engaging in theory building and logic modeling to examine how the inner context (individual provider characteristics and organizational factors) and the outer context (socio-political and interorganizational networks) might be interrelated and collectively influence whether ESPIs are implemented and, ultimately, whether clients achieve intended outcomes. Irrespective of their limitations, they provide insight into the “ingredients” or elements that may be needed to ensure clients have the opportunity to engage in the most effective services.
Acknowledging gaps in existing frameworks, Garcia et al. relied on interview data collected from scholars and agency leaders (2019) and caseworkers and supervisors (2020) to propose a model linking implementation processes to client outcomes. These data revealed that implementation climate, often influenced by the outer context (interorganizational networks and funding), was not optimized for engaging in cultural exchanges about TripleP. Cultural exchanges, defined as exchanges of knowledge, attitudes, and practices that occur when two actors (e.g., agency leaders, clinicians, or workers) representing diverse perspectives or identities engage in a process of debate and compromise (Palinkas et al., 2005), did not occur on an ongoing basis. This presented as a barrier, given that the climate eroded when leaders struggled to engage in transformational leadership and address intermittent use of research evidence by supervisors and staff. In particular, agency supervisors often did not model or encourage how to simultaneously balance (1) innovation of EBP implementation, (2) how to locate or apply evidence or information to effectively engage in the client referral and implementation process, and (3) crisis-oriented casework. However, they endorsed positive perceptions, attitudes, and beliefs about the interventions when exchanges of knowledge about the EBP process occurred. When exchanges did occur, directors, supervisors, and workers were more attuned to the concrete and perceptual barriers clients often grapple with during the implementation process. With perceptions solidified, practices were implemented, namely leaders provided adequate support and training, caseworkers referred clients to TripleP, and clinicians effectively engaged parents. In turn, scholars and directors were likely to endorse that EBPs play a fundamental role in achieving safety, permanency, and well-being.
Research aims
The current study sought to validate the model proposed by Garcia et al. (2019, 2020) by analyzing (1) survey data collected from workers and leaders involved in the early process of implementing TripleP and (2) pre- and post-data from caregivers who completed the intervention. By linking implementation process factors to client outcomes, we addressed the following questions: (1) What effect does TripleP have on achieving well-being outcomes (parental perceptions of resilience, social connectedness, concrete support, and their children’s social and emotional adjustment) in CW agencies? and (2) Which individual characteristics and organizational–social factors influence outcomes?
Methods
Setting
To address the aims of the current study, we analyzed data collected for the Promoting and Empowering Positive Perceptions of Evidence-Based Parenting (PEP2) in CW study (Garcia et al., 2019, 2018). Originating in 2015, PEP2 is a research–practice partnership developed to prospectively evaluate the process of implementing the TripleP. Of the five programmatic levels of TripleP, leaders in a large U.S. Mid-Atlantic city offered Level 3-group and Level 4-individual to parents between December 2015 and May 2017. Level 3 targeted parents who might benefit from learning new skills in a group setting to enhance parenting capacities, while Level 4 addressed the needs of parents with children who exhibited severe behavioral difficulties. This set of analyses focused solely on one agency that provided CW services and recruited and trained their own employees to deliver Level 3-group. The agency was responsible for monitoring and providing services to children who either (1) remained in the home with biological parents who agreed to engage in services to reduce harm or future risk of maltreatment, or (2) were placed in foster care and efforts were underway to work toward reunification.
Procedures
Agency workers, inclusive of case managers, supervisors, directors, and other staff were invited to complete a battery of surveys, listed below, about their attitudes toward EBP, intentions to refer parents to TripleP, and perceptions of the agency culture and leadership in September 2016 and February 2017. Recruitment procedures involved (1) attending agency all-staff meetings, (2) encouraging agency leaders to send out email reminders about the study, and (3) convening data collection meetings onsite in the mornings and afternoons to recruit participants and address any questions about the PEP2 study. After providing consent, participants completed the surveys, and the first and second authors assigned them a unique identifying number instead of listing their name for confidentiality. A separate list matching a study participant’s number with their real name was created and stored separately from the raw data. The PEP2 study was approved by the IRB from state, city, and university entities, and a data license agreement was obtained.
Measures
Demographics: Data on participants’ age, gender, race/ethnicity, level of education, language ability, years worked with at-risk youth, and role in the agency were collected by administering a demographic questionnaire developed for the PEP2 study.
Evidence-Based Practice Attitude Scale (EBPAS) is a globally used brief 15-item scale that assesses four general attitudes toward adoption of an EBP, including appeal (whether the EBP is appealing), requirements (extent to which the agency requires the EBP), openness (willingness to innovate), and divergence (difference between the EBP and current practice) (Aarons, 2004, 2005; Kim, 2019, 2020). Participants rated each item using the following scale: 0 (not at all) to 4 (very great extent). With alphas for the four domains ranging from .66 to .91 and an overall scale alpha of .76, the EBPAS has good internal consistency (Aarons, 2004,, 2005).
Structured Interview of Evidence Use (SIEU) (Palinkas et al., 2016) is a 45-item instrument that asks respondents to indicate the extent to which they obtain research evidence from a variety of sources, including the internet or academic journals (input subscale, 17 items). Questions are also asked to understand how providers assess the validity and reliability of research evidence (process subscale, 16 items), and the circumstances in which the evidence is used or not used when implementing an EBP (output subscale, 12 items). For each item, participants were asked to assess their level of agreement with questions using a Likert scale ranging from 1 (not at all) to 5 (all the time) for the items included in the input subscale, and a similar five-point Likert scale ranging from 1 (not important) to 5 (very important) for the items included in the process and output subscales. Palinkas et al. (2016) found that the total SIEU measure (α = .88) and all three primary scales (input = .80; process = .86; output = .80) have high internal consistency. In their same study, convergent and discriminant validity were also established. In light of the measure’s intention to focus on leaders’ decision-making, directors and supervisors completed the entire measure. While caseworkers are not expected to gather and interpret evidence to the extent leaders often do, they were asked to complete the “output” subscale. As “on the ground decision-makers,” caseworkers are often placed in situations that may or may not warrant using evidence.
Implementation Leadership Scale (Aarons et al., 2014) is a 12-item measure that examines the degree to which system leaders and supervisors support their staff in implementing an intervention. The measure asks staff to rate on a scale from 0 to 4 (0 representing not at all and 4 signifying very great extent) the degree to which they feel their supervisor is proactive, knowledgeable of EBP, supportive, and perseverant in the EBP implementation process. Directors and supervisors completed a version of the same types of questions to rate their own leadership skills. Aarons et al. (2014) concluded the measure has high internal consistency (α = .98) and strong discriminant and convergent validity.
Organizational Readiness for Change (ORC) encompasses the following four domains measured with 67 items divided in several subscales: motivational readiness (three scales), resources (two scales), staff attributes (two scales), and organizational climate (four scales) (Lehman et al., 2002). All items were rated on a five-point Likert scale (1 = strongly disagree to 5 = strongly agree). For the purposes of the PEP2 study, leaders had a keen interest in understanding how climate and motivational readiness influence implementation processes and client outcomes. Moreover, studies have concluded that both climate and motivation play significant roles in facilitating the implementation process (see Glisson & Green, 2008; Palinkas & Soydan, 2012). Accordingly, participants were administered the 38-item organizational climate subscale that measures the following: mission, cohesion, autonomy, communication, stress, change, and leadership. In addition, they were asked to complete the organizational motivation to change subscale, which included 23 items that assessed program needs, training needs, and pressure for change.
Cultural Exchange Inventory (CEI) (Palinkas et al., 2018) includes two subscales measuring (1) respondents’ time and energy toward maintaining meaningful collaboration and communication with colleagues (exchange process, seven items) and (2) the extent to which practice behaviors and/or opinions changed (exchange output, eight items). For this study, the “exchange output” subscale was administered to agency supervisors and directors only. The study’s central focus was to assess the impact of their modified practice behaviors and opinions due to exchanges with other key decision-makers. They were asked to respond to questions using a Likert scale ranging from 1 (not at all) to 7 (a great deal). Examples of questions included: “I feel like I am changing my opinion about something because of these exchanges,” and “I feel like we are changing their practices because of this collaboration.” A total exchange output score was calculated. Palinkas and colleagues conclused that the measure has strong internal consistency and reliability with an overall alpha for the process and outcomes scales ranging from .95 to .99, depending on which organizations were participating in the exchanges. In addition, they found some support for convergent validity (relative to the EBPAS) and strong support for the discriminant validity, as indicated by low correlations with organizational climate and structure.
Intention to Refer uses two item stems developed by Ajzen (2006). Case managers and staff were asked to rate on a scale from 1 (very unlikely) to 7 (very likely) their response to “How likely are you to refer all eligible families on your caseload to TripleP in the next four months?”. Next, they were asked to rate on a scale from 1 (strongly disagree) to 7 (strongly agree) their response to “I intend to refer all eligible families on my caseload to TripleP in the next four months.” A mean score of the two items was calculated.
Dependent variable: During the window of time the agency offered TripleP (between December 2015 and May 2017), parents completed the Parents' Assessment of Protective Factors, a measure that calculates a protective factors index (PFI) during the first session (Time 1) and after graduating from the program (Time 2). The PFI is a total measure of a parent’s self-reported beliefs, feelings, and behaviors regarding four protective factors: (a) parental resilience, (b) social connections, (c) concrete support in times of need, and (d) social and emotional competence of children (youngest child in home). As reported by Kiplinger and Browne (2014), the alpha coefficients for all four protective factors are greater than .85, and the alpha for the entire PFI is .95. The social connections subscale shows the strongest internal consistency (α = .93), while the parental resilience subscale demonstrates the highest overall reliability (CR = .95). Strong convergent validity of the subscales was established. Caregivers were asked to respond to the 36 protective factor items using a Likert scale ranging from 0 (This is not at all like me) to 4 (This is very much like me or what I believe). A total PFI score as well as subscale scores for each of the four aforementioned areas were calculated. Table 1 provides sample alpha coefficients for each of the aforementioned measures.
Cronbach’s alpha coefficients.
CEI: Cultural Exchange Inventory; EBAPS: Evidence-Based Practice Attitude Scale; ILS: Implementation Leadership Scale; ORC: Organizational Readiness for Change; PFI: protective factors index; SIEU: Structured Interview of Evidence Use.
Data merging process: Several steps were taken to merge the survey data collected from leaders, caseworkers, and staff with that of the parental protective factors measure completed by parents. First, leaders provided the name of the supervisor for each caseworker and staff member who completed a survey packet. Second, the authors' grouped all of the caseworkers by “units,” defined as those who were supervised by the same leader. Third, unit average scores for each of the following measures were calculated: ILS, ORC, EBPAS , SIEU, and CEI, thereby deriving an average score for each of the survey measures within each supervisory unit. Fourth, the names of caseworkers and supervisors who were assigned to manage the cases of caregivers who participated in TripleP were identified, and then caregivers were assigned to their respective “unit.” To that end, the one identifier linking survey data from agency workers to parental protective factors and capacities is “units.”
Samples
Agency workers: A total of 59 agency workers, including caseworkers (n = 23), intervention team members (n = 12), case aides or advocates (n = 9), and supervisors (n = 15), participated in the study. Of 59 workers, 69.5% identified as female, and the mean age of all participants was 34.6 years (SD = 8.9). The majority of them identified as African American (83.1%), followed by Caucasian (11.9%) and Other (5.1%). Over 37% had earned a master’s or PhD degree while the remaining participants had at least a high school diploma. Agency workers had been working with at-risk youth and families for an average of 7.2 years (SD = 7.3). A total of 14 units (or groups of workers) assigned to the same supervisor were included in the sample.
Caregivers: Of the entire sample of 57 parents who completed the pre- and post-PFI scale, nearly 90% identified as African American (n = 51). The remaining caregivers identified as Asian American (n = 3), Native Hawaiian/Pacific Islander (n = 2), or Native American (n = 1). In terms of gender, 85% reported they were female. Twenty-two percent of the sample had received a high school diploma, and less than a third of the sample (29%) earned more than $6000 per year. Most of the children in the family were 0–2 (36%) and 5–11 (36%) years of age, followed by ages 3–4 (23.4%) and 12–18 (4.5%). While an additional 27 parents completed the PFI at Time 1, they terminated services prematurely and did not complete the measure at Time 2; hence, they were excluded from the analyses.
Analyses
A paired samples t-test was employed to examine whether caregivers experienced changes in the PFI as well as in its four subscales before (Time 1) and after (Time 2) TripleP was offered by the agency. For the outcome measures that changed significantly between Time 1 and 2, linear regression, where the change scores between Time 1 and 2 were included as dependent variables (Allison, 1990), was conducted to investigate the extent to which the changes were associated with caregivers’ individual and organizational characteristics. Given the relatively small sample size of the study, regression models were conducted in which the two types or sets of predictors (i.e. individual characteristics and organizational characteristics) were included separately. When significant organizational predictors (i.e. implementation leadership and attitudes toward EBPs) were detected, additional models were estimated to examine whether the significant effects remain after controlling for significant individual characteristics (i.e. caregiver race, income, and education levels).
Results
Table 2 reports the results of the paired samples t-test. Results showed a significant increase between Time 1 and 2 scores in social connections: t(56) = 2.98, p < .01; and for the overall PFI scale: t(56) = 2.32, p < .05. Results from the regression models for social connections and the overall PFI are reported in Tables 3 and 4. The ORC, SEIU, CEI, and Intention to Refer measures were not included in the multivariate models, given that bivariate regression findings did not show a significant relationship between them and these two aforementioned scales.
T-test results.
Note: *p < .05, **p < .01.
Regression results (DV: changes in social connections).
EBAPS: Evidence-Based Practice Attitude Scale; ILS: Implementation Leadership Scale.
Note: † < .1, *p < .05, **p < .01.
Regression results (DV: changes in PFI).
EBAPS: Evidence-Based Practice Attitude Scale; ILS: Implementation Leadership Scale; PFI: protective factors index.
Note: * p < .05, ** p < .01, *** p < .001.
Regression models for the remaining subscales (i.e. parental resilience, concrete support in times of need, and social and emotional competence of children) were not conducted, given that significant changes in their scores were not detected, as shown by the paired samples t-test results.
Social connections
Parents with lower income (B = −0.56, p < .05) and educational attainment (B = −0.55, p < .05) were less likely to experience an increase in social connections. Parents who received services from agency workers who felt that EBPs were required were also found to experience fewer changes in social connections (B = −1.89, p < .05). However, the association lost its significance after controlling for income and education levels. On the other hand, parents who received services from organizations inclusive of supportive leadership (B = 1.14, p < .05) experienced greater increases in social connections, although the association became marginally significant (p < .10) after adjusting for caregivers’ income and education.
PFI
Similar to social connections, parents were likely to experience less change in protective factors when they (1) had lower educational attainment (B = −0.34, p < .05) and (2) received services from agency workers who felt EBPs were required (B = −1.74, p < .01); however, scores increased as supportive leadership increased (B = 0.84, p < .01). Parents who identified as African American scored less of an increase in PFI compared to those in other racial categories (B = −0.37, p < .05). Finally, parents who were engaged with agency workers who felt EBPs were appealing were likely to have greater increases in PFI (B = 1.41, p < .01).
Discussion
This study used staff survey data and parent clinical outcome data to determine (1) whether TripleP impacted parent and child well-being and (2) which individual characteristics and organizational–social factors influenced outcomes. Results showed that TripleP had an impact on protective factors and social connectedness. Additionally, both parent characteristics and implementation factors influenced whether TripleP engagement influenced these outcomes.
Effectiveness of TripleP on clinical outcomes
Study results showed an increase in both social connectedness and protective factors for parents. TripleP’s effect on social connectedness is supported by the extant literature that shows parent training programs can enhance parental social connections through the exchange of ideas and resources and through the giving and provision of peer support (Wilder Research, 2016). Furthermore, parents in the CW system have self-reported that being able to connect with other parents who were experiencing similar circumstances facilitated their engagement in an evidence-based parenting program (Garcia et al., 2018). TripleP has also been found to positively impact parents’ protective factors, specifically parental resilience, social connectedness, concrete supports, and social and emotional competence of children. While this finding corresponds with TripleP’s aim to enhance modifiable family protective factors associated with dysfunctional parenting practices (Sanders, 2008), previous studies have typically examined outcomes related to parenting style (De Graaf et al., 2008) and parent adjustment, satisfaction, and efficacy (Sanders et al., 2014). These findings add to the evidence base for TripleP’s ability to foster resiliency, though it should be acknowledged that protective factors have been measured differently across these studies. Future research is needed to examine these outcomes and determine whether these findings can be validated in other contexts and populations.
Impact of parent characteristics
Several parent characteristics moderated the impact of TripleP on social connectedness and protective factors. In particular, individuals who had lower incomes and lower educational attainment reported fewer increases in social connectedness and protective factors. This finding is consistent with results from a meta-analysis that found low income and educational attainment were associated with poorer treatment outcomes in parenting programs (Reyno & McGrath, 2006). In addition, the present study showed parent race/ethnicity impacted TripleP’s effectiveness. Parents who identified as African American reported fewer increases in protective factors than parents of other races and ethnicities. A meta-analysis of parent training interventions conducted by Ortiz and Del Vecchio (2013) showed that while the interventions were developed to address the needs of the majority population in mind, they have also demonstrated effectiveness for ethnic minority families. However, the studies they reviewed focused on parent outcomes that do not encompass protective factors. Perhaps it is the unique experiences of African Americans, including a greater likelihood of experiencing structural poverty and a lack of access to community resources (Boyd, 2014), that have a greater impact on protective factors than the skills emphasized in parent training programs. Future research directly exploring these connections is warranted (Ortiz & Del Vecchio, 2013), especially given the disproportionate rate at which African American children are involved in the U.S. CW system (Boyd, 2014; Harris, 2014). While African Americans account for 14% of the children in the U.S. (Annie E. Casey Foundation Kids Count Data Center, 2019), they represent 21% of the children who are placed in out-of-home care (UDHHS, 2019b). These inequities should prompt CW scholars and advocates to identify and amplify resilient contexts (e.g. social supports, safe neighborhoods and communities with ample resources) while implementing EBPs, such as TripleP, as a potential strategy to support African American caregivers and prevent out-of-home placement.
Impact of organizational–social factors
Results showed the impact of caseworker attitudes toward EBPs and implementation leadership on TripleP’s effectiveness in bringing about positive parent outcomes. Parents who had caseworkers who believed EBPs were required experienced less social connectedness and lower protective factors scores. However, parents assigned to caseworkers who viewed EBPs as appealing had higher increases in protective factor scores. While there is literature to show that requiring EBPs is more impactful than perceptions of appeal of EBPs for caseworkers (Leathers et al., 2016; Lopez et al., 2011), other researchers have found a positive impact of appeal of EBP on implementation (Czincz & Romano, 2013). In particular, Czincz and Romano (2013) showed that higher EBP appeal increases openness to EBP use among providers, adding that caseworkers need to believe that the EBP is beneficial for their clients for implementation to succeed (Aarons & Palinkas, 2007). Another implementation factor that moderated TripleP’s effectiveness was caseworkers’ ratings of implementation leadership. Parents with caseworkers who highly rated their organizations in supportive leadership had greater increases in social connectedness and protective factors. This finding is consistent with previous literature highlighting the importance of supportive leadership to caseworkers when implementing and sustaining new innovations in CW (Aarons et al., 2016; Michalopoulos et al., 2012). Interestingly, the effect of supportive leadership and attitudes toward EBPs on social connectedness diminished when accounting for income and educational attainment. Since family income and educational attainment were found to have large and moderate effects, respectively, on treatment outcomes in parenting programs (Reyno & McGrath, 2006), it may be that these factors overshadowed the moderating effects of caseworker and implementation factors. Future research is needed to explore these pathways further.
Implications
Results underscore the need to promote supportive leadership, embrace a culture of acceptance of EBPs—versus a culture of mandating EBPs—and to develop strategies to make EBPs more appealing and acceptable to staff. Supportive leadership is characterized by leaders who appreciate staff efforts to learn and use EBPs (Aarons et al., 2014). In CW organizations, supportive leadership may involve providing caseworkers with the time and emotional space to engage in cultural exchanges to learn more about EBPs and positively shift their perspectives (Garcia et al., 2019). Since high caseloads and constant crises may make it difficult for staff to research and learn about EBPs on their own, supportive leadership may entail offering caseworkers the opportunity to train with clinicians (Dorsey et al., 2012) and/or providing a central repository of available EBPs that staff can easily access (Garcia et al., 2015).
Study results also support the strategy of creating a culture of appeal as opposed to mandating EBPs. Simply mandating the use of an EBP, according to this study, may diminish the effect of EBPs. While many aspects of CW practice are constrained (Smith & Donovan, 2003), caseworkers do typically have some discretion in the services they refer families to and how they refer families. Caseworkers typically are more likely to refer families to services that align with case goals and are a good fit for a family’s needs (Aarons & Palinkas, 2007). Bunger et al. (2009) discussed how caseworkers have the ability to use informational referral strategies (providing information for services, verbal suggestions) as opposed to social referral strategies (helping clients complete paperwork, accompanying clients to appointments). While mandating an EBP may increase referrals, it may not increase the caseworker behaviors (i.e. social referral strategies) that will promote parental engagement and treatment effectiveness.
Of note is the impact of caseworker factors on EBP effectiveness. In CW contexts, caseworkers are not the EBP providers but are the referring agents to EBPs. Previous EBP implementation research has typically focused on intervention providers and EBP consumers. However, research has demonstrated that a lack of referrals or lack of caseworker participation in EBP implementation can have a deleterious impact on the EBP implementation process (Dorsey et al., 2012; Whitaker et al., 2012). Additionally, the current study demonstrates the effect that caseworker factors can have on treatment effectiveness. Therefore, it is critical that policymakers and organizational leaders intentionally engage, train, and support caseworkers throughout all phases of EBP implementation.
Taking these findings into account, we conclude that the hypothesized model presented by Garcia et al. (2019, 2020) is partially supported. In sum, results showed that individual adopter characteristics (i.e. attitudes toward EBPs) and transformational (i.e. supportive) leadership were salient factors in promoting parent and child well-being. When interpreting these findings in relation to the hypothesized model, it should be noted that the “external context” was not measured. The remaining key themes—implementation climate and cultural exchanges—did not emerge as significant processes toward achieving well-being. However, future research replicating this study with larger sample sizes in diverse populations and communities is necessary before it can be concluded that these processes are unnecessary.
While Garcia et al.’s (2019, 2020) model is based upon the experiences of workers, supervisors, and directors who implemented TripleP, the objective is to eventually develop a model that could be used to inform the implementation processes for any given EBP. To that end, the model assumes that fidelity is adhered to. In this case, TripleP clinicians were trained offsite in Columbia, South Carolina, by certified trainers to implement TripleP to fidelity, albeit with modified case studies to reflect the diversity, preferential values, and culture of the parents they served. The parents, for example, were given the opportunity to share their “challenging” parenting moments during group sessions (Garcia et al., 2018). Clinicians, with their permission, incorporated these challenging moments into the curriculum without disrupting the core components of the model.
Limitations of the study
While there are several key implications in the study’s findings, we must call attention to limitations. First and foremost, the sample size is relatively low, partly due to the fact that 27 parents completed the PFI at Time 1 but terminated services prematurely and did not complete the PFI at Time 2, thus decreasing the sample size from 84 to 57 parents. The lower than expected sample size is typical in CW research and practice, however. Parents involved in the CW system are transient and face numerous concrete barriers (transportation, lack of daycare, language, long waiting lists) and perceptual barriers (stigma, discrimination) to completing all of their required or court-mandated services (Garcia et al., 2019, 2018; Lewis et al., 2016; Schreiber et al., 2013). Scholars have previously reported similar challenges with attrition in longitudinal studies while tracking outcomes over time (Fernandez & Eyberg, 2009; Kazdin, 1996; Mersky et al., 2017; Weiner et al., 2009). For example, 36 out of 99 participants who enrolled in PCIT dropped out prematurely (Fernandez & Eyberg, 2009), and only 41 out of 205 families who consented to participate in SafeCare completed all three training components (Gershater-Molko et al., 2003). Thus, the overall sample size herein is not vastly different from that of previous studies. Regardless, future efforts should be devoted to engaging clients or participants in completing EBPs to fidelity by (1) inviting parents who graduated from EBPs to share their journey to new parents who are referred to EBPs and (2) providing caseworkers with adequate support and resources to eliminate concrete and perceptual barriers and improve the therapeutic alliance (Garcia et al., 2018).
Moreover, the current study did not identify moderators and mediators or variables that may indirectly influence the relationship between significant predictors and parenting outcomes. A larger sample size, coupled with administering additional measures that assess other dimensions of leadership, organizational functioning (e.g. culture, demands, and stress), and self-efficacy among caseworkers, would likely help address this gap (Collins-Camargo & Royse, 2010).
Given the limitations to some of the data obtained for this study, the authors' were unable to examine the relationship between individual and organizational characteristics and other outcomes, such as safety and permanency, and other dimensions of well-being (e.g. health/mental health and educational outcomes). Additional research is needed to link implementation processes to other outcomes. Related to that, it is often standard practice to statistically account for the nested structure of data (i.e. parent and youth dyads nested within CW service units). However, given the relatively low sample of dyads compared to units, we would induce more bias if hierarchical linear modeling was used to adjust for nesting (Raudenbush & Bryk, 2002).
Lastly, the authors were not authorized to collect data from clients who did not receive TripleP or other EBPs during the study timeframe, and thus, making comparisons between a control and an experimental group was not feasible.
Concluding remarks
Despite these limitations, this is a rare and important study, indicating which implementation factors or processes play a significant role in achieving intended outcomes. Results underscore the need to increase worker buy-in rather than mandate the use of EBPs, such as TripleP. Intended client outcomes are more likely to be achieved when agency leaders (1) provide workers support to learn and use EBPs and (2) rely on methods to increase EBP appeal. Future research is needed to (1) understand why the implementation context and/or some of the strategies embedded within the TripleP model are less effective for African Americans compared to other racial/ethnic groups, (2) validate findings in other agency contexts and with other EBPs, and (3) recruit a larger sample size in more diverse communities.
Footnotes
Ethics
University, state, and local IRBs gave ethical approval for the research.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The University of Pennsylvania provided financial support for the research study.
