Abstract
This special issue of Child Maltreatment (CM) offers a series of articles each addressing an aspect of an emerging area of understanding that lies at the very heart of our efforts to improve the quality of services for abused and neglected children and their families. Throughout my 40-year-career, I have often been surrounded by professional colleagues who aspired to consistently improve their practice. For much of that time, the pursuit of better practice was driven by the ideas and words of respected leaders in the field and delivered at countless conference workshops, in books, and professional publications. We would hear from others of some new innovation that sounded promising and we would rush to adopt or adapt it. The problem was (and to some extent, still is) very little of what was passed around as “best practice,” was based on scientific principles or previous research. Even less had been empirically tested in its own right to see if it actually worked.
Thankfully the CM field is maturing, and many now look upon their practice environment with an eye toward employing proven models, tested in rigorous studies. This trend emerged in the mid-2000s and has accelerated rapidly. At first, some saw the move to “evidence-based practices” (EBP) as the next fad in a long history of “splash in the pan” initiatives that have been all too common in child protection work over the last 40 years. But this time it was different. This time we were basing our decisions not on the latest clever idea delivered by a charismatic leader, but on actual results that had been empirically tested and proven in real-world settings. This movement to EBP is not a magic bullet. No one is saying we know all the answers or that one intervention can solve all our problems. On the other hand, we now have answers for some children and families and, I believe, we have an obligation to connect the right empirically supported practice to the right child and family when it does fit.
In 2004, with support from the Kauffman Foundation in Kansas City and the involvement of a group of very smart people, we published an article on the barriers to implementing three research-based “best practices” (Chadwick Center, 2004). This was published at a time when few were using the phrase “evidence-based practice” in relation to mental health or social services. We found over 100 identifiable barriers as to why professionals had not yet widely embraced three research-tested practices: Trauma Focused-Cognitive Behavioral Therapy (TF-CBT), Parent–Child Interaction Therapy, and Abused Focused-Cognitive Behavioral Therapy (AF-CBT; later retitled Alternative for Families—Cognitive Behavioral Therapy). Few of the barriers we found were really about the actual treatment models. Most of the identified barriers had to do with translating the research into everyday practice settings. We noted opposition to change and an entrenched status quo who resisted the idea of EBP on principal as major barriers to expanding the use of these and other proven interventions. We also found barriers around system issues such as funding to support the transition to EBP, including the cost of lost productivity associated with time spent in training or in learning a new practice.
As you review this issue, you will see some of the barriers we cited in 2004 remain challenging today, but significant progress has been made. In the space of seven short years, more and more professionals appear willing to adopt manualized interventions, as Allen et al. points out in this issue of CM. Perhaps, this is the result of advocacy for EBP, changing expectations of funders and accrediting bodies, such as Allen describes, and a growing number of organizational role models who have adopted EBPs with success. We have come a long way but as multiple authors in this issue illustrate we have much further to go.
Early in the last decade, the Governor of California asked a cross section of leaders in the state how best to redesign the child welfare system. Key among the recommendations that these stakeholders produced was that any change that counties undertook should be guided, as much as possible, by the available empirical evidence of what works. That recommendation was the foundation for the development of a web-based resource to inform child welfare administrators, many of whom were not conversant with the research literature, and had limited knowledge for what practices had empirical support and what did not. The web-based resource culminated in 2006 as the California Evidence-Based Clearinghouse on Child Welfare (www.cebc4cw.org). The initial goal of the clearinghouse was to help identify the strength of the science behind practices commonly used by child welfare across the nation to serve and support families. The Clearinghouse staff quickly concluded, however, that spreading the knowledge about the level of evidence behind common practices was important, but it was not sufficient.
Over the last 6 years, we have seen a dramatic increase in knowledge among policy makers as well as frontline clinicians and case managers about specific EBPs. The phrase evidence-based practice has flooded our lexicon and people who had never read a research article in their career can now talk with some authority about multiple evidence-based models. In this issue, you will find that Allen and colleagues discovered that a national sample of clinicians were able to correctly identify multiple evidence-based treatments. This would not have been true 5 or 10 years ago. This is all huge progress, but it is not enough.
Allen and colleagues also discovered the same clinicians who were prone to incorrectly believe some common interventions, which lack empirical support, were also “evidence based.” Many in the same study also persist in using unproven treatments they are accustomed to even knowing that there is no evidence of their efficacy.
The true test of movement toward EBP, however, is not in using the phrase correctly, knowing the names of well-supported interventions, listing specific EBP’s in agency marketing material, or even receiving the intervention training. The true test of EBPs is in their implementation: when and how the models are implemented, with whom, and how well the infrastructure in which they are embedded is prepared to take advantage of the power of these empirically supported models.
In 2005, Fixsen and colleagues described three levels of implementation, Paper, Process, and Performance (Fixsen, Naoom, Blase, Friedman, & Wallace, 2005). It is easy to move to EBPs on paper. All one needs to do is change what you call it and then pronounce your organization “evidence based.” We have seen some model developers and purveyors merely tell the world they are “evidence based” without conducting a single study. In these cases nothing changes other than the labels and marketing materials. Some organizations have been satisfied to do this or have deluded them into thinking using the right catch phrase is good enough.
Most respected organizations, however, want more. They legitimately seek to improve practice and so they go further. They change training, they change manuals, and they alter the patterns of supervision. They change their processes but they may still fall short, as process change alone often fails to produce the clinical results expected at the child and family level. In the end, we should all be seeking what Fixsen called performance-level change. There, we see the service provider and the child or family member together actually using the EBP with fidelity. This is a quantum leap beyond just changing nomenclature, policies, training, and processes. At this level providers develop, as Shanley and colleagues suggest in this issue, competency and fidelity to the model. When we have embedded the change with competency and fidelity in service delivery and organizational culture and it becomes “just how we do business now,” only then will we have truly succeeded.
Achievement of this level of performance change turns out to be very complex. Careful thinking, practical experience, and the use of emerging implementation research makes a real difference. The articles selected for this issue provide insights into the multifaceted world of implementation and build on an existing body of literature (Aarons et al., in press; Aarons, Hurlburt, & McCue Horwitz, 2011; Glisson, 2006; Mendel, Meredith, Schoenbaum, Sherbourne, & Wells, 2008; Proctor et al., 2011; Raghavan, Bright, & Shadoin, 2008) that can be used to drive change in ways that produce real improvements.
A serious move to implement EBP at the agency or community level requires preplanning and engagement of a number of key stakeholders long before the first training session. Those seeking to improve services must not only select the right practice but must build the infrastructure to support the practice.
A number of factors appear to influence successful uptake of EBP and when taken together serve as a road map for those aspiring to move to EBP service delivery environment whether at the state, community, or agency level.
Often at the heart of successful change is a leader or leaders who align the assets and resources needed to affect change in ways that translates to real improvement. Those leading the change must develop strategies that impact of at least four levels of system change, as outlined in the Institute of Healthcare Improvement (IHI). Leaders typically recognize they must determine how best to impart the necessary knowledge, skills, abilities, and attitudes to those who will be delivering the services. As pointed out by several of the authors in this special issue, however, focusing on this level of the system alone is not enough. Leaders must also win over what IHI calls the “micro system,” the unit level with coworkers and supervisors who have immense influence over the attitudes of those learning and delivering the new practice. Both these levels are influenced further by organizational-level messages and actions from the words and attitudes of leaders to the policy framework in which services are delivered all of which should be aligned. Even when accounting for the services delivery level, the microsystem and the organizational level, leaders of improvement must also address the community context that shape practice from regulatory and funding systems to referral sources, attorney’s, and the courts. Leaders need a road map, such as the one that follows, to help guide them through this process to ensure they address all the key levels.
Road Map to Implementation of EBP
Selecting the Right Practice
Most of the articles in this issue address the experience of implementing specific EBPs such as AF-CBT, TF-CBT, Triple P, and SafeCare®. For those planning to implement a major change, such as the introduction of a new practice or clinical model, it is imperative you slow down, examine how you are trying to improve or change your agency services, and first be sure you are seeking to implement the right practice. Perhaps, the first step to successful implementation is ensuring the leaders understand exactly what problem they wish to influence and what underlying factors are driving the condition they are focused on. Only then can they select an EBP best suited to address the specific problem. Any practice selected must also fit the community and agency environment where it will be delivered. Once the areas of clinical priority are identified (e.g., parent substance abuse, symptoms of traumatic stress, parenting knowledge and skills, behavior management, etc.) then one can begin the search for an EBP/EBPs that will have maximum impact upon the targeted issues.
Finding an EBP that is designed to fit the problem is only part of the process. One should actually consider all EBP models addressing the targeted issues and select the model that not only has strong empirical support but also is the best fit for the community and those who will be delivering it. Drawing from lessons learned in a systematic review of the diffusion of innovation in service delivery systems (Greenhalgh, Robert, Macfarlane, Bate, & Kyriakidou, 2004), the California Evidence-Based Clearinghouse for Child Welfare offers a simple implementation guide outlining the factors that those selecting practices should consider before deciding if a model fits. These factors include the compatibility of the model with (1) the workforce who will be delivering it, (2) the families who will be receiving it, (3) the referral sources, and (4) even those who will be expected to fund it. Factors such as how complex the model sounds to the person expected to deliver it and the related degree to which they believe they can be successful (efficacy) appear to link to effective uptake of change. In fact, many of the articles in this issue reinforce these points. Shapiro and colleagues as well as Shanley and colleagues stress the importance of the match between the intervention and the workforce that is to implement the new practice. Other articles in this issue highlight the importance of the fit between the intervention and how it is delivered and the clients to be served, such as Damashek and colleagues’ discussion of the role of culture in client engagement and minimizing service attrition. Once a practice or set of practices are identified for adoption, however, the work has just begun.
Building Assessment and Referral Pathways
If children and families are not matched with an appropriate EBP, the expansion of EBPs alone will have little meaning to families and little impact on outcomes. Ensuring that children and families are properly assessed and referred is a key step in the delivery of EBPs. Agencies such as child welfare and juvenile justice, who serve as key referral sources, must have strong screening and assessment processes able to broadly differentiate which EBP provider is the best fit for the unique needs of specific children and families being served. No implementation of EBP can succeed unless the referral sources are primed to connect the right children and families to the appropriate provider. In fact, as several authors point out in this issue of CM, the failure to educate and engage the referral sources early on can, as Whitaker and colleagues described, mean too few referrals. Insufficient referrals not only fails to fully utilize the new service capacity but, as several articles in this issue point out, can impede the pace of learning by the professionals implementing the new practice, who quite frankly need the practice while the training is fresh and the coaching is available to make the EBP their new way of business. Innovative referral agent interventions, such as described by Dorsey et al. in this issue, may be another step we eventually learn are a critical factor of successful intervention and services level changes.
Implementation Management
Implementation of a specific model needs to fit into a wider strategy to support the implementation. Kolko and colleagues briefly describes elements of such a strategy in their article in this issue as including eight phases: (1) readiness/preparation; (2) engagement; (3) orientation to new model; (4) systematic training; (5) consultation (although some EBP purveyors described in this issue prefer the term coaching that sometimes includes following the trainee into the field); (6) implementation guidelines; (7) activities that promote ongoing adoption; and (8) planning for sustainability. Others around the nation have developed formal implementation support models too such as the California Institute of Mental Health Community Development Teams (CIMH, 2006), the IHI’s (2011) Breakthrough Series Collaborative model, used extensively by the National Child Traumatic Stress Network, the Replication Effective Programs model (CDCP, 2012) and Texas Christian University’s Program Change Model (Simpson & Flynn 2007). Some of these, like the Availability, Responsiveness, Continuity Model (Glisson, 2006) are being empirically tested to provide an evidence base to implementation. Whether utilizing one of these models described in the literature or not, system-wide implementation will be enhanced by the presence of a change manager whose job is to facilitate and support the change and help build a sense of mutual accountability across the service delivery continuum.
Other Key Implementation Considerations
Reviewing the articles in this special issue, you will find many important themes outlined below that will help those seeking to implement a specific practice such as AF-CBT or SafeCare or leaders aspiring to shift their entire service delivery environment toward evidence based and informed orientation. You can also find other resources on the California Evidence-Based Clearinghouse where the developers of over 50 highly rated programs provide guidance on implementing their specific models (www.cebc4cw.org/implementation). These lessons are even more important in our current era of shrinking resources and the increasing demands placed upon the time of the very people that will be called upon to implement the changes at the practice level.
As the authors in this special issue have outlined there are some keys to implementation once it moves to the organizational level that can enhance the likelihood of success.
Organizational readiness
Like a gardener preparing the soil for planting, those leading implementation must, as Whitaker points out, ready the community, organization, and staff who will be expected to deliver the new practice. Some of the tools employed as measures in the studies in this special issue, such as those outlined by Shapiro and colleagues, can be used as part of a community or agency assessment process (e.g., attitudes toward EBP, organizational climate).
Well before the first training session the staff should be acquainted with the model and why it is in their and their clients’ interest to adopt it. Champions of change should be recruited and educated about the model and its benefits. These champions are needed at multiple levels within the organization and community from senior leadership to supervisors and extending to peer leaders who often carry a disproportionate amount of influence with their colleagues.
Training/consultation model
In this special issue, much attention is appropriately paid to the process of imparting knowledge, skills, and a sense of self-efficacy in the work force who will deliver the new practice. As Shanley and others point out, there is little evidence that workshop training alone will result in meaningful change (Herschell et al., 2009). Rather, most EBPs appear to be best implemented when training is experiential and is paired with ongoing consultation and/or coaching, feedback, and clinical supervision (see Kolko and colleagues, Nelson and colleagues, and, Whitaker and colleagues in this issue). This allows the user to understand and integrate the practice change in a meaningful way over time that simply would not occur based on a single training event.
Role of clinical supervision
Trainers and outside consultants, no matter how knowledgable about a new practice, lack the impact of effective day-to-day clinical supervision. In line with Kolko and colleague’s article, clinical supervisors need to be trained in any new practice and then given ongoing advanced training to enhance their supervisor competence and confidence. Failure to engage supervisors in the change process disempowers a vital member of the team and may lead to a lack of support by someone with enormous influence over agency climate and culture. Achieving competency in a new practice, however, is challenging if the supervisor is not implementing the new model. In Shanley and colleagues article in this issue, supervision around the EBP model was provided by the purveyor of that model, and article examines an innovative way to effectively provide that supervision using telemedicine. Traditionally, the dissemination of EBPs has been done with the expertise of the intervention primarily remaining within the purveyor or training center. This is not a cost-efficient model, however, and some purveyor groups are now training others to implement, supervise, and train new staff. Interventions that do this, such as SafeCare, may mandate that supervisors, or those that will provide coaching/fidelity monitoring, learn the model and carry cases before they can coach or train others. Supervisory competency in a new model would appear to be an important component of institutionalizing the new practice after the external consultants and coaches are gone. Those leading an implementation initiative would be wise to ensure supervisors receive special attention, support, and recognition in planning and implementing any new or expanded practices.
Implementation/fidelity monitoring
It will be important to ensure the implementation of any new or expanded model is on the right path and remains so. As Shapiro suggested in her article, providers are more likely to do what is required by their supervisor and organization. Monitoring and metrics have proved to be the most effective way to ensure the change in practice is being woven into everyday practice and became the new way of doing business.
Many EBP developers have developed fidelity tools that should be used periodically during the delivery process. Admittedly, few of these tools have been empirically tested but still represent the developers’ best resource to provide implementation feedback. Some of fidelity measures are brief and user friendly, like with TF-CBT, while for other models, such as Coping Cat, the developer must provide special training in the measurement of fidelity. Information on these tools can be found under implementation information about each model on the California Clearinghouse. These may be completed by the service provider (to ensure that the components of the practice are being followed as prescribed), the service recipient (to provide feedback on what has been delivered, the way in which it was delivered, and their satisfaction with services), or a combination of the two.
Conclusion
As we reflect on the last decade when research is increasingly finding its way into real-world everyday practice settings, we must acknowledge a degree of irony in that something as important as the implementation of EBPs has been largely guided by untested advice mixed with a good dose of common sense. Fortunately, we now have more actual knowledge and experience to base our efforts upon. There are a growing number of rigorous implementation studies underway where the focus of measurement is the implementation strategy itself, and not the underlying intervention which has previously been found effective in other studies.
The authors of the articles in this special issue are examining the processes of implementation and dissemination through an empirical lens that will help guide future efforts. A review of these articles will reinforce the importance of a broad implementation strategy that selects a practice best matched with the needs of the clients and the work force and community where it will be implemented. The articles in this issue provide insight into the complexity of training and coaching those implementing the practice which is often the key to successful implementation. When reading the articles you can see the important role of supervisors as change agent and the wisdom of training them first and with added support so they can be supportive change agents. Most of all, this special issue can help purveyors of EBPs and policy makers and agency managers understand how to structure their mutual efforts for maximum impact.
Other examples of “implementation science” are emerging (Proctor et al., 2009) including an online journal devoted to the topic (implementationscience.com) that is further shedding light on how to manage the change to EBPs in real-world settings. Researchers like Aarons and colleagues in this issue are refining how to conduct meaningful implementation research and the potential for ever increasing understanding of how best to implement EBPs is bright. This relatively new area of interest is welcome as, in the end; it holds the key to the long-term marriage of science and practice for the ultimate benefit of the families we all serve.
Footnotes
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
The author(s) received no financial support for the research, authorship, and/or publication of this article.