Making It Happen

Abstract

This article explores the progress that state governments across the country are making in implementing the three most widely used evidence-based programs (EBPs) for delinquent youth: multisystemic therapy (MST), functional family therapy (FFT), and multidimensional treatment foster care (MTFC). Rather than rank states, this study was designed to help state policy makers and practitioners identify strategies and techniques that can help expand the quality and availability of EBPs in their jurisdictions. Its explicit focus on implementation was purposeful. Most states are not yet in a position to begin to assess if their expenditures on these programs are having an impact (or at least an impact statewide) on juvenile recidivism, placements in residential facilities, or other key outcomes. We found that there are five states that are making substantially greater progress in implementing these EBPs: New Mexico, Louisiana, Maine, Connecticut, and Hawaii. In addition to the highest availability of these programs, ranging from 9.4 to 13.0 therapist teams per million population, these states share a number of key features that demonstrate that direct and purposeful state action is behind the expansion of these programs. Some of these features include structured involvement of all key stakeholders, effective leaders who championed not just the programs but a culture of using research to improve practice, pilot testing of new EBPs, special funding for designated EBPs, and technical assistance to counties to help get programs off the ground. Gaps in knowledge are identified and implications for policy are discussed.

Keywords

evidence-based program delinquency prevention implementation state government

Preliminary results were recently published on the first study to examine the ways that some state governments are promoting and supporting the use of evidence-based practice in the area of delinquency prevention (Greenwood & Welsh, 2012). The unit of analysis was brand-name evidence-based programs (EBPs). These programs are identified by lists of various expert groups (e.g., Blueprints for Healthy Youth Development,¹ Coalition for Evidence-Based Policy) and come with implementation and quality assurance packages offered by program developers. The latter features are particularly important in developmental and community contexts (Sullivan, 2013, 2014). The study focused on the three leading programs of the day: functional family therapy (FFT), multisystemic therapy (MST), and multidimensional treatment foster care (MTFC). Among the key findings were that several early adopter states showed a modest yet growing investment in these EBPs, and to support these programs states employ a number of key elements, including special funding, risk assessment guidance and support, assistance in needs assessment and program selection, and program evaluation. The authors concluded that “state and local governments and practitioners should be cautiously optimistic about the potential of evidence-based practice in delinquency prevention” (Greenwood & Welsh, 2012, p. 509).

Two insightful policy essays were published in response to this piece (Dodge & Mandel, 2012; Lipsey & Howell, 2012). Lipsey and Howell were generally praiseworthy but argued that there is an alternative to the use of brand-name EBPs. This involves using the results of large-scale meta-analyses as the basis for trying to improve existing programs. Identifying those factors that are significantly correlated with the most effective programs allows for the development of best practice guidelines. The next step involves applying this level of information to existing programs—as a way to inform local practice. To achieve this, Lipsey, Howell, Kelly, Chapman, and Carver (2010) developed the standardized program evaluation protocol (SPEP). In short, the SPEP is a hands-on tool that allows practitioners to compare programs operating in their local jurisdiction with what the research shows to be the most important factors associated with effectiveness. It is argued by the authors that one of the main advantages of this approach is to overcome the high start-up costs associated with brand-name EBPs.

A subsequent study tested the utility of these two approaches for translating research into evidence-based practice in juvenile justice (Welsh, Rocque, & Greenwood, 2014). Informed by prospect theory (Kahneman & Tversky, 1979), a first-stage analytic decision-tree model was developed that included three comparable EBPs (two brand names and one generic). Implementation success and financial costs and benefits were key variables in the model and analyses were conducted under two different conditions. Under the first condition, where brand-name programs have a large advantage in implementation success over generic programs, it was found that the brand-name programs had the highest expected values. Under the second condition, which considered the role of the SPEP as applied to generic programs, it was found that all three programs produced highly favorable expected values. Sensitivity analyses indicated that the results were robust.

The second policy essay by Dodge and Mandel (2012) was also generally praiseworthy but made the case for a greater connection between the availability of EBPs and their ability to make real and lasting change in the lives of at-risk children and youths and their families. In the words of Dodge and Mandel, “The goal is not to proliferate specific proprietary programs but to improve the well-being of a community’s population” (2012, p. 532, emphasis in original). This had everything to do with employing rigorous and comprehensive standards in evaluating the very framework—the evidence-based approach itself—to ensure that EBPs become part of policy and practice that can make a difference.

This article presents the final results of the first large-scale study of state progress in implementing EBPs for delinquent youth. The main aim of the study is to explore the progress that state governments are making in implementing the three most widely used EBPs for delinquent youth (MST, FFT, and MTFC). Rather than rank states, this study was designed to help state policy makers and practitioners identify strategies and techniques that can help expand the quality and availability of EBPs in their jurisdictions. While this article builds on the previous work, it represents an entirely new study because it uses more recent data, takes a national focus, and offers new insights about the translation of EBPs into policy and practice. The article begins with an overview of the role of state government and implementation.

Role of State Government and Implementation

Prior works have provided a fair amount of detail on the role that state governments can play in promoting and supporting the translation of EBPs into policy and practice (see Bumbarger & Campbell, 2012; Greenwood & Welsh, 2012; Rhoades, Bumbarger, & Moore, 2012). Importantly, these works build upon the lessons learned from community- and systems-based implementation research in a wide array of disciplines, including community psychology and medicine (e.g., Aarons, Hurlburt, & McCue Horwitz, 2011; Durlak & DuPre, 2008). A few notable points warrant brief mention here.

State governments can play a formidable role in advancing evidence-based practice in a wide range of human service sectors, including juvenile justice, mental health, and substance abuse prevention and treatment. This is true for both small and large states. However, in the case of less populated states, which are more likely to administer all social service and justice programs at the state level, this role becomes more pressing. Here, the state must initiate and then administer the process of transitioning to evidence-based practice. Maine is an example of such a state, and recent research on its efforts to implement and sustain evidence-based practice in juvenile justice bears this out (Rocque, Welsh, Greenwood, & King, 2013).

One key step that state agencies often take is to adopt their own short list of EBPs that they are willing to support in some way. In Washington State, for example, this initially included four programs, namely FFT, MST, aggression replacement therapy (ART), and coordination of services (Barnoski, 2004). Another important step is the establishment of a source of funding available to agencies proposing to implement one of the EBPs on the state’s shortlist. This can go some way toward defraying the costs of paying out-of-state trainers to implement and oversee the programs.² More states are now beginning to use such funds to develop their own capability to provide training and technical assistance for some EBPs. A further important role for states is to establish quality assurance procedures and mechanisms as a way to ensure fidelity to the model. In Washington State, there exists a state-level quality assurance committee that works closely with similar committees at the county level. Finally, states are in a unique position to set up some kind of evidence-based resource center or center for effective programming to aid with these and other important activities. Pennsylvania’s Evidence-Based Prevention and Intervention Support Center, located at Pennsylvania State University (Bumbarger & Campbell, 2012), and Connecticut’s Center for Effective Practice (Franks, 2010) are two examples.

It goes without saying that the process of implementation is a crucial component of evidence-based policy (Fagan, 2013; Mears, 2007, 2010). In a recent article on the implementation of EBPs, Fixsen, Blase, Metz, and van Dyke (2013) drew attention to a rather engaging and insightful framework for describing the diffusion and dissemination of innovations in human service organizations.³ Three approaches were identified “letting it happen,” “helping it happen,” and “making it happen” (Greenhalgh, Robert, McFarlane, Bate, & Kyriakidou, 2004). In the first approach, researchers are content with publishing their findings on an EBP and letting others implement and test it. Greenhalgh, Robert, McFarlane, Bate, and Kyriakidou (2004, p. 593) characterize this approach as “unpredictable, unprogrammed, uncertain, emergent, adaptive, self-organizing.” In the case of helping it happen, researchers take on the role of active and enthusiastic program developers in an effort to aid practitioners in making the EBP a reality.

Fixsen et al. (2013, p. 214) note that, despite the necessity of both of these approaches, neither are “sufficient for reliably producing intended outcomes of research in practice.” This is the task of the third approach, making it happen. Indeed, Fixsen et al. (2013) stress that it is this approach that links EBPs and effective implementation:

purveyors (i.e., developers who are supporting the use of a particular evidence-based program) and other implementation teams take responsibility for supporting practitioners, supervisors, and managers as they attempt to make full and effective uses of evidence-based programs and other innovations in their daily interactions with children, families, and stakeholders. (p. 214)

Ultimately, the goal of implementation is to have practitioners use innovations effectively. To accomplish this goal, high-fidelity practitioner behavior is created and supported by core implementation components also called “implementation drivers” (Fixsen, Blase, Naoom, & Wallace, 2009). These components are staff selection, preservice and in-service training, ongoing coaching and consultation, staff performance evaluation, decision support data systems, facilitative administrative supports, and system interventions. These interactive processes are integrated to maximize their influence on staff behavior and organizational functioning. This is the systematization of making it happen.

We draw upon Greenhalgh et al.’s (2004) framework to explore the progress that state governments are making in implementing the three EBPs. Importantly, as noted previously, this framework is not used to rank states. Instead, it is designed to focus on the diffusion and dissemination of EBPs, which includes drawing attention to both the challenges faced with implementation and factors that promote adoption of these programs at the state level. We believe this contributes to greater potential for learning about the implementation process.

Methodology

Data

The primary data for this study consist of total number of “therapist teams” for the three most widely used EBPs that serve juvenile offenders throughout the United States: FFT, MST, and MTFC. For all three of these programs, the therapist team is the basic unit of operation, supervision, and training. These data serve as a reliable indicator of the level of use of these programs on a state-by-state basis.

There are some differences in the composition of the teams across the programs. For example, MST teams include four therapists, while FFT teams include six therapists. This difference is largely explained by the additional responsibilities carried by MST therapists who are required to hold a master’s degree in counseling or an allied field and that they may be required to deal with school and community issues in addition to family communications and problem solving. For the typical start-up size of approximately 10 youths, MTFC teams include a full-time supervisor, 2 therapists, and 1 foster family for each placement (TFC Consultants, 2013).

The data were collected from the purveyors of the three EBPs. These include FFT, LLC; MST Services, Inc.; and (for MTFC) TFC Consultants, Inc. For comparative purposes, the data are presented as number of teams per million population in the participating states. States were included if they used a minimum of one of the three programs. Data are based on program availability for the calendar year 2011.

Additional data for this study was collected through interviews with a small number of key researchers, policy makers, and practitioners from the five leading states (Connecticut, Hawaii, Louisiana, Maine, and New Mexico).⁴ The main purpose of the interviews was to gain a better understanding of key historical and current developments related to the use of these EBPs in each state as well as the state’s orientation to evidence-based practice more generally.

The interviews were carried out in an open, semi-structured format either in person or on the phone. Interviews were conducted in 2012, with each member of the research team carrying out interviews with representatives from one or more of the five states. We relied on two main methods for selecting interviewees. In some states, where our knowledge of events was greater and we had personal contacts with key experts, we sought advice about whom to interview. Oftentimes, these experts were also amenable to being interviewed. In other states, we followed a snowball sampling approach. Finally, two regional meetings also served as vehicles for interviewing representatives from the states.⁵

Altogether 59 interviews were conducted, ranging from a high of 18 in Louisiana to a low of 2 in Hawaii (see Table 1). In some states, multiple interviews were carried out with the same individual. Excluding multiple interviews, nearly all of the interviewees were contacted on numerous occasions post-interview, either by phone or by e-mail, to follow up on key themes.

Table 1.

Number of Interviews Conducted in the Five Leading States.

State	Number of interviews
Connecticut	13
Hawaii	2
Louisiana	18
Maine	14
New Mexico	12

Interviews were supplemented with the collection of a range of documentation, including annual reports from state agencies, research reports from state and nongovernmental agencies, and evaluation studies. In one state (Maine), resources and on-the-ground access allowed us to employ a case study design (for details on this approach, see Gagnon, 2010; Gerring, 2004).

Programs

FFT involves modifying patterns of family interaction—by modeling, prompting, and reinforcement—to encourage clear communication of requests and solutions between family members and to minimize conflict. It is used for both high- and low-risk juvenile offenders (Alexander & Sexton, 2002).

MST is a multimodal intervention designed for serious and chronic juvenile offenders (Henggeler, Schoenwald, Borduin, Rowland, & Cunningham, 1998). The particular type of treatment is chosen according to the needs of the young person, and it may include individual, family, peer, school, and community interventions (including parent training and skills training); more often though, it is referred to as a family-based treatment.

MTFC involves individual-focused therapeutic care (e.g., skill building in problem solving) for adolescents in an alternative, noncorrectional environment (foster care) and parent management training (Chamberlain & Reid, 1998).

Measures

As noted previously, the key factor under investigation is per capita availability or implementation of these three EBPs on a state-level basis. This is more of an output than an outcome; that is, it does not tell us anything about the impact that EBPs may or may not have on key outcomes, such as juvenile arrests or placements in residential facilities. However, it can be hypothesized that states with a high per capita use of EBPs produce desirable effects on a range of key outcomes. There are several important factors that need to be considered. One is that multiple years of data are needed to help identify any trends and account for any lagged effects. Another consideration is what Dodge and Mandel (2012) refer to as the penetration rate: the percentage of eligible youths who receive EBPs. This is closely related to per capita use of EBPs, but it can be a more difficult statistic to track. Also, there is the need to take account of the well-known phenomenon of attenuation of effects, as a program gets scaled-up and “culminates in population-wide dissemination” (Dodge & Mandel, 2012, p. 526; see also Dodge, 2001; Welsh, Sullivan, & Olds, 2010).

Results

Figure 1 shows the total number of FFT, MST, and MTFC therapist teams per million population in each state that has implemented at least one of these EBPs. Altogether 35 states are included, representing all regions of the country as well as one (Hawaii) of the two noncontiguous states. There is a great deal of variability in the use of these EBPs, from a high of 13.0 teams per capita in New Mexico to a low of less than 1.0 team per capita in a number of states. Figure 1 also shows that MST is the most widely used of the three EBPs, by a factor of 2 and 4 times compared to FFT and MTFC, respectively. On a state-by-state basis, 91% of the states are using MST (32 of 35). This compares with 71% for FFT (25 states) and 40% for MTFC (14 states). More than one-third of the states (37% or 13) are using all three of the programs.

Figure 1.

Family therapy teams per million population by state, 2011. Source. Authors’ calculations.

Figure 2 presents the same data, but with states arranged by the highest to the lowest number of therapist teams per capita. The reason for presenting the data in this fashion is to show more clearly that there are five states that are making substantially greater progress in implementing the three EBPs. The five states are New Mexico, Louisiana, Maine, Connecticut, and Hawaii. We refer to these as the leading states; or in the parlance of Greenhalgh et al. (2004) these states are making it happen. (Based on other data collected, these states also share a number of key characteristics that are integral to a high level of effective implementation of the EBPs; see below). At 13.0 teams per capita, New Mexico has the highest availability of the EBPs. This is followed by Louisiana and Maine, each at 10.6 teams per capita; Connecticut, at 10.0 teams; and Hawaii, at 9.4 teams. For all five states, MST is by far the most widely used of the three EBPs, with MTFC not used in New Mexico and Louisiana.

Figure 2.

States arranged by family therapy teams per million population, 2011. Source. Figure 1.

Figure 2 also shows that there is a wide gap between the leading states and a group of states in the middle range of progress in implementing the programs. In fact, the median average of program availability in these 5 states is double that of the five states with the next highest availability (Colorado, North Carolina, Pennsylvania, Rhode Island, and New York): 10.6 versus 5.3 teams. This middle group of five states can be categorized as helping it happen (Greenhalgh et al., 2004). This group of five states is also distinguished from the remaining 25 states, which show a low availability of the programs. From the perspective of implementing EBPs, these states can be thought of as simply letting it happen (Greenhalgh et al., 2004).

Key Features of Leading States

As important as the measure of availability of EBPs may be, there remains a crucial question that is central to the five leading states: Is the enhanced availability of EBPs the result of some purposeful effort on the part of the state to expand the use of these programs or are these states statistical outliers, whereby, many local sites just happened to become interested in EBPs? The answer to this question is quite clear in that the expansion of EBPs in each of the leading states was the result of direct and purposeful state action. Support for this answer comes from the interviews we carried out with leading experts in the states as well as the collection and analysis of state documentation, which is best captured with the following themes.

Turning Crisis Into Opportunity

Three of the leading states were being sued by the federal Department of Justice over conditions in their juvenile institutions. In the other two states (Maine and New Mexico), there was a growing political consensus that many youths being sent to residential placement did not belong there. All five leading states were able to capitalize on this crisis of confidence by bringing together the key stakeholders and identifying capable individuals to take charge.

Structured Involvement of all Key Stakeholders

Effective changes in juvenile justice programming efforts require the cooperation of many state and local agencies, including state departments of children and families, mental health, probation, law enforcement, and school systems. This has a great deal to do with leveraging state resources for local communities. Each of the leading states created high-level stakeholder groups to oversee the process of rolling out EBPs. In Connecticut, it was the Governor’s Blue Ribbon Commission on Mental Health; in Maine, it was the Juvenile Justice Advisory Group (JJAG); in New Mexico it was the Behavioral Health Collaborative; in Louisiana, it was the Juvenile Justice Implementation Commission; and in Hawaii, it was the Empirical Basis to Services Task Force and local community councils.

In some states, the development of the key stakeholder groups goes back many years. For instance, Maine’s JJAG emerged as a result of work stemming from the passage of the federal Juvenile Justice and Delinquency Prevention Act in 1974 and was officially authorized in 1984. Comprising 22 members, all of whom operate on a voluntary basis, the JJAG is a powerful voice advocating best practices, evaluation, and dissemination of data to ensure juveniles are provided the most effective treatments available (Rocque et al., 2013).

Emergence of Champions

Each of the leading states had widely recognizable champions of EBPs, including key department heads, a leading public health doctor (in Louisiana), a governor (in Hawaii), and an associate commissioner of corrections (in Maine). Everyone knew who these champions were, and they were effective in this role.

Development of Local Expertise

Each of the leading states identified at least one person or a small team of people to become fully informed about the available EBP options and made the time available for them to do this, including travel to operational sites and training in specific models. This intermediary involvement coupled with efforts on the part of the state appears to be the primary means of developing local expertise.

Pilot Testing of New EBPs

When one jurisdiction in a state decides to adopt a new EBP, it provides an opportunity for others within the state to get a better sense of how it might work for them. Therefore, program fidelity is particularly important in pilot tests to make sure that the lessons learned apply to the selected program and not a watered-down version (Welsh et al., 2010). All but one of the leading states (not Hawaii) selected one or two sites to test the program models (i.e., MST, FFT, and MTFC) that had been selected as the best to suit their needs. The pilot tests were closely monitored and the results were widely shared.

Creation of Information Resource Centers

These centers became the primary bridge between the science of EBPs (e.g., review articles, assessment instruments, and training consultants) and the practitioners. Center staff would sit in on practitioner meetings to better understand their needs and then develop analytical or informational tools to help address these needs. Practitioners would ask staff for information about particular problems, or programs they may have heard about, and receive (for the most part) timely, unbiased answers.

Connecticut initially created several resource centers, known as Centers for Effective Programming. The first one established a partnership among an independent institute (the Child Health and Development Institute), state agencies that serve children, and the state’s major academic institutions with medical schools (Yale University and the University of Connecticut). The center worked closely with MST Services, Inc. to become a licensed systems supervisor, providing all the training, coaching, quality assurance, and outcome evaluation through collaboration with local and national partners (Franks, 2010).

New Mexico, unlike Connecticut and the other states, turned to an outside group for expertise. Seen as a cost-effective means of meeting their MST training needs, the state contracted with Colorado’s Center for Effective Interventions, which was already an MST Network Partner that was supervising programs in Colorado, to provide training and oversight for MST therapists in New Mexico.

Designation of a Small Number of EBPs to be Supported by the State

All of our leading states started out supporting just one EBP, either MST or FFT, and all of the states added additional programs to the list of what they supported, albeit slowly. New Mexico and Hawaii still mainly support MST, with just a few FFT teams currently in operation. Connecticut, while still a heavy adopter of MST, has elected to support more than a half-dozen other proven or promising models.

Special Funding for Designated EBPs

The availability of funds to support the crucial but costly preimplementation aspects of new EBPs can be a challenge. Each of the leading states made special funds available for this function, not to mention for wide-scale implementation. Data are beginning to emerge on the cost structure of brand-name EBPs, and figures for MST and FFT were obtained through contacts with the provider organizations. The standard operational cost for an MST or FFT team is about US$500,000 per year. This includes a group of trained therapists, a supervisor, and administrative overhead. MST teams use fewer therapists than FFT teams (4 compared to 6), but each MST therapist is supposed to have a master’s degree in counseling or an allied field and is reportedly paid more. For FFT, training and supervision in the first year costs about US$35,000 per therapist. Second and third year training and supervision costs are lower, on the order of US$25,000 and US$15,000, respectively, as the need for supervision and training becomes more routinized.

Technical Assistance to Counties for Needs Assessment, Program Selection, and Implementation

Since, in most states, counties are far from uniform in size or demographics, it is seldom likely that a state policy will meet the needs of all counties. Research has demonstrated that local communities will achieve better outcomes if they receive proper training in how to assess their needs, select programs, and then implement the programs. It has been shown that the spread of EBPs becomes much more rational and effective when states are able to serve local communities in this way (Fagan, Arthur, Hanson, Briney, & Hawkins, 2011; Hawkins et al., 2009). By all accounts, each of the five leading states embraced this approach.

Importantly, some of these themes are closely connected with Fixsen, Blase, Naoom, and Wallace’s (2009) implementation drivers. For example, special funding for designated EBPs draws attention to the staffing and training needs of effective practices in implementation science.

Discussion and Conclusions

The main aim of the present study was to explore the progress that state governments across the country are making in implementing the three most widely used EBPs for delinquent youth (MST, FFT, and MTFC). This study was designed with state policy makers and practitioners in mind, specifically, to aid them in identifying strategies and techniques that can help expand the quality and availability of EBPs in their jurisdictions. Its explicit focus on implementation was purposeful. Most states are not yet in any position to begin to assess if their expenditures on these programs are having an impact (or at least an impact statewide) on juvenile recidivism, placements in residential facilities, or other key outcomes.

We found that the following five states that are making substantially greater progress in implementing these EBPs: New Mexico, Louisiana, Maine, Connecticut, and Hawaii. In addition to the highest availability of these programs, ranging from 9.4 to 13.0 therapist teams per capita, these states share a number of key features that demonstrate that direct and purposeful state action is behind the expansion of these programs. Some of these features include structured involvement of all key stakeholders, effective leaders who championed not just the programs but a culture of using research to improve practice, pilot testing of new EBPs, special funding for designated EBPs, and technical assistance to counties to help get programs off the ground. According to Greenhalgh et al. (2004), these states are making it happen.

A number of policy drivers seem crucial to helping the five leading states get to a position where they can begin to assess if their efforts are having a statewide impact on key outcomes. There is a need for a sustained commitment to several of the key features that are behind the expansion of EBPs in these states. Structured involvement of all key stakeholders and development of local expertise speak to the need for a comprehensive approach backed up with specialized knowledge—characteristics of effective partnership models (Rosenbaum & Schuck, 2012). The establishment of an information resource center can be especially helpful to supporting the day-to-day needs of practitioners running EBPs (e.g., training and quality assurance) as well as serving as the hub for information sharing and analysis. Also, special funding for designated EBPs and technical assistance to counties for needs assessment, program selection, and implementation will allow for the expansion of effective intervention modalities.

Another policy driver involves states increasing the penetration rate or percentage of eligible youths receiving EBPs. But to return to a point made by Dodge and Mandel (2012), this is not just about increasing numbers; it also calls for bringing about real and lasting change in the lives of at-risk children and youths and their families. Achieving this, as Dodge and Mandel note, calls attention to evaluating—with rigorous and comprehensive standards—the evidence-based framework itself. We believe that the present study demonstrates that some states are making serious inroads on this front. It might be said that the leading states are well beyond the midpoint on this continuum. We also believe that this has the ability to provide policy makers and practitioners with a wider perspective on the importance of the implementation process.

The study’s findings also hold important policy implications for other states that are interested in expanding their use of EBPs or sustaining their present operations. The next group of states—the five states that are helping it happen—may be able to draw lessons from the leading states about what is needed to take the implementation of EBPs to the next level. For these states, it may be that many of the key elements are already in place and what is needed is to improve upon factor X or Y, more a matter of an extra push rather than wholesale change. One of the factors that could be improved upon in some of these states is greater attention to the need for technical assistance for needs assessment, program selection, and implementation.

In some respects, this relates to Shadish, Cook, and Leviton’s (1991) framework for how evaluative information can bring about social change by means of incremental change or major policy innovation. The former refers to program improvement and the latter refers to program replacement. In the case of the five helping it happen states, incremental change is very much in order, and this approach may also be far more manageable. As noted by Shadish et al. (1991, p. 173), “Programs change more through improvement than through replacement, through gradual rather than sudden and uniform shift. Change is nearly always incremental rather than radical.”

It is also important to recognize that for some of the states in this middle group it could just be that more time is needed for them to achieve a higher per capita availability of EBPs. Indeed, this is an important limitation of the study. We are only capturing the progress of the states at one point in time. Plans are underway to extend the research over multiple years (i.e., beyond 2011), offering the potential for valuable insights that come from taking a longitudinal perspective (Sullivan, 2013). In addition, interviews with key personnel in these other states would allow for further points of contrast with the leading states.

Several other limitations confronted our research. First, while it may be the case that the three EBPs under study are the most widely used across the country, some states have adopted other EBPs (e.g., ART) and the availability of these programs may surpass the likes of MST, FFT, and MTFC. This is certainly the case in two states: the Commonwealth of Pennsylvania and Washington State. This becomes more important when we consider what these two state have accomplished thus far in the area of evidence-based delinquency prevention and juvenile justice (see Bumbarger & Campbell, 2012; Drake, Aos, & Miller, 2009; Feinberg, Jones, Greenberg, Osgood, & Bontempo, 2010; Lee et al., 2012). It is even the case that Washington State has developed and tested two of its own EBPs, namely coordination of services and family integrated transition (Lee et al., 2012).

Along these lines, we also recognize that there are alternatives to brand-name EBPs. Lipsey and Howell (2012) identify the need to consider improving existing programs through the application of the SPEP. Importantly, this is not always a mutually exclusive question. There are several states (e.g., Connecticut, Florida, and Pennsylvania) that are presently experimenting with both approaches (Lipsey & Howell, 2012).

Another limitation is that we were not able to capture a more complete picture of the penetration rate of EBPs in the states. What is also needed is a measure of eligible youths who are receiving the available EBPs. State-level data were not available. Dodge and Mandel (2012) draw attention to this key statistic. They cite a study in California that found that only 5% of eligible youths in the state’s juvenile justice system were receiving EBPs in the mid-2000s (Hennigan et al., 2007). With our research showing that California is among one of the lower tier states for implementation progress, it is likely that this figure has not changed much in the intervening years. This differs considerably from Washington State where it is reported that upward of 30% of eligible youths are receiving EBPs (Washington State Institute for Public Policy, 2010). An example of how such data might be routinely reported comes from Jefferson Parish, Louisiana. Figure 3 shows the percentage of the juvenile treatment budget allocated to EBPs from 2007 to 2010, and Figure 4 shows the percentage of eligible youths participating in EBPs during the same period of time.

Figure 3.

Percentage of juvenile treatment budget allocated to evidence-based programs (EBPs) in Jefferson Parish, Louisiana, 2007–2010. Source. Phillippi (2013).

Figure 4.

Percentage of youths referred to evidence-based programs (EBPs) in Jefferson Parish, Louisiana, 2007–2010. Source. Phillippi (2013).

Closely tied to this point is the need for some caution in assessing the impact of EBPs on statewide outcomes. To return to Dodge and Mandel (2012), they argue that the percentage of eligible youths receiving these programs needs to be much higher to help bring about any state-level change. While they do not specify a threshold, we can imagine that it is much higher than 30%. Future empirical research needs to pay particular attention to this question.

One approach could be to draw upon time series data for the key outcomes described previously. It needs to be acknowledged that the data are also influenced by changes in the at-risk population and their behavior as well as the responses of the system, which make it difficult to disentangle the independent effects. Another approach could be to project the likely impact of these programs on key outcomes and outputs (e.g., crime and cost), drawing upon the body of evaluation research completed to date. This was used in Washington State and Pennsylvania (Aos, 2011; Jones, Bumbarger, Greenberg, Greenwood, & Kyler, 2008). Important here is to be mindful of the words of caution from Mears, Cochran, Greenman, Bhati, and Greenwald (2011, p. 516) who note that in the fields of child and youth development and juvenile justice the “state of evidence is far from definitive.”

If state governments are going to affect change on key outcomes for delinquent youth and on a statewide basis, they are going to need to put science at center stage in the policy-making process. This includes a heavy dose of the science of program evaluation and the synthesis of evidence—for determining what works, for whom, and under what conditions—and the science of program implementation (Fagan, 2013; see also Nagin & Weisburd, 2013). While the latter has long been neglected in crime prevention research in general, strides have been made toward Petersilia’s (2008, p. 350) call for “developing a science around program implementation.” Indeed, Fixsen et al. (2013, p. 214) report that the field of implementation science “is on the verge of having evidence-based implementation methods to reliably realize the promise of evidence-based programs in practice.” As documented here, a small number of states are far advanced in implementing EBPs for delinquent youth. Sustaining their successes and elevating other states to the level of making it happen will be critical steps toward building a safer society.

Footnotes

Acknowledgments

We are grateful to the editor and the anonymous reviewers for especially helpful comments. We also wish to thank Michael Rocque for his involvement in an early stage of this research and Sema Taheri for her timely research assistance.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was supported by a grant from a private foundation that wishes to remain anonymous.

Notes

References

Aarons

G. A.

Hurlburt

McCue Horwitz

(2011). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health and Mental Health Services Research, 38, 4–23.

Alexander

J. F.

Sexton

T. L.

(2002). Functional family therapy: A model for treating high-risk, acting-out youth. In Kaslow

F. W.

Lebow

J. L.

(Eds.), Comprehensive handbook of psychotherapy: Integrative/eclectic (vol. 4, pp. 111–132). Hoboken, NJ: Wiley.

Aos

(2011). Updates and new findings: Crime trends in Washington & policy options that reduce crime and save money. Paper presented at Senate Human Services and Corrections Committee, December 8, 2011.

Barnoski

(2004). Outcome evaluation of Washington State’s research-based programs for juvenile offenders. Olympia, WA: Washington State Institute for Public Policy.

Bumbarger

B. K.

Campbell

E. M.

(2012). A state agency-university partnership for translational research and the dissemination of evidence-based prevention and intervention. Administration and Policy in Mental Health and Mental Health Services Research, 39, 268–277.

Chamberlain

Reid

J. B.

(1998). Comparison of two community alternatives to incarceration for chronic juvenile offenders. Journal of Consulting and Clinical Psychology, 66, 624–633.

Dodge

K. A.

(2001). The science of youth violence prevention: Progressing from developmental epidemiology to efficacy to effectiveness to public policy. American Journal of Preventive Medicine, 20, 63–70.

Dodge

K. A.

Mandel

A. D.

(2012). Building evidence for evidence-based policy making. Criminology & Public Policy, 11, 525–534.

Drake

E. K.

Aos

Miller

M. G.

(2009). Evidence-based public policy options to reduce crime and criminal justice costs: Implications for Washington State. Victims and Offenders, 4, 170–196.

10.

Durlak

J. A.

DuPre

E. P.

(2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41, 327–350.

11.

Fagan

A. A.

(2013). Family-focused interventions to prevent juvenile delinquency: A case where science and policy can find common ground. Criminology & Public Policy, 12, 617–650.

12.

Fagan

A. A.

Arthur

M. W.

Hanson

Briney

J. S.

Hawkins

J. D.

(2011). Effects of communities that care on the adoption and implementation of evidence-based prevention programs in communities: Results from a randomized controlled trial. Prevention Science, 12, 223–234.

13.

Feinberg

M. E.

Jones

Greenberg

M. T.

Osgood

D. W.

Bontempo

(2010). Effects of the communities that care model in Pennsylvania on change in adolescent risk and problem behavior. Prevention Science, 11, 163–171.

14.

Fixsen

D. L.

Blase

K. A.

Metz

van Dyke

(2013). Statewide implementation of evidence-based programs. Exceptional Children, 79, 213–230.

15.

Fixsen

D. L.

Blase

K. A.

Naoom

S. F.

Wallace

(2009). Core implementation components. Research on Social Work Practice, 19, 531–540.

16.

Franks

R. P.

(2010). Role of the intermediary organization in promoting and disseminating best practices for children and youth. Farmington, CT: Connecticut Center for Effective Practice, Child Health and Development Institute.

17.

Gagnon

Y.-C.

(2010). The case study as a research method: A practical handbook. Quebec City, Canada: University of Quebec Press.

18.

Gerring

(2004). What is a case study and what is it good for? American Political Science Review, 98, 342–354.

19.

Greenhalgh

Robert

McFarlane

Bate

Kyriakidou

(2004). Diffusion of innovations in service organizations: Systematic review and recommendations. Milbank Quarterly, 82, 581–629.

20.

Greenwood

P. W.

Welsh

B. C.

(2012). Promoting evidence-based practice in delinquency prevention at the state level: Principles, progress, and policy directions. Criminology & Public Policy, 11, 493–513.

21.

Hawkins

J. D.

Oesterle

Brown

E. C.

Arthur

M. W.

Abbott

R. D.

Fagan

A. A.

Catalano

R. F.

(2009). Results of a type 2 translational research trial to prevent adolescent drug use and delinquency: A test of communities that care. Archives of Pediatrics and Adolescent Medicine, 163, 789–798.

22.

Henggeler

S. W.

Schoenwald

S. K.

Borduin

C. M.

Rowland

M. D.

Cunningham

P. B.

(1998). Multisystemic treatment of antisocial behavior in children and adolescents. New York, NY: Guilford.

23.

Hennigan

Kolnick

Poplawski

Andrews

Ball

Cheng

Payne

(2007). Juvenile justice data project phase I: Survey of interventions and programs: A continuum of graduated responses for juvenile justice in California. Los Angeles, CA: Center for Research on Crime, University of Southern California.

24.

Jones

Bumbarger

B. K.

Greenberg

M. T.

Greenwood

P. W.

Kyler

(2008). The economic return on PCCD’s investment in research-based programs: A cost-benefit assessment of delinquency prevention in Pennsylvania. University Park, PA: Evidence-Based Prevention and Intervention Support Center.

25.

Kahneman

Tversky

(1979). Prospect theory: An analysis of decision under risk. Econometrica, 47, 263–292.

26.

Lee

Aos

Drake

E. K.

Pennucci

Miller

M. G.

Anderson

(2012). Return on investment: Evidence-based options to improve statewide outcomes. Olympia, WA: Washington State Institute for Public Policy.

27.

Lipsey

M. W.

Howell

J. C.

(2012). A broader view of evidence-based programs reveals more options for state juvenile justice systems. Criminology & Public Policy, 11, 515–523.

28.

Lipsey

M. W.

Howell

J. C.

Kelly

M. R.

Chapman

Carver

(2010). Improving the effectiveness of juvenile justice programs: A new perspective on evidence-based practice. Washington, DC: Center for Juvenile Justice Reform, Georgetown University.

29.

Mears

D. P.

(2007). Towards rational and evidence-based crime policy. Journal of Criminal Justice, 35, 667–682.

30.

Mears

D. P.

(2010). American criminal justice policy: An evaluation approach to increasing accountability and effectiveness. New York, NY: Cambridge University Press.

31.

Mears

D. P.

Cochran

J. C.

Greenman

S. J.

Bhati

A. S.

Greenwald

M. A.

(2011). Evidence on the effectiveness of juvenile court sanctions. Journal of Criminal Justice, 39, 509–520.

32.

Nagin

D. S.

Weisburd

(2013). Evidence and public policy: The example of evaluation research in policing. Criminology & Public Policy, 12, 651–679.

33.

Petersilia

(2008). Influencing public policy: An embedded criminologist reflects on California prison reform. Journal of Experimental Criminology, 4, 335–356.

34.

Phillippi

(2013). Critical steps in expanding EBP on a statewide basis. Paper presented at Advancing Evidence Based Practice regional conference, Washington, DC, September 26, 2013.

35.

Rocque

Welsh

B. C.

Greenwood

P. W.

King

(2013). Implementing and sustaining evidence-based practice in juvenile justice: A case study of a rural state. International Journal of Offender Therapy and Comparative Criminology. doi:10.1177/0306624X13490661

36.

Rhoades

B. L.

Bumbarger

B. K.

Moore

J. E.

(2012). The role of a state-level prevention support system in promoting high-quality implementation and sustainability of evidence-based programs. American Journal of Community Psychology, 50, 386–401.

37.

Rosenbaum

D. P.

Schuck

A. M.

(2012). Comprehensive community partnerships for preventing crime. In Welsh

B. C.

Farrington

D. P.

(Eds.), The Oxford handbook of crime prevention (pp. 226–246). New York, NY: Oxford University Press.

38.

Shadish

W. R.

Cook

T. D.

Leviton

L. C.

(1991). Foundations of program evaluation: Theories of practice. Newbury Park, CA: Sage.

39.

Sullivan

C. J.

(2013). Enhancing translational knowledge on developmental crime prevention: The utility of understanding expert decision making. Criminology & Public Policy, 12, 343–351.

40.

Sullivan

C. J.

(2014). Individual, social, and neighborhood influences on the launch of adolescent antisocial behavior. Youth Violence & Juvenile Justice, 12, 103–120.

41.

TFC Consultants, Inc. (2013). Multidimensional treatment foster care: implementation. Retrieved September 23, 2013, from http://www.mtfc.com/implementation.html

42.

Washington State Institute for Public Policy. (2010). Washington State juvenile court funding: Applying research in a public policy setting. Olympia, CT: Author.

43.

Welsh

B. C.

Rocque

Greenwood

P. W.

(2014). Translating research into evidence-based practice in juvenile justice: Brand-name programs, meta-analysis, and key issues. Journal of Experimental Criminology, 10, 207–225.

44.

Welsh

B. C.

Sullivan

C. J.

Olds

D. L.

(2010). When early crime prevention goes to scale: A new look at the evidence. Prevention Science, 11, 115–125.