Implementation of a Capacity-Building Framework to Improve School Climate in an Urban School System

Abstract

Mounting evidence suggests that a positive school climate can promote learning and well-being for students and reduce bullying and other forms of violence. However, many schools lack the capacity needed to engage in comprehensive school climate improvement. As part of a broader evaluation, this study examines implementation of a capacity-building framework for school climate improvement that emphasizes shared leadership and data-informed decision making. Semi-structured interviews were conducted with points of contact at schools receiving technical assistance and those that were allowed to work through the framework on their own. Interviews were also conducted with technical assistance specialists working with schools randomly assigned to receive the intervention. Analyses explored three key research questions: 1 What factors contributed to schools’ initial decision to participate in the project? 2 What factors were associated with sustained engagement? and 3 To what extent did schools build organizational capacity through participation? Organizational readiness, including both motivation and general organizational capacity, emerged as a key facilitator of sustained engagement. In particular, the presence of a champion was facilitative despite the framework’s emphasis on shared leadership. Staff turnover and competing priorities were identified as common challenges. With technical assistance, schools that were able to invest time and effort in the framework perceived growth in their capacity for shared leadership and data literacy. However, technical assistance specialists did not feel adequately prepared to support schools in implementing the framework. Our discussion focuses on implications for evidence-based technical assistance and implementing comprehensive climate improvement efforts in urban schools.

Keywords

school climate school safety implementation capacity building technical assistance

Over the past several decades, education leaders have paid increasing attention to the role that school climate plays in advancing student learning, supporting student behavioral health and overall well-being, and preventing bullying and other forms of violence (Chriqui et al., 2019; Jordan & Hamilton, 2020; Thapa et al., 2013). School climate has been operationalized differently across studies; however, most conceptualizations of school climate include dimensions such as relationships, physical and emotional safety, physical environment, and disciplinary environment (see Ryberg et al., 2020 for a detailed discussion of the U.S. Department of Education’s Safe and Supportive Schools Model of school climate).

A robust body of research has linked positive school climate with important health and learning outcomes for students, such as higher academic achievement, improved behavioral health, and lower rates of bullying, violence, suspension, and substance use (e.g., Aldridge & McChesney, 2018; Greenberg, 2010; Kutsyuruba et al., 2015; O’Connell et al., 2009; Thapa et al., 2013; Wang & Degol, 2016). Much remains to be learned about the mechanisms that support these linkages; however, research is quite clear that supportive relationships promote student engagement in school (e.g., Klem & Connell, 2004; Shernoff et al., 2017; Suárez-Orozco et al., 2009) and that engagement, in turn, plays an important role in learning (e.g., Ladd & Dinella, 2009). Evidence also suggests that supportive peer relationships, strong connections with teachers, and feelings of safety at school promote student social and emotional well-being (Lester & Cross, 2015). Further, bullying occurs less frequently in schools when students feel supported and respected by adults and their peers (Gage et al., 2014). When it does occur, students may be more likely to seek help if they perceive that adults care about them and will respond appropriately, thereby raising adult awareness of the problem and creating opportunities for intervention (Eliot et al., 2010).

Even with strong evidence pointing to the positive effects of school climate on student outcomes, important gaps remain when it comes to identifying, implementing, and scaling effective approaches, especially in urban schools (Thapa et al., 2013; Wang & Degol, 2016). One challenge to systematic and large-scale efforts to improve school climate is the lack of school climate interventions that focus specifically on building schools’ capacity to select, implement, and sustain programs and practices that meet their students’ needs. In light of a large and growing body of research highlighting the importance of implementation quality for achieving desired outcomes of interventions (e.g., Bauer et al., 2015; Fixsen & Blase, 2009; Meyers et al., 2012), building school capacity for high-quality implementation is critical. For example, Durlak and DuPre’s (2008) seminal review of social and emotional learning interventions found that when evidence-based interventions are implemented poorly, they may achieve similar outcomes to no intervention at all.

Several frameworks have been developed to support high-quality implementation of evidence-based interventions. The Interactive Systems Framework (Wandersman et al., 2008) emphasizes identifying and delivering capacity-building supports to ensure high-quality implementation in a broad range of practice settings. Domitrovich and colleagues (2008) outlined a framework that focuses specifically on implementation of evidence-based interventions in schools, including factors that influence implementation at macro, school, and individual levels. Bertram et al. (2015) outlined a framework of quality implementation that highlights the importance of competency among implementers of an intervention, organizational facilitators, and supportive leadership. Each of these frameworks highlights the need to assess and enhance key capacities among organizations and individuals in order to successfully implement interventions that aim to enhance well-being.

A small number of well-studied capacity-building interventions have been found to positively impact outcomes for adolescents, including Communities that Care (CTC; Fagan et al., 2009), Promoting School-community-university Partnerships to Enhance Resilience (PROSPER; Spoth, Redmond, et al., 2007), and Schoolwide Positive Behavioral Interventions and Supports (SWPBIS; Horner & Sugai, 2015). With the exception of SWPBIS, these models focus on building capacity at the community level rather than within the school itself. All three focus on improving organizational capacity to engage in data-driven decision making through needs assessment, selection of effective interventions that specifically target identified concerns, and use of data to monitor implementation quality and assess outcomes (Miller & Shinn, 2005; Mrazek & Haggerty, 1994). In the cases of CTC and PROSPER, communities are expected to select and implement an evidence-based intervention from a pre-approved list that addresses a need identified during their needs assessment; SWPBIS emphasizes implementation of Multi-tiered Systems of Support (MTSS), providing universal supports for all students, targeted interventions for students who exhibit risk factors, and indicated interventions for students with identified needs.

Each model also relies on a team to provide leadership and the provision of training and technical assistance to build team capacity; CTC and PROSPER require dedicated, funded coordinators to lead teams (Hawkins et al., 2008; Horner et al., 2014; Spoth, Guyll, et al., 2007). Training and technical assistance for CTC and PROSPER are traditionally provided through partnerships with universities, whereas SWPBIS supports are generally provided through a network of regional PBIS centers funded by the U.S. Department of Education with coordinators in each state.

All three models have multiple decades of implementation and evaluation, which have identified several themes related to facilitators and barriers of implementing capacity-building models. For example, studies of SWPBIS have found that administrator support, staff buy-in, staff turnover, and competing initiatives can all influence schools’ sustained success (Turri et al., 2016). Studies of CTC and PROSPER have also identified team functioning, especially turnover of staff in leadership positions, as challenges to sustaining initiatives (Feinberg et al., 2007, 2008).

Less is known about the role readiness, in particular motivation, plays in successful and sustained engagement in capacity-building interventions. For example, a study of PROSPER found that previous success with change efforts in a community, a measure of readiness, was associated with better team functioning (Feinberg et al., 2007). However, they did not examine motivations for participation. A recent qualitative study examining community readiness for CTC in an urban setting focused on community members’ perceptions of readiness to undertake the initiative (Parker et al., 2018). Additionally, most implementation studies of these models have focused on fidelity and team functioning when it comes to capacity to implement the model. Few have examined the effects that participation in the intervention might have on organizational capacity that could be applied to other initiatives in the school such as increased capacity to effectively engage parents and students or increased data literacy among staff. These are particularly important factors to address implementation challenges that have been documented among urban schools and schools serving large proportions of students and families living in poverty (Feinberg et al., 2007; Nese et al., 2016).

The present study directly builds upon our current understanding of the facilitators and barriers for implementing and sustaining organizational capacity-building frameworks specifically within the context of urban schools. We leverage rich qualitative data to address gaps in our understanding of urban schools’ experiences with implementing capacity-building frameworks to improve school climate. In particular, we seek to better understand what motivates urban schools to participate in a capacity-building school climate initiative and in what ways implementation of a framework that emphasizes data-driven decision making in collaboration with key school stakeholders might ultimately increase organizational capacity.

Safe School Certification

Safe School Certification (SSC) is a 3-year organizational capacity-building model originally developed and implemented in Iowa by the Iowa Pride Network with funding from a U.S. Department of Education Safe and Supportive Schools grant. SSC is designed specifically for schools to build capacity to select and implement strategies to improve school climate that are aligned with needs identified in data and consists of three components: (a) a research- and practice-informed framework that defines success (“desired states”) across eight key elements of organizational capacity for school climate; (b) support from a technical assistance specialist (TAS) to help schools determine steps to reach the desired states; and (c) external reviews by a trained, community-based certification and advisory board (CAB) to assess schools’ progress and provide resources and recommendations. At the end of the process, if the school, as deemed by the CAB, achieves each of the desired states, the school can become “certified.”

The SSC framework consists of eight evidence-informed elements that are key to supporting a positive school climate: (a) Leadership: teams to guide the work that represent the whole school community; (b) Data and Data-Based Decision Making: data collection and analysis that drives decision making; (c) Buy-in to School Climate Improvement: support and input solicited from all members of the school community; (d) Policy and Enforcement: established rules for behavior that are agreed upon by the school community and enforced consistently; (e) Student Engagement: students who take leadership roles in shaping the school’s climate; (f) Family and Community Engagement: families and community partners who are integrated into school climate efforts; (g) Training for all school staff on school climate policies, programs, and practices; and (h) Programs and Practices: evidence-informed prevention and intervention programming based on needs identified during data collection.

These elements are grouped into two “checkpoints” such that schools work on the first three elements (Leadership, Data, and Buy-in) as part of checkpoint one, and after passing checkpoint one move on to the remaining elements for checkpoint two. To pass each checkpoint, schools must document that they have reached the desired state for each element, as defined in the framework. Schools compile this documentation in a workbook that they submit to the CAB. The CAB then provides written feedback and scores schools’ workbooks based on the SSC framework on a scale of 0–2, with a score of two indicating that a school has fully engaged on that element. Schools pass a checkpoint if they score at least 70% of the total potential points (11 out of 16 for checkpoint one, and 28 out of 40 for checkpoint two). At the end of the process, schools may apply for certification based on addressing all eight elements and developing a plan for sustaining their work long-term. During the Washington, DC implementation, schools were also eligible to receive grant funding to support evidence-informed programs and practices after completing each checkpoint to provide additional resources and incentive for continued participation.

Central to the SSC model are capacity-building supports provided by TASs to help schools build on their existing school climate and violence reduction work and to further develop their capacity to implement programs and policies to improve school climate. Once assigned to a school, TASs work with school leadership to assess existing school climate efforts and to provide school leadership team members with training on the framework elements, including how to interpret school climate survey data. TASs also engage in biweekly on-site and/or phone consultations to provide schools with individualized support.

Current Study

The present study represents the first formal implementation evaluation of SSC and focuses on the implementation of SSC in middle and high schools in Washington, DC. The current study uses qualitative methods to explore three key questions related to schools’ engagement and success with SSC:

1. What factors contributed to schools’ initial decision to engage in the SSC process?

2. What factors helped facilitate sustained engagement? What factors served as barriers? What role did technical assistance play in addressing these factors?

3. To what extent did schools build organizational capacity through participation?

Methods

Sample

The District of Columbia Public Schools (DCPS) identified a subset of public schools that served either seventh and eighth grade students or ninth and tenth grade students for recruitment into the study. In partnership with the Office of the State Superintendent of Education (OSSE), we additionally invited all DC public charter schools serving the same populations for participation. Ultimately, 26 middle and high schools agreed to participate, from which 13 matched pairs were created based on their student demographics (e.g., % of White students at the school, % of students with limited English proficiency at the school, % “at risk” based on a variety of factors including eligibility for Temporary Assistance for Needy Families [TANF] or the Supplemental Nutrition Assistance Program [SNAP]), school structure (e.g., K-8, 6–12, and 9–12), student enrollment, location within the city (i.e., District ward), school type (i.e., charter, public), and rates of reported physical fights and bullying from the 2015 Youth Risk Behavior Survey (Kann et al., 2016). Table 1 presents basic demographic information for each participating school as of the 2015–2016 school year; because of confidentiality agreements with the district, this information is presented in such a way as to not identify specific schools. One school within each pair was randomly assigned to one of two conditions: a technical assistance intervention group that received the full SSC model (intervention) and a control group that was given the opportunity to complete the SSC framework independent of technical assistance and received some basic supports (e.g., data reports, written guidance on the SSC elements) to facilitate that process.

Table 1.

Characteristics of Participating Schools as of SY 2015–2016.

School	Grade Level^a	Charter	Majority at Risk^b	Majority Black	Received TA	Participated Year 4^c	Passed Checkpoint 1	Passed Checkpoint 2	Certified
1	Middle	Yes	No	No	No	No	No	No	No
2	Middle	No	Yes	Yes	No	Yes	No	No	No
3	Middle	Yes	Yes	Yes	No	No	No	No	No
4	Middle	Yes	No	No	No	No	No	No	No
5	Middle	No	Yes	Yes	No	No	No	No	No
6	Middle	No	Yes	Yes	No	No	No	No	No
7	Middle	No	Yes	No	No	No	No	No	No
8	Middle	No	No	Yes	No	Yes	No	No	No
9	Middle	No	Yes	No	No	Yes	No	No	No
10	Middle	Yes	No	Yes	No	No	No	No	No
11	High	No	Yes	Yes	No	Yes	Yes	Yes	No
12	Middle	No	No	Yes	No	Yes	Yes	No	No
13	High	No	No	No	No	No	Yes	No	No
14	Middle	No	Yes	Yes	Yes	No	No	No	No
15	High	No	Yes	Yes	Yes	No	No	No	No
16	Middle	Yes	Yes	Yes	Yes	No	No	No	No
17	Middle	No	No	Yes	Yes	No	No	No	No
18	Middle	No	Yes	Yes	Yes	No	No	No	No
19	High	No	Yes	No	Yes	No	No	No	No
20	Middle	No	Yes	Yes	Yes	No	No	No	No
21	Middle	Yes	No	Yes	Yes	No	No	No	No
22	Middle	Yes	Yes	Yes	Yes	No	Yes	No	No
23	Middle	Yes	No	No	Yes	Yes	Yes	Yes	No
24	Middle	No	No	No	Yes	Yes	Yes	Yes	No
25	High	Yes	Yes	Yes	Yes	No	Yes	No	No
26	Middle	No	No	Yes	Yes	Yes	Yes	Yes	Yes

^aGrade Level refers to the students that participated in the school climate surveys. Some schools serve both middle and high school grade levels.

^bAt Risk is defined by OSSE as students identified as homeless, in the foster care system, qualified for Temporary Assistance for Needy Families (TANF) or the Supplemental Nutrition Assistance Program (SNAP), or high school students who are one or more years older than the expected age for their grade level.

^cParticipated Year 4 reflects the school’s participation in school climate survey data collection during Year 4 of the project.

All schools had the opportunity to participate in four years of school climate survey data collection and receive annual detailed data reports about their school climate survey results. All schools also received written guidance on the key elements of the SSC framework; had the opportunity to submit project workbooks detailing their efforts relative to each element and receive feedback from the CAB; and, if they met certain criteria based on their efforts, had the opportunity to apply for grant funding to support their school climate work. Schools receiving the full intervention additionally received individualized support from a TAS, described previously, and had the opportunity to participate in a community of practice. Detailed information about the sample of schools included in the project and the school climate survey that was administered each year can be found in a companion paper focused on quantitatively evaluating the effectiveness of the SSC model on improving student perceptions of school climate (Ryberg et al., in preparation). All study procedures were approved by the Child Trends Institutional Review Board (IRB) and the research board at (DCPS).

Data Collection

The study team conducted interviews with key informants: points of contact (POCs) from both control and intervention schools, TASs who worked with intervention schools, the lead member of the CAB, and the SSC program developer. Interviews took place by phone or in-person based on the participant’s preference, typically lasted 30–60 min, and were audio recorded with the consent of the participant and later transcribed verbatim for analysis.

Interviews with School Points of Contact.

POCs from all 26 schools were invited to participate in semi-structured interviews in Years 2 and 3 of the project to explore their experiences with SSC, including challenges, successes, and reflections on the value of the process for their school. In Year 4, POCs from the schools that passed both checkpoints were invited to participate in a follow-up interview. A total of 21 interviews were conducted with POCs from 14 schools, which included nine intervention schools and five control schools. POCs occupied various roles, including senior-level leadership roles (principal, head of school; n = 6), other leadership roles (assistant/associate principal, assistant head of school, dean, director; n = 9), and support staff roles (school counselor, intervention coach, restorative practices coordinator; n = 3). Written or verbal informed consent was obtained from all POCs, depending on whether the interview was conducted in-person or by phone.

Interviews with Technical Assistance Specialists.

Quarterly interviews were conducted with TASs working with intervention schools. Each TAS worked with multiple schools; therefore, each interview addressed the activities, supports provided, challenges, and successes of multiple schools. A total of 24 interviews were conducted with four TASs over the course of the project.¹

As can be seen in Table 2 below, the team had multiple TAS interviews and in many cases had POC interviews as an additional data source for most intervention schools. In contrast, for control schools, the research team had either one or more POC interviews or, more often, no implementation data whatsoever. As a result, our analyses naturally focused on those schools for which we had a minimum of one data source available and were more robust for the schools for which we had more data available.

Table 2.

Number of Schools with Multiple Data Sources.

School Condition	No Data Sources	TAS^a Interview(s)Only	POC^b Interview(s) Only	TAS and POC Interviews
Control (n = 13)	8	0	5	0
Intervention (n = 13)	0	4	0	9

^aTAS refers to technical assistance specialist.

^bPOC refers to school point of contact.

Interview with Certification Advisory Board Member.

An exit interview was conducted with the manager of the CAB after the project had ended to better understand the CAB’s composition and role in the project and to gather the CAB’s reflections on implementation.

Interviews with Safe School Certification Program Developer.

Interviews were conducted with the SSC program developer at two points in time, once during and once after the project, to explore the history of SSC, to understand the training and supervision provided to TASs, and to gather his observations and reflections on implementation based on check-ins with the TASs and conversations with the CAB.

Data Analysis

Interview transcripts were uploaded into Dedoose (Version 8.3.35, 2020) for analysis. Coding was conducted by two members of the research team. They used both a deductive and inductive approach to coding, beginning with an initial coding scheme based on the research questions and interview guides and then adding to and refining the coding scheme based on the content of the interviews. Throughout the coding process, the full research team—including the two coders and two other team members who had developed the interview guides and conducted most of the interviews—met regularly to discuss the evolving coding scheme, ensure alignment of understanding across coders, reconcile any discrepancies, and discuss emerging themes. These discussions not only helped to ensure the consistent application of codes across coders, but also provided ongoing opportunities to consider multiple interpretations of the data. Debriefing sessions such as these, involving colleagues who are familiar with the research topic but not directly involved in the coding process, help to reduce bias and strengthen the validity of qualitative analysis (Barbour, 2001; Guba, 1981; Shenton, 2004). In addition, consistency between the two coders was formally assessed through a double-coding process involving 25% of the interview transcripts. The coders achieved a level of agreement that ranged from 82% to 100% for 95% of codes, with an overall average agreement of 96% across all codes, thus exceeding the threshold recommended by Miles and Huberman (1994) of 80% agreement across 95% of codes.

The research team used a case-based approach to thematic analysis (Bazeley, 2013), involving both within-case examination of implementation at each school, including the sequence and timing of implementation, participants’ roles throughout implementation, and the context in which implementation occurred, as well as across-case comparisons to examine similarities and differences across schools. As part of this process, the research team developed brief summaries of each school’s implementation trajectory and experience, leveraging multiple data sources across multiple time points when available (see Table 2 above). In-depth team discussions were accompanied by targeted coding queries to explore patterns in the data related to the research questions and emerging themes.

Results

We first provide a brief description of the overall implementation of SSC in Washington, DC and a comparison of implementation across intervention conditions. We then present thematic results that address each of our three research questions: 1 What factors contributed to schools’ initial decision to engage in the SSC process? 2 What factors helped facilitate sustained engagement? What factors served as barriers? What role did technical assistance play in addressing these factors? and 3 To what extent did schools build organizational capacity through participation?

Descriptive Information about Safe School Certification Implementation

In Year 1, the project began with 26 schools (13 in each condition). However, a significant number of schools either formally withdrew or became nonresponsive to the project team over time, and one other school had closed by the end of the study. Table 3 presents the milestones achieved by schools in each condition. A total of eight schools (three control, five intervention) submitted and passed the first checkpoint workbook, four schools (one control, three intervention) submitted and passed the second checkpoint workbook, and one (intervention) school became certified after passing checkpoint two and submitting a written plan for sustaining its work long-term.

Table 3.

Participation by Treatment Condition.

Participation Milestone	Control (n = 13)	Intervention (n = 13)
Participated in survey data collection each year
YR1	13	13
YR2	9	9
YR3	6	6
YR4	5	3
Submitted workbook 1	5	5
Passed checkpoint 1	3	5
Submitted workbook 2	1	3
Passed checkpoint 2	1	3
Received a grant	3	3
Achieved certification	0	1

What Factors Contributed to Schools’ Initial Decision to Engage in the Safe School Certification Process?

During the first wave of interviews, we asked POCs to reflect on their decision-making process for joining the project, including who was involved, what interested them about the project, and why they chose to participate. POCs from nine schools (three control, six intervention) shared information about their motivations related to participation. Many POCs indicated that their school chose to participate because the project felt like a good fit in one way or another. They generally cited a combination of factors that contributed to their decision, including a desire to improve school climate, an affinity for the SSC approach, and external influences on their decision making. These factors were common across schools that experienced varying degrees of success with implementation. However, several schools were looking to strengthen existing school climate efforts and actively seeking programming for school climate improvement around the time that the project began. These schools generally experienced greater success with the framework in terms of the milestones they achieved. In contrast, schools that were looking to begin working on improving their school’s climate at the start of the project generally experienced less success with the framework.

Desire to Improve School Climate.

Many POCs noted that a driving factor for their participation was that the project aligned with a specific or general need related to school climate (e.g., address bullying, improve safety, reduce suspensions, or strengthen overall school climate). For example, one POC noted, “I think for us, we were definitely prompted by the struggle around our suspension rate. Really looking for support to help reduce that while ensuring that our school climate was safe.” Several POCs also stated that the project aligned well with their school’s mission or their personal beliefs about supporting the non-academic needs of students. For instance, one POC explained, “fell right in line with all those things that we wanted to do as it related to building positive relationship with young people, fostering or promoting their academic and socio-emotional success, and… young people feeling good about themselves and feeling good about the school environment.” As noted above, although some POCs perceived the project as an opportunity to further school climate efforts that were already underway, others expressed that they had been seeking ways to begin addressing school climate, and this project seemed to provide a path forward.

Appeal of the Framework.

Many POCs also discussed aspects of the SSC framework that felt like a good fit for their school, with the framework’s flexibility, focus on data, holistic approach, and comprehensible structure mentioned most often. For instance, one POC explained that the project aligned with his school’s desire to become more data-driven. Several POCs emphasized that they liked the flexibility inherent in the framework to tailor programming to their school’s unique needs as identified in the data.

Several POCs also mentioned the potential for grant funding and/or certification. However, this was more often discussed as something that initially caught a POC’s attention and prompted them to examine the project more carefully, or an added bonus of participation, rather than a reason to participate. One POC stated explicitly that although the grant funding caught his attention, the survey data was what ultimately motivated him to participate.

External Influences on Decision Making.

Several POCs also discussed external influences on their decision making, such as the project aligned with district priorities, was a district “sponsored” project, or that someone at the district-level directly encouraged the school to participate. For instance, one POC admitted that his impressions of the project were likely influenced by the fact that “it was recommended, if you will, by a highly reputable and accomplished leader” in the district. Additionally, one POC noted that she found it reassuring to know “there were reputable schools as part of the partnership” and they reflected a wide range of schools, “from charter to traditional district, high performing to schools that are in turnaround mode.”

Interestingly, one POC noted that she liked the project because she perceived it to be flexible in response to external influences. She explained, “With the fluidity of school mandates,” this particular framework is “something that can be used even as mandates change because we know certain things are not going to change as far as the need to address the needs of students.”

What Factors Were Associated with Sustained Engagement?

Among the 18 schools for which we have implementation data (i.e., POC interviews and/or TAS interviews), level of engagement in the project and progress with the SSC framework varied widely, not only among schools but also within individual schools over time. As indicated previously, 12 schools participated in just 1 or 2 years of data collection and then either actively withdrew from the project or became nonresponsive to project team outreach. Among those that persisted, some only participated in survey data collection and never established a core leadership team (CLT), an important first step in working through the framework. Others put forth effort on some, but not all, elements of the framework or demonstrated bursts of effort that later fizzled out. Many of these schools worked on the first three elements of the framework, Leadership, Data, and Buy-In, but either did or did not submit a checkpoint 1 workbook and ultimately did not progress much beyond this first stage. Just four schools were consistently engaged in the process and ultimately completed the framework, even if their levels of effort fluctuated over the 4-year duration. Our analyses brought to light six themes that address facilitators of sustained participation and progress with the framework: (a) schools’ baseline capacity as a key aspect of readiness for participation, (b) the critical role of a champion in leading the charge at their respective schools, (c) shared leadership as a buffer to help schools navigate common challenges, (d) the need to take time at the beginning of the process to think big picture, (e) the role of data in keeping schools engaged, and (f) the importance of technical assistance to foster progress. Each of these themes is summarized in Table 4 and discussed in more detail below.

Table 4.

Summary of Themes Related to Sustained Engagement.

Theme	Summary
1	Schools’ baseline motivation and capacity to take on a multi-stage school climate initiative was associated with their long-term commitment to the project
2	School champions played a critical role in establishing and maintaining momentum at their respective schools; their absence often resulted in a loss of momentum and sometimes dissolution of the project
3	When school teams invested in establishing shared leadership of the project, they tended to be more nimble and better able to navigate common challenges
4	Taking time to think “big picture” at the beginning of the project allowed team members to establish a shared vision that helped them remain focused on their school climate goals as they worked through the steps in the framework
5	Engaging in discussions about their school climate data, including seeking stakeholder input, enabled team members to deepen their commitment to the project through reflection on how the data aligned with their personal/group beliefs
6	School teams that actively engaged with their technical assistance specialists viewed them as coaches who helped them troubleshoot challenges and progress through the framework

Although the implementation data overall point to these six factors as facilitators of sustained participation and progress with the framework, there were exceptions. For instance, although the school that achieved certification appeared to benefit from all of these facilitators, the other three schools that completed the framework each benefited from some but not others. Moreover, some facilitators are likely more important than others. For instance, one school lacking in baseline capacity struggled to progress beyond the first checkpoint despite having several other facilitators in place. Ultimately, there is no singular profile of a school that was well-positioned to be successful with SSC. On the other hand, the challenges that schools faced were common across schools, although some were better able to navigate those challenges than others.

Baseline Capacity as a Key Aspect of Readiness.

It was common for competing priorities to slow a school’s progress or, in some cases, completely halt a school’s efforts towards school climate improvement during the project. However, the TASs observed that some schools struggled with this issue much more than others. They described these schools as relying almost exclusively on reactive rather than proactive approaches to addressing problems, and as a result were constantly “putting out fires,” “always in a frenzy,” and “stretched really, really thin.” In one case, a TAS noted that a school with a high level of enthusiasm for the project—more than most other schools—was lacking in the “sustainable infrastructure” needed because “everybody is at or past capacity in order to do the work that they’re doing.” The schools that fit these descriptions often made little or no meaningful progress with the framework and, in some cases, could not even make time for the school climate surveys to be administered. Many of the schools that withdrew from the project had struggled to maintain focus on their climate improvement efforts given a lack of capacity to manage competing priorities.

On the other hand, the schools that were able to make progress with their climate improvement efforts had staff that were able to attend CLT meetings, review data reports, and take on other additional responsibilities without feeling overburdened. As discussed earlier, they were also the schools where CLTs were able to build off of existing momentum for climate improvement rather than starting from zero. The school that was, by all measures, most successful in the project may have been especially well-positioned for participation given its baseline capacity. Referring to this school’s principal, the TAS explained that he was not routinely dealing with the kinds of crises that some other schools faced on a consistent basis such as fighting and other serious behavior problems, or parents coming into the building and “cursing the people out,” and had additional capacity-related advantages: “He’s got these structures in place already that allow him the room to do some things like this to move the school forward.” For instance, this principal had existing mechanisms for obtaining input from key stakeholders, such as student and parent representatives and teacher leaders from different grade levels and content areas.

The Critical Role of a Champion in Leading the Charge.

Despite the framework’s emphasis on shared leadership, POCs generally served as a linchpin of project work, especially in the first year or two of the project, and led efforts to assemble a CLT, schedule meetings, and write or coordinate the workbooks. When POCs struggled to dedicate enough time to the project and/or were not fully committed to the project, schools struggled to make progress either at points in time or throughout their involvement in the project. POC turnover and lack of capacity also posed substantial challenges to sustained engagement in the project, particularly in the absence of shared leadership. In cases of POC turnover, the process and challenges of identifying and onboarding a new POC often led schools to lose momentum on any project efforts previously underway. At times, POC turnover led to a full restart of the project at a school.

POC capacity in the form of strong people skills and a certain level of authority within the school also made it easier for POCs to build buy-in and motivate staff to participate. One POC reflected on his experience, saying, “It took a lot of people work, and if you don’t have that skill, I could see it being very troublesome to roll this out to a staff that could be up and down.” One TAS noted that POCs need enough influence within the school to achieve high rates of attendance at meetings, particularly at the beginning of the project. Although some principals had too many responsibilities to fill the role of POC and could not have dedicated the time needed to lead the project, one POC who was principal felt that the project would not have been as successful at his school were he not the lead because “the principal is the biggest message sender and signal sender in the school community.” Indeed, at some schools, POCs who served in instructional or support staff roles rather than administrative roles or who were newer to the school and not as well-known encountered challenges related to their lack of authority.

Shared Leadership as a Buffer to Help Schools Navigate Common Challenges.

Early in the project, when few CLTs were assembled and operating, many POCs felt overwhelmed by the process, sometimes to the point at which they became nonresponsive to TAS outreach or requests to schedule survey data collection. Although the SSC framework promotes a shared leadership approach to improving school climate in the form of a group that represents a cross-section of the school’s various stakeholders and drives the process of change, it took quite a bit of time for many POCs to begin implementing shared leadership of the project, if they did at all.

For some schools, the “staff turnover trap” profoundly hindered their ability to make progress with the framework. However, schools that implemented shared leadership found that it made them more resilient to such challenges. Having a lot of people involved in the project and a shared energy around the importance of the work helped schools sustain their focus on climate improvement over time, despite competing priorities. As one POC explained, “So it was not only my putting it on the radar…. a number of different teams in the school helped to make it a priority, too.” Another POC explained, “It's not just one person carrying the mantle for this, but it's more of a charge led by multiple people.” In other words, when more people were involved in the process, schools were less dependent on any given individual who might leave or become too busy with other responsibilities.

The Need to Take Time at the Beginning to Think Big Picture.

Schools tended to be more successful with the framework when CLTs spent time early in the process thinking and talking about the big picture (e.g., what school climate means to them, what their school’s strengths and growth areas are, and how the process set forth in the framework can help them reach their goals for improvement). Throughout the project, TASs consistently voiced concerns about schools that seemed to be going through the motions of the framework, checking boxes and trying to demonstrate that they had completed certain requirements when, in the TAS’s mind, they had not deeply engaged in the process or made any meaningful progress. For instance, at some of these schools, the CLT met “very sporadic[ally]” in capacities limited to reviewing the data reports. The TASs often attributed this going through the motions to not understanding the big picture and how all the framework components work together and/or not having engaged in a thoughtful process to identify areas of growth and envision how the framework could benefit the school over time. TASs agreed that when schools invested this time early in the process to become grounded in the framework, reflect on why they were participating in the project, and develop a shared vision of what they hoped to achieve, they were much more likely to sustain their efforts and make meaningful progress. During his Year 4 interview, the POC at the school that ultimately achieved certification recalled that the process initially moved more slowly than he anticipated “because the whole kind of understanding that was involved took some time to develop.”

The Role of Data in Keeping Schools Engaged.

Schools varied with regard to the ways in which and extent to which they engaged with the school climate survey data. When school teams actively engaged with the data, it helped to further school efforts around climate improvement. However, at some schools, the data report was not distributed beyond the POC or CLT, nor did it lead to substantive conversations or the development of next steps. Often this was due to limited time or other resources for reviewing, reflecting on, and discussing the data. Sometimes, however, TASs perceived that school teams were dismissive of survey results, such as when there were delays in receiving the data, low response rates, or concerns about representativeness.

Among those schools that actively engaged with the school climate data, some POCs described the data as eye-opening for their school, leading to meaningful conversations and decisions about how to address the needs identified in the data report. For instance, one POC indicated that, after receiving one of the data reports, they were able to contract with specific vendors and develop professional development plans to address the needs identified in the report. At another school, the CLT shared the data reports with student leaders to obtain further contextual insights and develop action steps based on their discussions of the data.

Other POCs felt that the data provided quantitative evidence for what they knew anecdotally. Although POCs shared different perspectives on the value of this type of evidence, some found it very motivating. For example, one POC commented, “When I found out that [the data] aligned with my own personal beliefs of the needs [of our students], that really kind of motivated me to continue working.”

The Importance of Technical Assistance to Foster Progress.

POCs overwhelmingly described TAS support as playing a facilitative role in their progress with the framework and any successes they experienced along the way. At a basic level, TASs helped with scheduling CLT meetings, developing meeting agendas, and onboarding new staff. They also worked hard to ensure that POCs and CLTs understood the structure of the project and what they needed to do, step by step. This was especially important in the case of POC turnover. One POC who came into the role in the middle of Year 2, like others in similar situations, explained that she heavily utilized the support provided by her TAS to get up to speed with the project:

The way it was presented to me in the meeting with my principal was just, “Oh, it’s this project that we’re doing. It’s really important to us and this is why.” So, I was like, “Oh. Oh, that’s great. That’s a great thing.” And then when I understood all of the steps that needed to be taken for the project, it was just a lot for me to—I guess not “cope with,” but I can’t think of a better word. So, even now that I’ve kind of gotten a couple months in… I reach out to [TAS] a lot, like, “Okay, what else do we have to do.... Did I get everything? Am I missing anything?”

When asked to reflect on the value of the TAS support they received, POCs frequently emphasized how much they benefited from support related to documenting progress within the workbooks in order to meet the requirements of each checkpoint. This was a task that felt burdensome to many POCs and CLTs, particularly those with limited time and/or discomfort with writing. It was especially challenging for those POCs who had not been involved with the project from the beginning. Given high rates of turnover among both POCs and other CLT members as discussed earlier, the TAS was often the only person with a complete understanding of what the school had been doing up until that point in time. More than one POC described their TAS as a “lifesaver” or “godsend,” referring specifically to workbook-related support.

Finally, TAS support appeared to have the greatest impact on school leaders’ capacity for school climate improvement when it went beyond basic guidance for meeting the requirements of the framework and successfully completing and submitting workbooks to more of a thought partnership and/or coaching relationship. One POC reflected on the coaching that his TAS provided, noting that it was the most beneficial aspect of participation for himself and his school:

That’s part of what makes it really valuable to me [and] makes it potentially different from other kinds of support that’s out there for a school…. And it’s not just coaching for me because [TAS] also works with our core leadership team. He’s present to support the team and provide the team with guidance and questions that help the team reflect. So, when he’s doing that, he’s coaching the team as well as modeling for me strategies that I can use…. As long as those opportunities are there for that kind of customized support, it seems really valuable and beneficial as a school. If that weren’t as present, then it wouldn’t be as helpful.

For their part, TASs received initial training focused on familiarizing them with the framework, including the elements and desired states, the workbooks, and the rubrics. One commented, “There was a huge learning curve there, and I did get training on that.” They also received ongoing support throughout the project from the SSC developer and other project partners. However, despite all this, they often felt like they were “building the plane while flying it,” especially in the first year of the project. This was partly the result of numerous unexpected challenges. For instance, they understood their job would entail helping their schools work through the framework; however, in practice early on, they also found themselves needing to foster buy-in among school leaders and POCs who had a tenuous understanding of the project for which they had signed up. As one TAS noted, “[Other TAS] and I kind of were the ones to have to sell this to people, which was not always the most comfortable job.”

In addition, TASs, like the schools they were supporting, were new to the SSC process. They had only an abstract understanding of the framework and no clear roadmap for how to support a school through it. One TAS reflected in Year 2:

I think that I would have liked more… training on the front end, more familiarity. I mean, hindsight is 20/20…. Going into this, we had no idea what messages were going to stick, and which ones weren’t, and which were going to fall just flat…. I feel like I was… learning the framework as I was also explaining the framework, which, I think, was a little bit challenging for the schools.

However, despite the challenges, the TASs learned through trial and error: “When something worked in one school, I would try it out in another.” They also leaned heavily on one another for support: “We would make suggestions to each other, back and forth, kind of bounce ideas off of one another.”

To What Extent Did Schools Build Organizational Capacity?

The SSC framework defines capacity for school climate improvement relative to the eight elements’ desired states. A minority of POCs expressed uncertainty about whether participation helped to increase their school’s capacity for school climate improvement, and TASs felt that the benefits of participation were dependent upon schools’ efforts. Many POCs and TASs felt that participation indeed helped to increase schools’ capacity for school climate improvement, particularly with respect to the data and leadership elements. Three themes related to increased organizational capacity are summarized in Table 5 and discussed in more detail below.

Table 5.

Summary of Themes Related to Increased Organizational Capacity.

Theme	Summary
1	School teams whose participation went beyond survey data collection, who successfully involved their school leadership in the project, and who understood that real change takes time, felt that they made substantive and positive improvements in how they support school climate
2	Beyond progress that school teams made toward their school climate goals, school points of contact and technical assistance specialists both saw increased data literacy as an important secondary benefit of participation
3	School points of contact and technical assistance specialists at several schools noted a growing capacity for shared leadership and greater commitment to engaging the perspectives of parents and community members, which they saw applied outside of the school climate project

“What You Put Into It You Get Out of It”

A few POCs at schools that withdrew early from the project or participated in survey data collection, but did not engage in the SSC process more broadly, indicated that their schools did not benefit from participation. For instance, one POC commented that because her school never progressed beyond “the developmental phase” of deciding where to focus their efforts, they did not experience any growth in capacity. However, a POC at a school that passed checkpoint 2 similarly felt that her school may not have benefitted from participation. She explained, “Does it seem like a good use of time? Is it benefitting the school at large? I wouldn’t say yes…. There were times when we were able to talk about different ways to improve the school climate and culture” Ultimately, however, she did not feel that the “talk” resulted in action. She emphasized that the challenges of bringing people together made it difficult to make meaningful progress: “We’d try to have these meetings that encompassed school leaders, parents, students, all the people together; and hopefully, we had people in the room that could make these decisions and say ‘yes’ and ‘no’ to things.” Unfortunately, something always came up at the last minute and the “people who make decisions” were rarely able to attend.

One TAS commented, “This is very much a ‘what you put into it, you get out of it’ experience.” The TASs also felt that there were schools that embellished information in their workbooks about activities that had not been sustained or were never implemented in the first place. Perhaps this helps explain how a school could pass checkpoint 2 and not perceive any clear benefits from participation. One TAS explained the potential impetus behind a less-than-honest workbook submission:

There is the “change the school” [approach], and then there’s “prove that you’re changing the school.” And only one of those things is rewarded. So, “prove that you’re changing the school,” which is to say submitting workbooks and hopefully getting grant funds… is the thing that schools tend to focus on because it’s tangible and you can get feedback immediately. Whereas if you’re trying to sort of change the school culture, it takes a much longer time to get that feedback. And… it’s less tangible.

Data Literacy.

While not all POCs found the data they received as part of the project to be compelling, those that did indicated the data element of the framework helped their school become more proactive and data-driven in decision making; TASs observed changes in this area as well. For instance, one TAS reflected on an increase in data literacy at the end of Year 3:

I saw [some of] them thinking about their data a little bit differently… not just their [school climate survey] data, but also other data that they collect and just kind of going through the process of asking why they collect it and what are some of the potential uses. What information can they actually glean from [it].... I think that sometimes there’s too much to digest, but I saw them developing the beginnings of a sense of data literacy…. a little bit more critical thinking around understanding that more information isn’t always better.

This embracing of data represents a large shift in culture for some schools, based on TAS observations of data-related challenges early in the project when they observed, “People turn off to data or are afraid of it because it's become used as a weapon” to threaten educators’ jobs or “label schools as failing.”

Shared Leadership.

Some POCs also felt that the framework helped their school transition to more of a shared leadership approach, although some equated this with welcoming input from diverse stakeholders whereas others equated this with directly involving diverse stakeholders in the decision-making process. Beyond helping to give students a voice and bringing issues at the school to light, one POC commented, “There are more school community members actively involved in working towards productive solutions to the school climate challenges that we face.” A POC at a different school indicated that they have taken steps to ensure the continued involvement of many staff members going forward by “making sure that candidates Washington, DC are a match for what we're looking for and what our plan is to improve the culture of our building.” Another POC indicated that his school’s orientation towards parent involvement shifted dramatically as a result of participation in the project:

We became intentional about including families into projects and activities and really seeing the need because initially… we tried to find parents to be a part of the process. We really couldn’t. So, it forced us to be like, “Do we really invite families into the building? Or do we mandate them to come for disciplinary behavior or academic issues?” That’s really the only conversations that we had [previously] with our parents…. It was a good way to welcome that environment back for the parents.

This embracing of shared leadership, especially in the form of inviting parents and community members to be involved in school climate improvement efforts, represents another large culture shift for some schools. This was something with which TASs saw many schools struggle early in the project. As one TAS noted, schools in Washington, DC, historically, have not been very welcoming of feedback and engagement from parents and community members and, in some cases, have actively discouraged their involvement: “You've got to be… transparent and honest about stuff with parents…[but] that’s not a mentality that’s common in this city where parents are seen as part of the school. It's more about separation between school folks and parents.”

Discussion

Mounting evidence suggests that a positive school climate can promote learning and well-being for students and reduce bullying and other forms of violence. In recognition of this, at least 13 states have chosen to assess school climate as part of their state accountability plans under the 2015 Every Student Succeeds Act (ESSA) reauthorization of the Elementary and Secondary Education Act of 1965 (Jordan & Hamilton, 2020). However, many schools lack the capacity that is critical to successfully engage in comprehensive school climate improvement to promote student health and learning. This study examined implementation of a capacity-building framework for school climate improvement and identified barriers and facilitators of sustained engagement with that framework.

Although the outcomes evaluation, described in a companion paper (Ryberg et al., in preparation), found a significant, albeit small, effect for the intervention such that students at intervention schools had more positive changes in perceptions of school climate over time than did students at control schools, most participating schools did not make substantial progress with implementing the Safe School Certification (SSC) framework. A considerable number of schools either actively withdrew from the project or became nonresponsive to research team outreach. The barriers we identified are not surprising in the context of urban public schools; many schools found it challenging to dedicate time to the project given competing priorities, and school teams were frequently affected by administrative and/or staff turnover. The school staff member serving as the research team’s point of contact (POC) often changed over the course of the project as well, compounding other challenges and frequently resulting in a loss of momentum on project work.

Although the implementation challenges that we identified are not surprising, several of our findings warrant discussion. First, we found that schools most in need of capacity-building were the least likely to be successful with a framework designed to build capacity. This has important implications for equity in research, in terms of which schools are invited to participate in these types of initiatives and how we position them for success. Second, somewhat paradoxically, we found that although the SSC framework promotes a shared leadership approach, schools typically did not get very far without an individual champion who was able to effectively mobilize a leadership team. Although consistent with prior literature addressing the importance of a champion for program implementation and change efforts more broadly, the literature has given much less attention to the process by which a school initiates a shared leadership approach and the resources and mechanisms needed to support that process. Lastly, we found that TASs did not feel adequately prepared to support schools in implementing the SSC framework. As we reflected on this finding, we turned to the literature to better understand available theories and models of technical assistance; however, our search came up short. The existing literature offers little to guide the preparation of TASs for capacity-building efforts. These three findings are discussed in more detail below.

Readiness for Capacity Building

As discussed previously, school capacity is critical to successful climate improvement efforts, which can in turn promote student health, learning, and overall well-being. The SSC framework is designed to build that critical capacity within schools. However, we found that schools most in need of capacity-building were the least likely to be successful with the framework.

Although many schools struggled to get the project off the ground, those that struggled the most, and made the least progress, seemed to be those most limited in their organizational capacity and/or commitment to the project. This finding aligns with previous research (e.g., Dellar, 1998) on the importance of organizational readiness for successful implementation of comprehensive school- or organization-wide programs. Scaccia and colleagues (2015) describe three components of such readiness: motivation, general organizational capacity, and innovation-specific capacity. In our study, schools that were most successful in implementing SSC may have had higher levels of readiness based on at least two of these three components. These schools reported that they were initially motivated to join the project to strengthen existing school climate efforts and/or were actively looking for school climate programming. This suggests they may have joined the project with a higher level of motivation and organizational capacity (e.g., staffing and time) compared to other schools.

Prior to joining the project, one individual at each school was required to sign a memorandum of understanding (MOU) to demonstrate their commitment. Moreover, all schools expressed interest in improving their school climate. However, the difficulties many schools encountered suggest that this level of motivation or commitment alone may have been insufficient to facilitate implementation. At many schools, the individual who signed the MOU left the school during the 4-year project or did not actively engage with project activities.

With regard to capacity, schools in the intervention condition were encouraged to complete a current capacity assessment during Year 1 of the project. The purpose of this exercise was to assess each school’s baseline capacity relative to the desired states of the eight framework elements to identify areas for improvement. However, this process did not go smoothly; few schools completed the assessment. One TAS described it as a “frustrating exercise” because of the time and resources required to complete the process, while another described it as a “contentious process” because school staff felt harshly evaluated by a scoring rubric that assigned a score of zero for elements that were not sufficiently far along to qualify as “progressing.” As a result, many schools did not have a clear sense of their areas of strength and growth until much later in the project, when they began the process of completing the first workbook.

Research on other comprehensive school programs highlight the importance of assessing readiness prior to implementation; in fact, in many studies, readiness indicators inform decisions about which schools are selected for participation (e.g., Flaspohler et al., 2008; Kingston et al., 2018). In contrast, the school selection process in the present study was more an open call for participation than a selection of schools based on a set of criteria. Clearly, some type of readiness assessment may have been helpful to inform and aid the selection of schools for participation. In a final quarterly interview, one TAS noted:

I think that there should be some sort of vetting and onboarding process for schools engaging in Safe School Certification. There needs to be a set of, like, “You need a minimum capacity to hold this.” I would even go out on a limb and say you need a staff member who’s 50% on this project, somehow, and whose time can be allocated that way—and protected—to coordinate it and keep it moving forward.

Alternatively, or in addition, a readiness assessment with fewer barriers to completion (e.g., one that is brief and easy to administer, provides non-evaluative feedback, and focuses on the most critical elements of capacity, such as staffing resources), conducted prior to the start of the project, could help the TASs tailor their support of schools. Improvement and change efforts often need to take place when school readiness is not optimal (Elliott &Mihalic, 2004; Novick et al., 2002). As demonstrated in the present study, those schools with low readiness may in fact have the greatest need for capacity building or improvement. Therefore, instead of screening out these schools for participation, tailored technical assistance based on needs identified in a readiness assessment could support implementation in low readiness schools. One way this support could be tailored is by adjusting the project timeline for schools with lower levels of readiness. Research demonstrates that such schools need significant additional time to build readiness (Elliott &Mihalic, 2004), and our findings confirm that the project timeline was unrealistic for some schools.

Championing Shared Leadership

An important element of the SSC framework is shared leadership of school climate improvement efforts, and our findings support the idea that shared leadership facilitates engagement with, and progress of, such efforts. One way that shared leadership appeared to facilitate climate improvement efforts in our study was by helping schools navigate common challenges such as staff turnover and competing priorities. This in turn helped schools maintain momentum on efforts currently underway, even when individual staff members left the school or became too busy with other responsibilities; it ensured that there were others who were invested in the work, understood what the group was working towards, knew what had been done up to that point, and knew what else needed to be done in order to continue moving forward.

However, we also found that POCs very much championed the work at their respective schools, despite not having been asked to do so by the project team. This was particularly true early in the project. It seems that unless a school has already begun implementing shared leadership for school climate improvement, an intervening period can be expected, during which someone begins the process of building buy-in and bringing people together. In other words, a school cannot begin implementing shared leadership without a period of transition. In our study, it took most schools much longer than anticipated to form their core leadership teams (CLTs), and it was, in large part, the championing work that POCs undertook during this time that helped some schools eventually get to the point of forming a CLT. At other schools, this never happened; either a POC was never identified by school leadership or the POC was unsuccessful during this critical championing stage because of the types of challenges discussed earlier, such as competing priorities, an inability to build buy-in among staff, or because the POC left the school.

This finding aligns closely with prior literature that has demonstrated the importance of having someone to champion change in schools (e.g., Storey et al., 2016) and other types of organizations (e.g., Kotter, 1995), particularly early in the implementation process. This finding also dovetails with the suggestion from one of our TASs that the presence of a champion may be a critical aspect of organizational capacity that signifies readiness for implementation. Indeed, based on their examination of a large-scale dissemination and replication of the Blueprints for Violence Prevention, involving several hundred schools nationwide, Elliott and Mihalic (2004) found that the presence of a champion was a critical readiness condition that was ultimately related to successful implementation.

Elliott and Mihalic (2004) further noted that a champion should be “well-connected and respected,” which aligns with Kotter’s (1995) point that key responsibilities of a champion are to build initial buy-in and create a “sense of urgency” for a change effort. Our findings support this as well. In our study, POC capacity was critical to schools’ sustained engagement and progress with the framework, and this person-level capacity went beyond an ability and willingness to prioritize the work; it also included characteristics of the individual. In particular, strong people skills and authority within the school made it easier for POCs to build buy-in and motivate staff to participate.

Preparing Technical Assistance Specialists

SSC is intended to be a capacity-building intervention, and the technical assistance component is critical to building capacity. The framework itself highlights different buckets of capacity (e.g., buy-in, leadership, and data) and the desired states for each; however, schools need support to understand how to reach those desired states. For instance, the framework operationalizes buy-in in terms of staff survey results (“Buy-in survey data indicates that 80% or more of staff are committed to addressing school violence and improving school climate”) rather than as an active process. POCs at intervention schools perceived TAS support as critical, both to understanding the framework and finding ways to put it into action. For instance, one POC reflected on the process of achieving the desired states: “And that’s obviously why I have [TAS] and she can help me understand like…. Okay, but how do I get to that point? What’s an example of a way I can get there?” Technical assistance was also critical to helping schools think beyond the framework to ensure that they had a clear, collective vision for where they wanted to go and how the framework could help them get there. When schools bypassed this important step, the desired states rather than school climate outcomes ended up being the goals that schools were working toward.

At the school that achieved certification, the principal repeatedly cited tailored technical assistance as a critical component of his school’s progress with school climate improvement. However, his description of the technical assistance he and his CLT received went beyond basic support to more of a coaching relationship. This personalized support (e.g., one-on-one coaching to help the principal navigate challenges, modeling a variety of leadership strategies) went somewhat beyond what was intended, given that the SSC model emphasizes developing shared leadership rather than individual leadership skills. However, this individualized support, which closely mirrors the “Intensive Technical Assistance” described by Fixsen and colleagues (2009), appears to have been highly beneficial for the school and was greatly valued by the principal. Fixsen and colleagues (2009) describe this type of support as particularly beneficial when structural changes are needed within an educational setting in addition to individual knowledge and skills.

Unfortunately, our TASs felt that they were “building the plane while flying it” despite receiving an initial orientation to the framework and ongoing support from the program developer. The existing literature does not offer much guidance for how to better prepare TASs in the future. Although prior research clearly indicates that technical assistance improves the implementation of prevention and health promotion programs in a variety of community settings including schools (e.g., Dunst et al., 2019a; Durlak & DuPre, 2008), the literature is relatively sparse with regard to operationalizing high-quality technical assistance or proper TAS preparation. Much of the existing literature focuses on dimensions of technical assistance such as dosage, mode of delivery, the extent to which it is collaborative, and the extent to which it is responsive to the needs of recipients (e.g., Wandersman et al., 2012). However, within these dimensions, much has yet to be unpacked. A recent review of technical assistance models identified 25 core elements of technical assistance (Dunst et al., 2019b), contributing much to our collective understanding in this area; however, it also highlighted critical gaps in understanding, such as best practices for building capacity through technical assistance. The authors stated, “Although nearly all TA model and framework developers noted the importance of TA provider professional development… not a single model or framework included a description of or reference to evidence-based adult learning or professional development” (Dunst et al., 2019b, p. 114). With so many interventions utilizing technical assistance as a key mechanism for capacity building, future research is needed to help build theory around high-quality technical assistance and proper TAS preparation.

Limitations

We conducted in-depth discussions with technical assistance specialists (TASs) and school points of contact (POCs) to understand schools’ experiences with implementing the SSC framework. Despite the rich insights that these interviews provided, several limitations should be noted.

Ideally, the implementation team would have conducted interviews with both TASs and POCs at all intervention schools, as well as with POCs at all control schools, across multiple time points. However, POC interviews were challenging to schedule given the many demands on school administrators’ and staffs' time. High levels of staff turnover, discussed earlier, may have contributed to these challenges, especially when there was turnover of the staff member serving as POC. New staff who were less familiar with the project may have been less inclined to participate in an interview. As a result, the implementation team relied heavily on TAS interviews, which occurred quarterly over the course of the project and provided insights into the experiences of all 13 intervention schools. In addition, because of the scheduling challenges we encountered during the first two waves of POC interviews, we focused our final wave of POC interviews on those schools that passed checkpoint 2, which included three intervention schools and one control school. Together, these factors resulted in having access to more data on intervention schools than control schools. We do not know how well, if at all, our findings reflect the experiences of the seven control schools for which we had no interview data.

Moreover, the TAS perspective alone may provide a limited view of implementation at intervention schools. There were times when a school was not responsive to TAS outreach or repeatedly rescheduled meetings with their TAS to the point where months passed without contact. If a school submitted a project workbook, this was a clear indication that they were still engaged, regardless of TAS involvement. However, in the absence of a POC interview, it was sometimes difficult to determine if a school was continuing to engage with the framework.

Finally, the SSC framework encourages the completion of particular activities in order to reach the desired states of the eight elements, but it does not explicitly promote depth or quality of engagement. As a result, it was difficult for the implementation team to assess implementation quality. The implementation team also was unable to assess progress with the framework (i.e., relative to a school’s baseline capacity) because the current capacity assessment was something that only intervention schools completed and, as noted earlier, they did not all complete it. We therefore chose to focus our analyses instead on the milestones achieved by schools and their sustained engagement over time.

Conclusion

This study offers some important lessons when it comes to implementing, in urban schools, comprehensive, school-wide efforts that have the potential to promote student health, learning, and overall well-being. First, it is possible to make cultural shifts when it comes to data and shared leadership in schools. This is no small accomplishment and one that can position schools to make sustainable progress in a range of areas when they are able to translate these assets to other endeavors. Indeed, in our study there was evidence that schools were able to apply increased data literacy skills to problems beyond school climate. Additionally, school leaders reported shifts in the way they intend to engage with families and community partners in the future. That said, the work of achieving sustainable change that is resilient to staff turnover and a constant barrage of competing priorities is difficult even in the best of circumstances. The field of school climate promotion and violence prevention must work to determine more effective ways to position schools for success. In particular, it seems a strategy for assessing readiness that also connects schools with resources to fill critical gaps in capacity could better position schools to engage in a process that requires leadership, buy-in, and an ability to make data-driven decisions. Finally, more attention must be paid to the core components of evidence-based technical assistance. There often seems to be a belief that technical assistance will improve outcomes, but it is critical to understand the process by which schools and technical assistance providers can establish the relationships that will not only foster intervention-specific improvements but also create enhanced capacity to sustain those improvements and tackle new challenges.

Footnotes

Declaration of Conflicting Interests

The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This paper was supported by Award No. 2015-CK-BX-0016, awarded by the National Institute of Justice, Office of Justice Programs, U.S. Department of Justice. The opinions, findings, and conclusions or recommendations expressed in this publication are those of the authors and do not necessarily reflect those of the U.S. Department of Justice.

Data Availability

The de-identified interview data used in this study have been submitted to and are currently in the process of being deposited in the NACJD archive with project ID NACJD_NIJ-123181.

Note

References

Aldridge

J. M.

McChesney

(2018). The relationships between school climate and adolescent mental health and wellbeing: A systematic literature review. International Journal of Educational Research, 88(1), 121–145. https://doi.org/10.1016/j.ijer.2018.01.012.

Barbour

R. S.

(2001). Checklists for improving rigour in qualitative research: A case of the tail wagging the dog? Bmj: British Medical Journal, 322(7294), 1115–1117. https://doi.org/10.1136/bmj.322.7294.1115.

Bauer

M. S.

Damschroder

Hagedorn

Smith

Kilbourne

A. M.

(2015). An introduction to implementation science for the non-specialist. BMC Psychology, 3(1), 32. https://doi.org/10.1186/s40359-015-0089-9.

Bazeley

(2013). Qualitative data analysis: Practical strategies. SAGE Publications Inc.

Bertram

R. M.

Blase

K. A.

Fixsen

D. L.

(2015). Improving programs and outcomes. Research on Social Work Practice, 25(4), 477–487. https://doi.org/10.1177/1049731514537687.

Chriqui

Stuart-Cassel

Piekarz-Porter

Temkin

Lao

Steed

Harper

Leider

Gabriel

(2019). Using state policy to create healthy schools: Coverage of the whole school, whole community, whole child framework in state statutes and regulations, school year 2017–2018. Child Trends Publication # 2019–09. https://www.childtrends.org/publications/using-policy-to-create-healthy-schools.

Dedoose (2020). Dedoose Version 8.3.35, web application for managing, analyzing, and presenting qualitative and mixed method research data. SocioCultural Research Consultants, LLC. www.dedoose.com.

Dellar

G. B.

(1998). School climate, school improvement and site-based management. Learning Environments Research, 1(3), 353–367. https://doi.org/10.1023/A:1009970210393.

Domitrovich

C. E.

Bradshaw

C. P.

Poduska

J. M.

Hoagwood

Buckley

J. A.

Olin

Romanelli

L. H.

Leaf

P. J.

Greenberg

M. T.

Ialongo

N. S.

(2008). Maximizing the implementation quality of evidence-based preventive interventions in schools: A conceptual framework. Advances in School Mental Health Promotion, 1(3), 6–28. https://doi.org/10.1080/1754730x.2008.9715730.

10.

Dunst

C. J.

Annas

Wilkie

Hamby

D. W.

(2019a). Review of the effects of technical assistance on program, organization and system change. International Journal of Evaluation and Research in Education (IJERE), 8(2), 330–343. https://doi.org/10.11591/ijere.v8i2.17978.

11.

Dunst

C. J.

Annas

Wilkie

Hamby

D. W.

(2019b). Scoping review of the core elements of technical assistance models and frameworks. World Journal of Education, 9(2), 109–122. https://files.eric.ed.gov/fulltext/EJ1215369.pdf.

12.

Durlak

J. A.

DuPre

E. P.

(2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41(3–4), 327–350. https://doi.org/10.1007/s10464-008-9165-0.

13.

Eliot

Cornell

Gregory

Fan

(2010). Supportive school climate and student willingness to seek help for bullying and threats of violence. Journal of School Psychology, 48(6), 533–553. https://doi.org/10.1016/j.jsp.2010.07.001.

14.

Elliott

D. S.

Mihalic

(2004). Issues in disseminating and replicating effective prevention programs. Prevention Science, 5(1), 47–53. https://doi.org/10.1023/b:prev.0000013981.28071.52.

15.

Fagan

A. A.

Hanson

Hawkins

J. D.

Arthur

M. W.

(2009). Translational research in action: Implementation of the communities that care prevention system in 12 communities. Journal of Community Psychology, 37(7), 809–829. https://doi.org/10.1002/jcop.20332.

16.

Feinberg

M. E.

Bontempo

D. E.

Greenberg

M. T.

(2008). Predictors and level of sustainability of community prevention coalitions. American Journal of Preventive Medicine, 34(6), 495–501. https://doi.org/10.1016/j.amepre.2008.01.030.

17.

Feinberg

M. E.

Chilenski

S. M.

Greenberg

M. T.

Spoth

R. L.

Redmond

(2007). Community and team member factors that influence the operations phase of local prevention teams: The Prosper project. Prevention Science, 8(3), 214–226. https://doi.org/10.1007/s11121-007-0069-2.

18.

Fixsen

D. L.

Blase

K. A.

(2009). Implementation: The missing link between research and practice. NIRN Implementation Brief #1. The University of North Carolina, FPG, NIRN. https://files.eric.ed.gov/fulltext/ED507422.pdf.

19.

Fixsen

D. L.

Blase

K. A.

Horner

Sugai

(2009). Intensive technical assistance. Scaling up Brief #2. The University of North Carolina, FPG, SISEP. https://nirn.fpg.unc.edu/resources/scaling-brief-2-intensive-technical-assistance.

20.

Flaspohler

P. D.

Anderson-Butcher

Bean

Burke

R. W.

Paternite

C. E.

(2008). Readiness and school improvement: Strategies for enhancing dissemination and implementation of expanded school mental health practices. Advances in School Mental Health Promotion, 1(1), 16–27. https://doi.org/10.1080/1754730X.2008.9715719.

21.

Gage

N. A.

Prykanowski

D. A.

Larson

(2014). School climate and bullying victimization: A latent class growth model analysis. School Psychology Quarterly, 29(3), 256–271. https://doi.org/10.1037/spq0000064.

22.

Greenberg

M. T.

(2010). School‐based prevention: Current status and future challenges. Effective Education, 2(1), 27–52. https://doi.org/10.1080/19415531003616862.

23.

Guba

E. G.

(1981). Criteria for assessing the trustworthiness of naturalistic inquiries. Ectj, 29(2), 75–91. https://www.jstor.org/stable/30219811.

24.

Hawkins

J. D.

Catalano

R. F.

Arthur

M. W.

Egan

Brown

E. C.

Abbott

R. D.

Murray

D. M.

(2008). Testing communities that care: The rationale, design and behavioral baseline equivalence of the community youth development study. Prevention Science, 9(3), 178–190. https://doi.org/10.1007/s11121-008-0092-y.

25.

Horner

R. H.

Kincaid

Sugai

Lewis

Eber

Barrett

Dickey

C. R.

Richter

Sullivan

Boezio

Algozzine

Reynolds

Johnson

(2014). Scaling up school-wide positive behavioral interventions and supports. Journal of Positive Behavior Interventions, 16(4), 197–208. https://doi.org/10.1177/1098300713503685.

26.

Horner

R. H.

Sugai

(2015). School-wide PBIS: An example of applied behavior analysis implemented at a scale of social importance. Behavior Analysis in Practice, 8(1), 80–85. https://doi.org/10.1007/s40617-015-0045-4.

27.

Jordan

P. W.

Hamilton

L. S.

(2020). Walking a fine line: School climate surveys in state ESSA plans (Report). FutureEd. https://www.future-ed.org/wp-content/uploads/2020/01/FutureEdSchoolClimateReport.pdf.

28.

Kann

McManus

Harris

W. A.

Shanklin

S. L.

Flint

K. H.

Hawkins

Queen

Lowry

O’Malley Olsen

R. E.

Olsen

E. O. M.

Chyen

Whittle

Thornton

Lim

Yamakawa

Brener

Zaza

(2016). Youth risk behaviour surveillance United States, 2015. MMWR. Surveillance Summaries, 65(6), 1–174. https://www.cdc.gov/healthyyouth/data/yrbs/pdf/2015/ss6506_updated.pdf.

29.

Kingston

Mattson

S. A.

Dymnicki

Spier

Fitzgerald

Shipman

Goodrum

Woodward

Witt

Hill

K. G.

Elliott

(2018). Building schools’ readiness to implement a comprehensive approach to school safety. Clinical Child and Family Psychology Review, 21(4), 433–449. https://doi.org/10.1007/s10567-018-0264-7.

30.

Klem

A. M.

Connell

J. P.

(2004). Relationships matter: Linking teacher support to student engagement and achievement. Journal of School Health, 74(7), 262–273. https://doi.org/10.1111/j.1746-1561.2004.tb08283.x.

31.

Kotter

J. P.

(1995). Leading change: Why transformation efforts fail. Harvard Business Review, 73(2), 56–67. https://hbr.org/1995/05/leading-change-why-transformation-efforts-fail-2.

32.

Kutsyuruba

Klinger

D. A.

Hussain

(2015). Relationships among school climate, school safety, and student achievement and well-being: A review of the literature. Review of Education, 3(2), 103–135. https://doi.org/10.1002/rev3.3043.

33.

Ladd

G. W.

Dinella

L. M.

(2009). Continuity and change in early school engagement: Predictive of children’s achievement trajectories from first to eighth grade? Journal of Educational Psychology, 101(1), 190–206. https://doi.org/10.1037/a0013153.

34.

Lester

Cross

(2015). The relationship between school climate and mental and emotional wellbeing over the transition from primary to secondary school. Psychology of Well-Being, 5(9), 9–15. https://doi.org/10.1186/s13612-015-0037-8.

35.

Meyers

D. C.

Durlak

J. A.

Wandersman

(2012). The quality implementation framework: A synthesis of critical steps in the implementation process. American Journal of Community Psychology, 50(3–4), 462–480. https://doi.org/10.1007/s10464-012-9522-x.

36.

Miles

M. B.

Huberman

A. M.

(1994). Qualitative data analysis: An expanded sourcebook (2nd ed.). Sage Publications, Inc.

37.

Miller

R. L.

Shinn

(2005). Learning from communities: Overcoming difficulties in dissemination of prevention and promotion efforts. American Journal of Community Psychology, 35(3–4), 169–183. https://doi.org/10.1007/s10464-005-3395-1.

38.

Mrazek

P. J.

Haggerty

R. J.

(Eds). (1994). Reducing risks for mental disorders: Frontiers for preventive intervention research. The National Academies Press. https://doi.org/10.17226/2139.

39.

Nese

McIntosh

Nese

Hoselton

Bloom

Johnson

Richter

Phillips

Ghemraoui

(2016). Predicting abandonment of school-wide positive behavioral interventions and supports. Behavioral Disorders, 42(1), 261–270. https://doi.org/10.17988/BD-15-95.1.

40.

Novick

Kress

J. S.

Elias

M. J.

(2002). Building learning communities with character: How to integrate academic, social, an emotional learning. Association for Supervision and Curriculum Development.

41.

O'Connell

M. E.

Boat

Warner

K. E.

(Eds), (2009). Preventing mental, emotional, and behavioral disorders among young people: Progress and possibilities. The National Academies Press. https://doi.org/10.17226/12480.

42.

Parker

C. J.

Winston III

III Simpson

Brady

S. S.

(2018). Community readiness to adopt the Communities That Care prevention system in an urban setting. American Journal of Preventive Medicine, 55(5), S59–S69. https://doi.org/10.1016/j.amepre.2018.05.022.

43.

Ryberg

Her

Temkin

Madill

Kelley

Thompson

Gabriel

(2020). Measuring school climate: Validating the education department school climate survey in a sample of urban middle and high school students. AERA Open, 6(3), 1–21. https://doi.org/10.1177/2332858420948024.

44.

Ryberg

Her

Temkin

Rodriguez

(2020). Associations between a school organizational capacity building model and improvements in student perceptions of school climate . Manuscript in preparation.

45.

Scaccia

J. P.

Cook

B. S.

Lamont

Wandersman

Castellow

Katz

Beidas

R. S.

(2015). A practical implementation science heuristic for organizational readiness: R = Mc2. Journal of Community Psychology, 43(4), 484–501. https://doi.org/10.1002/jcop.21698.

46.

Shenton

A. K.

(2004). Strategies for ensuring trustworthiness in qualitative research projects. Education for Information, 22(2), 63–75. https://doi.org/10.3233/EFI-2004-22201.

47.

Shernoff

D. J.

Ruzek

E. A.

Sinha

(2017). The influence of the high school classroom environment on learning as mediated by student engagement. School Psychology International, 38(2), 201–218. https://doi.org/10.1177/0143034316666413.

48.

Spoth

Guyll

Lillehoj

C. J.

Redmond

Greenberg

(2007a). Prosper study of evidence-based intervention implementation quality by community-university partnerships. Journal of Community Psychology, 35(8), 981–999. https://doi.org/10.1002/jcop.20207bib_Spoth_et_al_2007a.

49.

Spoth

Redmond

Shin

Greenberg

Clair

Feinberg

(2007b). Substance-use outcomes at 18 months past baseline. American Journal of Preventive Medicine, 32(5), 395–402. https://doi.org/10.1016/j.amepre.2007.01.014.

50.

Storey

K. E.

Montemurro

Flynn

Schwartz

Wright

Osler

Veugelers

P. J.

Roberts

(2016). Essential conditions for the implementation of comprehensive school health to achieve changes in school culture and improvements in health behaviours of students. BMC Public Health, 16(1), 1133. https://doi.org/10.1186/s12889-016-3787-1.

51.

Suárez-Orozco

Pimentel

Martin

(2009). The significance of relationships: Academic engagement and achievement among newcomer immigrant youth. Teachers College Record, 111(3), 712–749. https://research.steinhardt.nyu.edu/scmsAdmin/uploads/004/293/TCR%202009.pdf.

52.

Thapa

Cohen

Guffey

Higgins-D’Alessandro

(2013). A review of school climate research. Review of Educational Research, 83(3), 357–385. https://doi.org/10.3102/0034654313483907.

53.

Turri

M. G.

Mercer

S. H.

McIntosh

Nese

R. N. T.

Strickland-Cohen

M. K.

Hoselton

(2016). Examining barriers to sustained implementation of school-wide prevention practices. Assessment for Effective Intervention, 42(1), 6–17. https://doi.org/10.1177/1534508416634624.

54.

Wandersman

Chien

V. H.

Katz

(2012). Toward an evidence-based system for innovation support for implementing innovations with quality: Tools, training, technical assistance, and quality assurance/quality improvement. American Journal of Community Psychology, 50(3-4), 445–459. https://doi.org/10.1007/s10464-012-9509-7.

55.

Wandersman

Duffy

Flaspohler

Noonan

Lubell

Stillman

Blachman

Dunville

Saul

(2008). Bridging the gap between prevention research and practice: The interactive systems framework for dissemination and implementation. American Journal of Community Psychology, 41(3-4), 171–181. https://doi.org/10.1007/s10464-008-9174-z.

56.

Wang

M.-T.

Degol

J. L.

(2016). School climate: A review of the construct, measurement, and impact on student outcomes. Educational Psychology Review, 28(2), 315–352. https://doi.org/10.1007/s10648-015-9319-1.