Same,Same but Different: Comparing Program Fidelity at Early Versus Late Adopters of a Successful Long-Term School-Based Physical Activity Intervention

The Intervention

The program was initiated at six schools in 2008 and is currently running at all schools within the Danish municipality of Svendborg. All 15 public schools in the municipality of Svendborg were invited to participate in SP requiring them to uphold that (a) the pupils in preschool (age 5) to sixth grade (age 12) should increase their amount of weekly PE, going from 1.5 to 4.5 hours; (b) PE lessons should be distributed across a minimum of 3 days a week; (c) PE teachers were obligated to participate in a competence development course based on an Age-Related Training Concept (ATC; Pryce et al., 2005); (d) PE teachers had to plan 3 weekly hours of PE based on ATC-related content; and (e) pupils were to receive at least 20 minutes of high-intensity activity in each PE session. This made the PE teachers the main agents of change in SP, yet with a core focus on having school heads as important agents to qualify supportive structures and coordinate with municipal program managers.

Six schools stated that they had the capability (practically or economically) to prioritize the program. The six schools also stated that a main reason for them to become part of SP was the match to their existing culture, values, and experiences toward school-based PA (Nielsen, Klakk, et al., 2018; Nielsen, Skovgaard, et al., 2018). Conversely, lack of alignment was the main reason for the remaining schools in the municipality to decline participation (Nielsen, Klakk, et al., 2018). There was no significant difference in parental education level between participating and nonparticipating schools.

Program Evaluation

SP was initiated by municipal authorities and developed through a partnership between municipal employees, school heads, and school personnel. In SP, researchers only had an advisory role; thus, the researchers did not have direct influence on the initiation, design, implementation, or the maintenance of the program. Researchers could, however, follow the program from the point of initiation in 2008, with the purpose of evaluation. This led to SP being connected to a substantial and ongoing program of research, the Childhood Health, Activity, and Motor Performance School Study Denmark (the CHAMPS-study DK), being connected to SP. Evaluations of SP showed that the main components of the program had been adopted, implemented, and maintained with high fidelity (Nielsen, Skovgaard, et al., 2018). Furthermore, the effectiveness of SP was evaluated by including four schools in the municipality both matching the geographical area and parents’ socioeconomic status as a control group. Effectiveness evaluations showed that SP had produced a number of intended positive health outcomes. including pupils becoming more active during school time (Moller et al., 2014) as well as a decrease in the number of pupils being overweight and obese (Klakk et al., 2013), a reduction of cardiovascular risk factors (Klakk et al., 2014), and improved fitness in pupils with low fitness levels (Rexen et al., 2015).

Going to Scale

The successful evaluation of SP secured political interest and momentum to scale the program within municipality borders (Nielsen, Skovgaard, et al., 2018). Thus, in 2012 the remaining nine schools were once again invited and all decided to become part of SP. As a result of the program going to scale within municipal borders, SP now contains three different school groups engaged in the program: (a) six original intervention schools (OI-schools) enrolled in 2008; (b) four scaling control schools (SC-schools), included in 2008 but not implementing the program before 2012; and (c) five scaling standard schools (SS-schools) who also implemented the program in 2012. Thus, SP contains both early adopters (OI-schools) and two groups of late adopters (SC-schools and SS-schools). Table 1 displays how the different school groups have been involved in SP over time. Still, it is unknown if or how the original program has been implemented and maintained across the different school groups. Therefore, the aim of this study is to determine program fidelity across the different school groups and how early versus late adopters (schools with an initial ethos not matching the program) are adapting the intervention. Based on the results, scaling strategies for school-based PA programs will be discussed.

OPEN IN VIEWER

Table 1

The Three Different School Groups and Their Involvement in SP Over Time

Year	2008	2009	2010	2011	2012	2013	2014	2015	2016	2017	2018	2019
Original intervention schools (n = 6)	I	I	I	I	I	I	I	I	I	I	I	I
Scaling control schools (n = 4)	C	C	C	C	I	I	I	I	I	I	I	I
Scaling standard schools (n = 5)	—	—	—	—	I	I	I	I	I	I	I	I

Note. SP = Svendborgproject; PE = physical education; I = part of SP and implementing the added amount of PE; C = control group in the effect evaluation design.

Questionnaire Design

A questionnaire to determine the degree to which program elements were delivered as designed (fidelity) was developed for PE teachers. Fidelity questions covered the required program elements that schools had to uphold to be part of the program. In addition to fidelity, the questionnaire also included questions regarding PE teacher experiences of working at a school that was part of SP (e.g., if they were sufficiently informed and qualified to engage in program-related tasks) as well as general information on their employment. As PE teachers do not teach PE in all grades from preschool to sixth grade, they were asked to report grade-specific fidelity measures for all grades they were actually teaching, for example, “Do your fifth graders have 4.5 hours of PE a week?”

To strengthen content validity, the questionnaire was initially reviewed by two researchers who had in-depth knowledge of SP. Second, as municipal program managers had detailed knowledge on overall content and structure, they were asked to review all questions in relation to fidelity issues. Finally, to heighten face validity, two PE teachers reviewed the survey for any ambiguous or difficult questions the respondents might encounter.

Data Collection

The questionnaire was administrated electronically through the survey system SurveyXact. The use of electronic surveys made it possible to minimize irrelevant questions by only activating certain questions based on specific answers—for example, if teachers answered that they had participated in the professional development course, they would get additional questions about the usability of the course. The questionnaire was online for 4 weeks with two reminders: one midterm and one 3 days before the questionnaire was closed.

The questionnaire was distributed by the local school heads to all teachers who delivered PE at their individual school. Prior to the distribution, the questionnaire had been presented to all school heads, and they were given the opportunity to submit objections and proposals for its distribution. There were no objections. Questionnaire data were collected between November and December 2016 among all teachers who conducted PE (n = 169) across all 15 schools participating in SP. This included full-time and part-time PE teachers, substitute teachers, and teaching assistants. At this time, OI-schools had been engaged in SP for 8 years while SC-schools and SS-schools had been committed for 4 years.

Data Analysis

STATAv15 was used to handle questionnaire data. Due to the paucity of responses, the values “partially agree” and “agree” were collapsed, and the value “partially disagree” was collapsed with the value “disagree” in the analysis. Thus, two categories were used in the analysis: (a) agree and (b) disagree. Grade-specific responses (e.g., Do your fifth graders have 4.5 hours of PE a week?) were merged to estimate fidelity on the overall school level across all grades. Finally, descriptive statistics and proportions on implementation fidelity were produced.

Ethics

The questionnaire was anonymous and did not collect personally identifiable information such as name, address, or civil registration data. In addition, all PE teachers in the study received an information letter to inform them about the study. A passively informed consent procedure was used, automatically including the PE teachers unless they actively withdrew consent. It was made easy to withdraw from the study without providing any explanation. To withdraw from the study, participants had to inform the research team through the given contact information in the information letter. This procedure is in accordance with Danish regulations for anonymous and low-risk research. The program of research was approved by the Danish Research Ethics Committee (Project-ID: S-20080047 and S-20140105).

Schools Are Same, Same but Different

As noted in the MRC framework, complex interventions are often highly dependent on the local context which can lead to different implementation and effect outcomes (Skivington et al., 2021). If we consider behavior as shaped by the local school culture, values, and experiences, then two seemingly alike schools (e.g., in size, geographical location, and pupils’ socioeconomic status) can be substantially different (Skivington et al., 2021). Despite their different context and introduction into SP, both SC- and SS-schools chose to implement the program when it gained positive political attention and, at the same time, showed to be both feasible and effective (Nielsen, Skovgaard, et al., 2018). This kind of positive attention, together with tangible implementation experiences from other schools, can serve as important support for scaling schools and in general strengthen the implementation capacity within municipal borders (Bartelink et al., 2019; Skivington et al., 2021).

Findings from the current study also show rather good program fidelity across all school groups. Teachers in OI-schools, in general, report having implemented the program with higher fidelity than SC- and SS-schools. Although the results show variation in individual school fidelity among all school groups, the largest variation is found across SS-schools. This indicates a higher degree of implementation difficulty or resistance in these late adopters, who did not become involved in the program until municipal politicians promoted the need for the program to go to scale. Noticeably, SC-schools are also characterized as late adopters, yet these schools are showing a larger degree of fidelity with much less individual school variation. This suggests that SC-schools, due to their previous nonparticipating involvement in SP, were more familiar with and/or more willing to comply with program requirements when being allowed/asked to implement the program at a later time. Still, when addressing the action of going to scale, it is also important to acknowledge that schools are complex social adaptive systems in which the embedded individuals (e.g., heads, teachers, pupils, and parents) and their behavior frame the system (Duke, 2019; Keshavarz et al., 2010; Skivington et al., 2021). Notwithstanding, organizational change is required when implementing new practice in school settings and interventions have to be versatile enough to secure sufficient local adaptability to facilitate organizational change across various schools with diverse contexts (Daly-Smith et al., 2020; Duke, 2019; Parry et al., 2018).

As highlighted in the MRC framework, refinement of the intervention should be highly informed by user involvement (Skivington et al., 2021). Such user informed refinements were made prior to the scaling of SP, yet solely based on the involvement of OI-schools (Nielsen, Skovgaard, et al., 2018). Thus, future refinements based on the specific implementation experiences from SC- and SS-schools are recommended to improve the program. However, organizations and local implementation agents must still be willing and able to implement the required changes within the frame of the organization’s mission, priorities, and practices (Daly-Smith et al., 2020; Hirschhorn et al., 2018). For reasons such as these, it is crucial to find the right approach to making use of existing organizational capacity and practice when implementing something new.

The Influence of School Head Support

The MRC framework highlights the importance of stakeholders in improving the accessibility, feasibility, and implementation of a given program (Skivington et al., 2021). The local school head is viewed as one of the main stakeholders in SP. In general, backing from local school heads is an important factor to secure the motivation of school personnel and the implementation of school-based programs (Bush, 2019; Lee & Welk, 2021). The findings show that teachers at SS-schools experienced the lowest degree of backing from their school heads (44%)—despite displaying mid-range to high fidelity on SP requirements. However, it is worth mentioning that OI-schools have fairly high implementation fidelity, yet not many teachers (56%) experience that their local head is behind the SP.

Teachers’ low experience of school head backing could initially be viewed as a sign of implementation fatigue, yet OI-schools still show moderate to high implementation fidelity (63%–98% fidelity across various program requirements). Compared with SS-schools, OI-schools have been part of SP for the longest period of time why these results might imply that OI-schools have reached a degree of program sustainability instead. Based on this, we would like to hypothesize that although school heads are central in the decision to adopt a program, continual support from school heads might be less important for long-term sustainability when main program elements are in place. The MRC framework highlights that such switches in stakeholder priority during program stages are not unseen, yet potentially vital as they entail different strategies and engagements to uphold and improve the implementation process (Skivington et al., 2021).

This would imply that implementation and scaling should be addressed with different approaches. For example, Spence and Lee proposed that factors close to teachers (e.g., school heads on the organizational micro level) have a more direct influence on the implementation process than more distant factors (e.g., political structural level) (Spence & Lee, 2003). In relation, Cote-Boileau et al. (2019) found that health innovation programs that were going to scale, even in ill-suited contexts, might succeed if a sufficient part of the organization had adopted the program. These types of findings raise interesting questions whether support from the local school head might be less important, and whether the support for the program has been strengthened on other higher structural and organizational macro levels when school-based PA programs are going to scale.

Building Implementation Capacity Through Front-Runners and Courses

The MRC framework emphasizes that a program needs refinement going from one stage to another based on previous implementation experiences (Skivington et al., 2021). SP has gone to scale—including both SC- and SS-schools—and with a rather good implementation fidelity. However, when scaling up programs, it is important to also scale organizational capacity. This implies refining the program theory according to the changing features of the expected conditions and context of SC- and SS-schools (Skivington et al., 2021). A main goal of such refining is to assure program fidelity in schools with, perhaps, less motivated providers working within highly variable and sometimes insufficient infrastructures and, possibly, less attention on the program (Ogden & Fixsen, 2014).

In SP, engaging schools with previous experience of implementing PA into their school (OI-school) acted as a driving force to help strengthen the local organizational capacity and political backing that enabled the program to scale (Nielsen, Klakk, et al., 2018; Nielsen, Skovgaard, et al., 2018). During the early implementation process, OI-schools also engaged in the adaptation of the program to their individual local school context, hereby, establishing guideline implementation strategies that could aid less motivated schools (Nielsen, Klakk, et al., 2018; Nielsen, Skovgaard, et al., 2018). We, therefore, suggest that it is possible to qualify the scaling process by having motivated schools with preexisting experience and motivation of PA as “pilot implementers,” grinding the edges of the program and establishing solid political backing of the program. Still, OI-school shows better overall fidelity than SC- and SS-schools. To strengthen the organizational capacity and improve the fidelity on SC- and SS-schools, it might have been beneficial to actively engage OI-schools as a direct support (e.g., through a mentoring scheme) to these late adopters. This falls in line with the MRC framework recommendations on refining the program theory to support the delivery to a new local context and involving stakeholders with direct experience of the practical obstacles when implementing SP (Skivington et al., 2021).

To support organizational capacity and enable high-quality implementation, it is also considered a hallmark to have the local implementers (in SP this was the PE teachers) attend a training course (Adamowitsch et al., 2017; Naylor et al., 2015). Such training courses are deemed vital as they not only help develop the skills of teachers but also feed into their expectations, motivation, and continuous support of the program (Naylor et al., 2015; Simovska et al., 2016; Skivington et al., 2021). In the current study, the lowest fidelity measured across all school groups is the PE teachers’ participation in the competence development course and the application of its ATC content in their PE lessons. In SP, the course consists of four modules to be completed after the teachers’ normal working hours. Follow-up courses have been planned once every second year since program start-up in 2008, still there seems to be a need for more frequent courses. However, on SS-schools, 21% of the teachers had been employed within the last year at the point of the data collection. Thus, the competence development course might not have been available since their start of employment, and they simply have not had the chance to attend. This could be problematic as the course is a main method to ensure implementation and engage the teachers (the key stakeholders) with the program (Skivington et al., 2021). The need for follow-up activities is also reported in studies on similar long-term implementation and scaling programs (Adamowitsch et al., 2017; Durlak & DuPre, 2008; Naylor et al., 2015). Still, the availability of the courses is not the only barrier, as PE teachers still need to find the time and motivation to participate which calls for engaged leaders prioritizing and require their PE staff to attend.

Strengths and Limitations

SP generates relevant health outcomes (Klakk et al., 2013, 2014; Moller et al., 2014; Rexen et al., 2015) and has been successfully implemented and maintained on OI-schools for nearly a decade (Nielsen, Skovgaard, et al., 2018). This study provides unique insight into a real-world program containing a considerable amount of PE in primary school and offers unique insights for scaling processes within municipal borders. However, we recognize that the study has a number of limitations.

The questionnaire used in the current study was only distributed once. This entails some limits on the possible interpretation of the data due to capturing only a single point in time. It would have been beneficial to have collected a series of questionnaires over a longer period of time or to have included additional items that could help assess fidelity measures within a certain duration. This could have provided some perspective on possible change in fidelity over time. Furthermore, the characterization of fidelity as either “high” or “low” is often based on somewhat individual assessments (Durlak & DuPre, 2008) and there is a need for consensus on the operationalization and methods used for assessing fidelity (Schaap et al., 2018). Thus, what is characterized as “high” fidelity in one study may be characterized as “mid-range” fidelity in another (Durlak & DuPre, 2008). Although we have presented the implementation data in Tables 2 to 4, there is still a need for a larger discussion on when fidelity levels can be characterized as “good enough.”

Although SP has been proven effective at OI-schools, there have not been any studies looking into the program’s ability to be effective in scaling schools. The present study adopts a focus on program fidelity, showing that SP is implementable at SC- and SS-schools with rather high levels of fidelity. Still, insights into the effectiveness of the program on scaling schools are needed to heighten our understanding of exactly how such expansions affect outcome. In addition, qualitative insight might also have nuanced the results of the current study. As an example, results show that only 57% to 75% teachers report that pupils are receiving the required 4.5 hours of weekly PE. School timetables, however, reveal that all schools in SP have added the 4.5 hours of weekly PE as a mandatory part of all pre-preparatory classes and placed these PE lessons across a minimum of 3 days. This discrepancy could be due to teachers not recognizing time for changing and showering as part of the PE lessons (Nielsen, Skovgaard, et al., 2018). Qualitative interviews with teachers and school heads on scaling schools would in general have added valuable insights into the challenges and motives when joining an ongoing school-based PA program.