Key Considerations and Recommended Strategies for Conducting a School-Based Longitudinal RE-AIM Evaluation: Insights From a 28-School Cluster Randomized Trial

Foster Stakeholder Relationships

Building and maintaining relationships with implementation stakeholders is critical to program evaluation (Kwan et al., 2019). School-based projects involve many stakeholders including school administrators (e.g., principals), staff (e.g., teachers), and program facilitators. Relationships with those stakeholders can help assess the RE-AIM domains, as fostering buy-in helps ensure that even if difficulties arise, partners are willing to work with the team to collect the necessary evaluation data (e.g., providing access to students, scheduling survey sessions, etc.; Foster-Fishman et al., 2001; Kwan et al., 2019). Assistance from stakeholders is particularly needed to answer the REACH-related research questions (see Table 1 for example questions).

School-based program evaluators should consider forming an “evaluation advisory board” and include school-based program evaluators that have been successful with other evaluations—perhaps even with the same school(s)—and representatives of other stakeholder groups (e.g., administrators/teachers from evaluation schools and program facilitators). This board could be consulted about planned or considered processes and may help avoid challenges and increase buy-in from other stakeholders in their same role (e.g., teachers reiterating the importance of the evaluation to other teachers) when new processes are implemented. We found principals were busy and difficult to reach; appropriate incentives may be needed to encourage their involvement as advisors. Teachers may appreciate receiving gift cards or vouchers for school supplies, considering limited resources available. An advisory board could inform decisions about such incentives.

We built relationships with principals early, and they were charged with relaying program evaluation information to teachers. Yet, frequently, teachers were unaware or misinformed about the project—at the beginning, as well as in the Maintenance phase when we collected follow-up data. When study-enrolled students moved to new classrooms in subsequent semesters and years and were sought to participate in longitudinal assessments, some teachers appeared unfamiliar with the evaluation, did not understand that students needed to leave their classroom to participate, and did not send their students to the assessment location at the specified times. We suggest helping school administrators disseminate evaluation information. For example, evaluation staff—perhaps in conjunction with the advisory board—could create pamphlets, flyers, or email messages that clearly communicate expectations and planned activities in a friendly manner at key times. Especially for follow-up assessments, we recommend that these materials: (a) provide a project overview to remind previously involved teachers and educate new or previously uninvolved teachers about the evaluation and (b) provide timely information about upcoming activities and their rationale. Administrators could choose whether to distribute these materials as-is, to modify them, or create their own.

Regarding Implementation, our program facilitators were often invested in the implementation of TOP as part of their job but were less invested in program evaluation. To assess implementation fidelity, facilitators were asked to submit a log (~10–15 minutes of work) for every class meeting to track the actual (vs. prescribed) implementation. Sometimes logs were submitted late or not at all. Logs submitted late often lacked details related to successes and challenges with implementing a particular lesson, which threatened to limit our fidelity assessment. We recommend discussions with potential implementation staff and their supervisors before hiring, if possible, to relay that an equally important part of their job is on-time and accurate completion of implementation logs.

Initially, we were concerned our team had not spent enough time building relationships with the facilitators and communicating the importance of the facilitator logs for this evaluation. To rectify this, at annual facilitator recertification trainings we presented materials to show what happened to their facilitation logs once they were sent. We also trained evaluation staff to provide meaningful feedback to facilitators on a regular basis. Yet we continued to experience low numbers and poor-quality logs from some facilitators. In extreme cases, facilitators’ supervisors (FLDOH county health department staff, not the evaluation team) were informed of deficiencies and corrective actions were promised. Nonetheless, problems persisted. County health department supervisors shared that some program facilitators who were inclined to submit no, late, or minimally informative implementation logs may have feared punitive consequences for implementation infidelity.

The importance of this example is twofold—first, we had to obtain the facilitators’ buy-in, so they understood how their efforts contributed to the evaluation. But in some cases, their understanding was not enough. We learned during end-of-evaluation focus groups that facilitators experienced additional barriers, such as time constraints related to their positions at county health departments (e.g., additional work responsibilities that limited the time they could put toward the evaluation). In hindsight, we recommend involving both the facilitators and their supervisors in the process of developing fidelity instruments and procedures, consistent with true partnerships. A discussion of how completing the implementation logs falls with the broader context of the facilitators’ jobs may result in processes that foster more timely log completion. Even if not all facilitators and supervisors are hired during the development of these procedures, having input from a subset of these groups may help to foster a more collaborative spirit. Later, when fidelity instruments and procedures are implemented, an early check-in with these groups—even via conference call or group video-conferencing—may help address what does not work well in practice. Also, although we did not try it, perhaps rewards or competitions tied to log completion could help ensure that implementation data are provided promptly and completely.

Balancing Project Management Across RE-AIM Dimensions

The importance of good project management for Reach, Adoption, and Implementation was particularly salient. Effectiveness took priority over other components and required large numbers of staff at certain key points (e.g., data collection, which relied on paper and pencil assessments in schools due to limited technology in the schools, processing surveys, conducting quality assurance checks, and completing data entry). Some REACH-related tasks (e.g., getting demographic data from the school to describe the population from which our participants were drawn) were not completed due to lack of time and human resources, although better project management may have prevented that. Also, at critical times, staff hired for qualitative work on Adoption needed to forego their qualitative tasks (e.g., interviewing principals about adoption) to assist with quantitative data collection assessing Effectiveness. If possible, depending on the size of the project and the availability of resources, we recommend having a designated staff person for each component of RE-AIM. Each “RE-AIM Champion” would have several responsibilities including identifying tasks that must be completed related to their assigned component and ensuring those tasks are completed on time.

As with any multiyear project, it is imperative to plan for staff turnover, which can impede progress, especially with the Effectiveness and Maintenance domains. We had staff depart without sharing resources and knowledge gained. We recommend using project management systems to assist the evaluation team with task management, information sharing, document/data storage, and creating a historical project timeline. Each team should examine the contemporary project management options in conjunction with their needs, preferences, and budgets to find systems that work for them. Evaluators might also consider cross-training staff to maintain institutional knowledge and reduce interruptions in workflow when turnover occurs.

We recommend considering whether school or evaluation staff will be available to measure organizational or individual Maintenance after the grant-funded portion of implementation is completed. Turnover may occur, limiting the support for longer term follow-up. If staff connected to the project won’t be available, alternative strategies should be considered. Similar challenges have been reported in the broader literature on program sustainability (Scheirer, 2013; Scheirer et al., 2008) and in RE-AIM evaluations (Kwan et al., 2019).

Optimize Timing

Optimizing timing evaluation activities is critical for completing a RE-AIM evaluation. School systems produce unique climates, with precise and systematic schedules. In our evaluation schools, system-mandated student testing was planned for up to 90 of 180 school days (Maxwell, 2016; Yi, 2015), limiting the time for Effectiveness- and Maintenance-related assessments. Baseline assessments were delayed by weeks because schools wanted to finalize students’ schedules before we began assessments. Similarly, end of school-year activities (e.g., testing) limited time available for immediate postprogram and 1-year follow-up assessments. Understanding these limitations in the planning stage is critical.

It can be difficult to optimize timing for Maintenance assessment and other long-term outcomes measures. In our evaluation, pregnancy (and pregnancy prevention) was a primary outcome measure. Even with a very large sample, the effects of the program on unintended and early pregnancy may not be seen immediately postprogram implementation, or at 1-year follow-up assessments meant to measure maintenance. Alternative outcomes (e.g., condom use, delayed sexual initiation) were reported, but these outcomes did not reflect the intended program outcome (reducing unintended pregnancy). Careful consideration of the personnel, partner, and funding resources should be undertaken regarding how to reasonably assess maintenance.

Optimizing timing is also important for assessment of Adoption. When measuring adoption by interviewing school administrators, it is important to consider that some times—such as the beginning and end of the school year—are often exceptionally busy. Allow for flexibility of scheduling during the school year or try to schedule summer appointments with administrators.

Plan Ahead

Planning ahead is critical to the measurement of Reach, Effectiveness, and Maintenance. For example, we ran into challenges in securing IRB approval. Since the program was being implemented by the FLDOH, approval from the FLDOH IRB was required prior to funding. This led to a stepwise approach to instrument adaptation, protocols, and subsequent IRB approval. This is not atypical—rarely are all instruments and protocols finalized at the onset of a large evaluation. To begin collecting data as quickly as possible, we took a piecemeal approach to obtain approval for IRB amendments. We focused on the most pressing need at any given time, which did help us progress. Yet this was inefficient. We spent more time overall than if we had finalized all measures initially and submitted a single amendment. Evaluators should plan their budgets to devote substantial staff time early in the study to finalize instruments and reduce the number of subsequent IRB amendments making it easier to maintain the evaluation timeline.

We recommend considering what Adoption information will be important to collect for a RE-AIM evaluation and then developing or identifying instruments and databases for collecting this information. An example is related to identifying factors that affect administrators’ adoption decision (Table 1); to get needed data, develop protocols for documenting those factors before reaching out to potential partners—often before the grant application is completed.

As with adoption, it is important to have a clear plan and appropriate timeline in place to assess Implementation barriers and fidelity. Our evaluation staff regularly reviewed implementation data (i.e., attendance and facilitator logs) for quality and provided feedback to facilitators in weekly emails as needed. For facilitators with ongoing challenges, facilitators’ supervisors (separate from evaluation team members) were alerted. A good plan should include monitoring fidelity-specific metrics and clear steps for addressing implementation issues.

Anticipate Structural Barriers

Issues beyond control of the investigators can hinder the completion of evaluation-related activities. Opportunities to collect certain types of data may be lost. For example, to assess the population of students in a specific grade level at specific schools (potentially as the denominator in the Reach analysis), demographic data may be publicly available for some states or other entities and not others—determining this in advance is critical. We planned to pay a stipend to school personnel to help collect Reach-related data. However, policies prohibit contracts with individual schools, meaning we were unable to compensate personnel to assist us. We had no formally assigned Reach liaison at each school; consequently, some requested information was not received. We recommend having early conversations about how structural barriers may impede collection of Reach-related data (Kwan et al., 2019). Additionally, alternative measures or approaches may be needed to pragmatically determine the Reach of an intervention (Glasgow & Estabrooks, 2018; Kessler et al., 2013). Making these determinations early on will reduce unexpected gaps in Reach-related data.

Structural barriers to measuring Effectiveness may depend on the political environment, school system size, and state- and school-level policies and procedures. As noted, FLDOH and the Department of Education (Figure 1) were involved in our evaluation. The timing of testing, class management concerns, and program activities required understanding of and compliance with both bureaucratic systems, including their specific cultures, rules, and processes. Due to FLDOH rules, we were unable to provide incentives to youth—which may have affected participation in immediate and long-term follow-up assessments. We recommend critically assessing and understanding these structures prior to evaluation activities to determine clear avenues through which the evaluation plan can be successful. An advisory board of key stakeholders, recommended under Foster Stakeholder Relationships, would have given us insight regarding the cultures of each bureaucratic system and how to work quickly within them.

Structural barriers may also affect the assessment of Implementation. For an evaluation spanning a large geographic region, hiring evaluation staff at the county/regional levels might be preferable to maintaining a larger evaluation staff at the evaluation home base and having them travel broadly. For us, the high costs of travel arrangements and the time required in the field made overseeing implementation quality a challenge. Furthermore, if local staff are collecting data for assessing implementation (i.e., facilitators or their supervisors are collecting some of the data), consider potential barriers to receiving those data. If resources allow, we recommend utilizing web-based survey software and/or tablets to collect data, setting up an online portal for data submission, or arranging for a network of shared folders to facilitate data submission.

Be Flexible

As with all research, flexibility is a key to success (Chen, 2014; Kwan et al., 2019). Planned components may need to change in accordance with school system or community needs. We have outlined numerous cases in which flexibility is important throughout a school-based RE-AIM evaluation. Evaluation staff should be flexible. Funders and other partners may be less flexible; thus, some aspects of the evaluation may be nonnegotiable. We recommend relying on preestablished strong stakeholder relationships to negotiate processes that can be accepted by all. Keep in mind that changes to schedules, procedures, and even project goals may occur unexpectedly; being flexible will allow for a better evaluation.