Challenges of Applying a Comprehensive Model of Intervention Fidelity

Intervention Design

The fidelity strategies in the design category include addressing specific aspects of the contact sessions in each condition, and other factors that must be defined to evaluate and replicate a study. For interventions using ICT, it is necessary to describe the aspects of the contact sessions in each condition. An objective method of tracking the intervention dose in each condition is also needed. In the example case, the number of contact sessions during the 6-week Internet intervention was tracked electronically. Participants accessed the Internet intervention on their own time, progressed at their own pace, and reviewed and repeated the content as necessary. Thus, the length of the contact sessions was determined by the participant, not specified by the provider. The number of contacts recommended was to log on to the study website daily, Monday through Friday. New information was provided on the study website each day of the week to encourage daily activity. The intervention allowed more contact between the intervention and individual participant than would be possible in person, so an objective and automated method of tracking the intervention dose delivered was essential. Other aspects of the intervention content are detailed in Table 1. The nurse practitioner (NP) provided standardized feedback about percentage adherence to the National Cholesterol Education Program (NCEP, 2002) physical activity goals and positive reinforcement for goal setting as well as individualized feedback about the appropriateness of the physical activity goals. Reminders to participate in the electronic discussion board were also included each week.

The credentials of the providers involved must be detailed to demonstrate the level of expertise needed to replicate the intervention. In this case, the usual care condition was administered in person to both the intervention and usual care control groups, and thus, a description of provider credentials was relevant (see Table 1). The Internet intervention was delivered electronically, and thus, credentials were not relevant for delivery of the intervention. Credentials were relevant for the content expert developing the intervention and providing electronic feedback by email and on the discussion board (although electronic communication was standardized). Replication of an intervention using ICT may not require special expertise as long as the intervention is standardized. Describing the theoretical model and/or clinical guidelines on which an intervention is based is necessary to evaluate a study. The theoretical model and guidelines used for both the Internet intervention and the usual care condition are described in Table 1. A challenge for interventions using ICT is that theoretical constructs must be operationalized optimally for electronic delivery, as direct transfer of an in-person intervention to the Internet may not be as effective. For example, in an intervention using ICT, vicarious experience to build self-efficacy may best be provided by a video on the website showing participating in moderate intensity physical activity rather than observing others in person at a gym. Usability testing is recommended to determine if the theoretical constructs are operationalized optimally for electronic delivery.

An additional design strategy essential to the fidelity of Internet interventions includes providing learning objectives and competencies specifying what participants are expected to accomplish during the intervention. Because participants access Internet interventions independently, attention to the objectives and competencies is essential and must be visible to participants each time they access the study website. Standardized feedback must also be provided in a timely manner if objectives or competencies are not met.

Training Providers

This category involves describing the standardized protocol for interaction with participants. Provider training is essential in interventions involving participant and provider interaction to reduce the random and unintended variability that can threaten intervention fidelity. In contrast, the Internet intervention was delivered electronically, thus eliminating the need to train providers to deliver the intervention. However, a protocol was developed to ensure that standardized feedback was provided by email messages and on the electronic discussion board. Furthermore, adherence to the protocol can be routinely assessed by review of the archives. Providers must also be trained in other aspects of the study protocol, such as how to use the technology, and how to access and navigate the study website.

Delivery of the Intervention

Intervention delivery focuses on the provider to ensure that the content and the dose of the intervention are delivered as intended. In Internet interventions, it can be difficult to ensure participant access to the intervention content as intended. See Table 1 for details on how access was ensured in the physical activity intervention. Objective tracking of participant activity on the website is essential, including all linked pages and websites explored, and all supplemental components used. For simplicity, the linked websites explored by participants in the Internet intervention were tracked by asking participants if the site was helpful by selecting “yes” or “no” from a dropdown menu. Data recorded on the website was downloaded into Microsoft Access 2000 file format. The data in the tables contained participant login time, dates, physical activity goals, self-reported minutes of moderate or greater physical activity as well as participants’ yes or no responses about the helpfulness of the linked sites visited. The BlackBoard Academic Suite™ provided the platform for the electronic discussion board. All discussion threads posted by the NP, participant’s questions and comments, and NP feedback were retrievable in the BlackBoard archives for assessment. Provider adherence to the intervention plan and competence in delivery of an in-person intervention is essential, but this is not a major challenge in Internet interventions that are automated and use standardized feedback. The comprehensive fidelity model includes addressing the nonspecific intervention effects resulting from provider interaction with participants. To address nonspecific effects in the Internet intervention, participants did not receive interpersonal contact with the NP during the intervention period. All standardized messages were received electronically by participants. Participants were provided the credentials of the NP provider with whom they were interacting by email.

Finally, an intervention manual is typically developed for use by providers. The main purpose of the manual for the Internet intervention was a guide for development of the intervention website. In this case, the manual was provided to the website developer, who developed the intervention algorithms and published the program to the web. The intervention manual included the essential intervention content and skills needed by participants to meet the physical activity guidelines along with all of the hyperlinks. Thus, the website development was based on the intervention manual developed by the researcher.

Receipt of the Intervention

Receipt of the intervention focuses on subject behavior during the intervention period to ensure that the intervention was received and understood. Actual receipt of the intervention is difficult to assess and requires careful consideration. Receipt involves assessing participant comprehension of the intervention and providing strategies to improve comprehension and skill performance. When developing a plan to ensure receipt, the distinction must be made between strategies to ensure understanding and comprehension. In the Internet intervention, knowledge or understanding of the intervention content was easily assessed through administration of a quiz on the website at Week 3 and Week 5 of the 6-week intervention. Understanding was improved by providing the correct answers with explanations electronically in real time. Tests of comprehension relate to the participant’s ability to apply the intervention skills, along with standardized electronic feedback, to improve the performance of those skills. With ICT, there may be limited opportunity to directly observe participant–provider interactions or to observe participants performing a specific behavior during the intervention period. Indirect evidence of the participant’s comprehension by the ability to perform skills is necessary, such as a record of the expected behavior documented on the website (see Table 1) or remote real-time monitoring of vital signs, as described by Nangalia, Prytherch, and Smith (2010). Use of a webcam is another strategy that can be used to provide objective evidence of comprehension. Receipt of the intervention also includes strategies to improve performance beyond that included in the intervention. Receipt can be promoted by providing feedback using standardized electronic messages (to avoid nonspecific effects that can result from aspects of an intervention and it’s delivery that are unintended).

Enactment of Intervention Skills

The strategies to enhance enactment of intervention skills focus on participant performance of these skills in daily life. Ensuring enactment of ICT interventions may be more difficult when the intervention is delivered entirely in settings in which the skills may be applied, such as the home or work environment. Enactment also involves addressing performance of intervention skills in new settings or circumstances in which intervention might be applied. Monitoring of indirect electronic evidence of skill performance and providing feedback using standardized electronic messages is required to ensure enactment and to improve performance of intervention skills delivered by ICT. In the Internet intervention, the electronic discussion board allowed participants to post questions for the NP about implementing physical activity skills in various settings, such as when traveling, when the weather was bad, or when experiencing stress or depression. The NP anticipated participants’ questions and comments, and provided standardized feedback to overcome barriers to physical activity participation and to recover from relapse.

Evaluation of Fidelity

The final challenge of implementing a comprehensive fidelity plan involves evaluating fidelity for reporting. In the literature, behavioral interventions generally include a discussion of implementation of fidelity strategies, involving evaluation of provider’s adherence to the protocol and use of a treatment manual (Borrelli et al., 2005; Moncher & Prinz, 1991). The participant behaviors of receipt and enactment of intervention skills, however, are not always discussed. A strategy to assess if participants received an intervention as intended with particular relevance for interventions delivered using ICT is the qualitative manipulation check described by Donovan, Kwekkeboom, Rosenzweig, and Ward (2009). This involves evaluating transcripts from electronic discussions for themes reflecting study participation. Not only the finding of nonspecific factors but also the absence of specific factors in the postings identified by manipulation check indicates areas for further investigation. For example, the Internet physical activity intervention was based on Social Cognitive Theory (Bandura, 1997), with online activities to promote self-efficacy by mastery experience, vicarious experience, persuasion, and reinterpretation of physiological signs and symptoms. Specific factors indicating receipt of the intervention as intended might include participant postings on the electronic discussion board detailing how they could relate to the online testimonials of how others with the metabolic syndrome overcame difficulties and problems, solved relapses, and achieved physical activity goals (indicating receipt of vicarious experience); and/or relate to the discussion about dealing with minor aches and pains and reducing stress and anxiety levels related to initiating or increasing physical activity (indicating reinterpretation of physiologic signs and symptoms). Nonspecific factors might include participant postings of how the discussion board provided social support, which was not intended to be part of the Internet physical activity intervention content but indicates a theme for assessment in a future study.

Another challenge in the evaluation of fidelity is the lack of specific evaluation criteria. High levels of fidelity were defined by Borrelli et al. (2005) as those studies that had .80 or greater proportion adherence across all strategies. In the Internet physical activity intervention, the proportion of adherence to the fidelity strategies was calculated for each component of the comprehensive model by summing the number of strategies coded as present and dividing by the total number of strategies (see Table 2). Thus, the Internet physical activity intervention addressed most of all the strategies considered relevant to the study design. Using these criteria, fidelity was high. It may be most informative for researchers to identify the components of the comprehensive fidelity plan related to intervention outcomes to provide evidence for the levels of fidelity needed to observe intervention effects.

Evaluating enactment of intervention skills can be challenging. Enactment itself is not considered a study outcome but focuses on the skills required to achieve study outcomes (Resnick et al., 2009). Enactment can be quantified by random sampling of records and evaluation using descriptive statistics. For example, the levels of physical activity over the course of the Internet intervention can be compared with the level that the participants were expected to have completed based on the NCEP guidelines. Enactment of the intervention can be calculated to determine percentage achievement of the study goals.

Fidelity related to receipt of the intervention has been analyzed by the input of the dose of the intervention that was actually received as an independent variable in regression analyses (Santacroce, Macarelli, & Gray, 2004). This method, however, may be misleading, as the intervention dose delivered and the dose received by participants may be quite different. The qualitative manipulation check (Donovan et al., 2009) can be used to assess participant understanding, along with objective evidence of performance of intervention skills to assess comprehension. Researchers may encounter a final challenge in reporting intervention fidelity. Editorial issues or space limitations beyond the researcher’s control may limit reporting intervention fidelity along with the effects or feasibility. A separate article may be needed to report fidelity of a complex behavioral intervention, including all aspects of the comprehensive fidelity model.

Future Research

Fidelity is an essential part of conducting health behavior intervention research. A comprehensive model is necessary for incorporating essential aspects of fidelity in the design, implementation, evaluation, and reporting of complex health behavior interventions. The comprehensive plan described in this article is particularly relevant to ensure the fidelity of interventions delivered using ICT, such as the Internet. Interventions using technology are unique and offer some advantages compared with traditional in-person interventions, primarily consistency of implementation and ease of replication. These interventions also present unique challenges for ensuring fidelity. The greatest threat to the fidelity of Internet interventions shifts from implementing the intervention to participant receipt and enactment of intervention skills. Monitoring participant use of the intervention and providing timely, standardized feedback is necessary to ensure participant receipt and enactment. The strategies for evaluating fidelity are evolving, and more health behavior interventions are being conducted using technology. Future research is needed that focuses on the application and testing of novel methods for assessing intervention fidelity. Identifying the components of a comprehensive fidelity plan related to intervention outcomes can provide evidence for the levels of fidelity needed to observe intervention effects. Ultimately, reporting fidelity information is critical for translating the intervention to various clinical practice settings and populations.