Protocol Development,Treatment Fidelity,Adherence to Treatment,and Quality Control

Preparatory Development and Design Activities

Development of the research protocol is guided by institution-specific guidelines and scientific, ethical, and federal requirements (Chow & Liu, 2004; Rozovsky & Adams, 2003). Within the health and rehabilitation professions, researchers must balance the need for experimental control with the realities of clinical practice (Fetter et al., 1989). This challenge makes clinical research more difficult and requires that researchers take into account additional considerations during the design and planning stages of the research process.

Chow and Liu (2004) suggested preparation of the research protocol to address several key points; these activities are often performed by the principal investigator (PI). The PI should begin with careful assembly of a multidisciplinary research team. In doing so, the PI considers the following: Who should be involved? What will their responsibilities be in preparing and submitting the protocol? What will be required from each member in terms of time and effort? The process of assembling a wide range of content experts is necessary so that the research team is able to precisely define the target population and objectives for the protocol (Chow & Liu, 2004). In addition, early consultation with a statistician helps ensure that the research team selects an appropriate research design and procedures. These early steps help frame the research question.

Gallin (2002) suggested using visuals to frame the research question in a simple way. In addition to identifying the intended outcomes of the study, Gallin noted, the PI needs to ask the following: “How will you determine if your protocol will be a success? Will your protocol provide ‘proof of concept’? . . . an increased level of efficacy?” (p. 443). Investigators should also plan for how the protocol will handle adverse events and consider any regulatory or ethical issues that may arise. Addressing these items early in the predevelopment and planning stage of the research process will help the research team avoid common pitfalls (Chow & Liu, 2004).

Title, Abstract, and Introduction

Although institution-specific guidelines for protocol development may vary, most institutions require that general information be presented through some form of face sheet. This general information should include the title and identifying information for all study personnel and study locations (Chow & Liu, 2004; Rozovsky & Adams, 2003). The title of the protocol is a key element in communicating the purpose, design, and results of a research project. When creating a title, it is recommended that researchers use descriptive language that describes the population of interest, the treatment design, and how participants were allocated (Des Jarlais et al., 2004; Gutman & Murphy, 2012).

The reader of a research protocol abstract “should come away from the document with a clear understanding of why the study is being done, how it is being done, and the treatment outcome hoped for” (Gallin, 2002, p. 445). The introduction presents the scientific background and the significance of the identified clinical problem (Des Jarlais et al., 2004; Gutman & Murphy, 2012) and includes a summary of previous research, the results of previous studies or pilot testing, definitions of key constructs, discussion of potential risks, and identification of relevant gaps in the literature (International Conference on Harmonization [ICH], 1996). The introduction also includes a purpose statement, research questions, and the implications of the research for clinical practice (Gutman & Murphy, 2012; ICH, 1996).

Trial Design

Development of a rigorous yet feasible research protocol is largely dependent on selection of an appropriate design. The ICH (1996) E6 guidelines for trial design emphasize that “the scientific integrity of the trial and the credibility of the data from the trial depend substantially on the trial design” (p. 31). The research protocol should contain both written and schematic representations of the trial design (Gutman & Murphy, 2012; ICH, 1996). The written description of the trial design (e.g., cross-sectional, randomized controlled, longitudinal) should provide the research staff and reader with an appropriate amount of detail for understanding the procedures and stages of the design (ICH, 1996). An intention-to-treat analysis should be used and reported when appropriate (Gutman & Murphy, 2012).

In some cases, changes to the research protocol may be necessary. In preparation for this contingency, the research team should consider foreseeable deviations from the protocol and have a plan to address and report them (Chow & Liu, 2004; Moher et al., 2010). The TREND checklist does not contain an item for trial design per se but rather “emphasizes description of the intervention, including the theoretical base; description of the comparison condition; full reporting of outcomes; and inclusion of information related to the design needed to assess possible biases in the outcome data” (Des Jarlais et al., 2004, p. 362).

Participants

Identification of the target population is central to development of the research protocol (Gallin, 2002). Collaboration with content experts enables preparation of precise inclusion and exclusion criteria. Both the CONSORT and the TREND checklists emphasize that a complete description of the settings and locations where data are to be collected is helpful when reporting results (Des Jarlais et al., 2004; Schulz et al., 2010). Toward this end, we recommend that investigators take particular care during protocol development to describe the environments in which their study will occur. Additionally, it is helpful to plan for and describe the duration of patient participation, stopping rules, and withdrawal criteria (ICH, 1996). Although relevant to protocol development, specific strategies related to recruitment and retention of participants are discussed elsewhere in this issue (Page & Persch, 2013). Investigators should also describe the benefits of participation in developing these materials (Rozovsky & Adams, 2003).

Interventions

Within the health and rehabilitation sciences, the efficacy of interventions is often a focus of research. Descriptions of the intervention in the research protocol and in published manuscripts allows for replication (Moher et al., 2010). When designing a trial, the investigator must develop a description of the treatment, dosing schedules, methods of administration, and processes for assessing participant compliance (Des Jarlais et al., 2004; ICH, 1996). Implementation-related processes and strategies are discussed later in this article to provide researchers with additional resources to use when conducting trials. The TREND checklist provides a good rubric for breaking down the description of the intervention (Des Jarlais et al., 2004). The description should include the content of the intervention, the method and unit of delivery, the setting and duration of the treatment, and the personnel who will administer the treatment (Des Jarlais et al., 2004). Additionally, investigators may describe how the intervention was manualized or under what circumstances the intervention may be individualized (Gutman & Murphy, 2012). Consideration of these elements helps strengthen the internal validity of the protocol (Portney & Watkins, 2000).

Outcomes

Early collaboration with content experts helps identify primary and secondary endpoints and appropriate outcome measures. The intended purpose, population, and psychometric properties should be described for each outcome measure. Investigators should identify the level and type of data produced by outcome measures and cite studies reporting their reliability and validity (Gutman & Murphy, 2012). Additionally, the research protocol should describe the methods and schedule for administration of outcome measures, analysis of participant safety, arrangements for participant follow-up, and reporting of adverse events (ICH, 1996).

Sample Size

Justification of sample size and completion of a power analysis must be completed a priori. The CONSORT 2010 elaboration document states that investigators must “balance between medical and statistical considerations” during this process (Moher et al., 2010, p. 8). Calculation of sample size is critical during the planning stages to ensure that the study has enough power to detect changes in performance or differences between the tested groups. Power is defined as the probability of detecting a change or difference when one actually exists (Howell, 2010; Meyers, Gamst, & Guarino, 2006; Portney & Watkins, 2000). Power is affected by certain elements of the research design, some that are under the control of the researcher and others that are not. The easiest way to increase power is to increase α. Increasing α results in a decrease in the critical point and an increased likelihood of rejecting the null hypothesis. Making this type of change is not acceptable once the protocol has been initiated.

Increasing the number (N) of participants in the study is the easiest and most acceptable way to increase power. As N increases, the variance of the sampling distribution of the mean decreases, resulting in increased power. It is also possible, hypothetically, to increase power by decreasing σ² or increasing the effect size (d). These variables are typically not possible to manipulate and come with negative side effects (Howell, 2010). Calculating the required N is a relatively straightforward process once the investigator has estimated d, chosen α, and determined the preferred power (Howell, 2010). The TREND and CONSORT statements suggest reporting this process and the results of any interim analyses (Des Jarlais et al., 2004; Schulz et al., 2010).

Randomization

Many processes are available for assigning participants to intervention conditions. The requirements of such processes reflect the complexity of research design and the need to control for bias (Portney & Watkins, 2000). As related to the RCT, the CONSORT statement defines methods of random assignment appropriate for studies of intervention effectiveness (Moher et al., 2010). Use of randomization strategies such as blocking, stratification, and minimization “is an essential feature of experimental research, providing the greatest confidence that no systematic bias exists with respect to a group’s collective attributes that might differentially affect the dependent variable” (Portney & Watkins, 2000, p. 155). In a complementary way, the TREND checklist suggests use of eligibility criteria to minimize threats of bias (Des Jarlais et al., 2004). Early incorporation of a rigorous allocation scheme or inclusion process during protocol development contributes to limiting bias and enhances the generalizability of the findings (Des Jarlais et al., 2004; Schulz et al., 2010).

Blinding

Development of blinding procedures helps limit bias, increases the internal validity of the protocol, and enhances the generalizability of findings. The intricacies of and strategies for achieving blinding are reported elsewhere in this special issue (Page & Persch, 2013).

Statistical Methods

The ICH (1996) E6 guidelines provide detailed guidance for preparation of statistical methods. Specific statistical procedures should be developed and described in the research protocol document (ICH, 1996). Chow and Liu (2004) suggested translating all study objectives into discrete statistical hypotheses. A clear description of the data to be analyzed and plans for management and analysis is helpful in this preparation (Chow & Liu, 2004). Investigators need to determine whether interim analyses are required and, if so, describe the timing and procedures for such analyses (Chow & Liu, 2004; ICH, 1996). Procedures for early termination of the study on the basis of interim analyses are needed (Chow & Liu, 2004; ICH, 1996). The protocol should also detail the procedures for reporting deviations from the statistical plan resulting from missing or corrupted data (Des Jarlais et al., 2004; ICH, 1996).

Fidelity

Although often confused with adherence, which is concerned with participants’ behaviors (and is discussed later in this article), fidelity refers to the extent to which the study team complies with the study protocol. Occupational therapy clinical trial investigators should concern themselves with facets of treatment fidelity related to study design and the training of personnel.

Fidelity practices related to study design help investigators discern whether the study will adequately achieve the aims and test the hypotheses that have been set forth. Thus, when designing the study, the investigative team must make honest (and sometimes painful) assessments about the appropriateness of the study design. For example, are the study criteria and sample to be enrolled appropriate? Do the enrolled study groups approximate the target population such that the results will be generalizable? Is the study design matched to the phase of the research and to the primary study objective? Will the outcome measures capture the constructs of interest? In behavioral trials that are typical of occupational therapy, researchers may also want to consider what frames of reference are at play and whether the outcome measures and therapeutic procedures to be used are congruent with this frame of reference. Investigators are cautioned not to confuse institutional review board approval with fidelity as it relates to the design of clinical trials.

The manual of procedures (MOP; described later in this article) is a carefully constructed book that details the operating procedures for the study and procedures for training personnel in the administration of outcome measures and interventions. For example, our MOPs detail the ways in which each assessment will be administered, including the point at which it is given, by whom, in what environment, and using which equipment. With regard to equipment, we even define the attributes of the chairs and tables at which participants will sit and the distance that participants will be positioned from various equipment used during testing. Fidelity in training refers to the extent to which the outcome measures and treatment are administered in accord with the MOP. Thus, a team that has multiple protocol violations would be said to have low training fidelity, which would increase variability with which the protocol is administered. This variability is likely to artificially increase or diminish the results that are obtained, which can cause Type I or II errors to emerge.

To ensure treatment fidelity, our study coordinator oversees regularly scheduled checks of both our outcome assessors and our intervention therapists. For example, when a client who is not enrolled in the study is brought in to the laboratory, we ask the therapists to administer the measures or aspects of the intervention as they normally would and videotape them. We then review the videos to check for consistency with the protocol, and the therapists make modifications as needed. Occasionally, our treatment therapists meet and cotreat a participant, compare notes, check treatment fidelity, and fine-tune the protocol. The study coordinator is usually present to record any important points that come up and to ensure that any suggested changes to the protocol are nimbly implemented.

Adherence to Treatment

Adherence to treatment refers to the extent to which a participant’s behaviors comply with medical or health advice. For example, in the case of occupational therapy clinical practice, adherence may be exemplified when a client fully performs a suggested home exercise regimen. By comparison, a participant who follows the program that his or her randomization or grouping mandates would be considered to be adhering in an occupational therapy clinical trial. This attribute is fundamental to discerning the efficacy of the investigational approach because nonadherence may cause underestimation of the treatment effect or failure to detect a treatment effect that actually exists (i.e., Type II errors). In some cases, participant nonadherence may cause investigators to underestimate the safety of an intervention, which can cause adverse consequences in later trials or in the treatment’s subsequent clinical use.

Participants may exhibit two primary categories of treatment nonadherence: (1) underexposure to the treatment and (2) overexposure to the treatment. The former may occur because participants are unwilling to accept the group they are assigned to or do not fully adhere to the assignment because of a variety of factors (e.g., failure to see benefit; barriers to use, such as a lack of transportation; malaise). Overexposure is most commonly displayed when a mistake occurs in treatment administration by the person in charge of treatment (e.g., administration of longer duration or greater dosage than prescribed) or when the participant is administered similar rehabilitative therapy outside the confines of the study.

Many causes of nonadherence can be prevented before enrollment begins. For example, the study should be designed in such a way that it is easy for participants to adhere. Study designs that minimize or reduce the number of study visits are likely to increase adherence. In our laboratory, we combine the screening and pretesting visits when participants are capable of tolerating the total duration of the visit.

The investigative team also should consider whether the content of the study visits is conducive to participant adherence. For instance, some colleagues on our medical campus require participants to attend multiple testing visits that are located at different sites on the campus. This practice can be confusing, tiring, and frustrating for many participants and their care partners, particularly if they are not escorted by a research team member. The proliferation of academic medical centers that have access to a clinical translational science center can facilitate “one-stop shopping” in which testing, diet or medication regimens, and other study management features are available at one location. PIs can facilitate ease of access by ensuring that they have a dedicated single space in which to conduct their research. A study design that restricts the number of secondary and tertiary outcomes while not requiring excessive participant travel will also reduce participant burden and increase adherence.

In considering the content of the study visits, investigators should also examine the features of the intervention itself and the requirements that participants must meet to be eligible for the intervention. With regard to the former, investigators may wish to consider whether the intervention can be shortened in duration. For example, constraint-induced movement therapy (CIMT) is a recognized rehabilitative therapy targeting the hemiparetic upper extremity of stroke survivors (and, more recently, of people with cerebral palsy) that requires 6-hr upper-extremity therapy sessions. Data from a Phase 3 CIMT trial, however, showed that most patients can tolerate only about 3.9 hr of the 6-hr regimen before they have to stop because of fatigue (Kaplon, Prettyman, Kushi, & Winstein, 2007). Consequently, our laboratory was the first to shorten the CIMT parameters with a regimen titled “modified constraint-induced therapy” (mCIT; Page, Sisto, Levine, Johnston, & Hughes, 2001). mCIT requires clients to attend therapy sessions for just 30 min per day and continues to be used on an outpatient basis with high compliance (>90% in some clinics) and efficacy (Page, Levine, Leonard, Szaflarski, & Kissela, 2008). In other words, by simply shortening the duration of this well-recognized therapy, we were able to preserve its features and efficacy but also increase adherence.

With regard to eligibility requirements to participate in the intervention, an important factor in adherence is selection of participants. For instance, will the study criteria that have been established result in a participant sample that is generalizable and that will also be capable of participating in the intervention? Are the characteristics of the intervention—or even the screening or testing procedures—particularly difficult for the participants being sampled? Some research groups apply boilerplate exclusion criteria to many of their trials, such as excluding people who are addicted to certain drugs or alcohol, who live too far away, or who have particular concomitant diseases or medication regimens. Other groups pilot test a new intervention on participants from whom they expect high compliance and a high likelihood of change. This practice allows the team to maximize the likelihood of being able to determine the strength of the treatment effect, confirm the safety and feasibility of the intervention, and ensure selection of the most sensitive outcome measures. After gathering this information, the researchers then attempt the intervention on more impaired populations.

Finally, we have found that advocacy of the intervention by multiple members of the care team constitutes a feasible, effective approach to maximizing adherence. To accomplish this, we inform the participant’s physician and other care team members of his or her participation in the clinical trial and ask the physician’s office, rehabilitative team members, and other clinicians to provide instructions and reinforcement to the participant as appropriate. This approach allows the entire team to stay informed about the study and to feel a continued sense of investment in the participant’s care. It also provides the participant with multiple points of encouragement and information to facilitate trial adherence.

Additionally, a behavior contract may be used that requires the participant to record behaviors normally performed at home and to agree with the therapist about which behaviors the participant will carry out during the study and in what way. For instance, in our work on mCIT (Page et al., 2008), we used behavioral contracts to identify the particular movements for which each participant would use the affected upper extremity. During the course of our trials, we reviewed the contract at selected appointments to remind the participant of the study’s requirements. In several areas of behavioral research, a behavioral contract has been reported to increase protocol compliance and participant retention (Carroll, DiMeglio, Stein, & Marrero, 2011; Hartz, Brennan, Aulakh, & Estrin, 2010; Liberman & Rotarius, 1999; Solanto, Jacobson, Heller, Golden, & Hertz, 1994). This approach also meshes well with occupational therapy’s emphasis on client-centered care, because the client takes on an important role in identifying targets for therapeutic intervention for inclusion in the contract.

Quality Control

As is the case with fidelity and adherence, quality control—ensuring that the study adheres to the highest standards—is a multidimensional, shared responsibility among the trial’s stakeholders. Quality control is a process, and an extensive description is beyond the scope of this article; however, good clinical practices (GCPs; International Conference on Harmonization, 1996) and, in particular, the use of an MOP can guide efforts to ensure high study quality. Occupational therapists are challenged to apply these standards within our own research.

GCPs —guidelines for development and conduct of clinical trials—were originally developed to assist with clinical trials of medications but have increasingly been applied to behavioral trials and, in general, to trials sponsored by federal agencies and some private companies. GCPs have been described in the regulations governing research into new drugs (Investigational New Drug Application, 2006) and by the ICH (1996). Their use “provides assurance that the data and reported results are credible and accurate, and that the rights, integrity, and confidentiality of trial subjects are protected” (ICH, 1996, p. 4). Moreover, because GCP standards are regularly evaluated and updated, their use ensures that trials continue to measure up to the most current standards of rigor and participant protection. In the case of occupational therapy, their widespread implementation would provide the field with concrete guidelines that would reduce variability and interpretation, thus improving the quality and credibility of our trials and, ultimately, the field. These are worthy aspirations to which occupational therapy trials should aspire. We use GCPs to guide all aspects of our clinical trial protocols but especially to form our MOP.

The MOP for our studies includes the following elements:

Although the implementation of the MOP may vary from team to team, our study coordinator, who has academic and on-the-job training and coursework in GCPs, is usually the one who maintains and updates the MOP. Unlike some other teams, however, we feel that the PI must also be aware of all study activities and have a basic knowledge of the regulations and practices governing good clinical trial administration. Consequently, we encourage PIs of studies conducted in our laboratory to have the same training as the trial coordinators, be familiar with all aspects of the protocol, have regular meetings to facilitate communication about participants, and perform checks to ensure that the MOP is being followed. Many medical centers and institutions make available examples of MOPs, particularly in departments that are actively conducting clinical trials and through the clinical translational science center.