Abstract
Many factors can affect the successful implementation and validity of intervention studies. A primary purpose of feasibility and pilot studies is to assess the potential for successful implementation of the proposed main intervention studies and to reduce threats to the validity of these studies. This article describes a typology to guide the aims of feasibility and pilot studies designed to support the development of randomized controlled trials and provides an example of the studies underlying the development of one rehabilitation trial. The purpose of most feasibility and pilot studies should be to describe information and evidence related to the successful implementation and validity of a planned main trial. Null hypothesis significance testing is not appropriate for these studies unless the sample size is properly powered. The primary tests of the intervention effectiveness hypotheses should occur in the main study, not in the studies that are serving as feasibility or pilot studies.
Keywords
If you have built castles in the air, your work need not be lost; that is where they should be. Now put foundations under them. (Thoreau, 1854/2009, para. 10)
Planning intervention research takes creativity and innovation, a bit like building castles in the air. Feasibility studies are studies designed to build the foundation for the planned intervention study. For several reasons, it is challenging to define feasibility studies for occupational therapy research. Although many types of feasibility studies could be applicable to intervention research in occupational therapy, no typology has been developed specifically for the research done in our field. Published feasibility study typologies are rare and typically focus on preparing for drug trials in which a single “active” ingredient is being tested, such as a chemical that is posited to be the causal effect of intervention outcomes.
As we know, occupational therapy is not implemented with the assumption of a single active factor inducing change in our clients. Our intervention outcomes are derived from “blended” active agents. Person, environment, and occupation factors compose an interactional network of potential agents that create the quality of occupational performance and health outcomes (Law et al., 1996). Moreover, our outcomes are often theoretical constructs (e.g., occupational performance, quality of life) rather than direct observables (e.g., cure of disease or a change in bodily function), and they generally are measured along a conceptual continuum that is not a true count (e.g., cell counts, tumor size). Our interventions in actual practice are client centered and highly individualized as opposed to highly standardized.
Feasibility studies in occupational therapy must build the foundations that support intervention trials that involve a blending of active agents; a theoretical perspective that reflects an understanding of occupational performance and outcomes as being at the intersection of person, environment, and occupation; a measurement paradigm based on constructs and continua; and client-centered, individualized intervention. Feasibility studies are rare in our field. Many, if not most, occupational therapy pilot studies do not fulfill the definition of feasibility study according to the current paradigms emerging in the methodological literature on conducting intervention trials and as described in this article. Yet, feasibility studies are critical to the successful implementation of randomized controlled trials (RCTs), one of the top-tier designs for supporting intervention effectiveness.
Many factors can affect the internal, external, construct, and statistical validity of the design, implementation, and results of RCTs (Shadish, Cook, & Campbell, 2002). The primary purposes of a feasibility study are to ensure that study implementation is practical and to reduce threats to the validity of the study’s outcomes. Drawing on the emerging methodological literature in this area, this article first describes and defines feasibility and pilot studies; then presents a feasibility and pilot study typology that was designed for drug trials but that, with minor modification, is relevant to occupational therapy; and, finally, provides examples from my own and my colleagues’ work in Parkinson’s disease of how we used feasibility and pilot studies to design and refine a rehabilitation RCT.
Distinguishing Feasibility and Pilot Studies
One of the clearer definitions of feasibility study and its differentiation from pilot study comes from the United Kingdom’s National Institute for Health Research Evaluation, Trials and Studies Coordination Centre (NETSCC; 2012), which states, “Feasibility studies are pieces of research done before a main study in order to answer the question ‘Can this study be done?’. . . used to estimate important parameters that are needed to design the main study” (Research Methods section, para. 3). According to the NETSCC, a feasibility study differs from a pilot study in that a feasibility study tries out pieces of the RCT, whereas the pilot study tries out the operation of all pieces as they will be implemented in the planned RCT. A pilot study is
a version of the main study that is run in miniature to test whether the components of the main study can all work together . . . [and resembles] the main study in many respects, including an assessment of the primary outcome. (Research Methods section, para. 6)
Although the literature on feasibility and pilot study designs is relatively new and is not consistent with respect to these definitions, the emerging methodological literature suggests that both feasibility and pilot studies should be addressed specifically to descriptively assessing the feasibility and validity of the RCT plan and not to testing the hypotheses of the main RCT (Arain, Campbell, Cooper, & Lancaster, 2010; Leon, Davis, & Kraemer, 2011; Shanyinde, Pickering, & Weatherall, 2011; Thabane et al., 2010). Feasibility and pilot studies are not expected to have the large sample sizes that are needed to adequately power statistical null hypothesis testing. Indeed, pilot studies that are published often do not show statistically significant findings and rarely lead to larger trials to adequately power the hypothesis testing (Arain et al., 2010; Shanyinde et al., 2011). The outcomes of most feasibility and pilot studies should be measured with descriptive statistics, qualitative analysis, and the compilation of basic data related to administrative and physical infrastructure.
Unfortunately, editorial publication bias and peer review norms often require investigators to perform null hypothesis significance testing even when it is not scientifically reasonable to conduct these tests (Arain et al., 2010). In addition, reviewers and investigators alike tend to misinterpret nonsignificant statistical tests—those that fail to achieve the largely arbitrary criterion of p < .05 (Cohen, 1994)—of appropriately small-scale studies as indicative of the poor feasibility of future planned research or as the need for “more research” before research can be scaled up. These types of misguided conclusions can sidetrack or slow down the developmental progression of strong science and the rigorous testing of occupational therapy interventions.
Typology of Feasibility and Pilot Studies
A typology developed by a clinical epidemiology and biostatistics group at McMaster University (Thabane et al., 2010) appears to be one of the most systematic and comprehensive typologies developed to date. It focuses on drug trials; however, with minor modification, it can be a suitable rehabilitation intervention typology. Thabane et al. (2010) outlined four primary purposes for both pilot studies and feasibility studies: to test the (1) process, (2) resources, (3) management, and (4) scientific basis of the planned RCT.
To demonstrate these purposes, I draw on the experience of a recently completed RCT (Tickle-Degnen, Ellis, Saint-Hilaire, Thomas, & Wagenaar, 2010). The purpose of the RCT was to determine whether increasing hours of interdisciplinary self-management rehabilitation (physical, occupational, and speech therapy) had increasing benefits for health-related quality of life (HRQOL) in patients with Parkinson’s disease (PD) beyond best medical treatment, whether effects persisted at 2- and 6-mo follow-up, and whether targeted compared with nontargeted HRQOL domains responded more to rehabilitation. Participants on best medication therapy were randomized to one of three conditions for 6 wk of intervention: (1) 0 hr of rehabilitation; (2) 18 hr of clinic group rehabilitation plus 9 hr of attention control social sessions; and (3) 27 hr of rehabilitation, with 18 hr in clinic group rehabilitation and 9 hr in rehabilitation designed to transfer clinic training into home and community routines. Intervention was client centered in addressing participants’ specific quality-of-life concerns and also provided general strategies for addressing concerns that typically occur during the progression of the disease.
Process Assessment
Examples of questions for assessing the processes of a planned RCT include the following:
What are the expected
Numbers of eligible members of the targeted population?
Recruitment rates?
Refusal rates for participation and for randomization?
Retention and follow-up rates as the participants move through the trial?
Adherence rates to study procedures, intervention attendance, and engagement?
How feasible and suitable are
Eligibility criteria? Are criteria clear and sufficient or too inclusive or restrictive?
Data collection assessments? Do participants understand the questions and other data collection methods? Do they respond with missing or unusable data?
Amount of data collection? Do the participants have enough time and capacity to complete data collection procedures? Does the overall data collection plan involve a reasonable amount of time, or does it create a burden for the participants?
Process assessment is often documented in grant proposals in sections describing preliminary studies and human participant plans. At the time we were planning our RCT (2001–2002), we were unaware of a typology available to guide our preliminary planning. We did, however, have completed studies and collected evidence that we used to plan the RCT and support feasibility. Our neurological physician and nurse research team collected evidence about the number of eligible members of our targeted recruitment population—community-living adults with PD—whom the team followed in a movement disorder clinic. We developed potential recruitment, refusal, retention, and adherence expectations on the basis of a PD exercise trial led by our physical therapy investigators (Ellis et al., 2005) and on the research experience of the neurological team. We conducted small clinical interventions in occupational therapy, physical therapy, and speech–language pathology that involved elements of the planned RCT and asked clients to rate their satisfaction with the intervention, and we recorded adherence rates to the intervention.
For the feasibility and suitability of eligibility criteria, the research team members reflected qualitatively on their clinical and research experiences and combined these reflections with published standards for clinical trials for older adult and PD populations. To assess the quality and burden of our data collection procedures, we used experience from our physical therapy trial, our collective research and clinical experience, and a literature review of gold-standard PD HRQOL measures. We estimated the number of hours our assessments would take and took into consideration the interaction of data collection procedures with PD medication timing and fatigue. We also outlined factors that could contribute to our population completing the assessments as planned. We used this information to build snack and bathroom breaks into the study protocol and created a written data collection protocol that involved verbal administration and supervision of all data collection by a trained assessment team (blind to intervention condition). The primary investigator team collectively assessed the protocol and informally tested various aspects of the timing and administration of the protocol.
Resources Assessment
Examples of questions for assessing the resources for implementing a planned RCT include the following:
Do we have the
Physical capacity to handle the number of participants? What is the square footage as related to the stages and tasks of the procedures?
Phone and communication technology capacity to stay in touch with and coordinate the participants? Is there Web and teleconferencing capability?
Time to conduct each stage and aspect of the protocol? What are the time frames, and how do they coordinate with other responsibilities? How long does it take to connect with a participant or to send out mailings?
Equipment in the correct place at the correct time? What equipment is needed, and is it available when needed?
Ability to deal with broken, lost, or stolen equipment and materials? Are there backup plans for obtaining needed equipment and materials?
Adequate software to capture and use data? What software is available for conducting the research?
Institutional, departmental, and clinical centers’ willingness, motivation, and capacity to carry through with project-related tasks and to support investigator time and effort? What administrative services are in place for research at this level?
Documented evidence indicating that these centers abide by their commitments? What are the challenges in fulfilling research support commitments?
Access to basic services, such as copying, libraries, institutional technology, data servers, and purchasing?
Resource assessment often involves the collection and summarization of factual information at the investigators’ institutional settings. This information is often documented in grant proposals on facilities and resources forms and in the narrative description of research procedures. The gathering of resource evidence is the basis for determining what new materials, systems, and equipment must be obtained before the research activities can be implemented and for developing a study budget.
When planning our RCT, we measured the size of our labs and clinical spaces where assessments and clinic-based intervention would occur. We counted offices, desks, chairs, and computers; listed our software; located backup materials; and investigated and summarized all the services that our institutional settings provided for research such as ours. We spent considerable time assessing and developing institutional and departmental support for the research activities, clarifying and documenting decisions through e-mail. We secured rooms and obtained locking file cabinets for secure data storage and documented these in our research and human participant plans.
Although studies and documentation of resources may seem minor relative to process assessment and scientific development of the research plan, thorough resource assessment is fundamental to the success of research implementation. For example, Gardner, Gardner, MacLellan, and Osborne (2003) found that their otherwise thoughtfully conducted pilot study had not taken into consideration hospital process and environment factors that contributed to poor recruitment of participants and inadequate adherence of hospital staff to study protocol in the main study despite staff education related to project implementation. Consequently, the main study had to be aborted after startup. Such unfortunate events often entail great financial costs and potentially compromise future collaborations and the credibility and reputation of a research team.
Management Assessment
Examples of questions for assessing management issues in implementing a planned RCT include the following:
What are the challenges and strengths of
The investigators’ administrative capacity to manage the planned RCT?
Research investigator and staff capacities, expertise, and availability for the planned research activities?
Formats and structures of forms that document participant progress through the trial?
Accurate data entry into the computer? Are data lost, forgotten, or entered incorrectly? How are data files organized, named, and dated? Who is in charge of tracking the latest data entry and the quality of entry?
Matching of data to participants from different sources (e.g., participant screen data, consent and entry into the RCT, adherence, and responses on outcome measures)?
Management of the ethics of the research? To what extent do staff comply with the approved human participants protocol? How effectively are adverse events during implementation identified, documented, and reported? What happens if a participant experiences a clinical emergency or if family abuse is identified during the trial?
Management assessment is often documented in grant proposals in the investigator biosketches, data management and human participant plans, and budget justifications describing the specific responsibilities, activities, and roles of the research personnel. In our RCT, we drew on our collective experience to determine what our strengths and weaknesses were related to management and then planned specific activities and roles around investigator strengths to minimize weaknesses. Only after we started collecting data did we determine that our institution had a centralized data management service, and we were able to secure additional funding to use that service.
Because management of an RCT involves superb collaborative and communication skills and frequent collaboration and communication among research staff, these qualities must be assessed and documented before the initiation of the trial. For our RCT, our strongest systematic assessments involved compliance with human participant ethics and the storage of recruitment and screening data because, at the time, these were the most formalized aspects of research planning in our institution and in general practice. We completed other aspects of management assessment rather informally and on the basis of our collective research experience without a comprehensive typology guiding our planning. Current research planning practices demand more formal and systematic assessment.
Scientific Assessment
Examples of questions for assessing the scientific basis for implementing a planned RCT include the following:
What is the level of safety of the procedures in the intervention or interventions?
What is the level of safety and burdensomeness of the frequency, intensity, and duration of the intervention? Can these and other elements be standardized in a protocol without loss of a client-centered, individualized focus?
What are the reliability, validity, and trustworthiness of the assessments for the targeted population for this specific intervention? Do the assessments capture individual participants’ needs and measure their responsiveness to these needs?
What values constitute clinically meaningful differences on the primary outcome measures or assessment procedures?
What is the expected degree of change (i.e., responsiveness) of the participants?
What are the estimates of the intervention effect and the variance of that effect across the planned population?
What are the expected subgroup effects (i.e., specificity effects or moderator variables)?
When we think of pilot studies, the questions listed above often are the ones that come to mind. They are the research questions that we most clearly identify as necessary to investigate before conducting an RCT. Notice that there is no mediation question in the above list. Mediation is what causes the presumable effectiveness of the intervention. A whole set of other research studies—descriptive, observational, and experimental—occur before the implementation of the feasibility and pilot studies that set the foundation for the RCT. The RCT is often planned around a theoretical model of causality that has already been tested as representative of an underlying “active ingredient” involved in improving health or minimizing disability. Causality may be too restrictive of a construct for occupational therapy; we often use theoretical models of intervention that are targeted to the reduction of a set of risk factors or promotion of a set of protective factors related to health and disability, without the assumption that any one factor or variable is the critical active ingredient. Occupational therapists call on a multiplicity of ingredients to create adaptive responses in clients.
Assessing the feasibility of the scientific basis of the RCT largely involves assessing whether elements of the RCT will be likely to operate with low degrees of error and threats to validity (Shadish et al., 2002). In the case of our RCT, we drew on the emerging research literature on self-management of chronic disease and on theoretical models of health behavior and task performance to guide how we would promote HRQOL outcomes with a client-centered approach. We chose to conduct an interdisciplinary rather than a discipline-specific or multidisciplinary intervention on the basis of our scientific and clinical theory that health promotion would be greater with a task-specific than a discipline-specific or multidisciplinary approach. For example, if we wished to promote participants’ engagement in doing a favorite activity, we would most effectively do so by helping them integrate and manage their physical, occupational, and speech capacities in the service of doing the activity. Logically, this objective called for interdisciplinary intervention.
We conducted pilot studies to reduce threats to validity and documented these studies in grant proposal sections on preliminary studies, sample size estimation, the description and rationale for the research plan, and the human participant plan. These studies included our exercise trial (Ellis et al., 2005) and two meta-analyses on the effectiveness of rehabilitation for PD, one on occupational therapy effectiveness (Murphy & Tickle-Degnen, 2001) and one on physical therapy effectiveness (de Goede, Keus, Kwakkel, & Wagenaar, 2001).
We also planned the RCT around the evidence found in the literature on movement and speech science and quality of life research in older adults with PD. From this investigation, we ascertained the probable safety of our intervention and decided on our primary outcome measure. At the time of the initiation of the RCT, no research studies were available on clinically meaningful differences for our measure of HRQOL. This information became available by the time of publication of our study, and we reported our results accordingly.
Our pilot studies provided us with estimates of the probable effect of our intervention, enabling us to estimate an adequately powered sample size for our study. At the time, the standards of sample size estimation did not include an estimation of the variance of the predicted effect; however, this estimate is now advisable for planning an RCT (Lenth, 2001). Finally, we did not attempt to differentiate rehabilitation effects for subgroups of people with PD because little evidence was available in this respect. After completion of the trial, we performed post hoc analyses that suggested that more problematic baseline HRQOL predicted more participant responsiveness to the intervention, and from these tests we generated hypotheses to be tested in future intervention research.
Conclusion
Feasibility and pilot studies are important for building the foundation of large RCTs. These studies address all elements of the planned trial and ensure that the study is feasible and will be conducted in a manner that reduces threats to study validity. When conducted with proper aims and approaches, feasibility and pilot studies confront researchers with important facts before research stakeholders commit to a major investment in money and time for a large clinical trial. The publication of feasibility studies before a planned RCT, especially related to process, resources, and management of the RCT, was unusual in the past and is now becoming more common in the broader medical literature. Publication of these types of studies is rare in occupational therapy (e.g., Sturkenboom et al., 2012). The typology developed by Thabane et al. (2010) and slightly modified for occupational therapy intervention research provides guidance on how to approach pre-RCT studies systematically.