Intervention Effectiveness for Children and Youth

Efficacy of Practice and Effectiveness Trials

As the profession strives to meet the ambitious charge to position occupational therapy within higher evidentiary strata, it is important to define and distinguish efficacy of practice from effectiveness trials. Evidence for practice falls along a continuum, or into a hierarchy, with research designs with higher levels of rigor being less vulnerable to bias and error, better able to be generalized, and more likely to have outcomes attributed to the intervention being studied. Multiple methods are available for classifying evidence, in part because of discipline-specific methods that have distinct purposes, orientations, theories, and research methods uniquely developed to answer discipline-specific questions (Reichow, Volkmar, & Cicchetti, 2008). Efficacy refers to the ability of an intervention to produce the desired beneficial effects when implemented by expert clinicians and in ideal circumstances and essentially encompasses four key concepts, each typically assessed in a separate research study (Sussman, Valente, Rohrbach, Skara, & Pentz, 2006). Efficacy studies ask questions about the effectiveness of clinical interventions, their safety, their cost–benefit ratio in regard to time and cost, and their acceptability to clients.

Similarly, effectiveness intervention studies compare two independent variables such as intervention, medication, surgery, or therapeutic practice and evaluate the extent to which the intervention produces the intended change in the outcome of interest. Comparative effectiveness research compares existing health care interventions to determine which works best, for whom, and under what circumstances. Researchers have suggested that occupational therapy practitioners need to look at all levels of evidence (I–V) and not fixate on a rigid standard or they may miss effective and emerging treatments.

Pragmatic intervention studies look at questions related to the overall effectiveness of an intervention as it is delivered in treatment-as-usual methods, thus referring to clinical research that evaluates a therapy holistically, as it is used in normal practice. When everyday clinical practice is modeled, researchers are allowed to individualize protocols, adapt treatment, and ultimately test the overall package of care, but pragmatic intervention studies are not designed to empirically evaluate the contributions of an intervention’s components in isolation (McPherson, Smith-Lovin, & Brashears, 2006). The strength of pragmatic research is its ability to test the effectiveness of an intervention as it is meant to be used in practice. This ability is balanced against the benefit of controlling variables in a true experimental model. Many occupational therapy effectiveness trials may be better regarded as pragmatic interventions. What is important is to help clinicians learn to evaluate and analyze research to assess appropriate and reasonable interventions (Reichow et al., 2008).

Method

To analyze the 11 intervention effectiveness studies related to children and youth published in AJOT during 2012, we first identified the research design used in each study, determined whether it had the rigor to provide evidence for practice, and discussed implications of use of this evidence by practitioners, clients, and third-party payers. We then used the level-of-evidence hierarchy system developed by the AOTA Evidence-Based Literature Review Project (Lieberman & Scheer, 2002) to classify the reviewed studies. Level I is the highest level of research, encompassing systematic reviews, meta-analyses, and randomized controlled trials. Level II study designs consist of two-group pretest–posttest designs in which a control is present and randomization is not (e.g., cohort designs, case–control studies). Level III study designs include neither control nor randomization but instead use a one-group pretest–posttest design. Level IV includes studies involving single-subject designs, descriptive studies, and case series. Level V involves case studies or expert opinion studies that are not based on systematic review. Consistent with the review of 2011 articles on children and youth (Hilton & Smith, 2012), we used a pediatric adaptation of Baum’s (2011) expanded International Classification of Functioning, Disability and Health categories (WHO, 2007) to examine levels of mechanisms of rehabilitation science. Also, following Gutman’s (2008) charge, we identified alignment with research priorities for the Centennial Vision. Finally, to optimize clinicians’ ability to apply research to informed practice, we assessed the 11 articles against criteria of efficacy, effectiveness, comparative effectiveness, and pragmatic methodology (see Table 1).

Results

Intervention effectiveness studies in the area of children and youth have shown an increasing trend in numbers and evidence level of studies over the past 4 yr (Table 2). Ages of children examined in the 2012 studies ranged from <4 to 21 yr. All 11 studies involved quantitative data analysis, although 4 studies (27%) also incorporated qualitative measures (Case-Smith, Holland, Lane, & White 2012; de Brito Brandão, Gordon, & Mancini, 2012; Kinnealey et al., 2012; Schaaf, Hunt, & Benevides, 2012). Only 1 study enrolled children younger than age 3 (Silva, Schalock, Garberg, & Smith, 2012). Four of the studies enrolled children younger than age 6 (Case-Smith, DeLuca, Stevenson, & Ramey, 2012; Dunn, Cox, Foster, Mische-Lawson, & Tanquary, 2012; Schaaf et al., 2012; Silva et al., 2012), 6 enrolled children between ages 6 and 12 (Case-Smith, Holland, et al., 2012; de Brito Brandão et al., 2012; Dunn et al., 2012; Koenig, Buckley-Reen, & Garg, 2012; Palsbo & Hood-Szivek, 2012; Wells, Chasnoff, Schmidt, Telford, & Schwartz, 2012), and 2 enrolled adolescents (Gutman, Raphael-Greenfield, & Rao, 2012; Kinnealey et al., 2012). Four (36%) of the studies were Level I (Case-Smith, DeLuca, et al., 2012; de Brito Brandão et al., 2012; Silva et al., 2012; Wells et al., 2012), 2 (18%) were Level II (Gutman et al., 2012; Koenig et al., 2012), 3 (27%) were Level III (Case-Smith, Holland, et al., 2012; Dunn et al., 2012; Palsbo & Hood-Szivek, 2012), and 2 (18%) were Level IV (Kinnealey et al., 2012; Schaaf et al., 2012).

Researchers addressed five of the seven areas of the framework mechanisms. Specific mechanisms examined included body function and body structure (Kinnealey et al., 2012; Schaaf et al., 2012; Wells et al., 2012), functional limitations (Case-Smith, DeLuca, et al., 2012; Gutman et al., 2012; Koenig et al., 2012; Silva et al., 2012; Wells et al., 2012), activity (Case-Smith, Holland, et al., 2012; de Brito Brandão et al., 2012; Gutman et al., 2012), participation (Dunn et al., 2012), and environment (Kinnealey et al., 2012). See Table 1 for framework categories.

More than half (63%) of the studies used pragmatic trial methodology, assessing treatment in context as it is meant to be delivered (Case-Smith, DeLuca, et al., 2012; Case-Smith, Holland, et al., 2012; de Brito Brandão et al., 2012; Dunn et al., 2012; Gutman et al., 2012; Koenig et al., 2012; Wells et al., 2012; Table 3). One study addressed comparative effectiveness (de Brito Brandão et al., 2012). Additional criteria specific to the profession, including reliability and validity of test instruments, were examined in 2 of the studies (Palsbo & Hood-Szivek, 2012; Silva et al., 2012). Engagement in occupation and its ability to promote developmental milestones, health, and wellness were examined in 3 studies (Gutman et al., 2012; Kinnealey et al., 2012; Koenig et al., 2012). Three studies included examination of how specific disability experiences affect community and social participation (Dunn et al., 2012; Gutman et al., 2012; Schaaf et al., 2012), and 6 included questions that guide professional evolution (Case-Smith, Holland, et al., 2012; Dunn et al., 2012; Gutman et al., 2012; Kinnealey et al., 2012; Palsbo & Hood-Szivek, 2012; Silva et al., 2012). Acceptability of treatment was appraised in one study (Case-Smith, DeLuca, et al., 2012), although client satisfaction was considered more often (Case-Smith, Holland, et al., 2012; Dunn et al., 2012; Kinnealey et al., 2012). Schaaf et al. (2012) specifically explored factors that contribute to the success or failure of a specific frame of reference, namely the Ayres Sensory Integration approach.

Discussion

Multiple studies examined in this review were aimed at creating conceptual models, translating theory into practice, and informing program development as it relates to best practices for treatment, policy development, and program development. Studies that examined factors central to children, youth, and their families, such as finding a friend and participating in community life, are reflected in the studies examined for this article, but factors such as barriers to and supports for procuring and maintaining jobs are not.

Intervention effectiveness studies in the area of children and youth have shown signs of moving in a positive direction toward guiding practitioners to make evidence-based decisions affecting skills, processes, and foundations for childhood and adolescent occupations in comparison with studies published in 2011 (Hilton & Smith, 2012). One study examined children younger than age 3, and the majority were between ages 3 and 12, but 2 of the studies did address adolescents, an often overlooked population, which shows important progress toward meeting the objectives of the Centennial Vision charging researchers to examine the participation of children with special needs in their daily lives as they transition through childhood and adolescence into adulthood. Studies also included qualitative methodology, helping to better explain the roles and participation of parents as they experience disabilities in the context of family-centered services. Researchers must examine the emotional and social costs of occupational deprivation and occupational injustice, such as depression, alcohol and substance abuse, and suicide in disenfranchised youth, to children, youth, and their families, as well as to society.

Investigators are highly involved in examining interventions in context and across sites. Studies addressed a wide range of mechanisms from the framework areas: body function and body structure, functional limitations, activity, participation, and the environment, continuing to build evidence for the broad scope of practice within the field of occupational therapy. Discipline-specific methods to uniquely address the charge for science-driven practice have evolved, and pragmatic, comparative effectiveness, and mixed methods have all been used to inform practice, guide clinical decision making, and allow future researchers to attribute outcomes to interventions used in practice. For example, the majority of studies included manualized interventions, demonstrated adherence to intervention fidelity, and used established treatment protocols. Also, the inclusion of parental stress, teacher reflection, and student reflection demonstrates the profession’s commitment to understand intervention effectiveness framed within an ecological system.

Inclusion of measures of intervention fidelity has been identified as commonly absent in intervention effectiveness studies in AJOT (Murphy & Gutman, 2012). Intervention fidelity examines the extent to which the intervention is delivered as it was intended (Gearing et al., 2011). Describing the specific intervention protocol, which might be in the form of a manual, is the first step toward being able to achieve fidelity in adherence to the protocol. Use of intervention fidelity measures in several of the studies published during 2012 is promising, but it is an area for greater adherence in future studies. Inclusion of measures of client acceptability and cost–benefit analysis of services is lacking, and future research must address these issues to further establish the profession’s contribution to the practice area of children and youth. Discussing the major changes in health care delivery in the next decades will certainly include cost–benefit analysis of services within the larger conversation of improved quality of health care within a contracting budget.

Perhaps it is tacit knowledge and the commonness of the concern obscures its importance, but only 1 study addressed the acceptability of intervention. An even greater omission was that not one of the studies examining effectiveness of school-based intervention used alignment with the core curriculum standards as a variable of interest. Generalization and maintenance of improvement continues to be an area of weakness, and future investigators are encouraged to address this gap in the evidence.

Future reviews may want to evaluate trends in the number of efficacy, effectiveness, comparative effectiveness, and pragmatic trials because these trends are pertinent to fostering evidence that informs an evidence-based and science-driven practice. An even more important outcome is that such reviews will allow the profession to maintain a practice-informed evidentiary dialogue. Continued appraisal of identified research priorities (efficacy, reliability of test instruments, engagement, impact of disability on community and social participation, and the generation of questions that guide the future growth and evolution of occupational therapy) is essential and should become part of our self-assessment in the upcoming years.

In general, occupational therapy research is interested in understanding the pragmatic relevance of intervention and the extent to which intervention promotes participation, a core edict of the Centennial Vision. Future studies will need to include acceptability of the intervention and alignment with outcomes relevant to the context, such as the core curriculum standards in school-based interventions as well as the impact on families and children with disabilities, especially siblings and other family members. Cost–benefit analysis of intervention and comparative effectiveness trials, areas of concern neglected in the existing research, will likely become increasingly urgent in light of proposed health care reform laws and policies that will affect health care delivery in the next decade, with particular attention paid to the emotional and social costs to children, youth, and families—and to society—of occupational deprivation and occupational injustice.

Conclusion

Compared with studies published in previous years, the studies examined in this review showed improvement in their ability to guide practitioners to make evidence-based decisions by increasing the understanding of the pragmatic relevance of intervention and the extent to which intervention promotes participation in childhood and adolescent occupations. Studies’ level of evidence increased, as did efforts to increase scientific rigor. Intervention fidelity was, however, not consistently included in the studies. Several studies included qualitative methodology, helping to better explain the roles and participation of parents as they experience disabilities within family-centered services. Other strengths identified in this review include those aimed at creating conceptual models, translating theory into practice, and informing program development as it relates to best practices for treatment, policy development, and program development.

Although examination of factors central to children, youth, and their families was reflected in the studies examined for this article, barriers to and supports for procuring and maintaining jobs were not. Inclusion of longitudinal studies, measures of client acceptability, cost–benefit analysis, and alignment with educational standards (how coccupational therapy intervention supports academic success) also need more attention in future studies. In addition, occupational therapy research needs to do more to examine the emotional and social costs of occupational deprivation and occupational injustice for children and youth. Generalization and maintenance of improvement continue to be areas of weakness. Future studies should include acceptability of the intervention and alignment with outcomes relevant to the context and examine siblings and other family members.