Abstract
We reviewed 22 articles on children and youth published in 2011 in the American Journal of Occupational Therapy and organized the articles by level of research and research type according to a framework adapted from the International Classification of Functioning, Disability and Health (ICF; World Health Organization, 2007). The largest percentage of articles described intervention effectiveness studies classified as Level III or IV. The bulk focused on the Body Function/Body Structure construct of the ICF, but as a whole the studies addressed all the constructs except Biomedical Molecular/Cellular. Rigor remains a concern, although laudable efforts have been made to increase strength of the evidence. Longitudinal, efficacy, and qualitative studies, as well as studies examining adolescents and the transition to adulthood, were absent from articles in this review and are important areas for future investigation. Several studies explicitly addressed intervention fidelity, an imperative in evidence-based research needed to move the profession toward the Centennial Vision.
The Centennial Vision of the American Occupational Therapy Association (AOTA) states that occupational therapy will be a “powerful, widely recognized, science-driven, and evidence-based profession with a globally connected and diverse workforce meeting society’s occupational needs” (AOTA, 2007, p. 613). Supporting this effort, the American Journal of Occupational Therapy (AJOT) has the potential to guide practitioners in making evidence-based decisions by disseminating well-designed clinical trials; making practitioners, clients, and third-party payers aware of science-driven and evidence-based innovations that have the potential to improve participation and quality of life; and, eventually, influencing health care decisions for present and future clients.
Several of the elements viewed as relevant to a shared vision in AOTA’s Centennial Vision support engagement in rigorous research. They include the “power to influence,” “evidence-based decision making,” and “science-fostered innovation in occupational therapy practice” (AOTA, 2007, p. 614). For these elements, it is important to examine the level of rigor in research that informs occupational therapy practice. Another Centennial Vision element that may be addressed by research is “expanded collaboration for success” (p. 614), which could include collaborative research and measures to assess attitudes and opinions of various stakeholders.
AJOT has identified six main practice areas of focus with regard to publication of evidence-based practice: (1) productive aging; (2) children and youth; (3) rehabilitation, disability, and participation; (4) mental health; (5) work and industry; and (6) health and wellness (Corcoran, 2007). These areas of focus help the profession track research production and evolution with the desired result of facilitating practitioners’ use of evidence.
In 2006, the AOTA Children and Youth Ad Hoc Committee responded to a request by AOTA President Carolyn Baum to address issues related to children and youth. The committee’s report (AOTA, 2006) identified the following key areas of research to inform practice in the area of children and youth:
Basic and applied scientific studies related to skills, processes, and foundations for childhood and adolescent occupations
Factors that contribute to the success or failure of a specific frame of reference
Both qualitative and quantitative methodologies to address multiple facets of above
Efficacy studies that examine interventions (efficacy, effectiveness, outcomes development)
Theory development and development of conceptual models that promote integration of theory and practice
Empirical studies conducted in context
Translational research providing information on applications to practice, policy development, systems change, program development
The roles and participation of parents, siblings, and other family members within family centered services
Longitudinal studies of the participation of children with special needs in their daily lives as they transition through childhood and adolescence into adulthood
Studies that examine factors central to the children, youth and their families such as finding a friend, participating in community life, and procuring and maintaining jobs
Studies that examine the emotional and social cost of occupational deprivation and occupational injustice for children and youth such as depression, alcohol and substance abuse, and suicide in disenfranchised youth, and what this is costing emotionally to youth and family as well as to society. (p. 8)
In their most recent review of studies involving children and youth, Bendixen and Kreider (2011) used the International Classification of Functioning, Disability and Health (ICF; World Health Organization [WHO], 2001) and the ICF–Children and Youth Version (ICF–CY; WHO, 2007) as frameworks for categorizing the areas of study addressed in AJOT. Recently, Baum (2011) expanded the ICF categories to incorporate levels of rehabilitation science. We used this classification framework with a pediatric focus to examine the areas addressed for children and youth because it offers a more comprehensive categorization of these levels of research (Table 1).
Language of Rehabilitation Science
Note. ICF = International Classification of Functioning, Disability and Health. From Baum, C. (2011). “The John Stanley Coulter Memorial Lecture—Fulfilling the Promise: Supporting Participation in Daily Life,” by C. Baum, 2011, Archives of Physical Medicine and Rehabilitation, 92, p. 172. Copyright © 2011 by the American Congress of Rehabilitation Medicine. Adapted with permission.
Evidence-based research is particularly crucial in the focus area of children and youth because a substantial proportion of occupational therapy practitioners work with this population. According to the AOTA (2010) Workforce Study, 21.7% of occupational therapists and 21.4% of occupational therapy assistants currently practice in schools, in addition to the 5.2% of occupational therapists and 1.8% of occupational therapy assistants who practice with children and their caregivers in early intervention programs. In this review, we examined the AJOT articles published in the area of children and youth during 2011 and identified the types of research published; determined whether the published research has the rigor to provide evidence for practice; and discussed implications for use of this evidence by practitioners, clients, and third-party payers.
Method
We analyzed 22 research studies related to children and youth published in AJOT during 2011. We identified the research design used in each published study and categorized it into one of the following six research categories: (1) systematic reviews, (2) intervention effectiveness studies, (3) efficacy studies, (4) basic research, (5) instrument development and testing studies, and (6) professional issues.
We used the level-of-evidence hierarchy system developed by the AOTA Evidence-Based Literature Review Project (Lieberman & Scheer, 2002) to classify the reviewed articles. Level I consists of systematic reviews, meta-analyses, and randomized controlled trials. Level II includes two-group pretest–posttest designs in which control is present and randomization is not (e.g., cohort designs, case control studies). Level III designs involve neither control nor randomization but instead use a one-group pretest–posttest design. Level IV includes single-subject designs, descriptive studies, and case series. Level V involves case study or expert opinion that is not based on systematic review.
Results
Of the 22 articles, 45.5% (10) were intervention effectiveness studies, 22.7% (5) were instrument development and testing studies, 27.3% (6) were basic research, and 4.5% (1) was a professional issues study. We identified no systematic reviews or efficacy studies published during 2011. Table 2 summarizes the studies.
Articles on Children and Youth Published in 2011 in the American Journal of Occupational Therapy
Note. ADHD = attention deficit hyperactivity disorder; AOTA = American Occupational Therapy Association; ICC = intraclass coefficient; RCT = randomized controlled trial; SPM = Sensory Processing Measure; SSP = Short Sensory Profile; VABS = Vineland Adaptive Behavior Scales.
Intervention Effectiveness Studies
All the intervention effectiveness studies were quantitative. Ages of children examined in the studies were 3–12 yr. Three of the 10 studies were Level I (Golos, Sarid, Weill, & Weintraub, 2011; Pfeiffer, Koenig, Kinnealey, Sheppard, & Henderson, 2011; Silva, Schalock, & Gabrielsen, 2011), 2 were Level II (Collins & Dworkin, 2011; Lust & Donica, 2011), 3 were Level III (Bendixen et al., 2011; Case-Smith, Holland, & Bishop, 2011; Hahn-Markowitz, Manor, & Maeir, 2011), and 2 were Level IV (Fedewa & Erwin, 2011; Umeda & Deitz, 2011). Mechanisms examined included Body Function/Body Structure (Collins & Dworkin, 2011; Fedewa & Erwin, 2011; Golos et al., 2011; Hahn-Markowitz et al., 2011; Lust & Donica, 2011; Pfeiffer et al., 2011; Silva et al., 2011), Functional Limitations (Case-Smith et al., 2011; Golos et al., 2011; Hahn-Markowitz et al., 2011; Umeda & Deitz, 2011), Activity (Case-Smith et al., 2011; Golos et al., 2011; Lust & Donica, 2011), Participation (Collins & Dworkin, 2011; Fedewa & Erwin, 2011; Golos et al., 2011), and Environment (Bendixen et al., 2011; Umeda & Deitz, 2011).
A frequent limitation of past intervention effectiveness studies—a very small research cohort (n ≤ 10)—plagued 3 of the studies (Collins & Dworkin, 2011; Fedewa & Erwin, 2011; Umeda & Deitz, 2011). Limited generalizability of findings was also a concern for the 2 studies that examined only boys (Golos et al., 2011; Umeda & Deitz, 2011) and the 2 that examined children from only one classroom (Case-Smith et al., 2011; Umeda & Deitz, 2011). Issues regarding potential bias were a concern in several of the studies. In 1 study, the occupational therapist who was conducting the study also administered the assessments (Hahn-Markowitz et al., 2011). In another, the experimental group consisted of all students in one classroom, and the control group consisted of all students in another classroom, introducing a potential selection bias (Lust & Donica, 2011).
Several of the studies incorporated measures into their protocols to increase scientific rigor. Four included fidelity measures or examined procedural reliability to ensure consistency across interventions (Case-Smith et al., 2011; Collins & Dworkin, 2011; Hahn-Markowitz et al., 2011; Umeda & Deitz, 2011). Another study included a social validity scale to assess the degree to which teachers believed the intervention was helpful for the students, thus incorporating the opinions of a stakeholder into the study (Fedewa & Erwin, 2011). Three of the studies used blinded coders who were unaware of the study’s purpose or assignment of the participants for scoring to reduce the potential for assessment bias (Collins & Dworkin, 2011; Silva et al., 2011; Umeda & Deitz, 2011).
Instrument Development and Testing
The 5 instrument development and testing studies examined assessments addressing body function (Little et al., 2011; Spirtos, O’Mahony, & Malone, 2011; Vanvuchelen, Roeyers, & De Weerdt, 2011), participation (Taylor, Lee, Kramer, Shirashi, Kielhofner, 2011), and intervention fidelity (Parham et al., 2011). Two studies were Level II (Spirtos et al., 2011; Taylor et al., 2011), and 3 were Level III (Little et al., 2001; Parham et al., 2011; Vanvuchelen et al., 2011). All instruments were quantitative and addressed issues in assessment of children between preschool and 18 yr. All studies examined the validity and reliability of the instruments. The assessments addressed Body Function/Body Structure (Little et al., 2011; Spirtos et al., 2011; Vanvuchelen et al., 2011), Participation (Taylor et al., 2011), and Environment mechanisms (Parham et al., 2011). No studies examined assessments addressing Functional Limitations or Activity mechanisms.
The small size of the research cohorts (n ≤ 30) was a concern for 3 of the studies (Little et al., 2011; Parham et al., 2011; Spirtos et al., 2011). Little et al. (2011) used a larger sample for their examination of internal consistency (n = 358) but a small number for test–retest reliability (n = 24).
Basic Research
The 6 basic research studies were all quantitative and examined children from age 4 mo through adulthood or their care providers. None of the studies were longitudinal. Mechanisms examined included Biomedical (electroencephalography; Gavin et al., 2011), Body Function/Body Structure (sensory processing; Mailloux et al., 2011), Functional Limitations (postural control, fine motor skills; Bigsby et al., 2011; Clayton, Kaiser, de Pue, & Kaiser, 2011; Wang, Howe, Hinojosa, & Weinberg, 2011), and Environment (parent knowledge; Zachry & Kitzmann, 2011). Only 1 study examined a very small cohort (n ≤ 10; Clayton et al., 2011); most examined more substantial cohorts (n ≥ 100; Bigsby et al., 2011; Mailloux et al., 2011; Wang et al., 2011; Zachry & Kitzmann, 2011). For 3 of these studies, the implications for occupational participation were not clearly articulated (Bigsby et al., 2011; Clayton et al., 2011; Gavin et al., 2011). Four of the basic research studies were Level IV (Gavin et al., 2011; Mailloux et al., 2011; Wang et al., 2011; Zachry & Kitzmann, 2011), and 2 were Level III (Bigsby et al., 2011; Clayton et al., 2011).
Professional Issues
The study in the professional issues category examined factors associated with the Environment supporting the collaborative nature of early intervention practice (Myers, Schneck, Effgen, McCormick, & Shasby, 2011). Findings suggest that practices to enhance relationships among programs influence occupational therapist involvement. This study was Level IV; the research was quantitative and incorporated descriptive methods and regression analysis. The survey instrument used showed a high degree of internal consistency and reliability.
Discussion
The most common evidence level was Level III (36.4%), followed by Level IV (31.8%), Level II (18.2%), and Level I (13.6%). All studies were quantitative, indicating a failure to address the objective of including both qualitative and quantitative methodologies (AOTA, 2006) in this group of AJOT publications.
Intervention Effectiveness Studies
Inclusion of several Level I and Level II studies supports “power to influence,” “evidence-based decision making,” and “science-fostered innovation in occupational therapy practice” (AOTA, 2007, p. 614). Attention to scientific rigor continues to be an important priority in occupational therapy research, and continued efforts to increase rigor, such as manualizing intervention protocols, including procedural fidelity and social validity measures, and using blinded coders, all support the Centennial Vision elements.
Instrument Development and Testing
Each of the instrument development studies supports the increased ability of occupational therapists to provide “evidence-based decision making” (AOTA, 2007, p. 614) in client intervention by adding evidence that strengthens practitioners’ knowledge of available measurement tools. Because certain areas of occupational therapy intervention vary widely across geographic locations and practice settings, fidelity is a major concern affecting the validity of the research addressing these areas. Sensory integration is one of these areas, and research addressing its effectiveness has been plagued by problems with fidelity (May-Benson & Koomar, 2010; Miller, 2003; Parham et al., 2007; Pfeiffer et al., 2011). Efforts to achieve greater fidelity for sensory integration intervention in the Parham et al. (2011) study support “evidence-based decision making” (AOTA, 2007, p. 614).
Basic Research
Although basic research studies were generally categorized as lower-level studies, most were of a substantial size and examined a wide range of ages and levels of rehabilitation science. The lack of longitudinal studies indicates a failure to examine two needs identified by AOTA’s (2006) Children and Youth Ad Hoc Committee (2006): (1) “children’s and adolescents’ developmental and occupational patterns with the community, home, and school settings” and (2) “longitudinal studies of the participation of children with special needs in their daily lives as they transition through childhood and adolescence into adulthood” (p. 8).
Professional Issues
The focus of the article addressing professional issues (Myers et al., 2011) was on factors associated with the environment that support the collaborative nature of early intervention practice. This study addressed “power to influence” and “expanded collaboration for success” from the Centennial Vision. Continued research in the area of professional issues is imperative as our profession moves forward to fulfill the vision of becoming powerful, widely recognized, science driven, and evidence based. As we do with our clients, it is important to examine our own profession and its relationship with environmental supports and barriers. To best accomplish this, higher-level research studies examining occupational therapy professional issues are necessary to provide more evidence of influences on practice, development, and implementation of strategies to address these issues.
Conclusion
An important theme in intervention effectiveness studies is that of fidelity, defined as “the strategies that monitor and enhance the accuracy and consistency of an intervention to ensure it is implemented as planned and that each component is delivered in a comparable manner to all study participants over time” (Smith, Daunic, & Taylor, 2007, p. 121). Fidelity was explicitly addressed in several of the research studies but found lacking in others. Fidelity ensures that measurement of the effectiveness of our interventions is consistent; without it, we cannot claim that these interventions are evidence based. Use of larger study cohorts, blinded coders, and social validity scales are other important factors that contribute to strength of the evidence.
In their most recent review of occupational therapy research related to children and youth, Bendixen and Kreider (2011) used the concept of occupational therapy’s impact on positive youth development and the framework of the ICF to assess the research. They found that the bulk of studies fell within the ICF domains of Body Function/Body Structure and Activity. The levels of rehabilitation addressed in the studies published in 2011 were Body Function/Body Structure (50%), Functional Limitations (32%), Environment (23%), Participation (18%), Activity (14%), and Biomedical (5%), with none from Biomedical Molecular/Cellular. In the revised categorization used in this analysis, the largest group of studies continued to be those examining Body Function/Body Structure, but a representation was seen, at least modestly, across the levels, except for the most basic Biomedical Molecular/Cellular, which is not typically addressed by occupational therapy research.
The study authors did not always explicitly translate their findings to implications for participation, an omission that suggests a continuing need to document changes in children’s engagement in everyday life situations to build the evidence of occupational therapy’s efficacy in facilitating participation. None of the studies on children and youth published in AJOT in 2011 focused on efficacy (i.e., the ability of an intervention to produce the effect intended), and only 1 focused on professional issues. Longitudinal studies were not included among the 2011 studies. In addition, none of the intervention studies examined adolescents or issues involving the transition to adulthood. These are all important areas for future attention to enable a better understanding of developmental and occupational patterns of children and adolescents and better inform practitioners, clients, and third-party payers.
Gillen (2010) observed that occupational therapists frequently publish their studies in journals outside the discipline of occupational therapy. They may do so for two reasons: (1) the desirability of publishing in a variety of journals and (2) the difficulty of having manuscripts accepted for publication in AJOT. Although sharing our findings in journals outside our own profession may support our Centennial Vision of “expanded collaboration for success” and “power to influence” (AOTA, 2007), it is also important toward these objectives that researchers include AJOT and other occupational therapy journals in their choices of where to publish. Conversely, it is no less important that AOTA support these efforts by maximizing the capacity of AJOT to publish the rapidly expanding evidence being produced by members of this profession.
Footnotes
*
Indicates studies that were reviewed for this article.