Abstract
We reviewed 12 articles from 2012 that addressed development and testing of instruments for children and youths and American Journal of Occupational Therapy articles from 2009–2013 that addressed 11 activity and participation instruments to determine how well this group of instruments facilitates the generation of evidence sufficient to support practice in accordance with the Centennial Vision. We observed an increase in the number of instrument development and testing studies and in higher level studies and larger cohorts; funding was provided for almost half of the studies, and attention was given to use of blind testing and transition to adult-age assessments. Further development of performance-based activity and participation instruments; instruments that examine biomedical molecular–cellular, biomedical, and environmental mechanisms; and intervention fidelity measures and increased use of blind testing are necessary for occupational therapy to meet the Centennial Vision.
Our goal in this review was to critically appraise articles from 2012 addressing instrument development and testing for children and youths and to review more extensively articles published in the American Journal of Occupational Therapy (AJOT) in the period 2009–2013 that address activity and participation instruments. Doucet and Gutman (2013) asserted that the strength of our profession is in occupational therapists’ unique ability to thoroughly and comprehensively measure function. They stated further that the ability to evaluate it in a specific, objective, and quantifiable way is not only important to determine progress for reimbursement of services but is also critical for the profession’s survival. Others have added that to address this issue, it is necessary to discard home-grown assessments, which are neither reliable nor valid, and adopt the use of cost- and time-efficient standardized assessments (Radomski & Trombly-Latham, 2008). Attention to instrument development is a crucial step toward providing evidence-based practice that will allow the occupational therapy profession to become a “powerful, widely recognized, science-driven, and evidence-based profession with a globally connected and diverse workforce meeting society’s occupational needs” (American Occupational Therapy Association [AOTA], 2007, p. 613).
Our objective for this review was to examine how well this group of assessments facilitates the generation of evidence sufficient to support practice in accordance with the Centennial Vision, positioning occupational therapy as an evidence-based and science-driven profession to embrace excellence and sharpen its competitive edge (AOTA, 2007; Clark, 2011). This effort was designed to examine AJOT’s performance in three key areas: (1) guiding practitioners to make evidence-based decisions by disseminating well-designed clinical trials; (2) raising the awareness of practitioners, clients, and third-party payers of science-driven and evidence-based innovations that have the potential to improve participation and quality of life; and (3) ultimately influencing health care decisions for current and future clients. These areas echo AOTA’s Centennial Vision to conduct research geared to influence and support evidence-based decision making and foster innovation in occupational therapy practice (AOTA, 2007).
The International Classification of Functioning, Disability and Health (ICF) : Children and Youth Version (World Health Organization [WHO], 2007) describes activity as “the execution of a task or action” (p. 12) and participation as “involvement in a life situation” (p. 12). Participation in purposeful activities is a central aspect of the human experience (Wilcock, 1993). The Occupational Therapy Practice Framework: Domain and Process, 2nd Edition (AOTA, 2008) acknowledges the important connection between participation and health. The importance of participation in the lives of children has been examined in several studies. Law and King (2000) observed that participation in formal and informal activities is the “context in which children form friendships, develop skills and competencies, express creativity, achieve mental and physical health, and determine meaning and purpose in life” (p. 10). They proposed that by participating in activities, children develop an understanding of society’s expectations and acquire the physical and social skills needed to function and flourish. Moreover, participation in everyday activities plays an important role in a child’s social development and influences long-term mental and physical health (Law et al., 2004). Therefore, measuring the impact of occupational therapy interventions on participation and developing sensitive and psychometrically sound assessment tools to monitor client progress in participation are essential. It is also important that the assessment tools be able to evaluate performance or function to measure the changes in participation (Majnemer, 2009).
The AOTA Children and Youth Ad Hoc Committee (CYAC) identified 11 key areas of research that delineate the focus of the Centennial Vision for the area of children and youth (AOTA, 2006, p. 8):
Basic and applied scientific studies related to skills, processes, and foundations for childhood and adolescent occupations
Factors that contribute to the success or failure of a specific frame of reference
Both qualitative and quantitative methodologies to address multiple facets of the above
Efficacy studies that examine interventions (efficacy, effectiveness, outcomes development)
Theory development and development of conceptual models that promote integration of theory and practice
Empirical studies conducted in context
Translational research providing information on applications to practice, policy development, systems change, and program development
The roles and participation of parents, siblings, and other family members within family-centered services
Longitudinal studies of the participation of children with special needs in their daily lives as they transition through childhood and adolescence into adulthood
Studies that examine factors central to the children, youth, and their families such as finding a friend, participating in community life, and procuring and maintaining jobs
Studies that examine the emotional and social cost of occupational deprivation and occupational injustice for children and youth such as depression, alcohol and substance abuse, and suicide in disenfranchised youth, and what this is costing emotionally to youth and family as well as to society.
These areas advocate the conduct of rigorous research, evidence-based and systemically oriented to respective stakeholders, with the ultimate goal of enhancing treatment efficacy and outcomes. A substantial proportion of occupational therapy practitioners work with children and youth. According to the 2010 AOTA Workforce Study, 21.7% of occupational therapists and 21.4% of occupational therapy assistants currently practice in schools, in addition to the 5.2% of occupational therapists and 1.8% of occupational therapy assistants who are engaged in practice with children and their caregivers in early intervention programs. These practitioners, in particular, could benefit from quality evidence-based research that addresses activity and participation in children and youth and documents the validity and viability of a range of assessments.
Method
We first analyzed the 12 articles addressing instrument development and testing studies related to children and youth published in AJOT in 2012. We identified the research design used in each published study, determined whether the published research had the rigor to provide evidence for practice, and discussed implications for use of this evidence by practitioners, clients, and third-party payers. We used the level-of-evidence hierarchy system developed by the AOTA Evidence-Based Literature Review Project (Lieberman & Scheer, 2002) to classify the reviewed articles. Level I is the highest level of research, encompassing systematic reviews, meta-analyses, and randomized controlled trials. Level II consists of two-group pretest–posttest designs in which control is present and randomization is not (e.g., cohort designs, case control studies). Level III designs include neither control nor randomization but instead use a one-group pretest–posttest design. Level IV includes single-subject designs, descriptive studies, and case series. Level V involves case study or expert opinion studies that are not based on systematic review. Consistent with Hilton and Smith’s (2012) review of 2011 articles on children and youth, we used a pediatric adaptation of Baum’s (2011) expanded ICF categories (WHO, 2001) to examine levels of mechanisms in rehabilitation science. See Table 1 for details.
Language of Rehabilitation Science
Note. ICF = International Classification of Functioning, Disability and Health (World Health Organization, 2001). From “The John Stanley Coulter Memorial Lecture—Fulfilling the Promise: Supporting Participation in Daily Life,” by C. Baum, 2011, Archives of Physical Medicine and Rehabilitation, 92, p. 172. Copyright © 2011 by Elsevier. Used with permission.
Our second task in this review was to analyze 11 instrument development and testing studies of activity and participation instruments related to children and youth published in AJOT between 2009 and 2013. For this group, we focused on the instruments themselves. We analyzed frame of reference or theoretical model used to conceptualize development of the instrument, perspective used by the instrument (child, parent or caregiver, teacher, observation or administration), cultural relevance, population addressed by the instrument, purpose, description, test components, and item categories. We discussed practicality (time for completion, cost, training, and use for clinical vs. research purposes) and outcome measures. We compared the psychometric analyses completed on each assessment.
Results
Instrument Development and Testing Studies (2012)
The number of instrument development and testing studies rose from 5 studies in 2011 (Hilton & Smith, 2012) to 12 studies in 2012. (See Table 2 at the end of this article for a detailed analysis.) Among the 12 studies, 7 were Level II and 5 were Level III (Blanche, Bodison, Chang, & Reinoso, 2012; Bourke-Taylor, Law, Howie, & Pallant, 2012; Honaker, Rosello, & Candler, 2012; Ohl et al., 2012; Weiner, Toglia, & Berg, 2012), compared with 2 Level II and 3 Level III studies in 2011. Of the 12 studies in 2012, 1 examined a cohort of <30; 5, between 31 and 100; 5, between 101 and 265; and 1, >2,000. In 2011, 3 of the 5 studies had <30 and none had >400. One 2012 study reported using blinded testing (Brossard-Racine, Mazer, Julien, & Majnemer, 2012). Of the 13 studies, 5 were funded. Three of the instruments were designed for transitional ages of adolescence and young adulthood (Kao, Kramer, Liljenquist, Tian, & Coster, 2012; Saban, Ornoy, Grotto, & Parush, 2012; Weiner et al., 2012), and 2 were designed specifically for that age group. Two of the instruments address parent and family occupations (Bourke-Taylor et al., 2012; Honaker et al., 2012). One of the studies involved blinded testing (Brossard-Racine et al., 2012). No studies involved development of an intervention fidelity measure.
Four of the instrument development and testing studies examined body function or body structure mechanisms (Blanche, Bodison, et al., 2012; Blanche, Reinoso, Chang, & Bodison, 2012; Ohl et al., 2012; Weiner et al., 2012). Three instruments addressed functional limitation mechanisms (Griswold & Townsend, 2012; Saban et al., 2012; Silva & Schalock, 2012). Three instruments addressed activity mechanisms (Bourke-Taylor et al., 2012; Brossard-Racine et al., 2012; Kao et al., 2012). One of these addressed activity for mothers (Bourke-Taylor et al., 2012). Two instruments addressed participation mechanisms (Honaker et al., 2012; McDonald & Vigen, 2012).
Articles on Children and Youth Instrument Development and Testing Studies Published in 2012 in the American Journal of Occupational Therapy
Note. ASD = autism spectrum disorder; DCD = developmental coordination disorder; M = mean; SD = standard deviation.
Activity and Participation Instruments (2009–2013)
The 11 activity and participation instruments examine a range of activities from several perspectives. Three are performance based (Brossard-Racine et al., 2012; Josman, Goffer, & Rosenblum, 2010; Munkholm, Berg, Löfgren, & Fisher, 2010). The first, the School Version of the Assessment of Motor and Process Skills (School AMPS; Munkholm et al., 2010), is an observation of various aspects of school functioning and is performed in the classroom during the child’s regular routine. Similarly, the Do-Eat (Josman et al., 2010) evaluates the child’s performance of three activities of daily living (ADLs), instrumental activities of daily living (IADLs), and school activities and can be administered in any convenient environment. The third performance-based instrument, the Evaluation Tool of Children’s Handwriting–Manuscript (ETCH–M; Brossard-Racine et al., 2012), solely evaluates the performance of handwriting tasks. The School Function Assessment (SFA; Hwang & Davies, 2009), although not performance based, is an ecological test in which observation of participation and performance is conducted in the classroom.
The remaining seven tests are questionnaires, five from the child’s perspective and two from caregiver’s, teacher’s, or therapist’s perspective. Among those examining the child’s perspective, the McDonald Play Inventory (MPI; McDonald & Vigen, 2012) focuses solely on play, the Children’s Leisure Assessment Scale (CLASS; Rosenblum, Sachs, & Schreuer, 2010) evaluates leisure participation, the Children’s Assessment of Participation and Enjoyment and Preferences for Activities of Children (CAPE/PAC; Potvin et al., 2013) assesses activities outside of school, and the Quality of Life in School Questionnaire (QoLS; Weintraub & Bar-Haim Erez, 2009) focuses on quality of life in school. The Child Occupational Self Assessment (COSA; Kramer et al., 2010) is a self-report questionnaire of competence and values of everyday activities. As mentioned earlier, the remaining two questionnaires are completed by caregivers. Similarly, the purpose of the Revised Pediatric Evaluation of Disability Inventory (PEDI–CAT; Kao et al., 2012) is to measure and understand the functional abilities of children and youth and is completed by the caregiver. The final caregiver questionnaire, the Sensory Experience Questionnaire (SEQ; Little et al., 2011), examines sensory-based observed behaviors.
Seven of the studies addressed cultural relevance, and 5 of those stated that the instruments adequately addressed it (Josman et al., 2010; Kao et al., 2012; Kramer et al., 2010; Munkholm et al., 2010; Rosenblum et al., 2010). See Table 3 at the end of this article for details.
Activity and Participation Instruments (2009–2012)
Note. DCD = developmental coordination disorder; SD = standard deviation.
Analyzing the Tests From the Psychometric Point of View
Munkholm et al. (2010) evaluated the School AMPS for significant differential item functioning (DIF) and differential test functioning (DTF) and concluded that the instrument is valid when used in different regions of the world. Other studies examining well-established instruments evaluated the CAPE/PAC (Potvin et al., 2013), COSA (Kramer et al., 2010), MPI (McDonald & Vigen, 2012), and SFA (Hwang & Davies, 2009). These four articles reported research regarding the validity of the instruments, whereas test–retest reliability was reported only for the CAPE/PAC and MPI. The Do-Eat (Josman et al., 2010) and CLASS (Rosenblum et al., 2010) were both in the early stages of development with recommendations for further studies with additional populations and age groups. Nevertheless, both studies established reliability and validity. The current studies are the only ones published related to these instruments, other than the Do-Eat manual. Additionally, Weintraub and Bar-Haim Erez (2009) established the construct validity of the QoLS, yet mentioned no reliability details. In contrast to all of the other studies reviewed, no gender differences were found for the QoLS. Th PEDI–CAT authors described good levels of reliability and validity, although the studies conducted have not yet been published. Both the SEQ (Little et al., 2011) and ETCH–M (Brossard-Racine et al., 2012) are at the initial stages and require further research to establish validity (see Table 4 at the end of this article for more details).
Psychometric Comparison of Activity and Participation Instruments (2009–2012)
Note. ASD = autism spectrum disorder; ICC = intraclass correlation coefficient; SE = standard error.
Discussion
Instrument Development and Testing Studies (2012)
The increase in the number of instrument development and testing studies from 5 in 2011 to 12 in 2012 indicates an increase in the priority of measuring function among articles published in AJOT, which reflects adherence to the agenda of increasing the ability of occupational therapists to provide evidence-based decision making, a priority identified in the AOTA Centennial Vision (AOTA, 2007, p. 614). The larger percentage of higher-level studies and larger cohorts in 2012 indicate an increase in the rigor of instrument development and testing studies published in AJOT in 2012. Funding for almost half of the studies shows the promise of “expanded collaboration for success” in the support provided for instrument development (AOTA, 2007, p. 614).
The three instruments designed for the transitional ages of adolescence and young adulthood would be appropriate for the goal of “longitudinal studies of the participation of children with special needs in their daily lives as they transition through childhood and adolescence into adulthood” (AOTA, 2006, p. 8). The two instruments addressing parent and family occupations support the goal of examination of the “roles and participation of parents, siblings, and other family members within family centered services” (AOTA, 2006, p. 8). Not all of the instruments from the 2012 instrument development and testing studies have the potential for blind testing, but among those for which it would be appropriate, it was underused, thus suggesting an area of focus for future improvement.
Intervention fidelity measures examine how well the intervention is delivered as it was intended (Gearing et al., 2011). Consistency in adherence to intervention fidelity is a key element in guiding practitioners to make evidence-based decisions by disseminating well-designed clinical trials and raising practitioners’, clients’, and third-party payers’ awareness of science-driven and evidence-based innovations (AOTA, 2007). Fidelity ensures that measurement of interventions’ effectiveness is consistent; without it, one cannot claim that they are evidence based (Murphy & Gutman, 2012). Fidelity is critical in research examining intervention effectiveness, but developing fidelity assessments for areas of intervention that do not have consistent protocol agreement, such as sensory integration intervention (May-Benson & Koomar, 2010) or client-centered intervention, is also valuable. This area is an important one for future instrument development to address the goals of the Centennial Vision.
Instrument development and testing studies examined a wider range of mechanisms in 2012 than in 2011, as defined by the expanded ICF categories to examine levels of rehabilitation science (Baum, 2011), which ranged among body function and body structure, functional limitations, activity, and participation. None examined biomedical molecular–cellular or biomedical mechanisms. Environmental mechanisms were not examined, except for one study that looked at activities of mothers and one that looked at participation of families, which are both aspects of a child’s environment, so these assessments could be used to examine children’s and youths’ environments. These assessments addressing mother and family participation support the AOTA Children and Youth Ad Hoc Committee (CYAC) goal of informing practice in “the roles and participation of parents, siblings, and other family members within family centered services” (AOTA, 2006, p. 8).
Activity and Participation Instruments (2009–2013)
The analysis of activity and participation instruments indicates a recent shift in occupational therapy assessment development. Although for many years pediatric assessments focused on developmental outcomes, looking at motor, cognitive, sensory, behavioral, perceptual, speech, and social–emotional abilities with a focus on identifying delays in these developmental domains, only in the past decade has the focus on performance-based or functional outcomes (Majnemer, 2009) grown. Among the assessments examined in this review, three are performance based, evaluating ADLs and IADLs or school performance. From an ecological validity point of view, the performance-based tests more closely evaluate everyday function than other component-based tests and address the need to thoroughly and comprehensively measure function (Doucet & Gutman, 2013). The examination of DIF and DTF by Munkholm et al. (2010) supports the Centennial Vision goal of becoming a globally connected and diverse workforce. Examination of cultural relevance is a similarly important aspect of instrument development that has been found to be adequately addressed in 45% of the studies.
Activity and participation assessments are used to document what children actually do in their natural environments and are therefore helpful in developing individualized goals and treatment plans and for monitoring change over time. Activity and participation outcome domains may include walking; feeding; communication; social interaction; recreation, leisure, and play; education; work and chores; and community and social life (Baum, 2011, Table 1). The assessments in this review focus on many of these domains, suggesting their value in supporting the AOTA CYAC goal of informing practice in “basic and applied scientific studies related to skills, processes, and foundations for childhood and adolescent occupations” (AOTA, 2006, p. 8). Recent activity and participation measurement development, as examined in this review of recent AJOT publications, supports movement of the profession toward the Centennial Vision of becoming an evidence-based and science-driven profession. Increasing the focus on developing performance-based instruments is necessary to measure what the child actually does in his or her natural environment and to inform practice and raise practitioners’, clients’, and third-party payers’ awareness of science-driven and evidence-based innovations that have the potential to improve participation and quality of life. Continued development of questionnaire assessments is also valuable to provide different perspectives that will provide client-centered information for “efficacy studies that examine interventions (efficacy, effectiveness, outcomes development)” (AOTA, 2006, p. 8).
Conclusion
Our analysis of 2012 AJOT articles on children and youth instrument development and testing reveals several important steps to support occupational therapy’s Centennial Vision goal of becoming a “powerful, widely recognized, science-driven, and evidence-based profession with a globally connected and diverse workforce meeting society’s occupational needs” (AOTA, 2007, p. 613). We observed an increase in the number of instrument development and testing studies and in higher-level studies and studies with larger cohorts published in AJOT since 2011. Other positive indicators are funding of almost half of the studies, attention to use of blind testing, and more attention to transition to adult-age assessment. Attention to instruments that examine the other categories of rehabilitation science not addressed in 2012 (i.e., biomedical molecular–cellular, biomedical, and environmental) will further support the Centennial Vision. Inclusion of intervention fidelity measures and increased use of blind testing will also support this goal.
The increased focus on performance-based activity and participation instruments and continued development of questionnaire instruments identified by this review indicate further important forward steps toward addressing the Centennial Vision. Further increase in development of performance-based activity and participation instruments is necessary to further support this goal. Greater attention to examination of cultural relevance, such as DIF and DTF, and other culturally relevant aspects of instruments are important to becoming a globally connected and diverse workforce.
These findings indicate that the profession is moving in the right direction in instrument development and testing. The steps forward that we have observed, along with the suggestions for future instrument development, will help guide practitioners to make evidence-based decisions by disseminating well-designed clinical trials; raise practitioners’, clients’, and third-party payers’ awareness of science-driven and evidence-based innovations that have the potential to improve participation and quality of life; and, ultimately, influence health care decisions for current and future clients.
Footnotes
*
Indicates studies that were reviewed for this article.