Abstract
We extracted 35 articles published between January 2009 and September 2013 in the American Journal of Occupational Therapy (AJOT) that focused on children and youth instrument development and testing, summarized study details and traits of the 37 measures reported in them, and then critiqued the measures. Most of the articles contained Level III evidence (one-group nonrandomized and noncontrolled). The most common types of reliability reported in the articles were internal consistency, test–retest reliability, and interrater reliability; the most frequent types of validity reported were discriminant validity and construct validity. Most pediatric assessment tools were designed for school-age children between ages 5 and 12 yr. The two most common purposes for the assessments were reported as descriptive and discriminative. The continued publication of instruments that measure children and youth participation in meaningful occupations and life roles in home, school, and community environments is recommended.
Keywords
To commemorate the American Occupational Therapy Association’s (AOTA’s) 100th anniversary in 2017, the AOTA board of directors endorsed the implementation of the Centennial Vision, which was “designed to be a road map for the future of the profession” (AOTA, 2007a, p. 613). The Centennial Vision challenges the field of occupational therapy to generate high-quality evidence documenting its effectiveness and impact in six primary practice areas: (1) productive aging; (2) rehabilitation, disability, and participation; (3) children and youth; (4) work and industry; (5) health and wellness; and (6) mental health (Corcoran, 2007).
Gutman (2008b) articulated that achieving the goals identified in AOTA’s Centennial Vision would require adhering to five primary research priorities: (1) providing evidence of the efficacy of occupational therapy practice; (2) testing the reliability and validity of occupational therapy assessment instruments; (3) examining the connection between engagement in occupation and health and well-being; (4) using fundamental and basic research to explain how disability experiences affect people’s participation in community life; and (5) asking and answering topical questions that will provide insights for the occupational therapy profession’s ongoing development and evolution. “The sagacity of the Centennial Vision is that it has charged the profession to produce research needed to support the efficacy of practice in all major practice areas” (Gutman, 2008a, p. 501).
In 2006, AOTA’s Children and Youth Ad Hoc Committee generated a list of 11 research areas they deemed important to inform occupational therapy practice with children and youth (AOTA, 2006) and in turn outlined AOTA’s Centennial Vision for pediatric occupational therapy:
1. Basic and applied scientific studies related to skills, processes, and foundations for childhood and adolescent occupations.
2. Factors that contribute to the success or failure of a specific frame of reference.
3. Both qualitative and quantitative methodologies to address multiple facets of the above.
4. Efficacy studies that examine interventions (efficacy, effectiveness, outcomes development).
5. Theory development and development of conceptual models that promote integration of theory and practice.
6. Empirical studies conducted in context.
7. Translational research providing information on applications to practice, policy development, systems change, program development.
8. The roles and participation of parents, siblings, and other family members within family centered services.
9. Longitudinal studies of the participation of children with special needs in their daily lives as they transition through childhood and adolescence into adulthood.
10. Studies that examine factors central to the children, youth, and their families such as finding a friend, participating in community life, and procuring and maintaining jobs.
11. Studies that examine the emotional and social cost of occupational deprivation and occupational injustice for children and youth such as depression, alcohol and substance abuse, and suicide in disenfranchised youth, and what this is costing emotionally to youth and family as well as to society. (AOTA, 2006, p. 8)
Progress Toward Meeting the Centennial Vision: Children and Youth Instrument Development and Testing
The American Journal of Occupational Therapy (AJOT) is a significant vehicle to assist AOTA in achieving the Centennial Vision’s mandate (AOTA, 2007a). To monitor how AJOT is fulfilling this purpose, Sharon A. Gutman, AJOT’s Editor-in-Chief, commissioned several AJOT authors to write a series of review articles that chart the journal’s progress toward achieving the Centennial Vision. Several of these articles have focused on the identified primary practice area of children and youth. Examples of reviews published in AJOT focusing on this practice area include those written by Brown (2010a), Bendixen and Kreider (2011), Hilton and Smith (2012), and Hilton, Goloff, Altaras, and Josman (2013).
Many of the previous review articles published in AJOT have used the level of evidence hierarchy system developed by the AOTA Evidence-Based Literature Review Project (Lieberman & Scheer, 2002; Sackett, Rosenberg, Gray, Haynes, & Richardson, 1996) to classify articles included in them. Level I involves systematic reviews, meta-analyses, and randomized controlled trials. Level II consists of two-group pretest–posttest designs in which controls are used (e.g., cohort designs, case control studies) and randomization is not. Level III designs involve neither control nor randomization but instead use a one-group pretest–posttest design. Level IV includes single-subject designs, descriptive studies, case series, and case reports. Finally, Level V involves only expert opinion and is not based on any systematic research.
Children and Youth
Brown (2010a) completed a review and critique of 39 articles published in AJOT in 2008 and 2009 that fell under the practice area of children and youth. He found that the most frequent type of research published was basic research, accounting for 38.5% (n = 15) of all studies published on the topic of child and youth practice issues. Instrument development and testing and effectiveness studies were the next two most frequently noted research approaches, accounting for 25.6% (n = 10) and 20.5% (n = 8), respectively. Quantitative studies were the predominant research paradigm used, with 76.9% (n = 30) of the studies dealing with children and youth. Studies using a qualitative approach accounted for only 12.8% (n = 5), and 10.3% (n = 4) of the studies used a mixed-methods approach. When considering the level of evidence of children and youth studies published in AJOT in 2008–2009, Level III articles were the most common, accounting for 46.2% (n = 18). None of the 39 studies published in the 2008 and 2009 volumes of AJOT that focused on children and youth practice examined the link between health and well-being and occupational engagement.
Bendixen and Kreider (2011) conducted a systematic review of AJOT articles that focused on the practice category of children and youth published from 2009 to 2010. They used the International Classification of Functioning, Disability and Health (ICF; World Health Organization [WHO], 2001), ICF: Child and Youth Version (ICF–CY; WHO, 2007), and Positive Youth Development (PYD; Catalano, Berglund, Ryan, Lonczak, & Hawkins, 1999) frameworks to monitor how occupational therapy research was tracking toward AOTA’s Centennial Vision goals. They sorted 46 AJOT articles by research type and categorized them into either the ICF–CY or the PYD framework. Of the 46 articles, 12 reported on instrument development and testing, 15 were classified as basic research, 4 were designated as efficacy studies, and 15 involved evaluating the effectiveness of occupational therapy intervention. The ICF–CY breakdown of the variables that the 46 studies focused on was as follows: Body Functions, 31%; Activity, 31%; Participation, 16%; Environment, 12%; and Personal, 10% (Bendixen & Kreider, 2011). “Most of the effectiveness studies reported in AJOT during 2009–2010 focused on activity-based outcomes such as visual–motor integration, motor skill, feeding, and handwriting. As a whole, most treatment effectiveness studies measured clinical and activity-based outcomes of the intervention” (Bendixen & Kreider, 2011, p. 356). The authors concluded that to best meet the Centennial Vision goals, occupational therapists need to “document changes in children’s engagement in everyday life situations and build the evidence of occupational therapy’s efficacy in facilitating participation” (p. 357).
Hilton and Smith (2012) examined 22 articles published in the 2011 AJOT volume that dealt with the practice area of children and youth. Similar to the approach used by Bendixen and Kreider (2011), Hilton and Smith sorted the articles by type of research (Gutman, 2008b), level of evidence (Lieberman & Scheer, 2002), and the ICF categories expanded by Baum (2011) to incorporate levels of rehabilitation science.
Of the 22 articles analyzed, 10 were intervention effectiveness studies, 5 were instrument development and testing studies, 6 were basic research studies, and 1 was a professional issues study. “The most common evidence level was Level III (36.4%), followed by Level IV (31.8%), Level II (18.2%), and Level I (13.6%)” (Hilton & Smith, 2012, p. e49). The distribution of ICF rehabilitation levels in the 22 articles was as follows: Body Functions and Structures, 50%; Functional Limitations, 32%; Environment, 23%; Participation, 18%; Activity, 14%; and Biomedical, 5% (Hilton & Smith, 2012). Similar to Bendixen and Kreider’s (2011) findings for articles published in 2009–2010, Hilton and Smith (2012) reported that the largest percentage of children and youth articles published in 2011 presented results of intervention effectiveness studies classified as Level III or IV, with the majority focusing on the ICF Body Functions and Structures components. However, they noted that “longitudinal, efficacy, and qualitative studies, as well as studies examining adolescents and the transition to adulthood, were absent from articles in this review and are important areas for future investigation” (Hilton & Smith, 2012, p. e39).
Whitney and Hilton (2013) analyzed 11 intervention effectiveness studies related to children and youth published in the 2012 AJOT volume using evidence level (Lieberman & Scheer, 2002) and Baum’s (2011) expanded ICF categories. They also rated the 11 articles on four additional criteria outlined by Reichow, Volkmar, and Cicchetti (2008): efficacy, effectiveness, comparative effectiveness, and pragmatic methodology. Of the studies, 4 (36.4%) were Level I, 2 (18.2%) were Level II, 3 (27.3%) were Level III, and 2 (18.2%) were Level IV. The studies addressed five of the seven ICF rehabilitation science mechanisms: Body Functions and Structures (n = 3; 27.3%), Functional Limitations (n = 5; 45.5%), Activity (n = 3; 27.3%), Participation (n = 1; 9.1%), and Environment (n = 1; 9.1%). Note that some of the studies fell under more than one of the expanded ICF categories.
Whitney and Hilton (2013) found that 63% of the studies used pragmatic trial methodology (e.g., assessing treatment in context as it is meant to be delivered), and 1 study dealt with comparative effectiveness. They noted that the 11 studies included in this review “showed improvement in their ability to guide practitioners to make evidence-based decisions by increasing the understanding of the pragmatic relevance of intervention and the extent to which intervention promotes participation in childhood and adolescent occupations” (p. e163).
Instrument Development and Testing
Hilton et al. (2013) examined two groups of studies published in AJOT: 12 articles from the 2012 AJOT volume that concentrated on both children and youth and instrument development and testing and 11 AJOT articles published during 2009–2013 that reported on instrument development and testing studies with a specific focus on children and youth activity and participation scales. Hilton et al. wanted to investigate whether these instruments facilitated the production of credible evidence that endorsed the Centennial Vision principles. Among the 12 instrument development and testing studies published in 2012, 7 (58.3%) were Level II and 5 (41.7%) were Level III; 11 of the studies were on instruments that were the subject of the studies on child and youth activity and participation scales published from 2009 to 2013:
School Version of the Assessment of Motor and Process Skills (Fisher, Bryze, Hume, & Griswold, 2007)
Do–Eat (Goffer, Josman, & Rosenblum, 2009)
Evaluation Test of Children’s Handwriting–Manuscript (Amundson, 1995)
School Function Assessment (Coster, Deeney, Haltiwanger, & Haley, 1998)
McDonald Play Inventory (McDonald, 1987)
Children’s Leisure Assessment Scale (Rosenblum, Sachs, & Schreuer, 2010)
Children’s Assessment of Participation and Enjoyment/Preferences for Activities of Children (King et al., 2004)
Quality of Life in School (Weintraub & Bar-Haim Erez, 2007)
Child Occupational Self Assessment (Keller, Kafkes, Basu, Federico, & Kielhofner, 2005)
Pediatric Evaluation of Disability Inventory (Haley, Coster, Dumas, Fragala-Pinkham, & Moed, 2012).
Sensory Experiences Questionnaire (Baranek, David, Poe, Stone, & Watson, 2006).
Hilton et al. (2013) reported an increase in the number of instrument development and testing studies overall: More than half of the studies involved higher levels of evidence, external funding, and larger sample sizes. They also reported that blinding was a research method feature. Hilton et al. noted that “these findings indicate that the profession is moving in the right direction in instrument development and testing” (p. e34).
Meeting the Centennial Vision for Instrument Development and Testing in the Practice Area of Children and Youth
The occupational therapy profession worldwide seeks to consolidate theoretical underpinnings that explain, justify, validate, and develop existing models as well as to develop psychometrically sound instruments that will measure and substantiate occupational therapy practice. The end goal of achieving an evidence-based profession in occupational therapy is reliant on the development of psychometrically robust instruments, tests, and measures (Brown, 2010b). AOTA’s Centennial Vision itself charges the profession to produce both high-quality research evidence and substantive impact within practice. Consequently, occupational therapists having access to and being conversant with a range of psychometrically vigorous assessment tools and scales is crucial to achieving this aim (Coster, 2006a, 2008).
Assessment tools developed within occupational therapy have four main purposes: descriptive, discriminative, predictive, and evaluative (Brown, 2012; Fawcett, 2007). A tool’s purpose will influence the ensuing need for psychometric development and evaluation. Any research investigating an instrument’s psychometric properties should state the assessment’s purpose to guide potential clinicians or researchers who may want to use the instrument. Descriptive tools provide “information which describes the person’s current functional status, problems, needs and/or circumstances” (Fawcett, 2007, p. 98). Descriptive assessments provide a baseline or way of describing a child’s functional status to plan intervention. Examples of descriptive assessments include the Child Occupational Self Assessment (Keller et al., 2005), Assessment of Life Habits for children (Noreau et al., 2007), Paediatric Activity Card Sort (Mandich, Polatajko, Miller, & Baum, 2004), and the Self-Observation and Report Technique (Rintala et al., 1984).
Discriminative assessments are usually norm based and distinguish between individuals or groups on some characteristic or underlying dimension (Fawcett, 2007). Examples include any norm-based developmental assessment such as the Movement Assessment Battery for Children–Second Edition (Henderson, Sugden, & Barnett, 2007), the Peabody Developmental Motor Scales–Second Edition (Folio & Fewell, 2000), or the Vineland Adaptive Behavior Scales–Second Edition (Sparrow, Cicchetti, & Balla, 2005). Discriminative assessments seek to accurately determine whether a child or youth is within a specified range of typical development or performance. The assessment seeks to discriminate between children who may or may not be presenting with suspected occupational performance challenges. Consequently, the discriminant validity of such assessments becomes an extremely important part of their psychometric evaluation.
Predictive assessments “classify people into pre-defined categories of interest in an attempt to predict an event or functional status in another situation” (Fawcett, 2007, p. 99). An example is the Sensory Profile (Dunn, 1999), which predicts sensory responsiveness in other situations and also categorizes the child accordingly. Moreover, children’s performance on the Beery–Buktenica Developmental Test of Visual–Motor Integration (Beery, Buktenica, & Beery, 2010) in kindergarten has been shown to be predictive of future academic performance in subsequent grades (Kurdek & Sinclair, 2000). The Functional Independence Measure for Children (WeeFIM; Uniform Data System for Medical Rehabilitation, 2006) has also demonstrated the ability to predict future functional outcomes of children with neurological injuries (Msall et al., 1994). Consequently, the predictive validity of assessment tools and scales that are used for this specified purpose is crucial.
The fourth type of purpose is evaluative. Evaluative assessments detect the magnitude of change over time within one person or a group of people after an intervention or event (Fawcett, 2007). Evaluative assessments may be designed with the purpose of measuring changes in performance, competence, or satisfaction over time from the client’s perspective and are known as outcome measures. Examples include the Canadian Occupational Performance Measure (Law et al., 2005) and the Perceived Efficacy and Goal Setting System (Missiuna, Pollock, & Law, 2004). Other tools are designed to detect both development and progress over time, such as the Pediatric Evaluation of Disability Inventory (Haley, Coster, Ludlow, Haltiwanger, & Andrellos, 1992). Consequently, evaluative tests require sound test–retest and interrater reliability before thorough investigations of their responsiveness to change over time. For example, the Melbourne Assessment of Unilateral Upper Limb Function second edition (MAUULF–2; Randall, Johnson, & Reddihough, 2010) had established test–retest, intrarater, and interrater reliability before evaluation of its sensitivity to change over time. With such established properties as an outcome measure, the MAUULF–2 has been used to measure change in the functional status of children after occupational therapy intervention (Wallen, O’Flaherty, & Waugh, 2007) and botulinum toxin A (Rameckers, Duysens, Speth, Vles, & Smits-Engelsman, 2010; Speth, Leffers, Janssen-Potten, & Vles, 2005).
It is important to note that many assessment tools and scales have dual or multiple purposes. The usefulness of an instrument or tool in occupational therapy practice and research increases when it has two or more purposes. Examples include the Movement Assessment Battery for Children–Second Edition, which discriminates between children with and without motor differences and predicts whether a child exhibits the signs of developmental coordination disorder. Therefore, assessment tools with more than one purpose, as well as substantiated and appropriate psychometric evaluation, are the most robust instruments available to both clinicians and researchers in the field. The development of such instruments, scales, and tools is imperative for the future of occupational therapy in the evidence-based health care and education arena.
Finally, given the assessment tool’s purpose, it is important to describe the methods involved in establishing its psychometric body of evidence, namely its reliability and validity. Reliability refers to a test’s ability to collect data on a consistent basis, and validity refers to the available body of evidence indicating how well test items represent the construct they claim to assess. Specific subtypes of reliability include internal consistency, correlations between subscales and total scale score, test–retest reliability, intrarater reliability, interrater reliability, split-half reliability, and alternate-form reliability (Mertler, 2007). Usually, a test needs to have established validity before its reliability can be investigated; however, preliminary reliability scores are frequently reported (e.g., internal consistency, split-half reliability) before formal construct validity evidence is published. In other words, a test’s items can appear to reliably measure a construct without evidence that the items adequately represent the construct being assessed. Subtypes of validity include content validity, criterion-related validity, predictive validity, convergent validity, divergent validity, discriminant validity, and factorial validity (Brown, 2012; Fawcett, 2007).
In this article, we review 35 articles published in AJOT between January 2009 and September 2013 that were in the practice areas of children and youth and instrument development and testing. The purpose of this review was to describe these articles and their level of evidence (Gutman, 2008a). We also critique the specific pediatric assessment instruments cited in these articles, which provide a gauge of how well AJOT is meeting the challenges set forth in AOTA’s Centennial Vision.
Method
The AJOT Editor-in-Chief identified 35 articles published in AJOT between January 2009 and September 2013 that addressed both children and youth and instrument development and testing. We used a content analysis approach to summarize the characteristics of the 35 articles (Table 1).* Table 2 provides specific details about the pediatric instrument being investigated. Table 3 profiles and critiques the pediatric assessment instruments included in the 2009–2013 AJOT articles. Among other details, Table 3 includes the stage of instrument development and evaluation, which we based on DeVellis’s (2003) 10 stages of scale development:
Content domain specification (literature review, interviews with relevant audience, focus group) to ensure existence of construct
Item pool generation or reports on initial development of items
Content validity evaluation (content experts, relevant audiences) to ensure representativeness
Questionnaire development and evaluation and evaluation of scoring
Pilot study questionnaire
Sampling and data collection
Reliability assessment
Validity or dimensionality assessment (factorial, dimensionality, convergent, divergent, discriminant: statistical analysis and statistical evidence of construct)
Wider application of scale to new populations
Evaluation of scale’s measurement properties by others than the scale’s authors.
Summary of Articles Describing Development and Testing of Instruments for Children and Youth, Published in the American Journal of Occupational Therapy, January 2009–September 2013
Note. ADLs = activities of daily living; AMPS = Assessment of Motor and Process Skills; ANOVA = analysis of variance; ASD = autism spectrum disorder; CI = confidence interval; COP = Comprehensive Observations of Proprioception; DCD = developmental coordination disorder; DIF = differential item functioning; ETCH = Evaluation Tool of Children’s Handwriting; HPAS = Health Promoting Activities Scale; ICC = intraclass correlation coefficient; LPP = Life Participation for Parents; MANOVA = multivariate analysis of variance; OT = occupational therapist; PEDI = Pediatric Evaluation of Disability Inventory; ROC = receiver operating characteristic; SVMPT = Slosson Visual–Motor Performance Test–Revised; TVMI = Test of Visual–Motor Integration; VMI = Developmental Test of Visual–Motor Integration; WCPA = Weekly Calendar Planning Activity.
Suggested citation: Brown, T., & Bourke-Taylor, H. (2014). Centennial Vision—Children and youth instrument development and testing articles published in the American Journal of Occupational Therapy, 2009–2013: A content, methodology, and instrument design review (Table 1). American Journal of Occupational Therapy, 68, e154–e216. http://dx.doi.org/10.5014/ajot.2014.012237
Summary of Instruments for Children and Youth Described in Articles Published in the American Journal of Occupational Therapy, January 2009–September 2013
Note. DCD = developmental coordination disorder; ICF = International Classification of Functioning, Disability and Health (World Health Organization, 2007); PEO = Person–Environment–Occupation model.
Suggested citation: Brown, T., & Bourke-Taylor, H. (2014). Centennial Vision—Children and youth instrument development and testing articles published in the American Journal of Occupational Therapy, 2009–2013: A content, methodology, and instrument design review (Table 2). American Journal of Occupational Therapy, 68, e154–e216. http://dx.doi.org/10.5014/ajot.2014.012237
Critique of Children and Youth Instrument Development and Testing Articles Published in the American Journal of Occupational Therapy, January 2009–September 2013
Note. Level I = systematic reviews, meta-analyses, and randomized controlled trials; Level II = two-group nonrandomized pretest–posttest designs (e.g., cohort designs, case control studies); Level III = one-group nonrandomized, noncontrolled trials; Level IV = single-subject designs, descriptive studies, and case series; Level V = expert opinion, case study, not based on systematic research methods. DCD = developmental coordination disorder; ICF = International Classification of Function, Disability and Health; MOHO = Model of Human Occupation; PEDI = Pediatric Evaluation of Disability Inventory; PEO = Person–Environment–Occupation model; QoL = quality of life.
Suggested citation: Brown, T., & Bourke-Taylor, H. (2014). Centennial Vision—Children and youth instrument development and testing articles published in the American Journal of Occupational Therapy, 2009–2013: A content, methodology, and instrument design review (Table 3). American Journal of Occupational Therapy, 68, e154–e216. http://dx.doi.org/10.5014/ajot.2014.012237
We classified each study according to the level of evidence hierarchy system used by the AOTA Evidence-Based Literature Review Project (Lieberman & Scheer, 2002).
The Person–Environment–Occupation (PEO) model is a prominent occupational therapy model that drives client-centered practice and explains a person’s performance as the dynamic interaction among these three components (Law & Baum, 2005; Law et al., 1996). The person characteristics include physical, cognitive, affective, and other characteristics and capabilities. Environmental aspects are the social, cultural, physical, and institutional environment around the person, and occupations are classified as self-care, productivity, or leisure and play. The PEO model concepts operationalized in the occupational therapy assessment tools are listed in Table 4.
Classification of Instruments in the Children and Youth Instrument Development and Testing Articles Published in the American Journal of Occupational Therapy, January 2009–September 2013, by Practice Model
Note. ICF = International Classification of Functioning, Disability and Health (World Health Organization, 2001, 2007); PEO = Person–Environment–Occupation model (Law et al., 1996).
Suggested citation: Brown, T., & Bourke-Taylor, H. (2014). Centennial Vision—Children and youth instrument development and testing articles published in the American Journal of Occupational Therapy, 2009–2013: A content, methodology, and instrument design review (Table 4). American Journal of Occupational Therapy, 68, e154–e216. http://dx.doi.org/10.5014/ajot.2014.012237
The ICF provides an international framework that transcends professional boundaries, cultures, and countries. The ICF acknowledges and conceptualizes the influence of environmental (human and physical) and personal (i.e., characteristics and background of the person’s life such as age, gender, education level, socioeconomic status) factors on the health, capabilities, and capacity of the person to engage in activities and participate in opportunities available in his or her home and community (WHO, 2001). Environmental factors such as physical and human resources and personal factors such as age, gender, culture, and the person’s values, interests, life roles, habits, and self-efficacy provide a highly influential backdrop that affects what a person does and how a person interacts with the world around him or her. The ICF has had a significant impact on the development of assessment tools within and outside the field of occupational therapy. We used the ICF to categorize the assessment tools from the 35 AJOT articles (see Table 4).
To summarize, the assessment tools for children and youth included in the 35 AJOT articles published between January 2009 and September 2013 are summarized in Table 2, critiqued in Table 3, and classified using the PEO model and ICF framework in Table 4.
Results
We reviewed 35 journal articles that addressed some aspect of the psychometric development of 37 occupational therapy assessment tools or classification scales.
Two articles reported information about the Weekly Calendar Planning Activity (Toglia & Berg, 2013; Weiner, Toglia & Berg, 2012). Two articles were also published about the Comprehension Observations of Proprioception (Blanche, Bodison, Chang & Reinoso, 2012; Blanche, Reinoso, Chang, & Bodison, 2012). Similarly, the Evaluation Tool of Children's Handwriting was the subject of two articles (Brossard-Racine, Mazer, Julien & Majnemer, 2012; Duff & Goyen, 2010). Brown, Unsworth, and Lyons (2009) reported details of four visual–motor integration tests: Test of Visual–Motor Integration, Developmental Test of Visual–Motor Integration, Test of Visual–Motor Skills–Revised, and Slosson Visual-Motor Performance Test (SVMPT). Tsai, Lin, Liao, and Hsieh (2009) published an article about two visual perceptual tests: the Motor-Free Visual Perception Test–Revised and Test of Visual–Perceptual Skills–Revised. Kuijper, van der Wilden, Ketelaar, and Gorter (2010) reported about the Manual Ability Classification System and the Pediatric Evaluation of Disability Inventory (PEDI) while Kao, Karamer, Liljenquist, Tian, and Coster (2012) reported about a newer version of the PEDI referred to as the Pediatric Evaluation of Disability Inventory–Computer Adaptive Test (PEDI–CAT). However, only the PEDI–CAT was reported about in detail. This accounted for the difference between the number of published articles reported in Table 1 and the number of published tests described in Tables 2–4.
All tools were developed and evaluated by occupational therapists, except for the Manual Ability Classification System (Eliasson et al., 2010, discussed in Kuijper, van der Wilden, Ketelaar, & Gorter, 2010), which was evaluated by physicians. The research projects described in the articles were international (see Table 1): American (18 articles, 51.4%), Israeli (6 articles, 17.1%), Australian (5 articles, 14.3%), Finnish (1 article, 2.9%), Swedish (2 articles, 5.7%), Taiwanese (1 article, 2.9%), Belgian (1 article, 2.9%), and Canadian (1 article, 2.9%). The life stage of the participants varied, with the majority of participants falling into the following age ranges: preschool (≤4 yr; 5 articles, 14.3%), school age (5–14 yr; 22 articles, 62.9%), and youth (15–18 yr; 4 articles, 11.4%). Three studies (8.6%) were dedicated to families, and 1 study (2.9%) involved only therapists.
From January 2009 to September 2013, AJOT published articles outlining the initial measurement properties of nine new scales: the Health Promoting Activities Scale (Bourke-Taylor, Law, Howie, & Pallant, 2012), Adolescents and Adults Coordination Questionnaire (Saban, Ornoy, Grotto, & Parush, 2012), Quality of Life in School (Weintraub & Bar-Haim Erez, 2009), Children’s Leisure Assessment Scale (Rosenblum et al., 2010), McDonald Play Inventory (McDonald & Vigen, 2012), Sense and Self-Regulation Checklist (Silva & Schalock, 2012), Comprehensive Observations of Proprioception (Blanche et al., 2012), Do–Eat (Josman, Goffer, & Rosenblum, 2010), and Life Participation for Parents (Fingerhut, 2013).
The majority of studies described evaluation of the instrument’s reliability, validity, or both (see Table 1). A total of 19 (54.3%) articles dealt with both the reliability and the validity of an instrument, whereas 7 studies focused only on its reliability and 9 focused only on its validity. The specific subtypes of reliability assessed were internal consistency (n = 15), test–retest reliability (n = 15), intrarater reliability (n = 2), and interrater reliability (n = 10). The types of validity assessed were face validity (n = 2), content validity (n = 6), criterion-related validity (n = 1), discriminant validity (n = 15), construct validity (n = 19), concurrent validity (n = 7), and factorial validity (n = 4). No longitudinal studies investigated predictive validity. Only 1 study reported details of an instrument’s responsiveness to change (sensitivity) over time. All articles provided advancement of the psychometric properties of the instruments and measures on which they reported. Two articles were primarily authored by physicians (Kuijper et al., 2010; Silva & Schalock, 2012), and the others were all authored by occupational therapists (n = 33).
Thirty-five assessments are described in Table 2. The assessments are based on a variety of practice models, theories, and frameworks, with some instruments incorporating more than one: cognitive–behavioral (n = 2), occupational performance (n = 6), ecological (n = 6), PEO (n = 6), play (n = 2), perceptual–motor (n = 6), Model of Human Occupation (MOHO; n = 5), motor control (n = 2), sensory processing (n = 3), ICF (n = 8), biopsychosocial (n = 1), the Occupational Therapy Practice Framework: Domain and Process (2nd ed.; AOTA, 2008; n = 4), occupational adaptation (n = 2), sensory integration (n = 2), and biomechanical (n = 1).
In terms of client age groups, 18 scales are aimed at children from birth to age 4, 28 are designed for children between ages 5 and 12, 10 are meant for use with adolescents between ages 13 and 17, and 5 are designed for clients age 18 and older. Note that some of the scales covered more than one age group. For example, the Adolescents and Adults Coordination Questionnaire can be used with adolescents and adults, and the Assessment of Children’s Hand Skills can be used with children ages 3–12. The scores for the assessments are derived from a variety of perspectives: Eight scales are self-report, 10 are third party (usually completed by parents or caregivers who know the child or adolescent), and 22 are based on professionals rating the client’s performance on the basis of a set of criteria. Some instruments use a combination of third-party input and professional ratings.
Regarding DeVellis’s (2003) stages of scale development, we noted the following combinations: Stage 8 validity studies, n = 5; Stage 7 reliability studies, n = 5; both Stage 7 and 8, n = 10. One study was Stage 3 (content validity evaluation), and 3 studies were both Stages 7 and 10 (reliability studies) completed by those other than the scale developers. Thirteen studies were Stage 8 and 10 validity evaluations completed by researchers other than the original scale authors. No studies investigated responsiveness to change over time or changes in ratings over time to detect sensitivity.
No studies contributed Level I evidence. Thirty-three articles described studies at Level III (one-group nonrandomized and noncontrolled), 1 study was Level II (two-group nonrandomized), and 1 study was Level IV (single-subject or descriptive study; see Table 3). The purposes of most assessments were clearly multifaceted, with only 2 assessments seemingly designed to meet only one purpose. Twenty-nine assessments were clearly under development as descriptive measures, 32 tools were being developed as discriminative measures, 21 tools were being designed as predictive measures, and 27 tools were clearly under development as evaluative measures.
As shown in Table 4, we classified the tests and measures under the components of the PEO model and the ICF (i.e., Body Functions and Structures, Personal Factors, Environmental Factors, Activity, and Participation). Under the PEO model classification, 34 (91.9%) of the tests and measures included the Person, 12 (32.4%) included the Environment, and 19 (51.4%) included the Occupation. The ICF components were distributed as follows: Body Functions and Structures, n = 25 (67.6%); Personal Factors, n = 8 (21.6%); Environmental Factors, n = 10 (27.0%); Activity, n = 18 (48.6%); and Participation, n = 12 (32.4%).
Discussion
As evidenced by this review, considerable research activity has occurred within the field of occupational therapy to develop and evaluate tests and measures related to children and adolescents. In line with previous commentary, the occupational therapy profession remains committed to child- and youth-focused studies (Brown, Rodger, & Brown, 2005; Case-Smith, 2007; Gutman, 2008b). Before the period of time selected for this current review, Gutman (2008b) reported that instrument development and testing was a common research approach used in a large proportion (26%) of the studies published in AJOT in 2008 and 2009. Moreover, the current review concurs with results from other articles published in AJOT focusing on the children and youth practice area (Brown, 2010a; Bendixen & Kreider, 2011; Hilton & Smith, 2012; Hilton et al., 2013).
Characteristics of Children and Youth Instrument Development and Testing Articles
As noted in the Results section, the majority of the children and youth instrument development and testing articles (48.6%) were completed in the United States. However, researchers from several other countries also published articles dealing with pediatric assessment tools, including Israel (6 articles, 17.1%), Australia (5 articles, 14.3%), and Sweden (2 articles, 5.7%), thus demonstrating that AJOT has both a domestic and an international scope in articles about assessment tools aimed at children and adolescents.
The majority of pediatric assessment tools were designed for children between ages 5 and 12 (80.0%), and fewer were aimed at children ages birth–4 (51.4%) or youth ages 13–17 (28.6%). Several instruments were designed for use by more than one age group. For example, the Occupational Self Assessment can be used with adolescents and adults, and the School Version of the Assessment of Motor and Process Skills, Developmental Test of Visual–Motor Integration, and Evaluation of Social Interaction can be used with preschool-age and school-age children. Although identified as a priority in the Centennial Vision, youth have received substantially less attention in the sphere of scale and instrument development. The findings of this review suggest that future directions might emphasize instrument development for youth who receive occupational therapy services.
Three instruments focused on the perspective of families (Family L.I.F.E. [Looking Into Family Experiences], Life Participation for Parents, and the Health Promoting Activities Scale). AOTA has previously recognized the importance of caregivers and families in the lives of people with disabilities (AOTA, 2007b). The Children and Youth Ad Hoc Committee (AOTA, 2006) identified the need for research that would inform occupational therapy practice to address the roles, occupations, and participation of parents, siblings, and families of children and youth with disabilities. The findings of the current review suggest that, with only 3 published studies addressing instruments in this area of occupational therapy research and practice, further attention is indicated to successfully achieve AOTA’s Centennial Vision.
Of the 35 articles published in AJOT from January 2009 to September 2013, 10 reported the initial measurement properties of 9 new scales. Four of the 9 scales (McDonald Play Inventory, Sense and Self-Regulation Checklist, Comprehensive Observations of Proprioception, and Life Participation for Parents) were by U.S. authors, and 5 were generated by international authors: 4 from Israel (Adolescents and Adults Coordination Questionnaire, Quality of Life in School, Children’s Leisure Assessment Scale, and Do–Eat) and 1 from Australia (Health Promoting Activities Scale).
Practice Models, Theories, and Frameworks on Which Children and Youth Assessment Tools Are Based
The pediatric assessment tools included in the AJOT articles are based on a number of different practice models, theories, and frameworks. The most frequent ones were occupational performance (17.1%), ecological (17.1%), PEO (17.1%), perceptual–motor (17.1%), Model of Human Occupation (14.3%), ICF (23.0%), and the Occupational Therapy Practice Framework (11.4%). Several of the assessment tools were based on more than one practice model or theory, and several did not explicitly state on which practice theory they were based; thus, it had to be inferred.
Stage of Instrument Development and Level of Evidence of Children and Youth Assessment Tools
The extent of psychometric evaluation that an instrument has undergone contributes substantially to clinicians’ and researchers’ confidence in its utility (Fawcett, 2007). We used DeVellis’s (2003) stages of instrument development to evaluate how rigorous and advanced studies were in the psychometric evaluation of their measurement properties. Of the studies, 54.2% (n = 19) reported details on both the reliability and the validity of pediatric tests and measures, whereas 20.0% (n = 7) focused solely on instrument reliability and 25.7% (n = 9) focused solely on instrument validity. Given that 10 articles published the initial measurement properties of 9 new pediatric assessment tools, it is not surprising that more than half of the articles reported both reliability and validity data. The most common combined stages of instrument evaluation reported by external authors were Stages 8 (validity) and 10 (evaluation of scale’s measurement properties by other than the scale’s authors; 37.1%). It is commendable that Stages 8 and 10 were most frequently reported, because it provides evidence that researchers other than the original test authors are critically evaluating the validity of pediatric instruments used by occupational therapists. This evaluation contributes to the overall body of psychometric knowledge about these assessments.
Reliability studies indicate the degree of consistency or repeatability that a test’s scores achieve between Time 1 and Time 2. Researchers often report several subtypes of reliability. The most common types of reliability reported in the AJOT articles were internal consistency (42.9%, n = 15), test–retest reliability (42.9%, n = 15), and interrater reliability (28.6%, n = 10). No information was reported about split-half reliability or alternate-form reliability of the 37 instruments.
Evaluations of validity indicate how well a test measures the ability, trait, or construct it purports to measure. The most common types of validity reported in the AJOT articles were discriminant validity (48.6%, n = 17) and construct validity (54.3%, n = 19). Other less frequently reported types included face validity (5.7%, n = 2), content validity (17.1%, n = 6), criterion-related validity (2.9%, n = 1), concurrent validity (20.0%, n = 7), and factorial validity (11.4%, n = 4). None of the studies were longitudinal studies that involved investigating the instrument’s predictive validity.
Although all articles provided advancement in the psychometric data for the instruments and measures they reported on, further studies are needed at the later stages of an instrument’s psychometric development, namely evaluation of scoring and constructs within the assessment tools using advanced statistical techniques such as Rasch analysis or structural equation modeling. Evaluation of instruments and wider application to similar, different, or contrasting populations from scale developers and other unrelated researchers are needed. Only 1 study (3%) reported details of an instrument’s responsiveness to change (sensitivity) over time, and no studies investigated changes in ratings over time to detect test sensitivity.
Many of the assessments reported in the 35 articles were designed to meet more than one purpose. For example, 29 assessments were clearly designed as descriptive measures, 32 were developed as discriminative measures, 21 could be used as predictive measures, and 27 were potential evaluative measures. None of the articles provided comprehensive and definitive evidence that the assessment was entirely capable of measuring a characteristic or issue for the purpose for which it was intended. Rather, all articles provided substantial evidence that the tools were contributing to a growing body of evidence supporting their psychometric properties.
No studies contributed Level I evidence. Only 1 study was Level II (two-group nonrandomized), and 1 study was Level IV (single-subject or descriptive study). Most of the articles on pediatric assessment tool articles (94.3%) were at Level III (one-group nonrandomized and noncontrolled).
In sum, analysis of the type of instruments under development, stage of psychometric evaluation, and level of evidence illuminates future directions for occupational therapy researchers who are involved in ongoing development of psychometrically sound and clinically useful tools. First, the type of instrument is an important issue in the ongoing psychometric evaluation of a tool, as well as of its clinical utility. Ensuing evaluation of a tool must systematically confirm the purpose for which it was developed. Considering that 21 (56.8%) of the instruments appeared to be under development for predictive purposes and 27 (73.0%) were under development for evaluative purposes, relatively few studies investigated aspects of the instruments that would confirm their utility for these purposes. Worldwide, occupational therapy must defend and substantiate the need, impact, and efficacy of its interventions; therefore, psychometrically robust predictive and evaluative instruments are urgently needed.
Second, existing instruments require ongoing and higher stage evaluation to provide evidence of their measurement properties. Studies that complete evaluation at Stages 7–10 are important and are featured in the articles published in AJOT in this review period. However, ongoing evaluation is needed and might be published in future issues of AJOT. Finally, in addition to ongoing psychometric evaluation of instruments is the need for more rigorous research designs that produce higher levels of evidence. Considering that the highest level of evidence in the 35 articles was Level II (Kao, Kramer, Liljenquist, Tian, & Coster, 2012), researchers in the field would be well advised to consider research designs that contribute to higher level evidence and publications. Such studies are more likely to require larger cohorts and funding levels.
PEO and ICF Classification of Children and Youth Assessment Tools
The tests and measures reported in the 35 articles were classified under the components of one practice model, the PEO model, and one practice framework, the ICF. Under the PEO model, 97.1% (n = 34) of the tools included the Person, 34.3% (n = 12) involved the Environment, and 54.3% (n = 19) included the Occupation. In other words, nearly all the pediatric assessments published in AJOT included aspects of the person from the PEO model.
The ICF components were distributed as follows: Body Functions and Structures, 71.4% (n = 25); Personal Factors, 22.9% (n = 8); Environmental Factors, 25.7% (n = 9); Activity, 51.4% (n = 18); and Participation, 31.4% (n = 11). By far the most common component of the ICF that was assessed with the pediatric instruments was Body Functions and Structures, with Activity also being assessed by more than 50% of the instruments. These findings suggest that the majority of instruments currently published in AJOT and under development for use with children continue to focus on traditional reductionist approaches to assessment involving components of children’s body functions and structures (as defined by the ICF).
In their review of 46 child- and youth-oriented articles published in AJOT during 2009–2010, Bendixen and Kreider (2011) found that the distribution of the articles over the ICF domains were as follows: Body Functions and Structures, 31%; Personal Factors, 10%; Environmental Factors, 12%; Activity, 31%; and Participation, 16%. Of the 46 articles in the Bendixen and Kreider article, 12 focused on children and youth instrument development and testing. These 12 articles were classified in the following ICF categories: Body Functions and Structures, 25%; Personal Factors, 0%; Activity, 17%; and Participation, 58%. Bendixen and Kreider noted that “the higher levels of testing and development of Participation-level measures implies facilitation of the much-needed measurement of occupational performance in real-world life situations” (p. 356).
Similar to other reviews of AJOT publications, we found that although substantial attention was paid to PEO or ICF concepts of Activity and Occupation, a particular paucity of instruments under development and review were operationalized to evaluate either Participation or Environment. Participation has long been an important aspect of the assessment and intervention process for children with disabilities and other occupational therapy clients (King et al., 2003; Law, 2002). Growing evidence has supported the importance and influence of Environmental Factors, rather than issues around Body Functions and Structures, with regard to the Participation of children and youth in life situations at home, at school, and in the community (Coster et al., 2013; Fauconnier et al., 2009; Heah, Case, McGuire, & Law, 2007; Law et al., 2004).
In response to evidence suggesting that children and youth with disabilities require interventions aimed at altering environmental and contextual factors, occupational therapy as a profession has commenced developing instruments that assess and measure participation (such as the Participation and Environment Measure for Children and Youth; Coster et al., 2012, 2013) or an aspect of environmental support that enables participation (such as the Assistance to Participate Scale; Bourke-Taylor, Law, Howie, & Pallant, 2009; Bourke-Taylor & Pallant, 2013). The challenges of developing instruments that measure participation, involvement, enjoyment, and performance of children and youth with disabilities have been described (Coster, 2006b; Coster & Khetani, 2008) and continue to be discussed inside and outside of the profession (Granlund, 2013; King, 2013; Raghavendra, 2013). Finally, given that the AOTA Child and Youth Ad Hoc Committee (AOTA, 2006) highlighted the importance of research investigating the participation or consequences of lack of participation for children and youth in 4 of its 11 targeted areas, we recommend future support for the development and evaluation of measures of participation in future AJOT issues.
Limitations
Limitations of this review include that the classification of the 35 articles includes both subjective and objective judgment on our part. We did complete our ratings independently and then compared them to minimize influencing each other. Differences of opinion were negotiated between us until we reached a consensus.
We evaluated 37 instruments from the perspective of the information provided in the 35 articles. We did not otherwise confirm facts and data because the aim of this article was to summarize, synthesize, and comment on the research published in AJOT to date. Therefore, the descriptions of some instruments may have limitations and minor errors because we did not review the original test manuals, nor did we conduct a literature search to locate, peruse, and evaluate research published in other journals or manuals. Consequently, clinicians and researchers using this article to determine the current status of development of individual tools are encouraged to perform literature searches for up-to-date subsequent studies and contact the authors of the instrument. Researchers and clinicians are also advised to liaise with and consult colleagues about the clinical utility and practicalities of different instruments in their work setting and for their client group.
Recommendations for Occupational Therapy Practitioners
Occupational therapy practitioners who work with children and youth have both the need and the professional responsibility to access psychometrically vigorous tests, measures, and instruments. As Brown (2009) asserted, “If we [occupational therapists] are using assessment tools that do not have strong psychometric properties, then the test results that we use to inform our clinical reasoning and intervention planning are not sound or valid either” (p. 519). Practitioners need access to a variety of tools depending on the client group (specific disability, age group, ethnicity, etc.), individual therapists’ points of reference (MOHO, PEO, perceptual–motor, the Framework, cognitive–behavioral, etc.), and the practice setting (early intervention, school, private practice, rehabilitation setting, youth-oriented service, mental health community service for children and adolescents, etc.). This review demonstrates that a diverse set of assessment tools to address practitioners’ needs is being developed and evaluated.
AJOT has published studies that demonstrate the ongoing development of a substantial number of occupational therapy–specific assessment tools. Clinicians may feel confident that researchers and clinicians are working in collaboration to increase the number of psychometrically valid, reliable, sensitive, descriptive, discriminative, predictive, and evaluative tools. However, clinicians have the responsibility to use instruments for the purposes for which they are intended and to understand what validity and reliability studies have been completed and what still needs to be completed. For example, it is imperative that a test designed to be descriptive and discriminatory be used for such purposes and not as an evaluative or predictive test. Researchers and clinicians alike need adequate knowledge of assessment tools to critically evaluate research such as the AJOT articles included in this review. Moreover, clinicians might be encouraged to become involved in research or collaborate with researchers in the field to promote development of tests and measures that are clinically useful, practical, responsive, reliable, and valid.
Recommendations for Researchers
Researchers in the field of occupational therapy might be commended for taking charge of a discipline-specific instrument development and classification system (e.g., Manual Ability Classification System). The results of this review indicate that more work remains to be done, specifically in the areas of providing more rigorous evaluation of tools to obtain higher levels of evidence and evaluating existing tools in the field to provide reliability and validity studies that are conducted by experts other than the authors. Similarly, wider application of the tools and instruments to other populations (e.g., age, diagnosis, disability) is essential to inform practitioners about their clinical utility among other children and youth.
Other recommendations were described in the discussion and include the need for continued development of instruments that will measure salient aspects of the environment as well as operationalizing and measuring participation for children and youth who receive occupational therapy services. Within the profession, a need also exists for further development of self-report or rated scales that are implicitly client centered and for outcome measures that are otherwise described as evaluative scales or instruments. Moreover, we suggest that occupational therapy researchers use the ICF–CY (WHO, 2007) as a point of reference when developing new assessment tools because it more specifically focuses on the participation and function of children and adolescents than the initial version of the ICF.
Summary and Conclusion
We extracted and summarized 35 articles published in AJOT between January 2009 and September 2013 that focused on the Centennial Vision categories of both children and youth and instrument development and testing. The articles addressed some aspect of the psychometric development of 37 occupational therapy assessments or classification scales.
The majority of the articles contained Level III evidence. The most common types of reliability reported in the articles were internal consistency, test–retest reliability, and interrater reliability, and the most common types of validity reported were discriminant validity and construct validity. Most of pediatric assessment tools were designed for children between ages 5 and 12. The practice models and theories that the pediatric assessment tools were most frequently based on were occupational performance, ecological, PEO, perceptual–motor, MOHO, ICF, and the Framework. Many of the assessments were designed to meet more than one purpose; however, the two most common purposes were descriptive and discriminative.
We recommend seven preferences for future research articles addressing the Centennial Vision: (1) increased instrument development addressing younger children, youth, and families; (2) more rigorous research designs to contribute stronger research evidence; (3) increased client-centered or self- or proxy report measures; (4) higher level evaluation of instruments, including longitudinal studies, to address predictive and evaluative instrument properties; (5) international collaborations to further develop instruments; (6) consolidation and expansion of instruments that serve to address frameworks or models; and (7) emphasis on the publication of instruments that target measurement of the participation of children and youth in meaningful occupations and life roles in home, school, and community environments.
The occupational therapy profession is responsible for providing effective and best practice, value for money, and valuable services to all occupational therapy clients. In the context of children and youth, the client may be the child or youth, the family, the school, the community, the agency, or any other natural environment in which the child or youth participates in his or her daily occupations. AOTA’s (2006) Children and Youth Ad Hoc Committee targeted 11 areas for research development, and all of them rely on the development and evaluation of psychometrically sound measurement instruments.
As described by Moyers (2010), harm is more than actually causing injury. Harm also results when “our clients receive ineffective intervention or intervention not as effective as an alternative method in improving occupational performance and participation in daily life” (p. 457). It has been more than a decade since Cusick (2001) encouraged occupational therapists to reflect,
Am I doing the right thing, in the right way, with the right person, at the right time, in the right place, for the right result and am I the right person to be doing this . . . and is it at the right cost? (p. 103)
Instrumentation permits evaluation that guides intervention, substantiates the conceptual models that underpin the profession, and provides objective measurement of the efficacy and effectiveness of occupational therapy interventions and services. Further development and evaluation of stellar occupation-focused, participation-based tests, measures, and instruments is imperative to the future of the profession.
Footnotes
*
All tables appear at the end of this article, after the references.
†Indicates studies that were reviewed for this article.