Abstract
The authors describe a systematic process for identification and rating of performance-based outcome measures that can be used to identify appropriate outcome measures for occupational therapy clinical trials.
Autism spectrum disorder (ASD) is characterized by difficulties in social communication and the presence of restrictive and repetitive behaviors. These symptoms affect performance in daily life activities (Schaaf et al., 2011); thus, occupational therapy is frequently a component of a comprehensive program for children with ASD. Occupational therapy interventions often address areas that are important to families, including functional skills, participation in daily activities, and quality of life (Kuhaneck & Watling, 2015; Patten et al., 2013 ; Schaaf et al., 2011); as a result, occupational therapy interventions are one of the most valued and frequently requested services by parents (Goin-Kochel et al., 2009; Green et al., 2006, Peacock, 2012).
Despite the value and high use of occupational therapy for children with ASD, few studies have measured occupational therapy outcomes with objective, performance-based measures. Thus, guidance for choosing robust and psychometrically sound measures that assess meaningful outcomes of interventions is needed (Bennett & Bennett, 2000; Lami et al., 2018). Equally important is choosing measures consistent with the scope and aims of a particular study; these measures should be psychometrically strong, precise, and relevant for measuring outcomes reflective of family needs (Askari et al., 2015; Coster & Khetani, 2008; McConachie et al., 2015). In this article, we describe a systematic process for identification and rating of performance-based outcome measures that can be used to identify appropriate outcome measures for occupational therapy clinical trials.
For this study, we identified two main areas for outcome measurement—activities of daily living (ADLs) and socialization—because these areas showed considerable improvements in our pilot trials (Schaaf et al., 2012). In this pilot work, we used parent-reported outcome (PRO) measures to evaluate these constructs. PROs are an important strategy for outcome measurement because they provide the person’s perception of change; however, they have limitations, including potentially over- or underestimating function (Weldring & Smith, 2013). Hence, to increase the rigor of the outcome measurement plan for a future trial, we sought to identify performance-based outcome measures to pair with the PROs. Performance-based measures are administered by a trained evaluator who observes and rates the child’s performance on the basis of a standard scale (Kazdin, 2013; Lami et al., 2018; Schaaf & Lane, 2015).
Method
Design
Mixed methods were used to identify and rate existing performance-based outcome measures of ADLs and socialization for children with ASD ages 6 to 9 yr. A panel of experts reviewed and rated measures and then held a consensus meeting to identify the best measures for the planned trial.
Reviewers
Four experts in ASD and pediatric outcome measurement served as the reviewers. They included a PhD neuropsychologist with expertise in ASD diagnosis, a PhD occupational therapist with expertise in pediatric outcome measurement and instrument development, and two occupational therapists (one PhD and one OTD) with clinical and research experience in ASD and instrument development. All had university academic appointments.
Procedures
The review and consensus process occurred sequentially. First, we conducted a rapid review 1 of the literature to identify performance-based measures of ADLs and socialization appropriate for use with children with ASD ages 6 to 9 yr (the population for our planned clinical trial). Next, we adapted a quality indicator (QI) scale to rate each measure (described later). Ratings were collated, and a nominal group process was held to achieve consensus on identifying the best measures for the planned trial. Each step is described next.
Step 1: Identification of Measures
Toidentify outcome measures that met the inclusion criteria, the second author conducted a search of the research and gray literature (books and book chapters, online assessment resources, and websites). We also contacted professional colleagues in the fields of occupational therapy and autism research or practice to identify outcome measures used in their work. We used the following inclusion criteria: (1) is appropriate for children with ASD ages 6 to 9 yr, (2) evaluates ADL skills and socialization, (3) is administered as a performance measure, and (4) has established psychometric properties. We identified the following search terms: children, autism, performance-based measures, assessments, evaluations, evaluate, activities of daily living, daily living skills, daily activities, and socialization; we then searched PubMed, CINAHL, Google Scholar, and OTsearch. Measures meeting the inclusion criteria were organized on a secure shared drive, together with manuals and any relevant studies on the measure that described its psychometric properties.
Step 2: Modification of an Existing Quality Indicator Rating Scale
The QI rating scale used for this project is based on the work by Law and MacDermid (2014). This scale was adapted for our project needs in relation to clinical group (autism), age group, and focus on performance-based measures. Items on the QI scale address the psychometric properties of each measure on qualities such as reliability; validity; responsiveness to change; and characteristics of the measure, such as purpose, scope, scoring, and administration time requirements. The QI scale was reviewed and field tested for clarity, comprehensiveness, and redundancy by a measurement expert. On the basis of this review, 13 scale items whose content was redundant with inclusion criteria were removed. Two items related to a measure’s ability to interpret subscale scores were combined into one item, one item that addressed reliability was expanded into three items to more clearly define types of reliability (intrarater, interrater, and test–retest reliability), and two items that rated discriminant validity and use of Rasch analysis were added. The final QI rating scale is detailed in Supplemental Appendix A, available online with this article.
Step 3: Identification of Expert Reviewers
A convenience sampling of eight professionals with expertise in ASD or measurement received an email detailing the project and an invitation to serve as reviewers. Four accepted the invitation and received instructions for reviewing the instruments, which were placed on a secure shared drive.
Step 4: Review of Measures
Each expert independently reviewed and rated each measure. Ratings were submitted via encrypted email.
Step 5: Collation of Ratings
The scores for each QI rating from the four reviewers were compiled in a Microsoft Excel spreadsheet. The highest possible score obtainable for each measure by a rater was 23, and the highest possible sum score for a specific measure across all four raters was 92. Higher scores indicated stronger measurement characteristics and psychometric properties.
Step 6: Consensus Meeting
The expert reviewers used an online survey platform (Survey Monkey) to rate each measure and then met with the project investigators (first and second authors) virtually to discuss their reviews. The compiled results of the review were presented to the expert reviewers as the basis for discussion. Experts presented their rationale for ratings and their expert opinion about the measures. Group discussion facilitated the consensus process. Toward the end of the meeting, consensus was reached on the measures that best met the needs of our planned trial.
Results
Identification of Measures That Met Inclusion Criteria
Ten performance-based ADL measures and 11 performance-based socialization measures were identified (N = 21 measures). Of these, 7 measures met the inclusion criteria for this project: 5 ADL measures and 2 socialization measures. All identified measures are shown in Table 1. Note that the exclusion of a specific measure for this project is not an indication of its potential value; rather, it did not fit the identified needs for our future clinical trial.
Identified Outcome Measures
Note. ADL = activities of daily living; PDD = pervasive developmental disorder.
Quality Indicator Scale Ratings for Each Included Measure
The summed QI ratings for each measure are shown in Table 2. The Assessment of Motor and Process Skills (AMPS; Fisher, 2006) received the highest rating on the QI scale for the ADL measures, and the Evaluation of Social Interaction–Second Edition (ESI–2; Fisher & Griswold, 2010) received the highest rating for the socialization measures. These findings are described in Table 2.
Summed Quality Indicator Ratings for Each Included Measure
Note. N = 4 reviewers. ABLLS–R = Assessment of Basic Language and Learning Skills–Revised; ADLs = activities of daily living; AFLS = Assessment of Functional Living Skills; AMPS = Assessment of Motor and Process Skills; ASD = autism spectrum disorder; DASH–3 = Developmental Assessment for Individuals With Severe Disabilities–Third Edition; ESI–2 = Evaluation of Social Interaction–Second Edition; GOAL = Goal-Oriented Assessment of Lifeskills; N/A = not applicable; QI = quality indicator; SP = Social Profile.
Not scored by 1 reviewer.
Included Measures and Quality Indicator Points Obtained
Note. ADLs = activities of daily living; ASD = autism spectrum disorder; IADLs = instrumental activities of daily living; NA = not applicable; QI = quality indicator.
Descriptive Analysis of Included Measures of Daily Living Skills and Socialization
Table 3 lists the included measures and the QI points received. The strengths and limitations of each measure according to the needs of our planned trial are presented.
Nominal Group Process Consensus
The experts concluded that the AMPS and the ESI–2 were the best-suited outcome measures for our needs. Group consensus focused on the findings that both measures have norms for ASD, have strong standardization, are capable of measuring change between known groups, and used Rasch analysis.
Discussion
Recognizing that selection of outcome instruments for a clinical trial depends on the research hypothesis and study intent, this project was designed to guide the selection of performance-based outcome measures of ADLs and socialization for a planned clinical trial of an occupational therapy intervention for children with ASD ages 6 to 9 yr. ADLs and socialization were chosen as outcomes on the basis of a prior pilot study that showed that these areas were sensitive to change for the studied intervention (Schaaf et al., 2014). Thus, this article is not intended to be a systematic, comprehensive review of outcome measures but rather a means to identify and get expert opinion for the future trial. Although there have been at least two systematic reviews of outcome measures for people with ASD published in the literature (Askari et al., 2015; McConachie et al., 2015), none were specifically focused on performance-based outcome measures, and they did not target the outcome areas of ADLs and socialization.
One value of this project is in demonstrating a process for choosing performance-based outcome measures that are consistent with the aims and scope of a specific, planned clinical trial. Selection of appropriate outcome measures that are sensitive to assessing change in dependent variables is an important aspect of study design. Thus, the procedures used in this project may be useful to guide researchers and clinicians through the process of outcome measure selection for clinical trials. The sections that follow highlight five important considerations when choosing outcome measures for occupational therapy clinical trials.
Careful Assessment of the Psychometric Properties
Psychometrically robust outcome instruments that provide meaningful information are essential for clinical trials and intervention research as well as for building evidence in support of occupational therapy (Coster & Khetani, 2008; Mulcahey et al., 2010). Although several types of reliability and validity can be considered, sensitivity of an instrument to detect change in response to treatment is arguably the most important psychometric property of an outcome measure used in clinical trials (Fok & Henry, 2015; McConachie et al., 2015). Accordingly, sensitivity to detect change was an important aspect of the modified QI scale, and these measures scored highly on our QI scale.
Measurement Burden on the Family and Child
Although scientific integrity of outcome measurement in ASD research is a primary consideration, the impact of time burden on the child and family must also be considered (Ebesutani et al., 2012; Hinshaw et al., 2004). Excessive time required for participation in and completion of assessments and outcome measures may dissuade families from participation in research because of their daily life responsibilities. Moreover, measures that are time intensive may hamper optimal performance by children with ASD who may have shortened attention or focus during the assessment process (Hinshaw et al., 2004). Recognizing the impact of time burden, the American Academy of Child and Adolescent Psychiatry (Hinshaw et al., 2004) recommended that protocols be streamlined to reduce burden. Their recommendations include automation of assessment protocols or the use of item response theory (Coster, 2008).
Transportability of the Measurement Plan to Clinical Practice
Transportability is important when the application of outcome measurement shifts from research to practice (Ebesutani et al., 2012). Often, the assessment phase in research is “time-intensive, resource heavy and may be too costly for real-world implementation” (Ebesutani et al., 2012, p. 141). To address this issue, Ebesutani et al. (2012) developed an algorithm-based assessment protocol that reduced administration and interpretation burden but maintained accuracy in identification and classification of participants to target appropriate intervention. Following this recommendation, algorithm-based decision making may be a potential strategy for occupational therapy. Development and testing of algorithms that can provide adequate information for characterization of participants—as well as reliable, valid, and sensitive outcome measurement—are important next steps to enhance research participation and translation to practice.
Norm-Referenced Versus Criterion-Referenced Measurement
When considering an outcome measurement plan for a study, the approach to outcome measurement, including the decision to use criterion-referenced or norm-referenced measures, is important. For this study, we focused on performance-based, norm-referenced outcome measures because our aim is to compare ADL and socialization performance of our study groups with a normative sample. Our QI scale rated norm-referenced measures higher (better) than criterion-referenced outcome measures. The differences between criterion- and norm-referenced tests have important implications for the study design and outcome measurement plan.
Because our focus was on norm-based outcome measures, solid criterion-referenced measures of ADLs and social skills were rated lower than norm-referenced measures. Case in point is the criterion-referenced Goal-Oriented Assessment of Lifeskills (GOAL) measure (Miller et al., 2013). Although the GOAL scored highly on the QI items that evaluated standardization and responsiveness, it scored low on the items that rated normative reference because it is a criterion-referenced tool. One advantage of the GOAL is that it is sensitive to various degrees of change because it measures the magnitude of longitudinal change in ADLs, allowing comparison of the child’s performance in relation to their prior scores rather than comparing the score to a standard as in normed-referenced testing. Although this characteristic is useful for an outcome measure, it did not meet the needs for our future trial; thus, its rating may not adequately represent its many strengths as a criterion-referenced outcome measure. This outcome was also the case for the Social Profile (Donohue, 2013). Thus, this project highlights the importance of defining and describing the research question and expected outcomes clearly to determine whether norm-referenced or criterion-referenced measures, or some combination of each, are best suited to the study objectives and design; outcome measures should be chosen accordingly.
Measurement of Outcomes That Are Relevant and Meaningful to Families
In occupational therapy outcome research, an important area of interest is the participant’s ability to participate in meaningful life activities (Coster & Khetani, 2008). Thus, to accurately capture the diversity of performance skills and participation opportunities, it may be necessary to evaluate performance in context (i.e., real-life environments) and include activities that are meaningful to the child and family. To accomplish this goal, an outcome measurement plan that combines PROs may be needed (Askari et al., 2015; Coster, 2008; McConachie et al., 2015). PROs can provide a perspective on the value of a given outcome in the context of the individual’s daily life. One value of the top-rated measures in this project, the AMPS and the ESI–2, is that they are performance-based outcome measures that appreciate context and meaning by using tasks that are important to the individual in the most natural context possible. For these reasons, these measures were well suited for our occupational therapy clinical trials, which focus on measuring functional outcomes.
Limitations
Despite the usefulness of this study for guiding outcome measurement planning for clinical trials of occupational therapy interventions for children with ASD, it is important to note that the findings from this project are applicable to specific, predetermined criteria and may not be generalizable to other intervention trials. Moreover, expert reviewers’ prior knowledge may have influenced the results. Although the experts were highly qualified in ASD and outcome measurement, they had various degrees of expertise. We did not provide specific training on how to use the modified QI scale or establish competence in determining quality across reviewers; thus, interpretation and rating of measures may have been affected by this varying expertise.
In terms of the QI rating scale, the ratings for each item were unweighted for relative importance, which may have affected the final rating score. However, this limitation was somewhat mediated by the reviewers’ discussion during the consensus meeting where the needs of the planned clinical trial were considered in the final ratings.
For one instrument, the AMPS, the reviewers did not have access to the full manual, which may have limited their knowledge of the AMPS. Finally, the QI ratings of the Assessment of Functional Living Skills (Mueller & Partington, 2015) measure were affected because the assessment did not have published information related to reliability, validity, responsiveness, and time for administration at the time of this project. Thus, the clinicians and researchers seeking to use this criterion-referenced tool may want to check for updated data to evaluate its utility and rigor.
Implications for Occupational Therapy Practice
In this project, we identified the following essential elements when choosing outcome measures for clinical trials: Psychometric characteristics, including validity, reliability, and sensitivity to detecting change in the given construct Time (and attention) burden for the child and family as well as the clinicians when choosing outcome measures Ability to apply the outcome measurement plan to clinical practice The approach to outcome measurement, including the decision to use criterion-referenced or norm-referenced measures Outcomes that are relevant and meaningful to families.
Conclusion
This project takes an important step forward by disseminating specific characteristics of outcome measures for occupational therapy intervention trials for children with ASD. In this article, we introduce an approach to the review and identification of outcome measures for consideration in occupational therapy outcome studies. We highlight crucial considerations in identifying outcome measures for clinical trials, including the aims and scope of the planned study; we also point out considerations for transportability of research to clinical practice. The methodology used in this project may guide other researchers in appropriate outcome measurement selection.
Supplemental Material
Supplementary material for Choosing Performance-Based Outcome Measures of Daily Living Skills and Socialization for Clinical Trials in Autistic Children
Supplementary material, sj-pdf-1-aot-10.5014_ajot.2021.044602.pdf for Choosing Performance-Based Outcome Measures of Daily Living Skills and Socialization for Clinical Trials in Autistic Children by Roseann C. Schaaf, Amy Carroll, Elizabeth Conte Waskie, Rachel L. Dumont and Elizabeth Ridgway in The American Journal of Occupational Therapy
Footnotes
1
A rapid review provides a time-efficient strategy to identify, select, and critically appraise data from relevant research on a specific topic. It is a simplified approach to a systematic review where sources are limited because of time constraints (Khangura et al., 2012).
Acknowledgments
The American Occupational Therapy Foundation Intervention Research Grant Program supported this work. We declare no conflicts of interest. Ethics board approval was not required for this study. We thank Mary Jane Mulcahey, who provided her expertise in outcome measurement. We also thank the expert raters: Julianna Bates, Zoe Mailloux, Sarah Schoen, and Teresa May-Benson. Finally, we thank Sophie Molholm, who provided input to this project.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
