Abstract
The psychometric properties of assessments must be established for specific populations. The psychometric properties of the Children’s Assessment of Participation and Enjoyment/Preference for Activities of Children have been studied only in a sample of children with physical disability. We conducted a study to determine the appropriateness of drawing inferences from this assessment for children with high-functioning autism (HFA). The content validity and test–retest reliability (r > .7) were both found to be adequate for this population. Parents’ agreement with most of their children’s self-ratings on this assessment provided an estimate of interrater reliability. We also ascertained the feasibility of gathering recreational participation information from children with HFA and found that adaptations to facilitate the self-completion of the tool should be made available. The study findings support the use of this tool to assess recreational participation among children with HFA.
High-functioning autism (HFA) is a prevalent neurodevelopmental syndrome affecting people of average or above-average intellectual ability who present with one of three diagnoses within autism spectrum disorder (ASD): autistic disorder, Asperger disorder, and pervasive developmental disorders, not otherwise specified (American Psychiatric Association, 2000; Centers for Disease Control and Prevention, 2009). HFA, a disabling condition that emerges in childhood and affects social communication, is believed to affect children’s ability to participate in recreational activities. Recreational participation is an essential part of human performance and offers wide-ranging benefits across the life course (Bellini, 2004; King et al., 2003; Law et al., 2004; Mactavish & Schleien, 2004). Recreation can reduce behavioral and emotional disorders, help people develop social relationships and friendships, and improve physical and mental health (King et al., 2003; Mactavish & Schleien, 2004). Diminished and less varied recreational participation is found in children with disabilities and may have negative consequences in the long term (Faison-Hodge & Porretta, 2004; King et al., 2003; Mancini, Coster, Trombly, & Heeren, 2000).
The extent of involvement in recreational participation of school-age children with HFA has not been fully examined. To date, few studies of patterns of recreational participation in this population have been conducted. The studies that have been published have had important methodological limitations, including the weak psychometric properties of the tools used to measure recreation. Thus, in conducting more studies to better understand the recreational engagement patterns of children with HFA, estimating the psychometric properties of the tools chosen to measure this construct with this population is essential.
We sought a comprehensive measure of recreation that children with HFA would be able to complete with support. A review of recreational participation measurement tools yielded a dozen questionnaires, of which half could be used with children. Excluded were measurement tools requiring a high level of written or verbal language and those having a narrow scope of measured recreational activities, such as the Leisure Questionnaire (Passmore & French, 2001), the Occupational Therapy Assessment of Leisure Time (Soderback & Hammarlund, 1993), and the Godin Leisure Time Activity Scale (Godin & Shephard, 1997). We considered the Pediatric Activity Card Sort (Mandich, Polatajko, Miller & Baum, 2004); however, it measures the broad range of participation and includes fewer recreational activities than the Children’s Assessment of Participation and Enjoyment/Preference for Activities of Children (CAPE/PAC; King et al., 2004).
The CAPE/PAC (King et al., 2004) estimates a child’s participation outside of school. Although the CAPE/PAC estimates some broader aspects of participation, it is primarily a measure of recreational participation, with 46 of its 55 items related to recreation (Bedell & Coster, 2008; Hochhauser & Engel-Yeger, 2010; King, Petrenchik, Law, & Hurley, 2009; Shikako-Thomas, Majnemer, Law, & Lach, 2008). The CAPE/PAC was designed for children with and without disabilities, ages 6–18 yr. Its psychometric properties were studied with 427 children with limited physical functioning (Law et al., 2006). In addition to the diversity (overall number) of recreational activities in which children participate, the CAPE/PAC provides information about five dimensions of participation: (1) intensity (frequency of participation in the activity), (2) social aspect (with whom the child participates in the activity), (3) location (where the activity takes place), (4) the child’s degree of enjoyment in the activity, and (5) preference (the child’s desire to participate in the activity if he or she could “do anything in the whole world”). The CAPE/PAC is a newer measurement tool that has been used primarily with children with motor disability. Children with HFA were included in two studies.
The reliability and validity of a measurement tool’s scores must be established for different disability groups because psychometric properties have meaning only when applied to specific populations (Cook & Beckman, 2006; Streiner & Norman, 2003). Although the developers of the CAPE/PAC have specified that it can be used with children with other disabilities (King et al., 2004), children with varied disabilities were not included in their sample. Thus far, the psychometric properties of the CAPE/PAC have been examined for valid and reliable inference of its scores for children with motor disabilities (Law et al., 2006). In the two studies in which the CAPE/PAC was used with children with HFA, no discussion of the validity and reliability of drawing inference from the tool’s scores for this population was presented (Hilton, Crouch, & Israel, 2008; Hochhauser & Engel-Yeger, 2010).
Psychometric properties of measurement tools have traditionally been described in terms of types of reliability and validity such as convergent validity and interrater reliability (Streiner & Norman, 2003). More recently, Cook and Beckman (2006) proposed a novel approach using an overarching framework to describe all validity and reliability with five sources of data: content, response process, internal structure, relations to other variables, and consequences. In this framework, content corresponds to traditional content validity. Response process provides information about the respondents’ thought process in relation to the construct. Internal structure includes the degree of reliability as well as the factor structure of the tool. The correlation with scores from other instruments measuring similar constructs provides information about the relations to other variables. Finally, consequences allow one to determine whether inferences from this tool’s scores are clinically meaningful for the population in question. In the case of the CAPE/PAC, response process is especially important because it is a self-rated measurement tool on which children report their own recreational participation.
To understand the appropriateness of making inferences about the CAPE/PAC scores for children with HFA, we conducted a study to estimate its psychometric properties in this population. The objectives of the study were to
Examine whether the CAPE captured the breadth of activities in which children with HFA engage (content validity),
Estimate the test–retest reliability of the CAPE/PAC in children with HFA, and
Examine the response process of children with HFA while completing the CAPE/PAC.
Method
Research Design
The data for this study were collected as part of a larger cross-sectional study that examined recreational participation among children with HFA (Potvin, Snider, Prelock, Kehayia, & Wood-Dauphinee, 2012). Ethical approval was obtained from all appropriate institutions and agencies. For participating children ≤10 yr old, parents provided informed written consent; for children who were ≥11 yr old, both parental informed written consent and children’s informed written assent were obtained.
Participants
A sample of children with HFA (ages 7–13 yr) was recruited through multiple sources (e.g., parent support groups, therapists, health department) and through various modes (e.g., e-mail, word of mouth, postal mail) in an effort to achieve representativeness of the population. Peers were recruited from the same communities. Children with HFA were eligible to participate in the study if they had a current ASD diagnosis, had an IQ of ≥80 or an adaptive functioning score of ≥60, spoke English at home, and did not have neurodevelopmental co-conditions. The peer group consisted of typically developing children who met the same criteria except for a diagnosis of ASD. They also had not received special education supports during the past or current school year.
Procedures
The data collection took place over two or three visits at a location convenient to the participants. The first visit was used to ascertain the characteristics of the participating children and allowed Marie-Christine Potvin, the primary investigator (PI), to establish a rapport with them. The PI administered the following standardized measurement tools to the parents and children: Gilliam Autism Rating Scale (2nd ed.; Gilliam, 2006) to independently confirm the diagnosis, Test of Nonverbal Intelligence (3rd ed.; Brown, Sherbenou, & Johnsen, 1997) to estimate the participants’ IQ, and the Vineland Adaptive Behavior Scales (2nd ed.; Sparrow, Cicchetti, & Balla, 2006) to estimate adaptive functioning. These measurement tools have been used extensively with children with HFA and have demonstrated adequate psychometric properties with this population.
The second and third visits were used to gather data about the psychometric properties (i.e., content, test–retest reliability, response process) of the CAPE/PAC. The majority of the data related to response process were collected for the entire sample (children with HFA, n = 30; peers, n = 31) in the second visit. The PI kept detailed notes of the types of and reasons for providing adaptations to facilitate the children’s completion of the CAPE/PAC. Single questions were developed using Likert scales to gather information about parents’ responding to some of the CAPE/PAC questions when their children were unable and parents’ agreement with their children’s responses to that tool.
The data on the CAPE/PAC’s test–retest reliability and content validity, which required a third visit 1 mo later, were collected for a subsample of participants with HFA (n = 14) and peers (n = 13). Considering the potential burden of this repeated assessment on the children, they and their parents were given the opportunity to participate or not in the third visit. Only those who completed the first two visits with relative ease were considered for the third visit.
During this last visit, after the second completion of the CAPE/PAC, a brief probing interview was conducted with the children to gather further information about response process. This type of cognitive debriefing interview is commonly used in tool development, including with children, to ascertain the participants’ comprehension of the questions involved, what method they used to remember the information necessary to answer the questions, and how comfortable they felt answering the questions (Collins, 2003; Matza, Swensen, Flood, Secnik, & Leidy, 2004). Considering the challenges with abstract language experienced by children with HFA (Harris et al., 2006; Landa & Goldberg, 2005), the probing interview focused on concrete questions such as “Name any activities that you do that we have not talked about today,” “How often do you do this activity?” and “Name any activities that you would like to do that we have not talked about today.” These questions were followed by two questions that were more abstract: “How did it feel to answer these questions in the book about the activities?” and “Is there anything else you want to tell me about the recreational activities that you do?” The interview lasted approximately 5 min.
Data Analysis
The data collected about the CAPE/PAC’s content validity (Objective 1) were interpreted qualitatively. Recreational activities mentioned by the children with HFA that were not included in the CAPE/PAC were recorded by the PI as a list, examined for their meaning, and compared with the activities included in the CAPE/PAC (Rubin & Rubin, 1995; Seidman, 1998). This information was used to determine whether the CAPE/PAC’s content was reflective of activities in which children with HFA participate and thus valid for this population. An intraclass correlation coefficient was calculated to estimate the CAPE/PAC’s test–retest reliability (Objective 2) in children with HFA and peers. The response process of children with HFA was examined through descriptive statistics and qualitatively.
Results
The children in both groups (26 boys and 4 girls with HFA and 27 male and 4 female peers) and their families were similar on all measured key characteristics (i.e., child’s age and IQ, family composition and income) except for highest level of educational attainment (p < .05; Potvin et al., 2012). Participating families primarily had an intact family unit and were middle to upper-middle class. Participating families in both groups differed significantly (p < .001) from the broader population in socioeconomic status; consequently, this study’s results should be interpreted cautiously in the context of socioeconomic status.
Content
To examine the CAPE’s content in terms of its appropriateness for use with children who have HFA, these children were asked to name recreational activities in which they participated that were not included in the tool. Ten of the children did not name additional activities; the others mentioned four additional activities (i.e., inventing things, playing with a walkie-talkie, going online, and playing battle with a sister). “Going online” and “inventing things” did not appear to belong to any of the current CAPE items. However, the other two likely did. “Play with a walkie-talkie” falls under “playing with things or toys,” and “play battle with my sister” may belong to the CAPE activities “doing pretend play” or “playing games.” The CAPE includes an item about playing computer games but not about computer use beyond games, which is where “going online” would likely belong.
Similarly, a qualitative examination of the content of the PAC, the section of the tool that asks the children to describe their preference, was conducted. Two children with HFA answered “none.” The other children combined named 22 other activities. Of these, 10 fell under the activity “have a paid job” with professions such as “pace car driver” and “be the president” mentioned most often. Nine were either related to the child’s own area of intense interest such as “getting actual Pokémon training” or were a general wish of something the child would want if there were no rules (such as “stay outside as long as I want” or “not go to bed”) but were not a recreational activity per se. Three were recreational activities not listed in the PAC: “meeting a famous person,” “learn to fly,” and “be a hunter.”
Internal Structure: Test–Retest Reliability
For the HFA group (Table 1), the test–retest reliability of the CAPE/PAC overall scores for all dimensions was >.7, except the social aspect dimension, which was low (r = .196) according to a generally accepted standard (Streiner & Norman, 2003).
Test–Retest Reliability of the CAPE/PAC
Note. CAPE = Children’s Assessment of Participation and Enjoyment; HFA = high-functioning autism; PAC = Preference for Activities of Children.
To ascertain whether variation between the two administrations of the CAPE/PAC was the result of true changes within the families’ life, parents were asked during the second assessment visit whether the past 4 weeks had been typical for their family. Eighty-six percent of parents of children in the HFA group and 83% of parents of children in the peer group agreed or strongly agreed that the past 4 wk had been typical for their family. Parents’ rating of family change was compared between groups using a Mann–Whitney U test. Results were not statistically significant (p = .752). These results substantiate that CAPE/PAC scores are stable over time for children with HFA at least as much as for typically developing children.
Response Process
To gain an understanding of the response process of children with HFA when completing the CAPE/PAC, we collected information about the adaptations needed, parents’ agreement with their children’s self-rating of recreational engagement, and the amount of assistance provided by parents to their children.
The CAPE/PAC is a self-rated measure that children complete independently or supported by an adult through adaptations when necessary. To ascertain the type of assistance the children with HFA needed to complete the CAPE/PAC, we conducted a qualitative analysis of the adaptations provided (Figure 1).
Adaptations provided to children for completion of the Children’s Assessment of Participation and Enjoyment/Preference for Activities of Children (CAPE/PAC) and the rationale for them.
When parents were present as their child completed the CAPE/PAC (46.67% of sample), they were asked to rate their degree of agreement with their child’s self-rating on this measurement tool (Table 2). In the HFA group, 75% of parents agreed or strongly agreed with their child’s rating on the CAPE and 50% of parents agreed or strongly agreed with their child’s rating for the PAC. This information is not available for the peer group because too few parents observed their children’s completion of the CAPE/PAC (n = 3).
High-Functioning Autism: Parents’ Agreement With Child’s Ratings of Recreation
Note. CAPE = Children's Assessment of Participation and Enjoyment; PAC = Preference for Activities of Children.
The CAPE/PAC test manual notes that for the factual questions (i.e., intensity, social aspect, location), a parent may answer the questions “if it is clear to the parent that the child is having difficulty answering a question” (King et al., 2004, p. 27). In this study, parents rated to what degree they answered the three dimensions of the CAPE questions for their children (Table 3). The two groups differed significantly on the number of questions answered by the parents for all three dimensions measured (see Table 3; p < .001), with parents of the children in the HFA group answering more questions for their children: for intensity, 44%; social aspect, 37%; and location, 37%.
Parents’ Completion of the Children’s Assessment of Participation and Enjoyment
Note. HFA = high-functioning autism.
Finally, we explored the children’s thought process as they responded to the CAPE/PAC (HFA group, n = 14). Children were asked how they knew how often they did an activity. Not all children responded to this question; those who did shared the following information: “I guessed” (n = 2), “I just knew some of them” (n = 2), “I thought about it” (n = 2), “I tried to remember” (n = 3), “I counted the number of times I did some things like going to the movies” (n = 1), “I asked for help from mom” (n = 1), and “I don’t know” (n = 1). Children were also asked to talk about how it felt to answer the CAPE/PAC questions more generally. The children provided limited information for this question, ranging from “fun,” “cool,” and “felt good” (n = 6) to “so-so” and they “sort of liked it” (n = 2), to it was “boring” (n = 2). One child stated that it was “hard to do,” and 1 simply stated that he or she “did not know.”
Discussion
This study suggests that valid inferences can be drawn from the CAPE/PAC in terms of content for the HFA population. We attempted to determine whether children with HFA would report participating in recreational activities not captured by the measurement tool. Analysis of the study results revealed that the 55 activities included in the CAPE appeared to cover the range of recreational activities in which children with HFA participate. When asked to name additional recreational activities in which they participated, children named very few activities that did not fall under one of the items already included in the CAPE. Similarly, the 55 activities included in the PAC also appeared to cover the broad range of recreational activities in which children with HFA may want to participate (i.e., preference). However, the more idiosyncratic activities related to an individual child’s own area of intense or peculiar interests, a hallmark of this diagnosis, might not be included on the lists of recreational activities provided by the CAPE/PAC.
CAPE/PAC scores appeared to be stable over time among children with HFA, suggesting that reliable inferences can be drawn from this measurement tool for this population. For the HFA group, the test–retest reliability was high for most CAPE dimensions and the PAC (Table 1). In fact, the test–retest reliability was generally higher for the HFA group than for the peer group. We should note that parents of children in the HFA group answered the factual questions for their children, that is, the questions on the intensity, social aspect, and location dimensions of the CAPE, more frequently than parents of children in the peer group, which could facilitate better test–retest reliability in this group. However, further examination showed that the correlation coefficients for the factual questions were not systematically higher than those for questions answered by the children with HFA. Thus, parents answering more factual questions did not appear to explain the difference in reliability between the two groups. The test–retest reliability in the HFA group was similar to that reported in the CAPE/PAC manual for children with motor disabilities (King et al., 2004, p. 61) with the exception of the enjoyment dimension, which showed higher test–retest reliability in our study. King et al. (2004) noted in the CAPE/PAC manual that the lower enjoyment dimension reliability might be related to children rating their enjoyment of their most recent experience; thus, there may be less variation in enjoyment of activities in the HFA population.
A final point of consideration related to the CAPE/PAC’s test–retest reliability among children with HFA was that the subgroup of children who repeated the administration of the CAPE/PAC was different than the larger sample on two measured child characteristics, specifically, nonliteral language and nonverbal intelligence (p < .05), which were both higher. Thus, the reliability coefficient obtained in the HFA subsample may not have been representative of the test–retest reliability in the sample as a whole.
Most previous studies of recreational participation in children and adults with HFA and ASD had parents or guardians reporting on behalf of the person with the disability (Church, Alisanski, & Amanullah, 2000; Orsmond, Krauss, & Seltzer, 2004; Saldaña et al., 2009; Wagner et al., 2002). Several authors, however, have insisted on the importance of self-reporting in individuals with disability and questioned the accuracy of proxy reporting when measuring constructs related to recreational participation (De Civita et al., 2005; Rapkin & Schwartz, 2004; Verdugo, Schalock, Keith, & Stancliffe, 2005). Bearing this in mind, this study was designed for children with HFA to self-report on their recreational participation, and data were collected to ascertain the feasibility of this approach (i.e., response process).
The CAPE/PAC is a self-rated measure of recreational participation that allows the person administering the tool to provide adaptations to the instructions and parents to be proxy reporters for the factual questions included in the tool. We noted the type of adaptations provided by the PI to enable the children to self-report (Figure 1) as well as the amount of proxy reporting that occurred (Table 3). All the children in the study were able to answer the CAPE/PAC’s diversity, enjoyment, and preference questions. We should also note that a higher proportion of parents of children with HFA than parents of children in the peer group were proxy reporters for the factual dimensions of the CAPE (see Table 3). Parents’ agreement with their children’s self-reports (see Table 2) substantiated the appropriateness of using this tool as a self-report tool with this population. Thus, the study indicated that although children with HFA should be supported in self-reporting for the CAPE/PAC, parents’ input may also be necessary to complete the tool.
Implications for Occupational Therapy Practice
The results of this study have several implications for occupational therapy practice with children with HFA. Clinicians should keep the following information about the CAPE/PAC in mind:
Children with HFA can reliably self-report on their recreational engagement using the CAPE/PAC.
Children with HFA should be encouraged to list additional activities in which they participate because idiosyncratic activities related to a child’s own area of intense interests may not be represented in the CAPE/PAC.
Clinicians should be prepared to provide structural support and visual supports (Figure 1) to children with HFA when completing the CAPE/PAC.
When working with children with HFA and limited verbal language or attention to task, using only the parts of the CAPE/PAC that are necessary for their clinical application or breaking up the tool into small sections may be necessary.
With some children, parents should be allowed to answer the factual questions included in the tool to facilitate the completion of the CAPE/PAC in a timely manner.
Limitations
The study used a relatively small sample size, and some analyses had an even smaller sample; the results should be replicated in a larger study. We should also note that the subsample of children with HFA who repeated the CAPE/PAC had higher intellectual abilities and nonverbal language abilities than the whole study sample. Consequently, the CAPE/PAC scores’ test–retest reliability should be interpreted with caution in children with normal intellectual abilities in the lower range.
Future Studies
In addition to replicating the study with a larger sample, further examination of the CAPE/PAC’s content validity in this population would be important to investigate the understanding or interpretation of the CAPE/PAC items by people with HFA.
Conclusion
This study contributes to the body of literature on the validity and reliability of interpretation of CAPE/PAC scores for children with HFA. In terms of content validity, the activities included in the CAPE/PAC qualitatively appeared to cover the broad range of recreational activities in which children with HFA participate. The CAPE/PAC showed high test–retest reliability in children with HFA. This study is unique because we ascertained the feasibility of gathering recreational participation information from children with HFA. The data collected suggest that completing the CAPE/PAC with children with HFA is feasible, although adaptations to facilitate the children’s completion of the measurement tool should be made available and parents may need to assist their child in answering some or all of the factual questions. Findings supported the use of the CAPE/PAC with children with HFA.
Footnotes
Acknowledgment
This study was conducted with financial support from the Organization for Autism Research, the American Occupational Therapy Foundation, and the Judith Kornbluth Gelfand Pediatric Fellowship in partial fulfillment of a PhD Program in Rehabilitation Science. We gratefully acknowledge the contribution to this study of Sara Ahmed, Janice Dunn, and Alan Howard. The study was presented at the 2012 American Occupational Therapy Association Annual Conference & Expo in Indianapolis and as a poster at the 2011 Canadian Student Health Research Forum in Winnipeg, Manitoba.