Parallel-Forms Reliability and Clinical Utility of an Application Version of the Activity Card Sort Australia (18–64)

What Is the Activity Card Sort Australia for Adults Ages 18–64?

The ACS Aus (18–64) measures activity engagement of Australian adults (Gustafsson et al., 2014) and may be used to identify reductions in activity engagement, collaboratively identify priorities for goal setting, and measure changes in response to interventions. This version consists of 85 cards that depict adults engaged in representative activities assigned to one of three domains: (1) daily life, (2) recreation and relaxation, and (3) physical activities (Gustafsson et al., 2014). The daily life domain contains all daily activities completed for maintenance of self, the home, and life or roles in the community; the recreation and relaxation domain includes activities completed in free time for fun, as a hobby or an interest, including socializing with others; and the physical activities domain includes participation in any form of physical activity, whether of high or low intensity, or in team or in individual sports.

To complete the assessment, the client sorts each card to represent former and current activity engagement. The literature has identified the ACS as a unique assessment of activity engagement (Duncan & Earhart, 2011). The first unique advantage of the ACS is that the cards include pictures of people completing the activity, which is nonthreatening and effective in prompting a client’s memory recall of activity engagement (Doney & Packer, 2008; Sachs & Josmon, 2003). The second unique advantage is that the ACS is a client-centered measure of activity engagement—irrelevant activities are not counted in the final score, which provides a more accurate level of retained activity (Lyons et al., 2010; Packer et al., 2008).

Psychometrics

The psychometrics of the ACS for older adults have been comprehensively examined and supported cross-culturally (Laver-Fawcett & Mallinson, 2013). The findings of these studies can be considered collectively because the ACS is culturally adapted by modifying the activity cards to represent the given culture while the underlying methodology remains consistent (Katz et al., 2003). Initial psychometric testing of the ACS Aus (18–64) has also been completed with 54 healthy adults ages 18–64 yr, indicating high test–retest reliability (intraclass correlation [ICC] = .92, p < .001) and acceptable internal consistency (α = .83) for all domains (Gustafsson et al., 2017).

In a second study (Bryant et al., 2017), discriminative construct validity was supported by differences in activity engagement between healthy adults (n = 16) and adults with Parkinson’s disease (n = 16). In this study, participants with and without Parkinson’s disease were matched on age, gender, and education, and the authors reported significantly different activity retention for the physical activities domain (p = .035) in favor of the group with Parkinson’s disease (54% activity retention compared with 38% for healthy adults). The daily life domain demonstrated a nonsignificant trend (p = .80) in favor of the healthy adults (79% activity retention compared with 69% for the adults with Parkinson’s disease; Bryant et al., 2017).

Technological Developments in Occupational Therapy

Technology such as apps can be used in health care to ensure equity in access to services, regardless of geographical location (Stillerova et al., 2016; Street et al., 2009). However, research regarding which face-to-face assessments are viable as apps is lacking (Cason, 2014). Recently, the Canadian Occupational Performance Measure (COPM; Law et al., 2014) has been adapted and made available as a web-based app that can interface with electronic health records (COPM, 2017). A second assessment that is available as an app is the Multidimensional Task Ability Profile (MTAP; Mayer et al., 2005). The MTAP uses a pictorial card-sorting methodology similar to that of the ACS, in which participants with musculoskeletal injuries sort cards into categories presented on a 5-point Likert scale ranging from able to unable. Mayer and colleagues (2005) found evidence of high test–retest reliability (r = .95, p < .05) when MTAP scores were compared with a 2-day interval and high concurrent validity when they were compared with the Spinal Function Sort and the EPIC Lift Capacity test. The high levels of reliability and concurrent validity suggest the MTAP may be useful for tracking client changes and treatment program outcomes over time (Mayer et al., 2005).

Mayer et al.’s (2005) MTAP study provides preliminary evidence that assessments administered as apps may be valid, reliable, and clinically useful. However, app and mobile technologies are proliferating faster than clinical evidence, and the effectiveness of apps in improving health must be evaluated before their use in therapeutic and clinical settings (Anthes, 2016; Carter et al., 2015). An Android app version of the ACS Aus (18–64) has been developed to promote accessibility for clients ages 18–64 yr who are unable to attend an occupational therapy clinic, to provide a possible interface with electronic health records, and to support the use of the tool with people who have trouble holding and sorting physical cards, for example, people with Parkinson’s disease who have hand tremors (Bryant et al., 2017).

The primary aim of this study was to test the parallel-forms reliability of the ACS Aus (18–64) app with the original card-sort version. That is, we aimed to determine the consistency of results between two versions of a test constructed with the same content but differing forms of administration. The secondary aim was to assess the clinical utility, including acceptability, ease of administration, and participant experience, of the app version. We hypothesized that the ACS Aus (18–64) app would demonstrate sound parallel-forms reliability and clinical utility.

Method

For this exploratory research, we sought to recruit a cohort of healthy adults to complete both versions of the ACS Aus (18–64) and complete an additional questionnaire to provide feedback on the two versions of the tool. Before commencement of the study, ethical approval was obtained from the University of Queensland Behavioral and Social Sciences Ethical Review Committee (2015000143).

Measures

Gustafsson et al. (2017) demonstrated that the original card-sort version has high test–retest reliability and internal consistency, and it was therefore regarded as the accepted form. An Android app version of the ACS Aus (18–64) was developed and run on an Android tablet. To sort the cards within the app version, participants were required to tap the picture card and then tap their chosen category (Figure 1). Throughout the data collection period, the app version had two minor updates in response to participant and researcher feedback regarding size of font and a correction to an incorrectly labeled picture.

OPEN IN VIEWER

Figure 1.

Screen captures of the Activity Card Sort Australia (18–64) app.

For both the card and the app versions, participants were required to sort each of the 85 cards into five categories to represent their current activity engagement levels: never done, do not do now, do now, do less now, and given up. The current, previous, and retained activity engagement levels were then calculated. The current activity engagement level is the total number of activities classified as “do now” multiplied by 1, plus the number of activities classified as “do less now” multiplied by 0.5. The previous activity engagement level is the sum of activities classified as “do not do now,” “do now,” “do less now,” and “given up.” The retained activity engagement level is then calculated by dividing the current activity engagement level by the previous activity engagement level.

A feedback questionnaire was developed to evaluate the clinical utility of the app in comparison with the card version of the ACS Aus (18–64). Participants were asked to rate the experience of sorting each version on a Likert scale ranging from 1 (excellent) to 5 (terrible). Participants were also asked to identify their preferred version and why and to comment on aspects of the app version including screen layout, color contrast, and preferred action for sorting (swipe or touch).

Results

Discussion

The ACS Aus (18–64) app was developed in response to the current increase in technological developments that can create equitable access to health care (Stillerova et al., 2016; Street et al., 2009). Examining parallel-forms reliability and clinical utility was the first step in exploring the psychometric properties of the app version. The next step was to examine participants’ experiences and the clinical utility of the app version.

Before discussing the results, it is important to consider how the study demographics compare with those in the ACS Aus (18–64) study conducted by Gustafsson et al. (2017). The distribution of participants across age groups in this study (18–44 yr, 47.9%; 45–64 yr, 52%) was more equitable than the age group distribution reported by Gustafsson et al. (18–44 yr, 61%; 45–64 yr, 38.9%). Gender was very similarly distributed, but this study had a slightly higher percentage of male participants (31.35% compared with 24.07%). More participants identified as Australian/New Zealand ethnicity in this study (83.33% compared with 66.67%), and the distribution of participants across education levels was very similar. Because of a similar spread of demographics, this study appears to have the potential to demonstrate high levels of reliability, as found for the card version of the ACS Aus (18–64; Gustafsson et al., 2017).

This study found that the app version of the ACS Aus (16–84) demonstrated acceptable evidence of parallel-forms reliability for the overall retained score; however, the reliability was more variable within each of the domains. Use of technologies such as the ACS Aus (18–64) app may change thinking processes and thus the way people complete tasks (Barzilai & Zohar, 2006). Responses to the feedback questionnaire indicated that participants found both versions of the tool acceptable and that clinical utility does not seem to be an issue for the healthy adult population. However, results and responses indicate that people may have interacted with the card version and the app version in different ways. Participants reported that the app version was quicker or faster and easier to sort, which suggested that they spent less time engaged in decision making when sorting the app version.

Comments such as these have also been reported in the literature, where ease of use has been found to be one positive attitude associated with technology (Mitzner et al., 2010, p. 1720). In addition, participants commented that they thought more about the choice with the cards, which suggests that they may have spent more time engaging in decision making when sorting the card version. This is supported by the evidence, which has shown that engagement and stimulation of thinking processes is increased through tactile connection and the physical use of hand and fingers (Johnson, 2013). Collectively, the findings suggest that completing the app version of the ACS Aus (18–64) may engage a different sorting, decision-making, and thinking process that requires further exploration.

The underlying methodology of the ACS should also be considered. The original ACS was designed to prompt memory recall and reflection on participation through a physical sorting mechanism of pictorial cards (Laver-Fawcett & Mallinson, 2013). It is possible that using a different modality (an app) had an impact on this underlying methodology. Research has found that using different modalities can affect results obtained from subjective questions more than factual questions (Christensen et al., 2014). In light of this evidence and because the ACS Aus (18–64) is a subjective measure (there are no right or wrong or factual answers), it is appropriate to assume that the different modalities may have affected the reliability of results. In other words, introducing the app version may have meant that participants sorted the subjective cards differently because memory recall was prompted in a different way and because the physical sorting was completed in a different way.

The lowest parallel-forms reliability value was calculated for the physical activity domain. In the original study of the ACS Aus (18–64), the calculated reliability for the physical activity domain was also the lowest ICC value (Gustafsson et al., 2017). It is important to consider why this domain consistently produces lower scores across studies. There are two possible explanations.

First, this domain includes activities that are predominantly discretionary and tend to change throughout life stages. For example, younger adults report higher levels of participation in team sports than do middle-aged adults (Brunet & Sabiston, 2011). Therefore, during later life stages, participation in team sports (and other physical activities) is less common, more spontaneous, and therefore more variable, even over short periods such as the 2- to 3-wk interval used in this study (Brunet & Sabiston, 2011). The second consideration is the smaller number of activities in the physical activity domain (15 of the 85 cards). When cards are swapped between categories in this domain, the effects may be more pronounced. For example, in the event of a spontaneous involvement in a team sport during the data collection period, moving one card from the “do not do now” to the “do now” category would change the retained activity percentage from 73% to 80%.

Implications for Occupational Therapy Practice

The ACS Aus (18–64) is a tool that facilitates discussion regarding activity engagement, promotes occupationally focused goal setting, and allows remeasuring of activity engagement. The results of this study have the following implications for occupational therapy practice:

This study contributes to the body of evidence for the ACS by piloting research into the reliability of an app version of the tool. This study also confirms the importance of evaluating new technologies before adopting them in clinical practice (Anthes, 2016; Carter et al., 2015).

Limitations and Future Directions

The results of this study should be considered cautiously because of the following limitations. First, the convenience and snowball sampling used to recruit participants may have resulted in over- or underrepresentation of particular demographics. Second, participants were not asked to describe their thinking processes during their use of the assessments in this study, and a qualitative study should be conducted in the future to directly elicit and compare what participants consider while sorting the ACS across the different versions. Third, the time taken to sort the card and app versions was not recorded, although participants indicated that the app version seemed quicker or faster to complete. In the future, a human–computer interaction study could be conducted to explore the different interactions elicited with different versions of the tool.

Important considerations for future studies include reducing the time between administration of the two versions, statistical analysis of the observed variations in the two versions on the basis of demographic characteristics, and examination of the test–retest reliability of the ACS Aus (18–64) app. If the app is found to be psychometrically sound, future work could investigate different app settings (screen tap sensitivity, size of buttons, color contrast, audio and visual feedback) to enhance accessibility for people with motor, visual–perceptual, and sensory issues.

Conclusion

The ACS is a standardized tool that assists occupational therapy practitioners in measuring a client’s activity engagement patterns over time. This study provides a pilot investigation into the parallel-forms reliability of the ACS Aus (18–64) app compared with the card version. Results demonstrated acceptable parallel-forms reliability scores for all but the physical activity domain. Further studies are required to investigate the test–retest reliability of the app version and to better understand why the physical activity domain had less-than-acceptable reliability. This study demonstrates the importance of testing new technologies, and further testing is required before the app version can be used for research or clinical practice.