Abstract
Findings suggest that performance on the ACS3 is consistent with performance on the ACS, while providing administrators with more clinical efficiency.
Health care reimbursement has placed progressively greater emphasis on participation-level outcomes (Centers for Medicare & Medicaid Services, 2021). The focus on remediating impairments has been unfounded, with growing literature to support functionally oriented intervention and assessment (Lezak, 1982; Morrison et al., 2015). There is often an implicit assumption that improvements in impairment will be reflected in improvements in occupational performance. The assumption of “automatic” participation improvements often precludes the use of participation assessments in lieu of impairment-based assessments (Mohammed Alotaibi et al., 2009; Moulton, 1997). Measures of performance and participation are needed to ensure that clients demonstrate progress in meaningful, client-specific occupations (World Health Organization, 2001).
The Activity Card Sort (ACS) is a widely used measure of life participation across four domains: instrumental activities of daily living (IADLs), social activities, low-demand leisure activities, and high-demand leisure activities (Baum & Edwards, 2008). The ACS has been psychometrically evaluated in numerous populations (e.g., older adults, caregivers, people with multiple sclerosis, people who had a stroke; Doney & Packer, 2008; Hamed & Holm, 2013 ; Hartman-Maeir et al., 2007; Katz et al., 2003; Kniepmann & Cupler, 2014). Three different versions are available (institutional, recovering, and community living).
The ACS requires face-to-face administration, manual scoring, and use of paper materials. A considerable limitation in requiring face-to-face administration is the current unprecedented increase in the need for and use of telehealth in rehabilitation (Wosik et al., 2020). Teleassessment is a critical component of telehealth services. The increasing public acceptance of such approaches coincides with the 50% increase in growth of smartphone ownership from 2011 to 2020 (Pew Research Center, 2021). There is a need for remotely delivered measures of participation to support the growing use of telehealth. Therefore, the purpose of this study was to develop and test the concurrent validity and acceptability of the electronic ACS (ACS3).
Method
A cross-sectional, single-group design was used to evaluate the concurrent validity of the ACS3 compared with the original ACS in middle-aged and older adults. Convenience sampling through word of mouth, social media advertisements, and flyers was used from fall 2019 to spring 2020 to recruit community participants. Participants were recruited who were age >45 yr and able to speak, read, and write English fluently. There were no exclusion criteria in order to allow for a sample as representative of the general population as possible. A sample size of 20 was determined to be sufficient to accomplish the aims of this study. All assessments were conducted in a private room and were administered by someone trained by one of the authors. This study was approved by the University of Missouri institutional review board. All participants provided written informed consent.
Activity Card Sort
The ACS includes 89 cards depicting people completing IADLs, social activities, low-demand leisure activities, and high-demand leisure activities (Baum & Edwards, 2001, 2008). Cards are sorted to the following categories: (1) I have never done, (2) I have not done as an adult, (3) I do less, (4) I have given up, or (5) I continue to do. This sorting allows for identification of the level of current participation and whether the level of participation has lessened recently because of an acute (e.g., stroke) or chronic (e.g., aging) event. The ACS yields scores related to current activities and percentage of retained activities (current activity level divided by previous activity level) in total or by domain.
Development of the Electronic Activity Card Sort
The ACS was used as the basis for developing the ACS3, with a few modifications. The ACS3 uses the same categories as the ACS; however, whereas the ACS depicts one picture for each of the 89 activities, the ACS3 uses three pictures per activity to represent different iterations of that activity(Figure 1). Each photo in the ACS3 includes people performing the activity to reflect adults and older adults instead of just older adults, as in the ACS. In the ACS3, emphasis is placed on reflecting the diversity of the U.S. population and including activities in the realms of fitness, technology, and social interactions. The ACS3 was designed to support full inclusion of people who use mobility devices. The scoring system remains the same for the ACS3. The largest modification with the ACS3 is the electronic format, which may be used on any device with Mac or Windows operating systems. During the administration, the person sees one activity (with three photos) at a time and sorts it into the appropriate category. The ACS3 prompts the person to answer for each activity to ensure that an item is not accidentally skipped. The ACS3 automatically calculates scores and generates a score report. Results can assist in goal setting, action planning, and monitoring of progress with serial administration. It is also possible to link the data to the electronic medical record.
Acceptability Evaluation
Acceptability was evaluated with a survey developed for this study by an author of the ACS (CMB) on the basis of perceived areas of importance, including (1) ease of use, (2) visual appeal, and (3) representativeness of photographs and activities. The seven-item paper-and-pencil survey had a 5-point Likert scale ranging from 0 (strongly disagree) to 4 (strongly agree). The survey also included additional open-ended questions regarding utility and acceptability of the ACS3.
Procedure
Potential participants completed a brief survey to determine eligibility. Testing was completed within a single session at the University of Missouri campus. Assessment administration order was counterbalanced to help minimize order effects. People entering the study with even participant identification numbers received the ACS followed by the ACS3; those with odd participant identification numbers received the ACS3 followed by the ACS. Participants completed the assessments in a quiet room with minimal distractions and a researcher present. For the ACS, the researcher used standardized verbal instructions to guide the participant. For the ACS3, standardized instructions were provided by the electronic device. After completing each assessment (i.e., the ACS and ACS3), the participant completed the written acceptability survey for the respective assessment.
Data Analysis
All data were checked for accuracy. Measures of central tendency were calculated for continuous demographic data, and frequencies were calculated for categorical demographic data. All outcome data were first checked for normal distribution using Shapiro–Wilks tests. For any variables that were not normally distributed, nonparametric analyses were conducted.
To evaluate concurrent validity, Spearman’s ρ correlations were used to evaluate the relationship between the ACS and ACS3 for the current activity subscores and total current activity score. Mann–Whitney U tests were used to evaluate differences in group scores as a function of order administration (i.e., whether the order of testing influenced outcomes). To evaluate acceptability, qualitative response data were analyzed according to themes using inductive qualitative content analysis. Thematic coding was conducted by the first author, who has prior experience and publications with this method of qualitative analysis. Quantitative survey data were analyzed using descriptive statistics. All quantitative data analyses were performed with IBM SPSS Statistics (Version 25).
Results
Twenty participants were recruited and completed this study. Overall, the sample was middle-aged with a high level of education (Table 1). There were no overall trends for higher scores in either the ACS or ACS3 total scores; however, there were higher group subscores for the ACS in social activities and higher group subscores for the ACS3 in high- and low-demand leisure activities and IADLs.
Participant Demographics (N = 20)
Concurrent Validity
Results indicate very strong relationships for all current activity subscores, with Spearman’s ρ values equal to .836 or higher (Table 2). The total scores for ACS and ACS3 current activities also demonstrated a very strong relationship (r s = .863). Change in total scores (second test score − first test score) did not differ significantly as a function of which test was administered first (U = 38, p = .393).
Relationships Between ACS and ACS3 Domain and Total Scores
Note. ACS = Activity Card Sort; ACS3 = electronic Activity Card Sort; IADLs = instrumental activities of daily living.
p < .01.
Acceptability
The ACS3 was perceived by participants as being useful and acceptable. All 20 participants completed the acceptability survey, including open-ended responses. Six of seven items on the acceptability Likert scale questionnaire had a mean score of 3.7 or higher, indicating high levels of acceptability (Table 3). The only item that had a mean score <3.7, “The ACS3 helped me realize that there are things I have given up or was doing less that I want to do,” received a mean score of 3.0. Qualitative themes regarding the ACS3 include positive perceptions of ease of use, efficiency, and visual appearance (Table 4). A few participants noted that the ACS3 included stereotypical presentation of gender roles for select activities.
Quantitative Findings on Acceptability of the ACS3
Note. ACS = Activity Card Sort; ACS3 = electronic Activity Card Sort.
Qualitative Findings on Acceptability of the ACS3 and ACS
Note. ACS = Activity Card Sort; ACS3 = electronic Activity Card Sort.
Discussion
This study was conducted to evaluate the concurrent validity of the ACS3 by comparison with the gold standard of the original ACS. Additionally, this study aimed to establish the acceptability of the ACS3. High correlations were found between each domain (IADLs, social activities, low-demand leisure activities, and high-demand leisure activities) and total current activities score. This finding demonstrates high concurrent validity of the ACS3 because scores were comparable with the ACS. These findings are consistent with other studies exploring the relationships between paper and electronic versions of self-report tools. Gwaltney et al. (2008) and Muehlhausen et al. (2015) conducted sequential meta-analyses evaluating two different time periods of studies comparing paper and electronic versions of patient-reported outcome measures. Findings of both of these meta-analyses indicate a high degree agreement between modes of administration for a variety of different populations and outcome areas (Gwaltney et al., 2008; Muehlhausen et al., 2015).
Participants generally found the ACS3 to be highly acceptable and slightly easier to use compared with the ACS. Mean scores of ≥3 (on a Likert scale ranging from 0 to 4) for each item regarding acceptability and usability for the ACS3 indicate that it was acceptable to the end user. A few participants perceived the ACS3 as depicting stereotypical gender roles, with select activities having three photographs representing only one gender. This feedback informed a recently completed revision of the ACS3 to make activity photographs equally representative of genders. Participants found the ACS3 easier to use because of the fine motor demands of holding cards in the ACS. A lower rating for the acceptability item “The ACS3 helped me realize that there are things I have given up or was doing less that I want to do” may be due to the use of a community sample as opposed to a clinical sample for whom this item would have been more appropriate.
Our findings of a high level of acceptability align with prior research on the use of electronic patient-reported outcome measures. Other studies have supported client preference for electronic forms in spite of an assumption that older adults would experience a barrier in the use of electronics that could induce anxiety (Gwaltney et al., 2008; Howell et al., 2017). Additionally, the use of electronic systems appears to be more time efficient and to have greater reliability for data management (Howell et al., 2017).
During a time of immense focus on clinical efficiency, client satisfaction, and tangible functional outcomes, clinical utility is paramount. High clinical utility is required and is reflected in efficiency, technical feasibility, required training, cost, ease of access to materials, sound psychometrics, and meaningful outcomes (Smart, 2006). The ACS3 meets each of these components because it is cost-effective, is quick to administer, and does not require the printing or transporting of physical materials. It also eases the data management burden through automatic scoring and data transfer into electronic documentation. The ACS3 has high face validity with the inclusion of real-world activities and more diverse demographics in the photos used.
Limitations and Recommendations for Future Research
The generalizability of the findings from this study is limited because only the items shared in common by the ACS and ACS3 (∼95% of items) were compared. Use of a sample with lower education or with cognitive difficulties may lead to different findings. Future research is needed to confirm these findings in a larger, more diverse sample. Additionally, research establishing the sensitivity to change of the ACS3 is required to inform its use as an outcome assessment.
Implications for Occupational Therapy Practice
The results of this study have the following implications for occupational therapy practice: The ACS and ACS3 produce consistent scores on overlapping items, suggesting that the ACS3 may be an acceptable measure of daily life participation. The ACS3 may provide a visually appealing, ergonomic method of gathering participation data and is perceived as acceptable by stakeholders. Electronic forms of patient-reported outcomes should be considered for minimizing data entry errors, providing automatic scoring, and facilitating entry of results into documentation.
Conclusion
Health care is becoming increasingly electronic in both delivery (e.g., telehealth services) and documentation. Development of virtual forms of assessments may be easier and faster to implement. The ACS3 demonstrated high levels of acceptability and concurrent validity with the ACS. Future research is needed to further inform ACS3 psychometric properties and its use in clinical populations.
Footnotes
Acknowledgments
Research reported in this publication was supported by the National Institute of Nursing Research of the National Institutes of Health under Award Number R44NR016183. This research was fully funded with federal grant support. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. We acknowledge the work of Dershung Yang, BrightOutcome Inc., and all previous contributors, including faculty, clinicians, and students, who made this work possible.