Abstract
Cognitive impairment is a common consequence of mild stroke. Current performance-based assessments for mild stroke can detect mild impairments in executive function but lack alternate forms to be used as outcome measures. This study aimed to develop an alternate form of the Complex Task Performance Assessment (CTPA–Alt), a performance-based assessment of executive function, and to establish the alternate form reliability of the CTPA–Alt. A repeated-measures study was conducted with 26 community participants. Participants were screened for eligibility and administered both forms of the CTPA; administration order was alternated. Overall performance was significantly correlated (r s = .44, p = .03), but pattern of scoring differed by CTPA form and order of administration. Our results indicate that the CTPA forms were similar but that the specific tasks in each form were different. The CTPA may be used as an ecologically valid outcome assessment with further considerations.
Each year, more people live with chronic deficits resulting from stroke (Mozaffarian et al., 2016). People affected by stroke present with complex and diverse symptoms. One of the most complex chronic symptoms is cognitive impairment, with up to 70% of stroke survivors exhibiting some degree of deficit (Lindén, Skoog, Fagerberg, Steen, & Blomstrand, 2004). Up to 49% of the total stroke population experience mild stroke (Wolf, Baum, & Connor, 2009). People with mild stroke often exhibit good functional scores in basic activities of daily living (ADLs; e.g., dressing, eating, bathing), but executive function deficits are present in 30%–60% of this group (Wolf, Barbee, & White, 2011).
In stroke rehabilitation and research, evaluation of executive function is essential because of the role it plays in planning and executing complex ADLs and instrumental activities of daily living (e.g., medication management, cooking, driving; Crescentini, Seyed-Allaei, Vallesi, & Shallice, 2012). Neuropsychological tools are considered sensitive enough to detect specific executive function deficits; however, it is not uncommon for patients with milder neurological injuries to score within normal limits on these assessments and still report difficulty completing everyday life activities (Shallice & Burgess, 1991) because of the tools’ inability to detect deficits that occur when multiple cognitive skills are used simultaneously in real-world situations.
The Complex Task Performance Assessment (CTPA; Wolf, Morrison, & Matheson, 2008) is a performance-based, ecologically valid assessment of executive function that uses work-related tasks such as calculating, scheduling, and organizing. The validity of the CTPA was published for a sample of people with executive dysfunction after mild stroke; however, a second administration of the CTPA revealed strong learning effects (Wolf, Dahl, Auen, & Doherty, 2015), contraindicating use of the CTPA as an outcome measure to evaluate recovery or the impact of an intervention. The aims of this study were to develop an alternate form of the CTPA (CTPA–Alt) and to evaluate the alternate form reliability of the CTPA–Alt.
Method
Design
A repeated-measures design with two assessment time points 1 wk apart was used. The parallel forms method (Murphy & Davidshofer, 2005) of constructing a new form of the assessment to compare against the original was used in this study to develop and evaluate the CTPA–Alt. Community participants were recruited from Volunteers for Health, a patient registry maintained by Washington University School of Medicine in St. Louis, MO. This study was reviewed and approved by the university's Human Research Protection Office. All testing sessions occurred at the Program in Occupational Therapy at Washington University in St. Louis School of Medicine in a private testing room.
Personnel Training
Training for CTPA evaluators included review of the manual with an approved test administrator, practice in administering the instrument with participants, and test administration with approval by an appropriately trained clinician. The main investigator followed the same protocol to train two evaluators in the administration of the CTPA–Alt with consultation from the primary developer of the CTPA.
Participant Selection
Community participants met the following inclusion criteria: age 18–65 yr; ability to speak English, and score ≥85 (within normal limits) on the Wechsler Test of Adult Reading (WTAR; Psychological Corporation, 2001). The WTAR is a measure of intelligence in which respondents read a list of 50 words aloud and are scored on their ability to pronounce the words correctly, with higher scores indicating higher levels of estimated intelligence.
Participants were excluded if they had a history of neurological or mental health disorder or drug or alcohol abuse, blindness, score of ≥20 (severe depression) on the Patient Health Questionnaire (PHQ–9; Kroenke & Spitzer, 2002), or documented executive dysfunction indicated by the Delis–Kaplan Executive Function System (D–KEFS; Delis, Kaplan, & Kramer, 2001; scaled score ≤7 on Condition 4 of either the Trail Making Test [TMT] or Color–Word Interference [CWI] subtests). The PHQ–9 and D–KEFS had been used for participant inclusion in previous studies with the CTPA (Wolf et al., 2008, 2015). The PHQ–9 consists of nine questions addressing common symptoms of depression; results range from no depression (1–4 points) to severe depression (20–27 points). Condition 4 of the TMT and the CWI were used because they specifically target executive function; scores reflect both time to complete and number of errors (Delis, Kramer, Kaplan, & Holdnack, 2004; Delis et al., 2001).
Development Process
Construction of the CTPA–Alt.
Three key principles of ecological validity were considered in developing the CTPA–Alt:
Representativeness, or the extent to which the CTPA–Alt corresponds in form (i.e., clerical tasks) and context (i.e., office space setting) to the original CTPA
Generalizability, or the degree to which respondents’ performance on the test is predictive of struggles and success in daily life (Burgess et al., 2006)
Multitasking, or the ability to prioritize, organize, and execute various tasks concurrently (Burgess, Veitch, de Lacy Costello, & Shallice, 2000).
The development of the CTPA–Alt required fundamental changes to the nature of the task demands between versions to create a novel assessment, which is necessary for alternate versions of executive function performance-based measures. The reason for applying these principles in clinical evaluation is to better target how participants would react in real-life situations, in which they are rarely presented with structured choices.
To achieve uniformity between CTPA forms, we considered content equivalency, response processes, and analogous scoring (Murphy & Davidshofer, 2005). Content equivalency was addressed by incorporating the same multitasking criteria from the Multiple Errands Test (MET; Shallice & Burgess, 1991) and the CTPA (i.e., multiple tasks, interleaving, one task at a time, interruptions and unexpected outcomes, delayed intentions, differing task characteristics, self-determined targets, and no immediate feedback). Accuracy in response processes was addressed with an organized work setting and standardized instructions. Analogous scoring from the original CTPA was used, including inefficiencies, rule breaks, interpretation failures, task failures, and task accuracy.
Pilot Testing.
Six graduate students volunteered to take the CTPA–Alt in the 3 mo before study recruitment. Pilot participants then completed a questionnaire about clarity of the instructions, difficulty of tasks and distractors, and other aspects they believed needed to be changed. Feedback was analyzed and discussed among the research team, resulting in the reduction of verbal instructions, disambiguation of tasks, and simplification of the assignments and tasks displayed on the index cards.
Instruments
The CTPA is a performance-based evaluation that simulates working at a library. The CTPA was developed on the basis of criteria and scoring of the MET. In the CTPA, two tasks occur simultaneously: (1) calculation of library clients’ fees (inventory control) and (2) telephone messaging. Specific details about the CTPA have been previously published (Wolf et al., 2015), and the CTPA is available in the public domain (see Wolf, Morrison, & Matheson, 2011).
The CTPA–Alt simulates working in a financial institution. Like the original CTPA, the CTPA–Alt contains two tasks that occur simultaneously: (1) completion of a calendar sheet for scheduling appointments and (2) telephone messaging. Administration time and scoring of the CTPA–Alt follow the same criteria as the original CTPA.
Completion of the calendar sheet entails organizing the schedule of a bank manager in which lunchtime activities have been previously entered. The participant must enter 16 additional tasks and appointments. Instructions are displayed on 16 cards that include the name, length, and details of the tasks and appointments. The timing of appointments is fixed or immovable for some and flexible for others. Respondents are instructed to enter all activities displayed on the cards into the calendar sheet. They are also instructed to work as quickly and efficiently as possible.
Telephone messaging requires the participant to listen to prerecorded telephone messages and respond appropriately. Four messages are played within the first 15 min of the assessment. Messages fall under one of three categories described in the original CTPA: declarative, interactive, and reasoning. One caveat provided to CTPA–Alt respondents is that messages can also conflict with the information displayed on the index cards. In this scenario, messages take priority over the index cards, meaning that conflicting information must be resolved by prioritizing the information attained from the phone message.
Five primary scoring criteria, based on the original scoring criteria from the MET, are used with both versions of the CTPA: (1) inefficiencies, (2) rule breaks, (3) interpretation failures, (4) task failures, and (5) task accuracy. Inefficiencies occur when respondents could have used a more efficient method but completed the task correctly. Respondents are given a list of rules to follow (e.g., don’t talk to the examiner), and each time they break a rule it is counted as a rule break. Interpretation failures occur when respondents attempt to complete a task but, because they misinterpret the instructions, do not complete it correctly (e.g., scheduling an activity at a time that is not supposed to be scheduled). Respondents receive a list of tasks to complete, and any task not completed for any reason is counted as a task failure. Task accuracy relates to the primary task (e.g., CTPA–Alt calendar sheet, CTPA inventory control) and reflects the total number of items on the task that were either left blank or filled in incorrectly. All scores are summed to provide the overall score within each criterion and then totaled to provide the overall score for the CTPA or CTPA–Alt.
Procedure
All testing sessions occurred in a private testing room. After potential participants provided informed consent, they were administered the WTAR, D–KEFS, and PHQ–9 in two sessions scheduled 7 days apart. Those who met the eligibility criteria were administered a form of the CTPA. Administration protocol followed the counterbalancing method (Cozby & Bates, 2012) in which CTPA forms were alternated between administrations.
Analysis
All data were cleaned and checked for errors. Demographic characteristics of the participants and frequency tables or measures of central tendency and spread (continuous variables) are reported in Table 1. The data distribution was evaluated and found not to meet the assumptions necessary for parametric testing, so nonparametric analysis methods were used. Spearman correlations were used to measure the association between outcomes on both forms of the CTPA. Wilcoxon rank-sum tests were used to compare scores of participants who received the CTPA–Alt first with scores of those who received it second (Table 2). Variables of interest were determined as follows:
Percentage improvement in CTPA total score was calculated as the difference in scores between the first and second administration of the tests. This difference was standardized into z scores. Percentage improvement was designed to assess expected learning effect. Negative values indicate improvement over the first test, regardless of version.
Difference in task fail total score was calculated as the difference for the task fail total scores between the first and second administration of the tests. Difference in task fail was designed to assess the extent of task failure. Negative values indicate improvement over the first test, regardless of version.
Demographic Characteristics and Screening Results (N = 26)
Note. GED = general equivalency diploma; M = mean; SD = standard deviation; WTAR = Wechsler Test of Adult Reading.
Comparison of Mean Scores on Both Forms of the Complex Task Performance Assessment (N = 26)
Note. CTPA = Complex Task Performance Assessment; CTPA–Alt = Complex Task Performance Assessment alternate form; M = mean; NA = not applicable; SD = standard deviation.
p < .05.
Finally, χ2 and Fisher’s exact tests were used to determine statistical association for categorical outcome variables of interest. Statistical analyses and graphics were generated using IBM SPSS Statistics Version 23 (IBM Corp., Armonk, NY) and SAS Version 9.4 (SAS Institute, Cary, NC).
Results
Twenty-six participants completed the two assessment sessions. Performance on the CTPA–Alt was poorer on average compared with the original CTPA for all items except task accuracy (see Table 2).
Scores were compared using Spearman’s correlation. Overall CTPA scores were significantly correlated (r s = .44, p = .03). Correlations between subscales ranged from −.22 to .38 and were not statistically significant. These results indicate that although overall performance was related, the type and frequency of performance were different between the two CTPA versions.
Wilcoxon rank-sum tests comparing scores for participants who were given the CTPA–Alt first versus second revealed a significant increase in overall percentage improvement (d = 1.75, p < .01; Figure 1) but significantly more task failures when the CTPA–Alt was given first (d = 1.02, p < .01; Figure 2). χ2 tests used to determine any association between the order of test administration and demographic characteristics indicated no statistically significant associations between order of test administration and dichotomous measures of education (i.e., less than high school vs. high school or more), employment (i.e., full-time vs. not full-time employment), and race and ethnicity (i.e., White non-Hispanic vs. other). In addition, no age differences were found between those who received the CTPA–Alt first and second.
Standardized improvement (z scores) or expected learning effect for total scores.
Difference in task fail total scores.
Discussion
This study aimed to develop the CTPA–Alt and to establish its alternate form reliability. We found a moderate significant correlation (r s = .44, p = .03) between overall scores on the CTPA and CTPA–Alt, supporting the two forms of the test as related. In addition, the data related to the individual scoring constructs (e.g., task failures) suggest that the two versions of the CTPA are sufficiently different to evaluate mild deficits in higher level cognition. This finding reflects a fundamental aspect of ecological validity, in which some degree of novelty is required to represent real-life situations (Burgess et al., 2006). Findings are consistent with results from previous studies (Baum et al., 2016; Hahn et al., 2014) that found no statistically significant performance differences between forms of the assessment.
Executive function tests (i.e., D–KEFS subtests) correlated more strongly among themselves than with either form of the CTPA, likely because of clear differences in testing approaches; although the CTPA focuses on completion of several tasks that occur simultaneously, neuropsychological tests focus on single tasks that are more time oriented. This difference in testing approaches might explain why a ceiling effect in both versions of the CTPA was less likely.
The parallel test method provided a partial solution to characteristic weaknesses of test–retest procedures. Consistent with the principles explained by Murphy and Davidshofer (2005), carryover effects were not observed when comparing mean scores by level of education, age, race, and employment status as evidenced by our nonsignificant χ2 calculations, which showed no substantial differences among demographic groups when either version of the CTPA was administered first.
Although the Wilcoxon rank-sum test indicated significant differences in scores on the two forms of the CTPA, this difference is likely associated with the noticeably different tasks, which were necessary to have novelty in both forms of the assessment. Therefore, the change in this metric was likely to be higher for the group that undertook the more difficult form of the assessment first. The question as to which assessment is more challenging requires some additional explanation.
When comparing performance between CTPA forms, mean scores on all items but one (task accuracy) were higher for the CTPA–Alt. This finding means that participants had higher overall scores on the CTPA–Alt compared with the CTPA, regardless of whether it was given first or second. So although our sample was more likely to complete the calendar task (CTPA–Alt) more accurately than the math task (CTPA), the additional components added to the CTPA–Alt ultimately created a more challenging assessment overall.
Learning effects make serial administration of the CTPA undesirable (Wolf et al., 2015), so the CTPA–Alt provides an additional form of this instrument to use in detecting mild impairments in cognition. Although our sample showed significant improvement in overall score when taking the CTPA–Alt first, comparison of z scores of the overall scores indicates that the learning effect was markedly less than that reported with serial administration of the CTPA (Wolf et al., 2015). Even so, special consideration should be given to the order of administration of the CTPA versions to maximize utility and minimize learning effects; the original version of the CTPA should be given first. In addition, because the nature of the tasks was changed to create a novel assessment, the scoring pattern within each performance domain (e.g., task accuracy) was different between versions, and order of administration affected this pattern. Future studies with a powered sample should seek to quantify the learning effect and the pattern of errors to allow for score correction.
The primary limitation in this study was the small sample size, which contributed to high variance in the data that may compromise the generalizability of the results. Future investigation should evaluate the CTPA in a larger, homogeneous sample to allow for more robust parametric testing. Additionally, further research is recommended to evaluate sensitivity to change using both forms of the assessment as an intervention outcome measure. Validation studies of this instrument with other populations affected by dysexecutive syndrome are also recommended.
Implications for Occupational Therapy Practice
The CTPA is an ecologically valid performance-based assessment of executive function, and this study helped develop and evaluate an alternate form of the CTPA (CTPA–Alt). Results from this investigation have the following implications for occupational therapy practice:
The two forms of the CTPA may be used in conjunction as a performance-based outcome measure to evaluate changes in performance-based higher level cognitive function.
Order of administration matters with this test; the original CTPA should be given to clients first.
Both versions of the CTPA have a simple, analogous performance-based scoring method that evaluates adherence to the instructions and efficiency and accuracy in completion of tasks. Both CTPA forms also have analogous, standardized instructions for administrators to follow and read aloud. The simplified scoring criteria and instructions benefit administrators in that training for one form can be related to the other; however, attention must be given to the unique task demands of each CTPA form and how they may influence performance.
Footnotes
Acknowledgments
This study was funded by the Boston Rehabilitation Outcomes Center, Medical Rehabilitation Infrastructure Network, National Institutes of Health (NIH) Grant No. R24-HD065688-04. Timothy J. Wolf received salary support from the National Center for Medical Rehabilitation Research in the NIH National Institute of Child Health and Human Development under Award No. K23HD073190. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH. We thank students and staff of the Performance, Participation, and Neurorehabilitation Laboratory of Timothy J. Wolf at the Washington University School of Medicine in St. Louis for their support in carrying out this project. We also thank the students who volunteered to take the assessment and provided feedback and the community participants who took part in this research.