Abstract
Keywords
Executive functions are necessary to successfully complete instrumental activities of daily living (IADLs; e.g., grocery shopping, public transportation use, work- or school-related tasks) and are essential for completion of goal-directed behavior. Components of executive functions include initiation, sequencing, the ability to ignore irrelevant stimuli, identification and achievement of goals, and working memory (Connor & Maeir, 2011). Deficits in executive functions have been demonstrated in people with a wide range of diagnoses, including brain injury, stroke, multiple sclerosis, chronic heart failure, and Parkinson’s disease (Ferreira, 2010; Konrad et al., 2011; McKinlay, Grace, Dalrymple-Alford, & Roger, 2010; Nys et al., 2007; Pressler, 2008). These deficits are correlated with decreased performance in both activities of daily living and IADLs (Diamond, Felsenthal, Macciocchi, Butler, & Lally-Cassady, 1996; McKinlay et al., 2010; Tatemichi et al., 1994)
Seventy-four percent of people with stroke demonstrate acute cognitive impairment, with disorders in executive functioning being most common (Nys et al., 2007). Stroke survivors with cognitive impairments have greater functional impairment and a higher frequency of discharge to a dependent living environment in comparison to survivors without cognitive impairments (Diamond et al., 1996; Leśniak, Bak, Czepiel, Seniów, & Członkowska, 2008; Tatemichi et al., 1994). Even people with mild neurological impairment resulting from stroke (i.e., National Institutes of Health Stroke Scale score of <6) have reported decreased participation in IADLs, despite rarely being referred for rehabilitation services (Edwards, Hahn, Baum, & Dromerick, 2006; Hildebrand, Brewer, & Wolf, 2012; O’Brien & Wolf, 2010; Rochette, Desrosiers, Bravo, St-Cyr-Tribble, & Bourget, 2007; Wolf, Baum, & Conner, 2009).
Occupational therapy practitioners help people with executive function deficits after stroke improve their ability to perform and participate in their chosen occupations through the use of learned strategies associated with tasks that have meaning to each person. They are skilled in treating people with executive dysfunction using interventions such as metacognitive strategy training (including self-awareness training, goal management training, and cognitive orientation to daily occupational performance), functional task training, and environmental adaptations (Cicerone et al., 2011; Connor & Maeir, 2011; Goverover, Johnston, Toglia, & Deluca, 2007; Henshaw, Polatajko, McEwen, Ryan, & Baum, 2011; McEwen, Polatajko, Huijbregts, & Ryan, 2010).
Before occupational therapy practitioners can intervene with clients who have poststroke executive dysfunction, they need valid and reliable tools to identify the specific impairments that can limit performance in everyday life activities. Traditionally, neuropsychological assessments have been considered the gold standard for the identification of executive function deficits; however, the health care community has long known that people can perform within normal limits on neuropsychological assessments and still have performance-based impairments in activities that require executive function (Burgess et al., 2006; Shallice & Burgess, 1991). This discrepancy is a result of limitations in traditional pencil-and-paper assessments and contradicts current definitions of the components of executive function (Lezak, 1982). Therefore, it is critical for practitioners to use performance-based assessments of executive function with people with neurological injury.
An example of an established performance-based measure of executive function is the Executive Function Performance Test (EFPT; Baum & Wolf, 2013). The EFPT is easily administered and available in the public domain (Baum & Wolf, 2013). However, only one form of each EFPT task is currently available, which limits its use as an outcome measure because a potential learning effect may occur after administration.
This pilot study to develop the Alternate EFPT (aEFPT) had four objectives: (1) to develop additional tasks to complement those currently part of the EFPT, (2) to evaluate the alternate form reliability of the aEFPT, (3) to evaluate the discriminant validity of the aEFPT, and (4) to evaluate the concurrent validity of the aEFPT compared with the EFPT and standardized neuropsychological measures of executive function.
Method
Research Design
This cross-sectional study included poststroke participants (n = 25) and healthy control participants (n = 25). One of the three authors who were occupational therapists at the Rehabilitation Institute of Chicago obtained the written informed consent and administered all study screens and measures. All participants completed the EFPT and the aEFPT in two sessions that were approximately 1 wk apart. During the initial visit, we obtained written informed consent from all potential participants and then screened participants for inclusion. After it was determined that a person met the study criteria, we obtained demographic information, and the participant completed a short neuropsychological assessment battery. After completing the neuropsychological battery, participants were administered either the EFPT or the aEFPT on the basis of an a priori randomization sequence. One week after the initial testing session, we administered the form of the assessment (either the EFPT or the aEFPT) not administered during the initial visit. A local university institutional review board reviewed and approved the study.
Participants
We recruited people with stroke through a local stroke registry that includes information about approximately 500 people with stroke who consented to being approached about participation in research studies. In addition, we recruited participants with stroke and healthy control participants using flyers posted at the Rehabilitation Institute of Chicago’s day rehabilitation, inpatient, and outpatient sites.
Inclusion criteria included diagnosis of unilateral stroke at least 6 mo before study enrollment (poststroke participants only), ability to speak English, and ability to provide informed consent. Exclusion criteria were evidence of dementia on study enrollment as determined by a score of >12 on the Short Blessed Test (Katzman et al., 1983); severe depressive symptoms reflected by a score of >16 on the Center for Epidemiologic Studies–Depression Scale (Radloff, 1977); and current participation in any rehabilitation services addressing cognitive dysfunction, including any occupational or speech therapy.
Measures
All raters completed training on all the measures and were deemed proficient. Administrators were in contact with EFPT developers, and before study enrollment, all raters reviewed multiple videotapes of test administrations, scored the clients separately, and compared their scores to ensure consistency among raters.
Neuropsychological Battery.
The Trail Making Test Parts A and B (Trails A and B; Reitan & Wolfson, 1995) and the Wechsler Memory Scale–Revised Digit Span Forward and Backward Tests (Wechsler, 1987) were used to establish concurrent validity with performance-based assessments. Both Digit Span Tests were administered; however, we used data only from the Backward Test because of its increased complexity.
Executive Function Performance Test.
The EFPT evaluates five executive function constructs—(1) initiation, (2) organization, (3) sequencing, (4) safety and judgment, and (5) completion—as the client completes the four tasks of (1) cooking, (2) telephone use, (3) medication management, and (4) money management. The assessment is scored using a graded cueing system for each of the five constructs and is designed to measure the level of assistance the client needs to be successful with the task. The maximum score after all four tasks are administered is 100 points (25 points for each of the four tasks). The higher the score, the higher the degree of assistance required.
The EFPT has established interrater reliability (ICC = .91 for total score); internal consistency (α = .94); and validity with gold standards Trails B, Digit Span Backward Test and Story Recall from the Wechsler Memory Scale–Revised, Animal Fluency, and Short Blessed Test for the stroke population (all ps < .05; Baum et al., 2008). Reliability and validity of the EFPT have additionally been established in people with multiple sclerosis and schizophrenia (Goverover et al., 2005; Katz, Tadmore, Felzen, & Hartman-Maeir, 2007). Detailed instructions on how to administer and score the EFPT can be found in the EFPT manual available online (Baum & Wolf, 2013).
Alternate Executive Function Performance Test.
The aEFPT was developed to complement the original EFPT by providing additional tasks. The aEFPT tasks were developed by the authors of this article who were occupational therapists at the Rehabilitation Institute of Chicago (Hahn, Spoeri, & Doherty) in collaboration with Carolyn Baum, one of the original authors of the EFPT. The four additional tasks were designed to be similar to the four original tasks but with a novel component to prevent a learning effect from the original form. The additional tasks are all activities within the four original task categories (cooking, telephone use, medication management, and money management), which have been identified as essential skills for independent living (Lysack, Neufeld, Mast, MacNeill, & Lichtenberg, 2003). The aEFPT cooking task involves making pasta instead of oatmeal, as in the EFPT. The telephone use task involves calling a doctor’s office instead of a grocery store. Medication management involves sorting medications into a 7-day pill sorter instead of taking a medication. The money management task involves ordering a specific item from a catalog instead of paying two bills. Administration of the additional tasks is completed using the original EFPT structure. All tasks are scored using the same graded cueing system examining five constructs: (1) initiation, (2) organization, (3) sequencing, (4) safety and judgment, and (5) completion. Time taken to complete each task is recorded.
Data Analysis
All data analysis was completed with IBM SPSS for Windows (Version 19; IBM Corporation, Armonk, NY). Nonparametric statistical analyses were used because of the distribution and nature of the data. The Mann–Whitney U test and χ2 analysis were used to test for demographic differences between the control and poststroke participants. To evaluate the discriminant validity of the aEFPT, the Mann–Whitney U test was used to examine differences between groups on the EFPT and aEFPT. To evaluate the alternate form reliability of the aEFPT, the Wilcoxon signed-rank test was used to examine differences between EFPT and aEFPT scores from the poststroke group. To evaluate the concurrent validity of the aEFPT, Spearman’s rank order coefficients were calculated between scores on the aEFPT, the EFPT, and the neuropsychological measures (α was set at .05).
Results
Sixty-one participants were recruited. One healthy control participant and 7 poststroke participants were screened out because of depression, and 1 poststroke participant was excluded because of dementia. One participant from each group dropped out because of scheduling conflicts. In total, 50 participants, 25 in each group, completed the study.
Demographic characteristics of the study sample are presented in Table 1. Significant differences were found in race (Z = –2.52, p = .01), with poststroke participants being more diverse and having a higher percentage of African-Americans and Asian-Americans than the control group. The control group had a higher percentage of White and Hispanic participants than the poststroke group. Gender distribution was also different between the two groups, with the control group having a larger percentage of women. No significant difference between groups was found in age or education.
Demographic Characteristics, by Group
Note. n = 25 in each group. M = mean; SD = standard deviation.
p < .05.
Discriminant Validity
Task scores were significantly different between the control group and the poststroke group on all tasks for both the EFPT and the aEFPT (Table 2). Construct scores and completion times for the healthy control group and the poststroke group for both performance-based measures are listed in Table 3. Total aEFPT scores (Z = –5.28, p = .00) and times (Z = –5.38, p = .00) were significantly different between the control and poststroke groups. Total EFPT scores (Z = –5.44, p = .00) and times (Z = –5.07, p = .00) were also significantly different between the groups.
Task Scores, by Group
Note. n = 25 in each group. aEFPT = Alternate Executive Performance Test; EFPT = Executive Function Performance Test; M = mean; SD = standard deviation.
p < .05.
Construct Scores and Completion Times, by Group
Note. n = 25 in each group. aEFPT = Alternate Executive Performance Test; EFPT = Executive Function Performance Test; M = mean; SD = standard deviation.
p < .05.
Alternate Form Reliability
Wilcoxon signed-rank test was computed from poststroke participants’ scores on the EFPT compared with the aEFPT to establish alternate form reliability (Table 4). We examined scores for each task and each construct, as well as total score and total time. Total scores were not different between the two tests (Z = –0.768, p = .442). Similarly, construct scores were not different between the two tests. Total time was significantly longer on the aEFPT (Z = 3.579, p = .000). Two tasks, cooking (Z = –0.959, p = .337) and money management (Z = –0.262, p = .793), had no significant differences between tests. However, telephone use (Z = –2.074, p = .038) and medication management (Z = –2.495, p = .013) had significant differences between tests.
Alternate Form Reliability of the aEFPT Compared With the EFPT
Note. aEFPT = Alternate Executive Performance Test; EFPT = Executive Function Performance Test.
p < .05.
Concurrent Validity
Spearman’s rank order correlation coefficient was computed for concurrent validity testing using only scores from poststroke participants. Significant correlations (p < .01) existed between the aEFPT and Trails A (r 2 = .53), Trails B (r 2 = .55), Short Blessed Test (r 2 = .69), and EFPT (r 2 = .63). The correlation between the aEFPT and the Digit Span Backward Test (r 2 = –.38) was not significant.
Discussion
The aim of this study was to develop additional tasks for the EFPT and investigate the psychometric properties in poststroke and healthy control participants. The statistical objectives of this pilot study were to evaluate the alternate form reliability of the aEFPT, the discriminant validity of the aEFPT, and the concurrent validity of the aEFPT compared with the EFPT and standardized neuropsychological measures of executive function.
The total scores on the EFPT and the aEFPT were not significantly different, indicating that the two instruments can be used at two separate points in time to measure an outcome. However, scores on the telephone use and medication management tasks were significantly different between the two instruments. This difference could be attributed to the lower mean test scores and low variability on both these items; for example, 14 participants had a score of zero on the telephone use task, indicating that they required no assistance to complete the task. Another factor contributing to differences in the medication management task could be the increased complexity of the alternate medication management task (i.e., sorting two medications into a pill sorter) compared with the original version (i.e., simply taking a medication). Future research could further study the telephone use and medication management tasks with a larger sample size or a population with more severe impairments to clarify the relationship with the other tasks. Our data thus indicate that the two instruments should be administered in their entirety to provide comparable scores for an outcome. Additionally, the aEFPT tasks of cooking and money management could be used to measure change from the EFPT.
Comparisons between poststroke and control group scores demonstrated a significant difference between the groups on task scores, total scores, and time on both the EFPT and aEFPT. These findings indicate that the EFPT and the aEFPT are sensitive to detecting executive function deficits in the stroke population and are consistent with published research on the EFPT (Baum et al., 2008).
Initiation and completion were the only constructs not significantly different between groups (Table 3). This finding could be the result of poststroke participants having fewer deficits in initiation and completion than in the other constructs, with mean scores ranging from .00 to .36. These findings are consistent with previous research in which Baum and colleagues (2008) found that participants with mild stroke required little assistance with initiation and completion (mean scores were 0.83 and 0.88, respectively).
Task scores for poststroke participants (see Table 2) fell within the “mild stroke” category according to Baum and colleagues (2008), whose participants with mild stroke had mean task scores ranging from 0.92 to 2.98. In the current study, poststroke participants’ mean scores ranged from 0.80 to 2.92 on the EFPT and 0.60 to 2.92 on the aEFPT. The majority of participants in our study had mild executive function deficits, resulting in low variability in scores, particularly with the initiation and completion constructs and the telephone use and medication tasks.
Results of validity testing indicated that the aEFPT significantly correlated with all neuropsychological measures, with the exception of the Digit Span Backward Test. These results are consistent with EFPT validity established by Baum and colleagues (2008) and suggest that these performance-based measures correlate with gold standards for assessing executive functions.
Study limitations include the small sample size and the classification of the majority of the poststroke participants as having mild stroke, which could limit generalizability. Additionally, the small sample size limited our ability to complete a more advanced statistical procedure (i.e., multivariate analysis of covariance). Future research with a larger sample size is warranted to ensure that the differences between healthy control and poststroke participants’ test scores could not be attributed to any of the demographic characteristics. Average years of education was relatively high for all participants, which may have affected test scores; poststroke participants had an average of 14 yr of total schooling and control participants an average of 16 yr.
The increased time participants needed to complete the aEFPT could limit the clinical utility of the additional tasks if they are used collectively. The majority of the additional time was attributed to the money management task. However, the money management task in the aEFPT was generally more complex than the original money management task and therefore would provide additional information beyond what may be gained from the original money management task.
Suggestions for Future Research
Future research should focus on whether the aEFPT would identify executive function impairments in a poststroke population that is more heterogeneous. Further testing and development of the medication management and telephone use tasks should be completed to determine their relationship with the other tasks. Further research to revise the money management portion of the aEFPT and decrease the amount of time required to complete the aEFPT would enhance its clinical utility. Finally, additional tasks for the EFPT should be developed to further establish the clinical utility of this performance-based measure.
Implications for Occupational Therapy Practice
The results of this pilot study have the following implications for occupational therapy practice:
The aEFPT offers a promising way for clinicians to measure an outcome of clinical treatment of executive function deficits.
If administered in its entirety, the aEFPT is not significantly different than the EFPT and is able to detect change in deficits in executive function in the poststroke population.
The tasks of making pasta and ordering from a catalog could be used to measure change after administration of the original EFPT.
The aEFPT is a valid tool to capture performance-based executive function deficits in the poststroke population.
Footnotes
Acknowledgments
This study was carried out with the financial support of The Buchanan Family Fellowship through the Rehabilitation Institute of Chicago (RIC). We additionally thank Jason Raad for his statistical consultation and Donna Zahara, director, RIC Academy, Internal Staff Development, for her continued support of this project.