Abstract
The EFPT–E was found to be a promising novel measure of executive function in cancer-related cognitive impairment
The chronic effects of cancer treatment on cognitive function and life participation are generally acknowledged but often poorly identified and managed. Cancer-related cognitive impairment (CRCI) occurs in the majority of women who receive treatment for breast cancer and is associated with decreased occupational performance, community participation, and productivity (Janelsins et al., 2011; Reid-Arndt et al., 2009).
Neuropsychological assessments of executive function (EF) are considered the gold standard of evaluation but are unable to identify functional cognitive changes (Lezak et al., 2012). Performance-based assessments of EF provide more accurate estimations of cognitive abilities in real-world tasks and settings (i.e., functional cognition; Morrison et al., 2015). The Executive Function Performance Test (EFPT) is a reliable and valid performance-based assessment of EF that has demonstrated good reliability and validity (Baum et al., 2008; Wolf et al., 2010). Psychometric testing of the EFPT has revealed a ceiling effect that may reduce sensitivity to milder deficits in EF consistent with CRCI (Baum et al., 2008).
The purpose of this study was to develop and evaluate the Executive Function Performance Test–Enhanced (EFPT–E; Wolf & Baum, 2016), which uses more complex instrumental activity of daily living tasks to moderate the ceiling effect of the EFPT. Our aims were to develop the EFPT–E and to evaluate its (1) interrater reliability, (2) known-groups validity, and (3) concurrent validity in women who had been treated for breast cancer.
Method
Executive Function Performance Test–Enhanced
The EFPT–E is a performance-based assessment of mild executive dysfunction adapted from three of the four subtasks of the original EFPT: (1) Cooking, (2) Medications, and (3) Paying Bills (Baum & Wolf, 2013; Wolf & Baum, 2016). A short description of the EFPT–E subtasks and a comparison with the subtasks of the original EFPT are provided in Table 1. The EFPT–E is scored using the same standardized, hierarchical cueing system as the EFPT, beginning with verbal guidance and progressing to gestural guidance, verbal direct instruction, physical assistance, and completing the subtask for the respondent. Each subtask score represents the highest level of cueing required to support the completion of the task; a lower overall score indicates that less assistance was needed. The only difference in scoring between the EFPT and the EFPT–E is the latter’s addition of counting the number of cues to achieve more sensitive measurement.
Comparison of EFPT and EFPT–E Subtasks
Note. EFPT = Executive Function Performance Test; EFPT–E = Executive Function Performance Test–Enhanced.
Evaluation of Interrater Reliability Testing
Eight occupational therapy graduate students were trained by the test developers to administer the EFPT–E in three phases: (1) didactic training, (2) calibration, and (3) demonstration. During didactic training, graduate student raters met with the investigative team to review procedures for administration until they could competently describe the procedures. In the calibration phase, raters were required to administer the EFPT–E to a series of healthy volunteers. In the demonstration phase, each rater independently administered the EFPT–E. Mastery was considered established when raters reached an 80% level of agreement with an investigator for a given trial administration.
Next, all raters were asked to independently score a recorded administration of each EFPT–E subtask in which a simulated client made errors. Raters were required to identify each error and write on a scoring sheet the cues that would have been provided, in the correct order, to prevent that error during an actual administration of the EFPT–E. In the video, no administrator was present, only the client performing each of the subtasks while making errors. The investigative team independently assessed each rater’s scoring sheet for accuracy and then compared results to ensure accuracy. Each rater’s final accuracy score for each subtask and the overall EFPT–E were used in the analysis for evaluation of interrater reliability.
Evaluation of Validity
Participants and Procedure
A sample of women who had been treated for breast cancer (n = 12) and community control participants (n = 13) were recruited to evaluate the known-groups validity and concurrent validity of the EFPT–E. Both groups met the following inclusion criteria: age 18 or older; community dwelling; knowledge of how to use a checkbook register; able to read, write, and speak English fluently; no cognitive impairment (Montreal Cognitive Assessment [MoCA; Nasreddine et al., 2005] ≥26); and normal estimated intelligence (Wechsler Test of Adult Reading [WTAR; Wechsler, 2001] ≥85). The following criteria caused potential participants to be excluded: color blindness, severe depressive symptoms (>19 on the Patient Health Questionnaire–9 [Kroenke et al., 2001]), history of neurological or severe mental health condition per self-report, or executive dysfunction (≤7 on the Delis–Kaplan Executive Function System [D–KEFS; Delis et al., 2004] Trail Making or Colour–Word Interference tests). The MoCA, WTAR, and D–KEFS criteria were established to ensure our sample scored within normal limits on neuropsychological assessments.
Researcher assistants contacted potential participants by telephone for an initial screening. Those who passed this screening were scheduled for an in-person evaluation to provide written informed consent; complete a final screening; and, if qualified, complete the cognitive assessment battery (Table 2) and the EFPT–E. The University of Missouri institutional review board approved this study, and all participants provided written informed consent.
Descriptions of Cognitive Assessment Batteries
Measures
The assessment battery consisted of neuropsychological cognitive tests and self-report measures of cognitive dysfunction (see Table 2). Two research assistants administered the assessment battery after competency was established with the senior author.
Analysis
Statistical analyses were conducted using IBM SPSS Statistics (Version 25; IBM Corp., Armonk, NY). All data were checked for accuracy. The distribution of all data was evaluated, and nonparametric statistics were used when data deviated from the normal distribution. Interrater reliability of the accuracy among the raters for the EFPT–E total score and each subtask score was evaluated using intraclass correlation coefficients (ICCs).
Descriptive statistics were calculated for demographics and cognitive assessment outcomes for both the community control participants and the women treated for breast cancer. Chi-square tests (categorical data) and Mann–Whitney U tests (continuous data) were used to compare groups. To evaluate known-groups validity, Cohen’s d effect sizes with 95% confidence intervals (normally distributed data) or nonparametric effect size r (nonnormally distributed data) were calculated to estimate the difference between groups for total number of cues for the EFPT–E and each subtask. Concurrent validity between the EFPT–E and the measures in each cognitive assessment battery was evaluated across the entire combined sample. Spearman’s ρ correlation coefficients were used to compare the total number of EFPT–E cues with the cognitive measures.
Results
Evaluation of Interrater Reliability
The results indicated a high level of interrater reliability for all scores on the EFPT–E (total score ICC = .966, Cooking ICC = .902, Medications ICC = .981, Paying Bills ICC = .984).
Evaluation of Validity
Participant Characteristics
Descriptive data for demographics and performance on the cognitive assessments are presented in Table 3. The control participants reported a higher level of education (M = 15.9 yr, SD = 2.0) than the women who had been treated for cancer (M = 14.3, SD = 4.2), and the women with cancer were older (M = 54.0 yr, SD = 10.5) than the control participants (M = 41.2, SD = 18.9). In addition, the women with cancer self-reported greater levels of cognitive difficulties in everyday life on the Cognitive Failures Questionnaire (CFQ; Broadbent et al., 1982; M = 40.3, SD = 12.0) compared with the control group (M = 22.7, SD = 5.0). Means and standard deviations for the EFPT–E, each subtask, and the number of cues per subtask and in total are presented in Table 4.
Participant Demographics and Performance on Screening Measures (N = 25)
Note. CFQ = Cognitive Failures Questionnaire; D–KEFS = Delis–Kaplan Executive Function System; MoCA = Montreal Cognitive Assessment; PHQ–9 = Patient Health Questionnaire–9; WTAR = Wechsler Test of Adult Reading.
p < .05.
Performance Between Groups on the EFPT–E Subtasks and Total Score
Note. CI = confidence interval; EFPT–E = Executive Function Performance Test–Enhanced.
Nonparametric effect size was calculated when data were not normally distributed.
Known-Groups Validity
Moderate effect sizes between groups were found on the normally distributed variables of EFPT–E total score (Cohen’s d = 0.5), total number of EFPT–E cues (Cohen’s d = 0.4), Cooking score (Cohen’s d = 0.5) and the nonnormally distributed variables of total number of cues for Cooking (r = .3), and total number of cues for Medications (r = .3). For all comparisons, the women with breast cancer demonstrated lower levels of performance than the control participants.
Concurrent Validity
Spearman’s ρs revealed a weak correlation between the EFPT–E total number of cues and the MoCA (r = ‒.3). Little to no correlation was identified between the EFPT–E and the WTAR (r = ‒.1), the CFQ (r = ‒.03), the D–KEFS Colour–Word test (r = ‒.04), or the D–KEFS Trail Making test (r = .10).
Discussion
Evaluation of Interrater Reliability
The high levels of interrater reliability demonstrated in this study are consistent with those of other psychometric studies of performance-based cognitive assessments (Baum et al., 2008; Knight et al., 2002). Excellent interrater reliability indicates that, with adequate levels of training, EFPT–E administrators can expect consistency in scores between raters.
Evaluation of Validity
Known-Groups Validity
The effect sizes between groups indicated that the EFPT–E was able to identify a moderate difference between the control participants and the women treated for breast cancer, with the latter demonstrating poorer performance. The Cooking subtask revealed the greatest magnitude of difference, suggesting it may be the most sensitive in discriminating between persons with and without CRCI.
Concurrent Validity
We observed a weak correlation between the total number of EFPT–E cues and scores on the MoCA, but there were no statistically significant correlations with any other cognitive measures; therefore, the concurrent validity of the EFPT–E with neuropsychological measures was not supported. This lack of correlation between the EFPT–E and neuropsychological and self-report measures implies that they measure disparate constructs. Early assessments of functional cognition evaluated construct validity by making comparisons with neuropsychological instruments that gauge isolated cognitive components in tasks that bear little resemblance to everyday activities (Baum et al., 2008; Hartman-Maeir et al., 2009; Wolf et al., 2008). In contrast, functional cognitive assessments appraise the integration of multiple cognitive processes while allowing for the use of strategies to accomplish functional goals. As previously mentioned, mild cognitive impairment may be overlooked in an austere clinical environment, but in the context of a person’s lived experience it can have a negative impact on the fulfillment of complex life roles (Lezak, 1982; Shallice & Burgess, 1991); therefore, performance-based measures of functional cognition and standardized neuropsychological measures of isolated EF skills may not be sufficiently analogous to gauge concurrent validity. Future assessment of the EFPT–E’s concurrent validity should use a comparable measure of functional cognition as a comparator/gold standard (e.g., the Weekly Calendar Planning Activity; Toglia, 2015).
Limitations
This study has some limitations. First, the generalizability of our results is restricted because of the preliminary nature of the investigation. In addition, our sample was limited in that it comprised a small, nondiverse sample and lacked age-, gender-, and education-matched control participants.
Implications for Occupational Therapy Practice
The research presented in this article has the following implications for occupational therapy practice:
The EFPT–E demonstrates promise as a functional cognitive measure for people with CRCI.
Occupational therapy practitioners can expect a high level of consistency of EFPT–E scores between trained raters.
Future studies to validate the EFPT–E should use similar measures of functional cognition to evaluate concurrent validity.
Conclusion
This research yielded positive preliminary findings in support of the EFPT–E’s interrater reliability, and we encourage continued investigation to evaluate the measure’s validity. Comparison with another performance-based assessment will be required to evaluate concurrent validity. The intention of this study was not to definitively evaluate the validity of the EFPT–E but to determine whether further study is indicated. Future research should endeavor to not only establish validity but also evaluate the relationships between EFPT–E construct scores and neuropsychiatric measures of these constructs.
Footnotes
Acknowledgments
We acknowledge Meredith Spiers for her contributions in coordinating this study, and we thank the participants who gave their time and effort to take part in this study.