Abstract
People with stroke and older adults often experience executive function (EF) deficits, which negatively affect their ability to perform complex activities safely and efficiently (Gadidi, Katz-Leurer, Carmeli, & Bornstein, 2011; Gunning-Dixon & Raz, 2003; Katz & Hartman-Maeir, 2005; Nys et al., 2007). Adults with attention deficit hyperactivity disorder (ADHD) who function at a high level in the community are also known to have EF deficits (Barkley, 1997; Katz & Hartman-Maeir, 2005), which also negatively affect their daily occupations (Barkley & Murphy, 2010). Clinicians frequently use performance-based assessments for evaluating clients’ EF deficits to determine their strengths, limitations, and challenges (Baum & Katz, 2010) while performing a functional task. Gaining knowledge about the abilities of clients can help clinicians select the optimal means to promote their engagement in daily life. Neuropsychological assessments have been criticized for lack of ecological validity and accuracy in predicting real-life performance (Burgess et al., 2006).
The Executive Function Performance Test (EFPT; Baum et al., 2008) is used to evaluate components of EF while clients perform four everyday tasks (cooking, telephone use, medication intake, and bill payment). Recently, the alternate EFPT (aEFPT; Hahn et al., 2014), which uses different forms of the original four tasks, was developed. Both the EFPT and the aEFPT have been used with different patient populations, including stroke, and have been found to be valid and reliable (Baum et al., 2008; Hahn et al., 2014). However, two of the four original tasks include the use of checks for bill paying and the use of a phone book during telephone use, which are outdated. Therefore, with the permission of the EFPT authors, we updated both tasks for Internet use and tested both for alternate form reliability (Study 1) and validity (Study 2).
Our aims were (1) to assess the alternate form reliability of the Internet-based tasks in healthy adults and people with subacute stroke and to assess differences between the groups for both versions of the tasks (Study 1) and (2) to assess the criterion reliability of the Internet-based tasks for assessing EF deficits in different populations with known (or without) EF deficits by correlating their performance with that on the Trail Making Test (TMT; Reitan, 1986) in Study 2.
Method
The structure of the two Internet-based tasks was similar to the original tasks. Each task was also presented in much the same manner as the original tasks. The only differences for the bill-paying task were checks were replaced by a credit card and balancing the account was replaced by the line of credit. As in the original task, participants use a balance sheet with the last bank transactions and information about their line of credit. The only change between the original telephone-use task and the Internet-based task was that the Google search engine was used for the latter instead of a phonebook.
The Internet-based bill-paying task was created using a program that simulates an Internet search with payment using a mock credit card (LabVIEW, Version 12; National Instruments, Austin, TX. This software can be freely downloaded from http://www.tau.ac.il/∼portnoys/Internet-based_Bill_Paying_Task.html). As with an Internet search engine, the participant must type search words. The correct search word, such as gas or electricity (or misspellings, e.g., elecrticity), leads to the next page with five results, three of which lead to the correct page for bill payment (e.g., Electric company–private customers) and two that lead to an error page (e.g., What’s new in the Electric company). The participant then inserts the contract number, last four digits of the bill, total sum of payment, means of payment, credit card number and expiration date, and number of payments. If an error is made, after the information is entered, an X appears. When the page is completed correctly, an approval number is provided, and the participant clicks a button to return to the search page to pay a second bill. Similar to the original bill-paying task, participants have to problem solve when there is not enough line of credit to pay both bills.
Throughout the test, all keyboard and mouse data are recorded at a frequency of 100 Hz and saved to an Excel (Microsoft Corp., Redmond, WA) file. The computerized assessment provides computer performance data (e.g., time spent on each web page, number of mouse clicks, travel distance of mouse, number of delete and backspace clicks).
The task rating system was unchanged, with all the original components maintained: initiation, execution (organization, sequencing, judgment, and safety), and completion. However, the components were minimally modified. For example, rating the sequencing of the bill-paying task originally comprised observing whether the participant located the bill due immediately, checked the balance, wrote the check for the correct amount, put the check into an envelope, and sealed it. For the Internet-based task, rating the sequencing was changed to comprises observing whether the participant locates the bill due immediately, fills in the online information correctly, chooses to pay in one or more credit card payments, and writes down the confirmation number. The original cue-scoring system was not changed (from 0 = no cues needed to 25 = not independent).
Study 1: Alternate-Form Reliability of the Internet-Based Tasks and Differences Between Groups
Participants.
Participants with stroke (n = 15) were recruited from a large rehabilitation center. Inclusion criteria were at least 2 wk poststroke, age >18 yr, preserved basic cognition (Montreal Cognitive Assessment [MoCA; Nasreddine et al., 2005] score >21/30 points), lived independently in the community before the stroke, at least 12 yr of education, and used a computer before stroke. Healthy participants (n = 32) were recruited by advertisements posted at the university and in the rehabilitation center. Inclusion criteria were at least 12 yr of education, independent living in the community, and use of computers. Exclusion criteria were neurological or psychiatric conditions. The study was approved by Sheba Medical Center’s Helsinki committee and the university’s ethics committee, and all participants provided written informed consent.
Tools and Procedure.
This cross-sectional study included one session in which participants performed two tasks (bill paying and telephone use) twice: once using the original EFPT tasks and once using the Internet-based tasks. The order of the versions was counterbalanced to rule out potential order effects. Different bills were used for the original version (i.e., water and phone) and the Internet-based version (i.e., gas and electricity).
Data Analysis.
Data were analyzed using IBM SPSS Statistics (Version 21; IBM Corp., Armonk, NY). Descriptive statistics were used to describe the study population, the total and component scores, and the time for both the original and Internet-based tasks. An independent-sample t test was conducted to assess differences between tasks administered first and administered second. Alternative-form reliability was verified by the 95% limits of agreement using Bland–Altman plots (Bland & Altman, 1986; Giavarina, 2015) to assess agreement between the two versions of the tasks. Pearson correlations assessed the correlations between the scores and the time to complete each of the tasks in each group. Differences between groups (subacute stroke and healthy) for both versions of the two tasks were assessed using an independent-sample t test. The statistical level was set at p < .05.
Results.
Thirty-two healthy adults (10 men, 22 women) and 15 adults with stroke (11 men, 4 women) participated in this analysis. For the stroke group, 9 had ischemic and 6 had hemorrhage stroke, and mean months poststroke were 4.1 (standard deviation [SD] = 3.1). All had preserved cognitive abilities (MoCA score = 24.1 points [SD = 1.7]). In healthy adults, mean age was 42.5 yr (SD = 18.9; range = 22–68), and mean years of education were 14.0 (SD = 1.3). In adults with stroke, mean age was 53.6 yr (SD = 10.9; range = 35–68) and mean years of education were 14.3 (SD = 3.6).
No significant differences were found for the bill-paying total scores or time between participants who performed the Internet-based version first (t = –1.3, p = .187) and those who did it second (t = 0.3, p = .76). Similarly, differences were not found for the telephone-use total scores or time between participants who performed the Internet-based version first (t = –1.4, p = .16) and those who did it second (t = 0.56, p = .57). Therefore, we ruled out the order effect.
The mean total and EF component scores of the original and Internet-based tasks are presented in Table 1. From a visual examination of an average-difference Bland–Altman plot, the global agreement between the two measurements (original and Internet-based) for both tasks can be evaluated. The average-difference plot of the bill-paying task shows good agreement between the two versions because most data points fall within ±2 SD boundaries of the difference, and the mean difference is –0.9 points, with the 95% confidence interval (CI) ranging from –2.02 to –0.52. The limits of agreement are –4 to 3.1 points. For telephone use, the average-difference plot shows good agreement between the two versions because all the data points fall within ±2 SD boundaries of the difference, and the mean difference is 0.0 points, with the 95% CI ranging from –0.57 to 0.57. The limits of agreement were –3.0 to 3.0.
Original and Internet-Based Task Scores and Times for Healthy Participants and Those With Subacute Stroke
Note. EFPT = Executive Function Performance Test; M = mean; SD = standard deviation. Asterisks indicate significant differences between stroke and healthy groups: *t = –2.4 (p < .01); ** t = –2.2 (p < .03); *** t = –2.0 (p < .04).
For all the participants, a moderate to high significant correlation was found between the total scores of the original and Internet-based bill-paying tasks (r = .67, p < .001) but not for time to complete each task and not for the component scores. For telephone use, a moderate significant correlation was found for all participants between versions only for the time to complete the task (r = .58, p < .001) and not per group.
Bill paying performance was significantly better for the healthy participants for the original (total score and time) and the Internet-based (total score) task (see Table 1). Significant differences were not found between groups for the telephone-use task.
Study 2: Criterion Validity of the Internet-Based Tasks
The Internet-based bill-paying task was completed by the following groups, who have varying levels of executive functioning: healthy young (n = 30) and older (n = 17) adults, people with subacute (n = 15) and chronic (n = 15) stroke, and young participants with ADHD (n = 22). Telephone use was administered to three of these groups: healthy young and older adults and people with subacute stroke. To establish the criterion validity, we assessed the correlations between the Internet-based tasks to a pen-and-paper assessment of EFs for all participants and within groups. The TMT was selected because it is a well-known and commonly used quick assessment of EF. It specifically assesses visual scanning, divided attention, and cognitive flexibility, which are needed for searching a telephone number on the Internet but more so for bill payment online. The TMT was used previously to establish validity of the original EFPT tasks (Baum et al., 2008).
Participants.
In addition to the 47 participants reported in Study 1, 15 participants with chronic stroke and an additional 15 healthy participants were recruited. Twenty-two young adults with ADHD were also recruited. The sample consisted of 99 participants. Inclusion criteria were the same as for Study 1 for the healthy participants and for those with subacute stroke. People with chronic stroke were at least 6 mo poststroke, lived in the community with preserved basic cognition (as assessed by a Mini-Mental State Examination [MMSE; Folstein, Folstein, & McHugh, 1975] score >24/30 points), and used a computer. They were recruited via discharge lists from a large rehabilitation center and by snowball sampling. The inclusion criteria for the participants with ADHD were a valid diagnosis of ADHD by a neurologist or psychiatrist and no learning disabilities. The study was approved by Sheba Medical Center’s Helsinki committee and the university’s ethics committee. All participants provided written informed consent.
Tools and Procedure.
Internet-based bill paying was completed by all participants, but because of time restrictions, only 45 participants (30 healthy and 15 with subacute stroke) also performed the Internet-based telephone-use task. In the TMT, completed by all participants, the participant is required to draw a line from digit to digit in ascending order (Part A; TMT–A) or alternating between consecutive digits and letters (i.e., 1 to A, A to 2, 2 to B, and so forth; Part B [TMT–B]), and the completion time is measured (Bowie & Harvey, 2006). Because of time restrictions, TMT–B was stopped after 5 min.
Data Analysis.
Descriptive statistics were used to describe the sample, the scores of the Internet-based tasks, and the TMT scores. Pearson correlations assessed associations between the Internet-based tasks and the TMT for all participants and within each subgroup to support criterion validity. All analyses were set at a significance level of p < .05.
Results.
Five groups with a total of 99 participants took part in this study: 22 adults with ADHD (15 men, 7 women; mean age, 25.1 [SD = 2.3]), 30 healthy young adults (6 men, 21 women; mean age, 25.5 [SD = 2.2]), 17 healthy older adults (6 men, 11 women; mean age, 57.2 [SD = 13.8]), 15 people with subacute stroke (11 men, 4 women; mean age, 53.6 [SD = 10.9]; mean mo since stroke, 4.1 [SD = 3.1]), and 15 people with chronic stroke (9 men, 6 women; mean age, 61.1 [SD = 10.3]; mean mo since stroke, 16.8 [SD = 11.5]; mean MMSE score, 28.8 [SD = 0.91] out of a maximum of 30 points). The criterion validity was assessed by the correlations between the total scores and time of the Internet-based tasks and the TMT. Significant moderate correlations were found between Internet-based bill paying (total score and time) and the TMT (Parts A and B) for all participants and within groups, with stronger correlations for the groups with stroke (see Table 2). The time to complete the Internet-based telephone-use task also significantly correlated to TMT–A and –B for all participants, with stronger correlations for the older adults and the participants with stroke. However, telephone use (total score) was not significantly correlated to the TMT (see Table 2).
Pearson Correlation Coefficients Between the Score and Time of Internet-Based Tasks and the TMT
Note. ADHD = participants with attention deficit hyperactivity disorder; All = all participants and subgroups; CS = participants with chronic stroke; O = older adults; SS = participants with subacute stroke; TMT = Trail Making Test. For bill paying, N = 99; for telephone use, N = 32. Only significant correlation coefficients are presented.
p < .05. **p < .01. ***p < .001.
To further establish the criterion validity of the bill-paying task, we correlated the computer performance data with the TMT–A and –B (see Table 3). In each group, significant moderate correlations were found between the TMT and most of the computer performance measures, especially time to input credit card details and time to input correct numbers on the electricity page (see Table 3). The scores of the Internet-based tasks for each group are also presented in Table 4.
Pearson Correlation Coefficients Between Bill-Paying Computer Performance Data and TMT–A and TMT–B
Note. Only significant correlation coefficients are presented. ADHD = participants with attention deficit hyperactivity disorder; All = all participants and subgroups; CC = credit card; CS = participants with chronic stroke; O = older adults; SS = participants with subacute stroke; TMT–A = Trails Making Test Part A; TMT–B = Trails Making Test Part B.
p < .05. **p < .01. ***p < .001.
Performance on the Internet-Based Tasks by Group
Note. ADHD = attention deficit hyperactivity disorder; CS = chronic stroke; M = mean; SD = standard deviation; SS = subacute stroke; — = not applicable.
Bill-paying computer performance measure.
Discussion
Because of the development of technology (Lange, 2002), many instrumental activities of daily living are performed online. Therefore, for the EFPT to remain relevant and ecologically valid, we adapted two tasks for Internet use.
Study 1 confirmed the alternate-form reliability of the two Internet-based tasks. Therefore, clinicians can choose to use either the original or the Internet-based tasks to administer to clients, selecting the most relevant version for each client while taking administration time into account. The time to complete Internet-based bill paying was longer than the original bill-paying task because filling in details for the online payment takes longer than writing a check. This time difference was especially apparent for healthy participants, who filled in checks quickly but took more time to input the details online. Possibly, for this reason, significant correlations were not found for completion time of the two bill-paying versions.
Alternatively, Internet-based telephone use was slightly faster than the original version because using Google is quicker than manually searching a phone book. The Internet-based telephone-use scores, similar to the original version, are very low, indicating that healthy participants and those with stroke can perform this task independently. Because the scores were all low and similar, correlations between the original and Internet-based tasks were not significant.
Significant differences were found for the Internet-based bill-paying task between healthy participants and those with stroke. These differences provide initial construct validity of this task for assessing EF deficits because people with subacute and chronic stroke experience varying degrees of EF deficits (Gadidi et al., 2011), which restrict their performance in daily activities (Leśniak, Bak, Czepiel, Seniów, & Członkowska, 2008) and participation in the community. Future studies should include larger groups for assessing EF deficits. In addition, the difference found between these groups for the original bill-paying task supports previous studies that established the construct validity of the EFPT (Baum et al., 2008; Hahn et al., 2014). The construct validity of the Internet-based telephone-use task was not established because differences in scores or time between groups were not detected. This finding does not support the previous studies.
In Study 2, criterion validity of the Internet-based tasks was established by the significant moderate to high correlations found between the TMT and the total score (bill paying), the total time (bill paying and telephone use), and the bill-paying computer performance data for the different groups and all participants. Therefore, the Internet-based bill-paying task can be used to detect EF deficits in people who might experience difficulties in daily function as a result of these deficits. For example, older adults present aging-related EF deficits (Gunning-Dixon & Raz, 2003), and people with acute (Zinn, Bosworth, Hoenig, & Swartzwelder, 2007), subacute, and chronic (Gadidi et al., 2011) stroke experience varying degrees of EF deficits.
Adults with ADHD are also known to have EF deficits (Barkley, 1997; Katz & Hartman-Maeir, 2005) that negatively affect their daily occupations (Barkley & Murphy, 2010); however, because these adults function at a high level, their EF deficits are difficult to detect using objective measures. The computer data of the bill-paying task may provide a sensitive assessment to detect EF deficits in this population. Further studies should assess this task in larger groups of adults with ADHD. The TMT, which is commonly used as a tool to assess EF deficits, was used in this study to establish criterion validity of the Internet-based tasks, but because it is a pen-and-paper neuropsychological assessment, future studies should determine the criterion validity with other performance-based assessments to evaluate EF deficits.
Limitations of both studies are the small groups and the fact that the Internet-based telephone-use task was not administered to all participants. In addition, the cognitive status of the participants with subacute and chronic stroke was assessed using different tools (the MoCA for subacute and the MMSE for chronic), which possibly explains the better performance of the participants with subacute stroke.
Implications for Occupational Therapy Practice
This study has the following implications for occupational therapy practice.
The Internet-based tasks of the EFPT have alternate-form reliability and are relevant for assessing EF deficits in our technology-based world.
Internet-based bill paying is challenging and detects significant differences in EF between people with stroke and healthy adults.
Internet-based telephone did not detect differences between healthy participants and those with stroke; therefore, it should not be used in isolation to assess EF in clinical populations.
Conclusion
The Internet-based bill-paying and telephone-use tasks have alternate-form validity. The Internet-based bill paying task (but not telephone use) has initial construct validity, and both tasks have criterion validity to assess EF deficits of different populations. Similar to the original version, the Internet-based bill-paying task is more challenging than the telephone-use task. In addition, the bill-paying computer performance data might be valuable in detecting subtle deficits in EF in various clinical populations. Further research is needed.
Footnotes
Acknowledgments
We thank Professor Noomi Katz, Ono College; Professor Naomi Josman and Dr. Rachel Kizony, Haifa University; and Professor Adina Maeir, Hebrew University, for their help with the development of the Internet-based tasks. We thank Professor Carolyn Baum, Washington University, for permission to modify the original Executive Function Performance Test tasks. We also thank the following undergraduate and graduate occupational therapy students for their skillful data collection: Sapir Bibi, Shani Danziger, Mor Mois, Karin Glasberg, and Anat Keren. We thank all the participants for their participation. This work was performed in partial fulfilment of the requirements for a master of science in occupational therapy degree (Keren Lee Ben-Haim and Rachel Malka), Sackler Faculty of Medicine, Tel Aviv University, Tel Aviv, Israel.