Establishing the Validity of the Internet-Based Bill-Paying Task to Assess Executive Function Deficits Among Adults With Traumatic Brain Injury

Abstract

Importance: Executive function (EF) deficits are common after traumatic brain injury (TBI). During rehabilitation, it is important to identify EF deficits and understand their impact on daily function. The internet-based Bill-Paying Task, modified from the Executive Function Performance Test, has not yet been validated for use with people with TBI.

Objective: To examine the known-groups, convergent, and ecological validity of the internet-based Bill-Paying Task for assessing EF deficits after TBI.

Design: Cross-sectional study with two consecutive parts based on the study’s objectives.

Setting: Inpatient rehabilitation and community.

Participants: Part 1 included 42 adults with TBI and 47 healthy adults; Part 2 included 28 of the 42 adults with TBI.

Measures: Assessments included the Internet-based Bill-Paying Task, WebNeuro neurocognitive computerized battery, Semantic Verbal Fluency test, Behavioural Assessment of the Dysexecutive Syndrome (BADS), Dysexecutive Questionnaire (DEX), and cognitive items of the FIM^® and the Functional Assessment Measure (cognitive FIM+FAM).

Results: For Part 1, participants with TBI required significantly more cues and longer completion time to perform the internet-based Bill-Paying Task. For Part 2, moderate significant correlations were found between the internet-based Bill-Paying Task total score and the WebNeuro, Semantic Verbal Fluency test, BADS, DEX, and cognitive FIM+FAM.

Conclusions and Relevance: This study supports the known-groups, convergent, and ecological validity of the internet-based Bill-Paying Task for assessing EF deficits among adults with preserved basic cognitive abilities after TBI. Therefore, it can be used to assist with rehabilitation treatment planning after TBI.

What This Article Adds: The internet-based Bill-Paying Task, an online payment task relevant to today's technological world, is valid to assess higher cognitive abilities of people after a traumatic brain injury. This assessment may contribute to a better understanding of patients' cognitive profiles and their potential impact on daily performance.

Precis (for TOC): This study supports the validity and use of the internet-based Bill-Paying Task to identify executive function (EF) deficits and understand the impact on daily function for people with traumatic brain injury (TBI). The Bill-Paying Task can be used to assist with rehabilitation treatment planning for people with TBI.

The incidence and prevalence rates of traumatic brain injury (TBI) are high worldwide (Brazinova et al., 2016; Ma et al., 2014; Peeters et al., 2015; Siman-Tov et al., 2016), and TBI is a major cause of disability (Faul et al., 2010). One of the main consequences of TBI is deficits in executive function (EF; Wood & Worthington, 2017), defined as integrative cognitive processes that determine goal-directed and purposeful behavior (Cicerone et al., 2000; Stuss, 2011). EFs include high-level cognitive abilities such as attention, planning, problem solving, multitasking, and behavioral control, which are essential for performing complex and novel tasks (Burgess, 2000; Mateer, 1999; Shallice & Burgess, 1991a). EF deficits may therefore lead to difficulties in performing instrumental activities of daily living—such as household tasks, shopping, and driving (Lezak et al., 2012)—as well as poor participation (Faul et al., 2010; Lezak et al., 2012), which leads to decreased quality of life (Bottari et al., 2009 ; Siman-Tov et al., 2016).

Accurate assessment of EF is essential to plan a comprehensive rehabilitation program (Lewis et al., 2011). Because EFs are higher cognitive abilities, the assessment of EF is recommended after the screening and assessment of cognition to ensure that basic cognitive abilities are preserved (Bar-Haim Erez & Katz, 2018 ; Hartman-Maeir et al., 2009). Performance-based assessments, which use real-world objects or tasks, enable occupational therapy practitioners to link the observed performance to the underlying impairments. This practice helps practitioners understand the impact of EF deficits on daily function (Baum & Katz, 2010; Hahn et al., 2014) and thus to assess functional cognition (Toglia et al., 2019). Many performance-based assessments have been developed to assess EF, such as the Multiple Errands Test (Shallice & Burgess, 1991b) and the execution of a cooking task (Chevignard et al., 2000). However, none of these assessments entail the use of the internet, which is now an integral part of people’s lives.

The Executive Function Performance Test (EFPT; Baum et al., 2008) is a well-known, valid, and reliable performance-based tool that is used to assess EF (specifically, initiation, organization, sequencing safety and judgment, and task completion) while performing four everyday tasks (simple cooking, medication management, telephone use, and bill payment; Baum et al., 2008; Goverover et al., 2005; Katz et al., 2007). Recently, the Bill-Paying Task has been modified for internet use, in which the cheques were replaced by a mock credit card, and balancing the account was replaced by the line of credit. This modification makes the Bill-Paying Task more relevant for today’s technological world, especially for young adults. This internet-based task also provides computerized data regarding the person’s performance, making the scoring more sensitive. Validity of the internet-based Bill-Paying Task to assess EFs has been established for adults with stroke (Rand et al., 2018) but not for adults with TBI.

Therefore, we aimed to establish the validity of the internet-based Bill-Paying Task for assessing EF deficits among people after TBI. This study consisted of two parts: (1) establishing known-groups validity by comparing the performance of adults with TBI to healthy adults and (2) establishing convergent and ecological validity by correlating the internet-based Bill-Paying Task with assessments that measure EF and the participant’s behavioral and functional independence in the inpatient rehabilitation setting, respectively. We hypothesized that (1) significant between-groups differences would be found for performance of the internet-based Bill-Paying Task and (2) worse performance on the internet-based Bill-Paying Task would be correlated with worse performance on other EF assessments.

Method

A cross-sectional study was conducted with two consecutive parts.

Participants

Part 1 included 42 adults with TBI and 47 healthy adults who were administered the internet-based Bill-Paying Task. Part 2 included 28 (of the 42) adults with TBI who were administered the additional EF assessments.

We recruited participants with TBI (ages 18 yr and older) from inpatient rehabilitation wards using convenience sampling. Inclusion criteria were diagnosis of TBI, ability to understand verbal instructions as determined by their occupational therapy practitioner, motor ability to use the computer, previous computer experience (self-report), and intact or corrected vision. The Loewenstein Occupational Therapy Cognitive Assessment (Katz et al., 1989) was used to verify that participants’ basic cognitive abilities were preserved (score of 4/4 points on the Orientation, Visual Perception, and Spatial Perception subtests). Exclusion criteria were other neurological conditions, epilepsy, or psychiatric conditions (obtained from participants’ medical file) to control for confounding variables.

Healthy participants matched by age and sex were recruited through advertisements posted at the university and snowball sampling. Inclusion criteria were independent living in the community, at least 12 yr of education, previous use of computers, and no neurological or psychiatric conditions. The study was approved by the Loewenstein Rehabilitation Center’s Helsinki committee and by Tel Aviv University’s ethics committee. All participants provided written informed consent before their participation in the study.

Tools

Internet-Based Bill-Paying Task

The internet-based Bill-Paying Task was coded in LabVIEW (Version 12) and can be freely downloaded (https://www.tau.ac.il/∼portnoys/Internet-based_ Bill_Paying_Task.html). As in the original task, the occupational therapy practitioner actively observes the client’s performance, and scoring is based on the level of cueing that is provided to the client to complete the task: paying two bills (electricity, gas) online and using a mock credit card. Specifically, clients are scored on (1) initiation (5 points), (2) execution (15 points for organization, sequencing, safety, and judgment), and (3) completion (5 points). The total score ranges from 0 to 25 points; higher scores indicate more severe EF deficits (Baum et al., 2008, 2017). Computerized data include completion time (in seconds), time spent on each web page, number of mouse clicks, and mouse travel time (Rand et al., 2018).

In this study, the total, initiation, execution, and completion scores of the EFPT as well as the computerized data (specifically, completion time, time spent on electricity payment page, time spent on gas payment page, and total mouse travel) were all used as measures of EF. Criterion validity was established by the significant moderate to high correlations found between the total scores, total time, and computer performance data for the internet-based Bill-Paying Task and the Trail Making Test (TMT; Reitan, 1958) among healthy young and older adults, people with subacute stroke, people with chronic stroke, and young adults with attention deficit hyperactivity disorder (Rand et al., 2018).

WebNeuro Computerized Assessment Battery

The WebNeuro assessment battery (Silverstein et al., 2007) includes computerized adaptations of neuropsychological tests to identify deficits in EF. Five subtests from this battery, considered the gold standard for assessing EF components, were used: (1) digit span, which assesses the number of digits memorized (working memory; Wechsler, 1997); (2) TMT, which assesses the time taken to complete the task (cognitive flexibility; Reitan, 1958); (3) reaction time, which assesses the mean reaction time needed to choose between two stimuli (processing speed); (4) Stroop, which assesses the capacity to stop automatic reactions and thoughts as needed (inhibition; Stroop, 1935); and (5) maze, which assesses the time it takes to identify and learn a hidden path through trial and error (planning and organize behavior to meet a goal; Walsh, 1985). Scores for each subtest include raw scores and standardized scores adjusted for age and sex; however, scores are not recorded if the subtests are not completed within the time constraint. The WebNeuro battery was found to be moderately to highly correlated (r = .75–.86, p < .001) with IntegNeuro (Brain Resource Ltd., 2007), a well-established touchscreen-based computerized cognitive assessment battery that highly supports the criterion validity test–retest reliability of the different subtests (.73–.89; Silverstein et al., 2007). The WebNeuro battery has been validated for several adult populations, including people with mild TBI (Brain Resource Ltd., 2010).

Semantic Verbal Fluency Test

The Semantic Verbal Fluency test (Kavé, 2005) was assessed by obtaining the number of words generated in 1 min for each of the following three semantic categories: animals, fruits and vegetables, and vehicles. This test is a reliable and valid tool to assess EF deficits among adults after TBI (Henry & Crawford, 2004; Kavé et al., 2011), specifically, selective attention, mental set shifting, and self-monitoring.

Behavioural Assessment of the Dysexecutive Syndrome

The Behavioural Assessment of the Dysexecutive Syndrome (BADS; Wilson et al., 1996) is designed to assess effects of dysexecutive syndrome and includes the following subtests simulating everyday activities: Rule Shift Cards, Action Program, Key Search, Temporal Judgment, Zoo Map, and Modified Six Element. The BADS requires participants to plan, initiate, monitor, and adjust behavior in response to the explicit and implicit demands of a series of tasks. An overall profile score ranging from 0 to 24 is produced as a sum of individual subtest scores. It has high interrater reliability (.88–1.00), and construct validity was established for adults with TBI (Wilson et al., 2003).

Dysexecutive Questionnaire

The Dysexecutive Questionnaire (DEX) is a reliable and valid questionnaire (Wilson et al., 1996) for identifying EF deficits in everyday life (Bennett et al., 2005). This 20-item questionnaire inquires about everyday problems such as emotional, personality, motivational, behavioral, or cognitive changes. According to Emmanouel et al. (2014), the DEX was found to be an ecologically valid, highly accurate, and sensitive measure for patients with TBI because it was strongly correlated with several clinical measures of executive dysfunction, such as the BADS, the Everyday Description Task (Dritschel et al., 1998), and the Twenty Question Test (Laine & Butters, 1982).

Cognitive Items of the FIM™ and the Functional Assessment Measure

The participants’ cognitive abilities observed during daily functioning in the ward were rated by the occupational therapy practitioner using the five cognitive items (comprehension, expression, social interaction, problem solving, and memory) from the FIM^® (Keith et al., 1987) and the nine cognitive and psychosocial function items (orientation, concentration, safety awareness, emotional status, adjustment to limitations, leisure activities, reading, writing, and speech intelligibility) from the Functional Assessment Measure (FAM; Hall et al., 1993). These cognitive items of the FIM and FAM form the cognitive FIM+FAM. The total score ranges from 14 to 98 points; higher points indicate better cognition. In addition, participants’ total FIM score (18–126 points), which is reliable and valid to assess independence in 18 basic daily activities (Corrigan et al., 1997; Ottenbacher et al., 1996), was taken from their medical file.

Procedure

Participants with TBI in the inpatient wards at the rehabilitation center were approached by one of the occupational therapy practitioners and were provided with information regarding the study. If eligible and they agreed, they signed an informed consent form and were screened for basic cognition. If found eligible, they completed the internet-based Bill-Paying Task (N = 42; Part 1). Healthy participants also completed the internet-based Bill-Paying Task. All participants used a laptop with a keyboard and a mouse. Twenty-eight of the 42 participants with TBI from Part 1 also completed the WebNeuro computerized battery, BADS, and Semantic Verbal Fluency test, assessments known to assess EF deficits (Part 2). This testing took approximately 3 hr, divided into two sessions. In addition, the participants’ occupational therapy practitioner, on the basis of ongoing observations of the participants’ daily behavior and functioning, filled in the DEX and the cognitive FIM+FAM items. All assessments were administered in a quiet room by occupational therapy practitioners who were qualified assessors. Assessors were trained before the study, and we verified that the tools were scored in the same manner.

Data Analysis

We analyzed numeric data using IBM SPSS Statistics (Version 23). Because the data were not normally distributed, descriptive statistics were presented as median and interquartile range (IQR) values, and nonparametric statistics were used. Descriptive statistics were used to describe the demographic and injury-related information of the TBI group, the internet-based Bill-Paying Task scores, and the other measures.

For Part 1, known-groups validity, a type of construct validity, was established if an assessment could discriminate between groups with and without a known trait (Portney & Gross, 2020). Mann–Whitney U test were conducted to assess the differences between TBI and healthy groups. To understand the extent of the between-groups differences, we calculated the effect size using Cohen’s r = z/√N for nonparametric tests. The z value was retrieved from the Mann–Whitney U test output. Cohen’s r effect size values were considered small (0.1), medium (0.3), and large (0.5; Coolican, 2014).

For Part 2, convergent validity was established when measures believed to capture the same underlying phenomenon yielded similar findings (Portney & Gross, 2020). When the findings of an assessment can explain daily functioning or behavior in the participant’s own environment, it is considered to have ecological validity (Portney & Gross, 2020). Spearman correlations were used to assess associations between the internet-based Bill-Paying Task and the WebNeuro subtests, Semantic Verbal Fluency test, BADS (convergent validity), cognitive FIM+FAM, and DEX (ecological validity) for the participants with TBI. The statistical level was set at p < .05.

Results

For Part 1, the demographic and clinical data of the 42 adults with TBI (ages 19–70 yr) are presented in Table 1. Most participants sustained a TBI after a traffic accident; at the time of the study, 15 of the participants used a wheelchair for mobility. As per inclusion criteria, their basic cognitive abilities were preserved. The 47 healthy adults (15 men, 32 women), ages 21 to 68 yr, were younger (Mdn age = 27.0 yr, IQR = 25.0–60.0) than the participants with TBI (Mdn age 38.5 yr, IQR = 23.0–55.5; z = −0.1, p < .05); in addition, they had more years of education than the TBI group (Mdn = 14 yr compared with 12 yr; z = −4.2, p < .01).

Table 1.

Demographic and Clinical Characteristics of the 42 Participants With TBI and Demographic Data of 47 Healthy Participants

Characteristic	Mdn (IQR), Min–Max, or n (%)
Characteristic	Participants With TBI (n = 42)	Subgroup of Participants With TBI (n = 28)	Healthy Participants (n = 47)
Age, yr	38.5 (23.0–55.5), 19–70	29.0 (20.2–50.7), 19–65	27.0 (25.0–60.0), 21–68
Education, yr	12 (12–12), 8–20	12 (12–12), 10–20	14 (13–15), 12–17
FIM^®a	105 (92.7–110.7), 40–124	100 (85.7–110), 40–123
Days since injury	72.5 (44.7–116.5), 12–435	60 (36.7–112.7), 12–435
Sex
Male	34 (81.0)	22 (78.6)	15 (31.9)
Female	8 (19.0)	6 (21.4)	32 (68.1)
Residence
With partner	33 (78.6)	24 (85.7)
Alone	9 (21.4)	4 (14.3)
Cause of injury
Traffic accident	30 (71.4)	22 (78.6)
Fall	9 (21.4)	3 (10.7)
Gunshot	2 (4.8)	2 (7.1)
Sports injury	1 (2.4)	1 (3.6)
Mobility
Wheelchair	15 (35.7)	15 (53.6)
Walking	18 (42.9)	8 (28.6)
Walking with aid	9 (21.4)	5 (17.9)

Note. IQR = interquartile range; max = maximum; min = minimum; TBI = traumatic brain injury.

Total score can range from 18 to 126.

All 42 participants with TBI were able to complete the internet-based Bill-Paying Task; however, statistically significant between-groups differences were found (p < .01; Table 2). Participants with TBI needed a median of 7.0 cues (IQR = 3.7–10), whereas the healthy participants needed a median of 1.0 cue (IQR = 0–2). Significant group differences were also found for initiation, execution, and completion subscores; completion time; and the other computerized measures.

Table 2.

Median, IQR, and Differences Between the TBI and Healthy Groups for the Internet-Based Bill-Paying Task Scores and the Computerized Data (Study 1)

Assessment	Mdn (IQR), Min–Max		Differences Between TBI and Healthy Groups, Mann–Whitney z ^a	Effect Size, Cohen’s r
Assessment	TBI Group (n = 42)	Healthy Group (n = 47)		Effect Size, Cohen’s r
Total score (0–25)	7.0 (3.7–10.0), 0–20	1.0 (0.0–2.0), 0–12	−5.6	.63
Initiation score (0–5)	0.0 (0.0–0.8), 0–3	0.0 (0.0–0.0), 0–3	−1.6	.16
Execution score (0–15)	8.0 (4.2–9.7), 0–15	1.0 (0.0–2.0), 0–11	−5.6	.59
Completion score (0–5)	0.0 (0.0–2.0), 0–4	0.0 (0.0–0.0), 0–0	−5.0	.53
Completion time (min)^b	23.3 (17.7–29.9), 10.9–64.0	10.0 (8.4–15.6), 3.6–50.3	−5.8	.60
Time electricity page (min)^b	3.0 (1.9–4.9), 1.0–10.7	1.2 (0.7–1.9), 0–6.3	−4.9	.54
Time gas page (min)^b	1.8 (1.4–2.7), 0.7–13.4	1.2 (0.8–1.9), 0.3–7.9	−2.7	.30
Total mouse travel (pixels)^b	20,449 (15,667–34,082), 8,324–76,289	15,326 (10,132–15,326), 6,088–22,472	−2.7	.30

Note. In relation to the total score (0–25), a higher score indicates more executive function deficits. IQR = interquartile range; Max = maximum; Min = minimum; TBI = traumatic brain injury; time electricity page = total time spent paying electricity bill; time gas page = total time spent paying gas bill; total mouse travel = total travel path of the mouse cursor on the screen.

p < .01.

n = 36; missing data for 6 participants.

The effect sizes were large for the total score as well as the execution and completion subscores of the internet-based Bill-Paying Task; the effect sizes were also large for completion time and time on the electricity page. Medium effect sizes were found for time on the gas page and mouse travel time, and a small effect size was found for initiation, indicating that although statistically significant, these items had a small magnitude.

Part 2 was conducted subsequent to Part 1: Of the 42 participants with TBI in Part 1, 28 were administered the additional assessments. The characteristics of the 28 participants appear in Table 1. Overall, this subgroup was similar to the TBI group, except for age. The median age of this subgroup was significantly lower than the TBI group (z = −1.9, p < .05).

Table 3 presents scores on the internet-based Bill-Paying Task as well as the assessments to evaluate EF, daily behavior, and function. Assessment scores of the participants vary considerably, indicating mild to severe deficits. The median scores demonstrate EF deficits and difficulties in daily living (see Table 3).

Table 3.

Median, IQR, and Range on Executive Function Assessments, Daily Behavior, and Function of the 28 Participants With TBI (Study 2)

Assessment	Mdn (IQR)	Min–Max
Internet-based Bill-Paying Task (0–25)^a	9 (5–12)	0–20
Tools to establish convergent validity
BADS (0–24)^b	17 (12.2–19.7)	7–24
Semantic Verbal Fluency test^b,c	33.5 (27.5–51.5)	23–75
WebNeuro neurocognitive subtests
Digit span (no. of correct trials)	5 (3.2–7)	0–10
Trail Making Test (seconds)^d	91.6 (52.9–205.4)	61.6–123.2
Reaction time (seconds)	0.4 (0.4–0.6)	0.2–2.5
Stroop (no. of correct trials)^e	7.5 (4–11.2)	1–19
Maze (seconds)^f	404.9 (289.7–529.1)	115.8–1,000
Tools to establish ecological validity
DEX (0–80)^a	23.5 (8.2–35)	0–51
Cognitive FIM+FAM (14–98)^b	77.5 (64–87)	33–98

Higher score reflects greater executive function deficits.

Higher score reflects fewer executive function deficits.

Native Hebrew speakers, n = 22.

n = 27; score was not generated because task was not completed within the time limit.

n = 26; score was not generated because task was not completed within the time limit.

n = 25; score was not generated because task was not completed within the time limit.

Moderately significant correlations were found between scores on the internet-based Bill-Paying Task and the Semantic Verbal Fluency test, subtests of the WebNeuro, and the BADS (Table 4). Moderately to strongly significant correlations were found between completion time of the internet-based Bill-Paying Task and various WebNeuro subtest scores. In addition, moderately to strongly significant correlations were found between the computerized data from the internet-based Bill-Paying Task and several WebNeuro subtests (see Table 4). For example, longer completion time and higher scores on the internet-based Bill-Paying Task correlated with slower performance time on subtests examining mental flexibility, planning, and organization.

Table 4.

Significant Correlations Between Total, Subscores, and Computerized Data for the Internet-Based Bill-Paying Task and the Other Assessments to Establish Convergent and Ecological Validity (Study 2; n = 28)

Assessment	Total Score	Initiation	Execution	Computerized Data
				Total Time	Time Electricity	Time Gas	Time Visa Gas	Mouse Travel
				Tools to Establish Convergent Validity
BADS	−.46*		−.46*
Semantic Verbal Fluency test^a	−.50*		−.56**
WebNeuro neurocognitive subtests
Working memory (digit span)	−.46*		−.52**	−.53**		.42*	−.44*	−.49**
Flexibility (TMT)^b	.39*		.43*	.50**		.125	.364	.43*
Processing speed (reaction time)	.44*	.43*	.52**	.62**	.44*	.130	.218	.49**
Response inhibition (Stroop)^c	−.45*		−.55**	−.62**	−.53**	.44*	−.344	−.57**
Planning (maze)^d	.46*		.45**	.40*		.42*	.82**
Tools to Establish Ecological Validity
DEX	.65**	.48**	.67**
Cognitive FIM+FAM	−.66**		−.68**

Note. BADS = Behavioural Assessment of the Dysexecutive Syndrome; cognitive FIM+FAM = cognitive items of the FIM^® and Functional Assessment Measure; DEX = Dysexecutive Questionnaire; mouse travel = total travel path of the mouse cursor on the screen; time electricity = total time spent paying electricity bill; time gas = total time spent paying gas bill; time Visa gas = total time spent on the Visa page paying gas bill; TMT = Trail Making Test.

Native Hebrew speakers, n = 22.

n = 27; score was not generated because task was not completed within the time limit.

n = 26; score was not generated because task was not completed within the time limit.

n = 25; score was not generated because task was not completed within the time limit.

p < .05.

p < .01.

Moderate to strong significant correlations were found between the internet-based Bill-Paying Task and the DEX (r = .65, p < .01) and the cognitive FIM+FAM (r = −.66, p < .01), indicating that the participants’ EF deficits (detected during the internet-based Bill-Paying Task) might explain some of their functional and behavioral difficulties in daily living.

Discussion

This study supports the validity of the internet-based Bill-Paying Task, which simulates online payment of bills, as a performance-based assessment to evaluate EF deficits among people with TBI. Significant differences were found between the TBI group and a comparison group of healthy participants (Part 1). Significant moderate correlations were found between the internet-based Bill-Paying Task and EF assessments as well as the participants’ behavior and level of independent functioning (Part 2), which supports the convergent and ecological validity of this task.

As hypothesized, participants with TBI performed significantly worse than the healthy participants, which supports the known-groups validity of the internet-based Bill-Paying Task for assessing EF deficits. Previously, validity was established among adults with subacute and chronic stroke (Rand et al., 2018) who performed worse than healthy participants. We calculated the effect size to provide a quantitative measure of the magnitude of differences between groups (Fritz et al., 2012). Moderate to large effect sizes were found for the total score, execution and completion subscores, and the computerized data. A small effect size was found for initiation, indicating that this subscore cannot differentiate between groups. This finding is similar to the nonsignificant differences found between healthy people and those with mild to moderate stroke using the original EFPT (Baum et al., 2008).

As hypothesized, for the TBI subgroup, worse or higher performance on the internet-based Bill-Paying Task (total, subscores, and computerized data) was moderately correlated with worse or higher performance of well-known EF assessments, such as the Semantic Verbal Fluency test, several WebNeuro computerized battery subtests (i.e., digit span, TMT, reaction time, Stroop, maze), and the BADS. These findings concur with previous studies that used the internet-based Bill-Paying Task (Rand et al., 2018) and the original EFPT (Baum et al., 2008; Wolf et al., 2010). The moderate strength of these correlations emphasizes the fact that neuropsychological or simulated functional assessments can partly explain everyday performance of the Bill-Paying Task but are not able to capture the person’s full ability to perform complex everyday activities (Burgess, 2000; Wolf et al., 2010).

Ecological validity of the Kettle Test (Hartman-Maeir et al., 2005), another performance-based cognitive assessment, was established by correlations with the FIM (Hartman-Maeir et al., 2009). Similarly, in our study, the participants’ behavioral and functional independence at the inpatient rehabilitation center was observed and captured by the DEX and cognitive FIM+FAM. Correlations found between these measures and the internet-based Bill-Paying Task were stronger than correlations between these measures and the neuropsychological assessments, which establishes the ecological validity of the internet-based Bill-Paying Task. In other words, the difficulties seen during online bill payment (because of EF deficits) can generalize to difficulties in performance in the real world.

Limitations

Our study had several limitations, including the inclusion criterion for the healthy group of at least 12 yr of education, which resulted in a between-groups difference in years of education and might have led to better performance on the internet-based Bill-Paying Task. In addition, the sample of 28 participants with TBI who underwent all of the EF assessments is small. Because we included only patients with preserved basic cognitive abilities, these findings cannot be generalized to patients with TBI with impaired basic cognitive abilities. The use of the internet-based Bill-Paying Task is recommended only for adults with TBI with preserved basic cognition. We did not collect information regarding the emotional status of the participants with TBI, which may also lead to EF deficits. Although the neuropsychological tests used to assess EF are considered the gold standard, they were administered with computerized software and not with paper and pencil. The missing scores for the WebNeuro subtests (for 3 participants) and the internet-based Bill-Paying Task’s computerized data (for 6 participants) are another limitation.

Implications for Occupational Therapy Practice

The results of this study have the following implications for occupational therapy practice:

The internet-based Bill-Paying Task has known-groups validity and convergent validity for assessing EF deficits among adults with preserved basic cognitive abilities after TBI.

Because difficulties in performance of the internet-based Bill-Paying Task are associated with difficulties in daily functioning (ecological validity), this connection might help define specific goals for cognitive rehabilitation.

Conclusion

This study provides support for known-groups, convergent, and ecological validity of the internet-based Bill-Paying Task for adults with preserved basic cognitive abilities after TBI. Use of this online payment task may contribute to a better understanding of participants’ cognitive profiles and their potential impact on daily performance. Further research should assess the predictive validity of this tool.

Footnotes

Acknowledgments

This work was performed in partial fulfillment of the requirements for the Master of Science in occupational therapy degree for Yael Nadler Tzadok and Rotem Eliav at the Sackler Faculty of Medicine, Tel Aviv University, Tel Aviv, Israel. We thank the Loewenstein Rehabilitation Hospital Research Foundation for their support and the research assistants for help with data collection. We also thank all the participants for their contributions.

References

Bar-Haim Erez

, & Katz

(2018). Cognitive functional evaluation. In Katz

& Toglia

(Eds.), Cognition, occupation, and participation across the life span: Neuroscience, neurorehabilitation, and models of intervention in occupational therapy (4th ed., pp. 69–86). AOTA Press.

Baum

C. M.

Connor

L. T.

Morrison

Hahn

Dromerick

A. W.

, & Edwards

D. F.

(2008). Reliability, validity, and clinical utility of the Executive Function Performance Test: A measure of executive function in a sample of people with stroke. American Journal of Occupational Therapy , 62 , 446–455. https://doi.org/10.5014/ajot.62.4.446

Baum

C. M.

, & Katz

(2010). Occupational therapy approach to assessing the relationship between cognition and function. In Marcotte

T. D.

& Grant

(Eds.), Neuropsychology of everyday functioning (pp. 63–90). Guilford Press.

Baum

C. M.

Wolf

T. J.

Wong

A. W. K.

Chen

C. H.

Walker

Young

A. C.

, . . . Heinemann

A. W.

(2017). Validation and clinical utility of the Executive Function Performance Test in persons with traumatic brain injury. Neuropsychological Rehabilitation , 27 , 603–617. https://doi.org/10.1080/09602011.2016. 1176934

Bennett

P. C.

Ong

, & Ponsford

(2005). Measuring executive dysfunction in an acute rehabilitation setting: Using the Dysexecutive Questionnaire (DEX). Journal of the International Neuropsychological Society , 11 , 376–385. https://doi.org/10.1017/S1355617705050423

Bottari

Dassa

Rainville

, & Dutil

(2009). The factorial validity and internal consistency of the Instrumental Activities of Daily Living Profile in individuals with a traumatic brain injury. Neuropsychological Rehabilitation , 19 , 177–207. https://doi.org/10.1080/09602010802188435

Brain Resource Ltd. (2007). IntegNeuro™ user manual (Version 3).

Brain Resource Ltd. (2010). BRISC and WebNeuro assessment manual (Version 1.6).

Brazinova

Rehorcikova

Taylor

M. S.

Buckova

Majdan

Psota

, . . . Synnot

(2016). Epidemiology of traumatic brain injury in Europe: A living systematic review. Journal of Neurotrauma , 33 , 1–30.

10.

Burgess

P. W.

(2000). Strategy application disorder: The role of the frontal lobes in human multitasking. Psychological Research , 63 , 279–288. https://doi.org/10.1007/s004269900006

11.

Chevignard

Pillon

Pradat-Diehl

Taillefer

Rousseau

Le Bras

, & Dubois

(2000). An ecological approach to planning dysfunction: Script execution. Cortex , 36 , 649–669. https://doi.org/10.1016/S0010-9452(08)70543-4

12.

Cicerone

K. D.

Dahlberg

Kalmar

Langenbahn

D. M.

Malec

J. F.

Bergquist

T. F.

, . . . Morse

P. A.

(2000). Evidence-based cognitive rehabilitation: Recommendations for clinical practice. Archives of Physical Medicine and Rehabilitation , 81 , 1596–1615. https://doi.org/10.1053/apmr.2000.19240

13.

Coolican

(2014). Research methods and statistics in psychology. Psychology Press.

14.

Corrigan

J. D.

Smith-Knapp

, & Granger

C. V.

(1997). Validity of the Functional Independence Measure for persons with traumatic brain injury. Archives of Physical Medicine and Rehabilitation , 78 , 828–834. https://doi.org/10.1016/s0003-9993(97)90195-7

15.

Dritschel

Kogan

Burton

, & Goddard

(1998). Everyday planning difficulties following traumatic brain injury: A role for autobiographical memory. Brain Injury , 12 , 875–886. https://doi.org/10.1080/026990598122098

16.

Emmanouel

Mouza

Kessels

R. P. C.

, & Fasotti

(2014). Validity of the Dysexecutive Questionnaire (DEX). Ratings by patients with brain injury and their therapists. Brain Injury , 28 , 1581–1589. https://doi.org/10.3109/02699052.2014.942371

17.

Faul

Wald

M. M.

, & Coronado

V. G.

(2010). Traumatic brain injury in the United States: Emergency department visits, hospitalizations and deaths 2002–2006. U.S. Department of Health and Human Services. https://doi.org/10.15620/cdc.5571

18.

Fritz

C. O.

Morris

P. E.

, & Richler

J. J.

(2012). Effect size estimates: Current use, calculations, and interpretation. Journal of Experimental Psychology: General , 141 , 2–18. https://doi.org/10.1037/a0024338

19.

Goverover

Kalmar

Gaudino-Goering

Shawaryn

Moore

N. B.

Halper

, & DeLuca

(2005). The relation between subjective and objective measures of everyday life activities in persons with multiple sclerosis. Archives of Physical Medicine and Rehabilitation , 86 , 2303–2308. https://doi.org/10.1016/j.apmr.2005.05.016

20.

Hahn

Baum

Moore

Ehrlich-Jones

Spoeri

Doherty

, & Wolf

T. J.

(2014). Development of additional tasks for the Executive Function Performance Test. American Journal of Occupational Therapy , 68 , e241–e246. https://doi.org/10.5014/ajot.2014.008565

21.

Hall

K. M.

Hamilton

B. B.

Gordon

W. A.

, & Zasler

N. D.

(1993). Characteristics and comparisons of functional assessment indices: Disability Rating Scale, Functional Independence Measure, and Functional Assessment Measure. Journal of Head Trauma Rehabilitation , 8 , 60–74. https://doi.org/10.1097/00001199- 199308020-00008

22.

Hartman-Maeir

Armon

, & Katz

(2005). Kettle Test protocol. School of Occupational Therapy, Hadassah and Hebrew University of Jerusalem.

23.

Hartman-Maeir

Katz

, & Baum

C. M.

(2009). Cognitive Functional Evaluation (CFE) process for individuals with suspected cognitive disabilities. Occupational Therapy in Health Care , 23 , 1–23. https://doi.org/10.1080/07380570802455516

24.

Henry

J. D.

, & Crawford

J. R.

(2004). A meta-analytic review of verbal fluency performance in patients with traumatic brain injury. Neuropsychology , 18 , 621–628. https://doi.org/10.1037/0894-4105. 18.4.621

25.

Katz

Itzkovich

Averbuch

, & Elazar

(1989). Loewenstein Occupational Therapy Cognitive Assessment (LOTCA) battery for brain-injured patients: Reliability and validity. American Journal of Occupational Therapy , 43 , 184–192. https://doi.org/10.5014/ajot.43.3.184

26.

Katz

Tadmor

Gelzen

, & Hartman-Maeir

(2007). Validity of the Executive Function Performance Test in individuals with schizophrenia. OTJR: Occupation, Participation and Health , 27 , 44–51. https://doi.org/10.1177/153944920702700202

27.

Kavé

(2005). Phonemic fluency, semantic fluency, and difference scores: Normative data for adult Hebrew speakers. Journal of Clinical and Experimental Neuropsychology , 27 , 690–699. https://doi.org/10.1080/13803390490918499

28.

Kavé

Heled

Vakil

, & Agranov

(2011). Which verbal fluency measure is most useful in demonstrating executive deficits after traumatic brain injury? Journal of Clinical and Experimental Neuropsychology , 33 , 358–365. https://doi.org/10.1080/13803395. 2010.518703

29.

Keith

R. A.

Granger

C. V.

Hamilton

B. B.

, & Sherwin

F. S.

(1987). The Functional Independence Measure: A new tool for rehabilitation. Advances in Clinical Rehabilitation , 1 , 6–18.

30.

Laine

, & Butters

(1982). A preliminary study of the problem-solving strategies of detoxified long-term alcoholics. Drug and Alcohol Dependence , 10 , 235–242.‏ https://doi.org/10.1016/0376-8716(82)90017-5

31.

Lewis

M. W.

Babbage

D. R.

, & Leathem

J. M.

(2011). Assessing executive performance during cognitive rehabilitation. Neuropsychological Rehabilitation , 21 , 145–163. https://doi.org/10.1080/09602011.2010.543867

32.

Lezak

M. D.

Howieson

D. B.

Bigler

E. D.

, & Tranel

(2012). Neuropsychological assessment (5th ed.). Oxford University Press.

33.

V. Y.

Chan

, & Carruthers

K. J.

(2014). Incidence, prevalence, costs, and impact on disability of common conditions requiring rehabilitation in the United States: Stroke, spinal cord injury, traumatic brain injury, multiple sclerosis, osteoarthritis, rheumatoid arthritis, limb loss, and back pain. Archives of Physical Medicine and Rehabilitation , 95 , 986–995.e1. https://doi.org/10.1016/j.apmr.2013.10.032

34.

Mateer

(1999). The rehabilitation of executive disorders. In Stuss

D. T.

Winocur

, & Robertson

I. H.

(Eds.), Cognitive neurorehabilitation (pp. 314–332). Cambridge University Press.

35.

Ottenbacher

K. J.

Hsu

Granger

C. V.

, & Fiedler

R. C.

(1996). The reliability of the Functional Independence Measure: A quantitative review. Archives of Physical Medicine and Rehabilitation , 77 , 1226–1232. https://doi.org/10.1016/S0003-9993(96)90184-7

36.

Peeters

van den Brande

Polinder

Brazinova

Steyerberg

E. W.

Lingsma

H. F.

, & Maas

A. I.

(2015). Epidemiology of traumatic brain injury in Europe. Acta Neurochirurgica , 157 , 1683–1696. https://doi.org/10.1007/s00701-015-2512-7

37.

Portney

L. G.

, & Gross

K. D.

(2020). Concept measurement validity. In Portney

L. G.

(Ed.), Foundations of clinical research: Applications to evidence-based practice (4th ed., pp. 127–140). F. A. Davis.

38.

Rand

Lee Ben-Haim

Malka

, & Portnoy

(2018). Development of internet-based tasks for the Executive Function Performance Test. American Journal of Occupational Therapy , 72 , 7202205060. https://doi.org/10.5014/ajot.2018.023598

39.

Reitan

R. M.

(1958). Validity of the Trail Making Test as an indicator of organic brain damage. Perceptual and Motor Skills , 8 , 271–276. https://doi.org/10.2466/pms.1958.8.3.271

40.

Shallice

, & Burgess

P. W.

(1991a). Deficits in strategy application following frontal lobe damage in man. Brain , 114 , 727–741. https://doi.org/10.1093/brain/114.2.727

41.

Shallice

, & Burgess

P. W.

(1991b). Higher-order cognitive impairments and frontal lobe lesions in man. In Levin

H. S.

Eisenberg

H. M.

, & Benton

A. L.

(Eds.), Frontal lobe function and dysfunction (pp. 125–138). Oxford University Press.

42.

Silverstein

S. M.

Berten

Olson

Paul

Williams

L. M.

Cooper

, & Gordon

(2007). Development and validation of a World-Wide-Web-based neurocognitive assessment battery: WebNeuro. Behavior Research Methods , 39 , 940–949. https://doi.org/10.3758/BF03192989

43.

Siman-Tov

Radomislensky

Knoller

Bahouth

Kessel

Klein

, . . . Peleg

(2016). Incidence and injury characteristics of traumatic brain injury: Comparison between children, adults and seniors in Israel. Brain Injury , 30 , 83–89. https://doi.org/10.3109/02699052.2015.1104551

44.

Stroop

J. R.

(1935). Studies of interference in serial verbal reactions. Journal of Experimental Psychology , 18 , 643–662. https://doi.org/10.1037/h0054651

45.

Stuss

D. T.

(2011). Functions of the frontal lobes: Relation to executive functions. Journal of the International Neuropsychological Society , 17 , 759–765. https://doi.org/10.1017/S1355617711000695

46.

Toglia

J. P.

Golisz

K. M.

, & Goverover

(2019). Cognition, perception, and occupational performance. In Schell

B. A. B.

& Gillen

(Eds.), Willard and Spackman’s occupational therapy (13th ed., pp. 901–941). Lippincott Williams & Wilkins.

47.

Walsh

K. W.

(1985). Understanding brain damage: A primer of neuropsychological evaluation. Churchill Livingstone.

48.

Wechsler

(1997). Technical manual for the Wechsler Adult Intelligence and Memory Scale (3rd ed.). Psychological Corporation.

49.

Wilson

B. A.

Alderman

Burgess

P. W.

Emslie

, & Evans

J. J.

(1996). Behavioural Assessment of the Dysexecutive Syndrome. Thames Valley Test Company.

50.

Wilson

B. A.

Alderman

Burgess

P. W.

Emslie

, & Evans

J. J.

(2003). Behavioural Assessment of the Dysexecutive Syndrome (BADS). Journal of Occupational Psychology, Employment and Disability , 5 , 33–37.

51.

Wolf

T. J.

Stift

Connor

L. T.

, & Baum

; Cognitive Rehabilitation Research Group. (2010). Feasibility of using the EFPT to detect executive function deficits at the acute stage of stroke. Work , 36 , 405–412. https://doi.org/10.3233/WOR-2010-1045

52.

Wood

R. L.

, & Worthington

(2017). Neurobehavioral abnormalities associated with executive dysfunction after traumatic brain injury. Frontiers in Behavioral Neuroscience , 11 , 195. https://doi.org/10.3389/fnbeh.2017.00195