Abstract
Many people who sustain acquired brain injury (ABI) experience difficulty performing daily occupations that involve multiple goals or steps. For example, when making a trip to the shopping mall, people often have multiple tasks to complete, and those with ABI often fail to complete all tasks or do so inefficiently. Such difficulties have often been referred to as executive cognitive impairments, which include problems with planning, monitoring, task switching, inhibiting more routine responses (when novel ones are required), working memory, and prospective memory, and have long been associated with frontal lobe damage (Cicerone et al., 2006; Shallice & Burgess, 1991).
Executive dysfunction is most readily observed in nonroutine, complex, and unpredictable situations that occur frequently in everyday life (e.g., your family arrives for an unexpected dinner) and is often difficult to detect on standardized neuropsychological assessments (Burgess et al., 2006). In fact, it was an observation of everyday life difficulties in people with frontal lobe damage who performed relatively well on standardized tests that led Shallice and Burgess (1991) to develop the original Multiple Errands Test (MET). Although numerous versions of the MET have since been published (e.g., hospital versions: Clark et al., 2017; Dawson et al., 2009; Knight et al., 2002; Morrison et al., 2013; a shopping mall version: Alderman et al., 2003; and a virtual reality version: Cuberos-Urbano et al., 2013), there have been few investigations into patterns of performance on real-world versions of the MET (i.e., non–virtual reality versions) and scores on standardized tests of neuropsychological function. This study addresses this gap in the literature.
Published versions of the MET are modeled on the original versions (Knight et al., 2002; Shallice & Burgess, 1991) and generally include 12 tasks (e.g., buy a birthday card) that need to be completed while following nine rules (e.g., do not spend more than $X). People being tested carry the task list and rules with them throughout the assessment. Errors are scored when participants do not attempt a task (tasks omitted), make errors in completing a task (partial task failures; e.g., buying a get-well card rather than a birthday card), and break the rules (total rule breaks).
Common performance subscores on the MET include the number of tasks (out of 12) that were not attempted and the number of errors made while trying to complete tasks. All studies using real-world versions of the MET have shown that the performance of people with ABI is significantly worse than that of healthy controls (e.g., Alderman et al., 2003; Dawson et al., 2009; Knight et al., 2002; Morrison et al., 2013). These studies have also shown that the magnitude of the difference between the samples with ABI and healthy controls varies depending on the MET performance subscore and the sample. For example, Dawson et al. (2009) reported the effect size of the group difference between participants with ABI and healthy controls for rules broken was larger than the effect size of the group difference for tasks omitted. This finding may have clinical relevance if the underlying cognitive difficulties are also different. For example, if a participant breaking rules has a cognitive profile indicating working memory difficulties, a treatment plan may be focused in a different way from a plan for a participant who is breaking rules related to errors of inhibition.
Although it is hypothesized that performance on real-world versions of the MET relies primarily on executive cognitive functions, to our knowledge, only Knight and colleagues (2002) have explored the associations between scores on standardized neuropsychological assessments and MET performance. They investigated the relationship between MET performance, general cognitive ability, memory, and visual perception and found a statistically significant relationship between a general memory score (profile score on the Rivermead Behavioural Memory Test [RBMT]; Wilson et al., 1985) and task errors. They also found statistically significant relationships between MET performance and perseverative errors on a card-sorting test (standardized neuropsychological test). This study suffered from multiple comparisons with a small sample (n = 20 participants with ABI) but did provide preliminary evidence for different patterns of performance on the MET and cognitive tests. Notably, the study did not investigate relationships between MET performance and task switching or processing speed, although these factors are commonly reported cognitive difficulties in the ABI population (Chan, 2005).
Understanding what types of cognitive impairments contribute to specific performance patterns on the MET may provide important insight into the everyday life behaviors of people with ABI and be useful in developing treatment plans. Therefore, the objective of this study was to investigate the associations between cognitive functions (executive function [EF], memory, attention, and processing speed) and MET scores (for tasks omitted, partial task failures, and total rule breaks).
Method
Research Design
This study used a nonexperimental, correlation design with data originally collected by Dawson and colleagues (2009) for further development of the Baycrest-MET regarding standardized scoring, reliability, and ecological validity. These data, collected from June 2004 to April 2005 at Baycrest Health Sciences (Toronto, Ontario, Canada), were not changed, consolidated, or rescored in any way for this study. This study represents a unique analysis of a previously collected data set.
Participants
In the original study (Dawson et al., 2009), 27 participants with ABI (n = 14 stroke, n = 13 traumatic brain injury) were recruited using convenience sampling. All participants met the following inclusion criteria: sustained their ABI at least 3 mo before testing, were age 18 yr or older, were fluent in English, were able to walk independently for at least a half hour, and scored below the depression cutoff score of 16 on the Center for Epidemiological Studies Depression scale (Radloff, 1977). All participants provided written, informed consent for the original data collection. The secondary analysis performed for the current study was conducted in accordance with human ethics standards and received approval from the Baycrest Health Sciences and University of Toronto research ethics boards.
Measures
Age, gender, and completed years of education were obtained from each participant. The Mini Mental State Examination (MMSE; Folstein et al., 1975) was used to screen for cognitive impairments. The North American Adult Reading Test (NAART; Uttl, 2002) was used to estimate premorbid verbal intelligence. Because it is a naturalistic assessment, all versions of the MET have to be site specific. Therefore, the Baycrest-MET was used, which has the same format as previously published versions of the MET (including 12 tasks and nine rules) but with tasks and errands that were possible to carry out at Baycrest Health Sciences. Performance scores for the Baycrest-MET are as follows:
Tasks omitted: For each task not attempted, a score of 1 is given. A maximum score of 12 can be obtained, indicating that no tasks were attempted, that is, they were omitted. Thus, higher scores indicate worse performance.
Partial task failures: Accurate task completion involves multiple steps and the avoidance of some errors. Forty-six possible errors on the 12 tasks were identified a priori for the Baycrest-MET by Dawson et al. (2009). These errors correspond to partial task completions or errors in completion of a task. For example, for the task “mail something to the examiner,” a participant might obtain something to mail and address the envelope but not mail it (task is incomplete) or might put too much postage on the item before mailing (task is completed but not accurately). Participants are given a score of 1 for each error; thus, higher scores indicate worse performance.
Total rule breaks: The nine rules are scored as either broken or adhered to. Some rules may be broken more than once; thus, a score of greater than 9 is possible, and higher scores indicate worse performance.
Participants completed a series of neuropsychological tests in the domains of attention, memory, and EF, because these domains represent the primary areas of cognitive function and have separate although overlapping effects on everyday life function (Gillen, 2009). Because executive dysfunction can be difficult to detect on standardized tests (Burgess et al., 2006), we used several different measures that are sensitive to various aspects of executive cognitive function.
Standardized neuropsychological tests, in general, have excellent psychometrics (Spreen & Strauss, 1998). Whereas cognitive tests in the occupational therapy literature are often designed to measure cognitive processes in the context of functional tasks (e.g., the Executive Function Performance Test [Baum et al., 2008]), neuropsychological tests are designed to measure these processes more discretely. To gain a full understanding of the interrelationships between cognition and Baycrest-MET performance, we measured executive cognitive processes using the following neuropsychological tests:
Wisconsin Card Sorting Test (WCST; total categories completed, maximum is six; Heaton et al., 1993), as an indicator of organizational (specifically categorizing) ability, because this test requires participants to categorize cards without being told the rule, a skill that might be used in organizing a shopping list.
Zoo Map (sequence score; Wilson et al., 1996), as an indicator of organization and planning abilities, because this test requires choosing the most efficient route (on a pencil-and-paper map) to visit animals at the zoo, a skill that might be used in planning any multierrand trip (e.g., shopping at the mall).
Phonemic Fluency (Spreen & Strauss, 1998), as an indicator of verbal working memory, because this test requires participants to generate as many words as possible beginning with a set letter in 60 s, a skill that might be used in asking directions.
Digit Span Backward (Wechsler, 1958), as an indicator of numeric working memory, because this test requires participants to recite backward sequences of numbers presented orally, a skill that might be used in maintaining a mental budget during a shopping trip.
Trail Making Test Parts A (Trails A) and B (Trails B; Army Individual Test Battery, 1944), because Trails A provides a measure of visuomotor speed (Sánchez-Cubillo et al., 2009) and Trails B requires attentional shifting in which the participant quickly draws lines alternating between circles containing numbers and circles containing letters in sequence. A difference score (Trails B minus Trails A) was used, because it provides an indicator of executive control abilities (Sánchez-Cubillo et al., 2009). These skills might be used in a multierrand situation in which a person switches between tasks.
Because everyday life involves both verbal and visual memory, two tests of memory were used: the California Verbal Learning Test, second edition (CVLT–II; Delis et al., 2000), and the Brief Visuospatial Memory Test–Revised (BVMT–R; Benedict, 1997). The CVLT–II tests immediate and delayed recall of a word list. The BVMT–R tests the ability to reproduce six geometric figures in their correct location on a page immediately after a 10-s viewing time and after a 25-min delay. To measure long-term memory, the delay score was used for both the CVLT–II and the BVMT–R.
The ability to sustain attention over time was assessed using Digit Span Forward (Smith, 1978), a test that requires participants to repeat sequences of numbers back to the examiner. This test is an indicator of efficiency of attention and freedom from distractibility (Lezak et al., 2012). Being able to process information in a time-efficient way was assessed using the Digit Symbol test (Smith, 1978), in which participants are given a series of symbols assigned to Numbers 1–9, and then must fill in symbols corresponding to each number as quickly as possible.
All tests used have good to excellent test–retest reliability. However, it is likely that many participants would have some familiarity with some of these tests, which may have resulted in achievement of higher scores than would occur otherwise. To our knowledge, no participants had received any of these neuropsychological tests within 6 mo of testing for this study.
Planned Analyses
Descriptive statistics were calculated for all measures. Graphical displays indicated that the Baycrest-MET error scores and neuropsychological assessments were sufficiently normally distributed; therefore, parametric tests with an α level of .05 were used. The relationship between individual neuropsychological tests and individual Baycrest-MET error scores was assessed using Pearson correlation coefficients (r; one-tailed). When the correlation was significant at p ≤ .05, the percentage of variance accounted for between the pair of variables also was computed (r 2 multiplied by 100). Raw scores were used for all tests, and data were analyzed using IBM SPSS Statistics (Version 21; IBM Corp., Armonk, NY).
Results
Table 1 summarizes participant demographic and clinical characteristics. All participants were living in the community and were on average almost 10 yr after ABI, and the majority were male (70%). Table 2 lists assessment scores. All participants scored within the normal range on the MMSE and NAART.
Participant Demographics (N = 27)
Note. ABI = acquired brain injury; M = mean; SD = standard deviation; TBI = traumatic brain injury.
Mean Assessment Scores (N = 27)
Note. BVMT–R = Brief Visuospatial Memory Test–Revised; CVLT–II = California Verbal Learning Test, second edition; M = mean; MET = Multiple Errands Test; MMSE = Mini Mental State Examination; NAART = North American Adult Reading Test; SD = standard deviation; Trails A = Trail Making Test Part A; Trails B = Trail Making Test Part B; WCST = Wisconsin Card Sorting Test.
To investigate the associations between cognitive functions and performance on the Baycrest-MET, Pearson correlation coefficients were calculated between the neuropsychological tests and Baycrest-MET error scores (Table 3). All correlations were in the expected direction; that is, worse performance on the MET (i.e., more tasks omitted, more partial task failures, or more rules broken) was associated with worse performance on neuropsychological tests. Baycrest-MET tasks omitted was significantly correlated with measures of long-term verbal and visuospatial memory (CVLT–II, BVMT–R), accounting for 14.4%–31.4% of variance; tests of attention (Digit Span Forward), accounting for 16.8% of variance, and processing speed (Digit Symbol), accounting for 24.0% of variance; and measures of EF (Digit Span Backward, Phonemic Fluency, Zoo Map, and Trails B – Trails A), accounting for 15.2%–42.3% of variance. Baycrest-MET total rule breaks was significantly correlated with long-term visuospatial memory (BVMT–R), accounting for 18.5% of variance. Modest associations were also found with a measure of attention (Digit Symbol) and EF (Digit Span). Baycrest-MET partial task failures was not significantly correlated with any neuropsychological measures, although a modest association (r = −.31) was noted with EF (WCST). In summary, although a significant relationship between measures of attention, processing speed, EF, and the Baycrest-MET was found only for Baycrest-MET tasks omitted, both Baycrest-MET tasks omitted and total rule breaks were related to a measure of long-term visuospatial memory (BVMT–R), and Baycrest-MET tasks omitted was also associated with long-term verbal memory (CVLT–II).
Correlations Between Baycrest-MET and Neuropsychological Test Performance
Note. BVMT–R = Brief Visuospatial Memory Test–Revised; CVLT–II = California Verbal Learning Test, second edition; EF = executive function; MET = Multiple Errands Test; Prop Var = proportion of variance (r 2 × 100); Trails A = Trail Making Test Part A; Trails B = Trail Making Test Part B; WCST = Wisconsin Card Sorting Test. Dashes indicate that the proportion of variance accounted for is not significant.
p ≤ .10.
p ≤ .05.
p ≤ .01.
Discussion
This study was designed to investigate the cognitive functions associated with successful completion of the MET. Although the MET is understood to be a measure of the effect of executive dysfunction on everyday life tasks, few studies have examined the relationship between performance on standardized neuropsychological tests and performance on the MET. Using data collected from 27 participants with ABI on the Baycrest-MET, the results of this study suggest that although all measured cognitive functions (attention, memory, EF, and processing speed) are related to performance on the Baycrest-MET, their contribution varies depending on the aspect of Baycrest-MET performance examined. Moreover, because most of the variability in the data set was best accounted for by looking at relationships between the two sets of variables, this analysis also suggests that everyday life performance cannot readily be reduced to bivariate relationships between discrete aspects of that performance and specific cognitive functions.
In this data set, the score for Baycrest-MET tasks omitted was explained by performance on tests of EF, attention, processing speed, and delayed memory. Specifically, the tasks omitted score was significantly correlated with three aspects of EF: working memory (tested with Digit Span Backward, Phonemic Fluency, and Trails B – Trails A) and organization and planning (tested with Zoo Map and WCST). Participants with EF impairments (in working memory, organization, or planning) were more likely to omit tasks, despite having the task list available to them throughout the Baycrest-MET. In addition, tasks omitted was significantly correlated with a test of sustained attention (Digit Span Forward) and a measure of processing speed (Digit Symbol), suggesting that people who have impairments in sustained attention and processing speed omit more tasks.
The Baycrest-MET tasks omitted score was also significantly correlated with a test of delayed verbal recall (CVLT–II), and both tasks omitted and total rule breaks were significantly correlated with a measure of visuospatial memory (BVMT–R). Knight and colleagues (2002) also found that people who omitted tasks performed poorly on the RBMT, but they did not find an association between total rule breaks and memory. The RBMT measures multiple aspects of memory, including verbal and visuospatial. The weaker association of tasks omitted with verbal memory may be related to the fact that participants carry the task and rule list with them throughout the assessment. This factor may offset the demands on delayed verbal recall. In addition, before the assessment, participants have the opportunity to memorize the rules, which may account for the lack of association between total rule breaks and verbal memory. Furthermore, in this study, participants with poor visuospatial memory broke more rules, whereas those with poor verbal memory did not. This finding suggests that the rule memorization procedure may support MET performance for those with poor verbal memory.
The Baycrest-MET partial task failures score was not significantly correlated with any neuropsychological tests. Partial task failures comprises a large number of possible errors (>46). These errors are variable in nature, from location errors (e.g., went to the wrong place) to record-keeping errors (e.g., wrote incorrect price). Certain partial task failures could be related to specific cognitive functions, but the small sample size prevented examination of those potential relationships. The failure of the neuropsychological tests to explain Baycrest-MET partial task failures further supports the notion that some traditional task-based neuropsychological tests of EF may not effectively capture clinically significant executive impairments (Burgess et al., 2006).
Limitations and Future Research
Participants in this study were all living in the community and were on average almost 10 yr after ABI. It is likely that they had developed compensatory behaviors in their community-living skills, possibly resulting in fewer errors on the Baycrest-MET and limiting the associations with neuropsychological test performance. Associations may also have been attenuated because participants’ possible familiarity with the neuropsychological tests would have resulted in higher scores on the tests. In addition, the method of sampling (convenience) meant that medical histories were not accessible for some of the participants, and some of the participants may have had conditions too mild to have significant performance effects in everyday life.
Future research on the MET with a larger sample is necessary to validate these findings and will provide additional insight into the patterns of cognitive impairments that contribute to various errors on the MET. Future research should investigate patterns of strategy use in relation to cognitive impairments. A more in-depth understanding of performance errors and strategy use will enhance the clinical utility of the MET and may inform more effective intervention planning.
Implications for Occupational Therapy Practice
Given the small sample in this study, implications for clinical practice are made prudently. However, the results may be informative for clinicians because they suggest that clients will perform differently on the MET depending on their cognitive profile. For purposes of intervention, it follows that different strategies will be useful to different clients to improve everyday life performance. It may be particularly helpful to guide clients in identifying useful compensatory strategies in the context of a metacognitive intervention approach, many of which have been recently reviewed (Dawson et al., 2017; Radomski et al., 2016).
Conclusion
This study suggests that specific MET performance errors may provide practitioners with insight into clients’ cognitive functioning in areas in addition to EF. Specifically, the data from this study suggest that
The MET is mostly a measure of EF, but it also captures aspects of memory, attention, and processing speed.
Individuals who omit multiple tasks have greater impairments in EF.
Processing speed may be more important than simple attention to attempt tasks.
Total rules broken may be indicative of visuospatial memory impairments.
Footnotes
Acknowledgments
The authors acknowledge the participants who made this work possible. This project was completed while authors Hansen and De Amicis were graduate students at the Department of Occupational Science and Occupational Therapy, University of Toronto, Toronto, Ontario, Canada, and Rotman Research Institute, Baycrest Health Services, Toronto, Ontario, Canada. This research was presented in part at the Canadian Association of Occupational Therapists annual conference in April 2016.