Development,Reliability,and Validity of the Multiple Errands Test Home Version (MET–Home) in Adults With Stroke

Research Design

This study consisted of two phases: (1) adaptation of the MET with standardized administration and scoring protocol for use in the home environment and (2) psychometric analysis. Ethical clearance was obtained from the institutional review board of the Texas Woman’s University, and participants provided written informed consent before participating in the study.

Participants

Purposive, convenience, and snowball sampling techniques were used to recruit home-dwelling adults with stroke (n = 23) and control participants (n = 23) who were individually matched for age (±3 yr), education (±3 yr), and gender. A priori power analysis (with G*Power; Faul et al., 2009) indicated that 46 participants would be sufficient to detect difference with a power level set at .80, an α level of .05, and a moderate to high effect size of .40 for correlation analysis. We used the following inclusion criteria: ≥90 days poststroke; mild to moderate stroke severity; no dementia; independent before stroke; less than moderate depression; home dwelling >30 consecutive days; ≥18 yr old; and English speaking, reading, and understanding. Control participants met the same criteria except that they were fully independent at the time of the study. Exclusion criteria included history of previous stroke or other neurological condition, other conditions that could affect executive function, or other impairments that affect the ability to carry out the study protocol using National Institutes of Health Stroke Scale (Brott et al., 1989) in-scale cutoffs and clinical judgment.

Multiple Errands Test Home Version Development

The MET–Home was designed to reflect previously developed versions of the MET administered in various locations such as hospitals, shopping centers, and virtual environments. We initially developed a task and rule list designed to mirror other MET versions. However, several MET–Home task items were modified to fit the home. For example, the Baycrest MET (Dawson et al., 2009) and the MET–Revised (Morrison et al., 2013) provide maps for navigation; however, the home is familiar, and map use is not indicated. Instead, we added several tasks that involved locating items. We anticipated this modification would press participants to navigate to a greater number of rooms in the home and add to the complexity with spatial planning. Completing a monetary transaction is a task that also appears on other MET versions (e.g., Dawson et al., 2009; Shallice & Burgess, 1991); however, simple transactions for goods are not typical in the home. Therefore, the MET–Home integrated day-to-day purchases by including tasks for locating prices through whatever means were available in the home (e.g., phonebook, ads, computer, smartphones). Rules were modified to fit the home but included the same basic aims.

The developed task list was provided to adults with stroke at a local rehabilitation facility to review for applicability and was found to be relevant, contributing to its face validity. The tool underwent expert panel review, and revisions were made on the basis of the following feedback: integration and adaptation of the strategy classification sheet from the Baycrest MET (Dawson et al., 2009), addition of a prospective memory component to one task, and modification of wording for clarity. The tool was then given to five home health occupational therapy practitioners to use in the field who provided feedback after 2 mo. Revisions were made that included modifications to wording, and technology use was added to the strategy list. Next, the tool was piloted (N = 4), and additional revisions were made that included scoring simplification.

The MET–Home (Supplemental Figure 1, available online at http://otjournal.net; navigate to this article, and click on “Supplemental”) consists of a list of 14 tasks constrained by six rules. The assessment contains the following subscores: total time planning, total time elapsed, task completion (accurately, partially, or omitted), frequency and types of rule breaking, frequency and types of strategies used, number of passes (passing by a task that could have been carried out), number of tallied inefficiencies (becoming distracted and not being efficient with a task), and qualitative description of the environmental influence on performance. Findings from the subscores can provide information about how errors are made in context, potential influences on performance, and how naturalistic strategies are used to support performance.

Study Assessments

Participants initially completed a structured questionnaire on demographic characteristics and relevant clinical variables (e.g., age, gender, diagnosis, date of stroke). The Barratt Simplified Measure of Social Status (Barratt, 2006) was used to measure socioeconomic status (SES), which involves calculations of an aggregate score based on the education and occupation of the participant, the participant’s spouse or partner, and the participant’s parents.

We integrated the following series of assessments for screening and MET–Home analysis:

We selected a series of assessments to examine convergent and concurrent validity. Convergent validity is used to measure the degree to which one instrument is comparable with another that measures similar traits, and concurrent validity is used to examine the relationship between a specific instrument and another instrument that is already considered valid (Kielhofner, 2006). We selected neuropsychological assessments of executive function and a cognitive screen for examining convergent validity and the Executive Function Performance Test (EFPT; Baum et al., 2008) for examining concurrent validity:

Data Collection

All assessments were completed during a single visit in the participants’ homes. Each participant selected the time of day on the basis of his or her perspective of best performance and was asked to have the home inhabited naturalistically. Distractions were minimized during administration of the neuropsychological assessments to maintain instrument validity. The assessments were administered in a random sequence determined a priori, with the MET–Home administered first to mitigate potential learning from other assessments (i.e., EFPT). During MET–Home administration, participants were given a one-page task and rule list (Supplemental Figure 1), a clipboard with loop for ease of handling, and a pen or pencil. The evaluator read directions from the instruction sheet aloud and described the rules while panning the participant page with her hand to describe different test components. Participants were asked to repeat the rules to ensure comprehension and were prompted to ask questions, review the page for as long as desired, and verbalize when ready to begin, thus signaling the rater to initiate timing.

Data Analysis

We conducted all data analysis with IBM SPSS Statistics (Version 22; IBM Corp., Armonk, NY). Internal consistency of the MET–Home task list was calculated with Cronbach’s α, which increases as intercorrelations between test items increase. For interrater reliability, the two raters reviewed the scoring protocol before scoring and then independently scored video recordings. Results were not shared until after analysis. We conducted and estimated a two-way random method for calculating intraclass correlation coefficients (ICCs) using 95% confidence intervals for six subscores of the MET–Home: accurately completed, partially completed, omitted, rule breaks, passes, and inefficiencies.

We computed convergent and concurrent validity. Scores were visually inspected for normality with histograms, which indicated that distributions were heavily skewed. Spearman’s correlation coefficient (ρ) was used for data analysis because variation in data normality existed. Correlations were analyzed for the group of adults with stroke only because significant differences in scores were discovered (Supplemental Table 1). In addition, we examined MET–Home performance differences between adults with stroke and matched control participants. Because of violations of normality, the nonparametric Mann–Whitney U test was conducted to examine differences between adults with stroke and matched control participants. We calculated effect sizes to quantify the magnitude of difference between groups using Cohen’s d.

Participants

Demographic characteristics of the participants are reported in Table 1. Participants represented a variety of living situations. Adults with stroke ranged in chronicity and hemisphere localization. Significant differences in race/ethnicity and SES were identified (see Table 1). Control participants had a higher SES, and race/ethnicity varied between groups.

Internal Consistency and Interrater Reliability

We computed internal consistency reliability and found the MET–Home item task list acceptable (14 items; α = .73). Each task item α ranged from .68 to .74. For interrater reliability, the two raters achieved excellent interrater reliability of ≥.80 during MET–Home piloting. Interrater reliability for subscores ranged from .88 to .96, indicating excellent interrater reliability (Cicchetti, 1994; Table 2).

Convergent Validity

Some significant moderate correlations were discovered among the MET–Home subscores and the SDMT and D-KEFS Tower Test (Table 3). Significant positive and negative correlations were expected because MET–Home subscores increase both on the accuracy scale and with higher frequency of rule breaks, omissions, passes, and inefficiencies. Correlations with the MoCA, Dysexecutive Questionnaire, and D-KEFS Sorting Test were not significant.

Concurrent Validity

We analyzed scores for adults with stroke and found a significant negative moderate correlation between MET–Home accurate completion subscore and EFPT total score; we also found positive correlations among two EFPT cognitive construct subscores and MET–Home omissions, passes, inefficiencies, and rule breaks (see Table 2). Negative and positive relationships were expected as a result of scale differences because the EFPT is scored on levels of cueing in which fewer cues indicate better performance. Refer to Supplemental Table 1 (available online) for the assessment scores used for convergent and concurrent validity analyses.

Performance Differences

The sample of adults with stroke demonstrated significant differences in MET–Home performance compared with the matched control participants (Table 4). Adults with stroke completed fewer tasks accurately and had a greater frequency of tasks partially completed or omitted, rule breaks, passes, and inefficiencies. All effect sizes were large or very large except for “planning time.”

Reliability

This study revealed that the MET–Home subscores have adequate internal consistency and excellent interrater reliability. Similar, Knight et al. (2002) achieved adequate internal consistency (α = .77) and excellent interrater reliability (ICCs = .81–1.00) on the MET Hospital Version subscores. In addition, Dawson et al. (2009) achieved adequate to excellent interrater reliability (ICCs = .71–.88) on the Baycrest MET among adults with acquired brain injury. In our opinion, reviewing the assessment protocol before the scoring sessions supported the excellent interrater reliability for assessment subscores.

Validity

The MET–Home demonstrated evidence of convergent and concurrent validity. First, the MET–Home had significant correlations with two of the neuropsychological assessments. Associations were noted in some of the MET–Home subscores, suggesting that specific skills assessed with the neuropsychological assessments emerge during the MET–Home. The SDMT was associated with three MET–Home subscores, suggesting that skills such as processing speed and shifting attention may emerge during MET–Home execution. In addition, Nocentini et al. (2006) suggested that working memory substantially influences SDMT scores. Working memory can be described as a system that coordinates processing when several goals are present and guides behavior with information that may not be immediately present in the environment (D’Esposito & Postle, 2015).

In our opinion, working memory emerges throughout MET–Home performance. We also discovered associations between the D–KEFS Tower Test Rule Violations and MET–Home passes subscore. The Tower Test assesses skills such as spatial planning, inhibition of responses, and establishing and maintaining instructional set. It is likely that these skills emerge to prevent “passes” because to prevent a pass on the MET–Home, one must plan a route and follow rules (i.e., not enter a space more than once). We found no significant association with the D–KEFS Total Achievement Score, indicating that the skills for preventing rule violations may have primarily contributed to the association.

The D–KEFS Sorting Test, MoCA, and Dysexecutive Questionnaire demonstrated no significant associations with the MET–Home. It is surprising that no significant associations were found with the D–KEFS Sorting Test, which measures initiation of problem solving, flexibility of thinking (set-shift), and concept formation; however, this finding may be explicated by the fact that the Sorting Test uses modality-specific problem solving (verbal and perceptual) versus problem solving embedded in complex everyday task performance. This outcome is consistent with literature on integrating an ecologically valid approach to assess executive dysfunction, which suggests that real-life tasks are often complex and multidimensional (Morrison et al., 2015). The difference in how the skills were evaluated may contribute to the findings. We expected the limited associations with the MoCA because this tool contains a domain for executive and visuospatial screening, but the total score is used as a screen of general cognition.

In addition, we found no significant associations with the Dysexecutive Questionnaire, which is self-reported and reflects the personal experience of executive dysfunction. Toplak et al. (2013) suggested that rating measures of executive function assess different underlying neurocognitive constructs than performance-based assessments, and although both assessment categories are valuable, they provide different types of information. It is possible that the MET–Home and Dysexecutive Questionnaire are not correlated because they may not capture the same underlying process or neural substrates (Toplak et al., 2013). We anticipated the general strength of associations with neuropsychological assessments because it is likely that other skills affected performance. For instance, the neuropsychological assessments used in this study did not demand the same degree of motor involvement as the MET–Home.

The EFPT is an established and well-accepted assessment in stroke rehabilitation. Significant moderate correlations with the EFPT support the concurrent and ecological validity of the MET–Home. Correlations among the assessments were stronger than anticipated because we expected the novel and loosely structured method of presenting everyday tasks on the MET–Home to affect the strength of the association. Although the EFPT and MET–Home are both performance-based assessments of everyday task performance, the administration and assessment content are dissimilar. For instance, when administering the EFPT, the evaluator presents one task at a time (e.g., medication management, bill payment), and scoring is based on the level of cueing needed to support performance. Alternatively, the MET–Home task list is loosely structured, and the evaluator does not provide cues. The EFPT and MET–Home both use an occupation-focused approach, which likely contributed to the significance and strength of the correlation. In fact, the EFPT and MET–Home both lend themselves to the assessment of the effects of executive dysfunction on daily tasks typically carried out in the home environment and thus may provide complementary findings that may support an enhanced understanding of real-world performance.

Performance Differences

In this study, we found that the MET–Home distinguished between adults with poststroke executive dysfunction and matched control participants. The adults with stroke achieved significantly lower scores for tasks accurately completed and demonstrated higher frequency of omissions, tasks partially completed, rule breaks, passes, and inefficiencies. These findings are consistent with literature supporting other context-specific versions of the MET (e.g., Alderman et al., 2003; Dawson et al., 2009).