Abstract
This study found that performance-based assessments such as the Multiple Errands Test–Home Version (MET–Home) have the potential to guide occupational therapists when screening clients to determine when further, comprehensive assessment is indicated.
Driving is an important occupation for adults of all ages. However, driving is complex and therefore requires the interplay of high-level cognitive skills, including attention, memory, and executive functions for safe driving performance (Dickerson, 2014). Numerous medical conditions can cause cognitive impairments; however, a medical diagnosis alone is insufficient to predict driving performance (Classen et al., 2009). Ascertaining the level of impairment caused by health conditions and the impact on necessary skills and abilities for safe driving performance requires detailed assessment.
In clinical practice, there are two types of driver assessment: (1) screening and (2) comprehensive assessment. In-clinic screening involves identifying deficits with assessments that are rapidly completed. Regarding driving, these screening assessments can be administered by the medical practitioner, the driver licensing authority, and other allied health professionals as well as through self-screening by clients or family members (Austroads, 2022; Redepenning, 2006; Stav, 2003). The primary purpose of driver screening is to identity potential impairments that may impede safe driving and distinguish whether and when drivers should undergo further specialist assessment for driving safety (Korner-Bitensky et al., 2010). Comprehensive assessment is more extensive and structured, and it requires in-depth assessments.
Typically, a clinic-based and a behind-the-wheel (BTW) assessment is completed, and the latter can only be undertaken by specialist occupational therapists and other professionals in the United States, called certified driver rehabilitation specialists (CDRSs), who have undertaken training in driver assessment and intervention. In Australia, where this study was conducted, occupational therapists who have completed a specialist course are referred to as occupational therapy driver assessors (OTDAs; Fields & Unsworth, 2017). Although research into singular cognitive tests, assessment batteries, and driving simulators is extensive, no clear-cut evidence exists to accurately predict driving performance from the results of such tests alone; therefore, the BTW assessment is still considered to be the gold standard in assessing fitness to drive (Dickerson et al., 2014; Vrkljan et al., 2011). However, from the client’s perspective, getting access to and funding an OTDA are problematic in Australia, which pose sustainability issues for public safety.
The occupational therapy profession has a significant role in driving as identified in the community mobility domain of the Occupational Therapy Practice Framework: Domain and Process (4th ed.; American Occupational Therapy Association, 2020). This role is supported by evidence-based competency standards to guide both in-clinic and BTW assessments (Fields & Unsworth, 2017). Occupational therapists are well positioned to address driving as an occupational performance area, particularly with regard to client screening (Dickerson & Bedard, 2014; Dickerson et al., 2011). The generalist occupational therapy role focuses on commencing driving discussions and goal identification (Bedard & Dickerson, 2014; Scott et al., 2021); combining functional observation, standardized or performance-based assessments, and non-standardized assessments to evaluate how medical conditions may affect driving (Dickerson, 2014; Dickerson et al., 2011; Scott et al., 2021; Stapleton et al., 2015); providing information about medical clearance processes; and supporting clients with driving cessation and alternative community mobility options (Bedard & Dickerson, 2014; Scott et al., 2021).
However, use of these familiar assessment tools to guide clinical reasoning in relation to driving needs to be explored. Given the public safety concern regarding unfit drivers, occupational therapists need to screen their clients across the continuum of care, share assessment outcomes with OTDAs to reduce workload, and determine the appropriate type and timing of referral for comprehensive driver assessment with OTDAs (Bedard & Dickerson, 2014; Dickerson, 2014). Emerging research has suggested that a combination of tools (Dickerson et al., 2014) and evaluation of occupational performance in other complex occupations through performance-based assessments may be associated with clients’ driving potential (Dickerson et al., 2010, 2011), which supports occupational therapists to routinely screen drivers in clinic.
Occupational therapists struggle with screening clients because of reduced knowledge and confidence in being able to apply their assessment findings to driving (Scott et al., 2021). However, it is widely recognized that screening for intact cognitive processes is essential for determining driving suitability (Classen et al., 2009; Dickerson et al., 2014; Fields & Unsworth, 2017). A clinical reasoning process drawing on nonpropositional knowledge was used to identify the most suitable standardized assessments to help occupational therapists screen clients for driving potential. The most commonly used screening tools in the driving literature were identified but were excluded if the cost was more than $200 and if training was required for clinical utility purposes. Given the extensive literature in this area, an overview (a review of systematic reviews) by Fields and Unsworth (2017) was used that summarized the most commonly recommended standardized cognitive and perceptual impairment based-assessment tools used in the field of driving. These assessments included the Trail Making Test (TMT) Parts A and B (Reitan, 1958), Useful Field of View test (UFOV; McNamara et al., 2019), Rey–Osterrieth Complex Figure Test (ROCFT; Osterrieth, 1944, Standardized Mini-Mental State Examination (SMMSE; Marino et al., 2013), and the Clock Drawing Test (Christensen, 1984).
Of these, the TMT, SMMSE, and Clock Drawing Test were selected for inclusion in this study because they are cost-effective, have fewer training requirements, and are already commonly used in generalist occupational therapy practice compared with the UFOV and ROCFT. The SMMSE was selected because of unambiguous guidelines for administration and scoring that increase reliability (Molloy et al., 1991). Executive functioning is a necessary cognitive skill for driving, which is crucial to include in the screening process. To determine which performance-based assessment may be able to screen for driving potential, we examined a literature review by Poulin et al. (2013). These authors identified the following strongest performance-based assessment tools to examine executive functions on the basis of their psychometric properties: the Executive Function Performance Test (EFPT; Baum et al., 2008), the Multiple Errands Test (MET; Shallice & Burgess, 1991), and the Assessment of Motor and Process Skills (AMPS; Fisher & Jones, 2003). Although the AMPS has demonstrated predictive validity in driving outcomes (Dickerson et al., 2011), it requires specialized training for credentialing, takes a considerable amount of time to administer, and costs approximately $1,790, limiting its clinical utility.
The EFPT is used to examine functional cognition through observing occupational performance in instrumental activities of daily living, including cooking, phone use, medication use, and bill payment (Baum et al., 2008). However, in this assessment, therapists provided clients with cues to aid performance, which is not beneficial for assessing driving potential because driving requires independent task performance. The EFPT also lacks clinical utility and face validity because several items, such as writing a check, are now outdated. Although the MET is also used to observe task performance in occupations, such as phone use and medication management, it can evaluate the impact of executive function deficits in a broader range of everyday real-world tasks while also following predefined rules (Rotenberg et al., 2020), making it more analogous to driving than the EFPT. For example, when driving, people must perform multiple tasks, such as turning the steering wheel, checking speed, and performing head checks, while remembering to follow traffic rules. Moreover, pooled results of the psychometric properties of the MET by Rotenberg et al. (2020) outline strong evidence for its reliability and validity. Therefore, the MET was included in this study because we anticipated that if the MET could identify executive function deficits in everyday tasks, it could be a useful tool for screening executive function deficits in relation to driving outcomes.
The MET has several versions. Shallice and Burgess (1991) developed the MET in response to the disparity between clients’ high performance in neuropsychological tests and poor performance in everyday life tasks. Other versions of the MET have since emerged, including the MET–Simplified Version (Alderman et al., 2003), the MET–Hospital Version (Knight et al., 2002), and the Baycrest MET (Dawson et al., 2009). More recently, the MET–Home Version (MET–Home) was developed by Burns et al. (2019), acknowledging the importance of context in understanding real-world performance. This consideration is central for driving, which may explain why the BTW assessment remains the gold standard for assessing fitness to drive, because driving also occurs in a real-world environment (Dickerson et al., 2014). Therefore, further research to determine the association of the MET–Home with driving outcomes is warranted given (1) the ecological validity of the MET–Home, and its suitability to administer in the client’s home and local community where driving goals are also addressed, and (2) its reliability and validity as a performance-based assessment tool (Burns et al., 2019; Rotenberg et al., 2020).
To date, evidence is limited to support occupational therapists using and applying performance-based tools to the driving context. Therefore, the aim of this pilot study was to determine whether the MET–Home alone, or in combination with the TMT, Clock Drawing Test, and SMMSE, can be used to identify which drivers will pass their BTW assessment so that occupational therapists can better identify which clients require referral to a CDRS (or OTDA) for comprehensive driver assessment.
Method
Study Design
A cross-sectional pilot study was used to determine the accuracy of the MET–Home in identifying drivers who passed or failed their BTW assessment.
Participants
Over an 18-mo period, data were collected from consecutively referred, consenting clients age 18 yr and older who (1) had a change in their health status (any diagnoses with a potential cognitive decline or impairment likely to affect driving), (2) were previously driving and had a desire to return to driving, and (3) presented to private OTDA practice in the state of Victoria, Australia (population 6.7 million). To be eligible, clients needed to have gained medical clearance to drive, be able to communicate using simple English, and be independent with their mobility (either using a gait aid or motorized mobility device), which is a requirement for participation in the MET–Home. Clients were invited to participate through a small number of OTDA private practices selected for convenience for the researchers to travel to clients’ homes to complete the MET–Home.
Thirty people were approached to participate; however, 2 people did not meet the inclusion criteria (unable to read and write in English, moved interstate). The consenting 28 participants were divided into two groups according to outcome of “pass” (n = 20; 71.4%) or “fail” (n = 8; 28.6%). The most common diagnoses across both groups were dementia and stroke. Both groups were demographically similar in sex and age (for full details, see Table 1). Of those 20 participants who passed their BTW assessment, 80.0% (n = 16) had at least one license condition recommended to increase safety with resuming driving (for types of license conditions, see Table 2).
Comparison of Demographic Characteristics Between the Pass and Fail Groups on Their On-Road Driving Tests
Inadequate sample size for χ2 (p value) calculation.
License Conditions Recommended for the Pass Group (n = 20)
Note. Participants could have more than one recommended license condition.
Instruments
Multiple Errands Test–Home Version
The MET–Home consists of 14 tasks that involve the client locating various items (8 tasks), obtaining information and writing it down (4 tasks), asking the examiner what time it is 5 min after beginning of the test, and lastly stating when they have completed the assessment (Burns et al., 2019). Clients are scored on the number of these tasks accurately completed, partially completed, or omitted. While carrying out these tasks, the client must adhere to six predetermined rules, such as not going back to a room where they have already been (Burns et al., 2019). The number and occasion of rule breaks and the number of passes and inefficiencies observed are noted. A self-assessment performance interview is also conducted to ascertain levels of insight regarding performance strengths and limitations, which is scored on a 10-point Likert scale ranging from 1 (not well) to 10 (extremely well; Burns et al., 2019). The planning time and overall time taken are also recorded.
Trail Making Test Parts A and B
The TMT consists of two parts. Part A requires the client to keep their pen on the page and to join the numbers in a sequential order (e.g., 1, 2, 3). Part B requires the client to alternate between the numbers and letters (e.g., 1, A, 2, B). If the client makes an error, this mistake is pointed out immediately because the correction of errors is included in the completion time for the task. The task should be ceased if the client has not completed both parts after 5 min has lapsed; higher scores are correlated to greater cognitive impairment (Reitan, 1958). This test has been highly correlated with driving performance (Marshall et al., 2007); however, large amounts of variability exist in determining cutoff scores ranging from >39 to 178 s, indicating unsafe driving performance.
Clock Drawing Test
The Clock Drawing Test is a quick screening for cognitive impairment, which takes approximately 1 to 2 min to administer and asks the client to place all of the numbers in a predrawn circle so that it looks like a clock and to set the time to 11:10 (Christensen, 1984). The client is scored out of 7; a higher score represents better cognition (Freund et al., 2005). The Clock Drawing Test has demonstrated that cutoff scores of <4 = 100% specificity in clients failing their on-road driving test (Freund et al., 2005).
Standardized Mini-Mental State Examination
The SMMSE is a common assessment tool used to assist with predicting driving potential in older adults (Marino et al., 2013). Clients are scored between 0 and 30, with a score of 0 outlining severe cognitive impairment and a score of 30 outlining no cognitive impairment. Suggested cutoff scores consist of <24 indicating presence of a cognitive impairment and <20 likely unsafe to drive (Molnar et al., 2009).
Comprehensive Driver Assessment
The in-clinic assessment was conducted with the Occupational Therapy–Driver Off-Road Assessment Battery (Unsworth et al., 2012). The BTW assessment was conducted in a dual-controlled vehicle with a driving instructor present in the front seat and an OTDA seated in the rear, left passenger seat. The OTDA observed the client’s performance of driving tasks and skills on a public road within a drive time ranging from 45 to 60 min. Drivers were assessed and scored with standardized checklists, in line with BTW assessment guidelines (Catchpole & Di Stefano, 2018). At the conclusion of the assessment, OTDAs made recommendations to the licensing authority, which included able to return to driving, conditional driving (such as only driving within a kilometer radius), or unsafe to drive.
Data Collection Procedure
Eligible clients were contacted by Hayley M. Scott, and a time was organized to administer the MET–Home, TMT, Clock Drawing Test, and the SMMSE within their home environment. All these tests were completed on 1 day before the BTW assessment and took approximately 60 min. After the session, client test scores were collated and entered in a data sheet in preparation for data analysis. Clients completed the BTW assessment on a separate day with their regular OTDA, and a copy of all assessment forms were forwarded to the researchers. The study was approved by the Western Health and Federation University Human Research Ethics Committee.
Data Analysis
Statistical analyses were performed with IBM SPSS Statistics (Version 29.0). It was anticipated that for this pilot study an uneven small sample would be attained, because studies of driving assessment outcomes commonly have unbalanced samples with only a small number of clients failing their assessment. For example, the Unsworth, Russell, et al. (2019) study had a 79% pass and 21% fail rate, and McNamara et al.’s (2019) study had a 69% pass and 28% fail rate, often limiting analyses that can be performed. An a priori power analysis was conducted with G*Power, which estimated a sample size of 120 clients (60 per group) to achieve 80% power with a medium effect size (d = 0.5; Faul et al., 2007). However, this estimate far exceeds the realistic sample sizes recommended for pilot studies such as ours, as outlined by Hertzog (2008), who indicated sample sizes of 20 per group; it also exceeds the sample sizes obtained in previous research exploring associations between AMPS performance and driving outcomes, which reported samples of 20 participants per group (Dickerson et al., 2010).
Therefore, in our pilot study we aimed to attain a sample of 40 clients (20 per group). A simple approach to data analysis was used, with a focus on understanding the potential for the measures to screen for driving potential. As anticipated, a Box’s test for equivalence of covariance matrices showed that the data assumptions (including small and uneven sample sizes) did not hold to explore the effect of the tests on the dependent variable outcome of pass (with or without license conditions) or fail when a logistic regression analysis or a discriminant function analysis was used. Therefore, a Shapiro–Wilk test was used to test normality. For normally distributed variables, independent t tests were used to determine whether differences existed between the groups being considered (pass–fail). Finally, the Pearson r test was used to determine the relationship between variables. For non-normally distributed variables, Mann–Whitney U tests and χ2 tests were used. All tests were performed at a 5% level of significance. Participants with OTDA recommendations of being able to return to driving with or without license conditions (see Table 2) were classified into the pass group, whereas recommendations of unsafe to drive were classified into the fail group.
Results
Multiple Errands Test –Home Version
Table 3 presents the MET–Home test scores for this pilot study for the two driving outcome groups of pass or fail. The self-assessment score was the only subtest that significantly differed between the pass and fail groups (p = .014). The fail group scored on average 7 out of 10 (SD = 1.4), whereas the pass group scored on average 8 out of 10 (SD = 1.0). No relationship was found between self-assessment scores (participants’ perception of performance) and tasks accurately completed on the MET–Home (actual performance) for the pass (p = .892) and fail (p = .119) groups. Although no other subtests showed a statistically significant difference, there appeared to be a trend toward a higher number of MET–Home rules broken (M = 2.7, p = .153) that may be of clinical interest. Additionally, the time taken to plan out the MET–Home tasks (M = 2.8 min, p = .181) and the overall time taken to complete the MET–Home tasks (M = 17.7 min, p = .231) showed trends toward increased time taken by participants in the fail group, which may also be clinically important despite this result not being statistically significantly different.
Comparisons of Predictor Variables Between the Pass and Fail Groups on Their On-Road Driving Tests
Note. MET–Home = Multiple Errands Test–Home Version; SMMSE = Standardized Mini-Mental State Examination; TMT = Trail Making Test.
Mann–Whitney U test. Although expected median (interquartile range) values are not reported, the mean, standard deviation, and range have been included for easier interpretation of summary data.
Independent-samples t test.
TMT, Clock Drawing Test, and Standardized Mini-Mental State Examination
No differences were found among the TMT (Parts A and B), the Clock Drawing Test, and the SMMSE between the pass and fail groups. Participants in the fail group showed trends toward increased time required to complete the TMT Part A (76 s, p = .593) and Part B (199 s, p = .508); however, these differences were not statistically significant.
Discussion
The MET–Home is a common, performance-based assessment tool used by occupational therapists to assess a client’s executive function, which is an important skill for driving (Burns et al., 2019). In this pilot study, we sought to determine whether the MET–Home was associated with driving outcomes to help inform occupational therapists’ clinical reasoning of driving potential for client driving goals. However, our preliminary results suggest that the MET–Home is not associated with driving outcomes when used as a stand-alone tool or in combination with other standardized cognitive tools, noting that this finding needs to be confirmed with a larger sample of clients who pass and fail their BTW assessment. Although the self-assessment subtest was associated with driving outcomes, this result may not be of clinical importance, whereas conversely, the number of rules broken and time taken subtests were not associated with outcomes but may be more clinically important to assist clinical reasoning. This study reinforces that in-clinic tests are generally not good at identifying how a person will perform on road, and the BTW assessment remains the gold standard in assessing fitness to drive (Dickerson et al., 2014). However, further research is needed to support occupational therapists in interpreting the results of familiar performance-based assessment tools. This research will aid in determining the appropriate timing of referral for comprehensive driver assessment, ultimately enhancing public safety and client outcomes.
Although none of the core MET–Home subtests correlated with driving outcomes, such as tasks completed accurately, partially, or omitted, drivers who rated their overall performance on the MET–Home as 7 out of 10 or lower were significantly more likely to fail their BTW assessment. However, the reliability of this high score raises concerns regarding drivers’ self-awareness. Research on driver self-assessment and its connection to driving outcomes varies. Although a small number of studies have also reported that drivers’ self-assessment is associated with driving outcomes (Broberg & Willstrand, 2014; Rapoport et al., 2013), this relationship appears to be found within healthy adult populations; most research with clinical populations has suggested that drivers’ self-evaluations of their driving abilities are not correlated (Amado et al., 2014; Chen et al., 2021; Ross et al., 2012). Understanding these findings requires consideration of the accuracy of drivers’ self-assessment, which can be influenced by factors such as overestimation of skills because of reduced insight into driver-related impairments and drivers presenting themselves in a more favorable light than true abilities warrant, given the importance of driving to their lifestyle (Chen et al., 2021).
Because no relationship was observed between self-assessment scores and tasks accurately completed on the MET–Home, it raises questions about the efficacy of the MET–Home self-assessment item and its association with driving outcomes. Although the latter finding may have been statistically significant, its clinical applicability remains uncertain and should be interpreted with caution. Conversely, participants who failed their BTW assessments showed trends toward requiring increased time to plan and complete the MET–Home. Although this finding was not statistically significant, it may be more clinically relevant because trends with increased time taken were also found for completing the TMT, which is consistent with previous research in driving (Marshall et al., 2007).
When determining driving potential, the use of a combination of carefully selected assessment tools, rather than tools in isolation, is recommended (Dickerson et al., 2014). Our study revealed no significant relationship between the MET–Home and other cognitive–perceptual tests. However, researchers in prior studies have offered algorithms for predicting driving potential by using a combination of tests tailored to specific medical conditions. For instance, people with dementia have been assessed with the Clock Drawing Test, TMT Part A, and Snellgrove Maze test (Carr et al., 2011), whereas those poststroke have been evaluated with the Mini-Mental State Examination, Road Law Road Craft Test, and Drive Home Maze Test (Unsworth, Baker, et al., 2019). Combining assessment tools can strengthen the predictive validity of driving outcome; yet, it’s important to note that no study has consistently demonstrated 100% accuracy, reinforcing the BTW assessment as the gold standard (Dickerson et al., 2014). Nevertheless, these findings are important because they can still support occupational therapists in determining the necessity for more comprehensive driver evaluation, thereby potentially saving clients from unnecessary assessment cost and guiding appropriate referral timing.
Occupational therapists use evidence-based assessment tools or their expertise in observation of occupational performance to inform their clinical reasoning when evaluating driving as an instrumental activity of daily living (Dickerson et al., 2014). The evaluation of performance-based assessment tools and their relationship with driving outcomes is an emerging research area within the field of occupational therapy. Although the AMPS has been found to distinguish among drivers who pass, pass with conditions, and fail their BTW assessment (Dickerson et al., 2010, 2011), our findings show that the MET–Home was not able to discriminate among drivers. However, differences within our study that may have influenced the driving outcomes include the absence of healthy controls for comparison, multiple OTDAs undertaking the scoring with little evidence of the reliability in the findings, the utilization of different driving routes with the potential for assessments with different levels of complexity, and a smaller sample size.
Although the AMPS can support occupational therapists in deciding when to refer their clients for comprehensive assessment, it requires specialist training, which can be costly and not readily accessible for all occupational therapists. Consequently, additional research is warranted to ascertain whether the MET–Home, other MET versions, or other performance-based assessment tools can be used to support generalist occupational therapy practice regarding driving. If it is found that none of the existing performance-based assessments with acceptable clinical utility are associated with driving potential, then it’s important that further research be undertaken to develop a new assessment.
Limitations
Although the sample consisted of consecutive referrals, clients who potentially were not referred for assessment could not be tracked; however, systematic bias in the sample is unlikely. The OTDAs involved used various BTW assessment routes potentially affecting the reliability of the driving outcomes. In addition, the sample in this pilot study was small, which lowered the study power because the MET–Home contains 10 subtests rather than an overall score. Although a sample size of 40 participants was anticipated to detect differences in MET–Home outcomes for drivers who passed or failed, on the basis of an a priori power calculation and previous research (Dickerson et al., 2011; Faul et al., 2007), this sample size was not attained because of coronavirus disease 2019 (COVID-19) restrictions and significant time restraints. Moreover, the size of the fail group was lower than the pass group, which limited required assumptions to conduct a discriminant function analysis (as discussed in the “Data Analysis” section). Nonetheless, this pilot study showed trends that warrant further research.
Conclusion
Because evidence from this pilot study is insufficient to associate MET–Home scores with driving outcomes, this tool is not currently recommended for use by occupational therapists to screen driving potential. However, occupational therapists still require further supports to fulfill their role; therefore, more research is required to investigate the MET–Home or other performance-based assessments to help them address client driving goals.
Footnotes
Acknowledgments
We thank the occupational therapy driver assessors for their time and contributions. We also thank our biostatistician, Tanita Botha, for her support in selecting and conducting the analyses.