Abstract
The ScanCourse was found to have acceptable discriminatory ability for on-road driving performance; evidence supports its continued use as a screening tool to assess driver fitness.
Driving is a highly valued method of community mobility that promotes autonomy and participation for many people in developed countries (Dickerson et al., 2014; White et al., 2012). It is a complex activity requiring the integration of visual–perceptual, cognitive, and motor skills to rapidly process constantly changing environmental stimuli and to make safe, appropriate decisions on the road (Vrkljan et al., 2011; Yale et al., 2003). After neurological injury, deficits in these key areas may severely affect the ability to drive (George & Crotty, 2010). More specifically, the ability to visually scan the environment for information is critical for safety, given that driving is a highly visual task with many functional visual demands (Owsley & McGwin, 2010). Accurate tools to assess visual scanning abilities are necessary to identify safety risks and the potential need for interventions for people who intend to return to driving after injury.
To assess fitness to drive, the gold standard continues to be an on-road assessment (Gibbons et al., 2017; Korner-Bitensky et al., 2005); however, these assessments can be time consuming and costly, and the costs are often incurred by the client (Vrkljan et al., 2011; Wolfe & Lehockey, 2016). Moreover, clients may be assessed at a suboptimal period in their recovery (Unsworth et al., 2019). As a result, the predictive validity of commonly used clinical assessments (e.g., the Montreal Cognitive Assessment [MoCA; Nasreddine et al., 2005], the Motor-Free Visual Perception Test–3 [MVPT–3; Colarusso & Hammill, 2003], the Trail Making Test Parts A [Trails A] and B [Trails B; Reitan, 1958], the Useful Field of View Test [UFOV; Wood & Owsley, 2014]) have been studied extensively to facilitate screening and in-clinic assessment for better prediction of on-road performance. These assessments focus on several areas pertinent to driving, including cognitive impairment, speedy and fluid cognition, visual processing speed, and divided attention (Aslaksen et al., 2013; Classen et al., 2013; Gibbons et al., 2017; Kwok et al., 2015; Oswanski et al., 2007), but no articles on visual scanning and driver assessment were identified during a literature search.
The ScanCourse, a component of the Brain Injury Visual Assessment Battery for Adults (biVABA; Warren, 2006), is a functional assessment that gauges a person’s ability to scan for visual information while ambulating. The dual-task aspect of combining scanning and mobility in the ScanCourse mimics the functional demands of driving and makes it a promising candidate as a screening tool. It has face validity with clients because it is completed in a functional environment, and it does not require language skills, which allows use with a wide range of clients. In addition, the ScanCourse demonstrates good clinical utility, a concept encompassing ease of administration, training and qualifications, time requirements, and ease of results interpretation (Smart, 2006). Recently, Lund et al. (2020) reported that the ScanCourse is highly reliable among clients with neurological impairments (interrater reliability = .998, 2-wk test–retest reliability = .912) and that it correlates as anticipated with the Trails A and B. However, the predictive validity of the ScanCourse for driving performance is not known. Therefore, in this study we had four objectives:
Determine the overall predictive validity of the ScanCourse for on-road driving performance
Compare area under the curve (AUC) values between individual participating sites
Establish optimal cutoff scores for clinical use
Compare ScanCourse administration and physical setup between participating sites.
Method
Study Design
A retrospective cohort study design was used to review client driving assessment charts at four driver rehabilitation programs in Canada. An introductory letter outlining the details of the study was sent to a professional network of occupational therapists in Canada who performed driver rehabilitation as part of their services. Occupational therapists interested in taking part were asked to represent their respective programs as lead investigators for their site. These site lead investigators were responsible for the data reviewed at their own site. Approval was granted by the Clinical Research Ethics Board at the University of British Columbia, and operational approval was obtained from each participating site.
Inclusion and Exclusion Criteria
Charts at each participating site were included if they contained results from the ScanCourse and on-road driving assessments dated between September 1, 2008, and August 30, 2018, and contained a record of a neurological condition, vision condition, or both.
Measures
Data on the following demographic and clinical variables were collected: client medical diagnosis, age, sex, date of the ScanCourse trial(s), score(s) of the ScanCourse trial(s), date of the on-road assessment(s), and outcome (pass, fail, or referral to driver rehabilitation) of the on-road assessment(s).
The ScanCourse is administered in a hallway. Twenty numbered cards are placed on both sides of the hallway at various heights between floor level and eye level. The client is instructed to read the card numbers aloud as they walk or propel down the hallway. The clinician records the number of cards accurately identified by the client. One or two trials of the ScanCourse may be performed (Warren, 2006). Although the biVABA manual does not contain specific details with regard to practical administration instructions and procedures for the ScanCourse, Lund et al. (2017) developed a manual with guidelines that provide more standardized test instructions and card setup details (e.g., font style, font size, card dimensions, and card height placement) for use in clinical practice.
The on-road driving assessment is conducted by an occupational therapist who specializes in driver rehabilitation in collaboration with a qualified driving examiner (Classen et al., 2017; Korner-Bitensky et al., 2006). The on-road assessment appraises driving skills related to physical, cognitive, and visual–perceptual abilities (Canadian Association of Occupational Therapists, 2016). One of three recommendations is provided at the completion of the on-road driving assessment: (1) the client is safe to drive (pass), (2) the client is unsafe to drive (fail), or (3) the client is unsafe to drive at the moment but has the potential to benefit from additional rehabilitation (rehabilitation referral; Korner-Bitensky et al., 2007).
The ScanCourse administration survey was sent to each participating site’s lead investigator and contained questions related to hallway dimensions, numbered card size, content of administration instructions, number of trials conducted, and amount of cuing provided by the examiner.
Sample Size Calculation
MedCalc software (Version 19.0.7; MedCalc Software Ltd., Ostend, Belgium) was used to calculate the sample size. To detect an AUC of .70 with a null hypothesis of .60 (p = .05 and power = .80), a sample size of 263 was estimated. An AUC of .70 was chosen because values between .70 and .80 are considered to represent acceptable discriminatory power for diagnostic tools (Weaver et al., 2014).
Data Analysis
All statistical analyses were completed using IBM SPSS Statistics (Version 24.0; IBM Corp., Armonk, NY) and MedCalc. Descriptive statistics were used to highlight demographic information (age, sex, diagnoses) and total pass–fail rates. For Objective 1, we determined the predictive validity by generating a receiver operating characteristic (ROC) curve and calculating the AUC. An ROC curve is a tool used to quantify the probability of predicting dichotomous outcomes across various cutoff thresholds (Hajian-Tilaki, 2013). ROC curves were generated for each individual site, as well as for all sites as a group. Different combinations of the collected data (e.g., ScanCourse Trial 1 scores; Trial 2 scores; the average of the Trial 1 and 2 scores; the initial on-road dichotomous outcome [pass–fail], when a rehabilitation recommendation is also considered a fail at the time of assessment; and the final on-road outcome [pass–fail on subsequent on-road assessment(s) after driver rehabilitation training]) were used to generate the ROC curves. The AUC represents an overall value for predictive accuracy of outcomes (Weaver et al., 2014).
For Objective 2, we used a nonparametric approach outlined by DeLong et al. (1988) to analyze the AUC values between site-specific ROC curves for statistical differences. A logistic regression was also computed to control for different independent variables (e.g., site, age, sex, diagnosis, ScanCourse score), to determine which, if any, were independent predictors of on-road performance.
For Objective 3, we determined optimal cutoff scores for the ScanCourse by selecting the cutoff score with a higher sensitivity and calculating the Youden index (sensitivity + specificity ‒ 1). The Youden index represents the maximum distance from the chance line between cutoff scores on the ROC curve (Hajian-Tilaki, 2013). In relation to our study, sensitivity refers to the probability that the ScanCourse accurately predicts that a driver will fail the on-road assessment. Specificity refers to the probability that the ScanCourse accurately predicts whether a driver will pass the on-road assessment (Weaver et al., 2014). Choosing a cutoff score with a higher sensitivity at the expense of lower specificity is consistent in the driving literature because this minimizes issues associated with misclassifying at-risk clients as being fit to drive (Devos et al., 2011; George & Crotty, 2010; Kwok et al., 2015).
For Objective 4, we investigated potential differences in ScanCourse administration and physical setup between participating sites by comparing clinicians’ responses and using descriptive statistics.
Results
Sixteen individuals representing driver rehabilitation programs across Canada were initially contacted, and we received responses from 7 programs expressing interest in participation. Of these, 4 programs participated in our study. A total of 811 charts were reviewed across all sites, and 325 were included in the final sample for analysis. A total of 486 charts were excluded because of ineligible diagnoses (n = 406), dates of assessments that fell outside of the approved study time frame (n = 21), and missing ScanCourse scores or on-road outcomes (n = 59). The sample sizes for each respective site were as follows: Site 1, n = 79; Site 2, n = 28; Site 3, n = 104; and Site 4, n = 114. Charts predominantly belonged to clients who were male and had a diagnosis of cerebrovascular accident. A little more than 40% of clients had passed their on-road evaluation (Table 1). Chart medical diagnoses were categorized into one of four categories: traumatic brain injury, cerebrovascular accident, vision condition, or general neurology. In cases in which multiple medical diagnoses were listed, the primary medical diagnosis was recorded.
Client Characteristics (N = 325)
We were missing data for 3 charts.
Common examples of vision conditions included retinitis pigmentosa, glaucoma, and cataracts.
Common examples of general neurology conditions included multiple sclerosis, brain tumor, and dementia.
The overall AUC, pertaining to average ScanCourse score and final on-road outcome, for the entire sample was .702, with a 95% confidence interval (CI) of .645 to .754 and a standard error (SE) of .035. Site 1 had an AUC of .622, 95% CI [.497, .736], SE = .114; Site 2 had an AUC of .765, 95% CI [.538, .917], SE = .152; Site 3 had an AUC of .744, 95% CI [.638, .832], SE = .085; and Site 4 had an AUC of .600, 95% CI [.503, .692], SE = .054. No statistically significant differences were found when we compared AUC values between individual sites at a bivariate level; however, the logistic regression showed that site and average ScanCourse scores were statistically significant, independent predictors of on-road outcome (Table 2). Table 3 shows that although a cutoff score of 17/20 yielded the highest Youden index, the optimal cutoff score was 18/20 because a higher sensitivity (76.7%) was prioritized.
Logistic Regression of Independent Variables With On-Road Outcome
Note. Boldface indicates significance at p < .05. CI = confidence interval; Ref. = reference variable.
ScanCourse Clinical Cutoff Scores
Compiled clinician responses from the ScanCourse administration survey revealed that the assessment instructions and amount of cuing given to clients were generally consistent across sites. A wide range of values was recorded for many of the physical setup dimensions (Table 4). Several values fell outside of 1 standard deviation from the mean; however, no outliers were found for these measurements.
Hallway and Card Measurements From the ScanCourse Administration Survey
Discussion
The primary objective of this study was to determine the predictive validity of the ScanCourse for on-road driving performance among people with neurological or vision conditions. The overall AUC value of .702 is comparable to AUC values of other commonly used driver screening assessments (i.e., MoCA, .720–.889; MVPT, .758; MVPT–3, .922; Trails A, .810–.914; Trails B, .650–.979; UFOV, .775; and UFOV Risk Index, .860). Note that values on the lower end of the AUC range for each measure correspond to those in studies of samples who primarily had neurological conditions, whereas values on the higher end correspond to those in studies of older adults or generalized samples of clients with a mixture of neurological and nonneurological conditions (Aslaksen et al., 2013; Classen et al., 2013; Devos et al., 2011; George & Crotty, 2010; Gibbons et al., 2017; Kwok et al., 2015; Oswanski et al., 2007). The AUC value of .702 for the ScanCourse is considered to fall within the acceptable category (.7–.8) for discriminatory ability. Meanwhile, values between .8 and .9 and above .9 reflect excellent and outstanding discriminatory ability, respectively (Weaver et al., 2014). In observing these trends, we speculate that the prediction of on-road performance of clients with neurological conditions may be more challenging, given the complexity of their physical or cognitive functional impairment, which may reflect the lower AUC values seen.
When we compared AUC values between individual sites for statistical differences at a bivariate level, we found none; however, when we controlled for all other factors, site was found to be a statistically significant, independent predictor of on-road outcomes. Whether the significance of site is due to differences in client group composition at each site, differences in ScanCourse setup and hallway dimensions, or a combination of both is unclear. The effect of client group composition on driving performance, also known as spectrum bias, is commonly reported in driver rehabilitation–related studies (Classen et al., 2013; Gibbons et al., 2017). Spectrum bias refers to potential variations in test performance on a diagnostic test that are attributable to differences in client characteristics across clinical settings (Willis, 2008). On the one hand, differences in client group impairment severity and the resulting impact on sensitivity, specificity, and accuracy have been documented (Whiting et al., 2013). On the other hand, inconsistency in physical setup may be another contributing factor as to why site was an important predictor of on-road performance in our study. A wide range of values were observed for many of the dimension measurements noted earlier; variations in the physical setup may have affected clients’ performance on the ScanCourse. Statistical differences between these measurements could not be computed because of a significant imbalance in sample sizes between the four sites.
In terms of the optimal cutoff score of 18/20 for the ScanCourse, its corresponding sensitivity value of 76.7% is comparable to those of other assessments in the driving literature that have been used to examine neurological populations only (e.g., MoCA, 84.5%; Trails A, 85.0%; Trails B, 80.0%; UFOV, 85.7%). However, a cutoff of 17/20 yielded the highest Youden index, which represents a fair and unbiased way of determining an optimal cutoff score, giving equal consideration to both sensitivity and specificity (Weaver et al., 2014). Selecting a cutoff score of 18/20, with its higher sensitivity, will cause more at-risk people who score below 18/20 to be classified as unfit to drive, in exchange for a lower specificity and an increase in the risk of misclassifying potentially fit people who score below 18/20 as unfit to drive. Finding a balance in weighing the relative costs between these two test properties remains a challenge to improving overall diagnostic accuracy, and in clinical practice the consequences of placing a lower emphasis on specificity in favor of high sensitivity is an important consideration for client independence (Weaver et al., 2014).
Attempts at combining several in-clinic assessments or evaluating assessment batteries as a whole for driver fitness screening purposes have yielded prediction accuracy rates of 78.3% and 83.1%, respectively, for on-road outcomes (Gibbons et al., 2017; Unsworth et al., 2019). Future research should continue to explore the optimal combination of driver screening assessments to achieve the greatest accuracy in predicting on-road performance while giving ample consideration to clinical utility, for application in practice. In addition, in having higher sensitivity prioritized in many driver-screening assessments, it may be worthwhile to consider different assessments with higher complementary specificity to accurately classify more potentially fit drivers as well. Investigating all driver screening assessments of interest using the same sample in the same setting may minimize the potential effects that setting variation and client characteristics could have on diagnostic test performance.
Limitations of this study include the omission of low ScanCourse scores without on-road evaluation results, the effects of spectrum bias, and the potential impact that variations in physical site setup may have had on ScanCourse assessment scores. First, drivers who demonstrated poor in-clinic performance were deemed unfit to proceed to the on-road evaluation, and so no outcomes were recorded. Inclusion of these lower ScanCourse scores and outcomes may have otherwise increased the overall AUC value we found. Second, with a unique sample composition that also included vision conditions specific to our interest area of visual scanning, test performance—and, consequently, the AUC value—may have also been affected. Moreover, given the retrospective design of this study, changes to the ScanCourse after adoption of a standardized physical setup at some sites may have altered ScanCourse scores. It was also not possible to readminister the ScanCourse with the same group at different sites to observe any score variations.
Implications for Occupational Therapy Practice
The results of this study have the following implications for occupational therapy practice:
The ScanCourse has acceptable discriminatory ability for on-road performance and can be used in conjunction with other evidence-based clinical assessments.
The ScanCourse has an optimal cutoff score of 18/20 with a sensitivity of 76.7% and a specificity of 47.1%.
Assessment cutoff scores with a high sensitivity should be prioritized to ensure public safety in more effectively screening at-risk drivers as unfit to drive.
Conclusion
Because driving is a highly visual task (Owsley & McGwin, 2010), this study contributes to the limited body of knowledge of the evaluation of visual scanning abilities in the driving literature. With the predictive validity and measurement properties of the ScanCourse now established, there is good evidence to support its use as a clinical tool to capture essential performance components of driving as part of screening and assessment procedures in practice. Measuring the predictive ability of the ScanCourse assessment in relation to on-road driving performance provides occupational therapists with an acceptable clinical tool that can be used when screening at-risk drivers for fitness to drive.
Footnotes
Acknowledgments
We sincerely thank Sue Reil and Matthew Hamade for their contributions and assistance during the data collection process. W. Ben Mortenson’s work was supported by a New Investigator Award from the Canadian Institutes of Health Research.