Abstract
Driving is an important instrumental activity of daily living (IADL) that represents independence and a means of remaining active and connected with the community. The driving safety of older adults with medical conditions can be compromised, but revocation of driving rights may result in loss of independence and quality of life (Chihuri et al., 2016). Dementia has been found to be the most common medical condition among older adults reported to state licensing authorities for driving concerns (Meuser et al., 2009). Both clinicians and state licensing authorities have a role in determining driving competency in older adults. A recent study determined that in-person licensure renewal, an on-road assessment, or both, resulted in a lower prevalence of adults with dementia being hospitalized after a crash (Agimi et al., 2018; American Geriatrics Society, 2016).
Recognizing and understanding traffic signs is a critical component of safe driving. Various medical conditions associated with aging can impede a person’s ability to either perceive or comprehend traffic signs. For example, visual impairments such as decreased acuity and contrast sensitivity cause difficulty in identifying traffic signs (Desapriya et al., 2014; Owens et al., 2007; Wood & Mallon, 2001; Wood et al., 2009). However, studies have found that older adults with intact vision comprehend the majority of common traffic signs except when cognitive impairment is present (Carr et al., 1991; Scialfa et al., 2008). It is not surprising that identification of common traffic signs is worse among people with Alzheimer’s disease (AD) than among healthy older adults (Carr et al., 1998); one study found that one-fourth of people with dementia were not able to appropriately identify a stop sign (Brashear et al., 1998).
Sign comprehension requires cognitive abilities such as executive function, which can decline with age and some medical conditions (Scialfa et al., 2008) and result in the inability to accurately identify traffic signs (Aksan et al., 2013). When loss of general knowledge (i.e., semantic memory failure) or executive function occurs as a consequence of a disease such as AD (Luzzi et al., 2015; Venneri et al., 2016), people may lose the ability to recall a previously familiar sign or problem solve correct responses during a written driving examination. More specifically, understanding signs involves the ability to engage in abstract reasoning to interpret the meaning of a symbol or icon associated with the sign (Scialfa et al., 2008).
When determining fitness to drive, occupational therapy driving rehabilitation specialists (OTDRSs) often use traffic sign identification tests and written tests involving knowledge of rules of the road as part of their comprehensive driving evaluation (CDE). Many driving license renewal offices also routinely use a traffic sign test, written questions, or both as part of license renewal and testing. Although these tests are commonly used in both clinical and license renewal settings, very few studies have established the predictive validity of these tests for driving performance or driving safety. In one study, performance on a traffic sign recognition test successfully identified older drivers who had a recent motor vehicle crash, but the test lacked adequate sensitivity and specificity (MacGregor et al., 2001). In another study of older adults in a license renewal setting, a test of traffic sign knowledge was discontinued during the early phases of the study because of the confusion it caused among participants and its lack of predictive ability (Janke, 2001). The Road Law and Road Craft Test has been studied with clients with a wide range of medical conditions (n = 118) and was determined to have interrater reliability and to be a valid indicator of off-road driving skills. However, determining a cutpoint for those fit to drive versus those unfit to drive proved difficult because of the wide spread in the results (Unsworth et al., 2010, 2011).
Given the paucity of literature establishing traffic sign or written driving tests as reliable and valid predictors of driving outcomes, clinicians and driving licensure authorities are in need of evidence-based measures to assist in fitness-to-drive decisions for drivers with cognitive decline (Jacobs et al., 2017). It is important that clinicians better understand what constructs (i.e., construct validity) are being evaluated in traffic sign and written driving tests, especially if they are part of decisions related to the fitness to drive of older adults with conditions such as dementia. Therefore, the aim of this exploratory study was to determine the construct and predictive validity of performance on the Traffic Sign Naming Test (TSNT) and Written Exam for Driving Decisions (WEDD) and the interrater reliability of the TSNT as measures of fitness to drive among older adults with and without dementia.
Method
This cross-sectional study was funded by the Missouri Department of Transportation, Traffic and Highway Safety Division and conducted at Washington University Medical School (WUSM) in St. Louis in collaboration with the Rehabilitation Institute of St. Louis. The study was approved by the Human Studies Committee at WUSM, and informed consent was obtained from all participants.
Participants
Persons with dementia were recruited through the Memory Diagnostic Center of WUSM and private physicians from February 2008 through August 2013. Cognitively healthy older adults were recruited from the Volunteers for Health database maintained by WUSM and the local community. People with dementia were included in the study if they presented with a medical referral for a driving evaluation because of dementia, had a score of 2 or greater on the Alzheimer Detection 8 (AD8; Galvin et al., 2005), and could be accompanied by an informant to the CDE. Cognitively healthy older adults were included if they were older than age 55 yr and had an informant willing to answer questionnaires, an AD8 score less than 2, and a Short Blessed Test (Katzman et al., 1983) score less than 9. Exclusion criteria for both groups included lack of a valid driver’s license, visual acuity less than state guidelines, non–English speaking, less than 10 yr of driving experience, unstable medical condition, severe language or musculoskeletal impairment that would affect testing, sedating drugs, or having had a CDE within the past year.
A total of 220 people with dementia and 45 cognitively intact older adults were screened. We excluded 90 people from the sample with dementia for the following reasons: not interested or cancelled for unknown reasons (n = 54; 60.0%), licensure or consent concerns (n = 13; 14.4%); medical exclusion (n = 4; 4.4%); CDE in prior year (n = 4; 4.4%); no active license or already stopped driving, in nursing home, or went to the Department of Motor Vehicles (DMV; n = 6; 6.6%); test location (n = 3; 3.3%); scheduling issue (n = 4; 4.4%); and other miscellaneous reasons (n = 2; 2.2%). People were excluded from the group of cognitively intact older adults (n = 11) as a result of cancellation or lack of interest (n = 8; 73%) and medical exclusions (n = 3; 27%).
The final sample consisted of adults with dementia (n = 130) and cognitively intact older adults (n = 34). No significant differences were observed in demographic variables between those with and without dementia (p ≥ .06; Table 1). Scores on cognitive assessments reflected a sample of older adults with and without dementia (p < .001; Table 1).
Demographic Characteristics of and Cognitive Assessments Used With the Sample of Older Adults With and Without Dementia (N = 164)
Note. AD8 = Alzheimer Detection 8; DHI= Driving Health Inventory; SD = standard deviation; UFOV = Useful Field of View.
aAssesses for presence of dementia. bScreen for memory, orientation, and concentration. cScreen for executive function, visual–spatial skills and perception, and semantic memory. dScreen for visual attention. eScreen for visual attention and executive function. fScreen for visual perception. gScreen for speed of visual processing and visual memory. hScreen for executive function.
Procedures
Participants from both groups participated in a CDE that included a clinical assessment and an on-road assessment. A recommendation meeting with the informant and participant with dementia at the conclusion of the assessments provided them with individualized driving recommendations based on the clinical and on-road assessment. The CDE took place at the Rehabilitation Institute of St. Louis and WUSM in St. Louis.
Clinical Assessments
The standardized clinical driving assessments were administered by trained occupational therapists as described fully in previous publications. The clinical assessments were performed first, followed by the on-road assessments. Time for rest or a snack was provided between the clinical and on-road assessments if needed (Barco et al., 2015; Carr et al., 2011).
Cognitive Assessments
Screens for cognitive function included the Short Blessed Test, AD8, Freund Clock Drawing Test (Freund et al., 2005), Trail-Making Tests A and B (Armitage, 1945), Driving Health Inventory Useful Field of View (Subtest 2) and Visual Closure tests (TransAnalytics Health and Safety Services, n.d.), and the Snellgrove Maze Test (Snellgrove, 2005).
Vision Screens
Visual acuity was determined with the Early Treatment Diabetic Retinopathy Study (Arditi & Cagenello, 1993). Contrast sensitivity was assessed with the Pelli–Robson Tests for Contrast Sensitivity (Pelli et al., 1988).
Motor Screens
We tested motor function with the Nine-Hole Peg Test (fine motor coordination; Mathiowetz et al., 1985), Jamar dynamometer (grip strength; Bohannon et al., 2006), Rapid Pace Walk (gross motor speed; Marottoli et al., 1994), and Braking Response Time Monitor (cognitive processing speed and motor reaction time; Delta Integration Inc, 2018). Cervical range of motion measures were performed as part of the clinical assessment protocol.
TSNT and WEDD
Embedded in the clinical assessments were two novel tests, the TSNT and the WEDD. These tests include traffic sign identification and written tests that were largely adapted from tests currently in use by state licensing agencies and from sample questions provided online for people preparing for licensure tests. The TSNT includes 12 traffic signs selected from the Missouri Department of Revenue Drivers Guide (Missouri Department of Revenue, 2017); standardized scoring was developed. The TSNT is intended to assess participants’ working knowledge of various common traffic signs encountered while driving. Standardized instructions and scoring procedures were developed by the occupational therapists and OTDRS on the research team. Definitions of correct responses were developed and included examples of both correct and incorrect responses. Determinations of correct or incorrect responses were made through consensus with OTDRSs in the state of Missouri. The occupational therapists providing the clinical assessments were trained in standardized administration of the TSNT by the coinvestigator (PPB). The TSNT resulted in two scores: a naming score and a function score. All participants were asked to name the sign and then describe what they would do if they observed that sign while driving (i.e., function). There were 12 signs; a score of 12 on naming and function indicated a perfect score for each.
The WEDD is intended to assess a participant’s knowledge of rules of the road and ability to logically apply the rules to basic decision making and reasoning during common encounters that can occur while driving. It consists of 13 multiple-choice questions that require the older adult to read and choose a response. These questions were adapted from online driving practice questions, driving licensure guides, and common occurrences during everyday driving. The questions involved a common driving scenario (e.g., ambulance approaching from behind) and making a decision related to established practices and laws regarding appropriate action. An example question is listed in Appendix A.
Standardized On-Road Assessment
After completion of the clinical assessments, participants were evaluated with an on-road assessment, the Modified Washington University Road Test (mWURT; Barco et al., 2014; Carr et al., 2011). The mWURT is a standardized 13-mi on-road assessment that includes parking lots, low traffic, moderately high traffic, and self-directed driving. The scoring for this study used a qualitative scoring approach: pass, marginal, or fail. For analysis, we decided to combine the marginal category with the pass category (as was done in previous studies) to clearly separate out those with the most severe driving concerns (i.e., those who failed the on-road assessment; Barco et al., 2014; Carr et al., 2011).
Statistical Analysis
The final sample (N = 164) provided 90% power to detect odds ratios of .75 for the TSNT and .74 for the WEDD in logistic regressions with road test failure. The demographic data were analyzed with independent-samples t tests and χ2 tests. Benchmarks of substantial (κ = .61–.80) and moderate (κ = .41–.60) were used to assess κ estimates (Landis & Koch, 1977). Construct validity was determined by correlational analysis of the TSNT and WEDD with other clinical assessment measures. Relationships of road test failure with TSNT and WEDD were evaluated with contingency tables with relative risk and Pearson χ2 tests as well as logistic regression to determine predictive validity. Statistics were calculated with SAS (Version 9.4; SAS Institute, Cary, NC) and IBM SPSS Statistics (Version 23; IBM Corporation, Armonk, NY).
Results
Demographics
Of the total sample, 49% (n = 80) failed the road test. In a logistic regression, an age increment of 10 yr had an odds ratio of 1.645 (95% confidence interval [CI] [1.136, 2.384])—64% greater odds of road test failure (p = .0085). Other demographic variables (gender, race, education) were not related to road test failure (p > .098).
Interrater Reliability of the TSNT
TSNT interrater reliability was determined by an independent and blinded OTDRS who was trained in providing the TSNT and who reviewed the written responses and scored them accordingly. In a randomly selected subset of the sample of 40 participants, the naming portion of the TSNT had higher reliability (κ = .80; 95% CI [.69, .91]) than the function portion (κ = .54; 95% CI [.379, .700]). As a result, we chose to use traffic sign naming as the measure in this study.
Construct Validity of the TSNT and WEDD
To establish construct validity, we performed both convergent and discriminant Pearson correlations to determine the relationship between a participant’s total number of correctly named traffic signs and other clinical measures of vision, motor, and cognition. The TSNT and WEDD were significantly negatively correlated with all cognitive measures (correlations ranged from −.63 to −.029, p < .001; Table 2), providing evidence of the measures’ convergent validity to assess driving in older adults with dementia, a task requiring cognitive resources; these tests also have less relationship to visual and motor measures, indicating discriminant validity. Among the motor tests performed during the clinical assessment, the Nine-Hole Peg Test, Rapid Pace Walk, and Braking Response Time Monitor—timed tasks including a processing speed–cognitive component—were significantly correlated with TSNT and WEDD scores (p < .04; Table 2).
Relationship of TSNT and WEDD to Clinical Assessments in the Sample of Participants With and Without Dementia
Note. DHI = Driving Health Inventory; ETDRS = Early Treatment Diabetic Retinopathy Study; ROM = range of motion; SD = Standard Deviation; TSNT = Traffic Sign Naming Test; UFOV = Useful Field of View; WEDD = Written Exam for Driving Decisions.
Predictive Validity of the TSNT and WEDD
Both the TSNT (area under the receiver operating characteristic curve [AUC] = .748, p < .0001) and the WEDD (AUC = .709, p < .0001) had fair ability to discriminate performance on the standardized on-the-road test (Table 3). When deciding whether an older adult needs further assessment related to driving competency in a DMV license renewal office, three actions are usually available: continue driving (renew license), driving cessation (deny license), or require more testing (e.g., on-road testing, medical consult). Currently, DMVs already limit licenses on the basis of similar tests that typically lack validation. TSNT and WEDD scores are a first step in helping to guide this decision, and more studies are needed to determine their ability to predict driving performance.
Percentage Sensitivity and Specificity for the TSNT and WEDD
Note. TSNT = Traffic Sign Naming Test; WEDD = Written Exam for Driving Decisions.
For example, using the TSNT ranges listed in Table 4, 42% of all participants would not require further testing; however, 13.7% of poor drivers (described as those who fail the on-road test in this study) would be issued a license and 3.6% who passed the road test would be denied a license. The TSNT cutoff of 5 or fewer correct for driving cessation has specificity of 96.4% and sensitivity of 32.5% (Table 3), which is mitigated by a request for more testing. Similarly, using WEDD ranges, 21% of all participants would not require further testing, with 4.0% of poor drivers being issued a license and 1.2% of good drivers being denied a license (Table 4).
Examples of Licensing Decisions Based on Test Scores as Predictors of Passing or Failing a Road Test (N = 164)
Note. mWURT = Modified Washington University Road Test; TSNT = Traffic Sign Naming Test; WEDD = Written Exam for Driving Decisions.
Discussion
Determining fitness to drive of older adults with dementia continues to be an area of need for clinicians and state licensing agencies. Similar to the TSNT and WEDD, traffic sign and written tests (when standardized) make a unique contribution in predicting fitness to drive because they have direct application to driving, and similar tests are often used during license renewal in state licensing offices and during CDEs. When traffic sign tests and written tests are used by licensing agencies, individuals of all ages are denied a license on the basis of whether they achieve a passing score. This study took an empirical look at the types of tests given by licensing agencies (and often as part of CDEs) by standardizing the administration and scoring of similar tests and investigating their reliability and validity in a sample of older drivers with and without dementia.
The interrater reliability of the TSNT, when administered by occupational therapists as part of a standardized assessment in a CDE, is substantial for scoring participants’ responses to naming traffic signs in comparison with moderate for describing what they would do in traffic if they observed this sign (e.g., function). On review, we determined that the scoring of the traffic sign function measure (i.e., “What you would do if you saw this sign in traffic?”) had too much response variability. Increasing reliability of the function measure would involve better defining acceptable and unacceptable responses within the scoring parameters. It should be emphasized that this study has not determined interrater reliability when these tests are administered by examiners in a licensing agency, likely with a less standardized procedure.
Both the TSNT and the WEDD have construct validity with psychometric tests that are typically performed during CDEs. The results of this study showed that both the TSNT and the WEDD have a significant relationship to cognitive screens that are typically used by clinicians during fitness-to-drive evaluations and thus tap into some of the same cognitive domains that have previously been shown to be related to fitness to drive in people with dementia. Thus, licensing agencies are tapping into assessments related to the cognitive domains that are tested in CDEs. The TSNT and WEDD are not intended to replace other fitness-to-drive screening measures; rather, they can be used in combination with other assessments that are typically used.
We found that participants’ total TSNT and WEDD scores were able to predict with fair ability those who would pass an on-road driving test from those who would fail the test. An approach using high and low cutpoints for these tests may be useful in clinical or license settings to determine who requires medical assessment or a CDE to determine fitness to drive. More studies need to be performed with older adults with varied medical conditions to further validate the utility of these tests for use with people with conditions other than dementia.
The evidence supports a combination of tests and screens to guide fitness-to-drive decisions (Barco et al., 2015; Carr et al., 2011; O’Connor et al., 2010). To date, these combination models have not included tests that involve questions directly related to driving, such as those on the TSNT or WEDD, but other generic psychometric tests (Dickerson et al., 2014). Future research could examine whether level of prediction of driving competency would increase when tests such as the TSNT or WEDD are included as part of the prediction model.
Limitations and Future Directions
Limitations of this study include a sample that consisted mainly of older adults with dementia who were referred with a true need for a driving assessment and may not entirely reflect the population of all older adults renewing their driver’s license. Adding the relatively cognitively intact sample to this study likely inflated its predictability. However, inclusion of a wide spectrum of participants with varying levels of cognitive impairment would likely mirror the groups served by clinicians and licensing agencies.
The study’s exclusion criteria eliminated people with language impairment (expressive or comprehension) that would interfere with testing. People with language impairment would need to be considered separately to determine whether the TSNT or WEDD could be modified to be appropriate for them. It is also likely that mild language impairment (apart from other cognitive factors) influenced performance on both the TSNT and the WEDD and contributed variance to participants’ knowledge of traffic signs and driving decisions.
Although the majority of the rules of the road apply across state boundaries, some differences in signs and rules of the road may exist both nationally and internationally.
Implications for Occupational Therapy Practice
The results of this study have the following implications for occupational therapy practice:
State licensing agencies and OTDRSs can become collaborators in the process of determining fitness to drive. Standardized traffic sign tests and written tests (e.g., TSNT and WEDD) have the potential for use by driving licensing agencies as screening tools to determine who may need further referral to a clinician to further evaluate fitness to drive.
OTDRSs can use the TSNT and WEDD to supplement their clinical assessment battery in the CDE. These tests may have more relevance to the client because they more obviously relate to knowledge needed for driving.
The TSNT and WEDD, along with other clinical indicators, can be used as one of the means by which occupational therapy practitioners can determine which clients may need to proceed to a CDE. The TSNT and WEDD have fair predictability and are recommended for use in combination with other performance measures when determining fitness to drive. As with any psychometric test, it is important to not overuse assessments in a clinical setting or as treatment tasks.
Practitioners should consider adding the TSNT and WEDD to other tools and studying them as part of multimodal approaches to determining fitness to drive.
Conclusion
The TSNT and WEDD are reliable and valid measures of fitness to drive (when provided in a standardized manner) in older adults with and without dementia. They should be used by occupational therapists in combination with other clinical and performance-based assessments when clinically determining fitness to drive.
Footnotes
Acknowledgments
We acknowledge and express appreciation to the Missouri Department of Transportation, Traffic and Highway Safety Division for funding to support this study (Grants 17-DL-02-001, 16-DL-02-003, 15-DL-02-003, 14-DL-02-002, 13-DL-02-001, 12-DL-02-001, 11-DL-02-2, 10-PT-02-152, 09-PT-02-162, 08-PT-02-142). This work was presented at the 68th annual meeting of the Gerontological Society of America, Orlando, Florida, November 18–22, 2015.
A school bus ahead of you in your lane is stopped with red lights flashing. You should: