Abstract
In this study, we examined confidence and face validity or client acceptability of tests used in a Veterans Affairs Medical Center driving clinic. The clinic used evidence-based off-road tests and adopted the Washington University Road Test (WURT) as a performance-based on-road examination. Forty-three clients consented to participate in the study; most were male with an average age of 78.2 years (standard deviation = 12.6). In general, a trend existed toward higher client acceptability of tests adopted from the Neuropsychological Assessment Battery (Stern & White, 2003) and the WURT than of other off-road measures. Confidence decreased after administration of the psychometric test battery, yet it increased after the on-road evaluations despite a 47% failure rate in the sample. Additional study is needed on test acceptability because it may have the potential to increase understanding, compliance, or both with driving recommendations. Additional research is also needed to examine client confidence levels and their potential impact on performance during the driving evaluation process.
Keywords
Of the 38 million people age ≥65 living in the United States, 30 million are licensed drivers (National Highway Traffic Safety Administration, 2008). The automobile is the most frequently used mode of transportation, accounting for 90% of the trips made outside of the home (Collia, Sharp, & Geisbrecht, 2003). Older adults are likely to experience chronic illness that may affect their driving ability. Medical conditions known to affect driving ability include diseases affecting vision (e.g., cataracts, diabetic retinopathy, macular degeneration, glaucoma), cardiovascular disease, neurologic disease (e.g., dementia, multiple sclerosis, Parkinson’s disease), psychiatric disease, metabolic disease, and musculoskeletal disease, along with medication side effects (Wang, Kosinski, Schwartzberg, & Shanklin, 2003). Increased crash rates per miles driven have been noted for older adults (National Highway Traffic Safety Administration, 2000), and this finding has been attributed at least in part to medical illnesses (Meuser, Carr, & Ulfarrson, 2009).
Older adults are often referred for fitness-to-drive evaluation by physicians, other health professionals, or family members who are questioning their ability to drive. Driving rehabilitation specialists, many of whom are occupational therapists, assess these adults and make recommendations to the referring physician. A driving evaluation generally consists of a clinical interview, motor skills assessment, vision examination, psychometric tests, and performance-based road test. Occupational therapy practice guidelines for these evaluations have recently been published (Stav, Hunt, & Arbesman, 2006). A review of practices across the United States and Canada, however, has indicated that although the same domains are generally assessed, specific assessments vary significantly across programs (Korner-Bitensky, Bitensky, Sofer, Man-Son-Hing, & Gélinas, 2006). Although numerous psychometric tests of various functional abilities are believed to be key to operating a motor vehicle, their reliability and validity have not been fully established (Adler, Rottunda, Bauer, & Kuskowski, 2005).
Acceptance of Driving Evaluations
Few measures have been developed specifically for use during driving evaluations. Typically, driving clinics use standardized psychometric tests to address a wide range of key functional abilities. Although many of these assessments are being used for clinical and research purposes, they have not been adequately studied to determine their face validity or acceptability to the client.
Face validity can be defined as the characteristic that an instrument appears to test what it is supposed to and that it delivers a plausible method for doing so (Portney & Watkins, 2000). Face validity serves an important purpose because an instrument lacking face validity may not be acceptable to those who administer it, those who are tested by it, or those who use the results (Portney & Watkins, 2000). For those reasons, it is important that the most accurate, face valid, and acceptable driving evaluations be administered to older adults. If clients must bear the expense and stress of a driving evaluation, we must ensure that they understand and accept why specific tests have been adopted and that they believe that the test is connected to real-world driving. If people being tested do not understand how the test determines their ability to drive or find the test irrelevant to the driving task, they may be less likely to accept the occupational therapist’s or physician’s recommendation regarding their driving retirement.
Confidence During the Assessment Process
Confidence is the belief in oneself and one’s abilities. Little is known about how a person’s confidence affects his or her driving ability and especially how it affects driving during various testing procedures. Previous findings have suggested that confidence and self-rating of driving ability are closely related to each other and to driving frequency (Marottoli & Richardson, 1998). Confidence, however, was not related to demographic features, history of poor driving, or driving performance, indicating that even people who rate themselves as being confident and good drivers may still perform poorly on a driving evaluation. Additional studies have also found that older drivers generally have high confidence in their driving abilities (Hunt, Morris, Edwards, & Wilson, 1993; Marottoli & Richardson, 1998). Many people who are tested and fail, however, consider themselves safe drivers or at least rate themselves as good as or better than their peers before testing. Peduzzi et al. (2007) examined the effects of physical ability and self-confidence regarding driving ability. An enhanced self-confidence about speed of movement or physical ability was found to decrease anxiety while driving, whereas perceived sense of physical slowness or frailty was found to increase anxiety.
In this study, we had the following two objectives: (1) to examine the face validity or test acceptability of evidence-based assessment measures that were adopted in a driving clinic sample of older adults referred for fitness-to-drive evaluations and (2) to periodically measure client confidence levels during the entire driving evaluation process. We hypothesized that those tests that appeared to have more visual or actual driving content (e.g., driving scenes, map reading test, road test) would have higher rates of acceptability to clients. We further hypothesized that overall client confidence levels would decrease during the course of testing because of anticipated difficulty or failure of at least part of this sample on psychometric tests, the performance-based road test, or both.
Method
Study Location
The Jefferson Barracks Division of the St. Louis Veterans Affairs Medical Center (VAMC) provides psychiatric treatment, spinal cord injury treatment, a nursing home care unit, geriatric health care, rehabilitation services, and a rehabilitation domiciliary program for homeless veterans. A driving evaluation clinic located at Jefferson Barracks has served the VA system in the area for 30 yr. The driving program receives referrals from physicians within the VA system and evaluates approximately 150 clients per year. Fitness-to-drive evaluation referrals are usually made because of cognitive impairment or dementia, spinal cord injury, amputation, multiple sclerosis, or stroke.
Client Recruitment
Any client referred to the clinic between October 2007 and January 2009 was sent information on the research project by mail, including a copy of the informed consent. On arrival for their appointment, clients were asked whether they had received the research documents and whether they would like to participate in the study. Those who elected to participate were asked to sign the informed consent before the driving evaluation was initiated. Clients were excluded if they were <65 or did not hold a valid driver’s license or if the occupational therapist deemed them incompetent to understand the consent process. This study was approved by the Human Studies Committees at the Washington University in St. Louis School of Medicine and the St. Louis VAMC.
Driving Evaluation
Clinical Battery.
The evaluation begins with the client and family interview, which is followed by visual, cognitive, and motor functional tests; traffic sign recognition; and knowledge questions and ends with a performance-based road test. The evaluation ends with a client and family conference, at which time the occupational therapist conveys the final recommendation regarding driving ability. The entire evaluation takes approximately 2 hr. Evaluations in the VA system require a referral from a physician, and the results of the evaluation are sent back to the referring physician. Figure 1 shows the clinical evaluation process used in this study.
Clinical evaluation process.
Interviews.
All interviews and evaluations were conducted by either Patricia M. Niewoehner, an occupational therapist who is a certified driver rehabilitation specialist, or Jami L. Dalchow, an occupational therapy student intern, except for the Washington University Road Test (WURT), which only Niewoehner administered. An initial interview was performed with the client and family member to gather background information, driving history, current driving habits, and driving concerns.
Off-Road and Psychometric Tests.
Instruments or tests that are administered as part of routine care in the VAMC driving clinic include range of motion, strength, and sensation screening; the Rapid Pace Walk test (Marottoli, Cooney, Wagner, Doucette, & Tinetti, 1994); the Titmus 2 Screener, using the Drivers Testing and Education slides to evaluate visual acuity, traffic sign and traffic color recognition, depth perception, and visual fields (Titmus 2, 1987); traffic safety questions; the Clock Drawing Test (Freund, Gravenstein, Ferris, Burke, & Shaheen, 2005); Trail Making Test Parts A and B (Reitan, 1986); the Short Blessed Test (Katzman et al., 1983); and the Driving Scene, Map Reading, and Mazes tests from the Neuropsychological Assessment Battery (Stern & White, 2003).
Performance-Based Road Test.
The Jefferson Barracks Division driving clinic has adopted the WURT (Hunt et al., 1997) and created a course similar in structure and content with input from the WURT’s original creator, who provided on-site consultation (L. A. Hunt, personal communication, December 13, 2007). The road test takes 35 min and runs 15 mi on a predetermined route in a suburban residential area through various levels of traffic. It includes seven left turns, two of which are unprotected left-hand turns. The client drives the standard VAMC car with dual brakes, and the occupational therapist sits in the front seat observing the client’s driving ability.
The WURT course begins by allowing the client to become familiar with the car’s controls, at which time preignition skills are assessed (e.g., client’s ability to locate ignition, gas, and brake controls; signals; and seat belt and to adjust the seat and mirrors). The client is then asked to drive the VAMC grounds to become familiar with the vehicle. After the client becomes comfortable with the car, he or she is asked to drive off the grounds into low-density traffic and is progressed to higher levels of traffic density, eventually ending on a section of highway. The client is instructed to drive as he or she normally would and to follow the rules of the road. During the WURT, the occupational therapist assesses the client’s ability to maintain speed, obey traffic signs, signal, turn, yield right of way, change lanes, and react to other drivers. On completion of the WURT, the therapist gives the client a global rating of safe, marginal, or unsafe (Hunt et al., 1997). The client is typically not given feedback on his or her driving performance during the road test unless a safety issue arises.
Acceptability and Face Validity Measures.
No standardized test or measure of client acceptability exists that we were able to adopt for driving clinic settings. Therefore, we created our own measure of acceptability. After each psychometric test, clients were asked two questions regarding the acceptance of that particular test: “Do you believe this test can predict a person’s driving abilities?” and “Do you think this test accurately identifies a person’s driving strengths and weaknesses?” Clients rated each question on a 5-point Likert scale ranging from 1 (not at all predictive or accurate) to 5 (very predictive or accurate). The point scores were totaled for each test, giving an acceptability score range of 2–10.
Confidence Levels.
Before the client interview, clients were asked the following question regarding confidence in their driving abilities: “How confident are you in your driving abilities based on what you have experienced so far today?” Clients rated their confidence on a 5-point Likert scale ranging from 1 (not at all confident) to 5 (very confident). The same question was asked again after administration of the entire psychometric test battery and again after the WURT.
Statistical Analysis
We performed all analyses using SPSS (SPSS, Inc., Chicago). Descriptive statistics were used to report the demographics and clinical characteristics of all clients, including age, race, gender, highest level of education, reason for referral, medical conditions, and total acceptance scores. We used an independent-samples t test to describe the difference in test performance for those who passed and those who failed the driving evaluation. We used both an independent-samples t test and χ2 in the sensitivity analysis comparing clients and nonparticipants. We analyzed differences in confidence overall and at all three times between those who passed and those who failed by means of an independent-samples t test.
Results
Clients
We recruited 43 of 264 referrals or potential clients (including clients who showed for their appointment and those who did not) from the St. Louis VAMC Driving Evaluation Clinic. The original sample had 4 additional clients who did not receive the on-road evaluation because of severe impairment, weather conditions, or vehicle fit and were excluded from these analyses. No differences existed between those who participated and those who did not with respect to age, gender, or education. Descriptive statistics for client demographics are presented in Table 1. Clients had a mean age of 78.2 years (standard deviation = 12.6) and an age range of 65–92. Most clients were male (93%), 90% were White, and they had a mean education level of 12.6 yr (standard deviation = 2.8) with a range of 8–20 yr. Of the clients, 69.8% were referred to the clinic for cognitive disorders (e.g., dementia, cognitive impairment, traumatic brain injury), 20.9% for motor disorders (e.g., spinal cord injury, amputation, multiple sclerosis), and 4.7% for vision (macular degeneration, glaucoma, cataracts). It is interesting that 70% acknowledged that their physician or family were concerned about their driving, yet only 12% stated they had problems with their driving. Almost 30% had had a previous accident, ticket, or near miss within the past 2 yr. Nearly 40% required assistance in other instrumental activities of daily living (IADLs) such as finances, grocery shopping, or cooking. After the performance-based on-road driving evaluation, 53% (n = 23) were recommended to continue driving, and 47% (n = 20) were recommended not to drive.
Client Demographics
Note. N = 43. Four clients did not receive the on-road evaluations because of severe impairment, weather conditions, or vehicle fit and were excluded from the analyses.
Face Validity and Acceptability
Figure 2 shows the results of the sum of the acceptance scales for each evaluation used. The two 5-point scales were summed together to obtain a total score on a 10-point scale for this analysis. The WURT (avg. = 8.7), Traffic Safety Questions (avg. = 8.5), Driving Scenes Test (avg. = 8.5), Eye Exam (avg. = 8.4), and Map Reading Test (avg. = 7.9) were shown to be the most acceptable to our clients or to have the highest level of face validity of all the off-road tests. According to the t-test results, respondents who passed the driving test had higher acceptance scores on the WURT than those who failed the driving test (9.4 vs. 7.8, respectively, p = .017).
Client test acceptance.
Confidence
Figure 3 shows how confidence was affected during the evaluation process. For those who passed the WURT, confidence scores before all testing (Time 1) was 4.9, decreased to 4.6 after the psychometric tests (Time 2), and increased to 4.9 after the on-road evaluation (Time 3). The clients who failed the WURT had an average confidence score of 4.5 at Time 1, which decreased to 4.2 at Time 2 and increased to 4.5 at Time 3. As Figure 3 illustrates, those who received a recommendation to continue driving had a greater total confidence score than those who were recommended not to drive (14.4 and 13.1, respectively, p = .001). Results were significant at p < .05.
Confidence during evaluation.
Discussion
Our sample of older drivers was predominantely male and had a high school education. The demographic characteristics of this group were not surprising, and on the basis of comparisons with nonparticipants, we are relatively assured that our sample is representative of the types of clients commonly evaluated in VAMC driving clinic settings (Szlyk, Myers, Zhang, Wetzel, & Shapiro, 2002). Almost 70% of the evaluations were for cognitive impairment, but other reasons for referral were represented, including spinal cord injury, amputees, and older adults with visual impairment.
The tests that we have adopted in our clinical setting are not atypical of those that are used in driving evaluation settings (Korner-Bitensky et al., 2006), possibly with the exception of measures from the Neuropsychological Assessment Battery (Stern & White, 2003). Occupational therapists have myriad tests to choose from that may be helpful in identifying deficits in specific functional domains during driving evaluations. Tests that tap into working memory, executive function, visual search, and visuospatial abilities were more likely to indicate impairment in the cohort who failed our driving evaluation. We selected the psychometric tests used in our clinic on the basis of the evidence in the literature (Brown et al., 2005), but we also selected tests that we hoped would have a high degree of acceptability, face validity, or both for our clients.
In this study, we described the face validity or acceptance of common off-road and on-road tests used in driving evaluation clinics and the confidence levels of older adults across testing during the driving evaluation. The performance-based road test, visual acuity, and traffic safety questions had, not surprisingly, the highest levels of client acceptance. Of interest is that tests from the Neuropsychological Assessment Battery (e.g., driving scenes and map reading) were rated higher than some of the traditional tests often used in clinical settings such as the Trail Making Part A, Rapid Pace Walk, and the Clock Drawing Task. Recent work has noted that maze tests may be helpful in identifying unsafe drivers (Ott et al., 2003; Snellgrove, 2005). However, maze tests were rated fairly low on the basis of our acceptability rating measure.
Client acceptability should be an important consideration in the adoption of tests in driving evaluation clinics for several reasons. Clients may be more motivated to perform tests that they believe are actually related to the driving task. Thus, scores on these types of tests may be more reflective of actual performance. Administering tests such as those from the Neuropsychological Assessment Battery (e.g., Driving Scenes, Map Reading) with colored stimuli may also maintain interest. Finally, clients may be more willing to accept “failed” test results if the test is perceived to directly relate to driving abilities. These hypotheses should be further tested and explored in clinical settings.
Confidence levels appeared to drop for our entire sample after psychometric test performance. Those older adults who failed the overall driving evaluation also showed a significant lowering of confidence after psychometric testing in comparison with those who were recommended to continue active driving. We recognize that the drop (0.4 average points on our confidence scale) in the group that failed our driving evaluation is somewhat small and possibly of questionable clinical significance; however, the trend is interesting for the older adults who passed our driving evaluation as well as those who failed. This trend raises the possibility that the confidence of a subset of older drivers is lowered after psychometric testing. Lowered confidence after psychometric testing has the potential to negatively affect an older adult’s ability to pass a road test. Although we did not design this study to answer this question, we believe more research is needed in this area of confidence—how confidence changes across the course of the driving evaluation and whether it can have an adverse impact on performance. One way to empirically study this issue may be to perform road testing before psychometric testing once clients pass a safe brief screen of functional ability (e.g., meeting visual acuity standards for the state) or to design a study in which the sequence of on-road–off-road evaluation is randomized.
Our results show that older adults who ultimately fail the evaluation experience a decrease in confidence after off-road testing but an increase after the road test, indicating that older adults in our sample may not have recognized that they were unsafe drivers. This fact may not be surprising given the number of referrals of clients with cognitive impairment, dementia, or both. Alternatively, psychometric test scores associated with a drop in confidence could also affect clients’ final rating on the road test as the result of heightened anxiety or negatively bias the examiner toward rating clients as failing during road testing. To date, few studies have examined confidence in older adults during driving evaluations. The findings from this study are consistent with another finding that high confidence in driving ability does not relate to actual driving performance (Marottoli & Richardson, 1998). Unlike our study, however, Marottoli and Richardson (1998) did not examine how confidence levels changed during the evaluation process.
Limitations of the Study
This study has several limitations. First, our sample consisted mainly of male veteran clients. Although this sample may not be generalizable to all driving evaluation clinics, it appears to be typical of a clinic population at VAMCs. Additional studies are needed to determine whether women and nonveterans have similar results with regard to acceptance of tests and changes in confidence levels. Second, our sample size was fairly small. A larger sample may allow for a determination of the consistency of these findings. Limitations may have existed in asking for confidence levels at only three points in time in a sample with a high prevalence of cognitive impairment or dementia. Those clients with significant short-term memory loss may possibly have had difficulty recalling their performance during the previous section of testing. Finally, no standardized measures of acceptability or confidence in regards to driving exist. Further study of the clinimetric properties of these measures for the questions we created (e.g., validity and reliability) should be considered.
Implications
The fact that older adults experience a decrease in confidence after off-road testing may have an impact on the on-road test performance. As a result, professionals need to be aware of the potential for decreased confidence and make efforts to ease fears and make the evaluation more comfortable for older adults. Of additional concern was the finding of increased confidence levels after the on-road testing for clients who ultimately failed. Older adults may benefit from more immediate feedback to increase the likelihood that they understand their deficiencies. All older adults in this study, regardless of whether they passed or failed, were confident in their driving abilities, most likely as a result of a lifetime of driving experience and their own comfort level with their driving skill set. In this study, simply asking older adults about their driving abilities was not an accurate indicator of actual driving performance.
A new issue brought to the driving evaluation arena in this study is that of face validity or acceptance of driving evaluations. Professionals who work in driving evaluation clinics with older adults should find the results of this study useful with regard to test selection. We found several evaluations to be more acceptable, presumably because they appeared to have a link to actual driving performance. The process of a driving evaluation and subsequent recommendation to stop driving are difficult for both older adults and their families. The use of more acceptable tests studying driving in clinical settings may allow for a better understanding and compliance with the driving evaluation’s final recommendation. Occupational therapists should consider adopting tests that have more face validity or acceptability to their clients. These findings can also be used to encourage test developers to create items or stimuli that appear to have a link to driving, thereby making them more acceptable. More study is needed to determine whether off-road tests that are more acceptable to the client will increase confidence or result in a client being more compliant with driving recommendations.
Footnotes
Acknowledgments
This material is a result of work supported with the resources and use of facilities at the Department of Veterans Affairs Medical Center, Jefferson Barracks Division, St. Louis, MO; principal investigator, Patricia M. Niewoehner; co-principal investigators, David Carr, Thomas Meuser, and Jami L. Dalchow (occupational therapy student intern, Physical Medicine and Rehabilitation, Extended Care and Rehabilitation Services, Department of Veterans Affairs Medical Center, Jefferson Barracks Division, St. Louis, MO). We thank Thomas Meuser and Susan Stark for their knowledge and guidance throughout this project. This research is the result of work completed for requirements of Jami L. Dalchow’s doctor of occupational therapy degree, Program in Occupational Therapy, Washington University, St. Louis, MO.