Interrater Reliability of the Record of Driving Errors (RODE)

Research Design

This cross-sectional study was part of a larger study investigating medical fitness to drive funded by the Missouri Department of Transportation (Carr et al., 2011). It was conducted at Washington University Medical School in St. Louis, Missouri, through the Driving Connections Clinic at The Rehabilitation Institute of St. Louis (TRISL) and approved by the Human Studies Committee at Washington University and the Health South Research Committee Corporate Office, a managing partner of TRISL.

Participants

Participants were recruited from January to August 2013 through the Memory Diagnostic Center of Washington University School of Medicine. Inclusion criteria required a diagnosis of dementia and a physician’s referral for a driving assessment, an active driver with current driver’s license, an Assessing Dementia–8 (AD8; Galvin, Roe, Xiong, & Morris, 2006) score of 2 or greater by the informant, a Clinical Dementia Rating (CDR; Hughes, Berg, Danziger, Coben, & Martin, 1982) of 1 or greater, at least 10 yr of driving experience, an informant available to answer questions and attend portions of the driving assessment, visual acuity acceptable in state driving guidelines, and ability to speak English. Exclusion criteria were any major chronic unstable diseases or conditions (e.g., seizures), severe orthopedic or musculoskeletal or neuromuscular impairments that would require adaptive equipment to drive, sensory or language impairments that would interfere with testing, use of sedating drugs, or a driving evaluation in the past 12 mo. All participants went through a comprehensive driving assessment that included the clinical assessment and on-road assessment (Carr et al., 2011).

Performance-Based Road Evaluation

The modified Washington University Road Test (mWURT; Carr et al., 2011) replaced the Washington University Road Test (WURT; Hunt et al., 1997) as a result of route changes caused by road construction. However, the mWURT uses common traffic situations from the WURT. The course currently consists of 14 right-hand turns (5 at stop signs, 3 at traffic lights, and 6 unprotected), 11 left-hand turns (6 at traffic lights, 5 unprotected), 33 traffic lights, and 10 stop signs. The road test begins in a closed, quiet parking lot and then progresses to low-traffic conditions (in a park setting) and on to moderate- to high-traffic conditions (e.g., complex intersections, greater traffic demands, traffic lights), as safety permits. The course is 13 miles long and takes approximately 1 hr to complete.

For this study, the road testing occurred in a midsized driving evaluation car with an occupational therapy certified driver rehabilitation specialist (OT/CDRS) positioned in the front passenger seat. The OT/CDRS provided directions and maintained safety of the vehicle. Two OT/DRSs used the RODE to record driving errors made by the participant. The two OT/DRSs sat in the back seat (one in the back middle and the other behind the passenger seat) with a large board between them to prevent them from viewing each other’s scoring tool. Both OT/DRSs who were scoring driving errors were blinded to results of any clinical testing.

Record of Driving Errors

The RODE, developed and implemented in 2007, is an objective approach to quantify the number and type of driving errors in a standardized performance-based road test. As noted previously, a paucity of published studies on scoring driving errors exists. The RODE incorporates knowledge of common driving errors that were identified by reading the literature and numerous case examples from clinically based driving evaluations. Different from most road test scoring approaches, the RODE allows for a continuous measure of errors throughout the road test, without a maximum limit. When driving, a variety of type and frequency of errors can occur at any given time between Point A and Point B. To capture the true extent of the occurrence of the errors, it is necessary to measure the errors not only at designated points along the route but also consistently throughout the whole driving experience. The RODE incorporates categories of driving errors (i.e., driving situation and driving behavior errors) from previous studies (Wood et al., 2009) and operationally defines specific errors within each category.

The RODE scoring methodology follows a three-tiered approach. First, all driving errors are recorded according to three levels of traffic condition in which they occur (closed parking lot, low traffic, and moderate to high traffic). Second, within each traffic condition, driving errors are recorded according to the specific driving situation that is involved (parking or backing, left turn, right turn, straightaway driving, or lane change). Third, within each driving situation, errors are further described by the type of driving behavior errors that occur (operational, tactical, or information-processing errors; Figure 1).

OPEN IN VIEWER

Figure 1.

Record of Driving Errors scoring.

Building on the Michon model, the RODE defines operational errors as those that involve the direct operation of the car (e.g., use of turn signals and foot pedals), tactical errors as those that require driving tactics or skill in traffic (e.g., maintaining lane position, visual scanning, yielding), and information-processing errors as those related to higher processing skills (e.g., following directions and using memory, making decisions, anticipating maneuvers, knowing traffic signs and rules of the road). Information-processing errors are analogous to but extend beyond the “strategic/planning” driving errors described in other studies (Grace et al., 2005; Withaar, Brouwer, & van Zomeren, 2000). The RODE uniquely acknowledges that multiple driving behavior errors can simultaneously occur and can result in one driving situation error. For example, a driving situation error in making a left turn could be the result of multiple driving behavior errors (such as failure to signal, failure to yield to a vehicle or pedestrian, and turning from an incorrect lane). Finally, the evaluator records any necessary interventions to maintain safety (e.g., verbal cue, wheel intervention, brake intervention) and whether the errors resulted in an occurrence of a dangerous event (e.g., errors that resulted in immediate, imminent danger of a collision or hazard; see Figure 1).

Training on Scoring

We implemented a rigorous training protocol for scoring driving errors. Training occurred over a 3-mo period, and the primary investigator (Barco, an OT/DRS with 15+ years of experience in driving evaluation) served as the trainer. Training included approximately 30 hr of written and verbal practice activities involving scoring methodology (familiarity with the scoring sheets, tallying of errors, and objective definitions of the errors and driving situations) and 15 hr of practice trials of recording errors during road testing to ensure understanding of the methodology, discuss and resolve differences in scoring, and demonstrate proficiency in the administration of the RODE. All training was completed before data collection.

Statistics

Interrater reliability was assessed using ICCs (Shrout & Fleiss, 1979). ICCs reflect the degree of agreement between raters. Higher ICCs indicate better reliability, with guidelines as follows: ≤0 = poor, .01–.20 = slight, .21–.40 = fair, .41–.60 = moderate, .61–.80 = substantial, and .81–1.00 = almost perfect agreement (Landis & Koch, 1977).

Limitations

This study had several limitations. Although the road course was standardized, the traffic occurrences on each route were not; when driving in real traffic conditions, it is not possible to standardize the various traffic-related situations that occur at any given time on the performance-based road course. Moreover, the study was limited to one site and a small sample. Replication with larger samples and other standardized routes needs to occur to increase generalizability of these results.

Recommendations for Future Research

Future studies need to address how the types and quantity of errors affect passing or failing a road test in a comprehensive driving assessment. Studies should examine how the types of errors may differ across different medical impairments and whether the RODE can be validated with prospective or retrospective crash history.

Because the RODE uses a continuous measure of errors across a performance-based road test (in contrast to a point system at predesignated locations), it may be more easily transferred to use at other locations. Future studies are necessary to test the reliability and validity of the RODE across various sites and with different performance-based road tests.

It is unknown how the seating position in the car affects evaluator observation of driving errors. Therefore, future research could focus on whether error observations using the RODE differ because of evaluator seating position (back seat vs. front seat). Studies should also focus on the comparisons between error observations that are direct from an evaluator and errors that are detected with instrumented vehicles (i.e., vehicles equipped with various electronic mechanisms and sensors to record and review driving performance later, using video analysis). Most clinically based occupational therapy road assessments include only one driving evaluator (positioned in the front seat) who is responsible for observing errors, providing directions, and maintaining safety of the vehicle. In such cases, it may not be possible for this person to record errors with the level of detail included in the RODE. Future studies will focus on ways to adapt the RODE to make it more usable in clinically based driving assessments.