Abstract
We investigated (1) the psychometric properties of data gathered with the Performance Analysis of Driving Ability (P-Drive), an on-road assessment; (2) P-Drive’s predictive validity in relation to a summed decision of passing or failing a medical driving evaluation; and (3) the success of implementing P-Drive. Drivers (N = 99) with neurological impairment referred for a driving evaluation were observed on road and scored using P-Drive. Results showed evidence of person response validity and internal validity. P-Drive separated the drivers into four different strata (reliability coefficient = .92). A cutoff raw score of 81 was optimal, with sensitivity of 93% and specificity of 92%. Although only 9 occupational therapists (of 19 initially trained) contributed data for analysis, 17 reported using the assessment. Data gathered with P-Drive yielded evidence of excellent psychometric properties. Moreover, P-Drive was implemented successfully in clinical settings, although initial training to full implementation took longer than 1 yr.
Keywords
Driving is a highly valued occupation that is instrumental to many other meaningful occupations in everyday life. Research has shown the importance of driving (Vrkljan & Miller Polgar, 2007), especially to people who are at risk of losing their license as a result of aging or a medical condition such as an acquired brain injury (Patomella, Johansson, & Tham, 2009). Failing a driving evaluation and losing one’s license could have devastating consequences for one’s general health, well-being, and participation (Marottoli et al., 1997, 2000; Ragland, Satariano, & MacLeod, 2005). Thus, it is important that evaluations of driving performance be valid and reliable—that they prevent personal injuries but also allow people who are safe and competent to continue driving.
On-road tests are often seen as the gold standard, and they compose a significant part of the driving evaluation. However, on-road tests have been criticized for a lack of standardization, little evidence of validity and reliability (Di Stefano & Macdonald, 2003; Kay, Bundy, Clemson, & Jolly, 2008; Selander, Lee, Johansson, & Falkmer, 2011), and questionable ability to challenge drivers enough for examiners to be able to pick up critical and risky behaviors (Christie, 1996). The on-road test has been shown to be the most important determinant of whether a client will pass a driving evaluation and retain his or her driver’s license (Akinwuntan et al., 2002; Dickerson, 2013), thus reinforcing the need for valid and reliable on-road testing procedures.
The examiner’s clinical judgment is a critical part of all assessment. For clinical judgment to contribute in a meaningful way to decisions about driving, examiners must be well trained and equipped with valid and reliable assessment tools. Occupational therapists have a long history of training to become specialists and conducting on-road tests in North America (Korner-Bitensky, Bitensky, Sofer, Man-Son-Hing, & Gelinas, 2006) and Australia (Unsworth, 2007); they are becoming more involved in such assessment in other areas of the world. Many occupational therapists commence on-road tests at the client’s home and travel to a destination that the client visits frequently (e.g., shopping mall, workplace); the use of a route that is not predetermined, and the use of validated assessment protocols are not common (Di Stefano & Macdonald, 2003; Korner-Bitensky et al., 2006). Therefore, in on-road assessments the decision to pass or fail a driver is based primarily on the assessor’s clinical judgment rather than on an objective test score (Di Stefano & Macdonald, 2003; Shechtman, Awadzi, Classen, Lanford, & Joo, 2010).
To our knowledge, only two studies have established on-road assessment cutoff scores to determine pass or fail in a relatively objective way. Both studies mainly included healthy elderly volunteers. Shechtman et al. (2010) established cutoff scores based on the frequency of errors committed by elderly volunteers driving a set route. Unlike the sample typically referred for driving assessment, only 18% of Shechtman et al.’s participants failed the on-road test. Moreover, the route was specific and could be hard to replicate; thus, new cutoff scores would likely be required in different situations and for different routes. Kay et al. (2008) developed and validated an on-road error checklist. Their study was done on a standard route with readily replicable circumstances, but the time spent driving was relatively short (approximately 20 min), and pass or fail was based on a gestalt decision of the three assessors who had been in the car during the test. Both studies investigated predictive validity in relation to the on-road outcome (internal validity), not in relation to the client’s capacity to retain the driver’s license. Thus, a need exists for a validated on-road assessment of sufficient length, with established cutoff scores, developed with a population that includes at-risk drivers and that yields valid and reliable data.
The Performance Analysis of Driving Ability (P-Drive) was initially developed in the structured environment of an advanced driving simulator lab (Patomella, Tham, & Kottorp, 2006). Later, the items were redesigned for on-road tests (Patomella, Tham, Johansson, & Kottorp, 2010). The assessment tool was developed using occupational therapy theory (the Model of Human Occupation) and activity analysis; moreover, a psychological theory about decision making on road (Michon, 1985) was used in the development phase. The items are based on actions taken during driving and are different from those of other assessments that focus on frequency of errors (for further details about P-Drive’s development and theoretical background, see Patomella et al., 2006). P-Drive items are used to score the driver’s capacity to perform safe and competent maneuvers.
Earlier research has shown promising results for the psychometric properties of data gathered using P-Drive: evidence for internal validity (i.e., unidimensionality of the underlying construct, driving ability), internal reliability and construct validity (Patomella et al., 2010) and, recently, interrater reliability (Vaucher et al., 2015). Using a sample of healthy adults, Selander et al. (2011) suggested a cutoff score for P-Drive; however, this cutoff score needs to be confirmed with at-risk drivers to provide clinicians with pass–fail parameters on a clinical population and to further investigate validity parameters.
The three aims of this study were to investigate (1) the psychometric properties of data gathered with P-Drive, (2) the predictive validity of P-Drive in relation to a summed pass–fail decision regarding a medical driving evaluation, and (3) the success of adopting and implementing P-Drive in a clinical setting.
Method
Ethics approval was obtained from the local ethics committee at the Karolinska Institutet, Stockholm, Sweden.
Participants
Participants were a consecutive sample of 99 drivers referred to clinical occupational therapists for a driving evaluation. The participants’ mean age was 69.3 yr (standard deviation [SD]= 5.6; range = 21–85). Participants’ primary diagnoses were stroke (n = 43) and dementia, including suspected dementia and Alzheimer’s disease (n = 34); 15 clients had a diagnosis of mild cognitive impairment (Petersen, 2004), and the remaining 7 had other diagnoses (head trauma, tumor, cerebellar stroke, and psychosis). Twenty-one percent were female. Informed consent was obtained before participation.
Assessment Tool
P-Drive consists of a detailed structured manual and a scoring sheet with 27 items (Patomella et al., 2006, 2010). The items form four subgroups: (1) maneuver, comprising items such as steering and using turn signals; (2) orient, comprising items such as way finding and planning; (3) follow regulations, comprising items such as obeying stop regulations and following speed regulations; and (4) attending and acting on different stimuli in the traffic context, comprising items such as attending to and acting in response to regulatory signs and problem solving. The items are scored on a 4-point criterion-referenced scale measuring the safety and quality of performance (4 = competent and safe, 3 = questionable performance, 2 = problematic, and 1 = incompetent and unsafe performance). Driving performance is scored after observation on a local set driving route.
Procedure
In an intensive 3-day course, 19 experienced occupational therapists were trained in the use of P-Drive for on-road testing. The course was advertised in a newspaper that is directed to occupational therapists who are members of the Swedish Association of Occupational Therapists (90% of Swedish occupational therapists are members). The criteria for attending the course were to have practiced for at least 2 yr and to hold a valid driver’s license. In Sweden, driver licensing issues are commonly dealt with by physicians, but occupational therapists have recently begun to conduct cognitive screening; some complete on-road testing. There is no accredited course to become a driving rehabilitation specialist as there is in the United States and Australia.
The majority of the occupational therapists in this study had previously attended an Introduction to Traffic medical course. They came from different clinical settings (i.e., acute care, rehabilitation, and outpatient clinics). Five of the participating occupational therapists had previous experience doing on-road testing, and a majority screened cognitive abilities to contribute to physicians’ decision making about licensing. In Sweden, the medical driving evaluation is used to determine suitability to drive (i.e., whether clients can retain their driver’s license after an injury or illness). Physicians generally base their decision on the results of cognitive testing, the client’s clinical characteristics, and their clinical judgment. On-road tests have mainly been used as an adjunct when other test results have been inconclusive. Cognitive tests commonly used by occupational therapists in driving evaluations in Sweden are the Nordic Stroke Driver Screening Assessment, Cognistat, and Trail Making Test Parts A and B (Larsson, Lundberg, Falkmer, & Johansson, 2007). For the current study, occupational therapists were taught to base their final decision of pass or fail (i.e., driver’s license retention) on the results of the on-road test along with the clients’ results on cognitive tests and clinical characteristics (i.e., a total driving decision).
During training, the occupational therapists were taught to administer and score P-Drive in a standard manner. They were also taught to design a 60-min set route that included specific challenging traffic situations based on a list from the manual. Situations included intersections with yielding and stop regulations, problem-solving tasks using unstructured instructions (e.g., “Find your way back to the gas station where we recently parked”), and driving in different situations (e.g., at different speeds, in rural areas, on highways, and in city traffic). Each participating occupational therapist designed a local and set route in collaboration with a local experienced driving instructor; route maps were approved by the first author (Patomella) before the therapists began administering P-Drive. The therapists took notes during driving, and the actual scoring of P-Drive was done afterward. The therapists were instructed only to score what they observed and also to score the worst behaviors on each item. For example, if a driver was observed speeding on one occasion but followed speed regulations on other occasions, the one speeding occasion (worst behavior) determined the score without weighting up for better performance.
All participating occupational therapists were instructed to send in at least 10 P-Drive protocols to the first author within 1 yr of finishing the training. Thirty months after the course, an online survey with 10 questions was sent to the 19 occupational therapists who took the course to investigate their success in implementing P-Drive in their clinical setting.
Analysis
We conducted a Rasch partial credit analysis of the raw data from P-Drive using Winsteps Version 3.74.0 (Winsteps, Chicago). The Rasch analysis produces interval-level measures for each item and person. The Rasch program generates goodness-of-fit statistics for the items and persons, which are used to investigate the extent to which the data fit to the Rasch measurement model’s expectations of unidimensionality (internal validity) of the item and person response validity (Bond & Fox, 2007). The criterion for goodness of fit was set to MnSq ≤ 1.5 combined with a ZStd between −2 and 2 for both outfit and infit (Smith, 1996; Wright & Linacre, 1994). Infit statistics are weighted to give more value to on-target observations, and outfit statistics are unweighted and more sensitive to unexpected observations (Bond & Fox, 2007). Data from approximately 5% of items and persons are expected to fail to meet the criteria by chance alone (Bond & Fox, 2007).
To further investigate the unidimensionality of the scale, that is, to see whether any additional dimensions are present in the scale, a principal components analysis (PCA) of standardized residuals was used. A rule of thumb is that the dimension of the scale (the first component) should explain ≥60% of the variance and the variance unexplained by the first contrast should be <5%; other criteria suggest that the unexplained variance in the first contrast should not exceed 3.0 items (Linacre, 2013). A person separation reliability coefficient (the extent to which a test separates participants into different levels of ability) >0.7 indicates that the variable separates people into at least two strata (Fisher, 1992). In this study, we chose to be more conservative because of the potentially severe consequences of the loss of a driver’s license; we chose 0.8 as the criterion for the separation coefficient to receive at least three strata. According to Fisher (1992), a coefficient >0.9 reveals four distinct strata. More strata indicate that an assessment tool can distinguish drivers into more levels of ability and has responsiveness.
We used a Pearson correlation coefficient to investigate the relationship between raw ordinal scores and interval measures derived from the Rasch analysis. A high correlation indicates that the raw scores could be used to summarize the data retrieved from P-Drive. To investigate predictive validity, we used receiver operating characteristic (ROC) curves to determine related cutoff scores and contribute to the examination of P-Drive’s sensitivity and specificity. The ROC curve indicates the diagnostic accuracy of the scoring by displaying an area under the curve (AUC). An AUC close to 1 would indicate that P-Drive reliably distinguishes between pass and fail; a value of 0.50 would suggest that the assessment tool is no better than chance. We used IBM SPSS Statistics Version 20 (IBM Corporation, Armonk, NY) to calculate ROC curves for both raw and interval P-Drive scores.
Results from the online survey were analyzed using descriptive methods to summarize the contents.
Results
Psychometric Properties
An initial Rasch analysis of the 99 P-Drive protocols showed that data from 96% of the therapists were within the acceptable range for goodness of fit, evidence of person response validity. The person separation reliability coefficient was 0.92, revealing that P-Drive separated drivers into more than four levels of driving ability. Data from three items did not meet the cutoff criterion for goodness of fit, indicating too great a discrepancy between expected and observed scores. These items were “Obeying stop regulation,” “Controlling speed (driving too slow),” and “Using turn signals.” We then examined these items in detail and removed five individual scores that we deemed as probable coding errors. A subsequent Rasch analysis revealed that outfit data from only one item, “Controlling speed (driving too slow),” failed to conform to the expectations of the Rasch model (see Table 1). The PCA revealed that the scale (first component) explained 80.3% of the variance of the residuals and that the unexplained variance explained by the first contrast was 2.4%; both indicated unidimensionality in the scale. The unexplained variance in the first contrast was 3.2 items, which is 0.2 higher than the suggested criterion.
Goodness-of-Fit Statistics for All 27 P-Drive Items
Goodness-of-fit statistic did not meet the Rasch model’s expectations.
The correlation between the P-Drive raw scores and the interval measure scores yielded by Winsteps was .88, indicating that raw scores from P-Drive can be summed as a valid outcome score. The mean P-Drive raw score for clients who passed the on-road assessment was 94.6 of a possible score of 108 (SD = 9.6); for those who failed, the mean was 64.9 (SD = 10.2).
Figure 1 is an item map that shows the hierarchy of all items in P-Drive and how well the items target drivers’ abilities. It shows that P-Drive items do not target the full spectrum of drivers’ abilities. Although most of the items are too easy for very good drivers, the items are well targeted to the ability level of drivers who are at risk of failing.
Person ability measures in relation to item difficulty calibrations: Item map with a cutoff at 0.85 logits.
Predictive Validity
When inspecting the AUC in the ROC analysis, the P-Drive raw score and the Rasch-generated interval measure were almost identical (0.98 and 0.97, respectively) and very close to 1, evidence of the ability of P-Drive scores to distinguish between those who pass and those who fail. Analysis of sensitivity and specificity showed that a cutoff raw score of 81 was most optimal; this score yielded a sensitivity of 0.93 (i.e., 93% of drivers expected to fail did fail) and a specificity of 0.92 (i.e., 92% of drivers expected to pass did pass; see Figure 2). At a cutoff score of 81, the positive predictive value was 0.95 and the negative predictive value was 0.90. Taken together, these results demonstrate that a high proportion of clients were classified as would be predicted by a total driving decision.
Sensitivity and specificity of P-Drive using raw scores against a total driving decision of pass or fail.
Figure 1 shows the item hierarchy as well as the total driving decisions for pass and fail; a line is drawn at 0.85 logits, which is approximately equivalent to a raw score of 81. The distribution of participants with a total driving decision of pass or fail above and below the line confirms a cutoff raw score level at approximately 81. For a cutoff raw score of 85, the score Selander et al. (2011) suggested as optimal for healthy adults, the cutoff line would have been slightly higher, at approximately 1 logit. The occupational therapists passed 61 (61%) of the 99 clients tested.
Clinical Implementation
Of the 19 initially trained occupational therapists, 9 sent in a total of 99 protocols for analysis. The online survey was answered by 18 of those 19 occupational therapists. Occupational therapists who did not send in protocols reported being unable to implement the tool (n = 1) and being unable to perform 10 driving tests during the 1st year after the training (n = 7); 1 did not answer the question. Although only 9 occupational therapists contributed P-Drive scores for analysis, 17 of the 18 who answered the survey were, at the time of the survey (30 mo after training), using the assessment. Also, all 17 occupational therapists were using P-Drive each time they commenced an on-road test. The one who did not use P-Drive had changed workplaces and was not in a setting in which driving assessments were relevant.
In response to an open-ended survey question about the clinical implications of using P-Drive, many of the occupational therapists considered their work to have changed in a positive direction after the course. The two most common reasons were feeling more secure in conducting driving tests using the protocol and having a higher professional status because of the new credential of being a trained driving evaluator. Two respondents said that the assessment tool was useful for structuring the documentation using its rubrics. One of the therapists expressed the need for a pass or fail cutoff.
Discussion
In general, the analyses revealed evidence of sound psychometric properties and internal validity and reliability for data gathered with P-Drive by trained occupational therapists. Only one item (3%), “Controlling speed (driving too slow),” failed to conform to Rasch model expectations. Although difficulties with this item could be to the result of chance, in retrospect we consider this item too hard to score reliably. Driving too slow could be a sign of either being cautious or being unable to maintain a speed that does not hinder other traffic. Raters apparently found it difficult to distinguish between the two. Because we believe this is an important item, the manual requires a clearer description. Although the PCA indicated an unexplained variance of the first contrast, suggesting that a second dimension had the strength of more than three items (3.2), the results regarding the percentages of the variance explained were within criteria and indicated unidimensionality of the scale. This inconsistency in the results might not be attributable to the scale’s unidimensionality because some covariance between items in the same content area of a dimension could be expected (Linacre, n.d.).
The high correlation between raw scores and interval scores indicates that the use of raw scores is valid. This is an important finding because using raw scores directly rather than converting them to interval-level units will ease the interpretation of P-Drive scores for clinicians.
Selander et al. (2011) set the raw cutoff score for P-Drive at 85. We found a similar cutoff score (81), suggesting that a cutoff between 81 and 85 is appropriate for distinguishing between pass and fail; however, these cutoffs require further investigation. It can be seen in Figure 1 that a cutoff of 85 would have sensitivity of 82% and specificity of 97%. That is, at a cutoff of 85, nearly all drivers who are safe will be correctly identified, but 18% of drivers who are actually unsafe will be misclassified as safe. Instead of using a definitive cutoff point, we recommend that scores between 81 and 85 make up a gray zone in which the judgment of raters is particularly critical. These judgments should be based on factors beyond on-road performance. More important, a cutoff score is only a guide and should always be confirmed by clinical judgment.
Although only 9 of 19 possible occupational therapists contributed protocols for this study, almost all were administering P-Drive 30 mo after the course. These findings suggest that it takes time to successfully adopt a new assessment tool into the clinical setting. The increase in administration of driving assessments by these occupational therapists likely reflects an emerging need in Sweden for such evaluations to be conducted in the medical setting.
The current study had some methodological limitations. Each route used in the driving tests was local, and the cognitive tests used differed and thus were not standardized; however, this is how clinical driving tests are done and how they need to be conducted—that is, with appropriate tools and using clinical reasoning and judgment. Even though the participating occupational therapists did not have access to a cutoff score and were instructed to make a summed and total decision (including cognitive screening, on-road performance, clinical characteristics, and clinical judgment), the very high sensitivity and specificity suggest that the total driving decision was based mainly on results from the on-road test and possibly the P-Drive score. This assumption is in line with indications that on-road results are the main determinant of a pass or fail on a driving evaluation (Akinwuntan et al., 2002; Dickerson, 2013). A continued need exists to investigate the P-Drive’s predictive validity and external validity and establish and confirm valid and reliable cutoff scores. These aims could be achieved with a larger sample, with a more rigorous reference standard that uses the same cognitive tests and route, and by comparing decisions of expert examiners who are unaware of P-Drive results with those of examiners scoring P-Drive.
Implications for Occupational Therapy Practice
A valid and reliable assessment tool to evaluate on-road performance is critically needed. The current study has the following implications for occupational therapy practice:
P-Drive is an assessment tool that shows evidence of excellent psychometrics.
P-Drive could be implemented in clinical practice.
P-Drive could discriminate between pass and fail, but further studies are needed to confirm a suggested cutoff.
Footnotes
Acknowledgments
We thank all the raters and clients who agreed to take part. The study was funded by Skyltfonden, the Swedish Road Administration’s research foundation.