Reliability and Validity of the Closed-Course Version of the Standardized On-Road Assessment for Drivers (SOAD) With People With Mild Stroke

Participants

The criteria for inclusion were delineated as follows:

▪ Participants had to be referred by a medical professional for evaluation of driving capabilities at the affiliated medical institution.

In Japan, occupational therapy interventions for driving evaluation are prescribed under physician referral. Physicians first determine whether driving ability requires further assessment on the basis of clinical judgment, and patients who are deemed eligible are referred to occupational therapists for further off-road evaluation. These off-road assessments typically include cognitive screening (e.g., Trail-Making Test, Mini-Mental State Examination (MMSE; Folstein et al., 1975), visuospatial function tests, or driving simulator tasks. Usually, participants whose off-road results are considered borderline proceed to the on-road test. Those with clearly severe cognitive or physical impairments are often excluded at this stage for reasons of safety and cost-effectiveness.

The criteria for exclusion encompassed participants who exhibited significant cognitive deficits that would hinder comprehension of the study’s objectives, participants suffering from aphasia that would impede the evaluative procedures, and participants who did not provide informed consent for participation in this research endeavor. Although visual acuity is acknowledged as a factor that affects driving performance, it was deliberately omitted as a formal criterion in the present study. This is because, in practice, people with notable visual impairments would not meet the legal requirements for license renewal under Japanese road traffic regulations. MacCallum et al. (1999) suggested that a sample size of at least 100–200 participants is recommended for factor analysis. On the basis of this guideline, we set the sample size to over 100 in this study.

Instruments (SOAD Closed-Course Items)

The SOAD was developed using the Delphi technique, a method known for achieving expert consensus, by integrating components from earlier on-road tests and expert insights to enhance content validity. The content validity of the SOAD has been affirmed through this methodology (Sawada et al., 2019). The SOAD (closed-course version) administered in a controlled environment (driving school) encompasses an assessment of 40 items along with four indicators pertaining to hazardous behaviors (Table 1). The items of the SOAD meticulously address various dimensions of driving competency, including operational skill, vigilance, and compliance with traffic laws. Their detailed descriptions are provided in the SOAD manual; assessments were conducted with a training vehicle fitted with auxiliary braking systems. The closed-course version of SOAD uses the course delineated in the provisional licensing examination as mandated by Japanese law, which encompasses maneuvers such as altering the vehicle’s trajectory and maneuvering through constricted spaces.

The SOAD evaluation takes approximately 1 hr to complete, with each session lasting around 20 min. In Japan, the course design is based on those used for licensing examinations, incorporating both closed courses and public road courses. The SOAD is conducted on courses that closely resemble these official examination routes. Participants navigate the course on two occasions. After completing the first attempt, evaluators provide targeted feedback focusing on observed errors or areas for improvement. This feedback is intended to inform the second attempt, allowing participants to apply the guidance received. Scoring is contingent on the presence of errors in the assessed items during both attempts. Each item is evaluated on a ternary scale (ranging from 0 to 2 points); higher scores indicate better performance. Hazardous actions are assessed using a dichotomous scale (presence or absence).

Procedure

This study was conducted across five rehabilitation hospitals in Japan that are known for their involvement in driving rehabilitation for people with brain injuries, including academic presentations and clinical services. These institutions were selected on the basis of their relevance to the research topic and their ability to support recruitment and data collection.

At each facility, occupational therapists, in collaboration with attending physicians, recruited participants who met the eligibility criteria. After informed consent was obtained, participants underwent standard off-road assessments (e.g., MMSE, visuospatial tasks, driving simulator evaluations), which were conducted at each facility for clinical screening purposes. After this, participants completed the closed-course SOAD evaluation. The SOAD was conducted by occupational therapists with experience in driving support who had undergone training in the scoring method. They had no involvement with the research team. The data were collected from October 2020 to April 2024.

Data Analysis

Psychometric evaluation was conducted within the frameworks of CTT and IRT. In this study, we used item validity, structural validity, discriminant and convergent validity, and item response to investigate the validity of SOAD. Internal consistency reliability and item response were used to investigate reliability. We used the HAD statistical analysis program (http://norimune.net/had) for descriptive statistics, the normality test, and internal consistency reliability analysis. Mplus 8.11 (http://www.statmodel.com) was used for exploratory factor analysis (EFA), CFA, discriminant and convergent validity analysis, and IRT analysis.

Demographic Data

The study included 108 participants with stroke. Detailed demographic and clinical characteristics of the participants, including diagnosis, hemisphere of injury, driving frequency, motor and cognitive function, are summarized in Table 2. Among the 40 SOAD items, four optional items had missing data (Item 9, 60 cases; Item 12, 102 cases; Item 14, 87 cases; and Item 23, 26 cases). Four optional items were excluded before the factor analysis, resulting in 36 SOAD items.

Item Validity

Table 1 presents the results of the Jarque–Bera test, the PCCs for each item in the SOAD. Evidence of normal distribution was observed specifically in Item 23. The PCCs ranged from .056 to .756. There were seven items with a PCC less than .2.

Structural Validity

An EFA for categorical items was conducted using the full dataset, and models with one to six factors were fitted. For each factor solution, the model provided factor loadings for all items on all factors. Eigenvalues from the EFA suggested the presence of up to five factors (on the basis of the multiple criteria, including eigenvalues, cumulative variance explained, squared multiple correlations, and the minimum average partial test).

Table 3 presents the fit statistics for models with one to six factors. As shown in this table, the four- to six-factor solutions demonstrated good overall fit (RMSEA, ≤0.05; CFI and TLI, ≥0.90). However, the four-factor solution did not yield interpretable loadings. Both the five- and six-factor solutions resulted in interpretable loadings; however, the five-factor solution was more interpretable than the six-factor solution. As part of the factor analysis, items that tended to involve more than one factor (e.g., Items 10, 11, and 21) were excluded. Similarly, items related to traffic signs (Items 24–26) were removed because of conceptual redundancy.

As such, the five-factor solution was retained. Items were retained in the factor with the highest loading, and those with ambiguous cross-loadings were excluded. Table 4 presents the factor loadings for the five-factor solution (other factor solutions are presented in Table A.1, available online with this article at https://research.aota.org/ajot). Items in this Table 4 are grouped by factor and ordered by loading within factor (largest to smallest). The five factors were named on the basis of their corresponding items as follows: Basic Skill, Driving Attitude, Scanning Outside the Field of View, Hazard Prediction, and Merging.

We conducted CFAs on each of the five factors identified by the EFA. Model fit indices for all five retained factors were good to excellent (RMSEA, ≤.05; CFI and TLI, ≥.95).

Discriminant and Convergent Validity

The AVE values for the five factors ranged from .482 to .877, with three factors exceeding the threshold of .50 (Factor 1, .482; Factor 2, .819; Factor 3, .877; Factor 4, .591; and Factor 5, .633). The squared correlation coefficients (SCCs) between factors were as follows: Factors 1 and 2, .160; Factors 1 and 3, .275; Factors 1 and 4, .482; Factors 1 and 5, .349; Factors 2 and 3, .017; Factors 2 and 4, .125; Factors 2 and 5, .182; Factors 3 and 4, 0.187; Factors 3 and 5, 0.189; and Factors 4 and 5, 0.371. All SCCs were less than .50.

Internal Consistency Reliability

The internal consistency reliability coefficients for the SOAD (total score and all subscales) ranged from .657 to .829 for Cronbach’s α and .716 to .841 for Cronbach’s ω (Table 4).

Item Response

Table 5 presents the findings pertaining to the item slope parameter (α) and the item difficulty parameter (β). In general, the items on the SOAD exhibited commendable item response characteristics, with item slopes ranging from .386 to .829. The item difficulty parameters of the SOAD ranged from –2.508 to .113.

Factor Structure and Its Significance

Factor analysis identified five key factors: Basic Skills, Driving Attitude, Scanning Outside the Field of View, Hazard Prediction, and Merging. These factors effectively measure critical aspects of driving competency. Notably, the factors Scanning Outside the Field of View and Hazard Prediction reflect critical cognitive and behavioral processes during driving (Hale et al., 1990; Moore, 1969). These factors directly assess abilities that are essential for ensuring safety in daily driving scenarios, making them particularly valuable. Furthermore, the CFA results indicated good model fit (RMSEA, ≤0.05; CFI, ≥0.95; TLI, ≥0.95), providing statistical validation for the SOAD factor structure. These results confirm that SOAD establishes a robust foundation for evaluating driving competency. The CFA results indicated good model fit (RMSEA, ≤0.05; CFI, ≥0.95; TLI, ≥0.95), confirming the SOAD factor structure. The high fit indices (RMSEA = 0.016; CFI = 0.997; TLI = 0.996) provide statistical support.

Internal Consistency and Measurement Precision

Internal consistency was evaluated using Cronbach’s α and ω coefficients, yielding values of .829 and .841, respectively, for the overall SOAD, indicating that the evaluation items consistently measure specific aspects of driving competency. Some subscales had values below .8, likely because of the small number of items per factor (Cortina, 1993). In this study many of the identified factors consisted of only two or three items, and this likely influenced the results. However, the difference was minimal (approximately .1), and the RMSEA, CFI, and TLI values supported the SOAD’s reliability.

The IRT analysis confirmed that item discrimination (.386–.829) and item difficulty (−2.508 to .113) fell within acceptable ranges. This result indicates that the participants in this study had relatively high functional abilities. People with severe cognitive or physical impairments are frequently excluded from on-road testing because of safety concerns (Cox et al., 2010; Egeto et al., 2019). Because many participants had relatively preserved motor function, these results may reflect the characteristics of driving evaluations targeting people with relatively mild impairments. Additionally, one reason why 16 of the original 40 items remained is that participants had relatively intact motor function, making it difficult to assess operational skills. Consequently, operational items (e.g., Items 7–13) affected by the ceiling effect were excluded. Nevertheless, the excluded operational items may remain relevant for people with motor impairments and could be used optionally in certain clinical settings. These refinements enhanced the efficiency and specificity of the SOAD for evaluating practical driving competencies. This refinement eliminated redundancy, allowing SOAD to assess driving competency more effectively and efficiently.

IRT-Based Evaluation of SOAD

The IRT analysis demonstrated that SOAD is a reliable tool from the perspective of item response. The slope parameters (discrimination) for individual items showed that SOAD effectively captures differences in participants’ driving abilities. In addition to satisfactory item parameters, model fit indices such as the AIC (2,362.68) and BIC (2,491.42) indicated a good fit, further supporting the robustness of the IRT analysis. These results demonstrate that SOAD is a reliable tool for evaluating driving competency across drivers of mild stroke populations.

Uniqueness Compared With Other Assessment Tools

Existing on-road tests are predominantly designed for real-world driving assessments and are often difficult to use in legally constrained or high-risk environments (Bellagamba et al., 2020; Sawada et al., 2019). These results may be useful in countries with legal restrictions or in cases where a closed course is emphasized to ensure greater safety. From another perspective, previous reports have used IRT with only two on-road tests, which used Rasch analysis (a one-parameter model; Cheal et al., 2025; Kay et al., 2008). This study, however, used a two-parameter model, providing a more detailed examination of reliability and validity. This methodological advancement underscores the novelty of SOAD as a measurement tool.