Rater Reliability and Rater Effects of the Safe Driving Behavior Measure

Item Inclusion and Exclusion.

We excluded 27 items from the analysis, 22 items that were not observable by the driving evaluator at the time of testing (e.g., driving in snow) and 5 items that added little or no variance to the responses. For example, >95% of respondents used the same rating category (i.e., not difficult) for 5 items.

Interrater Reliability.

We conducted an interclass correlation (ICC) to examine rater reliability on the 41 remaining items. We used SPSS version 17.0 (SPSS, Inc., Chicago) for the analyses and considered p ≤ .05 significant for the correlations.

Rater Effects.

We used the MFRM to analyze rater effects using Facets software version 3.57 (Linacre, 2004). The MFRM extends the rating-scale Rasch model by adding one component or facet (Cj) to calibrate rater severity:

where P_nik is the probability of observing category k for person n who answers item i; P_{ni (k − 1)} is the probability of observing category k − 1; Bn is person ability; Dg_i is item difficulty for item i in group g; F_gk is the difficulty of being observed in category k relative to category k − 1 for an item in group g; and C_j is the severity of judge j, who gives rating k to person n on item i.

We used Facet ruler, fit statistics, fixed χ², and paired comparisons to investigate the rater effects. Facet ruler, displaying three facets (rater, item difficulty, person ability) in the same linear continuum, provides a visual map to compare the relative hierarchy within and between facets. To illustrate the relative distribution of the drivers’ abilities and item difficulties simultaneously, we anchored the mean of the rater severity to 0.

We used fit statistics (infit mean squares [MnSqs] and outfit MnSqs) to detect erratic raters, that is, raters who assign high scores to drivers with a low ability level and low scores to drivers with a high ability level. Infit statistics are more responsive to the variance of those well-targeted observations, and outfit statistics are sensitive to the variance of outliers or extreme observations. Ideal fit occurs when the observed response patterns exactly match the predicted pattern (MnSq = 1) of the model. We considered infit MnSq and outfit MnSq ranging from 0.6 to 1.4 an adequate fit for survey data (Bond & Fox, 2001). The measure represents the average ratings of the rater in logits, with higher scores indicating greater severity in rating.

We used fixed χ² to examine whether at least one rater group, on the overall scale level, consistently used the ratings differently from other rater groups. If the fixed χ² test was significant, then we performed paired comparisons to identify item-level rater effects. For example, if three rater groups are tested, a significant fixed χ² statistic means that at least one of these three rater groups is more severe or lenient in their ratings on the overall scale.

We then performed paired comparisons to identify which rater group was significantly more severe or lenient in its ratings or to show which items the raters rated significantly more severely or leniently. We used an α level of .05 to determine a significant rater effect.

Facet Ruler of the SDBM.

Figure 1 depicts three facets (raters, drivers, items) on the linear interval scale for the SDBM. The first column, Measure, is the interval scale expressed as a logit unit. The second column displays the severity of raters, representing, from bottom to top, lenient to severe raters. The third column shows the distribution of the drivers’ safe driving ability, from bottom to top, representing the drivers with safe driving abilities ranging from poor to good. The fourth column displays item difficulties representing, from bottom to top, the essentially easy items and then progressing to levels of increasing difficulty. The fifth column shows the likelihood of applying the rating scale in relation to the raters’ abilities; that is, when a driver’s estimated ability is between 1 and 2 logits, he or she will likely receive a rating of 4 on this measure. In the second column, the driving evaluator is located above the caregiver, indicating that the driving evaluator is the more severe rater. The distribution of the drivers’ abilities was on the upper part of the ruler, as displayed in the third column, and the distribution of the item difficulties was on the lower part of the ruler, as displayed in the fourth column. This distribution indicated that the drivers had, generally speaking, high safe driving abilities as measured on this 41-item scale.

OPEN IN VIEWER

Figure 1.

Facet ruler of 41 items of the Safe Driving Behavior Measure.

Fit Statistics of the Rater Groups.

The infit MnSqs and outfit MnSqs for both rater groups were between 0.97 and 1.05, well within the defined criteria of 0.6 and 1.4 (Bond & Fox, 2001; Linacre, 2002; Wright & Linacre, 1994).

Fixed χ².

The fixed χ² value, 166.9 with one degree of freedom, was statistically significant (p < .001). The ratings between the F–C and evaluator groups showed significant rater effects: The evaluator group was more severe, considering that the measure of the evaluator group is higher (mean = −3.32, SD = 0.03) than that of the F–C group (mean = −3.98, SD = 0.04).

Paired Comparisons.

The results of the paired comparisons showed significant rater effects on 17 items (Table 2 and Figure 2). Although the evaluators’ ratings were more severe on the overall scale, the F–C group rated 10 of 17 items more severely than did the evaluator group.

OPEN IN VIEWER

Figure 2.

Bias analysis map for the evaluator and family member–caregiver rater groups.

Facet Ruler of the SDBM.

The distribution of the drivers’ ability relative to the distribution of the items’ difficulty indicates that the participants in this study performed well on the instrument. As can be seen in Figure 1, many of our items are on the same logit level. Taking into account that only the means of the items are represented, we have more overlap among the items because each item consists of five difficulty levels corresponding to a 5-point adjectival scale. Having different items at the same difficulty level in the item pool may be redundant for paper-and-pencil tests; however, it will increase the item pool, which will in turn provide more choices for future applications, such as using computer adaptive testing (the next step in the development of our instrument).

Fit Statistics of the Three Rater Groups.

The fit statistics across the rater groups (evaluators and F–C) showed that no rater group was erratic and that overall the evaluators were more severe raters (Facets) than the F–C.

Fixed χ² and Paired Comparisons.

Although the evaluator group rated more severely than the F–C group on the overall scale, the F–C group rated 10 items more severely than did the evaluator group. However, the evaluator group rated 7 items more severely than did the F–C group. Evaluators have the formal training to rate driving behaviors according to the standards of regulatory bodies such as the Department of Motor Vehicle and Highway Safety licensure guidelines, and one can therefore expect that they will be more technical and more stringent in their ratings. The F–C group did not have such formal training and were rating the drivers on their perceptions of their loved one’s driving safety. The tendency for evaluators to rate more severely (than the F–C) may be influenced by their training to focus on identifying deficits. The F–C, however, may be influenced by concern for their loved one’s safety, thus rating more stringently, or may be concerned with maintaining their own independence in transportation and rating leniently, especially given that 31.3% of F–C stated that their independence would be affected if the older driver stopped driving. In future studies, we may want to control for this variable by stratifying F–C on the basis of whether their independence will or will not be affected if the older driver stops driving.

The generalizability of our findings is limited because we used only two evaluators, had a convenience sample, and had a sample size of 80 F–C and 80 older drivers. Our driver sample was skewed to include mainly White (97.5%) and educated participants (63.8% had some college education or a university degree). The F–C sample was mainly female (77.5%) and White (98.8%), and 48.8% had completed a college or university degree.

Despite the study’s limitations, our findings suggest that because of the significant relationship between F–C and evaluator findings, we may train caregivers to more accurately recognize older adults’ unsafe driving behaviors. As such, after a short training program for F–C, we expect that the paired comparisons of the identified items (displayed in Table 2) may show improved congruence between F–C and evaluators. We are developing a caregiver training protocol to test this hypothesis.