Cross-Cultural Validation of the Evaluation of Social Interaction (ESI)

RESEARCH QUESTIONS: Do any of the Evaluation of Social Interaction (ESI) items demonstrate differential item functioning (DIF)? Do any of the types of social exchange of the ESI demonstrate differential social exchange functioning (DSF)? Does the ESI demonstrate differential test functioning (DTF)?

RATIONALE: Social interaction is a recognized area of focus for occupational therapy assessment and intervention. The ESI is used internationally to evaluate clients’ quality of social interaction. There are, however, cultural differences in the way people interact socially, and there has been little effort to implement cross-cultural studies of social interaction assessments. The purpose of this study was to determine whether the ESI could be used to validly evaluate the quality of social interaction of people from different countries.

DESIGN: Retrospective, descriptive, cross-cultural validation study

PARTICIPANTS: A total of 9,811 participants from Australia, Canada, Denmark, Iceland, Ireland, Italy, Japan, South Korea, Sweden, Switzerland, the United Kingdom, and the United States, between the ages 2 and 101 yr, in good health and with a variety of diagnoses were selected from the international ESI database.

METHOD: When the ESI is administered, the participant is observed during two prioritized social exchanges in natural settings with typical social partners. The occupational therapist (rater) scores 27 ESI items using a 4-point scale (4 = competent, 3 = questionable, 2 = ineffective, 1 = severe) based on the degree to which the person’s quality of social interaction was impolite, disrespectful, poorly timed, or immature, giving consideration to the cultural context of the country where the person is evaluated. The ESI measures are free of gender bias and sensitive enough to differentiate among groups known to differ.

ANALYSIS: We used many-facet Rasch measurement methods to analyze participant raw data. Effect sizes were used to evaluate for DIF and DSF; criterion = 0.55 logit. Analyses of DTF were conducted by plotting country-specific participant ESI measures in relation to 95% confidence intervals.

RESULTS: A total of 1,752 (98.3%) of paired ESI item difficulty values showed no DIF, and none of the types of social exchanges showed DSF. Of the 30 (1.7%) significant DIF pairs, 10 were associated with only two raters each from the Italy, Switzerland, and United States, who scored Touches (Italy/Switzerland vs. United Kingdom/Australia/Japan) and Regulates (United States vs. Iceland/Switzerland/South Korea/Italy) differently than other raters from the same countries.

Two significant DIF pairs (Approaches/Starts, Canada vs. Italy; Heeds, United States vs. Italy) may have been related to participant diagnosis or age. The other 18 pairs were associated with Discloses, Empathizes, Accommodates, and Benefits, but we found no clear pattern to explain the differences. Finally, the existing DIF did not result in DTF.

DISCUSSION: There is only minimal (1.7%) DIF in the ESI and, combined with an absence of DSF and DTF, our findings suggest that the ESI measures are at minimal risk for cross-cultural bias. Rater scoring error associated with misunderstanding of the ESI scoring criteria explained a large proportion of the DIF. Future research is needed to examine for DIF associated with diagnosis or age.

IMPACT STATEMENT: The ESI is an occupation-focused and occupation-based assessment of a person’s quality of social interaction that supports occupation-centered practice. This paper presents validity evidence that the ESI is a culture-free assessment that can be used clinically and in collaborative research internationally.