Sensitivity of the Evaluation of Social Interaction Measures Among People With and Without Neurologic or Psychiatric Disorders

Assessments of Social Interaction

Although social interaction has always been an important behavioral domain of concern for occupational therapy, occupational therapists have reported that they lack a standardized tool to evaluate social interaction skills (Doble, Bonnell, & Magill-Evans, 1991; Doble & Magill-Evans, 1992). Doble et al. (1991) carried out a survey of 64 occupational therapists to identify their perceptions of the importance of evaluating social skills in their practice and to determine how they evaluated clients’ social skills. Of those who responded, 79% indicated that social skills evaluations were relevant, but only 29% used some type of formal evaluation; most reported that informal clinical observations were used almost exclusively for assessment.

Although other professionals (e.g., psychologists) have a long history of evaluating social skills for purposes of identifying general and specific social difficulties as well as evaluating treatment outcomes, their evaluation methods typically include self-reports of social functioning, informant ratings or checklists of social performance (i.e., proxy report), performance evaluations based on role play, and direct observations of underlying body functions or isolated skills (Sheridan & Walker, 1999). In contrast, occupational therapists often evaluate the quality of people’s daily task performances, including tasks that involve social interaction, by implementing evaluations based on direct observation as people are engaged in naturalistic daily life tasks that they have identified as meaningful and important for their fullest possible participation in society (Fisher, 2009). According to Fisher and Griswold (2010), the evaluation of social interaction skill in the natural context of engagement in occupation is what differentiates the occupational therapist’s assessment from that of other professionals.

To date, two standardized observation-based occupational therapy evaluations of social interaction have been developed: (1) the Assessment of Communication and Interaction Skills (ACIS; Forsyth, 1996; Forsyth, Lai, & Kielhofner, 1999, Forsyth, Salamy, Simon, & Kielhofner, 1998) and (2) the Evaluation of Social Interaction (ESI; Fisher & Griswold, 2010). The ACIS includes 20 items that are categorized into three theoretical domains: (1) physicality, (2) information exchange, and (3) relations. The ESI includes 27 items that are categorized into seven domains: (1) initiating and terminating social interaction, (2) producing social interaction, (3) physically supporting social interaction, (4) shaping content of social interaction, (5) maintaining flow of social interaction, (6) verbally supporting social interaction, and (7) adapting social interaction. The items in both tools are scored using a 4-point rating scale based on whether the skill observed is competent or supports ongoing social action, is questionable, interferes with social action, or is unacceptable or causes breakdown of the ongoing social interaction.

When the ACIS is administered, the person evaluated can be observed during either naturalistic situations or during role play, and the occupational therapist often becomes a social partner. The person is evaluated in more than one situation, and four social situations are suggested: (1) open, unstructured situations; (2) parallel tasks in which people work individually on tasks but in the presence of others; (3) cooperative groups, in which all social partners are working together toward a common goal; and (4) one on one, in which the person interacts with one other person (Forsyth, 1996; Forsyth et al., 1998, 1999). In contrast, the ESI always involves direct observation of the person’s quality of social interaction during at least two naturalistic, real-life social exchanges with typical social partners. The occupational therapist is always an unobtrusive observer, and the types of social exchange that can be observed are virtually unlimited (Fisher & Griswold, 2010).

When the ESI is administered, the familiarity and status of the primary social partner, the familiarity of the context, the primary social partner’s overall quality of social interaction, the person’s overall quality of social interaction, and the comfort level of the social interaction are also rated for each social exchange observed. Perhaps the biggest difference between the two instruments, however, is that raters who have trained and calibrated as reliable and valid ESI raters are able to use their individual copies of the ESI scoring software, which incorporates a many-facet Rasch analysis program (Linacre, 1993) to generate linearized quality of social interaction measures. These linearized ESI measures can be interpreted from a criterion-based and a norm-based perspective. Criterion-based interpretations include determining whether a person’s quality of social interaction measure is above or below a 1.0-logit competence cutoff on the ESI scale that delineates that the person’s observed quality of social interaction was likely competent during all observed social exchanges (Fisher & Griswold, 2010). Norms are available for people age 2 yr and older, and existing evidence has supported the validity and reliability of ESI measures and scales (Fisher & Griswold, 2010).

Client Groups That Experience Social Interaction Difficulties

Occupational therapy clients who often have problems related to social interaction include those with neurologic (e.g., stroke, traumatic brain injury [TBI]) and psychiatric (e.g., depression, schizophrenia) disorders. For example, people with schizophrenia are often unable to fulfill basic social roles and have significant problems with social relationships, and their social networks are often small (Harvey, Velligan, & Bellack, 2007; Yilmaz, Josephsson, Danermark, & Ivarsson, 2008). Among a wide range of interpersonal or social skills deficits that have been reported among people with schizophrenia are diminished eye contact and use of expressive gestures, inappropriate voice volume, and diminished verbal fluency (Halford & Hayes, 1995; Smith, Bellack, & Liberman, 1996). When a group of people with schizophrenia was compared with a control group (without identified diagnoses) using a performance-based measure, the group of people with schizophrenia demonstrated significantly more social skills deficits than did the control group (Patterson, Moscona, McKibbin, Davidson, & Jeste, 2001). Similar results were found by Sitzer, Twamley, Patterson, and Jeste (2008). Both Patterson et al. (2001) and Sitzer et al. used assessments that involved observation of role plays, not naturalistic social exchanges.

People with TBI have also been reported to often have problems with social interaction (Bornhofen & McDonald, 2008; Dahlberg et al., 2007; McDonald & Flanagan, 2004; McDonald et al., 2008; Robertson & Knight, 2008). People with TBI often have difficulty recognizing emotions, particularly negative emotions, communicated using facial expressions and tone of voice, and they are significantly slower than are people without identified diagnoses in initiating conversation. Inappropriate social communications by people with TBI have been shown to interfere with social reintegration for as long as 10–15 yr after a severe head injury, and loss of social contact with others has been described as the most disabling handicap in daily life (Dahlberg et al., 2007).

Although research has revealed that people with neurologic or psychiatric disorders often have problems with social interaction, studies based on standardized observation of quality of naturalistic social interaction of people with neurologic or psychiatric disorders have been lacking. They have most commonly been evaluated using role play, self-reports, and interviews. To better understand the unique needs of people with neurologic or psychiatric disorders, it is important that they be evaluated on the basis of direct observation and using a standardized tool.

A recent study has shown that the ESI measures are sensitive enough to differentiate quality of social interaction between participants with and participants without a disability (Simmons, Griswold, & Berg, 2010) and between an age-matched sample of people without identified diagnoses and people with neurologic or psychiatric disorders (n = 18 in each group; Søndergaard, 2009, cited by Fisher & Griswold, 2010). However, cross-validation of Søndergaard’s pilot evidence is needed using a larger sample and confirming that the ESI measures are sensitive enough to differentiate among people without identified diagnoses and people with either neurologic or psychiatric disorders, two important diagnostic groups who often demonstrate diminished quality of social interaction and who receive services from occupational therapists. Such evidence would further support the validity of the ESI measures when used with these client groups to identify people with diminished quality of social interaction. Therefore, we addressed the research question, “Do people with psychiatric or neurologic disorders have significantly lower mean quality of social interaction than do people with no identified diagnoses or disabilities?”

Method

Results

The demographic characteristics of the sample are shown in Tables 1 and 2. Most of the participants were people without identified diagnoses, and among the control group, most were female. The gender distribution within the groups with neurologic or psychiatric disorders was more equal. Most people with neurologic disorders had had a stroke or acquired brain injury, and most people with psychiatric disorders had thought (e.g., schizophrenia) or affective disorders. The mean age and ESI quality of social interaction measures for each group are shown in Table 3. The group with psychiatric disorders was younger on average than the other two groups.

As expected, the results of the first ANOVA revealed no significant age effect among the control group (F[4, 299] = 0.83, p = .51); a Levene’s test of homogeneity of variance among age groups for this sample revealed no violations of the assumption of equality. We therefore proceeded to evaluate for significant differences in quality of social interaction between the control group and the group with psychiatric or neurologic disorders.

The results of the second ANOVA revealed a significant group effect (F[2, 482] = 193.58, p < .001). Because the Levene’s test for homogeneity of variance among diagnostic groups was significant (F[2, 482] = 29.11, p < .001) and our group sizes were unequal, we elected to use a nonparametric Kruskal–Wallis test followed by post hoc Mann–Whitney U tests because they are more robust when both group sizes and variances differ among groups (Stevens, 1999). The results of the Kruskal–Wallis test were also significant (χ²[2] = 202.6, p < .001), and the post hoc Mann–Whitney U tests revealed that the group without identified diagnoses had significantly higher quality of social interaction (U = 3,172, p < .001) than did the group with psychiatric or neurologic disorders (U = 3,189, p < .001; see Table 3).

Figure 1 graphically compares the quality of social interaction among people without identified diagnoses and people with psychiatric or neurologic disorders. Although the three groups showed some overlap, almost 75% of the control group had ESI measures above the ESI’s 1.0-logit competence cutoff (Fisher & Griswold, 2010). In contrast, >75% of the people with neurologic or psychiatric disorders had ESI measures below this 1.0-logit cutoff, indicating they most likely had observed quality of social interaction that was at least mildly inappropriate or ineffective. More specifically, when we examined the occupational therapists’ ratings of the participants’ quality of social interaction during each observed social exchange, we found that among the control group (i.e., the group with no identified diagnoses), 215 of 304 (70.7%) demonstrated competent quality of social interaction during both social exchanges observed, whereas only 18 of 77 (23.4%) people with neurologic disorders and 14 of 104 (13.5%) people with psychiatric disorders demonstrated competent quality of social interaction during both observed social exchanges (see Table 3). Finally, Figure 1 also shows that the sample with psychiatric disorders varied more in quality of social interaction than did the sample with neurologic disorders.

OPEN IN VIEWER

Figure 1.

Boxplots showing the median (dark horizontal bar) and range of ESI measures for each group.

Discussion

Our finding that people with neurologic or psychiatric disorders have lower quality of social interaction than do people without identified diagnoses is congruent with the theoretical view that people with neurologic or psychiatric disorders often have problems with social interaction that have an impact on their daily lives (Yilmaz et al., 2008). This finding, therefore, supports the validity of the ESI quality of social interaction measure in terms of its ability to detect significant differences between groups. That is, our results support the sensitivity of the ESI measure with regard to detecting differences in quality of social interaction among groups known to differ.

An advantage of using the ESI is that it is a standardized tool, and using standardized tools better enables both clinicians and researchers to interpret and explain evaluation results to others. As noted by Doble and Magill-Evans (1992), because of the lack of standardized evaluation tools designed to evaluate a person’s quality of social interaction, occupational therapists have mostly used informal clinical observations. With the ESI, the occupational therapist is able to implement a standardized evaluation that yields valid and reliable measures that can be interpreted from both criterion-based and norm-based perspectives. This study highlights the use of criterion-referenced interpretations of a person’s ESI results. That is, the person’s ESI measure, generated by the occupational therapist using his or her individual copy of the ESI software, can be compared with an overall criterion of competence—social interaction that was mature, polite, and respectful. On the basis of this study’s results, healthy adults without identified diagnoses most often demonstrate competent quality of social interaction, but they sometimes also demonstrate questionable or mild problems with quality of social interaction. This finding can be compared with that of people with neurologic or psychiatric disorders, who most often demonstrate mildly to markedly diminished quality of social interaction. As has been noted in earlier research, some people with neurologic or psychiatric disorders do demonstrate competent quality of social interaction (Bornhofen & McDonald, 2008; Burleigh, Farber, & Gillard, 1998; Fisher & Griswold, 2010; McDonald & Flanagan, 2004; McDonald et al., 2008; Robertson & Knight, 2008).

Consistent with an occupational therapy perspective on assessment, the ESI is “designed to assess a person’s quality of social interaction as he/she engages in ‘real’ interactions, with client-specific intended purposes, and with social partners [with] whom the person would typically need or want to interact” (Fisher & Griswold, 2010, p. 1-3). An important feature of the ESI is that it is used to evaluate the quality of a person’s social interaction in the natural context of everyday “doing.” Other professionals most often evaluate with different kinds of assessments, especially self- and proxy reports and role play. Direct observations of naturalistic performance are rarely used because they are viewed as being more time consuming. In contrast, Coster (2008) advocated that occupational therapists need to use valid and reliable criterion-referenced assessments administered in the natural context of everyday life. The ESI meets this need.

The findings of this study—that people with neurologic or psychiatric disorders often demonstrate challenges with social interaction—support the role of occupational therapy in addressing problems with social interaction that affect the person’s full participation in needed or desired occupation. For many such people, diminished social skills mean that they withdraw from social interactions and, in the long term, become increasingly isolated and no longer interact with others (Dahlberg et al., 2007).

Conclusion

People without identified diagnoses and people with neurologic and psychiatric disorders differ in their quality of social interactions, and people with psychiatric disorders appear to have greater variability in their quality of social interaction than do people with neurologic disorders. These results support the sensitivity of the ESI quality of social interaction measures with regard to detecting differences in quality of social interaction among groups known to differ.