Abstract
The Evaluation of Social Interaction (ESI; Fisher & Griswold, 2008) assesses a person's performance of social interaction skills in the natural context with typical social partners during any area of occupation. We used Rasch analysis of 175 observations of 128 people, ages 4–73, to examine internal scale validity, the items’ skill hierarchy and intended purpose, and the ESI's ability to differentiate between people with and without disabilities. The ESI demonstrated validity for 24 of 27 skills and six intended purposes, with a hierarchy of performance. Of the observations, 95.3% demonstrated goodness of fit to the Rasch model, indicating person response validity. People without a disability demonstrated significantly higher social skills performance than those with a known disability (t = 4.468, df = 83 p = .000). The ESI has the potential to provide a quantitative assessment of social interaction performance in the natural context of a person's desired occupation and may be useful for intervention planning and outcome measurement.
Keywords
Supporting a person's occupational engagement and participation is the focus of occupational therapy (American Occupational Therapy Association [AOTA], 2002, 2008; Baum, 2003). The second edition of the Occupational Therapy Practice Framework (AOTA, 2008) specifically acknowledges the social interactions necessary for occupational engagement. People interact during work, leisure and play, education, and activities of daily living within the context of orchestrating daily occupations. Social skills often support or hinder success as people engage in occupations (Kopelowicz, Liberman, & Zarate, 2006). Social interaction skills are
individual actions or units of social behavior that are observable within the ongoing stream of performance that occur within the context of engagement in an occupation that involves social interaction. A person may exhibit more or less social interaction skill; diminished skill is characterized by “performance errors” that reflect decreased social appropriateness or effectiveness of the behavior. (Fisher & Griswold, 2008, p. 5)
The need for occupational therapists to consider social interaction skills in the performance of occupations (social interaction) is not new to the profession. Doble, Bonnell, and Magill-Evans (1991) reported that occupational therapists provide intervention to support social interaction skills. However, they also found that occupational therapists lacked an evaluation tool with which to plan an intervention or measure its progress. They argued that relying on observations that are not quantified results in incomplete and potentially biased information. From an occupational therapy perspective, Law (2002) and Fisher (1993) emphasized the importance of assessing a person's performance in a natural context as opposed to measuring a multitude of client factors out of context. However, many social participation assessments continue to measure client factor items, such as eye gaze, without considering the effects on overall social interaction and the influence of environment (Cruice, Worrall, & Hickson, 2005; Turkstra, 2005). Assessments of social interaction—such as the Assessment of Communication and Interaction Skills (Forsyth, Lai, & Kielhofner, 1999), Social Functioning Index (Schindler, 1999), and the Simulated Social Interaction Test (Penny, Mueser, & North, 1995)—have most commonly focused on people with psychiatric diagnoses. Many of these assessments are based on observing a person engaged in a simulated social interaction or on self-report. The Social Communication Skills Questionnaire–Adapted, also a self-report, evaluates people who had a traumatic brain injury (Dahlberg et al., 2006). In a 2002 meeting at the National Institutes of Health on research with people with autism, researchers advocated for assessment tools that would measure programs’ effectiveness in a natural context rather than in a clinical or artificial setting (Lord et al., 2005).
The Evaluation of Social Interaction (ESI) is an assessment tool designed specifically for occupational therapists to evaluate the quality of a person's social interaction performance in the natural context during occupational engagement (Fisher & Griswold, 2008). The ESI involves observing a person as he or she engages in a social interaction that is determined by the person, based on what he or she wants and needs to do. The social interaction takes place in a real context of occupation with typical social partners. The occupational therapist records relevant information about the observation, including the intended purpose of the social interaction and aspects of the environment and social partner(s). The intended purpose of a social interaction is the desired goal or purpose, as identified by the person, such as conversing with friends or collaborating with coworkers (Fisher & Griswold, 2008). The occupational therapist also records the familiarity and noise level of the environment; the number of social partners with whom the person interacts during the observation; and the familiarity, status, and overall quality of social interaction performance of the primary social partner. The ESI includes 27 social interaction skills that are scored using criteria specific to each skill. The ESI is analyzed using Rasch analysis. Coster (2008), in her Eleanor Clark Slagle Lecture, advocated for the use of Rasch statistical models in the development of criterion-referenced assessments.
History and Tool Development
Social interaction was included in Kielhofner's (1985) original performance subsystem in the Model of Human Occupation. Doble and Magill-Evans (1992) proposed a model of social interaction to guide assessment and intervention for people who are challenged in social interaction performance. They presented social enactment skills as observable actions that support one's participation in three specific domains of occupation: self-care, work, and leisure. Doble and Magill-Evans (1992) included 27 skills categorized as those for acknowledging, sending, timing, and coordinating one's social interaction. Englund, Bernspång, and Fisher (1995) constructed the Assessment of Social Interaction Skills (ASI), a criterion-referenced assessment using Doble and Magill-Evans's (1992) model. Englund et al. (1995) reported that Rasch analysis of the 16 people who were evaluated using the ASI supported the ASI's validity. However, they found that four skill items misfit the Rasch models and recommended that criteria for these items be refined or that the items’ inclusion in the ASI be reconsidered. They recommended further research be done to examine the ASI's validity and reliability for people with a broad range of disorders.
Fisher (2002–2006) refined the skills used in the ASI as part of her work on the Occupational Therapy Intervention Process Model. Fisher compared the skills to the actions included in the International Classification of Functioning, Disability and Health (World Health Organization, 2001). As a result of her review, Fisher added several new skill items and revised others.
After considering the need for an assessment tool to determine the quality of one's social interaction in natural social situations, Fisher and Griswold (2008) began a series of pilot studies, beginning in 2006, to develop the ESI. The change in the assessment's name helps differentiate the two tools. On the basis of the ASI's skills, as refined by Fisher (2002–2006), occupational therapists and occupational therapy graduate students from the United States helped to further develop and refine the criteria for all of the skills through an iterative process of watching videotapes and people in the community as they engaged in a variety of social interactions while carefully reviewing the criteria for each social skill item of the ESI. The goal at this point was to develop scoring criteria that clearly differentiated the skills from one another and captured errors in observation of people with varying levels of quality of social interaction across age and disability categories.
In addition, expert panels of occupational therapists (with experience in pediatric, adult physical rehabilitation, and mental health practice) identified challenges in social interaction demonstrated by clients that limited their participation in areas of occupation. The panels compared lists of social interaction challenges with the ESI skills to ensure that the ESI captured all types of challenging behaviors, in essence developing the ESI's theoretical construct.
Moreover, Fisher and Griswold met with Englund and Bernspång and updated them on the changes in the ESI. Work with occupational therapy colleagues from Sweden and Denmark resulted in further refinement of some skill items and criteria for scoring and led to the consideration of different intended purposes of social interactions that can be observed by people of all ages as they engage in many different types of occupations. By Fall 2007, the ESI included 27 skills. Of these, 22 had also been included in the ASI; 6 skills from the ASI were collapsed, resulting in 3 skills on the ESI (approaches and greets became approaches–starts; times duration and completes became times duration; and confirms and encourages became acknowledges–encourages). Eighteen of the ESI skills had been identified by Doble and Magill-Evans (1992). Two of their skills were collapsed to make one skill on the ESI (greets and initiates became approaches–starts). The ESI includes two new skills that had not been part of the ASI or Doble and Magill-Evans's list. Produces speech was added to capture a person's production of speech or sign language or use of an augmentative communication device to communicate. Accommodates was added to capture a person's anticipation and prevention of problems that occur during social interaction. The ESI taxonomy and skills are provided in Table 1. All skills are rated using a 4-point rating scale based on competence. Each skill has specific criteria that define competent, questionable, ineffective, or severely limited skill performance.
Social Interaction Skills Found on Evaluation of Social Interaction (
Once the ESI criteria had been refined as noted in the preceding paragraph, 12 occupational therapists and occupational therapy graduate students used the ESI to assess people of different ages and levels of ability. In this study, we report preliminary results of analyses on the ESI's internal scale validity and sensitivity, using Rasch analysis, and consider the instrument's value to the profession of occupational therapy. Research questions for the study were as follows:
Do the ESI's skill items and intended purposes of social interactions form a unidimensional scale of the quality of a client's social interaction performance?
Which skills are easy or more difficult in relation to one another?
Which intended purposes of social interactions are easier or more challenging? and
Is the ESI sensitive to differences in the quality of social interaction between people experiencing a known disability that affects social interaction performance and people not experiencing a known disability?
Method
Instrument
The ESI was administered by 12 occupational therapists and occupational therapy students who had been trained in its administration and scoring according to criteria in the ESI manual. The ESI consists of six intended purposes: (1) gathering information, (2) sharing information, (3) problem solving–decision making, (4) collaborating–producing, (5) acquiring goods and services, and (6) conversing socially–small talk. After the social interaction observation, the occupational therapist scores the client's quality of social interaction on 27 social interaction skills using a 4-point, criterion-referenced rating scale. The criteria against which social interaction skills are rated is a criterion of competence. Therefore, a score of 4 is given when the occupational therapist observes the client to consistently demonstrate behavior that is “socially appropriate, polite, respectful, and timely” (Fisher & Griswold, 2008, p. 60).
Participants
One hundred thirty-four participants (72 male and 62 female) were observed during social interactions and scored on the ESI. A total of 181 social interaction observations were completed; some of the 134 participants were observed during two social interactions. The participants were categorized into age groups for analysis, as defined in Table 2. Most participants were young adults (mean age = 32); however, their ages ranged widely (range = 4–73).
Participant Age Categories
Of the 181 social interaction observations, approximately half were of participants with a disability (47%; n = 85), and approximately half were of participants without a disability (53%; n = 96). Ten different diagnostic groups of participants with a disability were observed; the groups are listed in Table 3.
Diagnostic Groups
The Rasch analysis of the ESI data takes into account the relative challenge of the intended purpose of the social interaction. In this pilot study, the following percentages of intended purposes were observed: (1) information seeking, 5.5% (n = 10); (2) information sharing, 11.6% (n = 21); (3) problem solving–decision making, 5% (n = 9); (4) collaborating–producing, 28.7% (n = 52); (5) acquiring goods and services 17.7% (n = 32); and (6) conversing socially–small talk, 31.5% (n = 57).
Raters
Twelve raters—4 occupational therapists and 8 graduate occupational therapy students—were trained to ensure scoring accuracy with the ESI. We reviewed and explained the ESI training manual and the associated 27 skill items of the ESI and confirmed rater understanding. Training included administrative procedures and explanation of the skills, viewing five social interaction videos, and using the ESI manual to score the social interaction of people in the videos.
Procedure
The 12 raters observed 134 people over an 8-month period. Internal review board approval was obtained at both the University of New Hampshire and community sites participating in the study. Observations were completed at a community-based program for survivors of acquired brain injury, an elementary school, and public areas in the community in a northern New England town. People were observed as they engaged in social interactions in natural contexts during occupations of work, leisure and play, education, social participation, and instrumental activities of daily living. We collected the scored ESI assessment forms and entered the data into statistical software for analysis.
Data Analysis
Data that had obvious rater error were removed. This adjustment resulted in 6 observations being removed and a total of 128 participants and 175 observations used for the final analysis. Using a many-faceted Rasch (MFR) computer software program, FACETS (Linacre, 2008), we converted ESI performance skill item scores into equal-interval linear measures (Bond & Fox, 2007; Fisher et al., 1994; Linacre, 1989). The MFR model also allows for the discovery of the hierarchical order of ESI skill item difficulties and the relative challenge of intended purposes of social interactions (Bond & Fox, 2007; Linacre, 1989) because some social interaction skills and some intended purposes are easier than others.
We examined the ESI's construct validity to determine whether the ESI measures what it was designed to measure. On the basis of the tenets of Rasch statistical models, the ESI asserts that (1) a person has a higher probability of obtaining higher scores on easy ESI skill items and intended purposes of social interaction than on more difficult ones, (2) easy skill items and intended purposes are more likely to be easier for all people than are difficult skill items, and (3) raters are more likely to award higher scores for easy skill items and intended purposes. We therefore examined the ESI's internal scale validity to determine whether these assertions hold true as would be demonstrated by item and intended-purpose goodness of fit to the MFR model (mean square [MnSq] ≤ 1.4, z ≤ 2.0), thereby working together to form a unidimensional scale of the quality of social interaction (Baghaei, 2008). The ideal fit statistics predicted by the Rasch models are MnSq = 1 and z = 0. However, we used MnSq ≤ 1.4 and z ≤ 2 because these fit statistic cutoffs are those commonly used to evaluate validity of observation-based assessments and other tools similar to the ESI (Atchison, Fisher, & Bryze, 1998; Fisher, 1997; Wright & Linacre, 1994).
Finally, we explored the ESI's sensitivity to distinguish between participants with and without a disability. Because few children were represented in these pilot data and they were not well matched for age and disability, we used only data for adults—those in age groups 1–3 and >17—in the t-test analysis. Data from 85 participants were put into two groups: those with a disability (n = 37) and those without a disability (n = 48). We used the logit measures from Rasch analysis of the ESI data for an independent t-test analysis of differences between groups. Because the two groups were not even in size, we used the Levene's test for equality of variance to confirm that group size difference was not influencing the t ratio.
Results
Person-fit statistics demonstrated that 95.3% of the participants demonstrated goodness of fit to the MFR model. Of the 27 skill items, 24 demonstrated goodness of fit to the MFR model, indicating that they were measuring the same unidimensional construct of the quality of social interaction. Although 3 ESI skill items—discloses, concludes–disengages, and thanks—may not be contributing to the unidimensional social interaction scale, the ESI's internal scale validity is strong, and the ESI appears to be measuring the construct of social interaction as it occurs in the natural context during the completion of desired occupations. The analysis showed a person separation reliability of .89 and an item separation reliability of .98. Rasch separation reliability is equivalent to the Kuder-Richardson 20 or Cronbach's α test reliability statistic. Reliability statistics close to 1 are preferred (Linacre, 2008). The analysis showed a person separation index of 2.87. These results indicate that we can reliably differentiate the sample into at least three statistically distinct strata of social skill performance (Arnadottir & Fisher, 2008; Fisher, 1992).
The MFR analysis also produced an item difficulty hierarchy for skill items for the 175 observations analyzed in this study. The item difficulty hierarchy with fit statistics and logit measures is shown in Table 4. We also analyzed the hierarchy of difficulty for the intended purpose with Rasch analysis; each intended purpose with logit measures is shown in Table 5.
Item Difficulty Hierarchy and Fit Measures
Intended Purpose Hierarchy
The t-test analysis to consider ESI's sensitivity to measure the difference in social skill performance between adults with and without a disability was significant. The two groups were significantly different in logit measures, with those not having a disability demonstrating higher skill as measured by the ESI (t = 4.468, df = 83, p = .000). The ESI logit measure was the dependent variable in the t-test analysis. The difference in group size was determined to not influence the t ratio, and equal variances were assumed (F = 1.17, p = .281).
Discussion
The skills included in the ESI fit the construct of social interaction, indicating the ESI's internal scale validity as a measure of social interaction. Three of the social skills misfit the MFR model of the ESI: discloses (revealing information that is personal, sensitive, or inappropriate); thanks (thanking someone for objects, information, help, or a compliment); and concludes–disengages (ending a social interaction). Not all social interactions require someone to thank a social partner, and the social interaction may, in fact, not conclude but transition to another social interaction. Therefore, the skills of thanks and concludes–disengages do not have sufficient data because they were often left blank. Disclosing information that is personal, sensitive, or inappropriate was not often observed, resulting in high scores for this skill. However, when someone does disclose inappropriate information, it can greatly hinder the overall social interaction. These three skills may fit the model with more data. In addition, the criteria and scoring rules for these items should be reviewed for clarity of scoring criteria.
Most of the participant observations (95.3%) fit the MFR model, demonstrating good person-response validity of the ESI. The ability of the ESI measures to reliably separate people by quality of social interaction level is high, as noted by the separation reliability score from the MFR analysis. The ESI measures differentiated the quality of social interaction of adults with a wide range of disabilities from adults without a disability. The two groups were statistically different: The group without a disability demonstrated higher performance measures. The results of this pilot study indicate that the ESI has potential as a tool to quantify the quality of a person's social interaction when observed in a natural context with typical social partners.
The ESI is not diagnosis specific or limited to certain age groups. Results of an MFR analysis shows that the ESI skill items and intended purposes range in their relative difficulty, making the ESI useful for evaluating people with a wide range of social interaction skill. In other words, the ESI is useful for evaluating children as well as adults and across diagnostic groups. Further, the hierarchies for skills and intended purposes may be useful for intervention planning to support a person's social interaction. Occupational therapy will be more successful when addressing social interaction skills that are easier than those that are more difficult. Likewise, working on intended purposes that are easier will be more successful than working on those that are more challenging.
Certainly, the size of this pilot study is small, particularly given the range of ages of people observed. Continued analysis is essential with a larger sample of people with and without a disability. Many questions remain, including those involving the influence of being observed during a social interaction on one's performance, cultural differences in social interaction, and the change in social interaction performance with age and development. It is also important to further explore the tool's sensitivity to differentiate between people with varying levels of social interaction performance. Analysis of differential item functioning will be important to ensure that the instrument demonstrates no bias in relation to gender, age, and culture. Such analysis will require an increased sample size in the ESI database and data from several world regions beyond the United States. In the future, use of the ESI to plan intervention and measure change in quality of social interaction in a natural context should be explored, providing evidence of the effectiveness of occupational therapy services as related to social participation.
Conclusion
The ESI provides a tool to enable occupational therapists to measure a person's quality of social interaction during participation in occupations in their natural contexts. Preliminary data using Rasch analysis indicate that the ESI has good internal scale validity for social interaction. The social skills fit in a hierarchy of difficulty. Noting the intended purpose of a social interaction enables the ESI's use in all areas of occupation. The inclusion of social participation as an area of occupation in the Occupational Therapy Practice Framework (AOTA, 2008) will increase the focus of social interaction in clinical practice. As intervention to support social interaction and participation increases, it is imperative that the profession have both assessment and outcome instruments that are at the level of occupation and produce interval-level measurement to establish solid evidence of intervention effectiveness. The initial psychometrics of the ESI verify that it could achieve these goals.
Future research endeavors with the ESI will need to repeat this study with more participants, both with and without disability, to better understand the issues of the misfitting items and revise the skills as necessary. Finally, pilot studies that explore how social interaction intervention in natural contexts can guide practice will also be important to substantiate the ESI's central assumptions.
Footnotes
Acknowledgments
We thank Anne G. Fisher for her assistance in data analysis and guidance for this article.