Abstract
The General Social Outcome Measure (GSOM) is a performance-based measurement tool designed to assess the change in social skills performance during and after intervention for students with social skills deficits. In the current study, the psychometric properties of the GSOM, including the total score and social communication domains, were examined using a sample of U.S. in adolescents with social skill deficits. Internal consistency alphas were conducted, which showed promising results. Correlational analyses were conducted using the GSOM and the Children’s Communication Checklist-2 (CCC-2), School Social Behavior Scales (SSBS), and the Autism Diagnostic Observation Schedule–Second Edition (ADOS-2). A small but significant correlation was documented between the GSOM and the ADOS-2, with higher GSOM total scores (higher social skills) associated with lower ADOS-2 scores. Additional analyses revealed that individuals with different levels of symptomology according to the ADOS-2 (low, moderate, or high) differed significantly in the GSOM total score. The implications of these findings are discussed with respect to the utility of the GSOM as a progress-monitoring tool for targeted populations with social skill impairments.
Social skill impairments are central problems for students with different types of disabilities (autism spectrum disorder [ASD], social communication disorder, emotional behavior disorders, and others). Social skills are used to facilitate communication and interaction with others and to predict various important outcomes for children (Cummings et al., 2008). Impairments in social communication include challenges in building and sustaining social relationships, initiating and maintaining communication with people, understanding friendships, and making friends with peers (Shulman et al., 2020). The degree of social skills development directly impacts success (e.g., employment, independent living, romantic relationships) in and beyond the school environment (Zablotsky et al., 2014). Furthermore, research indicates that social skills deficits in early school years are linked to academic failures and antisocial behavior (aggressive and defiant behavior) and may lead to increased risk of dropout and a higher rate of bullying at school (Zablotsky et al., 2014). There is a clear and ongoing need to address the social skill deficits of individuals with autism and other neurodevelopmental conditions.
Social-emotional learning has been included as an essential learning domain in schools across the United States because these competencies are correlated with positive academic and social outcomes (Gresham & Elliott, 2014). Specifically, social skills interventions are highly correlated with improved outcomes, such as increased social competency and communication skills, cognitive development, and adaptive behavior skills (Smith et al., 2018). To effectively target social skills deficits, a mechanism to assess individual performance across multiple social skills and consistently monitor progress toward applied outcomes must be available to guide interventions.
Despite the highlighted importance of identifying difficulties, providing interventions, and monitoring social skills progress over time, there is a lack of measurement tools that are sensitive to changes in social skills performance (Salvia & Ysseldyke, 2007; Shulman et al., 2020). In their meta-analysis of social skills training, Gresham et al. (2001) highlighted the need to monitor intervention effectiveness using measures before, during, and after interventions. Assessment at all three time points not only provides an indication of progress but also provides prescriptive data earlier to enhance or modify programs to improve student success. In addition, most of the currently available tools are based on teachers’ and parents’ report questionnaires rather than direct communication and observation of students’ behavior, which is considered the gold standard for measurement (Greenslade, 2021).
A general lack of guidance and/or absence of a framework used to establish responsiveness to social skills interventions as well as appropriately timed use of progress monitoring for making necessary instructional changes exists and should be addressed. Progress monitoring measures add efficiency in assessing intervention outcomes, demonstrating program accountability, and corresponding interventions with treatment objectives (Bellini & Hopf, 2007). Currently, there are a few assessment tools available that detect specific social competency deficits in populations, and even fewer that are designed specifically to direct intervention and provide progress-monitoring tools for ongoing assessment of the outcomes of social skills treatments (Bellini & Hopf, 2007). For brevity, we highlight the measures used for those identified with social skills deficits.
Standardized Social Competency Measures
There are several standardized, norm-referenced assessment tools available to assess the social competency of individuals with ASD and other social skills impairment conditions. The tools described in the following sections were developed to assess social communication in educational settings and clinical trials. The following measures are described in subsequent paragraphs: Children’s Communication Checklist-2 (CCC-2; Bishop, 2006), School Social Behavior Scale, Second Edition (SSBS-2; Merrell, 2008), and Clinical Global Impressions-Severity Scale (CGI-S; Guy, 1976).
The CCC-2 (Bishop, 2006) is a questionnaire used to identify students with pragmatic language and social communication problems in children ages 4 to 16 years. It consisted of 70 items across 10 domains. CCC-2 is completed by caregivers, and it quantifies students’ ability in social communication from 0 (“less than once a week”) to 3 (“every day”). Four scales (speech, syntax, semantics, and coherence) were used to evaluate a child’s performance related to specific language impairment. Sample items include “Leaves off beginnings or endings of words” (speech); “Gets mixed up between he/him or she/her” (syntax); “Mixes up words that sound similar” (semantics); “Speaks fluently and clearly, producing all speech sounds accurate and without any hesitation” (coherent).
The SSBS-2 (Merrell, 2008), is another norm-referenced standardized rating scale used to assess the social skills and behavior of children ages 5–18 years across the K to 12th-grade levels. The SSBS-2 assists school professionals in identifying students at risk of behavioral challenges and antisocial behaviors and supports progress monitoring and development of appropriate instructions (Merrell, 2008). The SSBS-2 includes two scales, the Social Competence (peer relations, self-management/compliance, and academic behavior subscales) and antisocial behavior scales (hostile/irritable, antisocial/aggressive, and defiant/disruptive subscales), which consist of 32 items each and are scored on a five-point Likert scale. Higher scores on the Social Competence scale indicate a greater level of social skills, while higher scores on the Antisocial Behavior scale demonstrate greater levels of antisocial problematic behaviors.
The CGI-S (Guy, 1976) is an assessment tool that has been used in clinical settings to measure the progress of treatment over time in patients with diverse psychiatric conditions such as depression, anxiety, and attention deficit disorder and others (Busner & Targum, 2007). The CGI scale consists of two components: the CGI-Severity, which measures condition severity, and the CGI-Improvement, which assesses the change from the baseline of intervention to the completion of the intervention (Busner & Targum, 2007). There are no universal scoring guidelines for CGI scales; therefore, scoring is merely based on a clinical judgment of the patient, including the frequency, severity, and impact of symptoms (Busner & Targum, 2007). The CGI-S has been validated as a reliable assessment instrument for complex neurodevelopmental conditions, such as schizophrenia, anxiety, and depression (Busner & Targum, 2007; Morrens et al., 2020). However, the CGI-I has not been used to assess changes in social skill performance, which makes it problematic to evaluate the utility of this tool for that purpose (Busner & Targum, 2007).
Research indicates that assessments based on direct observation of social behavior are associated with higher sensitivity to changes in behaviors (Cummings et al., 2008). However, several disadvantages associated with direct observation exist, such as the feasibility of implementation in the natural setting, lack of generalizability across settings, time, and reliability over time (Stichter, Herzog et al., 2012; Cummings et al., 2008). Most notably, there is a lack of standardization of repeated measures that can occur with direct observations. In particular, natural settings can be highly variable from Time 1 to Time 2, and social behaviors exhibited in one setting may manifest functionally differently in another.
General Outcome Measures
General outcome measures (GOMs) are an assessment approach that addresses the limitations of the many current assessments described earlier (measure the change over time, reliability, and so on) and have been used to assess the social skills of individuals repeatedly over time (Cummings et al., 2008; Stichter et al., 2012). GOMs are used as repeated measures and demonstrate validity across settings and time (Cummings et al., 2008). Comprehensive coding procedures were used along with structured observations during the analog condition in the GOM assessments. Analog conditions represent arranged social situations that are challenging and promote student participation and reactions (Cummings et al., 2008; Greenslade, 2021).
The General Social Outcome Measure (GSOM) is an analog performance-based assessment that was developed to address gaps in the currently available tools and to evaluate and monitor changes in social skills performance for youth with social skills deficits through direct communication and observation of students’ social performance. Specifically, it was initially evaluated with students receiving the Social Competence Intervention for Adolescents (SCI-A, Stichter, O’Connor, et al., 2012). The development of the GSOM tool was modeled after academic progress-monitoring tools and the application of GOMs to social behavior (Cummings et al., 2008). Standardized assessments for social skills performance such as ADOS (Lord et al., 2001), Autism Diagnostic Interview-Revised, ADI-R (Lord et al., 1994), Social Responsiveness Scale (Constantino & Gruber, 2005), and SSIS (Gresham & Elliott, 2008) were analyzed to determine the common social skills constructs among the validated assessment tools (Stichter, Herzog, et al., 2012). Furthermore, the content of several social skills curricula for individuals with social skills deficits, but without significant cognitive impairment (IQ > 70), was analyzed (Stichter, Herzog, et al., 2012). Based on an extensive literature analysis, the following social skills constructs were considered and incorporated in the development of the GSOM assessment tool: affect recognition, conversational reciprocity (CR), recognizing and demonstrating emotional states, and social problem-solving (see Stichter, Herzog, et al., 2012).
To investigate the initial promise of the GSOM assessment tool, an exploratory study was conducted with participants of the SCI-A intervention, which was delivered to students across one academic semester, totaling approximately 20 h of group intervention (maximum of six participants per group). The SCI-A was designed to address the social skills deficits of adolescents, and research was conducted on the intervention (Stichter et al., 2010). A detailed description of the SCI-A program is beyond the scope of the current study, but more information can be found in previously published reports (see Stichter et al., 2016; Stichter et al., 2010; Stichter et al., 2018). Students with social skills deficits were evaluated with the GSOM assessments 2 weeks before the start of the SCI-A intervention, every 2 weeks after each learning unit (total 4 units), and 2 weeks after the intervention (Stichter, Herzog, et al., 2012). To assess the general use of the GSOM as a tool for assessing preintervention to postintervention changes, a t-test analysis of scores was conducted and demonstrated the overall improvement in social skills performance among study participants. These results suggest the potential utility of the GSOM as a measure for assessing students’ social skills (Stichter, Herzog, et al., 2012).
Following this initial investigation, the GSOM assessment tool was used in a clinical trial (Zamzow et al., 2016). In 2016, Zamzow et al. conducted a study to investigate the effect of propranolol on CR among individuals with ASD. Researchers have used the Conversational Reciprocity domain of the General Social Outcome Measure (GSOM CR) to measure changes. The findings of this study demonstrated that propranolol significantly increased CR among patients with ASD. The research conducted by Zamzow et al. (2016) demonstrated that the GSOM is sensitive to changes in social behavior over time (under the CR domain). The GSOM is currently being used to extend this line of inquiry within a randomized controlled trial study funded by the Department of Defense. As part of this research project, findings related to this work in a double-blinded placebo research study were presented at key conferences with publications to follow (Beversdorf et al., 2021).
The utility of the GSOM needs more research support, given the small sample size and other limitations of prior research. More research is needed to explore the psychometric properties of the GSOM with different measures of autism and social characteristics, including the ADOS-2, CCC-2, and the SSBS. It is also important to determine whether the GSOM can detect differences between individuals with complex social skill difficulties, including those with autism. The specific purpose of this study is to explore the utility of the GSOM through the following research question: What are the psychometric properties and sensitivity of the GSOM for the assessment and detection of alterations in social skills?
Method
Procedure
The current investigation was part of a longitudinal study on the efficiency of the SCI-A (Stichter et al., 2016; Stichter et al., 2010). The data for the current study were obtained during a federally funded 4-year cluster-randomized trial that evaluated the SCI-A (Stichter et al., 2010, 2016) program as compared to the “business as usual” (BAU) social skills intervention program carried out across 34 public middle schools within one state in the Midwest.
The study protocol was approved by the institutional review board, and all families provided written informed consent for participation. Each participant worked individually with one member of the research team to complete all the assessments. The administrators were trained in the assessment procedures and completed several practice assessments until the reliability of the lead administrator was established. An 85% threshold for inter-rater reliability between the trainer and trainee was established during the training phase.
Participants
Participants were recruited from 34 middle schools in one Midwestern U.S. state. A total of 283 middle school students (6–8th grades) participated in this study. Parent consent was obtained from all participants.
The social skills intervention (SCI-A), from which the GSOM and all other measures in this study were used, was designed for non-ID students with autism or emotional and behavioral disorders (EBD). The inclusion and exclusion criteria were consistent with previous research using this curriculum.
The inclusion criteria for the study included students who (a) did not have an intellectual disability (possess an IQ score above 75 from a reliable test), (b) demonstrated social skills deficits (as identified by school personnel), and (c) attended at least one general education class. Social skills deficits are described as difficulties in building a social relationship with peers and adults, challenges in expressing and reading emotions, limited problem-solving skills, and emotional dysregulation (angry outbursts, damaging behaviors, and aggression; Zager et al., 2012). Because students were inconsistently identified across schools and regions, we did not require specific special education eligibility. Schools do not have access to formal outside diagnosis by a medical care provider without parental consent, so this too is not a reliable way to define the population for the current study. In addition, the current work was funded by the Institute of Educational Sciences to target those with social skill deficits that impact educational performance. Two IQ tests were considered reliable in this study: Wechsler Abbreviated Scale of Intelligence (WASI; Wechsler, 1999) Full-Scale IQ score and Wechsler Intelligence Scale for Children, 4th Edition (WISC-IV; Wechsler, 2003) Global Ability Index score.
The exclusion criteria included the presence of intellectual disabilities (IQ scores below 70), as it is an indicator that students may have difficulties following the requirements of SCI-A, as well as a consistent demonstration of severe behavioral challenges such as physical aggression (e.g., hitting, kicking, biting) and noncompliance (refusal to comply with teachers’ requests and follow classroom rules) that result in repeated out-of-school suspension, as documented by the school. Students who had a significant record of aggression and noncompliance were often in some form of in-school or out-of-school suspension, as well as those with a high number of absences, and were typically excluded from the study due to the potential for reduced dosage of intervention. School personnel were provided with these exclusion criteria and were asked to nominate students and provide corresponding documentation consistent with the inclusion and exclusion criteria. Even though the inclusion criteria described characteristics associated with ASD (social skills deficits), the documented ASD diagnosis and/or special education eligibility were not required to participate in the study; however, these data were collected.
Table 1 provides a demographic description of the 283 students. Most participants were male (84.1%), white (84.1%), and 6th graders (n = 114) and had special education eligibility under the autism category (n = 158). The age of the participants varied from 11.38 to 15.40 years (M = 12.9, SD = 0.9). All the participants reported IQ scores at or above the inclusion criteria (M = 99.8, SD = 14.1). Most participants were eligible for special education under the categories of autism, emotional disturbance, and other health impairments. Finally, 60% of the consenting participants were randomly selected to participate in the ADOS-2 assessment (n = 158). Randomized sampling was designed to mitigate the high cost associated with the implementation of ADOS-2 and the lack of highly qualified professionals to administer this assessment, while providing a representative sample of reliable diagnostic data.
Demographic Characteristics for Students in Study of Psychometric Properties of the GSOM.
Note. N = 283. GOM = general outcome measures.
Measures
To address the current research questions, the data included from the measures described in the following sections were from the first time point before the treatment and BAU conditions began.
General Social Outcome Measure
The GSOM was developed as a progress-monitoring assessment of students’ performance in the following social skills domains: affect recognition, CR, recognizing and demonstrating emotional states, and social problem-solving (Stichter, Herzog, et al., 2012). The total number of items across the four domains was 24. The instrument was designed for middle school students (11–16 years old) with high cognitive abilities (IQ > 70). This measurement was developed after a thorough search and analysis of common social skills constructs among other valid measures and an examination of the literature on social curricula (Stichter, Herzog, et al., 2012).
Affect Recognition
Under the affect recognition domain in the GSOM, the participant was asked to look at the picture and identify the emotions (e.g., happy, sad, confused, angry, fearful) to evaluate the student’s ability to interpret typical facial expressions and emotions. The student is also asked to provide an explanation for the choice made (e.g., the person is happy because his mouth is wide open and he is smiling). The affect recognition domain in the GSOM is scored on a three-point scale: 0 is given for an incorrect match, a score of 1 for correct identification of emotion but incorrect rationale, and a score of 2 is given when the examinee correctly identifies both emotions and provides the correct rationale for his response.
Conversational Reciprocity
In the CR domain, the examinee is asked to demonstrate CR skills, such as staying on a topic, providing or eliciting information, taking turns in conversation, and using appropriate nonverbal communication (body posture, gestures, eye contact, and so on) during the conversation with the exam administrator. To start the conversation, the administrator provides prompts such as giving an opportunity to the students to choose the topic of conversation: “Today I would like to talk about pets or school. Which one would you like to talk about?” Additional prompts during the conversation were noted and deducted from the score, according to the administration instructions. The CR domain in the GSOM is scored on a five-point scale (inappropriate, poor, fair, acceptable, and appropriate), with higher scores indicating more socially appropriate behavior. General administration instructions are provided in the tool manual. For example, for the transition subscale under CR, the following instructions are included for “Appropriate” rating: “student used appropriate timing (e.g., wait for a pause before speaking, pause after speaking) and used appropriate transitions.”
Recognizing and Demonstrating Emotional State
In this domain, a scenario was provided to the student with a task to determine the character’s emotional state, followed by the task of matching the emotional state to the photo and explaining why the student felt it was a correct emotion. For example, the student is given a scenario text read by the teacher: “Carla has been looking forward to seeing a movie with her friend all week. On the day they are supposed to see the movies, her friend calls and says she cannot go because she has to work late” (GSOM, Stichter, Herzog, et al., 2012). Then the student is provided with four photos of a female with different emotions, and the student is asked to identify what emotion Carla is feeling, tell which photo demonstrates what she is feeling, and based on the story, give a reason why she would feel this way. For recognizing emotional state, a score of 0 is given when the interpretation of emotion and justification linking emotions to the story context is incorrect or absent, a score of 1 is given when the interpretation of emotion and justification is loosely tangential, and a score of 2 is given when the interpretation of emotion and justification is plausible. To test the comprehension of emotional differences, the students were asked to demonstrate a certain emotion (e.g., happy or angry) at three levels of intensity to exhibit an understanding of their own emotions and range of emotions. For example, the teacher prompts the student by saying, “People experience different levels of emotions at different times, and I want you to think of a time when you were a little sad, show me what a little sad looks like . . .” (GSOM, Stichter, Herzog, et al., 2012). Then, the student demonstrates the emotions prompted by the teacher. In addition, the student was asked to demonstrate the emotion across three levels (e.g., a little sad, sad, and a lot sad). To demonstrate the emotional state, a score of 0 is given when the student does not demonstrate the target emotion and intensity correctly, a score of 1 is given when the target emotion is demonstrated but intensity (e.g., a little sad, sad, a lot sad) is incorrect, and a score of 2 is given when the student demonstrates both the target emotion and intensity correctly.
Social Problem-Solving
Lastly, the enactive role-play method is implemented in the social problem-solving domain to measure students’ ability to deal with socially challenging situations in a role-play activity (Stichter, Herzog, et al., 2012). Socially problematic situations were given to students in the form of written vignettes and questions. The examinee is then asked to describe whether the situation described is problematic, why, and offer a possible solution to solve the problem and/or improve the situation. The social problem-solving domain was scored on a three-point scale (0 = did not provide a solution for the scenario, 1 = describe the plan of action that is unhelpful, 2 = provides a logical and helpful plan of action). Detailed instructions on GSOM administration are provided in the administration manual for examiners.
The GSOM addresses some gaps that were highlighted in the existing measures, such as direct communication and observation of students’ social behavior; feasibility of implementation as a repeated measure before, during, and after the intervention; and clarity in following instructions that make it possible to use the tool by diverse specialists who work with the student.
The GSOM administration took approximately 20 minutes and was implemented by the same trained professional. A pilot study conducted on GSOM revealed that significant improvements in the overall social skills performance of participants were observed, specifically in CR and social problem-solving domains, which supports the possibility of using the GSOM as a tool for assessing students’ social skills progress (Stichter, Herzog, et al., 2012). The CR domain in the GSOM has been shown to have strong psychometrics in previous research (Zamzow et al., 2016), as well as in the current study. In addition, repeated-measures analysis revealed utility of GSOM in detecting students’ progress after different units of instruction (Stichter, Herzog, et al., 2012).
Children’s Communication Checklist-2
CCC-2 is intended to differentiate children with and without pragmatic language and social communication deficits, which are characteristics of ASD. CCC-2 consists of 70 items within 10 subscales (Bishop, 2006). The tool consists of four scales (speech, syntax, semantics, and coherence) related to specific language impairment, four scales (initiation, scripted language, context, and nonverbal communication) related to pragmatic language, and two scales (social relations and interests) that measure the characteristics of ASDs that are not linked to language. The average completion time for CCC-2 was 15 min. The CCC-2 is recommended to be completed by an adult who knows the student well, as he/she must indicate the rate and frequency of each behavior on a four-point scale, 0 = less than once a week/never; 1 = at least once a week but not every day or occasionally; 2 = once or twice a day/frequently; 3 = several times (>2) a day/always.
Based on psychometric analysis (Bishop, 2006), scales have sufficient internal consistency (α = .73–.88) and inter-rater reliabilities (ICCs = .62–.83). Lastly, CCC (Bishop, 2006) established substantial correlations with scores on the ADI-R (r = .58) and ADOS (r = .36).
School Social Behavior Scales
The SSBS measures positive (social competence) and negative (antisocial) behaviors in the school setting and is designed for both screening and monitoring student behaviors (Merrell, 2008). The SSBS is a 64-item teacher-report measure that assesses both socially competent and antisocial behaviors (32 items per scale). Social Competence items describe positive or adaptive behaviors generally associated with positive social outcomes (e.g., positive relationships with peers, cooperation and compliance with school rules, engagement in academic tasks). The SSBS demonstrated good internal consistency (α = .94–.98) and test-retest reliability (rs = .60–.83). The SSBS also demonstrates high construct validity in comparison with other measures of social skills and has demonstrated success in discriminating between students in regular and special education (Merrell, 2008). School personnel used the SSBS as a measure of students’ social skills deficits. The average completion time of the SSBS was 10 min.
Autism Diagnostic Observation Schedule-2
Randomly selected consenting participants (60% of the total sample, n = 158) were assessed for specific autistic symptomology using the “gold standard” ASD classification measures ADOS-2 (Lord et al., 2001). The Autism Diagnostic Observation Schedule, Second Edition, (ADOS-2, Lord et al., 2001) is a semi-structured, standardized assessment instrument that measures a child’s communication, social interaction, and restricted and/or repetitive behaviors, which are key characteristics of ASD. The administrator of this assessment requires extensive prior experience and training. All administrators involved in this study met the reliability standards designated by the developers of the measure, and appropriate research-use algorithms were used to calculate the scores. The algorithm uses a priori selected items to calculate domain scores on Social Affect (SA, range = 0–20) and Restricted Behavior (RRB, range = 0–8) constructs, which are summed to form a total score. In the current study, the dataset included both the SA and RRB scores, in addition to the total. ADOS-2 uses the total score to create a classification category for each participant: non-spectrum (total score = 6 or less), autism spectrum (total score = 7–8), and autism (total score = 9 or more). ADOS-2 also provides an age-graded comparison score based on the total score. This comparison score was then used to classify the level of autism spectrum-related symptoms: minimal to no evidence (1–2), low (3–4), moderate (5–7), and high (8–10).
Data Analysis Plan
The data analysis steps included psychometric analyses to determine reliability and validity, as well as an exploratory analysis of group differences in key variables. First, internal consistency analysis was conducted for each domain. Validity data were analyzed according to the total scores on the revised domains and other measures deemed to assess similar skills (i.e., CCC-2 and ADOS-2). Group differences were analyzed using analysis of variance (for three levels of the independent variable) or t-tests (two levels). An analysis of variance (ANOVA) was conducted to determine whether the three types of forms (alternate forms A, B, and C) yielded significant differences (i.e., multiple forms used to control for practice effects). A t-test analysis was also conducted to determine if groups with high, moderate, or low scores on the ADOS-2 differed significantly in GSOM total scores.
Results
Internal Consistency
Internal consistency analyses (Cronbach, 1951) were conducted for each a priori domain of the items or a priori formed scale. The total number of items for the highest alpha was determined based on whether the domain alpha improved if the item was deleted. Items were systematically removed according to the item analysis to determine the optimal reliability for domains (e.g., CR). The internal consistency for the four domains was then determined, and each of the items across the domains was entered into a final total GSOM internal consistency analysis (n = 21 items).
For CR, the internal consistency alpha (α) was .757 and included six items (see Table 2). According to item analyses, the deletion of an item does not improve the internal consistency of the domain. The items were added to form a domain score and used for validity analyses.
Cronbach’s Alpha Values for the GSOM and Its Domains.
Note. GSOM = General Social Outcome Measure (Stichter, Herzog et al., 2012).
For social problem-solving, only three items were included in this domain, and the alpha was low (α= .290). Given the exploratory nature of this research, a total problem-solving score was calculated, and then items were deleted if the alpha was higher if items were deleted.
For the affect recognition items, when all 18 items were included, the alpha was low (α= .40s). However, after deleting items that would raise the alpha, the six items in this domain representing facial features yielded higher internal consistency (α= .60).
The analysis on recognizing and demonstrating emotional states, which also included 18 items, demonstrated low internal consistency (α= .40). Further analysis with reduction through analyzing items had a higher internal consistency score for six items (representing the “why” questions items) with an alpha (α) of .60.
For each of the four parts of the GSOM, CR (six items), recognizing and demonstrating emotional states (six items), affect recognition (six items), and social problem-solving (three items), a total of 21 items were entered to determine the internal consistency of the total GSOM score. The alpha with all together was almost as strong as that with CR (α) = .73. No item deletions would have improved the total alpha value.
Validity
Very small but significant correlations were observed across the total scores for the four domains. Table 3 includes the correlations among GSOM, ADOS-2, SSBS, CCC-2, and domains of the GSOM, such as total affect recognition, total perspective-taking “why” questions under recognizing and demonstrating the emotional state domain, total problem-solving, and CR domains. Four significant correlations are observed. The GSOM total score and domains of the GSOM and total scores of the ADOS-2, SSBS, and CCC-2 were examined with one another, and small positive and negative correlations were found ranging from −.017 (p < .001) to −.268 (p < .01). Furthermore, higher GSOM total scores (indicating higher social skills) were associated with lower ADOS-2 scores (r = −.216, p < .01). In addition, ADOS-2 scores were negatively associated with CR (r = −.231, p < .01). A low negative correlation was observed between the total GSOM and total CCC-2 scores (r = −.240, p < .01).
Correlations Among ADOS-2, SSBS, CCC-2, and GSOM.
Note. N = 214. ADOS-2 = Autism Diagnostic Observation Schedule–Second Edition (Lord et al., 2001); SSBS = School Social Behavior Scale–Second Edition (Merrell, 2008); CCC-2 = Children’s Communication Checklist-2 (Bishop, 2006); GSOM = General Social Outcome Measure (Stichter, Herzog et al., 2012); Recog. = recognition; Recogn. WHY = perspective-taking “why” questions; Conv. Recip. = conversational reciprocity; Dem. ES = demonstrating emotional state domain.
Correlation is significant at the 0.05 level (two-tailed).
Correlation is significant at the 0.01 level (two-tailed).
It is also important to determine whether the different forms of a scale are equivalent. Analyses of variance showed no significant differences (p < .05) among the three forms of GSOM for any domain.
Group Comparisons
A t-test was conducted with the independent variable of classification of high and moderate levels of symptoms on the ADOS-2 and GSOM total scores as the dependent variable. A significant group difference was found for the independent variable of high and moderate scores on the ADOS-2 and GSOM total scores. The group with high ADOS-2 scores had significantly lower total GSOM scores (M = 45.71; SD = 6.80) than the group with moderate ADOS-2 scores (M = 47.85; SD = 5.36). Cohen’s d effect size for the mean difference was small d = .34. A follow-up bivariate logistic regression was conducted after creating a dichotomous variable for the GSOM total score (with the lowest GSOM scores, specifically the bottom 15th percentile) and the two levels of ADOS-2 scores. A significant difference was documented in the odds of being in one group based on the scores from the other grouping (B = 1.29, p = .03, odds ratio [OR] = 3.61). OR analysis indicated that those in a high grouping on the ADOS-2 were 3.61 times more likely to be in the bottom 15th percentile on the GSOM; stated differently, those with high symptom levels on the ADOS-2 were 3.61 times more likely than those with moderate or low levels to be in the group with GSOM scores in the 15th percentile or below (with low scores indicative of less social-communicative competence).
Discussion
The purpose of the present study was to investigate the psychometric properties of the GSOM in a population of students with social skills deficits. The study examined the reliability and validity of the measure, and the results suggest that the GSOM is a promising social skills assessment tool for individuals with ASD and related social skills deficits (e.g., social communication disorder, emotional behavior disorder). The current study of the psychometric properties of the GSOM will contribute to more evidence-based use of this progress-monitoring instrument for social skills development.
The data analysis established good internal consistency for the total GSOM score and CR domain. These results support the reliability of the total GSOM as well as the separate domain of CR. The study conducted by Zamzow et al. (2016) on the effects of propranolol on CR in individuals with autism documented that the GSOM CR was sensitive to changes in CR between intervention and placebo conditions (Zamzow et al., 2016), which suggests the utility of using separate domains of the GSOM tool for the specific purpose of evaluation and intervention. However, the results of the “Social Problem Solving” domain were in the lower range (α = .290), which can be explained by the small number (n = 3) of items included in this domain.
The validity of GSOM has been investigated in several ways. First, correlations were conducted with other common measures of social skills and ASD characteristics, including SSBS, ADOS-2, and CCC-2. Significant, small correlations were found between the GSOM and SSBS and between GSOM and CCC-2 in the expected directions, considering that all included higher scores as reflective of higher social skills. Next, the GSOM total scores were negatively correlated with the ADOS-2 score (r = −.216, p < .01), which corresponds to higher scores on the GSOM associated with lower scores on the ADOS-2, which correlates with fewer ASD characteristics (Lord et al., 2001). Given the restrictive range of this clinical sample, we also conducted analyses that could assess the differences in the GSOM according to the level of symptomology on the ADOS-2 (moderate or high), using t-tests and bivariate logistic regression. The ADOS-2 groups were significantly different in their GSOM scores from the group with higher symptomology on the ADOS-2 with lower GSOM scores. Follow-up OR analysis found that individuals with high scores on the ADOS-2 were over three times more likely than those with moderate scores to score in the bottom 15th percentile on the GSOM.
The GSOM was designed to fill a need in the field for a unique assessment that can be utilized for intervention decision-making, including targeting specific skills in need of instruction and monitoring change with items that are more sensitive to more subtle and specific skills. In our sample, including adolescents with ASD or identified social deficits, we found evidence that those with high ADOS-2 scores on autism symptoms had lower scores on the GSOM than those with more moderate symptoms on the ADOS-2. This provides initial evidence that the GSOM can differentiate between groups with different severity levels. However, this is the only way to validate the GSOM. The next way is to determine its utility in making educational decisions for intervention targets and then to determine if it is sensitive to change over time.
Previous research on the GSOM has demonstrated significant changes in participants’ social skills performance over time (Stichter, Herzog, et al., 2012; Zamzow et al., 2016). The GSOM could be utilized to monitor students’ progress over time and make instructional decisions based on the results of the assessment (research was conducted on the intervention, plan for the intervention, adjust/change the intervention, and so on). With the implementation of the GSOM, educators and clinicians have the opportunity to assess students’ skills before and immediately following the intervention, which allows them to adjust their instructional programming or clinical intervention and plan accordingly (Stichter, Herzog, et al., 2012; Zamzow et al., 2016). For the Social Competence Intervention (Stichter et al., 2010), the GSOM was implemented before the intervention, after every unit of the intervention (every 3 weeks), and over (15 weeks). The examiners can use the GSOM as an assessment measure or simply utilize separate domains of the GSOM (e.g., CR) after the intervention (or unit of intervention) targeting specific social skills (e.g., reading facial expressions and CR) is delivered to the student. Practitioners may adapt the domains of the GSOM (e.g., CR) to the needs of specific students based on the student’s skill level. For example, if a student’s goal is to stay on topic and maintain eye contact, these specific skills can be assessed before, during, and after the intervention.
Within the field, there is a significant need to continue to create reliable assessments, especially performance-based ones, that transcend clinical and educational utility. Assessment instruments that can and are used both clinically and educationally may strengthen individual care and outcomes for individuals with social skills deficits by connecting educators and clinicians toward converging efforts that improve not only outcomes but also the social and ecological validity of treatment. These measures will support practitioners in evaluating the benefits of interventions and in planning for adjusting or changing interventions. Professional educators, researchers, psychologists, and others experienced in working with students with social skills difficulties can implement this measure.
Limitations
There are some limitations to the GSOM. It is important to note that the GSOM was initially designed as a performance-based measure to monitor the progress of students with ASD during the SCI-A intervention. As such, the generalizability of this tool to a broader population of students with social skills deficits and the evaluation of different interventions is warranted. In addition to the study by Stichter, Herzog, et al. (2012), future research should continue to examine the extent to which repeated exposure to the GSOM may influence the scores for those using it for progress monitoring. Next, the data analysis demonstrated high reliability for both the CR domain and the total GSOM score; however, other domains had lower internal consistency alphas. This limitation underscores the need to further investigate domains of the GSOM, such as affect recognition, recognizing and demonstrating emotional states, and social problem-solving domains, to further revise and potentially strengthen the measure.
Future Directions
There are several directions for future research that will contribute to the continued development of GSOM. Although the GSOM was developed for use in an educational setting, the tool has been successfully implemented in a clinical trial, and the CR domain has demonstrated sensitivity to changes in behavior (Zamzow et al., 2016). Therefore, to confirm the utility of the tool in clinical settings, additional research on the use of the GSOM with a larger sample size should be conducted. In addition, future work could evaluate the utility of the GSOM in evaluating the effects of diverse social skills interventions other than SCI-A. This measurement tool can be evaluated as a whole (all domains included) or specific subdomains of targeted social skills interventions. Because there were domain differences, more research should be conducted to understand why one domain (e.g., CR) demonstrated stronger reliability and validity than other domains. Future research should explore other domains and identify whether they can be used in isolation. Subsequent studies may also conduct a more in-depth analysis of the validity of the GSOM with other measures of social skills. Further investigation should be conducted around ongoing progress monitoring for the GSOM and similar types of assessments, so that clinicians and educators can evaluate and adjust accordingly during interventions.
Importantly, the GSOM has been developed based on Social Competence Intervention (SCI-A, Stichter et al., 2010) domains known to be representative of ASD characteristics and not unique to the intervention on which it was evaluated. Further evaluation is needed to evaluate the potential generalization of the tool implementation to other populations with different diagnoses but consistent social skills deficits.
Conclusion
Social skill deficits are a hallmark characteristic of ASD and other disabilities. Such deficits significantly impact students’ academic and social performance at school; accordingly, there is a clear need to support these social skills. The GSOM is an instrument that was developed specifically for this purpose to engage students in the assessment process, which allows examiners to collect data based on direct interaction and observation of students’ social performance. The present psychometric study demonstrated that the GSOM is a promising social skills assessment tool for students with social skills deficits. However, further research is required to confirm and extend these findings.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The research reported here was supported by the Institute of Educational Sciences, U.S. Department of Education, through grants R305A100342 and R305A130143 to the University of Missouri. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.
