Correlation of cognitive and social outcomes among children with autism spectrum disorder in a randomized trial of behavioral intervention

Abstract

Although social impairments are considered the hallmark deficit of autism, many behavioral intervention studies rely on cognitive functioning as a primary outcome. Fewer studies have examined whether changes in cognition are associated with changes in social functioning. This study examined whether cognitive gains among 192 students from 47 kindergarten-through-second-grade autism support classrooms participating in a year-long behavioral intervention study were associated with gains in social functioning. Children’s gains in cognitive ability were modestly associated with independent assessors’ and teachers’ evaluations of social functioning but were not associated with changes in parent ratings. Observed social gains were not commensurate with gains in cognition, suggesting the need both for interventions that directly target social functioning and relevant field measures of social functioning.

Keywords

autism spectrum disorder intervention social deficits

Introduction

Autism spectrum disorder (ASD) is characterized by three core domains: impairments in communication and language, impairments in social reciprocity, and abnormally restricted, repetitive behavior. Several comprehensive treatment approaches have shown promise in improving outcomes for children with ASD. The most commonly employed and empirically supported interventions have been based on the principles of applied behavioral analysis (ABA), a scientific approach to understanding behavior, and how it is affected by the environment (Reichow and Volkmar, 2010; Reichow and Wolery, 2009).

ABA includes a variety of established teaching strategies, such as discrete trial training (DTT; Smith, 2001) and pivotal response training (PRT; Koegel et al., 1989). DTT is a highly structured, one-on-one teaching strategy that breaks apart individual behaviors into smaller teachable units (Arick et al., 2004; Schreibman, 2000; Smith, 2001). PRT uses a more naturalistic, play-based approach to target crucial skills, including motivation and responsivity to the environment that are pivotal for many other skills (Koegel et al., 1989). Treatment programs that utilize these ABA principles have demonstrated dramatic improvements in young children’s cognitive functioning, adaptive behavior, language, and play skills (Dawson et al., 2010; Lovaas, 1987; Rogers and Vismara, 2008; Spreckley and Boyd, 2009; Stahmer, 1995).One such intervention, the Strategies for Teaching based on Autism Research (STAR), which incorporates DTT, PRT, and functional routines into a comprehensive program for young children with ASD, has shown improvements in functional communication, expressive language, and social interaction skills (Arick et al., 2003, 2004).

It is well documented that ABA strategies can be used alone or in combination with other treatment methods to address several core deficits associated with ASD across ages and functioning levels (Reichow and Volkmar, 2010). Although increasing in recent years, one area that has received comparatively less research attention is social functioning, one of the most challenging deficits to ameliorate associated with ASD (Reichow and Volkmar, 2010). Most large randomized controlled trials of ABA-based interventions have primarily measured cognitive, language, and behavioral outcomes of children with ASD, rather than social functioning, in part, because the definitions of social functioning are quite varied and difficult to measure across such a diverse population (Cunningham, 2012; Fombonne, 2010; Matson et al., 2007). Many school-aged children with ASD experience a range of enduring and complex deficits that fall underneath the larger umbrella of social functioning, which encompasses a broad range of abilities, including foundational skills (e.g. using eye contact, playing with an object, taking turns with peers) as well as more complex skills (e.g. having reciprocal conversations, playing social–interactional games with typically developing peers, establishing and maintaining friendships) that without intervention are unlikely to change (Kasari et al., 2011, 2012).

While ABA-based programs address several important domains of children’s development, including cognitive functioning, and have been widely used to improve the social skills of young children with ASD (Reichow and Volkmar, 2010), there are little data available examining the extent to which changes in cognitive gains realized through a behaviorally based intervention are associated with changes in social functioning. Thus, the objective of this study was to examine whether gains in cognitive functioning as a result of participating in an ABA-based curriculum, the STAR program, for children with ASD were associated with gains in social functioning as measured by independent clinicians, parents, and teachers. We hypothesized that children who received intervention through the STAR program would show considerable gains in cognitive ability over the course of the school year and that those gains would be associated with improvements in social functioning.

Methods

Participants

Data were drawn from a multiyear randomized field trial of kindergarten-through-second-grade (K-2) autism support classrooms called the Philadelphia Autism Instructional Methods Study (AIMS). This study includes data from students with ASD new to the STAR program in either Year 1 or Year 2 of the Philadelphia AIMS that had complete assessments at the beginning and end of the same school year (n = 192). A total of 47 K-2 autism support teachers with a wide range of autism-related teaching experience (1–29 years) from 43 schools participated. The average age of students was 6.1 years (standard deviation (SD) = 0.9 years) at entry into the study (range = 5–8 years). All children were in kindergarten through second grade; however, the exact number of children from each grade level was unknown. The majority was male (85.9%), and the ethnic breakdown was the following: 41.9% African American, 17.8% Caucasian, 2.6% Asian, 7.3% Latino, 4.2% Multiethnic, and 26.2% Unknown. The average IQ, as measured by the Differential Ability Scales–Second Edition (DAS-II), at baseline was 59.1 (SD = 23.1) with an average nonverbal IQ (NVIQ) of 68.5 (SD = 23.9) and verbal IQ (VIQ) of 50.7 (SD = 25.1). The average score on the Autism Diagnostic Observation Schedule (ADOS) severity algorithm was 6.2 (SD = 1.8).

Measures

ADOS

The ADOS is a standardized clinician-administered observational measure of social and communication skills used to classify children as meeting criteria for an ASD (Lord et al., 2000). ADOS symptom severity scores were calculated for each administration using the ADOS symptom severity algorithm (Gotham et al., 2009).

DAS-II

The DAS-II is designed to assess cognitive abilities in children ages 2 years 6 months through 17 years 11 months across a broad range of developmental levels (Elliott, 2007). The DAS-II yields a General Conceptual Abilities (GCA) score (M = 100, SD = 15), that is highly reliable, with internal consistency scores ranging from .89 to .95 and a test–retest coefficient of .90.

ABAS-II

The Adaptive Behavior Assessment System–Second Edition (ABAS-II) Parent Form is designed to measure adaptive behavior in three conceptual domains and 10 adaptive skill areas (i.e. communication, community use, functional academics, health and safety, home or school living, leisure, self-care, self-direction, social, and work) among children with developmental disabilities (Harrison and Oakland, 2003). Only the Social Subscale and Social Composite (e.g. items assessing play, friendships, emotion recognition, prosocial skills, etc.) scores were used in this study. The internal consistency of the ABAS is .83 and test–retest reliability ranges from .75 to .96.

Pervasive Developmental Disorder Behavior Inventory

The Pervasive Developmental Disorder Behavior Inventory (PDDBI)–Teacher Form is designed to measure behaviors unique to pervasive developmental disorders and consists of six domains: sensory/perceptual approach, ritualisms/resistance to change, social pragmatic problems, semantic pragmatic problems, social approach behaviors, and expressive language (Cohen and Sudhalter, 2005). The social pragmatic problem domain captures the difficulties children with ASD have in reacting to the approach of others, understanding social conventions, or initiating social interactions with others, whereas the social approach behaviors domain assesses the utilization of children’s social communication skills (Cohen and Sudhalter, 2005). Only the social pragmatic problems (e.g. understanding social nuances when approaching and interacting with others, self-awareness, inappropriate reactions to others) and social approach behavior (e.g. use of affect, visual attention to others, gestures, responsiveness to cues) domains were used in this study. Median internal consistency for the PDDBI teacher version is 0.91, and test–retest stability for teacher ratings ranges from .65 to .99.

Procedure

Participation in AIMS was voluntary. Teachers and classroom staff members participated in training in the STAR program in Years 1 and 2, and only students from participating classrooms that were newly implementing the STAR program were included in this study. Data collection occurred at the beginning (September/October) and end (May/June) of the school year. Psychologists trained to research reliability evaluated all participating students using the ADOS, to ensure they met criteria for an instrument classification of an ASD, and the DAS-II. Teachers were asked to complete a battery of assessments that included the PDDBI. Teachers received US$10 per student, per data collection point. Similarly, parents were asked to complete a battery of assessments, including the ABAS. They received US$50 for the first wave of data collection at the beginning of the school year, US$100 for the second wave of data collection at the end of the school year, and a summary report of their child’s assessment that was designed for use in individualized education program (IEP) planning.

STAR

The STAR program (Arick et al., 2004) combines three instructional approaches based on the principles of ABA: DTT, PRT, and functional routines into a comprehensive curriculum for children with ASD. These components are paired with a highly manualized curriculum with 169 lesson plans that provide educational programming in six key domains: receptive, expressive, and spontaneous language, preacademic concepts, functional routines, and play and social interaction concepts (e.g. turn-taking, functional and symbolic play, sharing with peers, pretend play, recess with peer, games) across three levels of functioning. Throughout the day, classroom staff facilitated DTT and PRT sessions as well as functional routines with their students (and peers when appropriate) to enhance academic, language, and social outcomes. DTT and PRT sessions occurred as needed (e.g. ranging from no one-on-one time to 15-min daily sessions) and were determined by the teachers’ schedules as well as children’s individualized student learning profiles, a STAR guide to program implementation and progress monitoring.

Teacher training included didactic instruction and in-classroom support (in vivo coaching). Didactic instruction occurred across seven professional development workshops throughout the school year in Years 1 and 2 of the Philadelphia AIMS. The professional development workshops were either half-day (3.5 h) or full-day (7 h) sessions. During these workshops, teachers and classroom staff were broadly trained in each component of the STAR program using direct instruction, small group instruction, role play, rehearsal, and modeling. In addition, six independent consultants trained by the developers of the STAR program were assigned a cohort of classrooms to provide in vivo coaching to teachers and classroom staff. Coaching included direct observation, modeling, and performance feedback that occurred in 45- to 60-min sessions every 2 weeks throughout the school year (approximately 16–17 coaching sessions a year). Coaching varied by classroom, based on the level of training required; however, consultants followed a scope and sequence beginning with arranging the classroom environment and establishing functional routines that promote students’ independence, then moving to DTT and PRT.

In addition, program fidelity measures for each program were designed based on the manuals and in consultation with the trainers (Mandell et al., 2013). Each teacher was filmed for 30 min (10 min of DTT, 10 min of PRT, and 10 min of functional routines) once per month. Blind raters coded all videos. Two tapes for each coder were recoded by a lead investigator every other month to measure criterion validity. If less than 90% agreement occurred, additional training until this level of agreement was achieved. Based on the videos, the average fidelity of STAR implementation was 57% (range = 12%–92%; Mandell et al., 2013).

Statistical analyses

Table 1 presents means, SDs, and change scores for each variable of interest at the beginning (baseline) and end (exit) of the school year. Change scores were calculated to capture the degree of difference between baseline and exit. Correlations were conducted between each change score to determine whether gains in cognitive ability were associated with gains in social functioning as rated by independent assessors, parents, and teachers.

Table 1.

Means, standard deviations, and change scores for child, parent, and teacher measures at baseline and exit.

	BaselineM (SD)	ExitM (SD)	Change scoreM (SD)
Child measures
DAS-II (N = 192)	59.12 (23.13)	64.68 (23.08)	5.56 (11.51)
ADOS Severity Score (n = 148)	6.32 (1.78)	6.18 (1.88)	−.15 (1.96)
Parent measures
ABAS – Social Subscale (n = 114)	4.04 (3.00)	4.08 (3.36)	0.04 (1.85)
ABAS – Social Composite (n = 112)	9.91 (5.76)	9.78 (6.56)	−.13 (3.47)
Teacher measures
PDDBI – Social Approach (n = 160)	56.13 (10.87)	58.04 (11.61)	1.91 (7.12)
PDDBI – Social Pragmatic Problems (n = 160)	53.66 (10.47)	55.39 (10.71)	1.73 (8.52)

DAS-II: Differential Ability Scales–Second Edition; ADOS: Autism Diagnostic Observation Schedule; ABAS: Adaptive Behavior Assessment System–Second Edition; PDDBI: Pervasive Developmental Disorder Behavior Inventory.

Results

The mean change in global cognitive ability scores on the DAS was modest (M = 5.56, SD = 11.51); however, the mean change scores from each of the measures of social functioning were negligible. Overall, the average change in (1) ADOS symptom severity was M = −0.15 and SD = 1.96; (2) parent-rated ABAS for the Social Subscale was M = 0.04 and SD = 1.85, and for the Social Composite was M = −.13 and SD = 3.47; and (3) teacher-rated PDDBI for the Social Approach was M = 1.91 and SD = 7.12, and for the Social Pragmatic Problems was M = 1.73 and SD = 8.52.

Overall, changes in cognitive scores on the DAS were negatively associated with independent assessors’ ratings on the ADOS (r = −.2, p = .03) and teachers’ ratings on the Social Pragmatic Problems subscale of the PDDBI (r = −.2, p = .01). Changes in cognitive scores on the DAS were not significantly associated with parents’ ratings on the Social Subscale of the ABAS (r = −.05, p = .59), the Social Composite of the ABAS (r = −.08, p = .39), or teachers’ ratings on the Social Approach Behaviors subscale of the PDDBI (r = .00, p = .95). In addition, changes in children’s VIQ and NVIQ on the DAS were not significantly associated with any measure of social functioning.

Discussion

This study examined associations between gains in cognitive ability and social functioning after participating in a year-long behaviorally based comprehensive intervention program, for children with ASD. Changes in cognitive ability were modestly correlated with changes in ADOS severity, and improvements on one teacher-rated measure of socialization (social pragmatic problems). There was no association between changes in cognitive ability and parent-rated measures of social functioning. Several limitations should be considered when interpreting these results.

First, the small correlations between cognitive and social functioning may be due to measurement selection. Previous studies of the STAR program demonstrated improvements in children’s social skills using the Autism Screening Instrument for Educational Planning–Second Edition (ASIEP-2; Krug et al., 1993) that measured children’s social interactions with adults (Arick et al., 2003). The Social Interaction Assessment component of the ASIEP-2 was designed to elicit individual social responses in a controlled setting and captures one distinct component of social functioning. Because social functioning encompasses several domains of social skills and with the limited number of psychometrically sound and ecologically valid measures of social functioning presently used in the field, it was challenging to select comprehensive measures of social functioning for a large heterogeneous group of children with ASD. While many social behaviors are part of the STAR curriculum, we were unable to capture global changes of social functioning through independent raters’, teachers’, and parents’ ratings despite using some of the most psychometrically sound measures possible for large-scale use in a heterogeneous group. These observations highlight that outcome measures should relate to the behaviors targeted for intervention, and should have validity in terms of direct ties to the intervention (Kasari, 2002). Other assessment procedures such as the ASIEP-2, students’ IEPs, and STAR student learning profiles as well as direct observation of children’s play skills may be more directly tied to the STAR intervention and thereby more aptly suited to capture any changes in children’s social functioning. Although the utilization of direct observation measures may be more aligned to curricular areas of the STAR program (e.g. turn-taking, play, engagement, etc.), conducting live observations in schools is quite challenging and costly for large randomized controlled trials and were not used in this study. Future studies should consider using a time series in which direct observations are conducted for a randomly selected subsample, so that data collection is practical and manageable for a large randomized controlled field trial.

In addition to measurement, the findings may be due to implementation-related issues such as fidelity, staffing, climate, and so on (see Mandell et al., 2013). The STAR program was developed and tested in controlled settings, but the conditions in the AIMS were naturalistic. There were a range of classrooms, with teachers who differed in autism training and experience, as well as challenges in consistent staffing and buy-in, to name a few. Some classrooms resembled ideal conditions and others did not. Although the purpose of AIMS was to train school district employees (i.e. teachers and classroom assistants) to incorporate and utilize the components of the STAR program into their autism support classrooms and curriculum, we encountered a number of barriers to implementation (briefly listed above) that may have impacted the results of this study.

Another limitation is that teachers’ participation in the study as interventionists may have influenced their completion of outcome measures. Teachers implemented and were the only school-based respondents on students’ progress, in large part because they were the only school-based respondents who could accurately report on children’s outcomes. It is difficult to use blind raters to independently assess children’s social functioning without using direct observational tools. In an attempt to partially ameliorate this issue, parents were also asked to complete questionnaires on children’s social functioning, and independent assessors evaluated children on the ADOS in addition to teachers’ reports. Multiple informants may be critical in evaluating outcomes in studies such as these, where the interventionist is also a reporter of progress. Unfortunately, reports from the multiple informants in this case did not coalesce, as parents reported few changes in their child’s social functioning, while teachers reported some progress. It remains important, however, to have this information from multiple sources in order to better understand whether reported improvements by one individual represent true progress. Finally, although we had a large sample of children with ASD, there were a number of missing surveys from parents and teachers and some missing ADOS evaluations. We were also unable to access (1) STAR student assessment tools that may inform children’s progress or (2) children’s IEPs to determine whether their IEP goals matched the targeted social domains included in the STAR curriculum.

Despite these limitations, there are some important implications of these findings. First, although behaviorally based interventions that target academic and cognitive ability have resulted in improvements in social functioning, socialization remains a challenge. In this study, social functioning did not improve even when the intervention resulted in cognitive gains, which suggests that we have more work to do in improving social skills and possibly need more targeted interventions that address social functioning. Individualized intervention programs that capitalize on children’s opportunities to socialize with peers at school may be more successful in enhancing children’s global social functioning.

Second, teachers’ reports of improved social skills among children with ASD who participated in the STAR program (where fundamental social skills, such as initiating, requesting, turn-taking, sharing, and so on, were directly addressed as part of the curriculum) were associated with gains in cognitive scores. However, parents’ and independent assessors’ reports of children’s social functioning scores showed relatively little change over the school year and were not correlated with gains in cognitive scores. These results suggest that on average children with ASD did not improve in the spontaneous application and generalization of learned skills outside of the STAR context. These results are supported by the existing literature, which suggests that some children with ASD acquire new social skills after treatment, but these skills often do not generalize to other contexts (Rao et al., 2008; Williams-White et al., 2007). The lack of agreement among measures and between raters may be expected considering the different contexts in which children were observed (e.g. home, school, structured and standardized assessment), the distinctive aspects of social functioning captured by each of these measures, and the lack of home–school coordination.

Finally, because improving social functioning is an increasingly common goal of many interventions, these data suggest a strong need for cost-effective, ecologically valid, and psychometrically sound measures of socialization for large-scale use in the field. Future intervention studies examining improvements in social functioning should carefully consider utilizing multiple informants and ecologically valid measures that can more fully assess generalization of intervention effects in this domain.

Footnotes

Acknowledgements

We thank the children, parents, schools, teachers, and classroom staff who participated, and the research associates who contributed countless hours of assessments, intervention, supervision, and data collection: Sujie Shin, Erica Reisinger, Andrea Sherman, Rukiya Wideman, Margaret Downey, Aubyn Stahmer, Melanie Pellachia, Megan Carolan-Tomkinson, Emily Cross, Laura Macmullen, Jessica Dodge, and Shawn Gilroy.

Funding

This study was drawn from the Philadelphia Autism Instructional Methods Study, which was supported by NIMH grant 1R01MH083717-01A1 and IES grant R324A08195. During the time of this study, the first author was funded by an Autism Science Foundation postdoctoral fellowship award.

References

Arick

Loos

Falco

. (2004) The STAR Program: Strategies for Teaching Based on Autism Research. Austin, TX: PRO-ED.

Arick

Young

Falco

. (2003) Designing an outcome study to monitor the progress of students with autism spectrum disorders. Focus on Autism and Other Developmental Disabilities 18: 75–87.

Cohen

Sudhalter

(2005) The PDD Behavior Inventory. Lutz, FL: Psychological Assessment Resources, Inc.

Cunningham

(2012) Measuring change in social interaction skills of young children with autism. Journal of Autism and Developmental Disorders 42: 593–605.

Dawson

Rogers

Munson

. (2010) Randomized, controlled trial of an intervention for toddlers with autism: the Early Start Denver Model. Pediatrics 125: e17–e23.

Elliott

(2007) Differential Ability Scales–Second Edition (DAS-II). San Antonio, TX: Harcourt Assessment, Inc.

Fombonne

(2010) Young preschoolers with autism: evidence for efficacy of early intervention. Pediatrics. Available at: http://pediatrics.aappublications.org/content/125/1/e17.abstract/reply#pediatrics_el_49749

Gotham

Pickles

Lord

(2009) Standardizing ADOS scores for a measure of severity in autism spectrum disorders. Journal of Autism and Developmental Disorders 39: 693–705.

Harrison

Oakland

(2003) Adaptive Behavior Assessment System. 2nd ed. San Antonio, TX: Harcourt Assessment, Inc.

10.

Kasari

(2002) Assessing change in early intervention programs for children with autism. Journal of Autism and Developmental Disorders 32: 447–461.

11.

Kasari

Locke

Gulsrud

. (2011) Social networks and friendships at school: comparing children with and without autism. Journal of Autism and Developmental Disorders 41: 533–544.

12.

Kasari

Rotheram-Fuller

Locke

. (2012) Making the connection: randomized controlled trial of social skills at school for children with autism spectrum disorders. Journal of Child Psychology and Psychiatry, and Allied Disciplines 53: 431–439.

13.

Koegel

Schreibman

Good

. (1989) How to Teach Pivotal Behaviors to Autistic Children. Santa Barbara, CA: University of California.

14.

Krug

Arick

Almond

(1993) Autism Screening Instrument for Educational Planning. 2nd ed. Austin, TX: PRO-ED.

15.

Lord

Risi

Lambrecht

. (2000) The autism diagnostic observation schedule-generic: a standard measure of social and communication deficits associated with the spectrum of autism. Journal of Autism and Developmental Disorders 30: 205–223.

16.

Lovaas

(1987) Behavioral treatment and normal educational and intellectual functioning in young autistic children. Journal of Consulting and Clinical Psychology 55: 3–9.

17.

Mandell

Shin

Stahmer

. (2013) The role of treatment fidelity on outcomes during a randomized field trial of a school-based autism intervention. Autism. Epub ahead of print 16 April 2013. DOI: 10.1177/1362361312473666.

18.

Matson

Rivet

(2007) Social-skills treatments for children with autism spectrum disorders: an overview. Behavior Modification 31: 682–707.

19.

Rao

Beidel

Murray

(2008) Social skills interventions for children with Asperger’s syndrome or high-functioning autism: a review and recommendations. Journal of Autism and Developmental Disorders 38: 353–361.

20.

Reichow

Volkmar

(2010) Social skills interventions for individuals with autism: evaluation for evidence-based practices within a best evidence synthesis framework. Journal of Autism and Developmental Disorders 40: 149–166.

21.

Reichow

Wolery

(2009) Comprehensive synthesis of early intensive behavioral interventions for young children with autism based on the UCLA young autism project model. Journal of Autism and Developmental Disorders 39: 23–41.

22.

Rogers

Vismara

(2008) Evidence-based comprehensive treatments for early autism. Journal of Clinical Child & Adolescent Psychology (Special issue: Evidence-based psychosocial treatments for children and adolescents: a ten year update) 37: 8–38.

23.

Schreibman

(2000) Intensive behavioral/psychoeducational treatments for autism: research needs and future directions. Journal of Autism and Developmental Disorders 30: 373–378.

24.

Smith

(2001) Discrete trial training in the treatment of autism. Focus on Autism and Other Developmental Disabilities 16: 86–92.

25.

Spreckley

Boyd

(2009) Efficacy of applied behavioral intervention in preschool children with autism for improving cognitive, language, and adaptive behavior: a systematic review and meta-analysis. The Journal of Pediatrics 154: 338–344.

26.

Stahmer

(1995) Teaching symbolic play skills to children with autism using pivotal response training. Journal of Autism and Developmental Disorders 25: 123–141.

27.

Williams-White

Keonig

Scahill

(2007) Social skills development in children with autism spectrum disorders: a review of the intervention research. Journal of Autism and Developmental Disorders 37: 1858–1868.