Preventing Academic Disengagement Through a Middle School–Based Social and Emotional Learning Program

Abstract

Behaviors that warrant school discipline (e.g., fighting, victimizing peers) is detrimental to school climate and the learning process. This study examines the effectiveness of preventing school disciplinary incidents in middle school through an experiential, social and emotional learning (SEL) program. A community youth development organization, two public middle schools in low-income communities and a local university collaborated to design and deliver the program to all seventh-grade students in social studies curricula. This article describes the design of the intervention and its effect on students’ suspensions, skipping class, and failing grades. The results of this study indicate that a school-based SEL service-learning program may reduce disciplinary incidents for middle school students. Other attitudes and skills, however, did not change significantly in the anticipated direction. This research demonstrates the effect of SEL curricula and service-learning programs embedded in school coursework. Implications for practice and research include understanding mechanisms of change in SEL processes.

Keywords

service learning experiential learning school discipline social and emotional learning middle school

Introduction

Many public schools in economically vulnerable communities are constrained by a lack of resources available to help students learn increasingly difficult academic concepts, while also preparing them for a successful launch into society. For resource-depleted schools, the primary mission of education is often complicated by the urgent need to address student behaviors, which can be exacerbated by the effects of poverty (Wadsworth et al., 2008). Poverty-related stress can manifest itself in adolescent internalizing and externalizing problems, deviant behavior, and school dropout (Wadsworth et al., 2008), further complicating the already difficult task of educating.

These types of stressors can create vicious cycles for vulnerable schools. Instead of proactively encouraging positive student behaviors, schools are left with little choice but to spend their limited time and resources addressing highly disruptive and oftentimes dangerous behaviors. On an individual level, paying the consequences of one’s actions can reduce time that a student remains in the classroom to learn. Fragmented attendance in the classroom further diminishes a student’s exposure to instruction, which may increase academic disengagement, causing students to fall farther behind academically (American Academy of Pediatrics, 2013). Lack of academic motivation may further exacerbate their participation in disruptive behaviors, creating a negative reinforcing loop of behaviors for schools to address.

The purpose of this study was to examine whether a universal social and emotional learning (SEL) curriculum delivered through a service-learning program could reduce school disciplinary incidents and increase academic engagement for middle school students. Specifically, this study included the following objectives: (a) work with an urban middle school to deliver an SEL program to an entire grade of seventh-grade students as part of their weekly social studies curricula, and (b) determine whether the intervention could reduce middle school disciplinary incidents and increase academic engagement, as found in previous replications of the program with high school students.

The intervention design tests a range of academic and social outcomes. First, this study tests whether the intervention could enhance academic efficacy, civic attitudes, and engagement in schools. To our knowledge, few SEL programs have been coupled with service learning and embedded in school curricula in this way. In addition, this study examines whether delivering the intervention in school curricula could reduce negative school behaviors when embedded in school curricula during the school day.

SEL as an Intervention to Prevent Negative School Behaviors

SEL is defined by the most widely used definition from the Collaborative for Academic, Social, and Emotional Learning (CASEL) and Zins, Bloodworth, Weissberg, and Walberg (2004) as follows:

SEL is a process for helping children develop the fundamental skills for life effectiveness. SEL teaches the skills we all need to handle ourselves, our relationships and our work effectively and ethically. These skills include recognizing and managing our emotions, developing caring and concern for others, establishing positive relationships, making responsible decisions, and handling challenging situations constructively and ethically. They are the skills that allow children to calm themselves when angry, make friends, resolve conflicts respectfully, and make ethical and safe choices. (Collaborative for Academic, Social, and Emotional Learning [CASEL], 2014, http://www.casel.org/social-and-emotional-learning/)

Similarly, Zins et al. (2004) define SEL as the process through which children and adolescents enhance their ability to incorporate thinking, feeling, and behaving to achieve important life tasks. These skills include recognizing and managing emotions, developing caring and concern for others, establishing positive relationships, making responsible decisions, and handling challenging situations constructively and ethically (Zins et al., 2004).

The American Academy of Pediatrics (2013) has recommended that early intervention programs identifying students at risk of suspensions and expulsions should also teach age-appropriate behaviors. SEL programs seem to align with this need. Previous research suggests that SEL programs can influence the development of appropriate emotion regulation techniques and positive peer interactions, encouraging young people to follow positive and productive trajectories (Anderson-Butcher, Stetler, & Midle, 2006; Durlak, Weissberg, Dymnicki, Taylor, & Schellinger, 2011; Zins & Elias, 2006).

In schools, SEL interventions have been used to reduce a variety of negative behaviors and increase positive development for youth. In a meta-analysis of SEL programs, successful programs included components that (a) use a connected or coordinated set of activities to achieve objectives related to skill development, (b) use active forms of learning to help youth learn new skills, (c) have at least one component devoted to developing personal or social skills, and (d) targets specific SEL skills rather than targeting skills in general terms (Durlak et al., 2011).

Social and emotional skills taught and demonstrated during childhood have been tied to numerous positive behaviors and qualities. A 3-year longitudinal study of a universal SEL program for elementary school students indicated that students who engaged in the program experienced preventive effects on population-level rates of aggression, social competence, and school engagement (Conduct Problems Prevention Research Group, 2010). In adolescence, there is evidence of indirect effects on the reduction of anxiety and depression or emotional distress (Durlak et al., 2011; Neil & Christensen, 2009); decreased incidence of conduct problems, such as drug use (Diekstra, 2008; Durlak et al., 2011); and improved performance in school (Diekstra, 2008; Durlak et al., 2011; Wilson, Lipsey, Jo, & Mark, 2006).

Although there are many different types of SEL programs, experiential activities embedded in service learning within a positive youth development framework may be one way to build social and emotional skills in early adolescence (Chung & McBride, 2015). Service learning is a pedagogy that is used to describe structured learning opportunities that engage students in service from research to reflection. These types of experiences have an explicit student learning focus integrated with an equally fundamental community focus, providing hands-on opportunities for students to research, practice, and reflect on their contributions to a community (Bringle, Clayton, & Hatcher, 2013; Eyler & Giles, 1999; Felten & Clayton, 2011; McBride, Pritzker, Daftary, & Tang, 2006).

Multiple studies have found that youth who participated in service learning were less likely to be referred for disciplinary measures and more likely to have increased standardized tests scores when compared with those who did not participate in such programming (Billig, 2000). In addition, studies have shown that school-based service learning increases grades, attendance (e.g., Bryant, Shdaimah, Sander, & Cornelius, 2013), academic interest, school engagement, and encourages stronger classroom task-engagement and skills (Schmidt, Shumow, & Kackar, 2006). Classroom-based programs using a positive youth development framework to incorporate service learning can also reduce school disciplinary incidents, especially for students who are most at risk (Allen & Philliber, 2001).

Despite evidence to support the development of SEL in children and adolescents (Durlak et al., 2011), many teachers and school administrators indicate that there are not enough resources or expertise to address the unique needs of individuals who exhibit disruptive school behaviors (Maras, Splett, Reinke, Stormont, & Herman, 2014). Changing the school environment may initially require reinforcements that are not available within the school. Mobilizing community resources that proactively support students’ social and emotional development can help address some of these challenges (Epstein, 1995, 2001). According to D’Agostino (2013), partnerships or collaborations can identify and prioritize student needs and strategize evidence-based ways to meet them.

The Intervention

Wyman’s Teen Outreach Program® (TOP) is a positive youth development program teaching social and emotional skills through curriculum and service learning. Research on the effectiveness of TOP has been published as early as the 1990s on topics of school suspensions and teen pregnancy (e.g., Allen, Philliber, & Hoggson, 1990; Allen & Philliber, 1991, 2001). Few studies, however, have examined how the program works when embedded within school curricula. In addition, few evaluations have been conducted in the past decade focusing on middle school youth in particular. This is important, as the intervention has been listed on a number of evidence-based practice lists from the Substance Abuse and Mental Health Services Administration’s (SAMHSA) National Registry of Evidence-Based Practices (“Wyman’s Teen Outreach Program,” 2015) to the Promising Practices Network: Programs That Work (2013). Most of these lists, however, use data from studies conducted over 30 years ago.

Overview of the Study

This article reports on outcomes of a 9-month intervention conducted over the course of 1 academic year. To our knowledge, this is one of the first studies using a quasi-experimental design to determine whether TOP, when closely aligned with middle school curricula, can improve academic engagement for adolescents. This study answers the following research questions:

Research Question 1: What is the effect of Wyman’s TOP at improving academic engagement for middle school students when embedded in the school curricula?

Research question 2: What is the effect of Wyman’s TOP on student’s behavioral, academic, and civic attitudes?

It is hypothesized that skills learned during the intervention will help students to refrain from engaging in negative behaviors that lead to disciplinary incidents (e.g., fighting, disrupting class, skipping class) for students in the intervention school. In addition, we hypothesize that students in this experiential curriculum will have increasingly positive beliefs and attitudes toward school engagement and civic duties.

Method

The Intervention

Wyman’s TOP is an SEL curriculum embedded in a service-learning program (Chung & McBride, 2015). In this study, there were four collaborative entities necessary to implement the intervention: a community-based youth development organization, two middle schools, and a local university. The youth development organization provided access to the intervention, trained and supervised program facilitators, and provided program materials and replication expertise. The intervention school delivered the program to seventh-grade students through the required social studies curriculum. The comparison school was asked to participate based on its similarity in demographic composition and size and received “business as usual” followed by a delayed intervention. The university designed the school and community partnership, provided access to master’s-level social work students to serve as program facilitators in exchange for practicum credits, and developed instruments for program evaluation and intervention effect testing.

Researchers, the youth development organization, and school administrators from both the comparison and intervention school were included in meetings to discuss the implementation. Given previous research outcomes on the program and existing relationships with the youth development organization, school leadership were open to implementing the program in their schools. Agreements from the school were contingent upon the university to provide master’s-level social work students to deliver the programming and to orchestrate the community service with training, oversight, and program support from the youth development agency.

The intervention was implemented during social studies class periods once a week (45-60 min sessions) across the entire school year (32 weeks). Program sessions (also called “TOP Clubs”) were led by program facilitators. During this time, classroom teachers would turn over their class to the program facilitator and use this time for lesson planning or to watch as a passive observer. Lessons were led in an interactive, facilitator-led group discussion format on social and emotional skill development. Sessions included topics that were planned alongside existing social studies lessons, such as decision-making, building healthy relationships, communication skills, and issues of social and physical development. The program also required students to participate in 20-hr of school-based service learning designed to engage students in the planning, execution, and evaluation of their service-learning activities. Students designed activities that coordinated with their existing social studies curriculum.

Program facilitators for the intervention were chosen from a pool of qualified master’s of social work (MSW) degree candidates at the collaborating university. Prior to the academic year, facilitators attended a 40-hr, weeklong training led by trainers from the youth development organization. A training manual and curriculum guide, including lesson plans for all sessions, was given to all facilitators for reference. Ongoing training, supervision, and professional development opportunities were provided throughout the year.

According to the fidelity requirements of the intervention, program clubs should meet weekly over the course of 9 months, each club provides a minimum of 25 curriculum-focused peer group meetings, students should complete a minimum of 20 hr of service learning, and meetings should be facilitated by a trained facilitator. Students in the intervention school participated in approximately 21.28 hr (SD = 2.89) of community service during the academic year and received approximately 31 program sessions (µ = 31.04, SD = 4.22), satisfying the fidelity requirements of the intervention.

Setting and Sample

The study took place in two public, Midwestern middle schools during 2012-2013. The protocol and design of this study was institutional review board (IRB)–approved by Washington University in St. Louis (No. 201207120). The intervention school included 661 sixth- to eighth-grade students in the 2012-2013 school year. A majority of the students (69.4 %) were eligible for free or reduced-price lunch. Racially, 87.4% of students in the intervention school were African American and 7.6% of students were White. The comparison school spanned Grades 7 to 8, with 1,028 students enrolled in 2012-2013 school year. A majority of the students (92.7%) were eligible for free or reduced-price lunch. In regard to race, 97.6% of students from the comparison school were African American.

All seventh-grade students were eligible to participate in the study. Parental consent for research was received for 148 students in the intervention school (71.8%) and 151 seventh-grade students in the comparison school (46.7%). Both samples from each school were mostly female (intervention school: 56.3%, comparison school: 60.4%) and African American (intervention school: 85%, comparison school: 95%). Chi-square analyses indicated no significant differences in gender, race, or living situation for these samples (Table 1).

Table 1.

Sample Description.

Demographic Characteristics	Intervention school		Comparison school
Demographic Characteristics	n	%	n	%
Total Participants in Each Sample	112	100	106	100
Gender
Males	49	43.8	42	39.6
Females	63	56.3	64	60.4
Race
African American	82	78.9	95	91.4
White	9	8.7	1	1.0
Hispanic/Latino	1	1.0	—	—
Asian or Pacific Islander	—	—	—	—
Multiethnic	5	4.8	4	3.9
Native American/Alaskan Native	1	1.0	1	1.0
Other	6	5.8	2	1.9
Mother’s education
High school diploma or less	16	18.4*	26	36.6*
Some college or more	71	81.6*	45	63.4*
Father’s education
High school diploma or less	21	31.8*	32	58.2*
Some college or more	45	68.2*	23	41.8*
Household family composition
Two-parent	61	59.2	55	52.9
Single-parent	33	32.0	39	37.5
Guardian/other	9	8.7	10	9.6

p < .05.

School faculty (e.g., teachers) from the intervention school had an average of 14 years of experience and 62% of staff had advanced degrees. The comparison school had an average of 17.9 years of experience and 70.3% of staff had advanced degrees. The intervention facilitators (n = 4) were all new to the intervention school setting but had prior experience with children and youth in other settings.

Method

At the beginning of the school year, the research team worked with facilitators, teachers, and school administrators to inform and recruit seventh-grade participants for the study. Pre- and posttest surveys were distributed to participating students in the fall and spring by the university research staff. Research staff explained the survey components to students and remained present during the survey administration period to answer questions. Of those who consented, 112 students completed both the pre- and posttest from the intervention school (75.6%). In the comparison school, 106 students completed both the pre- and posttests (70.2%).

Measures and Analytic Strategy

Surveys were used to understand changes in student attitudes regarding their belonging and engagement in school, self-worth, perceptions of community safety, and civic duty. Other constructs on prosocial bonding and student autonomy were also measured; however, these scales had poor Cronbach’s alpha scores (α < .70) and therefore were omitted from the analysis.

School belonging

Questions regarding school belonging were modified from items on the Psychological Sense of School Membership Scale (Goodenow, 1993; α = .79). Students were prompted with five statements: I feel like a real part of my school; Sometimes I feel as if I don’t belong here; I wish I were in a different school; I feel proud to belong to this school; and I am happy to be at this school. Likert-type response ratings ranged from 1 (not at all) to 5 (likely true). Negatively worded items were reversed while coding. The mean was taken across the sum of scale items.

Emotional engagement

Emotional engagement was assessed through a subscale (α = .78) of the Engagement versus Disaffection With Learning (EvsD) Measure (Skinner & Belmont, 1993): My classes are fun; I enjoy learning new things in my classes; When we work on something in class, I feel interested; When I am in class, I feel good; and In my classes, I work as hard as I can. Students rated items on a scale from 1 (never) to 5 (a lot), and the scale was summed and then averaged, with higher values indicating greater agreement.

Behavioral engagement

The Behavioral Engagement subscale of the EvsD (Skinner & Belmont, 1993) was used to measure students’ active participation in learning (α = .70). Survey items asked students to rate the following statements from 1 (never) to 5 (a lot): I pay attention in my classes; When I’m in class, I participate in class discussions; When I’m in class, I listen very carefully; I try hard to do well in school; and When we work on something in class, I get involved. Scale scores were then averaged, with higher scores indicating greater agreement.

Academic efficacy

Academic efficacy was measured through the Patterns of Adaptive Learning Scale (PALS; Midgley et al., 2000). PALS ask students to rate themselves on questions that gauge their perception of their ability to complete and learn difficult schoolwork, master the skills taught in school, and discern and solve difficult concepts (α = .79). Students were asked to respond with 1 (none) to 5 (a lot) to the following prompts: I’m certain I can master the skills taught in class this year; I’m certain I can figure out how to do the most difficult class work; I can do almost all the work in class if I don’t give up; Even if the work is hard, I can learn it; and I can do even the hardest work in this class if I try. Scores were averaged, with scores near 5 indicating higher self-efficacy.

Civic duty

Civic duty was measured using a 12-item scale on civic duty by Zaff, Boyd, Li, Lerner, and Lerner (2010) with an Cronbach’s alpha level of α = .83. The scale measured student attitudes on statements on a 5-point scale from 1 (not important) to 5 (very important). Scale items were summed then averaged for interpretation, with higher scores reflecting student attitudes suggesting that civic duty was very important. Statements included phrases such as the following: I believe I can make a difference in my community, helping to make sure all people are treated fairly, speaking up for equality, and it is important for me to contribute to my community and society.

School disciplinary events

Students were asked to report academic behaviors (failing grades/courses) and social behaviors (skipping class without permission and getting suspended) in both the pre- and posttest surveys. Students were asked, “Have you [obtained any failing grades; skipped school without permission; been suspended] in the past year?” Responses to these questions were dichotomously coded “yes/no.” If a student responded yes, he or she was asked to report how many times.

Logistic regression analysis in SAS 9.4 was used to determine whether there were any significant differences in the intervention and comparison group, while controlling for pretest responses, parent’s education, household structure, and gender.

Results

Social, Emotional, and Civic Attitudes

Overall, the differences between pre- and posttest surveys were not significant in the hypothesized direction. In fact, attitudes for certain scales were marked less favorably between pretest and posttest for the intervention students. The two areas where students from the intervention school were significantly different from students in the comparison group were areas of emotional engagement and academic efficacy.

Emotional engagement

At pretest, the mean score for students in the intervention school was 18.43 and dropped about 2 points at posttest to 16.09. Students from the comparison school only dropped about 1 point from 17.30 in the pretest to 16.48 in the posttest (t = 2.61, p < .05; Table 2). These scores suggest that students from the intervention school agreed less with the emotional engagement scale in the posttest than they did in the pretest when compared with their peers in the comparison school.

Table 2.

School Comparisons of Pre- and Postscales (Full Sample).

Variable	α	Pretest				Posttest				Difference		t
		Intervention		Comparison		Intervention		Comparison		Intervention	Comparison
		M	SD	M	SD	M	SD	M	SD	M	M
School belonging	.8	3.79	1	2.78	0.9	3.27	1.03	2.53	0.9	−0.54	−0.27	1.8
Emotional engagement in school	.8	3.69	0.8	3.46	0.8	3.22	0.93	3.3	0.9	−0.47	−0.15	2.61*
Behavioral engagement in school	.7	4.28	0.5	4.2	0.6	3.92	0.8	4.02	0.8	−0.34	−0.16	1.9
Academic self-efficacy	.8	4.31	0.6	4.2	0.7	4.03	0.84	4.16	0.7	−0.29	−0.05	2.25*
Civic duty	.8	4.04	0.6	3.7	0.7	3.76	0.71	3.59	0.7	−0.27	−0.13	1.5

p < .05.

Academic efficacy

Academic efficacy scales referred to students’ self-ratings on questions that gauge student’s beliefs on their ability to complete and learn difficult schoolwork, master the skills taught in school, and discern and solve difficult concepts. The highest score an individual could get on this scale is 25 points, with higher values indicating more agreement with the questions being asked. Overall, the scores on both the pre- and posttests were high for both schools (Table 2). Students in the intervention school decreased their ratings of efficacy by 1.43 points compared with the 0.27 increase in the comparison group (t = 2.25, p < .05).

There were no significant changes between the intervention and comparison group for attitudes on school belonging, behavioral engagement, or civic duty.

Student Behaviors

Control variables were chosen for regression analysis by examining existing literature on risk and protective factors for adolescents (Zolkoski & Bullock, 2012). Environmental factors, such as parental education and household structure, as well as demographic factors such as gender, were examined. Previous history of the adolescent’s targeted behavior (baseline pretest scores) was also included to adjust for the systemic bias and error variance (Barry, 2015). At baseline, there were no significant differences in the number of failing grades, suspensions, and classes skipped reported by students in the intervention and comparison school. By posttest, however, logistic regression results indicated that students who participated in the intervention were 62% less likely than the comparison group to skip class without permission (p < .05, χ² = 24.39, β = −0.96, odds ratio [OR] = .38, 95% confidence interval [CI] = [0.16, 0.92]) and 78% less likely to report failing grades (p < .05, χ² = 38.20, β = −1.50, OR = .22, 95% CI = [0.09, 0.54]) even while controlling for previous history of the outcome behavior, parent’s education, household structure, and gender (Table 3). The results suggest that students who went through the intervention were less likely to engage in behaviors that are correlated with negative academic outcomes and behaviors.

Table 3.

Logistic Regression for Behavioral Outcomes—Students Receiving Wyman’s Teen Outreach Program.

Outcome	Model statistics		Wyman’s Teen Outreach Program
Outcome	Model χ²	Max rescaled R²	β	SE	OR	95% CI
Failing courses	41.14	.31	−0.41	.42	.67	0.29	1.53
Failing grades	38.20*	.32	−1.50*	.46	.22	0.09	0.54
Skipping class	24.39*	.21	−0.96*	.45	.38	0.16	0.92
Suspensions	47.02	.39	−0.88	.47	.41	0.16	1.05

Note. Control items include the following variables: Gender, mother and father’s education, family composition, and previous history of the listed behavior. Coding was conducted as follows: Gender (1 = female), mother/father’s education (0 = less than high school; 1 = high school graduation or more), Intervention School (0 = Comparison School; 1 = Treatment School), household composition (0 = two-parent family; 1 = single parent; 2 = guardian/other), and previous history of X behavior (0 = no; 1 = yes). OR = odds ratio; CI = confidence interval.

p < .05.

Discussion

Wyman’s TOP, an SEL curriculum with experiential service-learning components, has been delivered and evaluated in a variety of contexts, though primarily in after-school, extracurricular high school settings (Chung & McBride, 2015). Few studies, however, have examined the effectiveness of this SEL service-learning program when delivered in middle school. This lack of nuanced research is important as the program continues to be implemented in a variety of school settings. This study’s focus was to understand whether delivering Wyman’s TOP within middle school curricula during the school day could prevent negative disciplinary outcomes for youth.

Results from this study suggests that adolescents who participated in the yearlong intervention experienced statistically significant reductions in failing grades and skipping classes compared with the comparison group, even while controlling for a variety of demographic factors. Although adolescents from the comparison group did not differ significantly in initial rates of failing grades and skipping classes without permission, there were significant reductions in these negative school behaviors for adolescents from the intervention group by posttest. These outcomes support the hypothesis that the intervention may have positive effects on behaviors related to academic success.

Limitations

Limitations of this study encompass issues with sampling and the occurrence of major unanticipated events. First, the sample included students whose parents actively consented for them to participate in the study. Therefore, students who did not participate in the research sample may represent those who are already disengaged and possibly most at risk. Future assessments of the program should be made to understand this sample and how they compare with students who elected to participate. In relation to the limitation of sampling, there may also be issues with social desirability present in this study. For instance, when students are asked questions about negative school behaviors, they may have answered in a way that they can be perceived more favorably, as they understand the intervention was focused on reducing these types of outcomes. We tried to minimize these effects by making responses anonymous in the survey administration process. Furthermore, the lack of randomization poses another limitation in regard to the generalizability of the results.

Another unforeseeable incident that may influence our findings is that the comparison school lost its accreditation shortly after the first intervention year. Therefore, underlying issues within the school may have affected students in ways that are unaccounted for in the design of this study. Although the comparison school was undergoing various accreditation issues, there are also aspects of the intervention school that may have altered our findings in unintended ways. Like many schools, there were a number of positive behavioral interventions being implemented. For instance, at the time of this study and unbeknownst to the researchers, the intervention school was also implementing strategies to reduce tardiness among students. It is unclear what effect these programs had on the intervention students, but it is important to point out that many school-based studies experience similar issues with conflation. Although the data are informative for the case study, they should not be considered generalizable to a larger context.

Examining the Findings

Why were there positive effects on academic engagement in the form of reduced failing grades? The intervention is highly experiential. Instead of lecture-based classroom sessions, lessons are tied to existing school curriculum through opportunities for rich group discussion and hands-on learning. Even the additional service-learning projects required by TOP were closely aligned with school coursework. Students were asked to participate in the planning, development, and execution of service-learning activities. These types of activities are considered to be critical to the success of service learning. In addition, when service learning is tied to opportunities for reflection and class time, as it was in the TOP intervention, it enhances the academic climate (Levesque-Bristol, Knapp, & Fisher, 2010). We conjecture that the hands-on activities connected with cognitive learning, as conducted in the intervention, were critical to engage students in the educational process. These outcomes have also been seen in other studies where service learning has been suggested to be an effective method of increasing academic performance (Celio, Durlak, & Dymnicki, 2011).

Contrary to our second hypothesis, however, behavioral, emotional, and civic attitudes did not differ significantly from pre- to posttest. The results of the student attitudes in our study are perplexing. The attitudes compared with the self-reported behaviors measured in this study seem to contradict each other. Although some behaviors, such as getting failing grades in school and skipping classes without permission, improved in the intervention school, attitudes on behavioral and emotional engagement did not. This paradox might be explained by a variety of possibilities related to school-based interventions and adolescent development. For instance, perhaps participation in this intervention made adolescents realize that their initial self-ratings in the pretest were unrealistic. It may be possible that middle school students increased in understanding of what it meant to be behaviorally engaged after receiving targeted instruction on how to engage in school. This type of measurement bias has been described as response-shift bias (Howard & Dailey, 1979).

Response-shift bias is unlike social desirability bias, when survey participants respond in a manner that would be viewed favorably by others. According to Howard & Dailey (1979), response-shift bias is a consequence of many interventions that aim to change a subject’s awareness or understanding of the variable being measured. It refers to a type of measurement bias where survey respondents rate themselves lower in the posttest because as a result of participating in an intervention, they learn their initial understanding of the question was flawed. Evidence of this bias has been noted in evaluations of education programs teaching assertiveness and learning basic helping skills (Howard & Dailey, 1979). Given the educational setting and age group, the use of retrospective pretests may remove this response-shift bias. These types of pretests demonstrate significantly greater validity than the measure of change used in traditional self-report pretests. Moreover, it would be helpful to use objective measures of change, such as grades or triangulation with teachers, peers, and parents.

Inconsistencies in attitudes on social and emotional outcomes are not new in SEL research. Whitehurst (2016) suggests that this has less to do with the attitudes themselves than it does with the nature of traits we measure to gauge these SEL constructs. In his Brookings Institution report, Whitehurst argues that the conceptualization and measurement of these soft skills are conducted in a way that psychologists approach human personality and therefore schools have difficulty influencing differences among students (Whitehurst, 2016). He gives examples of charter schools that have effectively raised student achievement by focusing on character development but have had no or reverse impact on students’ self-reported soft skills. Our findings are consistent with many other attitudes which indicate that studying the measurement of SEL is just as, if not more, important than implementing the interventions (Cohen, 2006; Humphrey, 2013). Although this study only compared students from two schools, the lack of change and unexpected impact on social and emotional attitudes support the theory that there may be a wider issue regarding SEL measurement in adolescence.

Although there were inconsistencies between behaviors and attitudes, the results of this study did point to possible benefits for student behaviors. Students in this program reported that they were less likely to skip class without permission and attain failing grades after a year of engaging in the intervention curriculum. And yet, though these behaviors improved, attitudes toward school engagement actually decreased. As previously discussed, these changes could likely be the result of response-shift bias or social desirability. These findings may also be indicative of processes that are inherent for early adolescent development. Given these paradoxical outcomes, future research is warranted. Consequent studies should examine how skills change during adolescence, refining the definition of constructs; whether outcomes from these types of interventions sustain beyond their grade level; and how to control for learning biases.

Conclusion

This study assessed the effects of delivering an SEL program utilizing service learning within middle school curricula. It updated existing studies on a popular youth intervention, the TOP, and provided a new perspective gauging whether the intervention was effective at reducing and preventing negative school behaviors, such as getting failing grades and skipping class. Although the intervention seemed to have positive effects on some behaviors, attitudes regarding behavioral and emotional engagement, academic self-efficacy, and civic responsibility did not seem to change in the desired direction. Further research is needed to understand the impact of SEL on this developmental phase, especially through service-learning curricula. Particular attention should be paid to sensitivity of design and measurement.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Author Biographies

Amanda Moore McBride is the Morris endowed dean and professor at the Graduate School of Social Work at the University of Denver. An international expert in civic and community engagement, she has authored or coauthored more than 70 books, special journal issues, and articles. Recent research focuses on higher education.

Saras Chung is a computational social scientist with the Brown School of Work at Washington University in St. Louis. Her research examines school-based initiatives to foster positive youth development and prevent negative mental health outcomes using systems science.

Anne Robertson is an adjunct professor at the University of Denver Graduate School of Social Work. Her research interests include social work and educational interventions and other interdisciplinary efforts focused on child and family well-being.

References

Allen

J. P.

Philliber

(1991, January). Evaluating why and how the Teen Outreach Program works: Years 3-5 of the Teen Outreach National Replication (1986/87-1988/89). New York, NY: Association of Junior Leagues.

Allen

J. P.

Philliber

(2001). Who benefits most from a broadly targeted prevention program? Differential efficacy across populations in the Teen Outreach Program. Journal of Community Psychology, 29, 637-655.

Allen

J. P.

Philliber

Hoggson

(1990). School-based prevention of teen-age pregnancy and school dropout: Process evaluation of the national replication of the Teen Outreach Program. American Journal of Community Psychology, 18, 505-524.

American Academy of Pediatrics. (2013). Out-of-school suspension and expulsion. Pediatrics, 112, 1206-1209. doi:10.1542/peds.112.5.1206

Anderson-Butcher

Stetler

E. G.

Midle

(2006). A case for expanded school-community partnerships in support of positive youth development. Children & Schools, 28, 155-163. doi:10.1093/cs/28.3.155

Barry

(2015, September 16). Data analysis of pre-post study designs. Cornell Statistical Consulting Unit. Retrieved from https://www.cscu.cornell.edu/news/statnews/stnews79.pdf

Billig

S. H.

(2000). Research on K-12 school-based service-learning. Phi Delta Kappan, 81, 658-665.

Bringle

R. G.

Clayton

P. H.

Hatcher

J. A.

(2013). Research on service learning: An introduction. In Clayton

P. H.

Bringle

R. G.

Hatcher

J. A.

(Eds.), Research on service learning: Conceptual frameworks and assessment (pp. 3-25). Sterling, VA: Stylus Publishing.

Bryant

V. C.

Shdaimah

Sander

R. L.

Cornelius

L. J.

(2013). School as haven: Transforming school environments into welcoming learning communities. Children and Youth Services Review, 35, 848-855. doi:10.1016/j.childyouth.2013.02.001

10.

Celio

C. I.

Durlak

Dymnicki

(2011). A meta-analysis of the impact of service-learning on students. Journal of Experiential Education, 34, 164-181.

11.

Chung

McBride

A. M.

(2015). Social and emotional learning in middle school curricula: A service learning model based on positive youth development. Children and Youth Services Review, 53, 192-200. doi:10.1016/j.childyouth.2015.04.008

12.

Cohen

(2006). Social, emotional, ethical, and academic education: Creating a climate for learning, participation in democracy, and well-being. Harvard Educational Review, 76, 201-237.

13.

Collaborative for Academic, Social, and Emotional Learning, (2014). What is social and emotional learning? Retrieved from http://www.casel.org/social-and-emotional-learning/

14.

Conduct Problems Prevention Research Group. (2010). The effects of a multiyear universal social-emotional learning program: The role of student and school characteristics. Journal of Consulting and Clinical Psychology, 78, 156-168. doi:10.1016/j.biotechadv.2011.08.021

15.

D’Agostino

(2013). Collaboration as an essential school social work skill. Children & Schools, 35, 248-251.

16.

Diekstra

R. F. W.

(2008). Evaluation of effectiveness of school-based social and emotional education programmes worldwide. In Fundacion Marcelino Botin (Ed.), Social and emotional education: An international analysis (pp. 255-284). Madrid, Spain: Fundacion Marcelino Botin.

17.

Durlak

J. A.

Weissberg

R. P.

Dymnicki

A. B.

Taylor

R. D.

Schellinger

K. B.

(2011). The impact of enhancing students’ social and emotional learning: A meta-analysis of school-based universal interventions. Child Development, 82, 405-432. doi:10.1111/j.1467-8624.2010.01564.x

18.

Epstein

J. L.

(1995). School/family/community partnerships: Caring for the children we share. Phi Delta Kappan, 76, 701-716.

19.

Epstein

J. L.

(2001). School, family, and community partnerships: Preparing educators and improving schools (1st ed.). Boulder, CO: Westview Press.

20.

Eyler

Giles

D. E.

(1999). Where’s the learning in service-learning? San Francisco, CA: Jossey-Bass.

21.

Felten

Clayton

P. H.

(2011). Service-learning. New Directions for Teaching & Learning, 128, 75-85. doi:10.1002/tl

22.

Goodenow

(1993). The psychological sense of school membership among adolescents: Scale development and educational correlates. Psychology in the Schools, 30, 79-90.

23.

Howard

G. S.

Dailey

P. R.

(1979). Response-shift bias: A source of contamination of self-report measures. Journal of Applied Psychology, 64, 144-150. doi:10.1037/0021-9010.64.2.144

24.

Humphrey

(2013). Social and emotional learning: A critical appraisal (1st ed.). London, England: Sage.

25.

Levesque-Bristol

Knapp

T. D.

Fisher

B. J.

(2010). The effectiveness of service-learning: It’s not always what you think. Journal of Experiential Education, 33, 208-224. doi:10.5193/JEE33.3.208

26.

Maras

M. A.

Splett

J. W.

Reinke

W. M.

Stormont

Herman

K. C.

(2014). School practitioners’ perspectives on planning, implementing, and evaluating evidence-based practices. Children and Youth Services Review, 47, 314-322. doi:10.1016/j.childyouth.2014.10.005

27.

McBride

A. M.

Pritzker

Daftary

Tang

(2006). Youth service: A comprehensive perspective. Journal of Community Practice, 14, 71-90. doi:10.1300/J125v14n04

28.

Midgley

Maehr

M. L.

Hicks

Roeser

Urdan

Anderman

E. M.

… Middleton

M. J.

(2000). Manual for the Patterns of Adaptive Learning Scales. Ann Arbor: University of Michigan. Retrieved from http://www.umich.edu/~pals/PALS2000_V12Word97.pdf

29.

Neil

A. L.

Christensen

(2009). Efficacy and effectiveness of school-based prevention and early intervention programs for anxiety. Clinical Psychology Review, 29, 208-215. doi:10.1016/j.cpr.2009.01.002

30.

Promising Practices Network: Programs That Work. (2013). Wyman’s Teen Outreach Program. Retrieved from http://www.promisingpractices.net/program.asp?programid=14

31.

Schmidt

J. A.

Shumow

Kackar

(2006). Adolescents’ participation in service activities and its impact on academic, behavioral, and civic outcomes. Journal of Youth and Adolescence, 36, 127-140. doi:10.1007/s10964-006-9119-5

32.

Skinner

E. A.

Belmont

M. J.

(1993). Motivation in the classroom: Reciprocal effects of teacher behavior and student engagement across the school year. Journal of Educational Psychology, 85, 571-581. doi:10.1037/0022-0663.85.4.571

33.

Wadsworth

M. E.

Raviv

Reinhard

Wolff

Santiago

C. D.

Einhorn

(2008). An indirect effects model of the association between poverty and child functioning: The role of children’s poverty-related stress. Journal of Loss & Trauma, 13, 156-185. doi:10.1080/15325020701742185

34.

Whitehurst

G. J.

“Russ.” (2016). Hard thinking on soft-skills. Brookings Institution. Retrieved from http://www.brookings.edu/research/reports/2016/03/24-hard-thinking-soft-skills-whitehurst?hs_u=saraschung@wustl.edu&;utm_campaign=Center+on+Children+and+Families&utm_source=hs_email&utm_medium=email&utm_content=27848577&_hsenc=p2ANqtz–YZuJXArwpHMQHSvx_jn1

35.

Wilson

S. J.

Lipsey

M. W.

Mark

(2006). The effects of school-based social information processing interventions on aggressive behavior: Part I: Universal programs: A systematic review. The Campbell Collaboration, 6, 1-37. doi:10.4073/csr.2006.5

36.

Wyman’s Teen Outreach Program. (2015). Substance Abuse and Mental Health Services Administration’s National Registry of Evidence-based Programs and Practices. Retrieved from http://legacy.nreppadmin.net/ViewIntervention.aspx?id=203

37.

Zaff

Boyd

Lerner

J. V.

Lerner

R. M.

(2010). Active and engaged citizenship: Multi-group and longitudinal factorial analysis of an integrated construct of civic engagement. Journal of Youth and Adolescence, 39, 736-750.

38.

Zins

J. E.

Bloodworth

M. R.

Weissberg

R. P.

Walberg

H. J.

(2004). The scientific base linking social and emotional learning to school success. In Zins

J. E.

Weissberg

R. P.

Wang

M. C.

Walberg

H. J.

(Eds.), Building academic success on social and emotional learning: What does the research say? (pp. 3-22). New York, NY: Teachers College Press.

39.

Zins

J. E.

Elias

M. J.

(2006). Social and emotional learning: Promoting the development of all students. Journal of Educational and Psychological Consultation, 17, 233-255.

40.

Zolkoski

S. M.

Bullock

L. M.

(2012). Resilience in children and youth: A review. Children and Youth Services Review, 34, 2295-2303. doi:10.1016/j.childyouth.2012.08.009