Service learning for improving academic success in students in grade K to 12: A systematic review

Evidence of service learning in primary and secondary education is inconclusive

School-based service learning is a teaching strategy that explicitly links community service to academic instruction. In this review, we aimed to find evidence of the effectiveness of service learning on students' academic success, personal and social skills, and risk behaviour. However, the evidence is inconclusive because of the small number of studies.

What is this review about?

Service learning is distinctive from traditional voluntarism or community service in that it intentionally connects service activities with curriculum concepts and includes structured time for reflection.

This review examines the evidence of impact of service learning on students' ‘neither employed, nor in education or training’ (NEET) status after compulsory schooling, academic success personal and social skills, and risk behaviour of students in primary and secondary education (from Kindergarten to Grade 12).

What is the aim of this review?

This Campbell systematic review examines the effects of service learning on academic success in students in primary and secondary education. The review summarises evidence from 10 studies undertaken in the USA that involved over 8,000 service learning participants in total.

What studies are included?

Included studies had to examine the impact of service learning in primary and secondary education. Studies had to have a comparison group.

Thirty-seven studies analysing 30 different populations were identified. Of these, only 10 studies, analysing nine different populations, could be used in the data synthesis.

The studies were all from the USA. There were eight randomised controlled trials (RCTs) reported in nine studies and one non-randomised study. The studies contained data for over 8,000 service learning participants.

What is the effect of service learning on academic success in students in primary and secondary education?

The evidence was inconclusive. The majority of studies available for meta-analysis reported on a very limited number of outcomes; in particular, few reported results on students' academic success even though the outcome was collected. At most, the results from three studies could be pooled in a single meta-analysis. Further, the majority of studies used in the meta-analyses reported implementation problems.

There was no evidence of adverse effects.

What do the findings of the review mean?

The current landscape of research on service learning in primary and secondary education (grades kindergarten to 12) in general education shows that it has yet to be evaluated thoroughly. The evidence was inconclusive because too few studies reported results on the same type of outcome.

Furthermore, all the available evidence used in the data synthesis was US-based, and so the findings may not be generalisable to other settings and systems outside the USA.

Also, the majority of studies used in the meta-analyses reported implementation problems.

These considerations point to the need for more rigorously conducted studies reporting a larger number of outcomes.

How up-to-date is this review?

The review authors searched for studies published up to October 2020.

Description of the condition

Completion of upper secondary education marks the minimum threshold for successful labour market entry and continued employability as suggested by the Organisation for Economic Co-operation and Developments (OECD's) annual indicators on education and associated labour market outcomes (OECD, 2015). On average across OECD countries, unemployment risk of younger adults (25–34 year-olds) who have not completed upper secondary education is almost double the risk of those with higher educational qualifications (upper secondary and postsecondary non-tertiary education). A maintained focus on completion rates is necessary. Even though enrolment rates among 15–16-year-olds (i.e., those typically in upper secondary programmes) are high; at least 95% on average across OECD countries in 2015 (OECD, 2018); far from all students graduate. According to OECD, only approximately 75% of students who had enroled had graduated after two years from the theoretical end date of the programme. Further, of the students who had not graduated, 80% were no longer enroled in education.

Many countries set specific targets for the completion rates of upper secondary education. For example, the countries in the European Union (EU) agreed on a 10-year strategy proposed by the European Commission on 3 March 2010 for advancement of the economy of the EU (Europe 2020). One of the main targets is to reduce the share of early school-leavers to 10% from the (at that time) current 15% and increase the share of the population aged 30–34 having completed tertiary from 31% to at least 40% (European Commission, 2010). Some countries go even further: Denmark has for example a specific target that upper secondary completion rates should be 95% and tertiary enrolment and completion rates should be 60% by 2020 (OECD, 2013a).

Not only graduation rates are important, the quality of the education received also matters for the educational prospects of young people and successful entry into the labour market. The shares of neither employed nor in education or training (NEET) are negatively related to the skill levels among young people (OECD, 2017a). The OECD's Programme for International Student Assessment (PISA) tests students near the end of their compulsory education (usually around age 15) on their reading ability, their skills in math and level in sciences. In general, the higher the percentage of low-performing 15-year-old students in PISA, the higher the percentage of NEETs among 15–19 year-olds (OECD, 2017a).

Having acquired some knowledge and skills that are essential for full participation in modern societies, particularly in reading, mathematics and science may be more reliable predictors of economic and social well-being than the number of years spent in school or in post-formal education (OECD, 2016). Research based on the 2012 Survey of Adult Skills (PIAAC) finds that poor proficiency in numeracy and literacy limits access to rewarding and well-paid jobs, and in addition is linked to poorer health and less social and political participation (OECD, 2013b).

There is, for these reasons, a significant interest in information about effective interventions to increase academic achievement and enhance educational prospects. The review we conducted focused on service learning in primary and secondary education. Service Learning is curriculum-based community service that integrates classroom instruction with community service activities. The connection with specific courses and having clearly stated learning objectives is what distinguishes service learning from other forms of volunteer work. Service learning should ‘address real community needs in a sustained manner over a period of time; and assist students in drawing lessons from the service through regularly scheduled, organised reflection of critical analysis activities, such as classroom discussions, presentations, or directed writing’ (Pritchard, 2002, p. 20). Well-designed service-learning activities can deepen learning and foster higher-order thinking skills by providing students with opportunities to apply their learning to a challenging situation or problem in their community.

The development of service learning as a pedagogical method that integrates community service into the course curriculum began in the 1970s, primarily in the USA (Spring et al., 2008). In the nineties, service learning became institutionalised in public education in the United States (Peterson & Seligman, 2004). In 1990, the National and Community Service Act created Serve America (later named Learn and Serve America), which was a federal programme dedicated to providing grants and other supports for service learning activities in schools and community-based organisations [1]. Further, in 1994, service learning became a recognised method for meeting the aims of federal school funding (included in the Elementary and Secondary Education Act). In addition to these federal policies, several states and school districts mandated the incorporation of service learning into the course curriculum (Education Commission of the States, 2014; Spring et al., 2008).

Service learning is not yet as widespread in the rest of the world. However, the OECD-project ‘Innovative Learning Environments’ mentions service learning as a pedagogical method to put learners at the centre (the first of the seven principles of learning needed to redesign the learning environments to meet the challenges of the 21st century) (Dumont et al., 2010). According to Furco (2010) ‘service-learning is one of the fastest growing educational initiatives in contemporary primary, secondary and postsecondary education’ (p. 228). Outside the USA, service-learning initiatives are part of the education systems of Argentina, Columbia and Singapore (Chua, 2010; Ierullo, 2016; Perold & Tapia, 2008). Argentina hosts the Latin American Center for Service-Learning (CLAYSS) which was created in 2002 to support students, educators, and community organisations in the development of service-learning projects in Latin America. Service learning is not part of any educational policy in Europe, although the EU recognises service learning as a way of achieving citizenship education (European Commission/EACEA/Eurydice, 2017). Service learning is however emerging in many European countries including Germany, Ireland, Italy, Spain and the United Kingdom (Furco, 2010), and currently CLAYSS is assisting in the creation of the Central and Eastern European Service-Learning Network (Regina & Ferrara, 2017).

In several European nations there are organisations (non-profit community-based) with programmes dedicated to providing supports for service learning activities in schools (Luna, 2012): Lernen durch Engagement in Germany, Center for Frivilligt Socialt Arbejde in Denmark, Lernen durch Engagement in Switzerland, Noi-orizonturi in Romania, MOVISIE in the Netherlands and Fundación Tomillo in Spain.

Description of the intervention

School-based service-learning is a teaching strategy that explicitly links community service to academic instruction (Billig, 2000). In the United States, 'service-learning' is an official term used by policymakers and educational leaders. Service-learning is distinctive from traditional voluntarism or community service in that it intentionally connects service activities with curriculum concepts and includes structured time for reflection. Service-learning is not an add-on to an existing curriculum, a requirement of minimum hours of service to graduate or service assigned as punishment. Rather, students are required to use academic knowledge and skills to address genuine community needs. A clarifying example is given by the National Youth Leadership Council (https://www.nylc.org/page/our-philosophy):

Picking up trash on a river bank is service.

Studying water samples under a microscope is learning.

When science students collect and analyse water samples, document their results, and present findings to a local pollution control agency—that is service-learning.

Service learning programmes can take many forms and are very diverse in content. However, a common set of elements are critical for a successful implementation of service learning. The National Youth Leadership Council and RMC Research Associates have developed a set of eight quality service-learning standards (the K-12 Service-Learning Standards for Quality Practice) with input from youth, teachers, administrators, youth agencies, policymakers, community members, and other stakeholders. The standards are:

Meaningful service: Service-learning actively engages participants in meaningful and personally relevant service activities.

The complete document can be accessed at https://www.nylc.org/page/standards.

How the intervention might work

Service learning, by connecting education to real world issues and allowing students to address problems they identify, may be particularly efficacious as it increases engagement and motivates students, in particular students who might not respond well to more traditional teaching methods (see, e.g., Bridgeland et al., 2008; Kraft & Wheeler, 2003; Scales & Roehlkepartain, 2005).

Motivation for learning and school engagement play a critical role in students' academic success (e.g., Fan & Wolters, 2014; Skaalvik & Valas, 1999). Motivated students tend to do better at school. According to OECD, students who are among the most motivated score the equivalent of more than one school year higher in PISA than the least-motivated students and motivation is further positively related to life satisfaction (OECD, 2017b).

Theoretically, Kolb's (1984) model of experiential learning is often referred to as the foundation for understanding how service-learning might work. Experiential learning theory defines learning as ‘The process whereby knowledge is created through the transformation of experience’ and knowledge is defined as: ‘a transformation process being continuously created and recreated, not an independent entity to be acquired or transmitted’ (Kolb, 1984, p. 38). Kolb further suggests that experiential approaches to learning such as service-learning are better at accommodating learners with different learning styles than traditional didactic approaches such as classroom-based teaching.

Experiential learning is inspired by pragmatist philosopher John Dewey's six-step process of experiential logical inquiry. According to Dewey the six steps are: (1) encountering a problem, (2) formulating a problem or question to be resolved (3) gathering information which suggests solutions (4) making hypotheses (5) testing hypotheses, and (6) making warranted assertions (Dewey, 1938; Giles & Eyler, 1994; Kolb, 1984). Kolb's (1984) model comprises these steps into a four stage experiential learning cycle involving: Concrete Experiences, Reflective Observation, Abstract Conceptualization and Active Experimentation (Cone & Harris, 1996; Kolb, 1984). Based on this conception, students participating in service-learning are engaged in a cycle in which their work in the community promotes written and/or oral reflection. Under the guidance of teachers or instructors, reflective work may be used to form abstract concepts and generate hypotheses, which may then be cycled back into further concrete experiences. According to Kolb this way of learning allows a variety of students with different learning styles and abilities to develop and integrate their skills (Cone & Harris, 1996).

Service-learning provides an opportunity for students to move between perceiving new information through experiencing the concrete, tangible, felt qualities of the world within the community and taking hold of new information through abstract conceptualization, thinking and analysing. The pattern in which a learner moves between these levels of experience are thought to reflect an individual learning style, and service-learning is thought to allow each student to move between the levels in a way consistent with their own learning style (Kolb et al., 2002).

Another strand of theory which offers a potential understanding of the theory of change behind service-learning is Situated Learning. The term ‘situated learning’ refers to learning that occurs within a particular and authentic context through the individual's social participation. Rather than focusing on learning as a primarily cognitive process involving a number of tasks, situated learning theorists study the process in which individuals become new members of a learning community. According to the theory newcomers within a learning community move from a state of legitimate peripheral participation to full participation through a process that involves continuous negotiation, collaboration, and reflection (Wolfson & Willinsky, 1998).

In their often cited work: ‘Situated Learning: Legitimate Peripheral Participation', Lave and Wenger (1991) focus on acquisition of skills and knowledge that takes place outside traditional schooling within communities of practice. Based on an ethnographic investigation of traditional and nontraditional apprenticeships in Mexico, Liberia and the United States, Lave and Wenger propose that learning should not be viewed as the mere transmission of knowledge but as a distinctly embedded and active process. Learning is thus perceived as a contextualised process in which content is learned through doing activities. Furthermore, Lave and Wenger suggest that motivation too is ‘situated', as learners are naturally motivated by their growing value of participation (Lave & Wenger, 1991). Based on this approach students participating in service-learning inherently become motivated to learn as this enables them to move from being novices to becoming full participants within the learning community. Furthermore, students participating in service-learning may become motivated as they experience how their own participation increases in value as they progress from being newcomers towards the center of the community of practice.

In situated learning the construction of meaning is seen as being tied to specific contexts and purposes. For students participating in service-learning this may be particularly important, as service learning may enable them to socially construct meaning which makes learning matter beyond school.

Why it is important to do this review

Two systematic reviews with meta-analyses are found in Conway et al. (2009) and Celio et al. (2011), both performing searches up to spring 2008. The review by Conway et al. (2009) analysed four outcomes: academic, personal, social, and citizenship outcomes. Many of the included studies did not have control groups. They furthermore included studies of community service or volunteerism as well as service learning without distinguishing between these very different types of interventions (except in a moderator analysis), participants were not limited to primary and secondary education (although all results were shown separately for grade kindergarten to 12 students but without distinguishing between community service or volunteerism and service learning).

The review by Celio et al. (2011) required included studies to analyse service learning using a control group, but participants were not limited to primary and secondary education. Five outcome areas were analysed: attitudes towards self, attitudes towards school and learning, civic engagement, social skills, and academic achievement. Separate results for primary and secondary education (grades kindergarten to 12) was only shown for the overall effect, that is, the mean of the five outcome domains attitudes towards self, attitudes towards school and learning, civic engagement, social skills, and academic achievement.

Besides being up to date, the major differences between these two systematic reviews and the current review are that we focused on service learning for primary and secondary education, only included studies with a control group, all relevant outcomes areas were analysed separately, and we took into consideration the dependencies between effect sizes.

In addition, there are several literature reviews of studies conducted in the United States (Billig, 2000, 2002, 2003, 2004). None of them is a systematic review and no data synthesis is performed in any of them. The review we performed differed in substantial ways from these existing reviews. It is systematic and several meta-analyses were conducted.

Criteria for considering studies for this review

Search methods for identification of studies

We implemented a wide range of search methods and strategies to maximise coverage of relevant references, while simultaneously attempting to reduce different types of bias related to publication and dissemination systems. The different strategies and methods are presented below.

Data collection and analysis

Selection of studies

Under the supervision of the review authors, two review team assistants first independently screened titles and abstracts to exclude studies that were clearly irrelevant. Studies considered eligible by at least one assistant or studies with insufficient information in the title and abstract to judge eligibility, were retrieved in full text. The full texts were then screened independently by two review team assistants under the supervision of the review authors. Any disagreement of eligibility was resolved by the review authors. Exclusion reasons for studies that otherwise might be expected to be eligible were documented and presented in section Characteristics of excluded studies. The study inclusion criteria were piloted by the review authors (see Supporting Information Appendix 1). The overall search and screening process is illustrated in Figure 1. None of the review authors were blind to the authors, institutions, or the journals responsible for the publication of the articles.

OPEN IN VIEWER

Figure 1

Flow diagram

Data synthesis

Meta-analysis of outcomes were conducted on each metric (as outlined in section ‘Types of outcomes measures') separately.

When the effect sizes used in the data synthesis were odds ratios, they were log transformed before being analysed.

Studies that were coded Critical risk of bias were not included in the data synthesis.

All analyses were inverse variance weighted using random effects statistical models that incorporate both the sampling variance and between study variance components into the study level weights. All meta-analyses were performed using Revman 5.4. The estimation of τ² was the DerSimonian and Laird (1986) estimate (DerSimonian & Laird, 1986). Random effects weighted mean effect sizes were calculated using 95% confidence intervals.

One study, Moskowitz (1981), provided results separately by numerous subgroups [type of service (2), grade (2) and gender (2)].

To take into account the dependence between multiple effect sizes from the same study, we planned to apply a robust variance estimation (RVE) approach (Hedges et al., 2010). However, as there was not a sufficient number of studies to use RVE, in accordance with the protocol, we conducted the data synthesis using a synthetic effect size (the average) to avoid dependence between effect sizes with one exception. Although random effects models applied when synthetic effect sizes are involved perform better in terms of standard errors than do fixed effects models (Hedges, 2007a), the method overestimates the standard error. As means and standard deviations for numerous subgroups within each condition (eight) were reported in Moskowitz (1981) the number used for calculating the standard errors for each subgroup effect size was very low and the standard errors were most likely heavily overestimated. Further, as the subgroups represented a breakout on several sample characteristics (grade and gender) the full within-group standard deviation is seriously underestimated because variability associated with the subgroup variable has been removed (variability in the outcome associated with grade and gender). We therefore used the formulas provided in Wilson (2015), section 3.19 Means and Standard Deviations with Subgroups to calculate the standardised mean difference for the overall population. We calculated the weighted mean for each condition (i.e., treatment and control) as

X ̲ = ( ∑ X ̲ j n j ) / ∑ n j ,

where j represents each subgroup in each condition. The pooled within-group standard deviation for each condition, ignoring any variance removed due to the subgroup variable, is calculated as

√ ∑ s j 2 ( n j − 1 ) / ∑ ( n j − 1 ) .

The subgroup variable is accordingly added back into the within group variance using the following formula:

√ ( ∑ s j 2 ( n j − 1 ) / ∑ ( n j − 1 ) + ∑ X ̲ j 2 n j − ( ∑ X ̲ j n j ) 2 / ∑ n j ) .

The within-groups pooled standard deviation is then computed using the standard formula, equation (5) in Wilson (2015), where T denotes treated and C denotes control:

s p o o l e d = √ s T 2 ( n T − 1 ) + s C 2 ( n C − 1 ) / ( n T + n C − 2 ) .

Moskowitz (1981) further provided results separately by two subscales for two outcome measures, Self-esteem and Locus of control. We conducted the data synthesis using a synthetic effect size (the average) of these two outcome measures to avoid dependence between effect sizes.

We provided a graphical display (forest plot) of effect sizes. Graphical displays for meta-analysis performed on ratio scales sometimes use a log scale, as the confidence intervals then appear symmetric. This is however not the case for the software Revman 5, which we used in this review.

Description of studies

Risk of bias in included studies

The risk of bias coding for each of the 37 studies is shown in a supplementary document (available here: https://osf.io/v8ceq/).

Fourteen studies reported on 10 randomised trials, two individually randomised trials and eight cluster randomised trials (reported in 12 studies). Four studies reported on the same CRCT, which included two cohorts, and two follow-up times. The four studies varied on the cohorts, follow-up times and outcomes they reported on, therefore they were all assessed for risk of bias. One study was a summary of five trials of which four were included in this review and this summary was not assessed for risk of bias.

Table 6 shows a summary of the risk of bias associated with the randomised studies.

Three of the studies did not report the method of randomisation nor was any baseline imbalances shown or discussed. We rated these three studies Unclear on the Randomisation Process item. Two studies reported an appropriate randomisation method and baseline balance on the pre-specified confounders and were rated Low risk of bias. Another four studies had some issues and were rated Some concerns and the remaining four studies were rated High risk of bias. On the Deviations from intervention item, the majority of studies, 10 studies, had some issues and were rated Some concerns, one was rated Low risk of bias and the two did not provide any information and were rated Unclear.

Concerning missing outcome data, two studies had no issues, and we rated them Low risk of bias, two studies did not report information concerning missing data and were rated Unclear, and six respectively three studies were rated Some concerns and High risk of bias. All but one study had some issues on the Measurement of Outcome item, we rated nine Some concerns, one High risk of bias, and two studies did not provide enough information and were rated Unclear. We rated no study Low risk of bias on the Selection of Reported Results item, all but one were rated either Some concerns (eight studies) or High risk of bias (four studies) and the last was rated Unclear. Overall, none of the studies were rated Low risk of bias, the majority were rated High risk of bias (ten studies), one was rated Some concerns and two studies provided insufficient information and were rated Unclear overall.

The remaining 23 studies used non-randomised designs, three studies (Melchior, 1995, 1998, 1999) used the same data set and modelling strategy so only one of them was risk of bias assessed. Table 7 shows a summary of the risk of bias associated with the non-randomised studies. As stated in the protocol, we stopped the assessment of a non-randomised study outcome when it was rated ‘Critical', therefore not all studies are rated on all domains.

Eighteen of the non-randomised studies were rated Critical risk of bias on the Overall judgement item corresponding to a risk of bias so high that the findings should not be considered in the data synthesis. The overall Critical risk of bias rating was mainly due to issues on the Confounding bias item; 16 were rated Critical risk of bias on this item; that is, they failed to establish a comparison group that was balanced on important confounders and further only a few controlled for any confounders. One study was rated Critical risk of bias overall due to a rating of Critical risk of bias on the Selection of Reported Results item. The remaining study rated Critical risk of bias on the Overall judgement item was rated Serious risk of bias on the Confounding bias and Selection bias items which lead to an Overall judgement rating of Critical risk of bias.

Two studies were rated Unclear overall as only very few, if any, of the domains in the risk of bias tool could be assessed due to very limited information provided. We excluded these two studies from the meta-analysis. One study was rated Serious risk of bias overall and was used in the data synthesis.

Effects of interventions

Ten studies (analysing nine different populations) permitted calculation of an effect size and standard error and were not rated Critical risk of bias. A large variety of different outcomes were reported in the studies. To carry out a meta-analysis, every study must have a comparable effect size. We synthesise effects separately by type of outcome (conceptual outcomes as outlined in section ‘Types of outcomes measures') and time point (end of intervention and follow up). Unfortunately each type of outcome was only reported in a small subset of studies (in many cases in only one single study). Thus, each meta analysis contains a very small number of effect sizes, at most three.

All continuous outcomes (effect sizes measured as Hedges g) were coded such that a larger effect size indicated better outcomes for the treated group. All binary outcomes (reported either as odds ratio or probability difference) were coded such that a smaller effect size indicated better outcomes for the treated group.

Academic success post intervention

Student achievement

Three studies reported students' general GPA.

Two of the reported results indicated a positive effect favouring the treated and one indicated a negative effect favouring the comparison; none of the study-level effects were statistically significant.

The weighted average was positive and statistically nonsignificant. The random effects weighted standardised mean difference was 0.09 (95% CI −0.02 to 0.21). Although the p value of the Q-statistic is notoriously underpowered to detect heterogeneity in small meta-analyses, the estimated τ ² is 0.00 and I ² is 0%, implying that heterogeneity among these three studies was not present. The forest plot is displayed in Figure 2.

OPEN IN VIEWER

Figure 2

(Analysis 1.1) Forest plot of comparison: 1 Academic success, outcome: 1.1 Grade point average

Two studies reported overall test results in reading.

One of the reported results indicated a positive effect favouring the treated and one indicated a negative effect favouring the comparison; none of the study-level effects were statistically significant.

The weighted average was positive and statistically nonsignificant. The random effects weighted standardised mean difference was 0.04 (95% CI −0.08 to 0.16). The estimated τ ² was 0.00 and I ² was 0%. The forest plot is displayed in Figure 3.

OPEN IN VIEWER

Figure 3

(Analysis 1.2) Forest plot of comparison: 1 Academic success, outcome: 1.2 Reading

Two studies reported overall test results in math.

Both reported results indicated a positive effect favouring the treated; one of the study-level effects was statistically significant and one was not statistically significant.

The weighted average was positive and statistically significant. The random effects weighted standardised mean difference was 0.21 (95% CI: 0.09 to 0.33). The estimated τ ² was 0.00 and I ² was 0%. The forest plot is displayed in Figure 4.

OPEN IN VIEWER

Figure 4

(Analysis 1.3) Forest plot of comparison: 1 Academic success, outcome: 1.3 Math

One study in addition to the overall math and reading test results, reported Social studies grades and Science grades and another study reported on a number of reading and math subscales in addition to the overall reading and math test results. We report the effect size in Table 8.

OPEN IN VIEWER

Table 8

Other outcomes

Study	Measure	Outcome	Effect size [95% CI]
Academic success
Melchior (1998)	School records	Social studies gradea	0.16 [0.02, 0.30]
Melchior (1998)	School records	Science gradea	0.15 [0.01, 0.29]
Melchior (1998)	School records	Overall/School GPA (including electives, other courses)a	0.10 [−0.04, 0.24]
Santmire (1999)	Metropolitan Achievement Test (MAT)	Math Processa	0.21 [−0.04, 0.46]
Santmire (1999)	Metropolitan Achievement Test (MAT)	Math Conceptsa	0.17 [−0.08, 0.42]
Santmire (1999)	Metropolitan Achievement Test (MAT)	Vocabularya	0.01 [−0.24, 0.26]
Santmire (1999)	Metropolitan Achievement Test (MAT)	Reading comp.	−0.03 [−0.28, 0.22]
Santmire (1999)	Metropolitan Achievement Test (MAT)	Languagea	0.12 [−0.13, 0.37]
Melchior (1998)	School records	Fail 1 or more coursesb	−0.04 [−0.08, −0.00]
Personal and social skills
Melchior (1998)	Personal and social responsibility (Search Institute scale: range 5–25):	Social welfare subscalea	0.18 [0.06, 0.30]
Melchior (1998)	Psychosocial maturity (Greenberger scale: range 1–4):	Communication Skills subscalea	−0.02 [−0.14, 0.10]
Melchior (1998)	Psychosocial maturity (Greenberger scale: range 1–4):	Work Orientation subscalea	0.06 [−0.06, 0.18]
Melchior (1998)	Connell scale	School engagement (Research/Assessment Package for Schools (RAPS))a	0.24 [0.12, 0.36]
Risk behaviour
Melchior (1998)	Involvement with Risk Behavior (Search Institute, Profiles of Student Life)	Consumed any alcohol in last 30 daysb	−0.02 [−0.08, 0.04]
Melchior (1998)	Involvement with Risk Behavior (Search Institute, Profiles of Student Life)	Used illegal drugs in last 30 daysb	−0.01 [−0.05, 0.03]
Melchior (1998)	Involvement with Risk Behavior (Search Institute, Profiles of Student Life)	Arrested in last 6 monthsb	−0.00 [−0.04, 0.04]
Melchior (1998)	Involvement with Risk Behavior (Search Institute, Profiles of Student Life)	Fought, hurt, or used weapon in last 6 monthsb	−0.05 [−0.11, 0.01]
Melchior (1998)	Involvement with Risk Behavior (Search Institute, Profiles of Student Life)	Ever been pregnant or made someone pregnantb	−0.03 [−0.07, 0.01]
Melchior (1998)	School records	Suspended last year (days)a	0.03 [−0.15, 0.21]
Melchior (1998)	Involvement with Risk Behavior (Search Institute, Profiles of Student Life)	Total number of risk behavioursa	0.10 [−0.06, 0.26]
Moskowitz (1981)	School records	Nondrug problemsb	0.04 [−0.39, 0.47]

Effect size is Hedges g and a positive effect favoures the treated.

Effect size is probability difference and negative effect favoures the treated.

Attendance

Two studies reported days absent from school.

Both reported results indicated a positive effect favouring the treated; none of the study-level effects were statistically significant.

The weighted average was positive and statistically nonsignificant. The random effects weighted standardised mean difference was 0.03 (95% CI: −0.10 to 0.16). The estimated τ ² was 0.00 and I ² was 0%. The forest plot is displayed in Figure 5.

OPEN IN VIEWER

Figure 5

(Analysis 1.4) Forest plot of comparison: 1 Academic success, outcome: 1.4 Absences

Drop out, NEET and other outcomes

None of the studies reported on drop out or NEET status. One study reported on failure of courses. The effect size is reported in Table 8.

Personal and social skills post intervention

Two studies reported on comparable self-esteem measures.

Both reported results indicated a positive effect favouring the treated; none of the study-level effects were statistically significant.

The weighted average was positive and statistically nonsignificant. The random effects weighted standardised mean difference was 0.13 (95% CI: −0.14 to 0.40). The estimated τ ² was 0.00 and I ² was 0%. The forest plot is displayed in Figure 6.

OPEN IN VIEWER

Figure 6

(Analysis 2.1) Forest plot of comparison: 2 Personal and social skills, outcome: 2.1 Self-esteem

Three studies reported on measures of locus of control. None of the study-level effects were statistically significant, and the weighted average was positive and statistically nonsignificant. The random effects weighted standardised mean difference was 0.07 (95% CI: −0.04 to 0.18). The estimated τ ² was 0.00 and I ² was 0%. The forest plot is displayed in Figure 7.

OPEN IN VIEWER

Figure 7

(Analysis 2.2) Forest plot of comparison: 2 Personal and social skills, outcome: 2.2 Locus of control

In addition, three studies reported on a number of other personal and social skills outcomes (measured by a variety of different standardised scales). These could however, not be meta analysed as each outcome was reported in only one study. The effect size is reported in Table 8.

Risk behaviour postintervention

Pregnancy

Two studies reported on a pregnancy outcome (Have you ever been/caused someone to be pregnant) measured as an odds ratio.

An odds ratio less than 1 indicates that the treated, that is, the participants in service learning, is favoured. That is, the odds of having been or caused someone to become pregnant is lower for participants in service learning. One of the reported results indicated an effect favouring the treated and one indicated an effect favouring the comparison. The weighted average favoured the comparison and was statistically nonsignificant. The random effects weighted mean odds ratio was 1.05 (95% CI: 0.63 to 1.74). The forest plot is displayed in Figure 8. There was some heterogeneity between the studies; the estimated τ ² was 0.10, Q = 4.12, df = 1 and I ² was 76% as displayed in Figure 8.

OPEN IN VIEWER

Figure 8

(Analysis 3.1) Forest plot of comparison: 3 Risk behaviour, outcome: 3.2 Ever been/made someone pregnant

In addition, one study measured this outcome as a probability difference (reported as the model result from a linear regression, hence it could not be transformed to an odds ratio). The effect size is reported in Table 8.

Risky sex

Three studies reported on a risky sex outcome (Engagement in unprotected sex) measured as an odds ratio.

Two of the reported results indicated an effect favouring the treated and one indicated an effect favouring the comparison; none of the study-level effects were statistically significant.

The weighted average favoured the treated and was statistically nonsignificant. The random effects weighted mean odds ratio was 0.96 (95% CI: 0.74 to 1.25). There was some heterogeneity between the studies; the estimated τ ² was 0.03, Q = 4.42, df = 2 and I ² was 55%. The forest plot is displayed in Figure 9.

OPEN IN VIEWER

Figure 9

(Analysis 3.2) Forest plot of comparison: 3 Risk behaviour, outcome: 3.1 Risky sex

Other risk behaviour

A number of other risk behaviours were reported in two studies but could not be meta analysed as each outcome was reported in only one study, see Table 8 for the effect sizes.

Risk behaviour at follow up

Pregnancy

Two studies reported on the pregnancy outcome (Have you ever been/caused someone to be pregnant) measured as an odds ratio approximately one year after the intervention.

One of the reported results indicated an effect favouring the treated and one indicated an effect favouring the comparison. The weighted average favoured the treated and was statistically nonsignificant. The random effects weighted mean odds ratio was 0.84 (95% CI: 0.39 to 1.82). There was heterogeneity between the studies; the estimated τ ² was 0.28, Q = 9.18, df = 1 and I ² was 89% as displayed in Figure 10.

OPEN IN VIEWER

Figure 10

(Analysis 3.4) Forest plot of comparison: 3 Risk behaviour, outcome: 3.5 Ever been/made someone pregnant Follow up

Risky sex

Three studies reported on a risky sex outcome (Engagement in unprotected sex) measured as an odds ratio approximately one year after the intervention.

Two of the reported results indicated an effect favouring the treated and one indicated an effect favouring the comparison; none of the study-level effects were statistically significant. The weighted average favoured the treated and was statistically nonsignificant. The random effects weighted mean odds ratio was 0.89 (95% CI: 0.73 to 1.105). We found no heterogeneity between the studies; the estimated τ ² was 0.00, and I ² was 0%. The forest plot is displayed in Figure 11.

OPEN IN VIEWER

Figure 11

(Analysis 3.5) Forest plot of comparison: 3 Risk behaviour, outcome: 3.4 Risky sex Follow up

Other risk behaviour

None of the studies reported other risk behaviour outcomes at follow up (one study actually did, but follow-up outcomes were rated Critical risk of bias, see the supplementary document here: https://osf.io/v8ceq/).

Sensitivity

Two studies used in the meta analyses needed correction for treatment given in clusters. One study was a cluster randomised trial (Giacalone, 2004) and the other was an individualised randomised trial (Santmire, 1999).

Although adjusting for clustering decreased the individual effect sizes slightly and increases the standard errors, the average effect size estimates were virtually unchanged, and the conclusions did not change (Figures 12–15).

OPEN IN VIEWER

Figure 12

(Analysis 4.2) Forest plot of comparison: 4 Sensitivity, outcome: 4.2 Sensitivity Grade point average

OPEN IN VIEWER

Figure 13

(Analysis 4.5) Forest plot of comparison: 4 Sensitivity, outcome: 4.5 Sensitivity Reading

OPEN IN VIEWER

Figure 14

(Analysis 4.6) Forest plot of comparison: 4 Sensitivity, outcome: 4.6 Sensitivity Math

OPEN IN VIEWER

Figure 15

(Analysis 4.7) Forest plot of comparison: 4 Sensitivity, outcome: 4.7 Sensitivity Self-esteem

Sensitivity analyses were further planned to evaluate whether the pooled effect sizes were robust across study design and components of methodological quality. However, there was no meta-analysis in which the number of studies contributing data was sufficient for further sensitivity analysis to be meaningful (no meta-analysis included more than three studies).

Summary of main results

Overall, there were too few studies included in any of the meta-analyses in order for us to draw any conclusion concerning the effectiveness of service learning on student outcomes. At most, the results from three studies could be pooled in a single meta-analysis. All the meta-analyses showed a positive weighted average except the pregnancy outcome and none of them was statistically significant except the weighted average of the two studies reporting math test results.

Overall completeness and applicability of evidence

We included in total 10 studies (analysing nine different populations) in the data synthesis and of these a maximum of three studies reported the same outcome and could be used in a specific meta-analysis. This number is lower than the number of studies (37) meeting the inclusion criteria. The reduction was caused by three different factors.

Six studies did not report effect estimates or provide data that would allow the calculation of an effect size. Eighteen studies were judged to have a Critical risk of bias and, in accordance with the protocol, we excluded these from the data synthesis on the basis that they would be more likely to mislead than inform. Finally, we could not use three studies as they reported on two clusters of studies using the same data sets and reporting on the same outcome(s) at the same time points.

If all the included studies had provided an effect estimate with lower risk of bias, the final list of useable studies in the data synthesis would have been larger, which again would have provided a more robust literature on which to base conclusions.

All studies used in the data synthesis were from the United States. A single study outside the United States was identified (from Nigeria) but did not provide data that would allow the calculation of an effect size. This narrow geographical coverage is a clear limitation of the review.

Follow-up analyses were only possible for two risk behaviour outcomes and none of the other types of outcomes. This is also a clear limitation of the review.

It was not possible to examine the impact of the moderators nor sensitivity analyses for each outcome to check whether the obtained results were robust across study design and methodological quality.

Quality of the evidence

The majority of studies (23) used non-randomised designs, and 14 studies reported on 10 randomised trials. Overall the risk of bias in the included studies was high. Among the non-randomised studies only one study was not rated Critical risk of bias (in addition, two studies provided too little information to be rated). The level ‘Critical’ means: the study (outcome) is too problematic in this domain to provide any useful evidence on the effects of intervention, and it is excluded from the data synthesis.

None of the randomised trials were overall rated low risk of bias, one was assessed to have some concerns while the rest were of high risk of bias (in addition two studies provided too little information to be rated).

We examined the risk of bias using Cochranes revised risk of bias tool, RoB 2 (Higgins et al., 2019) for the randomised studies and using the model ROBINS–I, developed by members of the Cochrane Bias Methods Group and the Cochrane Non-Randomised Studies Methods Group (Sterne et al., 2016a) for the non-randomised studies.

The quality of the evidence in this review was enhanced by excluding studies assessed to be at critical risk of bias using the ROBINS–I tool from the data synthesis. We believe this process excluded those studies that are more likely to mislead than inform.

There was overall consistency in the direction and magnitude of effects and there was no heterogeneity between studies except in a few cases (the risk behaviour outcomes).

Potential biases in the review process

We performed a comprehensive electronic database search, combined with grey literature searching, and hand searching of key journals. All citations were screened in teams by two independent screeners from the review team (TPC, MCTM, FSB, and LMTD), and one review author (TF) assessed all included studies against inclusion criteria (the review team is listed in section Acknowledgements).

We believe that all the publicly available studies on the effect of service learning on students' academic success, personal and social skills and risk behaviour up to the censor date were identified during the review process. However, six references were not obtained in full text.

We were unable to comment on the possibility of publication bias as at most three studies was included in the same meta-analysis. Thus, we cannot rule out that there are still some missing studies, which were not published or made public.

We believe that there are no other potential biases in the review process as two teams each with two members of the review team (TPC, MCTM, FSB, LMTD) independently coded the included studies. Any disagreements were resolved by discussion. Further, decisions about inclusion of studies were made by the two teams of each two members of the review team (TPC, MCTM, FSB, LMTD) and one review author (TF). Assessment of study quality and numeric data extraction was made by one review author (TF) and each study was checked by at least another review author (JD, NTD) and in addition in some cases by two members of the review team (TPC, MCTM).

Agreements and disagreements with other studies or reviews

The review by Celio et al. (2011) compared service learning interventions to control groups and is the only review we believe can be compared to our review. Celio et al. (2011) found 62 studies, of which 19 had participants from primary and secondary education only. Five outcome areas were analysed: attitudes towards self, attitudes towards school and learning, civic engagement, social skills, and academic achievement. Separate results for primary and secondary education (grades kindergarten to 12) was only shown for the overall effect, that is, the mean of the five outcomes attitudes towards self, attitudes towards school and learning, civic engagement, social skills, and academic achievement. The overall average effect (obtained from a random effects model) for these five measures combined was 0.20 for K-12 students (95% CI: 0.08 to 0.31); higher than any of the measures analysed separately in our review except for math. It is, however, unclear how much each of the five measures contributed to the size of the combined single outcome effect size. From the 62 studies (including those not analysing college and beyond) a total of 380 effect sizes were extracted and used in their meta-analyses. It is not reported how large a share of these effect sizes were from K-12 studies.

The approach followed by Celio et al. (2011) differ from ours in two other important aspects, making it difficult to compare the results. First, contrary to our inclusion criteria Celio et al. (2011) did not require outcome measures to be reliable or valid; of the total 380 effect sizes included, 120 was coded as not ‘Use of reliable outcome measures’ and only 169 were coded as ‘Use of valid outcome measures’ (tab. 3 in Celio et al., 2011). The average effect size in studies that used reliable outcome measures was markedly smaller than in those that did not (0.23 compared to 0.41), whereas the effect size calculated based on valid outcomes was similar to the one based on not validated measures (0.27 compared to 0.30). However, Celio et al. did not report these subgroup analyses separately for K-12 students and further they did not take into consideration that more than one outcome per study was included in this subgroup analysis (i.e., they did not take into account the statistical dependencies between the effect sizes).

Second, Celio et al. (2011) included all studies in their meta-analyses whereas we excluded studies rated Critical in at least one domain of ROBINS-I. RCTs and non-randomised studies have very similar effect sizes in their analyses (0.31 compared to 0.30). However, as this subgroup analysis is not reported separately for K-12 studies, it is difficult to say whether this result holds also in the subgroup of interest in our review.

Implications for practice

The current landscape of research on service learning in primary and secondary general education (grades kindergarten to 12) shows that it has yet to be evaluated thoroughly. The evidence was inconclusive because too few studies reported results on the same type of outcome.

Furthermore, all the available evidence used in the data synthesis was USA-based, and so the findings may not be generalisable to other settings and systems outside the United States. In fact, as the educational systems within the US differ between states and the studies examined service learning in different communities and settings, generalisations between contexts within the USA should also be made with care. However, it is important to point out that service learning is, in our view, potentially applicable in a wide range of contexts, and service learning interventions could be implemented in many more countries than those found in the studies we included. That is, the reason for the low number of studies from other countries may have less to do with institutional constraints and more to do with the tradition of quantitative educational research being stronger in the USA than elsewhere (see e.g., Dietrichson et al., 2020, 2021, for a similar pattern of USA-dominance regarding interventions targeting students with academic difficulties).

Implications for research

In this review, we aimed to find evidence of the effectiveness of service learning on students' academic success, personal and social skills, and risk behaviour. However, the evidence was inconclusive. We found only few randomised controlled trials and the risk of bias in the included non-randomised studies was very high leaving only one non-randomised study to be meta-analysed. The majority of the eight randomised trials available for meta-analysis reported on a very limited number of outcomes; in particular few reported results on students' academic success even though the outcome was collected. Furthermore, the majority of studies used in the meta-analyses reported implementation problems.

These considerations point to the need for more rigorously conducted studies reporting a larger number of outcomes.

It would be natural to consider conducting a large randomised controlled trial (or a series of large randomised trials) with specific allocation to implementation of high quality service learning as guided by the eight standards: (1) Meaningful service, (2) Link to curriculum, (3) Reflection, (4) Diversity, (5) Youth voice, (6) Community partnerships, (7) Progress monitoring and (8) Sufficient duration and intensity. Moreover, high-quality service learning practice occurs when it is shaped and adapted to the particular community and student contexts and conditions. When setting up an experiment, the intervention should therefore allow for adaptations to the community and student context. For example, the particular community service project should not be required to be the same across sites, as such a requirement would violate the standards of meaningful service, youth voice, and partnerships. What is important is that students can self-select into a service learning activity of their choosing, for the service learning effort to be considered of high quality. Students in the same class need not even participate in the same activity, teams of students within a class performing different activities of their own choice would meet the standards of high-quality service learning.

These features of high-quality service learning present some difficulties for the design and implementation of a high-quality randomised trial. As chosen projects can be collaborations between students and to reduce the risk of spill-over effects, the intervention should be assigned to clusters of students, not individual students. One could imagine a cluster-randomised trial where either classes within schools are randomised to take up service learning in a particular course, or schools are randomised to offer service learning in particular grades, or whole school districts are randomised to implement service learning. Larger clusters decrease the risk of spill-overs but may increase implementation difficulties. In this regard, there are examples of class (e.g., Schanzenbach, 2007), school (e.g., Gersten et al., 2015), and school district (e.g., Slavin et al., 2013) randomised trials in other areas of educational research, which we believe have yielded informative results and which much can be learnt from.

If schools can adapt and students self-select into the service learning activities, then it will be more difficult to find suitable outcome measures than if the intervention was the same across sites. However, some measures used by the included studies in this review were both validated and broad enough to capture effects that would be interesting to examine for any service learning activity. Examples include school absences, self-esteem, self-efficacy, social skills, and locus of control. No meta analysed study used drop out or on-time graduation but such measures would be interesting outcomes in any service learning intervention. Furthermore, while the service learning activity may differ, the subject in which service learning is implemented could be standardised across sites, possibly without decreasing the quality of the service learning experience. There are many validated tests available that could be used to examine effects on important student skills such as math and reading. By standardising for example state-level tests that all students are expected to take (e.g., by using the percentile rank within states), such tests can be meaningfully compared also across tests. Although measures of broader skills may not capture effects on aspects inherent to the service learning activity, this trade off would be acceptable in our view.

Specific attention would also have to be paid to stringency in terms of conducting a well-designed randomised trial with low risk of bias as well as ensuring that the sample sizes are large enough to enable sufficient power. The trial or trials should be designed, conducted, and reported according to methodological criteria for rigour to achieve high internal validity. For example, by following the criteria for risk of bias laid out in the RoB-2 tool (Higgins et al., 2019). To achieve high external validity, schools and students should be sampled from differing contexts and outcomes should be measured with validated instruments. If possible, research designs that allow for the evaluation of both short and long-term effects would be preferable.

Although we believe that implementing high-quality randomised trials is possible, we want to acknowledge that it is challenging (in all areas of education). Supplementing randomised trials with high-quality quasi-experimental studies will therefore be important to learn more about the effects of service learning. For instance, no included study used a ‘natural experiment’ to estimate the effects of service learning. The variation of service learning mandates across both time and school districts in the United States (Education Commission of the States, 2014; Spring et al., 2008) suggests that students have been differentially exposed to service learning because of factors that may be unrelated to student and school characteristics. Similar differences across regions and time have been used in other areas of education (see Gopalan et al., 2020 for a review), and may be useful also in the area of service learning.

Obtaining balance on important confounding factors may be difficult when students are not randomised or a natural experiment is not available, which adds to the importance of statistically controlling for relevant factors. In this review, we would have judged the risk of bias due to confounding to be of less concern had the primary study authors controlled for more relevant factors in their analyses. As data on for example performance at baseline, grade level, gender, or socioeconomic background were available in some studies judged to be at Critical risk of bias, we would recommend that this information is also used in the analyses to control for important confounding factors.

Lastly, in calling for more randomised trials and quasi-experimental studies, we do not want to downplay the importance of qualitative methods. On the contrary, qualitative methods are likely necessary to learn more about how effects come about and why they might differ between contexts. That is, a high-quality randomised trial ought to be combined with a detailed qualitative investigation of for example implementation fidelity and the operationalisation of service learning across sites.