Impact of summer programmes on the outcomes of disadvantaged or ‘at risk’ young people: A systematic review

Delivery period and duration

Summer education programmes included in this review take place between July and September (these are the summer months, when summer breaks commonly take place, in countries of the northern hemisphere, including the UK and US where studies for this review are concentrated). In the UK and US, some transition support programmes are offered either at the very beginning of summer vacation (Day et al., 2013b), to promote sustained engagement, or right before the start of new academic year, to facilitate the transition process (Anthony, 2019; Henson, 2018). Other summer education programmes are more diverse in terms of when they take place.

Programme durations vary: short summer programmes, of several days or up to 1 week, are more frequent in the UK (Burgess et al., 2021; Church, 2018; CooperGibson, 2022; Hayes et al., 2018; Lei et al., 2020; Sharp, 2018; Thompson et al., 2017) than the US where programmes tend to last between 2 and 6 weeks (Garcia et al., 2020; Henson, 2018; Herrera et al., 2013; Johnson, 2020; Mac Iver & Mac Iver, 2019; Mariano & Martorell, 2013; McEvoy, 2012; Robles, 2018; Snipes et al., 2015; Somers et al., 2015; Wachen et al., 2018). In the UK these can take the form of short residentials which are not common in the US. However, there is one example of a residential summer programme in the US taking place over 5 weeks (Wachen et al., 2018). Catch-up programmes are similar in duration in the UK and US. They tend to last 4 to 5 weeks, delivered over 4 to 5 days per week for around 5 h per day (Gorard et al., 2014; Gorard et al., 2015; Mariano & Martorell, 2013; McEvoy, 2012; Snipes et al., 2015; Somers et al., 2015; Torgerson et al., 2014).

Targeting approach

Summer education programmes tend to target students identified as being most able to benefit from additional academic support, such as those who come from low-income backgrounds, with lower academic attainment, or from areas with relatively poor higher education participation. Programmes also commonly target young people experiencing other forms of disadvantage, including those from ethnic minority backgrounds (Cohodes et al., 2022; Cosentino et al., 2015; Robles, 2018), disabled young people or those with a health condition (CooperGibson, 2022; Lei et al., 2020), students from the first-generation in the family to attend university (Anthony, 2019; Cohodes et al., 2022; Henson, 2018; Robles, 2018; Smith et al., 2013; Wachen et al., 2018), students in care (Kettlewell & Aston, 2014a; 2014b; Lamont et al., 2014; Martin et al., 2013b), immigrant students (Johnson, 2020), and young people at risk of or with a history of offending (Tarling & Adams, 2012). Some programmes target areas that have generally poor academic performance, but then select individuals within those areas that have high academic performance (Hoare & Mann, 2012; Smith et al., 2013). In both the US and the UK, summer education programmes target disadvantage at the individual, family, school or neighbourhood level (Church, 2018; Day et al., 2013b; Ferguson, 2018; Gorard et al., 2015; Herrera et al., 2013; Horton & Hilton, 2020; Lynch & Kim, 2017; Maxwell et al., 2014; McEvoy, 2012; Thompson et al., 2017).

It should be noted that some programmes have hard criteria that participants need to meet to be eligible, usually around socioeconomic status (e.g., coming from a neighbourhood with a high deprivation index) or academic performance (i.e., performing at/under a certain grade level), while others take a more holistic approach whereby the combination of a range of disadvantage factors may be taken into account, when deciding which young people get offered a place.

It should also be noted that most programmes in the review tend to target a ‘grey middle’, when it comes to academic performance, meaning students with poor performance but who may still be meaningfully supported by a developmental summer education programme. Young people with the poorest performance, who are often also those with the highest vulnerabilities, are often already recipients of more specialised and/or intensive alternative support outside of a summer programme.

Summer education programmes in the US tend to have a stronger academic focus, with a higher concentration of catch-up, alongside transition and raising aspirations programmes aimed at supporting increased participation in higher education and STEM (science, technology, engineering and mathematics) subjects. There is a stronger tendency in the US, compared to the UK programmes, to target first-generation (Anthony, 2019; Cohodes et al., 2022; Henson, 2018; Robles, 2018; Wachen et al., 2018) and ethnic minority students (Cohodes et al., 2022; Cosentino et al., 2015; Lynch & Kim, 2017; Robles, 2018). In the UK, programmes tend to look at wider social mobility factors, and targeting is more likely to be based on areas of socioeconomic disadvantage and where progression in education is low (Church, 2018; Day et al., 2013b; Ferguson, 2018; Gorard et al., 2015; Horton & Hilton, 2020; Maxwell et al., 2014; Thompson et al., 2017). Similarly, while both countries have programmes targeting transitions, the focus differs in each. In the UK, there is a strong emphasis in the literature surfaced on transitions from primary to secondary school (CooperGibson, 2022; Day et al., 2013a; 2013b; Martin et al., 2013a; 2013b; Maxwell et al., 2014; Sharp, 2018; Siddiqui et al., 2014), whereas US programmes most often target transitions from secondary to higher education – college ‘bridge’ programmes are common (Anthony, 2019; Barnett et al., 2012; Garcia et al., 2020; Gehring et al., 2018; Henson, 2018; Wachen et al., 2018).

There is a further distinction key to the way different programmes target eligible participants. Several programmes target individuals within a specific geographic area, community, or school district with high levels of disadvantage. Programmes that target disadvantage mainly by area (in terms of academic performance and socioeconomic status) are more concentrated in the UK. Specific criteria used include the proportion of students in a school eligible for Free School Meals (FSM) (Maxwell et al., 2014; Smith et al., 2013); the National Collaborative Outreach Programme (NCOP) (Church, 2018; Horton & Hilton, 2020) and the Participation of Local Areas in higher education (POLAR) (Ferguson, 2018) – two area-based indicators of low rates of progression to higher education; and the index of multiple disadvantage (IMD) (Taylor, 2022).

In the US, programmes rely far more on targeting disadvantage using individual-level criteria, particularly by targeting young people with relatively poor academic performance based on the individual's grades and/or test scores (Barnett et al., 2012; Gehring et al., 2018; Ghazzawi et al., 2022; Mac Iver & Mac Iver, 2019; Mariano & Martorell, 2013; Snipes et al., 2015; Somers et al., 2015; Wathington et al., 2016), those eligible for federal subsidies such as the Pell Grant (Anthony, 2019; Ghazzawi et al., 2022; Wachen et al., 2018), or who are from groups under-represented in specific academic disciplines (Cohodes et al., 2022; Ghazzawi et al., 2022; Mac Iver & Mac Iver, 2019; Robles, 2018; Snipes et al., 2015).

There are then some distinctions in targeting between different types of summer education programmes. Catch-up programmes tend to occur in schools located in socioeconomically disadvantaged areas, where students are at the highest risk of summer learning loss, and target students who are performing below the expected level or are at risk of falling behind academically (Gorard et al., 2015; Lynch & Kim, 2017; Williamson et al., 2020). By targeting schools in these areas, the programmes aim to provide educational opportunities to underperforming students who may otherwise have limited access to academic support. This can include learners of English as a second language or students who have been grade retained or are at risk of grade retention (the practice in the US whereby students repeat a grade level) (Johnson, 2020; Mariano & Martorell, 2013).

Transition support programmes tend to focus on supporting first-generation and low-income students, students from ethnic minority backgrounds, and those from rural areas, with the aim of supporting educational progression, particularly to higher education (Anthony, 2019; Day et al., 2013b; Henson, 2018; Martin et al., 2013b; Siddiqui et al., 2014). This is on the basis that students within these groups may lack external support systems in particular as they may have few, if any, relationships with people who have participated in higher education. Similarly, programmes aimed at raising students' aspirations are more likely to target disadvantaged learners who come from areas with lower-than-expected higher education participation rates (Hayes et al., 2018; Hoare & Mann, 2012; Kettlewell & Aston, 2014b; Taylor, 2022).

Recruitment methods

Approaches to recruitment tend to be similar in the UK and US with referrals from schools and academic staff, and parent involvement playing a significant role. Common advertising/awareness raising methods include flyers, posters, letters to parents, information evenings for pupils and parents, and referrals from school staff.

For catch-up programmes, which appear more widespread in the US than UK, recruitment methods include identifying eligible students from specific geographic areas or schools, often those with higher rates of underperforming students (Snipes et al., 2015; Somers et al., 2015; Torgerson et al., 2014). Information sessions are common to raise awareness among parents (Snipes et al., 2015; Torgerson et al., 2014). School staff play a crucial role as a source of information for eligible young people (Snipes et al., 2015; Wathington et al., 2016). In raising aspirations and transition support programmes, participants are often recruited through school referrals, or online application forms hosted on programme or university websites (CooperGibson, 2022; Herrera et al., 2013; Lei et al., 2020). Schools with higher rates of disadvantaged students are often targeted, and recruitment strategies include school visits, presentations, school assemblies, and word-of-mouth referrals (Cosentino et al., 2015; Ferguson, 2018). Many of these programmes identify parents as key, hence invitation letters, telephone calls, and face-to-face engagement through parent evenings, alongside use of multilingual teachers to remove barriers for parents with English as an additional language, support engagement (CooperGibson, 2022; Garcia et al., 2020; Henson, 2018; Martin et al., 2013b; Snipes et al., 2015).

Delivery period and duration

Summer employment programmes that met the inclusion criteria for this review were found only in the US. They tend to last between six and 7 weeks – the duration of the summer break (Gelber et al., 2016; Lansing et al., 2018; Leos-Urbel, 2014; Modestino & Paulsen, 2019a). Participants usually spend around 25 h a week on a work placement, alongside a training component, such as pre-work training or employability skills (Gelber et al., 2016; Heller, 2022; Leos-Urbel, 2014; Modestino, 2019b; Schwartz et al., 2021; Valentine et al., 2017). One programme, Urban Alliance, has a year-round training component, which requires participants to attend employability workshops from late September to July, plus 3 to 6 weeks of pre-work training before the work placement in the summer break (Theodos et al., 2017).

Targeting approach

Summer employment programmes tend to prioritise individuals from specific age groups and communities, often those facing high rates of poverty, unemployment, urban violence, or being at risk of not transitioning to higher education or meaningful work (Davis & Heller, 2020; Modestino & Paulsen, 2019a; Reich, 2018; Sum, 2015; Theodos et al., 2014, 2017). Geographic targeting is common, where programmes concentrate their efforts on neighbourhoods with significant socioeconomic disadvantages including high levels of poverty and unemployment as well as high levels of violence, with a particular focus on neighbourhoods with high rates of crime and high-violence schools (Davis & Heller, 2020; Sum, 2015).

Targeting based on academic outcomes is also common, with some programmes targeting students with lower attainment, or at risk of not transitioning to higher education or meaningful work (Theodos et al., 2014, 2017). Most programmes have a high proportion of ethnic minority participants, particularly African American and Hispanic participants – in many programmes, these groups make up over half of participants, although this is often a result of self-selection and the overlap between ethnic identity and the forms of disadvantage used as eligibility criteria, rather than an explicit eligibility criteria (Gelber et al., 2016; Heller, 2014; Leos-Urbel, 2014; Modestino & Paulsen, 2019a; 2019c; Modestino, 2019b; Reich, 2018; Sum, 2015; Theodos et al., 2014, 2017; Valentine et al., 2017).

Recruitment methods

Recruitment strategies for summer employment programmes include school staff leading general or targeted group presentations, individual meetings with students, and sharing information in newsletters and on posters to raise awareness (Reich, 2018; Theodos et al., 2014). Non-profit and community-based organisations and partners also spread awareness through newsletters and outreach, and may support potential participants to complete applications (Valentine et al., 2017). As a result, participants may apply for the programmes through specific providers or intermediaries, sometimes contracted by the relevant local authority or government agency (Modestino, 2019b; Modestino & Paulsen, 2019a; Modestino, 2019b; Valentine et al., 2017).

The application period for summer employment programmes is generally in early spring, and there tends to be more applicants than places, which sometimes leads to lottery systems for selection (N.B., this might be specific to the studies included in this review, as evaluators may have selected programmes where this was the case to randomise treatment status, which may not be generalisable to all summer employment programmes) (Gelber et al., 2016; Heller & Kessler, 2017; Leos-Urbel, 2014; Schwartz et al., 2021; Valentine et al., 2017). Other allocation methods include ‘first-come first-served’, merit-based assignment, or meeting eligibility criteria based on specified disadvantage characteristics, such as coming from a high-poverty or low educational progression area (Heller & Kessler, 2017; Lansing et al., 2018).

Engagement with, participation in and enjoyment of education

Summary: ▪

Summer education programmes appear to have a positive impact on this outcome, although we have low confidence in this finding as the majority of studies evaluating the outcome find no significant impact.

Five studies across five interventions report impacts of allocation to a summer programme on self-reported measures of engagement with, participation in and enjoyment of education activities. These studies which all focused on summer education programmes, and the outcomes they covered, were:

▪

Higher Achievement:

Herrera et al. (2013) – proportion that participated in range of academic/enrichment activities; spring after first programme summer.

Note that each of these studies source their self-reported outcome measures from surveys. The extent of attrition (overall and differential rates) for the surveys used by Hererra (2013) and Lynch and Kim (2017) to source the outcome measures is highly concerning, and moderately concerning for that used by Maxwell et al. (2014), introducing a potential source of bias into the estimates produced. Attrition is not a significant concern for the survey-based measures used by Somers et al. (2015) and Cohodes et al. (2022).

Herrera et al. (2013) also report this outcome two and four springs after first programme summer. However, as all other studies measure this outcome within 1 year after the programme summer these results are not included in the analysis. For each of the specific measures, the effect size is largest after 4 years, on average twice as large as after 1 year, whilst the effect sizes after 2 years are approximately equal to those after 1 year.

Figure 4 displays the forest plot from the meta-analysis of the impact of allocation to a summer programme on engagement with, participation in and enjoyment of education activities. As all the interventions included in this analysis are summer education programmes, there is no need to split by programme type.

OPEN IN VIEWER

Figure 4

Impact of summer programme allocation on engagement with, participation in and enjoyment of education activities. Source: IES (2024).

The average effect for summer education programmes is positive and significant (SMD = 0.12, 95% confidence interval = 0.03, 0.20), which suggests that allocation to a summer education programme has a positive impact on engagement with, participation in and enjoyment of education activities. An SMD of 0.12 translates to, for instance, approximately 60% of those allocated to the summer programme moving from never reading for fun to doing so once or twice a month as measured by Maxwell et al. (2014). Note that the two impact estimates underlying the effect size from Maxwell et al. (2014) are themselves separately insignificant – by pooling the effect sizes produced together the aggregate effect size for the study becomes marginally significant.

The p-value from the homogeneity test (p = 0.10) indicates that there is no evidence of statistically significant between-study heterogeneity. There is no significant difference in effect size between the one study of an intervention that targets ethnic minorities (Cohodes et al., 2022) and the others that do not.

As sensitivity analysis, we exclude the estimate from Herrera et al. (2013), the highest weighted study, from the analysis. Doing so reduces the average effect size estimate to 0.08, and this average effect size is now marginally insignificant (95% confidence interval = −0.00, 0.16). The effect size estimate from Herrera et al. (2013) is derived by first combining a range of dichotomous outcomes (whether the individual participated in range of academic/enrichment activities or not) before then converting to a standardised mean difference to be comparable with the other outcome measures used – this process might affect the quality of the effect size constructed. Given that it is the resulting effect size form this that drives the significance level for the overall finding for the outcome, this reduces our confidence in the security of this finding.

Additionally, one study reported impacts of participation in a summer programme on self-reported measures of engagement with, participation in and enjoyment of education activities: ▪

Elevate Math summer programme:

The extent of attrition (overall and differential rates) for the survey used by Snipes et al. (2015) to source the outcome is highly concerning, introducing a potential source of bias into the estimate produced.

Snipes et al. (2015) finds no significant impact of participation in the summer programme on engagement with, participation in and enjoyment of education activities.

Education skills, confidence and self-efficacy

Summary: ▪

Summer education programmes generally appear to have no impact on this outcome, although only three studies evaluate this outcome, one of which finds a positive impact.

One study reports impacts of allocation to a summer education programme on self-reported measures of educational skills, confidence and self-efficacy:

Note that Cohodes et al. (2022) sources their self-reported outcome measures from a survey, although the extent of attrition (overall and differential rates) is not a significant concern.

Cohodes et al. (2022) finds no significant impact of allocation to the summer education programme on educational skills, confidence and self-efficacy. There is also a consistent lack of significant impact across each of the specific indexes that they measured.

Two studies across two interventions reported impacts of participation in a summer education programme on self-reported measures of educational skills, confidence and self-efficacy. These are: ▪

Department for Education Summer Schools Programme:

Martin et al. (2013b) – pupil confidence index score/school readiness index score; autumn after the summer programme.

The extent of attrition (overall and differential rates) for the survey-based measure used by Snipes et al. (2015) is highly concerning, introducing a potential source of bias into the estimate produced. Insufficient information is provided by Martin et al. (2013b) to assess the extent to which attrition for their survey-based measure may introduce bias into the estimates produced, which is also concerning.

Martin et al. (2013b) finds that participation in the summer education programme has a positive significant impact on educational skills, confidence and self-efficacy compared with those that do not after participating in the programme, which equates to approximately 42% of those participating in the summer programme moving from agreeing to strongly agreeing that they understand most of the work at school. This translates to an effect size (SMD) of 0.07 (95% confidence interval = 0.05, 0.10).

Snipes et al. (2015) finds no significant impact of participation in the summer education programme on educational skills, confidence and self-efficacy.

Secondary education attendance

Summary: ▪

The one study of a summer education programme that evaluates this outcome finds a positive impact, although it is of moderate quality.

Six studies across three interventions report impacts of allocation to a summer programme either on secondary education (or high school in the US) attendance rates, or on the number of days attended or absent from which an attendance rate can be derived: ▪

New York City Summer Youth Employment Program:

Leos-Urbel (2014) – attendance rate; year after the summer programme.

Leos-Urbel (2014) estimates a positive significant impact of allocation to the summer employment programme on attendance rate, equal to a 1.3 percentage point increase in attendance rate which translates to an effect size (SMD) of 0.01 (95% confidence interval = 0.00, 0.01). Valentine et al. (2017) found no significant impact of allocation to the summer employment programme on attendance rate.

Modestino (2019b) finds a positive significant impact of allocation to the summer employment programme on attendance rate, equal to a 0.8 percentage point increase in attendance rate which translates to an effect size (SMD) of 0.07 (95% confidence interval = −0.02, 0.16).

Heller (2014), Davis and Heller (2020) and Heller (2022) all estimate no significant impact of allocation to the summer employment programme on attendance rate.

Eight studies across five interventions report impacts of participation in a summer programme on secondary education attendance rates, or report impacts of allocation to a summer programme which could be transformed into impacts of participation using differences in participation rates between the treatment and control groups. In addition to the six studies previously detailed, these are: ▪

Robotics Summer Learning Program:

Mac Iver and Mac Iver (2019) – attendance rate; year after the summer programme.

Figure 5 displays the forest plot from the meta-analysis of the impact of participation in a summer programme on secondary education attendance rates, split by summer education and employment programmes.

OPEN IN VIEWER

Figure 5

Impact of summer programme participation on secondary education attendance rates. Source: IES (2024).

The test of group differences indicates that the difference in the average effect size between summer education and summer employment programmes is statistically significant, therefore we do not focus on the combined findings for both summer programme types. The one evaluation of a summer education programme by Mac Iver and Mac Iver (2019) finds a statistically significant impact of participation in the summer education programme on attendance rates (SMD = 0.26, 95% confidence interval = 0.08, 0.44). Summer employment programmes have a positive although insignificant average effect size (SMD = 0.02, 95% confidence interval = −0.03, 0.07), suggesting that participation in a summer employment programme does not have a significant impact on secondary education attendance rates.

Within summer employment programmes, the p-value from the homogeneity test (p = 0.03) indicates that there is statistically significant between-study heterogeneity, reducing the applicability of the average effect size to all summer employment programmes. As there are less than 10 interventions covered in this analysis, sub-group analysis is used to examine heterogeneity. Instances of statistically significant differences on average effect size between sub-groups are: ▪

between raising aspirations and workplace exposure programmes (i.e., all the other included interventions), and those programmes that target individuals with poor academic performance and those that do not, although Mac Iver and Mac Iver (2019) is the only study that evaluated a raising aspirations programme or a programme that targeted individuals with poor academic performance; and

None of the interventions included in this analysis target individuals from ethnic minorities.

As the average effect size of summer employment programmes is insignificant, as is the effect size from the highest weighted intervention evaluated by Leos-Urbel (2014) and Valentine et al. (2017) which is weighted only slightly more highly than most of the other studies, there is no need to remove this estimate as a form of sensitivity analysis. Additionally, the estimates of impact of participation in the programmes from Leos-Urbel (2014), Valentine et al. (2017) and Modestino (2019b) are based on transformations of reported estimated impacts of allocation to the programme – removing these from the analysis has a negligible impact on the findings.

Additionally, two studies across two interventions report impacts of allocation to or participation in a summer programme on whether an individual is chronically absent: ▪

Urban Alliance:

Theodos et al. (2017) – proportion chronically absent; year after the summer programme.

Theodos et al. (2017) does not provide a definition for being chronically absent – the US Department of Education (2016) define it as missing at least 15 days of school in a year, but it is also commonly defined as missing more than 10% of the days in a school year (Centre for Research in Education & Social Policy, 2018). They do not find that allocation to the summer employment programme has a significant impact on the likelihood of being chronically absent.

Pyne et al. (2020) defines chronic absence as missing more than 10% of the days in a school year. They find that participation in the summer education programme has a significant negative (i.e., reducing the likelihood) impact on the likelihood that an individual is chronically absent, equal to a 1.4 percentage point reduction in the chronic absence rate. This translates to an effect size (log odds ratio) of −0.28 (95% confidence interval = −0.69, 0.14 – note that the specifics of the process for constructing the effect size means that it is insignificant despite the underlying impact estimate being significant) and an odds ratio of 0.76 (95% confidence interval = 0.50, 1.15). So, an individual who participated in the summer education programme is 0.76 times as likely to be chronically absent across the 3 years after the summer programme as someone who did not.

Passing tests

Summary: ▪

Summer education programmes appear to have no impact on this outcome – two of the three studies evaluating the outcome find a positive impact, although one of these is an outlier which may be explained by the specifics of the measure they use.

Six studies across five interventions report impacts of allocation to a summer programme on the likelihood of passing tests: ▪

Texas developmental summer bridge programme:

Wathington et al. (2016) – probability of passing first college-level mathematics/reading/writing course; within 2 years after the summer programme.

The extent of attrition (overall and differential rates) for the post-intervention test used to source the outcome measure for Snipes et al. (2015) is highly concerning, potentially introducing a source of bias into the estimates. For Cohodes et al. (2022) which uses a survey-based measure, the extent of attrition is not concerning.

Figure 6 displays the forest plot from the meta-analysis of the impact of allocation to a summer programme on the likelihood of passing tests, split by summer education and employment programmes.

OPEN IN VIEWER

Figure 6

Impact of summer programme allocation on likelihood of passing tests. Source: IES (2024).

Summer programmes and summer education programmes have positive although insignificant average effect sizes (log odds ratio (summer programmes) = 0.20, 95% confidence interval = −0.10, 0.50; log odds ratio (summer education programmes) = 0.41, 95% confidence interval = −0.13, 0.96), suggesting that allocation to a summer programme or summer education programme does not have a significant impact on the likelihood of an individual to pass tests. The average effect size for summer employment programmes is positive and statistically significant (log odds ratio = 0.02, 95% confidence interval = 0.00, 0.04). A log odds ratio of 0.02 translates to an odds ratio of 1.02, which means that individuals who were allocated to a summer employment programme are 1.02 times more likely to pass tests after the programme than those who were not. Moreover, using as the assumed comparator risk the average in the proportions of control individuals passing the test from Leos-Urbel (2014) and Schwartz et al. (2021), this equates to a need to treat 206 young people to see this outcome for one of them as a result of allocation to a summer programme. The test of group differences indicates that there is not a significant difference on average effect size between education and summer employment programmes.

Within summer education programmes, the p-value from the homogeneity test (p = 0.00) indicates that there is statistically significant between-study heterogeneity, reducing the applicability of the average effect size to all summer education programmes. This is not the case for summer employment programmes (p = 0.33). As there are less than 10 interventions covered in this analysis, sub-group analysis is used to examine heterogeneity. Instances of statistically significant differences on average effect size between sub-groups are: ▪

between intervention types – the one catch-up programme studied by Snipes et al. (2015) has a much larger effect size than the one raising aspirations intervention studied by Cohodes et al. (2022), the one transition support programme studied by Wathington et al. (2016), or the two workplace exposure interventions (average effect size = 0.02, 95% confidence interval = 0.00, 0.04).

There is no significant difference in effect size between the one programme that targets individuals from ethnic minorities, studied by Cohodes et al. (2022), and the other studies that do not.

The significance of the average effect size from summer employment programmes is driven by the combined result from Leos-Urbel (2014) and Schwartz et al. (2021). It should be noted that the test that Leos-Urbel (2014) and Schwartz et al. (2021) use to measure this outcome, the Regents Exam, is an elective, non-universal examination that can be taken multiple times, therefore the positive impact they estimate on the likelihood of passing the test may be a result of both an increase in ability as well as an increased willingness to take the test (Schwartz et al., 2021) estimates positive effects on the likelihood of taking the Regents Exam). Removing the result from these studies naturally results in the average effect size of summer employment programmes mirroring the effect size from Modestino (2019b), that is, the only other summer employment programme.

Additionally, Schwartz et al. (2021) estimates the impact of participation in the summer employment programme on the likelihood of passing tests. The result is broadly in line with their estimate of the impact of allocation to the summer employment programme on this outcome, that is, a positive and significant impact, equal to a 0.9 percentage point increase in the likelihood of passing the test which translates to an effect size (log odds ratio) of 0.04 (95% confidence interval = 0.01, 0.06).

Test scores

Summary: ▪

Summer education programmes appear to have a positive impact on any form of test score and English test scores specifically, although they appear to have no impact on reading, writing or mathematics scores.

A large number of studies report estimates of impact of allocation to/participation in a summer programme on scores across a wide range of tests. To explore these outcomes we first look at the impact of summer programmes on test scores within specific areas of English language – reading, writing, spelling and grammar, listening, and speaking.

Five studies across five interventions report impacts of allocation to a summer programme on reading scores: ▪

Higher Achievement:

Herrera et al. (2013) – standardised Stanford Achievement Test 10 reading score; spring after first programme summer.

The extent of attrition (overall and differential rates) to the post-programme tests used by Herrera et al. (2013) and Torgerson et al. (2014) are highly concerning, and are moderately concerning for Maxwell et al. (2014), introducing a potential source of bias into the estimates produced. Attrition is not a significant concern for Somers et al. (2015).

The specific measure used by Herrera et al. (2013) is referenced in Garcia et al. (2020). Herrera et al. (2013) also report this outcome two and four springs after the first programme summer. However, as all other studies measure this outcome within 1 year after the programme summer these results are not included in the analysis (they also do not find significant impacts at these time points).

Figure 7 displays the forest plot from the meta-analysis of the impact of allocation to a summer programme on reading scores. As all the interventions included in this analysis are summer education programmes, the analysis is not split by programme type.

OPEN IN VIEWER

Figure 7

Impact of summer programme allocation on reading scores. Source: IES (2024).

The average effect size for summer education programmes is positive although not significant (SMD = 0.01, 95% confidence interval = −0.04, 0.05), suggesting that allocation to a summer education programme does not have a significant impact on reading scores.

The p-value from the homogeneity test (p = 0.48) indicates that there is no statistically significant evidence of between-study heterogeneity. None of the interventions included in this analysis targeted individuals from ethnic minorities.

Removing the result from Herrera et al. (2013), by far the highest weighted study, means that the average effect size becomes negative (SMD = −0.01) although it is still insignificant (95% confidence interval = −0.10, 0.08).

Five studies across five interventions report impacts of participation in a summer programme on reading scores or report impacts of allocation to a summer programme which could be transformed into impacts of participation. The estimates of impact of allocation to the summer programme on reading scores noted above for Herrera et al. (2013) and Johnson (2020) cannot be translated to estimates of impact of participation in the summer programme, as they report standardised test scores with insufficient information to transform them. It is possible to convert the results from Somers et al. (2015), Maxwell et al. (2014) and Torgerson et al. (2014) previously detailed, and two further studies can be added into this analysis: ▪

Future Foundations summer school programme:

Siddiqui et al. (2014) – Key Stage 2 to post-test reading gain score; autumn after the summer programme.

The extent of attrition (overall and differential rates) to the post-programme test administered as part of the evaluation in Siddiqui et al. (2014) is highly concerning, introducing a potential source of bias into the estimate produced.

Siddiqui et al. (2014) and Williamson et al. (2020) both use gain scores (i.e., the change in score) as their outcome measures. By comparing differences in these between the treatment and control/comparison group, measures of the impact of the summer programme on academic performance at the point in time of the post-test are found, and therefore they can be considered alongside purely cross-sectional measures.

Figure 8 displays the forest plot from the meta-analysis of the impact of participation in a summer programme on reading scores. As all the interventions included in this analysis are summer education programmes, the analysis is not split by programme type.

OPEN IN VIEWER

Figure 8

Impact of summer programme participation on reading scores. Source: IES (2024).

The average effect size for summer education programmes is positive although insignificant (SMD = 0.04, 95% confidence interval = −0.06, 0.14), suggesting that participation in a summer education programme does not have a significant impact on reading scores.

The p-value from the homogeneity test (p = 0.78) indicates that there is no statistically significant evidence of between-study heterogeneity. None of the interventions included in this analysis targeted individuals from ethnic minorities.

Removing the result from Somers et al. (2015), by far the highest weighted study, has a negligible impact on the findings. Removing both the results from Somers et al. (2015) and Torgerson et al. (2014), that is, the two studies that report impacts of allocation that require transformation, also has a negligible impact on the findings.

The third investigation is into the effect of summer programmes on writing scores. Two studies across two interventions report the impacts of allocation to an educational summer programme on writing scores. These are: ▪

Discover Summer School:

Torgerson et al. (2014) – Progress in English writing score; 3 weeks after the summer programme.

The extent of attrition (overall and differential rates) to the post-programme test administered as part of the evaluation in Torgerson et al. (2014) is highly concerning, introducing a potential source of bias into the estimate produced.

Torgerson et al. (2014) finds no significant impact of allocation to the summer education programme on writing scores.

Johnson (2020) finds a positive significant impact of allocation to the summer education programme on writing scores, equating to an effect size (z-score) of 0.11 (95% confidence interval = 0.02, 0.20). It is worth noting this in comparison with the negative although insignificant effect of allocation to the summer education programme they estimated for reading scores. Johnson (2020) suggests as an explanation for this that reading skills may tend to develop more gradually compared with writing skills, and therefore the short duration of summer programmes may not allow sufficient time for as considerable an improvement in reading ability compared with writing.

Two studies across two interventions report impacts of participation in a summer education programme on writing scores: ▪

Future Foundations summer school programme:

The extent of attrition (overall and differential rates) to the post-programme test administered as part of the evaluation in Siddiqui et al. (2014) is highly concerning, introducing a potential source of bias into the estimates produced.

The estimate of impact of allocation to the summer programme on writing scores noted above for Johnson (2020) cannot be translated to estimates of impact of participation in the summer programme, as they report standardised test scores with insufficient information to transform them. Therefore, meta-analysis is not performed specifically for this outcome as only three studies across three interventions produce results that could be synthesised together, and so we discuss the results from Siddiqui et al. (2014) and Williamson et al. (2020) narratively.

Siddiqui et al. (2014) finds no significant impact of participation in the summer education programme on writing scores.

Williamson et al. (2020) estimates a positive significant impact of participation in the summer education programme on writing scores, equating to an effect size (SMD) of 0.22 (95% confidence interval = 0.05, 0.39). This study has a low-quality study design, and the intervention studied is the only one included in this review occurring outside the UK and US (in New Zealand).

Two studies report impacts of allocation to a summer education programme on other types of (English) language test scores: ▪

Discover Summer School:

The extent of attrition (overall and differential rates) to the post-programme test administered as part of the evaluation in Torgerson et al. (2014) is highly concerning, introducing a potential source of bias into the estimate produced.

Torgerson et al. (2014) finds no significant impact of allocation to the summer education programme on spelling and grammar scores.

Johnson (2020) finds a positive significant impact of allocation to the summer education programme on listening scores and on speaking scores, equating to an effect size (z-score) of 0.18 (95% confidence interval = 0.04, 0.31) for listening and of 0.16 (95% confidence interval = 0.06, 0.26) for speaking.

Eight studies across eight interventions report impacts of allocation to a summer programme (either education or employment) on English scores, either for the subject as a whole or for specific areas which can be treated (either on their own or in combination across areas) as the impact on English scores. These are: ▪

Future Foundations summer school programme:

Gorard et al. (2014) – Key Stage 2 to GL Assessment Progress in English gain score; autumn after the summer programme.

The specific measure used by Herrera et al. (2013) is referenced in Garcia et al. (2020).

Figure 9 displays the forest plot from the meta-analysis of the impact of allocation to a summer programme on English scores, split by summer education and employment programmes.

OPEN IN VIEWER

Figure 9

Impact of summer programme allocation on English scores. Source: IES (2024).

The test of group differences indicates that the difference in the average effect size between summer education and summer employment programmes is statistically significant, therefore we do not focus on the combined findings for both summer programme types. The average effect size for summer education programmes is positive and significant (SMD = 0.07, 95% confidence interval = 0.00, 0.13, p-value = 0.04), suggesting that individuals who are allocated to a summer education programme have significantly better English scores than those that are not post-programme. An SMD of 0.07 translates to, for instance, an increase in the English Grade Point Average (as measured by Modestino, 2019b) of 0.08 for those allocated to a summer education programme. The average effect size for summer employment programmes on English scores is negative and statistically significant (SMD = −0.03, 95% confidence interval = −0.05, −0.01), suggesting that individuals who are allocated to a summer employment programme have significantly worse English scores than those that are not post-programme.

Within summer education and summer employment programmes, the p-values from the homogeneity tests (p = 0.33 and p = 0.76 respectively) indicate that there is no statistically significant evidence of between-study heterogeneity within the two groups of summer programme, although across all interventions there is statistically significant heterogeneity (p = 0.03). As there are less than 10 interventions covered in this analysis, sub-group analysis is used to examine heterogeneity. Instances of statistically significant differences on average effect size between sub-groups are: ▪

between intervention types – the one transition support programme, studied by Maxwell et al. (2014), has the largest effect size, followed by catch-up programmes (SMD = 0.10, 95% confidence interval = 0.02, 0.18) whilst the one raising aspirations programme, studied by Herrera et al. (2013), and workplace exposure interventions (SMD = −0.03, 95% confidence interval = −0.06, −0.01) have much lower effect sizes;

None of the interventions included in this analysis targeted individuals from ethnic minorities.

The finding is largely driven by the result from Leos-Urbel (2014) who finds that allocation to the summer employment programme has a statistically significant negative impact on English Regents exam scores. It is suggested that this finding is likely a result of the summer employment programme encouraging lower performing students to take this elective, non-universal examination meaning that the average score of treatment individuals that take the test is lowered due to this ‘compositional’ effect, rather than allocation to a summer programme resulting in an actual decrease in English attainment. Removing the results from Leos-Urbel (2014) and Herrera et al. (2013), as these are by far the highest weighted studies, increases the average effect size (SMD) of summer employment programmes to 0.10 which now becomes significant (95% confidence interval = 0.03, 0.17) and increases the average effect size of summer education programmes to 0.10 (95% confidence interval = 0.03, 0.18) respectively.

The next analysis focuses on the effects of participation in summer programmes on English scores. Nine studies across eight interventions report impacts of participation in a summer programme on English scores, or report impacts of allocation to a summer programme which could be transformed into impacts of participation. The estimates of impact of allocation to the summer programme previously outlined from Leos-Urbel (2014), Herrera et al. (2013), Johnson (2020), and one of the measures evaluated by Modestino (2019b) cannot be translated to estimates of impact of participation in the summer programme, as they report standardised test scores with insufficient information to transform them. Hence, in addition to those others previously outlined, studies that can be included in this analysis are: ▪

Future Foundations summer school programme:

Figure 10 displays the forest plot from the meta-analysis of the impact of participation in a summer programme on English scores, split by summer education and employment programmes.

OPEN IN VIEWER

Figure 10

Impact of summer programme participation on English scores. Source: IES (2024).

Despite the test of group differences not finding evidence of statistically significant differences on average effect size between summer education and summer employment programmes, the overall findings for the two programme types do differ. The average effect size for summer education programmes is positive and significant (SMD = 0.07, 95% confidence interval = 0.03, 0.10), suggesting that individuals who participate in a summer education programme have significantly better English scores than those who do not post-programme. An SMD of 0.07 translates to, for instance, an increase in the English Grade Point Average (as measured by Modestino (2019b)) of 0.07 for those participating in a summer education programme. The one evaluation of a summer employment programme by Modestino (2019b) finds no statistically significant impact of participation in the summer employment programme on English test scores.

The p-value from the homogeneity test (p = 0.77) suggests that there is no statistically significant heterogeneity. None of the interventions evaluating this outcome target individuals from ethnic minorities.

The average effect size for summer education programmes is largely driven by the results from Mariano and Martorell (2013) and Pyne et al. (2020). Removing the results from these studies reduces this to 0.09 which is marginally insignificant (95% confidence interval = −0.00, 0.18), as well as the average effect size for all summer programmes to 0.06 which is also now insignificant (95% confidence interval = −0.01, 0.12). Removing the results from Gorard et al. (2014), Torgerson et al. (2014), Somers et al. (2015) and Modestino (2019b), so from those studies that report impacts of allocation to a summer programme that are transformed into impacts of participation, has a negligible effect on the findings.

The next outcome in focus is mathematics scores. Seven studies across seven interventions report impacts of allocation to a summer education programme on mathematics scores. These are:

Future Foundations summer school programme:

The extent of attrition (overall and differential rates) to the post-programme test administered as part of the evaluation in Herrera et al. (2013) and Snipes et al. (2015) are highly concerning, and moderately concerning for Somers et al. (2015), introducing a potential source of bias into the estimates produced. The extent of attrition to the post-programme test administered as part of the evaluation in Gorard et al. (2014) is not concerning.

The specific measure used by Herrera et al. (2013) is referenced in Garcia et al. (2020). Herrera et al. (2013) also reports this outcome two and four springs after the first programme summer, however as all other studies measure this outcome within 1 year after the programme summer these results are not included in the analysis. They estimate impacts two and four springs after the first programme summer that translate to effect sizes (SMD) of 0.10 (95% confidence interval = 0.00, 0.20) and 0.11 (95% confidence interval = 0.00, 0.22) respectively.

Figure 11 displays the forest plot from the meta-analysis of the impact of allocation to a summer programme on mathematics scores, split by summer education and employment programmes.

OPEN IN VIEWER

Figure 11

Impact of summer programme allocation on mathematics scores. Source: IES (2024).

Both summer education and summer employment programmes have positive (and larger than their equivalent for English scores) although insignificant average effect sizes (SMD (summer education programmes) = 0.14, 95% confidence interval = −0.09, 0.36; SMD (summer employment programmes) = 0.00, 95% confidence interval = −0.04, 0.05). This suggests that allocation to a summer programme of either type has no impact on mathematics scores. The test of group differences indicates that there is no statistically significant difference on average effect size between summer education and employment programmes.

Within the summer education programmes sub-group, the p-value from the homogeneity test (p = 0.00) indicates that there is statistically significant evidence of between-study heterogeneity, reducing the applicability of the average effect size to all summer education programmes. This is not the case for summer employment programmes (p = 0.92). There are no statistically significant differences on average effect size between any sub-groups of studies (by region of the intervention, intervention type, ‘in whole’ vs. ‘in part’ summer programmes, forms of disadvantage targeted, or study quality). None of the interventions evaluating this outcome targeted individuals from ethnic minorities.

Removing the result from Snipes et al. (2015), which is somewhat an outlier, reduces the average effect size of summer education programmes which remains insignificant (SMD = 0.02, 95% interval = −0.04, 0.07).

Turning to the effect of participation in summer programmes, seven studies across six interventions report the impact of this on mathematics scores, or impacts of allocation to a summer programme that could be transformed into impacts of participation. The estimates of impact of allocation to the summer programme previously outlined from Leos-Urbel (2014), Herrera et al. (2013), Lynch and Kim (2017) and one of measures evaluated by Modestino (2019b) cannot be translated to estimates of impact of participation in the summer programme, as they report standardised test scores with insufficient information to transform them. In addition to those others previously outlined, studies that can be included are: ▪

Future Foundations summer school programme:

The extent of attrition (overall and differential rates) to the post-programme test administered as part of the evaluation in Siddiqui et al. (2014) is highly concerning, introducing a potential source of bias into the estimate produced.

Figure 12 displays the forest plot from the meta-analysis of the impact of participation in a summer programme on mathematics scores, split by summer education and employment programmes.

OPEN IN VIEWER

Figure 12

Impact of summer programme participation on mathematics scores. Source: IES (2024).

Again, both summer education and summer employment programmes (noting that there is only one evaluation of the latter by Modestino (2019b)) have positive (and larger than their equivalent for English scores) although insignificant average effect sizes (SMD (summer education programmes) = 0.12, 95% confidence interval = −0.08, 0.32; SMD (Modestino, 2019b) = 0.00, 95% confidence interval = −0.10, 0.10). This suggests that participation in a summer programme of either type has no impact on mathematics scores. The test of group differences indicates that the difference on average effect size between summer education and employment programmes is not statistically significant.

Within summer education programmes, the p-value from the homogeneity test (p = 0.00) indicates that there is statistically significant evidence of between-study heterogeneity, reducing the applicability of the average effect size to all summer education programmes. The only statistically significant difference on average effect between sub-groups is between ‘in part’ (effect size = 0.57, 95% confidence interval = 0.36, 0.79) and ‘in whole’ summer programmes (effect size = 0.01, 95% confidence interval = −0.02, 0.05). However, this is because Snipes et al. (2015) finds a much larger effect size than any of the other studies included in this analysis and is the only ‘in part’ summer programme. None of the interventions included in this analysis targeted individuals from ethnic minorities.

Removing the result from Snipes et al. (2015) from the analysis reduces the average effect size of summer education which remains insignificant (SMD = 0.02, 95% confidence interval = −0.03, 0.06). Additionally, removing the results from Gorard et al. (2014), Somers et al. (2015) and Modestino (2019b) along with Snipes et al. (2015), that is, all studies that report impacts of allocation to a summer programme that are transformed into impacts of participation, has a negligible effect on the average effect size compared with just removing the result from Snipes et al. (2015).

Comparing studies that evaluate impacts on both English and mathematics scores, there is an even split between those that find greater effects for English than mathematics (Gorard et al., 2014; Mariano & Martorell, 2013; Modestino, 2019b; Siddiqui et al., 2014) and those that find greater effects for mathematics than English (Herrera et al., 2013; Leos-Urbel, 2014; Somers et al., 2015), although it should be noted that the majority of these studies find no significant impact on either outcome. Where greater effects or impacts are found on mathematics scores, reasons suggested by the studies are that this could be due to a difference in the quality of the teaching observed, that mathematics is less susceptible than literacy to summer learning loss (Gorard et al., 2014), that students may be more engaged in the mathematics curriculum, or that English curricula may not be as effective with students below the expected level compared with the mathematics curricula (Somers et al., 2015). In studies that find a greater effect or impact on English, there is not sufficient evidence provided to understand why this may be the case.

Next, we consider the five studies across three interventions that report impacts of allocation to a summer employment programme on overall academic scores. These are: ▪

Boston Summer Youth Employment Program:

None of these studies find a significant impact of allocation to the summer programme on overall academic scores.

Turning to the effect of participation on this outcome measure, six studies across four interventions report impacts of participation in a summer programme on overall academic scores, or report impacts of allocation to a summer programme which could be transformed into impacts of participation. In addition to those previously outlined, the other is: ▪

STEP-UP:

Figure 13 displays the forest plot from the meta-analysis of the impact of participation in a summer programme on overall academic scores. As all the interventions included in this analysis are summer employment programmes, there is no need to split by programme type.

OPEN IN VIEWER

Figure 13

Impact of summer programme participation on overall test scores. Source: IES (2024).

The average effect size for summer employment programmes is negative although insignificant (SMD = −0.01, 95% confidence interval = −0.08, 0.06), suggesting that participation in a summer employment programme does not have a significant impact on overall test scores. When pooled together, the previously mentioned insignificant impacts from the three studies evaluating One Summer Chicago (once transformed to impacts of participation) produces a significant effect size (SMD = −0.05, 95% confidence interval = −0.10, 0.00).

The p-value from the homogeneity test (p = 0.20) indicates that there is no statistically significant evidence of between-study heterogeneity. None of the interventions included in this analysis targeted individuals from ethnic minorities.

Removing the result from Modestino (2019b) as it is the only study in this analysis that reports impacts of allocation that require transforming to impacts of participation, or the combined result from the studies evaluating One Summer Chicago because this is the highest weighted effect size, has negligible impacts on the overall findings.

Lastly, the impacts of summer programmes on English, mathematics and overall academic scores are pooled. Fourteen studies across 12 interventions report impacts of allocation to a summer education programme on some test scores: ▪

Future Foundations summer school programme:

Figure 14 displays the forest plot from the meta-analysis of the impact of allocation to a summer programme on test scores, split by summer education and employment programmes.

OPEN IN VIEWER

Figure 14

Impact of summer programme allocation on all test scores. Source: IES (2024).

The test of group differences indicates that the difference in the average effect size between summer education and summer employment programmes is statistically significant, therefore we do not focus on the combined findings for both summer programme types. The average effect size for summer education programmes is positive and statistically significant (SMD = 0.14, 95% confidence interval = 0.00, 0.27), suggesting that individuals who are allocated to a summer education programme have significantly better test scores following the programme than those who are not. An SMD of 0.14 translates to, for instance, an increase in overall Grade Point Average (as measured by Modestino (2019b)) of 0.14 for those allocated to a summer education programme. In contrast, summer employment programmes have a negative although insignificant average effect size (SMD = −0.01, 95% confidence interval = −0.04, 0.01), suggesting that allocation to a summer employment programme does not have a significant impact on test scores post-programme.

Within the summer education programmes sub-group, the p-value from the homogeneity test (p = 0.00) indicates that there is statistically significant evidence of between-study heterogeneity, reducing the applicability of the average effect size to all summer education programmes. This is not the case for summer employment programmes (p = 0.73). As there are 12 interventions covered by this analysis, meta-regression as well as sub-group analysis are used to examine heterogeneity. The only statistically significant differences on average effect size between sub-groups are: ▪

between the one programme that targets disadvantaged areas (schools) with a high proportion of individuals with experience of or at risk of involvement with the criminal justice system, studied by Heller (2014), Davis and Heller (2020) and Heller (2022) (SMD = −0.02, 95% confidence interval = −0.07, 0.03), and those programmes that do not (SMD = 0.09, 95% confidence interval = −0.01, 0.18) (given this is based on a sub-group of one study, the external validity of this finding is quite limited); and

None of the interventions included in this analysis targeted individuals from ethnic minorities.

Removing the result from Snipes et al. (2015), which is something of an outlier, reduces the average effect size of summer education programmes to 0.04 which is still marginally significant (95% confidence interval = 0.00, 0.08).

Table 8 displays the results from running meta-regression for this outcome.

The results from the meta-regression indicate that, after controlling for other factors, there are no statistically significant differences on average effect size across any of the sub-groups included. Some of the estimated coefficients are relatively large however, particularly for the dummy of whether the intervention was an ‘in whole’ summer programme and whether it targeted socioeconomically disadvantaged areas. The high degree of overlap between various sub-groups, as well as the relatively small number of studies included in this analysis, is likely partly responsible for a lack of statistically significant differences. For instance, removing the study quality moderator, which is highly insignificant, results in the estimated differences in effect size between ‘in whole’ and ‘in part’ summer programmes and those that do and do not target socioeconomically disadvantaged areas becoming statistically significant.

Fifteen studies across 12 interventions report impacts of participation in a summer programme on test scores, or report impacts of allocation that could be transformed into impacts of participation. The estimates of impact of allocation to the summer programme previously outlined from Leos-Urbel (2014), Herrera et al. (2013), Johnson (2020) and Lynch and Kim (2017) cannot be translated to estimates of impact of participation in the summer programme, as they report standardised test scores with insufficient information to transform them. The additional studies in this analysis are: ▪

Summer Learning Journey:

Figure 15 displays the forest plot from the meta-analysis of the impact of participation in a summer programme on test scores, split by summer education and employment programmes.

OPEN IN VIEWER

Figure 15

Impact of summer programme participation on all test scores. Source: IES (2024).

The test of group differences indicates that the difference in the average effect size between summer education and summer employment programmes is statistically significant, therefore we do not focus on the combined findings for both summer programme types. The average effect size of summer education programmes is positive and statistically significant (SMD = 0.13, 95% confidence interval = 0.01, 0.26), suggesting that participation in a summer education programme has a positive impact on test scores. An SMD of 0.13 translates to, for example, an increase in overall Grade Point Average (as measured by Modestino (2019b)) of 0.13 for those participating in a summer education programme. Summer employment programmes have a negative although insignificant average effect size (SMD = −0.01, 95% confidence interval = −0.08, 0.06), suggesting that participation in a summer employment programme does not have a significant impact on test scores.

Within the summer education programmes sub-group, the p-value from the homogeneity test (p = 0.00) indicates that there is statistically significant evidence of between-study heterogeneity, reducing the applicability of the average effect size to all summer education programmes. This is not the case for summer employment programmes (p = 0.20). As there are 12 interventions covered in this analysis, meta-regression as well as sub-group analysis are used to examine heterogeneity. Instances of statistically significant differences on average effect size between sub-groups are: ▪

between the one programme that targets areas (schools) with a high proportion of individuals with experience of or at risk of involvement with the criminal justice system, studied by Heller (2014), Davis and Heller (2020) and Heller (2022) (effect size (SMD) = −0.04, 95% confidence interval = −0.12, 0.4), and those programmes that do not (SMD = 0.09, 95% confidence interval = −0.01, 0.18); and

As both of these findings are based on a sub-group of one study, the external validity of these findings are quite limited. None of the interventions included in this analysis targeted individuals from ethnic minorities.

Removing the result from Snipes et al. (2015), as an outlier, from the analysis reduces the average effect size (SMD) of summer education programmes to 0.05 which remains significant (95% confidence interval = 0.02, 0.08), and of all summer programmes to 0.03 which becomes marginally insignificant (95% confidence interval = −0.00, 0.07). Removing results that are produced by transforming impacts of allocation to impacts of participation has a negligible impact on the findings.

Table 9 displays the results from running meta-regression for this outcome (a dummy for interventions targeting socioeconomically disadvantaged areas was not included as these largely overlap with interventions targeting socioeconomically disadvantaged areas).

The results from the meta-regression indicate that, after controlling for other factors, the average effect size of ‘in whole’ summer programmes is 0.52 lower than ‘in part’ summer programmes which is significant at the 95% confidence level. Differences across the other sub-groups are not significant. Given the relatively small number of studies included in this analysis, and the significant overlap between various sub-groups, these results should be treated with a degree of caution.

On-time completion of secondary education

Summary: ▪

The one study of a summer education programme that evaluates this outcome finds no impact, although it is of low quality.

Three studies across three interventions report impacts of allocation to/participation in a summer programme on the likelihood of completing secondary education on time: ▪

New York City Summer Youth Employment Program:

Theodos et al. (2017) also evaluates the likelihood of graduating high school, however it is not certain that individuals within the sample will be graduating on time. In the context of the group of individuals studied by Johnson (2020) (ESOL learners that are recent migrants to the US), graduation from high school in 5 years constitutes graduating on time.

Valentine et al. (2017) found that allocation to the summer employment programme had no significant impact on the proportion of students that graduated high school on time.

Johnson (2020) found that allocation to the summer education programme had no significant impact on the proportion of students that graduated high school on time.

Reich (2018) found that participation in the summer employment programme had a significant impact on the likelihood of completing secondary education on time, equating to an effect size (log odds ratio) of 0.96 (95% confidence interval = 0.32, 1.60). This translates to an odds ratio of 2.61 (95% confidence interval = 1.38, 4.95), meaning that individuals who participate in the summer employment programme are 2.61 times more likely to complete secondary education on time than those that do not. To support interpretation of these findings, it is important to note that the two studies that find larger positive effect sizes, Johnson (2020) and Reich (2018) – whilst insignificant, the central estimate of the log odds ratio from the latter is 0.78 – have low-quality study designs, while Valentine et al. (2017), who finds a much smaller and overall insignificant effect size, has a high-quality study design.

Negative behavioural outcomes

Summary: ▪

The one study of a summer education programme that evaluates this outcome finds a negative (i.e., beneficial) impact, although it is of low quality.

Three studies across three interventions report impacts of allocation to a summer programme on the likelihood of having a negative behavioural outcome post-programme:

▪

Urban Alliance:

Theodos et al. (2017) does not find that allocation to the summer employment programme has a significant impact on the likelihood of being suspended or chronically absent.

Modestino (2019b) found that allocation to the summer employment programme had no significant impact on the likelihood of having a disciplinary incident.

Heller (2022) found that allocation to the summer employment programme had a significant positive impact on the likelihood an individual had a suspension, equating to an effect size (log odds ratio) of 0.17 (95% confidence interval = 0.01, 0.34) and an odds ratio of 1.19 (95% confidence interval = 1.01, 1.40). So, an individual who participated in the summer employment programme was 1.19 times more likely to have a suspension in the first year post-randomisation that someone who did not.

Five studies across five interventions report impacts of participation in a summer programme on the likelihood of having a negative behavioural outcome, or report impacts of allocation to a summer programme which could be transformed into impacts of participation. In addition to the three studies previously detailed, the others are: ▪

STEP-UP:

The extent of attrition (overall and differential rates) for the self-reported survey-based outcome measure used by Snipes et al. (2015) is highly concerning, introducing a potential source of bias into the estimate produced.

Reich (2018) largely finds that participation in the summer employment programme has no significant impact on the likelihood of having a negative behavioural outcome, apart from the likelihood of having a suspension where one of the two treatment arms has a significant positive (i.e., increasing the likelihood) impact.

Pyne et al. (2020) found that participation in the summer education programme has a significant negative (i.e., reducing the likelihood) impact on the likelihood that an individual has a suspension, equating to an effect size (log odds ratio) of −1.55 (95% confidence interval = −3.13, 0.03 – note that the specifics of the process for constructing the effect size means that the constructed effect size is insignificant) and an odds ratio of 0.21 (95% confidence interval = 0.04, 1.03). So, an individual who participated in the summer education programme is 0.21 times as likely to have a suspension across the 3 years after the summer programme as someone who did not.

Sum (2015) found that participation in a summer employment programme has no significant impact on the proportion of individuals who went from reporting that they had skipped classes without an excuse to not doing so from the start to the end of the summer (insufficient information was available to construct an effect size, therefore this is not included in the meta-analysis that follows).

Figure 16 displays the forest plot from the meta-analysis of the impact of participation in a summer programme on the likelihood of having a negative behavioural outcome, split by summer education and employment programmes.

OPEN IN VIEWER

Figure 16

Impact of summer programme participation on likelihood of having negative behavioural outcome. Source: IES (2024).

The average effect size for summer education programmes is negative whilst for summer employment programmes it is positive, although both are insignificant (log odds ratio (summer education programmes) = −1.55, 95% confidence interval = −3.14, 0.03; log odds ratio (summer employment programmes) = 0.05, 95% confidence interval = −0.43, 0.54). This suggests that participation in either a summer education nor employment programme does not have a significant impact on the likelihood that an individual has a negative behavioural outcome. Note that this finding for summer education programmes is only based on the result from one study. The test of group differences indicates that there is no statistically significant difference on average effect size between summer education and employment programmes.

Within summer employment programmes, the p-value from the homogeneity test (p = 0.00) indicates that there is evidence of between-study heterogeneity, reducing the applicability of the average effect size to all summer employment programmes. As there are less than 10 interventions covered in this analysis, sub-group analysis is used to examine heterogeneity: ▪

the one programme that targeted areas containing individuals with experience of or at risk of involvement with the criminal justice system, studied by Heller (2022), had a significantly different (higher, i.e., less of a reduction) effect size to those programmes that did not target these areas (average effect size (log odds ratio) = −0.24, 95% confidence interval = −0.59, 0.11).

None of the interventions included in this analysis targeted individuals from ethnic minorities.

Application to higher education

Summary: ▪

Consistent finding across studies of both summer programme types of no impact on this outcome, although this is based on only three studies.

Three studies across three interventions report impacts of allocation to/participation in a summer programme on the likelihood of applying to higher education. These are:

▪

Widening participation summer schools:

The extent of attrition (overall and differential rates) for the self-reported survey-based outcome measures used by Taylor (2022) and Theodos et al. (2017) are highly concerning, introducing a potential source of bias into the estimates produced.

Taylor (2022) finds no significant impact of allocation to the summer education programme on the likelihood of applying to higher education.

Cohodes et al. (2022) finds no significant impact of allocation to the summer education programme on the likelihood of applying to higher education.

Note that Taylor (2022) and Cohodes et al. (2022) study interventions that target ethnic minority young people.

Theodos et al. (2017) finds no significant impact of allocation to or participation in the summer employment programme on the likelihood of applying to higher education. Oddly, the reported estimated impact of participation in the summer employment programme is less than the reported estimated impact of allocation to the summer employment programme – this might suggest that drop-out from the treatment group was non-random.

Progression to higher education

Summary: ▪

Summer education programmes appear to have no impact on this outcome, although two of the five studies evaluating this find positive impacts.

Nine studies across eight interventions report impacts of allocation to a summer programme on the likelihood of attending higher education: ▪

New York City Summer Youth Employment Program:

The measure used by Modestino and Paulsen (2019a) is a somewhat different measures to those evaluated by other studies – intention to progress to higher education as opposed to realised progression to higher education – which should be borne in mind. The survey that this outcome is sourced from also has highly differential response rates between the treatment and control groups, introducing a potential source of bias into the estimates produced.

Figure 17 displays the forest plot from the meta-analysis of the impact of allocation to a summer programme on the likelihood of attending higher education, split by summer education and employment programmes.

OPEN IN VIEWER

Figure 17

Impact of summer programme allocation on likelihood of attending higher education. Source: IES (2024).

Summer education programmes have a positive although insignificant average effect size (SMD = 0.32, 95% confidence interval = −0.12, 0.76), suggesting that allocation to a summer education programme does not have a significant impact on the likelihood of attending higher education. Summer employment programmes also have a positive although insignificant average effect size (SMD = 0.10, 95% confidence interval = −0.07, 0.26) suggesting that allocation to a summer employment programme does not have a significant impact on the likelihood of attending higher education. The test of group differences indicates that there is no significant difference on average effect size between summer education and employment programmes. Overall, allocation to a summer programme has a positive although insignificant average effect size (SMD = 0.24, 95% confidence interval = −0.04, 0.52), suggesting that allocation to a summer programme does not have a significant impact on the likelihood of attending higher education.

Within each of the sub-groups, the p-values from the homogeneity tests (summer education programmes: p = 0.00; summer employment programmes: p = 0.02) indicate that there is evidence of statistically significant between-study heterogeneity, reducing the applicability of the average effect sizes to all summer programmes of each type. As there are less than 10 interventions covered in this analysis, we use further sub-group analysis to examine heterogeneity. There are no statistically significant differences on average effect size between any sub-groups of studies (by region of the intervention, intervention type, whether the intervention is an ‘in whole’ or ‘in part’ summer programme, the forms of disadvantage targeted, or study quality). Cohodes et al. (2022) and Robles (2018) study the one intervention evaluating this outcome that targets individuals from ethnic minorities (the impact estimates underlying the constructed effect size are themselves insignificant apart from one of the three treatment arms studied by Cohodes et al. (2022) 5 years after the summer programme).

Removing the result from Gehring et al. (2018) which is somewhat an outlier reduces the average effect size of summer education programmes (SMD = 0.09, 95% confidence interval = −0.11, 0.28) and all summer programmes (SMD = 0.09, 95% confidence interval = −0.03, 0.20) although both estimates are now marginally closer to being significant with narrower confidence intervals. Removing Modestino and Paulsen (2019a) which uses an intermediate (and potentially biased) outcome measure has a negligible impact on the findings.

Thirteen studies across 11 interventions report impacts of participation in a summer programme on the likelihood of entering higher education, or report impacts of allocation to a summer programme which could be transformed into impacts of participation. In addition to the nine studies previously detailed, the others are: ▪

University of North Carolina summer bridge programme:

Figure 18 displays the forest plot from the meta-analysis of the impact of participation in a summer programme on the likelihood of entering higher education, split by summer education and employment programmes.

OPEN IN VIEWER

Figure 18

Impact of summer programme participation on likelihood of attending higher education. Source: IES (2024).

Summer programmes have a positive and significant average effect size (log odds ratio = 0.33, 95% confidence interval = 0.03, 0.62), suggesting that participation in a summer programme has a positive impact on the likelihood of attending higher education. A log odds ratio of 0.33 translates to an odds ratio of 1.39 (95% confidence interval = 1.04, 1.90) meaning that individuals who participate in a summer programme are 1.39 times more likely to be in higher education than those that do not, post-programme. Moreover, using as the assumed comparator risk the average in the proportions of control individuals entering higher education across all the included studies where this is reported, and excluding for Modestino and Paulsen (2019a) which measures an intermediate outcome, the number needed to treat through participation in a summer programme to see this outcome among one more individual is 14. The average effect size for summer education programmes is positive although insignificant (log odds ratio = 0.35, 95% confidence interval = −0.04, 0.74), as is the case for summer employment programmes (log odds ratio = 0.27, 95% confidence interval = −0.16, 0.69). The test of group differences indicates that there is no significant difference on average effect size between summer education and employment programmes.

Within each of the sub-groups, the p-values from the homogeneity tests (p = 0.00 for both summer programme types) indicate that there is evidence of statistically significant between-study heterogeneity. As there are 11 interventions covered in this analysis, meta-regression as well as sub-group analyses are used to examine heterogeneity. There are no statistically significant differences on average effect size between any sub-groups of studies (by region of the intervention, intervention type, whether the intervention is an ‘in whole’ or ‘in part’ summer programme, the forms of disadvantage targeted, or study quality), including between the one intervention that targets ethnic minority individuals, studied by Cohodes et al. (2022) and Robles (2018), and those that do not.

Removing the result from Gehring et al. (2018), an outlier, reduces the average effect size of summer education programmes (log odds ratio = 0.17, 95% confidence intervals = −0.03, 0.38) and all summer programmes (log odds ratio = 0.21, 95% confidence intervals = 0.02, 0.39) although both figures are now more precisely estimated with narrower confidence intervals. Removing Modestino and Paulsen (2019a) which uses an intermediate (and potentially biased) outcome measure has an impact on the findings – participation in any summer programme no longer has a significant impact on the likelihood of progression to higher education (log odds ratio = 0.29, 95% confidence interval = −0.07, 0.11). Removing results that are produced by transforming reported impacts of allocation to impacts of participation has a negligible impact on the findings.

Table 10 displays the results from running meta-regression for this outcome.

Each of the coefficients are insignificant, suggesting that once controlling for other factors, there are no statistically significant differences in effect size across the sub-groups of interest. However, the direction and magnitude of some of the coefficients suggest some large differences across sub-groups; this may suggest insignificance is a function of the analysis being underpowered due to a small sample of studies as opposed to there being no true differences in effect size for participation in a summer programme with these different characteristics.

Completion of higher education

Summary: ▪

Summer education programmes appear to have a positive impact on this outcome.

Four studies across three interventions report impacts of allocation to a summer programme on the likelihood of completing higher education: ▪

New York City Summer Youth Employment Program:

Valentine et al. (2017) finds no significant impact of allocation to the summer employment programme on the likelihood of completing higher education.

The outcome measure from Theodos et al. (2017) is slightly different from the others, given the period these data cover. It might be expected that the impact of the programme on the completion of higher education would wear off over time, meaning that using this impact estimate would marginally overestimate the impact of allocation to summer programmes on completion of higher education. However, the effect size constructed using outcomes after 2 years from Theodos et al. (2017) that we include in the meta-analysis is greater than the one constructed after 1 year, which should alleviate these concerns. Theodos et al. (2017) finds no significant impact of allocation to the summer employment programme on the likelihood of completing higher education.

Cohodes et al. (2022) and Robles (2018) largely find no significant impact of allocation to the summer education programme on the likelihood of completing higher education, aside for one of the three treatment arms from Cohodes et al. (2022). When pooled together, the effect size produced is significant.

Eight studies across seven interventions report impact of participation in a summer programme on the likelihood of completing higher education, or report impacts of allocation to a summer programme which could be transformed into impacts of participation. In addition to those previously outlined, the others are: ▪

University of North Carolina summer bridge:

The outcome data in Consentino et al. (2015) is available up to a point, and there are three cohorts passing through the programme, therefore with different length evaluation periods.

Figure 19 displays the forest plot from the meta-analysis of the impact of participation in a summer programme on the likelihood of completing higher education, split by summer education and summer employment programmes.

OPEN IN VIEWER

Figure 19

Impact of summer programme participation on likelihood of completing higher education. Source: IES (2024).

The average effect size for summer education programmes is positive and significant (log odds ratio = 0.38, 95% confidence interval = 0.15, 0.62), suggesting that participation in a summer education programme has a positive impact on the likelihood of completing higher education. A log odds ratio of 0.38 translates to an odds ratio of 1.46 (95% confidence interval = 1.16, 1.86), meaning that individuals who participate in a summer education programme are 1.46 times more likely to complete higher education than those who do not, post-programme. Additionally, using as the assumed comparator risk the average in the proportions of control individuals completing higher education across all the included studies evaluating summer education programmes for which it is reported, the number needing to be treated through participation in a summer education programme to see this outcome for one individual as a result of participation is 13. For summer employment programmes, the average effect size is positive but not significant (log odds ratio = 0.07, 95% confidence interval = −0.19, 0.33), suggesting that participation in a summer employment programme does not have a significant impact on the likelihood of completing higher education. The test of group differences does not indicate that there is a statistically significant difference on average effect size between summer education and summer employment programmes (at least, at the conventional 95% confidence level). Summer programmes overall have a positive and significant average effect size (log odds ratio = 0.28, 95% confidence interval = 0.07, 0.48); however, the analysis split by programme types suggests that any significant impact of participation in any summer programme on the likelihood of completing higher education is a result only of summer education programmes.

For both summer programme types, summer education programmes and summer employment programmes, the p-value from the homogeneity test (p = 0.06, p = 0.07 and p = 0.00 respectively) indicate that there is statistically significant evidence of between-study heterogeneity, reducing the applicability of the average effect size to all summer programmes of that type. As there are fewer than 10 interventions covered in this analysis, we use sub-group analyses to examine heterogeneity. Instances of statistically significant differences on average effect size between sub-groups are: ▪

between programmes that targeted socioeconomically disadvantaged areas (log odds ratio = 0.07, 95% confidence interval = −0.19, 0.33) and those that did not (log odds ratio = 0.38, 95% confidence interval = 0.15, 0.62);

There is no statistically significant difference on average effect size between the interventions studied by Robles (2018), Cohodes et al. (2022), Ghazzawi et al. (2022) and Cosentino et al. (2015) that target ethnic minority individuals and those that do not.

Removing the result from Valentine et al. (2017), as it is the highest weighted effect size, increases the average effect size of summer programmes (log odds ratio = 0.32, 95% confidence interval = 0.16, 0.47) which is estimated more precisely with tighter confidence intervals. Also removing the result from Cohodes et al. (2022), as it is the other study, along with Valentine et al. (2017), that reports impacts of allocation which require transforming to impacts of participation, has a negligible additional effect on the findings.

Higher education outcomes relating to STEM fields

Summary: ▪

Summer education programmes appear to have a positive impact on this outcome, although studies evaluate a wide range of outcomes and studies covering only three interventions evaluated this outcome.

Four studies across three interventions report impacts of allocation to or participation in a summer programme on higher education outcomes relating specifically to STEM fields: ▪

STEM summer programmes:

Robles (2018) finds that allocation to or participation in the summer education programme has a significant positive impact on the likelihood of graduating with a STEM degree, with an effect size (log odds ratio) of 0.52 (95% confidence interval = 0.24, 0.80) after 4 years. This translates to an odds ratio of 1.68 meaning individuals allocated to or participating in the summer education programme are 1.68 times more likely to graduate with a STEM degree than those who do not, after 4 years. The effect size decreases to 0.45 (95% confidence interval = 0.12, 0.77) after 5 years, translating to an odds ratio of 1.57, that is, individuals allocated to or participating in the summer education programme are 1.57 times more likely to graduate with a STEM degree than those who do not, after 5 years. After 6 years the impact is no longer significant.

Cohodes et al. (2022) finds that allocation to the summer education programme had a significant positive impact on the likelihood of graduating with a STEM degree, with an effect size (log odds ratio) of 0.33 (95% confidence interval = 0.11, 0.55) after 4 years. This translates to an odds ratio of 1.39 so individuals allocated to the summer education programme are 1.39 times more likely to graduate with a STEM degree than individuals who are not, after 4 years. After 5 years, the effect size increases to 0.52 (95% confidence interval = 0.15, 0.90), translating to an odds ratio of 1.68 so individuals allocated to the summer education programme are 1.68 times more likely to graduate with a STEM degree than individuals who are not, after 5 years. Cohodes et al. (2022) also finds that participation in a summer education programme has a significant positive impact on the likelihood of graduating with a STEM degree after 4 years, with an effect size (log odds ratio) of 0.44 (95% confidence interval = 0.15, 0.73). This translates to an odds ratio of 1.55, meaning individuals participating in the summer education programme are 1.55 times more likely to graduate with a STEM degree than individuals who do not post-programme.

Ghazzawi et al. (2022) finds that participation in the summer education programme has a significant positive impact on the likelihood of graduating with a STEM degree from the initial field of study, estimating an effect size (log odds ratio) of 0.94 (95% confidence interval = 0.21, 1.67) and an odds ratio of 2.55 so individuals participating in the summer education programme are 2.55 times more likely to graduate with a STEM degree from the initial field of study than individuals who do not.

Cosentino et al. (2015) finds that participation in the summer education programme has a significant positive impact on the likelihood of applying to medical or dental school (effect size = 0.33, 95% confidence interval = 0.18, 0.48), enroling in medical or dental school (effect size = 0.44, 95% confidence interval = 0.28, 0.61) and achieving a bachelor's degree in a STEM field (effect size = 0.24, 95% confidence interval = 0.06, 0.41). These translate to odds ratios of 1.39 (95% confidence interval = 1.20, 1.62), 1.56 (95% confidence interval = 1.32, 1.83) and 1.27 (95% confidence interval = 1.06, 1.52) respectively. However, participation in the summer education programme has no impact on the likelihood of achieving a bachelor's degree in a health-related or medical preparatory programme.

It is worth noting that all three of the interventions covered by these studies target individuals from ethnic minorities.

Entry to employment

Summary: ▪

Summer employment programmes appear to have no impact on this outcome, although there is quite wide variation in findings across studies.

Four studies across three interventions report impacts of allocation to a summer programme on the likelihood of entering employment. These are: ▪

New York City Summer Youth Employment Program:

The measure used by Modestino and Paulsen (2019a) is somewhat different to those evaluated by other studies – intention to enter employment as opposed to realised entry to employment, that is, an intermediate employment outcome measure – which should be borne in mind. The survey that this outcome is sourced from also has highly differential response rates between the treatment and control groups, introducing a potential source of bias into the estimate produced.

Gelber et al. (2016) finds a positive significant impact of allocation to the summer employment programme on the probability of an individual being employed in the first year after the summer employment programme, equating to an effect size (log odds ratio) of 0.05 (95% confidence interval = 0.04, 0.07) and an odds ratio of 1.05 (95% confidence interval = 1.04, 1.07). So, individuals allocated to the summer employment programme are 1.05 times more likely to be employed in the first year after the summer employment programme than those who are not. Gelber et al. (2016), however, does not find any significant impacts in the second, third or fourth years after the summer employment programme.

Valentine et al. (2017) finds that allocation to the summer employment programme has a positive impact on the likelihood of entering employment in the short term, with a significant average effect size (log odds ratio) 1 year after the programme equal to 0.04 (95% confidence interval = 0.02, 0.05), which translates to an odds ratio of 1.04 (95% confidence interval = 1.02, 1.05). This means that individuals who are allocated to the summer employment programme are 1.04 times more likely to enter employment than those who are not, 1 year post-programme. However, there is no significant impact across the other periods across which the outcome is measured.

Modestino and Paulsen (2019a) finds that allocation to the summer employment programme has a significant negative impact on short-term work plans, equating to an effect size (log odds ratio) of −0.30 (95% confidence interval = −0.51, −0.08) which translates to an odds ratio of 0.74 (95% confidence interval = 0.60, 0.92). So, individuals allocated to the summer employment programme are 0.74 times as likely to report that they planned to work in the autumn at the end of the programme summer than those who are not.

Davis and Heller (2020) does not find any significant impact of allocation to the summer employment programme on the probability of an individual being formally employed in the first year and a half after the summer employment programme.

Five studies across four interventions report impacts of participation in a summer programme on the likelihood of entering employment, or report impacts of allocation to a summer programme which could be transformed into impacts of participation. In addition to the two previously outlined, these are: ▪

Urban Alliance:

Note that Theodos et al. (2017) uses a self-reported measure of employment and is based on a single point in time, whereas the other studies apart from Modestino and Paulsen (2019a) use administrative datasets to construct the outcome and consider whether the individual was employed across a period of time. The extent of attrition (overall and differential rates) for the survey used to source the outcome measure for Theodos et al. (2017) is highly concerning, introducing a potential source of bias into the estimates produced.

Theodos et al. (2017) finds no significant impact of participation in the summer employment programme on whether an individual reported as being employed 1 or 2 years after the summer employment programme.

Figure 20 displays the forest plot from the meta-analysis of the impact of participation in a summer programme on the likelihood of entering employment in the shorter term, that is, 1 year post-programme. As all the interventions included in this analysis are summer employment programmes, no split is included by programme type.

OPEN IN VIEWER

Figure 20

Short-term impact of summer employment programme participation on likelihood of entering employment. Source: IES (2024).

Summer employment programmes have a negative although insignificant average effect size (log odds ratio = −0.19, 95% confidence interval = −0.45, 0.08), suggesting that participation in a summer employment programme does not have a significant impact on the likelihood of entering employment in the short term. The impact estimate used to construct the effect size for Theodos et al. (2017) is itself insignificant – the effect size being significant is a feature of the construction process.

The two studies evaluating the New York City Summer Youth Employment Program, Gelber et al. (2016) and Valentine et al. (2017), both find positive and significant impacts on entry to employment. However, both also find that this is entirely a result of the design of the summer employment programme; the New York City Summer Youth Employment Program allows individuals to reapply in subsequent years, and individuals in the treatment group are more likely to reapply than individuals in the control group, and their summer employment programme is the employment picked up in the analysis. Both studies find that allocation to or participation in the summer programme has a negative impact on the likelihood of being employed outside of the summer programme. Davis and Heller (2020) also finds that participation in the summer employment programme has a negative (although in this case insignificant) effect on the likelihood of being employed by a non-provider employer post-programme. Using the impact estimates for employment outside of the summer employment programme for Gelber et al. (2016), Valentine et al. (2017) and Davis and Heller (2020) decreases the average effect size which is now significant (log odds ratio = −0.23, 95% confidence interval = −0.42, −0.03), suggesting that participation in a summer employment programme may decrease the likelihood of being employed outside of summer employment programme in the short term. Note that there is potential for a mismatch in the outcome measures between these studies where we are able to strip out future employment relating to the summer employment programme and the results from Modestino and Paulsen (2019a) and Theodos et al. (2017) where this is not possible.

The p-value from the homogeneity test (p = 0.00) indicates that there is evidence of between-study heterogeneity, reducing the applicability of the average effect size to all summer employment programmes. There were no statistically significant differences in impact across the sub-groups of interest. None of the interventions included in this analysis targeted individuals from ethnic minorities.

Removing the result for the New York City Summer Youth Employment Program from Gelber et al. (2016) and Valentine et al. (2017), as this is the highest weighted intervention, decreases the average effect size to −0.29 which becomes marginally significant (95% confidence interval = −0.57, −0.01). Additionally, removing the results from Valentine et al. (2017) and Modestino and Paulsen (2019a) which report impacts of allocation to a summer programme that are then transformed to impacts of participation has a negligible effect on the findings.

Figure 21 displays the forest plot from the meta-analysis of the impact of participation in a summer programme on the likelihood of entering employment, pooling the results over all time points together for Gelber et al. (2016), Valentine et al. (2017) and Theodos et al. (2017). Again, as all these interventions are summer employment programmes there is no split by programme type.

OPEN IN VIEWER

Figure 21

Overall impact of summer employment programme participation on likelihood of entering employment. Source: IES (2024).

The story is largely the same as when looking at impacts over the shorter term – summer employment programmes have a negative although insignificant average effect size (log odds ratio = −0.15, 95% confidence interval = −0.35, 0.05), suggesting that participation in a summer employment programme does not have a significant impact on the overall likelihood of entering employment. As before, using instead the impact estimates for employment outside of the summer employment programme for Gelber et al. (2016), Valentine et al. (2017) and Davis and Heller (2020) decreases the average effect size although in this instance it remains marginally insignificant (log odds ratio = −0.18, 95% confidence interval = −0.35, 0.00).

As before, the p-value from the homogeneity test (p = 0.00) indicates that there is evidence of between-study heterogeneity, reducing the applicability of the average effect size to all summer employment programmes. There were no statistically significant differences in impact across the sub-groups of interest. None of the interventions included in this analysis targeted individuals from ethnic minorities.

Again, removing the result from the New York City Summer Youth Employment Program studied by Gelber et al. (2016) and Valentine et al. (2017) decreases the average effect size to −0.23 and turns it significant (95% confidence interval = −0.46, −0.01). Additionally, removing the results from Valentine et al. (2017) and Modestino and Paulsen (2019a) has a negligible impact on the findings.

Overall, it appears then that summer employment programmes have a limited impact on entry to employment, and if anything, have a significant negative impact on entry to employment outside of extended employment relating to the summer employment programme itself. This is also interesting to note in combination with the positive although insignificant effect on progression to higher education found by Theodos et al. (2017) and the positive impact on intentions to progress to higher education found by Modestino and Paulsen (2019a). Indeed, Theodos et al. (2017) suggests that given the high rate of successive college attendance among programme participants, positive labour market outcomes are likely to develop over a longer timespan than examined in their evaluation. Modestino and Paulsen (2019a and, 2019b) also puts forward as explanations for these findings that: early work experience can raise career and academic aspirations through greater exposure to employment, which provides experiences (e.g., new occupations, different adult mentors, wider networks) that can help young people shape their goals; and that by enabling participants to shift their work experiences to a part of the year when they are not also attending school, summer employment programmes might enable young people to increase the time and attention that they can devote to academics during the school year and reduce their need to work in the post-programme period.

Earnings

▪

Studies evaluating summer employment programmes find a mix of no impact and negative impacts on this outcome.

Four studies across three interventions report impacts of allocation to or participation in a summer programme on earnings:

▪

Urban Alliance:

Theodos et al. (2017) finds no significant impact of allocation to the summer employment programme on average wages or money accumulated 1 or 2 years after the programme.

Davis and Heller (2020) also finds no significant impact of participation in the summer employment programme on earnings in the first year and a half after the programme.

Valentine et al. (2017) finds a statistically significant negative impact of allocation to the summer employment programme on earnings in the first and second years after the programme, equal to a $40 decrease in total earnings (or an approximately 3% decrease 1 year after the programme) and equating to an effect size (SMD) of −0.01 (95% confidence interval = −0.02, −0.00) in both years, although there is no significant impact in the third, fourth or fifth years post-programme. Furthermore, in combination with the impacts on entry to employment that Valentine et al. (2017) estimates, this suggests that allocation to the summer employment programme sees individuals earn less conditional on being in employment.

Gelber et al. (2016) finds a statistically significant negative impact of allocation to the summer employment programme on earnings in the first, second and third years post-programme, equal to decreases of $73.11, $68.66 and $81.04 respectively (or an approximately 2% decrease in each year), and equating to effect sizes (SMD) of −0.01 (95% confidence interval = −0.02, 0.00), −0.01 (95% confidence interval = −0.01, 0.00) and −0.01 (95% confidence interval = −0.01, 0.00) respectively, although there is no significant impact in the fourth year post-programme. Furthermore, in combination with the impacts on entry to employment that Gelber et al. (2016) estimates, this suggests that allocation in the summer employment programme sees individuals earn less conditional on being in employment.

Job readiness

Summary: ▪

Studies evaluating summer employment programmes find a mix of no impact and positive impacts on this outcome.

Three studies across three interventions report impacts of allocation to or participation in a summer programme on job readiness measures: ▪

Boston Summer Youth Employment Program:

Note that each of these studies source their outcome measures from surveys with highly differential response rates between the treatment and control groups, introducing a potential source of bias into the estimates produced.

Modestino and Paulsen (2019a) finds that allocation to the summer employment programme has a significant positive impact on five of the nine different job readiness measures examined, and no significant impact on the other four. When pooled together, these impacts equate to an overall effect size (log odds ratio) of 0.41 (95% confidence interval = 0.12, 0.71) and an odds ratio of 1.51. So, individuals allocated to the summer employment programme are 1.51 times more likely to report performing a range of job readiness behaviours than those not doing so at the end of the summer employment programme. This is an interesting finding especially when considering the same study, as previously detailed, found that allocation to the summer programme had negative impacts on intentions to enter employment. Within the job readiness measures that Modestino and Paulsen (2019a) evaluates, the least positive impact estimates are for those behaviours most directly measuring job search activity such as completing at least one online job application, asking an adult to serve as a reference, and searching for jobs online – allocation to the summer employment programme has no significant impact on any of these.

It is also worth noting the generally positive impact Modestino and Paulsen (2019a) finds on job readiness in combination with the significant negative impact they find on short-term work intentions. They suggest that whilst young people may gain job readiness skills through participation in the summer employment programme, because they are in school during the following year these short-term outcomes may not translate into longer-term improvements related to employment if they choose not to work while in school, especially during the short evaluation window considered.

Herrera et al. (2013) finds that allocation to the summer education programme has no significant impact on the proportion of individuals who had visited a business to see what it would be like to work there in the first year after the programme. Two and 4 years after the programme, however, they do find significant positive impacts, equating to effect sizes of 0.32 (95% confidence interval = 0.07, 0.59) and 0.61 (95% confidence interval = 0.34, 0.87) respectively, and odds ratios of 1.39 and 1.83 respectively. So, individuals allocated to the summer education programme are 1.39 and 1.83 times more likely have visited a business to see what it would be like to work there 2 and 4 years post-programme respectively. The intervention Herrera et al. (2013) evaluates lasts across 4 years which likely explains some of the increase in impact over time.

Theodos et al. (2017) finds positive impacts on the soft skills index from allocation to the summer programme after 1 year (SMD = 0.22, 95% confidence interval = 0.04, 0.41) although there is no significant impact after 2 years. Impact was detected from participation in the summer programme after 1 year (SMD = 0.44, 95% confidence interval = 0.26, 0.63) and after 2 years (SMD = 0.31, 95% confidence interval = 0.12, 0.49). Theodos et al. (2017) finds a significant impact of participation on hard skills after 1 year (SMD = 0.29, 95% confidence interval = 0.11, 0.48), apart from this they find no significant impact from allocation after 1 or 2 years, or from participation after 2 years. The specific outcome measure used by Theodos et al. (2017) means that the effect sizes provide the easiest interpretation of the impacts, therefore we do not translate them into another format.

Whether had a criminal justice outcome (including arrests, arraignments, convictions or incarcerations)

Summary: ▪

Findings for the impact of summer employment programmes on this outcome (including when broken down by the type of violent/offending behaviour) are mixed – studies largely either find no impact or substantial negative (i.e., beneficial) impacts.

Three studies across two interventions report impacts of allocation to or participation in a summer programme on whether an individual had an arrest, arraignment, conviction or incarceration for any type of crime or offence. These are: ▪

New York City Summer Youth Employment Program:

Kessler et al. (2022) finds that participation in the summer employment programme has a significant negative (i.e., a beneficial) impact on the probability that an individual has an arrest during the programme months, which translates to an effect size (log odds ratio) of −0.19 (95% confidence interval = −0.31, −0.07) and an odds ratio of 0.83 (95% confidence interval = 0.74, 0.93). So, participants in the summer employment programme were 0.83 times as likely to have an arrest as those that did not participate during the programme months, although there are no significant impacts post-programme. During the programme months, Kessler et al. (2022) finds a significant negative (i.e., a beneficial) impact on the probability of having a conviction during the programme months, which translates to an effect size (log odds ratio) of −0.38 (95% confidence interval = −0.60, −0.16) and an odds ratio of 0.68 (95% confidence interval = 0.55, 0.85). So, participants in the summer employment programme were 0.68 times as likely to have a conviction as those that did not participate during the programme months. Again, Kessler et al. (2022) did not find any significant impacts on this outcome post-programme.

Gelber et al. (2016) finds that participation in the summer employment programme has a significant negative (i.e., a beneficial) impact on the probability that an individual has an incarceration across the 4 years post-programme, which translates to an effect size (log odds ratio) of −0.11 (95% confidence interval = −0.18, −0.03), and an odds ratio of 0.90 (95% confidence interval = 0.83, 0.97). This means that participants in the summer employment programme are 0.90 times as likely to be incarcerated as those that did not participate across the 4 years post-programme.

Modestino (2019b) finds that allocation to the summer employment programme has no significant impact on the probability that an individual has an arraignment.

At the request of the review advisory group, given the particular interest of the organisations involved in the review (including YEF) in this domain, the requirement of studies covering four separate interventions to evaluate an outcome to perform meta-analysis was dropped to two for this and the subsequent violence and offending outcomes examined. The rule was introduced to manage the size of the review (i.e., not for methodological purposes, as it is possible to perform meta-analysis with just two effect sizes) given the large number of individual outcomes examined by a small number of studies across the range of domains of interest.

The effect sizes used were based on the impact estimates from the longest follow up point, that is, 5 years (60 months) post-programme for Kessler et al. (2022), as only one study (Kessler et al., 2022) produces usable estimates of impact during the programme months. Figure 22 displays the forest plot from the meta-analysis of the impact of participation in a summer programme on the likelihood that an individual had an arrest, arraignment, conviction or incarceration for any crime or offence across approximately 17–60 months post-programme. As all the interventions included in this analysis are summer employment programmes, no split is included by programme type.

OPEN IN VIEWER

Figure 22

Impact of summer employment programme participation on likelihood of having an arrest/arraignment/conviction/incarceration post-programme. Source: IES (2024).

The average effect size for summer employment programmes is negative although insignificant (log odds ratio = −0.05, 95% confidence interval = −0.15, 0.05), suggesting that they do not appear to have an impact on the likelihood of an individual having an arrest/arraignment/conviction/incarceration post-programme.

The p-value from the homogeneity test (p = 0.76) indicates that there is no evidence of between-study heterogeneity. None of the interventions included in this analysis targeted individuals from ethnic minorities.

Additionally, three studies across three interventions report impacts of allocation to or participation in a summer programme on whether an individual engaged specifically in drug-related, violent, or property-related criminal or offending activity. These are: ▪

Youth Violence Prevention Funder Learning Collaborative summer employment programme:

Note that the outcomes evaluated by Sum (2015) are self-reported measures based on surveys, and limited information is provided about the extent of attrition (overall and differential rates), potentially introducing a source of bias into the estimates produced.

Sum (2015) finds that participation in the summer employment programme had no significant impact on: the proportion of individuals that went from reporting that they bought or sold illegal drugs, to not doing so; the proportion of individuals who went from reporting that they had attacked or threatened someone with a weapon other than a gun, to not doing so; the proportion of individuals initially reporting that they had damaged or destroyed someone's property, to not doing so; and the proportion of individuals that went from reporting that they did not use marijuana, to doing so. Sum (2015) does, however, find a significant positive impact on the proportion of individuals that went from reporting that they use marijuana to not doing so, equal to an 8-percentage point impact on the likelihood of seeing this improvement (insufficient information is available to construct an effect size). It should be noted though that Sum (2015) has a low-quality study design and limited information on the construction of outcomes or the estimation approach used, therefore one should be cautious about the findings for this or any of the other violence and offending outcomes that Sum (2015) evaluates.

Kessler et al. (2022) finds that participation in the summer employment programme has a significant negative (i.e., beneficial) impact on the probability that an individual has a violent crime or offence conviction during the programme months, which translates to an effect size (log odds ratio) of −1.02 (95% confidence interval = −1.54, −0.51) and an odds ratio of 0.36 (95% confidence interval = 0.21, 0.61). This means that individuals that participate in the summer employment programme are 0.36 times as likely to have a violent crime or offence conviction as those that do not during the programme months. However, after 1, 3 and 5 years post-programme there is no significant impact of participation in the summer employment programme on this outcome. Additionally, Kessler et al. (2022) estimates no significant effect of summer employment programme participation on the probability that an individual has a violent crime or offence arrest.

Kessler et al. (2022) finds that participation in the summer employment programme has a significant negative (i.e., a beneficial) impact on the probability that an individual has a property crime or offence arrest during the programme months and after 1 year post-programme, which translate to effect sizes (log odds ratios) of −0.84 (95% confidence interval = −1.23, −0.46) and −0.28 (95% confidence interval = −0.42, −0.14) respectively. These translate to odds ratios of 0.43 (95% confidence interval = 0.29, 0.63) and 0.76 (95% confidence interval = 0.66, 0.87) respectively. This means that individuals who participate in the summer employment programme are 0.43 times and 0.76 times as likely to have a property crime or offence arrest as those that do not during the programme months and 1 year post-programme respectively. However, after 3 and 5 years post-programme there is no significant impact of participation in the summer employment programme on the probability that an individual has a property crime or offence arrest, and across all periods considered there is no significant impact on the probability that an individual has a property crime or offence conviction.

Kessler et al. (2022) finds that participation in the summer employment programme has no significant impact on the probability that an individual has a drug crime/offence arrest or conviction during the programme months and 1 year post-programme. However, they find that participation in the summer employment programme has a significant negative (i.e., a beneficial) impact on the probability that an individual has a drug crime or offence arrest after 3 and 5 years post-programme, and on the probability that an individual has a drug crime or offence conviction 5 years post-programme. The impacts on the probability that an individual has a drug crime or offence arrest 3 and 5 years post-programme translate to effect sizes (log odds ratios) of −0.10 (95% confidence interval = −0.15, −0.04) and −0.06 (95% confidence interval = −0.11, −0.02) respectively. These translate to odds ratios of 0.91 (95% confidence interval = 0.86, 0.96) and 0.94 (95% confidence interval = 0.90, 0.98) respectively. This means that individuals who participate in the summer employment programme are 0.91 and 0.94 times as likely to have a drug crime or offence arrest as those that do not after 3 and 5 years post-programme respectively. The impact on the probability that an individual has a drug crime or offence conviction 5 years post-programme translates to an effect size (log odds ratio) of −0.16 (95% confidence interval = −0.27, −0.06). This translates to an odds ratio of 0.85 (95% confidence interval = 0.77, 0.94), which means that individuals who participate in the summer employment programme are 0.85 times as likely to have a drug crime or offence conviction as those that do not 5 years post-programme.

Modestino (2019b) finds that allocation to the summer employment programme has no significant impact on the probability that an individual has an arraignment for violent, property or drug crimes/offences.

Supporting Information: Appendix 3 contains the results from meta-analyses of the impact of participation in the summer employment programmes studied by Kessler et al. (2022) and Modestino (2019b) on the likelihood that an individual had an arrest, arraignment or conviction for violent, drug or property crimes or offences across approximately 17–60 months post-programme.

Again, as only Kessler et al. (2022) produces usable estimates of impact during the programme months, we used effect sizes based on the impact estimates from the longest follow-up point, that is, 5 years post-programme for Kessler et al. (2022). The average effect sizes estimated for each of the three analyses are insignificant, suggesting that participation in a summer employment programme has no impact on the likelihood that an individual had an arrest, arraignment or conviction for violent, drug or property crimes or offences post-programme. The p-values from the homogeneity tests indicate statistically significant heterogeneity for the analysis of drug crimes/offences (p = 0.09), although not for violent or property crimes/offences. None of the interventions included in these analyses targeted individuals from ethnic minorities.

Number of criminal justice outcomes (including arrests, arraignments, convictions or incarcerations)

Summary: ▪

Findings for the impact of summer employment programmes on this outcome (including when broken down by the type of violent/offending behaviour) are mixed – studies largely either find no impact or substantial negative (i.e., beneficial) impacts.

Five studies across three interventions report impacts of allocation to or participation in a summer programme on the number of arrests, arraignments, convictions or incarcerations an individual has for any type of crime or offence. These are: ▪

New York City Summer Youth Employment Program:

Gelber et al. (2016) finds that participation in the summer employment programme has a significant negative (i.e., a beneficial) impact on the number of incarcerations that an individual has, which equals a 11% reduction in the number compared with the control group mean, and translates to an effect size (SMD) of −0.01 (95% confidence interval = −0.02, −0.00). Note that the magnitude of reduction reported here differs from the headline figure reported by Gelber et al. (2016) themselves, and this is also the case for several other instances of impact discussed in this section. This is due to differences in the results produced by different specifications of the models used by the study authors, who tend to take as their preferred results to report on those from the approaches that do not control for differences in baseline characteristics between the treatment and comparison groups. Conversely, where there is the option we choose to use those that do control for differences in baseline characteristics.

Kessler et al. (2022) finds that participation in the summer employment programme has no significant impact on the number of times the young person is arrested. Kessler et al. (2022) finds a significant, negative (i.e., a beneficial) impact on the number of convictions an individual has during the programme summer equal to a 28% reduction in the number compared with the control group mean and equating to an effect size (SMD) of −0.01 (95% confidence interval = −0.02, −0.00). However, in the years after the programme summer there is no significant impact.

Modestino (2019b) finds that allocation to the summer employment programme does not have a significant impact on the number of arraignments recorded for young people.

Davis and Heller (2020) finds that allocation to the summer employment programme has no significant impact on the number of arrests an individual has.

Heller (2022) finds that allocation to the summer employment programme has no significant impact on the number of arrests an individual has.

Firstly, we perform meta-analysis using effect sizes from the period covering the programme months, that is, for Kessler et al. (2022) from during the programme months and for Davis and Heller (2020) and Heller (2022) from the first year post-randomisation. Note that the results from Davis and Heller (2020) and Heller (2022) cover both during and post-programme periods.

Figure 23 displays the forest plot from the meta-analysis of the impact of participation in a summer programme on the number of arrests/convictions across a period that approximately corresponds to the period of the summer programme. As all the interventions included in this analysis are summer employment programmes, no split is included by programme type.

OPEN IN VIEWER

Figure 23

Impact of summer employment programme participation on number of arrests/convictions approx. during programme months. Source: IES (2024).

Overall, summer employment programmes do not appear to have a significant impact on the number of arrests/arraignments/convictions/incarcerations an individual has (average effect size (SMD) = −0.01, 95% confidence interval = −0.03, 0.00).

The p-value from the homogeneity test (p = 0.31) indicates that there is no evidence of between-study heterogeneity. None of the interventions included in this analysis targeted individuals from ethnic minorities.

Secondly, we perform meta-analysis using effect sizes based on the impact estimates from the longest follow up point, that is, 5 years (60 months) post-programme for Kessler et al. (2022) and 3 years post-randomisation from Davis and Heller (2020) and Heller (2022).

Figure 24 displays the forest plot from the meta-analysis of the impact of participation in a summer programme on the number of arrests/arraignment/convictions/incarcerations covering approximately 17–60 months post-programme. As all the interventions included in this analysis are summer employment programmes, no split is included by programme type.

OPEN IN VIEWER

Figure 24

Impact of summer employment programme participation on number of arrests/arraignments/convictions/incarcerations post-programme. Source: IES (2024).

Overall, summer employment programmes do not appear to have a significant impact on the number of arrests/arraignments/convictions/incarcerations an individual has (average effect size (SMD) = −0.02, 95% confidence interval = −0.05, 0.02).

The p-value from the homogeneity test (p = 0.37) indicates that there is no evidence of between-study heterogeneity. None of the interventions included in this analysis targeted individuals from ethnic minorities.

Additionally, four studies across two interventions report impacts of allocation to or participation in a summer programme on the number of arrests, arraignments or convictions an individual has specifically for drug-related, violent, or property-related criminal or offending activity. These are: ▪

Boston Summer Youth Employment Program:

Modestino (2019b) finds that allocation to the summer employment programme has no significant impact on the number of arraignments for drug crimes an individual has. However, they find that allocation to the summer employment programme leads to a significant reduction in the number of arraignments for violent crimes that an individual had, equal to a 52% reduction in the number compared with the control group mean and equating to an effect size (SMD) of −0.06 (95% confidence interval = −0.13, 0.01 – the raw impact estimate from Modestino (2019b) is significant at the 95% confidence level, but as a result of the assumptions used in order the construct an effect size from the reported data, the SMD is marginally insignificant at the 95% confidence level). They also find that allocation to the summer employment programme leads to a significant reduction in the number of arraignments for property crimes an individual has, equal to a 46% reduction in the number compared with the control group mean and equating to an effect size (SMD) of −0.06 (95% confidence interval = −0.13, 0.00 – the raw impact estimate from Modestino (2019b) is significant at the 95% confidence level, but as a result of the assumptions used to construct an effect size from these data, the SMD is marginally insignificant at the 95% confidence level).

Davis and Heller (2020) finds that allocation to the summer employment programme has no significant impact on the number of arrests for drug crimes an individual has. Davis and Heller (2020) finds that allocation to the summer employment programme has a significant negative (i.e., a beneficial) impact on the number of arrests for violent crimes an individual has in the first year post-randomisation. This impact is equal to a 41% reduction in the number compared with the control group mean and equates to an effect size (SMD) of −0.06 (95% confidence interval = −0.15, 0.04: note that the specifics of the process for constructing the effect size, which in this instance requires imputation of the standard deviation from another study, means that the constructed effect size is insignificant). However, they find no significant impacts on this in the second and third years post-randomisation. Davis and Heller (2020) finds that allocation to the summer employment programme has no significant impact on the number of arrests for property crimes an individual has in the first year post-randomisation, but a significant positive (i.e., an increase in the number) impact in the second and third years post-randomisation, equal to a 95% and 93% increase in the number compared with the control group mean respectively and translating to effect sizes (SMD) of 0.18 (95% confidence interval = 0.09, 0.28) and 0.19 (0.09, 0.28) respectively.

Heller (2014) finds that allocation to the summer employment programme has no significant impact on the number of arrests for drug crimes or property crimes an individual has. However, they find that allocation to the summer employment programme has a significant negative (i.e., a beneficial) impact on the number of arrests for violent crimes an individual has, equal to a 43% reduction in the number compared with the control group mean and equating to an effect size (SMD) of −0.07 (95% confidence interval = −0.16, 0.03: note that the specifics of the process for constructing the effect size, which in this instance requires imputation of the standard deviation from another study, means that the constructed effect size is insignificant).

Heller (2022) finds that allocation to the summer employment programme has a significant negative (i.e., a beneficial) impact on the number of arrests for drug crimes or offences that an individual has in the first year post-randomisation, equal to a 40% reduction in the number compared with the control group mean and equating to an effect size (SMD) of −0.04 (95% confidence interval = −0.09, 0.02: note that the specifics of the process for constructing the effect size, which in this instance requires imputation of the standard deviation from another study, means that the constructed effect size is insignificant). However, allocation to the summer employment programme has no significant impact on the number of arrests for drug crimes or offences an individual has in the second or third years post-randomisation, or at all on the number of violent or property crimes or offences an individual has.

Supporting Information: Appendix 3 contains the results from meta-analyses of the impact of participation in the summer employment programmes studied by Modestino (2019b) and Heller (2014), Davis and Heller (2020) and Heller (2022) on the number of arrests or arraignment for violent, drug or property crimes or offences an individual has across approximately 17–34 months post-programme.

We used effect sizes based on the impact estimates from the longest follow-up point, that is, 3 years post-randomisation (approximately 34 months post-programme) for Heller (2014), Davis and Heller (2020) and Heller (2022). The average effect sizes estimated for each of the three analyses are insignificant, suggesting that participation in a summer employment programme has no impact on the number of arrests or arraignments for violent, drug or property crimes or offences an individual has post-programme. The p-values from the homogeneity tests indicate statistically significant heterogeneity for the analysis of property crimes/offences (p = 0.09), although not for violent or drug crimes/offences. None of the interventions included in these analyses targeted individuals from ethnic minorities.

Socio-emotional skills and engagement

Summary: ▪

The two studies of summer education programmes find positive impacts on this outcome.

Three studies across three interventions report impacts of allocation to a summer programme on socio-emotional skills and engagement. These are: ▪

Boston Summer Youth Employment Program:

Note that each of these studies source their self-reported outcome measures from surveys. The extent of attrition (overall and differential rates) for the surveys used by Modestino (2019b) and Theodos et al. (2017) are highly concerning, introducing a potential source of bias into the estimates produced. Attrition is not a significant concern for the survey-based measure from Cohodes et al. (2022).

Modestino (2019b) finds that allocation to the summer employment programme has a positive significant impact on socio-emotional skills and engagement (SMD = 0.32, 95% confidence interval = 0.20, 0.45). Translating this back to the specific measures used by Modestino (2019b), this means that those allocated to the summer employment programme were (on average) 1.79 times more likely to report: feeling connected to people in their neighbourhood, knowing how to ask for help when they need it, knowing how to constructively resolve a conflict with a peer, knowing how to manage their emotions and their temper, and feeling that they have a lot to contribute to the groups they belong to; than those that were not.

Theodos et al. (2017) finds that allocation to the summer employment programme has no significant impact on socio-emotional skills and engagement.

Cohodes et al. (2022) finds that allocation to the summer education programme had a positive significant impact on socio-emotional skills and engagement (SMD = 0.23, 95% confidence interval = 0.07, 0.39). The specific outcome measure used by Cohodes et al. (2022) means that the effect size provides the easiest interpretation of the impacts, therefore we do not translate this into another format.

Two studies across two interventions report impacts of participation in a summer programme on socio-emotional skills and engagement. The estimates of impact of allocation from Cohodes et al. (2022) cannot be translated to estimates of impact of participation in the summer programme, as they report standardised index scores with insufficient information to transform them, therefore there are insufficient studies evaluating this outcome to perform meta-analysis. In addition to Theodos et al. (2017), the other is: ▪

Department for Education Summer Schools Programme:

Note that Martin et al. (2013b) sources their self-reported outcome measures from a survey. Insufficient information is provided to assess the potential for differential response rates between students in the treatment and comparison groups to introduce bias into the estimate produced.

Theodos et al. (2017) finds that participation in the summer employment programme has no significant impact on socio-emotional skills and engagement.

Martin et al. (2013b) finds that participation in the summer education programme has a positive significant impact on socio-emotional skills and engagement (SMD = 0.10, 95% confidence interval = 0.07, 0.12). An SMD of 0.10 translates to approximately 10% of students seeing a 1 level improvement (with each item scored from 1 to 5) in their response to each of the five items relating to socialisation (I made friends quickly when I started this school; I am bullied/picked on by people from my school; I feel safe in school; other people listen to what I say; I often feel left out).

Community engagement

Summary: ▪

The one study of a summer education programme evaluating this outcome finds no impact. The one study of a summer employment programme evaluating this outcome finds a positive impact.

Two studies across two interventions report impacts of allocation to a summer programme on measures of community engagement. These are: ▪

Boston Summer Youth Employment Program:

Note that each of these studies source their outcome measures from surveys with highly differential response rates between the treatment and control groups, introducing a potential source of bias into the estimates produced.

Modestino and Paulsen (2019a) finds that allocation to the summer employment programme has a significant positive impact on 5 of the 12 different sense of community measures examined, and no significant impact on the other seven. When pooled together, these impacts equate to an overall effect size (log odds ratio) of 0.26 (95% confidence interval = 0.12, 0.40) and an odds ratio of 1.30 (95% confidence interval = 1.13, 1.49). So, individuals allocated to the summer employment programme are 1.30 times more likely to respond ‘true’ to the range of statements relating to sense of community measured than those who are not, at the end of the summer employment programme.

Herrera et al. (2013) find no significant impact on the likelihood of engaging in community service and/or voluntary work as a result of allocation to the summer education programme.

Mental health

One study reports impacts of allocation to a summer programme on mental health: ▪

Boston Summer Youth Employment Program:

Note that Modestino and Paulsen (2019a) sources their outcome measures from a survey with highly differential response rates between the treatment and control groups, introducing a potential source of bias into the estimates produced.

Modestino and Paulsen (2019a) finds that allocation to the summer employment programme has a significant positive impact on all five of the depression scale measures examined. When pooled together, these impacts equate to an overall effect size (log odds ratio) of 0.43 (95% confidence interval = 0.31, 0.56) and an odds ratio of 1.54 (95% confidence interval = 1.36, 1.75), that is, individuals allocated to the summer employment programme are 1.54 times as likely to report hardly ever experiencing symptoms of depression, than those who are not. This is corroborated by the estimated significant negative impact on an individual's depression index based on these questions (insufficient information was available to construct an effect size for this measure).

Mortality rates

One study reports impacts of participation in a summer programme on mortality rates: ▪

New York City Summer Youth Employment Program:

Gelber et al. (2016) finds that participation in the summer employment programme has a significant negative (i.e., beneficial) impact on mortality rates, equal to a 18% reduction in the rate number compared with the control group mean and equating to an effect size (log odds ratio) of −0.20 (95% confidence interval = −0.32, 0.08) and an odds ratio of 0.82 (95% confidence interval = 0.73, 0.93). This means that young people who participate in the summer programme are 0.82 times as likely to die as those who do not, post-programme.

Personalisation

Summer education programmes often have a lower student-to-staff ratio than standard school classes (Day et al., 2013a; Martin et al., 2013b; Maxwell et al., 2014). This enables individualised attention and support to be delivered to young people, which in turn can support student engagement in the summer programme. This mechanism is suggested in Barnett et al. (2012) specifically in relation to the use of a single tutor participating in lessons alongside students in small-sized classes, or multiple tutors roaming between a small number of classes; and in Gorard et al. (2017), who observes that there were multiple adults in each class, and every child was seen receiving extra attention and time from the staff; although these studies do not detect impact.

Having programme staff who are trained to understand the unique circumstances and barriers faced by young people can also act as a motivator for young people and facilitate their participation in the programmes. The Scholars Academy specifically targeted low-income, first-generation students and had a dedicated programme director who was highly regarded by participants, which Henson (2018) argued supported them to do well and fostered their engagement. This mechanism is also suggested in Anthony (2019) who suggests that the Excel State University summer bridge programme recruited faculty who understood the unique circumstances and barriers of first-generation and economically disadvantaged students; although neither study detected impact.

As the primary focus of summer employment programmes is to provide work insight and experience for young people to improve their employability for future jobs, providers place a strong focus on ensuring a suitable match between participants and work placements, to maximise participation and mitigate the risk of disengagement. This personalised approach emphasises appropriate placement decisions that align with the participants' preferences and experiences. In one programme, providers conduct interviews with young people during enrolment to understand their work histories and interests and then broker suitable placements (Valentine et al., 2017). The One Summer Chicago programme finds that the majority of surveyed young people find their participation meaningful and fulfilling (One Summer Chicago, 2015). By matching participants with jobs that align to their interests and goals, the study authors suggest that the programme helps create a sense of purpose and relevance. When participants feel a strong connection between their work and aspirations, they are more likely to engage at work.

Relationship with programme staff

In summer education programmes, the presence of well-prepared staff who provide effective academic and socio-emotional support, and who are approachable and open to questions and discussion, can help create a positive learning environment which fosters participation. Younger (2017) suggests that this mechanism may support increased participation, although there is no impact evaluation associated with this study. Positive relationships with staff are also highlighted as playing an important role in supporting participation and engagement within catch-up programmes (Williamson et al., 2020). Similar views are expressed in Wathington et al. (2016) regarding the role of peer mentoring and counselling as a way of supporting integration; and Gorard et al. (2014) describes the use of a ‘student feedback corner’ where young people could openly share their views and concerns with staff at the end of each lesson; although both these studies do not detect impact.

In summer employment programmes, the positive relationships that staff – including programme staff and workplace supervisors – build with participants are also highlighted as likely to be playing a role in improving attendance and programme completion. Where participants in summer employment programmes report improved social skills, such as asking staff for help and support, this is positively correlated with improvements in attendance (Modestino & Paulsen, 2019c; Reich, 2018; Theodos et al., 2014, 2017). This suggests that where staff members foster the development of these social skills, attendance outcomes may improve. Additionally, factors such as gaining a mentor and increasing aspirations to attend the next phase of education also contribute to improved attendance (Modestino & Paulsen, 2019a). In some programmes, participants say they value the help, support, and guidance offered by supervisors during work placements (Sum, 2015). Positive relationships with supervisors are also valued by young people for finding appropriate and interesting tasks, discussing their career, and future education options, fostering a sense of reliability and accessibility (Reich, 2018).

Another factor which may play a role in sustaining engagement in some summer employment programmes is the involvement of job mentors who regularly visit participants' workplaces (Heller, 2014; Sum, 2015). The support received ranges from on-the-job assistance to coaching on future opportunities, as well as positive reinforcement to avoid negative behaviours, for example, those associated with street life. This support network is likely to foster a sense of belonging, empowerment, and personal growth, all of which contribute to increased engagement and active participation, as suggested by the evaluation of One Summer Chicago (2015).

Financial and non-financial incentives

In summer education programmes there are a range of incentives, which in UK-based programmes mainly consist of accessibility measures, such as providing free transport and removing financial barriers by eliminating the costs associated with attending. A number of authors suggest that incentives of this type may lead to increased summer programme attendance, including Ferguson (2018); Siddiqui et al. (2014); and Torgerson et al. (2014); although there is no impact evaluation associated with Ferguson (2018), and Siddiqui et al. (2014) and Torgerson et al. (2014) do not detect impact. Meanwhile, US programmes include stipends (i.e., fixed regular sums of money) and perks, such as gift cards, raffles, and field trips which support engagement and programme completion (Ghazzawi et al., 2022). This mechanism leading to increased summer programme engagement and completion is also suggested in Barnett et al. (2012) and Wathington et al. (2016); although these studies do not detect impact.

The motivations expressed by the participants may also serve as facilitators to participation. Many participants join summer employment programmes because they want something productive to do over summer (Lansing et al., 2018). The desire for personal development, understanding of the employability and technical skills required by work, and the opportunity to improve future prospects may support engagement and active participation. Additionally, where summer employment programmes provide financial benefits for participants (such as income), this is often cited as a facilitator to engagement and participation, given the young person's desire to receive the incentive as well as affecting how the opportunity is valued within the young person's household (Modestino, 2019b; Sum, 2015).

Recruitment and targeting approach

Recruitment strategies such as school open evenings, providing written invitations, alumni advocates, and offering incentives such as discount schemes may encourage young people's participation in summer education programmes (Day et al., 2013b).

Summer education programmes that specifically prioritise the identification and targeting of disadvantaged at the individual level, such as selecting those pupils eligible for Free School Meals – rather than targeting by area characteristics such as disadvantaged neighbourhoods, as well as those that make efforts to market the programmes to parents/carers in addition to young people, are more successful at engaging their target cohorts (Day et al., 2013b; Robles, 2018). This mechanism is also discussed by Henson (2018), although this study does not detect impact. Identifying and recruiting pupils with English as an Additional Language (EAL) is particularly difficult due to challenges in successfully engaging and gaining the support of their parents and carers (Martin et al., 2013a; Sharp, 2018). This mechanism is also discussed by Torgerson et al. (2014), although this study does not detect impact. Some summer education programmes also face challenges identifying and recruiting disadvantaged pupils due to difficulties in obtaining relevant pupil data from feeder schools (Martin et al., 2013b).

A common challenge to participation in many summer employment programmes is that they use combinations of first-come/first-served and merit-based recruitment (e.g., with applications, interviews, and employer selection) (Heller & Kessler, 2017). These approaches, which essentially allow eligible young people and providers to perform the allocation themselves, may lead to more disadvantaged young people not receiving places (Heller & Kessler, 2017).

Partnerships

A number of summer employment programmes establish formal partnerships with schools, for example by actively involving school staff in the identification and recommendation of suitable candidates (Theodos et al., 2014). This collaboration enhances the effectiveness of participant selection, as school staff bring valuable insights into students' skills, interests, and career aspirations. School staff distribute application forms and ensure their timely submission, fostering interest and enthusiasm among students. These collaborative efforts between programmes and schools may help secure participants' active involvement (Theodos et al., 2014).

Sponsorship and support from community action and workforce development agencies provide financial backing, resources, and expertise to engage and support young people (Sum, 2015). Multiple organisations, including community-based providers and employers alongside local government, often come together in summer employment programmes, fostering cooperation and creating a network of support and opportunity for participants (Gelber et al., 2016; Lansing et al., 2018; Modestino & Paulsen, 2019a; Schwartz et al., 2021). These partnerships help establish the summer programmes' infrastructure, enabling better coordination and collaboration with employers and community partners over time (Theodos et al., 2014; Valentine et al., 2017).

Partnerships also extend beyond the summer employment programmes. In one summer education programme, the Summer Learning Journey, the providers' collaboration with school staff strengthens communication channels and creates a stronger connection between the programme and participants' academic environments, improving referral processes and warm handovers (Williamson et al., 2020). Beyond logistical support, partnerships can also have legal and systemic implications. Participation in summer programmes may yield positive outcomes within the legal system, functioning as a mitigating factor and leading to more favourable responses from system actors (Kessler et al., 2022). These partnerships between programmes and legal entities create opportunities for better outcomes for young participants.

In a number of summer programmes, engaging parents/carers through activities like cookery classes or family learning, and celebration events, may support their increased involvement and promote a positive home–school partnership, which may in turn support young people's sustained participation and engagement (Day et al., 2013b).

Format

The format of a summer education programme may play an important facilitating role for some target groups in enabling their participation in the programme. This is discussed in particular in studies of online summer education programmes, especially for students with care responsibilities, part-time jobs, or other constraints on their availability, with their participation being facilitated through a flexible format. This mechanism leading to increased summer programme participation is suggested by Taylor (2022), although this study does not detect impact. Other barriers to young people's participation, including scheduling conflicts due to holiday plans or family commitments (Maxwell et al., 2014; Somers et al., 2015), in some instances might not be able to be overcome by the use of an online format.

Programmes that are delivered online can however pose a barrier to participation for students who lack access to reliable internet connections and IT resources, although the provision of laptops and IT support may mitigate, to some extent, this barrier, as suggested by Lynch and Kim (2017). However, this study does not detect impact across the outcomes previously examined. An additional challenge to sustained participation faced by online summer schools can be digital fatigue, particularly following the shift to online learning during the pandemic, which can make these online programmes less attractive to some students. This mechanism leading to decreased summer programme participation is suggested by Taylor (2022), although this study does not detect impact.

In face-to-face summer education programmes, social activities and opportunities to support the formation of connections may play a role in facilitating young people's engagement, especially for those who face additional barriers to participation such as disabled students and those with health conditions (Day et al., 2013b). This mechanism is suggested in Lei et al. (2020) in relation to the role of sessions about ‘play’, experiencing shared meals on campus, and social outings and opportunities for informal socialising, although there is no impact evaluation associated with this study.

Integration into the workplace

Some summer employment programmes facilitate pre-placement engagement between young people and employers. This can take the form of orientation days, job fairs, or introductions during the enrolment process. These allow participants to familiarise themselves with the work environment, establish connections with employers, and gain insights into their future roles, which may in turn foster motivation and participation (Valentine et al., 2017).

Skill acquisition

Improvements in social skills, such as feeling able to ask for help and resolve conflicts with peers, are positively correlated with improved attendance (Modestino & Paulsen, 2019c). Similarly, where young people cannot access the specific support they want from co-workers or supervisors, they may turn to each other, which may then supports their continued engagement with the programme, as suggested in Valentine et al. (2017).

Skill acquisition

The evidence highlights that skill acquisition, including academic, social, emotional, and life skills, resulting from participation in a summer programme plays an important role in supporting young people's outcomes in summer programmes.

Summer education programmes that combine a variety of social activities, such as sports, arts, and curricular activities delivered in a creative way, facilitate positive outcomes for young people given the holistic approach they take (Day et al., 2013b). By offering a blend of fun and educational experiences, they address the educational and psychosocial needs of young people. The majority of evidence for summer education programmes pertains to raising aspirations and transition support programmes, where the predominant focus is on social aspects of participation and enrichment. Martin et al. (2013b) highlights how students highly value these features, and particularly notes the significance of making new friends, participating in team-building activities, experiencing shared social activities, and interacting with peers in the structured learning environments, to their outcomes. These social interactions provide opportunities to develop social skills, build confidence, and form new friendships, while also improving motivation to engage in education thereby facilitating transitions. Day et al. (2013b) also highlights that incorporating activities that support the development of independent learning skills is an important facilitator of planned outcomes. Summer education programmes that support students to learn independently help to develop their autonomy and preparation for future academic endeavours. Martin et al. (2013a), evaluating the same intervention as these studies, estimates significant impacts of participation in the summer programme on self-reported measures of pupil confidence, school readiness, and socialisation.

Summer programmes focused on raising aspirations and transition support, in both the UK and US, often focus on the development of soft skills that can support transitions and educational progression. Alongside a focus on supporting students to develop socio-emotional skills and confidence to support personal growth, this also includes the ability to make effective university applications, study skills, and life skills, particularly in programmes focused on higher education. Cohodes et al. (2022) and Robles (2018), that highlight the importance of soft skill development in supporting transitions and progression, estimate significant impacts on the likelihood of applying to, progressing to and completing higher education. These studies evaluate a summer education programme with a focus on progression within STEM, and find larger effect sizes than Wachen et al. (2018) that evaluates a transition support programme, or Cosentino et al. (2015) that evaluates a raising aspirations programme, do on the likelihood of progression across all subject areas. This might indicate that the specialist nature of the skills this programme develops amplifies its impact on supporting transitions and progression to higher education, intermediate outcomes on the way to completion. Additionally, some programmes, particularly those supporting transitions from primary to secondary school, have dedicated activities on building relationships, understanding identity, navigating new challenges, and challenging stereotypes. This might lead to increased engagement in school, raising attainment: Pyne et al. (2020) estimates a significant positive impact of summer programme participation on English test scores, for instance.

In summer employment programmes (evidence for which is only available from the US) a focus on skills acquisition and employability attribute development, alongside the job-specific technical skills that the job placement provides, aims to instil transferable skills for the labour market, which should aid entry to employment and employment-related outcomes. In these programmes the main skills and attributes of focus are communication, problem-solving, work-readiness, social and life skills (Gelber et al., 2016; Leos-Urbel, 2014; Sum, 2015; Theodos et al., 2014; Valentine et al., 2017). Modestino and Paulsen (2019a) finds a significant impact of summer programme participation on self-reported measures of job readiness. However, there is no clear evidence of summer employment leading to positive labour market outcomes (although these may appear over the longer-term while the evaluations included in this review typically only consider relatively short term outcomes). If anything there is evidence of a negative impact on the likelihood of entering employment.

The opportunities summer employment programmes provide for young people to develop social and emotional skills (such as processing social information, managing thoughts and emotions, and setting and achieving goals), alongside participating in group discussions with co-workers and meeting new people who can support their growth (Heller, 2014), are also soft skills necessary to facilitate the achievement of wider education, socio-emotional and violence and offending outcomes. In one programme, the Youth Violence Prevention Funder Learning Collaborative summer employment programme, specifically targeting young people with a history of, or at risk of offending, there is a strategy of engaging participants in group talks and team problem-solving activities to foster communication, social skills, and critical thinking – Sum (2015) finds some evidence of participation in the programme reducing engagement in violent, offending and/or anti-social behaviour. Modestino (2019b), Heller (2014), Davis and Heller (2020) also estimate negative (i.e., beneficial) impacts on the likelihood of experiencing criminal justice outcomes from participation in the summer programme; Modestino and Paulsen (2019a) finds a positive impact on the likelihood of progressing to higher education, suggesting a diversionary effect through improved soft skills increasing the ability of summer employment participants to apply and progress to higher education.

Relationship with peers

Where summer education programmes, particularly those focused on transition support, involve older students as mentors, this provides valuable support to the young person while making their transition; building participants' networks in the new environment facilitates the transition, leading towards further outcomes (Day et al., 2013b).

Interactions with peers also provides valuable learning opportunities in summer employment programmes, as highlighted by the Urban Alliance programme where, during a pre-work training, an episode of confusion and frustration among teammates led to a spontaneous lesson on the importance of patience and helpfulness (Theodos et al., 2014), with Theodos et al. (2017) estimating positive impacts on soft skill comfort from participation in the programme.

Personalised and positive relationships with staff

The lower student-to-staff ratio that summer education programmes often have (Day et al., 2013a; Martin et al., 2013b; Maxwell et al., 2014) enables individualised attention and support to be delivered to young people that in turn promotes student engagement. This also means that programme staff can better address each student's specific needs and thereby ensure participants' academic and personal growth. Martin et al. (2013a), who studies an intervention with a lower student-to-staff ratio than would be expected in BAU, finds a positive impact of participation in the summer education programme on indexes of pupil confidence, school readiness and socialisation, and Maxwell et al. (2014) finds a positive impact on participants enjoyment of reading and motivation to read. Receipt of mentoring and opportunities for leadership within summer programme activities further enhance the positive relationships between staff and students. The evidence highlights that these positive relationships foster a sense of competence and belonging, promoting positive learning experiences and increased engagement and integration into the college or school community. Wachen et al. (2018), who highlights the importance of the relationships between staff and students, finds positive impacts of programme participation on retention in and completion of higher education. One of the benefits of transition support programmes, in particular, is that these provide students with the chance to meet their new teachers before the academic year commences, with emphasis also placed on the social aspect of the relationship. This early interaction helps students build positive relationships with staff, creating a sense of belonging, increasing confidence and comfort, and easing the transition. In the early phases of the new academic year, staff utilise the rapport established in summer transition support programmes to individualise support and foster engagement in the BAU classroom – note the aforementioned impacts of programme participation found by Martin et al. (2013b) on indexes of pupil confidence, school readiness and socialisation, and of Wachen et al. (2018) on retention in and completion of higher education.

In summer employment programmes, the positive relationships staff – including programme staff and workplace supervisors – build with participants also play a vital role in supporting positive outcomes, such as improved well-being. There is evidence of participants in summer employment programmes reporting improved social skills, such as asking staff for help and support. Furthermore, as detailed in the background section on how the intervention might work, adult relationships are also formed with employees in the employing organisation which, along with the employer's expectation of performance from the young person, builds responsibility, maturity and self-esteem. Modestino (2019b) finds positive impacts from programme participation on socio-emotional engagement.

Location of the summer programme

In the case of transition support programmes, there is evidence showing that allowing students to familiarise themselves with the new campus environment, can alleviate anxiety and help them feel more comfortable, which in turn promotes the achievement of wider soft outcomes including measures of school readiness (Martin et al., 2013a). Immersive experiences on university campuses, access to school resources, emphasis on navigating the college environment, and creating familiar learning environments all contribute to the effectiveness of these programmes: the 5-week University of North Carolina summer bridge programme provides students with experience in navigating the university campus, using the instructional technologies and accessing academic support services, with Wachen et al. (2018) estimating positive impacts on the likelihood of retention in and completion of college as a result of participation.

Location is also an important mechanism to the outcomes from summer employment programmes. Locating young people in an organisation for the job placement builds familiarity and confidence in this new setting. I also increases expectations for conduct in this adult environment, with Modestino and Paulsen (2019a) finding positive impacts of summer programme participation on job readiness.

Creating links to ‘business as usual’ learning over the summer

In some summer education programmes, it is highlighted that creating connections between the summer programme and the students' learning at home (through worksheets, activities, or recommended reading materials throughout the summer holidays) helps maintain continuity and reinforces learning (Day et al., 2013b).

Improving prospects and aspirations

A further notable finding pertains to summer employment programmes' effect on education outcomes. When participants find purpose and meaning in their work, potentially further facilitated through the provision of financial and/or non-financial incentives, they are more likely to see the importance of education in achieving their life goals (Leos-Urbel, 2014; Modestino & Paulsen, 2019c). As detailed in the background section on how the intervention might work, Modestino (2019) identifies a mechanism through building aspiration, self-belief, emotion control and a longer-term work ambition. The summer job encourages young people to improve their engagement with education as a precursor to achieving newly found higher quality employment goals. The post-participation survey results from One Summer Chicago reveal that 70% of participants recognised the importance of education in achieving their life goals. This suggests that the programme successfully instils the importance of gaining qualifications to building the career you want. By emphasising the connection between education and future aspirations, these programmes motivate participants to actively pursue further academic success. As previously mentioned, Modestino and Paulsen (2019a), who evaluates the similar Boston Summer Youth Employment Program, finds a positive impact of participation on the likelihood of progressing to higher education.

Financial and non-financial incentives

The financial incentives provided by summer employment programmes, as suggested in the background section on how the intervention might work, may also help to alleviate financial constraints on future education, increasing investment in human capital – Modestino and Paulsen (2019a) finds positive impacts of summer programme participation on the likelihood of progression to higher education.

Repeat participation

Long-term participation in a summer employment programme is also associated with larger positive impacts on academic performance and test taking. The effects tend to be more significant for second and third-time participants, suggesting that the benefits may accumulate over multiple years – self-selection may however play a role here, as motivated students who are more likely to achieve better outcomes are more likely to apply for additional years of participation (Schwartz et al., 2021). It should be noted here that ascribing any causality to this mechanism is especially problematic. While there may be a dosage effect whereby participating in the summer employment programme multiple times provides greater benefits, findings of a greater impact on outcomes because of this could be partly or largely a result of self-selection. For instance, those who benefit the most from participation may select into re-applying and repeatedly participating in the programme.

Design strengths and weaknesses

Building on the causal pathways and facilitators to participation and outcomes, some key design concepts can be gleaned for summer education programmes. Features of effective design include: ▪

Using approaches to learning that are interactive and/or differ from mainstream schooling to motivate students and promote their interest in learning activities (Cosentino et al., 2015; Sharp, 2018). This is particularly evident in catch-up programmes (Lynch & Kim, 2017; Somers et al., 2015; Torgerson et al., 2014; Wathington et al., 2016).

Across summer education programmes, the evidence highlights the importance of designing in planned progress indicators and developing tools to measure these (Day et al., 2013b). Delivery organisations also need to carefully assess what they can provide ‘in-house’ (based on their own strengths) and where partnerships are needed (Day et al., 2013b). ▪

The evidence also highlights design weaknesses and pitfalls to avoid. These involve a lack of consideration at the design phase of:

In deciding whether to take forward summer education programmes and the designs that are more feasible, it is also relevant to consider cost-effectiveness. Some evidence highlights that some programmes, particularly those centred on catch-up, have high ongoing costs per pupil, compared to other catch-up approaches delivered during the regular school year (Torgerson et al., 2014).

Implementation strengths and weaknesses

Overall, the evidence from summer education programmes highlights some key themes that underpin successful implementation. Successful summer education programmes include clear programme delivery guidance and good governance, high quality academic instruction, mentoring support, and strong partnerships.

Schools participating in a UK programme, highlighted funding and guidance provided by the funding and administering authority, which in this case was the Department for Education (DfE), as key to successful implementation. Clear communications about funding arrangements and involving the school governors in monitoring progress ensured transparency and accountability. Insight from this programme also highlights the importance of ensuring school administrators are aware of the funding arrangements and level of funding, are effectively managing these resources, and keep school governors informed about how funding is utilised. This allows foresight on funding adequacy, which means additional resources can be sought where necessary. In this example, additional funds were raised through local businesses, use of volunteers, and in-kind support (CooperGibson, 2022).

Another UK-based transition support programme, the Future Foundations summer school programme, highlights the importance of ensuring high-quality delivery through a structured curriculum, and engaged mentors and teachers. Regarding the latter, a key aspect of successful implementation is that in the third week of the Summer School, a changeover phase takes place where a new batch of teachers replaces the previous ones, which provides alternating teaching and rest periods to staff. It helps to ensure that those teaching the programme can bring their full energy, while allowing rest in between the standard academic terms. It also works to ensure that the Summer School maintains a high level of teaching quality. Before the changeover, a handover session is organised to allow new teachers to meet their colleagues, learn about the students, and understand the site's rhythms and routines. This enables a smooth transfer of information and ensures that the new teachers are well-prepared to continue the programme effectively. Both batches of teachers receive the same training, ensuring consistency to help maintain the quality of instruction and ensure that all teachers are well-prepared for their roles. A further key feature is that although the teachers changed, mentors and peer mentors remained the same. This consistency ensured some continuity for participants throughout the programme (Siddiqui et al., 2014).

The key role played by highly engaged mentors in facilitating successful delivery is also highlighted in other programmes. In one US-based transition support programme, Scholars Academy, participants cite peer mentors as one of the best parts of the summer school. Peer mentors are described by participants as someone to relate to outside of their teachers and families who can guide them on which societies to join, and introduce them to other students, facilitating their successful transition into their first year of university (Henson, 2018).

Partnering with external organisations and drawing upon their expertise in curricular and enrichment activities, such as sports, arts, and drama, may enhance the success of summer education programmes. In particular, in effective transition support programmes such as the DfE Summer Schools, secondary schools engage with feeder primaries early on and co-design programmes to ensure relevance and effectiveness for prospective participants. The DfE Summer Schools work particularly well where they offer curricular and enrichment activities with an emphasis on ‘fun’. This enables pupils to enjoy new experiences, build confidence, reinforce learning and develop positive patterns of behaviour. Additionally, embedding proactive measures to address students' concerns, such as bullying, or anxiety about making new friends, is highlighted as an aspect of successful delivery (Day et al., 2013b). Providing specific activities to help new bonds to form, alongside support and opportunities for pupils to mix with their peers and school staff, to become familiar with the expected behaviours and boundaries at secondary school, is suggested as fostering a safe and supportive environment which facilitates delivery.

Also key to successful implementation is integrating programme evaluation and the sharing of results. The UK DfE Summer Schools programme conducted a thorough review of implementation and delivery with staff, and shared the learnings and effectiveness information with stakeholders. The report identified key success factors to be built upon and improved in future planning, including: the provision of diverse and engaging learning activities; individual target-setting and mentoring for disadvantaged pupils; open discussion of topics that might be causing pupils concern (e.g., bullying, transitioning to the next stage of education, and the unfamiliar school environment); involving older students to welcome incoming students; involving parents in the delivery of the summer school and organising a celebration event; and identifying and assessing students' strengths and weaknesses during the summer school, to inform planning for the school year. Integrating evaluation activities also supports in identifying transferable learning from the summer programme to other areas of the school, such as wider transition support programmes, curricula, and learner support, ensuring that the benefits extend beyond the summer period.

In contrast to the implementation strengths, the review also highlights challenges encountered in implementing the summer education programmes. In some cases, these are the opposites of the strengths identified above. The key weaknesses cover: ▪

insufficient planning and lead in time, particularly common in transition support programmes between primary to secondary education, which in some cases results in schools not being able to run the programmes (CooperGibson, 2022);

Design strengths and weaknesses

The evidence that meets the inclusion criteria for this review, which is all drawn from the US, highlights a number of key design features in successful summer employment programmes which support their implementation. These include: ▪

Use of employer orientation materials and supervisor handbooks, such as those used by the STEP-UP programme – these resources are used by the public employment service to assist with recruitment, and to match participants with suitable positions, as well as providing guidance on payroll paperwork, and managing time reporting throughout the summer programme. These materials inform, engage, and prepare employers and supervisors, fostering successful work placements that feature structured work plans and measurable goals (Reich, 2018).

Some summer employment programmes also exhibit design weaknesses, which impact the quality of delivery. This can include: ▪

Uncertainty over funding and budget agreements – this was observed in the New York City Summer Youth Employment Program. Where providers do not receive final funding commitments within good time before a programme begins, it makes it challenging to plan services effectively (Valentine et al., 2017).

Implementation strengths and weaknesses

In successful summer employment programmes, effective job matching, supportive relationships, pre-work training, and mitigating attrition are all features of effective implementation.

In terms of job matching, insight from studies of One Summer Chicago highlights that matching young people to jobs that align with their interests and career goals enhances their overall experience of the summer employment programme, as they are more likely to be engaged and motivated in their work and work environment, and more likely to perform well (Lansing et al., 2018).

The quality of the summer work experience and the support provided by supervisors in the workplace are also important, as highlighted in the Youth Violence Prevention Funder Learning Collaborative summer employment programme. Positive relationships between supervisors and programme participants contributed to a valuable learning experience, a sense of contribution, and the development of soft skills. Supportive supervisors played a significant role, with the majority of participants in the programme evaluation study reporting that the programme supervisors had provided various types of help – ranging from positive assistance, including exploring new avenues for the future, to supporting the young person to avoiding negative behaviours by staying off the street (Sum, 2015).

Similarly, in the Urban Alliance programme coordinators played a key role in providing support and guidance to students outside of their placement, particularly for those requiring additional assistance or support. Successful relationships with programme coordinators and mentors were considered to lead to improved job performance among participants and overall better programme outcomes (Theodos et al., 2014). The STEP-UP programme also highlighted that the quality of the summer employment programmes is closely related to the availability of job supervisors. In the Discover track, 128 people supervised interns, of which 29 worked with a single intern and 99 had multiple interns. There were 385 Achieve supervisors, of which 298 worked with a single intern and 87 had multiple interns. Approximately 93% of Achieve supervisors and 95% of Discover supervisors attended an orientation before the summer programme began (Reich, 2018).

Some summer employment programmes also exhibited implementation challenges. An important element in the implementation of the New York City Summer Youth Employment Program was the use of ‘monitors’, provider staff members that visited each work site weekly to check with young people and supervisors, to ensure participants' regular participation, safety, and well-being. However, there were not sufficient monitors for the number of participants, especially at work sites employing a large number of participants. In the implementation study, young people reported having limited interactions with their monitors, and those mostly concerning timesheet collection, with some not having seen their monitors regularly at all (Valentine et al., 2017).

The review also highlighted that one programme, Urban Alliance, experienced challenges with employer availability at the planned commencement time, resulting in some young people starting their work placements later than anticipated. This had an effect on young people's overall experience and limited participants' engagement in meaningful work (Theodos et al., 2014). Urban Alliance also faced significant attrition at various stages of the programme, including between the application and the start of pre-work training, which happened during the school year, and throughout the work placement. The reasons for attrition were not fully observed or predictable, but factors such as changes in training classes timetables, extracurricular activities which clashed with the training, and family issues were highlighted as likely to be influencing attrition. Urban Alliance programme coordinators also reported that some participants needed a lot of support and guidance from their mentors and, if matched successfully, they performed well in their jobs. However, participants who were matched with mentors who were unsupportive or too busy with other work were at higher risk of disengagement (Theodos et al., 2017).