Interventions to improve the labour market outcomes of youth: A systematic review of training,entrepreneurship promotion,employment services and subsidized employment interventions

1.3.1 Training and skills development

Education and skills are considered a core factor in determining young people's opportunities in the labour market (Biavaschi et al., 2012). Skills training programmes are therefore the most widely used labour market intervention for young people worldwide and are increasingly delivered as a complement to other labour market measures (Betcherman, Godfrey, Puerto, Rother & Stavreska, 2007; Fares & Puerto, 2009). Training and skills development comprises programmes outside the formal education system that offer skills training to young people in order to improve their employability and facilitate their transition into the labour market. ¹ The objective of skills training programmes is to develop the employment-relevant skills of jobseekers. Broadly speaking, these skills refer to a set of job-specific technical skills, but also include non-technical soft skills, such as self-management, teamwork and communication. Increasingly, employers across the world are placing higher value on these non-technical skills than on technical competencies (Manpower Group, 2013; Cunningham, Sanchez-Puerta & Wuermli, 2010; Youth Employment Network & International Youth Foundation, 2009).

This analysis classifies training programmes according to the skill set which they target (Table 1): 1

First, training programmes that address a lack of trade- or job-specific technical skills demanded by employers. Such skills range from manual skills to computer literacy. Technical skills training programmes often include an on-the-job training component in order to increase practical work experience (i.e., by placing participants in internships, workplace training or apprenticeship schemes).

Technical training programmes are popular in development cooperation because many developing countries experience a skills mismatch between their labour force and emerging segments of their economies. However, pure training programmes have not proven to be particularly successful in many contexts (Betcherman et al., 2004). Therefore, most recent programmes tend to combine skills training with other types of interventions; for example, on-the-job training or employment services (Cunningham et al., 2010; Fares & Puerto, 2009). An example of a skills training programme is provided in Box 2.

A number of conditions determine whether skills training programmes are successful in bringing additional youth into work – most Not ably, correlation between the skills offered by a training programme and those demanded by the market. To this end, some programmes introduce a market-based (or bottom-up) approach in programme design. The application of this approach enables training curricula and programme components to respond much more effectively to the needs of employers (in both private and public sectors) and communities in a demand-driven fashion.

Furthermore, the success of all these interventions relies on the assumption that the (correct) target group participates in the training and that the training is appropriate and conducted in a way that actually augments the skill sets that are relevant to the labour market. Finally, a crucial element may be the award of a legitimate certificate on successful completion of a programme to prove the acquisition of increased knowledge and skills to potential employers in the job market.

1.3.2 Entrepreneurship promotion

Innovative entrepreneurial activities can promote job-rich growth and accelerate economic diversification paths through productivity and competitiveness. Entrepreneurship returns to economic development are maximized within business environments that are amenable to innovation and creativity and provide appropriate regulations, access to infrastructure services and finance (ILO, 2015b). However, entrepreneurship also carries substantial risks of failure and has the potential to contribute to job losses if increased productivity and competition leads to layoffs in existing enterprises (Kritikos, 2014).

Entrepreneurs are important income providers and job creators. They benefit booming economies by challenging existing enterprises to innovate and compete in order to keep up with rapidly changing technologies and global markets. They also benefit economies that are suffering from slow job growth or stagnation by boosting labour demand, developing innovative goods and services and stimulating competition.

Depending on the context, entrepreneurs can be driven by choice or by necessity. Entrepreneurs by choice select entrepreneurship over other employment options in order to increase their income or become more independent. Entrepreneurs by necessity, also known as subsistence entrepreneurs, face a market situation with insufficient labour demand and therefore lack formal employment opportunities, exposing their entrepreneurial ventures to the low productivity and precarious working conditions that prevail in the informal economy.

The enterprise size and its corresponding ability to grow and to create jobs also help to identify the rare “transformation entrepreneurs” or “gazelles”. These are the few entrepreneurs whose enterprises grow to become larger enterprises and generate most of the new jobs. Their high-growth enterprises create jobs and income for others, beyond the scope of an individual's subsistence needs (Cho, Robalino & Watson, 2014). In contrast, the enterprises of subsistence entrepreneurs usually do not grow, but provide income and employment for the owner of the micro-enterprise and their immediate family.

Entrepreneurship promotion programmes considered for this systematic review aim to lower the barriers and costs associated with young unemployed and underemployed people planning to establish or maintain a business. Since the scope of formal wage employment is often limited in developing countries, increasing (formal) self-employment among the labour force is considered an important anti-poverty strategy (Gindling & Newhouse, 2012). Because self-employed and small-scale entrepreneurs often face numerous internal and external constraints, a multitude of measures exist to support the process.

Access to capital is often a primary constraint for young entrepreneurs. Schoof (2006) identifies a number of constraints to accessing start-up finance. These range from inadequate personal savings and resources to a lack of securities and credibility, insufficient business experience and skills, strict credit-scoring methodologies and regulations, among others. Accordingly, many entrepreneurship programmes address the lack of access to (affordable) finance faced by young entrepreneurs. The review team disaggregated such programmes into three types: 1

Those providing or facilitating access to credit (including microfinance programmes)

ALMPs that facilitate access to finance often provide technical training and advice and support setting up partnerships and capacity-building schemes with (and for) microfinance institutions (MFIs) and banks.

In addition to access to finance, some programmes offer training on business and management skills as well as business advisory services and mentoring for soon-to-be or already self-employed youth. Finally, some interventions aim to reduce the barriers to business creation by assisting prospective entrepreneurs to enter established markets or existing value chains. The abovementioned interventions and their results chain are shown in Table 2. Some skills training programmes (as described in Section 1.3.1 above) incorporate features of entrepreneurship training and specific skills relevant for starting or maintaining a business.

Many entrepreneurship programmes take a multi-component approach; for example, combining access to credit with business skills training or the provision of post-programme consultation (i.e., mentoring and coaching).

Primarily, entrepreneurship programmes increase employment through their direct effect on the soon-to-be self-employed participant. The assumption is that beneficiaries actually plan to set up a new business after receiving credit and/or training (i.e., that targeted and trained individuals have been appropriately selected for the programme) and that they would not have done so without the intervention.

In order to generate additional jobs, entrepreneurship programmes have to assume that the intervention leads to either (i) increased marginal productivity of the input labour or (ii) increased output and profits resulting in additional investments and labour demand. To achieve this end, the training must suit the context and knowledge of the participants. Beneficiaries then have to apply the training or credit to their business and thereby increase performance and competitiveness. ³ Whether or not an entrepreneur will finally hire additional workers may also depend on the macroeconomic and labour market environment.

Box 3 describes the programme Start and Improve Your Business (SIYB), a widely used and adapted entrepreneurship training package designed by the International Labour Organization (ILO) and tailored for youth.

1.3.3 Employment services

Employment services programmes are generally based on the (matching and) intermediation approach to active labour market policy. Interventions within employment services are shown in Table 3. Job-placement programmes acknowledge the existence of information asymmetries and, particularly, incompleteness of information in the labour market. Hence, these programmes aim to improve the job-matching process by providing information and support to both sides of the labour market. On the one hand, they inform young jobseekers about suitable job opportunities (a service which is of particular relevance to youth who have only recently entered the labour market and are experiencing difficulties in marketing themselves or lack the knowledge, information and networks to find job openings). On the other hand, they provide information to potential employers about unemployed youth. The underlying idea is to facilitate the matching of employment opportunities with jobseekers while reducing the costs and risks to employers connected with recruiting young people.

The second type of intervention, job-search assistance services, includes job-search training, educational or career guidance, counselling and monitoring programmes. Such programmes primarily target disadvantaged or demotivated youth who are disconnected from the labour market. Their primary aim is to improve the intensity, motivation and effectiveness of participants’ job-searches.

Mentoring programmes are also provided to youth who are not currently unemployed but are in education or have just entered the labour market (post-placement support). Accordingly, in some circumstances, mentors encourage mentees to stay in education or in on-the-job training. In many countries, employment agencies adopt a case-management approach (identifying barriers to employment, designing individual action plans, referring jobseekers to appropriate interventions and monitoring job-search activity), which has been argued to be the most effective method of providing these services (Walther & Pohl, 2005).

While in some countries public employment agencies continue to be the main providers of employment services, other countries have moved into subcontracting, opening an important role for private employment agencies to address mismatches and information failures in the labour market. Box 4 illustrates a subcontracting model applied by a French public employment agency to facilitate counselling and job-placement for educated youth.

There are indications that involvement in employment services (and in ALMPs in general) has a stigmatizing effect on participants (Boone & van Ours, 2004; Kluve, Lehmann & Schmidt, 1999). Addressing this adverse effect is a prior condition for successful implementation. To this end, job-placement and job-search assistance programmes are often connected to financial incentives for jobseekers and/or employers. For example, such schemes may involve the imposition of sanctions on the unemployed for failure to comply with the terms of the intervention. Similarly, marketing of unemployed youth may be combined with the offer of short-term subsidies to employers.

1.3.4 Subsidized employment

Insufficient labour demand is one of the main constraints faced by young job market entrants – particularly in developing economies. Subsidized employment interventions comprise two main areas: wage subsidies and labour-intensive public employment programmes (Table 4), both of which are designed to increase the job and training opportunities available to unemployed youth. The main aim of both types of intervention is to ensure that individuals who do not find a job on the regular labour market remain integrated and connected to economic and social life. To that end, such programmes offer short-term interventions but primarily work towards longer-term labour market impacts.

Wage subsidies are transfers to employers or employees in order to fully or partially cover eligible individuals’ wage or non-wage employment costs. Most often, the measures aim to incentivize employers to hire members of a specific target group. Wage subsidies come in numerous forms and can be offered through various mechanisms, ranging from direct transfers to firms or workers to reductions in social security contributions or payroll taxes or tax credits.

Employer-side subsidies reduce the financial costs or risks associated with not knowing the productivity of the person to be employed. As with employment services, this is a scheme which is particularly relevant to youth entering the labour market for the first time, and whose (perceived) marginal productivity may be below market wages. Employer-side subsidies may also serve to lower the costs to employers of providing on-the-job youth training. Such training subsidies offer the possibility of expanding the number of work-based training places for disadvantaged young people.

Employee-side subsidies promote labour supply through increasing the returns from employment and hence increasing incentives to seek and retain employment. While it is believed that employer-side subsidies may also encourage more active job-search (because youths believe they will be able to find work), providing employee-side earning supplements may permit more effective targeting of specific socio-demographic groups. Furthermore, whereas employer-side subsidies tackle a lack of labour demand, employee-side subsidies may be more appropriate in countries that face labour supply constraints, for example due to reservation wages.

It is important to acknowledge the limited use and evidence of wage subsidies in developing countries. Almeida et al. (2014) detail the results of experimental and quasi-experimental impact evaluations around the world. Most evidence comes from the United States with rather mixed results concerning the effectiveness of wage subsidies as tools for fostering job creation.

Evidence on the impact of youth-targeted wage subsidies in developing countries is limited and results are mixed. Evaluations looking into wage subsidies in Jordan (Groh, Krishnan, McKenzie & Vishwanath, 2012) and South Africa (Levinsohn, Rankin, Roberts & Schoer, 2014) show positive though rather short-lived effects and a narrow participation from firms. Details of the Jordan New Opportunities for Women pilot are shown in Box 5: Subsidized employment: Jordan New Opportunities for Women (Jordan NOW). A recent review of wage subsidies for youth argues that, if well targeted, the interventions can be effective in improving employment outcomes of disadvantaged youth (Bördős, Csillag & Scharle, 2016).

The second type of labour market intervention analysed in this category is labour-intensive public employment programmes, also known as public works. These programmes are commonly used to increase aggregate demand for labour in contexts where markets are unable to create productive employment on the required scale. In addition to their ability to create direct jobs, public employment programmes also generate income and deliver public assets and services. Despite the strong association of these programmes with infrastructure and construction works, they can be quite versatile, with works and projects in the social sector, environmental services and multi-sectoral, community-driven programmes (Lieuw-Kie-Song, Philip, Tsukamoto & Van Imschoot, 2010; Lieuw-Kie-Song, Puerto & Tsukamoto, forthcoming).

In this type of intervention, basic social income recipients are recruited for public jobs and receive a small earning supplement to their unemployment assistance. Programmes usually target unskilled, disadvantaged or long-term unemployed workers with the aim of keeping them in contact with the labour market and mitigating the depreciation of human capital during periods of unemployment.

While public employment programmes have often been recommended as a measure in times of crises (such as seasonal shocks or economic recession), ⁷ they are increasingly used as a regular component of wider employment policies (Lieuw-Kie-Song et al., 2010). In addition, they have become popular as a mechanism for addressing youth unemployment (Grosh, del Ninno, Tesliuc & Ouerghi, 2008), serving both as an introduction to the world of employment and as a tool to maintain social integration. This is particularly relevant for youth service programmes, in which youth can “play an active role in community and national development while learning new skills, increasing their employability, and contributing to their overall personal development” (Cunningham, McGinnis, Verdú, Tesliuc & Verner, 2008).

Most wage subsidies and public employment programmes are designed to support employment only in the short or medium term. A positive effect on final outcomes is only attainable if the work experience and training received during the period of subsidized work also improves the longer-term employment prospects of participants. For this reason, (i) wage subsidies are often granted to firms that agree to provide additional training to subsidized employees (i.e., in connection with apprenticeship schemes) ⁸ and (ii) public employment programmes are often paired with exit strategies, such as skills training or entrepreneurship.

3.2.1 Population and context

The review was global in coverage and considered interventions from all countries, regardless of their level of development. Studies investigated active labour market policies (ALMPs) that were designed for – or targeted primarily – young women and men aged 15 to 35, in consideration of varying national definitions of youth.

3.2.2 Intervention

The ALMPs examined in the study (i) targeted the unemployed or those with low levels of skills or limited work experience or who were generally disadvantaged in the labour market and (ii) aimed to promote employment and/or earnings/wage growth among the target population, rather than simply providing income support (Heckman et al., 1999). Eligible studies evaluated an ALMP that provided at least one of the following categories of intervention (also shown in Section 1.3): training and skills development, entrepreneurship promotion, employment services and/or subsidized employment. An overview of the categories of intervention is presented in Table 6.

As discussed in more detail in the protocol (Kluve et al., 2014), this review made an important distinction between programmes, interventions and components of an intervention: A youth employment programme was considered to be a single entity that might consist of one or several interventions. In addition, each of these interventions could have different components: It was possible to find a comprehensive intervention that offered, for instance, both skills training and employment services (to the same participant). Some examples of such multi-component interventions included the Job Corps programme in the United States, the Economic Empowerment for Adolescent Girls programme in Liberia, the Projoven programme in Peru and the Employment Fund in Nepal.

Interventions were therefore specific tracks or sub-programmes of an overall programme that were offered to different samples of participants. They were defined based on their characteristics, such as the category of intervention or the population targeted. For example, if a programme had a training track and an employment services track and participants took one or the other, they were considered to be two interventions within the same programme. Note that, according to this definition of track, it was assumed that each intervention within a programme had separate groups of participants which did not overlap. In order to provide evidence on which interventions and combinations were shown to work best, these different types were evaluated separately in the meta-analysis in the empirical Section 4.3 on Synthesis of results.

Additional consideration was given to identifying primary intervention types among multi-component designs. The review defined “main category of intervention” as the largest and predominant intervention type within a programme. If several intervention types were equally distributed across the target population (i.e., an individual was exposed to more than one intervention type with the same level of intensity), the main category of intervention was classified as unspecified.

3.2.3 Comparison

The systematic review included studies that measured change in at least one outcome of interest among intervention participants and relative to non-intervention participants based on a counterfactual analysis. Eligible comparison groups (counterfactual) included those which received no intervention or were due to receive the intervention in a pipeline or waitlist study. Note that the comparison group of some studies might have been exposed to interventions other than the evaluated intervention. The review excluded studies that only measured the relative effects of two alternate interventions, without reference to a non-intervention comparator.

3.2.4 Outcomes

Eligible studies reported at least one selected outcome variable measuring the following primary outcomes of interest presented in Table 7: Employment outcomes, earnings outcomes and business performance outcomes. The review also captured outcomes which were measured conditional on other outcomes.

3.2.5 Study designs

The review focused on completed experimental and quasi-experimental evaluations and considered the following research design categories and impact evaluation methods to estimate quantitatively the causal effect of the intervention on the outcome it intended to influence: (i) randomized experiments, (ii) methods for causal inference under unconfoundedness (classical regression methods, statistical matching, propensity score matching) and (iii) selection on unobservables (instrumental variables, regression discontinuity design, difference-in-differences). 1

Randomized experiments: The most straightforward case for analysis occurred when assignment to treatment was randomized (in controlled conditions) and, therefore, independent of covariates X as well as the potential outcomes Y . In such classical randomized control trials (RCTs) it was relatively easy to obtain estimators for the average effect of the treatment using, for example, the simple difference-in-means by treatment status. Randomized experiments have been used in the evaluation of labour market programmes since the 1970s (starting in the United States), with an increasing trend over the past decade. The descriptive analysis by research design included in this review in Figure 3 confirmed that RCTs have increasingly been used to assess the impact of youth employment interventions in recent years.

3.2.6 Other inclusion criteria

The form of publication of eligible studies included peer-reviewed journal, working paper, mimeo, book, policy or position paper, evaluation or technical report and dissertation or thesis. Eligible studies could be published in any language. The date of publication or reporting of the study had to fall between 1990 and 2014.

3.3.1 Scoping search

Prior to implementing the primary and complementary search, the review team conducted a scoping search of potentially relevant sources to determine their relevance and to develop customized search strategies which would yield relevant results. The scoping search entailed an iterative process of testing and documenting several search strategies and identifying one or more preferred search strategies and search strings for each source in order to yield a comprehensive and precise set of potentially relevant results. The relevance of sources was determined by screening the results obtained from implementing each customized search strategy.

Based on a review of preferred search strategies and the results obtained during the scoping search, selected databases and websites were not included in the final search strategy if the review team did not have access to the source (e.g., SocIndex), the results obtained from the source were of low relevance (e.g., African Economic Outlook) or the source was covered by another source (e.g., ILO working papers are included in Labordoc). The final primary and complementary search strategy covered more than 70 sources, which included general databases, specialized databases, institutional websites, conference websites, dissertations and theses databases and grey literature databases. Section 9.6.1 in the Appendix presents the list of sources included in the final primary and complementary search.

3.3.2 Primary search

The primary search included 11 general databases and 12 databases that specialize in literature relevant to development economics and labour market issues. The search terms used for the primary search were based on the inclusion criteria and tried to strike a balance between sensitivity (e.g., finding all available articles in a topic area) and specificity (e.g., finding only relevant articles). For electronic databases with advanced search functions, the preferred search was based on a search of exposure, outcome and subject terms using Boolean operators in title and abstract from 2000 onwards. Highly relevant databases, such as EconLit, were searched for studies published since 1990 in order to include potentially relevant studies between 1990 and 2000. The search terms for electronic databases and examples for RePEc/IDEAS, EconLit and ERIC are presented in Sections 9.6.3 and 9.6.4 of the Appendix. The search strategy was modified according to the specifications of each database. Wherever possible, synonyms as well as wildcards and truncation symbols were applied as appropriate. The use of synonyms also accounted for British or American English spelling. To account for terminology differences across disciplines, database thesauri were consulted to ensure that all appropriate synonyms were included. Where available, the team also relied on the database's index terms and/or free-text terms. For databases or websites with basic search functions, the review team adjusted the search terms to accommodate the limited functionality of search functions and adapted these customized search strategies to relevant keyword searches and/or topic/theme searches based on the test results of keyword combinations of search terms. The search of electronic databases was completed in February 2014. From November 2014 to January 2015, the review team contacted experts and authors of included studies, screened reference lists of included studies and conducted citation tracking in order to identify additional studies. The search dates for each source used during the primary and complementary search process are presented in Section 9.6.1 in the Appendix.

3.3.3 Complementary search

The primary search was complemented by hand-searching and screening of 35 websites, such as institutional and conference websites, five dissertation, thesis and grey literature databases, nine other reviews and meta-analyses and literature snowballing as well as contacting experts and relevant institutions. The hand-searching strategy was customized for each relevant institutional website. Search terms were used for websites that included a search facility. Otherwise, relevant sections (for example, “documents” or “publications”) were searched. Websites of conferences that were deemed relevant to the research question were searched for potentially relevant studies. To include potentially relevant dissertations and theses that were not indexed in bibliographic databases, the review team searched national and international dissertation and thesis databases. The review team also conducted citation tracking and screened reference lists of included studies and relevant existing reviews and meta-analyses to identify further studies for inclusion. The review team contacted authors of previous reviews and included studies, as well as experts and individuals coordinating youth employment related topics in relevant institutions, to ask whether they knew of any studies that might be applicable in addition to the studies that were included after the full text review of full reports. Ongoing and unpublished studies within the grey literature were identified through the screening and hand-searching of relevant websites/gateways and conference websites, citation tracking and contacting experts and relevant institutions. In addition, a keyword search was undertaken for the grey literature databases.

3.4.1 Data extraction

Relevant information from included studies was systematically extracted using a coding tool and coding manual. The coding tool, which is presented in Section 9.7 of the Appendix, included information about variables related to study methods, the characteristics of the intervention and its implementation, the characteristics of the subject samples of analysis, the outcome variables and statistical findings, and contextual features.

At effect size level, the coding tool captured sub-group analysis of employment, earnings and business performance outcomes and estimated treatment effects by age cohorts, gender, educational level, income level and location, among other dimensions. Types of outcomes were further disaggregated by occupation category (dependent vs. self-employment), status of occupation (formal vs. informal) and conditional on other outcomes. To describe the data and empirical methods, the coding tool included information about the research design, statistical methodology, type of significance test, type and method of measurement, date of data measurement and data source. The coding tool also captured the form and year of publication.

For each category of intervention (i.e., skills training, entrepreneurship promotion, employment services and subsidized employment), the coding tool extracted information about the type of intervention, targeting and delivery mechanism, payment system and provider, duration of specific interventions, selection of participants and conditionality of eligibility. General programme characteristics recorded details of the target group by age, gender, educational level, income level, location and employment status as well as the type of organizations involved in designing, financing and implementing the programme. The coding tool kept a record of region, country, scale and average duration of the programme. In addition to any awareness-raising efforts and gender considerations integrated into programme design and implementation, it also captured the incentives, monitoring mechanisms and sanctions for non-compliance connected with the programme. ¹²

A separate section of the coding tool was used to record information when the study reported intermediary outcomes or outcomes other than the ones considered in this review. This section also captured additional sub-group analyses, relative treatment effects, general equilibrium effects, costs of the programme or cost-benefit analysis, as well as any implementation problems or empirical identification problems described by the author.

A coding manual provided detailed instructions for coders in order to ensure consistency in extracting and interpreting relevant information, in particular with regard to the selection of appropriate treatment effect estimates. Guidelines were provided to identify the treatment effect estimates with the lowest risk of bias when studies reported multiple estimates for the same types of outcomes. Coders selected the preferred method of estimating the effect and their choices were verified by a second reviewer. For example, estimates based on experimental designs were considered to provide the lowest risk of bias followed by natural experiments and quasi-experimental designs. Other considerations outlined in the manual to mitigate the effects of potential bias included the use of covariates, the type of data used and the statistical methodology applied for the estimation.

Information extracted from included studies was discussed with a second reviewer and coding decisions involving assumptions were documented by each researcher. Further information about the selection of studies and data extraction can be found in the protocol of the systematic review (Kluve et al., 2014).

3.4.2 Standardizing effect size estimates

To compare estimated treatment effects across studies, the standardized mean difference (SMD) was computed for both continuous outcome variables (e.g., income) and dichotomous outcome variables (e.g., employment probability) reported in the primary studies. In addition, researchers computed a binary variable holding the value of one if a treatment effect was positive and statistically significant (PSS). This report focuses on the SMD-based findings. The analysis of PSS indicators can be found at Kluve et al. (2016).

The SMD captured the relative magnitude of the treatment effect in a way that is dimensionless and hence comparable across outcomes and studies. It was the ratio of the treatment effect for a specific outcome relative to the standard deviation of that outcome within the evaluation sample used to estimate the treatment effect. Most studies reported either matching- or regression-based estimates of the treatment effect (even for RCT-based designs). ¹³ Hence, SMDs in most cases were computed using the formulae given by Waddington et al. (2004, p. 372f), namely:

For studies using parallel group or matching-based strategies Hedges’ g and its standard error SE_g were computed as

g = Y ¯ t − Y ¯ c S p * [ 1 − 3 4 * ( n t + n c − 2 ) − 1 ] S E g = [ n t + n c n c * n t + g 2 2 * ( n c + n t ) ] .

Where

Y ¯ t

Y ¯ c

Y ¯ t

Y ¯ c

S p = ( n c − 1 ) * s c 2 + ( n t − 1 ) * s t 2 n t + n c − 2

With S_t and S_c as the standard deviation in the treatment and comparison group respectively (Hedges’ approach). If either the comparison or treatment group's standard deviation was not reported, the standard deviation of the total sample S_T or the comparison group standard deviation was used to compute g . In the case of dichotomous outcome variables, the S_t and S_c were computed based on the number of observations and the proportion in the respective group, if available.

For partial effect sizes estimated using multivariate analysis, g and its standard error were estimated based on formula described in Keef and Roberts (2004):

g = β ^ σ ^

Where

β ∘

σ ∘

There are two approaches for the calculation of the pooled standard deviation from regression-based studies. In Hedges’ approach,

σ ∘

σ ∘

σ ^ = S T * ( n t − 1 ) − ( β 2 * ( n 2 * n t ) / ( n c + n t ) ) n t − 1

If information for calculating SE_g was not available, it was approximated by

S E g = g t

where t is the t-value associated with a t-test on the treatment effect of a regression.

If none of the values for S_p , S_T or S_c could be obtained from the report (or by contacting the authors), the standard deviation of the outcome variable was approximated using the formula from Borenstein, Cooper, Hedges and Valentine (2009)

S p = S E * n c * n t n c + n t

where SE is the standard error of a means test (e.g., regression coefficient). Since this formula is technically only correct for bivariate effect sizes, a sensitivity analysis was performed on the sample without these imputations.

For some studies, the review team transformed reported effect size statistics (often t, F, p or z-values) prior to calculating effect sizes following the procedures suggested in Lipsey and Wilson (2001).

Prior to synthesizing computed effect sizes, checks were made for outliers which could have been a result of erroneous coding or misleading assumption in the computation of SMD. In cases where SMDs or their standard errors seemed implausibly large, the original reports were revisited to check whether these were in accordance with the findings stated by the authors. In cases where the effect sizes where correctly coded and computed but still appeared implausible, the authors were contacted for clarification. As it was not possible to solve all outlier issues following this approach, the data from remaining outliers was censored, initially by winsorizing the data and then finally by dropping any remaining outliers. Winsorizing is a method of censoring data by limiting extreme values in the statistical data to reduce the effect of possibly spurious outliers. Winsorizing refers to a method where all outliers are set to a specific percentile. In this case, the top 1 per cent and bottom 1 per cent of observations were set to the value of the 1st and 99th percentile, respectively. The robustness of the results was tested with respect to the level of winsorizing and the cut-off ranges for trimming outliers.

3.4.3 Unit of analysis issues

Originally, it had been planned to correct the standard errors for a possible unit of analysis error by adjusting the standard errors () according to the formula suggested in Higgins and Green (2011, p. 502ff). A unit of analysis error typically arises if the study conducts analysis and programme placement at different levels and the analysis does not adequately account for this clustering (e.g., by using cluster robust standard errors or variance components analysis). In such cases, the analysis would yield narrower confidence intervals than the true confidence intervals, increasing the risk of Type-I error. This can be a problem in cluster randomized trials or in quasi-experimental studies in which treatment allocation is clustered. However, no studies were identified where there was a suspicion that the unit of analysis was not adequately addressed in the statistical analysis.

3.4.4 Dealing with missing data

In several instances, primary reports did not supply sufficient information to compute standardized effect sizes from reported treatment effect estimates. Most often, post-intervention mean and/or standard deviation of the outcome variable could not be obtained. The frequency with which missing information was encountered indicates that better reporting standards are required for impact evaluations studies.

As a first step, authors of included papers were contacted to provide missing information and to clarify discrepancies. This was an important and time-consuming measure, carried out via standardized letters and missing information forms, into which authors or research assistants could easily insert the results and data requested. ¹⁵

Initially, the review team reached out to the authors of 100 included reports (note that this number represents almost the entire sample of included reports in the systematic review), requesting additional information to facilitate the computation of the effect sizes or to achieve clarity on the quantitative results or interventions details. ¹⁶ In the event that an author did not reply, the same request was sent two more times. In total, the authors of 34 papers replied, while no response was received from the authors of 63 reports. In the remaining cases, no valid, up-to-date email addresses could be found for the authors.

In several instances, information was missing about, for instance, standard deviations, sample sizes or average outcomes in the comparison group follow-up data collection. In these cases, the missing data were imputed from available information based on specific assumptions. For instance, when the overall sample size was provided but not the sample sizes for the treatment and comparison groups separately, an assumption of equal sample sizes was made (splitting the overall sample size in half). The same assumption was applied in cases in which only the treatment or comparison group sample size was reported. Results from a meta-analysis were reported based on the more conservative sample (without imputing missing information) as part of the sensitivity analysis.

In cases where the information necessary to compute an effect size (e.g., sample size, mean outcomes and/or standard deviation) could not be derived from the available information, the effect size was excluded from the analysis. ¹⁷

3.4.5 Dealing with dependent effect sizes

In a meta-analysis, the unit of analysis is the study. Section 3.2.2 clarified that a single programme could include more than one intervention, which was regarded as the review's primary unit of interest (instead of the overall impact of one programme, the team was interested in the impact of each specific intervention). Each intervention may have been evaluated by more than one study (e.g., evaluation), each of which may have been published in multiple reports (e.g., working papers, technical reports or journal publications). Two reports were treated as part of the same study if they were based on the same data and hence could not be treated as independent, even if they were written by different authors. Therefore, an intervention population (all participants) might be different from the study population (all in one data set), which might itself differ from the sample population for a specific treatment effect estimate on a specific outcome construct.

Estimated treatment effects may be regarded as independent from each other when the underlying data were derived from different sample populations. To maintain the independence assumption, it was important that only one effect size per outcome construct and study was included in the analysis (Borenstein, Hedges, Higgins and Rothstein, 2009). However, each report might present different treatment effect estimates for the same outcome construct and the same sample population – for example for different sub-group analyses or employing different statistical methods. This implied that different estimates within each study (sometimes across reports) had to be combined into one effect size per sub-group.

Creating effect size aggregates and summary effect sizes (e.g., at the intervention or study level or across different sub-groups as part of the moderator analysis) required careful estimation to avoid the situation where a single group of participants influenced the summary effect size disproportionately. For example, a treatment effect might be reported in a study for the entire (pooled) sample and subsequently reported for sub-groups of the same sample, such as males and females. ¹⁸ The median number of treatment effect estimates per study in the sample was 12, with some reports providing more than 100 estimates. In such instances, a multitude of treatment effects could be reported for the same group where there was no a priori reason to give preference to one measure over another.

In these scenarios it was possible to mitigate the disproportionate influence on the aggregate effect sizes by applying the following steps. First, by identifying a set of effect sizes that were derived from the same independent group of participants and then, where applicable, selecting the effect sizes for this group where it was possible to establish a preference. (For example, keeping only pooled estimates and discarding sub-group estimates except when needed in the analysis.) By dropping some of the effect sizes derived from the sample this redundancy was removed from the analysis as far as possible. ¹⁹ This method provided a better approach to the data than averaging effect sizes across all overlapping sub-groups. ²⁰

Second, in cases where multiple effect sizes were reported for each independent group without clear justification for dropping some rather than others (e.g., the where same outcomes were reported at several points in time for the same group), the aggregate (“synthetic”) effect sizes were estimated for each independent group. Based on the method for combining effect sizes from the same independent population suggested by Borenstein, Hedges, Higgins and Rothstein (2009), the approach was as follows: Let g_ij and SE_ij be the i^th effect size, where i = (1,...,m), and its standard error, respectively, for the sample population identified by j. To arrive at a single combined (aggregate) effect size for group j the team took the simple average:

g j = 1 m ∑ i = 1 m g i j

and the standard error of g _i given by

S E j = ( 1 m ) 2 ( ∑ i = 1 m g i j 2 + ∑ i ≠ k ρ i k j σ i j σ k j )

where ρ_ijk is the correlation coefficient between g _ij and g _kj in study j. ²¹

Hence, the independent group aggregates were assembled at the relevant unit of analysis, such as at the intervention or study level (depending on the assumed correlation addressed in the procedure). Then the random-effects meta-analysis was applied to the aggregated data and estimated summary effect sizes.

3.4.6 Synthesis methods

Summary effect sizes are provided for the three outcome categories: (1) employment outcomes, (2) earnings outcomes and (3) business performance outcomes. The summary effect sizes were estimated via a random-effects meta-analysis based on the intervention-outcome level aggregates using the –metan– command in Stata. ²² Random-effects meta-analysis is recommended in settings which present significant contextual heterogeneity in terms of study population, intervention and implementation. To account for differences in individual studies’ sample sizes, effect sizes were averaged across studies by using inverse-variance weighting of the individual effect sizes. This weighting resulted in the individual effect sizes from studies with larger sample size being given more weight in the combined effect size. The summary effect sizes generated in this manner are presented alongside the 95 per cent confidence intervals in forest plots (Section 4.3). In addition to the aggregate effect size, these forest plots display the weight each intervention carries towards the summary effect size.

Heterogeneity tests were used to examine whether the variation in effect size estimates within outcome categories was larger than expected from sampling error alone (Deeks, Altman & Bradburn, 2001). To test for heterogeneity, the team employed I² statistics and Q-statistics. These statistics tested whether the percentage of variability in effect estimates was estimated due to heterogeneity rather than by chance. A significant Q (p-value <0.05) and an I² value of at least 50 per cent were considered to be indicators of heterogeneity.

3.4.7 Moderator analysis

Moderator analyses were performed when there was evidence of heterogeneity. The analyses tested hypotheses about whether variation in the (average) effect sizes reported in studies was associated with differences in study, participant and intervention characteristics (moderators). These moderator analyses also served as a test for correlations of effect size magnitude with specific characteristics of interventions and population groups. They therefore formed the basis for the answers to the research questions regarding factors of intervention effectiveness.

In a first step, a univariate approach was implemented, analogous to an analysis of variance (ANOVA) analysis, again via a random-effects meta-analysis based on the intervention-outcome level aggregates. Specifically, the review team investigated heterogeneity within outcome categories by (i) main intervention category, (ii) country income level, (iii) gender, (iv) participant income status and (v) time elapsed after programme completion. Results from these models are presented in the form of forest plots in Section 4.3.3.

Ideally, moderator analysis should be conducted with a minimum of ten studies for each individual moderator variable (Borenstein, Hedges, Higgins & Rothstein, 2009). A decision was made to present forest plots for sub-groups (e.g., intervention types) that had at least four individual interventions. The number of effect size estimates and individual interventions for each sub-group are displayed in the respective forest plots to provide the reader with an indication of the size of the evidence base.

A large array of study-level, intervention-level and contextual variables were identified and coded which it was assumed could be correlated with the reported effect size. The code description in Section 9.7 in the Appendix provides an overview of all variables that are included in the multivariate meta-regression. In addition to these, tests were conducted for the influence of various other moderator variables but the decision was made to exclude any that were deemed non-significant.

3.4.8 Supporting interpretation of effect sizes: The percentage change

In addition to the SMD, the review team computed the simple percentage change of the intervention over the control group mean as a more intuitive indicator of the intervention's impact. The percentage change was calculated by dividing the raw effect size (i.e., the mean difference between treatment and comparison group) by the mean value of the outcome variable for the comparison group. As a consequence, the percentage change indicates the direction of change for the treatment groups, with negative values meaning that the treatment group's outcome was lower than the comparison group's. This percentage change was then averaged by independent effect size group (i.e., by a grouped combination of intervention and study level). Subsequently, the team weighted the group-wise percentage changes using the inverse-variance weights (as throughout the analysis) and computed the final percentage changes. The computation of average percentage changes, however, comes with some caveats (cf. Lipsey et al. 2012) and in some cases the reported average percentage changes are at odds with the average SMDs computed across the same sample of (independent) effect sizes. For this reason, we do not emphasize the findings based on percentage changes but rather see it as a complement to reported SMDs, which are the main focus of our analysis.

3.4.9 Sensitivity analysis

A range of sensitivity checks were conducted to test the robustness of the results. Sensitivity analysis was carried out by restricting the meta-analysis to a subset of all studies included in the original meta-analysis. First, following guidance from the Campbell Collaboration (2014, p. 9), an examination was carried out to establish whether findings were influenced by the rigour of the evidence. Specifically, the team tested heterogeneity across study design (randomized vs. quasi-experimental) and publication status (published vs. unpublished studies). Second, the sensitivity of the results was tested with regard to the assumptions made for computing the SMD effect size in the presence of missing information. Third, the team tested the validity of the method of dealing with statistical outliers (dropping observations vs. winsorizing the data).

3.4.10 Risk of bias and study design assessment

During the research and coding process, the team found that impact studies often lacked important details that would allow a confident appraisal of the plausibility of the identifying assumptions on which the empirical analyses were based. This lack of detailed reporting in many publications limited the extent to which a full risk of bias assessment, for example, based on Waddington and Hombrados (2012), was possible. As a consequence, an alternative framework was adopted (proposed in Duvendack, Palmer-Jones, Copestake, Hooper, Loke & Rao (2011) and Duvendack, Hombrados, Palmer-Jones & Waddington (2012)) in order to assess the statistical rigour of primary studies. This approach combined an assessment of both research design and the method of statistical analysis. It did not incorporate detailed assessment of aspects of bias usually recommended in systematic reviews (see e.g. Higgins and Green, 2011) such as allocation method, confounding, selection bias (including attrition), performance bias, biases in outcomes data collection, and bias in analysis and reporting. In addition to the original approach, the assessment was further disaggregated by the statistical method (DiD, statistical matching, etc.) used for addressing potential confounders of the original research design (randomized experiment, natural experiment, etc.). By placing RCTs at one end of the spectrum and cross-section designs at the other, the tool aimed to reflect the potential capacity of different empirical identification strategies to control for possible confounding. ²³ In addition to the sensitivity analysis, the team therefore tested whether different research designs and empirical approaches yielded different effect sizes on average.

3.4.11 Assessment of reporting bias

Publication bias or “file drawer effects” refers to the underreporting of studies which establish a non-significant, negative or mixed evaluation finding (Franco, Malhotra & Simonovits, 2014). The review team assessed the danger of publication bias in the sample of included studies by several means. First, by testing the influence of study design and publication status as part of the sensitivity analysis. Second, by performing standard tests for publication bias: plotting the effect size against standard errors (funnel plots) using the –metafunnel– and –metacum– commands in Stata. Moreover, the team also implemented Egger, Davey Smith, Schneider and Minder's (1997) meta-regression test using the –metabias– command in Stata. The idea underlying the small-sample assessment to detect publication bias is that “researchers who have small samples and low precision will be forced to search more intensely across model specifications, data, and econometric techniques until they find larger estimates” hence “such considerations suggest that the magnitude of the reported estimate will depend on its standard error” (Doucouliagos & Stanley, 2012).

Tests were also made to establish whether there were observable differences in reported effect sizes between peer-reviewed and unpublished studies. For example, it was possible that estimates reported in journal articles might be more likely to be positive and significant (Stanley, 2013).

4.1.1 Search results and selection of studies

The primary and complementary search identified 32,117 records, based on a search of over 70 sources, including 12 specialized databases, 11 general databases, 35 websites, such as institutional and conference websites, five dissertation, thesis and grey literature databases, and nine other reviews and meta-analyses. The search in electronic databases was completed in February 2014. From November 2014 to January 2015, the review team contacted experts and authors of included studies, screened reference lists of included studies and conducted citation tracking in order to identify additional studies. The list of included sources as well as the search dates for each source used during the primary and complementary search process are presented in Section 9.6.1 in the Appendix. After removing duplicates in the reference management software EndNote, screening of 28,375 records by title and abstract was carried out by individual reviewers, applying the inclusion criteria of the screening questionnaire (see Section 9.6.2 of the Appendix). A total of 1,141 records were identified for full text screening.

In order to minimize bias, included and excluded results were cross-checked by a second researcher and discrepancies were resolved by both researchers. This systematic screening process led to the identification of 86 reports which were considered to be of adequate content and methodological rigour to inform the systematic review. The main reasons for excluding reports at full-text stage were the following criteria: study design, target group, intervention. In addition, several reports were excluded because a more recent or updated version of the same report was available, the report only focused on relative effects, the impact evaluation study was ongoing or the report did not examine any of the outcomes of interest considered in this review. Examples of excluded reports and the reasons for excluding them are presented in Section 7.2. ²⁴

After extracting data from the preliminary set of 86 included reports, the review team screened 6,782 additional records that were identified through reference lists and citation tracking of included studies, hand searching of key journals in which a large number of included studies were found and contacting authors and experts. This search process led to the selection of 27 additional reports. Overall, this comprehensive search and selection process identified 113 reports which were considered eligible for inclusion in this review. The search and screening process is illustrated in Figure 2.

OPEN IN VIEWER

Figure 2

Search results

4.1.2 Characteristics of included reports

The systematic screening process led to the identification of 113 reports that met all criteria for inclusion (Section 3.2). ²⁵ As shown in Table 8, panel A, more than half of the impact evaluations of youth employment interventions were conducted in high-income countries where there is an established practice of results measurement, particularly with regard to government employment measures. The large share of reports from high-income countries in this systematic review (65 out of 113 reports representing eleven of the 31 countries in the sample) is an important feature that justifiably suggests that some caution should be exercised when interpreting the results in global terms.

The number of reports assessing the impact of youth employment interventions has increased steadily over the past few years (panel B), with nearly half of the sample published after 2010 and 21 reports published in 2014 alone. ²⁶ Interestingly, this surge in evaluation has benefitted developing countries by providing a greater quantity of better quality evidence about what works to support youth in the labour market. There were 48 reports of interventions implemented in low- and middle-income countries, with a particular prevalence of impact evaluations in Latin America and the Caribbean.

The search process identified a variety of publications from the grey literature (panel C). Only around one-third of the reports come from peer-reviewed journals, with the remainder split between working papers, technical reports from implementing organizations and others, such as books or dissertations. Most of the reports published in 2014 were working papers, identified through the complementary search process.

While the review focused on counterfactual impact evaluations, the search process uncovered a large variety of different evaluation designs, namely experimental designs, natural experiments and quasi-experimental designs (as discussed in Section 3.2.5). In contrast to other systematic reviews, this review contained a significant share of randomized experiments (53 reports, as shown in Table 8, panel D). Many of the results from these randomized control trials (RCTs) have been published recently (66 per cent after 2010) and hence were not included in previous reviews. Figure 3 shows the recent surge in rigorous evidence. Prior to 2011 most RCTs in the sample were conducted in high-income countries (Figure 4), while the past five years have seen a remarkable increase in RCTs in developing countries. Most Not ably, in 2014, 12 out of 15 RCTs included in this review were from low- and middle-income countries; seven of them evaluating youth employment programmes in Africa (Box 7).

OPEN IN VIEWER

Figure 3

Total number of reports and reports relying on RCTs by year of publication

OPEN IN VIEWER

Figure 4

Total number of reports and reports from high-income countries (HICs) by year of publication

Quasi-experimental designs, such as panel and cross-sectional evaluations were the second most common study design (50 reports), frequently relying on propensity score matching (PSM) and difference-in-difference (DiD) for causal inference under unconfoundedness. Quasi-experimental designs have been more widely employed over the past decade, with approximately 40 reports published after 2004 in Latin America and the Caribbean and in OECD countries. Finally, the review included 11 natural experiments, all of which were implemented in high-income and upper middle-income countries between 2004 and 2014.

In relation to the evaluation features, 39 reports provided impact estimates at multiple time points. In addition, 71 reports measured changes in outcomes of interest over 12 months after treatment exposure (panel E). These longer term effects were estimated primarily across skills training interventions. Few studies provided a sub-group analysis in addition to the overall analysis (panel F). In particular, only half of the reports in the sample provided separate results for males and females (excluding those that evaluated gender-targeted programmes). Very few reports in the sample provided separate treatment effects for disadvantaged, low-income or low-educated youth.

Table 8 also provides an overview of the types of outcomes measured across the included reports (panel G). Three-quarters of the reports in the sample reported results for more than one type of outcome. Employment and earnings outcomes were extensively reported. Employment probability was by far the most commonly measured and reported outcome within the set of reports. More than 88 reports provided an estimate of the programme impact on employment probability. Another 35 reports estimated the effect of an intervention on hours worked.

Table 8 also displays the limited number of reports (ten out of 113) measuring changes in business performance outcomes. Nine of these related to RCTs. They were most commonly found among interventions aiming to promote entrepreneurship among young people.

4.1.3 Characteristics of evaluated interventions

As shown in Figure 2, the search process led to 113 reports that assessed impacts of 87 youth employment programmes. The review drew a key conceptual distinction between programmes, interventions and components (Section 3.2.2). Youth employment programmes can consist of one or more interventions. These are exclusive tracks offered to discrete samples of participants. For example, in the New Deal for Young People programme, implemented in the United Kingdom and described in Box 8, youth had to choose one of four different tracks, namely, (i) education or training; (ii) a job with a voluntary sector employer; (iii) a job on the environmental task force; or (iv) employment in a wage subsidy programme.

Interventions, on the other hand, have one or several components, which were classified as skills training, entrepreneurship promotion, employment services or subsidized employment measures.

Table 9 provides an overview of the 107 interventions in the review. Main category (panel A) refers to the interventions where it was possible to identify a primary component. In line with previous reviews (Kluve, 2010; Betcherman et al., 2007), skills training proved to be the most common type of main intervention category, followed by subsidized employment, entrepreneurship promotion and employment services.

There were six interventions for which no main category of intervention could be identified, and these were therefore classified as unspecified. Their components were bundled in such a way that made it impossible to identify one type of intervention as being predominant over the others. They were truly multi-dimensional in nature and formed part of the following programmes: Active labour market programme for disadvantaged youth in Germany (study by Ehlert, Kluve & Schaffner, 2012); the National Guard Youth Challenge Programme in the United States (study by Millenky, Bloom, Muller-Ravett & Broadus, 2011); the New Chance Programme in the United States (studies by Chang, Huston, Crosby & Gennetian, 2007 and Quint, Bos & Polit, 1997); the New Deal for Young People in the United Kingdom (studies by Blundell et al., 2004, De Giorgi, 2005 and Wilkinson, 2003); the Teenage Parent Demonstration in the United States (study by Maynard, Nicholson & Rangarajan, 1993); and the Youth Opportunity Grant Initiative in the United States (study by Jackson et al., 2007). Details of the New Deal for Young People in the United Kingdom are presented in Box 8.

While the remaining interventions had one main component to address the labour market constraints of youth, more than one-third extended the intervention's scope with one or more additional measures. As panel B shows, some 64 per cent of interventions in the review incorporated a skills training component; but almost half of these combined skills training with some other measure. The most common combination was skills training and employment services, observed in 27 interventions.

Entrepreneurship promotion interventions that focused on youth were comparatively scarce. Entrepreneurship-related components were only reported in 17 interventions, and these components often seemed to be delivered in a way that was disconnected from other active labour market measures. It is important to highlight that the results chain for entrepreneurship promotion (Table 2) already incorporates the delivery of training services in relation to entrepreneurial and business development and management skills, avoiding potential overlaps between skills training and entrepreneurship promotion categories. ²⁷

As discussed above, the majority of the reports included in this review assessed impacts of youth employment programmes implemented in high-income countries, which translated into a sample of 60 interventions (panel D). There were 56 interventions (52 per cent) from OECD countries alone (panel E), a proportion comparable to those seen in previous reviews (e.g., Card et al., 2010 and 2015; Betcherman et al., 2007).

The second largest share of impact evaluations stemmed from interventions in Latin America and the Caribbean, where many countries have experimented with active labour market policies (ALMPs) since the early 1990s – particularly through quasi-experimental designs embedded in the Jóvenes Programmes, a series of skills training interventions implemented throughout the region ²⁸ (see Box 2 for an example).

The review captured 17 interventions evaluated in Africa (15 in sub-Saharan Africa and two in North Africa, panel E), all of which were covered in recent impact evaluations published after 2011. In contrast, there was a relatively small number of evaluated interventions from other developing and emerging regions: Four in Europe and Central Asia, South Asia and the Middle East, respectively. There was no evidence from interventions implemented in East Asia and the Pacific.

With regard to scale (panel F), most interventions had a national coverage. In 30 cases the evaluations examined localized interventions implemented as pilots. The disaggregation across urban and rural areas demonstrated a significant lack of evidence about what works to support rural youth. The review's sample included only six evaluated interventions in rural areas, 33 in urban areas, and 62 interventions at a national scale with (imputed) coverage at both urban and rural level.

A close examination of programme targeting (panel G) led to the identification of 16 interventions (15 per cent) designed to serve only young women, 48 interventions (45 per cent) targeting youth who were unemployed prior to joining the intervention and 45 (42 per cent) that focused exclusively on low-income and disadvantaged youth.

Public and private sector actors were the most common implementing entities. Their implementing role was more prevalent among high-income countries, while evaluated interventions with an implementation role for non-governmental organizations (NGOs) and non-profit organizations tended to be more common in low-income countries.

Detailed descriptions of the intervention features and overall treatment effects are presented in the Appendix in Sections 9.1 to 9.5.

4.3.1 Descriptive analysis of effect size estimates

To synthesize the results of the 113 empirical reports of youth employment interventions, the review relied on the reported treatment effect as a measure of impact. The search and screening process led to the identification and coding of 3,629 treatment effects. Based on the reported (or acquired) information, it was possible to compute the direction and statistical significance for 3,105 treatment effect estimates. The computation of standardized mean difference (SMD) required further information (the minimum requirement being the number of observations in treatment and/or comparison groups). Even after using the methods of imputing missing information described above (Section 3.4.4), it was only possible to compute the SMD from 2,259 reported treatment effect estimates, as shown in the third column of Table 11.

It was possible to compute a substantially higher number of effect sizes due to efforts to acquire missing information from authors. There were 121 independent samples to account for dependencies within studies due to overlap of the study population across effect estimates, as described in Section 3.4.5 Dealing with dependent effect sizes.

4.3.2 Univariate random-effects meta-analysis

The following sections discuss results from the univariate meta-analysis approach to explore the differences in average effect size estimates across interventions in the sample. The analysis built on forest plots, which are commonly used to graphically describe the results of a meta-analysis. Forest plots are based on an inverse-variance weighted least squares random-effects meta-analysis model (see Box 9).

To improve the readability of this report, only “summary” forest plots are included in the main text. These provide the summary estimate for each sub-group in the respective analysis and, where appropriate, the respective overall summary SMD. ²⁹

Fifteen “disaggregated” forest plots, with study-level SMDs for each outcome category and main intervention category, are provided in the Appendix. ³⁰ Results presented in these forest plots were based on the sample using all imputations available and winsorizing the top 1 per cent of statistical outliers. Results from the restricted sample and/or obtained under different assumptions regarding outliers are presented as part of the sensitivity analysis in Section 3.4.9.

Synthesis of the overall evidence by outcome

Figure 6 and Table 12 present the overall summary effect sizes for each selected outcome category of interest – namely, employment outcomes, earnings or income outcomes and business performance outcomes. ³¹ The total sample size is calculated by aggregating the number of observations coded from individual studies, while avoiding double-counting of effect sizes measured for the same sample of participants. The aggregate sample sizes throughout the report are often strikingly large. The reason is a number of quasi-experimental impact evaluations based on administrative data. One example is the paper by Webb et al. (2014), who study a targeted employment subsidy using Canada's Labour Force Survey (LFS) and hence reach a sample size of more than 480,000 individuals. Note that an individual study may have contributed to multiple outcome categories and hence the individual sub-groups may not be independent (in other words, the same sample of participants may have provided an estimate for earnings and employment outcomes, in which case the two estimates are not independent. In addition, employment, earnings and business performance are different constructs. Consequently, an overall effect size is not reported.

OPEN IN VIEWER

Figure 6

Summary forest plot of all outcomes (full sample) by outcome category

Employment and earnings outcomes were the largest contributors to the overall meta-analysis: 105 of 119 independent studies estimated an employment outcome and 92 estimated an earnings outcome. ³² The overall effect on earnings outcomes across all intervention categories was 0.05 SMDs (CI = 0.03, 0.06; I² = 82 per cent; number of interventions = 92) and statistically significant at the 5 per cent level. The summary effect on employment outcomes was similar and also statistically significant (0.04 SMD; CI = 0.03, 0.06; I² = 64 per cent; number of interventions = 105). Only impact estimates from studies that measured business performance outcomes exhibited a relatively large confidence interval and the summary effect was not statistically significant (0.03 SMD; CI = -0.05, 0.12; I² = 49 per cent; number of interventions = 14).

At the same time, the plot also exposed high heterogeneity (represented by the I² statistic) within each outcome category, suggesting that a large share of the variation in effect sizes is explained by inter-study heterogeneity. Earnings outcomes displayed the highest I² value at 82 per cent, suggesting that more than three-quarters of the variation in the effect sizes is not by chance and rather due to heterogeneity between interventions.

In order to explore the factors driving such differences, the remainder of the report explored the effect sizes within each outcome category through moderator and sensitivity analyses. Since the number of independent studies that measured specific outcomes for certain moderators was small for some moderators, the review team only assessed those outcomes where at least four interventions were obtained.

4.3.3 Univariate moderator analysis

As a first step, the team tested whether summarizing effect sizes within the three outcome categories presented a viable procedure or whether significant heterogeneity was already detectable across outcome constructs in each outcome category. Following this, tests for heterogeneity were carried out by investigating the influence of several factors as part of the moderator analysis: (i) main intervention type; (ii) country income level; (iii) time after exposure to treatment; (iv) study-level summaries of participant characteristics, including gender and participant's income status; (v) programme characteristics, including scale of the programme and implementing organization. The moderator analyses generally provided results that were stratified by main category of intervention in order to avoid “comparing the incomparable”.

4.3.3.1 Outcome measure

To factor in the diverse nature of each outcome, the team assessed the effect size of each outcome measured separately by outcome category (see Figure 7, Figure 8 and The significantly smaller sample of effect sizes for business performance outcomes presented greater variability, with overall negative effects on profits (-0.02 SMD; CI = -0.09, 0.o5; I2 = 0 per cent; number of interventions = 7) and sales (-0.06 SMD; CI = -0.16, 0.04; I2 = 0 per cent; number of interventions = 5) and a large effect among capital and investment reported outcomes (0.15 SMD, CI = 0.05, 0.26; I2 = 0 per cent; number of interventions = 6).

OPEN IN VIEWER

Figure 7

Summary forest plot of employment outcomes (full sample) by outcome measure

Additional parameters are reported in Table 14 and Table 15.

While there was heterogeneity across the different outcome measures within each outcome category, this was not statistically significant based on the random-effects meta-analysis model within each outcome category (the 95 per cent confidence interval of the sub-group average included the overall mean represented by the dotted red line), except for the case of unemployment duration and capital and investment measures). Based on these results, the team was confident that it was viable to pool results across outcome measures in the subsequent analysis.

Figure 9). Employment probability represented the largest share of effect sizes and carried the greatest weight across employment measures, with an SMD of 0.06 (CI = 0.04, 0.08; I² = 71 per cent; number of interventions = 98) followed by hours worked, with an effect size of 0.03 SMD (CI = 0.00, 0.06; I² = 67 per cent; number of interventions = 35). See as well Table 13 for further details.

OPEN IN VIEWER

Figure 8

Summary forest plot of earnings outcomes (full sample) by outcome measure

OPEN IN VIEWER

Figure 9

Summary forest plot of business performance outcomes (full sample) by outcome measure

OPEN IN VIEWER

Table 13

Summary of results for measures within employment outcomes

Parameters of interest	Employment probability	Unemployment probability	Participation rate	Hours worked	Unemployment duration	Quality of employment
Standardized Means Difference I2	0.06	0.06	0.05	0.03	0.01	0.07
Standard errors of the effect size	0.01	0.05	0.03	0.01	0.02	0.03
95% confidence interval	0.04	-0.03	-0.02	0	-0.02	0.01
	0.08	0.15	0.11	0.06	0.05	0.13
I Squared	70.7	88.66	50.39	66.79	29.97	49.85
Number of SMDs	1029	50	18	137	30	48
Number of interventions	98	13	7	35	6	13
Sample size	2,015,670	958,542	495,992	469,482	172,543	56,715
Mean difference	2.73	0.04	0.05	-24.20	-1.60	0.01
Control outcome	0.31	0.23	0.51	299.27	21.11	1.14
Treatment outcome	0.31	0.23	0.57	289.76	16.92	1.09
Percentage change	0.1	-0.19	0.01	0.03	0.06	0.47

Within earnings-related outcomes, wages and reported earnings drove the overall effect size, with individual effect sizes of 0.03 SMD (CI = 0.02, 0.05; I² = 66 per cent; number of interventions = 36) and 0.06 SMD (CI = 0.03, 0.09; I² = 83 per cent; number of interventions = 66), respectively.

The significantly smaller sample of effect sizes for business performance outcomes presented greater variability, with overall negative effects on profits (-0.02 SMD; CI = -0.09, 0.o5; I2 = 0 per cent; number of interventions = 7) and sales (-0.06 SMD; CI = -0.16, 0.04; I2 = 0 per cent; number of interventions = 5) and a large effect among capital and investment reported outcomes (0.15 SMD, CI = 0.05, 0.26; I2 = 0 per cent; number of interventions = 6).

Additional parameters are reported in Table 14 and Table 15.

OPEN IN VIEWER

Table 14

Summary of results for measures within earnings outcomes

Parameters of interest	Earnings/Income	Consumption	Salary/wage
Standardized Means Difference I2	0.06	0.07	0.03
Standard errors of the effect size	0.01	0.04	0.01
95% confidence interval	0.03	-0.01	0.02
	0.09	0.14	0.05
I Squared	82.58	52.13	65.85
Number of SMDs	473	29	159
Number of interventions	66	10	36
Sample size	524,033	13,504	1,483,663
Mean difference	4,517.90	3,334.44	474.79
Control outcome	19159.10	10199.94	3,595.88
Treatment outcome	20740.19	13236.30	4,023.22
Percentage change	0.11	0.33	0.06

OPEN IN VIEWER

Table 15

Summary of results for measures within business performance outcomes

Parameters of interest	Profits	Sales	No. of employees/Jobs created	Capital and investment	Business creation
Standardized Means Difference I2	-0.02	-0.06	0.07	0.15	0.02
Standard errors of the effect size	0.04	0.05	0.06	0.05	0.06
95% confidence interval	-0.09	-0.16	-0.06	0.05	-0.11
	0.05	0.04	0.19	0.26	0.14
I Squared	0	0	43.51	0	45.94
Number of SMDs	59	36	35	26	9
Number of interventions	7	5	7	6	6
Sample size	4,191	2,252	3,449	3,487	3,035
Mean difference	-463.92	-141.49	75.54	268.39	0.00
Control outcome	917.73	1 021.32	142.91	1 115.14	0.25
Treatment outcome	356.31	920.79	203.94	1 198.96	0.25
Percentage change	-0.35	-0.18	0.06	0.65	-0.10##

While there was heterogeneity across the different outcome measures within each outcome category, this was not statistically significant based on the random-effects meta-analysis model within each outcome category (the 95 per cent confidence interval of the sub-group average included the overall mean represented by the dotted red line), except for the case of unemployment duration and capital and investment measures). Based on these results, the team was confident that it was viable to pool results across outcome measures in the subsequent analysis.

4.3.3.2 Main category of intervention

After restricting the analysis to cases where employment outcomes were reported (Figure 10), both entrepreneurship promotion (0.16 SMD; CI = 0.06, 0.26; I² = 71 per cent; number of interventions = 7) and skills training interventions (0.05 SMD; CI = 0.02, 0.07; I² = 65 per cent; number of interventions = 67) exposed larger than average effect sizes. It is important to note the evidence base behind this result. While there are 67 interventions supporting the magnitude of impact skills training measures, there are only seven related to entrepreneurship promotion. Detailed results are displayed in Table 16.

OPEN IN VIEWER

Figure 10

Summary forest plot of employment outcomes (full sample) by main category of intervention

OPEN IN VIEWER

Table 16

Summary of results for employment outcomes by main category of intervention

Parameters of interest	Skills training	Entrepreneurship promotion	Employment services	Subsidized employment	Unspecified
Standardized Means Difference I2	0.05	0.16	0.01	0.02	0.03
Standard errors of the effect size	0.01	0.05	0.01	0.02	0.04
95% confidence interval	0.02	0.06	-0.02	-0.01	-0.04
	0.07	0.26	0.04	0.06	0.10
I Squared	64.77	71.41	0.00	50	0.00
Number of SMDs	904	43	104	193	86
Number of interventions	67	7	10	16	5
Sample size	453,064	11,507	469,047	1,616,245	43,272
Mean difference	7.78	3.11	-1.09	0.51	-2.65
Control outcome	38.74	6.14	19.16	9.76	53.56
Treatment outcome	38.10	7.61	14.41	6.40	48.89
Percentage change	0.08	0.24	0.02	0.00	0.10

Interventions providing mainly employment services to youth were the least successful (0.01 SMD; CI = -0.02, 0.04; I² = 0 per cent; number of interventions = 10). In agreement with the descriptive analysis of interventions, interventions with skills training as the main category had the greatest weight within the overall employment-related effect size. In most cases, confidence intervals overlapped with the overall mean SMD, suggesting that there were no significant differences in average effect size across types of interventions. The I² tests, however, reported statistically significant heterogeneity within the sub-groups for skills training, entrepreneurship promotion and subsidized employment interventions.

There was no evidence of heterogeneity across cases where it was not possible to identify a main category of intervention (i.e., in the unspecified category). Such cases reported an SMD of 0.03 (C I= -0.04, 0.10; I² = 0%; number of interventions = 5) on employment outcomes. The category was dropped from the earnings outcome analysis due to insufficient sample size.

Qualitatively, the results of effect sizes from earnings- or income-related outcomes, across main intervention types, mimicked those from employment outcomes, though in this case subsidized employment interventions offered the lowest (and negative) effect size (-0.01 SMD; CI = -0.05, 0.03; I² = 61 per cent; number of interventions = 9).

The computed effect sizes (displayed in Figure 11) suggested that skills training (0.07 SMD; CI = 0.05, 0.08; I² = 86 per cent; number of interventions = 60) and entrepreneurship interventions (0.09 SMD; CI = 0.01, 0.18; I² = 64 per cent; number of interventions = 12) positively and consistently impacted both the employment and earnings prospects of young people, while evidence from other intervention types showed rather lower impacts on both outcome categories. However, significant heterogeneity was detected within all categories of intervention except for employment services (0.01 SMD; CI = 0.00, 0.02; I² = 0 per cent; number of interventions = 8). See Table 17 for further information on the results and parameters.

OPEN IN VIEWER

Figure 11

Summary forest plot of earnings outcomes (full sample) by main category of intervention

OPEN IN VIEWER

Table 17

Summary of results for earnings outcomes by main category of intervention

Parameters of interest	Skills training	Entrepreneurship promotion	Employment services	Subsidized employment	Unspecified
Standardized Means Difference I2	0.07	0.09	0.01	-0.01	0.06
95% confidence interval	0.05	0.01	0.00	-0.05	-0.03
	0.08	0.18	0.02	0.03	0.13
I Squared	85.7	63.81	0.00	61.24	51.9
Number of SMDs	495	50	36	57	32
Number of interventions	60	12	8	9	3
Sample size	232,334	11,909	43,522	1,315,479	Dropped from analysis
Mean difference	3,781.16	2,346.78	-16.53	-1524.22
Control outcome	15488.55	2,744.07	226.15	7,057.33
Treatment outcome	17274.87	3,639.95	186.38	5,535.96
Percentage change	0.11	0.22	0.00	-0.01

The summary forest plot for business performance outcomes (Figure 12 and Table 18) relies on a sample of 169 effect sizes, computed from treatment effects reported in 14 studies. Not ably, the impact of skills training interventions, which measured impacts on business performance outcomes (four cases), was negative with an average SMD of -0.09 (CI = -0.18, 0.01; I² = 0 per cent; number of interventions = 4). On the other hand, the magnitude of impact from entrepreneurship interventions on business outcomes was comparatively large (0.10 SMD; CI = 0.00, 0.19; I² = 39 per cent; number of interventions = 10) and just statistically significant at 10 per cent significance level.

OPEN IN VIEWER

Figure 12

Summary forest plot of business performance outcomes (full sample) by main category of intervention

OPEN IN VIEWER

Table 18

Summary of results for business performance outcomes by main category of intervention

Parameters of interest	Skills training	Entrepreneurship promotion
Standardized Means Difference I2	-0.09	0.1
95% confidence interval	-0.19	0
	0.01	0.19
I Squared	0.00	39.09
Number of SMDs	7	162
Number of interventions	4	10
Sample size	2,391	6,229
Mean difference		47.72
Control outcome		371.77
Treatment outcome		395.75
Percentage change		0.15

Notes: The table does not show report on all categories of intervention as related studies did not measure changes in business performance outcomes.

4.3.3.3 Country income level

This section explores differential impacts across country income levels. The analysis recognized (i) the differences in labour market barriers facing youth in the context of different country income levels (Robalino, Margolis, Rother, Newhouse & Lundberg, 2013); (ii) the role of context on the ability of youth employment interventions to shape labour market outcomes of youth (Betcherman et al., 2007); and (iii) the intrinsic and differentiated characteristics of labour markets and institutions across middle- and low-income countries in comparison to high-income countries (Fields, 2011; Cho, Margolis, Newhouse & Robalino, 2012).

The analysis capitalized on the sizable number of studies under each country income group. There were 65 and 48 reports of interventions implemented in high-income countries and low- and middle-income countries, respectively.

Interventions in high-income countries were typically national programmes, implemented and designed by government agencies. Evidence from local or pilot interventions was scarce (only 15 per cent of the total sample). In low- and middle-income countries, more than 40 per cent of the evidence was generated from small-scale local programmes. These programmes often targeted specific groups, such as young women. While only 5 per cent of interventions in high-income countries targeted young women, they were the focus of 27 per cent of the interventions evaluated in low-income countries.

Evaluated interventions also varied across country income levels. ³³ While, in high-income countries, evaluations of employment services, subsidized employment and skills training were common, only a negligible number of entrepreneurship promotion interventions were evaluated. In contrast, both entrepreneurship and skills training interventions were relatively frequently reported in countries outside the high-income economies, but there were few cases of evaluated interventions providing mainly employment services or subsidized employment interventions.

Research designs also varied across country income groups. A significant proportion (>50 per cent) of the recent evidence from middle- and low-income countries had been generated from relatively small-scale experimental evaluation designs. In contrast, quasi-experimental approaches using administrative data made up a large share (60 per cent) of the studies from high-income countries.

The review team also observed that many of the interventions in high-income countries were designed and implemented with the participation of government agencies. However, in some cases other stakeholders were involved, in particular the private sector (for example, in the form of private firms providing training or employment services).

Summary forest plots are provided for high-income and low- and middle-income countries. ³⁴ Effect sizes reported on both employment and earnings outcomes were generally higher among low- and middle-income countries (see Skills training interventions generated the highest magnitude of impacts in high-income countries, reportedly 0.04 SMD (CI = 0.01, 0.07; I2 = 68 per cent; number of interventions = 29) for employment (Figure 13, Table 19) and 0.02 SMD (CI = 0.00, 0.04; I2 = 72 per cent; number of interventions = 21) for earnings (Figure 15, Table 21) outcomes, respectively. In contrast, skills training interventions in low- and middle-income countries showed lower effect size (0.06 SMD; CI = 0.02, 0.10; I2 = 63 per cent; number of interventions = 38) among employment outcomes, compared to a 0.18 SMD (CI = 0.06, 0.29; I2 = 68 per cent; number of interventions = 5) for entrepreneurship promotion interventions.

OPEN IN VIEWER

Figure 13

Summary forest plot of employment outcomes by main category of intervention for high-income countries

While the effect size displayed wide variance, entrepreneurship promotion interventions offered positive prospects for stimulating the labour market outcomes of youth in the developing world. The limited number of effect sizes from entrepreneurship interventions in high-income countries caused the category to drop out of the analysis.

Evidence from subsidized employment interventions in low- and middle-income countries was rather limited but still presented positive impacts (on average) and non-heterogeneity within the sub-group for both outcome types. For high-income countries, the interpretation of the findings was less encouraging, as the studies reported no impact on employment and negative impacts on earnings.

Employment services interventions held an important position in terms of the effect of youth-targeted ALMPs in high-income countries. While their impact on employment was negligible, they tended to yield positive income gains among participating youth. The category dropped out of the low- and middle-income countries analysis due to its reduced sample size.

In conclusion, observing both outcome categories, the team ruled out statistically significant differences between intervention types for both high-income and low- and middle-income countries.

Figure 14, Table 20, Figure 16, and Table 22) with overall effect sizes of 0.08 SMD (CI = 0.04, 0.11; I2 = 64 per cent; number of interventions = 48) and 0.12 SMD (CI = 0.08, 0.15; I2 = 86 per cent; number of interventions = 53) for each outcome, respectively.

OPEN IN VIEWER

Figure 14

Summary forest plot of employment outcomes by main category of intervention for low- and middle-income countries

OPEN IN VIEWER

Figure 15

Summary forest plot of earnings outcomes by main category of intervention for high-income countries

OPEN IN VIEWER

Figure 16

Summary forest plot of earnings outcomes by main category of intervention for low- and middle-income countries

OPEN IN VIEWER

Table 19

Summary of results on employment outcomes across main categories of intervention in high-income countries

Parameters of interest	Skills training	Employment services	Subsidized employment	Unspecified
Standardized Means Difference I2	0.04	0.01	0	0.03
Standard errors of the effect size	0.02	0.02	0.02	0.04
95% confidence interval	0.01	-0.03	-0.03	-0.04
	0.07	0.04	0.04	0.1
I Squared	67.55	9.89	48.82	0
Number of SMDs	534	87	160	86
Number of interventions	29	7	11	5
Sample size	319,700	464,823	1,576,087	43,272
Mean difference	1.86	-1.57	0.54	-2.65
Control outcome	33.90	27.80	11.09	53.56
Treatment outcome	33.44	20.89	7.20	48.89
Percentage change	0.03	0.02	0.01	0.1

In comparison, reported effect sizes for high-income countries’ employment and earnings outcomes were 0.02 SMD (CI = 0.00, 0.04; I2 = 57 per cent; number of interventions = 52) and 0.01 SMD (CI = -0.01, 0.02; I2 = 70 per cent; number of interventions = 31), respectively. This suggested that active labour market measures had a greater impact among youth in developing countries compared to youth in advanced economies. The result coincided with Betcherman et al. (2007), which demonstrated that the probability that a programme has a positive impact on labour market outcomes declines as the country's income level rises.

Skills training interventions generated the highest magnitude of impacts in high-income countries, reportedly 0.04 SMD (CI = 0.01, 0.07; I2 = 68 per cent; number of interventions = 29) for employment (Figure 13, Table 19) and 0.02 SMD (CI = 0.00, 0.04; I2 = 72 per cent; number of interventions = 21) for earnings (Figure 15, Table 21) outcomes, respectively. In contrast, skills training interventions in low- and middle-income countries showed lower effect size (0.06 SMD; CI = 0.02, 0.10; I2 = 63 per cent; number of interventions = 38) among employment outcomes, compared to a 0.18 SMD (CI = 0.06, 0.29; I2 = 68 per cent; number of interventions = 5) for entrepreneurship promotion interventions.

While the effect size displayed wide variance, entrepreneurship promotion interventions offered positive prospects for stimulating the labour market outcomes of youth in the developing world. The limited number of effect sizes from entrepreneurship interventions in high-income countries caused the category to drop out of the analysis.

Evidence from subsidized employment interventions in low- and middle-income countries was rather limited but still presented positive impacts (on average) and non-heterogeneity within the sub-group for both outcome types. For high-income countries, the interpretation of the findings was less encouraging, as the studies reported no impact on employment and negative impacts on earnings.

Employment services interventions held an important position in terms of the effect of youth-targeted ALMPs in high-income countries. While their impact on employment was negligible, they tended to yield positive income gains among participating youth. The category dropped out of the low- and middle-income countries analysis due to its reduced sample size.

In conclusion, observing both outcome categories, the team ruled out statistically significant differences between intervention types for both high-income and low- and middle-income countries.

OPEN IN VIEWER

Table 20

Summary of results on employment outcomes across main categories of intervention in low- and middle-income countries

Parameters of interest	Skills training	Entrepreneurship promotion	Subsidized employment
Standardized Means Difference I2	0.06	0.18	0.11
Standard errors of the effect size	0.02	0.06	0.04
95% confidence interval	0.02	0.06	0.04
	0.1	0.29	0.18
I Squared	62.54	67.66	0
Number of SMDs	370	35	33
Number of interventions	38	5	5
Sample size	133,364	9,430	40,158
Mean difference	14.92	4.19	0.16
Control outcome	40.40	7.83	0.89
Treatment outcome	39.71	9.98	0.94
Percentage change	0.14	0.36	-0.04

OPEN IN VIEWER

Table 21

Summary of results on earnings outcomes across main categories of intervention in high-income countries

Parameters of interest	Skills training	Employment services	Subsidized employment
Standardized Means Difference I2	0.02	0.01	-0.02
Standard errors of the effect size	0.01	0.01	0.02
95% confidence interval	0	0	-0.07
	0.04	0.02	0.03
I Squared	72.32	0	79.91
Number of SMDs	303	32	46
Number of interventions	21	5	5
Sample size	109,379	40,007	1,309,712
Mean difference	4,800.83	-16.51	-2106.18
Control outcome	18537.80	215.34	9,585.32
Treatment outcome	20514.93	174.91	7,477.58
Percentage change	0.02	0	-0.03

OPEN IN VIEWER

Table 22

Summary of results on earnings outcomes across main categories of intervention in low- and middle-income countries

Parameters of interest	Skills training	Entrepreneurship promotion	Subsidized employment
Standardized Means Difference I2	0.12	0.14	0.02
Standard errors of the effect size	0.02	0.04	0.04
95% confidence interval	0.08	0.06	-0.05
	0.16	0.22	0.1
I Squared	88.62	49.45	0
Number of SMDs	192	39	11
Number of interventions	39	10	4
Sample size	122,955	10,363	5,767
Mean difference	726.10	3,013.40	16.09
Control outcome	5,520.71	3,286.44	551.43
Treatment outcome	6,198.67	4,447.00	588.67
Percentage change	0.18	0.3	0.08

4.3.3.4 Duration after treatment

Not all research studies reported information on the time lag between exposure to treatment and measurement of changes in outcomes. After imputations, only 72 per cent of the SMDs could be classified according to study timing after treatment into short- (data collected less than 12 months after the end of the treatment), medium- (12–24 months) and long-term studies (more than 24 months). Longer term outcomes were most common in evaluations from high-income country.

In the restricted sample, the overall effect size of employment and earnings outcomes was roughly the same (employment outcomes: 0.04 SMD; CI = 0.02, 0.06; I² = 66 per cent; number of interventions = 85; earnings outcomes: 0.05 SMD; CI = 0.04, 0.07; I² = 81 per cent; number of interventions = 79).

In both cases, short- and medium-term studies had a similar weight on the overall effect size of the entire meta-analysis (around 45 per cent and 35 per cent, respectively). While there was unaccounted heterogeneity within the different duration terms, it appeared that effect size estimates from longer term evaluations (>1 year) were relatively larger than shorter and medium term estimates in the case of studies measuring employment outcomes. As displayed in Figure 17 and Table 23, effect sizes for medium and long term were 0.05 SMD (CI = 0.03, 0.07; I² = 51 per cent; number of interventions = 43) and 0.06 SMD (CI = 0.02, 0.09; I² = 64 per cent; number of interventions = 21), respectively. This suggested a certain time lag before outcomes materialize.

OPEN IN VIEWER

Figure 17

Summary forest plot of employment outcomes by duration of period between individual exiting the intervention and data measurement (short, medium and long term)

OPEN IN VIEWER

Table 23

Summary of results on employment outcomes by duration

Parameters of interest	Short term (less than 12 months)	Medium term (12–24 months)	Long term (more than 24 months)
Standardized Means Difference I2	0.03	0.05	0.06
Standard errors of the effect size	0.01	0.01	0.02
95% confidence interval	0	0.03	0.02
	0.05	0.07	0.09
I Squared	71.98	50.81	64.08
Number of SMDs	348	377	180
Number of interventions	57	43	21
Sample size	748,302	736,124	594,572
Mean difference	3.39	6.42	3.58
Control outcome	18.36	25.97	112.76
Treatment outcome	15.97	25.09	109.56
Percentage change	0.07	0.1	0.11

Earnings outcomes showed a reversed pattern. Figure 18 (Table 24) shows impacts that decrease as duration between exposure to treatment and measurement increases. Medium- and long-term effect sizes were 0.06 SMD (CI = 0.03, 0.09; I² = 67 per cent; number of interventions = 38) and 0.05 SMD (CI = 0.02, 0.09; I² = 80 per cent; number of interventions = 20), respectively. While the effect size for short-term duration was 0.07 SMD (CI = 0.04, 0.1; I² = 84 per cent; number of interventions = 54).

OPEN IN VIEWER

Figure 18

Summary forest plot of earnings outcomes by duration of period between individual exiting the intervention and data measurement (short, medium and long term)

OPEN IN VIEWER

Table 24

Summary of results on earnings outcome by duration

Parameters of interest	Short term (less than 12 months)	Medium term (12–24 months)	Long term (more than 24 months)
Standardized Means Difference I2	0.07	0.06	0.05
Standard errors of the effect size	0.02	0.01	0.02
95% confidence interval	0.04	0.03	0.02
	0.1	0.09	0.09
I Squared	84.23	66.74	79.74
Number of SMDs	207	167	169
Number of interventions	54	38	20
Sample size	254,992	163,096	148,204
Mean difference	-654.38	7,715.05	1,359.53
Control outcome	4,361.13	25,542.80	9,785.48
Treatment outcome	3,530.38	29,645.58	11,207.11
Percentage change	0.08	0.12	0.2

At the same time, for both outcome types, the confidence intervals for each sub-group fell within the mean overall SMD and, hence, differences were not statistically significant. As the team suspected that other study-level characteristics might have confounded the analysis, this question was explored in more detail through multivariate meta-regression (Kluve et al., 2016).

4.3.3.5 Gender

The analysis by gender relied on whether an effect size was reported for female or male participants only. Pooled results (meaning those estimated on data that could not be disaggregated by gender) did not form part of this sub-group analysis.

OPEN IN VIEWER

Figure 19

Summary forest plot of employment outcomes (full sample) by gender

A large body of literature focuses on differences in the effectiveness of labour market interventions by gender. These differences were reflected in the interventions and studies in the sample. There were 39 interventions that reported male-only outcomes, of which more than half were located in high-income countries. Conversely, of the 54 interventions reporting outcomes separately for females, only 44 per cent were in high-income economies. More than one-quarter of these 54 interventions (15) specifically targeted only female participants (the vast majority of which (12) were located in low- and middle-income countries). In contrast, many other intervention characteristics were distributed relatively evenly between interventions that reported male-only and/or female-only estimates, in particular the main category of the intervention, age, education and income status of the target population, or programme implementers.

Summary forest plots showed greater effect sizes for young women compared to young men across employment and earnings outcomes. This suggested that interventions which specifically measured changes in outcomes by gender tended to have higher returns for women.

OPEN IN VIEWER

Table 25

Summary of results on employment outcomes reported by a specific gender or both

Parameters of interest	Male	Female	Both
Standardized Means Difference I2	0.06	0.08	0.04
Standard errors of the effect size	0.02	0.02	0.01
95% confidence interval	0.02	0.05	0.03
	0.1	0.12	0.06
I Squared	50.28	68.3	65.27
Number of SMDs	198	340	792
Number of interventions	28	42	84
Sample size	474,784	438,210	2,380,563
Mean difference	14.37	14.55	1.55
Control outcome	57.02	10.47	35.28
Treatment outcome	52.49	9.62	33.85
Percentage change	0.06	0.18	0.1

OPEN IN VIEWER

Figure 20

Summary forest plot of earnings outcomes (full sample) by gender

OPEN IN VIEWER

Table 26

Summary of results on earnings outcomes reported by a specific gender or both

Parameters of interest	Male	Female	Both
Standardized Means Difference I2	0.05	0.08	0.04
Standard errors of the effect size	0.02	0.02	0.01
95% confidence interval	0.02	0.05	0.02
	0.09	0.11	0.05
I Squared	30.38	77.51	83.76
Number of SMDs	134	190	346
Number of interventions	28	43	72
Sample size	308,752	315,493	1,470,935
Mean difference	2,157.98	1,368.85	2,162.41
Control outcome	16,718.45	8,816.04	10,010.51
Treatment outcome	18,855.68	11,016.96	10,828.26
Percentage change	0.07	0.2	0.07

4.3.3.6 Participant income status

This subsection looks at differential effects by participant sub-group, focusing on low-income/disadvantaged/at risk/vulnerable youth. Estimates for this sub-group (labelled “disadvantaged youth” for ease of reference) existed for almost half of the interventions (47 per cent) or more than half of the programmes (51 per cent), equally distributed across gender. The share of interventions containing separate estimates for the disadvantaged youth sub-group was considerably higher in low- and middle-income countries (57 per cent) than in high-income countries (38 per cent).

Figure 21 (Table 27) and Figure 22 (Table 28) display the results on employment and earnings outcomes respectively by participant income status. It appears that interventions had a greater impact on earnings outcomes of disadvantaged youth (0.12 SMD; CI = 0.08, 0.16; I² = 81 per cent; number of interventions = 53) compared to non-disadvantaged youth (0.02 SMD; CI = 0, 0.03; I² = 73 per cent; number of interventions = 37). Results on earnings for disadvantaged youth were larger than results across employment outcomes in the same target group (0.05 SMD; CI = 0.02, 0.08; I² = 66 per cent; number of interventions = 57).

Across intervention types, entrepreneurship promotion increased employment and earnings impacts among disadvantaged youth. Computed SMDs were large in magnitude and variance. Total overall effect size was larger for earnings outcomes (0.13 SMD; CI = 0.09, 0.18; I² = 82 per cent; number of interventions = 48) than for employment outcomes (0.06 SMD; CI = 0.02, 0.09; I² = 66 per cent; number of interventions = 54). These results contrasted quite sharply with the analyses of interventions which did not specifically target disadvantaged youth. For that sub-group, overall effect sizes were substantially smaller (employment outcomes: 0.03 SMD; CI = 0.01, 0.06; I² = 56 per cent; number of interventions = 40; earnings outcomes: 0.02 SMD; CI = 0.00, 0.03; I² = 73 per cent; number of interventions = 37) with skills training yielding the highest effects.

OPEN IN VIEWER

Figure 21

Summary forest plot of employment outcomes by participant income group (where yes is low-income, disadvantaged, at risk or vulnerable youth)

OPEN IN VIEWER

Table 27

Summary of results on employment outcomes by participant income group (where yes is low-income, disadvantaged, at risk or vulnerable youth)

Parameters of interest	No	Yes
Standardized Means Difference I2	0.04	0.05
Standard errors of the effect size	0.01	0.01
95% confidence interval	0.02	0.02
	0.07	0.08
I Squared	60.52	66.08
Number of SMDs	471	850
Number of interventions	44	57
Sample size	2,155,232	300,814
Mean difference	0.93	9.30
Control outcome	39.70	25.38
Treatment outcome	36.82	25.04
Percentage change	0.01	0.13

OPEN IN VIEWER

Figure 22

Summary forest plot of earnings outcomes by participant income group (where yes is low-income, disadvantaged, at risk or vulnerable youth)

OPEN IN VIEWER

Table 28

Summary of results on earnings outcomes by participant income group (where yes is low-income, disadvantaged, at risk or vulnerable youth)

Parameters of interest	No	Yes
Standardized Means Difference I2	0.02	0.12
Standard errors of the effect size	0.01	0.02
95% confidence interval	0	0.08
	0.03	0.16
I Squared	73.35	80.91
Number of SMDs	163	502
Number of interventions	37	53
Sample size	1,400,587	208,101
Mean difference	-577.28	9,001.62
Control outcome	3,138.52	27,440.73
Treatment outcome	2,477.55	31,707.80
Percentage change	0.07	0.16

OPEN IN VIEWER

Figure 23

Summary forest plot of employment outcomes by main category of intervention for low-income and disadvantaged participants

OPEN IN VIEWER

Figure 24

Summary forest plot of employment outcomes by main category of intervention for non-low-income/non-disadvantaged participants

OPEN IN VIEWER

Figure 25

Summary forest plot of earnings outcomes by main category of intervention for low-income/disadvantaged participants

OPEN IN VIEWER

Figure 26

Summary forest plot of income outcomes by main category of intervention for non-low-income/non-disadvantaged participants

OPEN IN VIEWER

Table 29

Summary of results on employment outcomes by participant income group (low-income participants includes also disadvantaged, at risk or vulnerable youth)

Parameters of interest	Low-income participants	Non-low-income participants
	Skills training	Skills training	Employment services	Subsidized employment
Standardized Means Difference I2	0.04	0.05	0.01	0.03
Standard errors of the effect size	0.02	0.02	0.02	0.02
95% confidence interval	0.01	0	-0.02	-0.01
	0.08	0.09	0.04	0.08
I Squared	64.38	66.87	0	50.54
Number of SMDs	743	152	102	174
Number of interventions	46	17	9	14
Sample size	130,157	185,818	466,578	1,461,817
Mean difference	11.49	2.24	-1.60	0.69
Control outcome	27.83	66.40	28.14	12.08
Treatment outcome	27.54	64.96	21.14	7.92
Percentage change	0.11	0.01	0.02	-0.01

Notes: Entrepreneurship promotion, employment services, subsidized employment and unspecified categories were dropped from the analysis for the group of low-income participants due to the small number of independent studies. Entrepreneurship promotion and unspecified categories were dropped from the analysis for the group of non-low-income participants due to the small number of independent studies.

OPEN IN VIEWER

Table 30

Summary of results on earnings outcomes by participant income group (low-income participants includes also disadvantaged, at risk or vulnerable youth)

Parameters of interest	Low-income participants		Non-low-income participants
	Skills training	Entrepreneur-ship promotion	Skills training	Entrepreneur-ship promotion	Employment services	Subsidized employment
Standardized Means Difference I2	0.13	0.18	0.03	0.01	0.01	-0.01
Standard errors of the effect size	0.03	0.06	0.01	0.05	0.01	0.02
95% confidence interval	0.08	0.05	0.01	-0.08	0	-0.06
	0.17	0.3	0.06	0.11	0.02	0.03
I Squared	82.04	71.08	84.62	23.86	0	66.08
Number of SMDs	431	14	59	36	35	33
Number of interventions	43	5	15	7	7	8
Sample size	86,072	6,548	145,166	5,361	40,968	1,209,092
Mean difference	11,202.67	4,535.91	-515.53	-1.00	-16.42	-1,898.11
Control outcome	34,760.16	5,106.97	2,868.19	162.03	215.01	8,792.60
Treatment outcome	40,082.61	6,863.33	2,245.09	145.55	174.83	6,899.06
Percentage change	0.16	0.23	0.1	0.26	0	-0.01

Notes: Employment services, subsidized employment and unspecified categories were dropped from the analysis for the group of low-income participants due to the small number of independent studies. The unspecified category was dropped from the analysis for the group of non-low-income participants due to the small number of independent studies.

4.3.3.7 Programme characteristics

This section analyses the presence of effect size heterogeneity across studies that evaluated programmes on a different scale or implemented by different actors. In the sample these two characteristics did not differ for interventions within the same programme, they were therefore referred to as programme-level characteristics.

The review team coded the scale of the programme using four categories, which generally referred to the level on which the programme was implemented, namely: 1

National level, which comprised programmes that were implemented across several regions in a country.

2

Regional level, referring to programmes that had clear geographical targeting on selected administrative regions.

3

Local level, when multiple areas in the entire country were selected (e.g., cities).

4

Pilot level, capturing programmes that were implemented as a trial, with relative low scope and the expectation of future scale-up.

Note that the variable was coded on the intervention rather than the study (sample) level: That is, the classification did not reflect whether the evaluation was conducted for a sub-sample of the entire programme, but rather the main objective was to test the difference between (small-scale) local or pilot programmes and national-level policies.

Results are presented in Figure 27 (Table 31) and Figure 28 (Table 32). Studies of national-level programmes generally reported somewhat smaller effect sizes for both earnings (0.03 SMD; CI = 0.02, 0.05; I² = 76 per cent; number of interventions = 47) and employment outcomes (0.03 SMD; CI = 0.01, 0.05; I² = 59 per cent; number of interventions = 55). However, the difference in terms of smaller scale programmes is not statistically significant. In addition, there was large unexplained heterogeneity within all sub-groups except the sample of pilot programmes.

OPEN IN VIEWER

Figure 27

Summary forest plot of employment outcomes by scale of the programme

OPEN IN VIEWER

Table 31

Summary of results on employment outcomes by scale of the programme

Parameters of interest	National	Regional	Local	Pilot
Standardized Means Difference I2	0.03	0.06	0.05	0.08
Standard errors of the effect size	0.01	0.02	0.03	0.03
95% confidence interval	0.01	0.02	-0.01	0.03
	0.05	0.11	0.11	0.14
I Squared	58.69	69.02	73.93	0
Number of SMDs	853	130	135	175
Number of interventions	55	17	24	7
Sample size	2,445,474	84,483	52,583	8,787
Mean difference	2.13	3.35	11.70	13.72
Control outcome	47.60	18.54	1.62	23.10
Treatment outcome	44.46	19.15	2.45	35.67
Percentage change	0.04	0.08	0.14	0.33

OPEN IN VIEWER

Figure 28

Summary forest plot of earnings outcomes by scale of the programme

OPEN IN VIEWER

Table 32

Summary of results on earnings outcomes by scale of the programme

Parameters of interest	National	Regional	Local	Pilot
Standardized Means Difference I2	0.03	0.17	0.07	0.05
Standard errors of the effect size	0.01	0.05	0.04	0.03
95% confidence interval	0.02	0.07	0	0
	0.05	0.26	0.15	0.1
I Squared	76.15	88.6	72.97	0
Number of SMDs	417	108	32	99
Number of interventions	47	21	16	6
Sample size	1,516,199	61,271	23,209	8,433
Mean difference	2,462.12	117.42	961.50	4,248.82
Control outcome	11,814.07	835.94	11,762.48	8,349.14
Treatment outcome	12,725.79	863.83	12,794.23	12,254.15
Percentage change	0.08	0.11	0.19	0.32

Figure 29 (and Table 33) and Figure 30 (and Table 34) provide summary SMDs for studies that analysed programmes implemented by different agencies. Implementers were categorized into public institutions, i.e., governments or multilateral organizations, and private entities, which were private sector firms or NGOs. In the analysis, the review team looked at the differential impact of programmes implemented by (i) governments and/or multilaterals, (ii) private sector firms and/or NGOs, or (iii) a combination of public and private sector (i.e., governments and/or multilaterals combined with private sector firms and/or NGOs). Any programmes that were not classified according to these three groups were called “other”; for example, when the implementing agency remained unknown to the reviewers.

OPEN IN VIEWER

Figure 29

Summary forest plot of employment outcomes by implementer

The team found that, for employment and earnings outcomes, a combination of public and private sector implementation led to the highest SMDs of around 0.06 (CI = 0.03, 0.08; I² = 48 per cent; number of interventions = 59) (employment) or 0.07 (CI = 0.05, 0.09; I² = 81 per cent; number of interventions = 57) (income) and significantly different from zero.

Private sector only implemented programmes (i.e., implemented by private sector firms and/or NGOs) led to moderate gains for both employment and income of around 0.04 SMDs (CI = -0.01, 0.10; I² = 80 per cent; number of interventions = 23) (employment) or 0.05 (CI = 0.00, 0.10; I² = 75 per cent; number of interventions = 21) (earnings) and with the summary SMD barely reaching significance at the 5 per cent level.

OPEN IN VIEWER

Table 33

Summary of results on employment outcomes by implementer

Parameters of interest	Government/Multilateral only	Private sector/NGO & Government/ Multilateral	Private sector/ NGO only
Standardized Means Difference I2	0.02	0.06	0.04
Standard errors of the effect size	0.02	0.01	0.03
95% confidence interval	-0.01	0.03	-0.01
	0.05	0.08	0.1
I Squared	63.32	48.38	79.82
Number of SMDs	225	825	265
Number of interventions	21	59	23
Sample size	1,294,748	1,241,524	54,691
Mean difference	2.39	7.35	0.96
Control outcome	19.69	54.38	1.67
Treatment outcome	16.76	52.65	2.09
Percentage change	0.02	0.06	0.15

OPEN IN VIEWER

Figure 30

Summary forest plot of earnings outcomes by implementer

The summary SMD of studies of public sector only implemented programmes (i.e., government and/or multilateral agency as implementers) was statistically insignificant for both employment and earnings outcomes.

However, as it was possible that the analysis could have been confounded with other intervention- or study-level characteristics that were correlated with programme scale (e.g., country income level), this difference was explored in more detail as part of the multivariate meta-analysis depicted in Kluve et al. (2016).

OPEN IN VIEWER

Table 34

Summary of results on earnings outcomes by implementer

Parameters of interest	Government/Multilateral only	Private Sector/NGO & Government/ Multilateral	Private Sector/ NGO only
Standardized Means Difference I2	-0.01	0.07	0.05
Standard errors of the effect size	0.03	0.01	0.03
95% confidence interval	-0.07	0.05	0
	0.05	0.09	0.1
I Squared	80.54	81.26	74.56
Number of SMDs	86	427	155
Number of interventions	12	57	21
Sample size	490,967	1,052,694	64,849
Mean difference	-1,526.65	3,859.60	1,296.64
Control outcome	7,180.16	15,993.60	3,242.02
Treatment outcome	5,285.78	17,770.68	4,003.11
Percentage change	0.06	0.09	0.16

4.3.4 Sensitivity analysis

In this section, the robustness of the results are tested. For the sake of brevity, this section discusses the sensitivity of the results from the overall synthesis of the evidence (pooled sample) and the moderator analysis by main intervention category. Hence, the robustness of each moderator analysis is not discussed, as these generally reflected findings in the pooled sample.

The following section focuses on three types of decisions which may have affected the overall results: First, different assumptions in computing (or imputing) effect sizes are tested. Second, the robustness of some of the decisions in the data synthesis (e.g., regarding outliers) is checked. Third, the question of whether the variance in effect sizes might be caused by factors related to the applied evaluation design (i.e., study type, risk of bias) is investigated.

For the univariate analysis, the respective summary forest plots are again included in the main text while, for this section, forest plots showing each intervention SMD separately have not been appended. ³⁵

In addition to the sensitivity analysis discussed in this section, the review team performed various other checks on the analysis (e.g., using Cohen's d instead of Hedges’ g; testing for differences between Intention-to-Treat (ITT) and Average Treatment Effect on the Treated (ATT) estimates; additional methods of data imputation). Since none of these checks significantly altered the main results, they are not discussed in detail.

4.3.4.1 Imputation of missing information

As discussed in Section 3.4.4, in some cases it was necessary to impute information and make assumptions in order to compute SMDs for specific studies. •

First, the review team had to make certain assumptions regarding the sample size of the treatment and/or comparison group if either of these was not reported.

•

Second, the team approximated SMDs using the formula provided by Borenstein, Cooper, Hedges and Valentine (2009) to approximate SMDs where information on the pooled standard deviation could not be obtained otherwise.

This section compares results using the entire sample of studies (including all imputed values) with a restricted sample excluding all studies where SMDs (or their standard error) could not be computed without these assumptions. ³⁶

Figure 31 replicates the forest plot displayed in Figure 6, displaying the summary SMDs by outcome category. Not imputing any missing information reduced the overall sample from 2,169 SMDs and 119 studies by almost half, to 1,116 SMDs (82 studies). The average SMD for employment outcomes increased, though the increase was not statistically significant. At the same time, the summary effect size (i.e., SMDs) for earnings outcomes was significantly reduced to 0.01; leading to an (insignificant) reduction in the overall SMD of youth employment interventions across outcomes.

Figure 32, Figure 33 and Despite reducing the sample size significantly, the basic results regarding the effectiveness of different intervention types held in the smaller sample. In fact, results and average effect sizes for employment and business performance outcomes were very similar to the main results that included all imputed values. Only in the case of earnings/income outcomes, was the average impact of skills training significantly reduced (the confidence intervals for skills training in the upper and lower panel did not overlap). Also, the precision of the estimate for entrepreneurship promotion intervention was somewhat reduced (i.e., had a larger confidence interval).

Figure 34 replicate the forest plots in the moderator analysis by main intervention category (Section 4.3.3.2) in the limited non-imputation sample. The sample size dropped from 1,312 effect sizes (105 studies) to 682 effect sizes (69 studies) for employment outcomes and from 661 effect sizes (90 studies) to 279 effect sizes (4 studies) for earnings outcomes. The number of business performance outcomes was reduced to 153 effect sizes (11 studies).

OPEN IN VIEWER

Figure 31

Forest plot of all outcomes by outcome category without imputations

OPEN IN VIEWER

Figure 32

Summary forest plot of employment outcomes by main category of intervention without imputations

OPEN IN VIEWER

Figure 33

Summary forest plot of earnings outcomes by main category of intervention without imputations

Despite reducing the sample size significantly, the basic results regarding the effectiveness of different intervention types held in the smaller sample. In fact, results and average effect sizes for employment and business performance outcomes were very similar to the main results that included all imputed values. Only in the case of earnings/income outcomes, was the average impact of skills training significantly reduced (the confidence intervals for skills training in the upper and lower panel did not overlap). Also, the precision of the estimate for entrepreneurship promotion intervention was somewhat reduced (i.e., had a larger confidence interval).

OPEN IN VIEWER

Figure 34

Summary forest plot of business performance outcomes by main category of intervention without imputations

4.3.4.2 Assumptions in the analysis

In the main meta-analysis, the team applied a procedure to remove implausibly large or influential estimates (cf. Section 3.4.2). Although meta-analysts would not want to erroneously exclude relevant estimates, balance demands that no single estimate, especially one among hundreds, should determine how an entire research literature is viewed or understood (Stanley & Doucouliagos, 2015).

As described in the respective sections, the team first winsorized the highest and lowest 1 per cent of coded SMD effect sizes estimates. Generally, this affected the (unweighted) mean SMD and its standard deviation only marginally. Subsequently any observations with an SMD or an SMD standard error of more than 0.75 were dropped. In the full sample, roughly 30 SMDs from 4 studies were excluded; most of these stemming from sub-group analysis and therefore all but one study was retained in the sample. This section tests whether the results are robust against these assumptions. The full results for all tests in the report are not displayed but those that appeared to be of major importance are highlighted. For example, the team also tested whether winsorizing at the 5 per cent level (instead of 1 per cent) altered the results, but could not find any definitive evidence and therefore this issue is not discussed.

The upper panel of Figure 35, Figure 36 and Figure 37 again replicates the results of the main moderator analysis regarding the main category of intervention but this time without winsorizing the data and only dropping outliers with an SMD or standard error above three (effectively not dropping any observations).

In fact, since very few observations were dropped, the results changed marginally and even confidence intervals did not increase as much as one might have expected. Correspondingly, the team found that results from the meta-regression model were not affected by the decisions to censor the specified data and so this robustness check is not discussed further in the current report.

Tests were also conducted on whether the results were robust to the level of aggregation of effect sizes before synthesizing results based on the random-effects meta-analysis. Specifically, the team checked whether aggregating effect size across all studies of one intervention, rather than only aggregating studies using the same data set, made a difference. The latter increased the number of observations in the full analysis (based on all outcome variables) from 100 interventions to 120 individual studies. Similarly, the reviewers tested whether the level of cluster in the meta-regression model significantly affected results but found no evidence that the level of first-step aggregation or cluster significantly altered results.

OPEN IN VIEWER

Figure 35

Robustness check: Summary forest plot of employment outcomes by main category of intervention, including outliers

OPEN IN VIEWER

Figure 36

Robustness check: Summary forest plot of earnings outcomes by main category of intervention, including outliers

OPEN IN VIEWER

Figure 37

Robustness check: Summary forest plot of business performance outcomes by main category of intervention, including outliers

In summary, the team tested the robustness of the results towards different decisions made in the process of compiling and analysing the data, such as the imputation of missing information or the handling of statistical outliers. Since all the sensitivity analysis yielded similar results to the main analysis, the team can be quite confident that the findings reported in sections 4.3.2 to 4.3.4 were not influenced by the method of analysis.

4.3.4.3 Research design

This section tests whether the results depended on the applied evaluation design. In the combined meta-analysis, studies of randomized control trials and quasi-experimental evaluation approaches were pooled but more rigorous evaluation designs might have systematically yielded different effect sizes than less robust evaluation designs.

Figure 38 and Figure 39 provide a moderator analysis by research design for both employment and earnings outcomes. In contrast to expectations, experimental studies actually produced larger effect sizes in both cases, but the difference between experimental and quasi-experimental was not statistically significant at the 5 per cent level in either case. Based on the one-way random effects ANOVA model, the team was able to rule out a systematic difference on average between effect sizes generated from experimental and quasi-experimental studies.

OPEN IN VIEWER

Figure 38

Summary forest plot of employment outcomes by research design

OPEN IN VIEWER

Figure 39

Summary forest plot of earnings outcomes by research design

OPEN IN VIEWER

Figure 40

Summary forest plot of employment outcomes by main category of intervention for experiments

OPEN IN VIEWER

Figure 41

Summary forest plot of employment outcomes by main category of intervention for quasi-experiments

OPEN IN VIEWER

Figure 42

Summary forest plot of earnings outcomes by main category of intervention for experiments

OPEN IN VIEWER

Figure 43

Summary forest plot of income outcomes by main category of intervention for quasi-experiments

However, this may differ according to the category of intervention. It is, for example, plausible that a particular intervention type consistently displays higher results when evaluated through experimental studies than through quasi-experimental ones. Therefore, the team tested whether the basic results on the effectiveness of different intervention types held when considering only evidence from experimental studies, which was arguably more reliable than quasi-experimental results. In a first descriptive step, Figure 40, Figure 41, Figure 42 and Figure 43 replicate the moderator analysis by main category of intervention for employment and earnings outcomes separately for experimental and quasi-experimental studies.

The evidence for entrepreneurship interventions was entirely based on empirical research and, hence, the review's results pertain. In contrast, all studies of subsidized employment interventions were derived from quasi-experimental approaches. To some degree, this correlation between intervention type and research design may have confounded the analysis. The spectrum of research designs employed for evaluating skills training interventions was more mixed. But regardless of evaluation design, skills training interventions appear to have been the most successful intervention type, along with entrepreneurship programmes. (The difference in SMDs between entrepreneurship and skills training interventions was still not statistically significant.) Unfortunately, studies of interventions classified as unspecified were dropped from the analysis, since fewer than four interventions were found which could have been classed as falling within either sub-group (experimental vs. quasi-experimental).

As with all univariate analysis, one issue was that the difference between experimental and quasi-experimental studies observed in forest plots might also have been driven by other factors (such as the fact that the majority of experiments were conducted in low- and middle-income countries, generally yielding larger effect sizes). Based on the univariate analysis, it is not possible to state with certainty that the aggregate effect size was actually downward biased by including evidence from quasi-experimental studies.

The meta-analysis results appeared robust to the type of evaluation design and the main findings were corroborated by rigorous evidence from experimental studies.

4.3.5 Analysis of small-sample bias and publication bias

This section uses funnel plots and Egger's tests to check whether there was any indication of publication bias in the sample of studies. Figure 44 and Figure 45 present funnel plots for the entire sample (including all outcomes and all sub-groups). The figure displays plots of the effect size (SMD) on the horizontal axis and the standard error of the effect size (SE SMD) on the vertical axis. In Figure 44, effect sizes are aggregated at the study level, and each dot represents an individual study. In Figure 45, the data is entirely disaggregated, meaning that each dot represents one effect size estimate. The solid line crosses the horizontal axis at the overall average fixed effect estimate.

OPEN IN VIEWER

Figure 44

Funnel plot of all outcomes and sub-groups, aggregated at study level

OPEN IN VIEWER

Figure 45

Funnel plot of all outcomes and sub-groups, disaggregated (on effect size estimate level)

Although most of the dots (studies) are spread around the solid line and within the triangular area (indicating the 95 per cent confidence interval), a degree of tendency towards the right is observable. These represent studies that reported positive effects (with a medium level of precision, as measured by the standard error). This slight asymmetry may be an indicator of publication bias.

Results presented in Table 35 from Egger's test for publication bias confirmed the visual indication: The coefficient of the variable bias was positive and statistically significant at the 5 per cent level.

As in the previous sections, the analysis is further disaggregated by outcome categories. Figure 61, Figure 62 and Figure 63 in Section 10.2 of the Appendix show funnel plots (aggregated at the study level) for the three outcome variables separately. ³⁷ As can be seen from these figures, effect sizes of earnings and income outcomes appeared most strongly skewed to the right. This was also confirmed when performing Egger's test for each outcome category separately. For business performance outcomes, Egger's test was not significant at the 5 per cent level.

This potential publication bias was accounted for in the multivariate meta-regression model using the procedure described in Doucouliagos and Stanley's study in 2009: The authors argued that including the standard error of the SMD in the random-effect model would account for the potential effect of publication bias and the resulting coefficient estimate would provide an indication of the magnitude (and significance) of the effect. Following this approach, the team found a clear indication of selection for statistically positive results. The point estimate was consistently positive and statistically significant. In addition, the summary effect estimate (represented by the constant) in the model, which pools all outcomes, turned non-significant when accounting for publication bias using this approach. This seemed to be largely driven by the negative (insignificant) results on business performance outcomes, while the summary effect on employment and earnings was still significant even accounting for publication bias.

In addition to the above test for publication bias, the team also tested whether reported effect sizes differed between peer-reviewed articles, working papers (some of which were unpublished at the time of publication search), evaluation reports/technical reports and other types of reports (such as books and dissertations). No statistically significant differences in average effect sizes by publication status were found, as can be seen by the forest plot shown in Figure 46. These results held when the analysis was disaggregated by intervention type or outcome category (not reported). Similarly, the dummy for publication status (peer-reviewed) in the multivariate results did not provide a clear picture.

Funnel plots and Egger's test indicated some publication bias towards studies showing positive effects of youth employment interventions on labour market outcomes. Using the procedure proposed in Doucouliagos and Stanley (2009), the reviewers accounted for publication bias in the multivariate meta-regression model. While the overall effect was significantly reduced, youth employment interventions still showed a significant positive effect on employment and earnings outcomes. Nonetheless, the team concluded that the summary effect size of youth employment outcomes probably represented an upper bound for the true impact of these interventions. At the same time, no correlation of reported effect sizes with publication status was detected.

OPEN IN VIEWER

Figure 46

Summary forest plot of employment outcomes by publication status

OPEN IN VIEWER

Figure 47

Summary forest plot of earnings outcomes by publication status

5.1 SUMMARY OF MAIN RESULTS

Table 36 displays the main results of the systematic review and meta-analysis.

To support informed decision making, the systematic review examined the existing evidence on the effectiveness of interventions that aimed to improve the labour market outcomes of youth. The review relied on a structured and comprehensive search that allowed the identification and assessment of all relevant impact evaluation studies carried out worldwide between 1990 and 2014 across the following intervention types: •

Training and skills development, which comprises programmes outside the formal education system (and therefore does not consider Technical and Vocational Education (TVE) programmes) that offer skills training to young people in order to improve their employability and facilitate their transition into the labour market.

The key labour market outcomes considered were the post-treatment measures of employment, earnings, and business performance. •

Employment outcomes include employment and/or unemployment probabilities, participation rates, hours worked, unemployment duration and quality of employment.

In the process of understanding what works, the review also focused on the way in which interventions work, relying on prior theories of change for the selected intervention types as well as on observed programme characteristics reported in the studies. 1

The systematic review showed that investing in youth through active labour market measures may improve outcomes. Interventions to support young women and men in the labour market may lead to positive outcomes, increasing their chances of finding or staying in employment and improving their income. The positive effect on employment and earnings was statistically significant with effect sizes measured by 0.04 and 0.05 SMDs, respectively, demonstrating the responsiveness of these outcomes to youth's exposure to active labour market programmes (ALMPs). With substantially less evidence, the effect on business performance outcomes was not statistically significant at 0.03 SMD; however, when only entrepreneurship promotion interventions were considered, its impact was larger and significant, at 0.10 SMD.

5.2 UNPACKING THE CAUSAL CHAIN ACROSS YOUTH EMPLOYMENT INTERVENTIONS

Section 1.3 proposed a series of causal chains connecting youth-targeted ALMPs to expected outputs such as direct job creation or changes in skills, knowledge, attitudes, or behaviours, and ultimately linking programme delivery to projected labour market outcomes as well as other closely correlated outcomes such as accumulation of human capital.

Some of these anticipated connections were confirmed by the results of the systematic review, shedding light on the impacts of skills training, entrepreneurship, employment services and subsidized employment on labour market outcomes of youth.

This section re-examines the proposed result chains, reflecting on the transmission channels that lead from activities to outcomes. It relies on the findings from the meta-analysis and digs deeper into the individual studies, unpacking features of programme design and implementation that triggered success in the achievement of intermediate and final outcomes.

5.3 AGREEMENTS AND DISAGREEMENTS WITH OTHER STUDIES OR REVIEWS

The effort to undertake this systematic review was initially motivated (see Section 1.4) by the statement that new evidence was needed to support decision-making on youth employment. Specifically, similar systematic reviews and studies either required urgent updating (Betcherman et al., 2007) or simply posed related, yet distinct, research questions (e.g., Card et al., 2010, 2015; Tripney et al., 2013; Grimm & Paffhausen, 2015; Filges et al., 2015).

The findings presented in the previous section are aligned with that motivation: the results of the empirical analysis are generally congruent with previous and related literature, but they (i) add much more depth given the rigour of the analysis and the comprehensive nature of the data, (ii) complement and carve out much more clearly the patterns indicated by previous studies, and (iii) add genuinely novel insights. So, in essence, the current study found few points of variance with related studies, but agreed on major lines, strengthening the existing knowledge base and identifying many new, detailed aspects: 1)

Agreement: The main result agreed with related studies – youth interventions are effective tools for improving labour market outcomes. With a broader sample of target population, i.e. not only youth, Card et al., 2010 and 2015 found that ALMPs have smaller effects for older workers and youth in comparison to women and the long-term unemployed. The ability of those studies to factor in other groups offers an important insight to the results of this review: While the impact of ALMPs on youth was positive, the magnitude of reported effect sizes is smaller than ALMPs targeting all individuals, without an age target. The systematic review and meta-analysis by Filges et al. (2015) also found a small increase in employment probability for unemployment insurance recipients who participated in ALMP programmes.

5.4 COMPLETENESS AND APPLICABILITY OF EVIDENCE

The evidence base on youth employment is growing and improving. While better study designs should lead to lower risks of bias, limitations in the evidence still shed only partial light on what works. 1)

Insufficient consideration of spillovers and general equilibrium effects. Exposure to ALMPs is expected to create a spillover effect among non-participants, as well as causing general equilibrium effects throughout the economy. While some of these spillovers may positively affect overall employment outcomes, in certain cases they can hamper the performance of programme non-participants. This is true of the substitution effects and windfall effects that can arise from wage subsidy programmes, which are rarely addressed in the empirical literature.

5.5 QUALITY OF THE EVIDENCE

This systematic review did not undertake a full risk of bias assessment as recommended in the systematic review methodology literature. However, it relied on a framework by Duvendack et al. (2012) to assess the analytical and statistical rigour of the included studies based on the studies’ design and statistical methodology. This framework makes some assessment of study design and confounding, but excludes other domains of bias such as performance bias, selection bias (including attrition) and biases in outcomes data collection. A future update of this systematic review would need to take these factors into account.

5.6 LIMITATIONS AND POTENTIAL BIASES IN THE REVIEW PROCESS

The review relied on evidence from indirect comparisons of programmes in different contexts. It excluded studies which only examined the relative effects of two or more interventions, limiting the extent to which review question 2, namely “Which of these interventions are the most effective on average?”, could be answered using direct comparisons of programmes in the same context. In addition, the review did not assess net outcomes resulting from the included interventions, such as whether employment creation of ALMP participants displaced non-participants thereby decreasing the overall employment impact of the programme. The summary effect size of youth employment outcomes may therefore represent an upper bound for the true impact of these interventions.

The review team made an extensive effort to collect missing information by contacting authors using a standardized template to solicit the data required for inclusion of the study. In addition, the team employed several methods to impute missing information where possible and extensively tested the adequacy of these procedures. This allowed effect sizes for a large share of the included studies to be computed. However, the main empirical analysis was based on 2,259 of the 3,629 coded treatment effect estimates, for which it was possible to compute the SMD. While this sample is much larger than in most other systematic reviews, it remains difficult to assess the degree to which missing information may impact the empirical findings (i.e., whether reporting quality is correlated with effect size magnitudes).

Relatedly, it was not possible to conduct a detailed assessment of the risk of bias in included studies. The main reason was that most reports did not provide the information needed to objectively code the information required. This is irrespective of the publication status and study design: Many experimental impact evaluations do not provide basic information on the randomization approach (allocation method and concealment from participants) or the participant flow through the study. For quasi-experimental studies, there are yet no common standards for reporting and analyzing bias that unifies various econometric approaches. It is therefore very difficult to objectively assess bias in studies without contacting original authors, who may not be inclined to respond to such queries. We therefore chose to adopt the approach to classification based mainly on study design.

Echoing other reviews in the social sciences (e.g., Tripney et al., 2013), the review found that the methods for calculating comparable effect sizes from studies using more complex multivariate econometric methods are underdeveloped and require further research. However, the review benefitted from the experience of the principal investigators and was carried out with frequent guidance from the Methods Coordinating Group of the Campbell Collaboration.

Finally, the search and selection of studies focused specifically on quantitative impact evaluations using a rigorous (quasi-) experimental design. While the team believes that this is a strength of the review, the method may have disregarded important findings from studies that were rather more qualitative in nature or did not attempt to provide causal effect estimates.

6.1 IMPLICATIONS FOR POLICY AND PRACTICE

The extent and urgency of the youth employment challenge and the level of global attention currently being given to this topic calls for more and better evidence-based action. Accordingly, this systematic review sought to examine the empirical evidence in order to understand what drives the success (or failure) of youth employment interventions. Investments in youth employment will continue, and even increase, as countries embark on the implementation of the 2030 Agenda for Sustainable Development; therefore, this review focused on identifying “what works” and, as far as possible, “how”.

This systematic review builds on a growing base of studies measuring the impact of youth employment interventions and offers a rigorous synthesis and overall balance of empirical evidence taking into account the quality of the underlying research. The review is systematic through a clearly defined and transparent inclusion and exclusion criteria, an objective and extensive search, a punctual data extraction process, a standardized statistical testing and analysis, and a thorough reporting of findings. These elements and underlying methods and tools were laid out and reviewed in the protocol (Kluve et al. 2014).

The evidence suggests that investing in youth through active labour market measures may pay off. The evidence also shows a significant impact gap across country income levels. Being unemployed or unskilled in a high-income country – where labour demand is skill intensive – puts youth at a distinct disadvantage in comparison to a cohort that is, on average, well educated. While ALMPs in high-income countries can integrate disadvantaged young people into the labour market, they are not able to fully compensate for a lack of skills or other areas where youth failed to gain sufficient benefit from the education system. On the other hand, in lower income countries, with large cohorts of disadvantaged youth, marginal investments in skills and employment opportunities are likely to lead to larger changes in outcomes. Youth-targeted ALMPs in low- and middle-income countries do lead to impacts on both employment and earnings outcomes. Specifically, skills training and entrepreneurship promotion interventions appear to yield positive results on average. This is an important finding, which points to the potential benefits of combining supply- and demand-side interventions to support youth in the labour market.

The evidence also calls for careful design of youth employment interventions. The “how” seems to be more important than the “what” and, in this regard, targeting disadvantaged youth as well as providing incentives for participation of youth, appropriate profiling mechanisms and schemes to motivate service providers to perform effectively may act as key factors of success.

The latter emphasises the ability of specific design features within employment interventions to affect individual behaviours – in this case among both young people and service providers. It also implies – and calls for – sensible interpretation of the results. The findings from this review need to be discussed vis-à-vis the local and national context and should be complemented by a long-term and holistic commitment towards youth development.

Achieving an understanding of the “how” element is not an easy task. Although the systematic review excluded studies which only reported relative effects, it is also the case that, frequently, impact evaluations do not assess relative effectiveness. Even more often, reports and papers fail to describe the underlying theory of change and observed transmission mechanisms behind an intervention. In some other cases, there is limited information about the characteristics of programme participants in the evaluation sample and their comparison group. Much remains to be done to improve reporting standards and advocate for more and better evidence examining the impact of youth employment interventions. The quality of the primary studies determines the quality of the systematic review and any subsequent synthesis of the evidence.

6.2 IMPLICATIONS FOR RESEARCH

Counterfactual studies examining youth employment interventions are comparatively well designed, with an increasing share of experimental evaluations conducted in recent years. While this assessment of the study design could only provide a partial picture, the analysis showed a relatively high overall level of rigour of included studies: Only 9 per cent of studies were judged to have a low level of rigour, based on their research design and empirical methodology. However, evidence for small study effects suggested publication bias was present, based on the sample of included studies.

A number of issues which placed limitations on this review could be mitigated with additional or improved primary research on youth employment interventions. 1)

Existing research is spread unevenly across the globe. While the evidence gathered was global in nature, capturing 31 countries and all regions of the world, slightly more than half of the evidence derived from interventions in high-income countries. While it was possible to include a number of recent experimental studies from middle- and low-income countries – Not ably sub-Saharan Africa and Latin America and the Caribbean – there was a distinct lack of evidence from Asia, Central Europe, the Pacific, the Middle East and North Africa. Furthermore, the evaluations of youth employment interventions in low- and middle-income countries were concentrated on rather small-scale, NGO-implemented interventions and there was a lack of evidence for larger, nationwide governmental programmes.