Technical and Vocational Education and Training (TVET) Interventions to Improve the Employability and Employment of Young People in Low- and Middle-Income Countries: A Systematic Review

1.2.1 Definition

There is no universally accepted definition of technical and vocational education and training (TVET). As a field, it is continually changing, usually in response to the demands made upon it (Maclean & Wilson, 2009). Broadly defined, TVET is concerned with the acquisition of knowledge and skills for the world of work. Here, we follow UNESCO's definition of TVET as ‘….a comprehensive term referring to those aspects of the educational process involving, in addition to general education, the study of technologies and related sciences, and the acquisition of practical skills, attitudes, understanding and knowledge relating to occupants in various sectors of economic and social life’ (UNESCO, 2010b).

A great diversity of TVET models can be found worldwide. Various terms are used to describe the diverse elements of the field that are now conceived as comprising TVET, many of them specific to particular geographical areas (for example, in the United States, the current term is career and technical education). Furthermore, the organisation of TVET varies widely, both between and within countries. With no internationally accepted set of definitions of the different types that can be distinguished, the following definitions have been used for the purposes of this systematic review:

Technical education: theoretical vocational preparation of students for jobs involving applied science and modern technology; compared to vocation education (which focuses on the actual attainment of proficiency in manual skills), technical education emphasises the understanding of basic principles of science and mathematics and their practical applications; usually delivered at upper-secondary and lower-tertiary levels to prepare students for occupations that are classified above the skilled crafts but below the scientific or engineering professions (although diploma- and degree-level courses also exist).

Vocational education: organised activities designed to bring about learning as preparation for jobs in designated (manual or practical) trades or occupations; traditionally non-theoretical and focused on the actual attainment of proficiency in manual skills; usually considered part of the formal education system and thereby falling under the responsibility of the Ministry/Department of Education.

Vocational training: prepares learners for jobs that are related to a specific trade or occupation; but, compared to vocational education, is better linked to the labour market and employment development system, and therefore usually falls under the responsibility of the Ministry/ Department of Labour.

On-the-job training: workplace-based training that uses real jobs as a basis for instruction and for practical purposes.

Apprenticeship training: combines on-the-job training for a highly skilled craft or trade (from someone who is already a skilled leader in the field) with academic/ theoretical instruction; ranges from informal work-based ‘learning-by-doing’ to formal structured programmes sponsored by large industrial firms.

1.2.2 How the intervention might work

The following logic model provides a very simple representation of the relationships among (a) the resources that are invested; (b) the activities that take place; and (c) the benefits or changes that result, as a sequence of events.

Preliminary thoughts on some of the underlying assumptions are indicated. Employability refers to a person's capability of gaining initial employment, maintaining employment (including the ability to make transitions between jobs and roles within the same organisation to meet new job requirements) and/or obtaining new employment if required (Hillage & Pollard, 1998). It is therefore a concept that can be applied to both employed people seeking alternative jobs or promotion and unemployed people seeking work. The concept of employability has become a cornerstone of labour market policies and employment strategies internationally, with many policymakers viewing the development of individual employability as a crucial step towards improving access to employment and as a means of offering workers the opportunity to develop the skills allowing self-sufficiency within the labour market (McQuaid & Lindsay, 2005). There is increasing recognition that employability is dependent not only on individual characteristics but also the environmental, social, and economic context in which work is sought (Department for Employment and Learning, Northern Ireland [DELNI], 2002).

Most interventions will have positive and negative impacts, both of which should be taken into account to assess the net difference that results from the intervention, over and above what would have taken place anyway. A key question concerning labour market interventions, including those offering TVET opportunities, is whether job creation is additional or not. It is changes in the net employment rate that are of primary interest to policy-level organisations and departments. Therefore, estimates of gross employment outcomes should, ideally, be adjusted to take into account displacement and substitution effects. ¹ In instances where no adjustments have been made, it will be important to remember that, even if found to be effective, the TVET intervention may not generate any additional employment; it may only be affecting who gets employed, not the level of employment. It is recognised that determining the ‘additionality’ of any employment effects is methodologically very challenging.

2.2.1 Interventions

Inclusion in the systematic review was restricted to TVET interventions ² with the following characteristics: •

Technical education, vocational education, vocational training, on-the-job training, apprenticeship training (as defined in the Introduction);

Multi-service interventions (for example, combining on the-job training with wage subsidies) were eligible for inclusion in the review. Also eligible were studies focused on the provision of financial assistance to purchase training where trainees' participation in such training was then evaluated. Labour market programmes that did not incorporate any training, but were restricted to the provision of other services, such as job search assistance or financial subsidies, were not eligible. Interventions promoting entrepreneurship through, for example, technical and business training, were eligible, whilst those promoting self-employment by providing technical assistance only were not. Continuing professional development (CPD) interventions (i.e., those designed to upgrade the knowledge and skills of practitioners in the medical and other professions) were not eligible. Interventions targeted specifically at youth with particular special needs, such as learning disabilities, physical disabilities, emotional problems, or behavioural problems, were outside the scope of this review. However, studies in which youth with special needs participated in mainstream TVET/skills training were considered eligible.

2.2.2 Participant characteristics

The focus of this review is on young people. Countries vary considerably in their definition of young people. The standard United Nations definition of youth as those belonging to the 15-24 years age group was applied to this review (UN, 1992).

Eligible participant populations included youth with the following characteristics: •

Age: Young people aged 15 - 24 years. In addition to samples in which all participants were aged 15-24 years, samples consisting of both young people and adults older than 25 years were eligible if (i) the average age of the sample lay between 15 and 24 years; (ii) the majority of participants were aged 15-24 years; or (iii) findings were disaggregated by age and reported for 15-24 year olds.

2.2.3 Research designs

To fulfil the eligibility criteria, studies were required to be impact evaluations that used an experimental or quasi-experimental design. Eligible research designs included those in which the authors used a control or comparison group, and in which: participants were randomly assigned (using a process of random allocation, such as a random number generation); a quasi-random method of assignment was used and pre-treatment equivalence information was available regarding the nature of the group differences (and groups generated were essentially equivalent); participants were non-randomly assigned but matched on pre-tests and/or relevant demographic characteristics (using observables, or propensity scores) and/or according to a cut-off on an ordinal or continuous variable (regression discontinuity design); participants were non-randomly assigned, but statistical methods were used to control for differences between groups (e.g., using multiple regression analysis, including difference-in-difference, cross-sectional (single differences), or instrumental variables regression).

For this review, the control or comparison conditions in eligible studies included youth receiving no treatment, treatment as usual, or an alternative treatment. No restriction was placed on duration of follow up.

2.2.4 Outcomes

To be included, a study had to assess intervention effects on at least one eligible outcome variable. Qualifying outcome variables were those falling into the following general construct categories: (a) employment and (b) employability. •

primary outcomes represented by the general construct employment: for example, gaining initial employment; re-entering employment; obtaining ‘better’ employment (e.g., through promotion or gaining employment in the formal sector); self-employment (starting a new business or expanding one); working hours; and payment levels (i.e., earnings, wages, salary or income)

Studies measuring either gross employment or net employment (i.e., where displacement and substitution effects have been taken into account) were eligible for inclusion.

Qualitative studies were not eligible for inclusion in the review, including any studies using perception measures only (i.e., those examining the views of employers and/or the workforce about their employability), regardless of whether or not they quantified their findings.

2.2.5 Other study characteristics

The date of publication or reporting of the study must have been in 2000 or later. This date was chosen due to time and funding limitations. The funder approved the cut-off for that reason. Eligible studies could be published in any language provided they met all other eligibility criteria. We did not exclude specific forms of publication, such as theses and dissertations.

2.3.1 Information Sources

A wide range of general and specialist electronic bibliographic databases was searched (see Appendix 9.3).

A tailored search query was developed for each bibliographic database relying on the database's index terms (where available) and/or free-text terms. In most cases, the search strategies combined a comprehensive list of search terms related to the intervention, outcomes, and research design. Database thesauri were consulted to ensure that all appropriate synonyms were included. Synonyms and wildcards were applied as appropriate. There were no country or language restrictions to the search. A publication year filter was used. The search strategy for ERIC is presented in Appendix 9.4.

To supplement the electronic bibliographic database search, we hand searched websites/gateways, checked the bibliographies of all included studies and relevant reviews, and performed forward citation tracking (through the ISI Web of Knowledge and Google Scholar). A list of websites/gateways searched is presented in Appendix 9.5. We emailed specialists in the field, including authors of included studies, for information about any potentially relevant studies. A specific request for assistance with the location of study reports published in languages other than English was made. Authors and funding sources were also contacted regarding the availability of translated versions of included studies. Requests for relevant literature were also made through the Network for Policy Research, Review and Advice on Education and Training (NORRAG) and the UNESCO-UNEVOC e-forum. Again, a specific request for assistance with the location of studies published in languages other than English was made. We did not undertake a keyword search using Google Scholar, hand search individual journals, or search for conference proceedings or dissertations separately.

2.3.2 Study selection

Selection of primary studies was based on the pre-developed selection criteria described above. These criteria were piloted by two researchers who screened a sample of reports independently and compared their results. Discrepancies were resolved by further review of the respective titles and abstracts. This process was repeated until consistency in application of the selection criteria was achieved. The study selection process then proceeded as follows: •

The review team manually examined the titles and abstracts of records identified through the searches of electronic databases to assess eligibility. The relevance of each item was assessed by an individual reviewer (i.e., single screening) and decisions recorded in the reviewing software, EPPI-Reviewer. Items were included at this stage if they appeared to meet the criteria on the basis of the information in the title and abstract, and excluded if they were clearly ineligible. Where there was any doubt as to their eligibility, items were marked as ‘unsure.‘ Where the title and/or abstract were not in English, the translation service offered by Google, http://translate.google.com/, was used to translate the information into English; screening against the selection criteria then proceeded as normal. In cases where only the title of the study was available, reference within the wording of the title to (a TVET intervention) AND (a relevant employment-related outcome OR a term suggesting the study was an evaluation) automatically warranted a full length review of the article.

2.3.3 Data collection

Reviewers used a coding tool to capture both substantive and methodological characteristics. The tool modified an existing EPPI-Centre tool in accordance with Campbell Collaboration guidelines, and also drew upon previous work by Wilson, Lipsey, Tanner-Smith, Huang, and Steinka-Fry (2010). Piloting of the coding tool was undertaken by members of the review team who worked independently on a random sample of eligible studies before meeting to compare their decisions. Reviewers were retrained on any coding items that showed discrepancies during this process and the coding manual was adapted accordingly. This process was repeated until a very high level of consistency in reviewers' application of the codes was achieved, at which point the tool was finalised. The remaining studies were double-coded.

For eligible studies published in English, different pairs of reviewers independently extracted information from each study report and then came together to compare their decisions. Any uncertainties and discrepancies were resolved by discussion, further review of the respective study reports, and consultations with a third reviewer (JT, MN or JH), where necessary. Guidance on advanced statistical issues was provided by JH. For eligible studies published in languages other than English, attempts were made to contact authors and funding sources regarding the availability of translated versions. Where these could not be obtained, two Spanish-speaking reviewers were identified and invited to join the research team (KS-F, EW). They used the coding tool to extract the relevant information and critically appraise the four studies in question.

Reviewers entered data directly into the EPPI-Reviewer 4 database (Thomas, Brunton, & Graziosi, 2010). Information was collected relating to general study characteristics (such as year of publication); participant characteristics; the nature of the intervention and its implementation; study methods; outcome variables; and findings. The coding process also incorporated a careful consideration of the potential methodological limitations of the included studies, focusing on the following key domains: selection bias, confounding, spillovers, outcome reporting bias, and analysis reporting bias. This involved use of a tool developed by researchers at the International Initiative for Impact Evaluation (3ie) specifically for assessing risk of bias in experimental and quasi-experimental designs based on statistical methods. The coding tool was detailed in the review protocol. ³

The approach taken to formulating summary assessments of risk of bias and study ‘quality’ is presented in Table 2.1 (adapted from the suggested framework in Higgins et al., 2011).

To help identify implications of the review findings for policy and practice, we used an interpretation framework that considered the strength of evidence for each set of results. This interpretation framework has evolved over several reviews by staff at the EPPI-Centre, spanning different disciplines.

2.4.1 Effect size indices

Where data allowed, effect sizes were computed for each study. Standardised mean differences (SMDs) were the metric used in the meta-analyses to synthesise the effects of TVET interventions on both continuous outcomes (e.g., earnings) and dichotomous outcomes (e.g., employment). Although other effect sizes such as risk ratios or odds ratios might be methodologically more adequate to synthesise dichotomous outcomes (Higgins & Green, 2011), most of the studies included in the review using dichotomous outcomes do not report sufficient information to use risk ratios and odds ratio. Thus, the use of risk ratios or odds ratios may lead to a substantial loss of information for the analysis. Therefore, we followed Petrosino, Morgan, Fronius, Tanner-Smith, and Boruch (2012) in the use of SMD for synthesising both continuous and dichotomous outcomes.

Each meta-analysis included a single type of outcome measure. In the primary studies, the causal effects of the programmes were expressed as a: •

Mean difference: i.e., a difference in the mean outcomes from the treatment and control groups (for example, in the mean level of earnings, weekly hours, etc.). For earnings, this value was reported in local currency or USD($).

In the majority of cases (see below for exceptions), SMDs were calculated using the following formulae.4 The numerator represents the causal raw impact of the programme on the outcome. In matching-based studies, this was the average treatment effect on the treated (ATT): the difference in outcomes between groups after matching (i.e., Y treatment - Y control). In a regression analysis, this is the coefficient of interest (β).

For matching-based studies:

S M D = A T T S p

To calculate the pooled standard deviation (the standard deviation of the outcome variable for both treated and control individuals) we used the Hedges' approach described in Lipsey and Wilson (2001):

S p = ( n t - 1 ) * S t 2 + ( n c - 1 ) * S c 2 n t + n c - 2

For regression-based studies we first used the formula described in Keef and Roberts (2004):

S M D = β σ

The denominator σ is the standard deviation of the error term in a regression. Where σ was not reported, we used the following formula as an equivalent ⁵ .

S p o o l e d = S t o t a l * ( n t o t a l - 1 ) - ( β 2 * ( n c o n t r o l * n t r e a t m e n t ) ) n c o n t r o l + n t r e a t m e n t n t o t a l - 1

Standard errors were calculated using the following formulae ⁶ , where t is the t test associated with the treatment effect of a regression:

S E ( S M D ) = S M D t

In a small number of cases7, SMDs were computed using the following input data: •

means, standard errors, and sample size information

The following effect-size calculator was used: http://gunston.gmu.edu/cebcp/EffectSizeCalculator/index.html

The review also corrected for sample bias in the effect sizes due to small sample sizes by using the correction for sample bias procedure developed by Hedges and Olkin (1985). All effect sizes were coded such that positive effect sizes represented positive outcomes (e.g., less unemployment, higher wages).

Basically corrected SMD and corrected SE were estimated as follows:

S M D c o r r e c t e d = S M D u n c o r r e c t e d * [ 1 - 3 4 * ( n t + n c - 2 ) - 1 ] S E ( S M D ) c o r r e c t e d = S E ( S M D ) u n c o r r e c t e d * [ 1 - 3 4 * ( n t + n c - 2 ) - 1 ]

2.4.2 Synthesis methods

A meta-analysis was conducted on a set of studies answering a particular question where there was a minimum of two studies rated medium quality (see Table 2.1). The data synthesis was carried out using random effects statistical models. To account for differences in sample sizes for individual studies, effect sizes were averaged across studies by using inverse variance weighting of the individual effect sizes. This weighting resulted in the individual effect sizes from larger n studies being given more weight in the combined effect size. To visibly examine between-study variability in the effect size estimates, forest plots are used to display the estimated effect sizes from each study along with their 95% confidence intervals. Heterogeneity tests were used to examine whether variation in effect size estimates was larger than expected from sampling error alone (Deeks, Altman, & Bradburn, 2001). Heterogeneity was explored using both the Q test and the I² index. The Q test reveals the presence versus absence of heterogeneity as indicated by a p value of <0.05, while the I² index quantifies the degree of heterogeneity (Higgins & Green, 2011). If significant heterogeneity was found, possible reasons for the differences between studies was explored through analysis of sub-groups of studies (see below).

2.4.3 Missing data

If we had studies that were missing essential data, our approach involved thorough attempts to contact the original investigators and funding sources, and discussion of the potential impact of missing data on the findings of the review (Higgins & Green, 2011).

2.4.4 Moderator analyses

Moderator analyses were performed to examine potential variability in effects due to study, participant, and intervention characteristics. An analogue to the ANOVA analysis (univariate) approach was used, as described in Lipsey and Wilson (2001). It was not possible to conduct multivariate moderator analysis using meta-regression models, as we did not have the minimum of 10 studies for each individual moderator variable (Borenstein, Hedges, Higgins, & Rothstein, 2009). Moderator analyses were used both to explore heterogeneity and also to answer questions about the effectiveness according to specific characteristics of interventions and population groups. The categorical variables that identified the sub-groups used in this exercise were specified in advance of the meta-analysis.

2.4.5 Publication bias analyses

Due to an inadequate number of studies, we did not attempt to detect or exclude the existence of publication bias using statistical methods, such as funnel plots or ‘trim and fill’ analyses (Duval & Tweedie, 2000; Egger, Davey Smith, Schneider, & Minder, 1997; Lau, Ioannidis, Terrin, Schmid, & Olkin, 2006). This issue is discussed further in Chapter 5.

3.2.1 Methodological characteristics

The majority of the 26 studies were conducted independently by teams based in universities and/or other research organisations; others were carried out exclusively by researchers who were closely related to the funding body or by teams comprising both independent and related evaluators.

The 26 studies employed different methodologies for evaluating the impacts of the interventions. Three studies used a randomised experimental design, and 23 used a non-experimental design (of these, 2 were natural experiments and 21 were quasiexperiments). There was heterogeneity within the 23 non-experimental studies in terms of the selection of the comparison group. In around half of the non-experimental studies, the comparison group was constructed ex-ante, for example, from registered applicants who did not start the course or from eligible non-applicants. The remaining non-experimental studies involved ex-post selection of the comparison group, for example, from similar individuals identified in household or labour force survey data. The non-experimental evaluations used different econometric techniques to address selection bias and net out the impacts of other factors (each of which imposes different assumptions and have different strengths with respect to internal validity). These techniques are broadly classified into two main types: matching and covariate/regression adjustment. The most common matching method used was propensity score matching (Rosenbaum & Rubin, 1983); other matching methods included log-odds ratio matching and the use of non-parametric reweighting techniques. Studies using covariate adjustment methods used different model specifications, most commonly ordinary least squares regression. The majority of studies used more than one estimation method, often as a means to verify robustness of the results. ¹⁷

In terms of measurement of outcomes, the majority of studies utilised a cross-sectional impact estimator; that is, they compared the outcomes for treatment and comparison group persons measured at some time period after the intervention ended. By exploiting the panel structure of the data, around a third of the included studies attempted to purge time-invariant unobservables through the use of the difference-in-difference estimator. This involves subtraction of the before and after change in outcomes for comparison group members from the before and after change for treatment outcomes (where the change is measured relative to some pre-intervention benchmark time period). ¹⁸

Most studies measured average treatment effects on the treated (ATT): the differential impact that the treatment showed for those individuals who actually participated in the intervention. A minority of studies measured average treatment effects (ATE), which can be defined as the average effect for the population, and/or conducted intention-to-treat (ITT) analyses which were based on the initial treatment intent, not on the treatment eventually administered. ¹⁹ Other parameters of interest included marginal treatment effects (MTE) and average treatment effects on the untreated (ATU).

There was variation between studies in relation to the time that had elapsed between completion of the TVET intervention and the measurement of outcomes (see Table 3.1). Nine of the 26 studies investigated the sustainability of treatment effects over time, with a maximum of three post-test measurements. To examine whether the impact of the intervention was robust to the data collection period, 7 studies collected data for several cohorts covering a number of years (with results reported separately for the different year groups and/or pooled).

In the case of 11 studies, a single cohort of a multi-cohort intervention was used for the evaluation. Two studies focused on the entire universe of trainees. Two-thirds of all studies estimated heterogeneity of treatment effects, most commonly by gender. Sixteen studies involved analysis with a sample size of more than 250 participants, and 7 with a sample size of 250 participants or less. The sample size was not specified in 3 studies. Finally, a very small number of studies dealt specifically with the issue of trainees with partial instruction.

3.2.2 Participant characteristics

The majority of studies included both male and female participants (fairly evenly balanced), with a single study focused exclusively on young women. In 19 studies, the average age of the study participants lay between 16 and 24 years. One study focused on 12-22 year olds, but no average age was provided. Of the 5 studies that did not report the average age, 2 reported the age range: 1 study focused on participants aged 16-35 years, and another included individuals aged 18-65 years. Each of these 5 studies conducted sub-group analyses by age. Finally, in 1 study, the average age of participants was 36 years; it too conducted sub-group analyses by age. The majority of study samples included some participants in employment and others who were not in employment. In a single study, none of the training beneficiaries were employed at the start of the study.

Argentina:

1.

Entra 21: an international programme (2001-ongoing) operating in 18 countries across South America and the Caribbean. The two Argentine projects evaluated are (i) Fundación SES (Sustentabilidad- Educación -Solidaridad), which provides training in five regions of the country; and (ii) Agencia para el Desarollo Económico de la Ciudad de Córdoba (ADEC), which trains students from the area of Rio Segundo. This intervention was evaluated by Alzuá et al. (2007).

Bosnia and Herzegovina:

3.

Emergency Demobilization and Reintegration Project: a national programme implemented over a four year period (1996-1999). This intervention was evaluated by Benus et al. (2001).

Bhutan:

4.

Rural Skills Development Project: operating between 2007 and 2010, this was a regional programme covering all sub-districts across the three rural districts of Haa, Trashigan, and Bumthang. This intervention was evaluated by Chun and Watanabe (2011).

Brazil:

1.

(see above) Entra 21: the two Brazilian projects evaluated are: (i) Centro de Ensino Profissionalizante Rotary (CEPRO) based in São Paulo; and (ii) Instituto de Hospitalidade (IH), which operates in the northeast of the country. This intervention was evaluated by Alzuá et al. (2007).

Chile:

5.

Chile Joven (Programa de Capacitación Laboral de Jóvenes): one of the earliest in a series of Latin American training programmes sponsored during this period by the Inter-American Development Bank, Chile Joven was available between 1991-2002 (phase I: 1991-1995; phase II: 1996-2002). This intervention was evaluated by Aedo and Pizarro (2004).

China:

7.

Retraining programmes for laid-off workers piloted in 30 municipalities in 1994 and expanded to 200 cities by 1996. Training in the cities of Shenyang and Wuhan is evaluated. This intervention was evaluated by Bidani et al. (2002).

Colombia:

8.

Jóvenes en Acción (Youth in Action): operating between 2002 and 2005, this was a national programme offered in seven of the largest cities of the country: Barranquilla, Bogota, Bucaramanga, Cali, Cartagena, Manizales, and Medellin. This intervention was evaluated by Attanasio et al. (2011).

Dominican Republic:

12.

Juventud y Empleo: one of a series of Latin American training programmes sponsored during this period by the Inter-American Development Bank, this programme operated nationally between 1999 and 2007. (NB: A programme by the same name continues in a slightly different format, with evaluation results due later this year). This intervention was evaluated by Card et al. (2011).

India:

13.

A livelihoods intervention implemented in 2001 as a year-long, city-wide pilot project (operating in Allahabad). This intervention was evaluated by Mensch et al. (2004).

Kenya:

14.

Technical and Vocational Vouchers Program: a regional initiative operating since 2008 in the area of Busia in Western Kenya (still ongoing). This intervention was evaluated by Hicks et al. (2011).

Latvia:

15.

Occupational training (OT) programmes for the unemployed implemented nationwide by the State Employment Agency of Latvia; operating since the early 1990s. This intervention was evaluated by Dmitrijeva (2009).

Mexico:

16.

Bécate (and the programme sub-component Capacitación en la Práctica Laboral): this ongoing initiative started in 2004, as a replacement for SINAT (see no.18 below). This intervention/sub-component was evaluated by Analítica Consultores (2006) and van Gameren (2010).

Panama:

19.

PROCAJOVEN: an independent sub-programme of the Assistance Program for the Building of a Training and Employment System in Panama (one of a series of Latin American training programmes sponsored during this period by the Inter-American Development Bank). Approved in 2002, PROCAJOVEN operated nationally until 2009. This intervention was evaluated by Ibarraran and Rosas-Shady (2006).

Peru:

20.

ProJoven (Programma de Capacitacion Laboral Juvenil): an ongoing large-scale intervention first implemented in 1996, and one of a series of Latin American training programmes sponsored during this period by the Inter-American Development Bank. Originally, the programme was intended to be implemented nationwide. In practice, reduced funding resulted in the programme starting in Lima (the capital) and progressively expanding to more cities (13 in total as of 2010). This intervention was evaluated by Chong and Galdo (2006); Chong et al. (2008); Díaz and Jaramillo (2006); Espinoza (2010); Jaramillo et al. (2007); and Ñopo et al. (2007).

4.2.1 Type of TVET intervention

The interventions involved different forms of TVET (see Table 4.1). The most common form of intervention was a two-phase TVET intervention that combined both theoretical and practical training (usually in the format of classroom-based vocational training followed by a period of on-the-job training to provide beneficiaries with work experience). Typically, these interventions were aimed at short-term semi-skill training in specific occupations demanded in the private sector, and provided basic job readiness skills and some trade-specific skills. Nine interventions were of this type. Two interventions consisted of different TVET-related sub-components, and young people seeking training could choose between the available options. The majority of the remaining interventions offered a single form of TVET.

4.2.2 Implementation

Various agencies were involved in the design and planning of the reviewed interventions, most commonly government agencies (Ministries of Labour, employment offices, etc.). Funding for the interventions came from a number of sources; many involved public/private partnerships between national and/or local government agencies, international development agencies (such as USAID) and multilateral organisations (most commonly, the Inter-American Development Bank). Social partners, in the form of employers, contributed to the financing of one intervention.

Participation of the private sector in the provision/delivery of training was a feature in around two-thirds of the reviewed interventions. Most of these adopted a ‘demand driven’ approach whereby the content of the courses they offered were customised to meet the needs of the local labour market: the assumption being that there would be job vacancies for the trainees when they graduate. In these cases, the government selected the training providers and courses competitively, through a bidding process where usually both private and public firms and/or training institutions could participate. For example, one intervention was executed by private sector NGOs from across 18 countries, with each NGO entirely responsible for eligibility criteria, obtaining the internships, defining the course contents, and so on. Of the remaining interventions, a small number adopted a centre-based model whereby the government was responsible for not only the financing and regulation of training, but also its content and provision (through a national training institution). ²³ A notable exception was an intervention that employed peer-to-peer instructors.

4.2.3 Target group

The target groups for the reviewed interventions fell into two main categories. The majority were specifically targeted at disadvantaged young people, based on criteria such as household income, education level, and employment experience. The remaining interventions tended to be occupation- rather than age-focused (targeting, for example, demobilised soldiers or unemployed former employees of state-owned enterprises). A small number of interventions were directed exclusively at either urban or rural residents.

4.2.4 Intervention aims

The aims of the reviewed interventions were broadly uniform, although on the whole they were not clearly or consistently reported. Overall, the main emphasis was at the individual-level. Most interventions sought to increase levels of employment/ employability for beneficiaries, with a small number having the joint aim of increasing the value of the wages received. The broader policy goal of tackling rising unemployment attributed to the global recession of the early- to mid-1990s underpinned many interventions. In a few cases, tackling a specific issue lay at the root of the intervention. For example, the stated aim of one programme was to mitigate the degree of poverty amongst rural residents who relied heavily on agriculture; another programme addressed the problem of displaced former employees of nationalised industries.

4.2.5 Intervention duration, frequency and format

The majority of the reviewed interventions lasted for periods of less than six months. The format for the two-phase interventions was most commonly three months classroom-based training followed by a further three months of on-the-job internships (with the shortest about three months overall, and the longest taking place over eight months). Interventions consisting solely of theoretical instruction delivered at training institutions ranged from one to six months. Those comprised solely of a period of practical on-the-job internship ranged from one to three months duration. For one intervention, the majority of trainees chose courses that lasted two years or more. Trainees on eight of the interventions were required to attend daily (Monday to Friday), and for one intervention the beneficiaries attended once a week. For the majority of the remaining interventions it was implicit that trainees attended daily. All interventions were delivered face-to-face.

Employment

•

Does participation in TVET have an effect on young people's chance of obtaining paid employment?

Income

•

Does participation in TVET have an effect on young people's earnings?

Hours worked

•

Does participation in TVET have an effect on the number of weekly hours worked by young people?

The synthesis is structured according to these three outcome categories, with findings reported in the following order: employment status (Section 5.2), income (Section 5.3), hours worked (Section 5.4), and other outcomes (Section 5.5).

Review Question 2 is concerned with finding out whether any observed relationship between TVET and these outcomes varies according to participant, intervention, or study characteristics. Using the analogue to the ANOVA approach, the following categorical variables were tested for moderating effects: study quality (medium/low quality), type of TVET intervention (two-phase models/other models), length of follow-up (short-/medium-term), and participant characteristics (female/male). The available data did not allow the use of this approach to assess the role of other potential effect size moderators, as outlined in the protocol. The moderator analysis results are presented in the text with additional statistics in Appendices 8.13 - 8.17. On the whole, given differences between studies in how many of the sub-groups were constructed, the small sample sizes in some of these analyses, and other sources of bias, we need to be wary of drawing strong inferences from the findings of these analyses.

In this review, authors of included studies often did not provide all the necessary data for calculating effect sizes. It was possible to calculate at least one effect size for only 10 of the 26 included studies. These 10 studies, and the interventions they evaluated, are detailed in Table 5.1. The intervention identification numbers correspond to the descriptions of the interventions in Chapter 4, and will be used throughout this Chapter. The findings from these 10 studies have been statistically combined using meta-analytic techniques to answer a range of sub-questions. The studies by Ibarraran and Rosas-Shady (2006) and Medina and Nuñez (2005) will occasionally appear in the same meta-analysis (as they evaluated more than one intervention). The studies by Aedo and Nuñez (2004) and Elías et al. (2004) both evaluated the Proyecto Joven programme using data from the fifth wave; however, the effect sizes from these studies are used in different meta-analyses.

The findings from the 16 studies not included in the meta-analyses will be discussed when they shed additional (or, in some cases, the only) light on a particular subquestion. Whilst meta-analysis is a more valid analysis strategy than narrative review, we decided to retain the 16 studies because their inclusion helps provide a clearer picture of the gaps in the knowledge base. We will not draw any conclusions regarding effective interventions based on these 16 studies.

The interventions evaluated in the 1o studies included in the meta-analyses comprised three distinct types (see Table 5.2).

All analyses were inverse variance weighted using random effects statistical models. The results of each meta-analysis are presented graphically in a forest plot. Each forest plot shows (i) the standardised mean difference for each individual study (represented by the dots) and the confidence intervals for that effect size (the bars on each side of the dot), and (ii) the overall weighted mean effect size (diamond) and its confidence interval (the points of the diamond represent the width of the confidence interval), which was obtained by combining the individual effect sizes from each study. ²⁴ A standardised mean difference greater than zero indicates that, on average, the group who received the TVET intervention had a better outcome than the group who did not (a positive effect). A standardised mean difference less than zero indicates that on average the group who received the TVET intervention had a worse outcome than the group who did not (a negative effect). Confidence intervals show the precision of the estimates of the effect size, by indicating the range within which the true mean is likely to be, given the observed data. For example, a 95% confidence interval of g=0.08 to g=0.51 around a mean effect size indicates a 95% probability that true mean effect size is somewhere between these two values. If the confidence interval does not cross zero (the ‘line of no effect‘) the calculated difference between the intervention and control groups can be considered as statistically significant, suggesting that the impact of the intervention is, on average, either positive or negative (depending on the direction of effect). However, interpreting the findings from the meta-analyses was challenging, due largely to the small number of studies involved.

5.2.1 Does participation in TVET have an effect on young people's chance of obtaining paid employment?

The reported data allowed the calculation of effect sizes for seven studies, and these were combined. The studies of interventions nos. 8, 12, 20, and 14 (Attanasio et al., 2011; Card et al., 2011; Espinoza, 2010; Hicks et al., 2011) were rated medium quality. The studies of interventions nos. 6, 5, and 19 (Acero et al., 2011; Aedo & Pizarro, 2004; Ibarraran & Rosas-Shady, 2006) were rated low. This analysis uses the data from each study that were closest in time to a 12-month post-training follow-up. The mean effect size and confidence intervals for each study are shown in the forest plot in Figure 5.1.

OPEN IN VIEWER

5.1

FOREST PLOT OF MEAN EFFECTS ON OVERALL PAID EMPLOYMENT

The pooled estimate of effect (g=0.134) suggests that the TVET interventions were, on average, effective; in other words, that those who experienced a TVET intervention had a greater chance of paid employment than those who did not. However, the high degree of heterogeneity between the studies (Q = 23.8; df = 7; p = 0.00124; I² = 70.6%; tau² = 0.0153) suggests differential effects across studies.

The first possible explanation that we considered for the pattern of variation seen in the meta-analysis was differences in study quality. Separate meta-analyses were conducted for (i) medium quality and (ii) low quality studies, and these were entered into a sub-group analysis. The mean effect size and confidence intervals for each study are shown in the forest plot in Figure 5.2.

OPEN IN VIEWER

5.2

FOREST PLOT OF MEAN EFFECTS ON OVERALL PAID EMPLOYMENT (BY STUDY QUALITY)

Treatment effects for the low quality studies, g=0.25 (95% CI [0.12, 0.38]), appear to be greater than for those rated medium quality, g=0.06 (95% CI [-0.01, 0.12]). Furthermore, the observed differences in mean effects were statistically significant (Q_b = 6.49; p = 0.0108).

Note: It was also observed that the majority of studies not included in this meta-analysis because effect sizes could not be calculated found that young people who had participated in TVET had a higher probability of being in paid employment than youth who had not participated (see Appendix 8.18).

5.2.1.1 Does participation in different types of TVET have different effects on overall paid employment for young people?

The TVET interventions evaluated in the seven studies included in the meta-analysis presented in Figure 5.1 comprised three distinct types (see Table 5.2). Separate meta-analyses were conducted for studies comprised of (i) two-phase TVET interventions and (ii) other TVET models. These meta-analyses were then entered into a sub-group analysis. The mean effect size and confidence intervals for each study are shown in the forest plot in Figure 5.3.

OPEN IN VIEWER

5.3

FOREST PLOT OF MEAN EFFECTS ON OVERALL PAID EMPLOYMENT (BY PROGRAMME TYPE)

For two-phase TVET interventions, a weighted average effect size of g=0.16 was observed (95% CI [0.04, 0.28]). For other TVET models, the pooled estimate of effect was positive, but negligible (g=0.01), and the confidence intervals do not exclude a negative effect (95% CI [-0.2, 0.22]). However, although treatment effects for two-phase TVET interventions appear to be greater than for other TVET modalities, the observed differences in mean effects were not statistically significant (Q_b = 1.43; p = 0.231).

5.2.1.2 Does participation in TVET have different effects on overall paid employment for young people in the short- and medium-term?

In this section, we explore whether differences in time since completion of the training might be a cause of any observed variance in outcomes. Studies included in the review varied in the length of time that elapsed between completion of the intervention and measurement of its impact on paid employment. Six of the seven studies included in the meta-analysis for overall paid employment (see Figure 5.1) assessed the impact of the intervention at a single point in time after training had ended, either at approximately 6 months or around 12-15 months. The remaining study examined impacts over time, measuring outcomes at 6, 12, and 18 months (Espinoza, 2010). Information about individual studies is presented in Appendix 8.9. Separate meta-analyses were conducted for studies comprised of (i) short-term and (ii) medium-term follow-up periods. These meta-analyses were then entered into a sub-group analysis. The mean effect size and confidence intervals for each study are shown in the forest plot in Figure 5.4.

OPEN IN VIEWER

5.4

FOREST PLOT OF MEAN EFFECTS ON OVERALL PAID EMPLOYMENT (BY LENGTH OF FOLLOW-UP)

Short-term treatment effects, g=0.18 (95% CI [0, 0.36]), appeared to be greater than medium-term effects, g=0.12 (95% CI [0, 0.24]). However, the observed differences in mean effects were not statistically significant (Q_b = 0.273; p = 0.601). When the sub-group analysis was re-run with only those studies rated medium, the differences remained insignificant (Q_b = 0.000628; p = 0.98).

5.2.1.3 Does participation in TVET have different overall paid employment effects on different sub-groups of young people?

Several studies explored this issue. Fifteen studies disaggregated the employment effects of TVET on young people by gender; six studies estimated other sub-group impacts of TVET (see Appendix 8.12). Information about individual studies is presented in Appendix 8.9.

Gender differences

The data from the following five studies were amenable to meta-analysis. The studies of interventions nos. 8, 12, and 14 (Attanasio et al., 2011; Card et al., 2011; Hicks et al., 2011) were rated medium quality. The studies of interventions nos. 2 and 19 (Aedo & Nuñez, 2004; Ibarraran & Rosas-Shady, 2006) were rated low. Separate meta-analyses were conducted for studies comprised of samples with different gender compositions. These meta-analyses were then entered into a subgroup analysis. The mean effect size and confidence intervals for each study are shown in the forest plot in Figure 5.5.

OPEN IN VIEWER

5.5

FOREST PLOT OF MEAN EFFECTS ON OVERALL PAID EMPLOYMENT (BY GENDER)

Treatment effects for female youth, g=0.1 (95% CI [0, 0.2]), appear to be slightly larger than for male youth, g=0.01 (95% CI [-0.08, 0.09]). However, the observed differences in mean effects were not statistically significant (Q_b = 2.1; p = 0.147). When the sub-group analysis was re-run with only those studies rated medium, the differences remained insignificant (Q_b = 1.49; p = 0.222).

Other population sub-group differences

The studies of interventions nos. 12 and 20 (Card et al., 2011; Espinoza, 2010) were rated medium quality. The studies of interventions nos. 18, 20, 19, and 20 (Delajara et al., 2006; Díaz & Jaramillo, 2006; Ibarraran & Rosas-Shady, 2006; Jaramillo et al., 2007) were rated low. As the two medium studies did not examine differential treatment effects for the same population sub-groups, no meta-analysis was performed.

One of the three studies that examined differences in impact by location (e.g., comparing impacts for young people from the capital city with those in other regions of the country) observed regional variation (Ibarraran & Rosas-Shady, 2006). Two of the three studies exploring treatment effect heterogeneity by level of education found larger positive effects for more educated workers, compared to the less educated (Card et al., 2011; Delajara et al., 2006). Two studies of the ProJoven intervention did not agree on the influence of poverty level. Espinoza (2010) observed that programme participation yielded no additional returns to individuals in the lowest household capita quartile prior to training, whereas Jaramillo et al.

(2007) concluded that that the strong treatment heterogeneity was not due to the variation in the initial poverty level of the beneficiaries. One of the two studies that were able to divide an evaluation sample of young people into two age groups found slightly higher point estimates among the youngest trainees in their sample (Díaz & Jaramillo, 2006). One study found that the programme increased the employment likelihood of individuals with no work experience prior to training (Espinoza, 2010).

5.2.2 Does participation in TVET have an effect on young people's chance of obtaining employment in the formal sector?

Several of the existing evaluations of TVET interventions have sought to capture not only whether trainees moved into employment as a result of the training, but also the quality of the position they secured. When talking about quality of employment, the focus is generally on the distinction between formal and informal employment. Whilst formal employment is government regulated, and workers are insured a wage and certain rights, informal employment tends to take place in unregistered enterprises, and often deprives people of financial stability and safe working environments.

In total, 10 studies assessed the impact of TVET interventions on formal employment (see Appendix 8.9 for details). Different benefit variables (proxies) were used to capture the quality of the employment position, including employment in a job with employer-provided health or social insurance, and/or formal written contract.

The reported data allowed the calculation of effect sizes from five of these studies, and these were combined in a meta-analysis. The studies of interventions nos. 8, 12, and 20 (Attanasio et al., 2011; Card et al., 2011; Espinoza, 2010) were rated medium quality. The studies of interventions nos. 5 and 19 (Aedo & Pizarro, 2004; Ibarraran & Rosas-Shady, 2006) were rated low. The mean effect size and confidence intervals for each study are shown in the forest plot in Figure 5.6.

The overall mean effect for formal paid employment is a standard mean difference of g=0.199. The confidence intervals do not cross the line of ‘no effect’ (95% CI [0.055, 0.344]). Although the analysis presented in Figure 5.7 provides evidence that the TVET interventions were, on average, effective, the heterogeneous nature of the distribution (Q = 11.1; df = 4; p = 0.0256; I² = 63.9%; tau² = 0.0131) suggests differential effects across studies.

OPEN IN VIEWER

5.6

FOREST PLOT OF MEAN EFFECTS ON FORMAL PAID EMPLOYMENT

The first possible explanation that we considered for the pattern of variation seen in the meta-analysis was differences in study quality (see the forest plot in Figure 5.7). Treatment effects for the low quality studies, g=0.37 (95% CI [0.24, 0.5]), appeared to be greater than for those rated medium, g=0.12 (95% CI [0.05, 0.19]). Furthermore, the observed differences in mean effects were statistically significant (Q_b = 10.6; p = 0.00116).

OPEN IN VIEWER

5.7

FOREST PLOT OF MEAN EFFECTS ON FORMAL EMPLOYMENT (BY STUDY QUALITY)

It was not possible to examine the variation in effect sizes (seen in Figure 5.6) by type of TVET or length of follow up due to an insufficient number of studies.

5.2.3 Does participation in TVET have an effect on young people's chance of obtaining self-employment?

Three studies separated salaried and self-employed workers and examined the impact of TVET interventions on the probability of self-employment among young people. The study of intervention no. 14 (Hicks et al., 2011) was rated medium quality. The studies of interventions nos. 18 and 17 (Delajara et al., 2006; López-Acevedo, 2003) were rated low. As there was a single medium study, no meta-analysis was performed.

Two of the three studies (Hicks et al., 2011; López-Acevedo, 2003) found a slight positive treatment effect. The remaining study (Delajara et al., 2006) found an irregular effect on self-employment; in some years, treatment effects were positive, and in other years they were negative.

5.3.1 Does participation in TVET have an effect on young people's earnings?

The available data allowed the calculation of effect sizes for eight studies. For this outcome, the studies of interventions nos. 8, 12, 2, and 14 (Attanasio et al., 2011; Card et al., 2011; Elias et al., 2004; Hicks et al., 2011) were rated medium quality, while studies of interventions nos. 6, 5, 20, and 19 (Acero et al., 2011; Aedo & Pizarro, 2004; Espinoza, 2010; Ibarraran & Rosas-Shady, 2006) were rated low.

Effect sizes from these eight studies were combined. The mean effect size and confidence intervals for each study are shown in the forest plot in Figure 5.8. The pooled estimate of effect is g=0.127 (95% CI [0.043, 0.21]).

OPEN IN VIEWER

5.8

FOREST PLOT OF MEAN EFFECTS ON EARNINGS

The overall mean effect size (g=0.127) suggests that, on average, young people who received a TVET intervention have higher monthly earnings than those who did not. The confidence interval for this point estimate (95% CI [0.043, 0.21]) is relatively precise and does not cross the line of ‘no effect.‘ However, the results of the statistical tests for homogeneity (Q = 25.5; df = 8; p = 0.00128; I² = 68.6%; tau² = 0.00815) suggest differential effects across studies.

We first explored whether the pattern of variation seen in the meta-analysis could be explained by differences in study quality (see the forest plot in Figure 5.9). It looks as if treatment effects for the low quality studies, g=0.15 (95% CI [0.01, 0.3]), are very similar to those rated medium, g=0.12 (95% CI [0.05, 0.18]). The slight differences in mean effects were statistically insignificant (Q_b = 0.204; p = 0.652).

OPEN IN VIEWER

5.9

FOREST PLOT OF MEAN EFFECTS ON EARNINGS (BY STUDY QUALITY)

5.3.1.1 Does participation in different types of TVET have different effects on young people's earnings?

The TVET interventions evaluated in the eight studies included in the meta-analysis presented in Figure 5.8 comprised three distinct types (see Table 5.2). Separate meta-analyses were conducted for studies comprised of (i) two-phase TVET interventions and (ii) other TVET models. These meta-analyses were then entered into a sub-group analysis. The mean effect size and confidence intervals for each study are shown in the forest plot in Figure 5.10.

OPEN IN VIEWER

5.10

FOREST PLOT OF MEAN EFFECTS ON EARNINGS (BY PROGRAMME TYPE)

For two-phase TVET interventions, the individual effect sizes were meta-analysed to produce a weighted average effect size of g=0.14 (95% CI [0.04, 0.23]). For other TVET models, a positive pooled estimate of effect was also observed (g=0.06), but the confidence intervals do not exclude a negative effect (95% CI [-0.15, 0.27]). Although the treatment effect appears to be slightly larger for two-phase models of TVET than for other models, the observed differences in mean effects were not statistically significant (Q_b = 0.397; p = 0.529).

5.3.1.2 Does participation in TVET have different effects on young people's earnings in the short- and medium-term?

Seven of the eight studies included in the meta-analysis for monthly earnings (see Figure 5.8) followed up participants once, after training had ended. The remaining study examined the impact of the intervention over time, measuring outcomes at 6, 12, and 18 months (Espinoza, 2010). Information about individual studies is presented in Appendix 8.9.

Separate meta-analyses were conducted for studies comprised of (i) short-term and (ii) medium-term follow-up periods. These meta-analyses were then entered into a sub-group analysis. The mean effect size and confidence intervals for each study are shown in the forest plot in Figure 5.11. Short-term treatment effects, g=0.22 (95% CI [-0.13, 0.58]), appeared to be greater than medium-term effects, g=0.14 (95% CI [0.05, 0.24]). However, the observed differences in mean effects were not statistically significant (Q_b = 0.186; p = 0.666).

OPEN IN VIEWER

5.11

FOREST PLOT OF MEAN EFFECTS ON EARNINGS (BY LENGTH OF FOLLOW-UP)

5.3.1.3 Does participation in TVET have different earnings effects on different sub-groups of young people?

Several studies examined whether some groups of young people benefited more than others, in terms of post-training earnings. Sixteen studies disaggregated the earnings effects of TVET on young people by gender, and seven studies estimated additional sub-group impacts (see Appendix 8.12). Information about individual studies is presented in Appendix 8.9.

Gender differences

Of these 16 studies, 6 were amenable to meta-analysis. The studies of interventions nos. 8, 12, and 14 (Attanasio et al., 2011; Card et al., 2011; Hicks et al., 2011) were rated medium quality. The studies of interventions nos. 2, 19, 9, and 11 (Aedo & Nuñez, 2004; Ibarraran & Rosas-Shady, 2006; Medina & Nuñez, 2005) were rated low. The mean effect size and confidence intervals for each study are shown in the forest plot in Figure 5.12.

OPEN IN VIEWER

5.12

FOREST PLOT OF MEAN EFFECTS ON EARNINGS (BY GENDER)

For female youth, the individual effect sizes were meta-analysed to produce a weighted average effect size of g=0.14 (95% CI [0.08, 0.21]). For male youth, the pooled estimate of effect was also positive (g=0.09) and again the confidence intervals do not include zero (95% CI [0.02, 0.16]). However, the observed differences in mean effects were not statistically significant (Q_b = 1.26; p = 0.262). When the sub-group analysis was re-run with only those studies rated medium, the differences remained insignificant (Q_b = 1.01; p = 0.315).

Other population sub-group differences

The study of intervention no. 12 (Card et al., 2011) was rated medium quality. The studies of interventions nos. 18, 20, 2, 20, 19, and 20 (Delajara et al., 2006; Díaz & Jaramillo, 2006; Elías et al., 2004; Espinoza, 2010; Ibarraran & Rosas-Shady, 2006; Jaramillo et al., 2007) were rated low. Three of the four studies that explored regional variation in impact observed some differences between sub-groups (Card et al., 2011; Elías et al., 2004; Ibarraran & Rosas-Shady, 2006). Two of the three studies exploring treatment effect heterogeneity by level of education found larger positive effects for more educated workers, compared to the less educated (Card et al., 2011; Delajara et al., 2006). Two evaluations of the ProJoven programme disagreed on the influence of poverty level on the earnings returns to young people. Espinoza (2010) found that ProJoven yields additional returns for those trainees in the lowest household income per capita quartile. In contrast, Jaramillo et al. (2007) found the programme to be ‘equity enhancing…as the evidence indicates similar returns for participants along varying poverty lines’ (p. 43). One of the two studies exploring treatment effects by age found slightly higher point estimates among the youngest trainees in their samples (Díaz & Jaramillo, 2006), whereas the other study (Card et al., 2011) found the estimated impacts on monthly earnings to be fairly similar for younger and older workers. Finally, Espinoza (2010) found that ProJoven yields additional returns for those trainees with no work experience prior to training.

5.3.2 Does participation in TVET have an effect on young people's self-employment earnings?

Two of the three studies that examined the impact of TVET on young people's earnings were amenable to meta-analysis. The studies of interventions nos. 8 and 14 (Attanasio et al., 2011; Hicks et al., 2011) were both rated medium quality. The mean effect size and confidence intervals for each study are shown in the forest plot in Figure 5.13.

OPEN IN VIEWER

5.13

FOREST PLOT OF MEAN EFFECTS ON SELF-EMPLOYMENT EARNINGS

The overall mean effect size (g=-0.025) suggests that, on average, young people who received a TVET intervention had lower self-employment earnings than those who did not, although this is a relatively imprecise estimate based on only two studies (95% CI [-0.11, 0.061]). The results of the statistical test for homogeneity was non-significant (Q = 0.206; df = 1; p = 0.65; I² = 0%; tau² = 0).

It was not possible to examine the variation in effect sizes (seen in Figure 5.13) by type of TVET or length of follow up due to an insufficient number of studies.

5.3.2.1 Does participation in TVET have different self-employment earnings effects on different sub-groups of young people?

Three studies examined whether the treatment effect on self-employment earnings varied across population groups. One study (Delajara et al., 2006) reported that, due to insufficient observations, many sub-groups could not be examined, and no clear pattern could be described. The remaining two studies examined gender differences only.

Gender differences

The studies of interventions nos. 8 and 14 (Attanasio et al., 2011; Hicks et al., 2011) were rated medium quality. The mean effect size and confidence intervals for each study are shown in the forest plot in Figure 5.14.

OPEN IN VIEWER

5.14

FOREST PLOT OF MEAN EFFECTS ON SELF-EMPLOYMENT EARNINGS (BY GENDER)

For female youth, the individual effect sizes were meta-analysed to produce a weighted average effect size of g=0.03 (95% CI [-0.08, 0.13]). For male youth, the pooled estimate of effect was negative (g=-0.06) although the confidence intervals cross zero (95% CI [-0.18, 0.05]). The observed differences in mean effects, however, were not statistically significant (Q_b = 1.27; p = 0.259).

5.3.3 Does participation in TVET have an effect on young people's hourly wages?

Five studies measured changes in the hourly wage received by workers. The study of intervention no. 12 (Card et al., 2011) was rated medium quality. The study of interventions nos. 20, 19, 17, and 20 (Díaz & Jaramillo, 2006; Ibarraran & Rosas-Shady, 2006; López-Acevedo, 2003; Ñopo et al., 2007) were rated low. The findings of the five studies were generally consistent, with four studies observing that young people who had participated in TVET experienced higher hourly rates of pay than those who had not participated.

5.4.1 Does participation in TVET have an effect on the number of weekly hours worked in paid employment by young people?

The data from five studies allowed calculation of effect sizes. The studies of interventions nos. 8, 12, and 14 (Attanasio et al., 2011; Card et al., 2011; Hicks et al., 2011) were rated medium quality. The studies of interventions nos. 6 and 19 (by Acero et al., 2011; Ibarraran & Rosas-Shady, 2006) were rated low.

OPEN IN VIEWER

5.15

FOREST PLOT OF MEAN EFFECTS ON HOURS WORKED

The forest plot in Figure 5.15 shows the mean effect size and the confidence intervals for each study. The majority of individual studies show a positive effect on the number of weekly hours worked. When the studies were pooled, the effect size favours the intervention (g= 0.043). However, as the confidence interval crosses zero (95% CI [-0.017, 0.104]) the result does not exclude a possible negative effect. Although the results of the statistical tests for homogeneity (Q = 1.8; df = 5; p = 0.876; I² = 0%; tau² = 0) suggest that variability in effect sizes between studies was not larger than expected from sampling error, visual indicators suggest that there is a degree of heterogeneity between studies.

We first explored whether the pattern of variation seen in the meta-analysis could be explained by differences in study quality (see the forest plot in Figure 5.16). Although treatment effects for the low quality studies, g=0.1 (95% CI [-0.01, 0.22]), appear to slightly larger than for those rated medium, g=0.02 (95% CI [-0.05, 0.09]), the observed differences in mean effects were statistically insignificant (Q_b = 1.41; p = 0.234).

OPEN IN VIEWER

5.16

FOREST PLOT OF MEAN EFFECTS ON HOURS WORKED (BY STUDY QUALITY)

Note: The majority of the studies not included in the meta-analysis because effect sizes could not be calculated also found that the number of weekly hours worked by young people increased as a result of participating in TVET (see Appendix 8.20).

5.4.1.1 Does participation in different types of TVET have different effects on the weekly hours worked by young people?

The TVET interventions evaluated in the five studies included in the meta-analysis presented in Figure 5.15 comprised three distinct types (see Table 5.2). Separate meta-analyses were conducted for studies comprised of (i) two-phase TVET interventions and (ii) other TVET models. These meta-analyses were then entered into a sub-group analysis. The mean effect size and confidence intervals for each study are shown in the forest plot in Figure 5.17.

OPEN IN VIEWER

5.17

FOREST PLOT OF MEAN EFFECTS ON HOURS WORKED (BY PROGRAMME TYPE)

For two-phase TVET interventions, the individual effect sizes were meta-analysed to produce a weighted average effect size of g=0.04 (95% CI [-0.02, 0.1]). For other TVET models, a pooled estimate of effect of similar magnitude was observed (g=0.07), and again the confidence intervals do not exclude a negative effect (95% CI [-0.14, 0.28]). The observed differences in mean effects were not statistically significant (Q_b = 0.0677; p = 0.795).

5.4.1.2 Does participation in TVET have different effects on the weekly hours worked by young people in the short- and medium-term?

All five studies included in the meta-analysis for weekly hours worked (see Figure 5.15) measured outcomes at a single point in time after training ended. Information about individual studies is presented in Appendix 8.9.

Separate meta-analyses were conducted for studies comprised of (i) short-term and (ii) medium-term follow-up periods. These meta-analyses were then entered into a sub-group analysis. The mean effect size and confidence intervals for each study are shown in the forest plot in Figure 5.18. Short-term treatment effects, g=0.07 (95% CI [-0.1, 0.24]), were very similar to medium-term effects, g=0.04 (95% CI [-0.03, 0.1]). The observed differences in mean effects were not statistically significant (Q_b = 0.109; p = 0.741).

OPEN IN VIEWER

5.18

FOREST PLOT OF MEAN EFFECTS ON HOURS WORKED (BY LENGTH OF FOLLOW-UP)

5.4.1.3 Does participation in TVET have different weekly hours effects on different sub-groups of young people?

Several studies examined whether the impacts of TVET on weekly hours were the same for different types of trainee. Five studies disaggregated the effects on weekly hours by gender, and two studies explored a number of additional subgroup impacts (see Appendix 8.12). Information about individual studies is presented in Appendix 8.9.

Gender differences

Three of the five studies had data that allowed effect size calculations. The studies evaluating interventions nos. 8 and 14 were rated medium quality (Attanasio et al., 2011; Hicks et al., 2011). The study of intervention no. 19 was rated low (Ibarraran & Rosas-Shady, 2006). The mean effect size and confidence intervals for each study are shown in the forest plot in Figure 5.19.

OPEN IN VIEWER

5.19

FOREST PLOT OF MEAN EFFECTS ON HOURS WORKED (BY GENDER)

For female youth, the individual effect sizes were meta-analysed to produce a weighted average effect size of g=0.16 (95% CI [0.04, 0.28]). For male youth, the pooled estimate of effect was negative (g=-0.09; 95% CI [-0.2, 0.01]). The observed differences in mean effects were statistically significant (Q_b = 10.1; p = 0.00151). When the sub-group analysis was re-run with only those studies rated medium, the differences remained significant (Q_b = 6.28; p = 0.0122).

Other population sub-group differences

Two studies estimated additional sub-group impacts of TVET on the number of weekly hours worked by young people. Both studies were rated low. When splitting the sample by age, Díaz & Jaramillo (2006) did not observe a clear difference in favour of either group (16-20 years or 21-25 years). Ibarraran & Rosas-Shady (2006) examined a number of different sub-groups and found effects relatively evenly distributed across the sample.

5.4.2 Does participation in TVET have an effect on the number of weekly hours worked in self-employment by young people?

One medium quality study (Hicks et al., 2011) examined the impact that training (intervention no. 14) had on the number of hours worked in self-employment. It found that young people in the treatment group worked fewer hours than their control counterparts. A similar pattern was observed for males and females in the sample.

5.5.1 Does participation in TVET encourage job searching amongst young people?

One study examined the impact of participation in TVET (following receipt of a training voucher) on the length of time spent job searching. It found that young people in the intervention group spent less time on job search compared to their control counterparts, where this gap was especially pronounced among men (Hicks et al., 2011).

5.5.2 Does participation in TVET lead to further additional training?

One study examined whether recipients of training went on to participate in further training activities (López-Acevedo, 2003). It found that participants of TVET interventions were more likely to participate in future training than control group individuals who had received no training.

Employment

•

The overall mean effect of TVET on paid employment was positive and significant (g=0.134, 95% CI [0.024, 0.243]).

Formal employment

•

The overall mean effect of TVET on formal employment was positive and significant (g=0.199, 95% CI [0.055, 0.344]).

Monthly earnings

•

The overall mean effect of TVET on earnings was positive and significant (g=0.127, 95% CI [0.043, 0.21]).

Self-employment earnings

•

The overall mean effect of TVET on self-employment earnings was negative and non-significant (g=-0.025, 95% CI [-0.11, 0.061]).

Weekly hours worked in paid employment

•

The overall mean effect of TVET on number of weekly hours worked was positive, but non-significant (g=0.043, 95% CI [-0.017, 0.104]).

6.3.1 Implications for policy and practice

Overall, the findings from this review provide evidence that young people in LMICs gain some benefit from TVET interventions. Statistically, the effect size may be small, or even negligible, but even a small increase in the rate of paid employment can translate into thousands, if not tens or hundreds of thousands, of young people entering the labour market, where the programme is delivered on a large scale. Notwithstanding that the crucial issue will always be whether the net employment rate has increased, it is both important and worthwhile to continue to invest in TVET provision for youth in developing countries and encourage access. It would be premature, however, to recommend for, or against, the use of any of the interventions included in this review. Interventions that were classified as multi-component (i.e., two-phase sequential programmes combining classroom-based training with on-the-job practical experience) produced mean effects that were substantially similar to simpler interventions comprised of a single mode of TVET (e.g., technical education). Although multi-component programmes may be able to target several barriers to employment, such as low levels of vocational skills and lack of work experience, thereby increasing their likelihood of success, the potential benefits of more complex interventions may suffer where there are difficulties with implementation. In the absence, therefore, of evidence in support of one particular (possibly expensive) intervention, opting for the cheapest and/or most culturally acceptable models may be the best approach.

As the effects observed in this review are generally small and were difficult to detect, it is therefore of some importance that future programmes are evaluated rigorously and that policymakers think carefully about how to improve programmes to create larger effects on the outcomes. To build the evidence base further, many more of the TVET interventions currently in existence in developing countries need to be rigorously evaluated, and the results reported and disseminated efficiently. It is important that policymakers, donors, and other relevant stakeholders coordinate efforts to identify why such research is not being conducted and/or is not being disseminated in a way that can inform others. They should also lend their support, financial and otherwise, to the systematic design, implementation, and evaluation of the full range of TVET models in operation in LMICs, in accordance with clear hypotheses of change that make explicit assumptions about causal links, implementation, and contextual and external factors. Assessment of the evidence should be undertaken for each key hypothesis. Doing so is likely to aid understanding of how an intervention is making a difference (i.e., though which channels or mechanisms), how feasible it is to extrapolate it to other settings, and ultimately help improvement of TVET interventions for the benefit of young people in developing countries.

6.3.2 Implications for research

Given the relatively small number of studies that met the criteria for inclusion in this review, there is a clear need for additional research in this area. The methodological inconsistencies and weaknesses of the current evidence base, and specific knowledge gaps, suggest a number of future research priorities. Acting on these will require the various stakeholders engaged in TVET research taking a critical look at the barriers affecting research production and dissemination.