PROTOCOL: Vocational and business training to increase women's participation in higher skilled occupations in low- and middle-income countries: protocol for a systematic review

The Problem

Although women's employment possibilities have improved with the rise of globalization, women in low- and middle-income countries tend to be overrepresented in informal labour markets, work in precarious conditions, receive lower salaries than men, and have few opportunities for learning and advancement (Borges Månsson & Färnsveden, 2012). Duflo (2012) reports that “women are less likely to work, they earn less than men for similar work, and are more likely to be in poverty even when they work” (p. 1052). Women often perform jobs that have low skill requirements and frequently work in occupations that are highly feminized, tend to be less socially valued, and pay lower wages (Aedo & Walker, 2012; Altman, 2006; International Labour Organization [ILO], 2015; ILO, 2016).

A recent ILO report has documented the limited opportunities for women in the labour market (ILO, 2016). The report shows that women face higher unemployment and underemployment than men, are more often employed in the informal labour market and in family enterprises, and are overrepresented in lower skill sectors. For example, a greater proportion of women are employed in the services sector (61.5 per cent versus 42.6 per cent of men), where women are particularly overrepresented in feminized positions such as “clerical, services, and sales” and “elementary” occupations (ILO, 2016). Moreover, although men and women face equal rates of wage and salaried employment (around 52 per cent), men are more likely to own their own business than women (3.7 per cent of men versus 1.4 per cent of women). Meanwhile, and while data on informal sector employment is scant, women are ‘believed to constitute most of the informal workforce in the developing world' (UNGEI 2012). Work in the informal sector is characterised by low pay and low productivity (ILO, 2016). Women working in informal employment do not gain access to social protection, such as pensions, and this may contribute to the fact that 71.8 per cent of all employed women do not have any type of maternity protection (ILO, 2016).

A range of factors contribute to the high proportion of unemployed and underemployed women. These factors include cultural norms regarding the place of women in employment, the role of women in domestic and care work, and the lack of adequate job market opportunities. Women all over the world spend a disproportionate amount of time doing domestic and care work. This time commitment is even higher in low- and middle-income countries, where the division of domestic labor often follows traditional patterns and women assume most, if not all family responsibilities (ILO, 2009). Women may also have a preference for jobs that are compatible with their domestic responsibilities, such as part-time and flexible jobs, both of which are scarce in low- and middle-income countries (ILO, 2009). However, even when part-time and flexible job opportunities exist, allowing women to combine work and family responsibilities, there is evidence that these jobs do not constitute ‘a path to decent work' (ILO, 2005). This lack of opportunities contributes to the choice of women to remain self-employed in small-scale enterprises or in domestic and care work, as opposed to pursuing professional careers in higher skilled occupations (ILO, 2009; ILO, 2012).

Evidence also suggests that lack of pre-service and in-service training opportunities leads many women to seek employment in low-skilled areas and in highly feminized occupations (ILO, 2016). The opportunities for women in other occupations are limited by the domestic obligations and limited skill sets of many women in developing countries (Katz, 2008; Mansson & Farnsveden, 2012). As a result, many young women seek employment in the informal sector, leaving them only with low-skilled employment characterized by minimum income potential, long working hours, and unequal power relationships, which may lead to exploitation. Limited data are available on informal labour markets in low- and middle-income countries. Nonetheless, evidence suggests that women constitute the large majority of the informal workforce in developing countries (UNGEI 2012).

The Intervention

National governments and development agencies have created a range of vocational and business training programs that aim to increase the participation of women in higher skilled and more secure, formal occupations. Such programs typically focus on improving the skills of women, foster entrepreneurship to expand employment, increase earning opportunities; and ultimately, reducing poverty (Blattman & Ralston, 2015; McKenzie & Woodruff, 2012), as discussed in more detail below.

Vocational training programs often target low-income, unemployed or underemployed individuals who have already left the formal schooling system, but they can also be part of the formal education system. These programs typically include preparing participants for jobs that are related to a specific occupation or trade through the acquisition of knowledge, practical cognitive and non-cognitive skills, and changing attitudes. Trainings include courses in administrative occupations (e.g., marketing, secretarial work, sales), manual occupations (e.g., electrician, cooking assistant), or fairly skilled occupations (e.g., account assistant, information technology (IT) specialist).

Many vocational training programs also combine the development of specific occupational skills with strategies to facilitate access to job opportunities, for instance, through internships, on-the-job training or by actively connecting participants with potential employers. For example, a representative program named “Jovenes en Accion” from Colombia provided three months of in-classroom and three months of on-the-job training to young people between the ages of 18 and 25 in the two lowest socioeconomic strata of the population (Attanasio, Kugler, & Meghir, 2008). Similarly, the Peruvian youth labour training program named “PROJoven” offered poor youths three months of training and a three-month internship with a local firm for beneficiaries who successfully completed their coursework (Ñopo, Robles, & Saavedra, 2007).

Additionally, some vocational training programs include the development of life skills among program participants. Program content ranges from specific job related life skills, such as preparing a Curriculum Vitae and interpersonal relationships to broader skills, such as those related to reproductive health and household economics. For example, Gap Inc's Personal Advancement and Career Enhancement program focuses on improving non-cognitive skills through learning modules that emphasize communication, problem solving and decision making, and time and stress management, among other skills (Gap Inc, 2015).

Business training programs provide active support to the poor to try to improve the performance of small and medium enterprises around the world (McKenzie & Woodruff, 2013). These programs attempt to offer skills in the form of business management training and assistance to small and medium enterprises. Business training may be offered by governments, microfinance organizations and non-governmental organizations. The length of these programs range from as short as two days to several months, and they are commonly offered to groups, although some programs also provide additional one-on-one follow-up training.

The majority of these trainings focus on general business skills that could be applied to most businesses (e.g., keeping business records and encouraging small business owners to separate household and business finances), while fewer focus on technical knowledge or sector-specific content or on trying to change entrepreneurial attitudes or aspirations. Several programs focus specifically on increasing women's understanding of the value chain for their product, sources of raw material and access to markets. For example, a PRASAC program in Cambodia helped weavers improve their businesses by identifying and building relations with suppliers, wholesalers, retailers, and designers (GTZ, 2003).

Finally, some business training programs are offered with additional incentives. These incentives could include asset or in-kind transfers, such as livestock or inventory, capital as a small cash loan or grant, or saving accounts. For example, a program implemented by BRAC Bangladesh named the Targeting the Ultra Poor (TUP) program offered a variety of business activities that ranged from livestock rearing to small retail operations to very poor and rural woman in selected rural communities. The program was combined with complementary and intensive asset-specific training and regular follow-up visits by asset specialists (Bandeira, Burgess, Gulesci, Rasul, & Sulaiman, 2013).

Both vocational and business training programs typically target low-income, unemployed or underemployed individuals, but there are also a range of programs that focus explicitly on building the skills of women. For instance the Jordan New Opportunities for Women program focuses on providing opportunities to young, female college graduates in Jordan to find work (Groh, Krishnan, McKenzie, & Vishwanath, 2012), while the Women for Women International Social Protection and Cash Transfer program in Nigeria offers biweekly meetings with business, vocational and life-skills training, including lessons in health awareness, decision making, negotiation, and civic participation (Mcilvaine & Oser, 2014). Both programs focus on women exclusively.

Vocational and business training programs that target women in particular may also include additional components to address barriers to women's participation in either the training itself or broader barriers to their participation in the labour market. Such components may include the provision of basic stipends to women to participate in the training and additional support for women with children (such as child care facilities or stipends). They may also include negotiating with women's families for permission to participate in the training (Ñopo, Robles, & Saavedra, 2007; Revenga, Riboud and Tan, 1994; Aedo and Nuñez 2004).

How the Intervention Might Work

Vocational or business training programs aim to improve outcomes such as women's labour market outcomes, income and empowerment by developing women's knowledge, skills and opportunities, and by doing so increasing women's participation in higher-skilled occupations (Revenga & Shetty, 2012). We developed a theory of change for both vocational training and business training programs, outlining how these programs might work in improving outcomes. They map out the causal chain between interventions, outputs, intermediate outcomes, and final outcomes, as well as the assumptions underlying the theory of change (White, 2009). Using such program theories help provide a framework for the analysis in the review and determine relevant outcomes for the systematic review. We outline the theories of change for each program area below.

Theory of Change for Vocational Training Programs Targeting Women

Figure 1 provides a theory of change for how vocational training programs may improve women's access to higher skilled, higher valued occupations and improve final outcomes such as income and empowerment. First, vocational programs can provide training that directly aims to increase women's skills (e.g., marketing, sales, account assistant, IT specialist), facilitate access to job opportunities (e.g., by on-the-job training or apprenticeship), or equip women with “life skills” to improve aspects of life such as interpersonal relationships and reproductive health. ³ Second, vocational training can result in improvements in women's employability by increasing women's occupational knowledge and vocational skills, women's knowledge of the labour market (e.g., job search skills, career management skills), and women's life skills such as attitudes toward work, motivation, self-esteem, career aspirations, and strategies to balance job and domestic responsibilities.

OPEN IN VIEWER

1

Theory of Change for Vocational Programs Targeting Women

In turn, these improvements in women's skills and employability may result in increases in women's participation in higher skilled, higher valued occupations. Access to higher skilled, higher valued occupations can in turn increase women's income and productivity; improve women's working conditions (such as safety, stability, social security) and opportunities for advancement; and reduce the degree of occupational segregation by improving access to occupations traditionally dominated by men. ⁴ Finally, the theory of change indicates that access to higher skilled, higher valued occupations may lead to an improvement in women's economic empowerment, such as women's control over earnings and decision making at home and at work.

Theory of Change for Business Training Programs Targeting Women

Figure 2 provides a theory of change for how business programs can contribute to increasing women's participation in higher skilled, higher valued occupations, and by doing so improve final outcomes such as income and empowerment. Business training programs typically provide training to equip women with business and life skills; provide assistance to small or medium enterprises; provide capital in the form of assets, cash, loans, or saving accounts and facilitate access to markets. Providing these inputs can result in improvements in a range of intermediate outcomes associated with entrepreneurship, such as business knowledge and skills (generic business skills and specific skills, such as knowledge about value chains), and their “life skills” such as attitudes toward work, motivation, self-esteem, career aspirations, and strategies to balance job and domestic responsibilities. These changes may in turn increase the likelihood that women start their own business, and/ or improve their business performance through the application of better business practices. Further, these improvements can result in increases in income (as measured by earnings, salaries, wages, profits, revenue, or productivity), improvements in working conditions (stability, social security, opportunity for advancement) and reduce the degree of occupational segregation by improving access to occupations traditionally dominated by men. Finally, changes in intermediate and final outcomes could lead to an improvement in women's economic empowerment, such as women's control over earnings and decision making at home and at work.

OPEN IN VIEWER

2

Theory of Change for Business Programs Targeting Women

Barriers and facilitators: The validity of the theory of change relies on a number of assumptions. If these assumptions do not hold the causal pathways in the theories of change may break down. One key assumption is that women are able to participate in training programs. But several barriers may limit women's participation in vocational or business training. Women require freedom of movement to participate in either vocational or business trainings, however restrictions on women's movement outside the house or on their participation in trainings by other household members may prevent this. Similarly, women may spend too much time on domestic chores or care work to allow them to participate in trainings, particularly if there is no access to child care or additional stipends for child care.

Other critical assumptions are related to the quality of the training, legislative conditions, and access to credit. If the training is of insufficient quality it may not develop women's knowledge, skills or entrepreneurship. Effective training requires high-quality trainers and relevant curricula with tailored content to teach women vocational or business skills. Moreover, legislative conditions may need to be in place to facilitate gains in women's participation in higher-skilled occupations. For example, women often require access to formal credit to benefit from business training because without access to credit, it is may be challenging to set up a small business. ⁵ Even if women are able to participate in training and develop their skills, women with children may require access to affordable child care facilities for the training to translate into participation in higher-skilled occupations. Finally, policies in the workplace, including those relating to discrimination, harassment and job security may also influence the extent to which programs succeed in changing final outcomes.

The barriers and facilitators are linked to various key stakeholders and different types of interventions may be effective depending on the barrier(s) that are most relevant in a specific context. For example, if discrimination in the workplace is a major barrier, vocational training may not be effective if it does not address women's ability to negotiate discrimination against women in the workplace.

A variety of market failures could be linked to the funding of vocational and business training programs. These market failures are associated with the question “if individuals can achieve high returns to participation in vocational and business training, why do these individuals not participate in these programs instead of waiting for the government to provide them?”. Market failures that prevent participation in the absence of government-funded training could include missing credit markets, information failures, and insurance market failures (Almeida et al., 2012). Missing credits markets could lead to an underinvestment of trainees in business training because of a lack of access to credit. Information failures could lead to a lack of knowledge about the availability of vocational or business training, which could in turn result in a lower participation in these training. Insurance market failures may lead to underinvestment in training when the returns to vocational and business training are uncertain (Almeida et al., 2012).

Why It Is Important to Do the Review

Primary Research Question

1.

What are the effects of vocational and business training programs on women's participation in higher skilled, higher valued occupations, women's income, working conditions, societal worth, and economic empowerment?

Secondary Research Question

2.

What are the barriers to, and facilitators of, the effectiveness of vocational and business training programs?

Criteria for Including and Excluding Studies

Types of Study Designs

The primary research question on the effectiveness of interventions (question 1) will be addressed using quantitative experimental or quasi-experimental studies. We will include both experimental studies that use random assignment to the intervention and quasi-experimental designs with non-random assignment. To be included, quasi-experimental studies need to use either known allocation rules (such as assignment on the basis of a threshold on a continuous variable or geographic variation in the assignment of the program) or include pre-and post-test measures of the outcome variable of interest. Knowledge about allocation rules may enable the use of regression discontinuity designs or natural experiments to determine the impact of the program, while the inclusion of pre-test and post-test measures of outcome variables will enable researchers to use methods that control for selection-bias such as interrupted time series models, difference-in-difference regression analysis, statistical matching (for example propensity score matching or covariate matching), instrumental variables, or Heckman selection models.

We will only include only quasi-experimental studies that use methods that can credibly address selection-bias. Cook et al. (2008) and Shadish (2011) demonstrate that quasi-experimental studies can address concerns with selection-bias, but only under certain conditions. Controlling for selection-bias from covariates and including pre-test measures of outcome variables are particularly effective in reducing selection bias (Steiner et al., 2010; Shadish, 2011). We will therefore only include quasi-experimental studies that either include a baseline measurement of the outcome of interest or other relevant confounding factors or where allocation rules enable the use of regression discontinuity designs or analyses on the basis of natural experiments. We will exclude quasi-experimental studies that do not include a baseline measure of the outcome of interest and do not enable the use of either regression discontinuity designs or analyses on the basis of natural experiments.

To address the secondary research questions (questions 2 and 3), we will include a broader range of evidence with relevant details on intervention design, implementation and context. To include these studies and documents we will use a similar approach as Snilstveit et al. (2015) in their systematic review regarding the effects of education programs. In line with that review, included studies and documents will focus on one of the interventions that was evaluated in the primary research question, and have to meet at least one of the following three criteria: 1.

A qualitative study collecting primary data using qualitative methods of data collection and analysis, and reporting some information on all of the following: 1) the research question, 2) procedures for collecting data and 3) sampling and recruitment.

Types of Participants and Settings

We will include studies that focus on interventions that include women who are 18 years or older in low- and middle-income countries, as defined by the World Bank. ⁷ In cases where the intervention participants are not exclusively women, studies will be eligible only if effects on women are assessed separately from those on men. We will include studies about the effects of vocational and business training regardless of the employment status or skill level of women at the time of the intervention.

Types of Interventions

The interventions included in this review will be vocational and business training programs that aim to increase the level of skills of the disadvantaged, unemployed, or underemployed; foster entrepreneurship to expand employment; and increase income prospects of women.

To identify interventions that focus on high skill occupations for women we take Acemoglu & Aedo's (2011) classification of occupations according to skill requirements as a starting point. They define three broad occupational categories: 1.

Low Skills occupations, that include: routine manual occupations (that require working at a pace determined by speed of equipment; spending time making repetitive motions); as well as routine cognitive occupations (that require repeating the same tasks; being exact or accurate).

We slightly adjust this definition of high skill occupations for the purpose of this review. While the definition of Acemoglu & Aedo (2011) is useful, we can expect a degree of variation as to what constitutes a high, medium or low-skill occupation in the context of low and middle-income countries. For example, a bank teller would constitute a low-value, low-skill, routine cognitive job in a high-income country, but in a low- or middle-income country, a job as a bank teller would qualify as a high-skill occupation. For this reason, we will include studies that focus on programs which provide women with either high skills or medium skills as defined by Acemoglu & Aedo's framework.

Vocational training programs could also include one or more of the following components (in addition to vocational training): •

Strategies to facilitate access to job opportunities (through internships, on-the-job training or by actively connecting participants with potential employers).

We will also include studies that focus on business training programs providing management training and assistance to small and medium enterprises. In addition to business training, these programs can include one or more of the following components: •

Strategies to facilitate access to markets and increase understanding of the value chain for the enterprise product.

We will only include evaluations of vocational or business training programs that include at least one of the components discussed above.

Excluded interventions

Several other criteria are required for interventions to be included in the review. We will exclude vocational and business training programs targeted exclusively at men. Interventions that train women to work in low-skill occupations fall outside the scope of this review. Thus, we will exclude studies that focus on interventions that aim to increase the labour market participation of women in low-skill occupations. We used a scoping review to further understand how low-skill occupations are defined within low- and middle-income countries. On the basis of the scoping review, we decided to exclude studies of programs that increase women's access to the domestic service or the agricultural labour market because our scoping review indicated that these sectors almost exclusively provide employment in routine manual jobs. We will also exclude those studies in this systematic review.

Some vocational training programs are offered within high schools or tertiary institutions. We will not include studies that focus on the latter types of vocational training programs because these programs are likely to influence outcomes through different mechanisms and will thus require a different theory of change.

Types of Comparison Conditions

Eligible comparison conditions will include no intervention, pipeline, or “business as usual.” In those studies that we include to address the secondary research questions, a comparison condition will not be necessary for the document to be included.

Types of Outcome Measures

To address research question 1 we will include studies that focus on a range of intermediate and/or final outcomes. Studies need to assess at least one of the outcomes outlined below.

Intermediate outcomes

Knowledge and Skills: Training programs aim to improve participants’ knowledge and skills. We will include any studies that measure women's business knowledge and skills; women's understanding of the value chain of their product/service; women's occupational knowledge and vocational skills and women's life skills. Studies that measure these outcomes through either tests, administrative or self-reported data will all be included.

Business Practices: We will include any measure of the adoption of business practices that is associated with good business outcomes. These practices may include regular bookkeeping, keeping clear records, good planning, and other practices that are taught during business trainings. We will include studies that measure the adoption of these and other good business practices (as reported by the authors of the study) through surveys.

Employment (status, occupation): We include all studies that focus on a range of measures of women's employment status and type of occupation, including whether women are employed, the status of their employment (full-time/part-time, permanent/temporary, formal/ informal) and the type of occupation they are employed in.

Final outcomes

Income (Earnings, Wages, or Salaries): We include measures of income that women earn through paid work, defined as earnings, wages, or salaries. We will include studies that measure earnings or wages through administrative data as well as self-reported data in which women are asked about their earnings, wages, or salaries. We will include only studies that measure these incomes at the individual level to ensure that we can distinguish between income of men and women. We will not include studies that proxy for income by relying on expenditure or other consumption data unless those data are able to distinguish between the expenditure levels of men and women.

Revenue or Profits: We define revenue or profits as the total amount of income generated by the sale of goods or services from a business and the total amount of income generated by the sale of goods or services from a business minus the costs, respectively. We will include studies that measure revenue or profits through administrative and self-reported survey data. We will include only studies that report the revenues or profits of women or women-owned businesses specifically.

Working Conditions: Working conditions are comprised of “a broad range of topics and issues, from working time… to remuneration, as well as the physical conditions and mental demands that exist in the workplace” (ILO, n.d.). More specifically, working conditions may include factors such as safety, stability of employment, social security, opportunities for advancement; job permanence; career and employment security; health and well-being at the workplace; and work–life balance; as well as respectful treatment at work. We will include all studies that measure these factors through administrative or self-reported data.

Societal Worth: We define improved societal worth as women's access to occupations traditionally associated with men, which in turn might lead to a reduction in the degree of occupational segregation. Occupational segregation and the feminization of certain occupations may contribute to gender inequalities in earnings and in the societal value attributed to women's work (Borges, Mansson, & Farnsveden, 2012). Thus, improved societal worth may improve the contribution of women on the labour market. We will include studies that measure societal worth or occupational segregation through survey or administrative data.

Economic Empowerment: In line with Brody et al. (2015), we define economic empowerment as “the ability of women to access, own, and control resources.” Women's empowerment can be measured through survey data, for example, by asking women about their ownership of assets and land, division of domestic labour across men and women, and control over financial decision making (Brody et al., 2015). In contrast to Brody et al. (2015), we, however, do not consider degree of women's participation in paid employment as an economic empowerment indicator because we emphasize the importance of high-skilled labour.

Studies addressing questions 2 and 3: Outcome measures will not be considered to filter qualitative studies, which will serve to address the secondary research questions. From these documents we will include descriptive information about programme design and implementation, context and resources, as well as any findings addressing questions 2 and 3 on barriers and facilitators of intervention success or failure.

Other criteria for inclusion/exclusion

We will consider studies published in English or Spanish after 1990. We will not exclude studies based on publication status.

Search Strategy

We developed a comprehensive search strategy in consultation with information specialists. Our search strategy will enable us to identify published and unpublished literature by focusing on relevant academic and institutional databases, citation tracking, and snowballing of references. For searches of electronic databases we will use a detailed search string, whereas searches of grey literature and institutional websites will typically rely on more simple, tailored searches.

The search strings are designed to return studies that include at least one keyword in the following four themes: 1.

Participants: woman, female, wife, mother, gender, occupational stratification, empowerment businesswoman, woman industry.

To capture both quantitative and qualitative literature relevant to the research questions, we will not include search strings for study design, comparison condition, and outcome measures. Using these criteria in the search strategy would exclude relevant qualitative studies in addition to quantitative and mixed-methods studies that omit this information from the title and abstract.

Appendix 1 provides an example of a search string. Each search string will be adapted to fit the syntax of the database host in order to utilize Boolean operators (AND/OR), wildcards, truncation, and other database search features. Below we list the sources covered by our search.

Electronic Sources

Comprehensive database searches will include the following paid-access and free-access electronic databases:

Paid-Access Databases

1.

Web of Science

Open-Access Databases

1.

Ideas-REPEC

Hand search: Selected Journals

We will hand search the most recent issues of the journals listed below for any papers not yet indexed in electronic databases. 1.

American Economic Review

Grey Literature

Grey literature searches will include a review of institutional websites and research funders. In addition to the following institutional websites, key papers will be examined in search of other relevant institutional sources.

Citation tracking

Forward and backward snowballing of the references of key papers will provide additional studies for review that may not be found in database searches. Citation searches will be conducted in Google Scholar, Scopus, and Web of Science. This set of key papers will include: anchor papers identified by the authors (listed below), key papers identified by an advisory group of reviewers of the scoping study we conducted by Chinen et al. (2015), key papers identified by Advisory Group, key papers identified by external funder (3ie), studies that pass the inclusion criteria and additional key papers identified from institutional websites.

Table 1 presents the list of key papers identified by authors and reviewers to be used in citation searches.

Targeted Search for Addressing Questions 2 and 3

After we have identified studies for inclusion to answer review question 1, we will undertake targeted searching for qualitative studies, as well as process evaluations, implementation documents and other relevant documents for those interventions evaluated in the included studies. We will conduct citation tracking of included studies to identify any relevant papers and conduct internet and database searches using the names of programs from included studies. We will also search databases of project documents and websites of implementing agencies to identify project documents. The targeted search will involve the following steps: 1.

Advisory group: contact advisory group members from UN Women, the Canadian Department of Foreign Affairs, Trade and Development, the Inter-American Development Bank, 3ie, the World Bank, BRAC and other key stakeholders to request project documentation about vocational and business training programs that were evaluated under the first research question.

Screening

Screening will take place in two phases: first on the basis of titles and abstracts and then on the basis of full texts.

Screening Phase 1

Titles and abstracts of the search results of database, grey-literature, and citation searches will be uploaded to the EPPI-Reviewer 4 online system (http://eppi.ioe.ac.uk/eppireviewer4). Duplicate studies will be identified and removed automatically using the duplicates feature built into EPPI-Reviewer 4. The inclusion criteria will be piloted by two reviewers; a sample of the studies will be independently assessed for relevance by each reviewer (see Appendix 2). Screening results will be compared, and the inclusion criteria will be adapted as needed. All titles and abstracts will then be screened against the inclusion criteria outlined above, retaining potentially relevant studies for screening at full text (phase 2).

Screening Phase 2

In the second phase, we will review the full text of all studies that pass Phase 1 screening. Multiple reviewers will independently assess studies against the inclusion criteria and studies meeting all criteria will be retained for inclusion in the review.

Studies that pass Phase 2, will be screened for the secondary research question based on the exclusion/inclusion criteria outlined above. Therefore, studies may pertain to only research question 1, or to both research questions 1 and 2. For example, a study that only uses quantitative methods may pertain only to research question 1. While a study that uses mixed methods –with a focus on measuring the impact of vocational training on women's labour market outcomes and on the barriers or facilitators that played a role- may pertain to both research questions 1 and 2.

Critical appraisal

Quantitative studies addressing question 1: Risk of Bias Assessment

We will determine the rigor of the quantitative studies using an adaptation of a set of criteria, to assess risk of bias in experimental and quasi-experimental studies (Hombrados & Waddington, 2012). Two researchers with expertise in experimental and quasi-experimental methods will assess the risk of the following biases: 1.

Selection bias and confounding, based on quality of identification strategy to determine causal effects and assessment of equivalence across the beneficiaries and non-beneficiaries.

The risk of bias assessment will not be used to exclude studies. Instead, we will rely on the risk of bias assessment to determine whether studies can be considered low, medium, or high risk of bias in the four risk of bias domains (selection bias and confounding, performance bias, outcome and analysis reporting bias, and other biases). Appendix 5 presents the risk of bias assessment tool.

Studies addressing questions 2 and 3: Critical Appraisal

To address questions 2 and 3, we will include different types of documents as outlined in the inclusion criteria. Critical appraisal of qualitative studies, process evaluations and project documents (including a broader range of evidence) can be complicated particularly as there is a lack of existing tools and criteria for quality (Noyes et al., 2011). Following Snilstveit et al. 2015, we adopt different approaches to appraise these types of studies/ documents, as outlined below.

We will assess the quality of the first group, included qualitative studies, using and adaptation of the nine-item Qualitative Research Checklist developed by the Critical Appraisal Skills Programme (CASP), presented in Table 2 below, making judgments on the adequacy of stated aims, the data collection methods, the analysis, the ethical considerations, and the conclusions drawn (Critical Appraisal Skills Programme, 2013). For each item, two researchers will determine whether the study had adequately met the item and gave “yes,” no,” or “can't tell” responses. If researchers disagree, they will discuss the item to reach consensus. We will rate studies that score 0–2 “no” or “can't tell” responses as with minor limitations, studies that score 3–5 “no” or “can't tell” responses as with medium limitations, and studies that score 6–9 “no” or “can't tell” responses as major limitations.

In the case of process evaluations, although there are no commonly used critical appraisal tools, assessment of methods of data collection should be considered. To achieve this goal, we propose to use the adapted version of the CASP checklist presented in Table 2.

Finally, project documents are typically descriptive and do not include much analysis of primary evidence. Therefore we will not formally appraise the quality of such documents. Rather we will assess the relevance of the documents for understanding program design and implementation and assess if they provide factual information about interventions. If several such sources are available we will extract data from all sources for purposes of triangulation.

Data Extraction

Two team members with expertise in quantitative research will work independently to extract information from each quantitative study included in the review. The data will include details of study design, intervention, comparison group, outcome, and data for effect-size calculation (more details below). Similarly, two team members with expertise in qualitative research will extract data from studies included to address questions 2 and 3. Data to be extracted will include descriptive details about intervention design, and more analytical findings on process, implementation and context. The draft data extraction forms are provided in Appendix 3 and 4. Any disagreements will be resolved through discussion.

Measures of Treatment Effects

We will extract information from each quantitative study to allow for the estimation of standardized effect sizes, and associated standard errors and 95 per cent confidence intervals. Where possible, we will use natural units, such as percentage changes in income and percentage point increases in employment and synthesize studies using those effect sizes. However, this approach will only be feasible when studies use the same outcome measures, which is not likely to be the case for all studies.

In all other cases, we will calculate two types of effect sizes. We will calculate the Hedges' g sample-size-corrected standardized mean differences (SMDs) for continuous outcome variables, which measure the effect size in units of standard deviation of the outcome variable. We will calculate odds ratios (ORs) for binary outcome variables.

We will calculate standardized mean differences (Cohen's d) by dividing the mean difference with the pooled standard deviation by applying the formula in Equation 1:

SMD = y t − y c s p

1

SMD refers to the standardized mean differences, Y_t refers to the outcome for the treatment group, Y_c refers to the outcome for the comparison group, and S_p refers to the pooled standard deviation.

The pooled standard deviation S_p can be calculated by relying on the formulas in Equations 2 and 3:

S p = ( ( S D y 2 ) * ( n t + n c − 2 ) ) − ( β 2 * ( n t * n c ) n t + n c ) n t + n c

2

S p = ( n t − 1 ) * s t 2 + ( n c − 1 ) * s c 2 n t + n c − 2

3

We will use Equation 2 for regression studies with a continuous dependent variable. In this equation, SD_y refers to the standard deviation for the point estimate from the regression, n_t refers to the sample size for the treatment group, n_c refers to the sample size for the control group, and β refers to the point estimate. We will use Equation 3 when there is information about the standard deviation for the treatment group and the control group.

We will correct the standardized mean difference for small sample size bias by relying on formula 4, which transforms Cohen's d to Hedges' g.

SMD corrected = SMD uncorrected * ( 1 − 3 4 * ( n t + n c − 2 ) − 1 )

4

We will rely on Equation 5 to estimate the standard error of the standardized mean difference:

SE = n t + n c n t * n c + S M D 2 2 * ( n c + n t )

5

Where possible, we will calculate odd ratios by relying on 2 by 2 contingency tables (Lipsey & Wilson, 2001), as described in Figure 3.

OPEN IN VIEWER

3

Estimation of Odds-Ratios

We will calculate the odds-ratio using Equation 6, where

E S ¯

E S ¯ = a d b c

6

Where studies use linear probability models, we will assume linearity in the estimation of standardized effect sizes as in Brody et al. (2015). This assumption will allow us to use the same 2 by 2 contingency table on the basis of the baseline means and impact estimates from the linear probability model. For example, if we observe a mean baseline value for the comparison group of 0.097 and an effect size of 5.1 percentage points, then we will assume that the follow-up value for the treatment group would be 0.097+0.051=0.148, and we will assume that the follow-up value for the comparison group will be 0.097.

We will use adjusted standard errors for those studies that use outcome variables that are clustered at a higher level of aggregation than the individual or household level but do not take this into consideration in the estimation of the standard errors and confidence intervals. For these studies, we will apply corrections to the standard errors and confidence intervals using the variance inflation factor (Higgins & Green, 2011):

S E c o r r e c t e d = S E u n c o r r e c t e d × ( 1 + ( m − 1 ) × I C C )

7

Here, m is the number of observations per cluster, and ICC is the intracluster correlation coefficient. We will estimate an intraclass correlation for each of the outcome measures from data of a study that can be considered representative for our sample of included quantitative studies and for which we are able to obtain data on the outcome measures. We will use a range of plausible ICCs to examine the effects of these ranges on the outcome of our synthesis.

Missing Data

In cases where it is not feasible to estimate the effect size because of missing information, we will contact the authors of the primary studies to request the missing information that is required to calculate the effect sizes. In those cases in which we are not able to retrieve missing data, we will extract or impute effect sizes and associated standard errors based on commonly reported statistics such as the t or F statistic or exact p or z-values using David Wilson's practical meta-analysis effect-size calculator. Where studies do not report sample sizes for the treatment and the control or comparison group, we will assume equal sample sizes across the groups.

Methods for Handling Dependent Effect Sizes

We will include only one effect size per study in any single meta-analysis. ⁹ Where studies report more than one effect size on the basis of different statistical methods, we will select the effect size with the lowest risk of bias. We will use the effect size from the most recent publication when several publications reporting on the same study report different effect sizes. We will check working paper versions for data on additional outcomes. Where studies present several impact estimates for different outcomes variables that measure the same construct, we will use a sample-size weighted average to create a “synthetic effect size.” In cases where more than one study use the same data set to measure an outcome variable, we will extract the effect size from the study with the lowest risk of bias. In cases where one study measures the same outcome variable at different periods in time, we will extract data on all time periods, but include the outcome measure that is closest to the time period of the measurement in other studies included in the same meta-analysis. Estimates from other time points will be discussed narratively. Finally, if studies include more than one treatment arm, we will include the effect size from the treatment arm that is most similar to the other programs that are included in the meta-analysis as recommended in Borenstein et al. (2009).

Methods of synthesis

Review question 1: Statistical analysis

If meta-analysis is feasible, we will address question 1 by synthesising studies using an inverse-variance, random effects model due to the anticipated heterogeneity in the included studies. We will only combine estimates from studies assessing the effect of comparable programs on similar outcome constructs (Wilson et al., 2011). Thus we will conduct separate meta-analyses for vocational and business training programs, running different analyses for different outcome constructs.

We will examine the heterogeneity of the effect sizes visually and by estimating the I-squared and Q as well as tau-squared (Borenstein et al., 2009). If we have sufficient studies we will investigate factors explaining heterogeneity by using inverse-variance weighted meta-regressions and stratified meta-analysis according to the following contextual and methodological moderator variables: (1) type of intervention component, (2) intervention components, (3) geography, (4) age of beneficiaries, (5) type of treatment effect estimate (eg: intention to treat, treatment effect on the treated etc.) (6) time to follow-up.

We will also examine the relative effectiveness of studies that evaluate different types of interventions through stratified meta-analysis. For this purpose we will use a similar approach as Fu et al. (2011), and Brody et al. (2015) who compare point estimates and then test differences in effect sizes between programs with different intervention constructs formally if the overlap in effect-sizes is minor or non-existent.

We will perform sensitivity analysis for two methodological effect size moderators: •

Risk of bias status for each risk of bias category.

We will start our analysis with separate meta-analyses of randomized controlled trials and quasi-experimental evaluations for determining the effects of vocational and business training. Then we will use an iterative approach to determine the potential bias from pooling randomized controlled trials and quasi-experimental evaluations and studies with low, medium, and high risk of bias for each of the types of bias we assessed in our risk of bias assessment. We will present the results of each meta-analysis, highlighting any differences in results between studies with different levels of risk of bias. This approach is in line with the approach used in a recent systematic review on the effects of women's self-help groups on women's empowerment (Brody et al., 2015).

Publication Bias

We will use various methods to determine the potential for publication bias. First, we will assess the potential for publication bias using funnel plots. Second, we will conduct Egger's test to determine the potential for publication bias in studies that focus on vocational or business training.

Qualitative synthesis

Techniques for integrating, comparing and synthesising qualitative literature have been variously described as “meta-ethnography” (Harden, 2010), “narrative synthesis” (Snilstveit, Oliver, & Vojtkova, 2012), and, more recently, as “meta-synthesis” (Sandelowski, Docherty, & Emden, 1997; Walsh and Downe 2005; Zimmer 2006).

In spite of the different terms, authors agree that unlike what happens in quantitative meta-analysis, where the aim is to aggregate the findings from multiple studies, in the case of qualitative synthesis, the main aim is not “aggregation” but rather “interpretation.” Rather than pooling findings from studies, key concepts are translated within and across studies, and the synthesis product is a new interpretation (Harden, 2010).

A specific challenge faced in this process is the highly contextual nature of qualitative findings, which means that findings or interpretations from a study cannot be merely extracted and compared but need to be considered closely to the context in which they were generated. Equally challenging is the fact that qualitative interpretations are even more mediated than quantitative findings by the epistemological and theoretical perspectives used by the researchers. All of this means that qualitative synthesis processes need to endeavour strongly to the characterization of included studies before moving on to develop middle-range and generalizing theories about their findings (Zimmer, 2006).

Thus the synthesis of included qualitative studies will proceed along the following stages: 1.

Characterization: During this stage, we will code and describe included studies along various items, including methodology (types of methods used, sample characteristics), epistemological perspective, theoretical framing, context, and thematic focus. This phase is important in order to appropriately locate studies within their empirical, theoretical, and epistemological context.

Integrated synthesis (review questions 1, 2 and 3)

Once we have completed the syntheses above we will provide an integrated synthesis of the findings. We will use the program theories provided above to present the findings from the different syntheses with the aim of providing an integrated narrative synthesis addressing the objectives of the review. We will use a summary of findings tables following the GRADE (Schünemann et al., 2011) and CerQual (Lewin et al., 2015) approaches to facilitate transparent and systematic presentation of our findings.

We will develop a series of matrices to identify the features of interventions and contexts that appear to act as barriers and facilitators of the interventions' effectiveness. We will initially conduct the cross-case analysis by intervention type, but we will also attempt to identify any overarching themes across intervention types. Finally, we will provide an integrated synthesis of the findings addressing all review questions.