PROTOCOL: Interventions for improving learning outcomes and access to education in low- and middle-income countries: a systematic review

Child-level interventions

We define child-level interventions, or ‘child settings’, as those interventions targeting children directly, focusing on improving their ability to benefit from schooling or their incentives and motivation for investing time and resources in their own education. Interventions falling under this category include the following: 1.

School feeding programmes typically aim to improve the general health of children, provide a safety net for vulnerable and food insecure families, and improve children's ability to learn (Jomaa et al., 2011). Such interventions fall into two categories: the traditional school feeding programme, where children are provided with meals in school, and take-home ration programmes where children are provided with food in school which they can take home to their family (Lawson, 2012). In many cases the food provided is fortified or supplemented in order to give additional nutritional benefits (Jomaa et al., 2011). School feeding programmes such as these are often targeted towards families and communities that are food insecure or have low incomes. Food for school feeding programmes is procured in a variety of different ways, but recently the focus has been on using local produce.

School-level interventions

We define education interventions taking place at the school level as those initiatives aiming to improve the quality of the teaching and learning environment. They include interventions providing physical inputs, or changes in how teaching is delivered. Typically a programme may include a combination of some or all of these components. These programmes can be delivered by governments, non-governmental organisations, religious organisations or international organisations. The education interventions implemented at this level typically fall under the following categories: 1.

Instructional approach, content, time and organisation interventions include a broad range of programmes that aim to adapt or improve educational content, the methods by which it is taught, or the time available for instruction. It includes for instance interventions introducing a new curriculum, innovative or specialised methods such as computer-assisted learning and multi-grade teaching. For instance, computer-assisted learning or use of other technologies in the classroom have been widely implemented as a means of tailoring learning to students individual needs (Kremer, 2013). Interventions to introduce a multi-grade teaching approach involve a shift in instructional approach, curriculum and materials to suit settings in which two or more grades are combined (Little, 2004). Tailored reading programmes introduce innovative methods and new materials to the classroom setting (Abeberese, 2007). Remedial classes or tutoring target those children seen to be most in need with tailored additional content (Banerjee et al., 2007). An increase in instructional time aims to increase instructional contact time and ultimately, to improve learning outcomes (NECTL, 2000).

Teacher-level interventions

We define teacher-level interventions as those interventions targeting teachers directly. These interventions include those designed to hire additional teachers and increase teacher-student ratios. Other interventions are designed to provide teachers with new skills, provide performance-related incentives or increase accountability. Interventions falling under this category include the following: 1.

Interventions providing teacher incentives and promoting accountability seek to improve the working conditions in schools so that teachers are motivated to come to work and improve their performance. Such interventions take many forms, such as providing direct payments to teachers based on their attendance or based on the achievement of their students (Glewwe et al., 2008). For instance, a programme in India offered teachers a cash bonus linked to their pupils’ performance in independent tests (Muralidharan & Sundararaman, 2009). Similarly, a programme in Kenya offered primary-school teachers in-kind rewards based on pupils’ exam scores (Glewwe et al., 2010). Alternatively, some interventions do not provide monetary incentives but infrastructural ones such as improvements to school facilities and classroom learning materials (Guerreo et al., 2012). Other interventions may use monitoring in order to keep track of teachers performance. Such monitoring may be undertaken by school principals, external assessors, or community members (Guerreo et al., 2012).

System-level interventions

We define this category of interventions as those aiming to improve education through changes to the education system at either the community, local government and district/state or national level. The interventions taking place at this level are primarily related to the management, governance and financing of education. Because of the nature of these interventions, they are typically implemented by governments, although non-governmental organisations, religious organisations or international organisations may be involved in delivery. The education interventions implemented at this level typically fall under the categories outlined below (drawing on Glewwe and Kremer [2005]): 1.

Decentralisation and local community participation interventions have been implemented in response to perceived failures of centralised education systems. At the core of such initiatives is the decentralisation of decision making authority to local levels and greater involvement of communities in making decisions and monitoring service providers. The intervention components of such interventions vary, but two commonly used modalities include school-based management and community monitoring, as outlined below. a

. School-based management (SBM) interventions involve de-centralising authority to the school level to improve the quality of school administration and leadership. SBM programmes may involve handing decision-making (for example, on budget, staffing and curriculum development) over to teachers, parents, students or other community members (Barrera-Osorio, 2009). For example, the School Management Initiative in Hong Kong gave school committees authority over staffing and devising the curriculum, as well as some financial matters, aiming to create greater flexibility in school finance, increase accountability, and encourage collaborative decision making (ibid). Committees may also devise school improvement plans and receive funds to finance implementation of these plans. The Education Quality Improvement Project in Cambodia encouraged school committees to identify their school's needs, suggest improvements and then carry out reforms using cash grants from the Ministry of Education (WDR, 2004). These types of intervention may also foster greater accountability to parents or the community, increasing capacity to demand improved services, although they do not always include a participatory component.

Review of existing literature

There is a relatively large literature of experimental and quasi-experimental evaluations assessing the impact of interventions in the education sector in low- and middle-income countries. Several authors have reviewed this impact evaluation literature in order to draw wider conclusions on the effectiveness of education interventions (Petrosino et al., 2012; Baird et al., 2013; Morgan et al., 2012; Morgan et al., 2013; among others). A comprehensive review of existing systematic reviews identified 19 systematic reviews, 1 protocol, and 13 non-systematic reviews (see the Education Evidence Gap Map of systematic reviews here: http://gapmaps.3ieimpact.org/evidence-maps/education-evidence-gap-map), two of which were using meta-analysis. Less than half of the systematic reviews include meta-analysis (Baird et al., 2013; Kabeer et al., 2012; Kristjansson et al., 2009; Petrosino et al., 2012; Taylor-Robinson et al., 2012), and apart from Petrosino et al. (2012) these reviews focus on a relatively discrete set of interventions. Below we review these studies in brief.

The majority of existing reviews focus on interventions to improve school enrolment and attendance. For instance, the impact of CCTs on schooling outcomes has been examined by five systematic reviews (Baird et al., 2013; Bouillon and Tejerina, 2007; Kabeer et al., 2012; Petrosino et al., 2012; Yoong et al., 2012), three of which used meta-analysis (Baird et al., 2013; Kabeer et al., 2012; Petrosino et al., 2012), and one meta-analysis (Saavedra & Garcia, 2013). These reviews all find that CCTs in education contribute to improving enrolment and attendance, though the evidence base on the effects on learning outcomes is limited, with available studies suggesting at best small effects. Two systematic reviews examine the impact of UCTs (Yoong et al., 2012; Baird et al., 2013). Yoong et al. (2012) report positive effects on enrolment, but only when the transfer is received by women. Baird et al. (2013) report positive effects, both on enrolment and attendance, although the size of the effect is lower than for CCTs.

School vouchers are another popular intervention implemented to improve access to education, primarily by enabling children from low-income households to access private education. Three systematic reviews include studies assessing the effects of school vouchers (Bouillon & Tejerina, 2007; Morgan et al., 2013; Petrosino et al., 2012), one of which included meta-analysis (Petrosino et al., 2012). Both Bouillon and Tejerina (2007) and Morgan et al. (2013) found positive effects of vouchers on attendance and performance, and an increase in private school enrolment among the poorest income groups respectively, while Petrosino et al., (2012) found no effects of school vouchers.

Morgan et al. (2012) reviewed the evidence on interventions that reduce or eliminate schooling costs (fees and uniforms) and found that such interventions strongly increased school enrolment and positively affected other education and non-education outcomes. Petrosino et al. (2012), on the other hand, reported no effect for user fee elimination, but found positive effects for provision of free uniforms.

School feeding and school-based health interventions are implemented to improve both school attendance and learning outcomes, and several systematic reviews have assessed the evidence on such interventions. Two systematic reviews found school feeding interventions to have positive effects on attendance as well as on attainment (Kristjansson et al., 2009; Petrosino et al., 2012). Petrosino et al. (2012) found generally positive effects of a range of interventions providing health care (de-worming, vitamin A intake, malaria prevention, and menstruation cups), while Taylor-Robinson et al. (2012) found positive but weak evidence of the impact of de-worming on school attendance.

Three reviews examined the effect of infrastructure improvements and new roads on schooling access, one of which used meta-analysis (Petrosino et al., 2012). Petrosino et al. (2012) and Bouillon and Tejerina (2007) reported positive effects on attendance, while Birdthistle et al. (2011) found no studies assessing the effects of separate sanitation facilities on girls’ attendance.

Ensuring a sufficient number of appropriately trained teachers who are present in classrooms is a key challenge for efforts to improve children's learning, and several systematic reviews have assessed different interventions targeting teachers, although only one review includes meta-analysis. For instance, Orr et al. (2013) examined the effect of teacher training and find mixed effects on educational attainment. Guerrero et al. (2012) found that teacher monitoring in combination with incentives is effective in tackling teacher absenteeism, although they did not find any effects on student achievement. On the other hand three systematic reviews assessing the evidence on effects of wage increases and incentives (Carr & Leggatt-Cook, 2011; Bouillon & Tejerina, 2007; Petrosino et al., 2012) found such interventions can have positive effects on students’ attainment. Kingdon et al. (2013) reviewed the evidence on the effects of contract teachers, and concluded that the use of contract teachers is more effective for improving student outcomes than teachers with permanent positions.

Finally, decentralisation programmes were reviewed by Bouillon and Tejerina (2007) who found that decision-making at the local level can improve performance in schools, provided that there is sufficient institutional capacity and appropriate human capital to support the intervention, without which effects may be non-existent or even negative. The review was conducted some time ago, and subsequent (non-systematic) reviews (Bruns et al., 2011) include additional studies on a broader range of school management interventions.

The above summary of findings from existing reviews reveals that while there is an increasing body of systematic reviews of education interventions of relevance to LMICs, existing reviews are scattered across a wide variety of interventions and outcomes, and reviewers seldom use statistical meta-analysis to synthesise findings, resulting in many reviews with mixed or contradictory results. The education Evidence Gap Map indicates that very few reviews link up the key stages between initial, intermediate and final outcomes. Some reviews focus mainly on enrolment and attendance and others examine effects only on attainment. Fewer reviews cover academic completion or progression outcomes, while only one provides findings on the cost effectiveness of the interventions examined. Several of the existing reviews also suffer from methodological shortcomings and rely on searches completed several years ago.

Knowing what works is not sufficient for policy makers who also need to know ‘how to make it work in different contexts and environments and with different groups of people’ (Davies, 2006:99). Twelve out of seventeen of the identified systematic reviews and meta-analyses focused on the effectiveness of interventions alone and reviewed only quantitative evidence to draw conclusions about what works. Among these reviews, only five performed sub-group analysis in order to explore effects across different groups of participants, quality of studies or geographical regions.

The systematic review and meta-analysis conducted by Petrosino et al. (2012) is the most inclusive (in terms of the interventions that it covers) conducted to date. However, the authors focused primarily on outcomes related to enrolment and attendance and reported progression outcomes only when included in those studies, thus excluding studies that evaluated learning outcomes only. Additionally, the search was conducted in 2009 and studies published after that date are not included. Moreover, the review included only experimental and quasi-experimental designs, and did not engage with the theory of change of interventions and qualitative literature. Finally, the review estimated overall intervention effects, pooling different types of interventions. Sub-group-analysis was conducted, but without specifying outcome types. The issues outlined above present drawbacks for policy-makers and donors interested in being able to compare the effectiveness of interventions across outcomes and across sub-groups of participants.

Relevance to policy and practice

The importance attached to the role of education for human development is reflected by the international community's continued focus on access to, and quality of, education, as demonstrated for instance by the Education for All (EFA) initiative (UNESCO, 2013) and the Millennium Development Goals (MDGs) (UN, 2013b). The targets of the education related MDGs is to achieve universal primary education for all boys and girls (MDG 2) and eliminate gender disparities in all levels of education (MDG 3) by 2015, while the Dakar Framework for Action on Education for All (EFA) provides a strategy for achieving those objectives.

Education interventions have focussed on getting children into school, whether by increasing enrolment in existing schools or building and staffing new ones where there was no school before. More recently commentators have called for a shift in focus from access to education to learning for all (Prichett, 2013; Robinson, 2011), and this is also increasingly reflected in the education policy of major agencies. For instance, learning is at the core of the education policy of the Department of International Development (DFID) in the United Kingdom (DFID, 2013). Similarly, the World Bank education strategy for the period until 2020 is focused on learning and skills development, with ‘Learning for all, beyond schooling’ as a primary objective (World Bank, 2011).

The Dakar Framework for Action on Education for All (EFA) included a commitment that no country should be left behind in making progress towards the EFA goals due to a lack of resources and significant funding has been dedicated to education over the last decade. For instance, domestic government spending on education increased at high rates in LMICs between 1999 and 2011, despite the global economic crisis and regional food crises (UNESCO, 2012). In low income countries, the average real annual government spending on education grew at a rate of 7.2 per cent, and at a rate of five per cent in Sub-Saharan Africa (ibid), suggesting a commitment to reaching global education targets in many countries. Moreover, between 2002 and 2010, aid to education increased by 77 per cent to US$13.5 billion, with the World Bank, the USA and the UK being the largest donors to the sector (ibid).

Nevertheless, the resources available for education in low-income countries still pales in comparison to the resources dedicated to education in high-income countries. In 2010, countries in North America and Western Europe spent $7916 on primary schooling per pupil (constant US$), in contrast to US$134 in Sub-Saharan Africa and US$263 in South and West Asia (UNESCO, 2012). In addition, donor spending trends from the last couple of years indicate a stagnation of aid to education and a general tightening of aid budgets as high-income countries around the world attempt to rein in their spending following the global financial crisis, reversing the aid expenditure trends of the last decade (UNESCO, 2013). These trends come despite significant remaining challenges in ensuring that all children have access to high-quality education, as noted above. To help inform decisions about how to spend limited resources, this review will provide a comprehensive review of the evidence on the relative effectiveness of education interventions in improving education access and learning outcomes for primary and secondary school children in L&MICs.

Types of study designs

To address questions 1a and 1b we will include studies that assess the effects of interventions using experimental and quasi-experimental study designs that allow for causal inference.

Specifically, we will include: 1.

Studies where participants are randomly assigned to treatment and comparison group (experimental study designs);

Quasi-experimental studies may be subject to bias in their estimate of treatment effects, however, studies have shown that if well conducted quasi-experimental studies can provide un-biased estimates of treatment effects (Cook et al., 2008; Shadish, 2011). In setting our inclusion criteria we have aimed to incorporate studies that adopt techniques which empirical research suggest are effective in reducing or removing bias. Including a pre-test measure of the outcome and controlling for appropriate covariates in particular have been found to be important in reducing selection bias in quasi-experimental studies (Steiner et al., 2010; Shadish, 2011).

Thus, studies without random allocation to treatment and comparison group that do not include a baseline measure of the outcome variables will be excluded. Similarly, studies without random allocation to treatment and comparison group that do not use matching or other statistical methods to control for selection bias and confounding will be excluded. The selection and measurement of appropriate covariates that are correlated with both the selection or allocation of the treatment and the outcomes is important in reducing selection bias in quasi-experimental studies (Steiner et al., 2010). We will not address this issue at the inclusion stage, but we will do so when assessing the risk of bias in all included studies (details below). To avoid confounding treatment effects with teacher or school effects studies of any classroom and school level educational interventions, whether they are randomised or not, will be excluded if they have less than two teachers or schools in each group.

Finally, our interest is in identifying the evidence on the effects of an intervention implemented as part of a program under circumstances that approach ‘real-world’ practice, so-called effectiveness studies. These types of studies stand in contrast to efficacy trials which test an intervention under ideal and controlled conditions in order to maximise the likelihood of observing an effect, if one exists.

Although there exists broad agreement on the type of study design characteristics of effectiveness (pragmatic) trials and efficacy (explanatory) trials, there is currently no validated definition of ‘effectiveness studies’ (Treweek et al., 2009; Gerthlener et al., 2006; Singal et al., 2014). Furthermore, as Thorpe et al. (2009) note, the distinction between the two types of trials can be regarded as a continuum rather than a dichotomy as very few trials are purely pragmatic or explanatory.

In order to distinguish effectiveness from efficacy studies we developed five criteria, drawing on two existing tools (Gartlehner et al., 2006; Thorpe et al., 2009). Studies will be considered efficacy trials and will be excluded if they fulfil at least one of the criteria outlined below:

Research Objective:

(1)

Is the study primarily designed to determine to what extent a specific technique, technology, treatment, procedure or service works under ideal condition rather than attempt to answer a question relevant to the roll-out of a large program?

Population:

(2)

Are the participants highly selected and therefore unrepresentative of the general population (Are strict inclusion and exclusion criteria used to enrol a homogenous population which may limit the generalizability of the results? e.g. students that truly have a disease of interest or are more likely to adhere to the treatment)?

Providers:

(3)

Is the intervention primarily delivered by the research study team rather than trained laypersons (parents, teachers, community members, or NGOs) who don't have extensive expertise?

Delivery of intervention:

(4)

Is the intervention delivered with high degree of assurance of delivery of the treatment? (Is the delivery tightly monitored and supervised by the researcher following specific protocols; Is adherence to the treatment monitored closely with frequent follow-ups?)

To address question 2a and 2b, we will draw on relevant evidence from studies included to address questions 1a and 1b. However, the lack of details about the study contexts, beneficiaries and interventions within primary studies can be a barrier for review authors seeking to incorporate this information in their reviews (Herbert & Bø, 2005; Roen et al., 2006). A broader range of evidence, including process evaluations (Armstrong & Waters, 2007), project documents and qualitative studies can help address questions about intervention implementation and context. Thus, to address questions 2a and 2b we will also include studies and documents that are linked to the interventions studied in the included impact evaluations AND meet at least one of the following criteria: 1.

A qualitative study collecting primary data using qualitative methods of data collection and analysis, and report some information on all of the following: the research question, procedures for collecting data, sampling and recruitment, and at least two sample characteristics.

A project document providing information about planned, ongoing or completed interventions. They may describe the background and design of an intervention, or the resources available for a project for instance. As such, these documents do not typically include much analysis of primary evidence, but they provide factual information about interventions. The purpose of including them in our review is to ensure we have sufficient information about the context and interventions in included studies.

Types of participants

The review will include interventions targeted at primary school and secondary school 1 age children in mainstream education in low- and middle-income countries (L&MICs), as defined by the World Bank, at the point in time that an intervention was carried out. We will exclude studies focusing on refugees, migrants and orphans only. We will also exclude studies of children with special educational needs.

All adult education interventions, including those that are university-based, will be excluded. We will also exclude studies from high-income countries as the differences with L&MICs, in terms of policy challenges, resources devoted to education systems, state capacity and broader contextual factors, are such that we consider this evidence to be of limited applicability

Types of interventions

Broadly defined, interventions will be included in the proposed review if they aim to improve the access and/or the quality of primary and secondary education in low- and middle-income countries. The interventions will cover a broad range of programs and in doing so we acknowledge that there are a wide range of options available to policy makers, making a comparison of different options important. Moreover, as outlined in Figure 1, the outcomes we are interested in are determined by a wide range of factors, meaning that a single intervention is unlikely to be sufficient to address the barriers faced by children, families and education systems across the diversity of contexts covered in this review. We use an intervention typology based on different settings to specify the specific interventions we will include in our review, as outlined above. We will not repeat the descriptions of all interventions here, but Table 1 summarises included interventions according to intervention level.

The following interventions do not meet the inclusion criteria and will be excluded from the review:

Early childhood development: While ECD is clearly an important part of education, it is a separate sub-component of education and does not directly address primary and secondary education. Moreover, a team at the World Bank is currently working on a systematic review covering all ECD interventions.

Girls’ sexual and reproductive health: This is a separate sub-component of education. While important for girls’ education, including such interventions would further add to the scope of the review as it would include a large literature on preventions of HIV and other STDs.

School-based nutrition and health promotion: Such interventions generally have a primary focus on improving knowledge and related health and nutrition behaviour, rather than on improving the primary outcomes of interest in this review. Any impact on education is likely to be indirect and such interventions will therefore be excluded from this review.

Interventions teaching physical activities: We will focus on interventions promoting key academic subjects such as maths, reading and science, and programmes to promote physical activities are thus excluded.

Distance education: These interventions tend to be focused on further education and adult education and are less common for primary and secondary schools. Distance education can be seen as a separate sub-category of education and may be better reviewed on its own.

Special Educational Needs interventions: The review is focusing on mainstream education and special education can be seen as a separate sub-component of education and thus, is better dealt with in a review on its own.

Interventions to address disruptive behaviour: While addressing disruptive behaviours and improving discipline are important outcomes, we consider such interventions to be ‘second order’ interventions and they are beyond the scope of this review.

Microcredit: Microcredit interventions are not primarily about improving education, and any impact on educational outcomes are likely to be indirect, through household income.

Roads and other community wide infrastructure: These are not primarily about improving education, and any impact on educational outcomes may only be incidental.

Community wide health interventions: We will not include community-wide or general health interventions as education is not a primary outcome, and if measured, educational outcomes are incidental.

Interventions extending the school year or duration of primary or secondary school.

Types of outcome measures

To be included, studies need to assess at least one of the education related primary or secondary outcomes described below.

Primary outcomes

1.

Enrolment: defined as the number of students registered for education at the start of primary or secondary education or a given grade year.

The Learning Metrics Task Force (LMTF) (2013) proposes a broad framework of seven different domains of learning outcomes that are important for children and youth to master, from physical well-being, to literacy and communication, learning approaches and cognition, numeracy and mathematics, and science and technology. The framework reflects a holistic approach to children's learning, and the LMTF argues that learning should not be oversimplified by focusing on only some domains. Nevertheless, it also recognises the challenges involved in measuring outcomes across all of these domains, and that efforts to measure learning outcomes at a global level may have to focus on a more narrow set of measures to be feasible.

It would not be feasible to include learning outcomes across all domains in our review and we will focus on outcomes measures assessing children's learning in a few key domains: we will include learning outcomes for students in numeracy and literacy (in indigenous language and any primary language(s) of country), measures of cognitive and problem solving skills, and composite assessment scores, whether based on test scores or other measures of skills and learning.

To avoid bias from studies using treatment inherent measurement (Slavin & Madden, 2008) we will code detailed definitions of outcome measures and exclude from the meta-analysis any studies which sensitivity analysis indicates is driving results.

Secondary outcomes

These include other education related secondary outcomes, including: (1) teacher attendance: defined as a measure of the proportion of total school days for which teachers are present; and (2) teacher performance: defined as any measure of teachers’ knowledge, practice, motivation or satisfaction (Orr et al., 2013).

In addition to the outcomes specified above we will also collect data on other secondary and intermediate outcomes if they are reported in studies that satisfy all other inclusion criteria. Relevant secondary outcomes will vary for the different interventions included in our review and will be identified based on intervention specific program theories. For instance, child labour is a relevant outcome for the household level interventions that are aiming to address financial barriers to access to education. We will ensure that any outcomes included in a single meta-analysis are substantively similar.

Duration of follow-up

We will not include or exclude studies based on duration of follow-up. If studies include multiple follow-ups we will include the outcome measures most similar to that presented in the other studies included in any single meta-analysis, and report any additional follow-ups narratively.

Types of settings

Studies may be conducted in any setting as long as all other inclusion criteria are met. We anticipate the majority of studies will be conducted at household, school or community settings.

Type of comparison

In order to answer question (1), we will include studies that compare students receiving an educational intervention with a comparison group that either receives no intervention (including wait-list comparisons as part of pipe-line designs)/ business as usual or receives a different form of educational intervention. We will not combine studies with business as usual and active comparison groups in the same meta-analysis. Comparisons may be between schools, groups of students or areas such as school districts.

Other criteria for including and excluding studies

We will exclude any studies published before 1990. A review of the systematic review literature showed that the earliest cut-off point employed by any review was 1990. Pilot searches carried out to help guide the protocol development returned no studies published before 1990. A review of study inclusion criteria are provided in Table 2 below.

Studies to address review question 1

A comprehensive search of the literature for a systematic review on a topic in international development should cover key bibliographic databases, those specific to international development, those specific to social sciences, and specific to the subject of the review (Waddington et al., 2012). The search strategy has been developed in collaboration with an information specialist (JE) and with reference to the guidance in Hammerstrøm et al. (2010). In addition we used pearl-harvesting – collecting keywords from studies that meet our inclusion criteria (Sandieson, 2006).

In order to capture the relevant literature as comprehensively as possible, we have developed both a general set of search terms and a series of sub-strategies designed around the typology of educational interventions set out above. A draft of these search strategies is included in Appendix 1. This strategy will be adapted to fit all the electronic databases included in the search and where appropriate, thesaurus terms will be used in addition to natural language terms in those databases where both can be searched. All searches will be limited by the L&MICs filter and by year, from 1990 onwards. In order to ensure sensitivity, the study methods filter has been excluded from the searches in accordance with Campbell guidelines, with the exception of the general education search where the large number of papers retrieved made scanning impracticable. With this general search, further refinements to reduce numbers were made using the Web of Science Research Areas topics. In the Web of Science search example given in Appendix 1, all results for each of the categories have been combined using the OR Boolean operator to achieve an overall total. Citation searches of included studies will be carried out in Web of Science (SSCI & AHCI), Scopus and Google Scholar.

Electronic searches

We will search a range of databases and websites, including subject-specific education databases as well as general social science databases. We will search the following academic databases: •

Africa Wide: http://www.ebscohost.com/academic/africa-wide-information

Health databases to search only using health terms: •

Global Health (CABI) (only school feeding and health terms)

We will search the following electronic libraries and registries of impact evaluations: •

3ie Systematic Reviews Database

Grey Literature Searching

To ensure maximum coverage of unpublished literature, and reduce the potential for publication bias, we will search the following organisational websites and databases for unpublished grey literature: •

Best Evidence Encyclopaedia (BEE)

We will search the following websites of bilateral and multilateral organisations relevant to this review: •

AUSAID

Other searches

We will screen the bibliographies of included studies and existing reviews for additional eligible studies and will conduct forward citation-tracking of included studies in Web of Science. We will also identify and contact key researchers and organisations working in the education field, including the UK's Department for International Development (DfID), The Abdul Latif Jameel Poverty Action Lab (J-PAL), the Rural Education Action Program (REAP), UNICEF, UNESCO, World Bank and key bilateral donors.

We will hand-search journals of particular relevance to the review in an effort to identify papers that have not yet been indexed, covering issues published in the last 12 months 2 . Similarly, following consultation with our advisory group and information specialist, a hand- search for relevant books will also be conducted in agreed recommended libraries.

Titles and abstracts will be screened against the inclusion criteria and relevant records will be downloaded into the review management software EPPI reviewer. The initial screening of records will be conducted by several reviewers screening the records from different databases. At this stage we will be over-inclusive to ensure relevant studies are not omitted because sufficient information is not reported in title or abstract. Two reviewers will then independently review abstracts that have been judged to be potentially relevant at the first stage in more detail to determine which papers should be retrieved and reviewed at full text.

Two reviewers will then independently assess full text studies for inclusion, with any disagreements determined by a third reviewer.

Targeted search for addressing review questions 2a and 2b

When we have determined which studies will be included in the review of effectiveness, we will undertake targeted searching for qualitative studies, as well as process, implementation and cost information for those interventions evaluated in the included studies. We will conduct citation tracking of included studies to identify any relevant sister papers and conduct internet and database searches using the names of programs from included studies. To identify project documents and process evaluations we will conduct targeted searches of databases of project documents and websites of implementing agencies. Finally, we will contact authors and implementing agencies to request available project documentation. A more detailed description of the targeted search is provided in Appendix 4.

Critical appraisal of qualitative studies, process evaluations and project documents

Including a broader range of evidence can complicate critical appraisal, particularly as there is a lack of existing tools and criteria for quality (Noyes et al., 2011). We anticipate the additional sources included to address questions 2a and 2b will fall into four main categories as outlined in the inclusion criteria. We will adopt different approaches to appraise these three types of studies and documents, as outlined below.

We will assess the quality of included qualitative studies and descriptive quantitative studies using an adapted version of the Critical Appraisal Skills Programme checklist (CASP, 2006), making judgments on the adequacy of reporting, data collection, presentation, analysis and conclusions drawn. The checklist is included in Appendix 3. We will filter out studies of particularly low quality at this stage (Noyes et al., 2011) and studies where questions 1-5 are assessed as “No” will be excluded at this stage. The results of the quality appraisal will be reported in the review.

There are no commonly used critical appraisal tools for process evaluations. Such analysis needs reliable data from a representative sample, so assessment of sampling and methods of data collection are obvious issues to consider. We have drawn on existing guidelines for process evaluations (Scriven, 2007) and again adapted the CASP checklist to better suit such evaluations. The checklist is included in Appendix 3.

Project documents provide information about planned, ongoing or completed programmes, providing information about the design or resources available for a project for instance. As such these documents do not typically include much analysis of primary evidence, but they provide factual information about interventions. The purpose of including them in our review is to ensure we have sufficient information about the context and interventions included in our review. Thus, we will not formally appraise the quality of such documents, but rather focus our appraisal on assessing the relevance of the documents against the interventions assessed in our review.

Before extracting any data we will ensure that the name of the intervention, the implementing agency, context and timeline of the intervention described in the project document corresponds to the intervention assessed in the impact evaluation included in our review. Finally, collecting data from a range of sources, especially if used for triangulation, can enhance confidence in the trustworthiness of the information included (Montgomery et al., forthcoming). If several sources are available we will extract data from all sources for purposes of triangulation. If we are doubt about the relevance of a particular document, we will contact the authors.

Effect size calculation

Where possible we will extract the necessary data to calculate standardised effect sizes. For continuous outcome variables, we will calculate the Hedges’ g sample-size corrected standardised mean difference (SMDs), its variance and standard error using formulae provided in Borenstein et al. (2009, Chapter 4). For dichotomous outcome variables we will calculate the risk ratio (RRs), its variance and standard error using formulae provided in Borenstein et al. (2009, Chapter 5). This will be done in a consistent way such that outcome measures are comparable across studies. Thus, an SMD greater than zero (RR greater than 1) will indicate an increase in the outcome with the intervention compared to the comparison group. An SMD less than zero (RR between 0 and 1) will indicate a reduction under the intervention compared to the comparison. An SMD equal to (or insignificantly different from) zero (RR equal to 1) will indicate no change in outcome over the comparison. Whether these relative changes represent positive or negative impacts will depend on the meaning of the outcome in the context of the programme being evaluated. For example, while positive impacts on school enrolment, attendance or attainment will be measured as values greater than 1, (in other words, fewer) positive impacts of drop-outs will be measured as values less than 1.

Given the fact that primary studies have become increasingly complex and multivariate, more frequently authors need to extract partial effect sizes in the context of meta-analysis. Keef and Roberts (2004) considered the case of treatment effects based on continuous variables, and proposed the use of a partial standardised mean difference for this situation. Keef and Roberts examined the case of the partial d-type (mean-difference) effect size, where a two-group comparison is examined by way of an analysis-of-covariance model. Specifically, their model was

Y j = α + γ D j + β 2 X 2 j + … β p X p j + e j ,

where Y is an outcome score, D is a dummy variable representing a treatment or group effect, and X₂ through X_p are covariates. The errors e_j are assumed to have common variance . Keef and Roberts proposed using gadj = as a partial index of treatment effects, since represents an adjusted mean difference (accounting for all covariates in the model) and is the residual variance – essentially the variance of the Y scores, partialling out the effects of all predictors. In practice, the meta-analyst does not have access to .

Thus, we will code also for the standard deviation (SY) of the outcome both before and after adjustment. The decision in how to standardise our partial effect sizes will be made taking into account what has been reported in the majority of our studies sharing common outcomes. Thus, we will standardise our partial effect sizes after each set of outcome is fully coded. Moreover, we will create variables to identify how each partial effect size has been coded.

Finally, without the data at hand, it is not possible to predict whether a d-type partial effect size will be smaller or larger than the zero-order effect size. Because two possible adjustments are at play – the adjustment to the mean difference and also a reduction in the standard deviation – gad_j can be either smaller or larger than the unadjusted effect size. It can be larger than the typical standardized mean difference if the adjusted mean difference does not differ much from the unadjusted mean difference, but the standard deviation is much smaller than the unadjusted standard deviation SY.

Dependent effect sizes

There are a range of different issues that give rise to issues of dependent effect sizes. For instance, there could be several publications from one study, or several studies based on the same data set. There might be studies with multiple treatment arms with only one control group. Similarly, studies may report outcome measurements from several time points, or use multiple outcome measures to assess related outcome constructs. In such cases, we cannot treat all outcome estimates as independent of each other (Borenstein et al., 2009).

We will only include one effect estimate per study in a single meta-analysis. Where we have several publications reporting on the same study we will use effect sizes from the most recent publication. Where several studies exist using the same data set or where multiple outcomes are reported from alternate specifications within the same study, we will select the study or specification according to likely lowest risk of bias in attributing impact. For studies with outcome measures at different time points we will follow De La Rue et al. (2013) and synthesise outcomes measured immediately after the intervention (defined as 1-6 months) and at follow-up (longer than six months) separately. If multiple time points exist within these time periods, we will use the most recent measure. Many of the interventions we include in our review are ongoing programmes and the follow-up will therefore reflect duration in a program, rather than time since intervention. When such studies report outcome measures at different time points we will identify the most common follow-up period and include the follow up measures that match this most closely in the meta-analysis. When studies include multiple outcome measures to assess related outcome constructs, we will follow Macdonald et al. (2012) and select the outcome that appears to most accurately reflect the outcome construct of interest without reference to the results.

If studies include multiple treatment arms with only one control group and the treatments represent separate treatment constructs, we will calculate the effect size for treatment A versus control and treatment B versus control and include in separate meta-analyses according to the treatment construct. If the treatments A and B represent variations of the same treatment construct we will calculate the weighted mean and standard deviation for treatment A and B before calculating the effect size for the merged group versus control group, following the procedures outlined in Borenstein et al. (2009, chapter 25).

Unit of analysis

We will assess studies for unit of analysis errors, where the unit of the treatment is different to the unit of analysis, without taking account of clustering in the analysis (The Campbell Collaboration, 2014). If unit of analysis errors exist we will correct for this by adjusting the standard errors using formula provided in Hedges (2009).

Missing Data

Where included studies do not provide the data required to calculate effect sizes, the team will attempt to contact the authors of the primary studies. Formulae will be used where necessary to extract or impute effect sizes based on other commonly reported statistics (Lipsey and Wilson, 2001).

Assessment of heterogeneity

We will assess heterogeneity of effect sizes graphically, and test for heterogeneity formally by calculating the Q-statistic. We will also calculate and report the I², and Tau² to provide an overall estimate of the amount of variability in the distribution of the true effect sizes (Borenstein et al., 2009).

Moderator analyses

If feasible, we will conduct moderator analyses to investigate sources of heterogeneity. The review may draw on the additional data sources sourced from phase 2 of the review to provide data for moderator variables. We will assess moderators falling into three broad categories of extrinsic, methodological and substantive characteristics (Lipsey, 2009): 1.

Extrinsic variables: funder, type of publication, publication date

If possible, we will use random effects meta-regression to investigate the association between moderator variables and heterogeneity of treatment effects (Borenstein et al., 2009) and sub-group analyses to investigate heterogeneity by treatment sub-groups (for example, girls and boys, poor and non-poor, and so on). If the latter strategies are not possible (that is, we do not have sufficient number of studies or data), we will discuss and explore the factors which may be driving heterogeneity of results narratively by conducting cross-case comparisons (Miles & Huberman, 1994).

Sensitivity analysis

We will conduct sensitivity analysis according to categories of risk of bias, study design (experimental and quasi-experimental, adjusted and unadjusted effect sizes), treatment effect (for example, intention to treat, average treatment effect on the treated, local average treatment effect) and follow-up.

Publication bias

We will attempt to reduce publication bias by searching for and including unpublished studies in the review, but we will also assess possible publication bias using meta-analysis, and for under-reporting of small sample studies, using funnel plots and Egger et al.'s (1998) test. Given the inherent subjectivity in assessing funnel plot asymmetry, we will assess sensitivity of meta-analyses using ‘trim and fill’ (Duvall & Tweedie, 2005), regardless of whether funnel plots suggest asymmetry 3 .

All statistical analyses will be conducted using R software (R Development Core Team, 2008).

Review questions 2a and 2b: Qualitative synthesis

As noted above, we will include qualitative studies, process evaluations and project documents related to the programmes studied in included impact evaluations. From these documents we will include descriptive information about programme design and implementation, context and resources, as well as any findings addressing questions 2a and 2b on barriers and facilitators of intervention success or failure.

The intention is to complement any statistical meta-regressions with a qualitative synthesis of data relevant to questions 2a and 2b (Rubenstein et al., 2009). After having completed the detailed coding of all of the included studies as described above, we will re-review the coding of data on context, intervention design and implementation to identify descriptive findings which remain close to the findings in the primary studies (following Thomas and Harden, 2008). We will then conduct cross-case analysis (Miles & Huberman, 1994), using a framework based upon the links and assumptions from the program theories of included interventions.

We will rank studies by effect size and develop a series of matrices to identify the features of interventions and contexts that appear to act as barriers and facilitators of improving educational outcomes. We will initially conduct the cross-case analysis by intervention type, but we will also attempt to identify any overarching themes across intervention types.

We will include an assessment of our confidence in each finding using the CER Qual (certainty of the qualitative evidence) approach. The CER Qual approach provides a transparent method for assessing the confidence of evidence based on reviews of qualitative research and may facilitate the use of these findings alongside review of effects and in guideline development processes. Using the approach we will base our assessment of confidence on four factors: the methodological quality of the individual studies contributing to a review finding; the coherence of each review finding; the relevance of a review finding; and the sufficiency of data supporting a review finding. The assessment of the overall degree of confidence in the review finding will be described and justified in a summary of qualitative findings table which includes narrative statements.

Integrated synthesis (review questions 1 and 2)

The overarching goal for the review is to provide an integrated synthesis of the findings from synthesis of review questions (1) and (2) in a narrative synthesis. We will use the overarching conceptual framework provided above to present the findings from the different syntheses with the aim of providing an integrated narrative synthesis addressing the objectives of the review.

We will produce summary of findings tables following the GRADE (Schünemann et al., 2011) and CerQual approaches to facilitate transparent and systematic presentation of our findings.