Abstract
Linked article:
BACKGROUND
The Issue: Food security and poverty reduction for smallholder farmers in Africa
A large proportion of the world's poor live in rural areas, dependent on subsistence farming for their survival (FAO, 2011). Smallholder farmers have been credited with providing up to 80 per cent of food in developing countries (IFAD, 2012) and have the potential to feed themselves and also supply urban markets. Vietnam's smallholder farmers are often credited, for example, with transforming the country from a net-importer of food, to a major exporter (ibid).
Whilst definitions of smallholder farming vary, the concept usually incorporates a number of key elements (Morton, 2007): farms on which labour is predominantly family (‘family farms’) (IFAD, 2009); farmers and farms that are resource poor (Nagayets 2005; Dixon et al., 2003); farms of a particular size, most commonly two hectares (Nagayets, 2005; Hazell et al. 2010; Wiggins et al., 2010; World Bank 2003; IFAD, 2011a); and farms which are predominantly subsistence, but might also include a mix of commercial and subsistence activities (Narayan and Gulati, 2002).
Smallholder farming is of particular significance to Africa for a number of reasons. Africa's economy is dominated by agriculture (Massett et al., 2011) and the vast majority of farmers in Africa are smallholders (World Bank, 2007). Smallholder farmers contribute significantly to food security on the continent, for example, in sub-Saharan African smallholder farmers contribute up to 80 per cent of the food supply (IFAD, 2011b). Smallholder farmers also include many of the continent's poorest and most marginalised people (World Bank, 2007). In southern Africa in particular, large numbers of women and girls rely on smallholder farming, and it provides a survival strategy for many of the continent's young people, many of whom are orphans and head of their households. Furthermore, supporting Africa's development is a priority within the G8, and working towards increasing food security in the region is high on the agendas of the majority of international donors, including the Canadian Foreign Affairs, Trade and Development agency (DFATD), who have commissioned this review.
Given both the importance of smallholder farming in Africa and its potential to contribute to the food security of so many, it is not surprising that considerable efforts are being invested in its success. Both national and international agencies are investing in improving the productivity of smallholder farming, including the International Fund for Agricultural Development (IFAD) and the Canadian Foreign Affairs, Trade and Development agency (DFATD). Additionally, in 2009, the G8's L'Aquila initiative pledged $22 billion USD for agriculture in developing countries (G8, 2009). In 2012, IFAD launched the Adaptation for Small Holder Agriculture Programme (www.ifad.org/climate/asap/). On a national level, heads of state in Africa are increasingly stressing the need for support for smallholder farmers. For instance, South African President Jacob Zuma emphasized the need for support of smallholder famers in his 2013 State of the Nation Address (RSA, 2013). Parallel efforts are being invested in agricultural research, such as impact evaluations and systematic reviews, to assess the effectiveness of these agricultural programmes.
Agricultural productivity of particularly starchy cereals is important since this category of crop accounts for two thirds of the region's energy intake as well as 70 per cent of the income of the extremely poor population living in Africa (AGRA, 2013). The 2013 Africa agriculture report issued by AGRA (ibid) identifies the general production trend for Africa as somewhat erratic, but with most countries reporting a steady increase in production. According to the report, Kenya, Ghana, Mali, Tanzania, Uganda, Zambia, and possibly Mozambique have all reported steady increases in agricultural production (AGRA, 2013, 21). Areas that have experienced civil unrest, political instability, or mismanagement of the macroeconomics of the country in the last decade have seen a decrease in agricultural productivity; included among these are Sierra Leone and Liberia (AGRA, 2013, 21). Technologies typically used to increase agricultural productivity in the region include the “increased use of agricultural inputs, modern farming techniques, and reduced market inefficiencies” (AGRA, 2013, 20). However, a much larger array of factors come to bear on agricultural productivity. Political, technological, physical environmental, and micro- and macroeconomic factors related to each country play a pivotal role in shaping the region's agricultural productivity. World prices of inputs and outputs, and international trade policies also influence the agricultural productivity within countries (AGRA, 2013). As such, any technology addressing any one of these aspects may be expected to have an influence on the food security or income of smallholder farmers in Africa. Examples of specific technologies include treadle pump irrigation technology (Adeoti et al., 2009); biofortification and health information (de Brauw et al., 2013); and adopting an export crop and marketing techniques (Ashraf et al., 2008).
In the context of this considerable and growing emphasis on smallholder farming, there is a need to understand the relative effectiveness of the different interventions targeting smallholder agriculture in achieving various outcomes.
Proposed solutions considered in this review and how they might work
Today, there are a multitude of agricultural interventions in place across Africa (Sapa, 2009). The focus of these interventions has shifted as understanding of the relationship between agriculture and poverty has developed (Massett et al., 2011). Early interventions focused on increasing productivity to meet a perceived lack of food. With the realisation that undernourishment persists alongside high levels of production (Reutlinger & Pellekaan, 1986), structural issues came to the fore and the concept of food security was introduced (Sen, 1981). Interventions shifted towards income generation, access to markets and the production of more nutritious and calorific foods. Two groups of interventions have specifically sought to increase food security and reduce poverty by training farmers and / or encouraging them to adopt agricultural innovations and new technologies.
Interventions that are categorised as innovations emphasise the introduction of a ‘new’ farming method, product, or service. An example of this kind of intervention is the introduction of home gardens to increase the intake of vitamin A. A new technology intervention places emphasis on the introduction of a previously unfamiliar agricultural input. This could be a different piece of equipment or genetically modified seeds. Training interventions would place emphasis on providing some kind of training to farmers. The content of such training may not necessarily be new to farmers, but rather previously unemployed. However, we acknowledge that some training interventions will centre on the introduction of new technology and/or innovation. For instance, an example of this kind of study is de Brauw et al.'s evaluation of biofortification and a health information intervention on the food security of smallholder farmers (2013).
A recent systematic map of the evidence of interventions targeting smallholder farmers (Stewart et al., 2013) found that there were gaps in the African evidence base, including: 1) a lack of systematic reviews addressing various interventions' impacts on the financial wealth of smallholder farmers, and 2) a lack of assessments of the impact of interventions on smallholders' food security. The scope of the present review has been influenced by these gaps as well as by consultation with our advisory group and funders.
Agricultural Training / Knowledge
Training interventions for farmers vary considerably. Some interventions focus directly on teaching farmers, using top-down ‘train and visit’ approaches (Hume, 1991). Such training interventions are also often packaged as ‘extension services’, a broad term for programmes which aim to “support and facilitate people engaged in agricultural production to solve problems and to obtain information, skills and technologies” (Anderson, 2007:6). Although traditionally considered as a top-down approach to training, extension services have over time become more participatory in nature (Waddington et al., in press). Specifically ‘farmer field schools’, which may be one component of broader agricultural extension services, use a more bottom-up approach to training and knowledge transfer (Waddington et al., 2009). Farmer field schools are participatory, empowering and experiential in nature and draw on problems and priorities identified by farmers themselves, rather than those determined by outsiders (Waddington et al., in press). Initially developed to tackle an over-reliance on pesticides, field schools have now been implemented across over 80 countries (van den Berg, 2004).
Another important aspect of these interventions relates to the training objectives: there is a clear distinction in the literature between courses that are directed to improve agricultural practices and increase yields (for example, training on natural resource management; integrated pest management; conservation agriculture), and those which focus on aspects of farm management (for example, social organisation; management; institutional development).
Perhaps the most straightforward way of considering the range of training interventions available is to consider three facets of the interventions: how experiential or participatory the training is; the duration of the training; and the content of the training – see Table 1.
Dimensions of training interventions
An example of an evaluation of this kind of intervention is Anyango et al. (2010), who evaluated a five-year project in Kenya aimed at improving the income and food security of smallholder farmers through a number of training interventions, including introducing new cultivars to the farmers and training them through farmer field schools, as well as providing training on marketing skills, amongst other things. Ashraf et al. (2008), on the other hand, evaluated agricultural interventions that worked with pre-existing farmer self-help groups and provided farmers with information and short orientation-sessions about switching to export crops and offered in-kind loans and facilitated transactions with exporters.
New Technology and Innovation
Agricultural innovation interventions aim to facilitate adoption of new technologies including: fertilisers; new crops (including genetic modification, Hall, 2010); more nutritious crops; and new industries (Ton et al., 2013); and incorporate these technical developments with new systems (Adjei-Nsiah et al., 2008). Sunding and Zilberman (2001) provide a useful framework of these interventions, in terms of mechanical, biological, chemical, agronomic, biotechnological, process and product innovations – see Table 2.
A framework of innovation and new technologies
The literature includes a number of examples of evaluations of these innovations and new technologies. For example, Bennett et al. (2003) evaluated the impact of a biological innovation - the introduction of insect-tolerant Bt cotton in South Africa. Panin (1995), on the other hand, assessed a mechanical innovation, evaluating the effectiveness of mechanisation (the introduction of tractor farm technology) on factors such as smallholder farming yield, income and resource utilisation in Botswana.
How the interventions might work
The intended outcomes of these innovations are wide-ranging: from investment (in seed, land, livestock, or labour), to increased yields, productivity, income generation, health, nutrition, food security, and poverty reduction (World Bank, 2007). In particular, there is increasing emphasis amongst international donors on the ‘end-point’ outcomes of food security and poverty reduction. Smallholder farming has long been credited with the potential to end food insecurity (Sen, 1981; Reutlinger & Pellekaan, 1986). The argument is that it is both an effective subsistence strategy and a potential income-generating activity enabling poor farmers to purchase additional food (IFAD, 2012). Furthermore, it is thought to benefit those segments of the population that are most vulnerable to the effects of poverty, namely women, children, and youth (World Bank, 2007).
Whilst there is demand for evidence of the effectiveness of training and innovation and new technology on the financial wealth and food security of smallholder farmers, the mechanisms by which these interventions work involve several intermediate steps. These are multi-faceted, and are dependent on factors such as the environmental context, political stability and economic climate, as well as more direct elements such as farmers' scope to change their practice and increase their productivity. As Figure 1 illustrates, there are key intermediate outcomes on the pathway to increased food security and financial wealth, specifically: investment, knowledge transfer, adoption of innovation, diffusion of innovation, and increase in yield and productivity.
An initial causal pathway
We will interpret yield broadly. We will include studies that refer to the extent of food production, the growth rate of crops, crop output, as well as crop losses (Stewart et al., 2013). These categories may include studies of the impact of interventions on: the improvement or conservation of soil fertility, of the quality of output, disease resistance or reduction, and food storage conditions (Stewart et al., 2013). Productivity is defined by Stewart and colleagues (2013) in the systematic map of African evidence as “the efficiency of production”.
This includes various aspects, such as “measures of technical efficiency, better resource management, reduced input costs, impacts on labour requirements, ‘stronger’ systems, and ‘better’ utilisation of available resources” (Stewart et al., 2013).
Why it is important to do this review
The importance of smallholder farming in Africa, and the multitude of interventions to increase the wealth and food security of smallholder farmers has been outlined already. We initiated discussions with government agencies and non-governmental organisations supporting these farmers to identify their priorities for evidence to inform their programmes. Having consulted widely on the range of interventions implemented and their intended outcomes, we identified the need for clear evidence on the effectiveness of innovation and / or training interventions, and their impacts on both poverty reduction and food security. An initial scoping review to ascertain the extent to which published reviews had already answered these questions highlighted how more focussed reviews provided evidence on one intervention, but did not answer the question that donors and NGOs raised around which intervention to invest in and why. (See Box 1 and Appendix 1 for more on this preliminary scoping work.)
Box 1: An overview of our ‘review of reviews’ (reproduced from Stewart et al. 2013, with authors' permission)
A total of 21 systematic reviews of relevance to smallholder farming in Africa were found. Of these, 18 reviews were complete, two protocols were published (Loevinsohn & Sumbug 2012; Knox et al., 2013) and a third protocol is currently under peer review (Dorward et al. 2013). The protocols both focus on agricultural infrastructure (Loevinsohn & Sumbug, 2012; Knox et al., 2013), whilst Dorward and colleagues' review will focus on agricultural finance. The scopes of the 18 completed reviews were categorised into four broad intervention categories: training, innovation and new technology, infrastructure and finance. Only one of the 18 focused on training, specifically farmer field schools (Waddington et al. 2013). Reflecting the search for new and better ways of farming, we found nine systematic reviews that evaluated the impacts of innovation and new technology (Bayala et al., 2012; Bennet & Franzel, 2009; Berti et al., 2004; Hall et al., 2012; IOB 2011; Girad et al., 2012; Gunaratna et al., 2010; Masset et al., 2011; Rusinamhodzi et al., 2011). These included evaluations of the effectiveness of conservation agriculture in general (Bayala et al., 2012, Bennet & Franzel, 2009, Rusinamhodzi et al., 2011), as well as specific conservation agriculture interventions, including: parkland trees associated with crops (Bayala et al., 2012), coppicing trees (Bayala et al., 2012), green manure (Bayala et al., 2012), mulching (Bayala et al., 2012), crop rotation and intercropping (Bayala et al., 2012; Rusinamhodzi et al., 2011), traditional soil and water conservation (Bayala et al., 2012), tillage management (Rusinamhodzi et al., 2011), and residue retention (Rusinamhodzi et al., 2011). These systematic reviews also considered the impacts of organic agriculture (Bennet & Franzel, 2009) and genetically modified crops (Hall et al. 2012), as well as specific interventions aimed at increasing nutritional status of households, such as home gardening (Berti et al., 2004; Girad et al. 2012; Masset et al., 2011), cash cropping (Berti et al., 2004), irrigation (Berti et al. 2004), and biofortification (Masset et al., 2011; Gunaratna et al. 2010). The impact of interventions to increase food production have been reviewed (IOB, 2011), including particular forms of agriculture, specifically livestock (Berti et al., 2004), and in particular poultry development (Masset et al., 2011), animal husbandry (Masset et al. 2011) and dairy development (Masset et al., 2011); fish ponds (Masset et al., 2011), aqua culture (Masset et al. 2011), and mixed garden and livestock (Berti et al., 2004). Five completed reviews have considered finance for farmers, specifically: index insurance (Cole et al., 2012), micro-credit (Duvendack et al., 2011; Stewart et al., 2010, 2012), micro-savings (Stewart et al., 2010, 2012), micro-leasing (Stewart et al., 2012), and agricultural investment grants (Ton et al., 2013). Lastly, three systematic reviews focused on the impact of agricultural infrastructure interventions, specifically agricultural interventions and food security (IOB, 2011); infrastructural investments in roads, electricity and irrigation (Knox et al., 2013); and land property rights (Hall et al., 2012).
Despite the somewhat extensive literature base outlined in Box 1, Stewart and colleagues (2013) found that there were three gaps in the African evidence base, two of which will be addressed by this review, namely the lack of systematic reviews addressing various interventions' impacts on 1) the financial wealth of smallholder farmers, and 2) on their food security.
Given the potential for African smallholder farmers to contribute to the food security across the region, coupled with the increasing investment in the industry, there is a need for evidence as to which interventions are most effective. The wide range of options facing policy-makers and practitioners and the need to focus the limited resources available increases the importance of this review.
OBJECTIVES
Our objectives in conducting this Campbell systematic review are to:
Systematically review the available evidence of effectiveness of i) training interventions and ii) innovations and new technologies on i) the financial wealth and ii) the food security of smallholder farmers' in Africa. Where available within the studies of the effectiveness of these interventions on financial wealth and food security, to explore any impacts on intermediate outcomes, specifically: investment, knowledge transfer, adoption of innovation, diffusion of innovation, yield and productivity.
In doing so, we will provide valuable information to decision-makers, not in the least being DFATD who have commissioned this work.
METHODOLOGY
I. Criteria for including studies in the review:
To be included in this review, a study must use an experimental or quasi-experimental design. Eligible designs include those in which the authors use a control or comparison group
Participants are randomly assigned (using a process of random allocation, such as a random number generation); A pseudo-random method of assignment has been used and pre-treatment equivalence information is available regarding the nature of the group differences (and groups generated are essentially equivalent); Participants are non-randomly assigned but matched on pre-tests and/or relevant demographic characteristics (using observables, or propensity scores) and/or according to a cut-off on an ordinal or continuous variable (regression discontinuity design); or, participants are non-randomly assigned, but statistical methods have been used to control for differences between groups (for example, using multiple regression analysis, including difference-in-difference, cross-sectional (single differences), or instrumental variables regression).
To be included a study must have:
pre-test data post-test data an intervention group a control group
For this review, the control or comparison conditions in these studies may include farmers receiving no treatment, treatment as usual, or an alternative treatment. No restriction will be placed on duration of follow up. Studies for which the impacts within Africa cannot be isolated, will be excluded from the review.
Given our focus on the end impacts of the interventions (that is, financial wealth and food security), studies for the review must include a minimum follow-up period of at least 6 months between receipt of intervention and measurement of these end impacts. Shorter follow-up may produce misleading results. For example, an intervention that introduces a new breed of cattle, could lead to increased access to meat in the diet in the immediate term, but it would be misleading to label the consumption of these cattle as an increase in food security.1
I a) Types of participants
To be included, a study must include African farmers of smallholder farms.
Farmers include both men and women who either own their farms or farm land owned by others. We will not limit by age as we acknowledge that there are large numbers of child-headed households in Africa, and it is feasible that smallholder farmers will be very young. For the purposes of sub-group analysis later in our review, we will define young farmers as those under the age of 20.
Smallholder farms can be defined in a number of ways. Whilst size of farm is often cited – most commonly less than 2 hectares – the productivity of the land can mean that in some countries much larger farms are considered to be ‘smallholdings’. In Tanzania for example, farms of up to 50 hectares have been classified as smallholder farms. The nature of the land, the crops grown and the types of livestock kept all shape the resource-level of farms. Farmers may own their land, although this is often not the case. Similarly smallholder farms are usually assumed to be rural, yet peri-urban farms can also be included. This review will employ a definition of smallholder farms as ‘resource-poor’, where the “resources of land, water, labour and capital do not currently permit a decent and secure family livelihood” (Chalmers, 1985). Table 3 provides a framework for how we will operationalise our definition of smallholder farms.
Defining smallholder farming for this review
Women farmers, young farmers and landless labourers have been highlighted by our advisory group as key populations of interest within this review. All three groups will be included within the review, and study populations coded accordingly.
I b) Types of interventions
This review focuses on two broad intervention types. Studies will be included in the review if they meet at least one of the following criteria:
Their main focus is the transfer of knowledge and / or experience to smallholder farmers They seek to train farmers in the use of one of the following types of innovation or new technology: mechanical, biological, chemical, agronomic, biotechnological, process or product innovations They introduce, or otherwise promote, a new technology or innovation to smallholder farmers They introduce, or otherwise promote, a technology to smallholder farmers which is new to the farmers, even if it may already be used by others. The means of introducing a new technology, that is, a previously unfamiliar agricultural input or innovation, such as the introduction of a ‘new’ farming method, product, or service, would primarily be through some form of what we are labelling broadly as ‘training’.
Interventions that do not target smallholder farmers specifically will be excluded.
Studies will not be excluded by duration or frequency of programme delivery.
I c) Types of outcome measures
Primary outcomes
This review focuses on two primary outcomes areas: financial wealth and food security.
Financial wealth
We define financial wealth as any form of finance or asset that a household generates, for example, income from selling food products or savings from not having to buy food products.
Specifically we will extract data on the following outcome measures for financial wealth:
Household income (including intra-household distribution of income if available) Household accumulation of financial assets Household accumulation of non-financial assets Household food expenditure
Bennet et al. (2006) is an example of a study in which the economic impact of genetically modified cotton on South African smallholders' profits is assessed.
Studies that do not consider one of these primary outcomes will be excluded from the review.
Food security
According to the 2009 Declaration of the World Summit, food security exists when all people, at all times, have physical and economic access to sufficient, safe, nutritious food to meet their dietary needs and food preferences for an active life (FAO, 2013). Therefore, food security is essentially the availability of food and one's access to it. We consider the above definition of food security in our review. Based on this definition, our review takes into account improved access, availability and nutrition.
We will extract data on the following specific outcome measure for food security:
Household food consumption by weight. Low et al. (2007) is an example in which the introduction of OFSP was assessed in an integrated agriculture and nutrition intervention, which aimed to increase vitamin A intake and serum retinol concentrations in young children. Per capita calorific intake (Low et al., 2007) Household perceptions of food security (for example, Ayalew et al. (1999) Reducing vitamin A deficiency in Ethiopia)
We have also included an ‘other category’ which will consider any measures which do not fall under those mentioned above.
Secondary outcomes
Secondary outcomes considered in this review will include:
Investment in capital Knowledge transfer Adoption of innovation Diffusion of innovation Yield Productivity
I d) Other criteria for including or excluding studies
Studies will not be excluded from the review on the basis of language. Searches will be conducted in English and translations obtained for foreign language papers where possible.
We will include only studies conducted since 1990. Both the methodologies for assessing impact of these interventions, and the nature of the interventions have developed significantly since 1990 (Romani et al., 2003; Sapa, 2009) making it highly unlikely that we will identify any relevant literature prior to this date. We will search only for papers published since 1990 and screen on study dates. Where any data in a study published after 1990 was collected prior to 1990 (e.g. baseline), the study will be excluded.
II. Search Strategy
In order to identify the literature for this review as comprehensively as possible, we have designed our search strategy to include both general and specialist sources, with both broad search terms and more specialist ones. We have taken advice from two search specialists, from the Campbell Collaboration and the EPPI-Centre, in the design of these searches.
Electronic Sources
CAB Abstracts Web of Science – specifically, the Social Science Citation Index and Science Citation Index 3ie impact evaluations database IDEAS Africa bib. databases (specifically, African periodical literature/ African Women's Bibliographic database). AGRIS, the research database of the FAO BLDS Agricola
Africa Wide
Other sources (including websites and grey literature)
IFAD evaluation reports
http://www.ifad.org/evaluation/public_html/eksyst/doc/index.htm
IFPRI publications JPAL evaluations The Millennium Challenge Corporation USAID Bill and Melinda Gates Foundation CGIAR
Search terms
The key concepts in our review are summarised below.
Smallholder farm Impact evaluation Africa Intervention (specifically training and innovation / new technology)
We have some concerns that combining four concepts in our searches may be too narrow and exclude some relevant studies. From test searches, the ‘Africa’ concept is challenging to search for (many search engines won't accept the large numbers of search terms required), and is also relatively easy to screen for (the country where a study is conducted is usually reported clearly in the abstract). We will therefore search for only three concepts in these new databases, combining the concepts for smallholder farms, impact evaluation and the interventions of interest in the following way:
((smallholder farm AND impact evaluation AND (training OR innovation))
We have developed detailed search strings in order to ensure we capture all possible search terms for the concepts we seek (see Appendix 2). However, some databases use relatively simple search functions making long strings of terms difficult to employ. The proposed strings will therefore be adapted to suit each of the databases as appropriate. Where available, we will search within the title and abstract fields. Where this option is not included, we will search the full record. We will also seek appropriate controlled terms where available.
Additional searches
Citation searches will be conducted using Google Scholar, Web of Knowledge, and Scopus for related systematic reviews and key impact evaluations, as listed in Appendix 2. Both the included and excluded lists of the identified overlapping systematic reviews will be screened for relevance to this review (see Appendix 1 for list of these SRs). We will search the reference lists of all potentially relevant impact evaluations. In addition, we will check the reference lists of a recently published scoping map of agricultural innovation in sub-Saharan Africa: Percy R, Tsui J and Sutherland A (May 2013) Agricultural Innovation in Sub-Saharan Africa and South Asia: A Scoping Study.
http://www.3ieimpact.org/media/filer/2013/06/28/3ie_scoping_study_report_1.pdf
Requesting relevant studies from key contacts: We will write to key contacts requesting relevant studies. These will include members of our project advisory group and first authors of relevant reviews, as listed in Appendix 2.
Selection of studies
Two reviewers will independently assess the full text papers against the inclusion criteria, and each author will extract data from included studies. Discrepancies will be resolved by consensus, and a third reviewer will be available to resolve any disagreements.
III. Description of methods used in primary research
Methods used in the primary research considered relevant to this review include randomised controlled trials, cluster randomised controlled trials, and a range of designs that employ non-randomised allocation approaches. These include regression discontinuity designs, natural experiments where external factors determine allocation, and self-selected assignment (by the research team, or the research participants) (Waddington et al., 2012). Of those study designs in which allocation to intervention or control is self-selected, we will only include studies that have a comparison group with before and after data, assessing impact using double difference approaches.
Only studies that have a well-defined intervention-control group will be included in this review. Included studies must assign participants at the individual, group, cluster, districts, or provincial levels.
Ashraf and colleagues (2008) study is an example of a randomised controlled trial which we anticipate including in this review. They collect baseline data and assess impacts at one year, across two treatment groups (both of which receive the ‘DrumNet’ intervention, and one of which also receive microcredit), and a control group.
In another example, Smale and colleagues (2012) assess the impact of three linked programmes using baseline and endline data, comparing beneficiaries and non-beneficiaries of the programmes using difference-in-difference analyses. They used propensity score matching to try to retrospectively match their intervention and control groups.
IV. Data extraction of study information
We will use a detailed coding sheet with screening information that determines whether a study is to be included or excluded for this review (see Appendix 3). We will then extract data from the included studies as outlined in the coding sheet, including details on the target population, the type of intervention, scale of intervention, outcomes and how they were measured, and funding agencies. We will use the coding sheet included in Appendix 4 to extract data to allow us to calculate effect sizes, including sample sizes, means, standard deviations, confidence intervals, and rates of dropouts for both control and intervention at each time of follow-up.
Initial coding and screening will be done on EPPI-Reviewer, and additional data extraction for included studies will be done in Microsoft Excel. This will facilitate standardisation of effect measures for outcomes in included studies that will be used in meta-analysis.
A randomly selected sample of the coded studies (10% of full texts) will be double-coded by an independent member of the review team, and inter-rater reliability scores (percentage matches) will be calculated and Cohen's kappa applied (Higgins et al. 2011). Disagreements will be discussed and resolved and a consensus code adopted.
V. Assessing risk of bias
We will assess the methodological quality of the included studies using the risk of bias tool developed by the Cochrane Methods group (Higgins et al., 2011) and adopted for non-randomised studies (Stern et al., 2013). Specifically, these will include screening questions to determine whether particular bias is controllable in a given study, and guidance for the reviewer to rely on while scoring the risk of bias for the outcome, and the justification for making a judgment for every domain and outcome reported. The six domains are listed below, with full details of the tool included in Appendix 5.
Bias due to baseline confounding
Bias due to selection of participants into the study
Bias due to departure from intended interventions
Bias due to missing data
Bias due to measurement of outcomes
Bias due to selection of results
Risk of bias assessments will be done for every relevant outcome in all the six domains. The risk of bias will be deemed as low, moderate, high, or critical, and where sufficient details to make judgment are unavailable, the risk will be deemed as unclear. The findings of those studies judged to be at critical risk of bias will be reported, but not considered for synthesis. See Appendix 5 for more details on each of these judgements.
VI. Effect sizes
We anticipate a variety of scales measuring effect sizes and we will standardise these measures in order to obtain a uniform scale to facilitate synthesis in a meta-analysis (Borenstein, 2009). For continuous outcomes we will use the Hedges' g statistic to calculate standardised effect sizes of included studies (Hedges & Olkin, 1985). For dichotomous outcomes we will calculate odds ratio and risk ratio as the measure of effect size. Effect sizes, standard errors and their 95% confidence intervals will be computed as recommended by Borenstein (2009).
Where information is missing, and can be calculated from other variables, we will do so, as per Higgins et al. (2011). Where crucial statistics are not reported, authors will be contacted, but if information is still unavailable manipulations will be conducted to derive desired statistics in order to perform meta-analysis, as specified by Lipsey and Wilson (2001).
For those studies where no standardised effect size can be extracted for meta-analysis, effect sizes and confidence intervals will still be reported but not included in the forest plot or meta-analysis. Furthermore we will seek to indicate on all forest plots how many studies reported relevant findings but could not be included due to an inability to extract the effect size. This is to avoid any over-simplification of the evidence base within our forest plots, without an awareness of the wider evidence.
VII. Selecting interventions in multi-armed studies
In cases where studies have used a single control group and multiple intervention groups:
If only one of the intervention groups is closely matched to our intervention group of interest, we will select one pair of intervention/control group and exclude others (Deeks et al. 2011; Borenstein et al. 2009; Higgins, 2008). The selected pair of groups will be closely matched to our intervention of interest using our pre-specified criteria. If more than one intervention group is closely related to our intervention of interest when compared to our pre-specified inclusion criteria, and if possible, all relevant experimental intervention groups will be combined to create a single experimental group to compare with the control group (Higgins, 2008). Similarly, all relevant control groups will be combined to create a single control group for comparison with relevant experimental groups (as per Borenstein, 2009). For dichotomous outcomes, both the sample sizes and the numbers of people with events can be summed across groups. For continuous outcomes, means and standard deviations will be combined as described in The Cochrane Collaboration handbook (Higgins, 2008). Where correlation measures are reported, multiple-treatment meta-analysis may be adopted, where findings from several arms of the same study will be included in the meta-analysis after adjusting for correlation.
VIII. Dependent effect sizes
We will follow Campbell Guidance (Becker et al., 2007) so that each meta-analysis pools only findings that are statistically independent. Dependent effect sizes may occur where more than one study is reported in a single paper, where several papers report the findings of one study, when studies include several intervention arms with only one control or measure outcomes at more than one point in time. Multiple measures of the same outcome within one study will not be synthesised. We will take steps to ensure that only independent findings are included in any one synthesis as outlined below.
Where more than one paper or report is identified on a single study, we will choose one as the ‘main’ paper (the one with the most relevant data) and the others will be considered ‘secondary reports’ which we draw on only for additional information about that one study, including outcome measures not reported in the main paper.
Where a single report describes more than one study, these will be separated into two or more ‘studies’, which will be coded and analysed separately. To help us identify papers which might be linked or split in these ways, we will collect information on funding bodies, and intervention programme names in our preliminary coding questions. Our aim will be to identify all affiliations between studies/reports before detailed coding takes place.
Where individual studies report multiple outcome measures for the same outcome construct we will select the outcome that is most commonly reported across included studies, and if not available, then the outcome that is most accurately measured will be used.
For studies reporting follow-up effects at multiple points in time, we will take the final follow-up measure, assuming more follow-up time has been accrued, thus increasing the statistical power of detecting an effect of the intervention.
Finally, some studies will report results for two or more instruments to assess the same participants, which provides basis for comparing the two instruments. Including both instruments into a statistical analysis violates the assumption of independent findings and moderator analysis will be performed to assess the potential bias of using different instrumentation. The specification with the lowest level of bias will be included in the synthesis.
IX. Unit of analysis and accounting for clustering
Unit of analysis errors occur when interventions are allocated at a different level than the unit of analysis. Clustering has the effect of narrowing the confidence intervals from the true confidence intervals because observations from individuals in the same cluster are likely to be more similar than individuals across the cluster (Higgins, 2011). Therefore, observations from the same cluster cannot be considered to be independent from one another. In clustered designs, reviewers will observe if the assignment units are the same units reported in analysis, and if this is not the case, whether appropriate statistical procedures of analysis including multi-level analysis have been applied. For clustered designs with unit of analysis errors, corrected standard errors and confidence intervals will be used to adjust for clustering
We will adjust standard errors or sample sizes from cluster randomised trials using the methods described in the Cochrane Handbook for Systematic Reviews of Interventions (Higgins, 2011). The adjustment method requires the intra-class correlation co-efficient (ICC) where available to estimate the design effect. The ‘design effect’ is given as
deff=1+ICC(m-1) where m is the mean cluster size
The ‘effective sample size’ ESS = total sample size/design effect
Standard errors associated with the SMD in cluster studies may be inflated by the square root of the design effect unless where authors have implicitly adjusted for the clustering.
X. Approach to Meta-analysis
Quantitative effect estimates will be synthesised using inverse-variance random effects model meta-analysis in STATA 13.0 (StataCorp, 2013). Using the random effects model allows the true effects in each study to vary according to some distribution.
We will separately analyse results from experimental studies and quasi-experimental studies as per The Campbell Collaboration guidelines. Only quasi-experimental studies that have adjusted for similar baseline covariates, or matched on related covariates, or applied similar statistical models to estimate effect sizes will be combined as these are deemed more comparable. If the outcomes are deemed to be too dissimilar to combine in a meta-analysis, we will display effect sizes in a forest plot, but without the final pooled effect size.
Sensitivity analysis will be conducted, where outlying studies that exhibit a high or low effect size will be excluded, as well as those with wide confidence intervals. Meta-regression will be performed to investigate sources of heterogeneity if sufficient studies are included for statistical synthesis.
We acknowledge that meta-analysis may not be always sensible if studies are not addressing the same question, measuring the effect of a similar intervention, or if the populations are different. This might imply that the outcomes are different and combining these for a mean estimate would not be sensible.
XI. Assessment of heterogeneity
Heterogeneity will be assessed using inspection of the forest plots for lack of overlap of confidence intervals, and using statistical tests of heterogeneity using the Q statistic (Hedges & Olkin, 1985). We will also calculate and report the I2, and Tau2 to provide estimates of the magnitude of the variability that is due to heterogeneity (Higgins, 2002; Higgins, 2003).
XII. Moderator analyses
It is possible that the effect sizes of intervention on outcomes vary according to levels of some moderating covariates. If feasible, we plan to investigate three types of moderators as laid out by Lipsey (2009): extrinsic, methodological, and substantive. The specific moderators are listed below.
Extrinsic moderators (unchangeable characteristics of the study): year of publication (to investigate whether treatment effects have changed over time), funder, type of publication.
Methodological (aspects of the study designs): study design, interventions evaluated, outcomes reported, and risk of bias status (Higgins et al., 2011)
Substantive moderators: type of implementer, settings, type of training received, age of participants, gender of participants, socio-economic status of participants, geographical location, type of crop or livestock, climate zone
Moderator analysis will be conducted as a form of sensitivity analysis according to the values taken by the moderator covariates, as specified by Borenstein (2009). The relationship between two categorical moderators will be examined by creating a two-way table where a chi-square test will be determined according to the number of studies with these particular moderator variables. The Pearson correlation will be used to examine the effect of continuous moderators (Cooper & Hedges, 1994).
XIII. Publication bias
We acknowledge that studies that may have evaluated similar interventions and outcomes may be missing. It is important to include all studies where possible and to get additional information, but there is also the possibility that the missing studies are different from the published studies. We will address publication bias in two ways. Firstly, we will conduct an extensive search of the published and unpublished literature and will include studies regardless of publication status. Secondly, we will use statistical methods to test and control for publication bias (Egger et al. 1997; Palmer et al., 2008). The presence of publication bias will be evaluated empirically in studies included in our meta-analysis using funnel plots. The funnel plot is based on the fact that precision in estimating the underlying treatment effect will increase as sample size increases. Results from small studies will scatter widely at the bottom of the plot, with the spread narrowing among larger studies. In the absence of bias the plot will resemble a symmetrical inverted funnel. However, if there is publication bias, funnel plots will often be skewed and asymmetrical.
XIV. Interpreting findings of syntheses
Our findings and judgments will be integrated into summary of findings tables to ease the readers' understanding of our how we have reached our review conclusions. We will apply the GRADE tool to our synthesis to enable transparent and structured interpretation of results (Guyatt et al., 2011).
XV. Treatment of qualitative data
We will not include qualitative studies, but we will extract information about intervention implementation and context from qualitative research or process evidence incorporated within the included studies, as outlined in the coding framework. We will use this information to inform our grouping and analysis of included studies.
SOURCES OF SUPPORT
This review has been made possible thanks to the generosity of many individuals and organisations. It is funded by Foreign Affairs, Trade and Development, Canada (DFATD, formerly CIDA), managed through the International Initiative for Impact Evaluation (3ie), who have also provided some financial support. As such, some extent of the review scope was predetermined. However, after a detailed conversation with funders following two pieces of additional work – a review of reviews and a systematic map – as well as consulting with our advisory group, the scope of the review was refined to its present form.
In addition, the Centre for Anthropological Research at the University of Johannesburg has generously allocated additional staffing to the review. With thanks too to our international advisory group, and peer reviewers for their input to our protocol.
DECLARATIONS OF INTEREST
None. No authors have any involvement in any of the primary studies included in this review. The same review team are also conducting a related review on the impacts of urban agriculture on food security and nutrition.
Stewart, R., Korth, M., Langer, L., Rafferty S., Rebelo Da Silva, N., & van Rooyen, C. (2013) What are the impacts of urban agriculture programs on food security and nutrition in low and middle income countries? A systematic review protocol. Environmental Evidence, 2(7): 1–13.
REVIEW AUTHORS
The lead author is the person who develops and co-ordinates the review team, discusses and assigns roles for individual members of the review team, liaises with the editorial base and takes responsibility for the on-going updates of the review.
ROLES AND RESPONSIBILITIES
We have a large team, deliberately formulated to enable us to build review experience within our Centre in Johannesburg. Some members will have only small roles on the review, whilst others will take the lead on specific elements as outlined below.
Content:
RS, YE, HZ and NRDS will be conducting much of the work required for this review, with occasional support from MK, LL and NM. RS and MK have both conducted systematic reviews of relevant literature (on microfinance including agricultural finance). The team is currently undertaking a parallel review on urban agriculture.
The team will be supported by the other PIs on the review, who have been chosen for their specific expertise: in international development in Africa (MK, YE and TW), in agricultural research (NR), and a new member of the team who specialises in biostatistics and meta-analysis.
Systematic review methods:
RS will be responsible for leading on methods, designing the study and taking responsibility for all stages of the review. Her depth of systematic review experience and previous reviews using a wide range of approaches (including reviews of reviews, systematic maps, traditional effectiveness reviews and rapid evidence assessments) and working with a number of organisations (Cochrane, Campbell, EPPI-Centre, 3ie and Collaboration for Environmental Evidence) means she is well equipped to lead this complex review.
TW, YE, MK, NR, LL and NRDS have all attended training in systematic review methods and have experience of working on reviews. They will draw on this experience in their varying roles in this review:
TW, and NR will be commenting on draft products as the review progresses, as well as systematic review tools (such as the coding framework), and leading our dissemination activities;
HZ and NRDS will be working day-to-day on the review conducting much of the many body of work, collecting, screening, coding, appraising and synthesizing studies alongside RS; An additional statistician is joining the team and will lead the statistical elements of the day to day tasks;
NRDS, supported by LL, will be responsible for the administration of the review, the management of our specialist database, and assisting with collecting, screening and coding the literature.
Statistical analysis:
RS will take the lead in the data analysis for this review, supported by our new statistician, a medical statistician with experience of conducting meta-analyses for systematic reviews.
In addition, an experienced Cochrane-trained bio-statistician, Alfred Musikewa, with expertise in conducting meta-analysis and in providing training to others has offered his advice to the team. Alfred has already has provided training to the team and is available to advise on the statistical meta-analysis for the review as necessary.
MK and RS are both trained in statistics for social research.
RS and NR have advanced training in statistical meta-analysis.
RS, NR and MK are all experienced in narrative synthesis in systematic reviews.
Information retrieval:
Additional technical input on systematic searching is being provided by the EPPI-Centre's Information Scientist, Claire Stansfield. Claire is a specialist in developing and implementing search strategies for systematic reviews in development. She will be supported in this role by RS, who also has experience of these systematic review tasks.
NRDS is experienced in collecting publications for inclusion in systematic reviews and is doing most of our ‘collecting’ supported by the rest of the team. We will benefit from having 3 centres included in this review (EPPI-Centre, Harper Adams University and University of Johannesburg), all of which have different access to publications.
N.B. While the following team are already committed to contributing to this review, we have been approached by a number of stakeholders from international agencies that specialise in agriculture. As well as contributing their expertise via our advisory group, they have expressed an interest in helping with the practical work of the review. It is therefore possible that additional agriculture specialists will provide input to the review. Should they be involved in conducting review tasks they will be trained by RS and will be ‘partnered’ by an experienced systematic reviewer to support their learning whilst assuring the rigour of the review.
PRELIMINARY TIMEFRAME
Approximate date for submission of the systematic review: April 2014
Scope agreed with funders: August 2013 Protocol submitted for peer review: October 2013 Searches and screening: until December 2013 Collection of full texts: September - December 20132
Detailed coding including quality assessment and data extraction: November - April 2014 Synthesis: May 2014 Preliminary findings available for discussion with our advisory group: early 2014 Writing and submission of full review: end May 2014
PLANS FOR UPDATING THE REVIEW
This review is reliant on external funding. Updates will similarly depend on the availability of funds, which is ultimately dependent on the importance of the subject to international agencies. There has been considerable interest in the review from not only our funders DFATD, but also other international agencies such as IFAD.
Our plans are therefore to approach possible funders for backing to update this review in 2016/2017. Ruth Stewart will take responsibility for exploring the potential for funding and liaising with the Campbell International Development Coordinating Group about updates.
AUTHORS' RESPONSIBILITIES
By completing this form, you accept responsibility for preparing, maintaining and updating the review in accordance with Campbell Collaboration policy. The Campbell Collaboration will provide as much support as possible to assist with the preparation of the review.
A draft review must be submitted to the relevant Coordinating Group within two years of protocol publication. If drafts are not submitted before the agreed deadlines, or if we are unable to contact you for an extended period, the relevant Coordinating Group has the right to de-register the title or transfer the title to alternative authors. The Coordinating Group also has the right to de-register or transfer the title if it does not meet the standards of the Coordinating Group and/or the Campbell Collaboration.
You accept responsibility for maintaining the review in light of new evidence, comments and criticisms, and other developments, and updating the review at least once every five years, or, if requested, transferring responsibility for maintaining the review to others as agreed with the Coordinating Group.
PUBLICATION IN THE CAMPBELL LIBRARY
The support of the Campbell Collaboration and the relevant Coordinating Group in preparing your review is conditional upon your agreement to publish the protocol, finished review and subsequent updates in the Campbell Library. Concurrent publication in other journals is encouraged. However, a Campbell systematic review should be published either before, or at the same time as, its publication in other journals. Authors should not publish Campbell reviews in journals before they are ready for publication in the Campbell Library. Authors should remember to include a statement mentioning the published Campbell review in any non-Campbell publications of the review.
I understand the commitment required to undertake a Campbell review, and agree to publish in the Campbell Library. Signed on behalf of the authors:
Form completed by: Dr Ruth Stewart
Date: October 2013
Footnotes
Appendix 1
Appendix 2
Appendix 3
Appendix 4
Appendix 5
1
2
Note that many full texts have already been identified, screened, and collected as part of our preliminary scoping work