PROTOCOL: Incentives for climate mitigation in the land use sector: a mixed-methods systematic review of the effectiveness of payment for environment services (PES) on environmental and socio-economic outcomes in low- and middle-income countries

Abstract

Background

The problem, condition or issue

Around a quarter of all anthropogenic greenhouse gas emissions originate from the agricultural, forest and other land use sector (AFOLU), driven primarily by deforestation, forest degradation and emissions from unsustainable livestock, soil and nutrient management practices (IPCC, 2014). But there is also a large potential for climate change mitigation in the sector, through removal of greenhouse gases in the atmosphere (carbon sequestration) and reduction in emissions from reduced forest and vegetation removal and improved agricultural practices.

AFOLU sector¹ also provides a range of other ecosystem services in addition to climate regulation. Forests and lands provide clean water, regulate soil and provide food, fuel, fiber and fresh water (MEA, 2005). Agriculture provides directly and indirectly for the livelihoods of billions of people, in addition to providing food for all the world's population (FAO, 2016a). The sector also offers livelihoods for an estimated 750 million of the world's extreme poor (FAO, ibid). Finally, forests provide paid employment for at least 100 million people and support the livelihoods of many millions more (FAO, 2016b).

The United Nations Framework Convention for Climate Change (UNFCCC) have recognised the critical importance of reducing emissions from deforestation and degradation for climate mitigation (UNFCCC, 2010). In addition, the IPCC highlights the importance of preservation and restoration of other ecosystems such as peatlands and mangroves for maintaining carbon stocks and reducing emissions (FAO & IPCC, 2017; IPCC, 2014). Improved livestock and crop management also represent practices with mitigation potential (FAO & IPCC, ibid).

The links between climate change, agriculture, forests and human wellbeing are complex.

The world's forest area declined from 4128 million hectares of forest in 1990 to 3 999 million hectares in 2015 (FAO, 2016c). Agriculture, both commercial and subsistence, was the main driver of this global deforestation, accounting for 73 per cent of forest clearance worldwide (FAO, 2016b). This is partially driven by an increasing global demand for food from increasing incomes and growing populations, which is expected to rise 60 per cent from 2006 levels by 2050 (FAO, 2016a). At the same time, climate change is expected to negatively affect all dimensions of food security, including agricultural production of food, quality, food access through the impacts on livelihoods, and food price stability (IPCC, 2014).

These complex relationships make sustainable preservation and management of forests and land, while at the same time ensuring food and livelihoods for the world's population, one of the biggest policy challenges facing the world (FAO, 2016a; FAO, 2016b). Concerns that climate change mitigation programming may have negative knock-on effects on human wellbeing and human rights, especially for the poor, remain. (Stickler 2009; Larson et al. 2013; Lawlor et al. 2013; Mutabazi et al. 2014). It is therefore important to identify strategies that reduce trade-offs between environmental protection and human wellbeing, and ideally programmes that offer win-win solutions.

The intervention

Economic incentives-based programmes, which aim to preserve or restore ecosystems services through financial incentives, have grown in popularity in the last two decades (Pirard, 2012; GEF, 2014; Ezzine-de-Blas et al., 2016). One such incentive-based mechanism is Payment for Environmental Services (PES). PES are a market-based approach, where users of an environmental service pay the owners or managers of the service, conditional on changes in behaviours that are likely to effect the provision of environmental services (Wunder, 2015). PES may be conditional on commitments to protect or restore forest areas or sustainable forest management, such as management of forest fires (Jayachandran et al., 2016; Alix-Garcia et al., 2014). Payments may also be tied to agricultural practices associated with reduction in GHG emissions or increase of carbon stocks, including introduction of agroforestry, silvo-pastoral or integrated crop systems, which combine crops, grazing lands and trees on agricultural land, improved tillage practices such as conservation agriculture, and reduced use of fire in rangeland management (Hedge & Bull, 2011; Garbach et al. 2012).

There is some debate on the definition of PES (Wunder, 2015; Muriadian et al. 2010; Engels et al. 2008). At the most simple level, PES is a voluntary transaction between service users and service providers, conditional on agreed rules for natural resource management that aims to generate environmental services or benefits that are felt off-site, for example carbon sequestration (Wunder et al. 2015). In practice, the service “user” is typically a government or NGO acting on behalf of beneficiaries of the environmental service and the service “providers” are individuals, households or community organisations that own or manage the land or forest areas in the programme.

There are a number of long-standing PES programmes in existence around the world, for example the Pago por Servicios Ambientales-Hidrologico (PSAH) in Mexico and the Sloping Land Conversion Programme (SLCP) in China. The PSAH in Mexico makes payments to landowners conditional on maintenance of certain level of forest cover, according to five-year contracts (Alix-Garcia et al., 2014). If forestland is converted to another land use such as agriculture, the landowner is removed from the programme. The SLCP in China is a large-scale programme that aims to incentivise the conversion of cropland back to forests or grassland through cash and in-kind payments to participating households, to reverse or prevent soil erosion and desertification (Démurger & Wan, 2012). In addition to these long-standing programmes, the number of new PES programmes has grown rapidly in the last decade (Börner et al., 2017). They increasingly also include goals around poverty alleviation.

For example, while the original goal of the PSAH was to maintain the provision of hydrological services from Mexico's forested land, in 2006 the objectives were extended to alleviating poverty (Alix-Garcia et al., ibid).

Because of the restrictions around land use from participating in the programme, implementers of PES programmes sometimes combine them with other activities to support behaviour change, such as awareness raising activities around environmental conservation or capacity building in sustainable resource use (Sharma & Pattanayak, 2015). In some cases they are also combined with more extensive support for livelihoods development. For example, a REDD+ pilot programme in Nepal made incentive-based payments to Community Forest User Groups (CFUGs). In addition to forest carbon monitoring, this programme included awareness raising and capacity building for improving local livelihoods and the use of alternative fuel and cooking technologies (Sharma & Pattanayak, ibid).

How the intervention might work

Payments for Environmental Services (PES) are frequently framed as a response to “market failure” (Arriagada & Perrings, 2009). A market failure occurs when the market does not provide a socially optimum level of a service or good because of the presence of positive externalities for society from providing the service. Carbon sequestration is an example of a public good with positive externalities felt at the global level (Alix-Garcia & Wolff, 2014). While households may get some individual benefits from environmental practices such as keeping trees on land, the larger benefits are felt externally but households are not compensated financially for these external benefits by market mechanisms. This leads to household or individual decisions that are sub-optimal for society, like deforestation.

The overarching theory of how PES works is quite simple. It is designed to act as an incentive for a household or community to contribute to the provision of a socially-optimal level of environmental services, thus correcting the market failure. Figure 1 presents a programme theory for how PES may influence environmental and socio-economic outcomes. The outcomes presented in the model are not the only potential outcomes of PES programmes, however we have chosen to focus on those that are of direct interest in this review.

PES programme theory

How PES may influence environmental outcomes

The intervention aims to influence environmental outcomes primarily through provision of a positive financial incentive to change environment-related behaviours (Pattanayak et al., 2010). Cash or in- kind payments are typically made to participating individuals, households or communities on a regular basis, conditional on the environmental behaviour, for example, payments to landowners to avoid deforestation on their land. Payments may come from private actors that directly benefit from the environmental service, but more typically come from government or non-governmental organisations acting on their behalf. If a participating household or community organisation fails to uphold the minimum environmental service provision, payments are suspended.

The theory underlying PES is that the financial incentives motivate participants to comply with the rules of the programme, resulting in improved land or forest management practices (Alix-Garcia & Wolff, 2014). The theory is that the increase in take-up of these improved practices will ultimately restore, maintain or enhance the provision of the environmental service that has wider benefits for society. The theory assumes that the conservation payments outweigh the benefits derived from business as usual, such as converting forests to agricultural uses, or harvesting wood for energy.

PES may have positive or negative spill-over effects on land that is not enrolled in the programme. If households or communities do not enrol all their land in a programme, resource exploitation pressures may simply move on to the non-enrolled areas, known as leakage or substitution effects (Sills et al., 2008). Similarly, increased household income because of the PES programme may have implications for spending patterns and put increased pressure on local resources (Börner et al., 2017). Conversely, positive spill-overs may occur due to increased forest monitoring resulting from the program or changes in social norms relating to resource use. Such indirect effects can affect the magnitude or even the direction of the effect of a PES programme (Pattanayak et al., 2010).

How PES may influence environmental and socio-economic outcomes

While not originally intended as a tool for poverty alleviation, PES may increase income for complying individuals or households. To directly increase household income, the assumption is that the cash transfer is greater than lost rents previously generated from the enrolled land. Alternatively, payments may also indirectly act as an incentive for households to diversify towards other livelihood activities that are less reliant on practices that reduce the provision of the ecosystems services. For example, participants may move away from agriculture that relies on regular forest clearing towards sustainable forest activities.

However, there are potential trade-offs between poverty alleviation and environmental goals. The effectiveness of PES in improving environmental outcomes is theorised to depend on effective targeting towards those actors that are the biggest threat to the provision of the environmental service (Wunder, 2007; Börner et al., 2017). If the biggest threat comes from larger, better off households or communities, the payment is best targeted towards them, but this will come at the cost of income transfers to poorer families that could support poverty alleviation (Alix-Garcia & Wolffe, 2014).

A range of programme design, implementation and contextual factors may influence the effectiveness of PES programmes. Below are some key design, implementation and contextual variables that are frequently theorised to moderate the effectiveness of PES schemes. In many cases, the theory is not conclusive on whether the impact on effectiveness would be positive or negative and thus on the direction of effects of PES schemes in general (Ferraro, 2017; Pattanayak et al., 2010). These factors will be explored in the review in the analysis of heterogeneity.

Targeting can influence whether PES programmes achieve their objectives. PES programmes are typically voluntary and there is a risk that households that already meet conditions will self-select into the programme. Depending on the opportunity cost of participating in the programme, households may choose to not enrol or only enrol some of their land (Ferraro, 2017). Land enrolled in PES programmes may therefore be land with the lowest value in terms of exploitation potential and thus the least likely to be exploited in the absence of PES. The result of this would be little or no added benefit of the programme in terms of environmental outcomes as land owners may have preserved resources even in the absence of payments.

The lack of additionality may therefore be more prevalent where pre-programme compliance with PES conditions is high (eg: low levels of resource exploitation, as indicated by low baseline deforestation rates for example). Thus, programmes targeted to land that is at a high risk of exploitation may result in higher levels of resource protection. However, this involves predicting where landholders will exploit resources in the future, information that is generally hidden from the policy-maker implementing the PES programmes (Alix-Garcia & Wolffe, 2014).

The size of payments may influence take-up and the extent to which programme participants change their behaviour: If the cost of lost rents from restrictions on land or resource use from participating in the programme are greater than the payments received, a land owner is unlikely to choose to enrol. This requires a payment that is large enough to overcome the opportunity costs for households to decide to participate in the programme and then to stick to conditions. However, because of missing markets the payment size that will induce people to participate in the programme is not observed (Börner et al. 2017).

Timing of payments: the timing of payments may influence how programme participants respond to the financial incentive. There is some suggestion that payments made at the end of the contracted period are most effective at incentivising changes in environmental behaviours (Alix-Garcia & Wolffe, 2014). However, this is often not feasible, particularly in low-income contexts, and often payments are made on a yearly basis.

The characteristics of PES conditions: Even if an improvement in an ecosystem service is the goal of a programme, few PES programmes are conditional on the provision of the ecosystem service itself (such as demonstrated increases in carbon sequestration in forests). In practice, PES program payments are frequently conditional on proxies or changes in behaviours that are likely to affect the provision of the ecosystem service (Wunder, 2015). For example, planting trees on agricultural land to improve carbon sequestration. While the use of proxies is typically easier to observe, there is no guarantee that changes in the behaviour will lead to improved ecosystems provision, particularly where the ecosystem service is heavily influenced by external factors to the programme (Pattanayak et al., 2010; Borner et al. 2017).

The extent to which conditions are monitored and enforced may moderate effects on environmental outcomes (Börner et al. 2017). Monitoring and enforcement of conditions may influence the extent to which programme participants change their behaviour and comply with conditions. A systematic review of the effects of cash transfer payments for building human capital found a larger effect on children's education outcomes when conditions were monitored and enforced (Baird et al., 2013).

Long run programme funding: permanent benefits of PES schemes may depend on continuous programme funding, which may be particularly difficult in government run PES schemes (Engel et al., 2008). On the other hand, payments may act to incentivise people to incur the fixed costs of switching to a more environmentally friendly practice and to “learn by doing” (learn about benefits and learn to reduce variable costs). And once they adopt a new practice, the marginal benefits may outweigh the marginal costs and the practice will persist even in the absence of payments.

Property rights system: weak property rights are a common driver of deforestation and lack of secure property rights may make PES implementation difficult (Alix-Garcia & Wolffe, 2014). Lack of secure property rights may reduce programme take-up rates and compliance as participants are less willing to invest in the sustainable management of land when they are uncertain if they will be able to reap benefits from those investments in the future.

Land tenure system: incentives to change behaviour around land management practices may depend on whether the land is privately owned, collectively owned, state owned or restricted in some way by the state (Robinson et al., 2017). For example, PES payments may have weaker effects on conservation behavior of users living in or near state owned lands than on private land or land held under collective title.

Credit markets: the presence of credit constraints for poor families in LMICs may be a barrier for them to make investments in, or exploit, land (Ferraro, 2017). There may be negative environmental consequences when payments to participating families allow them to overcome these constraints to make investments in unenrolled land, or enrolled land once payments stop, that result in less environmentally favourable land uses.

Why it is important to do the review

Review of existing literature

There is an emerging impact evaluation literature on payments for environmental services (PES) programmes. A 3ie evidence gap map (EGM) published in 2016 identified 41² experimental or quasi-experimental evaluations of PES programmes globally, with most taking place in Low-and Middle-Income Countries (L&MICs). We are only aware of one systematic review on the effectiveness of PES, published in 2014 (Samii et al., 2014). There have also been a large number of non-systematic literature reviews, either presenting narrative discussions on the effectiveness of PES (Börner et al., 2017; Pattanayak et al., 2010; Alix-Garcia & Wolffe, 2014) or presenting a range of effect sizes for PES programmes (Ferraro, 2017).

There are several reasons that warrant an update and extension of the Samii et al. (2014) systematic review. Firstly, the search for the review was completed in August 2013. 3ie's Evidence Gap Map of land use and forestry programmes (Snilstveit et al., 2016) identified at least six new evaluations of PES programmes that have been published since then, including studies from Uganda, Ecuador, Tanzania and new evaluations of long-term programmes in China, Mexico and Costa Rica. Secondly, Samii et al. (2014) were unable to do a meta-analysis for income and poverty related outcomes and for forest condition due to lack of data and heterogeneity between studies. Given the increase in the evaluation evidence base since then, we hope to be able to undertake additional meta-analyses.

Thirdly, Samii et al.'s review focused on PES for forest areas. We will expand the scope of the review to include PES in other settings such as farmland, mangroves and grasslands. A number of PES programmes target other important environmental behaviours of relevance to climate change mitigation programming, for example payments to incentivise farmers to take up agroforestry on their farmland (Hedge & Bull, 2011). This will be the first review that we are aware of to systematically cover the literature on the effectiveness of PES in these areas.

Finally, this review will answer new questions around design, implementation, context and costs of programmes, in addition to assessing programme effects. In doing so we will look at a broader range of literature, including process evaluations, programme documents and associated qualitative studies for the programmes evaluated in included impact evaluations.

Relevance to policy and practice

It is estimated that additional global investments of US$35 billion in the agriculture sector and US$21 billion in the forestry sector will be needed by 2030 to mitigate the effects of climate change (UNFCCC 2009). At the landmark United Nations Climate Change Conference (COP 21) in 2015, countries agreed to conserve and enhance sinks of greenhouse gases, including forests (UNFCCC, 2015). To ensure resources are used effectively to achieve agreed mitigation objectives it is important to ensure that decision-makers have access to reliable evidence.

The United Nations Reducing Emissions from Deforestation and Forest Degradation mechanism (REDD+) is one of the main frameworks for making payments to L&MICs to preserve and sustainably manage forests. There are significant resources pledged to the REDD+ initiative. At the COP21, Germany, Norway and the UK announced that they would provide US$ 5 billion between 2015 and 2020 to forest countries if they could demonstrate verified emissions reductions (BMUB, 2015). The UN-REDD Programme currently supports 64 countries across Africa, South and East Asia and Latin America and the Caribbean to enable their participation in REDD+, and 47 so far have qualified (UN-REDD, 2016).

PES are promoted as an important tool by REDD+ and are supported by a range of actors, from national governments to multi-national institutions such as IFAD, UNDP and the World Bank (GEF, 2014). The number of PES programmes operating in L&MICs has rapidly increased. A recent global review of PES identified hundreds of programmes mentioned in the literature, with 55 programmes currently in operation around the world that clearly fit the classic definition of PES (Ezzine-Blas et al., 2016). The Global Environmental Facility (GEF) alone has supported 57 projects containing elements of PES since its inception, totalling investments of over $225 million, in addition to $1.59 billion leveraged from co-financing (GEF, 2014).

Despite their popularity, key policy questions around the effectiveness of PES remain unanswered (Samii et al., 2014; Ferraro, 2017; Le Velly & Dutilly, 2016). One of these questions is the extent to which the environmental and poverty reduction goals of such a programme conflict or present strategies that can generate both environmental and poverty reduction benefits. A second, and equally important question is if PES generate environmental benefits that are additional to ‘business as usual’. To meet UNFCCC emissions targets, governments implement PES programmes on the assumption that by compensating some groups to reduce their emissions, emissions in other sectors are offset (Nhantumbo & Camargo, 2015).

Evaluations of PES programmes finding small effects have led some to dismiss it as an important mechanism. Indeed, a recent FAO-IPCC (2017) report on climate change and land use following the Paris Agreements stated that “[PES] effectiveness, however, is limited and they are more readily applied in some sectors (e.g. forest management) than in other emerging concerns (land restoration, soil health and soil carbon)” (FAO-IPCC, 2017: 28). The report concludes that for PES programmes to be effective, they must be better designed and informed by meta-analysis of the effects of previous programmes.

A range of policy alternatives to PES exist, including private sector zero-deforestation commitments (Climate Focus, 2015) and community forestry initiatives (Agrawal & Angelsen, 2009; Angelsen, 2009). The effectiveness of many of these approaches is also contested and should be subject of future reviews. While PES may be one of the most popular policy tools in the sector, it is important to assess the relative costs and effectiveness of the approach, facilitating comparison with other options in the future.

Given the resources dedicated to PES and the global importance of effective climate change mitigation activities, it is essential that rigorous and comprehensive evidence is available to policy-makers and implementers. To help inform decisions about how to use available resources most effectively we will provide a comprehensive review and synthesis of the evidence on the effects of PES, including an assessment of how intervention design, implementation and contextual factors moderate outcomes and cost-effectiveness.

Objectives

The objective of this review is to assess the effects of PES programmes on environmental and socio-economic outcomes in low- and middle-income countries (L&MICs). This will include identifying and synthesising evidence on how PES programme effects vary by programme design, implementation, context³; and by sub-groups of PES programme participants. We will also attempt to assess the cost-effectiveness of PES programmes.

To address these objectives, we will answer the following questions:

What is the effectiveness of PES programmes on intermediate, environmental and socio-economic outcomes in L&MICs? a)

Do PES programs simultaneously deliver positive environmental and socio-economic effects?

Do effects vary by sub-groups of people participating in PES programmes, including low-income groups, women and indigenous people?

Do effects vary by type of environmental services targeted?

To what extent do design and implementation features moderate the effectiveness of PES programmes?

In which contexts are PES programmes effective (or ineffective)? What are the contextual barriers to, and facilitators of, programme effectiveness?

What is the cost-effectiveness of PES programmes?

Methodology

The review will follow the Campbell and Cochrane Collaborations' guidelines to systematic reviewing (The Steering Group of the Campbell Collaboration, 2016; Hammerstrøm et al., 2010; Higgins & Green, 2011; Shadish & Myers, 2004). The review will also draw on the concepts of theory-based impact evaluation (White, 2009) and theory-based systematic reviews (Snilstveit, 2012; Waddington et al., 2012) to provide a mixed-methods systematic review and analysis along the causal chain, to also address questions related to intervention design, implementation and context.

To do so we will include studies in two phases. To address questions 1a, b and c, we will include studies meeting the impact evaluations study design criteria, presented below. To address questions 2, 3 and 4, studies that meet these criteria will be used as the basis for a second, targeted search to identify and include qualitative studies, project documents, process evaluations and cost data on the programmes examined.

Criteria for including and excluding studies

Types of population

We will include studies of programmes in countries classified by the World Bank as lower income, lower-middle income, or upper middle income (L&MICs). We use the classification of the country in the year of the initiation of the program under study. There are several reasons why we decided to focus on L&MICs only. Some scoping of the literature suggests that the impact evaluation literature on PES from high-income countries (HICs) is significantly smaller and does not typically use methods that would be included in the review (Snilstveit et al., 2016; Schomers & Matzdorf, 2013). It does not typically self-identify as PES (Schomers & Matzdorf, 2013; Ezzine-de-Blas et al., 2016) and would likely result in a need to search a separate literature. This is likely to add a significant amount of work to the searching and screening with only a potentially very small number of included studies. In addition, L&MICs contain most of the world's tropical forests, which offer the greatest potential for climate change mitigation in the AFOLU sector, such as climate regulation, watershed protection and carbon sequestration (Pattanayak et al., 2010). Similarly, the findings from the HIC literature will be less relevant for mechanisms such as REDD+. Finally, given that one of our main objectives is understanding the potential for PES to offer “win-win” environmental and poverty alleviation solutions, L&MIC settings offer a more likely setting for answering this.

We will include studies targeted at populations living in or near to forests, agricultural land, wetlands, grasslands and mangroves. Forests are defined as an area over 0.5 hectares with trees higher than five metres and canopy cover more than 10 per cent (FAO, 2012), including mangrove forest areas. Grasslands are areas with tree or shrub canopy cover below 10 per cent but with herbaceous plant cover (FAO, 2005).

Studies of programmes in HICs will be excluded.

Types of interventions

We will include studies of PES programmes, defined as those providing payments to owners or managers of land, conditional on some minimum environmental/ ecosystems service provision. Payments can be either cash or in-kind material transfers, such as seedlings, api-culture and fencing. Ecosystems services are defined as the benefits that humans get from ecosystems (MEA, 2005). In ideal type PES programmes, payments are conditional on the provision of the ecosystem service itself, for example payments for increased carbon sequestration in forests (Le Velly & Dutilly, 2016). However, in practice most PES program payments are conditional on changes in behaviours that are likely to affect the provision of the ecosystem service, for example reducing deforestation or planting trees on agricultural land. We will include payments tied either to the provision of an ecosystem service or to any of the following practices related to climate-regulating ecosystems services: forest protection or regeneration; sustainable forest management practices; sustainable watershed management; sustainable agricultural practices; sustainable livestock management.

The payments can be made to an individual, household, community or organisation and can either be conditional on a specified environmental commitment, for example on the fulfilment of an obligation to maintain a certain forest cover on land, or paid in advance of the PES programme. We will not limit inclusion of these programmes by the funder/ implementer (private versus public for example) or status of land (private land or state-owned/ protected land). Finally, we will include programmes that study PES alone or in combination with other intervention activities, for example interventions supporting alternative livelihoods.⁴

Types of outcomes

We will include studies that assess the impact of PES on either environmental, socio-economic or intermediate outcomes, as defined below. PES programmes often have multiple objectives, related to both the preservation or restoration of environmental services and human welfare. There is a considerable literature on the potential trade-offs or complementarities between these objectives. By looking at both sets of outcomes, we aim to inform this debate.

We will also include studies that assess intermediate outcomes such as changes in agricultural, forest or land management practices. This will allow us to report on effects at earlier stages of the PES causal chain.

Intermediate outcomes

We will include studies that assess changes in land or forest management practices, defined as measures of the type, frequency, intensity or adoption of such practices at the household or community level. We will also include studies that assess the adoption of sustainable agricultural practices or technologies, for example incorporating trees into agricultural or grazing lands. We will also assess measures of forest dependence, for example resource extraction.

Environmental outcomes

We will include environmental outcomes that are related to greenhouse gas emissions or carbon storage/ sequestration. This includes both direct measures of emissions (CO₂, CH₄, N₂0) or carbon storage/ sequestration and proxies for such outcomes. Based on previous mapping work in this area, we know that there are few evaluations that measure provision of environmental services such as carbon sequestration (Snilstveit et al. 2016). Proxy outcomes include deforestation rate, forest cover, forest condition/ degradation, forest fires, soil quality, and so on. We will accept whichever measure is used by the study authors. Once we have identified all studies, we will map all outcomes to determine if they are sufficiently similar for meta-analysis.

We will also include outcomes related to the spillover effects of PES programmes on to land or forests not enrolled in PES programmes.

Socio-economic outcomes

We will include any measures of socio-economic outcomes, including income, consumption, well-being, livelihood security and assets of communities / households / individuals participating in PES programmes. We will also include measures of food security across the four dimensions of food availability, access, utilisation and stability included in the Declaration on Food Security (FAO 2009). These include food consumption, food expenditure, prevalence of undernourishment and nutritional status (FAO 2013). We will accept whichever socio-economic measure is used by the study authors. Once we have identified all studies, we will map all outcomes to determine if they are sufficiently similar for meta-analysis.

Types of study designs

We will include studies in two stages, in a similar approach to Snilstveit et al. (2015). In the first stage, we will include studies that assessed the effects of interventions using experimental designs or quasi-experimental designs with non-random assignment that allow for causal inference (to address primary research question 1). Specifically we will include the following:

Studies where participants are randomly assigned to treatment and comparison group (experimental study designs);

Studies where assignment to treatment and comparison group is based on other known allocation rules, including a threshold on a continuous variable (regression discontinuity designs) or exogenous geographical variation in the treatment allocation (natural experiments);

Studies with non-random assignment to treatment and comparison group that include pre-and post-test measures of the outcome variables of interest to ensure equity between groups on the baseline measure, and that use appropriate methods to control for selection bias and confounding. Such methods include statistical matching (for example, propensity score matching, or covariate matching), regression adjustment (for example, difference-in-differences, fixed effects regression, single difference regression analysis, instrumental variables, and ‘Heckman’ selection models).

Studies with non-random assignment to treatment and comparison group that include post-test measures of the outcome variables of interest only and use appropriate methods to control for selection bias and confounding, as above. This includes pipeline and cohort studies.

Ferraro and Miranda (2014; 2017) argue that combining panel data with baseline observations and statistical matching is the most effective quasi-experimental method at reducing bias when evaluating conservation sector programmes. However, given the expected small size of the evidence base, we will include studies with post-intervention outcome data only as long as they use some method to control for selection bias and confounding. To account for the differences in the quality of study designs and analysis methods, we will appraise the risk of bias in all included studies and do sub-group analysis by risk of bias status.

Before-after studies and observational studies without control for selection bias and confounding will be excluded. Additionally, modelling based studies, commentaries and literature reviews will be excluded.

To address questions 2 and 3 on programme design, implementation and context, we will extract descriptive and qualitative data from the included experimental and quasi-experimental studies. In addition, we will conduct a targeted search for additional papers on the programmes covered by the included impact evaluations to provide additional detail on these areas. In order to be included, these papers must be related to the programmes in the included impact evaluations and also be one or more of the following types of studies⁵:

A qualitative study collecting primary data using qualitative or quantitative methods of data collection and analysis, and reporting some information on all of the following: the research question, procedures for collecting data, sampling and recruitment, and at least two sample characteristics.

A descriptive quantitative study collecting primary data using quantitative methods of data collection and descriptive quantitative analysis and report some information on all of the following: the research question, procedures for collecting data, sampling and recruitment, and at least two sample characteristics

A process evaluation assessing whether a programme is being implemented as intended and what is felt to be working more or less well, and why (HM Treasury, 2011). Process evaluations may include the collection of qualitative and quantitative data from different stakeholders to cover subjective issues, such as perceptions of intervention success or more objective issues, such as how an intervention was operationalised. They might also be used to collect organisational information;

A project document providing information about planned, ongoing or completed programmes. They may describe the background and design of an intervention, or the resources available for a project for instance. As such, these documents do not typically include much analysis of primary evidence, but they provide factual information about interventions. The purpose of including them in our review is to ensure we had sufficient information about the context and interventions in included studies

To address question 4 on cost-effectiveness we will include economic evaluations. We will also use any economic evaluation or cost data provided in any of the studies included under the criteria above.

Types of comparison

We will include studies with a comparison group that receives no intervention (including wait-list comparisons), business as usual, or a different environmental intervention.

Studies that only include a temporal (before-after) comparison will be excluded.

Other criteria for including and excluding studies

We will not impose any restriction on inclusion of studies by language of publication or publication status. However, we will undertake our searches in English. We will search the literature back to 1990, excluding any studies published before this date. This date date cut off is justified by both previous reviews of the literature, as well as the implementation of PES as a policy instrument for reducing deforestation. An evidence gap map covering PES interventions that searched back to 1990 did not identify any studies published before 2000 (Puri et al., 2016). Moreover, PES was pioneered by Costa Rica as an approach to reducing deforestation in the late 1990s (add ref) and REDD was first discussed at the UNFCCC conference of the parties in 2005 (UNFCCC, 2005). Thus implementation and studies of PES is unlikely to have taken place before 1990.

An overview of the inclusion criteria is provided in Table 1.

Table 1

Summary of inclusion criteria

Characteristics	Inclusion criteria
Population	Populations living in or near forests, wetlands, grasslands, mangroves and farmland areas in countries classified by the World Bank as Low-or-Middle Income (LMICs)
Interventions	Payments for environmental services programmes
Comparisons	Comparison group that receives no intervention (including wait-list comparisons), business as usual, or a different environmental intervention
Outcomes	Intermediate, environmental and socio-economic outcomes
Study design	To answer question 1, experimental and quasi-experimental studies. To answer questions 2 and 3, qualitative studies, descriptive quantitative studies, process evaluations, project documents
Other	No inclusion restrictions by publication status or language.

Search strategy: Studies to address review question 1

We will implement a systematic and comprehensive search strategy, developed in consultation with an information specialist, as outlined below.

Electronic searches

We will search a range of databases and websites, including general sources of social science literature as well as sources specific to climate change, forestry, agriculture and impact evaluation. To reduce the potential for publication bias, this will include both academic databases as well a range of specialist organisational websites and repositories of impact evaluations in international development. The sources covered by the search are listed below.

Bibliographic databases:

CAB Abstracts: http://www.cabi.org/publishing-products/online-information-resources/cab-abstracts/

Web of Science: http://wok.mimas.ac.uk/

Greenfile (EBSCO): https://www.ebscohost.com/academic/greenfile

Econlit: https://www.aeaweb.org/econlit/

AgEcon: https://ageconsearch.tind.io/?ln=en

IDEAS/RePeC (EBSCO Discovery): https://www.ebscohost.com/discovery

Agris (EBSCO Discovery): https://www.ebscohost.com/discovery

Specialist organisational databases:

Centre for International Forestry Research (CIFOR): http://www.cifor.org/library/

International Food Policy Research Institute Library (IFPRI): http://library.ifpri.info/discover/collections/

International Institute for Environment and Development (IIED): http://pubs.iied.org/about/

World Agrofresty Centre (ICRAF): outputs.worldagroforestry.org/

ATAI Research: https://www.atai-research.org/emerging-insights/?

Global Environment Facility Evaluation Office: http://www.gefieo.org/evaluations/all?f[0]=field_ieo_grouping%3A312

Conservation Evidence: http://www.conservationevidence.com/

Climate Change Agriculture and Food Security (CCAFS) publications: https://ccafs.cgiar.org/publications

Conservation International publications: http://www.conservation.org/publications/Pages/default.aspx

IUCN Library: https://portals.iucn.org/library/dir/publications-list

Biodiversity International: http://www.bioversityinternational.org/e-library/publications/

Bilateral and multilateral agencies and general repositories of impact evaluations in international development:

World Bank Open Knowledge Repository: https://openknowledge.worldbank.org/

DFID Research for Development (R4D): http://r4d.dfid.gov.uk/

Inter-American Development Bank Publications: https://publications.iadb.org/facet-view?locale-attribute=en&field=type_view

African Development Bank (AfDB): https://www.afdb.org/en/documents/publications/

Asian Development Bank (ADB): https://www.adb.org/publications

United Nations Development Programme (UNDP): http://www.undp.org/content/undp/en/home/library.html

United National Environmental Programme: http://www.unep.org/publications/

International Fund for Agricultural Development (IFAD): https://www.ifad.org/pub/overview

Food and Agriculture Organisation of the United Nations (FAO): http://www.fao.org/publications/en/

3ie Repository of Impact Evaluations http://www.3ieimpact.org/en/evidence/impact-evaluations/

3ie RIDIE (Registry for International Development Impact Evaluations): http://ridie.3ieimpact.org/

Innovations for Poverty Action (IPA): http://www.poverty-action.org/projectevaluations

J-Poverty Action Lab: https://www.povertyactionlab.org/evaluations

Targeted search for addressing review questions 2 and 3

Once we have identified our set of included impact evaluations, we will undertake targeted searching for qualitative studies, process evaluation, project documents and economic evaluations for those interventions evaluated in the included studies. We will conduct citation tracking of included studies to identify relevant sister papers and conduct internet and database searches using the names of programs from included studies. To identify project documents and process evaluations we will conduct targeted searches of databases of project documents and websites of implementing agencies. Finally, we will contact authors and implementing agencies to request available project documentation.

Screening

We will import all search results into EPPI-Reviewer 4. Once duplicates have been removed we will screen citations against review inclusion criteria at title/ abstract and full-text. At the title/abstract screening stage, we will make use of innovative text mining technologies to speed up the initial screening workload and test the potential for reductions in screening workload (O'Mara-Eves et al., 2015; Shemilt et al., 2016). We will use two functions in EPPI Reviewer to do this: the priority-screening function and inclusion/ exclusion classifier. We will rely on the first option in the list below to include studies in the review, but will compare the results of 2 and 3 retrospectively to assess reliability:

Full independent double screening using the priority screening function to order results by probability of inclusion, based on a training set of screening;

Single screening using the priority screening function with a “safety first” approach (an option to mark unclear studies for review by a second screener) (Shemilt et al., 2016);

Single screening using the priority screening function combined with the use of the classifier function to auto-exclude studies with a very low probability of inclusion;

The priority screening function can be used at the title/ abstract screening stage to prioritise the items most likely to be ‘includes’ based on previously included documents. This will involve screening a random test set of at least 500 citations to train the priority screening function, which will learn to identify relevant records based on key-words in the title and abstract of the included and excluded studies. Using priority screening in this way allows for the identification of includable records at an earlier stage in the review process so that work can begin earlier on full-text screening and data extraction. We will do this using both independent double screening. We will also have a single reviewer doing the screening independently with a safety first approach in order to compare results. We will also use the priority screening function to develop a classifier that will retrospectively classify studies into groups based on their probability of inclusion in the review. We will test the reliability of automatically excluding studies with a low probability of inclusion (for example less than a 10 per cent chance of inclusion), by comparing the results to the first approach.

Independent double screening is typically considered the most reliable approach to screening in systematic reviews. However, this approach is also very resource intensive. In the ‘single screening with text mining’ approach the machine effectively plays the role of the second screener. Moreover, before applying text mining all reviewers will be allocated the same set of 100 randomly selected records for independent screening to establish inter-rater reliability, followed by a meeting to discuss any disagreements.

At the full-text screening stage, a random sample of 20 per cent of the records will be independently double screened by two reviewers, followed by a meeting to discuss any disagreements, to establish inter-rater reliability in the application of inclusion criteria. For the remaining records, we will move to single screening using a ‘safety first’ approach (Shemilt et al., 2016), with an option for reviewers to put any papers where inclusion is unclear in a ‘provisionally include’ folder for screening by a senior reviewer.

Data extraction and coding procedures

We will use a standardised data extraction form to extract data from included papers (the full data extraction form is included in appendix 2). We will use Excel and EPPI reviewer and extract data on the following categories of information:

descriptive data on study design, intervention and context for purposes of descriptive analysis of the body of research;

data on the population, context, study design, intervention design, process and implementation and cost for purposes of moderator analysis and qualitative synthesis addressing questions 2 and 3

data on the outcomes of interest and sample size for purposes of effect size calculation

Critical appraisal

Assessment of risk of bias in experimental and quasi-experimental studies

We will assess the risk of bias in the included impact evaluations using criteria as suggested by an adapted version of the Cochrane Risk of Bias Tool (Hombrados and Waddington, 2012). We will assess risk of bias based on the following criteria, coding each paper as ‘Yes’, ‘No’ and ‘Unclear’ according to how well they address each domain:

Baseline confounding and selection bias: was the allocation or identification mechanism able to control for baseline confounding and sample selection bias?

Time-varying confounding: was the method of analysis executed adequately to ensure comparability of groups throughout the study?

Bias due to missing data: is the estimation method sensitive to non-random attrition?

Biases in outcome data collection: was the process of being observed causing motivation bias (Hawthorne and John Henry effects, courtesy bias, and recall bias)?

Departures from intended interventions: was the study adequately protected against performance bias and survey effects?

Outcome & analysis reporting biases: was the study free from outcome reporting bias and analysis reporting bias?

Two reviewers will undertake the risk of bias assessment independently, with disagreements resolved by a third reviewer. We will report the results of the assessment for each of the assessed criteria for each study. We aim to explore if there are systematic differences between primary studies with different risk of bias. If meta-analysis is feasible, we will conduct sensitivity analysis to assess the robustness of the results to the risk of bias in included studies.

Assessment of quality in descriptive quantitative studies, qualitative studies and process evaluations

We will assess the quality of included qualitative studies, process evaluations and descriptive quantitative studies using an adapted version of the Critical Appraisal Skills Programme checklist (CASP, 2006) and Pluye and colleagues' (2011) mixed-methods appraisal tool. The developed tool will make judgments on the adequacy of reporting, data collection, presentation, analysis and conclusions drawn. The appraisal will assess the quality of the included qualitative studies and descriptive quantitative studies using six appraisal domains:

The defensibility of the applied research design to answer the research question under investigation.

The defensibility of the selected research sample and the process of selecting research participants.

The rigour of the technical research conduct, including the transparency of reporting.

The rigour of the applied analysis and credibility of study's claims given the nature of the presented data.

The consideration of the study's context (for qualitative studies only).

The reflexivity of the reported research.

We will filter out studies of particularly low quality at this stage, using a fatal flaw approach following Dixon-Woods et (2005). Studies that do not meet either criterion of appraisal domains 1–4 above will be excluded from the synthesis. That is, they will be included in the review and we will report on the studies' descriptive data, for example applied intervention. However, no research findings will be extracted from these studies to feed into the review's synthesis. Each appraisal domain will be assessed from a scale of low quality to medium and high quality. An overall critical appraisal judgement per study will be allocated using a numerical threshold of the appraised quality domains (Appendix XX).

We will not undertake a critical appraisal of included project documents. They typically provide information about planned, ongoing or completed programmes, providing information about the design or resources available for a project for instance. As such these documents do not typically include much analysis of primary evidence, but they provide factual information about interventions. The purpose of including them in our review is to ensure we have sufficient information about the context and interventions included in our review. We will therefore focus the appraisal on assessing the relevance of the documents against the interventions assessed in our review. Before extracting any data, we will ensure that the name of the intervention, the implementing agency, context and timeline of the intervention described in the project document corresponds to the intervention assessed in the impact evaluation included in our review. Finally, collecting data from a range of sources, especially if used for triangulation, can enhance confidence in the trustworthiness of the information included (Montgomery et al., forthcoming). If several sources are available, we will extract data from all sources for purposes of triangulation. If we are doubt about the relevance of a particular document, we will contact the authors.

Effect size calculation

Where possible we will extract the necessary data to calculate standardised effect sizes. We expect most studies to be measuring continuous outcomes. For these outcomes we will calculate the Hedges' g sample-size corrected standardised mean difference (SMDs), its variance and standard error using formulae provided in Borenstein et al. (2009, Chapter 4).

The decision as to which formula we use to calculate effect sizes will be made taking into account what has been reported in the majority of the studies sharing common outcomes. We will use the most appropriate formulae for calculating effects sizes, considering the types of study designs we identify and the data they report. Based on our mapping of the literature we expect the majority of included studies to be quasi-experimental designs with outcome measures reported either as regression coefficients (partial (adjusted) estimates) or mean differences, with standard errors or t-statistics and sample sizes. Typically studies do not report standard deviations.

We therefore anticipate using one of the formulae listed below (in hierarchical order of preference) (Lipsey & Wilson, 2001):

For studies reporting regression coefficients:

d = \frac{2 t}{\sqrt{n_{t} + n_{c}}} V a r_{d} = \frac{2}{n_{t} + n_{c}} + \frac{2}{4 (n_{t} + n_{c})}

Where n denotes the sample size of treatment group (t) and control (c). We will calculate the t-statistic (t) by dividing the coefficient by the standard error. If the authors only report confidence intervals and no standard error we will calculate the standard error from the confidence intervals. If the study does not report the standard error, but report t we will extract and use this as reported by the authors.

Studies reporting other data than coefficients and standard errors:

Studies reporting mean differences

Δ \bar{X}

(between treatment (T) and control (C) and standard deviation (SD) at follow up (p+1) :

d = \frac{Δ {\bar{X}}_{p + 1}}{S D_{p + 1}} = \frac{{\bar{X}}_{T p + 1} - {\bar{X}}_{C p + 1}}{S D_{p + 1}}

Studies reporting mean differences between treatment and control, standard error (SE) and sample size (n):

d = \frac{Δ {\bar{X}}_{p + 1}}{SE \sqrt{n}}

Studies reporting means and standard deviations for treatment and control groups at baseline (p) and follow up:

d = \frac{Δ {\bar{X}}_{p} - Δ {\bar{X}}_{p + 1}}{S D_{p + 1}}

, where

\begin{array}{l} S D_{p + 1} = \sqrt{\frac{(n_{T p + 1} - 1) S D_{T_{p + 1}}^{2} + (n_{C p + 1} - 1) S D_{T_{p + 1}}^{2}}{n_{T p + 1} + n_{C p + 1} - 2}} \\ V a r_{d} = ((\frac{n T + n C)}{n T * n C}) + (\frac{d^{2}}{2 (n T + n C})) \end{array}

Studies reporting proportions (r) in treatment group and control:

d = \ln [\frac{r_{T} (1 - r_{T})}{r_{C} (1 - r_{C})}] \frac{\sqrt{3}}{π}

Dependent effect sizes can arise when one study provides multiple results for the same outcome of interest or multiple studies use the same dataset and report on the same outcome. Dependent effect sizes are problematic because the traditional estimation of a mean effect size relied on the statistical assumption of independence of each included estimation of effect (Gleser & Olkin, 2007). We expect a large number of PES evaluations will report multiple, dependent effect sizes and therefore this is an important issue to address (Snilstveit et al. 2016). We will therefore follow the rules laid out below for deciding on inclusion in meta-analysis.

We will only include one effect estimate per sample in a single meta-analysis. This is with the exception of cases where we identify ten or more effect sizes for the same meta-analysis; in these cases, we will combine dependent effect sizes within the same meta-analysis and use robust variance estimation (Hedges, Tipton, & Johnson, 2010; Tanner-Smith & Tipton, 2014).

Where we identify several papers that report on the same study we will use effect sizes from the most recent publication. Where several studies exist using the same data set or where multiple outcomes are reported from alternate specifications within the same study, we will select the study or specification which is the most similar to other estimates for the same outcome type to enhance the potential for meta-analysis. Where different studies report on the same programme, but use different samples (for example from different regions) we will include both estimates, treating them as independent samples.

Studies may provide estimates at several different time points. In such cases we will identify the most common follow-up period and include the follow up measures that match this most closely in the meta-analysis. Nevertheless we will extract data and calculate effect sizes for all time points and report these in the review.

If we identify studies with multiple treatment arms and only one comparison group, we will estimate a treatment effect from both arms. Ideally, we will code and synthesize both effects within the same meta-analytic model, accounting for the dependency using robust variance estimation. However, should the meta-analytic model have less than 10 studies, we will choose the effect estimate from the treatment arm that tests an intervention that most commonly resembles the other interventions included in the meta-analysis.

Unit of analysis

We will assess studies for unit of analysis errors, where the unit of the treatment is different to the unit of analysis, without taking account of clustering in the analysis (The Campbell Collaboration, 2014). If unit of analysis errors exist we will correct for this by adjusting the standard errors using formula provided in Hedges (2011).

Missing or incomplete data

We will contact study authors when there is missing or incomplete data for calculating effect sizes. If we are unable to obtain the necessary data, we will report on the descriptive characteristics of the study but state that it was excluded from the meta-analysis or reporting of effect sizes due to missing data.

Calculating cost estimates

We will estimate the incremental costs by building a profile of inputs, resource use and costs for each included intervention, drawing on the Ingredients Method (McEwans et al. 2012, Dhaliwal et al. 2012) and the resource-use data-coding tool proposed by Shemilt and colleagues (2012). The specification of key intervention “ingredients” will be based on the description of the intervention and the programme theory.

We will use key categories such as personnel, equipment, cash payments, overheads and other programme inputs. We will then capture the quantities of ingredients used, dividing these into fixed and variable costs, and value each input in monetary terms. Where costs of inputs are not available, we will estimate the costs drawing on overall budgets, or using comparable interventions in similar settings.

We will extract data on costs from the included impact evaluations and a range of additional sources including sister papers, as well process evaluations, economic evaluations and programme documents identified through the targeted searches. We will also contact the authors in an attempt to retrieve primary data that can help calculate or estimate intervention costs.

To ensure comparability of cost-estimates across studies, we will adjust costs for price inflation and currency exchange rates, converting all estimates into the same base year. When costs are provided/e in local currencies in nominal terms, we will convert these into US dollars. All cost conversions will be done using the CCEMG-EPPI-Centre Cost Converter (version 1.5, 2016)

Methods of synthesis

Review questions 1, 2 and 3: statistical meta-analysis and meta-regression

We will synthesise evidence on the effectiveness of PES programmes using meta-analysis where possible. We will use inverse-variance weighted, random effects model due to anticipated heterogeneity in the included studies (Higgins & Green, 2011). Where there are too few studies, or included studies are considered too heterogeneous in terms of interventions or outcomes, we will report on the individual effect estimates only. We will combine studies using meta-analysis when we identify two or more effect sizes using a similar outcome construct and where the comparison group state is judged to be similar across the two, similar to the approach taken by Wilson et al. (2011). We will use the metafor package in R software to conduct the meta-analysis (R Development Core Team, 2008; Viechtbauer, W., 2010).

Once we have identified all included studies we will map out all outcome measures provided in the included studies to determine how we will synthesise outcomes. At a minimum, we plan to synthesise deforestation outcomes and household income, based on the current assessment of the literature. Although we have a loose plan in place to guide this process, we also recognise that the nature of determining which outcome measures will be combined can only be determined after we have collected the studies and their corresponding information. Therefore, notably, we will consult with our stakeholders and Advisory Group to help finalize which effects will be synthesised in the same meta-analytic model.

Assessment of heterogeneity

We will assess the heterogeneity of effect sizes graphically using forest plots. We will also assess heterogeneity formally by calculating the Q-statistic, I², and Tau² to provide an overall estimate of the amount of variability in the distribution of the true effect sizes (Borenstein et al., 2009).

Moderator analyses

Depending on the size of the evidence base, we will conduct moderator analysis to explore heterogeneity in the included studies. If feasible, we will use multiple meta-regression to explore the association between the moderator variables and the outcomes of interest (Borenstein et al., 2009). We will use sub-group analysis to explore heterogeneity by different treatment sub-groups. We will undertake the moderator analysis by the following groups of variables:

Methodology: study design, risk of bias status

Substantive variables: intervention characteristics (length of programme exposure, size of transfer, type of condition, including whether the PES targets conservation, restoration of an environment or change to a different, more environmentally favourable land use, whether the PES scheme is government, NGO, multilateral / bilateral institution or user financed, and whether it is a national level, regional or local programme), context (region, country income level, tenure security), participant characteristics (gender, socio-economic status)

Sensitivity analysis

We will conduct sensitivity analysis to assess whether the results of the meta-analysis are sensitive to the removal of any single study. We will do this by removing studies from the meta-analysis one-by one and assessing for changes in results.

Publication bias

We will attempt to reduce publication bias by searching for and including unpublished studies in the review. In an exploratory manner, we will also test for suggestion of publication bias by using funnel plots and Egger et al.'s (1997) test. Given the inherent subjectivity in assessing funnel plot asymmetry, we will assess sensitivity of meta-analyses using ‘trim and fill’ (Duvall & Tweedie, 2000), regardless of whether funnel plots suggest asymmetry. Taken together, the totality of these tests will alert us to the possible presence of publication bias.

Questions 2 and 3: qualitative synthesis

To address questions 2 and 3 we will complement any statistical meta-regressions with a qualitative synthesis (Rubenstein et al., 2009). After having completed the detailed coding of all of the included studies as described above, we will re-review the coding of data on context, intervention design and implementation to identify descriptive findings which remain close to the findings in the primary studies (following Thomas and Harden, 2008). We will then conduct cross-case analysis (Miles & Huberman, 1994), using a framework based upon the links and assumptions from the program theories of included interventions. We will rank studies by effect size and develop a series of matrices to identify the features of intervention design, implementation and contexts that appear to influence effects.

In case where a sufficient number of studies report detailed qualitative data, we will conduct a thematic synthesis on intervention mechanisms and contexts that mitigate or reinforce intervention effects (Thomas & Harden 2008). In this, we will use inductive coding techniques to identify common descriptive themes based on the reported findings⁶ of the primary studies. We will use EPPI-Reviewer's coding software to illustrate the link between the inductive codes in the primary studies and the identified descriptive themes. Following the identification of descriptive themes, these will then be configured into higher level analytical themes, which present the results of the thematic synthesis. Analytical themes will be configured around mechanisms and contexts in relation to research question 2 and 3 of this review.

Questions 4: cost analysis

Costs and resource use are key considerations in the resource allocation choices of policy-makers and practitioners. Cost analysis and economic evaluation can help inform decisions about the relative efficiency of environmental programmes (Shemilt et al. 2008; Shemilt et al. 2012). The type cost analyses we will undertake will be determined by the availability and quality of data. Where sufficient data are available, we will assess costs and resource use, and conduct cost-effectiveness analysis using the formula provided below. We will discuss the limitations in the interpretation and generalisability of the cost-effectiveness estimates and clearly report all assumptions and underlying calculations used in our cost analysis. If we are unable to calculate cost-effectiveness ratios we will report costs descriptively based on available data.

If sufficient data is available, we will calculate the cost-effectiveness ratio using the following formula:

C E R = \frac{E}{C}

Where E is the incremental effect of the intervention on a given outcome, and C is the incremental cost of the intervention.

Integrated synthesis

The overarching goal for the review is to provide an integrated synthesis of the findings from synthesis of review questions 1, 2, 3 and 4 in a narrative synthesis. We will use the programme theory provided above to present the findings from the different syntheses with the aim of providing an integrated narrative synthesis addressing the objectives of the review. In doing so we will produce summary of findings tables following the GRADE (Schünemann et al., 2011) and CerQual approaches (Lewin et al., 2015) to facilitate the transparent and systematic presentation of our findings.

Author(s)

The review will be undertaken jointly by researchers at the International Initiative for Impact Evaluation (3ie) and the Africa Centre for Evidence at the University of Johannesburg. The team also includes a statistician, an information specialist, a substantive expert and an expert in cost-effectiveness analysis and systematic review methodology.

Sources of support

The Children's Investment Fund Foundation (CIFF) is providing funding for this review.

Preliminary timeframe

We plan to submit the draft review report in the first quarter of 2018, with review completion by June 2018 at the latest.

Footnotes

Appendix 1: Search strategies

Appendix 2: Data extraction

1

The value of ecosystems services to humans was concretised in the Millennium Ecosystems Assessment report published in 2005 (MEA, 2005). They define ecosystems services as the benefits that humans get from ecosystems.

2

This number is quite high as it is inclusive of a broad range of study designs, including cross-sectional studies with identification strategies considered to be at a very high risk of bias.

3

An initial list of potential moderating factors are presented in the section, Moderators analyses

4

We will code details of programme design and flag multi-attribute programmes in the analysis.

5

These criteria draw heavily on

6

We will not use the reported primary data in the included studies, for example interview quotes, as a unit of analysis in the synthesis. We do not expect as sufficient amount of qualitative primary data to be reported in order to allow for a meaningful thematic synthesis.

References

Alix-Garcia

Wolff

, 2014. Payment for Ecosystem Services from Forests. Annual Review of Resource Economics, 6, pp. 361–380.

Alix-Garcia

Aronson

Radeloff

Ramirez-Reyes

Shapiro

Sims

Yañez-Pagans

, (2014) Environmental and socioeconomic impacts of Mexico's payments for ecosystem services program, 3ie Impact Evaluation Report 20. New Delhi: International Initiative for Impact Evaluation (3ie)

Agrawal

, and Angelsen

(2009). Using community forest management to achieve REDD+ goals. In: Angelsen

S.A.

; Brockhaus

; Kanninen

; Sills

; Sunderlin

W.D.

; Wertz-Kanounnikoff (Ed.), Realising REDD: National Strategy and Policy Options, pp. 201–211. Bogor: Center for International Forestry Research (CIFOR).

Angelsen

(2009) Policy options to reduce deforestation. In: Angelsen

; Brockhaus

Kanninen

Sills

Sunderlin

W. D.

and Wertz-Kanounnikoff

(eds) Realising REDD+: National strategy and policy options. pp 125–138. Bogor: Center for International Forestry Research (CIFOR).

Baird

Ferreira

F. H. G.

Özler

, and Woolcock

(2013) Relative Effectiveness of Conditional and Unconditional Cash Transfers for Schooling Outcomes in Developing Countries: A Systematic Review. Campbell Systematic Reviews, 2013 (8).

BMUB (German Federal Ministry for the Environment, Nature Conservation, Building and Nuclear Safety) . (2015) Joint Statement by Germany, Norway and the United Kingdom of Great Britain and Northern Ireland. Unlocking the Potential of Forests and Land Use. Paris, COP21.

Borenstein

Hedges

L.V.

Higgins

J.P.T.

Rothstein

2009. Introduction to Meta Analysis (Statistics in Practice). Chichester: John Wiley & Sons.

Börner

Baylis

Corbera

Ezzine-De-Blas

Honey-Rose

Persson

U.M.

and Wunder

(2017) The Effectiveness of Payments for Environmental Services. World Development, 96, pp. 359–374.

CCEMG-EPPI (2016) CCEMG-EPPI-Centre Cost Converter, available from: http://eppi.ioe.ac.uk/costconversion/ - accessed 6/9-2017.

10.

Climate Focus . 2015. Progress on the New York Declaration on Forests - An Assessment Framework and Initial Report. Prepared by Climate Focus, in collaboration with Environmental Defense Fund, Forest Trends, The Global Alliance for Clean Cookstoves, and The Global Canopy Program.

11.

Critical Appraisal Skills Programme , (2006) 10 questions to help you make sense of qualitative research. Public Health Resource Unit. Available at: http://www.biomedcentral.com/content/supplementary/2046-4053-3-139-S8.pdf.

12.

Dixon-Woods

Agarwal

Jones

Young

and Sutton

, 2005. Synthesising qualitative and quantitative evidence: a review of possible methods. Journal of health services research & policy, 10(1), pp. 45–53.

13.

Duval

and Tweedie

, 2000. A nonparametric “trim and fill” method of accounting for publication bias in meta-analysis. Journal of the American Statistical Association, 95, 89–98.

14.

Egger

Davey Smith

Schneider

and Minder

(1997) Bias in meta-analysis detected by a simple, graphical test, British Medical Journal, no. 315, pp. 629–634.

15.

Engel

Pagiola

, and Wunder

(2008) Designing payments for environmental services in theory and practice: An overview of the issues. Ecological Economics, 65, pp. 663–674.

16.

Ezzine-de-Blas

Wunder

Ruiz-Pérez

Moreno-Sanchez

R.d P.

(2016) Global Patterns in the Implementation of Payments for Environmental Services. PLoS ONE, 11(3): e0149847.

17.

FAO and IPCC (2017) FAO-IPCC Expert meeting on climate change, land use and food security - meeting report. Rome, Italy, 23-25 January 2017.

18.

FAO (2016a) The State of Food and Agriculture: Climate change, agriculture and food security. Rome: FAO.

19.

FAO (2016b) State of the World's Forests 2016. Rome: FAO.

20.

FAO. (2013) The State of Food Insecurity in the World: The multiple dimensions of food security. Rome: United Nation Food and Agriculture Organization.

21.

FAO (2012) Forest Resources Assessment 2015: Terms and definitions. Rome: FAO.

22.

FAO. (2009) Declaration of the World Summit on Food Security. Rome: United Nation Food and Agriculture Organization

23.

FAO (2005) Grasslands of the world. Rome: FAO.

24.

Ferraro

P. J.

, and Miranda

J. J.

(2017). Panel Data Designs and Estimators as Substitutes for Randomised Controlled Trials In The Evaluation of Public Programs. JAERE, 4(1), pp. 281–317.

25.

Ferraro

P. J.

and Miranda

J.J.

(2014) The performance of non-experimental designs in the evaluation of environmental programs: A design-replication study using a large-scale randomized experiment as a benchmark. Journal of Economic Behavior & Organization. 107, pp. 344–365.

26.

Garbach

Lubell

, and DeClerck

F A. J.

(2012). Payment for Ecosystem Services: the roles of positive incentives and information sharing in stimulating adoption of silvopastoral conservation practices. Agriculture, and Ecosystems & Environment, 156, pp. 27–36.

27.

Gleser

L. J.

Olkin

(2007). Stochastically Dependent Effect Sizes. Technical report No. 2007-2. Stanford, California: Stanford University.

28.

Global Environment Facility (2014) GEF Investments on Payment for Ecosystem Services Schemes:GEF

29.

Hammerstrøm

Wade

and Jørgensen

A.-M.K.

(2010) Searching for studies: A guide to information retrieval for Campbell Systematic Reviews, Campbell Systematic Reviews 2010: Supplement 1. Available at: www.campbellcollaboration.org/lib/download/969/

30.

Hedge

, and Bull (2011) Performance of an agro-forestry based Payments-for-Environmental-Services project in Mozambique: A household level analysis. Ecological Economics, 71, pp. 122–130.

31.

Hedges

L. V.

Tipton

and Johnson

M. C.

(2010), Robust variance estimation in meta-regression with dependent effect size estimates. Res. Synth. Method, 1, pp. 39–65.

32.

Hedges

L. V.

(2011) Effect sizes in nested designs. In: Cooper

Hedges

L. V.

and Valentine

J. C.

(eds.) The handbook of research synthesis. New York: Russell Sage Foundation.

33.

Higgins

and Green

(2011) Cochrane handbook for systematic reviews of interventions. (version 5.0.2, updated September 2009), The Cochrane Collaboration.

34.

HM Treasury (2011) Magenta Book [online]. Available: http://www.hm-treasury.gov.uk/data_magentabook_index.htm

35.

Hombrados

G. J.

and Waddington

(2012) Internal validity in social experiments and quasi experiments: An assessment tool for reviewers. Unpublished working document. International Initiative for Impact Evaluation (3ie).

36.

IPCC (2014) Climate Change 2014: Synthesis Report. Contribution of Working Groups I, ii and iii to the Fifth Assessment Report of the Intergovernmental Panel on Climate Change. Geneva: IPCC.

37.

Jayachandran

(2013) Liquidity constraints and deforestation: The limitations of payments for ecosystem services. The American Economic Review, 103(3), 309–313.

38.

Jayachandran

deLaat

Lambin

E. F.

Stanton

C. Y.

(2016) Cash for Carbon: A Randomized Controlled Trial of Payments for Ecosystem Services to Reduce Deforestation. NBER Working Paper No. 22378. Cambridge, MA: NBER.

39.

Le Velly

and Dutilly

(2016) Evaluating payments for environmental services: methodological challenges. PLoS ONE, 11(2): e0149374.

40.

Larson

A. M.

Brockhaus

Sunderlin

W. D.

Duchelle

Babon

Dokken

Huynh

T. B.

(2013) Land tenure and REDD+: the good, the bad and the ugly. Global Environmental Change 23: 678–689.

41.

Lawlor

Madeira

E. M.

Blockhus

and Ganz

D. J.

(2013) Community Participation and Benefits in REDD plus : A Review of Initial Outcomes and Lessons. Forests 4: 296–318.

42.

Lewin

Glenton

Munthe-Kaas

Carlsen

Colvin

C.J.

Gülmezoglu

Noyes

Booth

Garside

and Rashidian

(2015) Using qualitative evidence in decision making for health and social interventions: an approach to assess confidence in findings from qualitative evidence syntheses (GRADE-CERQual). PLoS Med, 12 (10),

43.

Lipsey

Wilson

(2001). Practical Meta-analysis. Thousand Oaks, CA: Sage publications Ltd.

44.

Millennium Ecosystem Assessment (MEA) . 2005. Ecosystems and human well-being: Policy Responses: Findings of the Responses Working Group of the Millennium Ecosystem Assessment. Island Press, Washington, D.C

45.

Miles

M. B.

and Huberman

A.M.

1994. An Expanded Sources book: Qualitative Data Analysis. 2nd Edition. Sage Publications, London.

46.

Muradian

Corbera

Pascual

Kosoy

, and May

P.H.

, (2010) Reconciling theory and practice: an alternative conceptual framework for understanding payments for environmental services. Ecological Economics, 69 (6), pp. 1202–1208.

47.

Mutabazi

K. D.

George

C. K.

Dos Santos

A. S.

, and Felister

M. M.

(2014) Livelihood Implications of REDD+ and Costs-Benefits of Agricultural Intensification in REDD+ Pilot Area of Kilosa. Tanzania. Journal of Ecosystems & Ecography, 4 (144).

48.

Noyes

Booth

Hannes

Harden

Harris

Lewin

Lockwood

(Eds.). (2011) Supplementary guidance for inclusion of qualitative research in Cochrane systematic reviews of interventions (Version 1, updated August 2011). Cochrane 48 Collaboration Qualitative Methods Group. Available from: http://cqrmg.cochrane.org/supplemental-handbook-guidance

49.

O'Mara-Eves

Thomas

McNaught

Miwa

and Ananiadou

(2015) Using text mining for study identification in systematic reviews: a systematic review of current approaches. Systematic Reviews, 4(5), pp. 1–22.

50.

Pagiola

Rios

A.R.

and Arcenas

(2008) Can the poor participate in payments for environmental services? Lessons from the Silvopastoral Project in Nicaragua. Environment and Development Economics, 13, pp. 299–325

51.

Pattanayak

S. K.

Wunder

and Ferraro

P.J.

(2010) Show Me the Money: Do Payments Supply Environmental Services in Developing Countries? Review of Environmental Economics and Policy, 4(2, summer 2010, pp. 254–274

52.

Pirard

(2012) Market-based instruments for biodiversity and ecosystem services: a lexicon. Environmental Science & Policy, 19-20 (2012), pp. 59–68.

53.

Pluye

Robert

Cargo

Bartlett

O'Cathain

Griffiths

Boardman

Gagnon

M.P.

Rousseau

M.C.

(2011). Proposal: A mixed methods appraisal tool for systematic mixed studies reviews. Retrieved on 17/09-2017 from http://mixedmethodsappraisaltoolpublic.pbworks.com. Archived by WebCite® at http://www.webcitation.org/5tTRTc9yJ

54.

Puri

Nath

Bhatia

and Glew

(2016) Examining the evidence base for forest conservation interventions. Evidence Gap Map Report 4. International Initiative for Impact Evaluation (3ie): New Delhi.

55.

R Development Core Team (2008) R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0.

56.

Robinson

B. E.

Masuda

Y. J.

Kelly

Holland

M. B.

Bedford

Childress

Fletschner

Game

E. T.

Ginsburg

Hilhorst

Lawry

Miteva

D. A.

Musengezi

Naughton-Treves

Nolte

Sunderlin

W. D.

and Veit

(2017), Incorporating Land Tenure Security into Conservation. Conservation Letters.

57.

Rubenstein

L. V.

Williams

Danz

and Shekelle

, 2009. Determining key features of effective depression interventions. Los Angeles, CA: Greater Los Angeles Veterans Affairs Healthcare System/Southern California/RAND Evidence-based Practice Centre

58.

Samii

Lisiecki

Kulkarni

Paler

and Chavis

(2014) Effects of Payment for Environmental Services (PES) on Deforestation and Poverty in Low and Middle Income Countries. Campbell Systematic Reviews, 11.

59.

Schomers

, and Matzdorf

(2013) Payments for ecosystem services: A review and comparison of developing and industrialized countries. Ecosystem Services, 6, pp. 16–30.

60.

Scriven

, 2007. Key evaluation checklist. Unpublished manuscript. Available from: https://wmich.edu/sites/default/files/attachments/u350/2014/key%20evaluation%20ch

61.

Shadish

and Myers

(2004) Research design policy brief. Campbell Collaboration: Oslo. Available at: http://www.campbellcollaboration.org/artman2/uploads/1/C2_Research_Design_Policy_Brief-2.pdf

62.

Sharma

B. P.

, and Pattanayak

(2015) REDD+ Impacts: Evidence from Nepal. Kathmandu: South Asian Network for Development and Environmental Economics (SANDEE). Available at: http://www.sandeeonline.org/uploads/documents/publication/1064_PUB_Working_Paper_95_Bishnu_et_al.pdf.

63.

Shemilt

Mugford

Byford

Drummond

Eisenstein

Knap

Walker

, (2008) The Campbell Collaboration Economics Methods Policy Brief. Campbell Collaboration: Oslo. Available at http://www.campbellcollaboration.org/artman2/uploads/1/Economic_Methods_Policy_Brief.pdf

64.

Shemilt

Valentine

J.C.

Pössel

Mugford

and Wooldridge

D. T.

(2012) Costing program implementation using systematic reviews: interventions for the prevention of adolescent depression, Research Synthesis Methods, Vol. 3, pp. 91–201.

65.

Sills

Arriagada

R. A.

Ferraro

P. J.

Pattanayak

S. K.

Carrasco

L. E.

Ortiz

Cordero

Caldwell

, and Andam

. 2008. Private provision of public goods: Evaluating payments for ecosystem services in Costa Rica. Working Paper, North Carolina State University, Raleigh.

66.

Smith

Bustamante

Ahammad

Clark

Dong

Elsiddig

E.A.

Haberl

Harper

House

Jafari

Masera

Mbow

Ravindranath

N.H.

Rice

C.W.

Abad

Robledo

Romanovskaya

Sperling

A.F.

, and Tubiello

2014: Agriculture, Forestry and Other Land Use (AFOLU). In: Edenhofer

Pichs-Madruga

Sokona

Farahani

Kadner

Seyboth

Adler

Baum

Brunner

Eickemeier

Kriemann

Savolainen

Schlömer

vonStechow

Zwickel

and Minx

J.C.

. Eds. Climate Change 2014: Mitigation of Climate Change.

67.

Contribution of Working Group III to the Fifth Assessment Report of the Intergovernmental Panel on Climate Change. Cambridge University Press, Cambridge, United Kingdom and New York, NY, USA.

68.

Snilstveit

Stevenson

Villar

P.F.

Eyers

Harvey

Panfil

Puri

and McKinnon

M.C.

(2016) Land-use change and forestry programmes: evidence on the effects on greenhouse gas emissions and food security. Evidence Gap Map Report 3. International Initiative for Impact Evaluation (3ie): London.

69.

Snilstveit

Stevenson

Phillips

Vojtkova

Gallagher

Schmidt

Jobse

Geelen

Pastorello

, and Eyers

(2015) Interventions for improving learning outcomes and access to education in low- and middle- income countries: a systematic review. 3ie Systematic Review 24. London: International Initiative for Impact Evaluation (3ie).

70.

Snilstveit

(2012) Systematic reviews: from ‘bare bones’ reviews to policy relevance, Journal of Development Effectiveness, 4(3), pp. 388–408.

71.

Stickler

C. M.

Nepstad

D. C.

Coe

M. T.

McGrath

D. G.

Rodrigues

H. O.

Walker

W. S.

Davidson

E. A.

2009. The potential ecological costs and cobenefits of REDD: a critical review and case study from the Amazon region. Global Change Biology, 15, pp. 2803–2824.

72.

Démurger

and Wan

(2012). Payments for ecological restoration and rural labor migration in China: The Sloping Land Conversion Program in Ningxia. IZA Journal of Migration, 1 (10).

73.

Tanner-Smith

and Tipton

, (2014) Robust variance estimation with dependent effect sizes: Practical considerations including a software tutorial in stata and SPSS. Research Synthesis Methods, 5, pp. 13–30.

74.

The Steering Group of the Campbell Collaboration . (2016). Campbell Systematic Reviews: Policies and Guidelines (Campbell P). The Campbell Collaboration.

75.

Thomas

, and Harden

(2008) Methods for the Thematic Synthesis of Qualitative Research in Systematic Reviews. BMC Medical Research Methodology, 8(45), pp. 1–10.

76.

UNFCCC (2015) Adoption of the Paris Agreement, Decision -/CP.21. Bonn, Germany, United Nations Framework Convention on Climate Change (UNFCCC) (available at http://unfccc.int/files/meetings/paris_nov_2015/application/pdf/cop_auv_template_4b_new__1.pdf).

77.

UNFCCC (2010) Decision 1/CP.16 The Cancun Agreements: Outcome of the work of the Ad Hoc Working Group on Long-term Cooperative Action under the Convention.

78.

UN-REDD (2016) Key achievements of the UN-REDD programme, 2008-2016. Geneva: UN-REDD Programme Secretariat.

79.

Viechtbauer

(2010) Conducting meta-analyses in R with the metafor package. Journal of Statistical Software, 36(3), pp. 1–48.

80.

Waddington

White

Howard

Snilstveit

Birte

Hombrados

Jorge Garcia

Vojtkova

Martina

Davies

Philip

Bhavsar

Ami

Eyers

John

Koehlmoos

Tracey Perez

Petticrew

Mark

Valentine

Jeffrey C.

Tugwell

Peter

(2012) How to do a good systematic review of effects in international development: a tool kit, Journal of Development Effectiveness, 4:3, 359–387.

81.

White

(2009) Theory-based Impact Evaluation: Principles and Practice. Working paper series No. 3. New Delhi: 3ie.

82.

Wunder

(2015) Revisiting the concept of payments for environmental services. Ecological Economics, 117, pp. 234–243.