Abstract
BACKGROUND
The Problem, Condition or Issue
Since 1990, natural disasters have affected more than 200 million people every year (Leaning & Guha-Sapir, 2013). In addition, violent conflict affects the lives of 1.5 billion people every year across the globe (World Bank 2011). Although some crises can be directly attributed to a distinct event that occurs with little or no warning, most emergencies are complex, protracted and chronic, evolving from a series of related or unrelated events that exacerbate vulnerabilities of a population over a prolonged period of time. In many humanitarian crises, the needs of affected populations far outweigh available resources and capacity to respond.
For the purposes of this review, a humanitarian emergency (or crisis) is defined as a singular event or a series of events that threaten the health, safety or well-being of a community or large group of people (Humanitarian Coalition, 2013). There is no simple categorization of humanitarian emergencies, however, they are often classified into three broad categories: 1) natural disasters, 2) technological disasters (i.e. hazardous material spills, nuclear accidents, chemical explosions) and 3) conflict-related emergencies including civil strife, civil war and international war which are often referred to as complex emergencies. Humanitarian emergencies can also be categorized as ‘sudden onset,’ such as an earthquake or ‘slow onset’ such as a drought; for many conflict-related emergencies, this distinction is less clear and when extended over periods of many years they are considered ‘protracted.’
Internationally, the humanitarian response community has tended to distinguish between natural disasters and conflict and they respond frequently to both types of events (whereas responses by technological disasters are uncommon). The assistance provided by the humanitarian response sector in the case of a humanitarian emergency is defined as “aid to stricken population that complies with the basic principles of humanity, impartiality and neutrality” (WHO, 2014). Such assistance can be divided into three categories based on the way in which it is provided to the affected population. Direct assistance is the face-to-face distribution of goods, services or cash to affected populations; indirect assistance is one step removed from the affected population and involves activities such as transporting relief supplies or personnel; and infrastructure support that facilitate the relief effort but that are necessarily visible or solely for the benefit of the affected population.
The response to an emergency can be divided into phases which include ‘the emergency response phase’ during which initial response activities are implemented, operational capacities and systems are established, assessments are conducted and long-term planning takes place; this phase can last from several weeks to several months depending on the size, nature and complexity of the emergency. The continuing response phase includes support for recovery and may last anywhere from several months, in the case of smaller scale natural disasters, to several years in the case of large-scale natural disasters or conflict where the situation may be protracted and extended for several years (WHO, 2008).
The Intervention
Maximizing effectiveness and efficiency of assistance efforts is one of the most important challenges for the humanitarian sector (World Humanitarian Summit, 2013). In efforts to maximize the effectiveness and efficiency of assistance efforts, there is increasing interest in and support for implementation of cash transfer programs in both sudden onset and protracted emergencies, based on the assumption that cash programs enable affected populations to make choices about their own needs, can boost local markets, are both quick and cost-effective to deliver (GHA report 2013).
Cash-based approaches have been used for development purposes for a number of decades, particularly within social protection interventions in low and middle income countries. More recently cash transfer programs from development contexts have been increasingly applied in humanitarian settings with the aim of supporting affected populations in meeting basic needs or providing assistance for livelihoods recovery by stimulating demand and purchasing power (Creti & Jaspars, 2006). Cash-based approaches are now being used by multilateral organizations, national governments, international non-governmental organizations and national civil society groups for delivering assistance across all sectors, either on their own or in conjunction with in-kind provision of goods or services.
Cash transfers in humanitarian settings have been defined as “the provision of money to individuals or households, either as emergency relief intended to meet basic needs for food and non-food items or to buy assets essential for the recovery of livelihoods” (ECHO, 2009). Cash transfers may be singular events, usually larger sums of cash such as a grant for a specific purpose such as shelter or livelihoods recovery or recurring events, such as monthly cash grants/vouchers for rent or food or cash for work programs with multiple pay outs that extend over weeks or months. The defining characteristic of cash-based approaches is that choice resides with the beneficiary and not with the implementing organization, as is the case with in-kind assistance. A variety of different cash-based approaches exist and these are collectively are referred to as cash-transfer programming. The most common forms of cash transfer include:
The above types of cash transfers may be conditional or non-conditional. Unconditional cash transfers have no conditions on actions that have to be performed to ‘earn’ the cash (i.e. no work or behavioural requirements). There is no requirement to repay any money and recipients are entitled to use the money however they wish (CaLP, 2014). The underpinning assumption of unconditional cash transfers is that money will be used to meet basic needs. An example of an unconditional cash transfer would be a cash grant to displaced household which would be used to replace basic household items that were lost in displacement, purchase food or acquire productive assets that could be used to restart livelihoods activities.
Conditional cash transfers require that recipients meet certain requirements before the transfer is fulfilled. Conditions of a cash transfer may stipulate either 1) how the cash is earned (qualifying conditions), or 2) how the cash is used (use conditions). Cash transfers with qualifying conditions is a cash transfer given after the recipients have performed some task or activity as a condition of receiving the cash transfer (CALP, 2014). Common qualifying conditions include performing physical labor to contribute to disaster clean-up efforts, public works or the creation of community assets (cash for work programs); enrolling children in school or having them vaccinated; or attending training. Cash transfers with use conditions are a type of conditional transfer where the agency puts conditions on how cash is spent (ibid). Common conditions include requiring the money to be spent on food items, educational expenses or shelter materials.
How the Intervention Might Work
The conceptual underpinning of cash programs in emergencies is in part derived from Amartya Sen's (1981) entitlement theory. Entitlement theory states that famines are caused by an inability to gain access to, or purchase, food rather than an overall lack of food availability. Through this lens, income support measures could be an appropriate response in emergencies because people live in cash economies where they earn money and buy goods in markets, which in many emergency contexts are still functioning or quickly recover. Cash can be an effective means of meeting population needs because it increases access to food and other basic goods and services, and integrates humanitarian response with the local economy (Gairdner P, Mandelik F and Moberg L., 2011).
The conceptual roots of cash programs in emergencies are further derived from subsequent research and theories that illustrate the limits of entitlements in analysing famine in the context of war, such as de Waal's “health crisis” model which views famine as a result of various social and political pressures as well as the lack of entitlement (de Waal, 1990). Through this lens, cash transfer programming can be a more versatile means of addressing population needs than in-kind provision of goods or services so long as there is reliable supply and markets are able to respond to increases in demand resulting from cash infusions without inflation or other negative distortions (Gairdner P, Mandelik F and Moberg L., 2011).
In cash transfer programs, there are typically four groups of actors: (1) the funding source, (2) the implementing agency that administers the program, (3) the service provider that facilitates the actual transfer of cash, and (4) the recipient. These actors are outlined below:
Figure 1 (following page) describes how resources flow between the primary actors in cash transfer programs. In the case of unconditional cash transfers programs, there are no restrictions on how or where monies are spent. Conditional cash transfers may only be made upon verification that certain requirements (qualifying or use conditions) were met or vouchers may only be spent for certain goods or services from pre-designated providers. Although there is a large body of literature examining cash transfer programs and their outcomes, there is less understanding of the mechanisms and pathways by which outcomes are achieved. For example, a broad variety of theories of change models for cash transfer program exist which vary by the context and type of intervention. Most theories of change take a holistic approach and include macro- and micro-levels as well as contextual factors. In addition, theories of change draw either on human capital and productive assets theories or vulnerability and risk reduction (Browne, 2013).
Stakeholders and Resource Flow in Cash Transfer Programs in Emergencies
Across all models, there is consensus that cash-based approaches are appropriate only within certain contexts and that appropriateness is determined by the both the characteristics of the crises and the presence of certain enabling conditions. 1 According to Gairdner, et al. (2011), the five basic enabling conditions are required for cash-based approaches to be effective, in order of priority, include local availability of commodities for basic needs and recovery, private markets exist and function at a level that is adequate to provide the goods needed to meet the basic needs of the affected population, preference of beneficiaries for in-kind or cash-based approaches to delivering assistance, the security situation must be permissible for operations to occur, including both delivery of cash and the movement of goods, and financial infrastructure exists to facilitate cash or voucher transfers.
For example, cash-based approaches assume that goods will be channelled through local markets, that there is reliable supply and affordable prices, and that markets will be able to respond to increase in demand that result from cash infusions without inflation or other negative distortions (which can erode the value of the cash transfer and create hardships for non-beneficiaries). Whether the crisis is sudden onset or protracted can play an important role determining if markets are functioning or can be revived. In sudden onset conditions, availability of goods may be greater as markets recover from short term shocks whereas in chronic situations markets may be more difficult to revive because degradation occurred over an extended time period.
Why it is Important to do the Review
There is extensive documentation showing that individual donors and implementing agencies are delivering more of their programs through cash-based mechanisms (Gairdner P, Mandelik F and Moberg L., 2011). Spending on cash programming in emergencies has steadily increased since the 2004 Indian Ocean tsunami and a gradual shift away from inkind food as the default response towards a broader food assistance approach that includes cash-based approaches has occurred (Harvey et al., 2010).
Cash-based approaches to assistance have increased in other humanitarian sectors as well. Cash transfers are employed to enable emergency-affected populations to pay rent, rebuild or repair shelters and damaged public infrastructure, to support livelihoods recovery, and as a mechanism to improve access to food, water, hygiene supplies, productive assets and inputs, and basic services in a range of settings. Documentation from humanitarian donors and practitioners shows that there is a high degree of variation in humanitarian contexts where cash-based approaches are being used, the mechanisms by which cash is provided, and humanitarian objectives of cash programs (CaLP, 2014; ECHO, 2013).
Where markets are functioning, there is an emerging consensus that cash transfer programs are an appropriate alternative to in-kind provision of goods or services for populations affected by emergencies (Austin 2014; Gairdner P, Mandelik F and Moberg L., 2011). Advocates for cash transfer programs argue that they allow humanitarian actors to address immediate needs more rapidly than direct provision of goods or services because of reduced logistical complexity and are a preferable because they provide support in a way that maintains dignity and choice among affected populations (CaLP, 2014; Creti & Jaspars, 2006).
However, there is a lack of a systematic assessment of the effects of cash based approaches in humanitarian settings to support such claims. Little guidance is available to help humanitarian stakeholders determine which cash-based approaches might be most effective in which contexts and why.
A large and growing body of rigorous evaluations assessing the effects of conditional and unconditional cash transfers and voucher programs in development settings has been developed in recent years. This includes a relatively large number of systematic reviews assessing the effects of these programs on a broad range of outcomes, including health, education, and social protection outcomes. For instance, Hagen-Zanker et al (2011) assess the evidence on the effects of employment guarantee schemes and cash transfers on the poor. Moreover, Kabeer et al. (2012), Gaarder et al (2010) and Baird et al. (2013) assess the effects of cash transfer programs on economic, health and education outcomes respectively. Similarly, Yoong et al. (2012) systematically review the evidence on the effects of economic resource transfers, providing a comparison of outcomes for women versus men.
All these reviews focus on programs and populations in low and middle income countries, but they do not include an explicit focus on the effects of use of cash based approaches in humanitarian emergencies. Given the wide range of difficulties posed by humanitarian emergencies with respect to the implementation of cash programs and in many cases, the differences in objectives between cash programs in emergency and development settings, the findings from the existing literature are not easily transferable to the emergency context. The use of impact evaluations to assess the effectiveness and efficiency of cash based interventions in sudden onset or protracted emergencies is limited compared to the large literature of cash based programming in LMICs. The vast majority of publications identified so far are case-studies of specific programs, discussion pieces, or economic analyses of the cost-effectiveness of cash-based approaches over in-kind food assistance. Nevertheless, there is increased attention to the need for impact evaluations in humanitarian settings and several impact evaluations have been published in recent years (Aker, 2013; Gilligan, 2013).
A preliminary search of the peer-reviewed literature published in the last ten years identified no systematic reviews of cash-based approaches in humanitarian settings. Our review includes some overlap with an ongoing systematic review focusing on the effects of unconditional cash transfers (Pega et al., 2014) on health service use and health outcomes. However, we are interested in a broader range of interventions and outcomes. We include vouchers, cash for work and conditional cash transfers. Additionally, we will also be assessing the efficiency of different delivery channels, and factors influencing the implementation of cash based approaches in different settings.
The objective of this review is to assess and synthesize the existing evidence on the effects of cash-based approaches in humanitarian emergencies. The review aims to serve as a reference for humanitarian decisions makers and to support evidenced-based approaches to cash interventions in emergencies.
OBJECTIVES
The main objective of the review is to assess and synthesize the existing evidence on the effects of cash-based approaches in humanitarian emergencies. In doing so we aim to address the following research questions within the context of humanitarian emergencies:
Primary Research Questions:
What are the effects of cash-based approaches on individual and household level economic outcomes in humanitarian emergencies? What are the effects of cash-based approaches on sector-specific outcomes and cross-cutting issues such as dignity and resilience in humanitarian emergencies?
Secondary Research Questions: How efficient are different cash-based approaches and delivery channels in achieving their stated objectives in humanitarian emergencies? What factors have hindered or facilitated realization of cash program activities and their objectives in different emergency contexts?
METHODOLOGY
I. Criteria for including studies in the review:
a. Types of participants
Populations affected by humanitarian emergencies will be the focus of this review. For the purposes of this review, a humanitarian emergency (or crisis) is defined as a singular event or a series of events that threaten the health, safety or well being of a community or large group of people (Humanitarian Coalition, 2013). Humanitarian crises can either be natural disasters, man-made disasters or complex emergencies. Each humanitarian crisis is caused by different factors and as a result, requires a response targeted towards the specific sectors and populations affected. Thus, complex emergencies occur as a result of several factors or events that prevent large groups of people from accessing fundamental needs, such as food, clean water or shelter. Types of emergencies included in the review are: Sudden onset emergencies including both natural disasters and man-made or complex disasters for which there is little or no warning. Slow onset emergencies that do not result from a single distinct event but rather emerge gradually over time. Protracted emergencies where armed conflict and/or severe vulnerabilities to longterm disasters persist over periods of five years or more.
Affected populations could include those that were not displaced, those displaced within their home country, or refugees displaced in neighboring countries. Types of participants that will be included are individuals or households that received a cash intervention, such as vouchers, conditional or unconditional cash transfers, or cash for work beneficiaries. All beneficiaries of cash interventions meeting review criteria will be included, regardless of if the cash intervention was targeted to specific population sub-groups or the greater affected population.
b. Types of interventions
Types of cash transfer programs that will be included are: 1) cash grant programs, 2) cash for work programs and 3) vouchers programs. This includes both conditional and unconditional cash transfers that are delivered via these mechanisms. All of these interventions, as defined in the background section, are intended to benefit populations affected by emergencies.
Cash transfer programs that will be excluded are 1) fee waivers and subsidies where no cash transfer is involved, 2) microfinance interventions, including lending, saving and insurance where repayment is expected or no actual cash transfer occurred, and 3) direct budget support to organizations providing humanitarian assistance where no transfer to affected populations occurs.
c. Types of comparisons
We will include studies conducted in the context of a humanitarian emergency which compare beneficiaries that received cash or vouchers to populations that received no assistance or in-kind provision of goods or services. Comparisons of cash and voucher recipients will also be included, as will comparison of different transfer modalities (for example cards, mobile phones, physical cash). We will collect relevant information on the cash transfer(s) received by the comparison groups.
Due to the possibilities of spillovers, we will assess whether comparisons are geographically separated from intervention groups and to the extent possible, whether they were simultaneously benefiting from other similar assistance programs.
d. Types of outcome measures and other findings
Primary Research Questions
To address questions 1a and 1b, we will include studies that report the effects of cash interventions on at least one of the following outcomes: Individual and/or household level economic outcomes such as utilization of cash, household assets or economic status; Individual, household and/or population level sector-specific outcomes such as changes in health service utilization, food security, nutrition status, availability of shelter, access to clean water, school enrolment, etc.; or Individual, household and/or population level measures of cross-cutting outcomes such as protection and resilience.
A broad range of outcome measures may be used to assess these outcomes, including those listed in the Inter-Agency Standing Committee Humanitarian Indicators Registry for WASH, shelter, nutrition, health, food security, or education sectors (IASC 2014).
Secondary Research Questions
To address question 2a, we will include studies that report on at least one of the following: Costs of implementing a cash intervention Efficiency, defined as value for money, measured in terms of cost-utility, cost-benefit, or cost-effectiveness of a cash intervention. We will also include any measure of the difference between budgeted costs (staff needs, materials, running costs, etc.) and actual costs of implementing a cash intervention.
To address question 2b, we will also include studies that report on the following factors that may have hindered or facilitated realization of program activities and objectives: targeting of interventions delivery mechanisms, including use of new technologies staffing scaling up and phasing out interventions coordination security risk associated with different cash transfer modalities unintended consequences at beneficiary, organizational and societal levels beneficiary perceptions of participation in cash programs.
e. Study Designs
Primary Research Questions
To address questions 1a and 1b on the effects of cash interventions, we will include experimental and quasi-experimental study designs that allow for attribution. Specifically we will include the following study types: 1) Experimental studies using randomised assignment of an intervention to either the individual or community level; 2) Quasi-experimental studies where assignment is based on other stated allocation rules such as exogenous geographical variation or so called ‘natural experiments’, or a threshold on a continuous variable (regression discontinuity designs); 3) Studies where assignment to the intervention is based on other known rules such as self-selection by participants or based on program criteria, provided that data are collected on a comparison group (non-equivalent comparison group designs) and use appropriate statistical methods to control for selection bias and confounding, such as propensity score matching, covariate matching, difference-indifferences, and single difference regression analysis, instrumental variables, and ‘Heckman’ selection models); and 4) Studies where data are collected at multiple intervals before and after the interventions (interrupted time series design), provided data are collected at a minimum of three time points before and three time points after the intervention, and the study takes into account secular (trend) changes in the analysis or re-analysis is possible (Effective Practice and Organisation of Care 2013).
Studies which do not use a comparison group design, studies with a comparison group that do not use statistical analysis to address confounding, and studies that employ less than a six-period interrupted time series design will be excluded.
Secondary Research Questions
To address question 2a we will include experimental and quasi-experimental studies with an economic component/ data on costs, in addition to observational studies, process evaluations and economic evaluations. To address question 2b we will include observational or descriptive studies, qualitative studies, mixed methods studies and process evaluations. To be eligible for inclusion such studies need to be based on data collected from program beneficiaries, implementing agencies or stakeholders and report at least some information on the study's research question, procedures for data collection, and analysis methods.
Publication types that will be considered ineligible and excluded include: 1) opinion pieces, 2) commentaries, 3) editorials, 4) debates, 5) project implementation guidelines, 6) case studies of individual beneficiaries or households, 7) other reflective non-research based reports, 8) systematic and non-systematic reviews.
f. Other criteria for including and excluding studies
We will exclude any studies published before 2000. The year 2000 cut-off date was chosen because widespread uptake of cash programming in humanitarian emergencies began after this point.
The inclusion of exclusion criteria for the review are summarised in table 1 below.
Inclusion and Exclusion Criteria
II. Search Methods for Identification of Studies
The research team will conduct an extensive systematic search for grey and peer-reviewed literature, following the guidelines provided in the Campbell Collaboration's Information retrieval methods group policy brief (Hammerstrom, Wade, Hanz & Jorgensen, 2009). A common search strategy will be used to identify studies used to address the primary and secondary review objectives.
a. Electronic database searches:
To ensure a comprehensive search strategy in line with the Campbell Collaboration's Information retrieval methods group policy brief, searches will be conducted in multiple databases and high sensitivity will be sought (Hammerstrom, Wade, Hanz & Jorgensen, 2009). A Johns Hopkins University Welch Medical Library Public Health information specialist developed the search strategy after input from investigators on the research team. Results of the pilot search were reviewed by the research team and the strategy was developed iteratively. We also provided the information specialist with several key papers as a test set that was used to check to completeness of the search results. The search strategy includes a combination of controlled vocabulary and keywords for cash and emergencies. Databases and basic search terms that will be used are presented in Table 2 and a detailed example of the search strategy is presented in Annex 1.
Peer-Reviewed Literature Sources and Search Terms
We will limit the searches to studies published from 2000 to the present. The year 2000 cut off date was chosen because widespread uptake of cash programming in humanitarian emergencies began after this time point and a 14 year review period was consider sufficient given changes in humanitarian contexts, programming approaches and technological developments during this time period. We will include studies in any languages.
b. Other searches:
To identify an unbiased set of citations we will also identify studies in the grey literature from conference proceedings, databases of unpublished studies, studies published in supplements, theses, and dissertations (Higgins, 2005). In order to access grey literature, the peer-reviewed literature search strategy will be adapted to guide hand-searches of the following humanitarian donor, practitioner and research network collections presented in Table 3.
Grey Literature Sources
In addition, we will conduct forward citation-tracking of included studies in Web of Science, Scopus and Google Scholar, and also review bibliographies from systematic and non-systematic reviews of cash programs to identify additional studies that were not found using search strategies outlined above. All relevant documents meeting search criteria will be catalogued in an EPPI-Reviewer database.
III. Description of methods used in primary research
Several aspects of humanitarian situations make conducting rigorous evaluations a challenge. A recent review of studies on humanitarian cash transfer programming found that most research is generally undertaken by donors, implementing agencies, and private sector actors in order to inform development of organizational strategies (Austin, 2014). Only a sub-set of this research is empirical in nature and intended for widespread dissemination to inform programming or policy on a broader scale (Austin, 2014). This sub-set of studies is the body of evidence that forms the focus of this review. Examples of studies that may be included in the review include:
Aker J. Cash or Coupons? Testing the Impacts of Cash versus Vouchers in the Democratic Republic of Congo. CGD Working Paper 320. Washington DC: Center for Global Development. 2013.
The Center for Global Development (a research institution) has published a working paper presenting the results of a randomized evaluation of an unconditional cash transfer and voucher program implemented by Concern Worldwide (an international non-governmental organization) in an informal IDP camp in the Democratic Republic of Congo. The study involved household surveys conducted with members of 252 households before, during and after the program, as well as analysis of price data collected throughout the program period and administrative data from the agent distributing cash transfers and monitoring data collected during closed fairs where items were sold to voucher recipients. Results presented include uses of the transfer, effects on select measures of well-being and costs of each assistance modality to the implementing agency and recipients.
Gilligan D, Margolies A, Quinones E, Roy S. Impact Evaluation of Cash and Food Transfers at Early Childhood Development Centers in Karamoja, Uganda. International Food Policy Research Institute, 2013.
The International Food Policy Research Institute has published the findings an evaluation of food and cash transfers to households with children participation in Early Childhood Development centres in the Karamoja subregion of Uganda. The study was a cluster-randomized controlled study with 98 villages containing Early Childhood Development Centres randomly assigned to receive food, cash transfers or neither. Results reported include relative impact of food and cash transfers on household food security, frequency of child food consumption, child anthropometry, anaemia status, Early Childhood Development centre participation, and child cognitive and non-cognitive development.
IV. Data collection and analysis
a. Selection of studies
The screening of studies and application of inclusion and exclusion criteria will take place in two rounds. In the first round of screening, all titles and abstracts resulting from the search will be independently screened by two research assistants. Studies will be classified as either ‘exclude’ or ‘potentially eligible’. In round two, full text of all studies that have been classified as ‘potentially eligible’ or where there is a doubt or disagreement about potential eligibility, will be assessed independently by two reviewers. Studies will be classified as either ‘included’ or ‘excluded’, and reason for exclusion recorded. Disagreements will be reviewed by either Dr. Doocy, Dr. Tappis, or both, and a resolution achieved through discussion amongst the review team. The screening process will be managed using EPPI-Reviewer software to ensure completely independent reviews and resolution of disputes.
b. Data extraction
Two reviewers will independently extract and code all of the included studies using a data extraction form and codebook developed for the review (see Appendix 2 for data extraction codebook), and inter-rater reliability (i.e., percentage match) will be assessed.
Disagreements will be reviewed and when resolution is achieved a consensus code will be used. Information to be extracted will include intervention details, study details and study findings, as summarised in Table 4 below.
Information to be extracted
c. Critical appraisal of included studies
Studies will be stratified into three groups for critical appraisal: (1) experimental or quasi-experimental quantitative studies measuring the effectiveness of cash-based approaches for achieving sector-specific or cross-cutting outcomes; (2) quantitative studies measuring the cost-effectiveness, cost-utility, cost-benefit or cost-efficiency of cash-based approaches; and (3) observational, qualitative studies or mixed method studies that address issues related to implementation in a specific context. Studies with multiple purposes and study designs will be grouped and appraised according to their primary purpose.
Primary Research Questions
Studies will be independently assessed by two reviewers using a combination of the items from the Risk of Bias criteria outlined in The Cochrane Handbook of Systematic Reviews of Interventions (Higgins and Green, 2011), and the Effective Practice and Organisation of Care (EPOC) Cochrane Review Group criteria developed for non-randomised studies (Effective Practice and Organization of Care Group, 2013). For each item, a description of the information upon which each judgement decision is based will be provided, and a judgement will be made of: high, low or unclear risk of bias. Any disagreements in judgements on risk of bias will be reviewed by either Dr. Doocy, Dr. Tappis, or both, and a resolution achieved through discussion amongst the review team. Results of the appraisal will be reported by criteria across the included studies.
The body of included literature for measuring the effectiveness of cash based approaches for achieving sector-specific or cross-cutting outcomes will be assessed using criteria adapted from the Grading of Recommendations Assessment, Development and Evaluation (GRADE) system (Guyatt, 2008). The quality of evidence will be classified based on underlying methodology as high (randomized trials, or double-upgraded observational studies); moderate (downgraded randomized trials, or upgraded observational studies); low (double-downgraded randomized trials, or observational studies); and very low (triple-downgraded randomized trials, downgraded observational studies, or case series and case reports). Downgrading of study's quality will be based on limitations in the design and implementation of the study, indirectness of evidence, inconsistency or imprecision of results, and probability of publication bias. Upgrading of a study's quality will be based on the magnitude of effect or dose-response gradient (Guyatt, 2008).
Secondary Research Questions
Quantitative studies measuring the cost-effectiveness, cost-utility, cost-benefit or cost-efficiency will be assessed using criteria from the German Federal Ministry for Economic Cooperation and Development's Tools and Methods for Evaluating the Efficiency of Development Interventions (BMZ, 2011). The quality of evidence in these studies will be classified based on the analytic power of efficiency analysis as level 2 (cost-effectiveness analysis, cost-benefit analysis, cost-utility analysis, and multi-attribute decision making); level 1 (benchmarking of unit-costs and other efficiency indicators, stakeholder driven approaches and comparative ratings); and level 0 (entirely descriptive). Level 2 analyses are the most rigorous, assessing the efficiency of an intervention so that it can be compared with alternatives or benchmarks, while level 1 analyses with moderate analytic power identify potential for efficiency improvements within interventions. Both analyses serve specific purposes; level 2 analyses can assist in selecting the most efficient intervention strategy while level 1 analyses are needed to optimize efficiency within selected interventions (BMZ 2011).
Descriptive or qualitative studies that describe how implementation worked and effects on unintended outcomes will be assessed using the Mixed Methods Assessment Tool development by Pluye and colleagues at McGill University (2011). This tool was selected because it is designed to deal with quantitative, qualitative and mixed-methods research within the same appraisal.
d. Criteria for determination of independent findings
Several publications or studies based on the same data set, multiple treatment arms with only one control group, outcome measurements from several time points and the use of multiple outcome measures to assess related outcome constructs can lead to issues with dependent effect sizes, where we cannot treat all outcome estimates as independent of each other (Borenstein et al., 2009). We will follow Campbell Guidance (Becker et al, 2007) so multiple measures of the same outcome within one study will not be synthesised.
Where more than one paper or report is identified on a single study, we will choose one as the ‘main’ paper (the one with the most relevant data) and the others will be considered ‘secondary reports’. We will include any additional information about that one study, including outcome measures not reported in the main paper. For studies reporting follow-up effects at multiple points in time, we will take the final follow-up measure. If studies report results for two or more analyses to assess the same participants, we will include the specification with the lowest level of bias in the synthesis.
Where a single report describes more than one study, these will be separated into two or more ‘studies’, which will be coded and analysed separately. Where individual studies report multiple outcome measures for the same outcome construct we will select the outcome that is most commonly reported across included studies, and if not available, then the outcome that is most accurately measured will be used.
If studies include multiple treatment arms with only one control group and the treatments represent separate treatment constructs we will calculate the effect size for treatment A versus control and treatment B versus control and include in separate meta-analyses according to the treatment construct. If the treatments A and B represent variations of the same treatment construct we will calculate the weighted mean and standard deviation for treatment A and B before calculating the effect size for the merged group versus control group, following the procedures outlined in Borenstein et al. (2009, chapter 25).
e. Measures of treatment effect
Effect size estimates and 95 percent confidence intervals will be extracted when possible. If feasible, standardised mean differences (SMDs) for continuous outcome variables and risk ratios (RRs) for dichotomous outcome variables will be calculated. Intervention effects will be calculated as the ratio of, or difference between, treated and control observations consistently to enable comparison of outcome measures across studies. A SMD greater than zero (RR greater than 1) will indicate an increase in the outcome in the intervention group as compared to the comparison group. A SMD less than zero (RR between 0 and 1) will indicate a reduction in the intervention group as compared to the control group.
f. Unit of analysis issues
Unit of analysis error arises when the unit at which the intervention is allocated and the unit of analysis are different from each other, without the authors correcting for this in their analysis. For example, if the intervention is allocated at a cluster level, but the analysis of effects is carried out at the individual level. We will assess studies for unit of analysis errors, and if necessary and feasible with the information available, adjustments will be made to account for incorrectly analysed data using the formula provided in the Cochrane Handbook (Higgins and Green, 2011).
V. Synthesis Procedures and Statistical Analysis
Review questions 1a and 1b: Statistical analysis
We will synthesise evidence on the effects of cash-based interventions for only the primary objectives of the review. If meta-analysis is feasible we will synthesise studies using an inverse-variance, random effects model using Stata software due to the anticipated heterogeneity in our included studies (Stata Corporation, College Station, TX, USA). By accounting for the possibility of different effect sizes across studies, random effects meta-analysis produces a pooled effect size with greater uncertainty attached to it, in terms of wider confidence intervals than a fixed effect model (Higgins and Green, 2011).
Due to the heterogeneity of study settings, interventions and outcomes likely to be reported in the included studies, and challenges conducting research studies with comparison groups in emergency settings, statistical synthesis of effects might not be possible. We will only conduct meta-analysis of studies which we assess to be sufficiently similar. We will follow the approach adopted by Wilson et al. (2011) and conduct meta-analysis for only when we identify two or more studies with comparable effect-sizes for a common outcome construct, for the same intervention type and where the condition in the comparison group is judged to be similar.
If meta-analysis is not possible effect sizes will be calculated and presented in a common metric and to display those effect sizes using forest plots, but without producing a pooled effect size across studies (Waddington et al., 2012). In such cases a narrative synthesis of findings will be provided based on the size and direction of effects, together with 95% confidence intervals. Stata software will be used for all quantitative statistical analysis (Stata Corporation, College Station, TX, USA).
Assessment of heterogeneity
We will assess heterogeneity by inspection of forest plots for lack of overlap of confidence intervals, and test for heterogeneity statistically using the Q statistic (Hedges & Olkin, 1985). To provide estimates of the magnitude of the variability that is due to heterogeneity we will calculate and report the I2, and Tau2 (Higgins 2002, Higgins 2003).
Moderator analysis
We aim to investigate sources of heterogeneity through moderator analysis if feasible. Lipsey (2009) classifies moderators into three broad categories of extrinsic, methodological and substantive characteristics and we will use this framework to structure our moderator analysis. We aim to include the extrinsic variables funder, type of publication and publication date; we will also aim to assess methodological variables such as risk of bias/study quality characteristics. Finally, we aim to assess the following substantive variables: program design characteristics, emergency typology, humanitarian sector, intervention aim and assistance modality. Understanding how program design characteristics, cash delivery strategies and the contexts in which programs are implemented is important for providing conclusions that are relevant and applicable to the humanitarian community.
To investigate the association between continuous moderator variables and heterogeneity of treatment effects we will use random effects meta-regression and sub-group analyses to investigate the association between dichotomous moderator variables and treatment effects (Borenstein et al., 2009). We will supplement the statistical analysis of moderators with a narrative assessment as outlined below.
Sensitivity analysis
We will conduct sensitivity analysis according to categories of risk of bias, study design (experimental and quasi-experimental, adjusted and unadjusted effect sizes), treatment effect (for example, intention to treat, average treatment effect on the treated, local average treatment effect).
Publication bias
We will search a broad range of sources of unpublished studies to try to address publication bias in the review, but if feasible (at least ten studies included in a single meta-analysis with some variability in effect size standard errors) we will also assess possible publication bias using statistical methods. We will use funnel plots and Egger et al.'s (1998) test, and use ‘trim and fill’ (Duvall & Tweedie, 2005) regardless of whether funnel plots suggest asymmetry.
Missing data
Where included studies do not provide the data required to calculate effect sizes, we will contact the authors of the primary studies and request the missing data. If information is still unavailable manipulations will be conducted to derive desired statistics as specified by Lipsey and Wilson (2001).
Review questions 2a and 2b: Economic and thematic data analysis
Economic, observational and qualitative data that can help explain and/or contextualize primary review findings will be synthesized to complement the findings of the effectiveness review described above. The synthesis will be driven by the research questions and the factors listed in section 1d.
To address question 2a we will use tables presenting the key characteristics and findings of each included study, providing a narrative summary discussing and comparing the studies' main findings regarding the efficiency, cost and value for money of cash-based approaches in humanitarian emergencies (Shemilt et al., 2008).
To address question 2b we will use a thematic synthesis approach (Thomas and Harden, 2008), focusing on factors listed in section 1d as well as any additional themes emerging from the findings of included studies. After having completed the detailed coding of all of the included studies as described above, we will re-review the coding of data on the factors listed in 1d above to identify descriptive themes which remain close to the findings in the primary studies (following Thomas and Harden, 2008). We will then use these descriptive findings to generate analytical themes regarding factors that hinder and facilitate realization of cash program activities.
To improve the transparency of the synthesis we will use matrices to present the synthesis (Popay et al., 2006). The matrices for the descriptive themes will be organised according to a framework of studies and coding categories, displaying the descriptive themes and the findings from the studies these themes are based on. To identify the analytical themes we will identify common themes across these descriptive findings, and then use matrices with a framework based on the categories listed in 1d and the corresponding analytical themes for each category.
VI. External validity
The generalizability of the results obtained will depend on the scope of the studies identified, the extent to which relevant information about cash-based approaches is documented, and the representativeness of different contexts. In order to assess external validity, population characteristics, details of the intervention (such as implementation issues), reach, adoption, outcomes and sustainability must be reported by research studies. Any information provided about external validity that is discussed within each included study will also be captured as this information enables practitioners to determine if findings are relevant to the setting in which they work. Generalizability of review findings will be assessed across four domains: 1) population—if the studies included in the review are representative of emergency affected populations where cash interventions have been used in the humanitarian context; 2) intervention—extent of implementation and adaptation of the interventions reviewed; 3) outcomes for decision making characterized by the review; and 4) maintenance and institutionalization of cash intervention among humanitarian responders. The RE-AIM tool, developed by Green & Glasgow (2006), corresponds to these four domains will be adapted for to assess external validity for this review.
SOURCES OF SUPPORT
This systematic review is generously supported by the United Kingdom's Department for International Development (DFID).
DECLARATIONS OF INTEREST
The authors have no financial interests in this area and have not published any prior reviews on the topic. Dr. Doocy has published three primary research papers on cash interventions in emergencies. This should not be a significant conflict of interest because inclusion of primary research papers in the review will be governed the protocol.
REVIEW AUTHORS
ROLES AND RESPONSIBLIITIES
Both Dr. Doocy and Dr. Tappis have content expertise in humanitarian emergencies and methodological expertise, including prior experience conducting systematic reviews. A Johns Hopkins University informationist will provide support on retrieval in addition to student research assistants. Both Dr. Doocy and Dr. Tappis have statistical backgrounds; Dr. Tappis will lead the statistical analysis and if needed, additional support will be provided by the Johns Hopkins School of Public Health Biostatistics Consulting Service. Responsibilities for specific areas are as follows: Content: Doocy, Tappis Systematic review methods: Doocy, Tappis Statistical analysis: Tappis, with support from Biostatistics Consulting Service if needed Information retrieval: Doocy, Tappis, Johns Hopkins Informationist, Research Assistants
PRELIMINARY TIMEFRAME
PLANS FOR UPDATING THE REVIEW
There are no plans to update the review at this time.
AUTHORS' RESPONSIBILITIES
By completing this form, you accept responsibility for preparing, maintaining and updating the review in accordance with Campbell Collaboration policy. The Campbell Collaboration will provide as much support as possible to assist with the preparation of the review.
A draft review must be submitted to the relevant Coordinating Group within two years of protocol publication. If drafts are not submitted before the agreed deadlines, or if we are unable to contact you for an extended period, the relevant Coordinating Group has the right to de-register the title or transfer the title to alternative authors. The Coordinating Group also has the right to de-register or transfer the title if it does not meet the standards of the Coordinating Group and/or the Campbell Collaboration.
You accept responsibility for maintaining the review in light of new evidence, comments and criticisms, and other developments, and updating the review at least once every five years, or, if requested, transferring responsibility for maintaining the review to others as agreed with the Coordinating Group.
PUBLICATION IN THE CAMPBELL LIBRARY
The support of the Campbell Collaboration and the relevant Coordinating Group in preparing your review is conditional upon your agreement to publish the protocol, finished review and subsequent updates in the Campbell Library. Concurrent publication in other journals is encouraged. However, a Campbell systematic review should be published either before, or at the same time as, its publication in other journals. Authors should not publish Campbell reviews in journals before they are ready for publication in the Campbell Library. Authors should remember to include a statement mentioning the published Campbell review in any non-Campbell publications of the review.
I understand the commitment required to undertake a Campbell review, and agree to publish in the Campbell Library. Signed on behalf of the authors:
Form completed by: Shannon Doocy and Hannah Tappis
Date: 9 October 2014
Footnotes
APPENDICES
1
See Creti & Jaspars, 2006 p22 and ECHO, 2009 p4 frameworks on deciding between in-kind assistance and cash-based approaches.
2
Systematic and non-systematic reviews will be excluded from the analysis but may inform background discussions and synthesis of findings and recommendations.