Abstract
BACKGROUND
More than a billion people worldwide live on less than 1 US dollar a day (UN Millenium Project, n.a.). Lack of income is amplified by restricted access to a range of basic human capabilities such as education, health, self-determination, and self-worth (Sen, 1993 & 1992). Apart from research that has focused on the consequences of living in poverty, there is a large body of literature exploring how poor people manage their (little) financial resources from day to day (e.g. Collins, Murdoch, Rutherford, & Ruthven, 2009; Rutherford & Arora, 2009; Banerjee & Duflo, 2007). These analyses have found that, despite meagre incomes, the financial portfolios of the world's poor often go beyond mere hand-to-mouth survival and unfold in complex systems of money saving, borrowing, and various financial relationships (Collins et al., 2009). While it might appear that people living in poverty are ‘too poor to save‘, research has shown that certain events in their lives – such as health emergencies, traditional ceremonies (weddings or funerals), and educational requirements – necessitate considerably large sums of money and therefore make saving a prerequisite (Collins et al., 2009; Rutherford & Arora, 2009).
As access to formal banking is usually confined for poor households, a range of alternative financial instruments are needed. Over 80 per cent of poor people in low and middle income countries (LMICs) do not hold a formal bank account, but have developed informal mechanisms for saving money. These include: a) saving at home, for instance with lock boxes, b) money guards/deposit collectors who are entrusted with personal savings so as to ascertain that the money will not be spent, c) self-help groups and peer monitoring systems for implementing and formalising saving plans, and d) rotating savings and credit associations (ROSCAs) in which members pay into a shared savings pot at periodic intervals and receive pay-outs on a rotating basis (Karlan, Ratan, & Zinman, 2014; Beaman, Karlan, & Thuysbaert, 2014; Rutherford, 2000; Banerjee & Duflo, 2007; Ambec & Treich, 2007; Ashraf, Karlan, & Yin, 2006; Ashraf, Gons, Karlan, & Yin, 2003; Bouman, 1997). Alternatively, a large number of people living in poverty save in cash and in kind at home, by buying a goat or hiding money in a pillow or under their mattress (Karlan et al., 2014; Hulme, Moore, & Barrientos, 2009; Banerjee & Duflo, 2007).
The microfinance paradigm has evolved in response to the poor's need for financial systems and mechanisms that help them manage and potentially increase their (little) money. In the most general terms, microfinance institutions seek to lift their clients out of poverty by providing them with capital and schemes for investments in micro-enterprises and income generating activities. The movement's origins lie in Bangladesh where Nobel Peace Prize winner Muhammad Yunus had set up the Grameen bank 30 years ago, providing microloans to turn the country's rural poor into small entrepreneurs (Duvendack et al., 2011). However, recent evidence has highlighted potential adverse outcomes for microcredit clients, such as indebtedness, spiralling into poverty through taking out new loans to repay old loans, damages in reputation and social status, and impoverishment through the loss of collateral (Schicks, 2014; Agnihotri, 2013; Stewart et al., 2010, Bateman, 2010; Epstein & Yuthas, 2010; Copestake, 2002; Adams & Von Pischke, 1992). Echoing this critique more polemically, the Boston Globe writes: “Billions of dollars and a Nobel Prize later, it looks like ‘microlending’ doesn't actually do much to fight poverty” (Bennett, 2009).
In light of this recent disenchantment with micro-credit, an increasing number of microfinance institutions have broadened their focus from microloans to the provision of saving products (Karlan et al., 2014; Stewart, van Rooyen, Dickson, Majoro. & de Wet, 2010; Robinson, 2001). Banerjee and Duflo (2011) go so far as to portray microsaving as “the next microfinance revolution” (p. 190). Practitioners and scholars are now advocating for saving as the superior poverty alleviation tool (Hulme et al., 2009; Karlan et al., 2014; Duflo, Banerjee, Glennerster, & Kinnan, 2013; Berg, 2010; Stewart et al., 2010). This growing interest amplifies in 72 million microsaving clients globally (Microfinance Information Exchange, 2012). Yet, to date, there has been no systematic investigation of the impact of such interventions on a range of welfare-related outcomes for poverty-affected households and individuals.
The Intervention
In a constrained economic climate, it is essential to evaluate the poverty-alleviating potential of interventions that do not necessitate infusion of large external capital but instead emphasise behavioural change and sustainable management of available resources. In consequence, scholars and practitioners have touted saving promotion interventions as a promising poverty-reduction tool in international development. Different forms of saving interventions are outlined below, grouped into three main categories: 1) product-based interventions, 2) motivational-based interventions, and 3) knowledge-based interventions. In line with Pande et al. (2012), the former category can be understood as supply-side saving interventions seeking to increase access to saving services and schemes whereas the latter two categories denote demand-side saving interventions, targeting usage of saving services. It is important to note that all of these categories may well overlap and that some interventions can be hybrids of different categories. The intervention types described below include both formal as well as informal saving mechanisms. In consequence of constrained access to formal banking for poor individuals in developing countries, use of informal saving devices is more prevalent and more diverse in comparison to the developed world.
Supply side: Product-based Interventions
Access to formal banking services is commonly restricted to poor people due to far distance from bank branches and a range of demanding rules and requirements such as minimum balances, opening, maintenance, and withdrawal fees, or regulatory barriers (Karlan et al., 2014; Pande et al., 2012; Collins et al., 2009). A range of interventions have been developed to address this problem and facilitate access to formal bank accounts. For instance, a trial by Dupas and Robinson (2009) offered market vendors and bicycle taxi drivers in Kenya the opportunity to open a bank account at no costs to themselves by covering the opening fee and minimum balance. Likewise, in a study by Prina (2013), Nepalese women in the intervention arm were given access to a fully liquid bank account at a local branch without any associated fees for maintenance or withdrawals. Innovative designs and saving technologies can further promote usage and demand of banking services (Pande, Cole, Sivasankaran, Bastian, Durlacher, 2012). For instance, mobile banking has increased in popularity and may be particularly well suited for people living in remote distance to bank branches. Accordingly, a study in Sri Lanka (de Mel, Herath, McIntosh, & Woodruff, 2012) introduced scratch cards that clients could buy and use to make a deposit into their personal saving accounts. Other studies have used banking agents for improving access to banking in rural remote areas. For instance, Dupas et al. (2014) conducted a study in Kenya in which banking agents visited individuals at home in order to deliver information on account opening and maintenance as well as distribute vouchers for the opening of free saving accounts. Further, a project by Ssewamala and Ismayilova (2009) in Uganda adds saving incentives to youth accounts: Participants receive the matched sum of the amount of money they pay into their account as a top-up. Lastly, access to formal banking may be promoted on a state-level. Burgess and Pande (2005), for instance, examine the economic consequences of a policy-driven branch expansion across India in a quasi-experimental design.
Alternatively, interventions can provide simple saving devices such as metal boxes/safes, lock boxes, or piggy banks that could help to make saving more secure for poor households (Dupas & Robinson, 2011). Several interventions have combined the provision of saving devices such as lock boxes with a self-control mechanism where the key to the lock box is kept by a third person (here: the researcher) and is only returned to the owner after a certain period of time or after a specified saving amount has been reached (Dupas & Robinson, 2011; Rutherord & Arora, 2009; Ashraf, Gons, Karlan, & Yin, 2003).
Finally, a last method is to introduce group-based saving schemes such as ROSCAS, group-based bank accounts, or financial self-help groups that usually operate as a system of regular ‘pay-ins’ and rotating ‘pay-outs’ or ‘prizes’ (Rutherford & Arora, 2009; Ambec & Treich, 2007). Group-based approaches such as self-help saving groups might make saving practices more secure and regulated (when compared to saving at home) and additionally assert more control over the individual through peer pressure and common rules (Dupas & Robinson, 2011; Matthews & Ali, 2003; Ashe & Parrot, 2003; Zapata, 2003).
Demand-side: Motivation-based Interventions
Interventions can further leverage on motivational elements that seek to facilitate and encourage saving practices. For instance, Fiorill and colleagues (2014) and Soman & Cheema (2011) examine how visual representations of saving goals, the drafting of detailed saving plans, and sms- or mail-delivered saving reminders can increase the uptake of saving products or realised saving balances. Further, there is a growing body of literature on saving commitments (Karlan & Linden, 2014; Gine, Goldberg, Silverman, & Yang, 2012; Brune, Gine, Goldberg, & Yang, 2011; Ashraf, Karlan, & Yin, 2006). Scholars often distinguish between a) hard commitments that are either associated with flexibility constraints or economic penalties for failure and b) soft commitments that are primarily associated with psychological costs for failure. An example for the former can be found in a study by Ashraf and colleagues (2010) in the Philippines in which clients had to define a goal amount of money to be saved and agree that withdrawals from the account are restricted until the goal has been reached. Conversely, Soman & Cheema (2011) provide an example of a soft commitment where participants earmark amounts of money for specific purposes such as education. It is hypothesised that violations of these self-established rules or commitments would result in feelings of failure and guilt (Benabou & Tirole, 2004). The primary function of commitment devices – whether soft of weak – is thus to increase individuals' self-control and promote the motivation to save money.
Demand-side: Knowledge-based Interventions
In addition to the low provision of formal banking services, a lack of financial knowledge is commonly cited as a barrier to pro-saving behaviour (Karlan et al., 2014). In consequence, non-product interventions that emphasise financial literacy and knowledge have increasingly gained popularity (Karlan, McConnell, Mullainathan, & Zinman, 2010). Programmatic efforts are thereby based on the assumption that financial knowledge is an antecedent to healthy financial decision-making and that increases in financial literacy will ultimately increase savings and financial wellbeing (Fernandes, Lynch, & Netemeyer, 2014; Karlan et al., 2014). One of the most prominent examples here is the Aflatoun program that was designed to promote financial skills among children and adolescents. The curriculum includes lessons about financial planning, budgeting, saving and proper spending. The curriculum is available in over 30 languages and the programme has been implemented in 113 countries to date (see http://www.aflatoun.org/).
How the Intervention Might Work
The primary focus of saving promotion interventions typically lies on promoting saving and thus increasing saving amounts. And yet, the ultimate goal remains poverty alleviation. This review will therefore examine the effects of interventions on: a) intermediate outcomes, namely uptake of and increases in savings, saving-relevant motivations and knowledge, as well as b) distal outcomes, namely the wider multi-dimensional aspects of household poverty and economic wellbeing.
Although initially developed for health research, behavioural theories (e.g. Fishbein & Yzer, 2003; Bandura. 1986; Ajzen & Fishbein, 1980; Fishbein & Ajzen, 1975) are now widely applied in intervention research and can illustrate how saving promotion interventions may help to realise savings. Most behavioural theories postulate that performance of a specific behaviour (here: saving money) is determined by a person's intention to perform that behaviour. Intention, in turn, is viewed as a function of a) attitude towards the behaviour, b) perceived norms with regards to the behaviour (formed by a person's social network), and c) self-efficacy of performing the specific behaviour (Fishbein & Yzer, 2003). Following this, saving promotion interventions – particularly interventions with a motivational component – may change people's attitudes about the importance of savings, which could then translate in the uptake of saving practices. Similarly, product-based as well as knowledge-based interventions might raise individual's awareness about available saving devices and thereby affect their attitudes. In addition, interventions could promote self-efficacy, for instance by increasing financial planning skills or increasing knowledge about different saving techniques. Further, self-efficacy could be enhanced by providing participants with tools and mechanisms that make it easier and safer to save money, for instance through access to bank accounts. Alternatively, addressing self-control problems and providing financial tools such as commitment devices that can help inhibit spending on temptation goods or present-biased accounting could increase self-efficacy. Lastly, motivational and group-based saving approaches (e.g. promoting membership in saving groups) can seek to leverage on the dimension of perceived norms about the value and importance of saving money and thereby promote intentions to save. In the same vein, the presence of banking agents or administration of saving reminders can alter individual's perception of certain social values and expectations and thus have a significant impact on their perceived norms. While each type of saving promotion may have an impact on the different behavioural concepts outlined above, it is conceivable that a combination of several approaches is most likely to alter all three components and thus effectively instil behavioural change with regards to saving.
Turning to the hypothesised distal outcomes, it is essential to elaborate on the mechanisms of how increased savings may causally translate into poverty alleviation. First, savings can increase the resilience of individuals and households to unexpected economic shocks and adverse events. In consequence, vulnerability to poverty, that is, the threat of future poverty, is likely decreased (Klasen, Lechtenfeld, Povel, 2015). Unanticipated events such as illness or death of a household member can eliminate income sources and necessitate high expenditures on medical or funeral costs (Booysen, 2004; Bollinger & Stover, 1999). Hence, household savings can be viewed as substitute for a formal insurance mechanism and thus cushion the downstream impact of a financial crisis. Alternative coping mechanisms such as the sale of productive assets, borrowing at disproportionally high interest rates, or removal of children from school would likely lead to further impoverishment (Dupas & Robinson, 2009; Barnes, Gaile, & Kimbombo, 2001; Jacoby & Skoufias, 1997).
Second, while “behavioural anomalies” (Bryan, Karlan, & Nelson, 2010, p.681) such as the purchase of temptation goods or present-biased decision-making are characteristic of individuals across the globe, Banerjee and Mullainathan (2010) argue that the poor have less resources for absorbing such self-control problems. The authors demonstrate how the proportion of income spent on temptation goods is higher for individuals with very low income. Temptation spending and present-biased decision-making are therefore more consequential for the poor (ibid; Collins, Morduch, Rutherford, Ruthven, 2009; Prahalad & Hammond, 2002). Based on these considerations, poor individuals require mechanisms that can help them set aside money and thus decrease the immediate availability of cash. In consequence, time-inconsistent decision-making and purchase of temptation goods become less likely (Prina, 2013; Ambec & Treich, 2007).
Third, it has been argued that the accumulation of savings can take the form of investments in the future through the accumulation of larger lump sums (Collins et al., 2009). As Thaler (1990) described, once people dedicate a certain amount of their moneys to the purpose of saving, they are more likely to consider this money as unavailable for other expenses (Stewart et al., 2010; Dupas & Robinson, 2009; Rutherford, 2000). Such behaviour could potentially a) increase the overall wellbeing of poverty-ridden households with regards to health and housing quality, b) increase household income through sustainable accounting and investment in productive assets, and c) reduce the inter-generational transmission of poverty by investing in children's education, health, and nutrition (ibid).
Figure 1 illustrates the logic model of the above mechanisms. The arrows indicate how different types of saving promotion programmes may have an impact on the three behavioural aspects outlined above. Instilled behavioural change may then amplify in the uptake of saving and increases in realised saving amounts. In the long run, changes in these intermediate outcomes can have a downstream effect on distal outcomes such as poverty via the three causal pathways displayed in the bottom part of Figure 1.
Logic Model
Why it is Important to do the Review
Savings interventions are advocated as a new promising anti-poverty tool in international development. Yet, there is to date little systematic evidence on whether saving promotion interventions can effectively reduce poverty. A number of randomised controlled trials have found indication of a positive impact of pro-saving interventions on intermediate outcomes such as increases in saving amounts and uptake of saving products (Prina, 2013; Dupas & Robinson, 2009; Ashraf, Karlan, & Yin, 2010; Ssewamala, Han, Neilands, Ismayilova, & Sperber, 2010a; Ssewamala et al., 2010b; Ssewamala & Ismayilova, 2009). There is less empirical evidence, however, on whether saving interventions can effectively influence more distal outcomes such as household poverty and findings from primary studies vary.
Six systematic reviews have been published on related topics (Brody et al., 2015; O'Prey & Shephard, 2014; Fernandes et al., 2014; Pande et al., 2012; Stewart et al., 2012; Duvendack et al., 2011). However, none of these reviews have looked at the same kind of saving promotion programmes as proposed in this review. Fernandes et al. (2014) investigate the impact of financial literacy and financial education programmes that are, however, not exclusively savings-oriented. Further, the majority of studies that this review considered were implemented in populations of industrialised countries, which are likely to exhibit a range of different characteristics when compared to participants in low- and middle-income countries. Pande et al. (2012) focus exclusively on formal banking services and thus exclude a range of other saving interventions such as those promoting access to saving groups or training financial literacy on savings. Stewart et al. (2012) and Duvendack et al. (2011) examine a broader range of programmes, including microfinance interventions while Brody et al. (2015) analyse a range of self-help groups and not exclusively those with economic objectives. Lastly, O'Prey and Shephard (2014) examine the impact of financial literacy programmes, but thereby go beyond savings and include programmes that emphasise other literacy-relevant aspects such as numeracy in general. Also, again, their review includes studies from developed countries as well and findings might therefore not be valid for populations in LMICs. The proposed review project therefore attempts to exclusively look at saving interventions in LMICs that are not combined with microloan schemes or other cash transfers as these could potentially confound findings on poverty alleviation.
Following the above considerations, this systematic review intends to fill a gap in recent literature by exploring different kinds of saving interventions and their effectiveness in a) promoting intermediate outcomes such as increases in individuals' savings as well as b) promoting more distal outcomes such as decreases in household poverty and improvement in households' wider wellbeing.
In order to determine overall effect sizes, this review also aims to conduct a meta-analysis if possible – which no synthesis on savings has previously reported. We hope that the accumulated data will allow us to conduct meta-regression that would help establish which population groups benefit most from saving interventions and which programme types (e.g. access to bank accounts, financial literacy, etc.) have the largest impact on poverty alleviation. These findings would have important implications for policy makers and development activists and will provide essential information on the directing of funding resources towards the most effective types of saving programmes.
OBJECTIVES
The objective of this review is to synthesise evidence of individual studies investigating the impact of various kinds of saving promotion interventions on a) increases in savings (intermediate outcomes) and b) reductions in poverty levels (distal outcomes). The review will also look at how contextual and implementation factors, such as characteristics of study participants, setting of the intervention and programmatic components moderate intervention effects and thus affect outcomes.
Although there are previous reviews on the impact of related intervention, none has conducted a meta-analysis on the effects of saving promotion. We believe that quantitative aggregation of data from primary studies can add important insights to existing literature. Saving interventions have become considerably popular in international development practices over the recent years and organisations such as the Abdul Latif Jameel Poverty Action Lab (J-PAL) and Innovations for Poverty Action (IPA) have launched a range of impact evaluations of said programmes. Based on a preliminary scoping search, we are confident that we will find sufficient primary data to be able to conduct meta-analysis.
METHODOLOGY
INCLUSION CRITERIA
Criteria for including and excluding studies
Types of study designs
Eligible study designs should feature a credible comparison group or appropriate counterfactual condition to allow for an unbiased estimate of the treatment effect. Although there is a range of high-quality quasi-experimental study designs, comparison analyses have pointed to discrepancies in findings when compared to truly experimental study designs, with a tendency of the former to over-estimate effect sizes (Glazerman, Levy, & Myers, 2003; Shadish & Ragsdale, 1996). Therefore, randomised controlled trials (RCTs) have long been considered as a ‘gold standard’ in impact evaluation and are best equipped to maximise internal validity. Over the recent years, the evidence-based paradigm has gained prominence in international development. Duflo and colleagues (2004) note: “Rigorous evaluations through randomised experiments can revolutionise the social policies of the 21st century as randomised experiments have revolutionised the 20th century medicine.” Duvendack et al. (2011) have included quasi-experimental study designs in their microfinance review in light of a scarcity of randomised controlled trials. The authors conclude that almost all of the identified impact evaluations suffer from weak research designs. Since the publication of this review, organisations such as J-Pal and Innovations for Poverty Action have launched a large number of RCTs on microfinance and saving initiatives. Following this, the present review will draw on the most reliable and rigorous evidence and thus only include those study designs that are individual or cluster randomised controlled trials.
The quantitative synthesis of this review will exclude quasi-experimental studies, observational studies, as well as qualitative studies.
While the proposed review will not systematically search for qualitative evidence, efforts will be made to identify key qualitative papers that can help further elucidate the causal pathways that run from saving promotion to increases in savings (intermediate outcomes) and eventually reductions in poverty levels (distal outcomes) (Harachi et al., 1999). Further, the review project aims to identify relevant qualitative and contextual records for included studies. These could include qualitative outlets such as from focus group discussions or qualitative programme piloting. Further, detailed programme documentation will be searched such as intervention manuals or training plans. Lastly, major efforts will be made to identify information on programme implementation (e.g. was dose and duration of the programme as intended?), delivery (e.g. quality and fidelity to the manual), and participants' engagement with the programme (e.g. attendance and active participation). Understanding implementation/process and fidelity can help explain variations in programme outcomes and contribute to a more nuanced understanding on why some participants may show most improvements while others do not (Durlak & DuPre, 2008; Carroll et al., 2007; Dane & Schneider, 1998). We will mainly conduct Google searches and screen through trial reports on J-Pal, 3ie, and Innovation for Poverty Action websites. Further, all first authors will be contacted with requests for relevant qualitative material.
Types of participants
The review will include interventions that are targeted at poor individuals or households in low- and middle-income countries (as defined by the World Bank in 2015). The inclusion criteria are not further restricted to gender, age, or occupational status. If the retrieved data allows, meta-analysis will examine whether effect sizes vary for different subgroups (defined by gender, age, occupational status, etc.).
Types of interventions
The review will include all kinds of saving promotion interventions.
The review will exclude any intervention that combines saving promotion (as outlined above) with additional components that could hypothetically have an impact on poverty, financial stress, or savings behaviour. This may include microfinance programmes that combine microloans with micro-saving schemes, insurance programmes with saving schemes, or cash transfers with integrated saving schemes. It will further exclude programmes that integrate financial incentives to save, such as provision of monetary top-ups contingent on realised saving amounts. An exception from this can be multi-arm trials, as long as saving promotion, as defined above, is employed as a stand-alone treatment arm. A further exception would be if control and intervention group received the same kind of services with the only difference of an additional saving component in the intervention group. The potential poverty alleviating impact of micro-credits, financial incentives, or cash transfers would in this case be balanced across groups.
Types of outcome measures
The primary interest of this review lies on whether saving promotion interventions can effectively reduce poverty. Literature exhibits broad variations in the operationalisation and measurement of poverty. Essentially, one can distinguish between a) a money-metric approach that relies primarily on households' incomes or expenditures as proxies for poverty and b) a multidimensional approach that considers wider human wellbeing, namely aspects of health, education, food security, quality of housing, and standards of living (see Sen, 1993, 1992). A prominent example for the latter approach is the Multidimensional Poverty Index (Alkire et al. 2016; OPHI 2013; Alkire & Foster 2011) that aggregates indicators of nutrition, mortality, schooling, and living standard into a poverty ranking and is used in the annual Human Development that is released by the United Nations Development Program. Following this, the primary outcomes listed below seek to reflect the multiple dimensions that are commonly used for measuring poverty. Studies will be considered eligible irrespective of the operationalization of poverty they apply and will be grouped accordingly.
In addition, this review will examine the intermediate outcomes that lie on the hypothesised causal pathway from saving promotion to poverty reduction. It can be assumed that these outcomes are more malleable to change and easier to capture/measure and will therefore be more frequently reported by identified studies.
Primary/Distal outcomes
Poverty (chronic as well as transitory) Food security and nutrition Educational achievement (e.g. attending school, graduating from school) Sexual risk behaviour
1
Expenditures on health, education, nutrition Business profit
Intermediate outcomes
Aggregated savings in all forms
2
Uptake of a saving product/tool Knowledge and financial literacy about saving Attitudes towards savings Intentions to save Consumption smoothing
- Investments in profitable business
Types of control condition
The control group should be either untreated (i.e. ‘treatment as usual‘), be placed on a waiting list, or receive placebo treatment. Placebo treatment could exist in the form of an intervention focused on a fully unrelated topic such as health or hygiene.
Duration of follow-up
No restrictions are put on the duration of follow-up. However, it is conceivable that more distal poverty-related outcomes will only amplify after a certain amount of time. Therefore, longer follow-up periods will be considered as particularly important.
Types of settings
The review will include all studies that were implemented in low- or middle-income countries, defined as at the time the intervention was carried out. The
Any post-hoc changes to the eligibility criteria that may occur at the data extraction stage will be reported explicitly in the final review.
SEARCH STRATEGY
The following databases will be searched:
Cochrane Collaboration Library Campbell Collaboration Library EPPI-Centre Library DFID Database for Systematic Reviews (http://r4d.dfid.gov.uk/SystematicReviews.aspx) 3ie Database for Systematic Reviews (http://www.3ieimpact.org/en/evidence/systematic-reviews/)
PsycINFO International Bibliography of Social Science (IBSS) SCOPUS Web of Sciences Applied Social Science Index and Abstracts (ASSIA) Conference Proceedings Citation Index - social science&Humanities JOLIS (database of 14 World Bank and International Monetary Fund libraries) ECONLIT IDEAS/ RePEc Business Source Premier 3ie Impact Evaluation repository (indexed list of impact evaluations) 3ie RIDIE (list of on-going/registered impact evaluations) Abdul Latif Jameel Poverty Action Lab (J-PAL) Innovations for Poverty Action (IPA)
In addition, to identify relevant grey literature, the following databases of international organizations will be searched: Elvis USAID Development Experience Clearing House: World Bank Impact Evaluation Working Paper Series Research4Development (DFID) African Development Bank Evaluation Reports: Agence Française de Developpement: Impact Evaluations Asian Development Bank Evaluation Resources Inter-American Development Bank Evaluations
Lastly, we will hand-search reference lists of all included studies as well as of the following seven systematic reviews and research syntheses: Brody et al. (2015); Pande et al. (2012); Stewart et al. (2012), Duvendack et al. (2011), Fernandes, Lynch, & Netemeyer (2014); O'Prey & Shephard (2014); Karlan, Ratan, & Zinman (2014).
A PRISMA flow chart will be used to document the search process, including the flow of eligible impact evaluations and the numbers of linked programme implementation documents obtained. For each individual database, search date and number of retrieved records will be documented. Further, dates of title and full text screening will be recorded.
DATA EXTRACTION AND ANALYSIS
All identified titles will be screened by the first author (JIS). A subset of titles (20 per cent of all titles) will be double screened by the second review author (AM) to check for consistency in decisions on the inclusion/exclusion of studies. The screening process will be assisted by the online screening tool Rayyan. 3 Discrepancies between two authors will be flagged automatically by the tool and subsequently discussed and potentially referred to a third author (YS) in order to reach consensus. In case of a substantial number of discrepancies in screening decisions, the second review author will screen all titles rather than only a subset in order to increase accuracy and inter-rater reliability. Full text screening will be done by two review authors (JIS AM, JZ) for each individual study. Data from included studies will be extracted independently by two review authors respectively (JIS, AM, JZ, UF). Likewise, risk of bias assessment will be conducted by two review authors per individual study (JIS, AM, JZ, UF). Disagreements between the two review authors will be discussed and in case no agreement is reached, a third review author (YS) will be consulted. All extracted data will be entered into an Excel data extraction form (see Appendix 1). The form has been piloted by three review authors (JIS, AM, JZ) and applied to two pre-known eligible studies and was revised accordingly.
Coding of Included Studies:
Extracted data will be documented in a Table with ‘Characteristics of Included Studies‘:
Authors Publication type Publication date Country/Region of study
Program subtype Length of the program
Gender Age Ethnicity Initial income/poverty level
Study design Sample size Length of follow-up
Intermediate outcomes Distal outcomes Measurements/Scales used
For studies with multiple records/publications, the latest will be listed as identifying study reference. All records will be assessed in full in order to retrieve all relevant outcomes.
If information is missing from studies, efforts will be made to contact first authors. In case of no response, co-authors will be contacted and reminders sent. It is further possible that information required for meta-analysis cannot be retrieved or is not available. In this case, the study can only be reflected in the narrative synthesis of the review.
Risk of bias assessment:
The risk of bias assessment will be based on the Cochrane Risk of Bias Assessment Tool for Randomised Controlled Trials (Higgins et al., 2011). Six domains are assessed for risk of bias and quality of evidence: 1) random sequence generation, 2) allocation concealment, 3) blinding of participants/personnel, 4) blinding of outcome assessors, 5) incomplete outcome data, 6) selective outcome reporting. Each of the six domains is assessed according to the following categorization scheme: a) low risk of bias, b) high risk of bias, and c) unclear risk of bias. The Cochrane Handbook provides detailed guidelines on how to code the risk of bias for each of these six domains. However, recognising that blinding of participants to the intervention is impossible (even when authors use ‘no saving placebo’ for controls such as a hygiene intervention which cannot affect outcomes of interest), further potential sources of bias will be assessed including likelihood of spillover effects and motivation bias, drawing on the risk assessment tool developed by Hombrados and Waddington (2012), and implementation fidelity.
Meta-Analysis
Calculating Effect Sizes:
Individual study effects will be synthesised statistically using Stata SE/13. Effect size estimates and 95 per cent confidence intervals will be extracted for each individual study (if applicable). The following effect size measures will be used: For continuous outcome measures, standardised mean differences (SMD) will be calculated. To adjust for potential bias from small sample sizes, we use Hedges' g correction for all effect sizes.
4
An SMD greater than zero will indicate increases in the outcome of interest in the intervention group compared to the control group. An SMD less than zero will denote a reduction in the outcome in the intervention group as compared to the control group. For outcomes that are measured on a continuous scale in some studies and dichotomised in other studies (e.g. increases in saving amounts), we will transform odds ratios (ORs) into ##SMDs and use Hedge's g correction as described above (for transformation, see: Borenstein, Hedges, Higgins, & Rothstein, 2009; Sánchez-Meca, Marín-Martínez, & Chacón-Moscoso, 2003). For outcomes that are predominantly measured on a binary scale (e.g. uptake of savings), odds ratios will be reported as effect size measure.
Effect sizes across studies will be combined using the inverse variance weight method. A random-effects meta-analysis will be applied, assuming differences in individual studies. All meta-analyses will be conducted separately for a) different outcomes of interest (e.g. increases in savings, decreases in poverty levels, etc.), b) different intervention types (e.g. access to bank accounts, financial literacy, etc.), and c) different follow-up periods (e.g. six months, one year, etc.) corresponding to those listed above. In the case of multi-arm trials, each treatment arm will be analysed independently and coded as one intervention type.
Some of the specified outcomes such as poverty may show large variations in how they are operationalised across studies. Once all data are extracted, it will be judged whether meta-analysis of these outcomes is appropriate. If studies report multiple effect sizes for the same construct (e.g. expenditures and assets for household poverty), multilevel meta-analysis with random effects will be considered (Polanin, 2014; Van den Noortgate, López-López, Marín-Martínez & Sánchez-Meca, 2013).
Unit of analysis errors can arise if the unit of allocation and the unit of analysis differ. If not already accounted for in the included primary studies, we will adjust the effect estimates using an intraclass correlation coefficient to correct for unit of analysis error in cluster randomised controlled trials (see Littell, Corcoran, & Pillai, 2008). In case individual studies do not report ICC estimates, we will seek to include ICCs from similar cluster RCTs.
Assessing of Heterogeneity:
In order to adequately address heterogeneity of results, the proposed review intends to examine evidence by global regions. Therefore, the review project will initially focus on evidence from Sub-Saharan Africa, while maintaining plans to later extend the analysis to Asian/Pacific and Middle/South American regions.
Heterogeneity of results will be assessed by visual inspection of forest plots and statistical heterogeneity tests. I-squared and Q as well as tau-squared will be calculated to assess heterogeneity. Q will be tested for significance whereas I-squared and tau-squared are used to give some indication on the magnitude of heterogeneity. In contrast to Q, I-squared and tau-squared are not directly affected by the number of included studies (Borenstein et al., 2009; Littell, 2008). Acknowledging that I-squared and tau-squared are less reliable with a small number of individual studies, we avoid the use of simple thresholds to diagnose heterogeneity.
Subgroup Analysis:
Subgroup analyses will be run to assess whether programme features and participant characteristics show differential impacts on the intervention outcome. For this purpose the following moderator variables will be assessed, if possible: gender of participants age group of participants (dichotomous variable: children/adolescents versus adults) baseline poverty level of participants educational level of participants (ordinal variable: no schooling, primary school, secondary school, higher) programme component (categorical variable differentiating between different programme types: a) educational programmes, b) motivational programmes, c) product-based programmes, e) mixed-component programmes) duration of the programme (dichotomous variable brief vs. long whereby we define brief as three or less sessions) study setting (rural vs. urban) quality of programme implementation (dichotomous variable capturing whether a fidelity assessment suggested low or high quality of implementation)
A random-effects statistical model will be used to weight and estimate effect sizes for each moderator category. If the sample size within a sub-group is lower than 5, we employ a pooled estimate of the variance component (Borenstein et al., 2009). To test for differences in average effect sizes between subgroups, we apply a Q-test based on ANOVA.
An alternative methodological approach would be to conduct meta-regression in order to explain some of the variance in treatment effects. However, the use of meta-regression requires an appropriately large ratio of studies examining covariates (Borenstein et al., 2009). That is, decision to perform meta-regression will depend on the number of identified eligible studies for this review and will be considered as inappropriate for less than ten individual studies (Higgins & Green, 2011).
Sensitivity Analysis:
Sensitivity of results will be assessed via sub-group analyses based on risk of bias assessments. Three categories will be compared with regards to effect sizes, namely high, unclear, and low risk of bias. In cases where effects from studies using different operationalisations of an overall concept (e.g. household poverty) were combined, we will investigate sensitivity of results before and after pooling effect sizes.
Assessing Publication Bias:
To assess the potential for publication bias, we compare mean effect sizes of published studies with those from unpublished sources. If the number of eligible studies is sufficient (recommended minimum of ten studies), funnel plots will be inspected to assess whether estimates from individual studies are symmetrically distributed around the mean effect size. In addition, Egger's (1997) test will be utilized to test for funnel plot asymmetry. Duval and Tweedie's (2000) ‘trim-and-fill ‘method will be used to correct for possible publication bias. The method provides a) an estimate of the number of missing studies, and b) an adjustment of the estimated intervention effects. (Higgins & Green, 2011).
Treatment of Qualitative Data:
Retrieved qualitative data from programmatic records or possible process evaluation will be reflected in the narrative synthesis of study evidence. At the very least, this information will be used to report intervention components clearly. Clear information about intervention components can be used in moderator analysis, something which other systematic reviews on microfinance topics have not been able to do because they have not collected systematic information about intervention components (Vaessen et al., 2014; Brody et al., 2015). If we find sufficient information on programme fidelity and implementation, we will determine for included studies whether a fidelity assessment suggested low or high quality of implementation and run sub-group moderator analyses on this (for fidelity appraisal tool, see Appendix 3).
Summary of findings Assessment:
We will use the GRADE approach to interpret and assess the quality of evidence of the review findings (Guyatt et al., 2011). For this purpose, we will use the GRADE profiler (GRADEpro) to construct the “Summary of findings” tables. These tables effectively summarise the evidence for separate review outcomes providing information on the assumed and corresponding risks in each of the groups of the chosen comparison, the magnitude of effect (binary of continuous) and the quality of evidence rating. We will consider all the primary/distal outcomes of the review for inclusion in the “Summary of findings” table.
Footnotes
APPENDIX
1
A range of recent programmatic efforts have leveraged on economic empowerment which has been linked with sexual risk behaviours such as (quasi)-transactional or age-disparate sexual relationships (e.g. Baird, Garfein, McIntosh, & Berk, 2012; Ssewamala& Ismayilova, 2009;. Kim, Pronyk, Barnett, & Watts, 2008; Pronyk et al., 2005).
2
This denotes the total amount of realised savings, including savings held in a bank account, at home, in a savings group, savings in kind, etc. The outcome is intentionally defined broadly and aims to capture possible substitution effects, for instance reductions in savings held at home due to the opening of a saving account.
4
5
Adapted for social sciences publications from: Cochrane Highly Sensitive Search Strategy for identifying randomized trials in MEDLINE: sensitivity- and precision-maximizing version (2008 revision)
6
Using Cochrane EPOC LMIC filter which was developed for the World Bank classification 2009 and which we then updated for 2015