Abstract

1 Background
1.1 DESCRIPTION OF THE PROBLEM
Secure and predictable access to land as a productive resource is key to the livelihoods of millions of farmers around the world. Secure land rights enable farmers to invest in long-term improvements to farms and soils in the expectation that they will reap the benefits of those investments without fear that their land will be confiscated arbitrarily. Investments in improvements to soil fertility, and capital improvements such as irrigation equipment and fences, pay for themselves over multiple cropping seasons. Recent research on the use and management of common pool resources, such as forests and grazing lands, shows that ecological and livelihood outcomes are greater where local user groups have clear and secure rights to the resource; the right to exclude ineligible users often emerges as decisive to local communities' ability to manage their natural resources sustainably (see especially Persha, et.al [2010] and Porter-Bolland, et.al [2011]). A recent meta-analysis has also suggested that natural forests are better managed under community ownership rather than state ownership, because a greater share of the benefits of good forest stewardship accrue directly to local communities (ibid.). Formal and informal land rights are therefore seen as keys to improving the conditions of the poor in developing countries in terms of economic growth; agricultural production; food security; natural resource management; gender-related inequalities; conflict management and local governance processes more generally.
Many farmers in developing countries hold customary rights that are considered highly secure in the context of local social arrangements, but which are not accorded legal status in the country's statutory property regimes. Rather, land assigned under customary arrangements is statutorily categorized as public land, and subject to the stewardship and administration of public agencies. These areas of public land have been the principal targets of large-scale acquisition of land, or so-called “land-grabbing,” in many developing countries. In the process, the customary tenure arrangements that delivered secure tenure rights to generations of farming families have been over-ridden and thousands of farming families face displacement. An appropriate policy remedy may be to accord extant customary arrangements statutory status equal to that accorded land held under public land and freehold tenures (Knight, 2010; United Nations, 2012). The land-grabbing problem underscores the connection between clear statutory recognition of the land rights, whether they are based on customary rights or freehold rights, and tenure security.
Leading multilateral and bilateral development agencies accord high priority to policy reforms that strengthen tenure security, especially as elements of strategies to reduce poverty among women and other traditionally disadvantaged members of society. According to a 2003 World Bank study,
“Providing secure tenure to land can improve the welfare of the poor, in particular, by enhancing the asset base of those, such as women, whose land rights are often neglected. At the same time, it creates the incentive needed for investment, a key element underlying sustainable growth.” (Deininger, 2003, ix).
Other agencies, including USAID and FAO, have placed support for reforms promoting tenure security close to the centre of their funding strategies (see especially USAID and MCC [undated] and FAO [2011]). More about the historical and contextual understanding and success of land tenure interventions is discussed in section 1.4.
1.2 DESCRIPTION OF THE INTERVENTION
Land rights may include a wide range of rights to use, own and/or transfer land, as well as enforce rules and exclude outsiders. Strengthening of land rights can take a variety of forms that range from documenting customary uses to formalizing legal rights. Some forms may engage directly with the rights holder, for example through farm-by-farm land titling. Other forms of strengthening rights may act on a national level, for example constitutional reforms in Mozambique that recognize customary rights to land (van den Brink et al. 2006). National scale or even community level interventions that seek to strengthen rights may have differing impacts within populations, for example many interventions seeking to improve rights may lead to elite capture of benefits and subsequent loss of rights for poor and vulnerable sub-populations particularly in the absence of safeguards. The socially embedded nature of customary rights means land rights of many women depend on family relations that can be disrupted during interventions to strengthen rights resulting in men, but also women with greater status, exerting greater control over reform processes. Thus the observed impact depends on the type and scale of the assessment, across individuals, communities, regions and countries (Place and Swallow 2000).
The review will examine the specific impacts of two types of land rights interventions: Conversion of communal or non-demarcated rural land to freehold title and registration of such rights in an official registry. This kind of conversion has been a standard approach, under the presumption that communal land tenure rights are inherently insecure. Such conversion typically consists of adjudicating and assigning land rights, physically surveying boundaries, and registering rights and boundary demarcations in an official land registry. Subsequent to this conversion, all transactions involving the land are intended to be recorded in the official registry. Statutory recognition and codification of customary or communal rural land rights, and registration of these rights in an official registry. This approach recognizes that communal systems need not be inherently insecure. Rather, it is based on the idea that these rights might be protected from arbitrary taking through statutory protection. In practice, such registration can be used to protect against the use of statutorily recognized freehold or public land tenure from the grabbing of communally-held land by outside interests. Such processes typically involve law reform, with examples including Botswana's Tribal Land Act of 1968, which extended statutory recognition to the traditional customary tenure system while replacing chiefs as land administrators with civil land boards. More recently, the Kenya National Land Policy of 2009 places customary land rights on equal legal par with freehold tenure and public land.
Informal processes may resemble what we have described above but without statutory backing– conversion and communal rights registration interventions constitute a very prominent class of interventions in this sector. Despite the value of these informal interventions, this review will focus on the affects of the added value of formal registration of land rights. This decision was taken for the practical reason that effects of informal practices are less likely to be robustly and rigorously measurable and comparable, and because formal interventions are more relevant for development projects aiming to introduce and replicate effective interventions transparently and accountably.
We exclude from the review related justice interventions (e.g., paralegal, outreach, alternative dispute resolution interventions, etc.) and enforcement capacity interventions (e.g., training of justice sector actors, digital boundary marking, etc.). Land inheritance reforms are also excluded. 1 Relevant moderating factors and mechanisms/intermediate outcomes for these interventions are likely different, and analyzing them would require separate theories of change and literature searches. Associated with this exclusion, it should also be noted that we expect to find that many interventions relevant to this study will constitute only one part of a bundle of mutually supportive interventions affecting tenure undertaken simultaneously in a given context, and that extracting specific effect sizes– and accordingly differentiating out the clear-cut success or failure– of specific interventions within such bundles may not be possible. In our analysis, we will indicate whether or not interventions were carried out in isolation and if not, the other tools used will be identified and if studies include the information we will explain their perceived context specific roles in the qualitative section of the review. We will also suggest further work be done to review access-to-justice, enforcement, and other complementary interventions.
1.3 HOW THE INTERVENTION MIGHT WORK
A variety of factors are likely to influence effectiveness of land property rights interventions on productivity. Figure 1 presents the basic elements of a theory of change (causal chain) that draws on the research teams' own work in this area as well as the available literature (summarized below). The figure sketches out moderating factors, mechanisms of change and intermediate outcomes, and endpoint outcomes that we see as being important in understanding the effects of land property rights on productivity. The endpoint outcomes of interest include:

Theory of Change
Productivity of land use Welfare of pre-policy landholders, measured in terms of income and consumption, domestic violence, and gender equity. Welfare of post-policy landholders measured in the same way.
In theory, it is important to distinguish between the welfare of pre-policy and post-policy landholders in evaluating the welfare impacts of these interventions. To the extent that these groups differ, any analysis ought to incorporate the potential for adverse consequences for pre-policy landholders.
The theory of change proposes the following moderating factors as being important in determining the nature of the effects that are likely to follow land property rights interventions: Governance, including the nature of interests represented by those controlling policy. Social norms and practices, specifically ways in which gender, age, community standing, and other characteristics influence the other three moderating factors and individuals' ability interact with interventions in a particular social context. Land use, including population pressure on land, whether land is subject to mixed use (pastures and forests), as well as the types of cash crops grown, differentiated by prices and marketability. Markets, including the presence of credit markets and market demand for crops as well as demand for agricultural land, resulting from both local and international factors.
The theory of changes proposes that interventions and moderating factors operate through a number of intermediate drivers of change, including the following: Recognition of alienation right by those holding registered rights. This is presumed by conventional economic theory to provide collateral and, therefore, enable access to credit. Perceptions of tenure security, which is presumed by conventional economic theory to motivate investment of personal resources into production, investment Social conflict, including reducing amounts of land held in dispute and therefore not being used productively due to inheritance disputes, boundary demarcation disputes, or land use conflicts between, for instance, pastoralists and agriculturalists. At the same time, to the extent that such registration changes who has access to land, these interventions may indeed trigger violence or other forms of contention over these changes. Displacement of tenants whose rights to land are denied as a result of the intervention.
Intermediate outcomes include shifts in land, labor and agricultural inputs relevant to both short and longer term production. More specifically, these would include changes in: Investments of resources into short term production and land (fertilizer, pesticides, etc.) Investment of resources into longer term production and land (e.g., soil conservation, tree crops, etc.), Fuller employment of land through leasing-out or sharecropping.
1.4 WHY IT IS IMPORTANT TO DO THIS REVIEW
Existing evidence on the effects of land property rights interventions is mixed and to a considerable degree dependent upon the initial land rights conditions. In many cases where existing rights are already secure through stable informal and customary systems, the formalization of rights through land titling, one form of strengthening rights, may have little impact (Pickney and Kimuyu 1994, Atwood 1990). In other cases, as in the Brazilian Amazonian frontier in the early 1990s, mechanisms for formalizing property rights, where no formal institutions had previously existed, are argued to have increased productivity and slowed forest loss (Alston et al. 1996). Alternatively, if strengthening land rights simply results in formalizing a bundle of overlapping rights customarily distributed through a community into private property, this “strengthening” could lead to the exclusion and marginalization of large sections of the community, including the poor, as is argued to have occurred alongside Kenyan tenure reform (Meinzen-Dick and Mwangi 2005). Thus it is important to understand to what extent the strengthening of rights in any context leads to new institutional realities and who bears the costs and benefits of changes in how land rights are assigned (Fort 2008, Bellemare 2010). As a result, a systematic review of these lessons would allow policy makers to define specific contexts and allow for consideration of likely success of a variety of potential land tenure interventions for women and men.
These inconsistent conclusions from studies on the relationship between strengthening rights and productivity have led academics and policy makers in recent years to try to investigate these differences (Brasselle et al. 2002). A literature review published by Dickerman et Al. (1989) on efforts to formalize and register customary land rights in Africa found that formalization had significant positive effects on investment and agricultural productivity in only a small number of particularistic contexts where customary systems had broken down or were absent. Rarely did the benefits associated with surveying land, adjudicating and assigning rights and maintaining official registers outweigh the costs. The authors suggested that registration in many settings had deleterious effects on the poor and on women farmers, particularly where women were not listed as joint title-holders.
Recent research on land certification programs in Ethiopia (Deininger, et. al. 2011) found that new low-cost survey and titling technology, along with growing demand for land and the opening of new markets, appear to result in higher net benefits in the form of agricultural productivity and farmer incomes. In the case of Rwanda's land tenure regularization program, access to land for married women appears to have improved in the short term (2.5 years after interventions), and investment in, and maintenance of, soil improvements increased as well, especially in female headed households (Ali, et. al. 2011). Other evidence suggests that titling, especially where it is an option in peri-urban settings, can be an effective pathway for widowed, single and divorced women to purchase secure land rights not otherwise available to unmarried women under customary tenure.
These and other studies underscore the complexity of attribution and the importance of context (Place 2009) to understanding relationships between security, registration and productivity, and to understanding gender dimensions. They also suggest that tenure security alone is not the single factor ‘silver bullet’ leading directly to higher farmer incomes attributed to tenure reforms by writers such as Hernando de Soto (2000). Context matters, including whether markets and credit institutions are in place and input and other costs are at levels conducive to competitive pricing of agricultural products (Bruce, forthcoming). Relevant questions have recently been raised about the extent to which much of the available empirical research on the effects of tenure security has a handle on tenure security as a concept (Arnot, et. al, 2011).
To date, and with the exception of Porter-Bolland et.al (2011) which concerns forest management, the team is unaware of any systematic review or meta-analyses on the relationships between land property rights and productivity, or other outcomes. In addition, Fenske (2010) highlights study design limitations in many of the studies that have not found significant impacts of tenure security. The concerns about inconsistent effects and design limitations provide a strong motivation for a systematic review. Such a review might enable policy makers to better predict the outcomes of alternative interventions in particular social, economic and cultural contexts. In addition, our methods, which include both quantitative impact assessments as well as qualitative research, have been shown in the field of medicine to be useful for a variety of purposes, including: ensuring decision-makers have the most accurate evidence; assessing key population traits relating to a given intervention; establishing whether further primary research is required; and gaining new insights into relevant population or institutional traits (Ring et al 2011).
Finally, this review will be useful in that we plan to highlight areas in need of further assessment through rigorous impact evaluation and by providing guidance on how to make the most of evaluation opportunities.
2 Objective of the review
The objectives of the review are as follows: to understand impacts of interventions to strengthen land property rights on agricultural and livelihood outcomes in rural areas of low and middle income countries; to assess whether these effects are different for men and women, and under what circumstances; to assess specific mechanisms that enable or limit productivity improvement (barriers and facilitators).
3 Methods
3.1 CRITERIA FOR INCLUDING STUDIES IN THE REVIEW [PICOS]
Although the literature on the relationship between property rights and productivity in developing countries is large, with theoretical and applied research dating to the 1960s, rigorous impact evaluations are not the norm. Initial searches found 12 studies that may meet the standards for our review (see Appendix I); 12 studies are quantitative effectiveness studies, while three are qualitative investigations).
3.1.1 Participants
We will include studies investigating smallholders and communities in rural farming systems in low- and middle-income developing countries. We will include only studies that have data disaggregated at least to the household level. Where studies permit disaggregation by gender, we will examine any differential impacts for women and men, and will include a section in the synthesis specifically addressing gender relevant results of our findings. We will also disaggregate effects by other sub-populations, including by social status and age. Finally, whenever possible, we distinguish between effects on pre- and post-policy landholders, and we will stratify our quantitative and qualitative analyses on these sub-populations when possible.
3.1.2 Interventions
The review will examine the impact of two types of land rights interventions: Conversion of communal or non-demarcated land to free-hold and titling of such land. Statutory recognition and codification of customary or communal land rights, and certification of these rights.
These interventions constitute a prominent class among land tenure interventions. We are excluding from the review other reforms, including those relating to justice, capacity-building, outreach, and inheritance.
3.1.3 Comparisons
We will include studies which compare farmers and communities where formal and informal activities to strengthen land rights have been implemented to control or comparison groups where these efforts have not been undertaken. Thus, the comparison conditions are the ‘status quo’ property rights situation prevailing in the absence of the intervention. As is always the case with evaluation of interventions in natural field settings, our comparisons will be between intervention settings and prevailing non-intervention conditions in terms of land tenure security. This implies a range of counterfactuals across studies, but should nonetheless provide a suitable benchmark against which to measure impacts within a given setting. In addition, we will address sources of baseline and effect heterogeneity in our analysis of effect moderation due to characteristics related to governance characteristics, social norms and practices, land use, and market conditions.
3.1.4 Outcomes
Based on the theory of change outlined above, we will examine outcomes that we classify as “endpoint” outcomes and “intermediate” outcomes. Endpoint outcomes will be the basis of our primary analysis, and intermediate outcomes will be the basis of secondary analysis as well as analysis of causal mechanism linking the interventions to the endpoint outcomes. Fostering or inhibiting this change are a number of drivers of change expected to impact the outcome of tenure interventions.
3.1.4.1 Final outcomes of interest include:
productivity of land use (measured in terms of market value of agricultural output) income/consumption or poverty (measured in terms of standard consumption or income metrics), and gender-based welfare outcome measures (measured in terms of variability in income/consumption or poverty by gender).
3.1.4.2 Intermediate outcomes of interest include:
Changes in investments of personal resources into production (measured in terms of market value of inputs), investment in longer term production (e.g., tree crops), and fuller employment of land through leasing-out or sharecropping.
Our assessment of impacts on final and intermediate outcomes will draw on evidence from quantitative effectiveness studies.
In order to assess mechanisms that enable or limit productivity improvement, we will seek measures on the following
Recognition of alienation right and associated use of land as collateral and access to credit. Customary rights holders' perceptions of tenure security
Social conflict and associated quantities of land held in dispute based in inheritance or boundary disputes, and violence or overt contention. Displacement of pre-policy tenants.
Our assessment of the relevance of these drivers of change will come from both effectiveness studies as well as qualitative evidence.
3.1.5 Study Types
3.1.5.1 Study designs eligible for quantitative synthesis of effects
We will use quantitative studies to assess impacts on intermediate and final outcomes. We will consider counterfactual studies that compare outcomes observed at the point of intervention to those in an appropriate second context.
Specifically, the review will synthesize quantitative evidence only from studies characterized by all of the following (See Appendix II): (a.) Randomized experiments or (b.) quasi-experimental studies that employ strategies for causal identification with clearly delineated treated and control groups and use some method for removing biases due to non-random assignment of treatment, including, but not limited to, regression adjustment, difference-indifferences estimation, instrumental variables regression, fixed effects regression, regression discontinuity, matching, or inverse-propensity-weighted estimation. While application of such a method is sufficient for inclusion in our study, we appreciate that not all studies apply methods for causal identification with equal rigor. Therefore, each of the included studies will also be assessed in terms of “risk of bias,” as discussed below. Studies that estimate the impact of either of the two interventions described above. Studies that obtain measurement on at least one of the endpoint or intermediate outcomes described above. Studies that estimate impacts with outcome data measured at the individual, household, or village/community level. Studies undertaken in developing countries (as defined by the World Bank) and that measure outcomes at some point between 1980 and 2011.
3.1.5.2 Study designs eligible for qualitative synthesis
While this review will use evidence gathered solely from experimental and quasi-experimental research to evaluate how interventions impact final and intermediate outcomes, it will also include results from a wider range of empirical research (including qualitative research) in order to, inter alia: assess factors contributing to the success or failure of interventions; identify how and why intended or unintended outcomes occur; understand the context in which un/successful interventions are carried out; elucidate the views beneficiaries have of the interventions; as well as more generally broaden the evidence base and understanding of evidence of intervention effectiveness and address effectiveness questions more specifically than might be otherwise possible (Spencer et al 2003 and Ring et al. 2011).
Eligibility of non-impact evaluation studies will be determined via a two-stage screening process to facilitate review of the most relevant studies while quickly filtering out inappropriate research based on the Critical Skills Appraisal Program (CASP) tool (Hannes 2010; Waddington et al. 2010). The first stage will screen out studies based on intervention, location, population, relevance to review questions, and study type (See Appendix IIIa). The second round of screening will focus on study quality based on frameworks outlined in Kuper et al. 2008, Spencer et al. 2003, and Waddington et al. 2010. Specifically this second round will filter studies based on clearly defined: research objectives; links to relevant literature; context and sample selection; data collection; methods; as well as quality and relevance of analysis (see Appendix IIIb).
3.1.5.3 Examples of studies included and excluded
Appendix I provides a table of studies that we might consider under the quantitative and qualitative inclusion criteria. As an example of a study that would qualify under our quantitative inclusion criteria, consider the study by Ayalew et al. (2011). The intervention under study is Rwanda's National Land Tenure Regularization Program, which is an example of a conversion intervention in a developing country. The study examines impacts on various land-use investments at the household level, which contributes to our assessment of intermediate outcomes at the appropriate level of analysis. Finally, the study employs a geographic regression discontinuity design, which satisfies our requirement for causal identification. An example of a study that does not qualify under our quantitative effectiveness criteria is the study by Goldstein and Udry (2008). In this case, despite the use of sophisticated econometrics and a focus on household-level land-use investments in a developing country (Ghana), the study does not explicitly estimate the impact of one of the two interventions described above and is hence, not eligible for inclusion. We also exclude studies based on cross-country regression methodology (e.g. Deininger and Squire, 1998).
3.2 SEARCH METHODS FOR IDENTIFICATION OF STUDIES
3.2.1 Electronic searches
We will include studies, and as far as is feasible, in all languages. Translation cooperation may also be requested in this regard. We will search the following online electronic databases from 1980: Agricola database ASSIA British Library for Development Studies CAB Abstracts (to be conducted by IDCG TSC) EconLit (to be conducted by IDCG TSC) Econpapers ELDIS FAO Gender & Land Rights Database Google Scholar (for both subject searches and citation searches of included studies) HeinOnline International Bibliography of Social Science (to be conducted by IDCG TSC) JOLIS Networked Digital Library of Theses and Dissertations OpenGrey PAIS Web of Science (for both subject searches and citation searches of included studies -to be conducted by IDCG TSC)
Our searches will be based on the key terms in the titles and abstracts, but will be supplemented by thesaurus terms used by individual databases, where appropriate, and by an LMICs filter (eg Cochrane EPOC LMICs filter) where search results warrant it. land.ti,ab. (tenure or right* or property right* or conversion or freehold* or titl* or codification or recognition or customary or certification).ti,ab. (impact* or evaluat* or effect* or experiment* or trial or random* or quasi* or natural experiment* or discontinuity or fixed effect* or regression or difference in differences or instrumental variable* or matching or inverse propensity weight*).ti,ab. (alienat* or collateral or credit* or secur* or conflict* or dispute* or violen* or displac*).ti,ab. (qualitative or findings or interview* or themes or experience).ti,ab. 1 AND 2 AND 3 1 AND 2 AND 4 AND 5
In addition to the electronic database searches, we will search grey literature from the leading institutions working on land tenure, including the following back through 1980:
University of Wisconsin Land Tenure Center International Land Coalition Think tanks such as PLAAS, AIAS, TEGEMEO Institute, ASARECA. Reports from key national donors such as USAID, DFID, GTZ/GIZ, AFD, USAID Land Tenure & Property Rights portal. Reports from international development organizations such as the CGIAR group, FAO, IIED, IFAD, and the World Bank.
These searches will be supplemented by contact with key authors in the field, bibliographic snowballing, and hand searches of key journals back through 1980, such as: African Development Review Agricultural Economics American Economic Journal: Applied Economics American Economic Review American Economic Review: Papers and Proceedings American Journal of Agricultural Economics Development in Practice Economic Development and Cultural Change Econometrica Economics and Politics Journal of African Economies Journal of Agrarian Change Journal of Development Effectiveness Journal of Development Studies Journal of Development Economics Journal of International Development Journal of Political Economy Journal of Public Economics Land Economics NBER Working Papers Proceedings of the National Academy of Sciences Oxford Economic Papers Quarterly Journal of Economics Review of Economics and Statistics World Bank Research Observer World Bank Economic Review World Development
3.3 DATA COLLECTION AND ANALYSIS
3.3.1 Data extraction and management
For studies meeting the inclusion criteria laid out in section 1.4 of this protocol, the following data will be extracted (adapted from Waddington, et al 2010). We appreciate that some quantified versions of the moderators are quite coarse, but we will draw out more refined interpretations of the relevance of the moderators in qualitative synthesis, in the manner that they are conceptualized in the theory of change.
3.3.2 Assessment of risk of bias in included studies
Quantitative effectiveness studies that meet our inclusion criteria will then be quality rated in terms of risk of bias in estimating impacts. Risk of bias will be assessed using the IDCG Risk of Bias Tool (March 2012 version). These quality rating methods are based on guidance from Higgins and Green (2011), Campbell Collaboration (2011), and Cochrane Effective Practice and Organization of Care Group (2009), suitably adapted to development interventions (e.g., accounting for the fact that blinding is nonsensical for interventions such as changes in land property rights). The quality ratings reflect the following domains: Potential for selection bias due to non-random assignment, non-exogenous source of quasi-experimental variation in assignment, no adjustment for differences in baseline measurements, Potential for spill-over, non-intervention based differences in treatment, or other types of interference across intervention and non-intervention units, Selective outcome and analysis reporting based on systematic differences between reported and unreported findings, Potential bias due to non-compliance, attrition, or otherwise missing data, And other sources of bias.
For each study, a value of either “high risk of bias”, “low risk”, or “unclear risk” will be assigned for each of these five domains and support for judgment will be recorded. A summary judgment of the within-study risk of bias will be carried out using the criteria provided in Higgins and Green (2011, Table 8.7.a). These results will be recorded in a table that will be presented as part of the review.
3.3.3 Assessment of statistical power of included studies
Ex post power to detect standardized effects of low magnitude (0.2 standardized mean difference), moderate magnitude (0.5 standardized mean difference), and high magnitude (0.8 standardized mean difference) will be determined based on the guidance of Campbell Collaboration (2011, 6).
3.3.4 Measures of treatment effect
Treatment effects will be measured as standardized mean differences (SMDs) for continuous outcomes and risk ratios (RRs) for binary outcomes, reported alongside 95% confidence intervals, computed as per the formulas provided by Campbell Collaboration (2011, Appendix 2). Where sufficient data do not exist to calculate SMDs, we will convert effects on continuous outcomes into response ratios, which use the same formulae as RRs. As per Campbell Collaboration (2011, 6-7) sub-group estimates will be aggregated using sample-weighted averages. (We will aggregate prior to meta-analysis only within each intervention-outcome group.) When multiple estimates are presented, the estimate that is deemed to have minimal risk of bias will be reported. Treatment effect estimates on comparable outcomes will be displayed together using forest plots.
3.3.5 Unit of analysis issues
When units of analysis are at a lower level of aggregation than assignment units, the review will follow the guidance of Campbell Collaboration (2011, 7-8) to adjust confidence intervals based on the intra-class correlation (ICC) adjustment to standard error estimates. Plausible ICC values will be derived from available development survey data.
3.3.6 Dealing with missing data and incomplete data
When studies are found to have missing data on outcomes due to attrition or other sources of non-response, an assessment of sensitivity of study conclusions to missing data will be made by (1) computing worst-case bounds on effect estimates and (2) providing discussion of possible directions of bias given the nature of the missing data. When studies do not report on endpoint or intermediate outcomes, the study authors will be contacted to determine whether such outcome data do in fact exist and so estimates could be produced. If so, we will seek to obtain effect estimates on these outcomes either from the study authors or using the raw study data to compute them directly.
3.4 DATA SYNTHESIS
3.4.1 Quantitative Synthesis
Synthesis of effect estimates from quantitative effectiveness studies
We anticipate considerable heterogeneity in the manner in which intermediate and endpoint outcomes are measured. Nonetheless, if comparable measures are used for impact estimates on any intermediate or endpoint outcome for a reasonable number of studies, we will carry out a synthesis that includes the following (Campbell Collaboration, 2011, 8-10; Rothman et al., 2008, 675-677): Assessment of overall heterogeneity using the I-squared statistic. Analysis of effects within subgroups by moderator variables as operationalized above, quality rating domain assessments defined above, and gender, when data are available to do so. We will also include a section dedicated specifically to gender relevant findings of our review. Provided adequate degrees of freedom, a more structured analysis of heterogeneity using random effects meta-regression on moderator variables and quality rating domain assessments. Moderators will be deemed to significantly account for effect heterogeneity if the meta-regression yields statistically significant coefficients at the 90% level or above. We will also interpret the substantive importance of any such associations by translating the estimated coefficient onto a natural metric for the outcome. The estimate of the conditional between-study variance (tau-squared) will be reported in the table with the estimates of the moderator coefficients. Assessment of publication bias via a funnel plot and funnel plot regression (Egger et al., 1997).
The analysis of comparable measures will be carried out using the measures of treatment effect discussed above. The analysis will employ the suite of meta-analysis and random effects meta-regression functions in Stata Version 11. Such analyses will be restricted to sub-groups when necessary to ensure reasonable amounts of comparability.
3.4.1.1 Qualitative Synthesis
As indicated in section 3.1.5.2, aside from the analysis of effectiveness based upon evidence presented in quantitative impact evaluation studies, the qualitative portion of the review will use empirically-based non-impact evaluation studies to: assess factors contributing to the success or failure of interventions; identify how and why intended or unintended outcomes occur; understand the context in which interventions are carried out; and understand views of beneficiaries.
Once appropriate studies have been identified using the two stage CASP style criteria outlined in 3.1.5.2 and found in Appendices IIIa and IIIb, we propose to use the qualitative metasummary methodology pioneered by Sandelowski and Barroso (cf. Sandelowski et al. 2007; Voils 2008) to analyze their results. This methodology has been termed an “aggregative” approach in that it focuses broadly on quantitatively identifying the frequency of qualitative results found in the research, and is not used to synthesize concepts or create lines of argumentation (Voils et al. 2008).
Metasummaries involve a five stage process to process and evaluate findings: extraction of findings from the research; grouping them into categories; abstracting diverse findings into ‘themes’ with a comparable and coherent a format; establishing the frequency and intensity of findings; and presenting and interpreting results. While extracting findings, care will be given to ensure that these are separated from: data presented as evidence in the research; conclusions of other work used to support findings; methods used to arrive at findings; and elaborations on the relevance of findings. Creating a matrix of findings grouped by topic and similarity to one another will enable us to better compare results among disparate studies and elucidate possible trends or relationships. Carefully abstracting findings improves comparability by removing unnecessary context and detail while preserving their complexity helps reveal overarching trends and other important insights. Calculating frequency and intensity of findings helps to respectively understand the relative magnitude of findings and which studies contributed most or least to our overall sample of findings. The presentation of findings include a standalone analysis of the findings as well as a discussion of these findings together with those of the quantitative portion of the review (Sandelowski et al. 2007; Voils et al 2008). We will also include a section on gender within the qualitative result.
4 Timeline
Searches for studies: May - June 2012.
Assessment of relevance of studies: June - July 2012.
Extraction of data: July 2012.
Quantitative and qualitative synthesis: August 2012.
Preparation of draft report: September 2012.
Dissemination: October 2012.
Revision of draft report: November 2012.
5 Contribution of authors
6 Declarations of interest
The individuals involved in this study have no vested interests in the findings of this review, nor do they have incentives to misrepresent or bias findings. As such, there are no conflicts of interest exist that the authors are aware of.
7 Sources of support
This study is funded by the UK's Department for International Development (DFID) and the International Initiative for Impact Evaluation (3ie). Methodological assistance was provided by the Campbell Collaboration International Development Group.
Footnotes
9 Appendices
1
Research has shown the adaptability of customary tenure to changing demographic patterns, including large-scale rural to urban migration by men, by accommodating new rules permitting the inheritance of customary land rights by widows (
). The availability of evidence on inheritance reforms suggests that this topic should be devoted to a systematic review in itself.
