PROTOCOL: The effects of agricultural output market access interventions on agricultural,socio-economic and food and nutrition security outcomes in low- and middle-income countries: A systematic review

Farm to market transport infrastructure

Transporting produce to markets can be costly if farmers must rely on slow, poor quality transport networks and if trade, therefore, involves the risk of the loss of product quality and spoilage. Chamberlin and Jayne (2013) show rural marketing costs in many LMICs are often dominated by transportation costs and de Brauw and Bulte (2021) explain that such transaction costs reduce the benefits and incentives to trade. Farm to market transport infrastructure interventions include initiatives that improve infrastructure, such as road and bridges, used for delivering agricultural produce to domestic and international markets. This includes initiatives constructing, rehabilitating, or maintaining infrastructure. They are intended to reduce the risks and costs associated with transporting goods and produce. This may make market opportunities more lucrative and improve access to new or distant markets (Aggarwal et al., 2022; Ludwig et al., 2016).

Output market information

Output market information interventions include two different types of information interventions. One class of interventions include initiatives that directly provide farmers with information about the market, usually about prices, completed trades, or current market demand (including information on location, quality, variety, and quantity demanded). This can help inform farmers about markets where they might sell their produce, as well as help them to decide what to grow and how much to expect to sell it for. A second class of interventions increase access to technologies required to access market information. This includes investments in mobile phone services, internet or broadband access, and the basic provision of electricity.

Since Stigler's seminal work on the ‘Economics of Information’ (Stigler, 1961), a large body of literature has shown asymmetric information can affect market equilibria, creating an inefficient allocation of goods across markets, increasing price dispersion and instability, and decreasing trade and competition (Aker, 2008; Goyal, 2010). For example, one way information asymmetries can distort agriculture markets is that more informed traders may use their knowledge to exploit farmers and pay lower farm gate prices. Farmers who receive lower prices for their produce may be less inclined to trade and may limit their market operations accordingly (Magesa et al., 2020; Nakasone, 2013).

New marketplaces and alternative marketing opportunities

Interventions creating new marketplaces and alternative marketing opportunities aim to provide new ways for market participants to trade and/or reduce search costs by simplifying or improving connections between market participants. It is expected that these types of interventions will increase market competition, improve the prices farmers receive for their produce and, to the extent this creates market incentives, increase production and possibly alter the production of the types of crops grown (e.g., increasing the adoption of cash crops) (Goyal, 2010; Levi et al., 2020). Examples of this type of intervention include creating new marketplace by connecting geographically distributed agri-markets through online platforms (e.g., see Levi et al., 2020) or mobile trading platforms (e.g., see Bergquist & McIntosh, 2021). Other examples of alternative marketing outlets include setting up trading hubs or internet kiosks (e.g., Goyal, 2010) or curating commitments or stable arrangements where traders offer to buy farmers’ produce (e.g., Bold et al., 2022; Maertens et al., 2020).

Contract farming

Contract farming and outgrower schemes create arrangements where traders and farmers enter forward-looking agreements for the delivery of agricultural produce (Eaton & Shepherd, 2001).

Storage infrastructure

Improved storage infrastructure interventions provide access to storage facilities, such as sheds with off-the-ground storage, and storage deposit systems. These interventions intend to reduce the risks associated with the storage of agricultural produce, which may help to increase the cultivation of alternative high-yielding crops that do not otherwise store well. This can increase access to new or alternative markets (Kantunze et al., 2017; Omotilewa et al., 2018) and it may also improve market access by enabling farmers to perform a more extensive market search for buyers or enabling them to wait for more favourable market conditions (i.e., perform intertemporal arbitrage) (Lothoré & Delmas, 2009; Pingali et al., 2019). However, an increasing degree of controversy now surrounds storage interventions. For example, Cardell and Michelson (2020) show that the price for maize often declines after harvest (∼30% of the time), so storing after harvest may also be detrimental to farmers outcomes (especially when programs are combined with credit facilities).

Types of studies

We will include studies using experimental and quasi-experimental study designs to measure a change in outcomes that is attributable to an intervention. This is accomplished by identifying a counterfactual, which provides evidence about what would have happened in the absence of the intervention. The counterfactual may be inferred from a specific group of observations that do not receive the intervention (as in a randomised controlled trial [RCT]), or receive an alternative intervention, or may be constructed by researchers (as in propensity score matching or interrupted time series). This means studies may apply a wide range of potential designs, such as randomised controlled trials, regression discontinuity designs, instrumental variables, fixed effects regressions, interrupted time series models, matching methods, and the synthetic control method (see Supporting Information: Appendix 1 for a comprehensive list of included study designs). These study designs are widely recognised as being capable of establishing causal relationships between interventions and outcomes when carefully and credibly executed (Aloe et al., 2017; Fenton Villar & Waddington, 2019; Gertler et al., 2011; Reeves et al., 2017; Waddington et al., 2023).

We will not exclude studies based on the comparison condition of a control group. A study's control group may consist of observations subject to no intervention, on a wait-list, or a member of an alternative intervention or condition. However, we will exclude studies that only use simulation or forecast models, ex-ante impact assessments or scenario analyses, as well as evaluations and case studies that do not satisfy the methodological conditions described above. We will also exclude efficacy studies, feasibility studies, acceptability studies, and literature reviews and systematic reviews will not be included as primary studies.

The nature of many of the interventions we intend to study, particularly large infrastructure interventions and widespread information initiatives, may create spillover effects and market equilibrium outcomes. For example, Hildebrandt et al. (2020) provide evidence of spillovers from a price alert system in Ghana. We will document where studies present both partial and general equilibrium effects of interventions (e.g., Yanagizawa-Drott & Svensson, 2012) or control for this issue via a study design (e.g., Bergquist & McIntosh, 2021).

Types of participants

We will include studies of the effects of interventions consisting of participants residing in LMICs. We will use the World Bank income status classification for defining LMICs (see Supporting Information: Appendix 2) and studies will be classified according to their status in the year the intervention began.

Types of interventions

We will include interventions that aim to strengthen farmers’ output market access using one or more of the five types of interventions we describe above. We will consider both farmers with already access to markets (commercialised farmers) and subsistence farmers. This list of included interventions is not an exhaustive list of potentially relevant output market access interventions but provides a selection of key strategies. The selection of interventions included is informed by funder interests and expert advice.

As also noted in our discussion above, there are variations in the way interventions may appear in practice. Table 1 characterises further the different ways these intervention types may be sub-categorised, and our analysis is designed to consider variation within intervention types, as well as between them, where there is sufficient evidence to do so. However, at this point, we do not intend to be too prescriptive about the sub-categories of interventions included or those we will group together for the analysis. Practitioners are often highly industrious in finding innovative ways to address problems. We are adopting an iterative approach, where we will update these sub-categories of interventions again once we have identified the full population of studies included in the review. The sub-categories of interventions we present in Table 1 provide a starting point for this exercise based on expert advice and knowledge of existing studies from our initial searches to develop our search strategy (discussed below). We will perform a further mapping exercise analysing interventions for any other notable affinities that can be gleaned from the available information about the interventions, as well as the differences between them.

Interventions may include activities addressing these issues in isolation or in combination with activities. For example, a farming contract may include both a fixed price agreement and financing for inputs or, alternatively, an intervention may include multiple components consisting of both the provision of market information and a market-making activity using information technology. They may also include another intervention that is not in itself included in this review, such as agriculture extension services with a market access intervention (as in Bold et al., 2022). Our review will endeavour to record the nuances of intervention arrangements in its description and analysis.

We will include interventions targeting farmers of seasonal (such as grains, pulses, or root and tuber crops, etc.) and permanent crops (such as cocoa, coffee, tea, etc.). This includes horticulture products, and we will also include interventions concerning livestock and animal husbandry, which is the branch of agriculture concerned with animals that are raised for meat, milk, skins, hides, and so forth. However, to limit the overall breadth of the scope of the review, we will not include studies on market access interventions from related sectors (i.e., the fishery or forestry sectors). We add this clarification because some organisations, such as the Food and Agriculture Organization of the United Nations (FAO) and USAID, aggregate these economic activities into a single agriculture sector.

With respect to interventions creating alternative marketing opportunities or new markets, we will exclude public marketing (or commodity) boards, which are set up by a government to regulate the buying and selling of a certain commodity within an area (e.g., see Fung et al., 2020). We will also exclude minimum support price policies (MSPs), which are also recognised as a type of output subsidy and are regularly used by governments to safeguard farmers’ income against crop price falls and ensure a sufficient and balanced production of crops. Furthermore, we will exclude studies examining the effects of establishing traditional agricultural commodity exchanges or programmes promoting post-harvest practices and technologies, such as drying tarps (e.g., see Magnan et al., 2021). We will also exclude studies that concern certification only, such as UTZ Certified, Fair Trade, and Rainforest Alliance, unless a non-transferable fixed-term forward sales contract with a specific firm is offered. Sharecropping arrangements where a tenant farmer is provided with inputs in exchange for an agreed share of the harvest are also excluded.

Types of outcome measures

We will include studies that contain data on outcomes related to the conceptual framework above. This includes immediate outcomes (such as transactions costs, aggregation, use/adoption of improved inputs, technologies and practices, farm investment, access to credit and prices, crop losses), agricultural production outcomes (including measure of yields, volume, quality, produce type), intermediate outcomes (farm sales and income), non- and off-farm income and labour outcomes, and welfare outcomes (total household income and food security).

Table 2 provides further details of each of the outcome types included. This may not be an entirely exhaustive list of potential outcomes for these interventions. There are many channels through which these interventions may operate and some outcomes, such as farmers’ market risk, are also difficult to define or measure in the context of an impact evaluation. Informed by expert advice, this review focuses on a selection of key outcomes which provides good coverage of a broad range of relevant outcomes related to the interventions’ theory of change and the primary welfare outcomes of interest. All outcomes here are relevant for the five interventions types considered in this review. However, we expect the prevalence of outcomes featured in the primary research evaluations to dictate the final outcomes included for each intervention within our framework. We will reflect on the prevalence of these particular outcomes in the systematic review for each intervention type.

Interventions investing in societal infrastructure (such as roads and bridges concerning farm to market interventions or internet and electricity infrastructure under information interventions) may also have several possible channels explaining their effects on welfare outcomes. For example, other possible channels beyond the hypothesised effect on the agricultural sector may include improved access to health care, education, other labour markets, and so forth. Reflecting the theme of the review, since in this instance we are interested in understanding their effects through the agricultural development channel, we will include welfare, non- and off-farm income and labour market outcomes, but only from studies that also report agricultural production outcomes, intermediate outcomes, or off-farm (agriculture only) outcomes. In our analysis, we intend to examine the statistical correspondence between the estimated effects on agricultural outcomes and these other outcomes (e.g., confirming that evidence indicates infrastructure interventions improve both agricultural outcomes and welfare outcomes).

Other inclusion and exclusion criteria

Language: Studies published in any language will be included, although the search terms used will be in English only.

Publication date: We will include studies published in 2000 or later. This restriction is necessary to keep the review's search feasible within project resources. Furthermore, analyses indicate that although there were IEs published in international development before 2000, the number is relatively small, especially in the agriculture sector (see Sabet & Brown, 2018).

Electronic searches

To identify relevant studies for our review, we have developed a set of English search terms and a search strategy in collaboration with an information specialist. Our search terms combine Boolean terms with a list of keywords related to the review's inclusion criteria (see Supporting Information: Appendix 3). We will use and adapt these terms to search electronic databases and institutional websites with sufficient search functionality.

As suggested by Sandieson (2006), we have utilised pearl-harvesting techniques to develop our list of key search terms. This involves reviewing and identifying keywords from our inclusion criteria and known studies relevant to the review (see Supporting Information: Appendix 4 for further details). We identified an initial sample of relevant studies by consulting with members of the project's expert advisory group and searching the results of a selection of existing reviews (see Supporting Information: Appendix 5). We also performed targeted searches of 3ie's Development Evidence Portal (DEP), which is currently the world's most expansive repository of rigorous evidence on what works in international development. We used the DEP's intervention search filters, which index studies using a standardised vocabulary for classifying interventions (see Kozakiewicz et al., 2021), to help to identify a sample of studies listed in the portal that are most relevant to our review. Supporting Information: Appendix 6 provides a list of the initial list of studies identified for peal-harvesting and developing our search strategy.

We have also compiled a list of databases and websites we will search for relevant evaluations and studies (see Table 3). To reduce the risk of publication bias, these sources have been selected to cover a range of publication types, including journal articles, working and discussion papers, conference proceedings, thesis and dissertations, and institutional reports. We have identified relevant sources by consulting an information specialist, the project's expert advisory group, and systematic reviews on included or related interventions recording databases including impact evaluation evidence (including Hine et al., 2016, 2019; Ludwig et al., 2016; Oya et al., 2017, 2018; Ton et al., 2017; Waddington et al., 2014).

While some websites and databases have reasonably well-developed search functions, some do not support complex search strings or allow for the direct export of materials, and others must be browsed by keywords or even browsed in their entirety. We will customise our general search strategy according to the functionality of each database and website we search (using the website's thesaurus or keyword index if necessary to identify the most appropriate vocabulary). We will consult an information specialist who will help to troubleshoot problematic sources, as well as advise on the best ways of conducting targeted searches. We will document the literature search process and any necessary changes to the search strategy for each source.

Citation tracking

For the studies included in the review, we will also perform backward and forward citation tracking (Greenhalgh & Peacock, 2005). Backward citation tracking consists of screening the reference lists or bibliography of included studies for other eligible studies cited in the text. Here we will perform hand searches and manually record the number of studies screened. Forward citation tracking involves searching for eligible studies that cite the originally included study. For forward citation tracking, we will utilise Google Scholar.

Searching other resources

We will supplement these searches by contacting key researchers and organisations working on issues related to this review and we will engage our expert advisory group for suggestions concerning other relevant studies. Finally, we will search the included studies of other related evidence maps and reviews. Supporting Information: Appendix 5 provides a provisional list of relevant maps and reviews.

Selection of studies

After collating and de-duplicating records from our literature search, we will perform a two-stage selection process where trained reviewers will assess studies against the review's inclusion and exclusion criteria.

Data extraction and management

We will extract the following descriptive, methodological, quantitative, and cost and other qualitative data from each study included in the review using standardised data extraction forms (provisional forms are provided in Supporting Information: Appendix 7):

Descriptive data including authors, publication date and status, as well as other information to characterise the study including country, type of intervention and outcome, and intervention design.

Descriptive data, methodological information and cost data will be single coded by a trained reviewer and checked for agreement by another one. Two trained reviewers will independently code the quantitative data and any disagreement will be resolved through discussion with a third reviewer (who must be a core team member).

Assessment of risk of bias in included studies

We will assess the risk of bias in included studies using 3ie's risk of bias tool (see Supporting Information: Appendix 10). This examines both the internal validity and statistical conclusion validity of experimental and quasi-experimental impact evaluation designs (see Waddington et al., 2012). Two reviewers will undertake the risk of bias assessment independently. If there are disagreements, we will resolve them by discussion and the involvement of a third reviewer (who must be a member of the core team). We will compile a risk of bias assessment for each estimate we extract. This reflects that reflecting estimates on different outcomes in the same study may score differently in the assessment.

We will assess the risk of bias based on the following criteria, coding each estimate as ‘Yes’, ‘Probably Yes’, ‘Probably No’, ‘No’ and ‘No Information’ for each domain:

Factors relating to baseline confounding and biases arising from differential selection into and out of the study (e.g., assignment mechanism).

We will report the results of the assessment for each of the assessed criteria for each estimate. In addition, we will use the results of the risk of bias assessments to produce an overall rating for each study as either ‘High risk of bias’, ‘Some concerns’ or ‘Low risk of bias’, drawing on the decision rules in RoB2.0 (Sterne et al., 2019), rating studies as follows:

‘High risk of bias’: if any of the bias domains were assessed as ‘No’ or ‘Probably No’.

We will provide a description in our analysis of the outcomes of our assessment of reliability of included studies, and we also intend to explore whether there are systematic differences in estimated effects between primary studies with different risk of bias. We will conduct sensitivity analysis to assess the robustness of the results to the risk of bias associated with included studies (discussed below).

Measures of treatment effect

In this review, we intend to examine the effects of output market access interventions. An effect size (or treatment effect) expresses the direction and magnitude of the difference in outcomes between groups of observations, such as the difference in outcomes between observations in the intervention and comparison groups (Borenstein et al., 2009a; Valentine et al., 2015).

However, effect sizes presented in empirical studies are rarely independent of the scale or unit of the outcome in the study and the scale or unit of the outcome is rarely directly comparable across studies. For these reasons, to facilitate cross-study comparisons of the magnitudes of studies effects in our analysis, we will extract data from each study to calculate standardised effects sizes. We will choose the appropriate formulae for standardised effect size calculations in reference to, and dependent upon, the data provided in the included studies and the outcome type (see Supporting Information: Appendix 8 for details of the effect size formulae sheet).

If different outcome types exist under the same outcome construct (e.g., binary measure of employment and a measure of the number of hours worked off-farm), for comparability of estimated effect sizes, we will we convert estimates to the most common standardised metric. We will use common transformations outlined in Borenstein et al. (2009a) for converting between different measures of standardised effects.

Criteria for determination of independent findings

It is important our analysis accurately captures and reflects on co-dependencies between study estimates. This is because standard meta-analytic methods assume effect size estimates are independent and failure to qualitatively recognise estimates are derived from the same intervention or study can distort (inflate) our perceptions of the availability of evidence.

Dependent effect sizes can arise in several circumstances. For example, co-dependencies between estimates can arise when several publications stem from one study, or several studies are based on the same data set. Some studies might have multiple treatment arms that are all compared to a single control group. Other studies may report outcome measurements from several time points or use multiple outcome measures to assess related outcome constructs. All such cases yield a set of statistically dependent effect size estimates (Borenstein et al., 2009b).

We will assess the extent to which relationships exist across the studies included in the review. We will avoid double counting of identical evidence by linking papers before data analysis. We will utilise information provided in the studies included to help support these assessments, such as sample sizes, programme characteristics and key implementing and/or funding partners. Where we have several publications reporting on the exact same effect, one main study will be used for data extraction and the linked studies will be stored to help any required search for further or missing information. To identify the main study, priority will be given to journal articles and, in the case of multiple reports/working papers, the most recent one will be selected.

We will extract effects reported across different interventions, outcomes and subgroups within a study. We will address dependent effect sizes using data processing and selection techniques. We will utilise several criteria to select one effect estimate per outcome per study (further details of the criteria determining effect estimate selection are available in Supporting Information: Appendix 9).

Unit of analysis issues

Unit of analysis errors can arise when the unit of allocation of a treatment is different to the unit of analysis of effect size estimate, and this is not accounted for in the analysis (e.g., by clustering SEs at the level of allocation). We will assess included studies for the prevalence of these issues and, where they exist, account for them by adjusting the reported SEs according to the following formula (Hedges, 2009; Higgins et al., 2020):

( d ) ′ = ( d ) · 1 + ( m − 1 ) c , ${(d)}^{^{\prime} }=(d)\,\cdot \,1+(m-1)c,$

Dealing with missing data

In instances where there is missing or incomplete data, we will make every effort to contact study authors to obtain the required information. If we are unable to obtain the necessary data, we will report the characteristics of the study but state that it could not be included in the meta-analysis or reporting of effect sizes due to missing data. In line with recommendations on collating data in systematic reviews from study authors (see Mullan et al., 2009), we will report the number of studies for which authors were contacted, the information requested, any important details of the method of eliciting information, and the response of authors to the request. When pertinent, we will also report the impact that information obtained from authors has on the results (i.e., using sensitivity analyses discussed below).

Data synthesis

To synthesise the effects of market access interventions, we will combine a narrative synthesis of study findings with a meta-analyses of interventions effects.

We will include studies in the same meta-analysis when we identify two or more effect sizes using a similar outcome construct, the same intervention type, and where the type of comparison group is judged to be similar across the studies. This is similar to the approach taken by Wilson et al. (2011). Where there are too few studies, or the included studies are considered too heterogeneous in terms of interventions or outcomes, we will present a narrative discussion of individual effect sizes alone.

Because heterogeneity exists in theory due to the variety of interventions and contexts that could be included in the review, we will use inverse-variance weighted, random effects meta-analytic models (Higgins et al., 2020). We will use the metafor package (Viechtbauer, 2010) in R software to conduct the meta-analyses (R Core Team, 2020).

The narrative synthesis of study findings will provide information on the relevant information on the associated factors facilitating and moderating the effects. This information will be synthesised to possibly draw conclusions beyond the study level and design learnings on the necessary conditions for positive effects to occur. This analysis will closely follow the meta-analysis results and interpretations and the subgroup and heterogeneity analysis described in Section 3.3-10.

We will also examine the statistical correspondence between welfare outcomes and other more proximate outcomes in the conceptual model. For example, examining the correspondence in effects between immediate and agricultural outcomes and welfare outcomes. We will do this through descriptive analysis and, where possible, regression analysis and fuzzy Qualitative Comparative Analysis (QCA) (Thomas et al., 2014; Ton et al., 2017). We will also document the results and correspondence of estimated effects for studies that provide insights through general equilibrium analysis, as well as partial effects.

Assessment of reporting biases

If meta-analysis is feasible, we will assess reporting biases in the literature using a rank correlation test (see Begg & Mazumdar, 1994). We will also use a regression test (Sterne & Egger, 2005), using the standard error of the observed outcomes as predictor, to check for funnel plot asymmetry.

Subgroup analysis and investigation of heterogeneity

In our analysis, we intend to examine and discuss the distribution of estimated effects across intervention and outcome types. We will also statistically assess heterogeneity by calculating the Q statistic, I ², and τ ² to provide an estimate of the amount of variability in the distribution of study effects sizes (Borenstein et al., 2009a). We will complement this assessment with a graphical analysis using forest plots and, whenever feasible, we will conduct moderator analyses using meta-regression analysis to investigate sources of heterogeneity.

Following the PROGRESS-PLUS approach (Oliver et al., 2017), we will assess moderators falling into three broad categories of extrinsic, methodological and substantive characteristics. Examples of these categories include:

Extrinsic characteristics: funder of the study (e.g., NGO vs. private sector vs. government investments), publication type, publication date.

We will use random effects meta-regression to investigate the association between moderator variables and heterogeneity of treatment effects (Borenstein et al., 2009a), and subgroup analyses to investigate heterogeneity by treatment subgroups (e.g., men and women, poor and non-poor, and so on). If these strategies are not possible (e.g., if we do not have sufficient number of studies or data), we will discuss and explore the factors which may be driving the heterogeneity of results narratively by conducting cross-case comparisons (Miles & Huberman, 1994).

Analysis of intervention costs

We will collate and analyse any cost data reported in the set of studies included in our review or from any documents they reference. Following Shemilt and Mugford (2009), relevant studies will include full economic evaluations (e.g., cost–benefit, cost-effectiveness or cost-utility analyses), partial economic evaluations (e.g., cost analyses, cost-comparison studies, cost-outcome descriptions), or any other documentation reporting the costs associated with included interventions. Cost information will be tabulated and synthesised narratively with descriptions of details of the different approaches used to derive intervention costs also provided. Estimates show only around 15% of impact evaluations report information about intervention costs (see Brown & Tanner, 2019). Reflecting that it is likely that information on intervention costs will be limited, where possible, we will contact study authors and funders of interventions to try to retrieve further information. We will also collate any additional cost-effectiveness or other types of economic evaluations identified from our literature search.

Sensitivity analysis

We will conduct sensitivity analysis to assess whether the results of the meta-analysis are sensitive to the removal of any single study. We will do this by removing studies from the meta-analysis one-by one and assessing changes in results. We will also assess the sensitivity of our results to the inclusion of studies with a high risk of bias studies by removing these studies from the meta-analysis and comparing results to the main meta-analysis results. We will examine the sensitivity of results to the inclusion of specific outcome measures (e.g., limiting meta-analysis to the preferred measure of an outcome). We will also assess the sensitivity of our result to the inclusion of data obtained directly from study authors (as discussed above).

Furthermore, we will assess the sensitivity of our results to outliers. We will use studentised residuals to examine whether studies estimated effects may be outliers (Viechtbauer & Cheung, 2010) and studies with a studentised residual larger than the

100 × ( 1 − 0.05 / ( 2 × k ) ) $100\times (1-0.05/(2\times k))$

It is possible that we may also use both bivariate and multivariate (or partial) effects for calculating standardised effect sizes. A partial effect size is based on a regression coefficient measuring the treatment effect ‘holding all other variables constant’ and is, therefore, measuring a different quantity to a bivariate relationship. Our standardised effect sizes are only strictly comparable in studies using a common model (Keef & Roberts, 2004). However, only using bivariate effect sizes to calculate standardised effects would not be suitable in this context due to the likely high risk of bias this may cause quasi-experimental study designs that control for selection bias (Waddington et al., 2014). We will use sensitivity analysis to examine systematic differences in partial and bivariate effects (omitting them from the analysis or controlling for these characteristics in a meta-regression).

Finally, not all multivariate models control for the same covariates and nor should models estimated for different study designs using data collected in different contexts necessarily do so. The risk of bias assessment evaluates likely specification errors and the sensitivity analysis omitting high risk studies (discussed above) should capture most of these issues. Otherwise, we assume the possible resulting multicollinearity issues are inconsequential (see Waddington et al., 2014).

Analysis of intervention costs

We will collate and analyse any cost data reported in the set of studies included in our review or from any documents they reference. Following Shemilt and Mugford (2009), relevant studies will include full economic evaluations (e.g., cost–benefit, cost-effectiveness or cost-utility analyses), partial economic evaluations (e.g., cost analyses, cost-comparison studies, cost-outcome descriptions), or any other documentation reporting the costs associated with included interventions. Cost information will be tabulated and synthesised narratively with descriptions of details of the different approaches used to derive intervention costs also provided. Estimates show only around 15% of impact evaluations report information about intervention costs (see Brown & Tanner, 2019). Reflecting that it is likely that information on intervention costs will be limited, where possible, we will contact study authors and funders of interventions to try to retrieve further information. We will also collate any additional cost-effectiveness or other types of economic evaluations identified from our literature search.