Protocol for a Systematic Review: The Impacts of Business Support Services for Small and Medium Enterprises on Firm Performance in Low-and Middle-Income Countries: A Systematic Review

Participants

This review will focus only on studies that evaluate policies aimed at supporting SMEs in low-and middle-income countries (as defined by the World Bank's classification), with an emphasis on African countries wherever possible. The focus on LMICs is justified firstly because private firms in these countries tend to be more labour intensive and less innovative, and consequently are the main employer of a large proportion of the labour force. Secondly, restricting the scope to LMICs helps to identify the binding constraints that SMEs might face in similar institutional contexts, such as in some African countries. The term SME covers a wide range of definitions and measures, varying from country to country and between the sources reporting SME statistics. Some of the commonly used criteria are the number of employees, total net assets, sales and investment level (Ayyagari et al., 2007). The most common criterion used to classify SMEs is based on employment information, due to data availability, and the cut-off used to define SMEs is usually 250 employees4. This review will use this cut-off of 250 employees. Consequently, other types of interventions aimed only at supporting entrepreneurship and the creation of microenterprises, such as microfinance ⁵ , will not be part of this research. This is because self-employed and micro-entrepreneurs have a different nature in comparison to SMEs ⁶ . The former, especially in LMICs, are comprised of less productive or informal enterprises of few employees in the fringe of the markets. Furthermore, these enterprises are not eligible to most of the public interventions to be covered in this review. Thus, the definition of SME based on number of employees fits well our purpose of covering a broad set of interventions and of considering relevance for African countries ⁷ . Since our prior assumption is that there will be only a few studies examining public interventions in African countries, a proper contextualisation of the interventions, a comprehensive understanding of the designs, the target groups, and the moderator variables ranging from those related to firms themselves (size, sector, number of years in operation) to those related to the country where the intervention take place will be crucial to this review. This will allow us to be able to shed some light on whether the intervention has some external validity and consequently whether it could potentially work in an African context. ⁸

In order to address the likely problem of limited evidence, particularly of relevance to Africa, the scope of the review will include all studies identifying final and intermediate outcomes. This will also better inform the causal chain analysis which will help inform our tentative findings about generalisability to African countries.

In the studies selected, we will then search for any information on how and why interventions worked or did not work. The literature recommends that synthesis is informed by the theory of change embedded in the design of an intervention (see Waddington et al., 2012b). However, our focus is not only on the impacts directly anticipated by the intervention but also included unanticipated impacts.

Interventions

We will include the following interventions:

Formalisation/ Business Environment (Institutional Improvement): such as tax simplification, intended to provide incentives for informal SMEs to formalise. Underlying assumption: that formal firms are less credit-constrained than their informal counterparts and therefore formalisation would be an effective way to help entrepreneurs. Indirect support to SMEs may include policies regarding business registration, property registration and regulatory frameworks (Fajnzylber et al., 2011; Monteiro and Assunção, 2012; McKenzie, 2013).

Exports/Access to External Markets: defined as interventions that correct market failures such as information externalities and help SMEs overcome obstacles to exporting (Volpe and Carballo, 2010; Volpe et al., 2010; World Bank, 2010).

Support for innovation policies is based on the idea that social returns to innovation exceed private returns (Lundvall and Borras, 2005; Acs and Audretsch, 1988). Interventions designed to support innovation vary. This review will consider different types of innovation support such as matching grants, subsidies and tax incentives, as identified in the preliminary search. For instance, Chudnovsky et al. (2006) evaluate the case of matching grants provided after an open call for proposals. Alvarenga et al. (2012) analyse the impact of innovation-oriented matching grants and subsidised credit for innovation, and Czarnitzki et al. (2011) evaluate the effect of tax credits on innovation. Other forms of innovation support may also be identified during the search process.

Value Chain, Networks and Cluster interventions: defined as interventions that help individual firms benefit from agglomeration externalities and overcome the coordination failures that prevent SMEs from capturing these externalities (Schmitz 1995; Schmitz and Nadvi 1999; Giuliani et al., 2005).

Training and technical assistance: defined as interventions that provide support for employee training and technical assistance, based on the idea that skills improve employability and wages of workers and contribute to firm productivity (Attanasio et al., 2011; Rosholm et al., 2007). This type of intervention also includes consulting services and management practices such as those considered by the World Bank (2010), Bruhn et al. (2013) and Bloom et al. (2013).

SME Financing/Credit Guarantee: adverse selection and moral hazard in credit markets generate financial constraints, which in turn restrain SME activities (Beck and Demirguc-Kunt, 2006; Michelacci and Silva, 2007; Canton et al., 2012). The review will consider in this line of support, interventions that provide loans or insurance services to SMEs, such as those noted in World Bank (2010) for credit and in Oh et al. (2009) for credit guarantee schemes.

We note that sub-components within the business support interventions this review analyses may overlap. In this case, it will be important to categorise them as accurately as possible. If there is sufficient data, a sensitivity analysis will be conducted using detailed information on intervention sub-components, however, this may not always be possible if only a small number of examples are present.

Comparisons

To the best of our knowledge, most of the papers investigating the impact of a public policy targeted to SMEs compare a treated (or eligible) group with a control group (or comparison group in the case of quasi-experimental design). However, we will be distinguishing studies that compare treatment and control (or comparison) groups from studies that have more than two treatment arms. Besides, we will also separate the evidence according to the intervention design. In the case of RCTs, for instance, an intervention can use a phase-in design, an encouragement design, cluster (or block) randomisation, or pure randomisation (see Duflo et al., 2008). Different designs have two implications: (1) they almost always identify different parameters (ITT, ATT, LATE and so on); and (2) they almost always differ in terms of data collected (different take-up rates, different attrition rate, different risk of contamination bias and so on).

Outcomes of interest

The selected studies must report on at least one impact to do with firm-related outcomes, either intermediary or final. For the purposes of this review, we will define firm performance impacts to refer to objective indicators such as revenues, profits, job creation, innovation, formalisation, number of workers trained, and access to credit. Only factual/objective measures of firm performance impacts will be included: subjective measures on beliefs and perceptions will be excluded.

Study types: designs of interest to this review

The review will draw on a broad search to identify studies that relate to the interventions aimed at SMEs in LMICs.

To address questions i) to iii), the review will focus on quantitative analysis and include only studies that use experimental (randomised controlled trials) and quasi-experimental methods, such as regression discontinuity design (RDD), instrumental variables, difference-in-differences, matching on covariates, propensity score matching and any other methods that purport to control for selection bias (for example, Heckman two-step estimator) ⁹ . Studies selected must have controls for the endogeneity of program placement or self-selection into the program. Experimental and quasi-experimental methods are widely seen as the best tools when the main objective is to estimate the causal impact of an intervention or policy (see for example Duflo et al., 2008). When an intervention is carefully designed or the identification strategy of an observational study convincing enough, the findings on the impact of the program or intervention are said to have internal validity, that is, one can claim that the difference in the outcomes between treatment and control groups was caused by the intervention ¹⁰ .

This review will thus consider only studies that assess the impact of an intervention comparing the treatment (or eligible) and the control (or comparison) groups at one or more points in time. In cases where more than two treatment phases are considered, the estimates can also involve comparison of the two treatments ¹¹ . The studies considered will therefore be drawn from cross-sectional and panel data datasets. Quasi-experimental studies that rely on observation data must show balance tests or use a matching method to control for imbalances in observed characteristics. Studies using matching methods, for instance, should clearly state the intervention rule of the program to be able to make the case that the problem of selection bias is (mostly) due to observed characteristics. Most importantly, the studies included will document the impact of any business support service on SMEs compared to practice as usual. In addition, the review will compare the impact of different types of business support service on firm performance.

As discussed in Waddington et al. (2012b), focusing exclusively on studies that use experimental and quasi-experimental methods may significantly restrict the studies that can be included in the review. Although this might be a legitimate concern, particularly if one is interested in comparing different interventions, we choose this trade-off because findings of studies that do not control for their selection biases are of little relevance, and possibly misleading, for decision makers.

To address questions iv) and v), we will draw on background program documentation or ‘sibling studies’ (Snilstveit, 2012) on the interventions in question. Such studies are deemed relevant if they meet the following criteria:

They relate to the interventions included in the effectiveness review.

Electronic searches

Search terms

Table 2 provides the list of basic search terms used to identify studies in the systematic review. Based on these terms, a detailed search strategy was set up to account for US and British English spelling, to seek for the most relevant studies and to restrict the search to low- and middle-income countries. The details of the search terms are provided in Appendix 1. The search strategy was developed using the Social Science Citation Index (ISI) and Econlit databases, two of the most important databases in economics. These codes will be adapted for other databases that allow the users to construct detailed codes for search terms. Nevertheless, unfortunately not all databases listed above allow the users to generate such a specific chain of commands to identify studies related to SMEs. When a database does not make it possible to construct complex search codes, an iterative search strategy using the search terms provided in the appendix and Table 2 will be used. All searches performed will be provided in the appendices of the systematic review.

Other searches

Along with database searches, the research assistants will carry out manual back searches in bibliographies of studies and journals identified as relevant to the topic ¹² . Given that the search focuses on low- and middle-income countries, we will also contact authors of studies and experts in the field for further recommendations on studies dealing with under-researched aspect of the interventions of interest. In addition, we will contact authors of selected studies to obtain fuller information on the interventions of interest. The review will cover studies published in English, Spanish and Portuguese. We will also include sibling studies (Snilstveit, 2012): qualitative studies including project documents, process evaluations and monitoring and evaluation documents associated with the projects evaluated in the impact evaluations included in the review. This will help to understand how a particular intervention works and achieves the results shown and help inform how applicable the effectiveness findings are to African contexts.

Selection of studies

The selection of studies will be done in two stages. First, two independent reviewers will select the studies that meet the inclusion criteria described previously. Abstracts and full texts will be used to decide whether a study is relevant by the inclusion criteria. Conflicts over whether or not a study qualifies for inclusion will be resolved through discussion within the team.

Data extraction and management

For all studies included in the two stages of the review, the researchers will extract information on different stages of the research process as well as of the program intervention. This will be done through an independent and consecutive process of reading studies, and will thus involve double-coding.

For each included study we will extract the following information:

Assessment of risk of bias in included studies

The Cochrane Collaboration tool will be used to assess the ‘risk of bias’ in RCTs and the IDCG risk of bias tool (Hombrados et al., 2012) will be applied to assess bias in quasi-experimental studies. Since some interventions that will be covered in this research are likely to have general equilibrium effects, estimates of spillover effects are critical.

Assessment of statistical power of included studies

The impact estimates will be assessed on two grounds: (1) effect size, and (2) statistical significance. Statistical significance is directly affected by study design and sample size. Study design can affect the precision (or efficiency) of the estimates if, for instance, the randomisation (or intervention) occurs at cluster level and the unit of analysis are firms or workers. If that is the case, the standard errors should be computed correcting for intra-cluster correlation. This correction might imply a loss of precision if units within the same cluster are relatively similar in terms of the outcomes of interest.

For a similar reason, small sample size results in higher sampling variance and therefore lower precision. In this case, an intervention that is based on a small sample size will have to have a big effect size to be statistically significant. When studies report effect sizes but are mute with regard to power analysis, we will perform an ex-post power calculation or simulate the sample size that would be required so that the estimated effect size had a power of at least 80 per cent ¹⁴ .

Measures of treatment effect

The comparison of the effectiveness of different interventions requires standardised effect sizes. Whenever pooled standard deviation is provided, we will calculate standardised mean differences (SMDs) for continuous variables and Odds Ratios (ORs) for binary outcomes ¹⁵ . In the absence of an effect, SMD equals 0 and the OR equals 1. Whatever is the case, standardised effect sizes as well as their 95 per cent confidence interval (CI) will be provided. The computation of the CI is important to inform how precise (and therefore reliable) the effect size is and consequently the results of a study. Standard effect sizes are also important if one is considering implementing a meta-analysis or a meta-regression (Duvendack et al., 2012). To deal with outliers we will apply the Winsorising technique that consists of censoring the distribution of the effect sizes by replacing, say, the top and bottom two percentiles with the highest and lowest values of the censored distribution. ¹⁶

Methods for handling dependent effect sizes

In cases where impact is estimated for subgroups (for example, small, medium, and large firms), it is recommended that only one effect size per outcome per study is used in the meta-analysis to avoid over-representation and misleading average effect size computation (Borenstein et al., 2009: 226). In these cases, the effect sizes will be weighted averaged and variances recalculated accordingly prior to the meta-analysis. However, if the study investigates the impact of an intervention on several outcome variables, an effect size will be computed for each outcome and, in such cases, more than one effect size extracted from the same study will be considered in the meta-analysis.

Unit of analysis issues

Based on our preliminary review and background, most of the studies that assess the impact of the interventions that will be investigated in this review use data either at firm or worker level. However, the experimental or quasi-experimental unit may differ. For instance, an intervention may target firms in a specific region. In this case, the intervention would happen at cluster level and the analysis would be performed at firm or worker level. Since firms of the same region (cluster) are more likely to be similar than firms from different regions, the computation of the standard errors should correct for intra-cluster correlation (ICC - correlation between units within the same cluster) otherwise the standard errors would be underestimated and therefore the null hypothesis of no impact would be more likely to be wrongly rejected (a higher risk of Type 1 error). Whenever this is the case, we will contact the authors to request the estimate of the ICC. When the estimate is not made available, we will run power calculation with different values for the ICC to examine how sensitive the estimates and the study conclusions are to the presence of ICC ¹ ?.

Dealing with missing data

Since it is quite unusual to see studies in applied social sciences reporting the pooled standard deviation, we will first try to contact the authors when it is the case and eventually use alternative measures to compute the effect size, such as response ratios and standardised regression (beta) coefficients ¹⁸ . The response ratio is given by the following ratio (see Borenstein et al., 2009):

R = Y T Y C

where Y^T is the mean value of the outcome in the treatment (or eligible) group and Y_C is the mean value of the outcome in the control (or comparison) group. In this case, R corresponds to the effect of the programme on the level of the outcome variable. For difference-indifferences estimates, R will be given by the ratio between the variation in the mean outcome in the treatment group and the mean outcome in the control group. In this case, R provides the effect of the intervention on the growth rate of the outcome of interest.

Inter-rater reliability

To check code consistency across studies, what is called inter-rater reliability, different versions of Cohen's (1960) kappa will be used. The original version of Cohen's kappa is more suitable when there are two coders as it consists of a computation of a standardised index across studies based on cross-tabulated ratings. The index is given by difference between the observed percentage of agreement in ratings across studies, P(a), and the probability of expected agreement due to chance, P(e), divided by 1-P(e). The kappa index varies from -1 (total disagreement) to 1 (total agreement), with 0 representing completely random agreement.

According to Hallgren (2012), the original Cohen's kappa is subject to two main sources of problems. One problem arises from the nature of the coding system and it is called prevalence problem. This happens when one category of rating is over-represented compared to the others. This biases the kappa estimates downwards. The second source of problem is called bias problem and arises when the ratings differ considerably across coders. Although different versions of Cohen's kappa have been developed to deal with these problems, there is not a unique estimator that takes care of both problems simultaneously. Hallgren (2012) recommends Siegel and Castellan's (1988) kappa in the presence of bias and Byrt et al.'s (1993) kappa to correct for the prevalence problem.

Quantitative synthesis

Since this review will be covering a broad set of interventions, it may be difficult to find studies that satisfy the minimum requirements for direct comparability. However, whenever possible, this review will apply meta-analysis techniques to synthesise the effect sizes of an intervention. Borenstein et al. (2009) define meta-analysis as a rigorous way of synthesising the effect of an intervention investigated by several different studies. Although the interventions tested in different studies may be seen as equivalent by the researcher, the context in which they take place may be considerably different and by no means irrelevant for their success or failure. To be suitable for meta-analysis, the studies have to investigate the same intervention, and have similar outcome variables and effect sizes (Becker et al., undated; Waddington et al., 2012b). ¹⁹ In order to account for such idiosyncratic characteristics of each intervention, the meta-analysis will use random effects to compute the ‘average of the averages’, as it might be implausible to assume that there would be a unique effect size regardless of the differences between the studies. We will assess heterogeneity across contexts using moderator analysis (discussed below), as necessary.

Our criteria for studies to form part of the meta-analysis will be as follows:

The interventions are judged to be similar enough for comparison.

Forest-plots will be used to summarise the results for each study and the overall effect size will be estimated with inverse weighted variance. This is done to account for studies sample size. In addition to this, where we are able to control for external factors that might contribute to the success (or failure) of an intervention, the so-called moderator factors, such as the level of bureaucracy to deal with the paperwork and tax codes, and the labour legislation, we may be able to run a meta-regression controlling for such factors. The systematic review will use Stata 12 and 13 to perform the statistical analysis required.

Although quasi-experimental methods, such as difference-in-difference and regression discontinuity design, are usually seen as second best strategies to deal with the selection problems that characterise most of the studies that rely on observational data, there are many controversies regarding whether they should be included or excluded from a meta-analysis. In fact, different quasi-experimental methods have different identification strategies and in most of the cases identify different parameters, such as intention-to-treat, average treatment effect on the treated, local average treatment effects, marginal treatment effects, and so on. ²⁰

We will distinguish between direct and indirect comparisons in the analysis of comparative effectiveness of interventions. Comparative effectiveness will be assessed directly using factorial studies (i.e. studies with multiple treatment arms corresponding to different interventions). Comparative effectiveness will be assessed indirectly across studies using standard meta-analysis and mega-regression. We do not envisage using network meta-analysis due to the anticipated small number of studies reporting direct comparisons.

When meta-analysis is not feasible, for example because there are insufficient (less than two) effect sizes per outcome, forest-plots will still be used but without calculating the pooled effect size and we will discuss the heterogeneity within and across studies narratively. Whenever possible, the analysis will be complemented with reference to potential mediator factors and context variables such as those mentioned in Table 3 under Context and Causal Mechanisms.

Furthermore, given the number of rigorous studies we are aware of, at least for some interventions the review may end up with an insufficient number of studies to allow solid conclusions and therefore policy recommendations. In those cases, we will make clear that policy conclusions cannot be drawn from such a small sample of studies, and that more rigorous evaluations are needed in order for governments, multilateral organisations and donors to draw conclusions about the effectiveness of those interventions.

Heterogeneity Assessment

This review will use meta-analysis and, where possible, meta-regression to synthesise the effect sizes of different studies that looked at the same type of intervention. As stated above, whenever the use of meta-analysis is feasible, we will use random effects to account for the studies' idiosyncrasies.

Another advantage of using a random effects model is that it accounts for two sources of variance: between-study variance, and within-study variance. The former depends on the number of studies included in the review and the latter has to do with the sample size used in that study. To capture such heterogeneity stemming from these two sources of variability, we will compute the I ² statistic ²¹ . In fact, this statistic is considered a descriptive one that informs the inconsistency across the results of different studies (Borenstein et al., 2009). Where a forest plot is used to synthesise findings across the studies, we will also report the estimate for the variance of the ‘true effect size’, as an alternative measure for between-study variance ²² , τ ².

The effectiveness (or failure) of an intervention, particularly if small and very well targeted to firms of certain region or sector, is context-specific and consequently potentially heterogeneous. This review will collect information on contexts, treatment intensity (time exposed to the treatment), constraints and time span in order to scrutinise what underpins the success (or failure) of an intervention.

To shed light on the mechanisms underlying the variability, if any, of the results associated with the same type of intervention, we will conduct a meta-regression (or an AN OVA analysis) since it would be able to disentangle how much of the total variability observed in the results is explained by variables such as context (country), sector, firms size, or time span of the intervention^.

Sensitivity Analysis

We will report separate analyses for RCTs and quasi-experimental studies, as well as examining sensitivity of analyses to studies which report adjusted and unadjusted effect sizes (Becker et al., undated). We will follow Waddington et al. (2012a) and perform sensitivity analysis for different treatment effects (intent-to-treat, average treatment effect, average treatment effect on the treated, local average treatment effect). To test for publication bias, we will use funnel plots as suggested by Egger et al. (1997). To minimize the risk of such bias, the search methods will consider gray papers (unpublished studies), we will use Egger's meta-regression test for publication bias and, where there is evidence for publication bias for particular outcomes, we will re-estimate pooled effects using a ‘trim and fill’ procedure. We will assess whether there is indication of underreporting of negative or negligible effects of interventions under consideration, a practice named ‘file drawer effects’, as part of the risk of bias assessment (see Appendix 3).

External validity

The review will discuss the ways in which the sibling studies and background program documentation informs the potential generalisability of the findings, particularly with regard to Africa, since this is one key focus of this project. We will use the knowledge gained from these studies, in combination with the theory of change and program theory, to understand what factors may influence both the overall conclusions about the effectiveness of the interventions in question, and their generalisability. We will give particular attention to findings which may guide the application and adoption of particular types of intervention in Africa: the questions of how particular interventions may work in an overall low-income context; with regard to educational levels; or with regard to gender, for instance, all have place-specific aspects, and evidence on program design will help to guide the review's conclusions in this respect. To assess possible channels leading to external validity, we also aim to provide insights on potential causes of heterogeneous contexts in Africa by assessing the contextual factors (that is, population characteristics, implementation modalities, study characteristics using moderator analysis and by providing descriptive information about the characteristics of the studies).

We will discuss the extent to which the cause-effect relationships found in the studies evaluated have external validity and therefore may be expected to hold true across different geographic areas, or, as importantly, across income levels. For example, we expect to find more relevant studies in Latin America and Asia than in Africa, and probably more relating to higher-income countries within the group of interest. Since this review is being commissioned to determine the potential applicability of findings to Africa, transferability is also an important question, and we will pay specific attention to the conditions under which study conclusions may also hold true in African countries. This will involve considerations such as the geographic conditions of the study (urban, peri-urban, or rural), the types of SME involved in the study, the scale of the programme (if local, regional or national), and other country conditions such as economic policy, political stability and technological capacity. However, it is important to distinguish between African regions, and countries, in terms of the findings' transferability, given that very different conditions prevail in different regions across the continent. A consideration of these differences is also important when thinking about the related question of applicability: that is, whether a specific type of intervention, rather than the cause-effect relationship found in an evaluation, will play out in the same way in different countries ²⁴ . Although the review will discuss the possible channels that might lead to external validity, as noted in the objectives, assessing applicability of the results to specific local African context is not easy and is beyond the scope of the systematic review.