Within study comparisons and risk of bias in international development: Systematic review and critical appraisal

3.1 Literature review

Table 1 presents a list of known existing reviews of internal replication studies. Some are of particular literatures, for example, studies of labour market programmes (Glazerman et al., 2003) and education (Wong et al., 2017). Others cover particular methodological designs, such as RDD (Chaplin et al., 2018; Cook & Wong, 2008) and propensity score matching (PSM; Shadish, 2013), or map internal replication designs (Wong and Steiner, 2016). Only one known review is dedicated to evidence from social and economic development programmes in L&MICs (Hansen et al., 2013).

Hansen et al. (2013) surveyed four studies, involving two cluster-randomised conditional cash transfer programmes in Mexico and Nicaragua ⁵ and an individually randomised lottery balloting permanent migration visas in Tonga. ⁶ One study in Mexico examined the correspondence of estimates from an RDD analysis with estimates from a cluster-randomised controlled trial (Buddelmeyer & Skoufias, 2004). The remaining studies examined the correspondence of difference-in-difference (DID), matching and IV techniques (Diaz & Handa, 2006; Handa & Maluccio, 2010; McKenzie, Stillman, & Gibson, 2010). Findings from this review highlighted that across the four studies the nonrandomised estimators did offer instances where correspondence with randomised estimates was high (suggesting nonrandomised estimators can provide unbiased estimates), but that this was not always the case. In particular, in the context of the evaluation of development interventions, nonrandomised studies were more relevant in contexts where self-selection is negligible and the selection process is simple or well understood.

However, Hansen et al. (2013) did not use systematic approaches for study identification or formal critical appraisal of studies. In fact, few of the reviews in this body of literature appear to have been conducted systematically (White and Waddington 2012; Waddington et al 2012). Exceptions include a review by Wong et al. (2017), who report a systematic search strategy, and meta-analyses by Glazerman et al. (2003) and Chaplin et al. (2018), although concerns regarding the completeness of their search strategies are noted by the authors themselves. Glazerman et al. (2003) indicate that electronic searches failed to comprehensively identify many known studies. This was due to the lack of a common language to define an internal replication study. Furthermore, it is not uncommon that such studies feature as undefined empirical demonstrations in new methods papers or as a secondary piece of analysis in a broader study. Similar problems were also noted by Chaplin et al. (2018), who state that despite having searched broadly “we cannot even be sure of having found all past relevant studies” (p. 424).

Nevertheless, as the first meta-analysis synthesising this body of evidence, Glazerman et al. (2003) identified 12 studies on job training and employment services ⁷ where the dependent variable was earnings. All studies originated in high-income contexts, based on data collected on interventions in the United States and one in Norway; three-quarters of the interventions and data collection were concluded in the 1970s and 1980s. The analysis examined study findings from a range of different methodological approaches (including cross-section, panel and DIDs regression, statistical matching and selection models). It concluded that nonrandomised methods rarely replicated experimental estimates and the absolute magnitude of the differences was often quite large. ⁸

Non-systematic qualitative updates of this review by Cook et al. (2008) and Pirog et al. (2009) later highlighted that with “careful execution” nonrandomised estimators can recreate randomised estimates, and nonrandomised estimates based on inappropriately designed estimation procedures were associated with larger bias coefficients (Cook et al., 2008).

Reviews by Cook and Wong (2008), Cook et al. (2008) and Pirog et al. (2009) later expanded the scope of nonrandomised estimators examined, including ITS and RDD. Drawing from a limited base of evidence, they suggested that an ITS study could also create similar results to an RCT. Similarly, they concluded that studies using RDD provide estimates that are comparable to an RCT estimate when it is made for observations close to the discontinuity (or “cut-off”) in the assignment (or “forcing”) variable.

Building on these findings, Chaplin et al. (2018) further assessed the statistical correspondence of 15 internal replication studies with an RDD approach (including two studies based on data collected on programmes in L&MICs) using meta-analysis. They reported that the average of the difference between RCT and RDD estimates around the discontinuity is close to zero (approximately 0.1 standard deviations) and that the variability of results was also generally quite low. However, they warned that researchers should not assume based on these findings that individual RDD estimates will necessarily be near zero. They suggested factors such as larger samples, using nonparametric tests and the choice of bandwidths may prove important in determining the degree of bias in an individual RDD estimate.

Further reviews have included mapping internal replication designs and describing different measures of bias (or correspondence; e.g., see Jaciw, 2016; Wong & Steiner, 2016). Wong and Steiner (2016) describe broad categories of internal replication studies, including independent, synthetic, simultaneous and multisite simultaneous designs (Table 2).

However, authors such as Smith and Todd (2005, p. 306) warn against “searching for ‘the’ nonexperimental estimator that will always solve the selection bias problem inherent in nonexperimental evaluations”. Instead they argue research should seek to map and understand the contexts that may influence studies’ degrees of bias. For instance, Chaplin et al. (2018) consider that their review says little about instances when an experiment is logistically very difficult to implement, or noncompliance is likely to be large. Hansen et al. (2013) note the potential importance of the type of dependent variable examined in studies, suggesting simple variables (such as binary indicators of school attendance) may be easier to model relative to more complex outcome variables (such as consumption expenditure or earnings). Meanwhile, Glazerman et al. (2003) find factors such as the source of data, the quality of control variables and evidence of statistical robustness tests are related to the magnitude of estimator bias.

Jaciw (2016) provides a broader review of characteristics associated with bias in internal replication studies. The author notes that studies have found the comparison groups’ geographic proximity, the richness of background controls, use of baseline outcomes as control variables and the complexity of outcome variables to be related with the degree of bias among nonrandomised estimators. Meanwhile, investigating best practices for selecting covariates in education research, Wong et al. (2017) synthesise results from 12 internal replication studies (all from high-income countries) where the dependent variable included a standardised reading or math test score. Similarly, they describe baseline outcomes, geographic proximity and the richness of control variables are important factors that may determine the magnitude of bias among nonrandomised estimators. They also note where nonrandomised studies simply rely on a set of demographic variables, or those that prioritise local matching when local comparisons are not comparable to treated cases, they will rarely replicate similar estimates to RCTs. In Table 3 we summarise the list of factors which internal replication studies hypothesise may be associated with bias in nonrandomised studies.

Beyond bias being determined by a particular method, or magnified by a characteristic of a study, another potential source of discrepancy between randomised and non-randomised designs concerns whether they provide the same causal quantity. In other words, a factor explaining differences in findings across randomised and nonrandomised designs, over and above bias and sampling error, is that they may provide different causal estimands for different treatment populations. For example, Cook et al. (2008) articulate that confounding may occur when comparing an experimental intent-to-treat (ITT) estimate with a nonrandomised estimate which computes the average effect on those that receive the treatment of interest (i.e., the ATE on the treated, TOT). Here one issue that arises follows that the average effect reported by the ITT estimator becomes increasingly more conservative as noncompliance rises, making the two estimators not directly comparable.

In another example, Cook et al. (2008) highlight issues arising from estimators derived from LATEs, which are most commonly estimated during the conduct of RDD, to the ATE estimated from an RCT. In this instance, if we are to relax the assumption that the effects of an intervention are homogenous across a population, the size of the LATE effect would be conditional on the point in the population's distribution being assessed (e.g., the point in the distribution that a discontinuity occurs). Again, here the LATE estimate may be an unbiased estimate of the average effect of an intervention amongst the population in immediate proximity to the discontinuity. However, it may also be a very different magnitude to the average effect observed across the entire population that receive the treatment.

We used the following approaches for study inclusion criteria, searching and data collection.

Study design: Glazerman et al. (2003, p. 65) define a replication study as follows: “researchers estimate a program's impact by using a randomised control group and then re-estimate the impact by using one or more nonrandomised comparison groups”. We included studies that report treatment effects for a benchmark randomised causal study, alongside treatment effects for a nonrandomly assigned comparison replication. The replicated comparisons could be constructed using any quasi-experimental method (e.g., DIDs, IVs, statistical matching, RDD).

Population: populations in L&MICs were eligible; among general programme participants or lab studies conducted among students. The causal benchmark and comparison study needed to derive from the same study population so as to minimise the possibility of a factor other than study design confounding estimates of bias.

Intervention and comparator: studies could be of any social or economic intervention and of any comparison condition (e.g., no intervention, wait-list, alternate intervention). Where relevant, the causal benchmark and comparison study also needed to derive from the same intervention or comparison condition, and time period.

Outcome: studies could be of any outcome variable, provided the outcome was the same for the benchmark control and replication comparison groups.

Studies or study arms were excluded that:

made between-study comparisons (with no overlap in treatment group samples for causal benchmark and comparison);

3.3 Search results

Figure 1 summarises the search strategy and results for primary internal replication studies. The RePEc database search returned 3,271 records and was conducted in August 2016. Citation tracking searches, also conducted in August 2016, returned a further 951 records for screening. The search of institutional repositories, including Mathematica's (in September 2016) and 3ie's repository of impact evaluations (in July 2017), identified 307 records. Contacting authors of existing studies, and hand searches of our own personal repositories of known studies, identified 13 additional references. After removal of duplicates, a total of 3,904 records were included for screening at title and abstract.

OPEN IN VIEWER

1

Study search flow for nonrandomised internal replication studies

During screening at the title and abstract 3,328 records were excluded. The remaining 576 records were screened at full text. A further 443 records were then excluded during full-text screening, leaving 129 records which were assessed for eligibility. Of these 133 studies, 21 were removed due to being working papers of now published articles, and 102 internal replication studies were excluded due to the geographic location of the RCT not deriving from an L&MIC context. We eventually included 10 studies of social and economic programmes in L&MICs.

There were a number of excluded studies among L&MIC populations that made comparisons between randomised and nonrandomised estimates of programmes. For example, Friedman et al. (2016) were excluded because the results for the nonrandomised group were calculated using OLSs, a method which we would not expect to account for confounding satisfactorily. We similarly excluded OLS comparison group estimates from included within-study comparisons (McKenzie et al., 2010). Other studies did not meet the required criteria to be classified as within-study comparisons. Thus, we excluded Oosterbeek et al. (2008), Behrman et al. (2009) and Barham et al. (2014), on the basis that the populations did not overlap. Oosterbeek et al. (2008) compare the findings of a randomised experiment conducted among households with a poverty index between the 13th percentile and the 28th poverty percentile with an RDD analysis among households between the 33rd percentile and the 47th percentile. Behrman et al. (2009) provided a comparison of randomised and nonrandomised estimates using control populations with different variations in exposure to a cash transfer programme. Barham et al. (2014) compared randomised and nonrandomised estimates covering different calendar periods.

Another study by Cintina and Love (2014) also created nonrandomised treatment and control groups from an RCT by Banerjee et al. (2015), and as such, did not provide an estimate of effect of the same intervention using randomised and nonrandomised groups. Similarly, Glewwe et al. (2004) was also excluded because it formed a between-study comparison, examining differences in effects of similar but different interventions.

Included studies are summarised in Table 4. Four of these studies featured in the previous review of internal replication studies in international development (Hansen et al., 2013). These are based on data from conditional cash transfer schemes, PROGRESA in Mexico (Buddelmeyer & Skoufias, 2004; Diaz & Handa, 2006) and Red de Proteccion Social in Nicaragua (Handa & Maluccio, 2010), and a randomised lottery balloting permanent migration visas in Tonga (McKenzie et al., 2010). ¹¹ One study on Mexico's cash transfer programme examines the correspondence of estimates from an RDD analysis with estimates from an RCT (Buddelmeyer & Skoufias, 2004) and we were also able to locate an additional six replications of RCTs (Barrera-Osorio, Filmer, & McIntyre, 2014; Chaplin et al., 2017; Galiani & McEwan, 2013; Galiani, McEwan, & Quistorff, 2017; Lamadrid-Figueroa et al., 2010; Urquieta, Angeles, Mroz, Lamadrid-Figueroa, & Hernández, 2009).

All included studies use randomised control trials as the benchmark, with the exception of McKenzie et al. (2010) which uses a randomised natural experiment, where programme assignment was done by a public lottery by policymakers although the data itself were collected by the authors specifically to test the treatment effect. The studies test a range of nonrandomised replication methods including geographical discontinuity design (GDD), RDD, IVs, PSM and DIDs. In this section, we discuss narratively the results from the six additional internal replication studies identified in this updated search for literature. Extending the review of findings by Hansen et al. (2013), we provide a summary of the context, approach and highlights of each study.

3.5 Risk of bias assessment in benchmark studies

Existing reviews of internal replication studies do not provide comprehensive assessments of the risk of bias to the effect estimate in the benchmark study using formal risk of bias tools. Partial exceptions are Glazerman et al. (2003), who comment on the likely validity of the benchmark RCTs (randomisation oversight, performance bias and attrition), and Chaplin et al. (2018) who code information on use of covariates to control for pre-existing differences across groups and use of balance tests in estimation.

Our overall assessment of the risk of bias involving experiments in internal replications from L&MICs indicates that all of the experiments host low or moderate concerns. Concerns largely arise from a lack of sufficient evidence to confidently assign a “low risk” score. For instance, in some cases concerns could be alleviated with further provision of information or analysis of the underlying experiments data. For example, concerns involving the imbalances between treatment and control groups in Chaplin et al. (2017) could simply be resolved with appropriate analysis of baseline characteristics using distance metrics. Furthermore, more robustness testing and information on the sampling strategy at follow-up involving the control group may help to alleviate some concerns with regards to missing outcomes data in McKenzie et al.'s (2010) natural experiment.

In other instances, concerns may be more difficult to address. For example, none of the studies address the issues of blinding outcome assessors and it is unknown to what extent this could influence assessments of outcomes and participant selection. Furthermore, it is challenging and rare that widely implemented social programmes such as those that feature in many of the cluster-randomised trials assessed here can sufficiently capture data on confounding issues such as migration between clusters. This latter point may give rise to the argument that perhaps future within-study comparisons in L&MICs would also benefit from making use of smaller and more controlled environments in order to develop internal replication studies.

Finally, a caveat of the published risk of bias tool for RCTs is that it does not provide questioning to discern the sufficiency of the application of IV estimation as a correction for noncompliance. It merely provides a decision score of “some concerns”. This means that the IV results provided by McKenzie et al. (2010) would not be able to achieve a “low risk” assessment in relation to bias arising due to deviations from intended interventions, regardless of the rigour of the analysis done.

The rest of this section states the key factors, uncertainties and decision points influencing the scores associated with each domain of bias for each experiment assessed (Table 5). It proceeds by discussing each bias domain in turn.

3.6 Quantitative estimates of bias in nonrandomised within study replications

We collected data on treatment effects for the benchmark study and each corresponding nonrandomised replication presented, from 604 specifications. These data included outcome means in treatment and control/comparison (or treatment effect estimates from an analysis), outcome variances, sample sizes, tests of statistical significance, types of variables used in adjusted analysis and available measures of treatment compliance for RCTs (see coding tool in A). We used the estimate of effect from the RCT which most closely corresponded with the population for the nonrandomised arm (e.g., the bandwidth around the treatment threshold in the case of Buddelmeyer & Skoufias, 2004). Where there was nonadherence, we used the CACE, ¹⁶ as in the case of McKenzie (2010).

3.6.1 Quantitative measures of bias

There are two main types of measures of difference between the benchmark and nonrandomised replication study arms (Steiner & Wong, 2016): distance metrics which quantify the difference between the effect size estimates between the benchmark and nonexperimental replication; and conclusion-based measures which use information on sign, statistical significance or an effect threshold.

We calculated the standardised absolute difference between treatment effects in experimental and nonrandomised replication samples. We define D as the primary distance metric measuring the size of the bias between the nonexperimental and experimental results. D _s is a standardised measure of D which is defined in recent reviews by Wong et al. (2017) and Steiner and Wong (2016). We used the absolute difference in D to ensure consistency across studies reported effects; for example, Chaplin et al. (2017) only reported absolute direct standardised measures. In addition, in the subsequent results, we report averages over the large number of values of D collected in each study; we want a measure of the deviation of randomised and nonrandomised estimators, and not one that on average “cancels out” positive and negative deviations, hence potentially obscuring important differences. Formally D _s was computed as follows:

| D s | = | τ ˆ r e − τ ˆ n x | s r e , where | τ ˆ r e − τ ˆ n x | = | ( Y ¯ r e t − Y ¯ r e c ) − ( Y ¯ r e t − Y ¯ n x c ) = ( Y ¯ r e c − Y ¯ n x c ) | ,

1

Y ¯ n x c and Y ¯ r e c

are the mean outcomes of the nonexperimental comparison and experimental control groups,

Y ¯ r e t

is the mean outcome of the experimental treatment group and s _re is the sample standard deviation of the outcome in the experimental group or the pooled standard deviation of the experimental treatment and control group.

Where an appropriate standard deviation of the outcome in the experimental group or the pooled standard deviation of the experimental treatment and control group were not reported, the standardised mean difference (d) for each estimator was calculated using the following formula and then subtracted from one another to compute D _s, as follows:

d n x = 2 t n g + n c , d r e = 2 t n g + n c , D s = | d r e − d n x | ,

2

where n denotes the sample size of treatment group (g) or control/comparison (c) and t is the t-statistic for the difference in group means in either nonexperimental (nx) or experimental (re) samples. Where t was not reported, we calculated it by dividing the reported coefficient for the difference in group means by the standard error. If the authors only reported confidence intervals and no standard error we calculated the standard error from the confidence intervals (as in the case of McKenzie et al., 2010). ¹⁷

4.1 Risk of bias in Campbell systematic reviews

We downloaded records from the Campbell Library and collected the following data:

Study information: Lead coordinating group and study identifiers (lead author and year).

We reviewed all 99 Campbell systematic review reports published between January 2012 and December 2018, of which 80 (81%) included nonrandomised studies. ¹⁸ All reviews published by Education Coordinating Group (ECG) and International Development Coordinating Group (IDCG; including Nutrition Group) incorporated nonrandomised studies. Furthermore, 83% of the Crime and Justice Coordinating Group (CJG) and 55% of Social Welfare Group (SWG) reviews incorporated nonrandomised studies. In one Knowledge Translation and Implementation Group review, nonrandomised studies were included as eligible but none were found (Petkovic et al., 2018). Authors used different tools to assess risk of bias for included nonrandomised studies (Table 7), roughly corresponding to the group coordinating the registration process. SWG authors mainly used either an early version of Cochrane's risk of bias tool for nonrandomised studies of interventions (Reeves, Deeks, Higgins, & Wells, 2011), or Cochrane's risk of bias tool for RCTs (Higgins et al., 2011), as did half of ECG reviews. IDCG authors largely used the tool developed by 3ie (sometimes attributed to IDCG, other times as Hombrados & Waddington, 2012), although two used the tool developed by Cochrane Effective Practice and Organisation of Care (EPOC, u.d.), and one used Cochrane's Risk of Bias in Nonrandomised Studies of Interventions (ROBINS-I; Sterne et al., 2016; known as ACROBAT at the time the reviews were undertaken). CJG authors mainly used tools they developed for the specific purposes of the review in question.

We also collected data on the domains of bias contained in the tools used to assess studies incorporated in Campbell reviews. We focused on bias domains considered particularly relevant for RDDs: confounding and selection of the reported result (as defined in Sterne et al., 2014). 'Confounding bias' occurs when factors which predict the outcome also determine receipt of intervention. This includes selection to intervention by participants (e.g. on the basis of need) or practitioners (e.g. on the basis of eligibility), or programme placement bias by planners (e.g. on the basis of geographical unit). Confounding can be observed or unobserved (unmeasured or unmeasurable), time-invariant (fixed over the course of the study at baseline) or time-varying. Some types of confounding bias can be controlled in analysis, for example observables can be controlled in adjusted analysis, and time-invariant confounding (including unobservables) can be controlled through statistical modelling (e.g. difference-in-differences analysis of pre-test and post-test outcomes between intervention groups in order to account for), and most effectively in study designs which in theory can control for unobservable and observable confounders (e.g. RCTs and RDDs). We found that 94 percent of Campbell reviews incorporating non-randomised studies assessed risk of bias due to confounding. 'Bias in selection of the reported result' may refer to outcomes or analysis, and may be addressed through reporting of pre-analysis plans, transparent reporting of any analyses that were determined post hoc, and reporting on all outcomes and participant sub-groups measured irrespective of findings. Over 80 percent (83%) of Campbell reviews incorporating non-randomised studies specifically assessed bias in selection of the reported result. Other sources of bias defined by Sterne et al. (2014) which are relevant for RDDs are not covered here because the signalling required to answer them are unlikely to be different for RDDs as for other designs (e.g. 'performance bias', is when participants receive a different intervention to the one intended, and 'bias in outcomes data collection' due to measurement error in outcomes reported by participants or measured by outcomes assessors). One source of bias, 'bias in classification of interventions', was deemed irrelevant for RDDs, because assignment to intervention is based on a score measured at pre-test. RDDs are sometimes eligible for, and included in, Campbell systematic reviews. In 2012–2018, 35 reviews explicitly allowed for the inclusion of RDD (51% of reviews that incorporate nonrandomised studies, or 35% of all reviews)—that is they mention “regression discontinuity” explicitly in the study inclusion criteria, or define inclusion that would be consistent with RDD (e.g., allocation by “time differences, location differences, decision-makers, or policy rules”; Filges, Jonassen, & Jørgensen, 2018, p. 19). Twelve reviews were able to include at least one RDD in analysis (18% of reviews incorporating nonrandomised studies; Table 8). These include eight reviews in international development, which draws on a cross-disciplinary literature including econometrics where the method has become popular, plus three reviews in social welfare and one in education. In a few instances, authors differentiate types of discontinuity design such as Lawry et al. (2014) and Samii et al. (2014) who each include one GDD, and Piza et al. (2016) and Filges et al. (2018) who include “fuzzy” RDDs. In other cases, however, RDDs were excluded from analysis. Petrosino et al. (2012) differentiated experimental and observational designs, including two prospective RDDs and excluding two others that are natural experiments (defined by the authors as “investigator initiated not based on actual program rules”, p. 100). Turner et al. (2018, p. 16) note that “[a]lthough Regression Discontinuity Designs (RDDs)… generate data that can be used to make causal inferences, they were excluded from this review because statistical methods for incorporating RDD… data into meta-analyses are, to the best of our knowledge, not well-established”.

Previous research has found that existing risk of bias tools are not operationalised to assess nonrandomised studies with selection on unobservables (Waddington et al., 2017). This is partly because most tools were not developed to address particular designs like RDDs. For example, we are aware of four tools that present signalling questions for RDDs (Chief Evaluation Office, undated; Hombrados & Waddington, 2012; Schochet et al., 2010; Valentine & Cooper, 2008). The tools on which RDDs have been critically appraised in Campbell reviews are Hombrados and Waddington (2012), Reeves et al. (2011) and Higgins et al. (2011).

Given the increased interest in RDD as a method of programme evaluation, the incorporation of these studies in systematic reviews and concerns about methods for their synthesis, we elected to focus our work here on RDD. Using a mixture of literature review (of existing risk of bias tools, textbooks and recent journal articles) and expert consultation, we developed signalling questions under seven bias domains.

RDD, also called RD and “cut-off based design”, was conceived of as in educational psychology by Thistlethwaite and Campbell (1960) (cited in Shadish et al., 2002) and in economics by Goldberger (1972). In RDD, assignment to treatment is based on a specific score on an ordinal or continuous measure, such as a diagnostic test, that is given prior to treatment. For example, an educational authority might choose to administer a math test to all 10 year-old students, and provide extra support for students scoring below some threshold of proficiency. Students just on either side of the cut-off threshold should be highly similar to one another, and therefore if there is a treatment effect this should be noted in a discontinuity (or break) in outcomes between the treated and untreated groups at the point of intervention, which may be a change in intercept or slope, or a combination. Figure 2 presents two simple examples of the relationships between the assignment variable (pretest score with cut-off set at 50) and outcomes; there are many more possibilities allowing for nonlinear relationships between assignment variable and outcome (see, e.g., Shadish et al., 2002). RDD has become a popular approach to treatment effect estimation, ¹⁹ particularly in economics. Key review articles and practice guidelines are available in econometrics (Imbens & Lemieux, 2007; Lee & Lemieux, 2010), educational psychology (see e.g., Cook, 2008) and, latterly, epidemiology (Moscoe, Bor, & Bärnighausen, 2015).

OPEN IN VIEWER

2

Examples of RDD

We give several definitions of the RDD approach by leading authors in Table 9. The definition by Shadish et al. (2002) implies that it is the experimenter who designs the study prospectively. However, discontinuity assignment may also arise from natural processes of policy and practice—for example, age requirements for pensions and biometric tests used in medicine. Where outcomes data are available, or can be collected, this opens up the approach to retrospective analysis provided data are also available on the assignment variable used to allocate units prehoc; in other words, RDD as a natural experiment (Dunning, 2012).

Cook's (2014) definition as presented here suggests a deterministic relationship between the assignment variable and treatment status, but Cook also refers to “fuzzy” RDD which allows for a probabilistic relationship between assignment and treatment (in other words, there are additional factors, which may or may not be measured, affecting assignment status). The quote from Dunning (2012) also indicates that, at least in social science research and econometrics, the causal relationship is usually estimated close to the cut-off threshold. However, the global relationship may also be estimable under stronger assumptions (Angrist & Rokkanen, 2015; Bor, Moscoe, & Bärnighausen, 2015; Bor, Moscoe, Mutevedzi, Newell, & Bärnighausen, 2014; Rubin, 1977). ²⁰ Finally, Schochet et al. (2010) give guidance on the nature of the assignment variable, suggesting scope for ordinal scales to be included with sufficient units either side of the threshold.

In theory, RDD generates an unbiased treatment effect. This is because the assignment rule is known, hence the relationship between treatment and outcomes can be modelled with trivially small confounding, at least close to the assignment threshold (Cook, 2008). Its closest design relative is the RCT. ²¹ In economics, political science and many health applications (see Dunning, 2012; Moscoe et al., 2015), RDD has frequently been used retrospectively to evaluate existing threshold rules. Because RDD needs large samples, it is well suited to retrospective evaluation using existing data sources (e.g., administrative data).

Lee and Lemieux (2010) identified 67 RDD studies in economics between 1983 and 2009. Dunning (2012) reviews 22 examples of RDD natural experiments in social science research between 1999 and 2009. According to the literature review by Moscoe et al. (2015), there were—at time of writing—32 examples of RDD to assess the impact of interventions on health outcomes in PubMed. Most of the RDDs found evaluated social and economic interventions. Only two evaluated the effects of clinical interventions: Almond et al. (2010) evaluating health care programmes assigned by birth weight, and Bor et al. (2015) evaluating antiretroviral therapy assigned by CD4 count. A forthcoming systematic review covering health, social science and grey literature by Hilton-Boon et al. (2016) identified 177 RDDs published before March 2015 evaluating the effect of an intervention or exposure on a physical or mental health outcome. This indicates the great scope and practice of RDDs being undertaken in health and economics alone.

In RDD, treatment is assigned ex-ante according to a known rule—specifically, a threshold on a scale variable measured among participating units at pretest. Units scoring on one side of the threshold subsequently receive treatment, while those on the other do not. The treatment effect is estimated by comparing observations from different units observed contemporaneously, immediately on either side of the threshold. Different types of assignment variables have been used in RDD analyses (Dunning, 2012; Hahn, Todd, & van der Klaauw, 2001; Moscoe et al., 2015) including but not limited to:

Test scores: for example, continuous biomarkers in medicine (e.g., cholesterol, blood glucose, birth weight, CD4 count); entrance exams in education.

In GDD, exposure to the treatment depends on the position of observations with respect to an administrative or territorial boundary (e.g., Galiani et al., 2017). In the particular case where the assignment variable is time, RDiT is similar to controlled ITS (Hausman & Rapson, 2018; Shadish et al., 2002). Perhaps one difference between ITS and RDiT is that treated and untreated samples are different, which makes a difference to the length of follow-up period over which treatment effects can be credibly estimated. In ITS, the same participating units are followed-up over time, and the treatment effect is identified through variation in exposure to treatment over time, sometimes with respect to an untreated comparison (Somers, Zhu, Jacob, & Bloom, 2013). It is most credible in estimating treatment effects for observations immediately after the time of intervention in comparison with their values immediately before (i.e. the short term effect). In contrast, in RDD, the treatment effect is estimated by comparing observations from different units measured at the same time (or follow-up period); the comparison is made up of units who were eligible immediately before or after a threshold date on which a policy or practice change occurs. However, the outcome for those units could be assessed many years later.

In the basic design, assignment to treatment and comparison is based on the observational unit's pretest score on the continuum, relative to the assignment threshold (Figure 3)—whether above the threshold (panel a) or below it (panel b). Variants on the design might include multiple thresholds on the assignment continuum within which treatment is assigned (Table 10, panel c), or multiple thresholds for different levels of treatment (panel d; Shadish et al., 2002). While the forcing variable is necessarily measured at pretest, outcomes may only be available at posttest.

OPEN IN VIEWER

3

Examples of assignment rules used in RDD. RDD, regression discontinuity design

In the simplest formulation of the RDD, the treatment effect is estimated as the change in the intercept at the threshold in a linear regression of outcome measured at posttest on the assignment variable (Figure 2). The treatment effect is estimated as the vertical difference between the regression line for the observed outcome for treated units, and the extrapolated counterfactual outcome from the regression line for controls.

Estimating the correct functional form of the relationships between assignment variable and outcome is the main estimation concern. Researchers must correctly establish the functional form to avoid a potentially confounded relationship—for example, where a linear regression line is fitted to a nonlinear relationship (Angrist & Pischke, 2009, p. 254; Shadish et al., 2002). Variants on the estimation approach include allowing the treatment effect to manifest as a change in slope as well as, or instead of, a change in intercept (by incorporating interaction terms in regression estimation). For example, in the “regression-kink design” the treatment effect is estimated as the change in slope of the outcome variable at the threshold (Figure 2).

The counterfactual approach, and the modelling of functional form, can be strengthened by data. Availability of pretest outcomes data can strengthen the approach. The clear advantage of prospectively designed RDDs—over and above the advantage they have over retrospective studies in controlling for selection bias into the study—is that outcomes can be measured before assignment across participants, allowing estimation of the relationship between assignment variable and outcomes among different units at pretest. This can verify the counterfactual relationship at pretest for observations before they are assigned, in order to confirm the functional form. Where treatment occurs among units in time-dependent batches, there may be scope to estimate the counterfactual functional form relationship at the same time among different groups (Cook, 2008). ²² Recruitment into the study of a “pure control” group that is subjected to the same informed consent and data collection also has the advantage of enabling measurement of motivation biases in prospective studies (see below). The addition of a nonequivalent dependent variable function (“placebo outcome”) can also be added to provide a “no-cause counterfactual” (Trochim, 1984; cited in Cook, 2008). Tests for “placebo discontinuity” at different thresholds of the assignment variable can help rule out the existence of a chance relationship with the outcome of interest (Moscoe et al., 2015).

Most RDDs are analysed retrospectively using routine data collection. This has implications for the analysis that can be done to verify underlying assumptions of the approach (e.g., limits to testing the preintervention relationship between assignment variable and where outcomes baseline data are not available). But there may also be advantages of retrospectively conducted RDDs over prospective designs. One clear advantage of retrospective studies in general is that participants are not aware they are part of a research study at the time of intervention. Hence, threats to validity arising in prospective studies due to behavioural responses to being observed in treatment groups (Hawthorne effects) or control groups (compensatory rivalry [John Henry] or resentful demoralisation; Bärnighausen, Tugwell, et al., 2017b), or social desirability bias from outcomes data that are self-reported rather than directly observed (Schmidt, 2014), may be less relevant. A disadvantage of retrospective RDD is that one really needs to know a lot about the situation in which RDD is implemented to judge its validity—for example, whether placement scores could be manipulated (by selectively allowing some people to retake the placement test).

The main threats to internal validity in RDD concern inappropriate characterisation of the assignment variable or the relationship between assignment variable and outcome, and manipulation by participants or implementers of the assignment variable.

The assignment variable should be a scale variable with sufficient unique values. Usually, this would be a continuous variable, although the What Works Clearinghouse Standards for RDD states the minimum requirement is for an ordinal scale with four units either side of the treatment threshold (Schochet et al., 2010). ²³ The assignment variable does not need to be truly continuous, provided it is a scaled variable with sufficient possible values either side of the threshold in order for a credible linear line between the assignment variable and the outcome to be estimated.

In order for RDD to produce internally valid estimates, the minimum criterion is exchangeability at the threshold—that is, the potential outcomes would be same on average if treated units had been untreated and untreated individuals had been treated, as would be the case in a well-conducted RCT. One common way that this is violated is if the assignment variable itself is precisely manipulabled by participants or implementers, at least over the subsample of observations around the cut-off threshold. Threats to validity may arise where there is public knowledge among programme participants of a manipulable assignment variable, or where practitioners are able to assign to treatment on a discretionary basis. This is equivalent to assessing subversion of randomisation when random allocation is not concealed until after recruitment in RCTs. Hence, participants should either be blinded to the value of their assignment variable or unable to manipulate it (and practitioners should not be involved in assignment, or will not be able to manipulate it). Assignment variables which participants have manipulated include reported income, which may be incorrectly reported or manipulated to gain eligibility to programmes (Buddelmeyer & Skoufias, 2004).

In addition, the relationship between the assignment variable and outcome must be unconfounded over the sample in which the treatment effect is estimated. The most important cause of confounding is manipulation by participants or planners. This assumption is easiest to establish for the sample of observations closest to the threshold where there is random error in measurement of the assignment variable (Goldberger, 1972). Where an assignment rule determines eligibility for more than one intervention affecting the outcome of interest (treatment confounding), it will not be suitable for RDD analysis of the effects of the intervention of interest (Schochet et al., 2010). An example is evaluations of school lunch programmes in which family income is used for programme eligibility at the same cut-off as is used for other state welfare benefits. This is the same as a violation of the exclusion restriction in an IVs set up, or the selection-by-history effect that could also occur in an RCT. However, it still offers a valid opportunity to assess the overall ITT effect of assignment to the full package of interventions.

Hence three causal estimands can be distinguished, relating to assumptions about homogeneity of treatment effect across the full sample distribution:

1.

Where treatment effects are variable across the distribution, RDD is able to estimate the LATE (also called the ATE at the cut-off; Hahn et al., 2001). In the simplest case, where there is random noise in the measurement of a (nonmanipulable) assignment variable, units immediately around the threshold can be considered as randomly allocated to treatment and control (Goldberger, 1972). Where the assignment variable is measured without noise, a stronger assumption is needed, that is, that the relationship between the assignment variable and outcome is unconfounded in the region of the threshold. Another way of saying this is that the potential outcomes (expected outcomes conditional on the assignment variable) are assumed to be continuous at the threshold (Bor et al., 2015; Hahn et al., 2001).

2.

Where treatment effects are constant across the assignment variable distribution (i.e., slope of assignment–outcome relationship independent of assignment variable), RDD can estimate the ATE. In this case, the functional form of the relationship between assignment variable and outcome must be modelled appropriately over the full distribution of the assignment variable. Strong assumptions are needed for identification, such as the relationship between assignment and outcome being unconfounded over the entire distribution of the assignment variable (potential outcomes are continuous across the full distribution; Bor et al., 2015).

3.

Where the relationship between assignment variable and outcome is probabilistic rather than deterministic (“fuzzy RD”), IVs is used to estimate the CACE.

4.5 Statistical conclusion validity in RDD

The main factors affecting the validity sources of a statistical test of the treatment effect in RDD arise from: Choice of bandwidth around threshold, estimation of the functional form and adherence to assignment rule.

Statistical modelling to determine the sample over which to estimate the treatment effect is crucial in RDD. Under the weaker validity assumption for LATE, this means determining the appropriate bandwidth around the threshold for local linear regression, which is usually done using covariate balance tests to examine the assumption that groups on either side of the threshold are roughly equivalent in their baseline characteristics (Schochet et al., 2010).

Estimating the correct functional form of the forcing assignment variable is the main estimation concern. As noted above, the relationship between outcome and assignment variable may be nonlinear and confound the treatment effect estimate. There are two ways to estimate the relationship:

1.

Parametric approaches: The use of parametric approaches does not necessarily imply that the relation is linear. Ideally one should try and report different polynomial orders for the assignment variable, and then (using Akaike Information Criteria and visual inspection) identify the preferred specification under different bandwidths of the assignment variable.

2.

Nonparametric approaches: These do not make any assumptions about functional form. For this reason, Imbens and Kalyanaraman (2012) argue that this technique is preferable. The key issue when using this approach is in defining the optimal bandwidth. The state of the art option used by econometricians is the optimal bandwidth procedure defined by Calónico et al. (2014).

We can distinguish two forms of nonadherence in RDD analysis, which affect the estimation methods used: Nonadherence to the assignment rule; and fuzziness (cf. misclassification) in the relationship between assignment variable and assignment status. We note here that these are conceptually different from manipulation of the forcing variable, which is a major threat to validity in RDD, as noted above.

Imbens and Lemieux (2007) and Cook (2008), citing Trochim (1984) distinguish “sharp” and “fuzzy” RDD. In the case of a “sharp” RDD, the relationship between assignment variables and treatment status is known and deterministic. In analysis, this means there will be no value of the assignment variable at which different units may be both treated and untreated. There may still be overrides to the assignment due to nonadherence, which can be addressed in ITT analysis.

This is a qualitatively different case from fuzzy RDD, where the assignment scale only determines the probability of treatment status, for example because additional factors are taken into account to determine assignment. The distinction is between fuzziness in the assignment threshold value, as in the case of overrides to the assignment rule in sharp RDD, versus allowing for other factors to determine treatment uptake, with the assignment cut-off fixed, in fuzzy RDD. In fuzzy RDD there are elements of the rule determining assignment which may be “unknown” and in analysis, there may be values of assignment in which different units may be both treated and untreated, hence the need for two-stage least squares estimation, where RDD assignment is used as an IV.

Other implementation problems, such as nonrandom attrition and other sources of incomplete data, and biases in outcome measurement, can be assessed using standard approaches.

4.6 Reporting standards and a preliminary risk of bias tool

There is potential for bias if the assumptions for RDD are not met, or not demonstrated and reported to be met. For example, adequate reporting of the assignment mechanism is vital in RDD. Lee and Lemieux (2010) and Moscoe et al. (2015) discuss reporting criteria for RDD. We list the following as being important in justifying the approach:

1.

Clear presentation of the assignment–outcome relationship in a graph showing the discontinuity. Appropriate functional form may include local linear regression at assignment threshold, or ordered polynomial. The treatment effect may be measured as a change in intercept and/or change in slope (regression-kink design).

2.

Discussion of the validity conditions in the context of the study, particularly around manipulation of assignment variable score, demonstrating that: the assignment decision rule was adequately concealed from participants; the assignment variable was nonmanipulable by participants, practitioners or other decision makers; or the assignment variable was measured with random error.

3.

Confirmation tests:

Reporting the distributions of baseline characteristics above and below the cut-off.

Histogram of assignment variable demonstrating no data bunching around threshold, hence no manipulation of treatment status.

Addition of a phase in which intervention is not present (e.g., by estimating the preintervention relationship between assignment variable and outcomes at baseline), or among a “pure control” group that is not offered treatment assignment, to verify functional form and to adjust for nonlinearities in the relationship.

4.

Falsification tests:

Analysis of “placebo discontinuities” at different thresholds of the forcing variable showing no other discontinuities in the assignment variable within the window of interest.

Addition of a nonequivalent outcome, or “placebo outcome”. That is, assessing the effect on a second outcome variable that the intervention should not influence, as a falsification exercise.

5.

Reporting multiple specifications to check robustness. This includes testing the robustness of the results to the use of: nonparametric methods using different bandwidths; and parametric methods using different windows for the assignment variables and different polynomial orders when modelling the relation between assignment and outcome variables.

6.

Demonstration that there is no “treatment confounding”: that is, the allocation rule is only used to assign interventions of interest, and not additional interventions which may affect the outcomes of interest in the study.

7.

In the case of prospectively designed RDD (QEs), bias in selection of participants into the study can be addressed by controlling for baseline imbalances, similar to RCTs. Bias in outcomes measurement may be minimised by use of methods to account for biases, such as blinding of intervention or measurement of outcomes using direct observation rather than participant self-report.

8.

In the case of retrospectively designed RDD drawing on observational datasets (natural experiments), bias in selection of participants into the study may be problematic, but bias in outcomes measurement (e.g. due to self-reported outcomes) less so.

On the basis of this review, we propose signalling questions within relevant bias domains of ROBINS-I. ROBINS-I is grouped around seven bias domains as explained further in Sterne et al. (2016). Here we propose modifications (highlighted in bold in Table 10) for two domains: bias due to confounding, and bias in selection of the reported result.

Risk of bias due to confounding includes questions about the definition of the assignment scale (continuous or discrete), the specification of the relationship between assignment and outcome, treatment confounding and the assessment of balance. Thus we might expect credible RDDs to: use a continuous variable for assignment; use an appropriate method to examine the relationship with outcomes (e.g., nonparametric kernel regressions) as well as report sensitivity analysis; report a graph of the discontinuity to show no other discontinuities in the assignment variable within the window of interest; report a histogram (kernel density plot) of the assignment variable to spot bunching around the threshold which might be indicative of manipulation; and report baseline data to assess the preintervention relationship. Some of these reporting requirements, such as graphing the discontinuity, have only become common in RDD reporting latterly. For reporting bias, we may expect authors to present multiple findings for all outcomes prespecified and including multiple bandwidths.

5 DISCUSSION AND OPPORTUNITIES FOR FURTHER RESEARCH

The main results of this paper have shown that a wide range of risk of bias tools and internal study replications exist and may be drawn on in designing future internal study replications and risk of bias tools. In particular, drawing on a specialised and more in-depth systematic search of impact evaluations from international development, it highlights that a much larger body of evidence exists in this field than was previously known. Here the review identifies that at least 10 within study comparisons from international development exist, more than doubling the number previously reviewed. We consider that these studies can be used in further developing the accuracy of bias assessments and provide the grounds on which they can be evidenced.

A synthesis of findings from primary studies and existing reviews of internal study replications from international development has highlighted that nonrandomised estimators can provide estimates very similar to randomised estimates, but that they do not always eliminate bias. In particular, evidence suggests that reductions in bias may occur under conditions where the outcome or selection process can be simply or credibly modelled and when estimators use a richer degree of contextual and unit information. However, studies such as Lamadrid-Figueroa et al. (2010) and Galiani and McEwan (2013) also empirically demonstrate that randomised and nonrandomised estimators can provide very different results given the sensitivity of to the distributional effects of programmes. This point has numerous important implications. First, this highlights that during a synthesis of a literature, differences in findings between randomised and nonrandomised designs evaluating similar programmes should not necessarily be attributed to study bias. Second, it highlights the importance of experimental evidence looking beyond ATEs (examining the heterogenous effects of programmes). Third, it also empirically supports previous discussion by Cook et al. (2008) noting the need for internal replications to improve the similarity of the causal quantities reported by different estimators.

In this review, we have observed a number of strategies applied in internal replications featured in the international development literature to increase the similarity of randomised and nonrandomised estimators’ causal quantities. For example, studies used subsamples of randomised treatment groups or matching nonexperimental comparison groups with experimental control groups. We note that another innovative example featuring in the literature outside of international development includes applying transformations to estimators to increase their comparability. Gill et al. (2016) develop a method for converting nonexperimental estimators into an estimate that can be compared with experimental ITT estimates. This point highlights an interesting opportunity for future research in internal replication studies to further engage: comparing the findings of different techniques for adjusting estimators.

Despite the growth of the internal replications in international development, the total number of studies remains relatively small. This provides an impetus to continue to grow this existing body of evidence with new applications and studies. However, as highlighted in arguments by Smith and Todd (2005), future research would benefit from exploring the implications of context, as well as different statistical applications. Further research understanding whether aggregate conditions and target populations effect estimators bias seems intuitively of high importance to international development, which often works in very diverse contexts with specific and marginalised populations. Other examples of factors could relate to understanding the implications of different methods of survey administration for bias and the management of missing data. This reflects the context of limited resourcing driving demand for cost-effective methods of data collection and common issues of low levels of recorded data in many L&MICs.

Further research would also benefit from exploring whether specification tests can be used to “rule out” biased estimates. For instance, assessments by Heckman and Hotz (1989) of the predictive accuracy of diagnostics tests of pre-programme alignment of participant's characteristics first showed promise in helping to improve our ability to detect a biased estimator. Findings from Glazerman et al. (2003) further indicated that specification tests can help to eliminate the poorest performing non-randomised estimates. To date, the internal replication literature from L&MICs provides relatively little analysis to contribute to the understanding of the efficacy of diagnostic tests. In some cases, the L&MIC studies do use diagnostic tests—for example, Chaplin et al. (2017) assess the balance of variables of in their final propensity score models. However, these studies do not evidence whether such diagnostics can be used to predict consistently when biased estimators will occur.

Examples of more contemporary diagnostic tests that could also be considered include whether researchers can accurately detect sensitivity of nonrandomised estimators to unobserved biases (e.g., see Arceneaux, Gerber, & Green, 2010).

Finally, this review has further highlighted the need for risk of bias approaches to be further developed for other credible nonrandomised methods, due to the increasing use of these approaches including QEs and natural experiments in programme evaluation, as well as their incorporation in systematic reviews. Here we have suggested development of an approach for RD. Further work should take place to pilot this approach and, importantly, develop algorithms for reaching transparent “risk of bias assessments” for particular domains. Further work is also needed on other approaches, for example, statistical matching approaches and DIDs, popular methods of programme evaluation for which careful risk of bias assessment is not commonly undertaken in systematic reviews. This work can also draw on existing empirical studies of robustness (e.g., Dong & Lipsey, 2014; Rosenbaum, 2010).

Correspondingly, future research related to reviews of internal replication studies could consider an updated method-orientated review of nonrandomised matching studies in this literature. Chaplin et al. (2018) serve as an example of such a study on RDD. Other methods that may warrant reviews include panel regressions (such as DID), selection models, nonrandomised IVs and ITS analysis. We also note that the results of this study highlight that broader efforts to identify all existing internal replication studies should consider more specialised systematic search strategies within particular literatures (so to overcome the issue of a lack of systematic indexing of this evidence). With respect to international development, future updates of this review would be particularly warranted as systematic searches on databases such as the 3ie impact evaluation repository continue to be updated and expanded.