PROTOCOL: The effects of K-12 school enrollment interventions in developing nations

Criteria for inclusion and exclusion of studies in the review

For this project, we will only include those studies that have the following characteristics: (1)

Randomized controlled trials or quasi-experimental evaluations with evidence of equating. Our review includes evaluations that randomly assign entities (at any level) to intervention or control conditions. In the studies we have examined to date, the control or comparison group (the counterfactual) is either a condition that receives no intervention or receives the usual or standard practice. Only well-implemented randomized experiments can provide statistically unbiased estimates of an intervention's effect (Boruch, 1997). Although there are comparatively very few randomized experiments in developing nations compared to other types of evaluation studies (Glewwe and Kremer, 2006), there has been a “randomized revolution” in development work in recent years (e.g., Duflo and Kremer, 2005). However, because randomization is not possible in certain evaluation scenarios (e.g., in retrospective evaluations or when the program has already been implemented), we will also include evaluative studies that use quasi-experimental designs, provided that they offer evidence that the comparison groups were equated. Such quasi-experiments include regression discontinuity design, propensity scores, covariate matching, and other pre or post intervention matching. Although quasi-experiments cannot control for unknown or unmeasured confounding factors, there is conflicting literature on whether the estimates from such designs approximate those from randomized experiments (e.g., Oliver et al 2008). We include both types of studies and will include study design as a moderator in our later analyses (see discussion on moderator variable analysis in later section).

Example of studies that would be included in our review

The Burkina Faso evaluation conducted by Mathematica for the Millennium Challenge Corporation (Levy, et al. 2009) is an example of an evaluation that would be included in our review. Using regression discontinuity design, the study compares the results, at the village level, on girls' school enrollment for those villages that received new school construction and other interventions (via the BRIGHT program) versus those that did not. Results indicate that girls' enrollment 16–19% in the experimental villages. The background section included references to other examples of evaluations that would be eligible for our review. For example, the Ecuador lottery study that randomly assigned families to receive cash vouchers in exchange for enrolling their children in school would be included (Lopez-Calva, 2008).

Example of studies that would not be included in our review

One study that would not be included in our review is that of Bobonis, Miguel, and Sharma (2006), who evaluated a health program that provided iron supplementation and deworming medicine to preschool age children in poor urban areas of Delhi. Although this was a randomized evaluation in a developing nation that reported effects on school absenteeism, we would not include it in our review because it does not evaluate a K-12 strategy, but focuses on the preschool population. Another example of a study that would not be included in our review was conducted by Lockheed and her colleagues (1986) in Thailand. They studied the impact of providing textbooks to Thai students. Although there was a question of whether they used a truly equated comparison group, the study was nonetheless excluded because it did not include any outcome of enrolment, attendance, dropout, or persistence, but exclusively focused on academic achievement.

Search strategy for identification of relevant studies

Our goal is to identify both published and grey literature. Many of the databases in (1) below include grey literature (e.g., ERIC). The British Library indexes conference proceedings and makes these available in its “Integrated Catalogue.” Web searches will often identify literature that is made available at websites but not published in journals. Our contact with colleagues is designed to get at more of the grey literature. To accomplish our goal, we will use five major strategies: (1)

Electronic searches of bibliographic databases. Researchers will use available online resources and databases at WestEd, the University of Pennsylvania, Boston Public Library, and the University of Massachusetts, including ERIC, British Education Index, PAIS International/Archive, Sociological Abstracts and World Bank Documents. See Appendix A for additional databases that will be searched. We will also use the “advanced search” options in Google for broad searches of the World Wide Web.

Keyword strategies for bibliographic databases

The databases in Appendix A can be somewhat idiosyncratic. Thus, we believe the best strategy is to conduct a broad search of the available databases that errs on the side of sensitivity rather than specificity. In other words, we would rather get many titles and abstracts to sift through rather than potentially miss relevant citations because our search terms were drawn too narrowly.

We will use two different search strategies for these databases, depending on their focus. If the database is focused on education (such as ERIC), we will use broad searches that identify evaluation studies conducted in developing nations. To do this, we will use the following keywords to find experimental and quasi-experimental outcome studies: “random,” “experiment,” “control,” “evaluate,” “trial,” “impact,” “effect,” “comparison group,” “match,” “discontinuity,” “propensity” and “outcome.” Our plan is to use truncated versions of these words; for example, we will use “experiment*” to capture titles and abstracts with the word “experiment,” “experimental,” “experimenting,” “experimenter,” and “experiments.”

Second, we will combine those keywords with ones that focus the search on developing nations, such as the use of terms like “developing,” “third world,” and “impoverished” with “nation,” “country,” or “region.” In addition, we will use the names of specific developing nations, such as India, Mexico, etc., and the names of regions, such as Africa, Latin America, and Asia. If there is a geographic descriptor for country or region, we will incorporate that into our search process. We will also include keywords that focus on the outcomes of interest including dropout, enrollment, enrolment, and attendance. Such searching is an iterative process and we will modify as we retrieve studies. This strategy may produce a number of false positives, but our experience is that examining the abstracts is not time consuming and researchers can go through them quite quickly.

Wherever possible, we will limit our searches by a descriptor that indicates the grades that our review is targeting: K-12 (for example, selecting primary and secondary education, or elementary, middle, and high schools). This would have the advantage of screening out preschool and college age studies.

If the database is not focused on education (e.g., Sociological Abstracts), the above strategy must then be supplemented by something that identifies educational research. In some databases, that will be a classification code; for example, in Sociological Abstracts (Sociofile), one can limit the abstracts to those dealing with “sociology of education”. But in many of others, there is no classification code. Whether classification codes exist or not, we will use truncated versions of keywords related to the educational outcomes of interest, such as dropout, attendance, and enrollment/enrolment to try to reduce the number of false positives. Again, if there are geographic descriptors for country or region, we will incorporate that into our search process.

We recognize that specific search strategies may have to be developed for each database. What works in identifying potential studies in ERIC will not work in searching World Bank Documents. The appendices to our final review report will carefully document all keywords used for each database to permit replication.

Retrieving and Final Screening of Studies

Search methods will identify a large number of citations and abstracts. Many of these will be easily excluded as not being relevant to the proposed review. In some cases, however, they will identify potentially eligible studies. The first and second authors will review all citations and determine if the cited study should proceed to a second screening, i.e., is a potentially relevant study. If so, the full text documents of those potentially eligible studies will be retrieved and screened by the first two authors before the study can be formally included in the review. Fortunately, with the advent of the Internet and full-text electronic journal access, we will be able to rapidly retrieve the reports to do a more thorough reading. When a full text report is received, we will scan it to ensure that it includes randomization or quasi-experimental equating of study subjects and includes at least one outcome of school enrollment or persistence. If the first two Investigators do not agree on the inclusion of a particular study, it will be excluded and documented in the final report.

We have established a bibliographic reference database to maintain a log of all included and excluded studies. The log includes a field that allows the research team to document the reason for exclusion.

Extracting Information from Each Study

We have designed a preliminary instrument to guide us in recording information from each study (see Appendix B). Although the instrument contains several open-ended items, these will be collapsed when appropriate into a smaller number of categories to permit further analysis. For example, items such as “how equating was performed” can be collapsed into three or four larger categories representing the most frequent responses (e.g., discontinuity, covariate matching, propensity score, post-hoc statistical matching) and an “other” category that captures all those responses that do not fit into the most common methods of equating in this set of studies.

The instrument has items in the following areas:

Handling multiple reports on the same experiment

Note that investigators may publish several articles on the same study. Our unit of analysis is the individual evaluation and not the individual research article, and so it is reasonable to extract information from all documents to complete the coding instrument for one experiment. When reports on the same study contain conflicting information, we will employ a number of strategies, including contacting the original investigator(s) for resolution.

Criteria for determination of independent findings

Each study will be represented by a single effect size to prevent the analysis from being compromised by non-independence (multiple effect sizes from one study). Although some evaluations may report just a single outcome at one time interval, it is more likely that evaluation reports will include analyses at various time intervals and may use various constructs that reflect school enrollment and persistence. Therefore, decisions have to be made about what outcome will represent the effect size for that study.

For this review, we will keep outcomes distinct. That is, we will analyze enrollment, school attendance, dropout, and other learning or non-educational (health, behavioral, etc.) outcomes separately. The Table provides an example of how four such outcomes would be reported. We do not know as of yet how such outcomes will be reported, i.e., will they be prevalence measures (percentage of groups that enroll or attend) or incidence measures (the mean rate for some outcome of interest, such as the mean number of days attended per student). If results are varied and include prevalence and incidence rates, we will discuss the best way to report these (combine or separate out) and make such decisions explicit in our review.

We also propose to report three different analyses to handle the studies that report outcome data at various time intervals. As the Table indicates, we will report effect sizes at first follow-up (the first time interval reported), the middle effect (the middle time interval closest to the exact point between the first and longest), and the longest effect (the effect size for the longest follow-up period). If one time interval (e.g., 1 year) is reported in the study, it will be used in all three analyses. If two time intervals (e.g., 6 months, 1 year) are reported, the results will be averaged and the mean will be reported for the “middle effect.” If more than three time intervals are reported (e.g., 6, 12, 18 and 24 months), we will select the result that is closest to the exact middle. In this instance, the exact middle between 0–24 months is 12 months and would be reported as the middle.

If regression-adjusted estimates are reported for the experimental versus control groups, we will rely on them for any quantitative synthesis since they theoretically reduce statistical “noise” that may have come from chance fluctuations or randomization violations (in the case of well implemented experiments) or uncontrolled variables (in the case of quasi-experiments).

Some studies report analyses at multiple levels (e.g., Schultz, 2004), i.e., for schools or localities and for studies. Our rule is to capture this information separately, but to compute effect sizes for the analysis done at the level of assignment. So, for example, in Progresa, the randomization was done at the locality/community level, and so the main effect size will be that computed for treatment and control localities. We will code information, however, about the analyses done at the student, family or school level.

Some studies also report effects at all grade levels (e.g., Schultz, 2004). This is very important to policy and practice decision-makers. The main effect will again be computed at the larger analysis level, so that if schools are assigned to groups, the effect size will be computed for all schools in treatment versus all schools in control. However, we will record subgroup effects such as breakdowns by grade and gender.

Details of study coding categories

To ensure that we achieve good coding reliability, we will have two of the co-authors read and record information from a random sample of reports (25%). We will assess coding reliability (i.e., inter-rater agreement) by using the percentage of agreement for each item, rather than reporting a global inter-rater reliability statistic. This will avoid inflating reliability measures with study characteristics that generally achieve perfect agreement (e.g. year of publication) with those that do not. Items with lower rates of agreement (less than 80%) will be investigated to determine the source for conflict. The authors will meet to resolve disagreements and discussing coded items. We will drop those items from our database in which resolution could not be reached, as well as items that lack clear interpretation.

Statistical procedures and conventions

The data will be entered into the Comprehensive Meta-Analysis (CMA), version 2. We will use CMA to statistically combine results from the evaluations.

We will report standardized mean differences (Cohen's d), as it is a very flexible effect size metric and many formulae are available to estimate Cohen's d from information often reported in evaluation articles (e.g., statistical test data, probability levels). Forest plots will be used to display the results from the effect sizes. The plot will display, for each study, the effect size, confidence intervals and significance level. The plot will also display the same for the average effect across studies. Note that this will be reported assuming a random effects model, and the estimate will be weighted by sample size. When describing results in the text, we will report the effect size, the confidence intervals and whether the analysis indicates that the result is statistically significant.

Because of the likely heterogeneity in interventions, samples, countries, and outcomes, we will assume random effects models in our analyses, which tends to be more conservative than the fixed effects approach.

For our analyses, we will conduct tests for heterogeneity to determine if the average effect size is a good representation of the sample of studies being used in the analysis. We anticipate the heterogeneity will be present, given the variations in intervention type, nation, sample populations and the like in these development studies. Using CMA, we will confirm heterogeneity in each summary analysis (of each outcome at each of the three time intervals: first, middle and longest) through the Q-Value, which is reported as a summary indicator of the extent of variance across studies in the sample.

However, moderator analyses have to be approached carefully, as they are often based on small numbers of studies (the “small cell” problem), and that such analyses can be significant by chance if large numbers of variables are considered (the “capitalizing on chance” problem). We anticipate examining a small number of moderating variables as a source of heterogeneity, by comparing the effect sizes (assuming a random effects model) for the following potential moderators:

Randomized versus non-randomized studies. An important question is whether the estimates from randomized experiments are different than those reported in studies that used non-random assignment. This review will examine the average effect size for experiments and compare it to the average effect size for quasi-experiments.

We do not anticipate, at this time, conducting a study of publication bias. The reason is that economists conduct and report many of the studies relevant to this area. The tradition in the field of economics is to make unpublished papers available online. However, a large percentage of these are eventually published in economics journals. Therefore, papers may only be temporarily “unpublished.” We are tracking all relevant documents for an evaluation, but do not believe a comparison of “published” versus “unpublished” studies would be particularly useful at this time.

Treatment of qualitative research

It is very unlikely, because of our focus on experiments and quasi-experiments, and our focus on quantifiable outcomes that can be converted into an effect size metric, that we will uncover much qualitative research. However, we will code the presence or absence of ancillary qualitative studies, what the studies focused on, and what the main findings are. Certainly, qualitative data from the experiments and quasi-experiments will be used to illuminate three particular areas: (1) the context for the intervention; (2) the theory or mechanisms by which the program is supposed to impact the ultimate outcomes; and (3) the quality and nature of the intervention and comparison condition.

Treatment of economic data

We will report on any economic data included in the primary studies that are included in the review. This includes information on the costs of the program, any analysis of the cost-effectiveness of the intervention (e.g., the cost per child enrolled) and cost-benefit studies (e.g., the sum costs and benefits of the program). It is important that this information be linked in some way to the primary outcome studies so that it can be retrieved.