PROTOCOL: Early childhood education programs for improving the development and achievement of low-income children: a systematic review

School readiness

Nearly all ECE programs attempt to increase the “school readiness” of children. Although school readiness is defined in various ways, “experts agree that readiness is a multifaceted concept that goes beyond academic and cognitive skills” (Karoly, Kilburn, & Cannon, 2005, p. 8).

The Head Start Early Learning Outcomes Framework (see Table 1) provides a detailed description of school readiness, divided into five domains:

Approaches to learning (including behavior),

For each domain, a group of subdomains is provided, each comprising a set of goals and age-appropriate milestones/benchmarks for children from birth-to-five.

Objectives and expectations of ECE programs

Zigler and Styfco (2010) report that Head Start's initial goals were based on lessons learned from the Early Training Project, an ECE program that targeted children from “extremely poor inner-city homes headed by parents with weak education and occupational status” (p. 6), namely that (a) brief intervention cannot forever change the life of a poor child, (b) early enrichment and parental involvement can boost intellectual performance of children at risk for school failure, and (c) children's IQ gains fade over time. In fact, cognitive improvements were only a tertiary objective of the initial Head Start program, as listed in what is referred to as the Cooke Report (U.S. Department of Health, Education, and Welfare, 1965).

Proposed theories linking ECE participation to long-term outcomes (and impacts)

A number of theories of change have been posited to explain how ECE participation could lead to long-term positive outcomes and impacts. Woodhead (2004) summarizes the three most common:

Direct effects, wherein ECE participation “cause[s] a permanent change in children's cognitive functioning” (p. 33). In order for this theory to be validated, children's immediate cognitive gains are/should be sustained (and built upon) as they age.

OPEN IN VIEWER

Figure 1

Logic model of the school competence theory

Cunha and Heckman (2007) posit another model (referred to as the “skills beget skills” model) that is also sometimes referenced in the ECE literature and shares some similarities with the direct effects model, wherein “the skills produced at one stage [of a child's development] augment the skills attained at later stages” (p. 35). This is an economic model of skill formation that is based on six “facts” (perhaps better described as “assumptions”) established from the authors' review of the empirical research. The model views the skill formation process in multiple stages, each corresponding to “a period in the life cycle of a child.” It relies on three principles: first, “that the skills produced at one stage augment the skills attained at later stages” (referred to as “self-productivity”); second, that “skills produced at one stage raise the productivity of investment at subsequent stages” (referred to as “dynamic complementarity”); and third, that “together, dynamic complementarity and self-productivity produce multiplier effects which are the mechanisms through which skills beget skills and abilities beget abilities” (Cunha & Heckman, 2007, p. 35). Magnuson and Duncan (2016), however, conclude that “the hypothesis of dynamic complementarity in early childhood currently rests on a thin empirical base” (p. 124; see also Howard-Jones, Washbrook, & Meadows, 2012; Lubotsky & Kaestner, 2016).

Most research on the effectiveness of ECE programs does not discuss the underlying program theory (or theory of change) that links program participation to a reduction in the achievement gap. Instead, it generally looks for positive changes in various measures without consideration of the specific causal mechanism that might explain them.

Limitations of prior systematic reviews

Even among those 13 reviews that adopt a systematic approach, one or more weaknesses either threaten the validity of the findings or limit their usefulness (see Appendix B for a discussion of findings from these reviews).

First, those reviews with meta-analyses typically report one average effect size for a particular domain, combining effect sizes for a wide range of outcome measures (e.g., Manning, Homel, & Smith, 2010 combined, among other things, language and math test scores, school grades, and IQ test scores into one cognitive measure) and, often, without regard for the age of the child when measured. Camilli et al. (2010) combined “intelligence and cognitive/reading achievement domains…into a single “cognitive” domain” (p. 592). Although not explicitly stated, we presume that the calculation of the average effect size of the cognitive domain involved the combination of outcomes measured immediately after program completion, throughout primary school, and even up to and including secondary school.

This raises the potential for the “apples and oranges” problem; that is, the combination of disparate outcome measures, populations (e.g., demographic composition and ages of those evaluated), and/or interventions, which threatens the validity of the review. As Sharpe (1997) argues, in meta-analysis, it is important to strike a “balance between too broad or to narrow a focus” (p. 884). Many reviews, however, seem to paint with too broad a brush. The problem with this approach is twofold. Not only does the combination of apples and oranges potentially threaten the validity of the review, it may also obfuscate the variation in the changes across time and outcomes, thereby, limiting the usefulness of the findings. For example, one average effect size for a broad domain such as cognition, ignores that, over time, effects appear to diminish. It also ignores that there is variation in the size of the immediate effect and its general fade out. For example, McKey (1985) find modest variation in immediate effects between tests of IQ, achievement, and school readiness, but, strikingly, substantial differences in how the size of each of these outcomes decreased over time.

We think, therefore, that the better approach is to use more narrowly constructed domains, as described by the What Works Clearinghouse (see “Synthesis Procedures and Statistical Analysis,” below).

In addition, but less importantly, these reviews are not up-to-date and typically do not provide a full picture of ECE effectiveness. Of the 13 reviews mentioned above, only one (McCoy et al., 2017) includes studies conducted in the past 5 years, and even this one examined only medium- and long-term outcomes. None of the reviews include the most recent follow-up to the evaluation of Tennessee's pre-K program, which found negative overall effects for ECE participation at third grade.

These reviews are also typically narrowly focused, often considering outcome measures in one domain and/or in the short-, intermediate-, or long-term only. Only six reviews examined a comprehensive set of outcome measures, across a variety of domains and including those measured immediately after program completion and into adulthood. The result is that while research reviews on each of the outcome domains of interest and in the short-, medium-, and long-term (including adulthood) have been completed, the findings must be collected piecemeal, with each based on analyses using different sets of studies with varying methods for identifying and screening research and for evaluating the quality of the included studies (with substantial variation in the rigor of these evaluations).

Contribution of this review

Unfortunately, there is sharp disagreement among experts (and policy advocates) about the effectiveness of ECE programs.

ECE supporters argue that early intervention can compensate for the conditions that, at school entry, put disadvantaged children behind on various developmental dimensions—with the gap increasing as school becomes more demanding. On a selective basis, they point to various studies (e.g., the Perry Preschool Project and the Abecedarian Project) that show lasting gains in educational attainment, employment, and earnings, and substantial reductions in out-of-wedlock births and criminal behavior.

Naysayers argue that the ECE intervention is not strong enough to counter other forces in the child's life (such as family and neighborhood) and that these other factors are much more important to the child's cognitive and socioemotional development. They point to studies (e.g., Head Start Impact Study, Westinghouse Report) that show no or small effects or whose effects seem to disappear in a few years (“fade out”).

There is a certain selectivity in the way past studies are used to argue whether ECE programs “work,” rather than more balanced assessment of whether they can narrow the achievement gap to a socially significant degree. For example, Elango et al. (2015) claim that their review “organizes and synthesizes the literature on early childhood education” (p. 2), but includes evaluations of only 11 ECE programs (out of over 50 of which we are aware), one of which is a review of a program in Norway, that either have positive findings or, if they do not, can be easily criticized (in two of the studies, they turned nonstatistically significant findings into positive findings and increased the magnitude of already-positive findings).

Wise policy in this area calls for as full an accounting as possible of what is known about the effectiveness of ECE programs, including how they are administered, if they are cost effective, whether one model is more effective than another, and which model works best for whom.

Setting our project apart from other studies will be its emphasis on reconciling the capacities—and needs—of parents with the ability of ECE programs and other social institutions to help provide what children need for optimal social and cognitive development. We expect the findings to recognize the reciprocal importance of parent–child (and, as appropriate, grandparent–child) relationships. Thus, we will also review the findings of other programs that seek similar outcomes, for example, programs like Parents as Teachers or Nurse–Family Partnership that seek to improve children's cognitive outcomes through a change in parental knowledge, capacity, and/or behavior.

Moreover, our review will attempt to fill a void in the ECE literature by providing an up-to-date synthesis of ECE's effects on a wide range of outcome measures, across cognitive, socioemotional, behavioral, and health domains (as well as crime, teen pregnancy, and economic impacts) from early childhood through adulthood. We expect to take great care to avoid combing dissimilar effects, thus reducing the “apples and oranges” threat to the validity of our findings.

Because our report will be prepared under the auspices and rigorous standards of the respected Campbell Collaboration and with the guidance of a blue-ribbon advisory board—Jacob Klerman, Rebecca Maynard, Michael Puma, and Matthew Stagner (see Appendix D for biographical information)—we have high hopes that it will have broad policy influence.

In the current political climate, the role of the federal government in early childhood programing has become uncertain. But even if the federal government is less active in this area than previously, program expansion at the state and local level will likely continue. Although many would prefer bolder federal action, state and local action could present a greater opportunity for experimentation and innovation. In fact, many of the evaluations of the programs to be included in this review were funded and administered by organizations other than the federal government, including state and local governments, nonprofit organizations, and academic institutions.

Therefore, besides national policymakers and researchers, we will present our findings in a way that is useable by state and local planners, emphasizing the applicability of findings to particular populations, program settings, and geographical locations while also considering the applicability of decades-old findings to present day conditions. Available evidence, for example, points to different approaches for children from severely disadvantaged households as well as for children from homes where English is the second language.

Rather than using such findings as evidence that ECE programs “work,” we hope to use them as guidelines for how programs could be structured for greater effectiveness in particular situations. In doing so, we will attempt to reconcile early learning and development objectives with child care as a means for helping parents to work in the paid economy.

In addition to reaching out to federal and state officials, we would use our long-standing ties to the early childhood community to disseminate our findings.

Types of interventions

Eligible studies should evaluate center-based, classroom-style programs, defined as programs that take place in classroom-like settings in public school classrooms, faith-based institutions, or other public or private centers and that provide educational instruction directly to children, operating regularly for at least one academic year (approximately 180 days). Programs will not be excluded if they also provide ancillary services (such as parental or medical services).

Eligible programs need not provide curriculum-based instruction, be staffed by certified teachers, nor operate for a particular minimum number of hours per day. We will, however, collect information on these program characteristics during coding and, provided we have enough studies, use them as moderators in our analysis. We expect to use these moderators to learn about the plasticity of children's learning and what program variations might improve their skills and achievement.

Studies that use student outcomes to measure or compare the effectiveness of teachers or curriculum will not be included in our meta-analysis. These studies, however, might contain evidence that is relevant to our review, and we will, therefore, make note of these studies so that we might use them to interpret our findings.

Types of participants

Eligible studies should investigate the effect of an applicable intervention that serves disadvantaged children from low-income families sometime from birth to age five (with age breaks). We define “disadvantaged” or “low income” as below twice the official U.S. poverty line (or threshold, as adjusted for family size). Our preliminary research, however, found that many studies use these terms without providing a definition, or that the programs targeted children using factors other than income.

We have found that the following terms are compatible and used frequently in the literature: “eligible for free or reduced price lunch” (or “FRPL”), “low income,” “low socioeconomic status,” “poor,” and “at risk.” Studies that use these terms (or slight variations) to define the participants will be eligible for inclusion. We will base this decision on a combination of both the study author's description of the program as well as information reported about the characteristics of the enrolled students. We will capture information about the composition of the sample during study coding and use this information in moderator analyses.

When studies evaluate universal programs that serve children from all SES backgrounds, we will only include them if effects for low-income children are reported separately. When reported, however, these effects are usually the result of subgroup analyses, which are subject to issues related to sample size and statistical power. We will not, therefore, combine the results from universal programs and targeted programs but, instead, will report their results separately. We will also collect information on the percent of enrollees in these universal programs that are from disadvantaged families and, perhaps, use this as a moderator in our analysis.

We will exclude studies that target children with severe developmental disabilities, learning disabilities, or chronic medical disorders.

Types of outcome measures

Eligible studies should either report at least one immediate (or “short-term”) outcome measure or be a follow-on to a study that reported one (a “follow-on” is defined as a study that reports intermediate outcome or impact measures of a program on a group or groups that have previously been evaluated).

In addition, the outcome measure(s) reported should fall under one (or more) of three broad categories:

Cognitive, such as school performance; language, reading, and math skills; grades; and school performance;

More precise outcome domains are described under “synthesis procedures and statistical analysis” below.

Types of study designs

Eligible studies should investigate the effectiveness of participation in ECE programs compared to nonparticipation (the counterfactual) and report primary research findings (as opposed to being cross-study analyses or commentary) with at least one eligible outcome measure.

We do not intend to use study design alone as a criterion to make inclusion/exclusion decisions; rather, our plan is to rigorously review each eligible study for threats to causal validity, as described below (see Section 3.5), and exclude those that are rated as Insufficient Causal Validity. We think this is the best approach because it should provide a more precise justification for the decision to include or exclude each study.

In order to be included, a study must employ either a comparison group, provide a comparison time series, or use a control variable that is related to the extraneous variable.

We will include three-pronged studies that investigate the effectiveness of supplemental instruction or tutoring for children already in preschool classrooms only if one of the group's does not receive any preschool services. Without a no-preschool group, these studies are comparative effectiveness evaluations and, as such, do not measure the effectiveness of participating in ECE. As with studies that use student outcomes to measure the effectiveness of teachers or curriculum, we will use these studies to interpret our findings and identify what, if any, programmatic variations work better than others and what this means about the developmental plasticity of children.

Other criteria

Eligible studies should have appeared after 1960 and been written (or be available) in English. In addition, these studies should have been conducted in the United States or, because of their social and economic conditions, and research and evaluation infrastructures, a Western European country, as defined by the World Bank (i.e., Austria, Belgium, Denmark, Finland, France, Germany, Greece, Iceland, Italy, Luxembourg, Norway, Portugal, Spain, Sweden, Switzerland, the Netherlands, and the UK), or one of the following English-speaking countries: Australia, Canada, or New Zealand.

Application of the criteria

We expect to apply our inclusion criteria in two stages. In the first, the title and abstract for each reference will be screened using the following set of general questions: 1.

Is the study reported or available in English?

Given the large number of expected results, we will have two reviewers independently double screen the first 15% of references in order to establish inter-rater reliability. Reliability will be assessed by calculating Cohen's κ for the reviewers' responses to question eight. If inter-rater reliability meets or exceeds 0.80, the remaining references will be split in half with one half assigned to each reviewer. If κ falls below 0.80, discrepancies will be resolved by a third reviewer. In this case, another sample of references will be double screened and inter-rater reliability reassessed (this iterative process will continue until satisfactory inter-rater reliability is achieved).

Alternatively, if a κ of 0.80 seems impractical after a number of iterations, we would double screen all remaining studies and exclude only those for which both reviewers agree are not relevant.

In order to screen studies efficiently, we will immediately exclude a study when it fails to meet any one criterion, even if the exclusion happens after the first question of the screening tool. Studies that do not meet the criteria in stage one will be excluded from the review. In the event that there is insufficient information to answer one (or more) of the screening questions, the question will be marked as “cannot tell” and the study will remain eligible for the next stage of screening (the study may be excluded later, however, if the missing information is found during the full-text screening and it no longer meets the criteria in stage one).

The studies that remain eligible after stage one will be subjected to a full text screening using the following set of more focused questions: 1.

Was the study conducted in the United States or one of the following countries?

As with the first stage, the first 15% of the references will be independently double screened and Cohen's κ will be calculated to measure inter-rater reliability using the responses to question ten. If κ is satisfactory (0.8 or greater), the remaining references will be equally distributed among the reviewers. If not, a third reviewer will resolve the discrepancies and the process will repeat until satisfactory reliability is achieved. Or, if the desired κ does not seem attainable, we will double screen the remaining studies and only exclude studies that both reviewers agree should be removed.

In the event that there is insufficient information with which to answer one (or more) of the screening questions, the answer will be marked as “cannot tell” and the study will remain eligible for inclusion. Studies that meet the criteria in both stages will be assessed for risk of bias (see below).

If it is later determined (during the risk of bias assessment or full coding) that a study was improperly screened, it may be excluded from the analysis, and an explanation will be provided in the review.

Database searches

The bulk of our electronic database searches will be conducted using EBSCOhost, a collection of approximately one hundred indices and databases. We will individually search each of the following databases using the EBSCOhost interface (this is the only interface available to us through the University of Maryland libraries with which to search these databases):

Academic Search Complete

Entries in these databases and indices are well catalogued and include reference information and an abstract for each. Thus, we have designed the following search phrase, which concatenates keywords using the boolean operators “OR,” and “AND” to search only the abstracts for each entry:

(preschool OR pre-school OR prekindergarten OR pre-kindergarten OR pre-K OR preK OR “Head Start” OR “early childhood education” OR “early childhood care and education” OR “early N5 education” OR “nursery school” OR ECCE OR ECE) AND (experiment* OR quasi-experiment* OR quazi-experiment* OR “random* control* trial” OR “random* assignment” OR RCT OR trial* OR “control* study” OR “control* studies” OR “control* design*” OR “control* trial*” OR “control* group*” OR “control group design” OR “trial registration” OR “random* allocat*” OR evaluation OR “comparison group” OR outcome* OR impact* OR effect*) AND (achievement OR behavior* OR behaviour* OR cognit* OR intelligence OR IQ OR knowledge OR “school readiness” OR socioemotional OR socio-emotional OR socialemotional OR math* OR literacy OR vocabulary OR “language skills”) AND (child*) AND (poor OR “low income” OR “low-income” OR disadvantage* OR “at risk” OR FRPL OR “eligible for free school meal* OR universal* OR “state pre-K”)

We will not use date or language of publication as search limiters; rather, we will screen studies for date and language relevance during stage one of our inclusion criteria.

In addition to our EBSCOhost search, we plan to search individually the following databases:

JSTOR

For Web of Science, we will use a similar procedure to which we will use for EBSCOhost, except that we will apply the search phrase to the abstracts, titles, and keywords (referred collectively to as “topics” in Web of Science) of the references they contain instead of the abstracts only.

The search tools of the other three, however, are either limited to shorter search phrases or do not allow for (or, in some cases, discourage) the search term to be applied only to abstracts, and thus we will use the following, shorter boolean search phrase when searching them:

(preschool OR prekindergarten OR “nursery school”) AND (poor OR “low income” OR disadvantage*) AND (effect* OR outcome* OR impact*) AND (achievement OR cognitive OR health OR soci?emotional) AND (experiment OR quasi-experiment OR quazi-experiment OR random*) NOT (autis*)

If a search returns no results, we will search again using only the keywords contained in the first set of parentheses in the longer search term.

Other online searches

Our search will include reviewing the websites of research firms, advocacy groups, institutions of higher learning, and government agencies for technical reports, white papers, and conference proceedings. We will search the following websites:

Abt Associates

We will also search by hand the conference proceedings from the following United States and international conferences to identify those not yet indexed in the commercial databases.

Annual International Conference on Childhood Education (https://10times.com/childhood-education)

We will also use Google Scholar to search for additional studies, using various combinations of the keywords in our search phrase (search terms cannot exceed 32 words when using Google Scholar, and thus we cannot use our longer search phrase). Our preliminary searches yielded more than 100,000 results, which is far more than we can realistically screen, and thus we will limit the number of results to those contained in the first 20 pages (this appears to be a common approach in other systematic reviews, including Campbell Collaboration reviews).

Handsearches

We will conduct handsearches of 10–15 high-priority journals. The journals to be searched will be identified using the results from our database searches and based on their availability at the University of Maryland's libraries. Those journals available at the library and containing the greatest number of references returned in our results will be considered high-priority.

In conducting our handsearches, we will review the table of contents of each volume since 1960, record in Zotero any studies that match our search criteria, and apply our inclusion criteria on-the-spot (rather than adding to our reference database and applying at a later date).

Previous systematic reviews

During the preparation of our title registration and the description of prior reviews (see Appendixes A through C), as well as subsequent grant writing activities, we established a list of previous systematic reviews, meta-analyses, and literature reviews conducted in the area of ECE. We will use the references of these reviews to identify any studies that may have been missed during the first two stages of the search. Any studies that may be related to our topic will be included in our reference list for future screening.

Trial registers

The Campbell Collaboration provides a list of 22 trial registers. We will review these registers for any ongoing, concluding, or future evaluations of ECE programs. While these studies may not be completed in time to be included in this review, they may be useful in future updates.

Study rating process

Those studies that meet our criteria for inclusion will be subjected to an assessment of their causal validity prior to full study coding and, based on this review, will be assigned an overall rating of either Strong Causal Validity, Sufficient Causal Validity, or Insufficient Causal Validity.

To make these decisions, we have created a causal validity assessment tool. This tool is a modified version of the What Works Clearinghouse's (WWC) rating standards, and it draws from a number of additional sources, that is, the Cochrane Handbook for Systematic Reviews of Interventions (Version 5.1.0) (Higgins & Green, 2011), The Handbook of Research Synthesis and Meta-Analysis (Cooper, Hedges, & Valentine, 2009), The Maryland Scientific Methods Scale (Farrington, Gottfredson, Sherman, & Welsh, 2002; What Works Centre for Local Economic Growth, 2016), as well as our own work (Besharov, 2016).

Our rating tool comprises two standards, each composed of two or more criteria, meant to examine broadly the degree of causal validity in each study. These standards evaluate the extent to which the methods used by the researchers appear to account for threats to causal validity that may be the result of selection bias or attrition (as defined below).

As described below, each standard will be rated according to the extent to which the study satisfies the set of criteria that comprise it. Accordingly, the standards will be rated (borrowing language from the WWC) as either Meets Standard without Reservations or Does Not Meet Standard, with the exception of the selection standard, which can also be rated as Meets Standard with Reservations.

When each standard has been rated, an overall study rating will be assigned as follows:

Strong Causal Validity, if the study meets each of the standards without reservations. The only studies eligible for this rating, therefore, would be properly conducted randomized control trials (RCTs; likewise, only RCTs are eligible for the WWC's highest rating). Studies that evaluate and/or compare multiple variations of a specific program (e.g., comparing full-day to half-day programming) would be eligible for this rating, provided that the participants were randomly assigned and that there is a nonprogram group that did not receive services from the program.

One caveat to these rating rules is in the case of studies that use an age cutoff RDD. These studies typically use strict age cutoffs to assign children to either a program or nonprogram group. When limiting the comparison to children just above and just below the cutoff, the assignment is treated “as if” randomization has occurred. These studies, however, may have too few children near the cutoff to generate precise estimates, and thus the researchers may include children further from it to compensate. Presumably, the further children are from the cutoff, the more dissimilar they are, which threatens the causal validity of the findings. Indeed, Thistlewaite and Campbell (1960) argue that as observations further from the cutoff are added, the inferences from RDDs become “more and more suspect.” Moreover, these results may not be directly “directly comparable to either intent-to-treat or treatment-on-the-treated estimates from experimental studies” (Duncan & Magnuson, 2013, p. 119; see also, Gibbs, Ludwig, & Miller, 2011; Lipsey, Weiland, Yoshikawa, Wilson, & Hofer, 2015).

Given these two limitations, we will include age cutoff RDDs, but consider them separately, provided they meet the attrition standard of the Sufficient Causal Validity rating as outlined above.

We expect to use these overall study ratings to reduce the risk of bias in our analysis in two ways. First, we will exclude any studies that are rated as Insufficient Causal Validity, because, if included in our analysis, their findings, perhaps attributable to one or more of the threats to causal validity, might bias our results. Second, with the remaining studies, we will use the study rating as a moderator in our analysis. Provided we have enough studies, we plan to conduct sensitivity analyses in order to decide whether the results from studies with Strong Causal Validity should be combined with those of Sufficient Causal Validity. We anticipate that the results from these analyses will suggest that there is a statistically significant correlation between study rating and effect size and that we would therefore conclude that these two groups should be analyzed separately.

Moderators

As mentioned above, we consider contamination and implementation weaknesses to also be threats to causal validity, but we have decided to use them as moderators in our analysis instead of as standards or criteria in our study rating process. We think that this approach is best because, based on our review of a number of systematic reviews, these threats are not commonly assessed during the quality appraisal process in other systematic reviews. Moreover, we are worried that the effect of using them as standards or criteria might be to systematically exclude studies that have downward biases in their impact estimates. We think that it would be valuable to know if there is evidence that, despite the likelihood of downward biases in the impact estimates, a program found some evidence of effect. Our plan, therefore, is to treat contamination and implementation weaknesses as moderators, rating each threat during coding as either low, medium, or high, as described below. We plan to review the effects of this approach during the analysis stage.

For contamination, we will rate the threat overall as well as examine and rate four distinct types: substitution bias, crossover, diffusion of treatment, and contagion (this applies only to studies that use a nonprogram group).

Software

We will use R to calculate effect sizes (and their 95% confidence intervals) for the outcomes reported in the study findings, to combine effect sizes, and to perform moderator analyses.

Effect sizes

The majority of ECE studies report some measure of academic achievement (or other outcome) measured as a continuous variable, including reading and/or math test scores. For these, we expect to use the standardized mean difference as our index of effect, and although both Cohen's d and Hedges' g estimates tend to be biased upwards, the latter provides a more precise measure of effect when dealing with small sample sizes and when the SDs of the groups in the study are dissimilar (Ellis, 2009). Since many of the early studies of Head Start (e.g., Krider & Petsche, 1967; Larson & Olson, 1968; Nummedal & Stern, 1971) had small overall sample sizes and/or had treatment and comparison groups that were dissimilar in size (and are therefore likely to have dissimilar SDs), Hedges' g should produce a more conservative and accurate estimate under these conditions, and thus we plan to use it as our primary index of effect. This should produce a more conservative and more accurate estimate under these conditions. Furthermore, this is a common approach in meta-analyses of ECE programs, and the preferred option in WWC reviews (U.S. Department of Education, 2014a).

To calculate Hedges' g, we will use the formula proposed by Hedges (1981), and therefore, in order to be included in the meta-analysis, studies must report means, sample SDs, and sample sizes for each outcome measure and for each group in the study.

When studies report outcome measures that are dichotomus (e.g., placement in special education), we will use the odds ratio as our measure of effect. Occasionally, studies may vary in how they measure certain outcomes, sometimes measuring a given outcome construct as dichotomous and sometimes as continuous. In these situations, we will convert the dichotomous effect size first to Cohen's d and then to Hedges' g, as described in Borenstein et al. (2009).

Missing data and other issues

Although we expect missing data to be an issue in only a small percentage of studies, when the necessary information (i.e, means, SDs, and sample sizes) is not included in a study, we will not impute missing values (except for sample sizes when sufficient information is reported), but, instead, contact the study authors to request the missing data. If we are unable to obtain the necessary statistic, we will not include the result in our analysis. We would, however, discuss the finding in our report, especially as it compares to our results. In the event that an outcome is reported as an effect size in another form (e.g., r or Cohen's d), we will perform the necessary transformation using the appropriate conversion formula as described in Borenstein et al. (2009) (chapter seven of Borenstein et al., 2009 details how to perform transformations of effect sizes, converting either correlational or binary data to Cohen's d and from Cohen's d to Hedges' g).

In our initial searches, we did not find any evaluations using prepost, comparison-to-self designs. Should one or more study using this design be included, however, we would calculate an effect size using Glass' Δ—the typical practice whereby the mean of the outcome measure posintervention is subtracted from the mean of the measure preintervention and is then divided by the SD of the preintervention measure (Lipsey & Wilson, 2001). We will not combine these effect sizes with those calculated using Hedges' formula (or with odds ratios) because the effect sizes are dissimilar. Moreover, effect sizes from comparison-to-self designs tend to be significantly greater than for studies using a comparison group (which may be the result of autocorrelation), and thus the effect sizes from these different methodologies should not be combined even if the statistics were similar (Beretvas & Chung, 2008; Durlak, 2009).

If the comparison-to-self design does not include both a pre- and postintervention measure (e.g., postonly designs), we will include the findings in our review, but not in the meta-analysis, and they would be mentioned in our discussion of results as either supporting or contrasting evidence.

We are not aware of any evaluations of ECE that use a cluster-randomized design; however, should one be included in our analysis, we would calculate its effect size using the WWC's guidelines for doing so. In addition, we would apply the clustering correction described by the WWC to adjust the standard error and p-value for each effect size.

Outliers

Before data are combined, we will arrange each of the effect sizes and their confidence intervals by outcome domain (see below), and visually inspect them for apparent outliers. For each identified outlier, we will calculate the sample-adjusted meta-analytic deviancy (SAMD) statistic developed and described by Huffcut and Arthur (1995) to measure the degree to which the outlier effect size varies from what should be expected by chance. Given that the SAMD approximates a t distribution, any result of 2.0 or greater is considered large (about two SDs), and hence we would review the study for any apparent plausible explanation (e.g., the outlier is a parent-reported measure when all others are teacher-reported; Zalta, 2011). When a plausible explanation is identified, the outlier will be removed from the analysis. In the event that no explanation is obvious and we can identify no errors in our data entry, we will reduce the extremeness of the outlier (and, therefore, its effect on the average effect size) by setting the effect size to three SDs above the mean effect size, including outliers (referred to as either “Winsorizing” or “top-coding”; Cooper, 2017; Lipsey & Wilson, 2001).

Outcomes

As mentioned earlier, our review is a comprehensive look at the effectiveness of ECE programs across a broad array of outcome measures and ages. To reduce the risk of comparing apples and oranges, we will group outcome measures by age at measurement (using brackets) and then by outcome domain.

We expect to use 10 age brackets: (a) infant/toddler, 0–2 years old; (b) preschool, 3–4 years old; (c) kindergarten, 5–6 years old; (d) first grade, 6–7 years old; (e) second grade, 7–8 years old; (f) third through fifth grade, 8–11 years old; (g) middle school, 11–14 years old; (h) high school, 14–18 years old; (i) young adult, 18–25 years old; and (j) adult, 26 years old and over (we will reconsider these age brackets as we identify studies during the review).

We have identified the following domains and subdomains in which we intend to group effect sizes: (the first eight are identified by the WWC as the primary outcome domains for ECE research (U.S. Department of Education, 2014a), and we would use their definitions when categorizing outcomes).

Cognitive (general)

When a domain comprises subdomains, we will consider combining the effect sizes across the subdomains to generate an overall domain average effect size. The decision to do so will depend on a number of factors, including the number of effect sizes, the type of effect sizes and whether they can be combined (e.g., as mentioned earlier, odds ratios should not be combined with Hedges' g effect sizes), and the degree to which the outcomes appear to be sufficiently similar.

Risk of bias

As mentioned above, we will assess the degree of causal validity in each of the included studies and will combine the effect sizes for those rated as Strong Causal Validity separately from those rates as Sufficient Causal Validity. We will only combine the effect sizes across these groups if the results from our moderator analyses and sensitivity analyses reveal no statistically significant differences in the effect sizes related to this rating.

Combining effect sizes

When a study reports more than one outcome in a given domain (or subdomain) at a particular age, we will average the effect sizes using the inverse-variance weight function (Dong, Maynard, & Perez-Johnson, 2008) in order to reduce the likelihood of our results being influenced by “data mining.” We prefer this approach over the WWC's method of using a simple, unweighted average because the inverse-variance weight accounts for variation in the standard error of each effect size that may be due to differences in sample sizes or measurement error, and therefore seems more accurate.

Likewise, when more than one study reports the same outcomes for the same sample (i.e., they use the same dataset), we will calculate the average effect size across these studies for the common outcomes and use this average in our analysis.

We will then calculate, as is typical practice, the average across-study effect size for each domain (and subdomain) using random-effects inverse-variance weights and will report 95% confidence intervals for all mean effect sizes.

We will consider using robust variance estimation (RVE) if we ultimately perform a meta-regression and have a sufficient number of studies to apply the method. Hedges, Tipton, and Johnson (2010) caution that RVE “may yield accurate results with as few as 20-40 studies.” We are therefore reluctant to use this method unless we have upwards of 40 studies.

Assessment of heterogeneity

We expect to find substantial heterogeneity in the programs and populations included in our review, and thus we will use random-effects meta-analysis to combine effect sizes and perform moderator analyses in R. Nonetheless, as is common practice in meta-analysis, we will use the Q test to determine whether the level of heterogeneity is statistically significant and the I ² statistic to describe the percentage of variation across studies that is due to heterogeneity rather than chance (Higgins & Thompson, 2002; Higgins, Thompson, Deeks, & Altman, 2003).

Random-effects meta-analysis requires a measure of between-study variation, τ ², to calculate average across-study effect sizes, and, although there are many methods for computing τ ², each with their own advantages and disadvantages, we expect to use the approach recommended by the Cochrane Statistical Group. This involves using the Paule-Mandel method to estimate τ ² and the Hartung-Knapp adjustment to calculate confidence intervals for the resulting average across-study effect sizes (Knapp & Hartung, 2003; Veroniki et al., 2015; Veroniki & Salanti, 2013).

Moderator analyses

Provided we have enough studies and that the information within the studies permits, we will examine the following moderators for their influence on effect sizes either through subgroup analyses using random-effects assumptions or by using meta-regression, as detailed by Higgins and Thompson (2002):

Characteristics of the program (e.g., whether the intervention is a demonstration project, teacher qualifications, use of curriculum, ancillary services provided);

We expect that the number of studies that provide effect sizes for a given domain and age bracket will be small, and therefore our ability to conduct these moderator analyses may be limited.

Sensitivity analyses

We will conduct sensitivity analyses to compare the results of studies rated as Strong Causal Validity to those rates as Sufficient Causal Validity in order to decide whether they should be combined. In addition, we will explore whether and to what extent our decision to Winsorize outlier effect sizes may have affected our findings.