Statistical Power for Random Assignment Evaluations of Education Programs

Abstract

This article examines theoretical and empirical issues related to the statistical power of impact estimates for experimental evaluations of education programs. The author considers designs where random assignment is conducted at the school, classroom, or student level, and employs a unified analytic framework using statistical methods from the literature. Focusing on standardized test scores of elementary school students, this article discusses appropriate precision standards and, for each design, the required number of schools to achieve those standards using empirical values of intraclass correlations, regression R ² values, and other parameters. Clustering effects vary by design but are typically large. Thus, large school samples are required for education trials, and many evaluations will only have sufficient power to detect precise impacts for relatively large subgroups of sites.

Keywords

statistical power experimental designs evaluations of education programs

This article examines issues related to the statistical power of impact estimates for experimental evaluations of education programs. I focus on group-based experimental designs because many studies of education programs involve random assignment at the group level (e.g., at the school or classroom level) rather than at the student level. The clustering of students within groups generates design effects that considerably reduce the precision of the impact estimates, because the outcomes of students within the same schools or classrooms tend to be correlated. Thus, statistical power is a concern for these evaluations.

Until recently, evaluations of education programs where the student is the unit of analysis have often ignored design effects due to clustering; thus, many of these studies overestimated the statistical precision of their impact estimates (Hedges, 2004). As a consequence, there is currently much concern among education policy makers about how to interpret impact findings from previous evaluations of education programs and how to properly design future experimental studies to have sufficient statistical power to estimate impacts with the desired level of precision. This is a pressing issue because of provisions in the Education Sciences Reform Act of 2002 specifying, when feasible, the use of experimental designs to provide scientifically based evidence of program effectiveness and substantial taxpayer resources that are currently targeted to large-scale experimental evaluations of educational interventions by the Institute for Education Sciences (IES) at the U.S. Department of Education (ED).

There is a large literature on appropriate statistical methods under clustered randomized trials. Walsh (1947) showed that if clusters are the unit of random assignment, then conventional analyses will lead to an overstatement about the precision of the results, and the problem becomes more severe as the heterogeneity across clusters increases. Cochran (1963) and Kish (1965) discuss the calculation of design effects under clustered sample designs in terms of the intraclass correlation coefficient (ICC), which is the proportion of variance in the outcome that lies between clusters. In a seminal article, Cornfield (1978) first drew attention in the public health literature to the analytic issues presented by clustered randomized trials. Since that time, there have been extensive methodological developments in adjusting variance estimates for clustered designs (e.g., see Donner & Klar, 2000; Murray, 1998; Raudenbush, 1997). Much of this literature has focused on cluster randomized trials of medical and public health interventions. Despite this literature, however, Varnell, Murray, Janega, and Blitstein (2004) and Ukoumunne, Gulliford, Chinn, Sterne, and Burney (1999) found that only a small percentage of published impact studies that they reviewed in the public health field used appropriate methods to account for clustering.

Less attention has focused specifically on statistical power analyses in the education field. Bryk and Raudenbush (1992), Bloom, Bos, and Lee (1999), and Raudenbush, Spybrook, Liu, and Congdon (2004) discuss appropriate statistical procedures and provide examples but do not systematically consider statistical power issues for specific designs that are typically used to evaluate school interventions and that are based on up-to-date parameter assumptions.

This article builds on the literature in several ways. First, I use a unified hierarchical linear modeling (HLM) framework (Bryk & Raudenbush, 1992) to examine statistical power under commonly used experimental designs in a school setting with a single treatment and control group per site.1 Although the variance formulas used are known, the way in which they are used and the policy context is new, as is, in particular, my treatment of fixed versus random group effects. Second, I discuss appropriate precision standards for educational evaluations, a topic that has not been systematically addressed in the literature. Third, a list of key parameters (such as ICCs and regression R ² values) that are required to estimate power levels is compiled. Finally, this is the first article to jointly consider appropriate sample sizes for alternative designs that can serve as a guide for future designs of education-related evaluations.

My empirical analysis focuses on achievement test scores of elementary school and preschool students in low-performing school districts. I focus on test scores due to the accountability provisions of the No Child Left Behind Act of 2001 and the ensuing federal emphasis on testing interventions to improve reading and mathematics scores of young students.

This article is in five sections. First, general issues for a statistical power analysis, including techniques for assessing appropriate precision levels, are discussed. Second, presented is a unified mathematical model to examine sources of variance for school-based evaluations and, third, an application of this framework to specific designs. Fourth, I conduct power calculations for each of the considered designs and, finally, present conclusions.

General Issues for a Statistical Power Analysis

An important part of any evaluation design is the statistical power analysis, which demonstrates how well the design of the study will be able to distinguish real impacts from chance differences. To determine appropriate sample sizes for experimental evaluations, researchers typically calculate minimum detectable impacts, which represent the smallest program impacts—average treatment and control group differences—that can be detected with a high probability. In addition, it is common to standardize minimum detectable impacts into effect size units, that is, as a percentage of the standard deviation of the outcome measures (also known as Cohen’s d), to facilitate the comparison of findings across outcomes that are measured on different scales (Cohen, 1988). Hereafter, I denote minimum detectable impacts in effect size units as MDEs.

Mathematically, the MDE formula can be expressed as follows:

MDE = Factor (α, β, d f) * \sqrt{Var (impact)} / σ,

(1)

where Var(impact) is the variance of the impact estimate, σ is the standard deviation of the outcome measure, and Factor(.) is a constant that is a function of the significance level (α), statistical power (β), and the number of degrees of freedom (df, which equals the number of groups minus the number of strata minus 1).2 Factor(.) becomes larger as α and df decrease and as β increases (see Table 1).

As an example, consider an experimental design with a single treatment and control group and α =.05 and β =.80. In this case, for a given sample size and design structure, there is an 80% probability that a two-sample t test will yield a statistically significant impact estimate at the 5% significance level if the true impact were equal to the MDE value in Equation 1.

Precision Standards

A key issue for any evaluation is the precision standard to adopt for the impact estimates, which determines appropriate sample sizes. There are two key factors that need to be considered. First, the precision standard should depend on what impact is deemed meaningful in terms of future, longer term student outcomes (such as high school graduation, college attendance, earnings, etc.). Second, the precision standard should depend on what intervention effects are realistically attainable.

There is no uniform basis for adopting precision standards in educational research, and this critical issue has not been rigorously addressed in the literature, primarily because it is often difficult to determine what size impacts are meaningful, especially for young children. In this section, we discuss several procedures that can be used in practice.

Examine impact results from previous evaluations

One approach for adopting a precision standard is to use impact results found in previous rigorous evaluations similar to the one under investigation that found beneficial impacts. Another related and widely used approach is to use meta-analysis results from previous impact studies across a broad range of disciplines to examine the magnitude of impacts that have been achieved (Cohen, 1988; Lipsey & Wilson, 1993). Citing these meta-analysis results, many evaluations of education programs adopt standardized effect sizes of .20, .25, or .33 as the precision standard.

Although these approaches can be used to determine what impacts could be attainable for a particular intervention, they do not necessarily address what impacts are meaningful. As discussed next, I believe that these precision standards are somewhat high for testing the efficacy of education interventions on student standardized test scores.

Adopt a benefit–cost framework

One approach for assessing meaningful standardized effect sizes is to select samples large enough to detect impacts such that program benefits (measured in dollars) would offset program costs. For instance, several studies have indicated that a one-standard-deviation increase in either math or reading test scores for elementary school children is associated with about 8% higher earnings when the students join the labor market (Currie & Thomas, 1999; Murnane, Willet, & Levy, 1995; Neal & Johnson, 1996). Krueger (2000) estimates that the present discounted value of this higher earnings stream over a worker’s lifetime due to a one-standard-deviation increase in test scores is about $37,500.3,4 As a consequence, if an intervention improved test scores by .20 standard deviations, the present value of lifetime earnings would be $7,500, which was roughly the nationwide total expenditures per pupil in 1997–1998. Because most interventions are likely to cost less than $7,500 per pupil, these results suggest that a precision standard of .20 standard deviations might be too large from a benefit-cost standpoint.

Examine the natural progression of students

Another approach is to adopt a precision standard based on the natural growth of student outcomes over time. Kane (2004) found that the test performance of elementary school students in math and reading grows by about .70 standard deviations per grade. I found similar results using scaled SAT-9 test score data from the Longitudinal Evaluation of School Change and Performance (LESCP) in Title I schools.5 Assuming that test score gains occur uniformly throughout the school year, an average test score gain of about .70 standard deviations suggests that a standardized effect size of .20 corresponds with roughly 3 months of instruction (assuming a regular 10-month school year). This is a large impact given all else that is occurring in students’ lives. Thus, according to this metric, it might be appropriate to adopt a smaller, more attainable precision standard.

Examine the distribution of outcomes across schools

Another metric is to assess what an MDE implies about movements in mean student outcomes in a typical school relative to the distribution of outcomes across a broader set of schools. This approach again suggests that effect sizes of .20 to .33 are large.

For instance, I analyzed California Achievement Test (CAT/6) data for third graders using data from the 2004 California Standardized Testing and Reporting (STAR) Program. Consider a school at the 25th percentile of the math or reading test score distribution. A 33% effect size implies that the intervention would move that school from the 25th to 37th percentile of the score distribution, which is a large increase.6 A more attainable 10% effect size would move the school from the 25th to 29th percentile. LESCP data for SAT-9 scores of third graders in Title 1 schools yield similar findings.

A related method is to assess the magnitude of MDEs by translating them into nominal impacts for binary outcomes. For example, according to the National Assessment of Educational Progress (NAEP), nearly 70% of fourth graders nationally performed below the Proficient level in reading and math in 2003 (National Center for Education Statistics, 2004). For this binary outcome, effect sizes of .33 and .20 translate into large impacts of about 15.0 and 9.2 percentage points, respectively. A smaller, more realistic effect size of .10 translates into an impact of about 4.6 percentage points.

In sum, there is no standard basis for assessing appropriate precision standards for experimental impact evaluations of education programs. A precision standard of between .20 and .33 of a standard deviation is often used and is justified on the basis of meta-analysis results across a range of fields. This approach also represents a reasonable compromise between evaluation rigor and evaluation cost. However, it must be viewed as somewhat ad hoc. Other methods suggest that smaller effect sizes are meaningful for examining intervention effects on test scores.

Mathematical Framework for the Variance Calculations

The MDE calculations depend critically on the standard errors of the impact estimates. In this section, I use an HLM approach to develop a unified framework for identifying sources of variance under commonly used experimental designs in a school setting with a single treatment and control group per site.

I consider the following four-level model corresponding to students in Level 1 (indexed by h), classrooms in Level 2 (indexed by i), schools in Level 3 (indexed by j), and school districts in Level 4 (indexed by k):

\begin{array}{l} Level 1 : Students : & Y_{h ijk} = α_{0 ijk} + β_{0 ijk} R_{1} T_{h ijk} + e_{h ijk} \\ Level 2 : Classrooms : & α_{0 ijk} = α_{00 j k} + β_{00 j k} R_{2} T_{0 ijk} + u_{0 ijk}^{{random}_{2}} \\ β_{0 ijk} = β_{00 j k} + η_{0 ijk}^{{random}_{2}} \\ Level 3 : Schools : & α_{00 j k} = α_{000 k} + β_{000 k} R_{3} T_{00 j k} + r_{00 j k}^{{random}_{3}} \\ β_{00 j k} = β_{000 k} + θ_{00 j k}^{{random}_{3}} \\ Level 4 : Districts : & α_{000 k} = λ_{0} + λ_{1} R_{4} T_{000 k} + τ_{000 k}^{{random}_{4}} \\ β_{000 k} = λ_{1} + π_{000 k}^{{random}_{4}} \end{array} .

(2)

In the model, Y _hijk is a continuous outcome measure for a student; R _q (q =1, 2, 3, 4) is an indicator variable equal to 1 if the point of random assignment is at Level q, and 0 otherwise (ΣR _q = 1); T is an indicator variable equal to 1 for treatment group units and 0 for controls; and the superscript, random _q , equals 1 if the group effect in Level q is treated as a random error term, and 0 if it is treated as a fixed parameter (a topic that I discuss in more detail in the next section). The α terms represent level-specific intercepts, and the β terms represent level-specific treatment effects that are applicable depending on the point of random assignment. The λ₁ parameter represents the average impact estimate (the difference in mean outcomes between treatment and control group students), and the λ₀ parameter represents the control group mean. The random student-level errors, e _hijk , are assumed to be $iid N (0, σ_{e}^{2})$ .

Clustering arises due to the random error terms in the Level 2 through 4 equations. For HLM level q, group effects are considered to be random (that is, random _q = 1) if either (a) random assignment of units is conducted at level q, or (b) units at level q are considered to be randomly sampled from a broader universe of units before or after random assignment takes place. When included, I assume that $u_{0 ijk}^{1}$ are $iid N (0, σ_{u}^{2})$ classroom-specific random error terms that capture the correlation between the outcomes of students in the same classroom; $r_{00 j k}^{1}$ are $iid N (0, σ_{r}^{2})$ school-specific error terms that capture the correlation between the outcomes of students in the same school; $τ_{000 k}^{1}$ are $iid N (0, σ_{τ}^{2})$ district-specific error terms; and $η_{0 ijk}^{1}, θ_{00 j k}^{1}$ , and $π_{000 k}^{1}$ are $iid N (0, σ_{η}^{2}), iid N (0, σ_{θ}^{2})$ , and $iid N (0, σ_{π}^{2})$ random error terms, respectively, that represent the extent to which treatment effects (that is, average impacts) vary across classrooms, schools, and districts, respectively. The random error terms across equations are assumed to be distributed independently of each other.7

To more concisely examine sources of variance for the considered designs, I use a single-equation version of the HLM framework (e.g., Murray, 1998) by recursively inserting the Level 2 through 4 equations into the Level 1 equation, which yields the following expression:

\begin{array}{l} Y_{h ijk} = λ_{0} + λ_{1} T_{h} + {L_{1} e_{h ijk} + L_{2} [u_{0 ijk}^{{random}_{2}} + R_{1} T_{h} η_{0 ijk}^{{random}_{2}}] + L_{3} [r_{00 j k}^{{random}_{3}} \\ + (R_{1} + R_{2}) T_{h} θ_{00 j k}^{{random}_{3}}] + L_{4} [τ_{000 k}^{{random}_{4}} + (R_{1} + R_{2} + R_{3}) T_{h} π_{000 k}^{{random}_{4}}]}, \end{array}

(3)

where, for simplicity, I subscript T by h only, and where, for future reference, I include L _q indicator variables (q =1, 2, 3, 4), which equal 1 if Level q is included in the considered design, and 0 otherwise (in which case, I omit subscripts corresponding to Level q). In all designs, L ₁ = 1.

Equation 3 yields a simple differences-in-means estimator. As discussed later, the model can be generalized to include level-specific covariates measured at baseline that can improve the precision of the impact estimates.

Variance Calculations for Commonly Used Designs

In this section, I use Equation 3 to examine sources of variance for designs where the following units are randomly assigned to a research status: (a) students within sites (schools or districts); (b) classrooms within schools; and (c) schools within districts. Each of these designs is nested within the four-level model discussed above.

Table 2 summarizes the various designs that I consider with a single treatment and control group per site. For each design, it displays values for R _q , L _q , and random _q in Equation 3 and displays equation numbers in the text for the variance formulas. Power analyses for these designs can be conducted using the Optimal Design software program (Raudenbush et al., 2004).

Random Assignment of Students Within Sites: Fixed-Effects Case

In some designs, students in purposively selected schools or districts are randomly assigned directly to the treatment and control groups without regard to the classrooms or schools that the students attend, for example, the 21st Century Community Learning Centers Program (Dynarski et al., 2004) and the Impact Evaluation of Charter Schools Strategies (Gleason & Olsen, 2004). It is clear that these designs are not appropriate for testing classroom-based interventions where random assignment at the classroom or teacher level is required. Furthermore, these designs are appropriate only if potential “spillover” effects are expected to be small, so that students in the control group are expected to “receive” little of the intervention through their contact with students in the treatment group.

A central issue in the variance calculations for this design is whether site effects should be treated as fixed or random. Under the fixed-effects case, the impact estimates are viewed as generalizing to the study sites only, whereas under the random effects case, the impact estimates are viewed as generalizing to a broader population of sites similar to the study sites, which introduces design effects.

Although this issue needs to be addressed for each study, I believe that the fixed-effects case is usually more realistic in evaluations of education interventions. For most evaluations, sites are purposively selected for the study for a variety of reasons (such as the site’s willingness to participate, whether the site has a sufficient number of potential program participants to accommodate a control group, and so on). Furthermore, most evaluations are efficacy trials where a relatively small number of purposively selected sites is included in the study. Thus, in many instances, it is untenable to assume that the study sites are representative of a broader, well-defined population. Furthermore, inflating the standard errors to incorporate between-site effects will slant the study in favor of finding internally valid impact estimates that are not statistically significant, thereby providing less information to policy makers on potentially promising interventions. Instead, I believe, in general, that it is preferable to treat site effects as fixed and to assess the generalizability of study findings by examining the pattern of the impact estimates across sites. This approach is likely to yield credible information on the extent to which specific interventions could be effective and whether larger scale studies are warranted to examine whether they truly are effective.

Using Equation 3, the impact estimate for site g under the fixed-effects design is ( ${\hat{λ}}_{1} + {\hat{θ}}_{0. g .}^{0}$ ) if sites are schools, and ( ${\hat{λ}}_{1} + {\hat{π}}_{0 .. g}^{0}$ ) if sites are districts. The variance of these site-specific impact estimates can be expressed as follows (see Design I in Table 2):

Var ({\hat{λ}}_{1 g}) = \frac{σ_{e}^{2}}{m_{g} p_{1 g} (1 - p_{1 g})},

(4)

where m _g is the total sample size of treatment and control students in site g, and p ₁ _g is the proportion of students assigned to the treatment group in the site.

Pooled impact estimates across sites are calculated as a weighted average of the impact estimates in each site, and the associated variances are obtained by aggregating the site-specific variances in Equation 4 as follows:

Var (pooled impact) = \sum_{g = 1}^{s} w_{g}^{2} \frac{σ_{e}^{2}}{m_{g} p_{1 g} (1 - p_{1 g})},

(5)

where s is the number of sites and w _g is the weight associated with site g, where the weights sum to unity. Each site could be given equal weight in the analysis, or weights could be constructed to be inversely proportional to site-specific variances if they are allowed to vary (Fleiss, 1986).

To reduce notation and to facilitate comparisons with the other designs discussed below, I use the following simplified (approximate) version of Equation 5:

Var (pooled impact) = \frac{σ_{e}^{2}}{s [m p_{1} (1 - p_{1})]},

(6)

where m is the average sample size per site and p ₁ is the average sampling rate to the treatment group per site.

I note that estimating pooled impacts is appropriate in many site-based evaluations, because even in cases where the tested interventions differ somewhat across sites and serve different populations, the interventions are usually within the same general category (such as a reading or math curriculum, an after-school program, a technology, or a teacher preparation or mentoring model) and often share common features and a common funding source. Thus, it is typically of policy interest to examine the overall efficacy of promising interventions within a general class of treatments, even though the results must be interpreted carefully.

Random Assignment of Students Within Sites: Random-Effects Case

In some designs, site effects are treated as random. This can occur in two ways. First, purposively selected sites could be considered representative of a broader population of similar sites. Second, in some evaluations, sites are actually randomly sampled from a larger pool of sites. This type of design is typically employed in large-scale studies of a well-established program that require externally valid impact estimates and where the burden of evidence of program effectiveness is set high, for example, the National Evaluation of Upward Bound (Myers & Schirm, 1999) and the National Job Corps Study (Schochet, Burghardt, & Glazerman, 2001).

In these random-effects designs, study results are generalized more broadly than in the fixed-effects designs but involve a cost in terms of precision levels. Consider a two-level model corresponding to students and schools (see Design II in Table 2). For this design, Equation 3 yields the following variance formula for the pooled impact estimate (λ̂₁):

Var (pooled impact) = \frac{σ_{θ}^{2}}{s} + \frac{σ_{e}^{2}}{s [{m p}_{1} (1 - p_{1})]},

(7)

where $σ_{θ}^{2}$ is the variance of the impacts across schools, and where other parameters are defined as above. Design effects occur under this design because of the first variance term, which is deflated by the number of schools (not students). If sites are considered to be districts rather than schools, then, in Equation 7, $σ_{θ}^{2}$ is replaced by $σ_{π}^{2}$ and s is replaced by d, the number of districts.

The variance formulas presented above can be easily generalized to account also for additional levels of clustering within sites. For instance, if classrooms in study schools were considered to be representative of a broader population of classrooms in these schools, then Equation 3 yields the following variance formula (see Design III in Table 2):

Var (pooled impact) = \frac{σ_{θ}^{2}}{s} + \frac{σ_{η}^{2}}{s c} + \frac{σ_{e}^{2}}{s c [{n p}_{1} (1 - p_{1})]},

(8)

where c is the average number of classrooms per school and n is the average number of students per classroom. Additional variance terms could also be included to account for potential “treatment-induced” correlations between the outcomes of treatments if the intervention is administered in small groups, thereby creating potential correlations between the outcomes of treatments within each small group (Murray, Varnell, & Blitstein, 2004; Raudenbush, 1997).

Random Assignment of Classrooms Within Schools

A design that is commonly used in evaluations of school interventions is when classrooms or teachers within study schools are randomly assigned to the treatment or control groups, for example, the Evaluation of the Effectiveness of Educational Technology Interventions (Dynarski et al., 2004). This type of design is appropriate for interventions that are administered at the classroom or teacher level and where potential spillover effects across classrooms are deemed to be small.

One way to interpret this design is that a “miniexperiment” is being conducted in each school. If school effects are treated as fixed, pooled impact estimates across schools are calculated by averaging the impact estimates from each miniexperiment. The variance formula for these pooled impacts can be expressed as follows (see Design IV in Table 2):

Var (pooled impact) = \frac{σ_{u}^{2}}{s [{c p}_{2} (1 - p_{2})]} + \frac{σ_{e}^{2}}{s [{c p}_{2} (1 - p_{2})] n},

(9)

where p ₂ is the average proportion of classrooms assigned to the treatment group per school.

Several important features of this variance formula are worth mentioning. First, in some evaluations, all children in the study classrooms are included in the study. Under the fixed-effects scenario, one could then argue that student effects should not be included in the variance calculations. However, it is customary to include these student-level terms, because it is usually the case that some children will not provide follow-up data due to study nonconsent, attrition, and interview nonresponse. Thus, students in the follow-up sample are often considered to be representative of a larger pool of students in the study schools. An alternative approach is to apply a finite population correction to the student-level variance term (Schochet, 2005).

Second, in some evaluations, students within each of the participating schools and grades are randomly assigned to classrooms at the start of the school year, for example, the Teach For America (TFA) Evaluation (Decker & Glazerman, 2004). This design ensures that the average baseline characteristics of students in the treatment and control group classrooms are similar. Although this design reduces classroom effects, it does not remove them. This is because classroom effects arise from two sources: (a) differences in the quality of teachers within schools, and (b) systematic differences in the types of children who are assigned to different classrooms. The random assignment of children to classrooms reduces the second source of variance but not the first source.

Third, if school effects are treated as random, then Equation 3 yields the following variance expression (see Design V in Table 2):

Var (pooled impact) = \frac{σ_{θ}^{2}}{s} + \frac{σ_{u}^{2}}{s [{c p}_{2} (1 - p_{2})]} + \frac{σ_{e}^{2}}{s [{c p}_{2} (1 - p_{2})] n} .

(10)

Finally, due to limitations in the number of available classrooms, it is often the case that only one treatment and control classroom can be randomly assigned per school. In this case, there are not enough degrees of freedom to estimate $σ_{u}^{2}$ , which represents the extent to which classroom-level outcomes vary within schools and treatment condition (Murray, 1998). One approach is to set $σ_{u}^{2}$ to 0 in Equation 10 and to use the resulting variance formula for either the random or fixed-effects specifications (see Design VI in Table 2). Another possibility for the fixed-effects specification is to use Equation 9 and ignore the school fixed effects in the analysis, that is, the stratification by school. A final approach is to combine similar schools into larger strata, thereby making it possible to estimate between-classroom effects.

Random Assignment of Schools

In some designs, schools within districts are randomly selected to the treatment and control groups, for example, the Social and Character Development Research Program (Schochet, James-Burdumy, & Kisker, 2004). These designs are necessary for testing interventions that are (a) school based or (b) classroom based, but where potential spillover effects across classrooms are deemed to be serious. I focus on designs where school districts volunteer for the study and, thus, where district effects are treated as fixed.

Clustering at the school level only

For a school-based evaluation, one design option is not to sample classrooms within the study schools. For this option, either all relevant classrooms in the selected schools are included in the research sample or students are sampled directly to the research sample without regard to the classrooms that they are in.

Under this design and using Equation 3, the impact estimate in district g is ( ${\hat{λ}}_{1} + {\hat{π}}_{0.0 g}^{0}$ ). The pooled impact estimate across districts is then calculated as a simple or weighted average of the district-specific impact estimates, with the following variance:

Var (pooled impact) = \frac{σ_{r}^{2}}{{d s p}_{3} (1 - p_{3})} + \frac{σ_{e}^{2}}{[{d s p}_{3} (1 - p_{3})] c n},

(11)

where d is the number of school districts, s is now the average number of schools per district, and p ₃ is the average proportion of schools per district assigned to the treatment group (see Design VII in Table 2).

Clustering at the school and classroom levels

For a school-based design, there could also be clustering at the classroom level. This would occur if, to conserve project resources, classrooms were sampled within the study schools or if the full set of classrooms in the study schools were considered to be representative of a larger population of classrooms. In this case, the variance formula for the pooled impact estimate can be expressed as follows (see Design VIII in Table 2):

Var (pooled impact) = \frac{σ_{r}^{2}}{{d s p}_{3} (1 - p_{3})} + \frac{σ_{u}^{2}}{[{d s p}_{3} (1 - p_{3})] c^{*}} + \frac{σ_{e}^{2}}{[{d s p}_{3} (1 - p_{3})] c^{*} n},

(12)

where c* is the number of sampled classrooms (which is assumed to be the same in the treatment and control schools).

Estimating Intraclass Correlation Coefficients

A critical issue for the MDE calculations is what estimates to use for the error variances in the above variance formulas. In this section, I present empirical values for these error variances based on those found in the literature and from new tabulations from several recent large evaluations conducted by Mathematica Policy Research Inc. I report ICCs (denoted by ρ) for standardized test scores of elementary school and preschool students in low-performing schools. An ICC represents the proportion of variance in the outcome that lies between clusters and is the relevant parameter for the power calculations because MDEs are in standard deviation units.

Table 3 displays empirical values for ρ _r . The table shows that estimates of ρ _r vary somewhat by data source and grade level and typically become smaller when adjusted for district fixed effects. Thus, although applicable values will depend on the study context, the examined data sources suggest that ρ _r values often range from .10 to .20 for standardized test scores. Thus, in the illustrative power calculations below, I use the midpoint, .15.

There is less evidence on plausible values for the ICC at the classroom level, ρ _u , because there are fewer data sources that have student-level data on multiple classrooms within schools. LESCP and TFA data suggest a ρ _u value of about .16, which is similar to ρ _r . Thus, mean student test scores tend to differ as much across classrooms within schools as they do across schools. In the power calculations, I assume the same .15 value for ρ _u and ρ _r .

A plausible value for the impact-related ICC, ρ_θ, will depend on the relative effectiveness of the tested interventions across schools. Data indicate, however, that this ICC is typically small. For instance, in the evaluation of the 21^st Century Program, the value of ρ_θ for math and reading test scores was about .04 across elementary schools and .08 across middle schools. Similarly, in the evaluation of the School Dropout Demonstration Assistance Program, the value of ρ_θ for student grades was .06. Finally, in the Early Head Start evaluation, ρ_θ was about .06 for Bayley scores and .08 for the MacArthur Communicative Development Inventories. However, because of the uncertainty of this ICC, I assume a conservative ρ_θ value of .15.

Finally, I have less information on plausible values for ρ_η, which require data on multiple treatment and control classrooms within schools. Test score data from the TFA evaluation suggest a value of about .15 for ρ_η. However, I assume a more conservative value of .20 in the power calculations to reflect the uncertainty in this parameter.

Incorporating Baseline Covariates

For a given sample design, the most effective strategy for improving precision levels for clustered designs is to use regression models to estimate program impacts (Raudenbush, 1997). The inclusion of relevant baseline student-, classroom-, and school-level explanatory variables in the regression models in Equation 3 can increase power by explaining some of the variance in mean outcomes across schools and across classrooms within schools, that is, by increasing regression R ² values.

To demonstrate how to incorporate the use of regression models into the variance formulas, I consider the design where the school is the unit of random assignment and generalize Equation 12 as follows:

Var (pooled impact) = \frac{σ_{r}^{2} (1 - R_{B S}^{2})}{{d s p}_{3} (1 - p_{3})} + \frac{σ_{u}^{2} (1 - R_{B C}^{2})}{[{d s p}_{3} (1 - p_{3})] c^{*}} + \frac{σ_{e}^{2} (1 - R_{W}^{2})}{[{d s p}_{3} (1 - p_{3})] c^{*} n} .

(13)

In this expression, $R_{B S}^{2}, R_{B C}^{2}$ , and $R_{W}^{2}$ are, respectively, the proportion of the between-school, between-classroom, and within-classroom variances that are explained by the regression model.9 The most effective explanatory variables are likely to be preintervention measures of the outcome variables.

To obtain benchmark regression R ² values, I examined the fit of models using baseline and follow-up test score data on elementary school students from various data sources: (a) the LESCP, (b) the national evaluation of the 21st Century program, and (c) the TFA evaluation. My analysis indicated that $R_{B S}^{2}$ and $R_{W}^{2}$ values were at least .50 in regression models that included student-level baseline test scores as explanatory variables. Gargani and Cook (2005) and Bloom et al. (1999) found similar values using test score data from Louisville, Kentucky, and Rochester, New York, respectively. However, in the absence of these preintervention measures of the outcome variables, R ² values were closer to .20.

Illustrative Precision Calculations

In this section, I collate formulas and results from above and calculate illustrative MDE calculations for standardized test scores for the most common designs that I have considered.

Presentation and Assumptions

Tables 4 and 5 display, under various assumptions and for each of the considered designs, the total number of schools that are required to achieve precision targets of .10, .20, .25, and .33 of a standard deviation, respectively. Because the amount and quality of baseline data vary across evaluations, I conduct power calculations assuming conservative R ² values of 0, .20, and .50 at each group level. My estimates assume a two-tailed test; a balanced allocation (Table 4) or 2:1 allocation (Table 5) across the treatment and control groups; and the ICC values discussed above. Finally, all calculations were conducted under the following assumptions: (a) 80% power, (b) a 5% significance level, (c) the intervention is being tested in a single grade with an average of three classrooms per school per grade and an average of 23 students per classroom (with no subsampling of students), and (d) 80% of students in the sample will provide follow-up (posttest) data. Schochet (2005) presents a more complete set of precision calculations under a wider range of plausible parameter values.

Results

The main results can be summarized as follows:

Clustering matters in school-based evaluations that aim to improve test scores

Precision levels decrease substantially as clustering effects increase. For instance, assuming a zero R ² value, 14 schools under Design I (student-level random assignment) are required to detect an effect size of .20 standard deviations, compared with 55 schools for Design VI (classroom-based random assignment) and 130 schools for Design VII (school-level random assignment; see Table 4).

Relatively large school sample sizes are required for the most commonly used designs

For classroom-level random assignment, about 30 to 50 schools are required to detect an effect size of .20 (depending on the R ² assumption), and the required number of schools is about 65 to 130 under school-level random assignment (see Table 4). As a consequence, because of resource constraints, many evaluations will only have sufficient power to detect precise impacts for relatively large subgroups of sites. Thus, the chosen point of random assignment and the treatment of group effects as random or fixed have important implications for evaluation costs and the range of study questions that can be addressed.

Achieving effect sizes of .10 may not be attainable in many evaluations

As discussed, relatively small test score gains might be meaningful from a benefit-cost standpoint and realistic in terms of the natural progression of students over a school year. However, results suggest that very large sample sizes are required to detect relatively small test score gains. For instance, even with an R ² value of .50, to detect an effect size of .10, Design VI requires 106 schools and Design VII requires 259 schools (see Table 4). As a consequence, because of cost constraints, some interventions should be tested only if they can be expected to have a relatively large effect on student outcomes.

R ² values matter

An effective strategy for improving precision levels for group-based experimental designs is to use regression models to estimate program impacts. For example, under Design VII, 86 schools are required to achieve an effect size of .25 for an R ² value of 0, compared with only 44 schools for an R ² value of .50 (see Table 4). Thus, the availability of detailed baseline data at the aggregate school or individual student level (and in particular, data on preintervention measures of the outcome variables) can substantially improve statistical power under clustered designs.

A 2:1 split of the research groups does not materially reduce precision levels relative to a balanced allocation

The required school sample sizes are only slightly larger under a design with twice as many treatments as controls (or vice versa) than under a design with equal sample sizes across the research groups (Tables 4 and 5).10 This is an important finding because an unbalanced design might be preferred in some evaluations for operational reasons (e.g., to minimize the number of controls).

Finally, I note that for each of the considered designs, the optimal sample size of districts, schools, classrooms, and students can be calculated by minimizing the MDE subject to a cost function or minimizing costs subject to a fixed MDE. The cost function will depend on study sample sizes as well as unit costs (from recruiting districts, schools, and teachers, obtaining parental consent, conducting interviews and assessments, and so on). The precision calculations presented above are based on designs that are likely to be relatively cost-effective for attaining a given MDE level, because they assume no subsampling of students. Clustered designs can usually attain the same MDE level with a smaller total student sample if the sample contains more schools and fewer students per school. However, these designs could cost more due to the significant expense of recruiting additional districts and schools, but this will depend on unit costs specific to each project.

Summary and Conclusions

In this article, I have examined theoretical and empirical issues related to the statistical power of impact estimates under commonly used experimental designs for evaluations of education interventions that seek to improve students’ standardized test scores. My main conclusion is that clustering effects cannot be ignored when groups such as schools or classrooms are the unit of random assignment. I find that relatively large school sample sizes are required to achieve targeted precision levels under these designs. Furthermore, the required sample sizes of schools and classrooms increase substantially as clustering effects increase and as precision standards are made more stringent. Design effects due to clustering cannot be ignored because of the relatively large ICCs for standardized test scores at the school and classroom levels.

The implication of these findings is that because of study resource constraints, many impact evaluations of education interventions will only have sufficient statistical power to detect impacts at the pooled level and for relatively large subgroups of sites or schools, but not for smaller subgroups. Furthermore, it might not always be feasible from a power standpoint to randomly assign multiple treatments to units. In addition, some evaluations may not have sufficient power to obtain precise estimates for other types of analyses that are often conducted in impact evaluations (such as mediated analyses, latent variable analyses, and so on). As a consequence, we must recognize that many impact evaluations of education programs can be expected to rigorously address broad research questions only and, hence, should be structured to focus on a narrow set of issues. Results from more disaggregated analyses, although important, must be deemed heuristic.

This discussion has stressed that education researchers who conduct power analyses—using, for example, the Optimal Design software program (Raudenbush et al., 2004)—must carefully specify the sources of clustering under their designs and the assumptions underlying them. In particular, the treatment of group effects as fixed or random is an important issue that has major implications for sample size requirements and has often been ignored in the literature. Finally, I emphasize that the collection of detailed baseline data is an important way to reduce clustering effects.

Footnotes

1

Under designs with multiple interventions (treatment groups), multiple comparison issues must be considered, which are beyond the scope of this article.

2

Specifically, Factor(.) can be expressed as [T ⁻¹ (1 − α)+T ⁻¹(β)] for a one-tailed test and [T ⁻¹ (1 − {α/2})+T ⁻¹ (β)] for a two-tailed test, where T ⁻¹(.) is the inverse of the student’s t distribution function with df degrees of freedom (see Murray, 1998, and Bloom, 2004, for derivations of these formulas). Equation 1 ignores the estimation error in the standard deviation.

3

This figure was calculated (a) using the age-earnings profile in the March 1999 Current Population Survey, (b) a 4% discount rate, (c) assuming workers begin earning wages at age 18 and retire at age 65, and (d) a productivity (wage) growth rate of 1% per year.

4

find similar results.

5

Kane’s results are based on separate cross-sections of students (which could be affected by cohort effects), whereas the Longitudinal Evaluation of School Change and Performance results are based on the same students over time.

6

The 25th percentile of the school distribution is 605 for reading and 602 for math, and the standard deviation of CAT/6 scores is about 20 scale points.

7

My framework is a variant of a General Linear Mixed Model where it is assumed that random effects are independent and normally distributed (e.g., see Murray, 1998; Searle, Casella, & McCulloch, 1992). I do not consider models with a more general covariance structure for the random effects.

8

As shown in , a small correction factor needs to be applied to the variance formulas when group-level covariates are included in the regression model.

9

It is usually the case that $R_{B S}^{2}$ and $R_{B C}^{2}$ are positive, although the school and classroom variance components could increase with regression adjustment, in which case, Equation 14 does not apply (Murray, 1998, provides a more general formulation).

10

also documents this finding.

Tables

References

10.

11.

12.

13.

14.

15.

16.

17.

18.

19.

20.

21.

22.

23.

24.

25.

26.

27.

28.

29.

30.

31.

32.

33.

34.

35.