Hold the Bets! Should Quasi-Experiments Be Preferred to True Experiments When Causal Generalization Is the Goal?

Abstract

By design, randomized experiments (XPs) rule out bias from confounded selection of participants into conditions. Quasi-experiments (QEs) are often considered second-best because they do not share this benefit. However, when results from XPs are used to generalize causal impacts, the benefit from unconfounded selection into conditions may be offset by confounded selection into locations. This work shows that this tradeoff can lead to situations where estimates from QEs are less-biased from selection than are estimates from uncompromised XPs when drawing causal generalizations. This work establishes the conditions theoretically, demonstrates the idea empirically, and discusses the implications of the results.

Keywords

Quasi-experiments RCTs within study comparisons impact evaluation generalizability

It is natural for policy-makers and practitioners to ask about the potential for programs to achieve impact for an inference population that they are directly concerned with. Ideally, an uncompromised randomized experiment (XP) would be conducted in their specific context to answer the question decisively. However, if an XP is not possible, they may look to other sites, where the program has been used, to support an inference about the potential causal impact for their site.

Consider two options for doing this. The first, a more obvious and standard one, is to use an XP-based result from one or more study sites where the program has been evaluated—the “generalized from” site(s)—to infer impact for the “generalized to” site. This result may be adjusted for possible differences in the distribution of baseline characteristics that moderate the impact. This approach has the advantage that at the “generalized from” site, the result is not biased from confounded selection into conditions; however, because it involves a cross-site comparison of outcomes, the result may be biased from confounded selection into locations on characteristics that moderate the program effect (Hotz et al., 2005).

A second option is to make a cross-site comparison to infer just the missing outcome at the inference site. There are two possible situations for this depending on the missing outcome at the inference site. In the first scenario, the treatment has been used at the inference site, and the goal is to compare the performance given treatment at that site, to the performance in the absence of treatment using a comparison group from another site that has not used the program. This is the standard observationalist's application of a non-equivalent comparison group design (CGD), a type of quasi-experiment (QE) (Shadish et al., 2002). In the second scenario, the treatment has not been used at the inference site, and the goal is to compare the performance without treatment at that site, to the performance in the presence of treatment using a group from another site that has used the program. This is akin to a CGD, except it uses performance under assignment to treatment at another site to infer performance in the counterfactual condition for the group that has not received treatment at the inference site.

The second option has the advantage that it essentially uses an estimate of half the true impact result for the inference site (e.g., achievement in the presence of treatment for the first scenario, and performance in the absence of treatment for the second scenario); however, because this option involves a cross-site comparison of outcomes, it may be biased from confounded selection into locations on characteristics that either affect average achievement or that moderate program impact (Hotz et al., 2005).

The reader may well wonder why one would ever use the comparison group-based approach (the second option) in place of the XP-based one (the first option). The result based on an uncompromised XP has the advantage of being unbiased from selection at the study site, and therefore seems advantageous, even if it is achieved at a different site than the one for which the generalized impact is sought. Intuition suggests that adjusting results from an XP is the better option. In this work it is shown that, contrary to intuition, this is not always so. This finding is important because the less-biased option should be chosen whenever possible; therefore, it is critical to establish the conditions under which one approach yields less biased results than the other.

Clarifying the structure of this work will help the reader navigate it. There are five main sections. The first provides background. The second lays the groundwork for the method by describing three core scenarios investigated in the work. Each scenario offers a different solution to generalizing a causal effect of a program to an inference site using a comparison of outcomes from another site. The development of this section, especially the details of the first two scenarios, is important because the third scenario is a composite of the first two—it shows how a novel solution can be achieved by applying previous solutions in a new way. This section derives expressions for bias for the simplified scenarios of comparisons between just two sites. Both graphical and algebraic representations are given.¹ The basic example gets at the gist of the main idea—that generalized causal impact quantities, whether from experiments or non-experiments, can be biased for the inference site; and the levels of bias from each design can be quantified and compared. The third section sets up the empirical example by deriving expressions for average magnitudes of biases when generalizing impacts to multiple sites. The fourth section uses data from the Tennessee STAR (Student-Teacher Achievement Ratio) class size reduction multisite experimental evaluation to demonstrate an empirical approach to evaluating the alternative cross-site generalization approaches represented by the three scenarios. The fifth section concludes the work.

Background

Recently there has been a groundswell in advances in methods of evaluation for generalizing impact findings from experiments. The starting point of these methods is an impact finding from an XP conducted at one or more study sites. The generalization step involves adjusting the internally valid XP-based impact finding so that it reflects the local conditions for the inference population. Reweighting is a chief example (Schochet et al., 2014). The approach adjusts for differences between the study sample and the inference population in the distribution of moderators of impact.² Adjustments may also be made in terms of an index that summarizes the effects of multiple moderators on one dimension. Subclassification methods (e.g., Tipton, 2013) are an example.

Such XP-based approaches reflect an internal validity-first orientation: the starting point is an estimate of average impact from a true experiment, which is assumed to be internally valid. External validity—which addresses the extent to which a causal relationship holds over variations in persons, settings, outcomes or treatment variants (Shadish et al., 2002, p. 256)—is achieved after, and is based on the impact finding from a completed XP. This is consistent with a specific orientation in program impact evaluations that recognizes internal validity as the sine qua non (literally, “without which there is nothing”). The prioritization of internal validity makes establishing the causal nature of the relation under study the foremost concern—“the basic minimum without which any experiment is uninterpretable” (Campbell & Stanley, 1963, p. 5, in Shadish et al., 2002, p. 97). The implication is that when internal validity is of primary interest, then the priority is to first establish the causal relationship between variables, and only then address questions about the reach of the causal inference to contexts beyond the study.

The XP-based approach, however, is not the only one. Consider the case where a principal has implemented a program school-wide and would like to know if the program had an average positive impact on student achievement at her school. With the XP-based approach to generalization described above, she would look for an average impact finding from an XP conducted elsewhere, preferably from a locale similar to hers. She might then reweight the impact result from the remote site to more-closely reflect the distribution of characteristics of persons at her site. Alternatively, using a quasi-experimental (QE-based) approach, she may estimate impact for her site by comparing the average performance at her site, in which everyone has received treatment, to contemporaneous performance from one or more similar locales where the program has not been used. This is a standard non-equivalent comparison design (Shadish et al., 2002), with the inference locale being the treated site (i.e., the comparison is with a sample that provides a plausible counterfactual value for what the performance at the inference site would have been in the absence of treatment).

The comparison group-based approach accommodates also an alternative “reversed” scenario, where the program has not been used at the principal's (inference) site, and where she would like to know about the potential impact that could be achieved for her site. In this situation, the principal can again use the XP-based result, or, applying the comparison group-based strategy, she can infer impact to her site by comparing average performance at other similar locales, where the program has been used, against the average performance (in the absence of treatment) at her site. This approach is a version of the non-equivalent comparison design (Shadish et al., 2002), but where the comparison is made with a treated group to support a causal inference for the untreated site (i.e., the comparison is with a sample that provides a plausible counterfactual value for what the performance at the inference site would have been if the treatment had been used at that site).

For either of these scenarios, the XP-based result has the advantage that it is unbiased for the source sample (assuming the XP has not been compromised in some way). The success of generalization depends on completely identifying and adjusting for the effects of factors that produce a difference in impact between the experimental and inference sites (Cole & Stuart, 2010; Hotz et al., 2005; Imai et al., 2008). On the other hand, the comparison-group based approach has the advantage that it uses information from the actual inference site; however, the comparison with another site puts the result at risk of being biased from selection on confounders that affect average achievement, or on moderators that affect achievement by way of their interactions with the treatment. Put another way, the XP-based option uses an estimate of the true (unbiased) result, but it is from someplace other than the inference site; whereas the comparison-group based approach makes use of an estimate of the true outcome for the inference site itself; however, it is observed for just one condition (i.e., it is half of the unbiased solution for the inference location, and the other half must be inferred from someplace else).

Based on these scenarios, is not a priori clear that the XP-based generalization always yields a better (more accurate) result than the QE-based alternative. There is the prospect that under certain conditions, the QE strategy is the better option. This work addresses the following questions: (1) What are the expressions for bias for the XP- and QE-based generalizations described above? (2) Under which conditions is net bias in the XP-based generalization lower than in the QE-based result, and vice versa? (3) In a single empirical application, what are estimated levels of each type of bias, prior to and after applying site-level covariate adjustments?

The questions raised and explored in this work are motivated by an alternative perspective on the relationship between internal and external validity. Rather than seeing internal validity as the sine qua non, it treats the two forms of validity as interdependent. There is precedent for this in the program evaluation literature. Shadish et al. (2002), although accepting that internal validity is the sine qua non, also stressed that it is inseparable from external validity. They considered the latter as “the desideratum” (the purpose or objective) of educational research (Shadish et al., 2002, p. 97). The two kinds of validity— internal and external—are deeply complementary (Shadish et al., 2002). Further, they also interpreted the sine qua non of internal validity “in a strong sense,” that pertains to the situation where an experimenter has the option to exercise choice among several validity types. In that situation, internal validity may be prioritized at one stage of research, and external validity at another. Taking a stronger position, some evaluators outright rejected the precedence of one form of validity over the other, instead emphasizing the indispensability of one form of validity to another, and the role of contextual factors as co-causes of the results (e.g., Cronbach, 1975, 1982; Scriven, 2008). Cronbach, for example, stressed the “limited reach” of internal validity. For him identification of “the cause” of an effect has limited value without understanding the conditions and scope of that effect. Put another way, Cronbach considered that “if observations are not generalizable, then causal validity is irrelevant” (Albright & Malloy, 2000, p. 338). According to this interpretation, what we observe as the marginal impact in an XP-based evaluation, is the product of both the experimental manipulation plus the interactions of treatment with observed and unobserved moderators of the effect for that specific context. Any generalization requires making additional strong assumptions about the role and impact of those factors in the “generalized-to” context.

This work provides an alternative perspective on the positioning of internal and external validity. It shows that when causal generalization is the main concern, under certain plausible conditions, quasi-experimental comparisons can yield generalized inferences that are less-biased than XP-based ones. We develop the idea that both the XP- and QE-based inferences have potential for bias—either from confounders that affect average performance across sites, or from moderators that contribute impact heterogeneity across sites. Net bias in generalized impacts from QE-based designs depends on whether effects of confounders and moderators compound or offset one another. QE-based inferences may be preferred especially when this offset occurs because it can result in a cancellation of biases. An implication of this is that when causal generalization is the concern, the role of factors affecting internal validity (e.g., confounders) and external validity (e.g., moderators of impact) must be considered simultaneously. This point can be made theoretically, and studied empirically through so-called Within Study Comparisons (WSC) methods. We explore both in the sections that follow.³

Three Scenarios to Motivate the Main Ideas of This Work

Scenario 1: Establishing the Experimental Benchmark

The goal is to estimate the impact of a program T relative to counterfactual C on outcome Y at a given site N. Express the true average impact quantity for this site as follows (XP stands for “experimental” to denote a quantity that would be estimated without bias through an uncompromised randomized experiment):

Δ_{X P | N} = Y_{N} (T) – Y_{N} (C)

(1)

This work assumes that there is a true benchmark average impact quantity for each site. If it is possible to randomize individuals to

T

or C at an inference site, then this unconfounded assignment of individuals into conditions allows the true value to be estimated without bias. For instance, if the site is a school, this quantity would be estimable if students and teachers were randomly assigned to conditions and if their outcomes were observed. The true value average impact is represented pictorially in Figure 1.⁴

The problem addressed in this work arises when information is missing about performance in one of the conditions at the inference site, N, thereby requiring the use of information from elsewhere (i.e., from other sites) to generalize impact to the inference site. These cases are explored in the next two scenarios.

Scenario 2: Inferring Average Impact at Site N When Treatment Is Provided to Everyone at the Site

In Scenario 2 the goal is to infer average impact for site N, $Δ_{X P | N}$ , in the situation where an experiment is not possible at the site because the program is provided to everyone. As a result, the average performance of individuals in the absence of treatment at the site is unknown.

This situation would arise if a program is being implemented school-wide. The administrator may want to know if the program achieves positive average impact on student outcomes, compared to if it had not been used.

For this scenario, two options for inferring average impact at N are considered.

Option 1. The first option under Scenario 2 is to infer impact for N using the result of a randomized experiment conducted at a different location $M .$ ⁵ That is, the following impact quantity can be generalized from the source site M to the inference site $N$ :

Δ_{X P | M} = Y_{M} (T) - Y_{M} (C)

(2)

Figure 2 displays both the average impact at the inference site N,

Δ_{X P | N}

, and the average impact from the comparison site M,

Δ_{X P | M}

. For Scenario 2, it is assumed that performance in the absence of treatment is not observed at N (represented by the empty horizontal bar), which prompts the use of the result from M.

The difference between the true average impact at M, and the one at inference site N, is the bias in the former when used to generalize impact to the latter site. This is shown as $B i a s 1$ in Figure 2, and is expressed as follows:

B i a s 1 = Δ_{X P | M} – Δ_{X P | N}

(3)

B i a s 1

is represented in Figure 2 as the difference in the length of the vertical bars.

B i a s 1

is present if there is a difference between the sites in the average impact of treatment.⁶

What Are Possible Sources of $B i a s 1$ ? $B i a s 1$ is attributable to factors that result in systematic differences in impact between sites (i.e., not counting random sampling error). Among possible sources, most commonly considered are “unit characteristics”—person attributes that moderate the program's impact. That is, $B i a s 1$ will be present if there is imbalance between sites N and M in the distribution of characteristics of persons that interact with treatment (Cole & Stuart, 2010; Jaciw, 2016). For example, if the program achieves greater impact for students with higher incoming achievement, and if inference site N has a systematically lower proportion of students with higher incoming achievement than site M, then, unless this difference is adjusted for analytically, the average impact at M will have positive bias if generalized to N. Random assignment of individuals to locales would eliminate this confound; however, such assignment rarely occurs in practice (Hotz et al., 2005). A common strategy to limit $B i a s 1$ is to adjust for effects for imbalance on moderators through reweighting or subclassification methods (Tipton, 2013). An assumption of these methods is that the adjustment strategies result in conditionally unconfounded selection into locations on variables that interact with treatment—a necessary condition to mitigate $B i a s 1$ .

Lack of $B i a s 1$ implies that the selection of individuals into locations across which outcomes are being compared are unconfounded (or is conditionally unconfounded) with factors moderating impact; however, the converse of this is not necessarily true. Macro effects (Hotz et al., 2005) are another possible source of $B i a s 1$ . These are site characteristics measured through variables that have values specific to sites. For example, if N and M are different in terms of principal leadership style or school technological capacity, and if the impact of the treatment depends on (interact with) these variables, then they will induce $B i a s 1$ if $_{X P | M}$ is used to infer impact at N. Macro variables may also be site-averages of person characteristics measured at each site. If a site is a school, then this may include the school average of teachers’ years of experience, or the school average of student performance assessed before the study. When the comparison involves just two sites (in our case, N and $M$ ) the effects of macro variables on outcomes cannot be analytically de-confounded from the effect of treatment.⁷

$B i a s 1$ may be attributable to other sources also, including if the treatment itself varies across locations either in its offering (program practices) or uptake (participation in activities) (Bloom et al., 2003), if fidelity of implementation is different (Hulleman & Cordray, 2009), or if the control condition or programs vary across sites (Weiss et al., 2014). An additional potential source of $B i a s 1$ is a difference between the sites either in the measures or the comparability of the scales used to assess outcomes.

Option 2. The second option is to make a quasi-experimental (QE) comparison between N, where it is assumed everyone is receiving treatment, and individuals at a different site M, where it is assumed the treatment is not implemented. The resulting difference quantity may be represented as follows:

Δ_{Q E 1} = Y_{N} (T) – Y_{M} (C)

(4)

Figure 3 displays both the benchmark impact quantity at site N,

Δ_{X P | N}

, and the average impact using the QE comparison with site M,

Δ_{Q E 1}

. As before, the short horizontal bars represent performance levels. The solid-filled black bars represent observable quantities. The white-filled horizontal end-bar (in this case for average performance in the absence of treatment for

N

) represents a quantity that is not observed.

The difference between the QE result, $Δ_{Q E 1}$ , which involves a comparison of outcomes from M, and the XP-based benchmark impact at N, is the bias in the former quantity when used to infer impact to N. This is shown as $B i a s 2$ in Figure 3. It is expressed as follows:

\begin{aligned} B i a s 2 = & Δ_{Q E 1} – Δ_{X P | N} \\ = Y_{N} (T) – Y_{M} (C) – [Y_{N} (T) – Y_{N} (C)] \\ = Y_{N} (C) – Y_{M} (C) \end{aligned}

(5)

B i a s 2

is represented in Figure 3 as the difference between the inference site N and the comparison site M in the vertical heights corresponding to average performance in the absence of treatment. That is,

B i a s 2

is present if there is a difference between the sites in their average performance without the program. This is the standard expressions for bias in a CGD-based difference quantity relative to an experimental benchmark for an inference site. (Estimation of this bias is usually the starting point in WSC empirical studies that evaluate levels of and conditions for bias in impact estimates from CGD studies relative to benchmark impacts from XPs. In these studies, treatment group performance typically is differenced away in calculating bias [Heckman et al., 1997; Weidmann & Miratrix, 2021]).

What Are Possible Sources of $B i a s 2$ ? $B i a s 2$ can result from imbalance between the locations being compared on any factor that produces a difference in outcomes in the absence of treatment. As with $B i a s 1$ , this includes “unit characteristics,” in this case person attributes that affect the outcome. They include student baseline achievement in impact evaluations in education (Unlu et al., 2021), and individual prior earnings in impact evaluations of employment training programs (Glazerman et al., 2003). Such variables are important because they are highly predictive of later outcomes. Stated more formally, an important potential source of $B i a s 2$ in Scenario 2 is confounded selection of persons to locations (Hotz et al., 2005) on factors that affect performance in the absence of treatment. As is the case with $B i a s 1$ , random assignment of individuals to locales would eliminate this confound; however, such assignment rarely occurs in practice (Hotz et al., 2005). A common strategy to limit $B i a s 2$ is to adjust for effects of confounders, or a corresponding balancing score (Rosenbaum & Rubin, 1983) to achieve conditionally unconfounded assignment to locations. It is notable that “unit characteristics” (the person attributes) that result in $B i a s 1$ may or may not be the same as those that produce $B i a s 2$ . This point is important in a later discussion of the implications of this work.

Lack of $B i a s 2$ implies that the selection of individuals into locations that are being compared is unconfounded (or is conditionally unconfounded) with factors affecting performance in the absence of treatment; however, the converse is not necessarily true. $B i a s 2$ may be present for reasons other than person-level selection. For example, outcomes may be affected by site-level characteristics—the “macro variables” that have site-specific values (Hotz et al., 2005), which were discussed under Option 1. Using the example from the last section, if locales N and M are different in terms of principal leadership style or school technological capacity, and if these variables have a bearing on the outcome, such as achievement, then they will induce $B i a s 2$ in $Q E 1$ . As was pointed out under Option 1, if the comparison involves just two sites (in our case, N and $M$ ) the effects of macro variables on outcomes cannot be analytically de-confounded from the effect of treatment. Other potential sources of $B i a s 2$ include a difference between the sites in the “business as usual” (BAU) program or a difference between them in the measures or the comparability of the scales used to assess outcomes.

Recap of Scenario 2. Scenario 2 assumes treatment is implemented across site N, and the goal is to infer average impact of treatment for the site. The unbiased “benchmark” average impact for site N is $Δ_{X P | N}$ .

Option 1 uses the uncompromised XP-based impact quantity from site M, $Δ_{X P | M}$ , to infer impact at N. The result, used to infer average impact for site N is biased by the amount $B i a s 1 = Δ_{X P | M} – Δ_{X P | N}$ .

Option 2 makes use of a quasi-experimental comparison of average performance between the treated sample at inference site N, and average performance in the absence of treatment at the comparison site $M$ : $Δ_{Q E 1} = Y_{N} (T) – Y_{M} (C)$ . The result, when used to infer average impact for site N is biased by the amount $B i a s 2 = Y_{N} (C) – Y_{M} (C)$ .

The two options in Scenario 2 raise the following question. In Scenario 2, which of the two options yields a lower-bias result when inferring impact at $N$ . This question is explored empirically later in this work.

Scenario 3: Inferring Average Impact at Site N when the Treatment has Not Been Implemented at that Site

This is like Scenario 2, except no one at the inference site, N, has received the treatment. Scenario 3 may seem to be more-obviously about generalization, in the sense that an externally valid causal inference is being sought for a population that has not yet received the program or been involved in an experiment of the program. It is assumed that a randomized experiment cannot be conducted at N in the short term, and a plausible value for impact at N is needed immediately—perhaps to guide programming decisions for which an answer is urgently needed—and before a randomized experiment can happen.

As with Scenario 2, two options are considered to infer impact at N.

Option 1. The first option is to use the experimental quantity from location M, as in Case 1 under Scenario 2. With this option, bias is the same as in equation 3 above, reflecting the difference in impact between the two sites.

Option 2. The second option is less familiar. It consists of a quasi-experimental comparison between treated individuals at a different site M, and untreated individuals (everyone) at the inference site N. The resulting quasi-experimental difference quantity is represented as follows:

Δ_{Q E 2} = Y_{M} (T) – Y_{N} (C)

(6)

This is the difference in outcomes between the treated at comparison site M, and individuals at the inference site N, where treatment has not been implemented.

Figure 4 displays both the quantity of interest, which is the average impact for the inference site N, $Δ_{X P | N}$ , and the average impact based on a comparison of outcomes with site M, $Δ_{Q E 2}$ . As with the prior figures, the solid-filled horizontal black bars represent observed values, and the white-filled bar (in this case representing average performance under assignment to treatment at $N$ ) represents a value that is not observed. The absence of this value prompts the comparison with the treated group at M.

$Δ_{Q E 2}$ in this scenario is different from $Δ_{Q E 1}$ in Scenario 2. In Scenario 2, the inference is of the counterfactual to the average performance to T at N using outcomes in condition C at M. In Scenario 3, the inference is of the counterfactual to the average performance to C at N using outcomes in condition T at M. The comparison is inverted between the two scenarios, reflecting the information that is available or needed to infer the average causal effect for site N. This is evident in Figures 3 and 4: under Scenario 2 (Figure 3) the known and unknown quantities (the solid black bars and white bars) at N are for the treated and untreated groups, respectively. Under Scenario 4, the shading of the bars is reversed for the two groups at that site.⁸

To formulate bias in $Δ_{Q E 2}$ , first the expression in equation (6) is expanded:

Δ_{Q E 2} = Y_{M} (T) – Y_{N} (C) = [Y_{M} (T) – Y_{M} (C)] + [Y_{M} (C) – Y_{N} (C)]

(7)

The difference between the QE result obtained through a comparison of outcomes at M,

Δ_{Q E 2}

, and the experiment-based benchmark impact at N,

Δ_{X P | N}

, is the bias in the former quantity when used to generalize impact at the inference site N. This is expressed as follows:

\begin{aligned} Δ_{Q E 2} – Δ_{X P | N} \\ = {Y_{M} (T) – Y_{M} (C) + [Y_{M} (C) – Y_{N} (C)]} – {Y_{N} (T) – Y_{N} (C)} \\ = [Y_{M} (T) – Y_{M} (C)] – [Y_{N} (T) – Y_{N} (C)] + [Y_{M} (C) – Y_{N} (C)] \\ = {[Y_{M} (T) – Y_{M} (C)] – [Y_{N} (T) – Y_{N} (C)]} – [Y_{N} (C) – Y_{M} (C)] \\ = Δ_{X P | M} – Δ_{X P | N} – (Y_{N} (C) – Y_{M} (C)) \\ = B i a s 1 – B i a s 2 \end{aligned}

(8)

B i a s 1

and

B i a s 2

are represented in Figure 4. The figure is essentially a composite of Figures 2 and 3. The sources of bias are the same as before.

Δ_{Q E 2}

has two bias terms when used to infer impact at

N

: due to the difference between the sites in average achievement in the absence of treatment (

B i a s 2

) and due to the difference between the sites in their impact

(B i a s 1)

. This implies that the difference quantity,

Δ_{Q E 2}

, potentially reflects selection into sites both on factors that affect average achievement in the absence of treatment, and on factors that moderate impact.

Recap of Scenario 3: Scenario 3 assumes treatment is not implemented across site N, and the goal is to infer average impact of the treatment for the site. The unbiased “benchmark” average impact for site N is $Δ_{X P | N}$ .

Option 1 uses the uncompromised XP-based impact quantity from site $M,$ $Δ_{X P | M}$ , to infer impact at N. As with Scenario 2, the result is biased by the amount $B i a s 1 = Δ_{X P | M} – Δ_{X P | N}$ .

Option 2 makes use of a quasi-experimental comparison of average performance between the treated sample at site M, and average performance in the absence of treatment at the inference site $N$ : $Δ_{Q E 2} = Y_{M} (T) – Y_{N} (C)$ . The result, used to infer average impact for site N is biased by the amount $B i a s 1 – B i a s 2$ .

The two options in Scenario 3 raise the following question: For Scenario 3, what is the better choice for achieving lower net bias, when inferring impact at N ? Is it the comparison group-based result, $Δ_{Q E 2}$ , which is susceptible to both $B i a s 1$ and $B i a s 2$ , or the experiment-based quantity $Δ_{X P | M}$ , which is susceptible to $B i a s 1$ only? On an intuitive level, the second option seems preferable because it is potentially affected by only one form of bias. But is the second option always better?

This question is addressed in the next section. The alternatives discussed above are summarized in Table 1.

Table 1.

Scenarios for Inferring the Average Causal Impact of Program T to Site $N$ Through a Comparison of Outcomes from Site $M$ .

Causal quantity of interest	Missing component	Quantity used to infer impact at N	Bias
$Δ_{X P \| N} = Y_{N} (T) - Y_{N} (C)$	None (Scenario 1)	$Δ_{X P \| N}$	0
	$Y_{N} (C)$ (Scenario 2)	Option 1: $Δ_{X P \| M} = Y_{M} (T) - Y_{M} (C)$	$B i a s 1 = Δ_{X P \| M} - Δ_{X P \| N}$
	$Y_{N} (C)$ (Scenario 2)	Option 2: $Δ_{Q E 1} = Y_{N} (T) - Y_{M} (C)$	$B i a s 2 = Y_{N} (C) - Y_{M} (C)$
	$Y_{N} (T)$ (Scenario 3)	Option 1: $Δ_{X P \| M} = Y_{M} (T) - Y_{M} (C)$	$B i a s 1 = Δ_{X P \| M} - Δ_{X P \| N}$
	$Y_{N} (T)$ (Scenario 3)	Option 2: $Δ_{Q E 2} = Y_{M} (T) - Y_{N} (C)$	$\begin{aligned} B i a s 1 - B i a s 2 \\ = Δ_{X P \| M} - Δ_{X P \| N} \\ - [Y_{N} (C) - Y_{M} (C)] \end{aligned}$

Clarifying the Conditions Under Which a Quasi-Experimental Generalization Is Preferable to an Experiment-Based One

Each of Scenarios 2 and 3 explores two options for inferring impact at N, and the two scenarios suggest different rules for deciding the preferred alternative.

Scenario 2

When everyone at N (the inference site) receives treatment, impact may be inferred using $Δ_{X P | M}$ which is susceptible to $B i a s 1$ , or $Δ_{Q E 1}$ , which is susceptible to $B i a s 2$ . This work assumes that the preferred option is the one with a lower magnitude of net bias. The question is addressed below by empirically comparing the magnitudes of estimates of each type of bias.

Scenario 3

When no one at N (the inference site) has received treatment, impact may be inferred using $Δ_{X P | M}$ , which is susceptible to $B i a s 1$ , or $Δ_{Q E 2}$ , which is susceptible to both biases, specifically, net bias expressed in terms of the difference: $B i a s 1 – B i a s 2$ . As with Scenario 2, this work assumes the better choice is the one with a lower magnitude of net bias. To determine this, a comparison can be made between $| B i a s 1 |$ and $| B i a s 1 – B i a s 2 |$ , or similarly, between $B i a s 1^{2}$ and $(B i a s 1 – B i a s 2)^{2}$ .

Before moving to an empirical exploration of the questions, a further numerical comparison of the two options clarifies the conditions under which each should be preferred. Applying the criterion of lower net bias, $Δ_{Q E 2}$ should be preferred when the following inequality is satisfied:

(B i a s 1 – B i a s 2)^{2} < B i a s 1^{2}

(9)

This can be rewritten as:

(B i a s 1^{2} – 2 B i a s 1 B i a s 2 + B i a s 2^{2}) < B i a s 1^{2}

(10)

Cancelling terms on both sides:

(B i a s 2^{2} – 2 B i a s 1 B i a s 2) < 0

(11)

For this to be true,

2 B i a s 1 B i a s 2

has to be positive, which implies the biases have the same sign. If both biases are positive then the following must hold:

B i a s 1

> 1/2

B i a s 2

, and if both are negative the following must be true:

B i a s 1

< 1/2

B i a s 2

Figure 5 shows the relations graphically. It displays the space in which $B i a s 1$ and $B i a s 2$ are distributed. The gray region designates the region in which $Δ_{Q E 2}$ is preferable to $Δ_{X P},$ given the criterion of lower net bias. A circular shape representing a hypothetical confidence interval for a joint distribution of $B i a s 1$ and $B i a s 2$ is overlaid, and represents the case in which there is no correlation between $B i a s 1$ and $B i a s 2$ . Various distributions are imaginable and will depend on the program type, the attributes of persons in the sample as they affect outcomes and moderate impact, and the outcome on which impact is being evaluated using a given assessment.⁹

Before further discussing the alternatives, it is important to consider whether satisfying either of these conditions is even plausible. First, can the conditions in equation (11) be satisfied for certain plausible values of bias? Past empirical work shows that without adjustment for effects of covariates, $B i a s 2$ can achieve levels of .20 standard deviations of the outcome variable (Unlu et al., 2021), and $B i a s 1$ can achieve similar levels (Jaciw et al., 2022; Orr et al., 2019). Using these values and allowing some variability in magnitudes of each type of bias, suggests that the inequality in equation (11) is not unexpected (e.g., based on past empirical results one can imagine situations in which $B i a s 1$ > 1/2 $B i a s 2$ [when both biases are positive] or $B i a s 1$ < 1/2 $B i a s 2$ [when both biases are negative]).

Second, given that the advantage of $Δ_{Q E 2}$ over $Δ_{X P}$ is expected to occur when the two biases have the same sign, is this condition plausible? That is, are there experimental evaluation scenarios in the real world in which the biases take the same sign? Presence of $B i a s 2$ implies a difference in average achievement between N and M in the absence of treatment. Presence of $B i a s 1$ implies average impact varies between N and M. For the biases to have the same sign, the differences have to be in the same direction; that is, average performance in the absence of treatment must be lower, and impact higher, at the comparison site, M, relative to the inference site (see Scenario A in Figure B1 in Appendix B) or average performance in the absence of treatment must be higher, and impact lower, at the comparison site, M, relative to the inference site (see Scenario D in Figure B1 in Appendix B). These situations are plausible under implementation of more-equitable programs that focus improvements for sites in which students on average perform on the lower end of the incoming achievement scale. With the use of such programs, it is reasonable to expect greater impacts for lower achieving students. Under these conditions, same-sign biases are a precondition to achieving lower net bias when using $Δ_{Q E 2}$ instead of $Δ_{X P | M}$ to infer impact at $N .$ ¹⁰

Having established the condition under which $Δ_{Q E 2}$ is preferable to $Δ_{X P | M}$ for assessing impact at N using outcomes from M, and having shown that the level of each form of bias to support a preference for $Δ_{Q E 2}$ is plausible both numerically, and can be imagined for real-life scenarios, this work turns to an empirical example.

Method

Setting Up the Empirical Example

The empirical example of this work extends the methodology of WSC studies to empirically evaluate the levels of each type of bias discussed above. Traditionally, WSC studies are used to evaluate bias in non-experimental estimates relative to experimental benchmarks (pioneering studies are by Lalonde, 1986, and Fraker & Maynard, 1987).

The starting point for a WSC study is typically the impact finding from an uncompromised randomized experiment. This result serves as an unbiased benchmark quantity. A non-experimental result is generated by substituting the outcomes from the experimental control group with those from a different comparison group. The measured change in impact that results from this substitution estimates bias in the quasi-experimental comparison (corresponding to $B i a s 2$ in the current study). WSC studies include use of multisite trials in which controls at one or more sites serve as the non-experimental comparison group for an inference sample consisting of one or more of the other site(s) (e.g., Bloom et al., 2005; Wilde & Hollister, 2007). After the levels of bias are established, design and analysis strategies are evaluated for their potential to reduce bias. There are many examples of WSC studies (Wong et al., 2018 cite 66, and there are more recent ones, such as by Unlu et al., 2021).

Recently, WSC methods and other similar approaches have been extended to empirically evaluate discrepancies in generalized causal inferences from benchmark experimental impacts (Dehejia et al., 2021; Jaciw, 2016; Jaciw et al., 2022; Kern et al., 2016; Orr et al., 2019). The approaches parallel the standard one described above—the benchmark impact estimate from an uncompromised experiment at a site is replaced by a generalized impact quantity based on XP-based findings from one or more other sites. The difference between the generalized impact and the benchmark impact for the site reflects bias in the former quantity (corresponding to $B i a s$ 1 in the current work). Consistent with standard applications of WSC studies, design and analytic strategies are then evaluated in terms of their capacity to reduce bias.

The current work uses the “multisite variant” of the WSC method noted above (Bloom et al., 2005; Michalopoulos et al., 2004; Wilde & Hollister, 2007). With this approach, each site in a multisite experiment has a benchmark true value that may be estimated without bias through an uncompromised XP (i.e., each can play the role of inference site $N$ ), and controls at some or all of the remaining sites serve as the comparison group that yields an alternative quasi-experimental result (i.e., they play the role of $M$ ). The empirical study of the current work is an application of WSC methods in which estimates of the alternative generalized impact quantities $Δ_{X P | M}$ , $Δ_{Q E 1}$ , $Δ_{Q E 2}$ are compared to the estimated benchmark impacts for each inference site, N, and estimated bias is evaluated for each alternative (i.e., through estimates of $B i a s 1$ , $B i a s 2$ , and $B i a s 1 – B i a s 2$ , respectively.) The steps in the application of this WSC methods are discussed in the following section.

Going forward, it is important to keep in mind that each WSC study is just one of potentially many. Multiple WSC studies are necessary to establish the empirical distributions of bias. That is, no single WSC study yields results that are definitive for understanding general conditions for bias. The accumulation of results from many studies provides the opportunity to develop research summaries of the findings, including specific procedural rules for avoiding or lowering bias that have been shown to work across many WSC evaluations. Summaries of this kind in traditional WSC evaluations include works by Bloom et al. (2005), Cook et al. (2008), and Glazerman et al. (2003). The results from the current study should therefore be considered as providing one piece of the evidence for evaluating how discrepant XP-based and QE-based generalizations are from the XP benchmarks for individual sites. Multiple similar efforts will help to consolidate what is known about conditions for bias more generally. The current work should also be considered a proof of concept, because it is a novel application of WSC methods.¹¹

Steps in the Method

Before describing the three steps of the method, one modification is presented. The causal generalizations discussed so far have been between just two sites, the inference (generalized to) site N, and the comparison (generalized from) site M. Going forward, instead of comparisons of outcomes being drawn with just one other site ( $M$ ), they will be drawn between the outcome for the inference site and the average of outcomes across all other sites. This has three advantages: (1) it allows testing the accuracy of “large to small” generalized inferences—corresponding to the meaningful and policy-relevant question of whether findings assessed on a larger scale apply on the smaller scale (Jaciw, 2016; Jaciw et al., 2022; Orr et al., 2019; Shadish et al., 2002), (2) it allows discrepancies from benchmark site-specific impacts to be summarized in a convenient way, specifically, as variance expressions (a point we develop below), and (3) with multiple such comparisons, it is possible to evaluate the reduction in the average magnitude of bias conditional on the effects of site-level (macro) variables (i.e., through application of specific types of “model-based adjustments” noted by Hotz et al. (2005)).

Step 1. Express $B i a s 1$ and $B i a s 2$ for a Given Site j .

$B i a s 1$ with respect to inference site $S = j$ is impact averaged across all sites except j, ${\bar{Δ}}_{X P | S \neq j}$ , minus impact at site j, ${\bar{Δ}}_{X P | S = j} . (S = 1, 2... N)$ .

B i a s 1_{j} = {\bar{Δ}}_{X P | S \neq j} – Δ_{X P | S = j}

(12)

B i a s

2 with respect to in inference site j is the control performance at site

S = j

Y (C)_{| S = j}

, minus the average of control performance across all sites except j,

\bar{Y} (C)_{| S \neq j}

B i a s 2_{j} = Y (C)_{| S = j} – \bar{Y} (C)_{| S \neq j}

(13)

Step 2. Express Bias in

Δ_{X P}

Δ_{Q E 1}

, and

Δ_{Q E 2}

for Each of N Sites:

Site-specific biases in $Δ_{X P}$ are as follows:

B i a s 1_{| S = 1} = ({\bar{Δ}}_{X P | S \neq 1} – Δ_{X P | S = 1})

(14)

B i a s 1_{| S = 2} = ({\bar{Δ}}_{X P | S \neq 2} – Δ_{X P | S = 2})

(15)

…

B i a s 1_{| S = N} = ({\bar{Δ}}_{X P | S \neq N} – Δ_{X P | S = N})

(16)

Site-specific biases in

Δ_{Q E 1}

are as follows:

B i a s 2_{| S = 1} = (Y {(C)}_{| S = 1} – \bar{Y} {(C)}_{| S \neq 1})

(17)

B i a s 2_{| S = 2} = (Y (C)_{| S = 2} – \bar{Y} (C)_{| S \neq 2})

(18)

…

B i a s 2_{| S = N} = (Y (C)_{| S = N} – \bar{Y} (C)_{| S \neq N})

(19)

Site-specific biases in

Δ_{Q E 2}

are as follows:

B i a s 1_{| S = 1} – B i a s 2_{| S = 1} = ({\bar{Δ}}_{X P | S \neq 1} – Δ_{X P | S = 1}) – (Y {(C)}_{| S = 1} – \bar{Y} {(C)}_{| S \neq 1})

(20)

B i a s 1_{| S = 2} – B i a s 2_{| S = 2} = ({\bar{Δ}}_{X P | S \neq 2} – Δ_{X P | S = 2}) – (Y (C)_{| S = 2} – \bar{Y} (C)_{| S \neq 2})

(21)

…

B i a s 1_{| S = N} – B i a s 2_{| S = N} = ({\bar{Δ}}_{X P | S \neq N} – Δ_{| S = N}) – (Y (C)_{| S = N} – \bar{Y} (C)_{| S \neq N})

(22)

Step 3. Summarize Bias for Each Alternative ( $Δ_{X P}, Δ_{Q E 1}$ and $Δ_{Q E 2,}$ ) Using Means of Squared Differences (i.e., Mean Squared Bias [MSB])

If overall bias was summarized by simply averaging over site-specific biases, then cancellation of positive and negative values would result in the underestimation of the average magnitude of bias (Bloom et al., 2005). Summarizing average levels of bias using the mean squared bias is one way to avoid this problem:

For $Δ_{X P}$ :

M S B_{X P} = \frac{1}{N} \sum_{j = 1}^{N} ({\bar{Δ}}_{X P | S \neq j} – Δ_{X P | S = j})^{2}

(23)

For $Δ_{Q E 1}$ :

M S B_{Q E 1} = \frac{1}{N} \sum_{j = 1}^{N} (Y {(C)}_{| S = j} – \bar{Y} {(C)}_{| S \neq j})^{2}

(24)

For $Δ_{Q E 2}$

\begin{aligned} M S B_{Q E 2} & = \frac{1}{N} \sum_{j = 1}^{N} [({\bar{Δ}}_{X P | S \neq j} – Δ_{X P | S = j}) – (Y {(C)}_{| S = j} – \bar{Y} {(C)}_{| S \neq j})]^{2}, \\ = \frac{1}{N} \sum_{j = 1}^{N} (Y {(C)}_{| S = j} – \bar{Y} {(C)}_{| S \neq j})^{2} + \frac{1}{N} \sum_{j = 1}^{N} ({\bar{Δ}}_{X P | S \neq j} – Δ_{X P | S = j})^{2} \\ - \frac{2}{N} \sum_{j = 1}^{N} (Y {(C)}_{| S = j} – \bar{Y} {(C)}_{| S \neq j}) ({\bar{Δ}}_{X P | S \neq j} –_{X P | S = j}) \\ = \frac{1}{N} \sum_{j = 1}^{N} (Y {(C)}_{| S = j} – \bar{Y} {(C)}_{| S \neq j})^{2} + \frac{1}{N} \sum_{j = 1}^{N} (Δ_{X P | S = j} – {\bar{Δ}}_{X P | S \neq j})^{2} \\ + \frac{2}{N} \sum_{j = 1}^{N} (Y {(C)}_{| S = j} – \bar{Y} {(C)}_{| S \neq j}) (Δ_{X P | S = j} – {\bar{Δ}}_{X P | S \neq j}) \end{aligned}

(25)

A point to emphasize here is that

M S B

for

Δ_{X P}

(equation 23) is an expression for the cross-site variance in impact. Similarly,

M S B

for

Δ_{Q E 1}

(equation 24) is an expression for the cross-site variance of average performance in the absence of treatment, and

M S B

for

Δ_{Q E 2}

(equation 25), is the sum of these quantities plus twice the covariance between terms

– B i a s 1

and

B i a s 2

These expressions are convenient for addressing the main questions of this work, including about the degree of bias in comparison-group-based and XP-based generalizations when compared to experimental benchmarks. It is expected that on average the magnitude of bias in $Δ_{Q E 2}$ is less than the magnitude of bias in $Δ_{X P}$ , when the following condition is satisfied:

\begin{aligned} M S B_{Q E 2} < M S B_{X P} \\ \Leftrightarrow \frac{1}{N} \sum_{j = 1}^{N} (Y {(C)}_{| S = j} – \bar{Y} {(C)}_{| S \neq j})^{2} + \frac{2}{N} \sum_{j = 1}^{N} (Y {(C)}_{| S = j} – \bar{Y} {(C)}_{| S \neq j}) (Δ_{X P | S = j} – {\bar{Δ}}_{X P | S \neq j}) < 0 \end{aligned}

(26)

This requires that each of the following conditions is met:

\frac{2}{N} \sum_{j = 1}^{N} (Y {(C)}_{| S = j} – \bar{Y} {(C)}_{| S \neq j}) (Δ_{X P | S = j} – {\bar{Δ}}_{X P | S \neq j}) < 0

(27)

| \frac{2}{N} \sum_{j = 1}^{N} (Y {(C)}_{| S = j} – \bar{Y} {(C)}_{| S \neq j}) (Δ_{X P | S = j} – {\bar{Δ}}_{X P | S \neq j}) | > | \frac{1}{N} \sum_{j = 1}^{N} {(Y {(C)}_{| S = j} – \bar{Y} {(C)}_{| S \neq j})}^{2} |

(28)

Equation (27) shows the requirement of a negative covariance between site deviations in average achievement, and site deviation in impact. The implication is that

B i a s 1

and

B i a s 2

must have the same sign, in expectation.¹² (The conditions described here capture, on the aggregate, the criteria discussed earlier for the two-site case [i.e., the condition in equations 9

–

11]).

Step 4. Adjust for Effects of Confounders and Moderators

As with standard WSC studies, a salient question is whether quantities that summarize bias, in this case the values of $M S B$ , are reduced in magnitude after conditioning on effects of covariates. More specifically, the question is whether adjusting for effects of covariates lowers (1) $M S B_{X P}$ by reducing the influence of moderators of impact, (2) $M S B_{Q E 1}$ by limiting the influence of confounders that affect average achievement, and (3) $M S B_{Q E 2}$ by reducing the influence of either confounders or moderators. To address this question, estimation will be first of the quantities in equations 23 $–$ 25 above without conditioning on effects of covariates: ${\hat{M S B}}_{X P}$ , ${\hat{M S B}}_{Q E 1}$ , and ${\hat{M S B}}_{Q E 2}$ ; and second, after conditioning on the set of covariates, X : ${\hat{M S B}}_{X P | X}$ , ${\hat{M S B}}_{Q E 1 | X}$ , and ${\hat{M S B}}_{Q E 2 | X}$ . The main results will be reported as the square root of these quantities divided by the pooled standard deviation of the outcome variable (i.e., in the metric of the standardized effect size), to allow a direct comparison with average program impacts, which are commonly also reported in terms of effect sizes. The main reported quantities are as follows: $\frac{{\hat{R M S B}}_{X P}}{S D}$ , $\frac{{\hat{R M S B}}_{Q E 1}}{S D},$ $\frac{{\hat{R M S B}}_{Q E 2}}{S D},$ $\frac{{\hat{R M S B}}_{X P | X}}{S D},$ $\frac{{\hat{R M S B}}_{Q E 1 | X}}{S D},$ and $\frac{{\hat{R M S B}}_{Q E 2 | X}}{S D}$ .

Identifying Sources of Variation in Outcomes Across Sites

Estimates of $M S B$ reflect sources of random sampling error. A secondary question of this work is whether, in the case of multisite designs with randomization of both students and teachers within sites, results are sensitive to modeling the intermediate teacher level. It is important to know whether ignoring the intermediate level results in substantive differences in estimates of the $M S B$ terms. The generalizations discussed in this work are concerned with differences in impact across sites, and not with differences between sites that arise from class-level perturbations in outcomes within sites; therefore, it is important to make sure that measurements of the former are not conflated with effects of the latter.

Research Questions

Using the multisite experiment version of a WSC design, what are the estimated average magnitudes of the discrepancies between site (benchmark) impacts and the three different versions of generalized impact corresponding to $Δ_{X P}$ , $Δ_{Q E 1}$ and $Δ_{Q E 2}$ , expressed in the metric of the Standardized Root Mean Squared Bias, and are the estimated average magnitudes statistically significant?

How do the magnitudes of the RMSB quantities in (1) compare to each other?

These questions will be addressed prior to and after covariate adjustments, and with and without adjusting for effects of class-level random sampling error. Given the focus of this work, the contrast of main interest is between estimates of Root Means Squared Bias for the experiment-based generalization,

\frac{{\hat{R M S B}}_{X P}}{S D}

, and for the comparison-group based result

\frac{{\hat{R M S B}}_{Q E 2}}{S D}

Estimation

Hierarchical Linear Models (HLMs) (Raudenbush & Bryk, 2002) are used to obtain the following estimates: ${\hat{M S B}}_{X P}$ , ${\hat{M S B}}_{Q E 1}$ , ${\hat{M S B}}_{Q E 2}$ , ${\hat{M S B}}_{X P | X}$ , ${\hat{M S B}}_{Q E 1 | X}$ and ${\hat{M S B}}_{Q E 2 | X}$ . Random Intercept Random Coefficient (RIRC) models are used in estimation (Jaciw et al., 2022; Miratrix et al., 2021). The precedent for using HLMs in applications of WSC is in Jaciw (2016) and Jaciw et al. (2022).¹³ (The full details of the models used are provided in Appendix C.)

Application

The application uses data from the Tennessee STAR (Student-Teacher Achievement Ratio) class size reduction multisite trial. Results from the study are reported by Finn and Achilles (1990), Mosteller (1995), and Nye et al. (1999, 2000, 2001, 2002). The multisite trial started in 1985 and lasted 4 years. In the original study, 6,400 kindergarten students and their teachers were randomly assigned to (1) small classes (13–17 students), (2) regular classes (22–25 students), or (3) regular classes with an aide within each of 79 schools (sites). Approximately 100 classes were allotted to each arm of the trial. The intervention continued through third grade. Teachers were randomly assigned to conditions within grades as the student cohort moved from kindergarten through third grade. The design aimed to retain students in the condition to which they were originally assigned over that period. Students who joined the study in intervening years were randomly assigned to conditions. In previous studies, regular classes, with or without an aide, are considered the control group, and this approach is adopted in this work. Having multiple control classes per school, allows estimation of sampling error for intermediate (class) units, which helps to address the question of the effect of ignoring the teacher level in estimates of MSB.¹⁴

In the original study, outcomes were assessed on reading and math in kindergarten through third grade using the SAT-7 assessment. For the demonstration of this work, impacts are evaluated on second grade reading outcomes. The sample consists of students with posttests, who joined the study either in kindergarten or in first grade, and who remained in their assigned condition through second grade. To maintain a hierarchically structured dataset (i.e., with students nested in schools) analysis is limited to students who remained in the same school through second grade. Applying these criteria, the final dataset consists of 3,452 students among 314 second grade classrooms, among 73 schools.¹⁵

The estimates of RMSB reported below are obtained: (a) prior to covariate adjustments; (b) after adjusting for effects of student-level school-centered variables (gender, eligibility for Free or Reduced Price Lunch, minority [non-White] status, years of experience teaching by the student's teacher, whether a student's teacher holds a Master's degree or higher, and end-of-kindergarten scores in math and reading, and the interactions of these covariates with treatment); and (c) after adjusting for the effects of both the variables in (b) and site-level (i.e., macro) variables, including site averages of uncentered student-level variables, and school urbanicity (i.e., whether a school is inner-city, suburban, rural or urban) and their interactions with treatment.¹⁶

Results

The main results are displayed in Table 2 and Figures 6a and 6b. Each triplet of bars shows, from left-to right: $\frac{{\hat{R M S B}}_{X P}}{S D}$ (light gray), $\frac{{\hat{R M S B}}_{Q E 2}}{S D}$ (dark gray), and $\frac{{\hat{R M S B}}_{Q E 1}}{S D}$ (black). where $S D$ is the pooled standard deviation of the outcome variable. (Estimates of corresponding variance components are shown in Table E1 in Appendix E.¹⁷)

Table 2.

Estimates of Root Mean Squared Biases for Evaluating Several Approaches to Generalization.

Research question:	Is there bias in the experiment-based (XP) generalization?	Is there bias in the comparison group-based generalization that infers treatment group performance from the comparison site (QE2)?	Is there bias in the comparison group-based generalization that infers control group performance from the comparison site (QE1)?	Is the average magnitude of bias different between the experiment-based (XP) and the non-experimental (QE2) generalizations? (Scenario 3)?	Is the average magnitude of bias different between the experiment-based (XP) and the non-experimental (QE1) generalizations (Scenario 2)?	Is average magnitude of bias different between the comparison group-based generalizations (QE1 and QE2)?
Null hypothesis $(H_{0})$ :	$M S B_{X P} = 0$	$M S B_{Q E 2} = 0$	$M S B_{Q E 1} = 0$	$M S B_{Q E 2} - M S B_{X P} = 0$	$M S B_{Q E 1} - M S B_{X P} = 0$	$M S B_{Q E 1} - M S B_{Q E 2} = 0$
Corresponding bars in Figures 5a and 5b	Height of light gray bar	Height of dark gray bar	Height of black bar	Difference: dark gray bar – light gray bar	Difference: black bar – light gray bar	Difference: Black bar – dark gray bar
	Estimates of RMSB (or differences in RMSB) expressed in SD units (not controlling for class-level random sampling error)
No covariates	.31****	.46***	.45****	.15*	.14**	-.01
^aStudent-level (site-centered) covariates only	.31****	.47***	.44****	.16*	.13**	-.03
^a Student-level (site-centered) and site-level covariates	.29***	.34***	.25****	.05	-.04	-.09
	Estimates of RMSB (or differences in RMSB) expressed in SD units (controlling for class-level random sampling error)
No covariates	.18	.40**	.43****	.22*	.25**	.03
^aSite-centered student-level covariates only	.17	.42**	.42****	.25**	.25**	0.00
^a Student-level (site-centered) and site-level covariates	.14	.27	.19**	.13	.05	-.08

Note. The values displayed are the square roots of estimates of corresponding MSB divided by the standard deviation of the outcome variable. Expression in the metric of the standardized effect size allows comparison with values for average impact and yearly expected growth. For example, the average impact of small classes on second grade reading performance in this experiment is .24-.25 (similar to results in Nye, Hedges and Konstantopoulos, 2000). For reference, annual expected growth in second grade reading scores is approximately .60 standard deviations (Hill et al., 2008).

Covariates at the student level (all school centered) are: gender (1=male, 0=female), eligibility for Free or Reduced Price Lunch (1=eligible, 0=non-eligible), and minority (non-White) status (1=minority, 0=non-minority), the years of teaching experience of a student's teacher, whether the student's teacher holds a Master's degree or higher (1=yes, 0=no), and end of kindergarten scores on tests of math and reading. Covariates at the school level are school averages of uncentered student-level covariates and variables indicating school urbanicity (whether a school is inner-city, suburban, rural or urban.)

Models include the main effects of the covariates and their interactions with treatment.

****p<.001, ***p<.01, **p<.05, *p<.10.

What Is Observed? How Large on Average are the Discrepancies of Generalized Quantities From Experimental Benchmarks?

Estimates of RMSB/SD for XP. Before adjusting for class-level random sampling error, estimates of standardized $R M S B$ for the experiment-based (XP) generalizations are 0.31, 0.31 and 0.29, without covariate adjustment, with adjustment for student-level site-centered covariates, and with further adjustment for school-level covariates, respectively. After figuring in effects of class-level random sampling error, the corresponding quantities are 0.18, 0.17 and 0.14. All estimates are statistically significant before adjusting for the class-level random sampling error (at level α=.05), but none are after this adjustment.

Estimates of RMSB/SD for QE2. The estimates of standardized $R M S B$ for the comparison group-based generalization that involves drawing a comparison with outcomes for cases assigned to treatment at other sites (QE2), are 0.46, 0.47 and 0.34 in the metric of the standardized effect size, before controlling for the class level random sampling error, and 0.40, 0.42 and 0.27, after. (They are reported in the same order as for XP above.) All estimates are statistically significant (at level α=.05) with the exception of the model that adjusts for class-level sampling error and includes all covariates.

Estimates of RMSB/SD for QE1. The estimates of standardized $R M S B$ for the comparison group-based generalization that involves drawing a comparison with outcomes for cases assigned to control at other sites (QE1), are 0.45, 0.44 and 0.25 (reported in the same order as for XP above), before controlling for the class level random sampling error, and 0.43, 0.42 and 0.19, after. All estimates are statistically significant (at level α=.05).

What Are the Main Takeaways From the Results?

The main takeaway points are separated into two types. The first address specifically results from Project STAR. The second address general implications for WSC studies of this kind. The focus is primarily on the difference between XP and QE2.

With reference to the results from Project STAR: How do the magnitudes of the RMSB quantities compare to each other? (1) The generalization from XP consistently outperforms the QE2 result, in terms of yielding a lower-bias result; (2) Modeling teacher effects leads to a substantial reduction in between-site variation in outcomes. Overall, adjusting for the class-level random effect leads to a statistically significant reduction in the site-level variance components based on the change in the deviance statistic (the results are shown in Appendix E). The effect is largest for the XP design, with standardized RMSB decreasing from 0.31 to 0.18, without covariate adjustments, and from 0.29 to 0.14 with covariate adjustments. The reductions are less for QE2, with standardized RMSB decreasing from 0.46 to 0.40 without covariate adjustments, and from 0.34 to 0.27 with covariate adjustments. (3) Adjusting for effects of site-centered student-level variables is not expected to result in reduction in between-school variation in outcomes because such variables essentially difference away between-cluster differences. On the other hand, adjusting for effects of site-level variables, including site means of student variables, is expected to lower between-site variation in outcomes, and therefore values of standardized RMSB discussed in this work. This was confirmed. Adjusting for effects of site-level covariates more-fully reduces the standardized RMSB for QE2 than for XP. Specifically, the estimate of RMSB/SD for QE2 is lowered from 0.46 to 0.34 in the absence of teacher effects, and from 0.40 to 0.27 with teacher effects; in contrast, the estimate for RMSE/SD for XP is reduced from 0.31 to 0.29 in the absence of teacher effects, and from 0.18 to 0.14, with teacher effects. A possible explanation for this is that in the case of QE2, there is more bias to begin with, and therefore, greater opportunity to reduce $B i a s 1$ and $B i a s 2$ through modeling variables that operate both as confounders that affect average achievement and as moderators of impact that contribute between-site differences in program impact; (4) With adjustment for both teacher effects, and effects of student- and site-level covariates, XP still outperforms QE2. The corresponding estimates of RMSB/SD are 0.14 and 0.27, respectively (5) Even after adjusting for effects of covariates and teacher effects, the remaining levels of average bias (in 4, above) can be considered substantive (Hill et al., 2008).

Main takeaways with a view to WSC studies generally. As is the case with all WSC research efforts, it is important to not read too much into the results from a single study. Each WSC study is just one in a potential sample of studies that may be used together to establish more-completely the conditions for bias, and reductions in bias, for alternative generalized impact quantities. The cumulative results of many WSC studies, like the one conducted here, can provide more robust answers. Summaries such as those developed by Glazerman et al. (2003) and Bloom et al. (2005) in their reviews of standard WSC studies, will begin to give such answers. The results using Project STAR in this work may be considered an initial step in a potential cumulative and empirical evaluation effort that applies WSC methods to compare alternative solutions to problems of causal generalization.

One conclusion from the derivations of this work is that when there is no opportunity for covariate adjustments and no information about the relationship between $B i a s 1$ and $B i a s 2$ , the result from an uncompromised XP conducted elsewhere is likely the better option using the criterion of lower net bias. Figure 5, which represents the simpler case, is sufficient to make the point. Under the assumption that $B i a s 1$ and $B i a s 2$ are uncorrelated (their bivariate distribution is circular), the gray-shaded region, which indicates the cases where net bias is lower for QE2 than for XP, is less extensive.

However, under different situations and with more information, the preference could change. Consider first if there is a positive correlation between $B i a s 1$ and $B i a s 2.$ This may be due to effects of variables, non-uniformly distributed across sites, that influence both performance in the absence of treatment and impact. A positive correlation would lead to some degree of cancellation of terms in $B i a s 1 – B i a s 2$ , and therefore a reduction in the net bias in QE2. Looking at Figure 5, a positive correlation implies greater coverage of the area in which QE2 yields lower net bias. Further, covariance adjustments may reduce bias due to effects of moderators, or confounders, or both, and to different extents, thereby reducing the area and shape of the distribution for bias in Figure 5. If covariate adjustments have a predictable effect on the magnitudes of, and more completely, on the distributions of $B i a s 1$ and $B i a s 2$ (and $B i a s 1 – B i a s 2$ , the net bias for QE2) then this would clarify the more general conditions under which one or the other solution for generalizing is preferable. Identifying the situations in which the two alternative generalized quantities perform close to equally well on average may be especially important because it would allow more evidence, from different study design types, to inform causal generalizations.

Addressing Possible Limitations of the Empirical Application

Sampling Error. Estimates of $M S B$ used to index generalizability reflect between-school differences in average performance and impact, as well as random sampling error attributable to classes within schools, and to students within classes. An important result from this study is that estimates of $\frac{R M S B_{X P}}{S D}$ , $\frac{R M S B_{Q E 1}}{S D}$ and $\frac{R M S B_{Q E 2}}{S D}$ are sensitive to the inclusion of class-level random sampling error. Ignoring the variance attributable to class-level differences within schools leads to its absorption at the school level, resulting in upward-bias in variance at that level. This is important because estimates of variances at the school-level, both in average achievement and impact, are used to measure the proximity of generalizations to benchmark values. Ignoring class-level sampling variation biases these results. This effect may be more apparent if there are few classes per school, as was the case in the STAR experiment.

In the STAR experiment, both students and teachers were randomly assigned to conditions, making the inclusion of the class random effect sensible; however, with multisite trials, modeling the intermediate level may be important even under different randomization schemes. For instance, if students are randomly assigned to conditions, but teachers are not, then bias may result from teachers’ selecting into conditions (e.g., certain teachers may jockey for the position to teacher students assigned to treatment). Even if randomization is blocked on classes, resulting in balance between conditions on student characteristics within classes, if treatment interacts with class averages of students attributes (or other class-level characteristics) this will be reflected in heterogeneity in impact among classes within sites. Unless these within-school effects are modeled explicitly, they may be misinterpreted as reflecting variability of impact across schools.

Potential for Bias from Overfitting. A possible concern is that the covariate adjustments at the site level induce bias from overspecification, which would raise doubt about the effects of those adjustments. To address this possibility, results were examined from 22 models that included specific subsets of main effects of school-level covariates and their interactions with treatment. The first set of models adjusted for class-level random sampling error. The second set excluded the class-level effects. This yielded 44 results.

Results showed that the relative changes in estimates in school-level variance components were very similar when adding the same sets of school-level covariates to less- or more-parameterized models. That is, inclusion of specific sets of school-level covariates produced similar relative changes in estimated variance components regardless of how many school-level covariates were already in the model. If the models were overfitted, one would expect instead that changes in estimates of variance components from inclusion of additional school-level covariates to depend greatly on how saturated the model already is. The details of these tests are provided in Supplement A.

Continuing Efforts

Replication is needed. Knowledge from WSC studies about conditions for bias is cumulative, and requires replication. As noted earlier, Wong et al. (2018) cite 66 WSC studies that have led to more-general understanding about conditions for $B i a s 2$ . These efforts have led to overarching strategies in the design and analysis of quasi-experiments to reduce selection bias that corresponds to $B i a s 2$ in this work. Other more-recent efforts have applied WSC or WSC-like approaches to questions of external validity (Hotz et al., 2005; Jaciw, 2016; Jaciw et al., 2022; Kern et al., 2016; Orr et al., 2019) with primary focus on $B i a s 1$ . The current work, in a sense, subsumes earlier efforts by considering effects of $B i a s 1$ and $B i a s 2$ as they simultaneously affect the accuracy of causal generalizations. Given the importance of cumulative knowledge about conditions for bias from multiple WSC studies, the results from the single STAR data set explored in this work should not be considered as definitive. However, a qualification is necessary about what is meant here by cumulative knowledge. It is a contention of this work that the focus of cumulative efforts should be on optimizing solutions for a given inference sample. For instance, if it is observed that XP performs better on average, using the criterion of lower magnitude of bias, then this does not justify the conclusion that, as a result, it should always be the first choice by default. The aim instead should be to demarcate when QE2 should be preferred, to allow the best possible design option for a given situation.

This study suggests a focus for future studies of this kind. The results of this work motivate an important question for subsequent replication efforts: Is it the case that, normally, differences between experiment-based and comparison-group-based generalizations are diminished, after adjusting for macro effects, and are resulting differences between the two solutions small enough to not matter? This is important because, if under specific conditions generalizations from comparison group-based designs are as reliable as experiment-based ones, then both designs may be considered well-suited for targeted applications, and as noted above, potentially yield more abundant evidence from evaluations of program effectiveness. Important to exploring this idea is the need to understand the function of covariates both in their direct and moderated relationships to the outcome. For example, we can expect that a baseline measure of achievement to be routinely related to the outcome, but whether a pretest moderates impact will depend on the treatment and whether it is designed to have, or inadvertently has, a different impact depending on a person's incoming performance. The dependence of $B i a s 2$ on specific combinations of moderators and treatments may help to explain why it has been found harder to account for effect heterogeneity across sites by adjusting for effects of moderators (albeit this limitation has been found in studies where adjustments are made in terms of a relatively small number of all-purpose demographic variables [Jaciw et al., 2022; Orr et al., 2019], and methods with more extensive covariates have been more successful.¹⁸)

Conventional WSC studies at providing direction for future work. Continuing work may focus on some of the standard questions typically addressed with traditional WSC studies, including about the benefits of local matching, the effects of adjusting for standard demographic variables as opposed to productive and theory-driven covariates (Shadish et al., 2006) the role of different types of adjustments, including matching on propensity scores (Heckman et al., 1997; Rosenbaum & Rubin, 1983). Recent work has also evaluated different metrics for capturing correspondence between QE and XP results (Steiner et al., 2019), and focused on quantification of discrepancies from benchmarks in terms of policy-relevant metrics (Kruger, 1999, 2000; Orr et al., 2019; Wilde & Hollister, 2007). These extensions may be adapted to the problems studied in this work, and may be earmarked them for the future. It is also important to understand the role of other factors, in addition to moderators of treatment and confounders that limit replication in causal generalization (see Steiner et al., 2019). A search for effective confounders and moderators to account for impact heterogeneity may be fruitless if the variation is attributable to other sources. Separately from WSC studies, the current work also may be developed further through application of a framework for separating estimation error into components due to sample selection and treatment imbalance (Imai et al., 2008), especially in situations where it is assumed that the components are not additive, and in the context of generalization from large-to-small as we have been considering in this work.

Discussion

In their discussion of the prioritization of types of validity, Shadish et al. (2002) emphasize internal validity, as the sine qua non, in the context where “the experimenter can exercise choice within an experiment about how much priority to give each validity type” (p. 100) (with four types being considered: statistical conclusions, construct, internal, and external validity). They note further that “across a program of research, all validity types are a high priority” (p. 102). The approach discussed in the current work may be interpreted as putting internal and external validity on an even keel.

This work may also be seen as making a stronger claim: in application, causal generalization requires consideration of problems of internal and external validity as one, at least in the context of WSC studies. This work presented two options for drawing causal generalizations, XP, subject to $B i a s 1$ , and QE2, subject to net bias $(B i a s 1 – B i a s 2)$ . $B i a s 2$ is the main concern of traditional WSC studies. It is usually regarded as the primary threat to internal validity of inferences from non-experiments. In this work, where the focus is on external validity, it becomes merely one component of an expression for bias among the two causal quantities considered for generalization.

Put somewhat differently, a key idea of this work is that when causal generalization is the goal, bias from confounded selection on factors that affect average achievement in the absence of treatment (factors resulting in $B i a s 2$ ), cannot be considered separately from bias due to confounded selection on factors that affect achievement by way of their interactions with treatment (i.e., moderators leading to $B i a s 1$ ). As noted earlier, traditional WSC studies are almost exclusively concerned with $B i a s 2$ . Calculation of this bias “differences away” performance in the treatment condition (Bloom et al., 2005). Consequently, in standard applications of WSC studies, the role of treatment is immaterial to the discussion of bias. In the case where causal generalization is the goal, this misses half the problem.¹⁹

A central idea of this work is that QE2, expressed as $B i a s 1 – B i a s 2$ , holds a potential advantage. Susceptibility to both forms of bias, as expressed through this difference, does not necessarily mean greater net bias, because the two types of bias may cancel-out each other. One way that this could happen is if variables that are imbalanced between comparison and inference sites affect both average achievement and moderate impact; that is, if confounders are also moderators, and if bias from confounding is opposite in sign to bias from effect moderation. As noted earlier, this situation may arise in situations where impact is more positive for persons who in the absence of treatment tend to perform lower. Specifically in this scenario, impacts from QE2 designs may be equally valid to those from XP designs, which is something to be studied further in future applications of WSC. This work adds to the literature advocating for greater use of quasi-experimental designs to address questions about program impacts (Jaciw, 2023).

The quantity QE2 may seem counter intuitive. An attendee at a conference where this work was presented objected to the possibility that bias in $Δ_{Q E 2}$ could be less than for $Δ_{X P | M}$ . The attendee summarized this intuition by saying that “two wrongs don't make a right.” However, it is shown in this work that because the biases may offset each other, sometimes “two wrongs” when represented in relation to each other, may be closer to the benchmark solution.

Another possible concern is that researchers would never consider a site in which a program has not been implemented as the starting point for estimating impact. The normal course for an inference is to either run an experiment, or if a program has been used at the site, to seek a matched comparison group to establish a counterfactual outcome to persons who are receiving treatment (a standard QE). This work asserts that QE2 is valid, and prioritization of this causal quantity is in keeping with the idea that preferences for validity types are situated, and dependent on the aims of researchers and the maturity of the research program (Shadish et al., 2002). QE2 situates causal validity directly in the practitioner's world, where one must look to either experimental results from studies conducted somewhere else (e.g., through WWC), or a result that uses half the true solution for impact at the inference site (i.e., performance in the absence of treatment) and seeks a matched comparison group to establish a counterfactual outcome to persons who have not received treatment; that is, to infer how the inference group would have performed had they been assigned to treatment. This is the QE2 alternative.

Building on the idea of the prioritization of validity types that is described by Shadish et al. (2002), what are the implications of the approaches developed in this work for how validity types are prioritized? This work shows that the approach to establishing external validity that is dominant in experimental research, namely, run experimental study somewhere to obtain an internally valid causal result → adjust experimental result to support the external validity of a causal generalizations to a specific inference population, is not the only option for establishing a causal generalization. This order, which clearly prioritizes internal validity over external validity, may be the preferred approach for pure researchers who have time to amass a series of internally valid results before exploring the boundary conditions for effects. The alternative casual quantities compared in this work recognize that, in contrast, decision-makers on the ground may need causal evidence that applies directly to their specific context, and that prioritizes the use of performance outcomes from their site in order to increase relevance. The QE2 quantity may seem as an unfamiliar and hard-to-motivate alternative for some, but one much closer to evaluation purposes and relevant to decision points for local decision-makers.

Supplemental Material

sj-docx-1-aje-10.1177_10982140241246208 - Supplemental material for Hold the Bets! Should Quasi-Experiments Be Preferred to True Experiments When Causal Generalization Is the Goal?

Supplemental material, sj-docx-1-aje-10.1177_10982140241246208 for Hold the Bets! Should Quasi-Experiments Be Preferred to True Experiments When Causal Generalization Is the Goal? by Andrew P. Jaciw in American Journal of Evaluation

Footnotes

Declaration of Conflicting Interests

The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Andrew P. Jaciw

Supplemental Material

Supplemental material for this article is available online.

Notes

Appendix

References

Albright

Malloy

T. E.

(2000). Experimental validity: Brunswik, Campbell, Cronbach and enduring issues. Review of General Psychology, 4(4), 337–353. https://doi.org/10.1037/1089-2680.4.4.337

Bell

S. H.

Orr

L. L.

Blomquist

J. D.

Cain

G. G.

(1995). Program applicants as a comparison group in evaluating training programs. Upjohn Institute for Employment Research.

Bloom

H. S.

Hill

C. J.

Riccio

J. A.

(2003). Linking program implementation and effectiveness: Lessons from a pooled sample of welfare-to-work experiments. Journal of Policy Analysis and Management, 22(4), 551–575. https://doi.org/10.1002/pam.10154

Bloom

H. S.

Michalopoulos

Hill

C. J.

(2005). Using experiments to assess nonexperimental comparison -group methods for measuring program effect. In Bloom

H. S.

(Ed.), Learning more from social experiments (pp. 173–235). New York: Russell Sage Foundation.

Campbell

D. T.

Stanley

J. C.

(1963). Experimental and quasi-experimental designs for research. Boston: Houghton Mifflin.

Cole

S. R.

Stuart

E. A.

(2010). Generalizing evidence from randomized clinical trials to target populations: The ACTG-320 trial. American Journal of Epidemiology, 172(1), 107–115. https://doi.org/10.1093/aje/kwq084

Cook

T. D.

Shadish

W. R.

Wong

V. C.

(2008). Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within-study comparisons. Journal of Policy Analysis and Management, 27(4), 724–750. https://doi.org/10.1002/pam.20375

Cronbach

L. J.

(1975). Beyond the two disciplines of scientific psychology. American Psychologist, 30(2), 116–127. https://doi.org/10.1037/h0076829

Cronbach

L. J.

(1982). Designing evaluations of educational and social programs. San Francisco, CA: Jossey-Bass.

10.

Dehejia

Pop-Eleches

Samii

(2021). From local to global: External validity in a fertility natural experiment. Journal of Business and Economic Statistics, 39(1), 217–243. https://doi.org/10.1080/07350015.2019.1639407

11.

Finn

J. D.

Achilles

C. M.

(1990). Answers and questions about class size: A statewide experiment. American Educational Research Journal, 27(3), 557–577. https://doi.org/10.3102/00028312027003557

12.

Fraker

Maynard

(1987). The adequacy of comparison group designs for evaluations of employment-related programs. The Journal of Human Resources, 22(2), 194–227. https://doi.org/10.2307/145902

13.

Gelman

(2005). Analysis of variance = why it is more important than ever. The Annals of Statistics, 33(1), 1–53. https://doi.org/ 10.1214/009053604000001048

14.

Glazerman

Levy

Myers

(2003). Nonexperimental versus experimental estimates of earnings impacts. American Academy of Political and Social Science, 589, 63–93. https://doi.org/10.1177/0002716203254879

15.

Heckman

J. J.

Ichimura

Todd

P. E

. (1997). Matching as an econometric evaluation estimator: Evidence from evaluating a job training programme. Review of Economic Studies, 64(4), 605–654. https://doi.org/10.2307/2971733

16.

Hill

C. J.

Bloom

H. S.

Black

A. R.

Lipsey

M. W.

(2008). Empirical benchmarks for interpreting effect sizes in research. Child Development Perspectives, 2(3), 172–177. https://doi.org/10.1111/j.1750-8606.2008.00061.x

17.

Hotz

V. J.

Imbens

G. W.

Klerman

J. A.

(2006). Evaluating the differential effects of alternative welfare-to-work training components: A reanalysis of the California GAIN Program. Journal of Labor Economics, 24(3), 521–566. https://doi.org/10.1086/505050

18.

Hotz

V. J.

Imbens

G. W.

Mortimer

J. H.

(2005). Predicting the efficacy of future training programs using past experiences at other locations. Journal of Econometrics, 125(1–2), 241–270. https://doi.org/10.1016/j.jeconom.2004.04.009

19.

Hulleman

C. S.

Cordray

D. S.

(2009). Moving from the lab to the field: The role of fidelity and achieved relative intervention strength. Journal of Research on Educational Effectiveness, 2(1), 88–110. https://doi.org/10.1080/19345740802539325

20.

Imai

Kong

Stuart

E. A.

(2008). Misunderstandings between experimentalists and observationalists about causal inference. Journal of the Royal Statistical Society: Series A, 171(Part 2), 481–502. https://doi.org/10.1111/j.1467-985X.2007.00527.x

21.

Jaciw

A. P.

(2016). Assessing the accuracy of generalized inferences from comparison group studies using a within-study comparison approach: The methodology. Evaluation Review, 40(3), 199–240. https://doi.org/10.1177/0193841X16664456

22.

Jaciw

A. P.

(2023). Do social programs help some beneficiaries more than others? Evaluating the potential for comparison group designs to yield low-bias estimates of differential impact. American Journal of Evaluation. https://doi.org/10.1177/10982140231160561

23.

Jaciw

A. P.

Unlu

Nguyen

(2022). A within-study approach to evaluating the role of moderators of impact in limiting generalizations from “large to small.”. American Journal of Evaluation, 43(1), 108–131. https://doi.org/10.1177/10982140211030552

24.

Kern

H. L.

Stuart

E. A.

Hill

Green

D. P.

(2016). Assessing methods for generalizing experimental impact estimates to target populations. Journal of Research on Educational Effectiveness, 19(1), 103–127. https://doi.org/10.1080/19345747.2015.1060282

25.

Krueger

(1999). Experimental estimates of education production functions. Quarterly Journal of Economics, 114(2), 497–532. https://doi.org/10.1162/003355399556052

26.

Krueger

. (2000). The class size policy debate. Economic Policy Institute Working Paper No.121.

27.

Lalonde

(1986). Evaluating the econometric evaluations of training programs with experimental data. The American Economic Review, 76(4), 604–620.

28.

Michalopolous

Bloom

H. S.

Hill

C. J.

(2004). Can propensity-score methods match the findings from a random assignment evaluation of mandatory welfare-to-work programs? The Review of Economics and Statistics, 86(1), 156–179. https://doi.org/10.1162/003465304323023732

29.

Miratrix

L. W.

Weiss

M. J.

Henderson

(2021). An applied researcher’s guide to estimating effects from multisite individually randomized trials: Estimands, estimators, and estimates. Journal of Research on Educational Effectiveness, 14(1), 270–308. https://doi.org/10.1080/19345747.2020.1831115

30.

Mosteller

(1995). The Tennessee study of class size in the early school grades. The Future of Children, 5, 113–127.

31.

Nye

Hedges

L. V.

Konstantopoulos

(2000). The effects of small classes on academic achievement: The results of the Tennessee class size experiment. American Educational Research Journal, 37(1), 123–151. https://doi.org/10.3102/00028312037001123

32.

Nye

Hedges

L. V.

Konstantopoulos

(2001). Are effects of small classes cumulative? Evidence from a Tennessee experiment. The Journal of Educational Research, 94(6), 336–345. https://doi.org/10.1080/00220670109598771

33.

Nye

Hedges

L. V.

Konstantopoulos

. (1999). The long-term effects of small classes: A five-year follow-up of the Tennessee class size experiment. Educational Evaluation and Policy Analysis, 21(2), 127–142. https://doi.org/10.3102/01623737021002127

34.

Nye

Hedges

L. V.

Konstantopoulos

(2002). Do low-achieving students benefit more from small classes? Evidence from the Tennessee class size experiment. Educational Evaluation and Policy Analysis, 24(3), 201–217. https://doi.org/10.3102/01623737024003201

35.

Orr

L. L.

Olsen

R. B.

Bell

S. H.

Schmid

Shivji

Stuart

E. A.

(2019). Using the results from rigorous multi-site evaluations to inform local policy decisions. Journal of Policy Analysis and Management, 38(4), 978–1003. https://doi.org/10.1002/pam.22154

36.

Raudenbush

S. W.

Bryk

A. S.

(2002). Hierarchical Linear Models (2nd ed). Thousand Oaks, CA: Sage.

37.

Rosenbaum

P. R.

Rubin

D. B.

(1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41–55. https://doi.org/10.1093/biomet/70.1.41

38.

Schochet

P. Z.

Puma

Deke

. (2014). Understanding variation in treatment effects in education impact evaluations: An overview of quantitative methods (NCEE 2014–4017). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Analytic Technical Assistance and Development. Retrieved from http://ies.ed.gov/ncee/edlabs

39.

Scriven

(2008). A summative evaluation of XP methodology: & an alternative approach to causal research. Journal of Multidisciplinary Evaluation, 5(9), 11–24. https://doi.org/10.56645/jmde.v5i9.160

40.

Shadish

W. R.

Cook

T. D.

Campbell

D. T.

(2002). Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin.

41.

Shadish

W. R.

Luellen

J. K.

Clark

M. H.

(2006). Propensity scores and quasi-experiments: A testimony to the practical side of lee sechrest. In Bootzin

R. R.

McKnight

P. E.

(Eds.), Strengthening research methodology: Psychological measurement and evaluation (pp. 143–157). Washington, DC: American Psychological Association..

42.

Smith

J. A.

Todd

P. E

. (2005). Does matching overcome Lalonde’s critique of non-experimental estimators? Journal of Econometrics, 125(1-2), 305–353. https://doi.org/10.1016/j.jeconom.2004.04.011

43.

Steiner

P. M.

Wong

V. C.

(2018). Assessing correspondence between experimental and nonexperimental estimates in within-study comparisons. Evaluation Review, 42(2), 214–247. https://doi.org/10.1177/0193841X18773807

44.

Steiner

P. M.

Wong

V. C.

Anglin

(2019). A causal replication framework for designing and assessing replication efforts. Zeitschrift fur Psychologie, 227(4), 280–292. https://doi.org/10.1027/2151-2604/a000385

45.

Tipton

(2013). Improving generalizations from experiments using propensity score subclassification: Assumptions, properties, and contexts. Journal of Educational and Behavioral Statistics, 38(3), 239–266. https://doi.org/10.3102/1076998612441947

46.

Unlu

Lauen

Fuller

S. C.

Berglund

Estera

. (2021). Can quasi-experimental evaluations that rely on state longitudinal data systems replicate experimental results: Findings from a within-study comparison. Forthcoming at Journal of Policy Analysis and Management, 40(2), 572–613. https://doi.org/10.1002/pam.22295

47.

Weidmann

Miratrix

(2021). Lurking inferential monsters? Quantifying selection bias in evaluations of school programs. Journal of Policy Analysis and Management, 40(3), 964–986. https://doi.org/10.1002/pam.22236

48.

Weiss

M. J.

Bloom

H. S.

Brock

(2014). A conceptual framework for studying the sources of variation in program effects. Journal of Policy Analysis and Management, 33(3), 778–808. https://doi.org/10.1002/pam.21760

49.

Wilde

E. T.

Hollister

(2007). How close is close enough? Evaluating propensity score matching using data from a class size reduction experiment. Journal of Policy Analysis and Management, 26(3), 455–477. https://doi.org/10.1002/pam.20262

50.

Wong

V. C.

Steiner

P. M.

Anglin

K. L.

(2018). What can be learned from empirical evaluations of nonexperimental methods? Evaluation Review, 42(2), 147–175. https://doi.org/10.1177/0193841X18776870

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.14 MB

Hold the Bets! Should Quasi-Experiments Be Preferred to True Experiments When Causal Generalization Is the Goal?

Abstract

Keywords

Background

Three Scenarios to Motivate the Main Ideas of This Work

Scenario 1: Establishing the Experimental Benchmark

Scenario 2: Inferring Average Impact at Site N When Treatment Is Provided to Everyone at the Site

Clarifying the Conditions Under Which a Quasi-Experimental Generalization Is Preferable to an Experiment-Based One

Scenario 2

Scenario 3

Method

Setting Up the Empirical Example

Steps in the Method

Step 3. Summarize Bias for Each Alternative ( Δ X P , Δ Q E 1 and Δ Q E 2 , ) Using Means of Squared Differences (i.e., Mean Squared Bias [MSB])

Step 4. Adjust for Effects of Confounders and Moderators

Identifying Sources of Variation in Outcomes Across Sites

Estimation

Application

Results

What Is Observed? How Large on Average are the Discrepancies of Generalized Quantities From Experimental Benchmarks?

What Are the Main Takeaways From the Results?

Addressing Possible Limitations of the Empirical Application

Continuing Efforts

Discussion

Supplemental Material

sj-docx-1-aje-10.1177_10982140241246208 - Supplemental material for Hold the Bets! Should Quasi-Experiments Be Preferred to True Experiments When Causal Generalization Is the Goal?

Footnotes

Declaration of Conflicting Interests

Funding

ORCID iD

Supplemental Material

Notes

Appendix

References

Supplementary Material

Step 3. Summarize Bias for Each Alternative ( $Δ_{X P}, Δ_{Q E 1}$ and $Δ_{Q E 2,}$ ) Using Means of Squared Differences (i.e., Mean Squared Bias [MSB])