Disentangling Person-Dependent and Item-Dependent Causal Effects: Applications of Item Response Theory to the Estimation of Treatment Effect Heterogeneity

Abstract

Analyzing heterogeneous treatment effects (HTEs) plays a crucial role in understanding the impacts of educational interventions. A standard practice for HTE analysis is to examine interactions between treatment status and preintervention participant characteristics, such as pretest scores, to identify how different groups respond to treatment. This study demonstrates that the identical patterns of HTE on test score outcomes can emerge either from variation in treatment effects due to a preintervention participant characteristic or from correlations between treatment effects and item easiness parameters. We demonstrate analytically and through simulation that these two scenarios cannot be distinguished if analysis is based on summary scores alone. We then describe a novel approach that identifies the relevant data-generating process by leveraging item-level data. We apply our approach to a randomized trial of a reading intervention in second grade and show that any apparent HTE by pretest ability is driven by the correlation between treatment effect size and item easiness. Our results highlight the potential of employing measurement principles in causal analysis, beyond their common use in test construction.

Keywords

causal inference heterogeneous treatment effects item response theory psychometrics educational measurement

1. Introduction

Heterogeneous treatment effects (HTEs) are a topic of growing importance and interest as understanding HTE is crucial for determining the policy relevance of interventions. HTE analysis allows researchers to identify the target population for which interventions are most effective. Further, identifying subgroups for whom there might be much larger or smaller treatment effects can lead to efficiency gains if policies are targeted to groups that are most responsive to interventions (Abenavoli, 2019; Blundell et al., 2005; Brand et al., 2014; Bryan et al., 2021; Schochet et al., 2014; Torche et al., 2024).

One challenge with estimating HTE is the risk that researchers may engage in questionable research practices such as selectively reporting subgroups with large effects or p-hacking, thus providing misleading conclusions about the generalizability and replicability of HTE analyses (Schuetze & von Hippel, 2023). Many methods have been proposed to address these problems, such as preregistration (Olken, 2015) and machine learning approaches (Chernozhukov et al., 2022; Wager & Athey, 2018; Wallace et al., 2023; Yeager et al., 2019). These approaches address spurious HTE as a problem of inference. However, even when a researcher takes a principled approach to estimating HTE or employs novel machine learning methods, we show that they may still detect spurious HTE if the relevant outcome is a psychological construct or other form of latent outcome measured using psychometric techniques. This spurious HTE can occur when the items used to construct outcome measures, such as a test of reading comprehension based on a set of test items, themselves exhibit HTE. The present study thus complements the broader HTE literature by reconsidering spurious HTE as a problem of identification instead of a problem of inference. In particular, we show that two qualitatively different data-generating processes (DGPs) can yield identical patterns of HTE when psychometric outcomes, such as educational test scores, are used to estimate treatment effects.

To illustrate this problem, consider a common approach to estimating treatment effects. Education researchers often collect item-level assessment data, generate test scores for each individual from the item responses, and use these scores as outcomes in a standard regression framework to estimate treatment effects. The regression model can then be extended to include interactions between person characteristics and treatment status to probe potential sources of HTE. However, recent scholarship has proposed models where the treatment might differentially impact the individual assessment items (Ahmed et al., 2023; Gilbert et al., 2023b; Sales et al., 2021). Thus, by not modeling individual item-level responses directly, researchers are potentially ignoring the HTE among the items that constitute the outcome measure.

We offer two examples that could give rise to treatment effects that vary at the item level. First, in education, a treatment effect may reflect “teaching to the test” or “score inflation” rather than an improvement in the latent ability giving rise to the item responses (Koretz, 2005). For example, instruction in test-taking strategies such as “process of elimination” could have the effect of making multiple choice items uniformly easier without improving latent student ability. Second, in psychology, spousal loss induces a systematic change in the reporting of depressive symptoms, such that people with otherwise similar levels of depression are much more likely to report the specific symptoms of loneliness and sadness, as opposed to other symptoms such as a loss of motivation (see figure S7 in Domingue et al., 2021). The interpretation of this change for the purposes of diagnosis has been a long-standing challenge (e.g., Olivera-Aguilar & Rikoon, 2024; Zisook et al., 2007). These illustrations are not meant to be exhaustive but rather suggestive of the possibilities that we have in mind when we discuss item-dependent HTE.

In this study, we show that correlations between item-specific treatment effects and item easiness parameters can generate observed HTE patterns indistinguishable from those generated by HTE arising from treatment interacting with person characteristics. We demonstrate this resulting confounding both analytically and through Monte Carlo simulation. To resolve this issue, we propose a novel approach that leverages item-level data to simultaneously estimate treatment by person characteristic interactions as well as the correlation between item-specific treatment effects and item easiness parameters. We show that our approach identifies the relevant DGP using Monte Carlo simulation. Finally, we apply our approach to a randomized evaluation of a second-grade content literacy intervention and show that the observed HTE by pretest scores is driven by the correlation between item-specific treatment effects and the item easiness parameters.

Our study contributes to the burgeoning literature on the estimation of HTE (Athey & Imbens, 2016; Bryan et al., 2021; Chernozhukov et al., 2022; Künzel et al., 2019; Schuetze & von Hippel, 2023; Wager & Athey, 2018; Wendling et al., 2018). In particular, our study extends a literature on the identification and estimation of a latent heterogeneity, that is, the variation in treatment effects that is not driven by baseline variables (Jeon et al., 2021; Lyu et al., 2023; Pearl, 2022; VanderWeele & Batty, 2023; Winship & Morgan, 1999; Xie et al., 2012). We build on recent work describing how item-level assessment data, typically only used to construct outcome measures, can provide additional insights into the nature of treatment effects (Ahmed et al., 2023; Gilbert et al., 2023b; Sales et al., 2021), and more generally, how estimates of treatment impact may be sensitive to measurement properties such as the alignment between interventions and outcomes (Francis et al., 2022), the comparability of effect sizes across studies (Wolf & Harbatkin, 2023), effect moderation and mediation (Montoya & Jeon, 2020; VanderWeele & Vansteelandt, 2022), and the consequences of scoring decisions and outcome metric properties for inference (Domingue et al., 2020, 2022; Gilbert, 2023a; McNeish & Wolf, 2020; Skrondal & Laake, 2001; Soland et al., 2022; Widaman & Revelle, 2023). Our study leverages a measurement model (Briggs, 2008; De Boeck, 2004) within a potential outcomes framework (Holland, 1986; Rubin, 1974) to identify and estimate HTE, effectively integrating the three foundational elements of empirical research—measurement, identification, and inference—into a cohesive framework.

The study is structured as follows. Section 2 sets up a potential outcome framework for the causal inference model and reviews person-dependent HTE and item-dependent HTE DGPs. Section 3 examines how the two DGPs can yield identical observed patterns of heterogeneity and our proposed solution to this problem. Section 4 presents a Monte Carlo simulation study, demonstrating the identification challenge and its resolution. Section 5 presents an empirical application to a randomized evaluation of a content literacy intervention. Section 6 discusses the implications of applying measurement principles in impact evaluation in education and other fields.

2. A Model of Treatment Effects

We begin with a model of treatment effects for item response data. We consider the following DGP for dichotomous item responses on a test used to evaluate the efficacy of a given treatment. The probability of a correct response to item i by person j is

Pr (y_{i j} = 1) = f (η_{i j}) = f (θ_{j}^{1} + b_{i j}),

where f is a monotonically increasing function bounded within $(0, 1)$ . Common choices for f include the inverse logit ( ${logit}^{- 1}$ ) and inverse probit ( $Φ^{- 1}$ ) functions, following a standard Rasch item response model (Baker, 2001; Embretson & Reise, 2013), or more flexible nonparametric approaches (Sijtsma, 2005). $θ_{j}^{1}$ denotes person j’s postintervention ability—we denote preintervention ability as $θ_{j}^{0}$ —and $b_{i j}$ is a parameter that indexes item easiness and that may vary across people. Our use of b as item easiness is the negative of an item’s “difficulty” in the item response theory (IRT) literature. A basic version of Equation 1, where $θ_{j}^{1}$ and $b_{i j}$ are constants and $f (x) = {logit}^{- 1} (x)$ is shown in Panels A1 and A2 of Figure 1.

Figure 1.

Item response functions (IRFs) and treatment effects. Notes. The plots in the top row present the linear predictor and the plots in the bottom row present the probability of correct response. A1 and A2 depict a standard IRF that describes the probability of a correct response as a function of the sum $θ_{j}^{1} + b_{i}$ . In all plots, the dashed black line shows the probability of a correct response under the control counterfactual, or $η_{ij} (0)$ . In B1 and B2, the solid black line shows a uniform treatment effect on the standard IRF. In C1 and C2, solid lines depict the heterogeneous treatment effect. The IRF curves represent potential difference in functioning across items after exposure to treatment, with the blue IRF more impacted by treatment than the red IRF.

Both $θ_{j}^{1}$ and $b_{i j}$ (and therefore $η_{i j}$ ) may be the functions of treatment status and other relevant parameters such as preintervention ability ( $θ_{j}^{0}$ ). Using the potential outcomes framework (Holland, 1986; Rubin, 1974), we define the causal effect of treatment $τ_{i j}$ as the difference in $η_{i j}$ under the counterfactual treatment and control conditions:

τ_{i j} = η_{i j} (1) - η_{i j} (0) .

Suppose that an individual is randomly assigned to treatment or control conditions; we write ${treat}_{j} = 1$ if person j is assigned to treatment. Consider a uniform treatment effect of magnitude $β_{1}$ , so that the intervention leads to an increase in ability for treated individuals,

θ_{j}^{1} = θ_{j}^{0} + β_{1} {treat}_{j},

b_{i j} = b_{i},

which implies $η_{i j} (1) = θ_{j}^{0} + β_{1} + b_{i}, η_{i j} (0) = θ_{j}^{0} + b_{i}$ , and $τ_{i j} = β_{1}$ . The implications of this kind of treatment effect are shown in Panel B of Figure 1; the vertical shift in panel B1 and the horizontal shift in the IRF in B2 would be $β_{1}$ in this example. Note that the shift uniformly affects all items and treated students, though the vertical distances between the curves differ because of the nonlinear transformation to the probability space; the treatment effect is uniform in the latent linear predictor $η_{i j}$ space (Breen et al., 2018; Colnet et al., 2023; Mood, 2010).

As an alternative, if we suppose that the treatment effect is entirely related to change in the functioning of the items,

θ_{j}^{1} = θ_{j}^{0},

b_{i j} = b_{i} + β_{1} {treat}_{j},

we similarly produce $τ_{i j} = β_{1}$ . The fact that item response models are not identified under such scale translations in single timepoint, single-population analysis (San Martín, 2016) is a key component of our argument.

We now introduce HTE into the model. At the person level, we can allow the treatment effect to depend on person characteristic X_j (e.g., pretest scores, gender, socioeconomic status [SES]) by introducing an interaction between ${treat}_{j}$ and X_j :

θ_{j}^{1} = θ_{j}^{0} + β_{1} {treat}_{j} + β_{2} X_{j} + β_{3} {treat}_{j} \times X_{j},

b_{i j} = b_{i} .

At the item level, we can similarly allow the treatment effect size to depend on item characteristic X_i (e.g., item content, item type, position of item in the test):

θ_{j}^{1} = θ_{j}^{0},

b_{i j} = b_{i} + β_{1} {treat}_{j} + β_{2} X_{i} + β_{3} {treat}_{j} \times X_{i} .

We consider HTE in Panel C of Figure 1. The red and blue lines represent different increases in $η_{i j}$ or $Pr (y_{i j} = 1)$ for different values of $η_{i j} (0)$ . One possible interpretation is that the treatment effect depends on student characteristics. For example, if X_j represents baseline ability, a lower ability student represented by the red line shows some increase in their probability of a correct response but this was less of an increase than the one observed by the higher ability student represented by the blue line. An alternative—and mathematically equivalent given that they lead to equivalent changes in $η_{i j}$ —model would be to assume that items are differentially sensitive to treatment. The red IRF would then capture the posttreatment functioning of an item that showed a relatively smaller effect as compared to the blue IRF. We now formalize these ideas, which we describe as person-dependent HTE or item-dependent HTE, respectively, and discuss their relevance for empirical investigations of HTE.

2.1. Person-Dependent HTE

We first consider a DGP in which HTE arises due to variation in the treatment effect as a function of pretreatment ability ( $θ_{j}^{0}$ ):

θ_{j}^{1} = β_{0} + β_{1} {treat}_{j} + β_{2} θ_{j}^{0} + β_{3} {treat}_{j} \times θ_{j}^{0},

b_{i} \sim N (0, σ_{0}) .

$τ_{i j}$ is a function of both $β_{1}$ and $β_{3}$ :

τ_{i j} = η_{i j} (1) - η_{i j} (0),

= (θ_{j}^{1} (1) + b_{i}) - (θ_{j}^{1} (0) + b_{i}),

= (β_{0} + β_{1} + (β_{2} + β_{3}) θ_{j}^{0} + b_{i}) - (β_{0} + (β_{2}) θ_{j}^{0} + b_{i}),

= β_{1} + β_{3} θ_{j}^{0} .

Here, $β_{3}$ is the person-dependent HTE parameter. Suppose $β_{3}$ is 0. In this case, the treatment effect is constant and independent of baseline ability. Further, the average treatment effect ( $ATE = \bar{τ_{i j}}$ ) is simply $β_{1}$ . In contrast, if $β_{3} < 0$ , then there are larger treatment effects when $θ_{j}^{0} < 0$ (and reversed if $β_{3} > 0$ ). We emphasize the fact that the heterogeneity in this model is person-dependent; specifically, it is due to preintervention variation in ability. For identification purposes, we assume here that b_i are normally distributed with mean 0 and some standard deviation (SD) $σ_{0}$ .

2.2. Item-Dependent HTE

We now contrast the person-dependent HTE model with an alternative DGP:

θ_{j}^{1} = γ_{0} + γ_{1} {treat}_{j} + γ_{2} θ_{j}^{0},

b_{i j} = b_{i} + ζ_{i} {treat}_{j},

[\begin{matrix} b_{i} \\ ζ_{i} \end{matrix}] \sim N (0, [\begin{matrix} σ_{0} & ρ \\ ρ & σ_{1} \end{matrix}]) .

Again consider the treatment effect $τ_{i j}$ . In contrast to the person-dependent DGP, here, $τ_{i j}$ is constant as a function of j and, in particular, invariant as a function of $θ_{j}^{0}$ . However, items are differentially affected by treatment, represented by $ζ_{i}$ :

τ_{i j} = η_{i j} (1) - η_{i j} (0),

= (γ_{0} + γ_{1} + γ_{2} θ_{j}^{0} + b_{i} + ζ_{i}) - (γ_{0} + γ_{2} θ_{j}^{0} + b_{i}),

= γ_{1} + ζ_{i} .

The $ζ_{i}$ parameter has been previously described as “item-level” HTE, “treatment by item interactions,” or “item-specific effects” (Ahmed et al., 2023; Gilbert et al., 2023b; Sales et al., 2021). For example, if $ζ_{i} > 0$ , then responses to item i are more likely to be correct if ${treat}_{j} = 1$ , above and beyond the ATE $γ_{1}$ (assumed here to be impacting $θ_{j}^{1}$ directly), independent of $θ_{j}^{1}$ . The $ζ_{i}$ offsets are equivalent to uniform differential item functioning (DIF) effects caused by the treatment (Camilli, 2006; Gilbert et al., 2023b; Montoya & Jeon, 2020; Santelices & Wilson, 2010). In this model, we assume that b_i and $ζ_{i}$ come from a multivariate normal distribution with mean 0 and SDs $σ_{0}$ and $σ_{1}$ and correlation $ρ$ . $ρ$ captures potential associations between item easiness and item-specific treatment effect size. For example, if $ρ > 0$ , then easier items are more likely to be positively affected by treatment. The parameter $ρ$ will play a key role in the appropriate identification of HTE.

A directed acyclic graph representation of the two DGPs is presented in Figure 2. Concretely, the person-dependent HTE model may be theoretically justifiable when an intervention targets students by ability level. For example, an intervention that provides more resources and supports to lower ability students would presumably have a larger effect on low ability students and this effect would be constant across all items (i.e., $β_{3} < 0 {and σ}_{1} = 0$ ). In contrast, the item-dependent HTE model may be theoretically justifiable when an intervention targets skills by complexity level. For example, an intervention that provides foundational skills training to all students would potentially help students at all levels of $θ_{j}^{0}$ equally but show the largest impact on the easiest test items (i.e., $ρ > 0 {and β}_{3} = 0$ ). A treatment that leads to greater gains for low-scoring students will have different policy implications than a treatment that leads to greater learning of easier content for all students.

Figure 2.

Directed acyclic graphs for person-dependent heterogeneous treatment effect (HTE; top) and item-dependent HTE (bottom) data-generating processes. Notes. Squares indicate observed variables, hollow circles indicate latent variables, and solid circles represent cross product interaction terms. I_n are item responses and T_j is the treatment indicator. $β_{3}$ represents the treatment by baseline ability interaction. $ρ$ represents the correlation between item easiness and item-specific treatment effect size. Path coefficients are fixed at 1 unless otherwise indicated.

3. Appropriate Identification of HTE

3.1. The Problem of Identification

We now describe a scenario under which conventional analysis of HTE would leave the DGP unidentified. By this, we mean that the person-dependent or item-dependent HTE processes produce data with equivalent patterns of treatment effects on an estimated posttreatment sum score S_j conditional on $θ_{j}^{0}$ . Our argument hinges on the $ρ$ parameter from Equation 19, which indexes the correlation between item easiness b_i and residual item-specific treatment effect size $ζ_{i}$ .

We use sum scores for clarity of exposition and because—despite active debate among psychometric researchers concerning their properties (McNeish & Wolf, 2020; Soland et al., 2022; Widaman & Revelle, 2023)—they are commonly used as outcomes in empirical research (Flake et al., 2017). However, our argument is not dependent on the use of sum scores. When items are equally discriminating, the sum score is a sufficient statistic for an IRT-based score. In fact, sum scores will be an injective (i.e., one-to-one) function of the true abilities (Birnbaum, 1968; Borsboom, 2005), but monotonic transformations can induce HTE or remove it (Ding et al., 2019; Domingue et al., 2022). Further, our simulations will show that our argument applies equally to latent variable models that estimate the measurement and regression models simultaneously (Lockwood & McCaffrey, 2020).

Let $E (S_{j})$ be the sum of the IRFs. That is, $E (S_{j})$ is the test response function, or TRF, which can also be interpreted as the classical test theory true score (Borsboom, 2005). We define the TRF for the person-dependent process as $E_{P} (S_{j})$ and the TRF for the item-dependent process as $E_{I} (S_{j})$ :

E_{P} (S_{j}) = \sum_{i = 1}^{I} f (η_{i j}) = \sum_{i = 1}^{I} f (β_{0} + β_{1} {treat}_{j} + β_{2} θ_{j}^{0} + β_{3} {treat}_{j} \times θ_{j}^{0} + b_{i}),

E_{I} (S_{j}) = \sum_{i = 1}^{I} f (η_{i j}) = \sum_{i = 1}^{I} f (γ_{0} + γ_{1} {treat}_{j} + γ_{2} θ_{j}^{0} + {treat}_{j} \times ζ_{i} + b_{i}) .

We define the causal estimand at the test score level as $τ_{j}$ , where $S_{j} (\cdot)$ is the potential value of the sum score under the treatment or control counterfactuals:

τ_{j} = E [S_{j} (1) - S_{j} (0)] = E (S_{j} (1)) - E (S_{j} (0)) .

We emphasize that $τ_{j}$ is distinct from $τ_{i j}$ described earlier in Section 2, in which treatment effects were defined in terms of $η_{i j}$ . Here, $τ_{j}$ is averaged across items.

We first illustrate the identification problem with a toy example, setting $β_{0} = 0$ , $β_{1} = 0$ , $β_{2} = 1$ , $γ_{0} = 0$ , $γ_{1} = 0$ , and $γ_{2} = 1$ . In the person-dependent HTE model, we set $β_{3} = - 0.28$ , and in the item-dependent HTE model, we set $ρ = 1$ , so that $ζ_{i} = b_{i}$ . Thus, the treatment is most effective for students with lower $θ_{j}^{0}$ under the person-dependent DGP (for all items), and the treatment is most effective for the easiest items under the item-dependent DGP (for all persons). Further assuming $f () = {logit}^{- 1}$ (i.e., a Rasch or 1PL IRT model), the equations simplify to

E_{P} (S_{j}) = \sum_{i = 1}^{I} {logit}^{- 1} (θ_{j}^{0} - 0.28 {treat}_{j} \times θ_{j}^{0} + b_{i}),

E_{I} (S_{j}) = \sum_{i = 1}^{I} {logit}^{- 1} (θ_{j}^{0} + {treat}_{j} \times b_{i} + b_{i}) .

In Table 1, we calculate $E (S_{j})$ for three items with $b_{i} = - 1, 0, and 1$ and three individuals with $θ_{j}^{0} = - 1, 0, and 1$ under both treatment and control counterfactuals. We see that $E (S_{j})$ are identical under both DGPs, which implies that $τ_{j}$ is identical. We also see that $τ_{j}$ is a linear function of $θ_{j}^{0}$ under both DGPs, with the largest $τ_{j}$ emerging for the lowest values of $θ_{j}^{0}$ even though there is no treatment by $θ_{j}^{0}$ interaction in the item-dependent HTE DGP. A visualization of the data underlying Table 1 is shown in Figure 3. As we will show, without item-level outcome data supplementing preintervention data, there is no way to distinguish between these two competing causes of HTE.

Table 1.

Toy Example Illustrating the Identification Problem on a Three-Item Test

j	$θ_{j}^{0}$	$E_{P} (S_{j} (0))$	$E_{P} (S_{j} (1))$	$E_{I} (S_{j} (0))$	$E_{I} (S_{j} (1))$	$τ_{j P}$	$τ_{j I}$
1	−1	0.88	1.04	0.88	1.04	.16	.16
2	0	1.50	1.50	1.50	1.50	.00	.00
3	1	2.11	1.95	2.11	1.95	−.16	−.16

Note. Toy example shows how both the person-dependent and item-dependent data-generating processes produce treatment effects on sum scores that depend on $θ_{j}^{0}$ . f is a ${logit}^{- 1}$ function. $E (S_{j})$ is calculated for three items with $b_{i} = - 1, 0, and 1$ and three individuals with $θ_{j}^{0} = - 1, 0, and 1$ under both treatment and control conditions. We set $β_{0} = 0$ , $β_{1} = 0$ , $β_{2} = 1$ , $γ_{0} = 0$ , $γ_{1} = 0$ , and $γ_{2} = 1$ . In the person-dependent heterogeneous treatment effect (HTE) model, we set $β_{3} = - 0.28$ , and in the item-dependent the model, we set $ρ = 1$ , so that $ζ_{i} = b_{i}$ . We calculate $τ_{j}$ by subtracting the expected scores under control condition from expected scores under treatment condition.

Figure 3.

Toy example illustrating the identification problem on a three-item test. Notes. Toy example based on the parameters used to generate Table 1 shows how both the person-dependent and item-dependent data-generating processes produce treatment effects on sum scores that depend on $θ_{j}^{0}$ .

This phenomenon holds in more general settings. For example, Figure 4 shows a more realistic case with 16 items, where the top row depicts a person-dependent HTE scenario, where $β_{3} < 0$ , and the bottom row depicts an item-dependent HTE scenario, where $ρ = 1$ . The first column shows $E (S_{j})$ as a function of $θ_{j}^{0}$ by summing the IRFs for all 16 items, and we can see that the pattern is essentially identical in both DGPs, similar to the result in Table 1. That is, given just $E (S_{j})$ , the two DGPs are empirically indistinguishable (Spoto & Stefanutti, 2023), which provides a serious interpretational problem, given the common use of psychometric outcomes in randomized-controlled trials (RCTs) and the typical analysis focusing on test-level aggregates. Using the items, however, may help resolve this problem. The second and third columns show the individual IRFs on the probability scale and linear predictor $η_{i j}$ scale, respectively. When examining the item-level data, we can clearly distinguish the DGPs. In the person-dependent HTE model, the IRFs cross for each item, and summing the IRFs creates the overall crossing pattern in $E (S_{j})$ . In the item-dependent HTE model, the IRFs are parallel for each item, but the relative vertical distance between the groups varies across items, creating the identical crossing pattern in $E (S_{j})$ . The stark distinction between IRFs for each DGP suggests that these two processes can potentially be distinguished when the appropriate model is applied to item-level data.

Figure 4.

Different data-generating processes (DGPs) can produce the same pattern of sum score responses. Notes. This figure illustrates how different DGPs can produce the same pattern of sum score responses. The top row depicts person-dependent heterogeneous treatment effect (HTE) and the bottom row depicts item-dependent HTE. We first generated the item-level data from the item-dependent HTE model, where we set $ρ = 1$ , implying that the treatment effect is largest on the easiest items. We then fit the misspecified person-dependent HTE model to the data generated by the item-dependent HTE model and used the parameter estimates from this model to generate the data for the top row of the figure.

We can formalize the intuition provided by Table 1 and Figure 4 analytically. We begin with an arbitrary set of $β$ for the person-dependent HTE model and find a set of $γ$ from the item-dependent HTE model that will yield identical $τ_{j}$ . While there is no simple formulation of $E_{P} (S_{j} (\cdot))$ in general (Baker, 2001, pp. 69–70), we have assumed that items are equally discriminating and their easiness parameters are normally distributed, which enables a tractable solution. Under these assumptions, $E_{P} (S_{j} (\cdot))$ can be approximated by a logistic curve scaled by the number of items I, as the sum of logistic curves can be approximated by a logistic curve when the slopes are equal and the curves have considerable overlap (Reed & Pearl, 1927, p. 733). In fact, our simulations show that the logistic approximation of $E (S_{j})$ is incredibly accurate, with an R ² of greater than 99.9%.¹ We can therefore approximate $E_{P} (S_{j} (\cdot))$ as follows:

E_{P} (S_{j} (\cdot)) = \sum_{i = 1}^{I} {logit}^{- 1} (θ_{j}^{1} (\cdot) + b_{i}) \approx {logit}^{- 1} (θ_{j}^{1} (\cdot)) I .

Applying the above equation to each potential outcome yields the following logistic functions, each of which can in turn be approximated by a linear equation because logistic curves are approximately linear between about 20%–80% of the upper asymptote (Long, 1997; Von Hippel, 2015), where $\frac{I}{2}$ represents the midpoint of the scale and $\frac{I}{4}$ represents the linear approximation of the slope at the midpoint in a Taylor expansion:

E_{P} (S_{j} (1)) = \sum_{i = 1}^{I} {logit}^{- 1} (β_{0} + β_{1} + β_{2} θ_{j}^{0} + β_{3} θ_{j}^{0} + b_{i}),

\approx {logit}^{- 1} (β_{0} + β_{1} + β_{2} θ_{j}^{0} + β_{3} θ_{j}^{0}) I,

\approx \frac{I}{2} + \frac{I}{4} (β_{0} + β_{1} + β_{2} θ_{j}^{0} + β_{3} θ_{j}^{0}),

E_{P} (S_{j} (0)) = \sum_{i = 1}^{I} {logit}^{- 1} (β_{0} + β_{2} θ_{j}^{0} + b_{i}),

\approx {logit}^{- 1} (β_{0} + β_{2} θ_{j}^{0}) I,

\approx \frac{I}{2} + \frac{I}{4} (β_{0} + β_{2} θ_{j}^{0}) .

The treatment effect, $τ_{j}$ , is the difference in these quantities:

τ_{j P} \approx \frac{I}{4} (β_{1} + β_{3} θ_{j}^{0}) .

For the item-dependent HTE model, we assume $ρ = 1$ , so that $ζ_{i} = b_{i}$ , yielding the following approximations for $E_{I} (S_{j} (\cdot))$ . The division by 2 in $E_{I} (S_{j} (1))$ comes from the presence of two b_i terms (as in Figure 3; the extra b_i essentially “stretches” the x-axis with respect to $θ_{j}^{0}$ ):

E_{I} (S_{j} (1)) = \sum_{i = 1}^{I} {logit}^{- 1} (γ_{0} + γ_{1} + γ_{2} θ_{j}^{0} + b_{i} + b_{i}),

\approx {logit}^{- 1} (\frac{γ_{0} + γ_{1} + γ_{2} θ_{j}^{0}}{2}) I,

\approx \frac{I}{2} + \frac{I}{8} (γ_{0} + γ_{1} + γ_{2} θ_{j}^{0}),

E_{I} (S_{j} (0)) = \sum_{i = 1}^{I} {logit}^{- 1} (γ_{0} + γ_{2} θ_{j}^{0} + b_{i}),

\approx {logit}^{- 1} (γ_{0} + γ_{2} θ_{j}^{0}) I,

\approx \frac{I}{2} + \frac{I}{4} (γ_{0} + γ_{2} θ_{j}^{0}) .

We again compute $τ_{j}$ :

τ_{j I} \approx \frac{I}{8} (γ_{1} - γ_{0} - γ_{2} θ_{j}^{0}) .

Clearly, $τ_{j}$ is a linear function of $θ_{j}^{0}$ in both DGPs, analogous to the visualization shown in Figure 4.²

We now have two expressions for $τ_{j}$ , one for each DGP, both of which can be approximated by a linear function of $θ_{j}^{0}$ . Thus, there will be no way of differentiating between the two DGPs if we can identify the values of these functions that lead to equivalent expressions for $τ_{j}$ in the above equations. By setting $τ_{j P} = τ_{j I}$ , we see that the DGPs provide equivalent $τ_{j}$ when

γ_{1} - γ_{0} = 2 β_{1},

γ_{2} = - 2 β_{3},

as the $γ$ values are fixed but unknown constants. Thus, models for $E (S_{j})$ alone cannot distinguish between the two DGPs.

3.2. The Solution

We can identify the relevant DGP with item-level data by including both the baseline ability by treatment interaction $β_{3}$ and the correlation between item easiness and item-specific treatment effect size $ρ$ in a single model, such as an explanatory item response model (EIRM). The EIRM is a special case of the generalized linear mixed model that allows for the simultaneous estimation of an IRT measurement model and an explanatory regression model with person- or item-level predictors in a single procedure (Briggs, 2008; De Boeck, 2004; De Boeck et al., 2011; Gilbert, 2023b, 2024; Gilbert et al., 2023b; Petscher et al., 2020).

We propose the following flexible EIRM to disentangle the causes of HTE:

logit (y_{i j} = 1) = η_{i j} = θ_{j}^{1} + b_{i j},

θ_{j}^{1} = β_{0} + β_{1} {treat}_{j} + β_{2} θ_{j}^{0} + β_{3} {treat}_{j} \times θ_{j}^{0},

b_{i j} = b_{i} + ζ_{i} {treat}_{j},

[\begin{matrix} b_{i} \\ ζ_{i} \end{matrix}] \sim N (0, [\begin{matrix} σ_{0} & ρ \\ ρ & σ_{1} \end{matrix}]) .

Because this model allows for both person- and item-dependent HTE and models the item-level data directly, we can determine the extent to which person or item HTE is better fit to the underlying item-level data and identify the appropriate causes of HTE. We turn now to a simulation study demonstrating this result.

4. Monte Carlo Simulation

A key implication of the empirical indistinguishability of the person-dependent and item-dependent HTE processes is that we will estimate a nonzero value of $β_{3}$ if $ρ$ is the true data-generating parameter and vice versa. To demonstrate that person-dependent and item-dependent HTE can become confounded, we perform a Monte Carlo study across a range of $β_{3}$ and $ρ$ values to demonstrate the effects of model misspecification across a range of more realistic testing conditions. To maintain focus on our two parameters of interest, we fix the following parameters across all trials: 500 subjects, 40 items with constant discrimination of 1, $β_{1} = 0.20, β_{2} = 1, σ_{0} = 3, σ_{1} = 0.5, and SD (θ_{j}^{0}) = 1$ . We vary $β_{3}$ and $ρ$ independently from −1 to 1 in increments of 0.25. We fit three EIRMs to each simulated data set, one with a treatment by baseline ability interaction $β_{3}$ , the other with a $ρ$ term, and the third with both $β_{3}$ and $ρ$ , using the glmer function from the lme4 package in R (i.e., a Rasch model; De Boeck et al., 2011; Gilbert, 2023b), collect the output of the models for further analysis, and repeat the process 100 times for each combination of parameters.

The simulation results are shown in Figure 5. The plots in the first row show that each model appropriately recovers the target parameter with minimal bias. That is, whether the DGP includes treatment by baseline ability interaction $β_{3}$ or treatment by item easiness correlation $ρ$ , ${\hat{β}}_{3}$ and $\hat{ρ}$ estimates from the appropriately specified model are unbiased, as are estimates from the flexible model that allows for both $β_{3}$ and $ρ$ . The plots in the second row show the consequences of model misspecification. We observe that a nonzero data-generating value of $ρ$ causes bias in ${\hat{β}}_{3}$ and vice versa. In contrast, when we fit the flexible model that allows for both $β_{3}$ and $ρ$ , we can see that the bias is removed. Therefore, when we observe an apparent treatment by pretest interaction a test score outcome, we cannot determine whether this is due to true person heterogeneity or a correlation between item treatment effect size and item easiness, unless we model the item-level data directly.

Figure 5.

Simulation results showing bias in parameter estimates. Notes. Figures depict bias in parameter estimates. The top row presents properly specified models and the bottom row presents misspecified models. The following parameters are fixed across all trials: 500 subjects, 40 items with constant discrimination of 1, $β_{1} = 0.20, β_{2} = 1, σ_{0} = 3, {and σ}_{1} = 0.5$ . $β_{3}$ and $ρ$ are varied independently from −1 to 1 in increments of 0.25.

5. Empirical Application

For our empirical application, we analyze a public use data set from Kim et al. (2023), which examined the impact of the Model of Reading Engagement (MORE) literacy intervention on the reading comprehension test scores of 2,174 Grade 2 students through an RCT. The researcher-designed reading comprehension test contained 20 dichotomous items. Figure 6 shows the standardized outcome sum score on the standardized pretest score, where the right panel appears to show that the treatment effect is larger for students with higher preintervention scores.

Figure 6.

Scatter plot of standardized sum-score outcome on pre-intervention scores. Notes. The figure presents a scatter plot of standardized reading comprehension sum scores on preintervention measure of academic progress reading scores by treatment status. The left panel depicts the main effect of the treatment and the right panel also includes the interaction effect. Hollow circles indicate the conditional means of the outcome at each decile of the baseline score.

However, it would be premature to assert that the treatment is more effective for students with higher preintervention achievement (i.e., $β_{3} > 0$ ) without conducting an item-level analysis; the data are equally consistent with a treatment that produces the largest impacts on the most difficult items (i.e., $ρ < 0$ ). As such, we replicate the middle panel of Figure 4 using our empirical item-level data in Figure 7. Items are ordered from most to least difficult. Visually, it appears that the treatment effects are largest on the most difficult items rather than the treatment showing the strongest effect on high achieving students across all items.

Figure 7.

Fitted logistic curves by treatment group for each item. Notes. The figure presents the fitted logistic curves of the probability of a correct response on preintervention measure of academic progress reading scores by treatment status. The items are arranged by difficulty, with percentage correct values listed in the headings. The logistic curves are derived from a fixed effects model of the correct response as a function of treatment, pretest, and item indicator, with two-way interactions between treatment and pretest and treatment and the item indicators.

To formally test whether the person- or item-dependent model is a better fit to the data and determine the source of HTE, we fit four EIRMs to the item-level data. Our baseline specification is a constant treatment effect model as described in Equation 3. We then fit a person-dependent HTE model that incorporates an interaction between treatment and pretest scores, $β_{3}$ , as described in Equation 11 and an item-dependent HTE model that includes a correlation between item easiness and treatment effect size, $ρ$ , as described in Equation 18. Our final specification fits the flexible model, integrating both $β_{3}$ and $ρ$ , as specified in Equation 45. All models also include a residual variance term for $θ_{j}^{1}$ to allow for imperfect prediction of postintervention ability by the covariates. The results are displayed in Table 2.

Table 2.

Empirical Application of Explanatory Item Response Models

	(1)	(2)	(3)	(4)
Constant ( $β_{0}$ )	.12 (.15)	.12 (.15)	.11 (.16)	.12 (.16)
Treat ( $β_{1}$ )	.19 (.04)***	.18 (.04)***	.18 (.05)***	.18 (.05)***
Pretest ( $β_{2}$ )	.65 (.02)***	.61 (.03)***	.65 (.02)***	.62 (.03)***
Treat $\times$ Pretest ( $β_{3}$ )		.08 (.04)*		.06 (.04)
Akaike information criterion	52,345.52	52,343.10	52,310.11	52,309.33
Bayesian information criterion	52,388.92	52,395.18	52,370.87	52,378.77
Log likelihood	−26,167.76	−26,165.55	−26,148.05	−26,146.66
N observations	43,480	43,480	43,480	43,480
N students	2,174	2,174	2,174	2,174
N items	20	20	20	20
Var: Person ( $σ_{θ}^{2}$ )	.46	.46	.46	.46
Var: Item ( $σ_{0}^{2}$ )	.41	.41	.51	.50
Var: Treatment ( $σ_{1}^{2}$ )			.03	.03
Cor: Item-treatment ( $ρ$ )			−.74	−.74

Note. We apply the EIRM to data from Kim et al. (2023), which examined the impact of a content literacy intervention through a randomized-controlled trial. Column (1) presents a baseline constant effects model, column (2) presents the person-dependent heterogeneous treatment effect (HTE) model, column (3) presents the item-dependent HTE model, and column (4) presents the flexible model allowing for both person-dependent and item-dependent HTE. The unit of observation is item-person for all the models. Standard errors are in parentheses.

*p < .05. ***p < .001.

Column 1 shows a clear positive ATE in log-odds units. That is, the treatment is estimated to cause a 0.19 logit increase in the probability of a correct response across all students and items, holding constant pretest scores. Adding the treatment by pretest interaction term in Column 2 shows a small but significant positive interaction ${\hat{β}}_{3} = 0.08$ , suggesting that the treatment is more effective for previously high achieving students, in line with Figure 6. When we add $ρ$ in Column 3, we see that $\hat{ρ} = - .74$ , indicating that treatment effects are most pronounced on the most difficult assessment items, as depicted in Figure 8. Finally, in the last column, we allow for both $β_{3}$ and $ρ$ and see that the interaction term is rendered nonsignificant but $\hat{ρ}$ is unchanged. Likelihood ratio tests indicate that the specification in Column 3 is preferred. Therefore, the observed HTE appears to be not due to the treatment helping the highest achieving students most, but rather, impacting the hardest test items that better distinguish among previously high achieving students.

Figure 8.

Correlation between item easiness and item treatment effect size. Notes. The points represent empirical Bayes's estimates of item easiness in the control group (b_i ) and item-specific treatment effect size ( $β_{1} + ζ_{i}$ ) derived from the item-dependent heterogeneous treatment effect model. The horizontal dashed lines represent the average treatment effect (ATE) and a null effect.

6. Discussion

Our study shows that using summary measures of student performance as outcome variables can lead to scenarios where two distinct DGPs produce virtually identical observed results. A treatment may appear more beneficial for students with high or low baseline ability levels when the effectiveness of the treatment genuinely varies based on students’ baseline abilities (person-dependent HTE) or when the treatment impacts easier items more (or less) than hard ones (item-dependent HTE). These DGPs have distinct interpretations, and without item-level data, they cannot be distinguished. A treatment that disproportionately benefits low-scoring students across all items has different policy implications compared to one that uniformly enhances learning of easier content across all student groups. Consequently, standard analytic practices to estimate HTE may lead to misleading conclusions and poor policy decisions when incorrectly applied.

We show that this identification problem can be resolved by using an EIRM that accounts for both variation in treatment effect along a preintervention participant characteristic and a correlation between item easiness and treatment effect size that can correctly determine the relevant DGP and draw correct conclusions. The EIRM is flexible and can be extended beyond the relatively simple application explored here to a wide array of data analytic settings, such as the inclusion of additional covariates at the person or item level, treatment by item cluster interactions, randomization blocks, multiple treatments, additional levels of hierarchy, such as the nesting of students within schools, polytomous item responses, 2PL or 3PL models, or Bayesian approaches (Bulut et al., 2021; Bürkner, 2021; De Boeck et al., 2011; Gilbert, 2023b, 2024; Gilbert et al., 2023b, 2024; Stanke & Bulut, 2019), underscoring the applicability of our approach to diverse contexts. For researchers interested in applying our model to their own data sets, our references provide tutorials in the R programming language: the general EIRM for dichotomous items (De Boeck et al., 2011), extending the EIRM to polytomous items (Bulut et al., 2021), the item-dependent HTE model for dichotomous items (Gilbert, 2023b), and a Bayesian approach that allows for extensions including 2PL or 3PL models (Bürkner, 2021; Gilbert, 2023b). Furthermore, we provide both a detailed replication tool kit in our Online Supplemental Materials (OSM), and a brief Appendix showing the basic R syntax to fit our models.

The problem of identification here hinges on item easiness varying in a systematic way such that $ρ \neq 0$ . How realistic a problem is this likely to be in applied settings? This is a question that will merit more attention in future studies. One study of a 29-item reading comprehension assessment showed $ρ = .20$ (Gilbert et al., 2023b, p. 902). Similarly, an analysis of item-level data from 15 RCTs offers some evidence that $ρ \neq 0$ in that data (Ahmed et al., 2023, p. 8). Conceptually, the presence of DIF-by-easiness correlations in other settings (Duflo et al., 2011; Santelices & Wilson, 2010) is related and also suggestive of the possibility for such systematic variation. We largely leave this as an open question; whatever the value of $ρ$ observed in any given setting, future research on HTE will be more sound if its value is scrutinized because it clearly plays an important role in subsequent inferences.

We used the example of pretest ability to motivate this study, but our argument extends to any pretreatment characteristic correlated with outcomes, such as age or SES. In our OSM, we provide three additional empirical examples with alternative covariates (mathematics pretest, race indicator, and high SES indicator) and an additional data set with item-level outcome data (a 36-item vocabulary assessment) to demonstrate this point. In all cases, $ρ$ is strongly negative, and the estimates of treatment by covariate interaction terms are attenuated when the model allows for $ρ$ , findings fully consistent with our results showing bias in $β_{3}$ when $ρ \neq 0$ . The generalizability of this phenomenon suggests that empirical researchers examining HTE may wish to adopt latent variable models, such as the EIRM, to avoid spurious conclusions about HTE. Furthermore, even if HTE is not of primary interest, $ρ \neq 0$ will necessarily create bias in main effects of covariates, because the main effect in a model without interactions an average of the two subgroup effects, weighted by the sample size of each subgroup.

While our exposition is primarily drawn from the perspective of education research, where the analyses of student assessment data are ubiquitous, the implications of our study are relevant across various disciplines, where both interest in HTE is high and detailed item-level data may potentially be available. Our results apply to economists, who collect item-level data on consumption, spending, or attitudes (Jackson et al., 2021), psychologists studying traits such as depression through survey scales (Beevers et al., 2007; Gilbert et al., 2024; Schmitt et al., 2009), political scientists measuring political knowledge through questionnaires (Baek & Wojcieszak, 2009), or medical practitioners using surveys to supplement biometric measurement or clinical evaluation (Domingue et al., 2021; Hieronymus et al., 2019; Jessen et al., 2018). By making item-level data available, as advocated by Domingue and Kanopka (2023) researchers can better analyze and interpret the impact of policies and interventions in education and other fields.

We acknowledge four limitations of our approach. First, item-level data are not always available to the researcher, which may preclude item-level analysis in many applications. Accordingly, it is unknown how prevalent or large item easiness by treatment effect size correlations may be in empirical applications, and thus, the scope of the problem outlined in this study is not well understood. Second, the logit coefficients produced by the EIRM are generally difficult to interpret compared to conventional test score metrics like standardized sum scores. While there are existing methods available to convert logit coefficients to standardized effect sizes (Breen et al., 2018; Gilbert et al., 2023b; Hox et al., 2017; Mood, 2010), these methods rest on strong assumptions and may add an extra layer of complexity for the researcher. For example, we can convert the main effect of the MORE treatment (Table 2, Model 1) to an effect size by dividing our coefficient $β_{1}$ by the pooled SD of $θ_{j}^{1}$ , estimated as the residual SD of the person random effect from a model with only the treatment indicator (Briggs, 2008, p. 110; Gilbert et al., 2023b, pp. 907–908, Footnote 4). Applying this process to our data yields an estimated effect size of .20 SDs, which is likely to be more interpretable to practitioners than the logit coefficient. Third, the EIRM makes additional computational demands compared to alternative methods, due to the numerical integration required for parameter estimation (Rabe-Hesketh & Skrondal, 2022), particularly when employing Bayesian approaches requiring Markov chain Monte Carlo estimation (Bürkner, 2021; Gilbert, 2023b), and thus may be time-consuming to employ on large data sets. Finally, we examined the relatively simple case of a randomized trial with two groups and a single time point, and as such, it is unknown to what extent the issues identified here may generalize to alternative experimental and quasi-experimental contexts, such as regression discontinuity, difference-in-differences, multisite trials, instrumental variables, and longitudinal analyses, though an emerging literature on the synthesis of latent variable and causal inference methods has begun to shed light on these areas (Gilbert et al., 2023a; Kuhfeld & Soland, 2022, 2023; Rabbitt, 2018; Soland, 2022, 2023; Soland et al., 2023).

Despite these limitations, our findings emphasize the critical role of measurement principles in program evaluation and demonstrate the useful implications of IRT-based causal analyses, beyond the traditional use of measurement tools solely for test construction. The adaptability of the EIRM makes it a powerful tool for any field that relies on detailed, item-level data to uncover patterns of treatment heterogeneity not readily apparent through more traditional data analysis methods.

Footnotes

7. Appendix

Authors' Note

A replication tool kit containing our code and Online Supplemental Material is available via Research Box for researchers interested in replicating or extending the analyses in this study at the following URL: https://researchbox.org/2221. The code includes links to download the empirical data from .

Acknowledgments

The authors wish to thank Zach Himmelsbach, Kenji Kitamura, Alex Bolves, Andrew Ho, the HGSE Measurement Lab, and the Stanford Psychometrics Lab for their helpful comments on drafts of this article.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article:This research was supported in part by the Jacobs Foundation.

ORCID iDs

Joshua B. Gilbert

Luke W. Miratrix

Benjamin W. Domingue

Notes

References

Abenavoli

R. M.

(2019). The mechanisms and moderators of “fade-out”: Towards under-standing why the skills of early childhood program participants converge over time with the skills of other children. Psychological Bulletin, 145(12), 1103–1127.

Ahmed

Bertling

Zhang

A. D.

Loyalka

Xue

Rozelle

Domingue

B. W.

(2023). Heterogeneity of item-treatment interactions masks complexity and generalizability in randomized controlled trials (EdWorkingPaper: 23-754). Annenberg Institute at Brown University. https://doi.org/10.26300/1nw4-na96

Athey

Imbens

(2016). Recursive partitioning for heterogeneous causal effects. Proceedings of the National Academy of Sciences, 113(27), 7353–7360.

Austin

P. C.

Merlo

(2017). Intermediate and advanced topics in multilevel logistic regression analysis. Statistics in Medicine, 36(20), 3257–3277.

Baek

Y. M.

Wojcieszak

M. E.

(2009). Don’t expect too much! Learning from late-night comedy and knowledge item difficulty. Communication Research, 36(6), 783–809.

Baker

F. B

. (2001). The basics of item response theory. ERIC.

Beevers

C. G.

Strong

D. R.

Meyer

Pilkonis

P. A.

Miller

I. W.

(2007). Efficiently assessing negative cognition in depression: An item response theory analysis of the dysfunctional attitude scale. Psychological Assessment, 19(2), 199–209.

Birnbaum

(1968). Some latent trait models and their use in inferring an examinee’s ability. In Lord

F. M.

Novick

M. R.

(Eds.), Statistical theories of mental test scores (pp. 397–479). Addison-Wesley, Reading.

Blundell

Dearden

Sianesi

(2005). Evaluating the effect of education on earnings: Models, methods and results from the national child development survey. Journal of the Royal Statistical Society: Series A (Statistics in Society), 168(3), 473–512.

10.

Borsboom

(2005). Measuring the mind: Conceptual issues in contemporary psychometrics. Cambridge University Press.

11.

Brand

J. E.

Pfeffer

F. T.

Goldrick-Rab

(2014). The community college effect revisited: The importance of attending to heterogeneity and complex counterfactuals. Sociological Science, 1(25), 448–465.

12.

Breen

Karlson

K. B.

Holm

(2018). Interpreting and understanding logits, probits, and other nonlinear probability models. Annual Review of Sociology, 44, 39–54.

13.

Briggs

D. C.

(2008). Using explanatory item response models to analyze group differences in science achievement. Applied Measurement in Education, 21(2), 89–118.

14.

Bryan

C. J.

Tipton

Yeager

D. S.

(2021). Behavioural science is unlikely to change the world without a heterogeneity revolution. Nature Human Behaviour, 5(8), 980–989.

15.

Bulut

Gorgun

Yildirim-Erbasli

S. N.

(2021). Estimating explanatory extensions of dichotomous and polytomous Rasch models: The eirm package in R. Psych, 3(3), 308–321.

16.

Bürkner

P. C.

(2021). Bayesian item response modeling in R with brms and Stan. Journal of Statistical Software, 100(5), 1–54. https://doi.org/10.18637/jss.v100.i05

17.

Camilli

(2006). Test fairness. Educational Measurement, 4 (pp. 221–256).

18.

Chernozhukov

Demirer

Duflo

Fernandez-Val

. (2022). Generic machine learning inference on heterogeneous treatment effects in randomized experiments, with an application to immunization in India (tech. rep.). National Bureau of Economic Research.

19.

Colnet

Josse

Varoquaux

Scornet

. (2023). Risk ratio, odds ratio, risk difference…which causal measure is easier to generalize? arXiv preprint arXiv:2303.16008 .

20.

De Boeck

. (2004). Explanatory item response models: A generalized linear and nonlinear approach. Springer Science & Business Media.

21.

De Boeck

Bakker

Zwitser

Nivard

Hofman

Tuerlinckx

Partchev

(2011). The estimation of item response models with the lmer function from the lme4 package in R. Journal of Statistical Software, 39(12), 1–28.

22.

Ding

Feller

Miratrix

(2019). Decomposing treatment effect variation. Journal of the American Statistical Association, 114(525), 304–317.

23.

Domingue

B. W.

Duncan

Harrati

Belsky

D. W.

(2021). Short-term mental health sequelae of bereavement predict long-term physical health decline in older adults: US Health and Retirement Study Analysis. The Journals of Gerontology: Series B, 76(6), 1231–1240.

24.

Domingue

B. W.

Kanopka

Trejo

Rhemtulla

Tucker-Drob

E. M.

(2022). Ubiquitous bias and false discovery due to model misspecification in analysis of statistical interactions: The role of the outcome’s distribution and metric properties (Version 3) [Computer software]. PsyArXiv. https://doi.org/10.31234/osf.io/932fm

25.

Domingue

B. W.

Trejo

Armstrong-Carter

Tucker-Drob

E. M.

(2020). Interactions between polygenic scores and environments: Methodological and conceptual challenges. Sociological Science, 7, 465–486.

26.

Duflo

Dupas

Kremer

(2011). Peer effects, teacher incentives, and the impact of tracking: Evidence from a randomized evaluation in Kenya. American Economic Review, 101(5), 1739–1774.

27.

Embretson

S. E.

Reise

S. P

. (2013). Item response theory. Psychology Press.

28.

Flake

J. K.

Pek

Hehman

(2017). Construct validation in social and personality research: Current practice and recommendations. Social Psychological and Personality Science, 8(4), 370–378.

29.

Francis

D. J.

Kulesz

P. A.

Khalaf

Walczak

Vaughn

S. R.

(2022). Is the treatment weak or the test insensitive: Interrogating item difficulties to elucidate the nature of reading intervention effects. Learning and Individual Differences, 97, 102167.

30.

Gilbert

J. B

. (2023a). How measurement affects causal inference: Attenuation bias is (usually) more important than scoring weights (EdWorkingPaper: 23-766). Annenberg Institute at Brown University. https://edworkingpapers.com/index.php/ai23-766

31.

Gilbert

J. B.

(2023b). Modeling item-level heterogeneous treatment effects: A tutorial with the glmer function from the lme4 package in R. Behavior Research Methods, https://doi.org/10.3758/s13428-023-02245-8

32.

Gilbert

J. B.

(2024). Estimating treatment effects with the explanatory item response model. Journal of Research on Educational Effectiveness, 1–19. https://doi.org/10.1080/19345747.2023.2287601

33.

Gilbert

J. B.

Hieronymus

Eriksson

Domingue

B. W

. (2024). Item-level heterogeneous treatment effects of selective serotonin reuptake inhibitors (SSRIS) on depression: Implications for inference, generalizability, and identification. arXiv preprint arXiv:2402.04487 .

34.

Gilbert

J. B.

Kim

J. S.

Miratrix

L. M

. (2023a). Leveraging item parameter drift to assess transfer effects in vocabulary learning (EdWorkingPaper: 23-868) . Annenberg Institute at Brown University. https://doi.org/10.26300/hpzb-2f41

35.

Gilbert

J. B.

Kim

J. S.

Miratrix

L. W.

(2023b). Modeling item-level heterogeneous treatment effects with the explanatory item response model: Leveraging large-scale online assessments to pinpoint the impact of educational interventions. Journal of Educational and Behavioral Statistics, 48(6), 889–913.

36.

Hieronymus

Lisinski

Nilsson

Eriksson

(2019). Influence of baseline severity on the effects of SSRIS in depression: An item-based, patient-level post-hoc analysis. The Lancet Psychiatry, 6(9), 745–752.

37.

Holland

P. W.

(1986). Statistics and causal inference. Journal of the American Statistical Association, 81(396), 945–960.

38.

Hox

J. J.

Moerbeek

Van de Schoot

. (2017). Multilevel analysis: Techniques and applications. Routledge.

39.

Jackson

K. C.

Porte

S. C.

Easton

J. Q.

Blanchard

Kiguel

(2021). Linking social-emotional learning to long-term success: Student survey responses show effects in high school and beyond. Education Next, 21(1), 65–72.

40.

Jeon

Jin

I. H.

Schweinberger

Baugh

(2021). Mapping unobserved item–respondent interactions: A latent space item response model with interaction map. Psychometrika, 86(2), 378–403.

41.

Jessen

A. D.

Corrales

C. E.

Yueh

Shin

J. J.

(2018). Improving measurement efficiency of the inner ear scale with item response theory. Otolaryngology–Head and Neck Surgery, 158(6), 1093–1100.

42.

Kim

J. S.

Burkhauser

M. A.

Relyea

J. E.

Gilbert

J. B.

Scherer

Fitzgerald

Mosher

McIntyre

(2023). A longitudinal randomized trial of a sustained content literacy intervention from first to second grade: Transfer effects on students’ reading comprehension. Journal of Educational Psychology, 115(1), 73–98.

43.

Koretz

(2005). Alignment, high stakes, and the inflation of test scores. Teachers College Record: The Voice of Scholarship in Education, 107(14), 99–118.

44.

Kuhfeld

Soland

(2022). Avoiding bias from sum scores in growth estimates: An examination of IRT-based approaches to scoring longitudinal survey responses. Psychological Methods, 27(2), 234–260.

45.

Kuhfeld

Soland

(2023). Scoring assessments in multisite randomized control trials: Examining the sensitivity of treatment effect estimates to measurement choices. Psychological Methods. Advance online publication. https://doi.org/10.1037/met0000633

46.

Künzel

S. R.

Walter

S. J.

Sekhon

J. S.

(2019). Causaltoolbox—Estimator stability for heterogeneous treatment effects. Observational Studies, 5(2), 105–117.

47.

Lockwood

McCaffrey

D. F.

(2020). Recommendations about estimating errors-in-variables regression in Stata. The Stata Journal, 20(1), 116–130.

48.

Long

J. S

. (1997). Regression models for categorical and limited dependent variables (Advanced Quantitative Techniques in the Social Sciences, vol. 7, p. 219). Sage Publications, Inc.

49.

Lyu

Kim

J. S.

Suk

(2023). Estimating heterogeneous treatment effects within latent class multilevel models: A Bayesian approach. Journal of Educational and Behavioral Statistics, 48(1), 3–36.

50.

McNeish

Wolf

M. G.

(2020). Thinking twice about sum scores. Behavior Research Methods, 52(6), 2287–2305.

51.

Montoya

A. K.

Jeon

(2020). Mimic models for uniform and nonuniform dif as moderated mediation models. Applied Psychological Measurement, 44(2), 118–136.

52.

Mood

(2010). Logistic regression: Why we cannot do what we think we can do, and what we can do about it. European Sociological Review, 26(1), 67–82.

53.

Neuhaus

J. M.

Kalbfleisch

J. D.

Hauck

W. W.

(1991). A comparison of cluster-specific and population-averaged approaches for analyzing correlated binary data. International Statistical Review/Revue Internationale de Statistique, 59(1), 25–35.

54.

Olivera-Aguilar

Rikoon

S. H.

(2024). Intervention effect or measurement artifact? using invariance models to reveal response-shift bias in experimental studies. Journal of Research on Educational Effectiveness, 1–29. https://doi.org/10.1080/19345747.2023.2284768

55.

Olken

B. A.

(2015). Promises and perils of pre-analysis plans. Journal of Economic Perspectives, 29(3), 61–80.

56.

Pearl

. (2022). Detecting latent heterogeneity. Probabilistic and Causal Inference: The Works of Judea Pearl, (pp. 483–506).

57.

Petscher

Compton

D. L.

Steacy

Kinnon

(2020). Past perspectives and new opportunities for the explanatory item response model. Annals of Dyslexia, 70(2), 160–179.

58.

Rabbitt

M. P.

(2018). Causal inference with latent variables from the Rasch model as outcomes. Measurement, 120, 193–205.

59.

Rabe-Hesketh

Skrondal

. (2022). Multilevel and longitudinal modeling using Stata. Stata Press.

60.

Reed

L. J.

Pearl

(1927). On the summation of logistic curves. Journal of the Royal Statistical Society, 90(4), 729–746.

61.

Rubin

D. B.

(1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5), 688.

62.

Sales

Prihar

Heffernan

Pane

J. F

. (2021). The effect of an intelligent tutor on performance on specific posttest problems [Conference session]. Proceedings of the 14th International Conference on Educational Data Mining (EDM21), Paris, France, June 29–July 2, 2021, pp. 206–215. https://educationaldatamining.org/edm2021/

63.

San Martín

. (2016). Identification of item response theory models. In Linden

W. J. van der

(Ed.), Handbook of item response theory (Vol. 2, pp. 127–150). Chapman and Hall/CRC.

64.

Santelices

M. V.

Wilson

(2010). Unfair treatment? the case of freedle, the sat, and the standardization approach to differential item functioning. Harvard Educational Review, 80(1), 106–134.

65.

Schmitt

A. B.

Bauer

Volz

H. -P.

Moeller

H.-J.

Jiang

Ninan

P. T.

Loeschmann

P. -A.

(2009). Differential effects of venlafaxine in the treatment of major depressive disorder according to baseline severity. European Archives of Psychiatry and Clinical Neuroscience, 259(6), 329–339.

66.

Schochet

P. Z.

Puma

Deke

(2014). Understanding variation in treatment effects in education impact evaluations: An overview of quantitative methods (NCEE, 2014-4017). Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Analytic Technical Assistance and Development.

67.

Schuetze

B. A.

von Hippel

P. T.

(2023, October 23). How not to fool ourselves about heterogeneity of treatment effects (Version 1) [Computer software]. PsyArXiv. https://doi.org/10.31234/osf.io/zg8hv

68.

Sijtsma

(2005). Nonparametric item response theory models. Encyclopedia of Social Measurement, 2, 875–882.

69.

Skrondal

Laake

(2001). Regression among factor scores. Psychometrika, 66(4), 563–575.

70.

Soland

(2022). Evidence that selecting an appropriate item response theory–based approach to scoring surveys can help avoid biased treatment effect estimates. Educational and Psychological Measurement, 82(2), 376–403.

71.

Soland

(2023). Item response theory models for difference-in-difference estimates (and whether they are worth the trouble). Journal of Research on Educational Effectiveness, https://doi.org/10.1080/19345747.2023.2195413

72.

Soland

Johnson

Talbert

(2023). Regression discontinuity designs in a latent variable framework. Psychological Methods, 28(3), 691–704.

73.

Soland

Kuhfeld

Edwards

(2022). How survey scoring decisions can influence your study’s results: A trip through the IRT looking glass. Psychological Methods. Advance online publication. https://doi.org/10.1037/met0000506

74.

Spoto

Stefanutti

(2023). Empirical indistinguishability: From the knowledge structure to the skills. British Journal of Mathematical and Statistical Psychology, 76(2), 312–326.

75.

Stanke

Bulut

(2019). Explanatory item response models for polytomous item responses. International Journal of Assessment Tools in Education, 6(2), 259–278.

76.

Torche

Fletcher

Brand

J. E.

(2024). Disparate effects of disruptive events on children. RSF: The Russell Sage Foundation Journal of the Social Sciences, 10(1), 1–30.

77.

VanderWeele

T. J.

Batty

C. J

. (2023). On the dimensional indeterminacy of one-wave factor analysis under causal effects. Journal of Causal Inference, 11(1), 20220074.

78.

VanderWeele

T. J.

Vansteelandt

(2022). A statistical test to reject the structural interpretation of a latent factor model. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84(5), 2032–2054.

79.

Von Hippel

. (2015). Linear vs. logistic probability models: Which is better, and when. Statistical Horizons. https://statisticalhorizons.com/linear-vs-logistic/

80.

Wager

Athey

(2018). Estimation and inference of heterogeneous treatment effects using random forests. Journal of the American Statistical Association, 113(523), 1228–1242.

81.

Wallace

M. L.

Mentch

Wheeler

B. J.

Tapia

A. L.

Richards

Zhou

Redline

Buysse

D. J.

(2023). Use and misuse of random forest variable importance metrics in medicine: Demonstrations through incident stroke prediction. BMC Medical Research Methodology, 23(1), 144.

82.

Wendling

Jung

Callahan

Schuler

Shah

N. H.

Gallego

(2018). Comparing methods for estimation of heterogeneous treatment effects using observational data from health care databases. Statistics in Medicine, 37(23), 3309–3324.

83.

Widaman

K. F.

Revelle

(2023). Thinking thrice about sum scores, and then some more about measurement and analysis. Behavior Research Methods, 55(2), 788–806.

84.

Winship

Morgan

S. L.

(1999). The estimation of causal effects from observational data. Annual Review of Sociology, 25(1), 659–706.

85.

Wolf

Harbatkin

(2023). Making sense of effect sizes: Systematic differences in intervention effect sizes by outcome measure type. Journal of Research on Educational Effectiveness, 16(1), 134–161.

86.

Xie

Brand

J. E.

Jann

(2012). Estimating heterogeneous treatment effects with observational data. Sociological methodology, 42(1), 314–347.

87.

Yeager

D. S.

Hanselman

Walton

G. M.

Murray

J. S.

Crosnoe

Muller

Tipton

Schneider

Hulleman

C. S.

Hinojosa

C. P.

Paunesku

Romero

Flint

Roberts

Trott

Iachan

Buontempo

Yang

S. M.

Carvalho

C. M.

Richard Hahn

P.…

Dweck

C. S

. (2019). A national experiment reveals where a growth mindset improves achievement. Nature, 573(7774), 364–369.

88.

Zisook

Shear

Kendler

K. S.

(2007). Validity of the bereavement exclusion criterion for the diagnosis of major depressive episode. World Psychiatry, 6(2), 102–107.