No Need to Be Indirect: On the Role of Data in the Validation of Theoretical Models

Phenomena Selection

This first issue concerns the problem of selecting phenomena for the validation of a particular model. It can be very difficult to find a set of sufficient phenomena—that is, a set of phenomena which can validate a theory in its entirety. To see this, we draw an instructive parallel to the concept of sufficient statistics in statistical theory. In statistics, a sufficient statistic contains all the information in a sample that is relevant for estimating the parameters of a model. For instance, when sampling from a normal distribution, the sample mean and variance together form the sufficient statistics for estimating the population parameters. However, the mean alone is insufficient: Multiple normal distributions can produce the same mean through variations in their variance. Similarly, in theory testing, a single phenomenon is often insufficient to discriminate between competing theories, as multiple theoretical mechanisms could generate the same phenomenon. Practically, this problem implies that researchers should be able to identify those phenomena that are sufficient in order to validate that theory. While we believe that there are sufficient phenomena to be found, identifying them becomes troublesome in more complex psychological theories where numerous underlying mechanisms could generate similar surface-level phenomena (e.g., Yu et al., 2023).

Now imagine four competing models, each specifying different expected response time distributions based on distinct underlying cognitive processes. Model A posits that response times follow a normal distribution, with differences between experimental conditions arising from a shift in the mean of this distribution—representing a simple additive effect. Model B posits a bimodal distribution, where the observed mean difference results from a strategic mixture between two response modes (e.g., fast guessing versus deliberate responding). Model C posits a skewed distribution, reflecting a threshold-based evidence accumulation process (such as in drift-diffusion models), where condition differences emerge from changes in the decision threshold or drift rate. Model D posits a heavy-tailed distribution, where rare but extreme response times play a substantial role, and mean differences are driven by changes in the tail behavior (e.g., increased likelihood of lapses or extreme delays). Through applying the phenomenon-based approach in isolation, we found that despite the fundamentally different assumptions and predictions of the four theories, we are not able to distinguish between them based on only this single phenomenon (see Figure 1). This shows how a selective focus on a single phenomenon can lead researchers to incorrectly confirm their preferred theory.

OPEN IN VIEWER

Figure 1.

Illustration of the selective focus issue. Using the models described in the section, we simulated 150 data points for each model and each condition separately. Panel (a) shows the four distributions of the model, each having distinctly different shapes representing different psychological processes: normal (simple additive effects), bimodal (strategic choice between options), skewed (threshold-based accumulation), and heavy-tailed (processes with occasional extreme responses). Panel (b) shows that all four models produce identical mean differences of 0.5 between control and treatment conditions, which serves as the phenomenon of interest. Panel (c) visualizes the different skewness and kurtosis values of the models, providing clear evidence that they represent different underlying mechanisms.

It is useful to repeat that we do think that there are sufficient phenomena to be found: In this example, we may have looked at other distributional parameters such as the skewness and kurtosis to distinguish between the four competing models (see Panel (c) in Figure 1). However, identifying such sufficient phenomena becomes increasingly difficult with more complex models that share prediction of many different phenomena.

Model Complexity

A single phenomenon can be implied by many formal theories, but the converse is also true, so that one model can imply a wide range of different phenomena. Indeed, while it is typically argued that verbal theories lack specificity, we should appreciate that mathematical models can suffer from the same limitation (Oude Maatman, 2025). In extreme cases, models can predict virtually any phenomenon if their parameters or boundary conditions are “tweaked” in a certain way. More generally, one can always come up with a more complex model that generates several phenomena equally well as long as it is used in a particular way, as we illustrate in the following example.

A : y t = ϕ j y t − j ϕ ∈ [ 0 , 1 ) B : y t = β δ j y t − j β ∈ [ 0 , 1 ] , δ ∈ [ 0 , 1 )

where j ∈ {1, · · · , T} represents the lag in the variable y.

OPEN IN VIEWER

Figure 2.

Illustration of the functional form of emotional inertia according to Model A (exponential, left) and Model B (quasi-hyperbolic, right). For the exponential model, we vary the defining parameter ϕ ∈ {0.10, 0.30, 0.50, 0.70, 0.90} to show the strength of the regulation that is expected with each of these parameter settings. For the quasi-hyperbolic model, we fixed the parameter δ = 0.70 and vary the value of the parameter β ∈ {0.10, 0.35, 0.60, 0.85, 1.00}. Observe that the quasi-hyperbolic expectation for the value of parameter β = 1 corresponds to the expectation of the exponential model for the value of ϕ = 0.70.

Following the phenomenon-based approach, we have to find out which of the two models best reproduces the observed decay rates. However, if we imagine that in reality affect decays exponentially, then we will not be able to distinguish between these two models, as Model B is able to produce both exponential and quasi-hyperbolic decay through variation in its parameter β. In other words, reproducing the phenomenon of interest is not enough to differentiate between these two competitor models.

To get around this issue, one could consider the parsimony of a model to distinguish between alternatives (Borsboom et al., 2021). However, to our knowledge, the phenomenon-based approach provides no principled way of doing so, meaning there is no inherent limit to the complexity of the models we consider. Exacerbating this issue is that parsimony is difficult to define when based on phenomena alone, as even relatively simple models can produce complicated ranges of phenomena. One example is the linear differential equation which, under the right specification of the eigenvalues of its drift matrix, allows not only for the typical exponential decay function, but also for more complex oscillatory behavior (Strogatz, 2018). Should this be considered a “simple” model due to its limited number of parameters, or as a “complex” model due to the wide range of behavior that the model predicts? At this moment, the phenomenon-based approach does not answer this question.

The relevance of the complexity debate for modeling is further illustrated in an anecdote involving physicist Enrico Fermi, as retold by Dyson (2004). At the time, Dyson was investigating the strong nuclear force using mathematical equations that he had previously used to understand the weak nuclear force and which had their roots in the theory of quantum electrodynamics, which states that electrons and protons interact through weak electromagnetic forces. After finishing his calculations on the strong nuclear force, Dyson asked Fermi for his opinion on his work. Fermi responded by asking “How many arbitrary parameters did you use for your calculations?,” afterwards concluding by quoting an adage of his friend John von Neumann: “With four parameters I can fit an elephant, and with five I can make him wiggle his trunk” (Dyson, 2004, p. 297). Initially disheartened, Dyson abandoned the project and only later appreciated the depth of Fermi’s critique with the discovery of the quarks, which enhanced the understanding of the strong nuclear force more so than his complex model ever could have.

Circularity

The final limitation of the phenomenon-based approach is the fact that it takes phenomena as a given. If a phenomenon is indeed present in nature, a valid model should necessarily recover that phenomenon. However, this recovery becomes problematic if a phenomenon turns out to be non-existent (see also van Dongen et al., 2025). The strength of inference within the phenomenon-based approach therefore depends heavily on the phenomenon’s existence.

Y = α + β X + γ X 2 + δ X 3 .

where α, β, γ ∈ R, and δ > 0. Now imagine that researchers routinely fit a linear regression model to their data in order to estimate the relationship between X and Y:

Y = α + β X + v t . v t i i d ~ N ( 0 , σ 2 ) .

Across many studies, researchers would discover a seemingly linear relationship between X and Y, in the sense that the slope β ^ would consistently be estimated to be positive. Because of the robustness of this phenomenon, theory builders conclude that their formal theories should accommodate the expected positive linear relationship between X and Y.

When it comes to theory testing, however, the ability of the model to reproduce a curve of the form Y = α + βX is clearly irrelevant, if not counterproductive: The more accurately the model reproduces the phenomenon, the less likely it will be to conform to the true relationship Y = α + βX + γX² + δX³. Reproducing the phenomena does not bring the literature closer to discovering, understanding, or explaining the true relationship between the variables Y and X. If the practice of validating a theory using the phenomena it was designed to explain becomes mainstream, it may be to the detriment of new ideas or findings which challenge the current way of thinking. More strongly, a pure application of the phenomenon-based approach creates an echo chamber where theory prescribes which phenomena to model without questioning whether these phenomena were correctly identified or without allowing for data to disagree.

Auxillary Assumptions

As we previously mentioned, researchers are required to make some auxiliary assumptions when formalizing a theory into a model (Suppes, 1962). These assumptions concern many different aspects, including the properties of the measurements (Krosnick & Presser, 2010; Stevens, 1946), structures that relate theoretical constructs to observables (Kellen et al., 2021; Regenwetter & Robinson, 2017; Robinson et al., 2025), or the structure of the error (Westfall & Yarkoni, 2016). Critically, when we use models to test theory, we never test theories in isolation. Rather, we test a theory plus auxiliary assumptions (Earp & Trafimow, 2015; Lakatos, 1978; Suppes, 1962). This creates a fundamental attribution problem. When a model fits data poorly, this failure could come from multiple sources: The core theory could be wrong, but so could any of the auxiliary assumptions of the model. Unfortunately, data alone cannot indicate the source of the misfit, meaning that current judgment of the performance of the model is largely determined by its theory-irrelevant auxiliary assumptions.

Estimation

A more practical issue concerns that of estimation. To be able to leverage on the data-driven approach, one requires their model to be estimable. This can be difficult to achieve for more complex models, especially given that classical estimation methods require the specification of a likelihood function (Spanos, 2024; Viscardi et al., 2025). Recent developments have tried to address this problem, using for example simulation-based methods (Mestdagh et al., 2019) or neural networks (Radev et al., 2020, 2023) to allow for model estimation without the explicit derivation of its likelihood function. Such developments are very useful, and allow researchers to focus on the structure of the models rather than on how one should go about estimating and performing inference on the parameters. However, even with these developments, it still holds that more complex models will be more difficult to estimate, hindering the application of these models in research.

Maybe more problematic is the implicit assumption that a model should be estimable for it to be useful. However, one can argue that models that are not estimable can still lead to important insights (Bramson et al., 2017; Campanella et al., 2014). For example, the boid model designed by Reynolds (1987) shows the possibility of flocks of birds emerging because of the tendency for these birds to account for their immediate neighbors rather than for the flock as a whole. Similarly, the model proposed by Robinaugh et al. (2024) allows for predictions that, despite its inestimability, may provide critical insight into the emergence of panic disorders. By requiring a model to be estimable, these insights would not see the light of day, severely limiting both the scope of the models that the data-driven methods apply to as well as what can be discovered through these methods.

Atheoretical Models

Finally, there exists a fundamental tension within the data-driven approach: Theoretical models often fit data less well than atheoretical models that were designed merely to predict. For example, machine learning models such as neural networks, random forests, or support vector machines routinely outperform theory-driven models on standard data-driven metrics (Yarkoni & Westfall, 2017), achieving superior fit by exploiting patterns in the data without requiring any understanding of the underlying mechanisms (Gelman et al., 2014; Piironen & Vehtari, 2017; Susko & Roger, 2020). This indicates that data-driven testing is mechanistically blind. Data-driven methods do not distinguish between mechanistic understanding and mathematical convenience, treating successful fit as equal evidence across these types of models. Taken to the extreme, applying a pure data-driven approach to model selection might lead to a proliferation of atheoretical models, a situation that is undesired when taking theory seriously (Eronen & Bringmann, 2021).

Phenomena Selection

In Example 1, we showed that four different formal theories produced identical mean differences of 0.5 between conditions in a decision-making task, making them indistinguishable based on this phenomenon alone. As we have explained in the section around this issue, one could try to alleviate this issue by trying to find another set of phenomena that is sufficient to distinguish between different theories. However, one could also address the issue more directly, namely through assessing the fit of the formal theories model to the data. Indeed, by fitting the model, we are imposing all of its assumptions on the data and we need not select individual phenomena.

In Table 1, we show the relative fit as assessed through the information criteria AIC and BIC. The results paint a clear picture: When a model was fit to data that it itself generated, it outperformed the alternative models. By considering data, we are now able to distinguish between the different models because all implications of the models, including their differences with respect to phenomena other than the mean, have been imposed on the data.

OPEN IN VIEWER

Table 1.

Model Comparison Results Using Information Criteria to Solve the Problem in Example 1. Best-fitting models for each dataset are indicated in bold.

Model	Information criteria (AIC/BIC)
	Normal	Bimodal	Skewed	Heavy-tailed
Normal	850 / 865	1026 / 1040	1040 / 1054	1006 / 1021
Bimodal	856 / 884	909 / 938	969 / 997	1006 / 1035
Skewed	956 / 978	1009 / 1030	938 / 960	1182 / 1204
Heavy-tailed	854 / 876	1030 / 1051	1019 / 1040	993 / 1014

While one may be tempted to conclude that within this example, fit serves as a sufficient means of validation, we would disagree with such a conclusion. Fit only provides us with one piece of the puzzle and, as argued above, pursuing only fit to data would lead to a preference of atheoretical models. If our goal is to use models as a form of theory, fit to data as a sole criterion would be inadequate. Instead, it is the combination of the data-driven model fit together with the phenomenon-based predictions that lead us to believe one model is a better representation of reality than the other.

Model Complexity

One of the primary advantages of the integrative approach is that it allows for a principled way to handle the complexity of a model. In Example 2, we described a situation in which the reproduction of a phenomenon alone did not allow us to find out whether affect decays in an exponential or quasi-hyperbolic fashion. However, by assessing fit more directly and penalizing the models according to the number of parameters they have, we are able to get around this issue.

To illustrate this, we report the results of a simulation-based model comparison of the situation described in Example 2. We simulated 20 time-series of five datapoints with the exponential model, setting its parameter ϕ = 0.25 and its error variance σ² = 1 (see Figure 2). When we then assess the AIC and BIC of the fit of the exponential and quasi-hyperbolic models, we find that the former (AIC = −15.44, BIC = −9.63) outperforms the latter (AIC = −14.40, BIC = −5.98). This again illustrates the value of using the integrative approach rather than an approach based on only phenomena.

Like in the previous example, we do not wish to suggest that the data-driven fit of the model is sufficient to assess the models’ validity. If either Model A or Model B had been unable to reproduce the phenomenon of interest (in this case, exponential decay), then we follow the phenomenon-based approach in saying that this would count as evidence against that particular model. Instead, our argument is that only looking at this reproduction is not enough to validate a model, and that one should additionally use data-driven approaches to have a more comprehensive test of the model and the theory that it formalizes.

Additionally, we need to make explicit that in our example here, we use well-known fit metrics that provide a principled way of accounting for the complexity of a model. However, this complexity is only based on the number of parameters of the model and doesn’t consider the complexity of the behavior implied by the model. Yet, considering the parsimony of a model should entail an investigation in the range of behavior that the model predicts, which may often but not always go together with the number of parameters of a model (see again the linear differential equation example; Strogatz, 2018).

Unfortunately, we don’t know of a principled way of accounting for the complexity of a model’s qualitative behavior within the data-driven approach. Yet, one option may exist within the integrative approach: By investigating within which range of parameter values of the model a particular behavior occurs, one can leverage on the comparison of the fit of restricted and unrestricted versions of this model to get evidence for the occurrence of the more complicated behavior, as this behavior will “trade off” against parameter restrictions.

Circularity

In Example 3, we showed that the ability of a model to reproduce a phenomenon can be counterproductive if the phenomenon itself is incorrectly specified. This is the case due to the absence of a principled correcting mechanism. Within the integrative approach, one is able to avoid this issue through a systematic study of misfit. Specifically, when one fits a model to data, one can examine the residuals of this model to identify missing or incorrectly specified phenomena (see Figure 3). Conversely, one may also spot overfit of a model to the data through the observation of extremely small residual variances, therefore also accommodating the issue of model complexity. Finally, one can check the values of the estimated parameters of the model, which may lie outside of the boundaries that are imposed by the theory.

OPEN IN VIEWER

Figure 3.

Inspection of the residuals of the linear regression model when fitted to N = 100 data points generated from a third-degree polynomial with α = 0, β = γ = δ = 1; the situation we described in Example 3. Panel (a) shows the fit of the linear regression to the raw data. Panel (b) zooms in on the residuals of this fit, showing that residuals are not independently and identically distributed, which signals model misfit.

Note that the use of data for the validation of a model could also be argued to be circular, as the models one fits to the data in principle need to be correctly specified to retrieve unbiased estimates of the model parameters (Ariens et al., 2023). However, a key advantage of the integrative approach is that it allows for the data to speak against the merits of the model we fit to it while keeping the model itself theory-informed and capable of making theoretically interesting predictions. These assumption violations are therefore important sources of information rather than nuisances and allow for researchers to detect that something has gone wrong. Unfortunately, this kind of correction mechanism is often neglected (for exceptions, see Revol et al., 2025; Vanhasbroeck et al., 2022).

We must admit, though, that if a researcher encounters an assumption violation, this might indicate a failure of an auxiliary assumption rather than a theoretical one. In this case, the task of the model builder will be to reconsider the method with which data has been collected, the auxiliary assumptions of the model, or the theory itself in light of this mismatch.

Strengths

A major strength of the integrated approach is the direct testability of a model to data and, critically, the ability to estimate a realistic set of parameters from those data. The importance of a realistic set of parameters for evaluating a model cannot be understated: Parameters govern the behavior of the model. The ability to estimate the parameters of a model is therefore useful, as it provides us with a realistic parameter set that can be used to predict phenomena that can be realistically expected to occur—the hallmark of strong scientific theories. Additionally, it allows for an empirical check of whether the parameter values satisfy theoretical constraints, rather than merely assuming that the parameters do. This type of analysis corresponds to what Pitt et al. (2006) referred to as local analysis, which focuses on evaluating a model’s behavior at specific parameter values. In contrast, global analysis explores the full range of qualitative behaviors a model can produce across its entire parameter space. Critically, Pitt et al. (2006) argue that while global analysis offers valuable theoretical insights into the range of behaviors a model can produce, it should not stand alone. Instead, it must be complemented by local analysis based on parameter estimates derived from actual data. We fully agree with this position.

Parameter recoverability and identifiability can furthermore serve as critical diagnostics for whether theoretical constructs in a model can be empirically grounded. When different combinations of parameter values generate the same phenomena, or when parameter estimates fluctuate erratically across similar datasets, the correspondence between parameters and psychological constructs becomes vague, despite mathematical formalization. Pushing this argument a bit further, well-designed models may use their parameters as indirect measurements of psychological constructs, extending the reach of psychological science beyond directly observable behavior (e.g., Brown & Heathcote, 2008; Heathcote et al., in preparation; Kahneman & Tversky, 1979; Yu et al., 2023, 2025). When a model includes parameters which represent mechanisms like learning rates, generalization gradients, or attentional focus, estimating these parameters allows researchers to investigate how specific psychological processes vary across individuals, developmental stages, or experimental conditions. These parameter-based insights provide a deeper understanding of psychological functioning than would be possible through analysis of behavioral measures alone.

Limitations