Introduction to a Special Issue on Simulation Studies as a Means of Informing Psychoeducational Testing and Assessment

Educational and psychological tests are used in almost every industrialized country, and testing has been characterized as “the flagship of applied psychology” (Embretson, 1996, p. 341). Thousands of tests have been developed to meet market needs for children, youth, and adults (Oakland, Douglas, & Kane, 2016). Although the Standards for Educational and Psychological Testing (American Educational Research Association [AERA], American Psychological Association [APA], & National Council on Measurement in Education [NCME], 2014) provides criteria for developing tests and evaluating the quality of tests and testing practices, literal satisfaction of every standard is unlikely and thus an unrealistic expectation when evaluating the acceptability of score interpretations and uses (Plake & Wise, 2014). Moreover, even well-designed, well-researched tests are sometimes used in ways that have not been validated.

The interpretation and use of test scores is informed by statistical analysis. When there is insufficient evidence to support the interpretation and use of test scores, it is difficult to safeguard against unfair and potentially harmful consequences that may arise from testing and assessment practices. Unfortunately, it is often impractical to sample actual data. For example, it is difficult to sample participants who are normatively rare (e.g., individuals diagnosed with a psychiatric or educational diagnosis that has a low base rate or individuals at the extremes of normally distributed variables such as intelligence or academic achievement). Fortunately, artificially generated data can be utilized when practical constraints limit the availability of actual data. The process of generating artificial data to reflect relevant characteristics of data we would expect to obtain in the real world, then subsequently using these data to answer relevant questions, is known as a simulation study.

Simulation studies can be used to inform score interpretations, clinical decision making, and test development. The principle underlying the use of simulation studies is that the performance of data in the real world can be approximated using computers to generate numerous samples specified to have statistical characteristics of interest. The goal of this special issue is to equip applied researchers and clinicians with knowledge and skills needed to conduct simulation studies.

Many of the statistical innovations touted by quantitative methodologists are not implemented in substantive and applied research (Sharpe, 2013). Two interrelated objectives that address the goal of promoting statistical innovations include communicating effectively and reducing cognitive complexity. Sharpe suggests that these objectives can be achieved by (a) providing a tutorial that shows how to implement the innovation in concrete terms and (b) demonstrating how an innovation meets the statistical needs of applied researchers using real-world examples. Accordingly, the first article in this special issue provides a tutorial on data simulation, whereas the other articles provide real-world examples of how simulation studies can inform clinical decision making and test development. In his tutorial, Beaujean (2018) describes the general steps of a simulation study and provides a didactic example. As adoption of statistical innovations often depends on the accessibility of software (Aiken, West, & Millsap, 2008), Beaujean has provided R syntax that will allow readers to replicate the didactic example.

Two articles in this special issue demonstrate how simulation studies can inform the process of identifying learning disabilities via the pattern of strengths and weaknesses (PSW) approach. The assumption underlying the PSW approach is that an academic achievement deficit can be linked to a core cognitive deficit, and this pattern of weakness exists within an otherwise normatively average profile of intrapersonal strengths. Miciak, Taylor, Stuebing, and Fletcher (2016) present a study regarding classification accuracy of the concordance/discordance method for identifying learning disabilities (Hale & Fiorello, 2004). Their results extend previous research by examining improvement in classification accuracy as a function of including multiple measures of abilities. Also with respect to the use of multiple measures, Schneider and Roman (2018) present their study involving examination of potential bias in ability estimates resulting from the cross-battery assessment (XBA; Flanagan, Alfonso, Mascolo, & Sotelo-Dynega, 2012) procedure for selecting cohesive scores. Their results inform the practice of applied cognitive assessment with respect to the interpretation of divergent test scores and the calculation of composite scores within XBA.

Two articles in this special issue pertain to the interpretation of progress monitoring data. Christ and Desjardins (2018) describe the use of simulation data for the longitudinal modeling of progress data, specifically curriculum-based measures of reading. The authors compared ordinary least squares and Bayesian estimation as methods for examining progress. Hintze, Wells, Marcotte, and Solomon (2018) present results from a simulation study that compared the effects of alternative decision rules, with varying standard errors of estimates for trend lines, on diagnostic accuracy as defined by the proportion of observations correctly classified as either meeting or not meeting expectations for progress. The goal of this research is to inform decisions regarding treatment effect so that the likelihood of detecting positive intervention effects is maximized while the likelihood of prematurely altering goals and/or interventions is minimized. Both studies provide examples of how simulation studies can be used to inform formative decision making within multitiered systems of support.

Finally, Morgan, Moore, and Floyd (2018) discuss the use of simulation studies to inform the development or revision of measurement instruments. These authors provide an illustrative example in which a simulation study is used to guide the development of a universal behavior screener. This article demonstrates how test developers can utilize simulation studies proactively to help ensure that test scores will have adequate evidence of validity to support proposed interpretations and uses. Proactive analysis seems particularly useful when developing norm-referenced tests. Given the expense of collecting standardization data, it is important to use data to inform decision making to protect the investment of financial resources and labor.

In summary, the articles presented in this special issue represent some of the various ways in which simulation studies can be utilized. In fact, the only limitations to the potential of simulation studies lie in the limits of creativity among experienced users and the absence of knowledge and skills among inexperienced users (Mooney, 1997). It is hoped that this special issue both sparks creativity and equips applied researchers, clinicians, and test developers with sufficient knowledge and skills to implement simulation studies. Furthermore, it is envisioned that increased adoption of this innovative practice will ultimately improve the quality of psychoeducational tests, testing and assessment practices, and clinical decision making.