Simulation methodologies to determine statistical power in laboratory animal research studies

Step 1: Establish study objectives

A researcher should establish clear objectives prior to beginning experimentation. Null and alternative hypotheses are specified during this phase. In this example, we studied the potential toxicity of a chemical on male HSD rat liver and testis weights using Jonckheere’s trend test. Before using Jonckheere’s test on real data, we wanted to determine the power of the test to detect an effect if one truly existed. Power is calculated in reference to a specific true effect size, and our objective in this simulation study was to calculate the power of Jonckheere’s test with a 15% effect size in the high dose group, a 10% effect size in the medium dose group, and a 5% effect size in the low dose group. In our example, a 15% effect size represents a 15% reduction in organ weight between the control and high dose groups. The low and medium dose groups exhibited a 5% and 10% reduction in mean when compared with the control group, respectively. Under our experimental assumptions, H 0 : θ 1 = θ 2 = θ 3 = θ 4 , and H 1 : θ 1 ≥ θ 2 ≥ θ 3 ≥ θ 4 , where θ 1 represents the median in the control dose group, θ 2 represents the median in the low dose group, θ 3 represents the median in the medium dose group, and θ 4 represents the median in the high dose group, with at least one strict inequality.

Step 2: Collect pilot data

Once the objectives of the study have been established, collect pilot data. Biological variation is always present, ⁵ and quality pilot data allow the researcher to sufficiently assess the endpoint of interest to formulate representative simulation parameters that account for variability. Pilot data are often collected for control animals (for example, from an online database like CEBS ¹² ) to determine variables such as the empirical mean, SD, and distribution in the sample data. These data should be representative of the population, as summary statistics serve as the baseline for the simulation, and the variability in these data influences the overall power. Control male HSD rat data from four NTP subchronic toxicology studies on post-natal day 90 were averaged and used as baseline parameters in the organ weight example, as summarized in Step 2 of Figure 1. These summary statistics are assumed to be representative of expected control group liver and testis weight values.

Step 3: Determine data distribution and verify assumptions

After examining the empirical distribution of the control pilot data in Step 2, propose a probability distribution from which to simulate representative data. While a normal distribution may be a suitable approximation for many pilot data sets, further investigation is warranted before defaulting to any distributional choice. For brevity, we refer the reader to alternate sources that detail how to determine an appropriate distribution due to the introductory scope of this manuscript and the multipronged, intricate nature of determining a distribution dependent on the study design.^11,13,14 In the organ weight simulation, pilot data were approximately normally distributed.

After the distribution is determined, a statistical test is specified. Verify that the assumptions of the chosen statistical test are met by the proposed simulation distribution. Parametric statistical tests rely on certain assumptions about the distribution of the population from which the data were collected, including normality, equality of variances between experimental groups, and independent errors. ⁷ Parametric tests are more powerful than non-parametric tests, meaning they generally require smaller sample sizes to achieve the same statistical power. When these assumptions are violated, non-parametric approaches may be used instead, since they do not rely on assuming a specific probability distribution for the data. ⁷ Although assumption violations should be given due consideration, the violation of parametric assumptions does not immediately warrant the use of non-parametric tests and vice versa. Researchers should carefully assess their data in tandem with their research question to determine the best approach. ⁸ In the example simulation, we used Jonckheere’s trend test, the NTP standard statistical test for evaluating a dose-response relationship in organ weight data. Jonckheere’s test is a non-parametric test that assumes ordinal or continuous data, independent observations, and two or more independent groups that follow the same distribution. ¹⁵

Step 4: Specify simulation parameters

Once the simulation distribution and statistical test have been chosen, specify the simulation parameters for each simulated dose group. Our pilot data were approximately normally distributed with unequal variance (heteroscedasticity) across dose groups. Therefore, each dose group was simulated from a normal distribution with unequal variances that mimicked the historical data. Simulated multi-dose group data also depend on a predetermined effect size. In our example, the simulated high dose group data had a mean 15% lower than the control group mean (Step 4, Figure 1). Linear interpolation was used to calculate the effect size for the low (5% reduction in mean) and medium (10% reduction) dose groups.

Step 5: Simulate data

Various statistical software packages can be used to simulate data according to the assumed distribution and parameters from Steps 3–4. Before simulating the data, it is prudent to set a seed for the software’s random number generator to ensure exact reproducibility each time the code is run. Here, liver and testis weight data were simulated with one control group and three treated dose groups. Simulated data should be plotted to ensure that it resembles the pilot data from Step 2 and to verify that it follows the simulation distribution specified in Step 3. For a more comprehensive discussion of techniques for simulating data, refer to Arnold et al. ¹⁶

Step 6: Perform statistical testing

After simulating a data set, perform the statistical testing using the test selected in Step 3. Record whether the statistical test correctly rejects H₀ at the prespecified significance threshold (p < α).

Step 7: Repeat N sim times

Repeat Steps 5–6 (simulate data and perform statistical testing) for a total of N sim iterations, where N sim is chosen to be sizable, often ≥ 1000 unless a large number of iterations is not computationally feasible. In the example organ weights simulation, N sim  = 10,000, so that a new dataset of 40 organ weight values ( N = 10 per four dose groups) was simulated and tested 10,000 times. The total number of the N sim iterations where H 0 is correctly rejected is denoted T (see Figure 1).

Step 8: Calculate power

Once the simulated data are analyzed, calculate power as T N sim * 100 % . In our analysis, T N sim * 100 % =  76.58% power for liver weights and 93.55% power for testis weights when N = 10 per dose group. Large-scale studies with multiple endpoints may not achieve 80% power in the analysis of each endpoint, though it is important to check that the power of the statistical test is reasonable for all endpoints of interest. In this example, the power differed for liver and testis weights due to varying means and SDs in the empirical pilot data for those organs. With reliable levels of power, we can proceed with confidence in the statistical analysis of these data given a sample size of N = 10 per dose group.

Step 9: Repeat for different sets of simulation parameters

Repeat all previous steps for each desired combination of sample size, effect size, distribution, true parameter values, significance threshold, and endpoint of interest. ¹¹ In our example, Steps 1–8 were conducted for both male liver and testis weights, with N = 10 animals per dose group. Though the example simulation determined the power based on the given sample size and distributional assumptions, researchers can conversely choose a target power (i.e., 80%) to determine an adequate sample size. Additional factors can be varied for a power analysis under this framework (Figure 1).