Abstract
This article discusses the interrelated issues of statistical power, sampling, and effect sizes when conducting rigorous quantitative research. Technical and practical connections are made between these concepts and various inferential tests. To increase power and generate effect sizes that merit practical or clinical notice, not only must the research aims and associated design be well devised, reflecting best scientific practice, state of the art sampling procedures should be applied with sufficiently large and representative number of participants. Applications to research conducted in the counseling profession are included.
As most experienced researchers understand, a publishable study with generalizable and meaningful findings necessitates a well-conceived and executed research design. In contrast, quantitative studies lacking the rigor and quality essential for scholarly journals may be criticized for a variety of research-related flaws, such as inadequate controls, unrepresentative sampling, loose data collection procedures, as well as instrumentation deficiencies, outcomes with trivial professional significance, and other confounding factors affecting internal and external validity (Babbie, 2010; Moss et al., 2009; Rowland & Thornton, 2002). One of the major reasons for poor research performance by academics is the lack of an effective research skill set, including method skills (Wheeler, Seagren, Becker, Kinley, & Mlinek, 2008).
Even though these research concerns are largely avoidable, investigator errors continue with some regularity, often with serious unintended consequences. For example, these errors may reduce statistical power and the magnitude of effect sizes (ESs). In certain situations, the researcher might end up rejecting the wrong hypothesis and advancing erroneous conclusions (Huck, 2009). If the ultimate goal of most counseling-related research is to positively affect the profession and the work of practitioners and their clients, knowing the basics and the nuances of quality research is indispensable. Succinctly put, counseling research must reflect best practices (Osborne, 2008). In an attempt to inform nascent and perhaps more knowledgeable counseling researchers, the central intent of this article is to discuss the primary interconnections between sampling, statistical power, and ESs. To illustrate how these notions may influence the results and conclusions of a counseling-related investigation, a hypothetical study is initially provided.
Research Scenario
A researcher investigated the efficacy of narrative therapy (NT) on client sense of self-efficacy using two convenient (nonprobability) samples of caregiver-referred adolescents (ages 14 to 18). All participants received a minimum of 10 counseling sessions from licensed mental health practitioners at several urban mental health clinics. Prior to the quasi-experimental 1 study’s onset, one group of participants, the experimental (e) group, reported feeling mild-to-moderate depression and anxiety for at least 3 months. Participants comprising the comparison (c) group exhibited for at least 3 months moderate school behavior problems (SBP; e.g., acting out in class, fighting), as reported by the school counselor. For reasons made clear later, to improve statistical power, the researcher was able to obtain 50 participants for each group. Each participant was individually pretested on a self-efficacy measure by the investigator. Following a 10-intervention session interval, the groups were readministered the inventory (posttest), and 5 weeks later, again asked to complete the instrument (follow-up test). The independent variable of primary interest was “group” with two levels (experimental and comparison) and dependent variable was client perceptions of self-efficacy (total score) measured at three intervals (pre, post, and follow-up). Based on the research literature, the researcher hypothesized (i.e., alternative hypothesis) that the experimental sample after NT would report significantly higher self-efficacy scores than the comparison group at the time of post- and follow-up testing. In other words, the researcher anticipated, based on theoretical assumptions and previous research, that NT is more efficacious for clinical samples showing anxiety and depression (internalizing behavioral symptoms) than school-based samples with SBP (externalizing behavioral symptoms).
Using this scenario, we pose two related questions: (a) What is the best way to ensure the research study has sufficient statistical power to find statistically significant group differences on the self-efficacy measure as well as to generate consequential outcomes (sizable ESs) for clinicians to take note? (b) What role do sampling and sample size exert on statistical power and ESs? In response, we first tackle the challenging notion of statistical power, moving then to sampling issues, and finally, to practical significance.
Statistical Power
The goal of inferential statistics is testing particular hypotheses about potential group differences or correlations between variables. Statistical power, a concept directly linked with inferential testing, concerns the ability to detect group differences or nonzero correlations. Competent researchers estimate power early on as they design their studies (Huck, 2009). Statistical procedures that are “powerful” have a greater likelihood of finding any true effect that may exist. Metaphorically, the concept of statistical power can be likened to the process of magnification (Meyers, Gamst, & Guarino, 2006). A more powerful magnifying glass has the ability to show greater detail. Similarly, a more powerful statistical test used to examine data can better reveal a significant result. In practitioner language, power is the odds that a researcher will observe a treatment/intervention effect when it occurs (Trochim, 2006) or as Cohen (1988) suggests, statistical power is the probability that the researcher will come to the conclusion that the phenomenon under investigation actually exists.
In our research scenario above, the investigator’s research goal is to detect any differential group effect following the NT intervention period (pre- to posttesting). In other words, our experimenter wants to correctly reject the null hypothesis (μexperimental = μcomparison) and instead affirm the alternative hypothesis (μexperimental > μcomparison). 2 Retaining the alternative hypothesis when it was true would indicate that at the time of posttesting adolescents in the experimental group (i.e., participants exhibiting anxiety and depression) reported significantly higher self-efficacy scores than those adolescents in the comparison group (i.e., participants exhibiting SBP). To accomplish this end, the researcher will need to maximize statistical power.
Power is often expressed as 1 − β, where β represents the likelihood of committing a Type II error (i.e., the probability of incorrectly retaining the null hypothesis). Betas can range from .00 to 1.00. When the beta is very small (close to .00), the statistical test has the most power. For example, if the beta equals .05, then statistical power is .95. Multiplying statistical power by 100 yields a power estimate as a percentage. Thus, 95% power (1 − β = .95 × 100%) suggests that there is a 95% probability of correctly finding a significant result if an effect exists. Typically, a power index greater than .80 (or β ≤ .20) is considered statistically powerful (Park, 2003). In summary, rigorous quantitative studies need to be well designed to generate adequate power to detect any statistically significant group effects or correlations (Huck, 2009). A priori power analysis helps researcher determine (a) how large the sample sizes need to be to generate sufficient power, or (b) whether the power for fixed sample sizes is large enough to justify moving ahead with the study.
Several statistical power calculators exist online, which allow researchers to estimate how large the samples must be to generate sufficient power. In some variants, the online calculators will yield sample error computations as well. An extensive website developed by StatPages.org provides a wide array of statistical options to choose from including power, sample size, and experimental design calculations (see http://statpages.org/index.html#Power). Another useful option is available through Lenth (2006–2009). This resource provides Java applets (software) for power and sample size, allowing the researcher to more effectively plan statistical studies.
Factors Affecting Power
All statistical procedures and their power estimates are influenced by various research-related factors. For instance, use of a one- versus two-tailed test will increase power. Relatively, less evidence is required to find a significant effect with a one-tailed test than with a two-tailed test. The use of parametric versus nonparametric tests also affects power. Parametric tests are generally more powerful than nonparametric ones. Other factors include the alpha level (α), ES, sample size (n), and distribution variance (Park, 2003). These are discussed in more detail below.
Alpha Level
Most readers with a basic knowledge of statistics are familiar with the concept of the alpha (α) level or the p value. This numerical index reflects in part the level of risk researchers are willing to tolerate when they reject chance as a plausible explanation for the significant results derived from an inferential analysis (e.g., t test). By convention, this level is set at .05, signifying that the researcher is willing to accept 5% chance of rejecting the null hypothesis erroneously. To illustrate, if our fictional researcher wants to virtually rule out chance as a potential reason for finding significant mean differences between experimental and comparison groups, when conducting inferential tests the researcher needs to set the alpha level very low (e.g., α = .01). In other words, to reduce the likelihood of committing a Type I error (i.e., one incorrectly rejects the null hypothesis or saying there is a true difference between groups when there is not), the researcher selects a priori a conservative alpha level of .01.
As Kline (2004) elucidated, when an investigator designs a study to purposely reduce the chances of Type I error, there are perhaps unintentional consequences. For example, setting the alpha level to a very stringent value increases the potential for committing a Type II error (β). A more liberal alpha level (e.g., α = .10) translates into a lower β, which in turn leads to higher value for statistical power (1 − β). When the practical consequences of Type I error are not serious, the investigator may choose a higher alpha level to increase the likelihood of finding a relatively minor difference between group means. For instance, when considering the study of the NT effect on self-efficacy, the counseling method may be benign and present no drain on resources. In this case, selecting a conservative alpha level should have little, if any, practical consequences. However, given that NT is a relatively new technique, our researcher may want to detect even the smallest group difference on the self-efficacy measure. To do so, the investigator may increase the alpha level to .15 to enhance statistical power. The appropriate balance of alpha level and power is informed by the desired ES. It should be noted here that for purposes of journal publication most reviewers consider an alpha level of .05 the upper limit.
To recap, investigators test null hypotheses presuming them to be false. It is important, therefore, to consider the probability of committing a Type I error (α) and Type II error (β), and the role statistical power (1 – β) exerts in the process (Davey & Savla, 2010). Decisions about how to maximize statistical power, while at the same time balancing Type I and Type II errors, require a relatively sophisticated knowledge of research design and relevant statistical properties underlying inferential analyses.
ES
Since this topic is detailed later in the article, we only mention here how this concept influences statistical power. As accomplished researchers understand, when a study involves large sample sizes (e.g., over 500), the likelihood of finding even a minimal effect (e.g., main or group effect) is near 100%. However, in most cases where the group mean differences are negligible but statistically significant, the ES will be trivial, allowing the researcher to conclude that the statistically significant finding has little, if any, application to the real world.
There are multiple types of ESs that quantify the extent to which groups differ or the strength of two or more correlated variables (Thompson, 2008). In the circumstance of estimating the statistical power of t or F tests, researchers are generally interested in determining a standardized ES (e.g., Cohen’s d). As the ES value increases, the power to detect actual group differences increases (i.e., there is positive relationship between 1 − β and ES). For example, if our fictitious researcher’s study was designed in such a way to maximize statistical power (e.g., increase sample size for each group to 500), it is more likely that even a small ES for group differences on the dependent variable (self-efficacy scores) would be detected. If the inferential procedure deployed has low power, it is less likely to distinguish group differences even when they exist and the associated ES may be small. One of the best ways to enhance statistical power is to design a study with a more than adequate sample size.
Sample Size
Sample size is an important element not only affecting statistical significance but also statistical power (Maxwell & Delaney, 2004). Because larger samples are associated with more stable sample statistics, reduced sampling error (i.e., a lower standard error of the mean) and narrower confidence intervals, an increase in sample size is generally commensurate with a boost in statistical power. By maximizing power and minimizing sampling error, statistically significant effects with trivial ESs are more readily detected. That is, mean differences that may have little importance are detected. Researchers are cautioned, therefore, to keep in mind that a large sample can result in a statistically significant effect even when the ES is minimal. In brief, power analysis allows researchers to determine the size of the sample required to obtain a statistically significant result (Mertens, 2010).
Variance
Data variability is influenced by the reliability and validity of the measures used, the design of the study, and the extent to which extraneous variables are controlled. When there is less variability (e.g., sum of squares) in the population, the estimated standard error of the mean tends to be smaller (Maxwell & Delaney, 2004). Under these circumstances, the sample statistics provide relatively good estimates of the population parameters. Because lower variance also tends to be associated with higher levels of statistical power, researchers must seriously consider this factor.
For situations where two groups are compared, such as the one presented in our research scenario, the researcher assumes that the groups have equivalent variance in self-efficacy scores. For instances where the assumption of equality of variance is violated, the probability of committing a Type II error becomes inflated, and this in turn, lowers the level of statistical power. In an effort to restrict sample variability, researchers, for example, can be very specific in defining the sample to “moderate school behavior problems” or “mild-to-moderate depression and anxiety.”
Application of Power Analysis to Research Designs
This section considers briefly how power analysis is relevant to various research designs often utilized in counseling research.
Single Participant Design
In single participant designs, the researcher’s major concern is effectively increasing internal and external validity of the findings. Using a multiple baseline design is a good way to do so (Ferron & Sentovich, 2002). How participants are randomly assigned to the various baselines affects the level of statistical power. Power is largest in cases where there is an alternating treatment design (Ferron & Sentovich). That is, the participant receives one of two treatments at each experimental session. For instance, in our NT study, the researcher would design the investigation so each participant randomly receives NT in one session and perhaps another intervention in the next. However, to be more technically accurate, it is more common to observe the dependent variable across a number of baselines and then NT sessions; subsequently, in a reversal design (e.g., ABA or ABAB) the treatment is withdraw for several sessions to observe whether improvement measured by the dependent variable decreases, thus increasing the researcher’s confidence in the study’s internal and external validity.
Group Experimental Design
Increasing sample size, as previously discussed, is often a useful means to enhance statistical power. This can be costly in terms of money and effort, necessitating perhaps other more realistic ways of increasing power. For example, adding a pretest as a covariate while retaining one’s original sample size may lead to a lowering of the standard deviation of the error term for the same sample size. As a result, the ES, precision of statistical estimates, and statistical power increase. Although impractical in most settings, Venter, Maxwell, and Bolig (2002) recommended at least five repeated measures for the posttest to achieve significant gains in power. Any fewer than five repeated measures (testings) is likely no more beneficial than simply using a pretest-with-one-posttest-only design.
When groups are being compared on dependent variables, they need be relatively equivalent, for example, in their demographic characteristics. Researchers attempt to “equalize” the samples by randomly selecting participants for the study and then randomly assigning participants to groups. Obviously, when the study is unable to use randomization procedures, the findings from inferential statistical analyses are problematic (Jo, 2002). The lack of randomization confounds the treatment effect and tends to reduce the statistical power of the study.
To apply this information to our research scenario, the lack of randomization lowers statistical power. If the experimental group has a large proportion of treatment “no shows,” the power to detect differences in self-efficacy is greatly compromised. Other than encouraging participation, there is little else a researcher can do about noncompliance. However, an investigator can perhaps further adjust the study’s method (e.g., use more psychometrically sound instrumentation and improve sampling procedures) to minimize error and gain adequate power.
Meta-Analysis
Contemporary researchers are using meta-analysis as a quantitative method of research synthesis (Cooper, 2010; Konstantopoulos, 2008) and, at times, a statistical approach to increase statistical power in a fixed-effect model 3 where samples are from the same population (Cohn & Becker, 2003). As more studies are included in the meta-analysis, not only does power improve but the precision of descriptive statistics (estimates) to represent population values increases as well.
There are several criticisms leveled against meta-analytic studies. For instance, publication bias (i.e., journals largely publish articles with significant findings) and the lack of attention to statistical power are often cited as serious weaknesses (Muncer, Craigie, & Holmes, 2003). In cases where the null hypothesis is rejected, the statistical power issue is less of a concern. When the null hypothesis is not rejected, however, there are two possible explanations that need to be considered: the null hypothesis is true or the study included in the meta-analysis was underpowered. As we have mentioned previously, research conducted with low power requires a large ES to produce significant findings. In this pursuit, researchers may artificially inflate ES estimates and thus increase the heterogeneity of ESs in a meta-analysis. This practice tends to mask both the extent and the direction of the true ES. There are various ways to address this challenge (e.g., weighting the mean), but power analysis is the most effective method. The bottom line for researchers is “power level” should be one of the inclusion criteria in selecting studies for meta-analyses; failing this, at the very least, the statistical power of included studies should be discussed.
Closing Remarks on Statistical Power
Because hypothesis testing and statistical power are so interconnected, these topics are critical for researchers to address in their studies (Shieh, 2003; Thompson, 2008). Before funding a research study, granting institutions such National Institute of Mental Health want evidence that the study under consideration has sufficient statistical power. Power analysis may be an ethical issue as well (Miles, 2003). Consider, for example, counseling-related research. Clients may consent to participate in a relevant study with the hope of aiding future recipients of mental health services. It behooves researchers to do all they can to generate reliable and valid results with meaningful implications for clinical practice. As noted previously, an underpowered study will reduce the probability of finding a significant and practical effect.
Up to this point, we have discussed power analyses conducted before the results are fully analyzed. Although discouraged as an unsound research practice (e.g., Huck, 2009; Millis, 2003; Nakagawa & Foster, 2004), naïve investigators may conduct a retrospective or post hoc power analysis. This “after-the-fact” analysis determines the level of statistical power subsequent to a nonsignificant effect. Researchers deploying this method are tempted to conclude, on one hand, when a nonsignificant finding was attained with high power, that the finding reflects “reality;” on the other hand, a nonsignificant effect with low power is deemed “inconclusive.” Figure 1 provides a graphical representation of a retrospective power analysis (n = 40, p < .05). The statistical power estimate (y-axis) is a function of the detected ES (x-axis).
Graph representing a retrospective power analysis (n = 40, p < .05)
Critics of the retrospective approach point out that the technique disregards the impact of sample size and ES (Aguinis, Beaty, Boik & Pierce, 2005; Huck, 2009) by essentially detecting the power necessary to determine the ES already found. Depending on whether the investigator uses an a priori or retrospective approach, the same sample size and alpha level can lead to the equivalent level of statistical power regardless of ES and p values.
The preferred method is to conduct a prospective power analysis (a priori analysis) before the study is fully instituted. Millis (2003) underscored this point, suggesting that failure to conduct such a prospective analysis a tantamount to committing a research “deadly sin.” The procedure requires the researcher to determine the needed sample size to attain the desired power to detect a predetermined ES. Figure 2 shows a graph of statistical power as a function of sample size. For 80% power (usually considered to be sufficient power), the group size required is about 50. In the case of a two group comparison as in the above research scenario, 100 participants total would be then adequate.
Graph representing statistical power as a function of sample size
Although there a multiple commercially available products to choose from (e.g., nQuery 7.0, PASS: Power Analysis and Sample Size 2008), a no-cost online tool for conducting various statistical power analyses, including the a priori method, is called G*Power III 4 (available at: http://www.psycho.uni-duesseldorf.de/abteilungen/aap/gpower3/). Ultimately, the goal is to “prevent” investigators from conducting time-consuming and costly studies with little chance of detecting significant treatment/intervention effects. In the event that the power analysis produces undesirable outcomes, the researcher can reconceptualize the study and improve its design and execution.
Sampling
As alluded to previously, sample size is a strategic component to enhancing statistical power and potentially ESs. If researchers want to capitalize on the power gained from collecting data from large samples, effective representative sampling procedures must be used. Because this is a complex issue and there are numerous texts to consult (e.g., Cochran, 2007; Creswell, 2009; Houser, 2008), we only address the major considerations affecting statistical power and ESs. Obviously, the study’s context and goals direct what sampling method is deployed. In most situations, the most advantageous sampling approach is one that is realistic, efficient, and, perhaps most importantly, minimizes potential error variance from sampling bias (i.e., systematic error associated with nonrandom sample from a population) and produces representative samples. Researchers want to reduce bias in the data, which can often lead to severe non-normal distributions, including those that show substantial kurtosis and/or skewness.
One of the best ways to maximize power and to increase the likelihood of obtaining a normal distribution of error variance is through true probability sampling involving various randomization techniques (e.g., simple, stratified, multi-stage, and systematic). Ideally, counseling researchers should attempt to include 30 or more participants per group (Judd, McClelland, & Ryan, 2009). Given the nature of research ethics, availability of clients, and financial constraints, for most counseling studies using random sampling is not appropriate. When this is the case, the next option researchers turn to is nonprobability 5 sampling such as selective (e.g., purposive, expert, and snowball) and convenience methods (e.g., use participants on hand). Typically then, counseling studies resort to soliciting volunteers from an opportune location(s) (e.g., a local mental health clinic or school), where researchers use naturally formed groups (e.g., clients in a counseling center, families in faith community) in that setting as study participants (Creswell, 2009). Regrettably, with all approaches to nonprobability sampling, investigators cannot be assured of representative samples and that the data they collect will be appropriate for parametric inferential analyses.
We recommend that researchers limited by funding, potential participant groups, and ethical considerations, use a modified form of randomization with nonprobability sampling. Using fairly large sample sizes (n ≥ 30 per group) drawn from intact groups, researchers can randomly assign therapists to treatments and participants to groups (e.g., experimental vs comparison). If random assignment is not feasible, matching participants to groups is the next best alternative. If possible, match participants in the experimental and comparison groups on key demographic variables (e.g., gender, ethnicity, age, socioeconomic status [SES]). Prior to computing inferential statistics, ensure the similarity of groups and the appropriateness of the data for parametric analyses through data screening techniques (e.g., scatterplots, box-and-whisker plots) and reviewing relevant descriptive statistics (e.g., Ms, SDs, kurtosis, skewness) 6 . In brief, rigorous sampling reduces error variance in the groups, which in turn improves statistical power and the chances for sizable and consequential effects.
ESs
There is a wealth of literature detailing the statistical properties, the need for, and value of ESs in quantitative psychology-related research (see, e.g., Cortina & Nouri, 2000; Grissom & Kim, 2005; Huberty, 2002; Kline, 2004; Sink & Stroh, 2006; Thompson, 2006a, 2006b, 2007, 2008; Trusty, Thompson, & Petrocelli, 2004; Vacha-Haase & Thompson, 2004, for a detailed discussions). These numerical indices essentially provide an estimate of the magnitude of the effect, which in turn represent the practical value or clinical utility of the statistical finding. In this section, we briefly underscore the rationale for including ES estimates in studies, review the types of ESs and their interpretation, as well as clarify the relationship between ES and sample size, which, as explained above, is also an important element of calculating statistical power (Cohen, 1988, 1992; Huck, 2009).
Applying the research scenario summarized above, let us assume, for whatever reason, that the counseling investigator was mainly concerned with possible group mean differences at the follow-up phase of the study and ignored possible mean differences at the pre- and posttest phases. Furthermore, the experimenter anticipated finding a statistically significant result as well as a modest ES (e.g., Cohen’s d = .40), suggesting that NT is more clinically useful with adolescent clients with internalizing disorders than with youth exhibiting school behavior issues in improving self-efficacy. After consulting a basic statistics text, the researcher opted for a simple independent samples t test, computing it using a typical statistical software package like SPSS (2009). A statistically significant result for group differences (t = 10.95, two-tailed, p < .001; Me = 70.5, SD = 7.5; Mc = 65.5, SD = 9.5) was revealed.
The investigator, pleased with the statistically significant t value, next asked a follow-up question: Is the mean difference between samples, favoring the experimental group (clients with depression/anxiety), practically or clinically significant? In response, depending on what statistical procedure (e.g., t test or F test using the general linear model 7 [GLM]) one uses, SPSS (2009) generates a relevant ES index, indicating the size of the significant effect (see Trusty et al., 2004, for discussion of SPSS and ESs in counseling research). Given the mean of experimental group was significantly larger than the comparison group’s average score and the samples have comparable standard deviations, the researcher anticipates at least a moderate ES. However, experimenters must be careful about making this type of prediction, because practical significance as measured by an ES can be near zero even though statistical significance was found (Huck, 2009; Kline, 2004).
Rationale
As implied above, there are good reasons for including ES estimates when reporting on the findings of a quantitative study. Most notably, doing so makes statistical sense and reflects “best practices” (Fidler & Cumming, 2008; Kline, 2004; Thompson, 2008). The American Psychological Association’s (APA, 2010) Publication Manual underscores this point, reiterating that the traditional manner of presenting quantitative findings—the null hypothesis statistical testing method 8 —is insufficient. By only reporting the inferential statistics (df, value of a t, F, or χ2 test, derived p value) and whether the null hypothesis was rejected or not, readers have little perspective on the wider meaning of the results (see Kline, 2004, for technical discussion). Similarly, in an attempt to bolster the rigor of educational research, the Coalition for Evidence-based Policy (2003) argued that studies must report the size of the intervention’s effects, so that readers can judge their educational import (p. 9). The Results section of a manuscript, therefore, should include ES indices, and, if relevant, associated confidence intervals. In summary, without a measure of practical significance, the “real-world” importance of the statistical findings may be lost.
Types of ESs and Interpretation
Thus far, we have attempted to correct a common research misconception: statistical significance does imply practical significance (Fidler & Cumming, 2008; Huck, 2009). Statistical significance involves null and alternative (experimental) hypotheses, a priori significance level (commonly called an alpha level or α), and the derived p value. Practical significance is determined using the statistical data generated after the inferential statistical procedure is computed.
As summarized below, two major classes of ESs (group difference and relationship- or variance-explained indices) are generally discussed in related technical publications (e.g., Grissom & Kim, 2005; Huberty, 2002; Kline, 2004; Rosnow & Rosenthal, 1996; Thompson, 2006a, 2006b, 2008; Vacha-Haase & Thompson, 2004). Complicating matters, however, numerous ES indicators are available to researchers and ESs exist for both parametric and nonparametric tests (see Sheskin, 2007; Sink & Stroh, 2006; Thompson, 2006a, 2006b, 2008, for application-oriented summaries). Because a synopsis of all these alternatives is beyond the scope of this article, we consider here only those indices most often reported in quantitative studies using parametric statistical tests.
Group difference indices
These values 9 , often categorized as part of the “d family,” should be included when conducting group comparison studies. For instance, when experimental and control (or comparison) groups are statistically compared after an intervention, various d indices (e.g., Cohen’s d) are frequently reported. In meta-analytic studies, Glass's delta (Δ) is widely reported as the d family ES. Essentially, a group difference index involves calculating the size of the mean difference between two samples (e.g., experimental and comparison), taking into account group size (n) and standard deviation (SD) (Kline, 2004). More precisely, the d estimate—a standardized ES—is calculated as the difference in the mean outcome between the intervention/treatment and comparison/control groups, divided by the pooled standard deviation (Thalheimer & Cook, 2002). Standardized ESs are understood in a similar way as a z score, where a d is calculated on a common scale, allowing the researcher to evaluate the “success” of different intervention programs using the same metric.
Our fictional investigator computed an independent samples t test revealing a statistically significant effect for Group (t = 10.95, two-tailed, p < .01). As anticipated, the participants in the experimental sample on average outperformed the adolescents in the comparison group. To determine the magnitude of the independent variable’s impact on the dependent variable, descriptive statistics (ns, Ms, and SDs) from the case study are used to calculate Cohen’s d, a well-recognized group difference index. This indicator is easily computed from one of several online ES calculators (see Table 1 for sample websites). After inputting the relevant data, the d = .58. To adequately grasp the nuances of Cohen’s d, the relevant statistical formulae are overviewed next.
Online Resources for Effect Size (ES) Calculations and Interpretation
The generic formula
10
for Cohen’s d is:
To extend the previous discussion, suppose another researcher computed a univariate or one-way analysis of variance (ANOVA) comparing posttest mean differences from three groups of participants after each received 10 weeks of intensive NT. In this situation, a d index is computed from the ANOVA’s F test statistics (Cortina & Nouri, 2000). Here, the experimenter could generate a d for the mean comparison between Groups 1 and 2 (M
1 with M
2). Subsequently, ds can be calculated for other group comparisons (e.g., M
1 with M
3) as well. The required formula is as follows:
Another standardized mean difference ES most often reported in meta-analyses is Glass's delta (Δ) (Sink & Mvududu, 2009). The formula looks similar to Cohen’s d:
After the calculation of Cohen’s d or Glass's Δ, the researcher still needs to interpret the derived estimate in practical terms, that is, to evaluate “whether an intervention makes a meaningful difference in an applied setting” (Kline, 2004, p. 135). In our fictional study, the researcher computed the d, yielding a value of .58. So what does this number mean? Before we answer this question, some further information is still required. Theoretically, ds and Δs can range between 0.00 and ±4.00 (Kline), but rarely in research with human respondents do they even reach the ±1.50 threshold. Generally, they range between 0.0 and 1.0. If the experimental group’s mean is significantly higher than the comparison group’s mean (i.e., the alternative hypothesis is retained and null hypothesis is rejected), the d or Δ will be a positive number. In the odd situation when the comparison group outperforms the experimental group, the ES will be in the negative range. According to Cohen’s (1988, 1992) “rules of thumb,” an ES of .20 is considered relatively small; .50 is the benchmark to be considered medium-sized; a d of .80-plus is a large ES. Actual ESs from counseling, social science, and educational research are often less than optimal, suggesting that Cohen’s estimates may be somewhat inaccurate for meta-analytic research in the helping professions (Lipsey & Wilson, 2001). Table 2 provides a quick reference guide to interpreting d-family ESs.
Guide to Interpreting Group Difference (d-family) Indices
Note. Adapted from Becker (2000) and Cohen (1988); d or Δ can exceed 2.0 but this situation is extremely rare; see Kline (2004) for discussion of distribution overlap; benchmarks are tentative.
Group difference indices provide researchers with estimates of how much the experimental group surpassed the control (comparison) group following an intervention using a standardized quantitative index. For example, when a d or Δ equals .08, the ES can be understood in several related ways. First, participants in the experimental and control (comparison) groups produced relatively similar means (i.e., the intervention “produced” only a trivial or near-zero mean difference between groups, Me ≈ Mc ). Second, the mean of the experimental group is around the 50th percentile (actual = 53rd percentile) of the comparison group. Third, the distribution of scores for the experimental group overlaps almost completely with the distribution of scores for the comparison group (i.e., the percentage of score distribution overlap is near 100%; actual = 94% overlap). Finally, a near-zero d is a very small ES. Similarly, to interpret a d of .5, a moderate ES, Table 2 shows that the mean of the experimental group is at the 69th percentile of the comparison group. In this case, the experimental group outperformed the control (comparison) group by .5 of a SD, with the percentage of distribution nonoverlap for both samples being approximately 33%.
We now have enough background information to answer the researcher’s primary question regarding practical significance. A d of .58 is considered a moderate ES, indicating that the experimental group’s mean was around the 73rd percentile of the distribution of the comparison sample. The researcher is able to conclude that clients in the experimental sample when tested at the study’s follow-up phase had higher self-efficacy scores (~ .6 SD) following NT than did participants in the comparison group. The researcher must also report the caveats (e.g., issues related to internal and external validity) to making this assertion.
Relationship or Variance Explained
The second category of ESs, commonly referred to as the “r family” indices, reflect the degree of association or covariation (i.e., shared variance) between the independent and dependent variables (Kline, 2004). Readers are perhaps familiar with the Pearson r, which provides an estimate of the observed strength of the relationship between two continuous variables (e.g., variables using interval data). Bivariate correlations range from 0.00 to ±1.00. As rs approach 1.00, the stronger the relationship between variables. If r is squared (r 2) then this value represents the proportion of explained (shared) variance between two variables (i.e., coefficient of determination), ranging from .00 to 1.00. If the r 2 is multiplied by 100, one obtains the percentage of explained or shared variance, ranging from 0% to 100%. In quantitative research, the aim is produce a large r, and thus, a strong r 2, indicating a large percentage of score overlap between two variables.
With this information in mind, we return to our invented research example. An investigator used an ANOVA to compare posttest self-efficacy mean differences from three groups of participants after each received 10 weeks of intensive NT. In this scenario, we indicated that a d could be calculated as a useful ES index. However, in comparative studies with two or more independent variables (e.g., participant group, counseling treatment, clinic location), researchers often analyze the effect of these variables on the dependent variable(s) (e.g., participant scores on self-efficacy, mood, or personality inventories) using GLM factorial procedures (e.g., factorial ANOVA, analysis of covariance [ANCOVA], or multivariate analysis of variance [MANOVA]).
Keeping the technical details of the GLM procedure to a minimum and somewhat oversimplified, the typical statistical software package like SPSS (2009) will generate a multiple correlation ratio (R 2) for the overall ANOVA model 13 (see, e.g., Field, 2009; Green & Salkind, 2008, for SPSS procedures and statistical output) and an eta square for each ANOVA effect. Specifically, the R 2 is the ES for the overall ANOVA model (all variance components are included; e.g., total sum of squares, error sum of squares, main effect[s] and interaction effect[s] sum of squares). This multiple correlation ratio represents the amount of explained variance in the dependent variable by the combined main and interaction effects, excluding mean square error (MSE) term.
Moreover, SPSS conveniently outputs an additional ES for each main (independent variable [IV]) and interaction (e.g., IVa by IVb effect). Although interpreted in the same way as a R 2, depending on the type of analysis of variance requested, SPSS produces an eta squared (η2) or a partial eta squared (ηp 2). The former ES represents the ratio of the sum of squares for a particular effect of interest (e.g., a main effect/independent variable) to the ANOVA’s total sum of squares (i.e., total sums of squares for all main and interaction effects plus error). Therefore, η2 = SS effect/SS total. When η2 = .40, for example, this means that 40% of the variability in the dependent variable can be explained or accounted for by the independent variable. However, a more precise ES to report for each main and interaction effect in a GLM factorial design is the ηp 2. Technically, this ES is defined as the ratio of sum of squares for a particular effect (main or interaction) to an adjusted total sum of squares, which is composed of two variance components: (a) the sum of squares for the effect under consideration and (b) the sum of squares for the error term associated with that effect (Huck, 2009). Thus, the partial eta square is symbolically represented as: ηp 2 = SS effect/(SS effect + SS error). The ηp 2 gives researchers a more precise ES estimate for a specific effect (e.g., independent variable or main effect A, independent variable or main effect B, or the A × B interaction effect). In other words, the ηp 2 is understood similarly to a partial correlation squared, in that, this ES is the proportion of variance explained by a particular effect (main or interaction) after removing any variance associated with the other effects. Practically speaking then, when a counseling researcher computes a factorial ANOVA and SPSS (2009) outputs a partial η2 of .15 for the main effect (independent variable) of Group, one can report that the effect for group accounted for 15% of the group-differences plus related error variance.
Researchers should also note that when computing a one-way ANOVA, the η2 as the ES for the particular independent variable (the main effect) is reported. However, this is not the case in studies where there are two or more independent variables (factorial designs). If, for instance, the researcher wants to investigate the effects of Group (2 levels: experimental and comparison) and SES (3 levels: low, medium, and high) on the dependent variable (e.g., self-efficacy total score), a two-way ANOVA could be computed. Here, we have two main effects (Group and SES) and the interaction effect (Group by SES). For each of these three effects, the researcher should report a partial η2. For the overall two-way ANOVA model, the η2 can be reported in the findings as well. Table 3 provides user-friendly guidelines for determining what constitutes a small, medium, or large variance-explained index. Finally, because counseling researchers often use ANOVA or an associated procedure as the inferential approach of choice for group comparison studies, consulting Howell (2008) for a practitioner-friendly discussion of this statistical method is recommended.
Guide to Interpreting Variance Explained Indices
Note. Each squared ES ranges from 0 to 1; multiplying each ES value by 100 yields a percentage of explained variance.
a Threshold values are usually smaller than those from an eta squared; hence, small, medium, and large benchmarks for ηp 2 are probably too large, so they must be interpreted with caution (Green & Salkind, 2008; Sink & Stroh, 2006).
Additional Comments and Caveats on ES Interpretation
Given the numerous ESs available to researchers, readers may wonder which ones are best to report in quantitative studies. Like most issues in research and statistics, there is not a universal rule of thumb to apply to all studies. Because d family and r family ESs can be computed for most inferential statistics in group comparison research designs, both are easily reported. However, following Thompson’s (e.g., 2007, 2008) lead, APA (2010) and Coalition for Evidence-based Policy (2003) recommended that “variance-accounted-for” ESs might be better understood by practitioners if they were converted to unsquared indices associated with the d family of ESs. Coalition for Evidence-based Policy (2003) authors provided sample statements from published research that assist readers to understand the practical importance of a study’s findings. For example, an investigator might report the following:
Based on the ESs derived from the significant findings, students in the experimental group in contrast to the control group improved their vocabulary skills by two grade levels, as well as showed a 20% reduction in weekly use of illicit drugs, and a 20% increase in high school attendance.
When reviewing previous research, authors might want to follow this example from Wasik and Slavin (as cited in Coalition for Evidence-based Policy, 2003):
Evidence from randomized controlled trials, discussed in the following journal articles, suggests that one-on-one tutoring of at-risk readers by a well-trained tutor yields an ES of about 0.7. This means that the average tutored student reads more proficiently than approximately 75% of the untutored students in the control group. (p. 9)
Finally, the decision as to which ES to include in a research manuscript depends on the study’s aims, samples, context, and target audience (e.g., Thompson, 2006a, 2006b, 2007). Moreover, when interpreting ESs researchers must consider individual differences among client groups and the ES findings from previous research with comparable interventions and target groups (Hill, Bloom, Black, & Lipsey, 2008). Correspondingly, investigators and consumers of research are cautioned not to apply Cohen’s (1988, 1992) guidelines too rigorously when characterizing what ESs are small, medium, or large. Again, ES interpretation must be situated to the particular research application and with any d-related ES, researchers should, if appropriate, include confidence intervals (Cohen & Lea, 2004; Thompson, 2007).
Conclusion
In this article, we reviewed and practically discussed the interdependency of statistical power, sampling, and ESs within the context of quantitative counseling research. Reflecting in part Osborne’s (2008) recommendations, it is our hope that future studies will use research designs that are well conceptualized, methodologically sound, and executed in a manner that minimizes error and maximizes statistical power and ESs. If possible, a priori power analyses should be conducted. Furthermore, the proposed sampling method must be carefully scrutinized, ensuring that the samples are relatively similar in composition and representative of the population. It is essential that counseling investigators move beyond the null hypothesis significance testing approach when computing and reporting on inferential statistics. All significant findings should be accompanied by their relevant ESs and associated confidence intervals. By adopting these best practices, counseling studies will reflect the highest standards set for quantitative research.
Footnotes
Notes
The authors declared no potential conflicts of interests with respect to the authorship and/or publication of this article.
The authors received no financial support for the research and/or authorship of this article.