Abstract
For a variety of reasons, researchers and evidence-based clearinghouses synthesizing the results of multiple studies often have very few studies that are eligible for any given research question. This situation is less than optimal for meta-analysis as it is usually practiced, that is, by employing inverse variance weights, which allows more informative studies to contribute relatively more to the analysis. This article outlines the choices available for synthesis when there are few studies to synthesize. As background, we review the synthesis practices used in several projects done at the behest of governmental agencies and private foundations. We then discuss the strengths and limitations of different approaches to meta-analysis in a limited information environment. Using examples from the U.S. Department of Education’s What Works Clearinghouse as case studies, we conclude with a discussion of Bayesian meta-analysis as a potential solution to the challenges encountered when attempting to draw inferences about the effectiveness of interventions from a small number of studies.
Keywords
A number of evidence-based practice repositories or clearinghouses have been created that attempt to summarize evidence in ways that are useful in public policy contexts, including the What Works Clearinghouse (WWC; http://ies.ed.gov/ncee/wwc/), SAMHSA’s National Registry of Evidence-Based Programs and Practices (NREPP; http://www.nrepp.samhsa.gov/), the Office of Justice Programs’ CrimeSolutions.gov (http://www.crimesolutions.gov/), and others. These repositories review and summarize evidence on programs and practices in education, social welfare, and crime and justice and share a similar mission of attempting to produce reliable and valid syntheses of the literature in their coverage areas. Clearinghouses often make methodological choices that tend to limit the number of studies that are available for review. These choices include (a) narrowly defining the research question and (b) not carrying out thorough and systematic searches for potentially relevant studies. In the sections below, we briefly describe these choices and their implications. We then discuss options for synthesizing evidence, including narrative reviewing, vote counting, and traditional approaches to meta-analysis. When there are few studies to synthesize, we show that many common synthesis options are suboptimal and that even the most recommend synthesis options (classical fixed effect or random effects meta-analysis) are problematic. We conclude by demonstrating that a Bayesian approach to meta-analysis is a potential solution to the issues raised when trying to synthesize a small number of studies.
The Scope of the Research Question
Many evidence-based clearinghouses focus on narrowly defined programs (e.g., aggression replacement training) rather than focusing on more broadly defined practices (e.g., social skills interventions) or programs with targeted outcome interests (e.g., any school-based program that targets dropout). Although this narrow focus is aligned with the types of decisions typically made by practitioners and policy makers (i.e., “Will this program work for my school?”), it has implications for the types of conclusions that might be drawn from a limited number of studies. First, among programs that target the same outcome (e.g., dropout prevention programs), specific intervention components are likely to be similar across different interventions. That is, if there are 10 school dropout prevention programs and each of these programs has five elements, it is safe to say that there will not be 50 unique components across the 10 programs. Carrying out separate reviews of these interventions will tend to mask the similarities across programs and will impede efforts to investigate the extent to which different intervention components are associated with program success. The second important implication of narrowly defined research questions is that, almost by definition, there will be fewer studies to review. That is, there surely will be a larger number of studies of social skills interventions than there will be of one specific type of social skills intervention like aggression replacement training. Therefore, the narrow focus of most research question posed by clearinghouses limits the number of studies that can be reviewed.
Searching the Literature
Regardless of the scope of the intervention at issue (whether we’re interested in finding all of the studies on a particular branded program or have a broader question about the impacts of social skills training), it is clear that including different sets of studies in a synthesis can lead to different conclusions about the effects of a program or practice. And, generally speaking, if we want to know about the effects of an intervention on an outcome, we are better off having access to all of the studies that have been conducted on that intervention rather than to only a selected portion of that literature. Furthermore, the most defensible conclusions can only be drawn from having the complete set of available research. One major challenge to identifying this comprehensive set of studies arises from publication bias, which refers to the tendency of the published literature to suggest effects that are larger than those observed in the unpublished studies (Dwan et al., 2000). Publication bias occurs because study authors are less likely to submit for publication, and journal editors and peer reviewers are less likely to accept studies that do not have statistically significant findings on their primary outcomes. These behaviors are attributable in part to a persistent disregard for statistical power and to common misconceptions regarding the interpretation of probability values arising from null hypothesis significance tests. Sample size and probability values are related (i.e., holding everything else constant, a study with a larger sample will yield p values that are smaller than a study with a smaller sample). Therefore, studies with small samples that by chance happen to find relatively large effects will be more likely to be published—and hence easier to find—than studies with small samples that do not find relatively large effects. This results in an evidence base that suggests effects that are larger than they really are and therefore in a bias against the null hypothesis.
A second challenge in identifying the complete evidence base is associated with both the capacity of clearinghouses to conduct comprehensive literature searches and with the practice of intentionally selecting studies with particular characteristics or from particular sources for review. These actions limit the literature reviewed from the outset. For example, some clearinghouses accept nominations from outside researchers or developers for inclusion. If the clearinghouse only has studies voluntarily submitted by an intervention’s developer, a clear concern is that the developer might submit studies for review that might not be representative of the effects that the intervention actually produces. As can be seen in Table 1, even though best practice is to conduct a comprehensive and systematic search, not all clearinghouses attempt to locate all of the potentially relevant studies that have been conducted. In addition, the scientific skills needed to implement a comprehensive search should not be underestimated, and generally speaking, a professional librarian with training in retrieval for systematic reviews should be involved (Rothstein & Hopewell, 2009). Finally, a high degree of subject matter expertise is often needed to find the so-called “gray” literature, for example, studies commissioned for foundations and governmental agencies for which there is not a strong publication incentive. In our experience, evidence-based clearinghouses face a particular challenge in this regard because some of them may not have the degree of expertise, either in the substantive research question or in general literature retrieval techniques, to carry out a robust literature search even if they have the goal of doing so.
Study Identification, Assessment, and Synthesis Across Eight Evidence-Based Clearinghouses.
Synthesizing the Studies
Once studies have been located, they are typically screened for relevance, coded, and quality appraised. The next major task confronting researchers and evidence-based clearinghouses is to synthesize the studies, that is, to determine what the studies collectively reveal about the effect under investigation. Because meta-analyses allow researchers to (a) transparently reach a conclusion about the extent to which an intervention is effective and (b) statistically investigate sources of heterogeneity (Higgins & Green, 2011), when multiple studies that examine the same outcome are available, there is little debate among statisticians that the best way to integrate the results of the studies is by using some form of meta-analysis. In practice, however, researchers (including the evidence-based clearinghouses) use a variety of techniques to arrive at conclusions or statements regarding the effectiveness of interventions. These include (a) narrative reviewing, (b) vote counting, (c) setting rules regarding the number of studies that have statistically significant results, and (d) a variety of forms of meta-analysis. Each of these techniques is discussed below.
Narrative reviewing
In the past, reviewing the literature on a set of related studies relied exclusively on a narrative review, in which a scholar would gather some studies that were relevant, read them, then pronounce on what those studies had to say. Typically, little attention was paid to whether the studies could claim to be representative of the studies that had been conducted, and almost nothing was said about the standards of proof that were employed during the review (in other words, the pattern of results that would lead the reviewer to conclude that the intervention “works”). This lack of transparency, forethought even, leads to conclusions that are more likely to be a product of the reviewer’s experiences, preferences, and cognitive algebra. Further, the results of narrative reviews—which unfortunately continue to be common—are often presented in impressionistic terms, with little insight provided about the magnitude of the observed effect (i.e., how much of an effect the intervention had on participants). Scholars have increasingly recognized that narrative literature reviews do not meet the standards of rigor and transparency required in primary research, precipitating the increased use of systematic review methods (see Cooper & Hedges, 2009, for a review of the history and issues related to narrative reviewing).
Vote counting
Sometimes used in conjunction with a narrative review, vote counting in its most common form is based on counting (as a “vote”) the statistical significance of the results observed in the studies that are being reviewed. For example, if one study found statistically significant effects for an intervention, the researcher would count that as a vote that the intervention works. If another study failed to reject the null hypothesis, the researcher would count that as a vote that the intervention does not work. If another study finds harmful effects, the researcher would count that as a vote that the intervention “harms.” When all studies have been processed this way, the category with the most votes wins.
Vote counting is a seriously limited inferential procedure, in part because it requires that most studies must have statistically significant results in order for the claim to be made that an intervention works. Unfortunately, in most circumstances when using vote counting, it is unacceptably probable that studies will not reach the same statistical conclusion, even if they are estimating the same population parameter (e.g., if the intervention really is effective). The fundamental problem is that for vote counting to work reasonably, the underlying studies all need to be conducted with very high levels of statistical power. Unfortunately, relatively few studies have very high statistical power, and on average in the social sciences, statistical power is only about .50 (Cohen, 1962; Pigott, Valentine, Polanin, & Williams, 2013; Sedlmeier & Gigerenzer, 1989). If two independent studies are conducted with statistical power of .80 (meaning that both have an 80% chance of correctly rejecting a false null hypothesis), in only 64% of cases will both studies result in a correct rejection of the null hypothesis. If both studies are conducted with statistical power of .50, then in only 25% of cases will both studies result in a correct rejection of the null hypothesis. If there are three studies, only 50% of time would we expect at least two of the studies to be statistically significant, given that all three studies were conducted with power of .50. As such, because studies are typically not highly powered, in most current real-world contexts, vote counting is an approach with an unacceptably high error rate (by failing to detect real intervention effects when they exist). In fact, Hedges and Olkin (1985) demonstrated the counterintuitive result that, in many situations common in social research (i.e., interventions with moderate effects investigated in studies with moderate statistical power), vote counting based on statistical significance has less statistical power as more studies are available.
Rule setting
A particularly egregious version of vote counting involves setting rules for the number of studies that need to have statistically significant outcomes in order to conclude that an intervention works. Most commonly, the requirement is for at least two “good” studies that reveal statistically significant effects. As a form of vote counting, it shares the overreliance on statistical significance testing and the neglect of statistical power with that technique. It has the additional disadvantage of neglecting the number of studies that have been conducted. It is one thing if two of the two studies conducted on an intervention yielded statistical significance. It is quite another if 2 of the 20 studies conducted reached statistical significance. Unfortunately, as will be seen, these types of rules are relatively common, and systems that simply require two studies to meet some threshold of quality while ignoring the total number of studies that have been conducted run the risk of making this serious inferential error.
Meta-analysis
Meta-analysis is another method for synthesizing the results of multiple studies. As will be seen, it is not a perfect solution (especially when there are few studies), but given the need for a synthesis, it is better than the alternatives (Valentine, Pigott, & Rothstein, 2010). Although there are many varieties of meta-analysis, we focus below on three: fixed effect, random effects, and unweighted meta-analysis. We briefly introduce the strengths and limitations of each.
Fixed effect meta-analysis starts with the assumption that all of the studies in the meta-analysis are estimating the same population parameter. One way of thinking about this assumption is that if the studies in the meta-analysis were all infinitely large, they would all have exactly the same effect size. Meta-analysis usually involves arriving at a weighted average of the study effect sizes—this means that the mean effect size from a meta-analysis is computed like any other weighted mean. The trick is in finding the right weights. Most fixed effect meta-analyses use inverse variance weighting, in which studies are weighted by a function of their sample size. Therefore, large studies are given relatively more weight in the meta-analysis than smaller studies.
The main limitation of a fixed effect meta-analysis is the assumption that all studies in the analysis are estimating the same population parameter. This assumption implies the fixed effect model is most appropriate if the studies are highly similar to one another along important dimensions that contribute to variation in effect sizes (Hedges & Vevea, 1998) or in other words, if the studies are close replications of one another (i.e., they are very similar on all the dimensions that matter, including they specifics of intervention implementation, the sample, the measured outcomes, and so on). In reality, this is a difficult bar to reach, as most studies of the same intervention are probably not close replicates of one another. Instead, they are likely “ad hoc” replications that vary in known and unknown ways (Valentine et al., 2011). This means that conceptually the fixed effect model is often not a good fit.
Random effects meta-analysis relaxes the assumption that all studies are estimating the same population parameter. Instead, studies are conceptualized as part of a distribution of plausible effect sizes that vary around a mean population effect size. Effects from studies in a meta-analysis are therefore expected to vary from one another due to both known and unknown study characteristics in addition to random sampling error. Like fixed effect meta-analysis, random effects meta-analysis involves computing a weighted effect size. In the random effects model, studies are weighted by a function of their sample size and by an estimate of the extent to which the study estimates “disagree” with one another (called the between-studies variance). Relative to the fixed effect model, the random effects model is generally more conservative. The confidence intervals arising from a random effects analysis will never be smaller and are usually larger than their fixed effect counterparts, making it less likely that the statistical conclusion following from an inferential test involving a random effects estimate will be a type I error (i.e., an incorrect rejection of a true null hypothesis).
However, one critical limitation of the random effects approach is that the estimate of one key parameter in the analysis, the between-studies variance, is poor when the number of studies is small (as a rule of thumb, a bare minimum of five studies in the meta-analysis is needed to support estimation of the between-studies variance, although very often many more studies will be needed; Viechtbauer, 2005). As a result, the estimated mean effect size and confidence intervals can be either too large or too small relative to what they “should” be, depending on whether the between-studies variance is over- or underestimated.
The last meta-analytic technique we will discuss in this section is unweighted meta-analysis. Here, the mean effect size is computed as straight mean of the observed effects (i.e., the sum of the effect sizes divided by the number of studies). While simple, this method of meta-analysis has two undesirable properties. First, it is a type of fixed effect analysis but is less efficient than its more commonly implemented cousin. That is, the standard errors arising from an unweighted meta-analysis are larger than the standard errors from a fixed effect analysis using inverse variance weights (unless the sample sizes are equal across studies in which case the standard errors will be equal in the inverse variance weighted fixed effect and unweighted meta-analyses). This means that in most cases, the unweighted meta-analysis will have a larger confidence interval than the inverse variance weighted model. The other undesirable property is that an unweighted meta-analysis is more vulnerable to the effects of publication bias than the inverse variance weighted fixed effect model. One signal that publication bias might be operating is the commonly observed negative correlation between sample size and effect size in a meta-analysis (i.e., as sample size increases, effect size decreases). This negative correlation means that smaller studies are observing larger effects than larger studies are observing possibly because small studies that find small effects are being systematically censored from the literature. Because the unweighted model does not involve weighting by sample size, it will be more affected by publication bias than the inverse variance weighted fixed effect model will be. Stated differently, weighted models generally have a degree of built-in protection against the effects of publication bias because relatively small studies have relatively less weight.
Evidence-Based Clearinghouses
With these quality indicators as background, we turn now to a brief overview of the ways that several evidence-based clearinghouses carry out their syntheses. We present how evidence is located, how the quality of the located studies is assessed, and how the results of multiple studies are synthesized. In all, we researched practices of eight registries, using only information provided on their websites. Table 1 summarizes study identification, study review, and outcomes synthesis practices across eight registries across a variety of funding sources: Blueprints for Healthy Youth Development, California Evidence-Based Clearinghouse, Coalition for Evidence-Based Policy, Crimesolutions.gov, NREPP, Promising Practices Network, WWC, and What Works in Reentry.
Across these eight registries, three of them identify studies through nominations, supplemented by periodic literature searches (Blueprints, Crimesolutions.gov, and NREPP). One registry, the Coalition for Evidence-Based Policy, identifies studies through nomination only. For the California Evidence-Based Clearinghouse for Child Welfare, nominations of studies are secondary to the literature searches. For the WWC and What Works in Reentry Clearinghouse, only systematic literature searches are used. Seven of the eight registries include published and unpublished studies. The California Evidence-Based Clearinghouse for Child Welfare only searches for published, peer-reviewed studies.
Most of the registries reviewed for this article use rules to synthesize the results of studies. For example, What Works in Reentry assigns its highest rating for interventions with one high-quality randomized controlled trial or two high-quality quasi-experimental designs (QEDs). Three of the registries use some form of vote counting (Blueprints, Crimesolutions.gov, and NREPP). Only one clearinghouse (the WWC) conducts meta-analysis in order to synthesize effect sizes reported across studies.
Synthesizing Studies in an Information Poor Environment
Although not all evidence-based clearinghouses conduct formal syntheses, our review suggests that when they do it is clear that most of these are based on a very small number of studies (e.g., two or perhaps three). For example, between January 2010 and September 2014, the WWC conducted 56 meta-analyses. The mean number of studies in the meta-analyses was 2.8, and both the mode and the median were 2.0. As we discussed earlier, the small number of studies evident across clearinghouses is partly a consequence of the relatively narrow focus of most synthesis efforts carried out by these organizations. That is, these clearinghouses tend to focus on specific interventions themselves rather than on the interventions as a class (e.g., they focus on a specific brand of a nurse visiting program rather than on nurse visiting programs as a whole). Shifting the focus to a more abstract level might admit more studies into the meta-analysis and might also allow for moderator tests that directly address dimensions relevant to decision-making (e.g., comparing studies that involve more or less intensive doses of nurse visits).
Assuming that, for whatever reason, only a small number of studies is available for review, how should evidence-based clearinghouses synthesize the evidence across studies? As we have seen, relative to more common alternatives the unweighted meta-analytic model almost certainly yields upwardly biased estimates of the mean effect size (i.e., effect size estimates that are larger than they should be) and implies a confidence interval that is almost always wider than would be produced under the more common fixed effect analytic model (i.e., the estimates are less precise than they should be). Both the unweighted model and the inverse variance weighted fixed effect model invoke a strong assumption—that the studies are highly similar to one another—and this assumption is probably not true in most syntheses. The random effects model provides a better conceptual fit than these, but statistically, when the number of studies is small, the estimate of the between-studies variance is poor. As such, the random effects model is statistically not a good choice for most syntheses, as they are currently conducted by evidence-based clearinghouses. For these reasons, none of the common meta-analytic models seem to be highly appropriate for evidence-based clearinghouses.
What, then, should researchers and evidence-based clearinghouses do, given that they have few studies to synthesize? Researchers and evidence-based clearinghouses could consider not synthesizing studies if there are too few to support a synthesis (Valentine et al., 2010). If waiting for more evidence is not an option, meta-analysis is the best option. It is a better option than narrative reviewing because it is more transparent and less reliant on the reviewer’s cognitive algebra. It is also better than vote counting and rule setting because these latter techniques have properties that make them even more objectionable than meta-analysis. Thus, we recommend meta-analysis not because it is ideal but because it is better than the alternatives. We recommend fixed effect meta-analysis because we believe it is a better option than the random effects model when there are few studies to synthesize.
However, a viable alternative is to consider the possibility of carrying out a Bayesian meta-analysis. As we will describe, a Bayesian meta-analysis can be carried out in a way that is analogous to inverse variance weighted fixed and random effects meta-analysis and has two major benefits relative to the more common meta-analytic methods. First, we have already noted that, conceptually, the random effects model is usually the best fit. Bayesian meta-analysis provides a better way of carrying out a random effects meta-analysis when the number of studies is small. The second benefit is that Bayesian statistics are easier for users to interpret accurately and lend themselves more naturally to decision support than the statistics that arise from a classical statistics framework.
Bayesian Statistics: An Introduction
Many readers will likely have encountered the term “Bayesian statistics” perhaps without understanding what it implies for data analysis. Briefly, classical statistics and Bayesian statistics conceptualize probability in fundamentally different ways. In classical statistics, the probability of an event is the proportion of times that the event occurs in a long series of situations in which it is given the opportunity to occur. For example, if we flip an equally weighted coin a very large number of times, we expect that heads will be on top half the time. An example of the complexities of this view can be seen in the proper, formal interpretation of a probability value arising from a null hypothesis significance test. Specifically, a p value can be formally defined this way: Assuming that the null hypothesis is true, the p value represents the probability of observing an effect at least as large as the one observed (with probability defined as the proportion of times that an effect at least as large as the observed effect would be obtained given a very large—really, infinite—number of samples).
The interpretation of a confidence interval in classical statistics relies on similar reasoning. A 95% confidence interval can be described this way: “Assume that a very large number of samples of size n are drawn, and a confidence interval is computed for each sample. Of these confidence intervals, 95% will contain the true population parameter.” Note that this is not equivalent to saying that there is a 95% chance that the parameter is within the confidence interval: The probability statement is about a very large number of intervals, not about the interval that was observed, and not about population parameter.
Most users of statistics find these correct definitions unwieldy and difficult to understand. In fact, misconceptions are common. Many users, for example, interpret a p value as the probability that the null hypothesis is true. But because the interpretation of the p value is conditional on the null hypothesis being true, we can’t then say that the null hypothesis is false when we collect the data and observe some small value of p. Tricky logic like this is one of the main reasons that misconceptions in the classical statistical framework are so common.
In classical statistics then, the locus of uncertainty (the probability) lies in the event (i.e., the event has a certain probability of occurring). Bayesian statistics start with a different view of probability, specifically that events are fixed (they either will or will not happen), and the locus of uncertainty is in the observer (e.g., the intervention is either effective or it is not, but we are uncertain about its effectiveness). In fact, Bayesian statistics can be thought of as a formal method for using the results of a study to update what we thought we knew about the effects of the intervention before we conducted that study. And happily, Bayesian statistics can be implemented in all of the major statistics software packages and are easier to understand than statistics arising from the classical framework. The Bayesian framework also lends itself more naturally to decision support. For example, a Bayesian analysis can provide an estimate of the probability that the intervention’s effects are greater than 0 (analogous to a null hypothesis significance test in classical statistics, but with an accurate interpretation that many users will be able to understand), or the probability that an effect is larger than some critical threshold. As an example of the latter, assume that a teen pregnancy program is operating in a context in which 15% of the female population experiences a pregnancy as a teen. A policy maker might believe that for a pregnancy prevention intervention to be worthwhile, the intervention’s effect must be greater than 4 percentage points (i.e., the intervention would need to reduce the teen pregnancy rate below 11%). A Bayesian analysis can support statements like “There is a 97% chance that the intervention is effective and a 74% chance that its effects are large enough to be important.” Classical statistics offers no framework for making statements like these.
Of course, as is true of life in general (where nothing is free), all of these good things come with a cost: The need to specify what is known about the effects of the intervention before conducting the study. In Bayesian statistics, this is known as a prior. Priors can be based on anything (including subjective beliefs), but most Bayesians advocate using formal data (e.g., a previous meta-analysis on a related question) or expert opinion (ideally elicited from a range of experts). Priors can be thought of as varying in terms of how informative they are, that is, how precise they are about the distribution and location of the parameters under study. Weakly informative priors suggest a position of prior ignorance of not knowing very much at all about the effects of a given intervention. At other times, a stronger prior might be appropriate. For example, if several good studies on the effects of a well-established program have been conducted, then the distribution of effects observed in those studies might be a good basis for quantifying what is known about the intervention prior to conducting the new study.
There is a long history of debates between classical statisticians and Bayesian statisticians. These debates do not center on the math involved but instead on the nature of probability and, in particular, on the proper interpretation of Bayesian results given the prior. That is, many classical statisticians worry that the results of a Bayesian analysis are dependent on the essentially arbitrary priors chosen. Because a Bayesian analysis can be thought of as a weighted synthesis between the prior and the actual data, in large data situations (e.g., a meta-analysis with several studies of reasonable size or a large randomized trial), this concern tends to be mitigated because the data will usually overwhelm the prior. In information-poor environments (e.g., a meta-analysis based on two studies), the prior has the potential to be more influential—in fact, very strong priors can overwhelm the data in these situations.
Options for selecting priors in public policy contexts
Because priors can exert a large influence on results in many public policy contexts, they need to be selected with care. In public policy contexts, we believe two strategies for choosing priors are most defensible. First, as articulated by Press (2003), it may be important to choose a prior that most people will find acceptable. Generally speaking, broad acceptability is more likely when the prior is not very informative. For example, the prior might specify that the intervention’s effects are likely to be distributed over a certain range. This distribution could be uniform over that range, suggesting that we know very little about the intervention’s effects or have some other form (e.g., they could be normally distributed, which would suggest that we know somewhat more about the effects of the intervention). Press refers to these as “public policy priors” because the fact that the inferences that arise from the analysis will rarely be strongly influenced by the prior, even in an information poor environment, makes them particularly useful for public policy problems.
When feasible, an alternative that should also gain broad acceptance is to use empirical information to form the prior. For example, between January 2010 and September 2014, the WWC released eight intervention reports on the effects of elementary school math interventions. The effect sizes in these studies ranged from a low of d = −.09 to a high of d = +.52. If a 95% confidence interval is placed around each study’s effect size, then this suggests that the smallest plausible value in this area is about −.23 and the largest plausible value is about +.76. As such, a relatively uninformative public policy prior suggesting that the effects of elementary school math interventions range from −.23 to +.76 seems both reasonable and defensible. Alternatively, if the effects of elementary math interventions are thought to be reasonably normally distributed, then the prior could reflect this and the width of the distribution could be estimated empirically. The main difference between a prior based on a uniform distribution of effects and a prior based on a normal distribution of effects is that in a uniform distribution, all effect sizes in the distribution are equally likely. In a normal distribution, effect sizes around the center of the distribution are thought to be more likely than effects in the tails. For the WWC’s elementary school math interventions, the variance of the distribution of effect sizes is about .033. The prior could be centered at 0 (which is advantageous as it makes the analysis closely analogous to a null hypothesis significance test in classical statistics) or at the center of the empirical distribution (e.g., the median effect size in this distribution is about d = +.15, so that could be used as the center). In this case, the difference in results generated by these two different centers will be negligible (because the variance is relatively large, suggesting that 99% of effect sizes should fall within a range of −.38 to +.71).
Illustrating the effects of priors
To illustrate the conditions under which the prior has a large effect on the conclusions arising from a meta-analysis, we chose three WWC intervention reports—one relatively large and two relatively small—and conducted random effects Bayesian meta-analysis using two different priors for the population mean μ: A relatively uninformative prior that assumes a uniform effect size over the range of WWC effect sizes observed in meta-analyses conducted to date and a somewhat informative prior that assumes a normal distribution with a variance based on the WWC’s existing meta-analyses. Random effects meta-analyses require an additional prior on the between-studies variance component τ2, and for this, we used a relatively uninformative prior that assumes a half normal distribution with a large variance (which was derived using all of the studies that the WWC included in meta-analyses conducted between January 2010 and September 2014).
Linked learning communities
The “large” meta-analysis was taken from a WWC intervention report on learning communities (U.S. Department of Education, 2014b). There were six studies that met inclusion criteria that measured the effects of learning communities on student credit accumulation; all six of these studies were randomized experiments. The sample sizes ranged from 1,071 to 1,534 with a total student sample size of 7,419. Standardized mean difference effect sizes ranged from −.02 to +.08. Using inverse variance weights, both fixed effect and random effects estimates suggested a mean effect size of +.025 with a standard error (SE) of .023 (p = .28).
With the results of classical inverse variance weighted fixed effect and random effects meta-analyses presented for reference, the results of the Bayesian meta-analyses are:
As can be seen, these results are not sensitive to the choice of the prior and are very close to what was obtained in the inverse variance weighted meta-analyses. These happy things occurred because the data—six relatively large randomized experiments—essentially swamped the prior. Unfortunately in public policy contexts, most analyses are not based on this much information, and as such the choice of the prior will often have the potential to be more consequential as illustrated below. It is worth noting that the standard errors for the Bayesian random effects meta-analyses were larger than their classical random effects counterpart. This occurred because the classical empirical estimate of the between-studies variance component was 0 (when the assumptions of the fixed effect model are met, the random effects model reduces to the fixed effect model). However, as discussed earlier, the estimate of the between-studies variance is poor when there are few studies, and in fact experts in meta-analysis assert that the choice between the fixed effect and the random effects model should be made conceptually, not empirically (in part because the empirical estimates can be poor; see Hedges & Vevea, 1998). The Bayesian framework influenced the estimate of the between-studies variance component by bringing in information from the WWC’s prior experiences, and this resulted in a somewhat more conservative estimate of the SE that was also more consistent with the underlying conceptualization of the random effects model.
Repeated reading
The first “small” meta-analysis is repeated reading (U.S. Department of Education, 2014a). Two studies met inclusion criteria; these assessed the effect of the intervention on reading comprehension. The sample sizes for the two studies were 16 and 62. Standardized mean difference effect sizes were +.28 and +.05. Using inverse variance weights, both fixed effect and random effects estimates suggested a mean effect size of +.097 with a SE of .227 (p = .67).
Using the same priors as in the previous example, the results of the Bayesian meta-analyses are:
In this example, the impact of the informativeness of the prior is easier to see. Using the somewhat informative prior, which assumed a normal distribution, the meta-analytic effects were “pulled” toward the center of that distribution (i.e., 0). The relatively uninformative prior, which assumed a uniform distribution, resulted in effects that were slightly larger and slightly more variable (as reflected in the increased standard error). Again, the standard errors from the Bayesian analyses were larger than the SE from the classical random effects analysis (in which the estimate of the between-studies variance was 0).
Doors to discovery
The second small meta-analysis, on the preschool literacy curriculum doors to discovery, included two studies that assessed the effects of the intervention on print knowledge (U.S. Department of Education, 2013). The sample sizes for the two studies were 37 and 365. Standardized mean difference effect sizes were +.69 and +.08. It is worth noting how different both the effect sizes and the sample sizes are and that the large effect is paired with a very small sample. In cases like this (i.e., very different effect sizes estimated with very different sample sizes), the random effects estimate will tend to converge on the unweighted estimate (here, about .39). Indeed, using inverse variance weights, the random effects meta-analysis suggested a mean effect size of d = .300 (SE = .293), p = .31.
Using the same priors as in the previous examples, the results of the Bayesian meta-analyses are:
Here, the estimate from the classical random effects meta-analysis is larger than both of the estimates from the Bayesian analyses. In essence, the priors from the Bayesian analyses moderated the effects of the large degree of observed heterogeneity, resulting in estimated effects that are somewhat closer to the fixed effect mean of d = +.10.
The effects of very strong priors
As a final demonstration of the impact of priors on small meta-analyses, assume that we have a very strong belief that the real effect size for doors to discovery is uniformly distributed between +.40 and +.60. In this case, the mean meta-analytic effect size would increase to d = +.43 with a very small standard error. Of course, this illustrates a point: In an information-poor environment, priors matter, and poorly chosen priors can have a tremendous influence on the final results. In this case, the prior is a poor and indefensible choice in part because the range of expected effect sizes is far too narrow. As such consumers of research that utilized a Bayesian approach need to attend to the priors that were used, the justification for those priors, and to any sensitivity analyses that were done illustrating the impact of the priors.
Conclusion
Several key points emerged from our review of evidence-based clearinghouses, which face a number of challenges to successfully fulfilling their missions. Chief among these are the difficulties inherent in assembling the evidence base, quality appraising the evidence, and synthesizing the evidence is a manner that permits valid and useful inferences. We discussed the importance of each of these aspects of systematic reviewing and highlighted some of the traps lying in wait for unsuspecting researchers. Based on our review, there is wide variation in how evidence-based clearinghouses approach their work. One aspect of work across clearinghouses that is consistent is the tendency to define research questions narrowly. This means that clearinghouses are often in an “evidence poor” environment (i.e., their syntheses have a small number of studies). This feature of clearinghouse syntheses has important implications for the methods used determine what the body of relevant studies reveals about the research question being asked. In particular, the most appropriate synthesis method (i.e., random effects meta-analysis) requires more studies than are usually available to clearinghouses. We recommend the Bayesian approach to statistics in general, and to meta-analysis in particular, as a partial solution to the problems associated with arriving at inferences regarding program effectiveness when there are few studies to synthesize. We do this by showing how clearinghouses can exploit two different types of priors (i.e., noninformative priors and priors based on similar studies) to generate more credible estimates of program effectiveness. The Bayesian approach has additional advantages of yielding statistics that are interpreted in a way that is similar to the way that most people incorrectly interpret classical statistics and supporting decision-making more effectively than classical statistics (e.g., by allowing researchers to estimate the probability that the effect is larger than some preestablished critical threshold). However, the need for and importance of priors in a Bayesian analysis means that care needs to be exercised in selecting them. Even with this caution in mind, we believe the Bayesian approach offers a promising way of addressing the problem that most evidence-based clearinghouses face, namely, that synthesis is difficult when there are few studies to review.
Footnotes
Authors’ Note
The views expressed are those of the authors and do not necessarily represent the positions or policies of the Institute of Education Sciences or the U.S. Department of Education.
Acknowledgment
Betsy Becker, Spyros Konstantopoulos, Bruce Sacerdote, and Chris Schmid provided helpful feedback on portions of this article.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The work reported herein was supported in part by the U.S. Department of Education’s Institute of Education Sciences (Contract ED–IES–12–C–0084).