Priming,Replication,and the Hardest Science

The Importance of Replication: What Does It Do for Us?

As a starting point, I take as a given that replication is an essential research activity for science. Although nearly every scientist agrees on this general point, few explicitly discuss why this is true. What does replication tell us about the nature of science, and how does it allow us to distinguish science from other means of knowing the world (such as religious approaches)?

Heidbreder (1933) argued that when we say “scientific knowledge is objective,” there is a very precise meaning at play for the terms objective and subjective. Objectivity here does not refer to metaphysical claims about what exists outside one’s own mind. When we say that physics yields objective knowledge, we are not making claims about the metaphysical status of the substances that physicists study. In point of fact, it is this disregard for the question of metaphysics that allowed physics and other natural sciences to advance as mature sciences. Instead, what we mean by objective knowledge is that such knowledge has the quality of social verification. As Heidbreder stated:

It is not the source of knowledge but its verification that determines its objectivity in science. The physicist does not pause at length to ask: Do these events which I observe exist only in my perception? He asks: Do they lead to communicable judgments based on repeated and verified experience? (p. 272)

Replication is important not for its own sake, but because it is a sign that the knowledge being generated will pass the social verification test. We know that physics generates objective knowledge precisely because we will both get the same answer if you measure the rate of the ball dropped from the tower and I measure it as well. I can observe your stopwatch and confirm its reading; we (and others in the field) agree that this is an appropriate measurement device to begin with. It is only through the process of doing the same thing and arriving at the same conclusion that we have a science that generates knowledge we call objective.

What Is Replication?

If replication is the key to objective knowledge, what exactly is replication and how does it play out in the area of priming? If replication is to serve the function of social verification of generated knowledge, then it must mean “doing the same thing” in terms of the methods used in an original investigation. On the surface, this appears quite straightforward, and in some cases it will be. If a researcher finds that responses on two questionnaires positively correlate in a sample of Michigan State University students, it is a mostly straightforward matter to replicate this study—that is, to do the same thing in terms of the methodology. I can distribute in the same way those same questionnaires to another sample recruited in the same way from this same population. If I repeat this sampling 100 times, one of two things will happen: (a) I generate a large proportion of nonzero positive correlations and get a more precise estimate of the true population effect size, or (b) I fail to generate a substantive number of positive correlations and we conclude that the original finding was likely Type I error. ²

If some feature of the methodology is changed, then the inference one can draw about a set of failed replications becomes ambiguous. That is, being able to infer that the original finding was Type I error depends absolutely on the methodology being the same as in the original finding. If and only if I did the same thing can I conclude that my replication failures indicate Type I error. If I change some important variable of the methodology, then all bets are off with respect to the inference I can draw about that failure. (As explicated in more detail below, a change in the population from which a researcher is sampling constitutes a change in methodology, in which case having a theory for predicting and explaining such differences becomes of central importance in interpreting a failure to replicate; see also Asendorpf et al., 2013; Henrich, Heine, & Norenzayan, 2010.)

Priming Methodologies and Replication

Critics of priming methodologies not only desire direct replications of priming effects (which is reasonable), they have also voiced the expectation that priming effects should not vary by any but the most obvious and salient situation or population characteristics. When replication failures occur, critics frequently conclude either that original effect was likely Type I error or that the effect in question is so sensitive as to be unimportant and capricious. But are these reasonable conclusions?

The first step in answering this question is to ask whether we should, in fact, expect any but the most basic cognitive and perceptual priming effects to be consistent across broad segments of populations and across variable experimental contexts. I argue that this expectation is entirely misguided and inconsistent with the predominant understanding of the brain as a computational organ that uses informational inputs to execute adaptive decision rules. To illustrate, consider the well-known priming effect obtained by Bargh, Chen, and Burrows (1996) that subliminally priming pictures of Black males results in more aggressive responses to provocation (relative to priming pictures of White males). What are the appropriate expectations for the replicability of this effect?

We know that there are some features of the original methodology that must be reproduced exactly for a replication attempt to be informative because we know that changing them would change the nature of the effect. So, for example, we would want to use exactly the same pictures of Black and White males as in the original, the same prime duration, and the same magnitude of provocation. But how do we know that these features are important and that changing them would change the original effect (thereby making the results of the replication attempt noninformative)? We know this only because we have relevant theories that tell us that these features should matter. With respect to the types of pictures used as primes, for instance, we know that people have different automatic responses to Blacks with high or low Afrocentric features or with dark or light skin (Blair, Judd, & Fallman, 2004; Hagiwara, Kashy, & Cesario, 2012; Ma & Correll, 2011), which would lead to different predicted effects if the pictures varied along these dimensions. Similarly, we know that prime duration influences the strength of a prime, and shorter prime durations might eliminate the effects of priming young Black males (Higgins, 1996).

The central problem with replications in priming research is that, at the current time, we do not yet have enough well-developed formal theories of priming that can pinpoint exactly what features and what effects should be important. Because of this, replication failures at this stage will necessarily be ambiguous because we cannot be sure that features that appear incidental to the researcher are not actually integral to obtaining the original effect. This is not a problem inherent to priming in general, or priming social behaviors, or priming motivation. It is simply a function of the relatively young state of priming research. ³

To illustrate this point, I have provided a few examples of some moderating variables that have been uncovered since Bargh and colleagues published the priming effect described earlier but that were probably unknown to researchers at the time. These include both individual difference variables and experimental variables; thus, changes to the population being sampled or changes to the experimental context would render replication failures ambiguous (yet still may provide meaningful information about underlying process.)

In terms of variables that might be specific to the effect of priming Black males on aggressive responses, my colleagues and I found that the degree to which an individual stereotypes Black males as dangerous moderates this effect (Cesario, Plaks, Hagiwara, Navarrete, & Higgins, 2010; see also Dijksterhuis, Aarts, Bargh, & van Knippenberg, 2000). This is based on our understanding that aggressive behavioral responses to the priming event in the Bargh et al. paradigm are the output of participants preparing to interact with a dangerous outgroup male (see Cesario, Plaks, & Higgins, 2006). To the extent that participants do not represent Black males as dangerous, then there would be no need to prepare and execute defensive threat behavior. In addition, one might expect that a participant’s physical size should also influence responses to the primes in this paradigm. Given that physically larger individuals can perform aggressive behaviors more effectively and incur lower costs doing so than would smaller individuals, it follows that one’s physical formidability may influence aggression in response to the provocation (Sell, Tooby, & Cosmides, 2009).

Besides such individual differences, one can also identify features of the experimental context that, if varied, could eliminate or even reverse priming effects. Recently, we found that the physical surroundings in priming experiments could influence participants’ responses to primes, given that different physical situations allow for different actions to be executed (Cesario et al., 2010). Drawing on the nonhuman animal literature on defensive threat behavior, we predicted and found that White participants who were seated in a physical space that restricted their ability to escape (a sound-resistant booth) showed increased accessibility of fight-related action semantics when primed with pictures of Black males. However, when primed with the same pictures while seated in a physical space that allowed distancing behavior (an open field), participants instead showed increased accessibility of flight-related action semantics. Similarly, we have found that priming participants while they are surrounded by ingroup members changes the nature of the response to various primes, given that being around one’s coalition allows behaviors to be executed that could not be performed easily by oneself (Cesario & Jonas, 2013; see also Fessler & Holbrook, 2013). Therefore collecting data individually versus in a group setting could eliminate or reverse a sample-wide average response to a prime.

In addition, the outcomes of replication attempts could be influenced by a range of variables now known to influence priming effects in general (not just the specific Black male priming effect described above): interdependent self-construals (rather than independent self-construals) lead to assimilative priming effects (Bry, Follenfant, & Meyer, 2008), low (rather than high) self-monitors are more likely to show priming effects (DeMarree, Wheeler, & Petty, 2005), the positivity associated with a prime influences responses (Custers & Aarts, 2005), associating a primed goal with ingroup or outgroup members can change responses to primes (Loersch, Aarts, Payne, & Jefferis, 2008), and the prime’s effect can change based on whether the prime is self- or other-generated (Mussweiler & Neumann, 2000; see also Mussweiler & Förster, 2000).

The point here is that if some critical feature of the experimental context has been changed, even if it is deemed irrelevant by the researcher, we would expect such a change to eliminate or reverse a previous priming effect (see also Katzko, 2006). Similarly, if a researcher is sampling from a population that differs markedly on some key individual difference variable from the population sampled by the original researcher, such as drawing from a population of chronically high or low self-monitors, then we would expect that the priming effect will not be obtained (or will even reverse).

As noted above, given the infancy of priming research we do not yet know all the relevant moderators and process-related variables; thus, there will be a host of variables that we fail to identify as important and that may be changed unwittingly by a replicating researcher. To the extent that, as noted earlier, “doing the same thing” is required to draw accurate inferences about replication failures, then not having a full accounting of the relevant variables will prevent meaningful conclusions when a different researcher fails to replicate a priming effect. ⁴

Is the Search for Invariant Effects Even Reasonable?

The argument that priming effects should vary considerably is not post hoc excuse making for failures to replicate. Variability should be expected because this is what available data would suggest with respect to important social and motivated behaviors. Varying responses (across individuals, across contexts) to an identical stimulus is the norm for both human and nonhuman animals. To provide just a few examples illustrating the range of behaviors for which this is true, we know this to be the case for defensive threat behavior in rodents (R. J. Blanchard, Flannelly, & Blanchard, 1986) and humans (D. C. Blanchard, Hynd, Minke, Minemoto, & Blanchard, 2001), hormonal effects on aggression and intimacy (Bos, Panksepp, Bluthé, & van Honk, 2012), effects of chronically active self-concept attributes (Brown & McConnell, 2009), and the effects of physical postures on power (Cesario & McDonald, 2013). With the exception of purely reflexive behaviors, it is difficult to find many examples in which behavior is executed inflexibly in response to a stimulus without regard to other information in the surrounding environment.

The sensitivity of priming effects to a range of individual difference and experimental context variables is consistent with current approaches to understanding the evolved functions of the mind. The widely held perspective from evolutionary psychology is that the mind is a computational organ designed to incorporate information from different sources to regulate behavior. Adaptive decision rules are followed using such information as inputs (see Tooby & Cosmides, 2005).

With respect to social interactions, the brain cannot successfully regulate the body to any reasonable degree if it does not take into account information about the self, target others, and the broader contexts in which interactions take place. In other words, the brain must operate on information beyond target stimulus features. In terms of priming an aggressive outgroup male, decisions about whether to execute costly aggressive behaviors and escalate a conflict involve a number of variables beyond simply the aggressiveness of the target other (i.e., whether the target is stereotyped as aggressive). Such decisions include, but are not limited to, a calculus of (a) one’s own physical formidability, (b) the formidability of the opponent, (c) external contingencies such as barriers to escape and distance to the threat, and (d) assessments of one’s own coalitional support relative to the coalitional support of the opponent (see, e.g., Benson-Amram, Heinen, Dryer, & Holekamp, 2011; Cesario & Navarrete, in press; Fessler & Holbrook, 2013; Grinnell, Packer, & Pusey, 1995; McComb, Packer, & Pusey, 1994; Wilson, Britton, & Franks, 2002; Wilson, Hauser, & Wrangham, 2001).

In other words, more than just the primed stimulus information is needed to predict behavior following priming and we should not expect responses to be broadly uniform, at least not when priming socially relevant stimuli. Critics calling for invariant priming effects are effectively arguing that only stimulus information should be important in regulating behavior. This expectation is inconsistent with broader understandings of behavioral regulation and evolutionary understandings of adaptive decision making.

Are Any Priming Effects Real?

So how is it that any one researcher could ever find an effect given what appears to be an overwhelming avalanche of potential variability? It is important to be cautious when answering this question because although there are reasonable answers, there are also unreasonable answers that devolve into excuse making for poor research practices by those of us in the field of priming. My suspicion is that, more often than not, a researcher is accidentally hitting on the right level of an important moderating variable, such as drawing from a population with a low enough mean level of self-monitoring as to obtain a priming effect, or placing participants into cubicles without thinking much of it, or having an experimenter who is low in perceived aggressiveness (allowing participants to respond to the provocation with aggression). And when researchers do not get the “right” combinations of variables, the failures end up in the file drawer. Indeed, this might be what is meant when researchers talk about having “insight” or “intuition” in conducting priming experiments in which they cannot verbalize why they made a decision but knew to make it. In the end, there is really nothing magical about this.

However, there is a danger in this explanation—it may be used to cover up poor research practices and prevent findings that actually are Type I error from ever being corrected. This short circuits the self-correcting nature of science, and it is essential to prevent this from happening. If “researcher degrees of freedom” and questionable research practices (Simmons et al., 2011) are allowing priming researchers to publish findings that are actually Type I error, then it would be a mistake to explain away any subsequent failures and to continue to defend such findings. Along these same lines, priming researchers cannot appeal endlessly to “unknown moderation” without doing the work to provide evidence for such moderation. At some point, the evidence may shift, and it would be more reasonable to conclude that the original effect was wrong or was so specific as to be rendered meaningless. Indeed, the combination of small sample sizes, very large effect sizes, and few direct replications should be a cause for concern. As priming researchers, we must be open to the revision of our beliefs about an effect, not just in theory but in actual practice. The recommendations offered below are designed in part to address these concerns.

Recommendations

At this stage in the development of priming research, it seems apparent to me that the most important step is that we first establish the basic replicability of a given priming effect: Can a single research lab replicate a reported effect multiple times? Only when we have a handle on an effect size of a given effect and we know that the obtained effect is not Type I error can we then move to having other labs attempt replications. At that point, subsequent failures can be understood as holding the promise of an interesting case of moderation. Absent that initial foundation, we will be caught in a perpetual cycle of failures and uncertainties.

Given this, it is the responsibility of priming researchers themselves to provide direct replications of an effect upon initial publication. Having surveyed the literature on behavioral and goal priming, the number of direct replications currently published does not instill a sense of confidence. ⁵ This is unacceptable for our field and we must do better.

As others have pointed out, replication by multiple independent labs is essential. However it is essential for a given purpose and at the appropriate time. The given purpose is to understand generalizability. The appropriate time is when we have a good enough theory to help us understand the moderating variables that might differ across labs and therefore introduce variability in the original effects. In my assessment, now is not the appropriate time and we cannot yet aim for the purpose of generalizability. This hesitation is due in part to the near-certainty that previous priming effects have been published under methodologies that we are now recognizing as being suboptimal (e.g., questionable research practices, inappropriate analytic techniques: see Judd, Westfall, & Kenny, 2012)—such practices increase Type I error. This point also suggests that priming researchers themselves have to be more conservative and resist making extreme claims about the widespread, highly important, and strong effects of priming. We do not yet know whether this is true, and only widespread direct replications across multiple labs will tell us this information.

It is also important to be clear about what conceptual replications (as opposed to direct replications) can and cannot tell us, as priming researchers have tended to overstep the limits that can be drawn when discussing conceptual replication. Although a conceptual replication can provide important information, it cannot give the kind of confidence in an effect that direct replication can (nor can it provide information about the effect size of the conceptually replicated effect).

One might be tempted to argue, as priming researchers have (e.g., Bargh, 2012), that a conceptual replication provides additional evidence for the underlying psychological process being tested and therefore instills confidence about other related results. For example, a manuscript may report a study showing that priming “elderly” causes participants to walk more slowly, followed by a study showing that priming “young Black male” causes participants to respond to provocation with greater aggression (Bargh et al., 1996). One might wish to say that the second case of conceptual replication provides good evidence for priming effects in general and decreases the likelihood that the first effect is Type I error. However this is true only if researchers have perfect research and reporting practices. Given the almost certain routine problems of our current practice (such as not being forthright about failed replications), conceptual replications without direct replications will only serve to perpetuate problems in the field. Suppose a manuscript reports three studies, the latter two of which are “conceptual replications” of the first. Perhaps the latter two show interesting moderation of the effect reported in the first study by two different individual difference variables. (This pattern will be familiar to readers of social psychology articles.) Three studies showing an initial effect and subsequently moderation of that effect may appear to be very strong evidence of the researcher’s claims. However, rather than being strong evidence, the same set of studies can be generated by including multiple individual difference measures in three separate studies and only reporting such measures when they produce a (false) rejection of the null hypothesis (Simmons et al., 2011). Only direct replication can guard against the use of questionable research practices and other “researcher degrees of freedom,” because it would be unlikely to obtain the moderation multiple times if it was simply Type I error. Therefore, it is inappropriate for priming researchers to point to conceptual replications of a given effect as evidence for the robustness, reliability, or generalizability of the priming effect in question.

Conclusion

As priming researchers, we must do better in our methodological practice and provide direct replications when publishing an effect. At the same time, the expectation of widespread invariance in priming effects is inappropriate, and critics of priming must stop drawing strong inferences about replication failures because we are not yet at a stage where we have the quality of theories that render such replications informative. Priming research will get there eventually, but complex social and motivational behaviors take time to understand. Although basic spreading activation effects in cognitive science have been studied for over 50 years, behavioral and goal priming is in its infancy and is, by its nature, more complex.

Theories inform us as to which variables are important and which are unimportant (i.e., which variables can be modified from one research study to the next without consequence). This is one of the reasons why psychology is the hardest science: social objects are variable in a way that inanimate objects are not (Meehl, 1978). This difficulty is amplified in priming research, given that we do not yet know all the relevant contingencies of the person and context (though of course, many of the problems of this area are shared throughout psychology and beyond; e.g., Gelman, in press).

The linguist Noam Chomsky recently stated:

Take, say, physics, which restricts itself to extremely simple questions. If a molecule becomes too complex, they hand it over to the chemists. If it becomes too complex for them, they hand it to biologists. And if the system is too complex for them, they hand it to psychologists. (Lawton, 2012)

We should not expect priming effects to be obtained “under all conditions, with all people,” a kind of invariance restricted to questions of classical physics. To expect such invariance would be to deny the complexity of human behavior. It is also inconsistent with what we know about the evolved workings of the mind, which center around computing information from different sources (including the physical context) and executing decision rules using this input. Therefore the requirement by some to find a priming paradigm that will work everywhere, every time, and with everyone is a requirement that is simply not reasonable.

There needs to be an appreciation by all parties that every study is merely one data point in the cumulative, ongoing practice of science. For priming researchers, this means to do our part to first establish the replicability of an effect and obtain a reasonable estimate of its effect size and to stop drawing sweeping conclusions about the ubiquity of priming effects before such conclusions are warranted. For critics, this means to give priming research time. The relevant theories are not yet there, but with better practice on the part of priming researchers, we will get there.