Stickier Priors: The Effects of Nonanalytic versus Analytic Thinking in New Product Forecasting

Abstract

The author investigates scenario generation and analogical reasoning as potential sources of bias in new product forecasting. In a series of studies, scenarios and analogies are shown to have persistent effects on judgment, despite subsequent use of corrective analytic techniques (e.g., counterfactual reasoning, counterscenarios, counteranalogies, decomposition, accountability). These findings demonstrate the robustness of nonanalytic processes on judgment and the need to be aware of their seductive effects.

Although most research on market forecasting has focused on developing quantitative models as decision aids, surveys indicate that judgmental forecasting is ubiquitous (Armstrong, Brodie, and McIntyre 1987; Dalrymple 1987). New product forecasting represents a particularly compelling context because the critical early stages of product development frequently lack the necessary data for model development. Moreover, reliance on a wide range of judgmental processes has produced disappointing results in terms of forecast accuracy.

In the best instances of judgmental forecasting, managers will consider all relevant and available evidence. Such behavior can be viewed as “analytic,” inasmuch as it reflects people's efforts to apply reasoning in a relatively objective and evenhanded manner. In contrast, forecasts that apply reasoning in a one-sided or subjective manner can be viewed as “nonanalytic.”¹ For example, a manager may recall a salient analogy to a previous experience that maps only partially onto the new situation or may envision a particular scenario of the future to the exclusion of alternative possibilities. In an extreme case, an optimistic manager may envision only success for a new venture and elaborate on the path to that success.

Nonanalytic or analytic thinking is useful nomenclature for this distinction (Alba and Hutchinson 1987; Sloman 1996).

Intuition suggests that a forecaster will be better served by engaging in analytic rather than nonanalytic thought. However, all forecasts have a starting point. Frequently, the starting point may involve casual or cursory consideration of the evidence. For example, it would not be unusual if prediction of product success or failure began with a particular mental scenario or an analogy to a previous product launch. The question addressed in the current research pertains to the case in which diligent managers engage in analytic thinking following nonanalytic thought. Specifically, to what extent can analytic thinking neutralize the biasing effects of preliminary intuitions about an outcome?

The focus of this research is on the persistence of preliminary beliefs despite subsequent use of corrective analytic techniques. The risks of belief perseverance should be self-evident. For example, “sticky priors” could contribute to managerial perseverance in a course of action despite evidence to the contrary (e.g., continuing to support a failing product; Boulding, Morgan, and Staelin 1997). However, are all priors sticky? Belief perseverance is a complex phenomenon. Although strong priors tend to persevere (Anderson, Lepper, and Ross 1980), many recency effects in judgment suggest that belief perseverance is not a forgone conclusion (Hoch 1984). Primacy and recency effects have been attributed to several factors, including strength of priors and evidence for updating (Anderson and Sechler 1986), judgment delays and memory (Hoch 1984), and amount of elaboration during judgment (Haugtvedt and Wegener 1994). This research investigates whether belief perseverance is a function of the type of elaboration (i.e., nonanalytic versus analytic thinking). In other words, are some priors stickier than others?

Two nonanalytic sources of priors used in business—scenarios and analogies—are compared with priors based on traditional analytic thinking. Systematic differences in persistence resulting from how priors are generated and updated constitute evidence of procedural bias because such processing differences are causally unrelated to actual outcomes. The fundamental proposition of the research is that scenarios and analogies create stickier priors than analytic thinking does (i.e., a procedural bias in favor of initial nonanalytic thinking). Scenarios and analogies are shown to have persistent effects on judgment despite subsequent corrective analytic thinking.

In the remainder of this article, a series of studies that provide support for the proposition that nonanalytic priors are stickier than analytic priors is described. The majority of the studies are controlled laboratory experiments that simulate real-world managerial decisions. In particular, the use of undergraduate students in these experiments may be scrutinized on the grounds of external validity, which usually arises from the belief that managers differ from students on important characteristics (e.g., accountability, expertise) that are believed by some to confer immunity to error. However, learning from new product experiences—as evidenced by new product failure rates—may be difficult for managers (e.g., Fischhoff 2001; Paich and Sterman 1993). Moreover, prior work suggests that managers are overconfident (e.g., Mahajan 1992) and not immune to the kinds of judgmental biases that have been primarily established through the study of undergraduate students in controlled experiments (e.g., Boulding, Morgan, and Staelin 1997; Tetlock 2000). Nonetheless, experimental findings should be generalized with caution to real-world managerial behavior. Anticipating this issue, Study 2 and Study 3A examine accountability and experience and find that neither appears to confer immunity to stickier priors. Evidence for this procedural bias is established by comparing scenario generation and reasoning by analogy with a benchmark of traditional analytic judgment. The first set of studies considers scenarios.

Scenarios

The role of scenarios in new product development and forecasting appears to be growing in acceptance, both anecdotally (Hamel and Prahalad 1994) and in the development of formal techniques (Ringland, Todd, and Schwartz 1998; Schoemaker 1993). For purposes of this research, a scenario is defined as a story that describes how events unfold over time and includes vivid details that create a mental picture or vision of the future. This definition is consistent both with related research on mental simulation (Kahneman and Tversky 1982) and with views that describe scenarios as “flowing narratives” that trace the progression of events (Schnaars and Topol 1987, p. 405), as stories about the future that are “both causal and concrete” (Kuhn and Sniezek 1996, p. 2), and especially as “script-like narratives that paint in vivid detail how the future might unfold in one or another direction” (Russo and Schoemaker 1992, p. 13). Thus, scenarios include aspects of imagination and use a story to link beliefs in support of a focal hypothesis.

Prior research has separately demonstrated the biasing effects of imagination (e.g., Carroll 1978; Gregory, Cialdini, and Carpenter 1982), explanation (Ross et al. 1977), event conjunction (Tversky and Kahneman 1983; see also Sloman 1996), and story construction (Pennington and Hastie 1992) on probability and confidence judgments. Each of these processes can be implicated in scenario generation and may be contrasted to the generation or processing of a set of discrete, evenhanded reasons that argue for and against the likelihood of a particular outcome. The hypothesis in the present research is not merely that scenarios can be biased, but that scenarios may resist debiasing if they are encoded as an elaborated and integrated schema in support of a focal hypothesis (i.e., the likely success or failure of a new product).

Study 1

The fundamental hypothesis regarding scenario-induced bias is investigated in the context of scenario analysis, which has been advanced as a tool that prompts people to consider alternative future scenarios and thereby improve judgment (Wack 1985). From a psychological perspective, the effectiveness of scenario analysis may be rooted in two processes: counterfactual reasoning (mediated by the availability of arguments; Hoch 1984) and multiple explanations (mediated by the ease of simulation of alternative outcomes; Hirt and Markman 1995; Koehler 1991, 1994). Prior psychological research and several studies of scenario analysis suggest that both processes improve judgment and may reduce overconfidence (Connolly and Dean 1997; Griffin, Dunning, and Ross 1990; Schoemaker 1993). However, several other studies are only weakly supportive, if at all (Byram 1997; Kuhn and Sniezek 1996; Schnaars and Topol 1987). Interpretation of these conflicting results is made difficult by variation in operationalizations. Some studies use quite elaborate scenarios, whereas others use simple reasons—a distinction that is formally recognized in the present experiments.

Perhaps the litmus test for the effectiveness of scenario analysis is whether it is possible to generate and evaluate multiple scenarios in an unbiased manner. Biases in the generation and evaluation process would undermine the effectiveness of scenario analysis. In the case of simple reasons, prior research has found order effects in the generation of discrete reasons for and against an outcome such that both reasons and predictions are biased toward the initial category of reasons generated (Hoch 1984). Thus, expectations of an outcome are stronger when reasons for its occurrence are generated before reasons for its nonoccurrence. However, this primacy effect is transitory. When a short delay is inserted between the generation of each set of reasons, reasons are evenhandedly generated, and predictions show a recency effect.

Inasmuch as scenario analysis incorporates pro versus con reasons into competing scenarios, similar results may be expected. However, the research on imagination and story construction described previously suggests that a different result is possible. After an elaborate and integrated schema has been developed in support of an initial scenario, it may be difficult to “undo” the scenario completely and consider an alternative.

H₁: Predictions will be biased in the direction of the initial scenario generated (i.e., a primacy effect). Generating counterscenarios will not completely debias judgment.

H₂: The primacy effect for scenarios will be stronger than the primacy effect for discrete reasons (i.e., priors based on a scenario will be stickier).

Method

Subjects and Design

Subjects were undergraduate students who participated for extra credit in an introductory marketing class. Each subject was randomly assigned to one of four cells in a 2 (type of elaboration: generate scenarios versus reasons) × 2 (order: success–failure/failure–success) × 2 (predictions: initial versus final) mixed design. A total of 92 subjects participated, and the number in each cell ranged from 21 to 25.

Materials and Procedure

The experimental materials were contained in a booklet that was distributed to each subject. The cover story asked subjects to assume the role of a business executive while thinking about the future of Internet shopping. Subjects were given some background information on the current status of Internet shopping and asked to give an initial opinion. Subjects in the scenario conditions then generated either a best-case or a worst-case scenario for the likely success of Internet shopping; subjects in the reasons conditions generated reasons for the success or failure of Internet shopping. Afterward, subjects were asked to generate a contrary scenario or set of reasons.

Each writing task was timed and lasted five minutes. For the scenario/success task, the title of the task was “Best-Case Scenario: Internet Shopping Is a Success.” Subjects were instructed as follows:

Please think about a best-case scenario in which Internet shopping will succeed. (Recall the characteristics of a scenario—it is a story that describes how events unfold over time and includes vivid details that create a mental picture or vision of the future.) Write a scenario that describes this success story for Internet shopping.

For the scenario/failure task, the title of the task was “Worst-Case Scenario: Internet Shopping Is a Failure.” Subjects were instructed as follows:

Please think about a worst-case scenario in which Internet shopping will not succeed. (Recall the characteristics of a scenario—it is a story that describes how events unfold over time and includes vivid details that create a mental picture or vision of the future.) Write a scenario that describes this story of failure for Internet shopping.

For the reasons/success task, the task was titled “Best-Case: Internet Shopping Is a Success,” and subjects were instructed as follows:

Please think about reasons why Internet shopping will succeed. List as many reasons as you can to support the success of Internet shopping.

For the reasons/failure task, the task was titled “Worst-Case: Internet Shopping Is a Failure,” and subjects were instructed as follows:

Please think about reasons why Internet shopping will not succeed. List as many reasons as you can to support the failure of Internet shopping.

After completing both scenario or reasons tasks, all subjects were asked a series of questions about the future of Internet shopping, including a revised opinion, market-share predictions over time, likelihood and confidence measures, recommendations for launch of online sales in various product and service categories, and predictions of attitudes of various population groups. Subjects were then presented with a set of evenhanded, experimenter-generated pro and con reasons that were depicted as another opinion about the future of Internet shopping (which may be “helpful or not in forming your professional opinion”). After processing these reasons, subjects repeated their opinion and market-share judgments.

Results and Discussion

The hypotheses predicted a two-way interaction between type of elaboration and order. Judgments were expected to be more biased toward an initial scenario than toward initial reasons generated. Results across all the dependent measures are consistent with the hypotheses.

Consider subjects' binary prediction of the future of Internet shopping (see Table 1 and Figure 1). A categorical analysis on the binary choice between success and failure indicates a significant two-way interaction of type and order of elaboration. Subjects predicted failure outcomes more often when their initial scenario supported failure or their second set of reasons supported failure (χ² = 4.40, p = .04). This pattern of results is supported by analyses of other dependent variables, including likelihood ratings for success and failure (F_{1, 87} = 8.13, p < .01, χ² = .08), launch recommendations for online sales in a variety of product and service categories (e.g., lawyer services, housing, furniture, hardware, clothing; F_{1, 88} = 3.86, p = .05, χ² = .03), confidence in launch (F_{1, 86} = 3.40, p = .07, χ² = .04), and predicted attitudes toward Internet shopping for various population groups (i.e., adults, teens, seniors, manufacturers, retailers, catalogers, and self; F_{1, 87} = 6.48, p = .01, χ² = .06).² Overall, predictions were more favorable toward Internet shopping when subjects generated a success scenario initially or when subjects generated reasons for success more recently. In other words, a counterscenario failed to eliminate the effect of an initial scenario; however, counterexplanation was effective after initial reasoning. The reasons control groups rule out fatigue and demand as alternative explanations for the results, whereas the primacy effect for scenarios supports H₁. Moreover, the differential order effect supports H₂: An initial scenario influenced judgment more than initial reasons, demonstrating stickier priors for scenarios than reasons.

Effect sizes are reported throughout, and they range from small (χ² = .01) to medium (χ² = .05) to large (χ² = .15) for behavioral research (Keppel 1991). Any attempt to interpret effect size must be viewed in light of the modest strength of the manipulation used to induce bias (scenarios and analogies may pervade a manager's thinking for much longer in the real world) and the relatively heavy-handed attempts to eliminate it (typically through multiple attempts to induce subjects to be more analytic). Nonetheless, scenario and analogy effects persist.

Figure 1

Study 1: Percentage of Subjects Predicting Success of Internet Shopping After Ordered Elaboration

Table 1

Study 1: Judgments After Ordered Elaboration for Internet Shopping (IS)

Type of Elaboration	Order	Number of Subjects Predicting Success	Likelihood that IS Succeeds	Likelihood that IS Fails	Predictions of Others' Attitudes (Averaged)
Reasons	Success–Failure	14/20 (70%)	4.76 (1.51)	3.48 (2.71)	4.28 (.72)
Reasons	Failure–Success	19/21 (90%)	5.48 (1.12)	2.71 (1.38)	4.68 (.74)
Scenarios	Success–Failure	22/25 (88%)	5.28 (1.14)	3.12 (1.20)	4.63 (.69)
Scenarios	Failure–Success	18/25 (72%)	4.63 (1.47)	3.92 (1.56)	4.15 (.96)

Type of Elaboration	Order	Change in Attitude (After Ordered Elaboration)	Change in Attitude (After Presented Evidence)	Average Change in Market-Share Estimates (Across Presented Evidence)
Reasons	Success–Failure	–.19 (1.25)	–.05 (1.02)	–1.4 (6.5)
Reasons	Failure–Success	.31 (.93)	.62 (1.20)	+.9 (9.8)
Scenarios	Success–Failure	.32 (.90)	.24 (.72)	–1.4 (3.6)
Scenarios	Failure–Success	.12 (1.42)	–.36 (1.55)	–4.1 (6.6)

The dependent measures that were repeated after presentation of the evenhanded, experimenter-generated reasons also show an interesting pattern. Recall that attitude measures were taken initially, after the scenario or reasons tasks and after the presentation of evenhanded evidence. These attitude judgments can be used to construct two attitude change measures to examine whether changes in attitude are induced by the elaboration tasks and by presentation of pro and con information. A multivariate analysis of variance (MANOVA) of both attitude change measures produced the expected two-way interaction of type of elaboration and order (F_{1, 88} = 4.97, p = .03, χ² = .04), showing a stronger primacy effect for scenarios than reasons. The interaction with repetition was not significant; however, the two-way interaction became directionally stronger, suggesting that processing of evenhanded reasons can actually exacerbate the order-driven bias rather than reduce it. Indeed, a similar analysis of the change in market-share estimates revealed a marginal two-way interaction of type of elaboration and order (p = .09), qualified by an interaction with success estimation period (p = .10). As shown in Table 1, market-share estimates shifted in the direction of the initial scenario or second set of reasons. This result suggests that presenting pro and con evidence may exacerbate rather than correct bias in judgment. Although far from conclusive, this latter finding merits additional research. At a minimum, there is no indication that presenting evenhanded evidence reduces the bias in judgment. Instead, the results are consistent with prior research that shows that nondeterminative evidence can exacerbate existing biases (for a review, see Kuhn and Lao 1996). Note that the biases in this experiment resulted from procedural differences in which subjects generated evidence (i.e., the order and type of elaboration) and are therefore non-normative.³

This research investigates the effects of scenario generation on judgment. I concur with the anonymous JMR reviewer who suggests that breaking down the generation task into creating the prior versus updating the prior would be a fruitful line for further inquiry, particularly from a Bayesian updating perspective.

Overall, the results of this study provide considerable support for order effects in judgment for scenarios and reasons. First, primacy order effects were stronger for scenarios than reasons: Generating an initial scenario influences judgment more than generating initial reasons. In other words, scenarios lead to stickier priors than reasons. Second, order effects endured despite counterfactual thinking; therefore, judgments remained biased in the direction of an initial scenario generated, even after the generation of a counterscenario. This susceptibility to order effects raises questions about the efficacy of scenario analysis, which relies on the generation of multiple alternative scenarios. Some evidence suggests that subsequent attempts to debias judgment through presentation of evenhanded evidence may have exacerbated the procedural bias.

Study 2

Study 1 shows that the biasing effects of an initial scenario are not eliminated by counterfactual thought. Study 2 examines a corrective procedure that is more motivational than cognitive in nature. A feature of managerial decision making that should enhance motivation and the overall quality of thought is accountability (i.e., expecting to have to justify one's opinions to others). Managers of new products in particular understand that their forecasts will come under scrutiny. Popular wisdom suggests that accountability should help managers avoid many judgmental biases that are demonstrated in a laboratory setting. Some research suggests that accountability improves judgment by inducing “more integratively complex” thinking (Tetlock 1983, 1985; Tetlock and Kim 1987, p. 700). However, accountability is not a panacea (Sanbonmatsu, Akimoto, and Biggs 1993; Simonson and Nye 1992; Tetlock and Boettger 1989). Lerner and Tetlock (1999) argue that the effects of accountability may vary as a function of the information that forms the basis for judgment. Although accountability should generally enhance analytic thinking by motivating people to consider multiple or opposing alternatives in anticipation of needing to justify their views, it may amplify bias that “results from naïve use of normatively (but not obviously) irrelevant cues” (Lerner and Tetlock 1999, p. 264). It could be argued that scenarios, which incorporate imagination and storytelling, include nonanalytic features or irrelevant cues that are self-generated and bias judgment. If so, accountability could exacerbate rather than reduce bias. Study 2 examines how this motivational intervention influences judgments that are generated from scenario-like processing.

Method

Subjects and Design

Subjects were undergraduate students who participated for extra credit in an introductory marketing class. The experiment was a 2 (scenario) × 2 (accountability) between-subjects design. The order of scenario and accountability manipulations (when both were present) was counterbalanced, for a total of five cells to which subjects were randomly assigned. A total of 190 subjects participated, and the number in each cell ranged from 25 to 33.

Materials and procedure

The experimental materials were contained in a booklet that was distributed to each subject. The cover story asked subjects to assume the role of a new product manager. Subjects were given a brief description of several new product ideas and asked for their “gut reaction” for screening purposes. Subjects were then instructed to consider the future of a target new product, the Electronic Book.

In the scenario conditions, subjects imagined and wrote a scenario describing the future success of the target new product. The scenario manipulation was incorporated into the task as follows:

This stage of the new product process asks you to imagine the idea's potential success…. In doing so, you may find that you need to fill in some details regarding the product, how it might be used, its features, and its appeal to customers. For example, what would it take for this new product to be successful? What would it be like to use this new product? What would the world be like if this new product were widely available and popular? Use your imagination to form a vision of the future success of the EBook.

Some subjects also received an accountability manipulation. Its wording was consistent with the role-playing task: “You may also be called upon to explain or justify your position to ‘senior management’ (in a brief interview with the experimenter at the end of this session).” In the scenario conditions, subjects received the accountability manipulation either before or after the scenario generation task (i.e., counterbalancing their order) or not at all. In the no-scenario conditions, subjects either received the accountability manipulation or not. (The no-scenario conditions serve as control groups to enable testing for differential accountability effects.)

All subjects then performed a feasibility assessment and generated and listed reasons for and against the success of the Electronic Book. Subjects then proceeded to the judgment task and made a series of predictions, including probability of success over time, market share over time, and the number of years until national new product launch and until E-Book ownership reaches one million consumers. At the end of the judgment task, subjects were asked to report whether they recalled the accountability manipulation and were also advised that interviews would occur only if time permitted at the end of the session (to maintain the fiction of accountability). The entire experimental procedure was self-paced.

Results

Subsequent analyses treat the data as a 2 (scenario) × 2 (accountability) design.⁴ The MANOVAs of predictions over time as a function of scenario generation and accountability revealed a two-way interaction for estimates of market share (F_{1, 145} = 3.52, p = .06, ω² = .02) and probability of success (F_{1, 124} = 3.31, p = .07, ω² = .02). This finding is central, because it indicates that the overall effect of accountability on judgment depends on whether subjects generated scenarios.⁵ The overall pattern of means is consistent: Without scenarios, accountability led to more conservative judgments; with scenario generation, accountability exacerbated optimism. This pattern is evident in the slopes of each of the lines or, more simply, the average prediction mean across all time periods (as shown in Table 2 and Figure 2). A similar pattern of results was also found on other dependent measures, including predictions for time-to-launch (directional only) and one-million ownership (F_{1, 147} = 5.10, p = .03, ω² = .03).

The lack of order effects due to counterbalancing (all not significant) justifies combining conditions in which subjects received the accountability manipulation before or after scenario generation. The accountability manipulation also had the predicted effect on responses to the manipulation check (χ² = 28.40, p < .01). To simplify the experiment, only positive priors were investigated (and reported in the main text); therefore, screening eliminated 38 subjects who indicated a negative gut reaction to electronic books. It was expected that the negative priors of these subjects would thwart their ability and motivation to generate scenarios of future success. Indeed, these subjects provided lower ratings of their ability to picture the new product (p = .05) and the amount of thought given to the task (p = .03). A follow-up analysis that included these subjects and reverse-coded their responses revealed consistent interaction patterns on market share (p = .06) and probability of success (p = .07) that support the study's findings. For the sake of completeness, note also that probability estimates that did not (logically) increase over time were omitted for 22 subjects. Key analyses conducted on other dependent variables with these subjects included were consistent.

Follow-up tests of the simple effects of accountability were also examined for each prediction variable. For probability of success estimates, accountability led to more optimistic judgments when scenarios were generated (F_{1, 125} = 4.48, p = .04); otherwise, accountability had no effect (F < 1). For market-share estimates, accountability led to marginally more conservative estimates when scenarios were not generated (F_{1, 145} = 2.65, p = .11); otherwise, accountability had no effect (F < 1).

Figure 2

Study 2: Average Market Share (A) and Probability of Success Estimates (B) as a Function of Scenario Generation and Accountability For Electronic Books

Table 2

Study 2: Judgments as a Function of Scenario Generation and Accountability For Electronic Books

Scenario	Accountability	N	Average Market Share	N	Average Probability of Success
No scenario	Not accountable	25	54.40 (11.75)	24	55.49 (11.00)
No scenario	Accountable	30	47.35 (17.64)	27	53.19 (17.04)
Scenario	Not accountable	32	49.13 (16.16)	26	49.15 (13.32)
Scenario	Accountable	62	52.50 (16.52)	52	56.56 (15.16)

Scenario	Accountability	N	Number of Years Until National Launch	Number of Years Until One-Million Ownership
No scenario	Not accountable	26	4.02 (2.96)	9.20 (5.43)
No scenario	Accountable	30	4.83 (4.13)	13.13 (10.30)
Scenario	Not accountable	33	4.41 (4.62)	11.76 (7.75)
Scenario	Accountable	62	3.99 (3.48)	9.75 (6.84)

Notes: N is the number of subjects in each condition (and may vary because of missing values).

Overall, these results suggest that accountability may not improve judgment. This may be viewed as a relatively conservative test, inasmuch as the scenario task was only a few minutes long and all subjects also generated evenhanded reasons before judgment. Consistent with Study 1, generating evenhanded reasons was insufficient to improve judgment after nonanalytic thinking. Worse still, accountability exacerbated a scenario-induced bias by increasing its resistance to evenhanded reasoning. It is possible that accountability motivated subjects to think harder, not smarter, to tell a better story. In doing so, generating a more elaborate scenario may have exacerbated scenario-induced bias, which is consistent with the notion of polarization (e.g., Millar and Tesser 1986). A follow-up study (not reported here for the sake of brevity) manipulated the amount of elaboration directly. Results show that predictions of success for electronic books polarize significantly as a function of scenario elaboration despite subsequent evenhanded reasoning (p = .03). In other words, scenario effects are not the result of superficial thinking, but instead reflect one-sided elaboration that is difficult to recalibrate.⁶

Consistent with prior literature on accountability and justification, this study examines process or procedural accountability and its effect on procedural bias. One JMR reviewer suggests that forecasters are held accountable for outcomes as well as process, and if so, outcome accountability might improve judgment. (Tetlock [2000] attributes differences in attitudes toward outcome and process accountability to managers' world views.) Although prior research suggests that neither process nor outcome accountability is sufficient to eliminate judgmental bias (Siegel-Jacobs and Yates 1996), other motivational mechanisms that might reduce onesidedness merit further investigation.

The accountability effects of Study 2 are particularly disconcerting inasmuch as accountability is generally thought to improve judgment by inducing evenhanded and analytic thought. In contrast, accountability appears to exacerbate one-sidedness in nonanalytic thinking (see also Sieck, Quinn, and Schooler 1999), which suggests that accountable decision makers will be particularly susceptible rather than immune to scenario-induced effects.

Taken together, the results of Studies 1 and 2 support the proposition that scenarios bias judgment and that scenario-induced bias is difficult to neutralize with analytic thinking. Scenarios led to stickier priors than reasons, and scenario effects persisted despite corrective action with various analytic tools (generation and presentation of evenhanded evidence, counterfactual thinking, and accountability). Further discussion of these findings is reserved until the end of the article when they can be considered in conjunction with a second source of nonanalytic influence on product forecasts, namely, analogical processing.

Analogies

An analogy is a comparison of two objects on certain dimensions, usually for the purpose of inference making. Reasoning by analogy about a target appears to consist of several steps: access or retrieval of a base analog, mapping of the base and target, inference making, and perhaps higher-order learning (Holyoak and Thagard 1997). Prior research on analogies has focused on factors that influence each step of reasoning by analogy. Evidence suggests that analogical retrieval and transfer is influenced by structural relations, surface details, problem context, and goals, though their relative importance is a matter of continued debate (for reviews, see Gregan-Paxton and Roedder John 1997; Reeves and Weisberg 1994). The influence of structural relations reflects causal reasoning and analytic thinking; the influence of surface details, similarity-based inferences, and other salient noncausal information reflects nonanalytic thinking.

Prior research has demonstrated analogy effects on judgment. For example, people were more likely to recommend direct military intervention in a conflict scenario when irrelevant details (e.g., refugees fleeing in boxcars versus small boats) cued an analogy to World War II versus Vietnam (Gilovich 1981). This result is notable because the irrelevant analogy biased judgment for both experts and novices, though some researchers speculate that these effects may be amplified for novices (Gregan-Paxton and Roedder John 1997). As Holyoak and Thagard (1995a, p. 132) state, “people … make use of superficially appealing analogies to highly familiar source analogs, which actually miss most of the important causal relations in the target.” Moreover, even relevant analogies based on causal relations may be over-applied, leading to emphasis on mapped dimensions of the problem at the expense of others (Gentner and Markman 1997) and driving one-sided elaboration and inferences (see Zaltman 1997). Sieck, Quinn, and Schooler (1999) provide some evidence for one-sidedness. Their subjects weighted surface similarity (rather than deep structure) more heavily when asked to justify their judgments.

In contrast to these views, other researchers suggest that people prefer analogical mappings that are rational and based more on deep structure than surface similarities. For example, people prefer to generate analogical mappings that honor basic coherence constraints (Spellman and Holyoak 1992) and to make inferences based on systematic correspondence between the target and the analogy (Markman 1997; see also Read and Cesa 1991). Thus, the fallibility of reasoning by analogy is subject to debate, and no research on analogy has tested whether its effects can be reduced or eliminated by analytic thinking.

Study 3

The first objective of this study is to test whether counterexplanation can neutralize the effects of preliminary reasoning by analogy. On the basis of the research cited previously, as well as the results of Study 1, it seems plausible that the one-sided and noncausal aspects of analogies may confer resistance to analytic reasoning. After a schema has been elaborated to support an initial analogy, it may be difficult to undo. If so, counterexplanation may be less effective when it follows a one-sided analogy than when it follows onesided reasons. In other words, an initial analogy may be more resistant to debiasing with counterexplanation than initial reasoning.

H₃: Predictions will be biased in the direction of an initial analogy (i.e., a primacy effect). Generating counterreasons will not debias judgment.

H₄: The primacy effect favoring an initial analogy will be stronger than the primacy effect favoring initial reasons (i.e., priors based on an analogy will be stickier).

The second objective of this study is to investigate the effectiveness of counteranalogies in neutralizing an initial analogy. Several researchers have speculated that the use of multiple analogies may improve judgment (Einhorn and Hogarth 1987; Holyoak and Thagard 1995b). An alternative analogy may break the conditional frame of reference established by the initial analogy (Koehler 1994) and reframe and highlight different aspects of the judgment task. Therefore, prompting people to consider multiple analogies may induce more evenhanded consideration of the relevant dimensions (a characteristic of analytic thinking). In addition, and consistent with the line of argument used in work on multiple explanation (Hirt and Markman 1995) and scenario analysis (Schoemaker 1993), consideration of alternative analogies may be sufficient to debias judgment. In contrast, when faced with multiple analogies, people may prefer the analogy that best fits the target or that best supports their goals (Spellman and Holyoak 1996). Such favoritism is evident in Holyoak and Simon's (1999) research on jury decision making, in which subjects' analogy ratings shifted in favor of their verdict in a case. Evenhanded consideration of analogies may be more difficult if previous reasoning by analogy and analogy-driven inference making serves to reinforce the initial analogy. Anecdotal evidence suggests that initial analogies can interfere with the acceptance of new analogies (for a discussion of analogies in science, see Holyoak and Thagard 1995b). After a schema has been elaborated in support of an initial analogy, it may be difficult to undo this schema completely and then evenhandedly consider an alternative analogy.

H₅: Predictions will be biased in the direction of the initial analogy considered (i.e., a primacy effect). Counteranalogies will not debias judgment.

Order effects constitute direct evidence of procedural bias and undermine the effectiveness of multiple analogies as a forecasting aid. Recall that, in a similar fashion, Study 1 examined the use of multiple scenarios. Although prior research has investigated scenario analysis, direct work on multiple analogies as a judgment tool is lacking.

Method

Subjects and Design

Subjects were undergraduate students who participated for extra credit in an introductory marketing class. Each subject was randomly assigned to one of six cells in a 3 (type of elaboration: analogy then counteranalogy, analogy then counterreasons, reasons then counterreasons) × 2 (order: success–failure/failure–success) between-subjects design. A total of 125 subjects participated, and the number in each cell ranged from 20 to 21.

Materials and Procedure

All experimental materials were contained in booklets distributed to each subject. The cover story asked subjects to assume the role of a marketing manager and think about the future of vitalia-based snack foods at SnackTime. (Vitalia is a fictitious product ingredient.) Subjects were given some background information as follows:

Imagine that you are a marketing manager. You worked for a number of years in the new product division at a major beverage company. This experience has helped you to land a new job as head of new products at a snack food company, SnackTime Inc. This large company makes a variety of snack foods, including chips, cookies, crackers, and so on. SnackTime's snack food products are very popular with consumers. SnackTime is facing an important decision concerning their products and has asked for your views as new product manager.

The Research and Development Division at SnackTime has developed a new product, called vitalia(tm). When vitalia is added to the snack food formula during the manufacturing process, snack food flavors change and become stronger. In blind taste tests, consumers slightly preferred vitalia-formula snacks to regular-formula snacks. The research department is proposing that SnackTime switch to a vitalia formula for all of its snack foods.

Subjects then considered reasons for and against the success of vitalia-formula snack foods. Some subjects also received an analogy manipulation (indicated in parentheses in the following instructions). In their instructions, subjects were told to think about success as follows:

Think about reasons why switching to vitalia-formula snack foods could be a success for SnackTime. (As you think about reasons for success, you may find it helpful to consider Coca-Cola's experience with the success of Diet Coke.) List as many possible reasons as you can why SnackTime should switch to vitalia-formula snack foods.

Subjects were asked to think about failure with the following instructions:

Think about reasons why switching to vitalia-formula snack foods could be a failure for SnackTime. (As you think about reasons for failure, you may find it helpful to consider Coca-Cola's experience with the failure of New Coke.) List as many possible reasons as you can why SnackTime should not switch to vitalia-formula snack foods.

Both writing tasks were timed and lasted three minutes. Between each task, subjects engaged in a three-minute dis-tractor task. Subjects received the writing tasks in either order (i.e., success-failure or failure-success). Some subjects received no analogies (i.e., reasons then counterreasons), some subjects received the initial analogy only (i.e., analogy then counterreasons), and some subjects received both analogies (i.e., analogy then counteranalogy). Subjects then answered a series of questions about their recommendation to switch to vitalia-formula snack foods, confidence in this recommendation, opinion toward vitalia-formula snack foods, and estimates of the percentage of people consuming this new product over time.

Results

Results across the various dependent measures are consistent with the hypotheses. An analysis of the 2 (analogy versus reasons) × 2 (order) design was expected to reveal a significant two-way interaction, with a stronger primacy effect in the analogy-then-counterreasons cells than in the reasons-then-counterreasons cells. In addition, H₅ predicts a primacy order effect in the analogy-then-counteranalogy cells. No prediction is made regarding its strength relative to the other order effects at this time. Note also that an order effect in considering multiple analogies (as well as the reasons control groups) helps rule out demand as an alternative explanation for these results.

Specifically, subjects were asked to rate the future of vitalia-formula products at SnackTime on four seven-point scales: “unlikely/likely to succeed,” “will sell very poorly/very well,” “not a good idea/a very good idea,” and “do not/definitely recommend launch.” A MANOVA of attitude ratings revealed a two-way interaction (F_{1, 117} = 6.27, p = .01, ω² = .04): The primacy order effect was stronger in the analogy-then-counterreasons cells (F_{1, 117} = 15.56, p < .01) than in the reasons-then-counterreasons cells (F < 1). In other words, subjects were more optimistic (pessimistic) when considering an initial positive (negative) analogy. Counterreasons failed to eliminate completely the bias induced by an initial analogy, though it did eliminate the effect of initial reasons generated. Thus, H₃ and H₄ are supported. Moreover, there was an overall primacy effect in favor of the initial analogy when subjects considered both analogies (F_{1, 117} = 4.48, p = .04, ω² = .03), in support of H₅. That is, a counteranalogy failed to eliminate the effect of an initial analogy. This pattern of means is depicted in Table 3 and Figure 3 and is supported by analyses of other dependent variables (omitted for the sake of brevity).

Figure 3

Study 3: Average Attitude Ratings as a Function of Type of Reasoning and Order

Table 3

Study 3: Judgments for a New Food product as a Function of Reasoning Order and Analogy Cues

Type of Reasoning and Order
First Reasoning Task	Second Reasoning Task	Switch (Yes)	Confidence in Switching	N	Attitude Ratings (Average)	N	Percentage of Regular Users (Average)
Analogy + Pros	Analogy + Cons	19/21	72.14 (13.03)	21	4.83 (1.26)	21	34.70 (18.83)
Analogy + Cons	Analogy + Pros	10/21	50.00 (25.10)	21	4.10 (1.42)	21	28.77 (17.53)
Analogy + Pros	Cons	19/21	68.81 (18.23)	20	5.00 (.86)	21	42.25 (12.54)
Analogy + Cons	Pros	8/21	40.71 (23.15)	21	3.61 (1.00)	21	29.53 (13.89)
Pros	Cons	18/21	64.76 (18.54)	21	4.86 (.97)	21	42.79 (20.40)
Cons	Pros	15/20	64.00 (22.98)	19	4.72 (1.16)	20	34.03 (19.05)

Notes: N is the number of subjects in each condition (and may vary because of missing values).

Overall, the results of this study provide considerable support for the notion that there are order effects in judgment for analogies and reasons. First, primacy order effects for analogies appear to be stronger than for reasons. An analogy leads to a stickier prior. That is, the effect of an initial analogy is more persistent than the effect of initial reasons. Second, order effects endure despite counterfactual thinking. In contrast to prior research by Hoch (1984) on explanation and counterexplanation, counterfactual reasoning failed to eliminate the effect of an initial analogy. In addition, a counteranalogy failed to eliminate the effect of an initial analogy. Moreover, the consistency between this study and Study 1 supports the proposition that nonanalytic thinking has persistent effects on judgment that are difficult to eliminate with subsequent analytic or nonanalytic counterfactual thinking.

Study 3A

The previous study demonstrates that counteranalogies and counterexplanation are insufficient to neutralize the effect of an initial analogy. Study 3A is designed to address the use of multiple analogies in a relatively conservative test environment. First, subjects were experienced in the prediction task. Second, the use of a simulation exercise enabled the provision of additional relevant information (market research data), which should reduce the impact of less relevant information (e.g., analogies) on judgment. Third, the analogies were designed to be irrelevant, which should make it easier for subjects to discount or argue effectively against the analogies. Fourth, the analogies were again presented to subjects, which eliminated availability biases that could inhibit the generation of alternative analogies. Finally, the analogies did not include any explicit numerical information, and thereby anchoring was reduced as an alternative explanation for the results. As in Study 3, the intent of Study 3A is to demonstrate that the schema developed under an initial analogy is difficult to undo, which in turn undermines evenhanded consideration of multiple analogies. Thus, it provides another test of hypothesis H₅, which argues that judgments will be biased in the direction of an initial analogy (i.e., a primacy effect) and that consideration of an alternative analogy will not debias judgment.

Method

Subjects and Design

Subjects were MBA, Executive MBA, and undergraduate students who participated as part of a class exercise. Each subject was randomly assigned to one of two cells in a 2 (analogy: positive/negative) × 2 (predictions: after initial analogy and after counteranalogy) mixed design. A total of 162 subjects participated; of these subjects, 21 failed to complete the exercise and were eliminated from the analysis. The remaining 141 subjects consisted of 40 MBAs, 27 Executive MBAs, 65 undergraduate students, and 9 others who were drawn from various universities.

Materials and Procedure

Subjects worked through the materials individually and at their own pace. The study took place within the context of the BrandMaps simulation game (Chapman 1997). All subjects were experienced players. In BrandMaps, firms compete against each other in a “vaporware industry” (i.e., a fictitious product category used by the simulation game). Firms can market a varying number of brands in each region, and participants are responsible for the launch, development, and marketing of their firm's brands. Each region, firm, and brand has a numerical designation (e.g., Firm 2–1 refers to Firm 2 and its Brand 1). Firms and regions are typically named in BrandMaps (though brands are not).

In the BrandMaps exercise, subjects were given background information about the competitors in a target region and their own firm's position in the region, as follows:

Imagine that you are part of the management team for “Coca-Cola Vaporware” (Firm 1). You have just joined the firm and have been asked for your assessment of Coca-Cola Vaporware's position and future in the U.S. (Region 1). Summary statistics for Quarter 2 in the U.S. region are shown below. Briefly, you can see that Coca-Cola Vaporware faces 2 main competitors in its industry: “Pepsi-Cola Vaporware” (Firm 2) and “No-Name Vaporware” (Firm 3). Each of these competitor firms has one active brand in the U.S. Coca-Cola Vaporware also has one active Brand 1–1 in the U.S., its flagship “Classic Coke” brand. Classic Coke is the leading brand in terms of earnings and ROI [return on investment] in the U.S.

Subjects were told that Coca-Cola Vaporware is considering the launch of a new brand as part of a project. The project name served as the manipulation of the initial analogy, either to an irrelevant past product failure or success. In the success analogy condition, subjects were told the following:

Coca-Cola Vaporware has a project underway, code-named “Diet Coke,” to assess the feasibility of launching a second Brand 1–2 in the U.S. Marketing research regarding the Diet Coke project—including competitor and customer information in Region 1, as well as specific research concerning Diet Coke—is attached (see Table of Contents).

In the failure analogy condition, the same instructions were provided with the label “New Coke” instead of “Diet Coke.”

In both analogy conditions, subjects were given detailed information about the firm's marketing plan for the new brand (e.g., its formulation, marketing mix). Subjects also received the same firm summary statistics and marketing research (e.g., financial reports, concept and preference tests for the new brand, test market data, other customer and competitor information). Subjects were asked for a launch recommendation for the new brand, their confidence in this recommendation, and their predictions of market shares for the firms in the next quarter if the brand were launched.

Subjects were then asked to “briefly describe what you know about real-world Coca-Cola's experience with Diet Coke and New Coke in the soft drink market.” After doing so, subjects were prompted to consider both the analogy and the counteranalogy in an evenhanded manner as follows:

You may or may not find real-world Coca-Cola's experience with New Coke and/or Diet Coke in the soft drink market useful in assessing Coca-Cola Vaporware's position in the BrandMaps vaporware market. Now that you have had a chance to give the matter some more thought, please answer the following questions related to the project launch described earlier.

Subjects then repeated their judgments for launch recommendation, confidence, and market-share predictions.

Results

The pattern of results is similar across all subject populations and support H₅. Initial predictions were biased by the analogy to Diet Coke versus New Coke. Recommendations for launch were biased in the direction of the initial analogy (χ² = 4.23, p = .04). Similarly, confidence in launch of the new product was higher for subjects working on the Diet Coke versus New Coke project (F_{1, 138} = 4.02, p = .05, ω² = .02). In addition, market-share predictions were significantly different for the existing brand, Classic Coke, in the market (F_{1, 131} = 3.89, p = .05, ω² = .02) and for the new brand if launched (F_{1, 131} = 3.67, p = .06, ω² = .02). The differences appear to reflect the real-world reasoning that New Coke, compared with Diet Coke, may cannibalize sales of Classic Coke. This is evident in the pattern of the means in Table 4, and it was also mentioned in subjects' cognitive responses.

Table 4

Study 3A: New Product Launch Decisions and Forecasts after an Initial Positive/Negative Analogy and After a Counteranalogy

Initial Analogy	Launch? (Yes)	Confidence in Go Decision	Market Share (Percentage) New/Diet Coke	Market Share (Percentage) Classic Coke
True values	—	—	8.5	25.4
		Initial Predictions (Given an Analogy)
Diet Coke (+)	67/74 (91%)	75.7 (19.4)	20.9 (14.0)	35.0 (13.8)
New Coke (–)	52/67 (78%)	67.9 (26.4)	26.1 (17.1)	30.2 (14.4)
		Final Predictions (Given both Analogies)
Diet Coke (+)	66/74 (89%)	75.6 (21.0)	18.8 (12.5)	35.9 (14.1)
New Coke (–)	37/67 (55%)	49.4 (30.0)	20.1 (14.1)	31.5 (13.0)
	Change in Percentage Launch	Change in Confidence	Change in Market Share (Percentage) New/Diet Coke	Change in Market Share (Percentage) Classic Coke
Diet Coke (+)	–2%	–.1 (18.0)	–1.6 (7.2)	+1.3 (4.8)
New Coke (–)	–23%	–18.4 (26.8)	–6.1 (16.5)	+1.0 (7.8)

After subjects were prompted to think about both New Coke and Diet Coke, their repeated predictions continued to be biased by the initial analogy. The effect of the analogy on launch recommendations and confidence remained significant (respectively χ² = 18.01, p < .01; F_{1, 138} = 36.16, p < .01, ω² = .20). Market-share predictions for Classic Coke also continued to marginally differ by condition (F_{1, 120} = 3.31, p = .07, ω² = .02); the difference for New versus Diet Coke was nonsignificant (F < 1). Overall, these results suggest that prompting consideration of both analogies did not eliminate the effects of the initial analogy.⁷

Some subjects also responded to an attitudinal rating scale after making both sets of predictions (because of session constraints, not all subjects received these questions). Subjects reported more favorable attitudes toward the new product in Diet Coke versus New Coke conditions (F_{1, 68} = 12.14, p < .01). This result is consistent with analysis on the repeated measures and helps rule out explicit anchoring on previous judgments as an alternative explanation for the findings.

Another view of the data considers the change in responses when subjects were asked to consider both analogies. As previously shown, the initial analogy biases both initial and final judgments, so this attempt at debiasing was ineffective overall. However, the question still remains regarding whether the attempt at debiasing had at least some effect on improving subsequent judgment. The answer appears to be negative. An analysis of variance (ANOVA) of the change in confidence toward the new product launch indicates a main effect of the initial analogy (F_{1, 138} = 22.96, p < .01, ω² = .11). Instructions to subjects to think about both analogies actually exacerbate the preexisting bias, presumably by prompting additional analogy-driven reasoning beyond a mere anchoring effect. An ANOVA of the change in market-share predictions indicates that Classic Coke forecasts did not change as a function of the initial analogy (F < 1); the biasing effect of the initial analogy endures. However, New Coke predictions dropped more than Diet Coke predictions (F_{1, 120} = 3.80, p = .05, ω² = .02). This analysis suggests that any initial difference in predictions for New Coke versus Diet Coke was exacerbated after thinking about both analogies.

Overall, the data suggest that the initial analogy continued to drive reasoning, even for experienced decision makers in an environment rich in analytic information. Confidence in the launch decision dropped for New Coke but did not change for Diet Coke. Market share dropped for New Coke, but not for Diet Coke; there was no change in predictions for Classic Coke. These changes seem to reflect reasoning by analogy to the real-world experience of Coca-Cola, which suggests that New Coke will fail but Diet Coke will succeed. The major impact is from the analogy to New Coke, suggesting that spontaneous consideration of failure is otherwise unlikely, at least in this exercise. Indeed, as the true values for market share indicate in Table 4, all subjects were optimistic and overestimated market share for the new brand if launched.⁸ Taken together, the results of Studies 3 and 3A demonstrate that analogies create stickier priors than reasons and that even experienced decision makers are unable to use multiple analogies evenhandedly. Study 4 investigates whether decomposition neutralizes the effects of an initial analogy.

The notion that prompting people to consider failure analogies particularly improves judgment is intriguing and merits further investigation. However, New Coke's greater impact must be generalized with caution because a ceiling effect could limit Diet Coke's influence on judgment. Indeed, the overall optimism found in this experiment is itself a calibration issue, because outcome bias (the deviation of forecasts from actual outcomes) also depends on the favorability of information provided to subjects. In the experiment, subjects were provided with standard marketing research from the BrandMaps simulation environment, and their overoptimism appears to be consistent with what is known about managerial overconfidence and new product failure rates. Nonetheless, these calibration issues illustrate why procedural bias (i.e., comparisons of nonanalytic judgments to a benchmark of traditional analytic thinking) rather than outcome bias has been emphasized in this research.

Study 4

Research on decomposition suggests that a “divide-and-conquer” approach to problem solving can lead to improved judgment (for a recent review, see MacGregor 2001). In a typical multiplicative decomposition, an algorithm is used to break down a target quantity into subcomponents that, when multiplied together, provide an estimate of the target quantity. A decompositional algorithm may be viewed as a simplified representation of a quantitative forecasting model used as an aid to judgment. The main benefit of decompositional algorithms may arise from encouraging people to apply analytical reasoning to the prediction task. Experimenter-provided algorithms improve accuracy because they cue relevant information and provide a structure for analyzing the problem (MacGregor and Lichtenstein 1991). Thus, decompositional algorithms should prompt data-driven, analytic judgment. However, prior research has not investigated whether decomposition is equally effective at improving judgment that is already biased by nonanalytic thinking. Nonanalytic thinking that creates prior beliefs may induce top-down processing that reduces the benefits of decomposition by creating systematic bias in the decomposed estimates, which could carry over to the recombined estimates as well. A pilot study (available from the author) suggests that decomposition is ineffective following initial reasoning by analogy. The present study manipulates top-down and bottom-up processing directly and makes the following prediction.

H₆: Decomposition will fail to neutralize judgment that is already biased by an initial analogy (i.e., top-down processing); otherwise, decomposition will reduce reliance on an initial analogy (i.e., bottom-up processing).

Method

Subjects and Design

Subjects were undergraduate students who participated for extra credit in an introductory marketing class. Each subject was randomly assigned to one of eight cells in a 2 (analogy: to a product with high versus low sales) × 2 (processing: bottom-up versus top-down) × 2 (product name: similar versus dissimilar) between-subjects design. A total of 92 subjects participated, and the number in each cell ranged from 10 to 12.

Materials and Procedure

Subjects worked through the survey booklet at their own pace. The instructions asked subjects to assume the role of a marketing manager and briefly described the firm and its first product (shown here in the low-analogy condition).

We would like you to imagine that you are the new marketing manager for HomeAlert, a manufacturer of safety equipment targeted at the 60 million Americans who own their own home. HomeAlert began as a one-product company in 1993, manufacturing RadonAlert (a kit that enables homeowners to test for unsafe levels of the dangerous gas, radon, in their homes). RadonAlert rapidly penetrated the marketplace, and has achieved 1.2 million unit sales to date for a 2% market share.

In the high-analogy condition, RadonAlert achieved 30 million unit sales and 50% market share.

A new product idea in an entirely different product category was then described (shown here in the similar-name condition).

ChildAlert is a security system designed to monitor the whereabouts of children playing outside the home. The system consists of a small bracelet (which is worn by the child), and a monitoring unit located in the home. The monitoring unit sends out a radio-frequency signal, which detects the presence of the child's bracelet and buzzes if the child moves more than a pre-set distance away from the house. Parents set the distance themselves (e.g., based on size of yard, or length of street), and the system also buzzes if the bracelet is removed without authorization. Note that technical limitations restrict the use of the product to buildings no greater than 3 stories tall, thus excluding most apartment buildings. The target market for this product is homeowners with children.

In the dissimilar-name condition, the product was called ChildMinder.

In the top-down processing condition, subjects first gave an initial estimate of market share and unit sales of this product in the first three years before proceeding to the decomposition task. In the bottom-up condition, subjects proceeded directly to the decomposition task. The decomposition task prompted subjects to make the following component estimates:

•

Percentage of homeowners with children,

•

Percentage of homeowners with children of the appropriate age group,

•

Percentage of parents concerned about their child's whereabouts,

•

Percentage of parents who would consider a monitoring system,

•

Percentage of parents who would learn of the existence of the ChildMinder,

•

Percentage probability that competitors will offer a similar product in the next three years,

•

Percentage of consumers who would prefer ChildMinder to similar competing products, and

•

Percentage of stores where these consumers shop who agree to carry ChildMinder.

All subjects then provided final estimates of market share and unit sales in the first three years. Subjects also estimated the percentage probability that the product will succeed, made a recommendation to launch, rated their confidence, and indicated the basis for their decision.

Results

Subsequent analyses focus on the two-way interaction between analogy and processing on the dependent measures. Although CATMOD and LOGIT indicated that this interaction was not significant for the binary decision to launch variable, ANOVAs on the continuous dependent measures support the predicted interaction for final market share (F_{1, 79} = 2.99, p = .09, ω² = .02) and unit sales (F_{1, 71} = 5.82, p = .02) and directionally for probability of success in both conditions (dissimilar: F_{1, 83} = 2.42, p = .13, ω² = .02; similar: F_{1, 83} = 2.33, p = .13, ω² = .01).⁹ As predicted, the analogy had no effect in the bottom-up processing condition (all simple effects tests were nonsignificant) but had a significant effect in the top-down processing conditions for market share (F_{1, 38} = 7.45, p < .01, ω² = .06) and unit sales (F_{1, 38} = 8.71, p < .01) and probability of success in the dissimilar-names condition (F_{1, 21} = 12.32, p < .01, ω² = .33) and similar-names condition (F_{1, 19} = 7.34, p = .01, ω² = .23). These results for probability of success, sales, and market-share forecasts are shown in Table 5 and Figure 4 and support hypothesis H₆. Thus, in between-subjects comparisons, it is evident that subjects who provided an initial estimate were susceptible to reasoning by analogy and that analogy effects were not neutralized by subsequent task decomposition. In contrast, subjects who did not provide an initial estimate but instead proceeded directly to a decomposition task did not appear susceptible to analogous product information. These results are consistent with a top-down versus a bottom-up processing view.

The original rationale for this study predicted a three-way interaction between name, analogy, and processing on predictions. An analogous product with a similar versus dissimilar name (i.e., a manipulation of surface similarity) was expected to be particularly influential on top-down judgments. The ANOVAs indicated that this three-way interaction was not significant for any of the dependent measures, with the exception of probability of success (F_{1, 83} = 4.75, p = .03). Thus, the analysis focuses on the two-way interaction of analogy and processing. There is an interaction of analogy and name on probability of success (F_{1, 83} = 17.68, p < .01), market share (F_{1, 79} = 3.97, p = .05), and unit sales (F_{1, 71} = 2.71, p =.10), which is consistent with the notion that analogy effects are enhanced when names are similar. Indeed, subjects rated the analogy as more important when the product names were similar than dissimilar (F_{1, 81} = 5.56, p = .02, ω² = .05). Taken together, these effects suggest that subjects are also susceptible to irrelevant similarities such as product name; this merits further investigation.

For the sake of completeness, note that there were also effects of analogy (F_{1, 79} = 3.31, p = .07) on market share, and analogy (F_{1, 71} = 7.73, p < .01) and process (F_{1, 71} = 3.65, p = .06) on unit sales. Note that analysis of unit sales must be interpreted with caution because of variance heterogeneity. An alternative analysis with sales estimates categorized as high or low yielded similar results.

Figure 4

Study 4: Market Share (A) and Unit Sales (B) as a Function of Analogy and Processing

Table 5

Study 4: Predictions of Success, Market Share, and Sales as a Function of Analogy, Processing, and Name

Name	Analogy	Processing	N	Probability of Success
Similar	Low	Top-down	11	43.64 (21.22)
Similar	Low	Bottom-up	11	55.00 (27.02)
Similar	High	Top-down	10	72.00 (26.69)
Similar	High	Bottom-up	12	63.33 (18.38)
Dissimilar	Low	Top-down	11	75.00 (16.43)
Dissimilar	Low	Bottom-up	12	57.25 (20.99)
Dissimilar	High	Top-down	12	45.08 (23.46)
Dissimilar	High	Bottom-up	12	47.08 (18.27)

Analogy	Processing	N	Final Market Share (Percentage)	N	Final Sales (Millions)	N	Initial Market Share (Percentage)	N	Initial Sales (Millions)
High	Top-Down	21	44.24 (32.19)	22	17.15 (23.70)	20	44.86 (33.57)	21	16.37 (22.76)0
High	Bottom-up	22	37.86 (32.43)	17	4.94 (7.21)		—		—
Low	Top-Down	21	21.45 (24.31)	20	2.29 (3.05)	22	20.33 (27.13)	22	1.19 (1.38)
Low	Bottom-up	23	37.17 (33.65)	20	3.75 (5.91)		—		—

Notes: N is the number of subjects in each condition (and may vary because of missing values).

There are several reasons that the decomposition task may be ineffective at debiasing judgment. First, it must be noted that the decomposition task did not provide an explicit algorithm instructing subjects how to combine their decomposed estimates into an overall judgment. However, a pilot study (available from the author) provided an explicit algorithm and yielded consistent results. Second, it is possible that the decomposition task itself was susceptible to analogy-driven effects. To investigate this possibility, a MANOVA was conducted on the component estimates in the decomposition task. It revealed a marginal main effect of the analogy (F_{1, 74} = 3.13, p = .08, ω² = .03), indicating that component estimates tended to be higher in the high-analogy (mean 60.5, standard deviation = 11.8, n = 37) versus low-analogy (mean 55.7, standard deviation = 14.8, n = 45) condition. Pilot work replicated this analogy effect on component estimates in another decomposition task. Given this finding, task decomposition could even exacerbate analogy effects on judgment if systematic error were inflated as a result of the nature of the decompositional algorithm.

Overall, two conclusions can be drawn from the results of this study. First, immediate use of a decomposition task appears to induce bottom-up processing, thereby limiting the effects of analogies on judgment. Second, making an initial estimate appears to induce top-down processing; after people adopt the analogy, the effectiveness of decomposition is reduced. The first conclusion suggests a boundary condition on the influence of analogies and anchors;¹⁰ the second conclusion suggests a boundary condition on the effectiveness of task decomposition in improving judgment. Nonetheless, the partial effectiveness of decompositional techniques merits further investigation.

It is difficult to distinguish analogy-induced biases from anchoring effects in this study because a numerical anchor was included with the analogy. Other studies reported in this article, as well as the pilot study, have examined analogy effects without explicit anchors. These phenomena may be naturally confounded much of the time in the real world. If so, the results are still interesting whether viewed as demonstrating analogy or anchoring effects.

General Discussion

The studies provide evidence to support the fundamental proposition of this research. Nonanalytic thinking about scenarios and analogies had persistent effects on judgment despite subsequent analytic thinking. Study 1 demonstrates that a counterscenario does not eliminate the effect of an initial scenario, thereby undermining the potential effectiveness of scenario analysis as an analytic tool for improving judgment. Study 2 provides evidence that accountability exacerbates, rather than reduces, the effects of scenarios on judgment, which suggests that managers will not be immune to this nonanalytic bias. Studies 3 and 3A demonstrate that counteranalogies fail to eliminate, but instead may exacerbate, the effect of an initial analogy, thereby undermining the effectiveness of multiple analogies as a judgment tool. Study 4 provides evidence that decomposition fails to debias judgment that is already influenced by an initial analogy. Decomposition appears to reduce reliance on reasoning by analogy, though the decomposition task itself may be susceptible to the analogy's effect. Overall, these findings attest to robust effects of nonanalytic thinking on subsequent analytic judgment. Moreover, analytic tools are susceptible to nonanalytic thinking and therefore fallible—a finding that deserves emphasis for its contrast to prior research.

Stickier Priors

Another important finding of this research is that nonanalytic thinking leads to stickier priors than analytic thinking. In Study 1, judgments were biased in favor of an initial scenario, and a counterscenario was insufficient to eliminate its effect. More important, this primacy order effect was stronger for scenarios than reasons. Similarly, in Study 3, judgments were biased in favor of an initial analogy, and counterexplanation was insufficient to eliminate its effect. Again, this primacy order effect was stronger for an initial analogy than for initial reasons. The basic finding of order effects in judgment is not new. Certainly, there is a great deal of evidence to suggest that people are non-Bayesian and experience difficulty in generating and evaluating multiple hypotheses. Taking this finding one step further, the present research provides evidence that analytic and nonanalytic thinking are differentially susceptible to order effects in judgment. Belief perseverance is therefore a function of the type of elaboration (i.e., nonanalytic or analytic thinking). People favor an initial scenario or analogy more than initial reasons. In other words, the initial scenario or analogy leads to greater fixation or a stickier prior than does more analytic reasoning.

Process Issues

The purpose of this research was to provide robust evidence for the resistance of nonanalytic thinking to subsequent analytic thinking. Thus, a series of studies investigated this proposition using two sources of nonanalytic thinking (i.e., scenarios and analogies) and a variety of analytic tools thought to improve judgment (i.e., counterfactual reasoning, decomposition, scenario analysis, multiple analogies, and accountability) and by testing across multiple products and dependent variables with some variation in subject population. The usual caveats regarding laboratory-based experimental work apply when these results are generalized to the general population of managers. However, the article speaks directly to this issue on two points: Neither accountability (which exacerbated nonanalytic bias in Study 2) nor experience (in Study 3A) appears to limit the generalizability of these results.

The quest for robustness necessitated a sacrifice in understanding process—an area ripe for further research. Indeed, reasoning by analogy and scenario generation are complex processes, and their effects are likely due to multiple processes (e.g., availability, ease of simulation, perceptual fluency, biased hypothesis testing). For example, scenarios include aspects of imagination and explanation. Imagination may be unduly influenced by associations that are causally irrelevant to new product success (e.g., mental imagery in scenarios), yet such details may enrich the scenario schema and enhance its persuasiveness. An additional experiment was conducted to investigate the effects of imagination versus explanation on judgment (details available from the author). Subjects who imagined success of electric cars were more optimistic about the probability of success over time than subjects who explained success (F_{1, 43} = 7.35, p = .01, ω² = .12). This optimism endured despite a one-week delay and efforts to debias judgment with evenhanded reasoning. A more direct examination of the role of causally irrelevant information could be conducted by manipulating the degree to which scenarios contain mental imagery. Similarly, reasoning by analogy may be influenced by salient associations among noncausal details in the analogy. Further research aimed at understanding nonanalytic processes and correcting for stickier priors is needed, and managers should be aware of the seductive effects of scenarios and analogies.

References

Alba

Joseph W.

, and Wesley Hutchinson

(1987), “Dimensions of Consumer Expertise,” Journal of Consumer Research, 13(March), 411–54.

Anderson

Craig A.

, Lepper

Mark R.

, and Ross

Lee

(1980), “Perseverance of Social Theories: The Role of Explanation in the Persistence of Discredited Information,” Journal of Personality and Social Psychology, 30(6), 1037–49.

Anderson

Craig A.

, and Sechler

Elizabeth S.

(1986), “Effects of Explanation and Counterexplanation on the Development and Use of Social Theories,” Journal of Personality and Social Psychology, 50(1), 24–34.

Armstrong

J. Scott

, Brodie

Roderick J.

, and McIntyre

Shelby H.

(1987), “Forecasting Methods for Marketing: Review of Empirical Research,” International Journal of Forecasting, 3(4), 355–76.

Boulding

William

, Morgan

Ruskin

, and Staelin

Richard

(1997), “Pulling the Plug to Stop the New Product Drain,” Journal of Marketing Research, 34(February), 164–76.

Byram

Stephanie J.

(1997), “Cognitive and Motivational Factors Influencing Time Prediction,” Journal of Experimental Psychology: Applied, 3(3), 216–39.

Carroll

John S.

(1978), “The Effect of Imagining an Event on Expectations for the Event: An Interpretation in Terms of the Availability Heuristic,” Journal of Experimental Social Psychology, 14(1), 88–96.

Chapman

Randall G.

(1997), BRANDMAPS: The Competitive Marketing Strategy Game, 4th ed. Upper Saddle River, NJ: Prentice Hall.

Connolly

Terry

, and Dean

Doug

(1997), “Decomposed Versus Holistic Estimates of Effort Required for Software Writing Tasks,” Management Science, 43(7), 1029–46.

10.

Dalrymple

Douglas J.

(1987), “Sales Forecasting Practices,” International Journal of Forecasting, 3(4), 379–91.

11.

Einhorn

Hillel J.

, and Hogarth

Robin M.

(1987), “Decision Making: Going Forward in Reverse,” Harvard Business Review, 65(January-February), 66–70.

12.

Fischhoff

Baruch

(2001), “Learning from Experience: Coping with Hindsight Bias and Ambiguity,” in Principles of Forecasting: A Handbook for Researchers and Practitioners, Scott Armstrong

, ed. Boston: Kluwer Academic Publishers, 543–54.

13.

Gentner

Dedre

, and Markman

Arthur B.

(1997), “Structure Mapping in Analogy and Similarity,” American Psychologist, 52(1), 45–56.

14.

Gilovich

Thomas

(1981), “Seeing the Past in the Present: The Effect of Associations to Familiar Events on Judgments and Decisions,” Journal of Personality and Social Psychology, 40(5), 797–808.

15.

Gregan-Paxton

Jennifer

, and John

Deborah Roedder

(1997), “Consumer Learning By Analogy: A Model of Internal Knowledge Transfer,” Journal of Consumer Research, 24(December), 266–84.

16.

Gregory

W. Larry

, Cialdini

Robert B.

, and Carpenter

Kathleen M.

(1982), “Self-Relevant Scenarios as Mediators of Likelihood Estimates and Compliance: Does Imagining Make It So?” Journal of Personality and Social Psychology, 43(1), 89–99.

17.

Griffin

Dale W.

, Dunning

David

, and Ross

Lee

(1990), “The Role of Construal Processes in Overconfident Predictions About the Self and Others,” Journal of Personality and Social Psychology, 59(6), 1128–39.

18.

Hamel

Gary

, and Prahalad

C.K.

(1994), Competing for the Future. Boston: Harvard Business School.

19.

Haugtvedt

Curtis P.

, and Wegener

Duane T.

(1994), “Message Order Effects in Persuasion: An Attitude Strength Perspective,” Journal of Consumer Research, 21(1), 205–18.

20.

Hirt

Edward R.

, and Markman

Keith D.

(1995), “Multiple Explanation: A Consider-an-Alternative Strategy for Debiasing Judgments,” Journal of Personality and Social Psychology, 69(6), 1069–86.

21.

Hoch

Stephen J.

(1984), “Availability and Interference in Predictive Judgment,” Journal of Experimental Psychology: Learning, Memory, and Cognition, 10(4), 649–62.

22.

Holyoak

Keith J.

, and Simon

Dan

(1999), “Bidirectional Reasoning in Decision Making by Constraint Satisfaction,” Journal of Experimental Psychology: General, 128(1), 3–31.

23.

Holyoak

Keith J.

, and Thagard

Paul

(1995a), “The Construction of Similarity,” in Mental Leaps: Analogy in Creative Thought. Cambridge, MA: MIT Press, 101–37.

24.

Holyoak

Keith J.

, and Thagard

Paul

(1995b), “The Analogical Scientist,” in Mental Leaps: Analogy in Creative Thought. Cambridge, MA: MIT Press, 185–209.

25.

Holyoak

Keith J.

, and Thagard

Paul

(1997), “The Analogical Mind,” American Psychologist, 52(1), 35–44.

26.

Kahneman

Daniel

, and Tversky

Amos

(1982), “The Simulation Heuristic,” in Judgment Under Uncertainty: Heuristics and Biases, Kahneman

Daniel

, Slovic

Paul

, and Tversky

Amos

, eds. New York: Cambridge University Press, 201–208.

27.

Keppel

Geoffrey

(1991), Design and Analysis: A Researcher's Handbook, 3d ed. Upper Saddle River, NJ: Prentice Hall.

28.

Koehler

Derek J.

(1991), “Explanation, Imagination, and Confidence in Judgment,” Psychological Bulletin, 110(3), 499–519.

29.

Koehler

Derek J.

(1994), “Hypothesis Generation and Confidence in Judgment,” Journal of Experimental Psychology: Learning, Memory, and Cognition, 20(2), 461–69.

30.

Kuhn

Deanna

, and Lao

Joseph

(1996), “Effects of Evidence on Attitudes: Is Polarization the Norm?” Psychological Science, 7(2), 115–20.

31.

Kuhn

Kristine M.

, and Sniezek

Janet A.

(1996), “Confidence and Uncertainty in Judgmental Forecasting: Differential Effects of Scenario Presentation,” Journal of Behavioral Decision Making, 9(4), 231–47.

32.

Lerner

Jennifer S.

, and Tetlock

Philip E.

(1999), “Accounting for the Effects of Accountability,” Psychological Bulletin, 125(2), 255–75.

33.

MacGregor

Donald G.

(2001), “Decomposition for Judgmental Forecasting and Estimation,” in Principles of Forecasting: A Handbook for Researchers and Practitioners, Scott Armstrong

, ed. Boston: Kluwer Academic Publishers, 107–123.

34.

MacGregor

Donald G.

, and Lichtenstein

Sarah

(1991), “Problem Structuring Aids for Quantitative Estimation,” Journal of Behavioral Decision Making, 4(2), 101–16.

35.

Mahajan

Jayashree

(1992), “The Overconfidence Effect in Marketing Management Predictions,” Journal of Marketing Research, 29(August), 329–42.

36.

Markman

Arthur B.

(1997), “Constraints on Analogical Inference,” Cognitive Science, 21(4), 373–418.

37.

Millar

Murray G.

, and Tesser

Abraham

(1986), “Thought-Induced Attitude Change: The Effects of Schema Structure and Commitment,” Journal of Personality and Social Psychology, 51(2), 259–69.

38.

Paich

Mark

, and Sterman

John D.

(1993), “Boom, Bust, and Failures to Learn in Experimental Markets,” Management Science, 39(December), 1439–58.

39.

Pennington

Nancy

, and Hastie

Reid

(1992), “Explaining the Evidence: Tests of the Story Model for Juror Decision Making,” Journal of Personality and Social Psychology, 62(2), 189–206.

40.

Read

Stephen J.

, and Cesa

Ian L.

(1991), “This Reminds Me of the Time When…: Expectation Failures in Reminding and Explanation,” Journal of Experimental Social Psychology, 27(1), 1–25.

41.

Reeves

Lauretta M.

, and Weisberg

Robert W.

(1994), “The Role of Content and Abstract Information in Analogical Transfer,” Psychological Bulletin, 115(3), 381–400.

42.

Ringland

Gill

, Todd

Keith

, and Schwartz

Peter

(1998), Scenario Planning: Managing for the Future. New York: John Wiley & Sons.

43.

Ross

Lee

, Lepper

Mark R.

, Strack

Fritz

, and Steinmetz

Julia

(1977), “Social Explanation and Social Expectation: Effects of Real and Hypothetical Explanations on Subjective Likelihood,” Journal of Personality and Social Psychology, 35(11), 817–29.

44.

Russo

J. Edward

, and Schoemaker

Paul J.H.

(1992), “Managing Overconfidence,” Sloan Management Review, 33(2), 7–18.

45.

Sanbonmatsu

David M.

, Akimoto

Sharon A.

, and Biggs

Earlene

(1993), “Overestimating Causality: Attributional Effects of Confirmatory Processing,” Journal of Personality and Social Psychology, 65(5), 892–903.

46.

Schnaars

Steven P.

, and Topol

Martin T.

(1987), “The Use of Multiple Scenarios in Sales Forecasting: An Empirical Test,” International Journal of Forecasting, 3(4), 405–19.

47.

Schoemaker

Paul J.H.

(1993), “Multiple Scenario Development: Its Conceptual and Behavioral Foundation,” Strategic Management Journal, 14(3), 193–213.

48.

Sieck

Winston R.

, Quinn

Clark N.

, and Schooler

Jonathan W.

(1999), “Justification Effects on the Judgment of Analogy,” Memory and Cognition, 27(5), 844–55.

49.

Siegel-Jacobs

Karen

, and Frank Yates

(1996), “Effects of Procedural and Outcome Accountability on Judgment Quality,” Organizational Behavior and Human Decision Processes, 65(1), 1–17.

50.

Simonson

Itamar

, and Nye

Peter

(1992), “The Effect of Accountability on Susceptibility to Decision Errors,” Organizational Behavior and Human Decision Processes, 51(3), 416–46.

51.

Sloman

Steven A.

(1996), “The Empirical Case for Two Systems of Reasoning,” Psychological Bulletin, 119(1), 3–22.

52.

Spellman

Barbara A.

, and Holyoak

Keith J.

(1992), “If Saddam Is Hitler Then Who Is George Bush? Analogical Mapping Between Systems of Social Roles,” Journal of Personality and Social Psychology, 62(6), 913–33.

53.

Spellman

Barbara A.

, and Holyoak

Keith J.

(1996), “Pragmatics in Analogical Mapping,” Cognitive Psychology, 31(3), 307–46.

54.

Tetlock

Philip E.

(1983), “Accountability and Complexity of Thought,” Journal of Personality and Social Psychology, 45(1), 74–83.

55.

Tetlock

Philip E.

(1985), “Accountability: A Social Check on the Fundamental Attribution Error,” Social Psychology Quarterly, 48(3), 227–36.

56.

Tetlock

Philip E.

(2000), “Cognitive Biases and Organizational Correctives: Do Both Disease and Cure Depend on the Politics of the Beholder?” Administrative Science Quarterly, 45(2), 293–326.

57.

Tetlock

Philip E.

, and Boettger

Richard

(1989), “Accountability: A Social Magnifier of the Dilution Effect,” Journal of Personality and Social Psychology, 57(3), 388–98.

58.

Tetlock

Philip E.

, and Kim

Jae Il

(1987), “Accountability and Judgment Processes in a Personality Prediction Task,” Journal of Personality and Social Psychology, 52(4), 700–709.

59.

Tversky

Amos

, and Kahneman

Daniel

(1983), “Extensional Versus Intuitive Reasoning: The Conjunction Fallacy in Probability Judgment,” Psychological Review, 90(4), 293–315.

60.

Wack

Pierre

(1985), “Scenarios: Uncharted Waters Ahead,” Harvard Business Review, 63(September-October), 73–89.

61.

Zaltman

Gerald

(1997), “Rethinking Market Research: Putting People Back In,” Journal of Marketing Research, 34(November), 424–37.