Abstract
Marketing models are often intended to guide managers to make decisions. In this rejoinder, the author reviews three questions that must be addressed before a model can be established as useful for policy analysis.
The relevance of econometric models in marketing is often demonstrated by simulated outcomes under different scenarios of the marketing mix. Such simulations are helpful to demonstrate what the model means exactly. Many present-day econometric models used in marketing research are not linear in their parameters and sometimes contain unobserved components. In addition, various models deal with time-series variables with possibly complicated properties, such as trends, whereas other models assume spatially correlated error terms. All these aspects simply mean that modelers must go beyond determining whether estimated parameter values are significant and have the “right” sign. Therefore, creating and analyzing data from these models in various marketing-mix scenarios can provide valuable insights.
It is tempting to interpret the outcomes of hypothetical scenarios as instrumental to policy analysis; that is, the outcomes might suggest the answer to what-if questions. However, the argument made by Franses (2005) and emphasized by Bronnenberg, Rossi, and Vilcassim (2005) and Van Heerde, Dekimpe, and Putsis (2005) is that for econometric models to be really useful for policy analysis, additional issues must be addressed. In this short article, I formulate these issues as questions that researchers should address when contemplating the use of a model for policy analysis. 1 These questions appear as the titles of the next three sections.
It is not my intention to specify a set of rules to editors and reviewers to make the rejection of articles easier. Both Bronnenberg, Rossi, and Vilcassim (2005) and Van Heerde, Dekimpe, and Putsis (2005) worry about the possibility that if a buzzword is not used by authors, a manuscript is prone to rejection. As one of the reviewers rightfully pointed out, this possibility could lead to authors mindlessly using tests or methods without having strong awareness of their consequences. The main purpose of Franses (2005) and this rejoinder is to place concepts such as endogeneity, expectations, parameter stability, model diagnostics, and the Lucas critique at the forefront of marketing researchers' mind sets.
Does the Model Match Reality?
For an econometric model to be useful for policy analysis, it must summarize the available data related to the situation at hand. If the model does not reflect the “real world” it tries to describe, then what does it do?
There are various views on what that “real world” is. For example, for promotion planning of a brand, a manager might want to include knowledge of the closest competitors and what they would do if they became aware of the brand's promotion schedule.
Next, the model should reflect that real world, and model evaluation depends on the marketing question and the availability of data. For example, for long-term inference about the adoption of new products, a manager can rely on annual data, and hourly sales of these products are then less relevant. In contrast, for decisions about shelf space allocation, highly disaggregate data on the behavior of individual consumers who look at these shelves are needed.
These two issues—namely, the reality that the model aims to describe and the level of abstraction of the model—determine the size, shape, and relevance of the model. Therefore, there is no optimally designed model, because the very design depends on each situation and question.
Theoretical Properties
A theoretical requirement of a useful policy model is that it includes all variables and components that are relevant for the marketing question at hand. If the model is to be used to understand the long-term impact of the marketing mix, it should include time-series variables that allow for correlations over time. If the model aims at a better understanding of the competitive structure, the model should include retailers and manufacturers. If the setting is such that one or more of these market participants can foresee what other participants might do, these expectations should be addressed too.
Not everything that is going on in the real world needs to be included in the model. After all, it is a model, which by definition is an abstraction of reality. It would be nice to be able to model everything, but whether anyone wants or needs to do that again depends on the marketing question. 2
As an illustration, Reiss and Wolak (2004) develop models based on the microeconomic foundations of consumer choice on the demand side and on the behavior of firms on the supply side, in which they also allow for competition. Note that this excellent article is 159(!) pages long. Although it is a wonderful article, sophisticated and elegant, could hands-on marketing questions also be answered using simpler models?
Statistical Properties
Most modelers would agree that models should reflect reality to some extent. Otherwise, the models are not of much use. Many diagnostic tests exist to guide the researcher to improve models. Models are based on assumptions, and these assumptions should be checked. In Franse (2005), I single out the case of a variable x influencing y for illustration purposes, but this notion also holds for the models mentioned by Van Heerde, Dekimpe, and Putsis and Bronnenberg, Rossi, and Vilcassim. In my first article, I argue that the more the model is supposed to do (from descriptive, to forecasting, to policy simulation), the more assumptions need to be checked.
At times there are debates about the usefulness of sophisticated econometric models to describe and forecast economic, marketing, or other variables. 3 Large macro models may not appear to forecast well, even in the long run, so why should more involved marketing models be better? In many “horse races” in terms of out-of-sample time-series forecasting, simple extrapolation models seem to do just fine or often much better (for an extensive demonstration, see Makridakis and Hibon 2000; for recent statistical support of their finding, see Koning et al. 2005). However, this finding is true only for the numerical values of those forecasts. These extrapolation rules are far from informative if the projections turn out to miss the true observations completely. An extreme case is that a random walk forecast might predict tomorrow's sales quite well, even when evaluated over several years, but suppose tomorrow's sales are completely different from those of today; what can managers then learn from looking again at the random walk model? How can they understand why the forecast went wrong?
The discussion of the level of abstraction of models for policy analysis has a long history in macroeconomics, in which large-scale macro models survive next to four-equation vector autoregression models.
In summary, given a research question, the quality of the available data, and the purpose of the modeling exercise, modelers always must verify whether the model adequately summarizes the data. This applies to any type of model, be it a panel data model, a time-series model, or a new empirical industrial organization model. 4 The confidence in such diagnostic measures may not be high, and some of the tests have notoriously low power. Therefore, marketing researchers must continue to design better measures to determine the quality of models. In other sciences, they do that too.
I do not argue for carrying out only new empirical industrial organization–type studies. I view their benefits, but, as do Bronnenberg, Rossi, and Vilcassim (2005), I also see their drawbacks.
Can Consumers and Competitors Expect Policy Changes?
Models contain equations with variables on the right-hand side, some of which are assumed to be controllable by the manager, and at least one variable on the left-hand side, which measures the response by consumers or competitors. In a sense, this setup assumes that other agents respond with full surprise to what managers do. In some cases, as in the example of Albert Heijn's lower prices, described by Franses (2005) and Van Heerde, Dekimpe, and Putsis (2005), this assumption can be valid. Consumers could not possibly expect that the prices of more than 1000 stock-keeping units would be decreased by Albert Heijn so suddenly, nor could the competition (which responded two days later). In predicting what consumers would do the day of the price decrease, the retailer might have used models that link unannounced price changes with sales in the past.
Things would have been very different if Albert Heijn had announced the price changes a few days before. Things would also have been different if this retailer had substantially reduced prices quite regularly in the past. In that case, consumers can form expectations and, given these, change their behavior because of some knowledge of the pricing strategies. The Lucas critique carries this notion one step further by noting that neglecting this possibility may make econometric models that describe all available data (i.e., before and after price changes) difficult, if not impossible, to use for policy analysis.
For retailers (and this is just one example), it thus becomes important to understand what consumers expect, for them to quantify the effect of, say, price changes. If consumers base these expectations on their knowledge about the same model that Albert Heijn would use to set prices, these expectations are associated with rational expectations. A typical solution in econometric modeling is to include additional equations into the model, such as equations for the pricing process, which are used to form expectations.
This process immediately raises the question of how to know how consumers form expectations. Researchers might include convenient mathematical descriptions into models to reflect typical schemes (rational, adaptive) but cannot be sure whether that inclusion is done correctly. However, marketing seems to have an advantage relative to other economic disciplines. Because the data and the questions sometimes pertain to the very micro level, marketers could simply ask consumers what they would do. Modern conjoint analysis techniques and clever ways to elicit future revealed preferences from current stated preferences, also developed by marketing research, can then be used.
Does the Marketing Mix Depend on Expectations?
Another intriguing phenomenon, certainly in a modeling context, is endogeneity (for marketing studies, see Besanko, Gupta, and Jain 1998; Bronnenberg and Mahajan 2001; Manchanda, Rossi, and Chintagunta 2004; Villas-Boas and Winer 1999; Villas-Boas and Zhao 2005). It is an often used concept in macroeconometrics, where it dates back to the days of developing the first simultaneous equation models, and it remains an important concept. The econometric literature on endogeneity, and its counterpart exogeneity, is vast. Unfortunately, this literature is difficult to understand.
Basically, endogeneity entails the following: If a manager wants to know what the effects are of changing the marketing mix, he or she might make a model to link, say, prior changes in sales with prior changes in prices. However, the manager may have set price levels that depended on what he or she thought the response would be. If this price-setting behavior led to sales levels very different from sales levels given normal prices, it would be reflected by larger disturbances, which are correlated with those prices again. Thus, the x variable is correlated with the error term. In that case, the manager cannot infer the net effects of changing x by just considering a regression of y on x.
The interesting outcome seems to be that if econometric models are regularly used for policy analysis, marketing-mix variables become more endogenous than exogenous. Thus, strictly speaking, it might be better to assume that all variables are endogenous from the start. Again, it is left to the modeler to decide whether the degree of endogeneity matters. In that sense, it would be mindless to dismiss all studies that do not assume endogeneity, and marketers should prevent the mindless use of instrumental variable techniques, which suffer from all kinds of other problems.
Final Thoughts
In closing, I mention two research areas in marketing that seem to have received little attention. The first is the need to incorporate into models better ways of describing how consumers, competitors, and other stakeholders form expectations. The second area is the incorporation of the last stage of the whole exercise: those who actually implement models. Often, these decision makers have other loss functions than the modelers have, which might affect model choice.
Finally, the three questions in the titles of the sections highlight issues that must be addressed when planning to use a model for policy analysis. I hope that this article, along with the discussions in Franses (2005), Bronnenberg, Rossi, and Vilcassim (2005), and Van Heerde, Dekimpe, and Putsis (2005), stimulates further thinking about these issues and that marketing research studies take these issues into account. With Bronnenberg, Rossi, and Vilcassim, I strongly believe that if there is one discipline in economics and business in which this development can take place at a high and sophisticated level, it must be marketing. Data are plentiful and carefully measured, researchers are highly skilled, and practitioners increasingly perceive the need for quantitative support of their decisions.