Abstract
The central question of David Berri and Martin Schmidt’s most recent book, Stumbling On Wins, is how so many people paid so much to make good managerial choices can consistently and repeatedly make bad ones. Often these choices are bad not just in retrospect, but appear predictably ill informed even in a profession inundated by a veritable deluge of quantitative data. Indeed, amateur bystanders and academics have used publicly available data to create a vibrant cottage industry disseminating statistical analysis and (ex post) testable predictions. Many of these modelers have developed “favorite toys” that consistently predict athlete performance better than the professionals. The book’s title, an allusion to Daniel Gilbert’s Stumbling on Happiness, is intended to point out how elusive the secret of building winning sports teams (like the secret of happiness) remains. While the vast amounts of interest and effort put into the respective searches are similar, the soundness of the implied analogy is crucial to the authors' thesis, yet is largely ignored. What if “Wins” in sports aren’t always the same as “Happiness”?
The greatest strength of Stumbling is the enumeration of many examples that individually and collectively appear to show highly questionable, if not irrational, managerial decision making. Their examples are drawn from all four major North American team sports to demonstrate that the anomalies are not league specific, and they are presented in a lucid and entertaining manner. The authors' writing style is engaging and witty, and it is clear that they are drawing from deep knowledge of their subject matter (both sporting and statistical) and that they enjoy their work.
By presenting a large number of short subjects with a broad unifying theme, Stumbling joins a rapidly expanding genre of “observational economics,” our discipline’s version of a Jerry Seinfeld stand-up routine or an Andy Rooney monologue. Of course, sports economists were writing books of analytical essays on a range of topics well before Freakonomics and The Undercover Economist hit the best-seller lists, with these authors' previous work with Stacey Brook, The Wages of Wins, holding a place alongside Fort and Quirk’s Pay Dirt, Goff and Tollison’s Sportometrics, and Bradbury’s The Baseball Economist, among the best examples.
In this volume, the common approach for discussion of each topic is to lead with an anecdote, then present the current best practice for statistical analysis of that topic. A table lists the factors shown to be important in managerial decisions, followed by a second model that identifies a list of factors that are most predictive of future performance. The differences between the lists are invariably attributed to cognitive errors.
In perhaps the best example, Chapter 6 presents an analysis of the National Basketball Association (NBA) draft. Table 6.6 answers “What Explains Draft Position in the NBA?” while Table 6.7 counters with “What Explains a Player’s Productivity in the NBA?” These lists do not coincide for several factors such as rebounding (increases productivity; no effect on draft position) and playing for an NCAA Champion the Year Drafted (improves draft position; reduces productivity in the NBA).
Why don’t the lists match? Berri and Schmidt present their thesis: “We believe this is what’s happening on draft day. Decision-makers try to consider everything, but the limits of the human mind undermine this effort. For a decision to be made, the human mind has to simplify the vast list of factors considered. The simplification process ends up emphasizing the factors that are most conspicuous. In other words, the final decision is dominated by scoring, age, height, and Final Four appearances. [That] list of factors, though, is not really related to future productivity in the NBA” (p. 100).
For each of these cases, and indeed for all of the numerous examples of anomalies presented in the book, the disparities are categorized in the concluding chapter either as cases where “decision makers have the information, but it’s not interpreted correctly” or where “the predictive power of information is misunderstood” (p. 137).
The problem with this taxonomy, of course, is that it does not allow the possibility that the anomalies are not examples of cognitive error. Although the authors are swinging a hammer, not every unresolved puzzle in sports economics is a nail. Although behavioral economics is an exciting new subfield that promises to greatly increase understanding of decision making over the coming years and decades, many of these anomalies—in fact, the vast majority of them, can be explained far more plausibly as examples of the principal-agent problem. If these are agency problems, the disparities found are not due to a lack in the manager’s cognitive function but rather in the analysts' understanding of what motivates the decision maker. Put another way, which choice is “best” depends upon who is choosing.
Seen in this light, several of the authors' conclusions come into question. Is the voting for NBA All-Rookie teams (Chapter 2) misaligned with productivity factors because the coaches do not understand how basketball games are won? Or could it be because the coaches are not hired or fired for their All-Rookie team selections, but instead for watching enough scouting films and preparing their own players to win more games better than would the proverbial deck chair (p. 125)? Is Coach Kevin Kelley’s (Chapter 7) refusal to have his Arkansas prep football team punt on fourth down a shining example of triumph over the win-sapping risk aversion from which most coaches suffer, as illustrated by Romer’s model? Perhaps risk aversion is rational for a coach who obtains exponentially less job security from converting a fourth down attempt than job insecurity from failures to convert that are featured on ESPN’s SportsCenter for the next 6 days. Perhaps the real reason Coach Kelley (and Marty Schmidt’s son on Madden NFL 2008) can always “go for it” on fourth down is that they are not running league average offenses against league average defenses; they can be (rationally) very confident that their respective opponents are highly unlikely to stop them from gaining the needed yardage.
As for the NBA draft, can it really be true that general managers (GMs) are incorrectly screening the same mountain of data year-after-year-after-year and selecting the stars of one of last year’s NCAA Final Four team in frustration? Does not it seem more plausible that a GM might have an objective function that includes items such as criticism avoidance and team financial success in addition to wins or that there might still be some tweaks necessary in the regression model needed to fix the specification?
If a GM passes over a recognizable player who has been a leader on a Championship college program and that player goes on to greater NBA success than the relatively unknown player he chose for the team, the fallout could be career threatening. Given the high variance in draftee success that Berri and Schmidt themselves document, the NCAA star offers more career insurance to the GM in the event the pick is a dud. Moreover, the big-name player can help the team sell a lot more tickets and rake in more revenue now, whereas the underrated player may (or may not) win a few more games for the team years down the road. What is the appropriate discount rate for a GM when the position’s job turnover is often likened in the press to a revolving door?
Generally speaking, the fundamental question that does not seem to have been adequately addressed during the analysis of these anomalies is: are a team’s decision makers really being judged and rewarded in such a way that they are even trying to maximize wins? If the standard profit-maximizing assumption used for most firms outside of sports is applied, many of these puzzles look much less puzzling. Chapter 8 contains a summary of the current literature on aging and peak athletic productivity and uses these findings as a baseline to show that playing time is overallocated to older players relative to their productivity. However, an opportunity to fill more seats and generate higher television ratings by giving people the opportunity to watch an aging legend rather than by winning a few more games might be attractive to some teams.
When the NBA’s Washington Wizards in 2001 signed 38-year-old Michael Jordan, it is doubtful that they expected him to perform at the level he had with the Chicago Bulls. The Wizards won more with Jordan in 2001-2002 than they had in the previous season, but they still missed the 16-team playoffs by several games. Depending upon what evaluation system one uses, Jordan’s performance relative to his teammates could be regarded as very good (1st on the Wizards in Player Efficiency Rating [all ratings from www.basketball-reference.com]), or fairly good (4th on the team in Win Shares, behind Chris Whitney, Richard Hamilton, and Popeye Jones), or mediocre (11th on the team in Win Shares per 48 minutes). While Jordan’s on-court productivity could be disputed, what is clearer is that the Wizards' attendance rose from 18th in the NBA to 3rd, and that the Wizards received substantially increased national television exposure and merchandise revenue. Even when not launching shots from beyond the 3-point line, Jordan could still “make it rain” financially. Was giving him so much playing time an error, or was it a successful realization of their management objective?
The authors' chief innovation is perhaps their statistical modeling of productivity in professional basketball, which builds upon earlier work such as that in Dean Oliver’s Basketball on Paper. But using modeling results to claim inefficient draft selections and salary negotiations by team GMs leaves their assertions open to challenge on two fronts. First is the possibility raised in the Jordan example; that it remains unclear whether teams are profit-maximizing, win-maximizing, or win-maximizing subject to a budget or profitability constraint. Berri and Schmidt show in Table 6.3 that wins and lagged wins correlate with team revenue, a result consistent with the classic Scully model, but the relationship between the marginal cost and the marginal revenue of an expected win is still unknown. Second is the possibility that, despite their best efforts, the model is producing estimates for player productivity that are incorrect.
Econometricians might sit around a pub table and disagree until closing time over the proper specification of a model or the functional form particular variables should take within it, and then repeat the process upon their next meeting ad infinitum. This being the case, I shall refrain from nitpicking with their specification here. The differences in the ratings of Jordan’s 2001-2002 performance (as in other seasons) illustrate how widely evaluations of individual performance can vary while still predicting team performance better than free-agent salaries negotiated by GMs.
Without a natural experiment, all the authors have is a joint hypothesis. Given that the model’s expected salaries for players do not match labor market outcomes, all one can truly conclude is that either the labor market is inefficient or that the model is generating incorrect estimates. The labor market strategy of baseball’s Oakland Athletics, as portrayed in Michael Lewis's Moneyball, presented just such a natural experiment. Researchers looking at salary and performance data from Major League Baseball (MLB) prior to the turn of the century would have only gotten as far as Berri and Schmidt have here: establishing a disparity between market prices and the value to winning of factors of production. It was not until the baseball labor market shifted so as to align with a model’s estimates—presumably as other General Managers reacted to the Athletics visible success—that the hypothesis of model misspecification could be rejected in favor of the hypothesis of prior salary inefficiencies.
Requiring evidence that is only available after the fact, however, is decidedly inconvenient for researchers hoping to discover potential market improvements. Without such empirical confirmation, however, a myriad of possible errors must be explained away to show that misspecification or mismeasurement is not a concern, and a similar multitude of potentially “correct” models can vie for preference. But more importantly, in some situations, as when evaluating draft picks of NFL quarterbacks and NBA players in Chapters 5 and 6, reliance upon regression analysis to estimate individual expected player performance is to neglect a fundamental concept in finance and risk management. Even a well-specified productivity regression does not estimate probabilities of excellence, but rather the expected value of the individual player’s performance. In a sports industry organized to distribute revenues disproportionately to the top teams in relative performance and where roster constraints create rents for teams that can concentrate high production in a few roster spots, one superstar who wins X games is more valuable than three players who can win X/3 games. The expected value approach neglects the concept of risk and reward both for an individual pick and also for the “portfolio” of picks made over the course of a draft.
All these concerns aside, for those choosing to accept the most plausible results of the authors' modeling as a blueprint for improved team performance, this book leaves a tantalizing question: Is there an NBA equivalent of the Athletics' GM Billy Beane waiting in the wings who is willing to make an entrepreneurial gamble for a chance (at least until the market corrects) of on-court success at discount prices?
Anecdotal evidence suggests that the future, if it is not already here, may be soon. In the same manner that MLB teams in the past few years have hired (and sometimes even listened to!) statistical consultants, many teams in other major professional sports leagues have also “geeked up.” Teams in basketball, soccer, hockey, and other sports are spending enough money to nurture an emerging market to provide data on movements of all the players and the ball quickly enough to provide analysis in (almost) real time. It is possible that data from a system such as Stats LLC’s SportVU will either support one of the current models of NBA productivity, or perhaps provide insights leading to further model evolution. In the meantime, Stumbling On Wins provides an engagingly written analysis and entertaining anecdotes and examples, and it also represents the state of the art in sports econometrics. Although certainly not the final word on the subject, the book presents a bit more light on the subject of how success is created, and would be enjoyed by nearly everyone … except perhaps Isiah Thomas.