Value-Added Models for Teacher Evaluation and Accountability: Commonsense Assumptions

Introduction

Value-added models (VAMs) are intended to objectively measure the amount of “value” that a teacher “adds” to (or detracts from) student learning and achievement from one school year to the next. More specifically, VAMs are statistical tools meant to measure the observable relationships between a teacher’s instruction and how that instruction contributes to student learning and achievement over time. This is done by statistically measuring growth in achievement from one year to the next using students’ large-scale test scores, while controlling for students’ prior testing histories, and sometimes controlling for student-level demographics (e.g., race, poverty, English language proficiency, special education status), other student-level variables (e.g., attendance, suspension, retention), and other classroom- and school-level variables (e.g., class size, average prior achievement) as available and as specified per model.

Prior to the recent January 2016 passage of the Every Student Succeeds Act (ESSA; 2016), which helped to curb the stronger teacher and teacher education accountability reforms ongoing across the United States since President Obama’s Race to the Top (2011) initiative (by which states were incentivized/required to adopt VAM-based teacher evaluation policies), 44 states and D.C. (88%; n = 45/51) had adopted and had at least begun implementing VAMs to evaluate and, in many cases, make important decisions about teachers in effect (e.g., teacher tenure, merit pay, teacher termination). In addition, 30 states and D.C. (61%; n = 31/51) had state legislative acts and regulations that mandated such teacher evaluation initiatives (Collins & Amrein-Beardsley, 2014). Given the recency with which ESSA was passed, however, it is now somewhat uncertain what states are doing post ESSA. Although some states seem to still be moving forward with their teacher evaluation and accountability plans regardless of ESSA (e.g., Florida, New Mexico, New York, Ohio, Tennessee, Texas), others have at least paused or begun to reframe their stronger accountability efforts (e.g., Alabama, Arizona, Georgia, Oklahoma), although these states still seem to have some measure of value-added included within what still appear to be similar teacher evaluation systems, albeit with less consequential stakes attached.

Perhaps the fact that states appear to still be moving forward in these regards, albeit in varying ways, is because the idea behind the development and implementation of VAMs is not necessarily wrongheaded; hence, and especially conceptually, these reform ideas continue to resonate among many state-level educational policymakers and educators. However, it can also be argued that educational policies in support of such strategies are nonetheless misguided given the research surrounding VAMs, including VAM adoption, implementation, and use, and especially when said uses are tied to high-stakes consequences (e.g., teacher tenure, merit pay, teacher termination). Specifically, contemporary concerns about using VAMs for teacher evaluation and accountability purposes, as per the current research, primarily pertain to (a) methodological concerns (e.g., reliability, validity, bias), (b) logistical concerns (fairness, transparency, usability), and (c) consequential concerns (e.g., high-stakes attached to outcomes, effects on school culture and the teaching profession). The purpose of this article is to explore these concerns, while attempting to understand the persisting debate that surrounds them.

There is a growing divide between academic scholars who have taken to either side of the VAM debate, with many educational economists often (although not always; see, for example, Rothstein, 2009, 2010, 2014) on one side, promoting the strengths and potentials of these models (see, for example, Chetty, Friedman, & Rockoff, 2014a, 2014b; Hanushek, 1970, 1971, 1979, 2009, 2011; Kane & Staiger, 2008, 2012), and educational researchers often on the other side, more often representing VAM critics (see, for example, B. D. Baker, Oluwole, & Green, 2013; E. L. Baker et al., 2010; Briggs & Domingue, 2011; Corcoran, 2010; Eckert & Dabrowski, 2010; Gabriel & Lester, 2013; Graue, Delaney, & Karch, 2013; Hill, Kapitula, & Umlan, 2011; Newton, Darling-Hammond, Haertel, & Thomas, 2010; Papay, 2011; Rothstein, 2009, 2010, 2014). Given the still current momentum of VAM adoption and use, then, it is reasonable to posit that economists have more or less taken the lead in influencing now state-level policies in this area, as well as international initiatives and policies in this area (Araujo, Carneiro, Cruz-Aguayo, & Schady, 2016; Sørensen, 2016), which is not surprising in light of their rising influence in the public policy arena at large (Fourcade, Ollion, & Algan, 2015; Lazear, 1999, 2001)

The Rise of the Economist in Education Policy

Of all the social science disciplines, economics has risen to the top in terms of its political and social influence (Lazear, 1999, 2001). Fourcade et al. (2015) describe this rise in clout, arguing that despite economists’ inability to predict the financial crisis of 2007-2009, their analytic tools to make predictions about social processes are trusted more now than they were 30 years ago. Fourcade et al. also point out that economists see their discipline as the most rigorous, and accordingly rely almost exclusively on what they view as the most rigorous statistical models and measures to conduct their research, to often yield economics-based (e.g., “bottom-line”) answers. Related, the epistemological underpinnings of the economics discipline, that is about what is known and can be known using such methods and models, often contradict those of other social science disciplines. This creates an increasingly insular network of economists that rarely incorporate, and often marginalize other interdisciplinary traditions (e.g., sociology, political science, psychology, education). Lazear (1999) further explained,

It is the ability to abstract that allows us [economists] to answer questions about a complicated world . . . I have argued elsewhere that the strength of economic theory is that it is rigorous and analytic. But the weakness of economics is that to be rigorous, simplifying assumptions must be made that constrain the analysis and narrow the focus of the researcher. It is for this reason that the broader thinking sociologist, anthropologist and perhaps psychologist may be better at identifying issues, but worse at providing answers. Our narrowness allows us to provide concrete solutions, but sometimes prevents us from thinking about the larger features of the problem. This specialization is not a flaw [however, as] much can be learned from other social scientists who observe phenomena that we often overlook. (pp. 5-6)

Given that economists largely rely on parsimonious frameworks, it is in the nature of the economist’s work to make sense of complex issues using sophisticated tools, in or to simplify the issue. However, from an epistemological standpoint, statistical tools that intentionally disregard uncontrollable factors cannot always help social scientists get at the complex social processes that they are tasked to investigate. Thus, while econometrics might appeal to policymakers for the tangible, “simple” outcomes and solutions they can yield, other analytical tools might better serve our understandings of complex social domains, such as those resident within education.

Related, factions of educational policy have come to be based on such economic models and measures, especially given the policy-popular accountability initiatives related to teacher quality (e.g., VAMs). For example, after the sociologist, Coleman and colleagues (1966) reported that teacher quality was the most influential in-school variable when explaining differences in student achievement scores, the economist, Eric Hanushek, began to conceptualize teacher effects in economic terms as based on the relationship between inputs (e.g., education status, years of experience) and outputs (i.e., student achievement scores). After applying an economic-based analytical framework to a set of data in one California school district, Hanushek (1970) argued that (a) years of experience and graduate education were not related to higher student achievement, (b) teacher effects did not explain Mexican American students’ achievement outcomes, and (c) teacher effects did have an impact on White students’ achievement outcomes, regardless of socioeconomic status. Hanushek (1979), encouraged by these findings, proceeded to call upon additional econometric instruments to develop a model that could measure the amount of value that a teacher added or detracted from student learning. Thus, the VAM—which was most commonly used in business and agriculture prior—made its way into the education scene. Soon thereafter, a different set of econometricians further established the VAM in more applied and pragmatic terms, for example, in the state of Tennessee (Sanders, 2003; Sanders, Wright, Rivers, & Leandro, 2009).

VAMs have since taken a progressively significant role in teacher accountability policies and practices, regardless of the extent to which the methods might be thwarted by theoretical, methodological, and practical problems, some of which we mentioned prior (see, for example, B. D. Baker et al., 2013; E. L. Baker et al., 2010; Briggs & Domingue, 2011; Corcoran, 2010; Eckert & Dabrowski, 2010; Gabriel & Lester, 2013; Graue et al., 2013; Hill et al., 2011; Newton et al., 2010; Papay, 2011; Rothstein, 2009, 2010, 2014; Schochet & Chiang, 2012). Accordingly, VAMs continue to be the source of concern surrounding discussions of educational reform. This tension might also be due, at least in part, to the theoretical and epistemological differences of the economists who have had the most influence on education policy/practice, and other social scientists who have criticized VAM-use on the grounds of simplicity.

In this conceptual piece, we argue this might just be the case, as we attempt to disentangle these discrepancies by reviewing a large set of the VAM literature coming from multiple disciplines, as well as nontraditional sources, such as op-ed pieces, blogs, and newspaper articles. We do this while tracking and coding the implicit and explicit assumptions that have, over time, been accepted as “truth” despite interdisciplinary disagreement and interdisciplinary research evidence countering the assumptions being made. Ultimately, we argue that a multidisciplinary approach should be used to understand VAMs and VAM-use.

Conceptual Framework

As mentioned, there is a group of economists subscribing to a particular ideological camp who accept and therefore work from a set of theoretical and epistemological assumptions that underpin VAMs and VAM-use. These particular economists work from a discipline-specific assumption that teachers are rational beings who make rational decisions. Logically, VAM methodology is seen as a legitimate means for measuring teacher effects and using VAM outcomes to incentivize teachers to perform better. This is in stark contrast to many other social scientists (including some economists) who conceptualize education and teaching as complex social domains that cannot possibly be understood in such rational and simplistic ways. This former set of economists who, again, subscribe to this particular ideological camp, tend to work from the presupposition that “the properties of ‘collectivities’—groups, institutions, societies—can be reduced to statements about the properties of individuals” (Ingham, 1996, pp. 245-246).

In contrast, for example, sociologists tend to work from the presupposition that individuals are part of larger social structures that influence the makeup of the individuals (Ingham, 1996). Accordingly, the methodological approaches to understanding human behavior and society are quite different. Other social science disciplines have entirely different sets of epistemological assumptions within which they ground their work. While the field of economics is unquestionably made up of far more than a monolithic group of scholars and analytical approaches, there is a particular faction of the discipline that has embraced VAMs by anchoring their arguments in assumptions that fail to hold up in other disciplines’ approaches.

Going back to the previous example, the sociologist who typically sees individuals as parts of larger social structures might seek to understand teacher quality as a complex mixture of social factors that cannot, entirely, be statistically observed or measured, or much less statistically controlled. Sociologist Raudenbush (2004) suggests something similar in his statement: “[T]he estimates from VAM, when combined with other information, have potential to stimulate useful discussion about how to improve practice. But they should not be taken as direct evidence of the effects of instructional practice” (p. 128). Hence, the differences in epistemological positions create challenges when scholars representing multiple disciplines attempt to understand a single phenomenon or subject, such as teacher quality or how teacher accountability mechanisms can improve it. Likewise, this calls for an multidisciplinary approach to understanding VAMs and VAM-use.

To this end, our purpose for writing this article was to address the conflict that specifically relates to VAMs and VAM-based policies and practices. We sought to understand the core of the debate by investigating two primary research questions: (a) to identify the assumptions, or conditions, that have not been made explicit when calculating teachers’ value-added, but have been implicitly assumed as “true” conditions upon which VAM-based output can be used; and (b) to map these assumptions onto the greater VAM literature to interpret the feasibility of such assumptions as being “true” given a multidisciplinary lens.

For the purposes of this article, we define “assumption” broadly to include room for all social science disciplines’ definitions, for we find that such inclusion is not only important but also critical for understanding the complexities involved in the process for measuring teacher value-added. Specifically, we define an “assumption” to mean a condition which has been accepted as “truth,” with the understanding that violations of said assumptions can ultimately result in invalid inferences, spurious conclusions, and sometimes unintended consequences at some level of VAM development, implementation, or use. As such, assumptions can include traditional, statistical-based assumptions that must be met for VAM estimates to be considered reliable and valid, or assumptions can include more publicly accepted beliefs about, for example, teachers’ attributional or causal effects on student achievement over time (i.e., value-added).

Consequently, and regardless of the validity or truth of the assumptions made, or whether the assumptions made are done so in implicit or explicit terms, such assumptions are inextricably linked to the rationales behind VAMs, as well as the justifications used to perpetuate the VAM narrative. Accordingly, once assumptions are made explicit, violations of assumptions regardless of assumption type must be acknowledged and then questioned as to “whether plausible departures from those assumptions would lead to substantially distorted inferences” (Reardon & Raudenbush, 2009, p. 493).

As educational policy researchers, what sparked our interest in this study was the ongoing acceptance of VAMs and VAM-based policies at high levels of policy decision-making (e.g., federal and state governments), despite mounting concerns coming from educational scholars, but also given the mounting influence of many economists in the area of VAMs. As we argued prior, econometric methods are increasingly being regarded above all other disciplinary methods of inquiry (see also Fourcade et al., 2015), as are economists themselves as the sage protectors of the public good; hence, we argue that making explicit and examining these assumptions is a necessary precondition to better understanding VAMs and VAM-use, with implications for educational policy.

Specifically, we were interested in looking at the assumptions surrounding all sides of the debate in an effort to understand the epistemological conditions that might be obscuring the possibility for reaching similar “truths” regarding VAMs. As such, we conducted a systematic review of the VAM literature, including both traditional (i.e., peer-reviewed academic articles) and nontraditional (e.g., newspaper articles, blog entries) sources, while focusing on the various implicit and explicit assumptions that contributed to the sources’ conditions, methods, findings, and conclusions.

Research Methods

As duly noted, we worked from the position that a group of economists have led the work, as well as the narrative, on VAMs (Chetty et al., 2014a, 2014b), especially as it relates to the asserted need for VAMs (Weisberg, Sexton, Mulhern, & Keeling, 2009), the development of VAMs (Hanushek, 1970, 1971, 1979, 2009, 2011), and the use of VAMs for teacher evaluation purposes (Harris, 2011; Kane & Staiger, 2008, 2012; Sanders, 2003; Sanders et al., 2009). Accordingly, VAMs and VAM-based policies and practices have been built upon a set of assumptions that are appropriate to the discipline of economics, such as the notion that teacher effects can be measured in isolation of or controlling for outside factors, and that teachers can (or will) make better instructional decisions given the right conditions and incentives (e.g., the promise of merit pay).

However, given the complexities that are inherent in educational systems, it might serve our understandings to apply multidisciplinary approaches to think about not only the capabilities of VAMs but also the assertions regarding the need for VAMs, the practical application of VAMs, and the potential consequences related to VAM-use. We argue that by looking at the VAM literature in terms of these disciplinary assumptions, we might better understand the nature of the debate, while unpacking the problems that can arise by depending almost entirely on one disciplinary approach to any problem.

To this end, we conducted a systematic review of the VAM literature, covering a total of 470 unique sources, both in the traditional and nontraditional academic senses. Our goal was to locate and identify the implicit and explicit assumptions that were most commonly made across pieces, while situating these assumptions within a multidisciplinary framework. In other words, and as mentioned prior, we did not limit our definition of “assumption” to the strict statistical sense of the word. Rather, we took “assumption” as more broadly conceived, to encompass that which is accepted as “truth,” devoid of empirical proof. For example, while some economists might assume that factors affecting student learning can be statistically controlled for, other social scientists might not work from the assumption that this can truly be achieved. These epistemological differences can and should be unpacked, which is what we attempted to do.

Data Collection

Sources for this study included 470 distinctly different pieces that we collected via multiple sources. These sources included traditional journal outlets, for example, via journals’ official publication announcements and emails, multiple series of Education Resources Information Center (ERIC) searches (e.g., using search terms such as value-added, value-added models, VAMs, teacher growth, teacher accountability, teacher evaluation), and via a snowball sample approach, whereby the references in one article also lead us to others. As for our nontraditional sources (e.g., newspaper articles, blog entries), we used multiple Google Alerts (e.g., using the same search terms used for the ERIC searches) to collect articles on a daily basis. Although we also employed a similar snowball approach here if, for example, a blog post leads us to a news story or journal article.

Employing these systematic albeit imperfect data collection methods on both the traditional and nontraditional ends, we attempted to safeguard against our own biases that may have otherwise subconsciously informed our selections, and also ensure that we included counterfactuals as also key. While we do not have evidence that the proportions of articles that we pulled in favor or in opposition to VAMs indeed match the true proportions resident within the entire population of traditional and nontraditional pieces out there on this topic, given this would likely be impossible to gauge, via the data collection methods that we used we believe that our article sample was appropriate, rich, and representative of the general arguments surrounding VAMs and VAM-use.

It must be mentioned again, however, that in no way is this list complete or comprehensive given the controversial nature of this policy trend and the rapidity with which pieces were being published at the time of this study, in some cases multiple times per day. To read everything that has ever been written on this topic would be nearly if not entirely impossible. Regardless, we made extensive efforts to stay as current, comprehensive, and unbiased and objective as possible, all the while remaining open to both traditional and nontraditional sources.

In order by volume, resources that we read and analyzed included peer-reviewed research studies (29%, n = 138); articles published in media outlets (21%, n = 98); organization, foundation, business, and think tank research studies (20%, n = 95); editor- and self- or author-reviewed studies (14%, n = 66); federal, local, and other promotional materials (10%, n = 46); and blog posts (6%, n = 27) (see Figure 1).

OPEN IN VIEWER

Figure 1.

Types of VAM-based articles read and analyzed for this study.

More specifically, we read and analyzed peer-reviewed research sources (29%, n = 138) including most prominently articles published in American Educational Research Association (AERA) journals (e.g., Journal of Educational and Behavioral Statistics), open-access journals (e.g., Education Policy Analysis Archives), economics journals (e.g., American Economic Review), National Council on Measurement in Education (NCME) journals (e.g., Journal of Educational Measurement), and other peer-reviewed academic journals (e.g., Education Finance and Policy).

Next, we read and analyzed articles published in media outlets (21%, n = 98) including national news outlets (e.g., the New York Times), professional press outlets (e.g., Education Week), and state and other local news outlets (e.g., the Chicago Tribune). In addition, we read and analyzed non–peer-reviewed research articles and technical reports (20%, n = 95) published by research organizations (e.g., the National Bureau of Economic Research [NBER]); foundations (e.g., the Bill and Melinda Gates Foundation); businesses, companies, and research firms (e.g., Mathematica Policy Research); and think tanks (e.g., Education Sector).

Finally, and least in terms of volume, we reviewed articles published in editorially and author- or self-reviewed outlets (14%, n = 66) including journals (e.g., Phi Delta Kappan), books and book chapters (e.g., Doug Harris’s, 2011, Value-Added Measures in Education), doctoral dissertations, and papers presented at national and other relevant conferences (e.g., the National Conference on Value-Added Modeling). We reviewed promotional and other advocacy materials (10%, n = 46) produced and released by federal (e.g., the U.S. Department of Education), local (e.g., Tennessee Office of Education Accountability), and some VAM-relevant businesses, companies, and research firms (e.g., SAS Institute Inc.). We read and reviewed blog posts (6%, n = 27) as well (e.g., the Huffington Post).

Findings

In this section, we identify each assumption (in italics) and then provide the multidisciplinary-based consensus regarding the assumption to demonstrate not only the complexities related to the assumption but also the implications for either systematically disregarding the assumption or accepting the assumption as “truth.” To be clear, we treated the 470 pieces as qualitative artifacts and, using a grounded approach, allowed the assumptions and multidisciplinary research consensuses to emerge accordingly, after which we investigated each one within the external research. Again, we are using “assumption” in the broadest sense of the word to include that which has been accepted as truth regardless of empirical support. The following assumptions have been accepted as truth, as evidenced in policy discussions and policy initiatives, despite the empirically supported concerns coming from disciplines outside of economics.

Conclusion

Not only are value-added analyses convoluted and technically intricate, the contexts within which they are to work—classrooms—are also highly complex, or rather complicated settings that vary by place, time, context, the individuals involved, and so forth. Appropriately, scholars from varied theoretical and methodological traditions have attempted to make sense of such complexities. However, as of the past few decades, the education narrative, generally, and the VAM narrative, specifically, have been more or less dominated by a single academic discipline—economics—which has largely shaped the conversation, as well as many large-scale policies surrounding this narrative. In alignment with economics-based principles, then, there seems to be a dominant view that complex matters, such as education, can be understood in parsimonious and quantifiable and, subsequently, measurably objective ways.

Teachers and teacher quality, specifically, have been reconceptualized in terms of a production function model where teachers produce a product that is consumed by students and is measurable in terms of how much “value” a teacher “adds” to the production. Given this formulation, and the economics-based epistemological position that society is made up of individuals who make rational decisions given the right conditions (Lazear, 1999), teachers are positioned as people who make good/bad decisions that result in added/detracted value to student learning, respectively. When approached from a different epistemological position, such as that of a sociologist, this notion fails to stand because teachers are presumed to be parts of larger social structures that influence not only teacher behavior but also the “outcomes” of teaching (e.g., student achievement). For this reason, the study of VAMs and VAM-based policies and practices should be approached from multiple disciplines to better capture the complexities involved in understanding the relationship between teaching and learning.

In this study, we have attempted to situate VAM practices and the VAM narrative within a multidisciplinary context by unpacking the conditions upon which VAMs and VAM-use are built. We attempted to do this by also mapping these conditions from within and onto the larger value-added literature. When taking a multidisciplinary approach, it becomes rather clear that the conditions that must stand as “true” for VAMs and VAM-use to be possible, feasible, and appropriate for that which they are intended and tasked to do, is much more difficult than is currently represented in the public narrative. The many assumptions that must be accepted and met to “truly” measure the isolated impact of teacher effects on student learning are, therefore, nearly (and most likely) impossible. Nonetheless, policies across the country are requiring school administrators not only to apply such methods to their teacher evaluation systems but also to often attach high-stakes personnel decisions to such evaluative outcomes.

This is where the true policy implications from this work dwell. If only policymakers, and perhaps more importantly those who inform policymakers, could begin to better understand the assumptions with which the educational community writ large must accept in order to accept teacher-level value-added estimates, they might be more critically aware of not only whether the assumptions are supported by research but also reasonable given the research. To critically consume and understand this information, perhaps educational policies that are more informed about versus bent on these models would help others move forward with the general teacher evaluation enterprise in much wiser and research-based ways.

Similarly, if there is any “truth” that can be gleaned from this work, it is that teachers work in highly complex institutions that are constantly affected by uncontrollable factors. Assuming that it is possible (or appropriate) to reduce teaching to a single numerical outcome is indefensible when taking into account the entire breadth of the literature on the topic. While VAM-based policies and their associated narratives would have the public assume that teaching can be simplified to a value-added estimate, other disciplines warn that doing so is impossible at best, and unethical at worst (Ravitch, 2014). Whereas a high level of methodological precision is almost always implied by those advocating VAM-use, the actual precision with which VAM estimations are made is almost never made transparent. This is particularly true in terms of transparency with data errors (e.g., missing data and data limitations), statistical errors (e.g., confidence intervals and standard errors of measurement), levels of reliability (e.g., consistency over time), and related types of evidence of validity (e.g., correlations with other similar measures; see, for example, B. D. Baker et al., 2013; Braun, 2005; Kane & Staiger, 2012). VAM estimations are, by definition, rough calculations that are largely imperfect, so “users of VAM[s] must resist the temptation to interpret estimates as pure, stable,” or even “true” measures of teacher effectiveness (Lockwood et al., 2007, p. 61; see also Rubin et al., 2004). Regardless, many state-level policymakers and other promoters still continue to endorse VAMs.

Lazear (1999) reminds us that it is the intention of the economist to make “simplifying assumptions” (p. 5) to sustain a level of statistical rigor. He also argued that economists are better at offering solutions than their noneconomics colleagues. It is feasible to assume, then, that such work has a certain appeal that might lure politicians and the public into trusting their analytical instruments and analyses. On the contrary, policy analysts, sociologists, psychologists, and other social scientists (e.g., educational researchers) tend to focus more on issues than solutions, which might be more challenging to directly translate into policy. This is not to say, however, that policy should depend solely on economics, but that policy should incorporate multidisciplinary approaches and analyses into policy consideration, development, implementation, and assessment. VAM research is but one example that demonstrates the problems that can arise with a reliance on a single discipline.