Abstract
The Past of Test-Based Educational Accountability
According to its preface, The Future of Test-Based Educational Accountability “began as a tribute to Distinguished Professor Robert Lee Linn” (p. xi). Unquestionably, Linn’s contributions to educational measurement are noteworthy, and a tribute is appropriate. It is difficult to conclude, however, that this volume does justice to Linn’s work. The collection of essays suffers from many weaknesses, some of which are described in the following paragraphs.
The first problem is a mismatch between the stated purpose of the volume and what actually appears between its covers. The editors, Katherine E. Ryan and Lorrie A. Shepard, state that “the purpose of this book is to identify and analyze key issues associated with test-based accountability independent of the particular provisions of NCLB” (p. xii, italics added). One immediately wonders if this focus was communicated clearly to the authors of the 16 chapters constituting the book. For example, in the 20-page opening chapter alone, the No Child Left Behind Act (NCLB) and its adequate yearly progress (AYP) provisions are referenced at least 54 times. NCLB is invoked as the context for nearly all of the remaining chapters, with 11 of the 16 chapters citing the NCLB context on their very first pages.
This is not to deny that NCLB has had a dramatic influence on the practice of educational accountability and the technology of measuring growth in student achievement, nor to minimize the shortcomings of that legislation. Unfortunately however, with few exceptions, The Future of Test-Based Educational Accountability focuses almost exclusively on what the contributors believe to be the flaws of NCLB. One issue related to the pervasive focus on NCLB is simple redundancy: Nearly all of the recounted flaws are well known and have been chronicled in detail elsewhere. A second issue related to that focus is that the volume takes on a decidedly backward-looking perspective instead of the more future-oriented focus suggested by its title. For example, it seems odd that the chapter by Lorraine M. McDonnell would ask “Can the Clock Be Turned Back?” (p. 47) rather than probing how we might push forward efforts to reduce achievement gaps and increase educational achievement for all students by building on the strengths of current accountability systems. Across the entire volume, although the limitations of past initiatives are chronicled, not a single chapter contains detailed, comprehensive proposals for alternative accountability systems that would address the identified weaknesses.
Even if an analysis of NCLB had been the stated purpose of the volume, a second problem is that the book presents no new insights into that legislation or fresh findings about its effects, and the overwhelming sentiment conveyed about NCLB and accountability is decidedly negative. For example, the chapter by Lorrie A. Shepard contains a section that addresses NCLB’s “Contradictions and Controversies” (p. 37), but no companion section is included that might have spelled out its “Contributions and Consensus.” In an appeal to guilt by association, the chapter begins by identifying hereditarianism and “a climate of Social Darwinism and survival of the fittest” as “precursors to today’s school accountability” (pp. 25–26). The book’s fairly uniform negativity with respect to NCLB and accountability suggests that a more accurate title might have been Why We Don’t Like NCLB . . . and, Come to Think of It, Any Strong Educational Accountability System at All.
The homogeneity of perspective on accountability is analogous to a volume that purports to address “the future of cancer treatment” in which every chapter was authored by an oncologist, or a book on “the future of personal computing” by Microsoft. Such homogeneity is especially unfortunate given that the primary intended audience of The Future of Test-Based Educational Accountability is stated to be “legislative aides conceptualizing policy or journalists following an educational assessment story” (p. xii)—audiences whose members would likely be enthusiastic consumers of a broader, more balanced, and nontechnical treatment of educational accountability and who would play key roles in facilitating advances in accountability policy development and improving public understanding of assessment.
In addition to what the volume contains, the narrowness of perspective is evident in what is not found among its chapters. For example, the volume is missing chapters by many of the major research and policy contributors to the literature on testing and accountability, such as Hess (with Finn, 2007) and Phelps (2008), who would have provided some diversity of perspective. Big gaps are apparent in the omission of chapters devoted to the gaming of accountability systems (e.g., Carey, 2007), to the increasingly popular growth or value-added models (e.g., Braun, 2005), to documenting the beneficial effects of accountability (e.g., Cizek, 2001), to other lenses that might be applied to gauging the future of test-based accountability (e.g., Loveless, 2005), to the major problem with the symmetry in current accountability systems (see Porter & Chester, 2002), or to a general defense of strong accountability systems in education. Overall, the point is that readers who seek a comprehensive treatment of accountability systems will need to seriously supplement this book with other sources.
Perhaps the biggest concern about the book lies in the implicit messages it conveys about what is to be valued and protected in American education. The book clearly articulates a pervasive concern about how accountability affects institutions and those who work in those institutions, as opposed to how accountability systems affect students. For example, much ado is made about the “serious consequences” and “severe” sanctions faced by schools (in Vonda L. Kiplinger’s chapter, p. 94), whereas the effects on children are essentially ignored. Kiplinger sees it as unfair that “schools that fail to make AYP in two consecutive years are required to offer supplemental services and school choice and to submit a school improvement plan” (p. 94). This position implicitly advocates erring on the side of protecting institutions from inappropriate labels or sanctions. But that foundational value premise is never critically examined. Are the primary intended beneficiaries of accountability systems supposed to be schools or kids? And which of these is most in need of protection? Why is it so objectionable to demand that a school be required to offer—even mistakenly—additional resources to struggling students or to explicitly focus on school improvement planning?
Finally, on the positive side, some chapters in the book do provide thoughtful and balanced treatments of a topic. Ironically, however, those chapters tend to be the ones for which NCLB (or even accountability more generally) is not the primary focus. For example, the chapter by Kiplinger and Laura S. Hamilton provides a good introduction to the concepts of scaling, linking, and equating, including the difficulties of vertical scaling and of using the National Assessment of Educational Progress as a check on the truthfulness of state assessment results. A chapter by M. David Miller provides a similarly solid introduction to reliability and validity. In his chapter, Daniel Koretz urges that the psychometric concern about error be broadened to include the traditional focus on measurement error as well as sampling error. A chapter by Stephen B. Dunbar describes current assessment designs and sampling options and highlights the applicability of the Standards for Educational and Psychological Testing (American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, 1999) to testing and accountability systems. The chapter by Derek C. Briggs and Edward W. Wiley, summarizing the challenges inherent in making causal attributions in the social sciences, would make good assigned reading for introductory research methods courses. In the final chapter of the book, Michael J. Feuer presents a general analysis of accountability systems through the lens of political economy. And, despite its nearly exclusive focus on NCLB, the chapter by Robert L. Linn could stand alone as a concise primer on current accountability strategies by one of the most thoughtful contributors on the topic.
In summary, although there are some good aspects of The Future of Test-Based Educational Accountability, the strongest chapters are those that have the least to do with that title, and the problems probed are predominately those of the past. The chapters that focus most squarely on accountability are decidedly uniform in perspective, uncritical in stance, and under-developed in terms of specific, forward-looking proposals for the future. As of this writing, The Future of Test-Based Educational Accountability can be ordered from its publisher for about $100 (or about $50 in paperback). Before ordering this volume, however, potential purchasers would do well to acquaint themselves with the range of available resources on test-based accountability, many of which they are likely to find to be more comprehensive, inclusive, and progressive.