Reflections on the National Mathematics Advisory Panel Final Report

Commentaries in This Issue

This issue features commentaries by a number of researchers who seek to redress what some see as an imbalance in the report. These articles are written from different perspectives: (a) mathematics education research (articles by Boaler; Borko & Whitcomb; Cobb & Jackson; Confrey, Maloney, & Nguyen; Lobato; and Thompson), (b) a more expansive view of the mathematics necessary for lifelong learning (Roschelle, Singleton, Sabelli, Pea, & Bransford), (c) other research genres (Greeno & Collins), (d) a more comprehensive view of the role of randomized clinical trials (Sloane), (e) assessment practices (Shepard), and (e) policy implications (Spillane). A rejoinder to their comments is provided by Benbow and Faulkner (the vice chair and chair of the report, respectively).

Borko and Whitcomb

As journal editors, Borko and Whitcomb are aware of issues of quality in education research and sympathize with the task that faced the NMAP. However, they disagree with the Panel that randomized clinical trials represent the summative research event and that other methods are merely preparatory for this event. They argue that different research methods have different functions and answer different research questions. Limiting research findings to quantitative studies, they assert, provides a distorted view of the field of mathematics education research, particularly theorizing about the role and impact of teachers. For example, considerations of teachers’ mathematical knowledge for teaching in the report were narrow, and the integrated act of teaching and learning was fragmented across task group reports, diluting the construct and potentially distorting the policy implications for teacher education.

Cobb and Jackson

Cobb and Jackson articulate a concern of many of the other commentators that the Panel over-relied on quantitative studies, particularly randomized trials, in defining the quality of mathematics education research. They assert that the imposition of such a filter betrays an ideological position (i.e., experimentalism) that serves to downgrade the importance of context, particularly around issues of equity and measurement (a topic of direct focus by Shepard). They fault the Panel for valuing regularity over process views of causality (citing Maxwell, 2004) and recommend expanding the mathematics topics of interest to include statistics.

Thompson

Thompson faults the Panel for adopting a narrow view of research methods that eliminates from consideration vast amounts of research literature, while retaining for itself a subjective character in its judgments. He agrees with Cobb and Jackson on their characterizing the report as ideological but also sees the choice of research filter as having political overtones.

Boaler

Boaler believes that the authors of the report knowingly created definitional dichotomies (such as between teacher- and student-centered instruction) that undermine scholarship in this area and mislead practitioners. She argues that “gold standard” randomized trials (e.g., Raudenbush, 2005) are unsuited to field settings in education. Boaler shares Holland’s aphorism (cited in Briggs, 2008, p. 20) that “a randomized controlled experiment is just a quasi-experiment waiting to happen.” She believes that research methods must reflect the phenomenon of learning and teaching mathematics that is realized in messy social settings and that is shaped by, and requires, nontrivial amounts of time, resources, and personnel to enact. By insisting on a filter of strict randomized studies, the Panel makes the ideal the enemy of the good—the good in this case being quasi-experimental and qualitative studies more suited to the realities of complex field research.

Lobato

Lobato faults the Panel on its valuing of regularity over process models of causality. She shows that detailed attention to disciplinary definitions of mathematics concepts, together with students’ misconceptions of these concepts, has clear implications for interpretation of the results of regularity studies. Her analyses contribute directly to Shepard’s concerns about the quality of mathematics assessments, which, if in doubt, cast doubt on the dependent measures that form the backbone of the recommendations from randomized trials. Thus the downplaying of qualitative analyses in the report impoverishes its ability to form causal explanations of phenomena (Shadish, Cook, & Campbell, 2002). A key concern for mathematics instruction is not only assigning meaning to symbols but also developing capacity to transfer mathematical ideas to everyday life, to problems in engineering, and to other areas of science. Lobato argues that the report adopts an information-processing model of transfer, which is inadequate to the task of accounting for failures to transfer learning; she proposes alternative models. Lobato also criticizes the Panel’s understanding of Vygotsky’s theory by noting that core assumptions vary across the theories and are not interchangeable. She refuses to accept the Panel’s assumption that the validity of Vygotsky’s theory could be tested by fitting it to the assumptions of the behaviorism operative in the use of randomized clinical trials.

Shepard

Shepard criticizes the report on a number of grounds that have significant implications not only for the construct validity of the measures used but also for the validity of large assessments of mathematics learning. Shepard notes the problem of creating a national test in the absence of a national curriculum, a problem she claims is addressed neither by using narrow measures of mathematics skills nor by attempting to test too many topics. She critically reviews the Panel’s take on formative assessment. She faults the Panel for not addressing the problem of teaching the test, where instruction targets the indicator, not the curriculum. She points out how difficult it is to write good items and design reliable and valid tests, and she holds that the Panel lacked the requisite expertise in this critical area. Shepard dismisses the Panel’s claim that multiple-choice and open-response items are interchangeable and asserts that the form of the test can directly influence what is assessed (for a physics education example, see Mestre, 2000).

Roschelle, Singleton, Sabelli, Pea, and Bransford

Roschelle and his colleagues approach the Panel’s report from the perspective of a National Science Foundation–supported Science of Learning Center on learning in formal and informal settings. Roschelle et al. suggest that the report should (a) embrace the use of a broader set of research methods, (b) seek a more expansive view of mathematics as a modeling tool both inside and outside of the classroom, (c) advocate for a theoretically integrated view of resources for improving mathematics learning (including the use of technology, teacher knowledge, assessment practices, etc.), (d) recommend moving away from methods that assess the impact of any one factor in isolation, and (e) call for a better integration of the findings of the Task Group on Learning Processes into the synthetic analysis of the report and its recommendations.

Greeno and Collins

Greeno and Collins also fault the report for its narrow filter for research. Like Thompson, they believe that the judgment as to what constitutes quality research and evidence is best determined by the entire community of researchers in a field, and not by governmental edict. They argue that important research methods that have grown out of mathematics education research itself (e.g., Kelly & Lesh, 2000) and newer methods that have arisen from the learning sciences, such as design research (e.g., Barab, 2004; Kelly, 2004; Kelly, Lesh & Baek, 2008), together with sociocultural (or activity-theoretical, or situative) framings of learning and teaching, cannot be simply dismissed as nonscientific activities. Greeno and Collins cite a National Academy of Education (1999) project as a viable alternative to the Panel’s approach to improving mathematics achievement.

Sloane

Sloane is generally sympathetic to the research direction chosen by the Panel. He values the “what works” framing of the Panel. He faults the use of randomized trials of small experimental studies and large field studies as a means for answering “what works” questions on the grounds that the approach is piecemeal. He argues for a more explicit continuum of research projects in a larger portfolio or program of research. He outlines possible phases of research that he believes better map the drug trial model from medicine to the practice of teaching and learning. In the absence of such an articulated phase model, he asserts that the Panel conflated efficacy and effectiveness trials in its call for more experimental research.

Confrey, Maloney, and Nguyen

Drawing on prior experience in writing national reports, Confrey and her colleagues outline six conditions that they claim lead to high-quality reports. They claim that three conditions were violated in preparing the NMAP report: (a) The composition of the panel was unbalanced with respect to the appropriate expertise and thus misrepresented the consensus view of mathematics education research; (b) the report failed to articulate and fairly and consistently apply coherent methodological standards; and (c) the articulation between the task group reports and the NMAP report was inaccurate and lacked intellectual integrity. These flaws, they claim, weakened the report. Confrey et al. end their article with an anecdote of the misuse, as they see it, of the report at the state level in Missouri.

Spillane

Spillane examines the NMAP report from the perspective of policy and implementation. He notes that the report is primarily an exhortative document (one without the force of law) and, as such, depends on the goodwill and commitments of hierarchies of actors for its realization. These actors include not only those at the federal level but also state actors, school policy makers and implementers (including teachers and parents), and a growing set of “extrasystem” actors such as book publishers, testing companies, and tutoring services. Spillane argues that the impact of the report is unlikely to be direct and immediate. Rather, it will be hindered by differing interpretations of the meaning of its recommendations and their incomplete and inconsistent implementation, by variable application of the use of incentive schemes, and particularly under the current economic conditions, by scarcity of resources.

As the guest editor, I hope that this special issue of Educational Researcher will not only present diverse perspectives on substantive research in mathematics education but also contribute to the current discussions of how methodological approaches are matched to and shaped by questions and objectives. The articles herein are intended to broaden the terms of the ongoing discussion of effective instruction as well as to draw sharp distinctions where there is disagreement.