Abstract
Within each cabinet-level department of the federal government there are offices responsible for research, evaluation, and statistics. These offices are critical to producing evidence for social policy and encouraging its use. An evidence agenda within a department will flounder, or never even emerge, if its research office is weak. The Institute of Education Sciences (IES), established in 2002, is markedly different from the iterations of a federal education research offices that preceded it, and it has been successful in developing an evidence agenda in the Department of Education. Here, I use the IES example to address the challenge of improving the functioning of research offices in the federal government. I identify key ingredients in the success of IES that may be relevant to the reform of other federal research offices.
Keywords
The federal government plays a senior role in promoting evidence-based policy not only because it invests so much in social programs, but also because the responsibility for advancing scientific knowledge as a public good is uniquely federal. At its core, the evidence agenda blends the functions of delivering social programs, producing knowledge of how they work, and using that knowledge to improve program performance. In each cabinet-level department (and in some organizational units below the cabinet level), research offices play a critical institutional role in this process. They are responsible for research, evaluation, and statistics intended to guide program policies, designs, and implementation. 1
These research offices face many challenges, both internal and external, in carrying out their work. In too many instances they are a weak link in the chain (U.S. Department of Education 2007). This was the case for the U.S. Department of Education in the past, but it now houses what is generally acknowledged as a model research office in the form of the Institute of Education Sciences (IES). This article provides an overview of that office and identifies features of its organization and function that may be relevant to other federal research units.
How IES Is Organized
Established by the Education Science Reform Act of 2002 (ESRA), IES houses research, evaluation, and statistics branches. The president appoints and the Senate confirms the IES director for a fixed term of six years, with the option of renewal for additional terms. The director’s term spans two presidential administrations of potentially different political parties, motivated by the intent expressed in ESRA that the activities of IES be nonpartisan. As ESRA requires, the secretary of education formally delegates to the director all powers necessary to carry out the office’s responsibilities, such as appointing and evaluating staff and awarding grants and contracts.
The director’s activities are overseen by the National Board for Education Sciences, 2 which advises the director and approves research priorities. Its most important responsibility, at least with respect to the board’s power, is to submit an annual report to Congress and the secretary evaluating the office’s performance. A report that identifies management and leadership problems within IES would be a significant rebuke of the director and could lead to resignation or dismissal by the president.
ESRA makes explicit that IES is a science agency. It may seem obvious that a research office would be so. But in fact, ESRA’s language represents a clear departure from past practice. We can see this clearly in the qualifications and backgrounds of the individuals who have served in leadership roles at IES since its inception—all established researchers with doctorates in relevant quantitative fields, including psychology, economics, measurement, sociology, and political science—compared with those who served in previous versions of the education research office. That list includes historians, political operatives, civil rights activists, educators, and education policy specialists—people with talent and good intentions among them, to be sure, but some with no academic training beyond an undergraduate degree and others with graduate degrees and experience in fields not characterized by rigorous quantitative research, such as educational administration. A dramatic indication of change and progress in the nation’s education research office is that it is currently as unimaginable that someone would be appointed as IES director without solid quantitative research credentials as it would be for a science historian or workaday physician to be appointed head of the National Institutes of Health. Effective science agencies need to be led by scientists.
Among the important provisos in ESRA under which the director and the center commissioners operate are those related to the publication of the reports. Every substantive report published by IES must, by law, undergo rigorous, independent peer review. This process is managed within IES by the Standards and Review Office, which enlists qualified external reviewers.
By law, these reports are not subject to review or approval before publication by any other office of the U.S. Department of Education, including the Office of the Secretary. In practice, IES provides copies of its forthcoming reports to the secretary and other interested federal officials only after the reports are final and no sooner than 10 business days before their scheduled release to the public. This independent publication provision in the statute and the way that IES has implemented it (by providing final reports to the secretary only shortly before their public release) are foundational to the ability of IES to carry out its work without preemptive political interference. It is one thing for political appointees in the executive branch to be unhappy with a report after it is released. Their options at that point, up to and including having the president dismiss the director, are constrained by the increased public attention that would be drawn to the very report that is causing discomfort. In contrast, forewarned political leadership can put considerable pressure on IES to alter or delay a report, starting with simple requests, delivered in a friendly tone, to be helpful for reasons that make sense from a political perspective. A firewall between political leadership and the conduct and reporting of IES research is essential, once the decision to carry out the work has been made.
IES has a staff of about 180. The career professional staff and commissioners have enough expertise to function as professional coequals with the contractors and grantees that receive IES funding. The office operates with an annual line item budget of $605 million, plus another roughly $60 million from set-asides for evaluations and national activities in the budgets of programs in the Department of Education that are outside of IES, for example, Title I of the Elementary and Secondary Education Act. These set-asides fluctuate based on the line item program appropriations in which they are embedded. Although the IES budget is small compared to the $68 billion discretionary budget of the Department of Education, it has been steady (U.S. Department of Education 2017).
IES Products and Activities
IES has a number of signature products and activities. A review of these products and activities yields a clear idea of why IES has been successful.
The National Assessment of Educational Progress (NAEP)
Every two years, NAEP assesses student achievement in math and reading in grades four and eight, using large representative samples in each state of the nation as well in twenty-one urban school districts. And every two years, education reformers, politicians, educators, and education beat reporters anxiously await the answer to the question, “Is NAEP up?” NAEP assessments in reading and mathematics are the epitome of modern large-scale cognitive assessments from the perspective of methodology, management, and dissemination. It is a demonstration of what sound science in education looks like, and proof that government can and should play a central role in the production of such large-scale assessments.
The What Works Clearinghouse (WWC)
The WWC reviews research on programs, products, practices, and policies in education to determine how they affect important outcomes. For example, does Everyday Mathematics®, a curriculum for pre-K through grade six, raise student achievement in math? The WWC makes its determinations through explicit, well-documented protocols for systematic review that cover how to search the relevant scholarly literature, grade the studies’ methodologies, and summarize and report the results. Since its inception, the WWC has reviewed roughly 10,000 individual research reports and identified about 350 that both meet rigorous methodological standards and find positive impacts on intended outcomes. The WWC also produces practice guides, which go beyond evaluations of specific interventions to offer coherent recommendations for educators on important broad goals, such as how to improve math instruction for young children. The practice guides’ recommendations are grounded in and graded by the quality of the evidence behind them, as determined through WWC’s individual study reviews.
Impact evaluations of federal programs and practices
Since IES was founded, the Center for Education Evaluation and Regional Assistance has launched forty-one large-scale evaluation studies designed to provide unbiased estimates of the impact of federal education programs using randomized trials and regression discontinuity designs. There were no more than two such evaluations carried out in the entire prior history of the department (Mosteller and Boruch 2002).
Frequently, program advocates and opponents have a lot riding on the results of IES evaluations. For example, an IES report demonstrating the negative impact of the DC Opportunity Scholarship voucher program on student achievement put the Trump administration and Secretary Betsy DeVos on the defensive at the beginning of their administration, just as they were rolling out budget plans to support the expansion of similar voucher programs nationwide: “When Secretary DeVos’s own Department’s independent research office tells her that siphoning taxpayer dollars into private schools has a negative impact on students, it’s time for her to finally abandon her reckless plans to privatize public schools across the country,” said Democratic Senator Patty Murray (Askarinam 2017).
That IES can release such reports without bringing political retribution on itself is testimony to the rigor of the research and the stick-to-the-facts nature of IES reports. For example, in response to Senator Murray’s thrust, Secretary DeVos parried by saying that “DC’s traditional public schools have not suffered as a result of being part of a system that allows choice. Rather, they have greatly improved since the 2004 inception of the District of Columbia Opportunity Scholarship Program” (Askarinam 2017). Thus, Secretary DeVos shifted the focus to something the evaluation did not cover—the impact of vouchers on traditional public schools—rather than quarreling with the findings.
Annual education research grant competitions
Since its inception, IES has awarded $2.7 billion, largely to university-based research teams, through roughly 1,320 competitive grants. About one-quarter of these grants (338) are for randomized trials, which, before IES came along, many thought were somewhere between impractical to impossible to conduct in education. 3
The yield from this taxpayer investment has been considerable across topics as important and diverse as student cognition and postsecondary success. For example, a large-scale, multistate randomized trial examined the impact of a number of scholarship programs that gave low-income postsecondary students a modest financial stipend contingent on their meeting certain academic benchmarks, for example, registering for the required number of courses to make expected progress to degree completion. The programs produced positive impacts—students randomly assigned to the scholarship programs earned more credits toward their degrees and had higher graduation rates than their control-group peers (Mayer et al. 2015).
Predoctoral interdisciplinary research training programs in the education sciences
Before IES, too little high-quality, relevant education research was available. The implicit mission of IES, as seen by its founders, was to correct that. Two primary supply-side strategies were used to do so. The first was the annual grant competitions described previously, which were intended both to encourage individuals already doing high-quality education research and to draw in top researchers, such as leading labor economists, who had been working on topics outside of education. The conceptual model was “If we build it (orderly, predictable, fair, and well-capitalized research competitions), they will come.”
The second strategy was to expand and reconfigure the pipeline for new researchers. Before IES, training in education research was largely the province of schools of education. But schools of education were not equipped with the faculty or the values that aligned with training graduate students to carry out methodologically rigorous education research focused on questions relevant to education policy and practice (Levine 2007). The predoctoral training grants are awarded to universities willing to adopt new training programs that can produce graduates with the skills to compete successfully in IES grant competitions and to take positions in the education research and evaluation organizations.
To date, IES has dispersed $161 million through thirty-six predoctoral training grants. The graduates of these programs have largely alleviated the shortfall in the supply of researchers able and willing to carry out high-quality education research. They now constitute a substantial portion of the education researchers who compete successfully for research grants, fill new positions at contract research firms, and work in other venues where research skills are critical.
They have also influenced schools of education themselves to hire and promote faculty with quantitative research skills who can compete for IES and other federal research grants (Levine 2007). These faculty, in turn, alter the interests and aptitudes of students who are admitted to the schools’ doctoral programs through those faculty’s roles in selecting and mentoring the entering classes. It is a virtuous circle that pulls schools of education back toward the canons of the empirical social sciences.
Why Is IES Important?
Of course, education research, evaluation, and statistics funded by the federal government should be appropriately conceptualized, effectively managed, and usefully disseminated. To the extent IES does these things competently, it is important on the face of it. But there is more to it. The success of IES compared with the failure of its predecessors offers lessons for the organization and legislative authorities of other federal research offices.
The Cooperative Research Act of 1954 authorized the Office of Education, then a component of the Department of Health, Education, and Welfare (HEW), to make grants to universities and state education agencies to carry out education research. Since then, several federal entities have been responsible for education research.
In 1972, the Bureau for Research within the Office of Education in HEW was superseded by the National Institute of Education (NIE) within the Education Division of HEW. Then in 1979, responsibility for education research and evaluation was shifted to the Office of Educational Research and Improvement (OERI) in the newly established Department of Education. OERI morphed several times until it was replaced by IES in 2002.
The churn in organizational structures responsible for education research from 1965 through 2002 occurred in large part because Congress and others were dissatisfied with the management of the federal investment and its yield to education. This dissatisfaction began early on—in 1967 a U.S. House Special Subcommittee on Education concluded that the federal education research enterprise had failed to provide useful products (Vinovskis 2001). In 1975, the National Council on Educational Research concluded that the NIE was “not paying sufficient attention to considerations of quality and relevance” (quoted in Vinovskis 2001, 96). In 1987, the assistant secretary for educational research, Chester Finn, concluded that he did not “see any evidence that the taxpayers’ substantial investment in labs these past two years has yielded any results of any sort” (Vinovskis 2001, 48). A 1992 report on OERI by the National Academies of Sciences concluded that “OERI needs to be rebuilt” (Atkinson and Jackson 1992, 2).
Contrast these appraisals with the Office of Management and Budget’s (OMB’s) conclusions about IES through its performance assessment rating tool (PART). 4 PART was a rigorous effort to evaluate the effectiveness of federal programs, including a requirement that each program specify the results for which it would be held accountable and periodically report evidence on its success. PART won Harvard University’s Innovations in American Government Award, and served as a model for related efforts in other countries (Government Innovators Network 2005). As of 2008, only 17 percent of federal programs had been rated effective (the highest score on PART). Only 6 percent of Department of Education programs were so rated. IES was among these. About IES, PART concluded: “Since its creation by the Education Sciences Reform Act of 2002, IES has transformed the quality and rigor of education research within the Department of Education and increased the demand for scientifically based evidence of effectiveness in the education field as a whole” (IES 2008, 3).
Features of IES That Have Produced Success
The federal offices that managed education research in the last half of the twentieth century failed to produce much in the way of usable scientific knowledge or to support meaningful evidence-based education reform. The consistently poor outcomes across so many iterations of leadership and organization suggest that design failures were primarily to blame.
By contrast, the operational success of IES has persisted under two long-serving directors, several leaders delegating the responsibilities of the director, and three presidential administrations. This stability in performance despite changes in leadership suggests that IES has design features that set it apart from its predecessors and that might be widely relevant for other federal research offices. What are those features?
A clear and focused mission
Consider the legislative mandate that established the Office of Education Research and Improvement in 1979: “There shall be in the Department an Office of Educational Research and Improvement, to be administered by the Assistant Secretary for Educational Research and Improvement. … The Assistant Secretary shall administer such functions concerning research, development, demonstration, dissemination, evaluation, and assessment activities as the Secretary shall delegate” (Department of Education Organization Act 1979).
Compare this with the Education Sciences Reform Act of 2002, where the mission of IES is defined as “expanding fundamental knowledge and understanding of education … in order to provide … reliable information about … the condition and progress of education in the United States. … educational practices that support learning and improve academic achievement and access to educational opportunities for all students; and the effectiveness of Federal and other education programs” (Education Sciences Reform Act 2002).
That statutory clarity of the IES mission, compared to the emptiness of OERI’s, is critical to the success of IES and important to any federal agency that is expected to conduct and support scientific research.
Statutory independence
Federal agencies whose work is highly relevant to politics and policy cannot succeed unless they are shielded from the inevitable disappointments that their findings will produce in the executive branch and among influential people outside of government. Of all the federal research offices that carry out program evaluation, only IES has statutory independence. Other agencies seek some of the benefits of such a shield through formal principles of conduct that they hope political appointees up the line will honor (National Academies of Sciences, Engineering, and Medicine 2017). Though it is better to have principles in writing than not, those principles have no authority beyond a voluntary commitment. Here Congress could play a significant role by crafting legislation that gives IES-like independence to similar federal research units.
Highly qualified staff
Producing high-quality research, evaluation, and statistics requires highly qualified staff. Many if not most of the federal research agencies scattered through the federal bureaucracy have no staff members who, for example, understand at an expert level how to carry out an impact evaluation or a large sample survey (National Science and Technology Council 2008). Short of giving lots of small research offices the resources and motive to employ such experts, Congress or the executive branch might consider a cross-agency entity, perhaps housed in the OMB, that could help research offices with the responsibilities that they do not have the human capital to carry out well on their own.
Strong internal quality controls
The external review processes in IES have been essential to ensuring the quality of its reports. Further, the Standards and Review Office, which makes decisions independent of the IES director and commissioners, provides an additional layer of political cover for high-stakes, politically charged products. Such independent review is required by the IES statute. Other federal research agencies without such a statutory mandate may find in worthwhile to incorporate independent peer review as part of their standard operating practices. OMB might encourage or require peer review by other federal research offices.
High-impact products
Increases in funding, recognition, and influence for research, evaluation, and statistics follow from having products of those enterprises that draw attention and inform, while being rock solid in terms of quality and free from any form of perceived partisanship. Federal research units that feel underappreciated and underfunded might consider whether they can invest more in products that become central to administrative decision-making at higher levels of government and garner media attention as a result.
Investments to strengthen the workforce
Every research office depends on the health and vitality of the academic and contract workforce that is relevant to its work. Investing in that workforce, as IES did through its predoctoral training grants, can pay many dividends.
Sufficient and predictable funding
A research office cannot perform well over the long term if most of its activities are necessarily short term because of fluctuating and insufficient funding. Evaluation and statistical data collection done on the cheap may be worse than no data collection at all. And when a program of work cannot be programmatic because funding forces all projects to be one-offs, it will fall far short of the yield from projects that can build on prior efforts and set the stage for future work. Congress could directly fund research, evaluation, and statistical work and ensure that the funding remains stable or increases. However, a statutory percentage set aside for research, statistics, and evaluation in the line item program budgets of cabinet agencies would have wider impact.
Conclusion
The nation needs policy-makers, practitioners, and concerned citizens to see the value of rigorous evidence and to push for policies that cannot wait for evidence to be tested as they are implemented. This requires a transformation—we must become a learning society, committed to learning how to improve our social programs. IES has been at the center of that transformation, because that is its statutory mission and there is substantial, bipartisan political support for evidence-based education policy (IES 2007). IES and the education evidence-based enterprise as a whole have made substantial progress in this regard over the past 15 years. Much work remains, and many challenges lie ahead. But the fact that no one thought it would happen, and then the corner was turned and progress came quickly, offers both reasons for optimism and lessons for practice for the many other areas of the federal government in which research, evaluation, and statistics are central to mission but lag in performance.
Footnotes
Notes
Grover J. (Russ) Whitehurst is a senior fellow in economic studies at the Brookings Institution. He was previously director of the Institute of Education Sciences, chair of psychology at Stony Brook University, and vice president of the Merrill-Palmer Institute. He was also the last person to serve as assistant secretary for educational research and improvement in the U.S. Department of Education (2001–2002). His current areas of focus include early childhood policy and school choice.