Abstract
Evaluation evolves, responding to developments in the technical, social, and political environments in which it unfolds as well as influencing how those contexts are to be understood and assessed. Evaluation has typically been understood as a reliable means of providing some assurance in the appraisal of the value of social interventions, but has increasingly been seen as having to address a variety of uncertainties and risks not simply in those appraisals per se but in the design and management of the interventions as well. Some new ways of thinking and forms of evaluation practice linked to planning and decision making are emerging reflecting assumptions of unpredictability as well as incompleteness, instability, and a plurality of perspectives in value determination. This may signal the emergence of “post-normal evaluation.” This article explores several key features of this development.
Introduction
The call for papers for 13th Biennial Conference of the European Evaluation Society (EES) invited rethinking evaluation in turbulent times of financial, geopolitical, environmental, and humanitarian crises. This article is a bold and perhaps risky illustration of such rethinking and an invitation to reconsider the current self-understanding of evaluation theory and practice. It begins with a sketch of some signs of the social, cultural, intellectual, and political times in which we live and then suggests that our customary understanding of evaluation, what I will call “normal” evaluation, may be inadequate for the zeitgeist of our era. The possibility of rethinking evaluation, namely, “post-normal” evaluation, and what that might entail is then introduced.
Evidence of ideas and beliefs of the times
A relatively rapid scan of the global landscape for ideas, practices, considerations, and issues with particular relevance to the aim of evaluating what it is we are doing as a society yields several developments. They are presented here in no particular order of importance.
The fourth industrial revolution
Klaus Schwab, founder and executive chairman of the World Economic Forum, characterizes the Fourth Industrial Revolution as follows: “We stand on the brink of a technological revolution that will fundamentally alter the way we live, work, and relate to one another. In its scale, scope, and complexity, the transformation will be unlike anything humankind has experienced before.” 1 Among the damaging outcomes is increased inequality: The largest beneficiaries of innovation tend to be the providers of intellectual and physical capital—the innovators, shareholders, and investors—which explains the rising gap in wealth between those dependent on capital versus labor. Schwab adds, “the Fourth Industrial Revolution will change not only what we do but also who we are. It will affect our identity and all the issues associated with it: our sense of privacy, our notions of ownership, our consumption patterns, the time we devote to work and leisure, and how we develop our careers, cultivate our skills, meet people, and nurture relationships.” A recent report from Southern Voice, a think tank on sustainable development, argues that current education and research systems in most middle-income countries are not preparing a new generation of citizens and leaders to address the changes that the fourth industrial revolution will bring, including new approaches to policy design and formulation at the national and regional levels (Pellini et al., 2019).
Transformational development
The 2030 Agenda for Sustainable Development holds forth the promise of transformational development. 2 The word “transformation” is not used descriptively to refer to the level and nature of change as it occurs, but normatively to convey the desire that something about what we do and achieve and why and how should be fundamentally different in the future from the way it is now. Transformational development requires changes in social structures and relations, including addressing the growing economic and political power of elites and patterns of stratification related to class, gender, ethnicity, religion, or location that can lock people (including future generations) into disadvantage and constrain their choices and agency. It also means changing norms and institutions, both formal and informal, that shape the behavior of people and organizations in the social, economic, environmental, and political spheres (United Nations Research Institute for Social Development (UNRISD), 2016). Taking a similar line of argument, the SDG Transformations Forum (https://www.transformationsforum.net) emphasizes three types of change and their purpose—incremental (improve performance), reform (understand and change the system and its parts), and transformational (to innovate and create previously unimagined possibilities).
The state of democracy around the world
The International Institute for Democracy and Electoral Assistance in its 2017 report The Global State of Democracy points out that governments, parliaments, and political parties are increasingly viewed by their electorates as unable to cope with complex policy problems. 3 Many see a crisis of legitimacy in democratic institutions and processes, coupled with a creeping erosion of public trust, which exposes democracies as fragile and vulnerable. In its 2018 report, Freedom House (https://freedomhouse.org/report/freedom-world/freedom-world-2018) argues that democracy faced its most serious crisis in decades in 2017 as its basic tenets—including guarantees of free and fair elections, the rights of minorities, freedom of the press, and the rule of law—came under attack around the world. 4 A total of 71 countries suffered net declines in political rights and civil liberties, with only 35 registering gains. This marked the 12th consecutive year of decline in global freedom. The United States retreated from its traditional role as both a champion and an exemplar of democracy amid an accelerating decline in American political rights and civil liberties.
Complexity turn in the applied social–behavioral sciences
While there are multiple ways of understanding the term complexity and its relation to systems thinking, it is undeniable that some kind of complexity turn is happening not just in the natural sciences, but in the applied social sciences as well. This is evident in efforts both to explain the relationship between complexity theory, systems thinking, and the social sciences (e.g. Byrne and Callaghan, 2014; Midgley, 2000; Reynolds and Holwell, 2010) and to illustrate its application in contemporary concerns such as development thinking and practice (Chambers, 2017; Ramalingam, 2013). Importantly, complexity science and systems thinking is not just influencing scientific research and evaluation, but has significant implications for policymaking from resisting engineering solutions to complex problems to embracing experimentation and failure.
Reimaging measurement
The Monitor Institute, a team within Deloitte (2017) that works with social impact–focused organizations and their leaders, recently undertook a year-long, multi-funder “Reimagining measurement” initiative to highlight existing notable work in the use of data and information in the social sector. The goal was to hold up a mirror to the field and, in the process, help individual organizations and the social sector as a whole explore and influence possible futures for measurement, evaluation, and learning. The report paints a negative picture of the current performance of monitoring and evaluation.
Thinking and working politically
The Thinking and Working Politically (TWP) Community of Practice (CoP) that first met in Delhi in November 2013 aims to help better understand how to translate into operationally relevant guidance the evidence that political factors are usually much more important in determining developmental impact than the scale of aid funding or the technical quality of programming. 5 The CoP encourages political engagement as an iterative and adaptive approach. It aims to deepen and extend the analysis and understanding of the politics that drive developmental processes. It focuses on what is distinctively political about politics—power, interests, agency, ideas, the subtleties of building and sustaining coalitions, and the role of contingency. It argues in particular for more focus on recognizing and working with the different forms of power, on understanding how and where interests develop, and on the role of ideas.
Problem-driven iterative adaptation and doing development differently
The Manifesto of Doing Development Differently (DDD) states that successful initiatives reflect common principles that include (1) focusing on solving local problems that are debated, defined, and refined by local people in an ongoing process; (2) blending design and implementation through rapid cycles of planning, action, reflection, and revision (drawing on local knowledge, feedback, and energy) to foster learning from both success and failure; and (3) managing risks by making “small bets”—pursuing activities with promise and dropping others. 6 Problem-driven iterative adaptation (PDIA) is a management strategy of try-learn-iterate-adapt that reflects DDD principles and promotes local solutions for local problems, problem-driven positive deviance, and scale through diffusion (Andrews et al., 2017).
Behavioral economics
The 2015 World Development Report argued for a redesign of development policy on the basis of insights emerging from behavioral economics. 7 Paying attention to how humans think (the processes of mind) and how history and context shape thinking (the influence of society) can improve the design and implementation of development policies and interventions that target human choice and action (behavior).
Megatrends
A 2017 report from Deloitte’s Center for the Long View entitled Megatrends examines 35 drivers of change in society, technology, the environment, the economy, and politics that intersect to create 10 megatrends—universal trends affecting multiple aspects of our world and resulting in overlapping stories of the future. 8 Two that particularly caught my eye are polarization—the rise of divisiveness and divergence—and the erosion of governance—the decline of the traditional world order of direct democracy.
The state of public discourse
Finally, around the globe, we witness the effects of a toxic cocktail impairing our ability to reason together: a cocktail composed of a deadly mixture of the spread of post-truth politics, Babel that passes for public discourse, the Backfire effect, and the collapse of civic virtue. Post-truth politics means a political culture in which debate is framed largely by appeals to emotion disconnected from the details of policy and by the repeated assertion of talking points to which factual rebuttals are ignored. This is facilitated by the fact we are in the middle of a transition from a society of facts to a society of data. During this interim, confusion abounds surrounding the exact status of knowledge and numbers in public life, exacerbating the sense that truth itself is being abandoned. Data are not the same thing as truth.
The Backfire effect dispels the misconception (widely shared in the field of evaluation, evidence-based policy, and so forth) that, when your beliefs are challenged with facts, you alter your opinions and incorporate the new information into your thinking. The truth is that, when your deepest convictions are challenged by contradictory evidence, your beliefs get stronger. 9
I can only speak with some assurance about this in the United States, but there is a great deal of Babel that passes for public discourse, as examined in a book with the devastating title, “I’m right and you’re an idiot” (Hoggan, 2016) along with several others (Kakutani, 2018; Kavanaugh and Rich, 2018). The public square where we meet to discuss and debate problems of the commons should be forums for open, honest, higher quality debate but are polluted by a toxic mix of polarized rhetoric, propaganda, and miscommunication. 10
The collapse of civic virtue—The Greek word for virtue is Arete. In the Politics, Aristotle spoke of citizenship as consisting not of political rights, but rather of political duties. Civic virtue refers to the honesty and trust that enables a country to function as a considerate, forward-looking, optimistic nation. Civic virtue is often conceived as the dedication of citizens to the common welfare of their community even at the cost of their individual interests (see also Oxfam’s Briefing Note (July 2018), “Space to be Heard”).
Implications for evaluation theory and practice
So, what does all this have to do with the theory and practice of evaluation? I am not claiming that there should be one-to-one correspondence between these issues and actions that characterize the present zeitgeist and responses from the field of evaluation. Rather, my thesis is that these events, ideas, and experiences, taken collectively, present an opportunity to rethink what, for lack of a better term, I will call “normal” evaluation.
I realize the danger in claiming that one can talk about normal or typical evaluation given the proliferation of evaluation models and practices across the globe. Yet the official definitions and mission statements from evaluation associations and societies as well evaluation policies and frameworks as spelled out by major organizations including the European Commission, the World Bank, United States Agency for International Development, the United Kingdom Department for International Development, and the United Nations Evaluation Group display remarkable consistency in defining what evaluation is and what role it plays in society.
From a governance perspective, normal evaluation is a practice that exists within and simultaneously promotes a delivery mode of governance—citizens are recipients, beneficiaries of social, economic, and environmental interventions given them by government. Evaluation is dedicated to appraising the success of these interventions as solutions to society’s problems. Stame and Furubo (2019) recently described three central assumptions of normal evaluation, although they did not use that label: (1) policymaking happens through piecemeal interventions targeting specific social problems (as opposed to seeing governing as an ongoing struggle with interrelated wicked problems) and evaluation is a tool to improve existing interventions, (2) the political environment is assumed to be relatively stable, and (3) evaluations of past interventions is a primary strategy for achieving knowledge for future interventions. In its alliance with governance arrangements, the practice of evaluation has typically been directly or indirectly committed to ideas of liberalism or neoliberalism, providing determinations of the value of social programming suited to state- or market-based solutions to social, economic, and environmental problems.
Normal evaluation is wedded to notions of scientific rationality, social progress, effectiveness and efficiency in social programming, and the broad ideology of modernization. As Peter Dahler-Larsen (2012) has pointed out, evaluation is genuinely modern in its ambition—in its optimistic belief in the possibility of improving society via data gathering and rational decision making. Since its inception, normal evaluation has concerned itself with issues of rigor, independence, accountability, logical frameworks (and its refinement known as theory of change), measurable results, and assisting in risk minimization. Specifically, evaluation has typically been understood as a reliable means of providing some certainty in the appraisal of value. It has prided itself on its ability to deliver independent, external, expert determinations of value by employing the unique logic of evaluation. It is about building an evidence base of best practices that help achieve stated goals and targets while attending to both intended and unintended effects of programs and policies (although, parenthetically, I might note that the most important impacts of our choices and decisions are precisely those held to be unintentional, or side effects in the language of Ulrich Beck).
At the risk of a caricature (and borrowing a metaphor used recently by one of my colleagues), normal evaluation has aimed to play the roles of scientific watch-dog and guide dog, both guarding the key role that valid and reliable determinations of the value of policies and programs play in democratic decision making and providing those determinations, while studiously avoiding the role of lapdog—that is, resisting the pressure to act as subservient partner to management and commissioners by leading the office of scientifically looking good news.
Normal evaluation is also, with few exceptions, institutionalized and has acquired status as an industry run by large commissioners and providers (e.g. The World Bank Independent Evaluation Group, Green Climate Fund Independent Evaluation Unit, United Nations Development Programme (UNDP) Independent Evaluation Office, Independent Evaluation Unit of the Asian Development Bank, Independent Office of Evaluation at the International Fund for Agricultural Development). This institutionalization is also evident in the evaluation framework for the European Commission; a variety of evaluation capacity building efforts sponsored by foundations, United Nations Agencies, and other organizations; as well as in efforts to formulate government evaluation policies as promoted, for example, by both the American Evaluation Association’s Evaluation Policy Initiative and the Global Parliamentarian Forum for Evaluation.
Post-normal evaluation?
The characteristics of the zeitgeist identified earlier present a challenge to this normal conception of evaluation. Stated somewhat differently, my question is, “Are modernity’s blueprints for evaluation theory and practice exhausted?” Are we perhaps seeing the emergence of “post-normal evaluation?” I have liberally borrowed the idea of post-normal evaluation from the literature on post-normal science: The concept of postnormal was first introduced by Jerome Ravetz, the celebrated British philosopher of science, and the Argentinean mathematician Silvio Funtowicz (Funtowicz and Ravetz, 1993). They argued that the old image of science, where empirical data led to indisputable conclusions and scientific reasoning led to correct policies, was no longer plausible. In other words, science was no longer functioning in the “normal” way. They wrote, “Whenever there is a policy issue involving science, we discover that facts are uncertain, complexity is the norm, values are in dispute, stakes are high, decisions are urgent and there is a real danger of man-made risks running out of control” (p. 737). They described the emerging developments as “post-normal science,” which has now become an established field of inquiry. Post-normal science has been extended by philosophers and social scientists to all of society using the phrase “post-normal times” and pointing to complexity, chaos, and contradictions as defining characteristics of these times (see Sardar, 2010).
Before speculating on the outlines of post-normal evaluation theory and practice, I offer a few caveats: First, I am not talking about a revolution in evaluation theory and practice. More simply, I am suggesting that some significant rethinking of evaluation theory and practice may be taking place, and more may yet come. As Sherlock Holmes often said (quoting Shakespeare), “the game is afoot.” The field is already showing some signs of post-normal thinking, so to speak. The growing interest in developmental evaluation is an example as is the influence of ideas from systems thinking and complexity science on evaluation practice, evident in the growing number of publications devoted to the topic, particularly in this journal (e.g. Gerrits and Verweij, 2015; Larson, 2018). The recently published document “Principles for Effective Use of Systems Thinking in Evaluation Practice” (https://www.systemsinevaluation.com/principles-for-systems-thinking/) from the Systems in Evaluation Topical Interest Group of the American Evaluation Association is another example. Other signs of post-normal thinking include the South-to-South Evaluation Initiative launched in 2017 that aims to address long-standing asymmetries in global practices of evaluation and the Organisation for Economic Co-operation and Development (OECD)/Development Assistance Committee (DAC) Network on Development Evaluation sponsorship of an examination of the long-standing DAC criteria although whether this latter development is leading to post-normal thinking is suspect (see, for example, Zenda Ofir’s blog “Evaluation for Development”: zendaofir.com).
Second, there are multiple evaluation communities, for example, the development evaluation community, the environmental evaluation community, the government-oriented evaluation community, evaluators committed to indigenous knowledges, and so forth. Each of these communities may well find different salience or currency in the aspects of post-normal evaluation as discussed below.
Finally, it is unlikely that post-normal evaluation will mark a radical break with normal evaluation. Borrowing and extending Baumann’s (2000) idea that modernity has an ambivalent, dual nature, I believe we might say the same for evaluation practice. On the one hand, it is largely characterized by a need for order—to define, appraise, and rationalize the world so it will be controllable, predictable, and understandable. This ordering, rationalizing tendency is the characteristic force of modernization. On the other hand, normal evaluation has, for some time, been also characterized by change, by new ways of thinking that challenge this rationalizing tendency and heavily criticize and seek to overthrow traditional practices and forms of evaluation as perhaps most evident in some forms of culturally responsive and transformative evaluation.
Intimations of post-normal evaluation
The first indication is really a question about the very idea of innovation in evaluation, namely, “Has strategy replaced substance in evaluation discourse?” By that I mean are the primary concerns of the field of evaluation really about how evaluators go about doing their work, or alternatively, what evaluators’ work is about? Should not the central rationale for evaluation—valuing—signify something that is not just quantitative and instrumental but qualitative and substantive? Should not valuing contribute in a significant and direct way to debates about social direction and not simply be a matter of oversight and judging the performance of interventions? Is innovation in evaluation really only about how the work is done—about method? A recent webinar on innovations in evaluation sponsored by Better Evaluation, United Nations Children’s Fund (UNICEF), and EVALSDGs is a case in point. The promotion for the webinar claimed, “When existing evaluation tools, processes, methods and systems are not enough, you need to draw on innovations in evaluation, and the innovations discussed were negative program theory; triple-row logic models; data rehearsal; big data; collecting, disaggregating, and reporting by sub-groups, and rubrics.” I am not saying innovation in technique is not necessary, only that it is far from sufficient to address current challenges. The next two indications relate to how we understand the notion of resilience.
Resilience thinking as a rationality of governing
Resilience is commonly defined as the capacity of individuals and societies to recover, adapt, and thrive in the face of difficulties. But David Chandler (2014b) in his book, Resilience: The Governance of Complexity and related articles (Chandler, 2014a) argues that resilience is a mode of thinking and acting in the world grounded in an ontology of emergent complexity—the idea that life, that being, is complex, relational, embedded, and contextual.
This perspective on life is at odds with a modernist ontology that seeks to understand life in terms of its parts, their properties, and their relations. Without emergent complexity, Chandler (2014a) argues that there is no demand for resilience. Moreover, he explains that “resilience-thinking should not be understood narrowly, as merely building the capacities of individuals and societies … but more broadly, as a rationality of governing which removes the modernist understanding of government as instrumentally acting in a world potentially amenable to cause-and-effect understandings of policy-making” (p. 58).
Given this understanding of resilience thinking, the idea of governance shifts from a delivery mode—based on supply-side, goal-based instrumental policymaking—to a relational mode—governance based on understanding processes and capacities that already exist and how they can be integrated into policy understandings. This marks a shift from normal evaluation that functions as independent observation and appraisal toward evaluation as a systemic intervention—purposeful action to create change in relation to reflection on boundaries (Midgley, 2000) where boundaries determine which empirical observations and which value considerations count as relevant and which are left out or considered less important. Post-normal evaluation generates concrete and practical knowledge and is thus more closely aligned with an action-research orientation rather than independent expert-generated appraisal (Chandler, 2014b)
Politics returns to the people
Normal evaluation is associated with the idea of democratic discourse unfolding in the formal political sphere of decision making. Evaluation information feeds into and informs the decision-making process and the formation of the collective democratic will (Chandler, 2014b; see also Dzur, 2008). This is the assumption underlying the oft-quoted goal of evaluation serving social betterment, as well as efforts to build national evaluation capacity as undertaken by, for example, EvalPartners and UNDP.
However, given the idea of governance in a relational (vs a delivery) mode, politics returns to the people, to the sphere of everyday practices, interactions, and understandings (Chandler, 2014b), to viewing democracy as a way of life—as John Dewey put it—rather than composed of all the features of formal politics. This is the realm of public work. It is “public” in that the work that is visible, open to inspection, and its significance is widely recognized. It is cooperative civic work of “a public: a mix of people whose interests, backgrounds, and resources may be quite different” (Dzur, 2008). Public work means something more than the idea of public deliberation. It raises the expectation that citizens act as co-creators of a public world, citizens making a public life together (Dzur, 2008). As politics returns to the people, the independent professional stance characteristic of normal evaluation gives way to a role for the evaluator as a facilitator of public discussion in ways that share power and responsibility with citizens (Schwandt, 2017, 2018). The political scientist Albert Dzur (2008: 130) defines this role as “democratic professionalism”: “Sharing previously professionalized tasks and encouraging lay participation in ways that enhance and enable broader public engagement and deliberation about major social issues inside and outside professional domains.”
Recovery of practical reasoning
A corollary of the return of politics to the people is a recovery of the importance of practical reasoning as an antidote to the unwavering acceptance of scientific reasoning in policymaking. One current characteristic of our times is acceptance of a rationality that relies on the natural sciences and economics to the exclusion of history, culture, and politics, in short, scientific rationality “unchecked by experience, empathy, and moral grounding” (Sarewitzd, 2011). When practical reasoning is understood as situated within political communities, when reasoning is viewed not in terms of a technocratic and bureaucratic rationality but as inseparable from how we regard people and what they do in everyday and public life (Edmondson and Hüsler, 2012), we then realize how scientific reasoning must incorporate rationality as understood in social, moral, and historical contexts. Practical reasoning is precisely what is needed in situations where decisions or actions are required, yet knowledge is constantly changing and insufficient and where both expert and everyday expectations are unable to provide clear direction. The problems that evaluation seeks to address in its unique way lack recipe-like solutions and demand responses combining cognitive, emotional, social, political, and moral capacities (Edmondson, 2012: 121). Post-normal evaluation serves practical reasoning not scientific rationality. Post-normal evaluation does not aim to become an even stronger science or technology but an exercise in phronesis—that is, a moral–practical means to deliberate and act in relation to substantive issues in everyday life. 11
Co-production
A fifth intimation of post-normal evaluation, closely related to the previous two, is realization of the importance of the phenomenon known as co-production, originally introduced by the late Eleanor Ostrom (1996). Admittedly, the term has multiple meanings in policymaking, governance, and research in different fields. In healthcare, for example, co-production is described as a way of working together to improve health and of creating user-led, people-centered healthcare services. In the United Kingdom, co-production has become a mainstream term in government and public policy discourse. I am interested in co-production as a means of redefining the relationship between professionals delivering a public service and citizens. Mutuality and reciprocity replace the relationship of a client or consumer depending on an expert. Citizens are not merely beneficiaries in receipt of expert services but collaborators who contribute their knowledge, experience, skills, and capabilities to creating social innovation. It is contrasted with a transaction-based method of service delivery in which citizens consume public services which are conceived of and provided by governments. Co-production is also an exploratory space and a generative process that leads to different, and sometimes unexpected, forms of knowledge, values, and social relations. These dynamic processes take the form of interactions between individuals and services, and involve interactions between different rationales for participation and policy agendas, between different modes of knowledge production and between different kinds of value (e.g. economic, equity, social justice; Filipe et al., 2017).
Ethical accountability
The final intimation of post-normal evaluation is the discovery of the centrality of ethics to knowledge production and action. Neither the well-known interrelationship between politics and ethics represented in the profession’s sense of democratic obligation “to conduct defensible, just and fair evaluations in political contexts where the values and interest of different stakeholders often conflict” (Simons, 2006: 255), nor the usual emphasis on the ethical virtues that the professional ought to display (e.g. honesty, integrity, and so on) is the issue here. Nor am I concerned here with the familiar question of whether scientific knowledge production is in any significant way subject to moral, social, or political value judgments. (This is essentially the long-standing issue of value neutrality and the central concern when practices of evaluation and science become forms of political activism or issue advocacy.) What I am concerned with is ethical considerations inherent in knowledge production in evaluation.
Following an argument advanced by the late Paul Cillers (2005) and others, acceptance of an ontology of emergent complexity transforms the way we think about knowledge. The ethical consideration related to knowledge production is this: The failure to acknowledge uncertainty and complexity is not simply a technical error but also an ethical one. It is an ethical failure when we fail to take responsibility for our knowing. Ethics is not something that is supplementary to our understanding of the world. Ethics is always already part of what we do.
It has been said many times by advocates of systems thinking but bears repeating, one cannot study a situation in its entirety. One must choose what is in and what is out, what is important and what is not. We frame a situation in order to understand it—a frame is a perspective that makes it possible to address a situation in a particular way. Framing involves setting boundaries and thus placing limits on our understanding. And when we are dealing with the limits of our understanding, we are dealing with ethics (Cillers, 2005: 261). Because boundaries must be set, the evaluator, as a professional, bears responsibility for the consequences of setting boundaries in a particular way. As Werner Ulrich has explained, professional evaluators and social researchers cannot justify their claims about framing and boundary setting on the basis of their theoretical and methodological expertise. For, if that were so, we would cede authority to these professionals for they then must surely know better than ordinary people what is best for everyone. In reaching and offering boundary judgments, evaluators have no in-principle advantage over ordinary citizens. What they do have is an ethical obligation as professionals to pursue a “self-reflective, self-correcting, and self-limiting approach to inquiry … [one] that question[s] [an] inquiry in respect of all conceivable sources of deception,” for example, in its presuppositions, methods and procedures, findings and the translation of findings into evaluative claims (Ulrich, 2017).
Last words
I am not wildly optimistic that these indications of a post-normal evaluation will be fully realized soon, especially given that the vast majority of evaluation is conducted within or sponsored by large bureaucratic agencies and institutions where instrumental accountability and performance management reigns (Vosselman, 2012). I do not see these institutions as the cutting-edge sites that will nurture post-normal evaluation. I hold out hope for the Emerging Evaluators Thematic Working Group in EES and EvalYouth sponsored by EvalPartners for new thinking.
Post-normal evaluation can be seen, on the one hand, as a result of the failure of normal evaluation to rationalize the world and, on the other hand, as the amplification of the inevitability of and capacity for constant change which can only be managed. We will gauge the success of the future practice of evaluation in terms of whether it is resilient enough to adapt to the ontological realities of complexity, uncertainty, and contradiction in ways other than being methodologically innovative. That resilience is largely an ethical matter, of evaluators taking full responsibility for the choices they make in framing and bounding the evaluations they conduct with the public.
Footnotes
Author’s note
This is a revised version of a keynote address given at the 13th Biennial Conference of the European Evaluation Society, Thessaloniki, Greece, on 4 October 2018.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.