Abstract
The evaluation discipline has long been put at the service of liberal democratic values. But contemporary evaluation practice is threatened by vested interests, western democracy is under stress and internationalization has propelled evaluation towards illiberal and patrimonial states. What is to be done in contexts where democracy is absent and/or evaluation has been captured by powerful interests whether globally, within countries or within organizations? Are existing democratic evaluation approaches still relevant? Is it time to try something new? This article reviews the evidence and recommends adoption of a progressive evaluation model designed to complement, update and renew existing democratic and social justice evaluation approaches.
‘The capture of evaluation by its sponsors is the greatest threat the evaluation community has faced for some time. In fact, the credibility of the field is at risk.’ (House, 2013: 64)
Introduction
The International Year of Evaluation (2015) is premised on the notion that evaluation is an indispensable tool of good governance. According to its sponsors 1 ‘evaluation can enable leaders and civil society to develop and support better policies, implement them more effectively, safeguard the lives of people, and promote well-being for all . . . [It] can help leaders report to their constituents honestly about the impact of their decisions and to listen to feedback from every group’.
The underlying hypothesis is that evaluation favours democracy and that democracy favours evaluation, i.e. that evaluation contributes to (and benefits from) an enabling environment where authority is accountable to citizens and valid evidence is sought to guide decision making. But is this ‘theory of change’ valid in a world where private interests dominate and/or where democracy is absent, dysfunctional or in retreat? Are current evaluation approaches adapted to the challenges of rapid internationalization, rampant privatization, economic inequality and social fragmentation? Is the traditional democratic evaluation model designed half a century ago still fit for purpose?
To explore these issues this paper unearths the democratic roots of the evaluation discipline; traces its subsequent evolution; draws the evaluation governance implications of internationalization and privatization; and reconsiders existing evaluation approaches within a transformed international context. Against this background, it proposes a revision of the democratic evaluation model.
The antecedents of democratic evaluation
A common narrative about the birth of the evaluation discipline holds that it came into being at a time of ‘can do’ idealism – when widespread faith in the capacity of governments to improve livelihoods through national planning and public investment prevailed. During this imagined age of innocence (1946–57; Hogan, 2007) the United States economy experienced rapid growth and the social sciences flourished. As research methods and availability of statistical data improved, the government sought guidance regarding the design and implementation of public policies and evaluation offered a convenient transmission belt between social science departments in universities and policy makers.
In parallel, out of the ashes of World War II the victorious allies turned swords into ploughshares and recognized that they had an interest – indeed an obligation – to make the world safe for democracy and to promote human wellbeing in poverty stricken, faraway lands. To be sure these admirable goals concealed geopolitical and commercial interests but they were nevertheless significant drivers of international relations and the development idea eventually emerged as a major influence on public policies worldwide.
Initially the new evaluation discipline focused on the retrospective assessment of domestic government programs. Given the separation of powers embedded in the United States Constitution, the burgeoning evaluation discipline sprouted in a wide range of government and voluntary sector entities. By the late 1950s evaluation had become embedded in the policy cycle of education, health, public housing, law enforcement and community-based programs.
In particular the US Congress engaged in evaluation to buttress the work of its oversight committees with expert support from the General Accounting Office and other policy research agencies. State governments and local authorities joined the fray. A wide range of nongovernmental organizations and think tanks relied on evaluation to monitor, examine and influence government policies and programs. Thus, at least in principle, pluralistic and participatory principles characterized this evaluation governance model.
To be sure evaluation has had to struggle to make a mark and it has yet to acquire the status of a fully-fledged professional occupation. But there is little doubt that evaluation ‘at the creation’ was a novel and significant feature of democratic governance in the post-war era and that its adoption by development assistance agencies – starting at the World Bank in the early 1970s – gave impetus for its spread across national borders.
The evolving context for democratic evaluation
Evaluation has many facets and serves a host of purposes, large and small. But in the last analysis its value to a democratic society hinges on the extent to which it improves the effectiveness of decision making towards meeting citizens’ needs and aspirations. The democratic ideal provides for all citizens to be equal before the law and for all voices to be heard. Responsiveness and accountability – the very characteristics promoted by evaluation – are acid tests of democratic governance. But in order to turn the democratic ideal into reality a host of collective action dilemmas must be overcome. Evaluation helps to resolve them through collection, assessment and dissemination of reliable information about the social merit, worth and value of policy interventions and programs.
Under the big tent of evaluation various models vie for influence. The concepts, methods and approaches most influential at any one time reflect the dominant paradigm that drives public policy. The resulting dynamics have been aptly captured by Evert Vedung’s metaphor of evaluation diffusion. It portrays the history of evaluation as a succession of waves driven by the prevailing winds of political ideology (Vedung, 2010). Each wave reflects currently popular ideas and doctrines. After transforming the evaluation landscape the wave gradually loses energy and subsides leaving behind layers of intellectual sediment that enrich the discipline and shape its contours.
Don Campbell’s ‘experimenting society’ was emblematic of the first wave. Rooted in rationalist thinking it was science based and robustly democratic. Its flag bearers held the view that ‘the organization design which enables a disputatious community of active truth seekers is that of an open society based on trilateral democratic participation. Policy makers in tandem with members of the policy sciences community and citizens seek to discover solutions for social problems by systematically testing the plausibility of alternative policies’ (Dunn, 1997: 29).
In reaction to the positivist forces underlying this experimental wave a dialogue-oriented, constructivist, participatory and pluralistic wave surged in the late 1960s. The ‘jewel in the crown’ of this approach was democratic evaluation conceived and practiced by Barry MacDonald in the mid-1970s. The thrust of this article is that it is time to renew its promise in a transformed international context.
The neo-liberal reaction
Whereas the political winds had boosted deliberative and progressive evaluation in the 1960s and 1970s they shifted sharply to the right in the 1980s. Suddenly big government was perceived as the problem rather than the solution. A powerful neo-liberal wave swelled and engulfed the evaluation discipline: it was called upon to promote free markets; public-private partnerships and results-based incentives in the public sector. Imbued with new public management thinking the third wave supplanted the constructivist, dialogical, participatory and democratic evaluative approaches of the second wave.
We are now surfing a fourth wave. It has carried experimental evaluation to the top of the methodological pyramid. It is evidence-based and it takes neo-liberalism for granted. The scientific aura of randomization steers clear of stakeholders’ values. By emphasizing a particular notion of impact evaluation that clinically verifies ‘what works’ it has restored experimentalism as the privileged approach or the evaluation enterprise. By doing so it has implicitly helped to set aside democratic politics from the purview of evaluation – the hallmark of the prior dialogical wave.
The evidence-based wave is aligned with the interests and the world view of ruling international elites. It is welcomed by the chief operating officers of corporations as well as the leaders of authoritarian regimes. It legitimizes value free evaluation by clothing it in technocratic apparel. It gives pride of place to the achievement of pre-determined goals. It thrives on tracking progress through indicators. In an international context where multinational corporations dominate the international economy and democracy is in retreat the democratic evaluation model, never widely adopted, has become even more marginalized. It has been supplanted by a technocratic, positivist, utilization-based evaluation model highly reliant on impact assessments.
To be sure scientific concepts are precious assets for the evaluation discipline: they guarantee the rigour of evidence-based evaluations. However, in the real world of policy making a major threat to their validity looms: conflict of interest. For example private pharmaceutical companies’ tight controls over evaluation production and dissemination processes have introduced systematic biases in the conduct of drug control trials and interpretation of research results (House, 2008). Tobacco companies have conspired to deliberately confuse the public debate about smoking and co-opted scientists through funding of shadowy research organizations (Cummings et al., 2007). Conservative groups have spent up to $1bn a year on the effort to deny science and oppose action on climate change (Goldenberg, 2013).
Evaluation has not escaped the growing encroachment of emboldened private interests over public affairs. In a recent issue of New Directions for Evaluation (Griffith and Montrosse-Moorhead, 2014: 101) Ernest House stated that ‘because of structural changes in society itself, we have a new set of potential biases, a family of biases that we have to deal with or should deal with’. These structural changes include the creeping encroachment of private interests in the workings of western democracies and the spread of evaluation around the world including countries where democracy has yet to take hold. Both trends have brought to light the limits of traditional democratic evaluation approaches.
Evaluation internationalization
The Vedung ‘wave’ metaphor connotes perpetual change and fluidity but it does not preclude the co-existence of countervailing eddies, cross cutting streams and overarching tides. In fact, the recent doctrinal waves aptly described by Vedung have been amplified by a strong and powerful tide that began to surge in the early 1990’s: internationalization. This mega-trend was shaped by the growing interconnectedness of nations made possible by the new information and communications technologies. This was presaged by Eleanor Chelimsky in 1995 at an American Evaluation Association Conference in Canada, the country that hosted the first evaluation society, just ahead of the American Evaluation Association.
By then, evaluation had crossed the Atlantic and the centre of gravity of the evaluation discipline had begun to shift east and south. The United Kingdom Evaluation Society was set up in 1992. France and the European Evaluation Society followed in 1994. Surveying the evaluation terrain Eleanor Chelimsky visualized evaluation ‘becoming international in the sense of being at the same time more indigenous, more global and more trans-national’ (Chelimsky and Shadish, 1997). She proved prescient: by the turn of the century 20 evaluation associations were operating and by the end of 2012 Eval-Partners had identified 114 evaluation groupings globally.
Closing the circle Canada featured Evaluation across borders as the overarching theme of its 2013 Conference and the International Organisation for Cooperation in Evaluation (IOCE) that hosts the EvalPartners network launched a campaign which ultimately bore fruit: in December 2014 the United Nations General Assembly consecrated 2015 as the International Year of Evaluation and asked each member country to take two landmark steps: (1) strengthen its capacity to conduct evaluations, in accordance with its national policies and priorities and (2) to report back to the UN in 2016 on the progress it has made. The historic Resolution brings into sharp focus the new global politics of evaluation and it highlights the urgent need to renew and reinvigorate democratic evaluation in an international context characterized by rising economic inequality, social exclusion and popular protest.
The curse of inequality
Evaluation is not value free. It is steeped in politics and politics lie at the root of economic inequality. Through massive lobbying vested interests have used their wealth and influence to secure the lion’s share of the benefits unlocked by open trade and free capital movements. For example multinational corporations have pressed governments to negotiate trade treaties that allow them easy entry into foreign markets and to defend their commercial interests around the world. At the same time they have used the very same foreign bases as well as offshore tax heavens to escape corporate taxation.
As a result the wealthy have been getting special treatment. This has undermined meritocracy and constrained social mobility. Cartels have consolidated the hold of the few over the many. Fiscal policies have failed to correct for the inequitable distribution of wealth and income. Inequality has been fuelled by deregulation, privatization and the smashing of unions. A lot of talent has been devoted to financial engineering rather than innovative entrepreneurship in the real economy (Stiglitz, 2013).
Reducing inequality and bolstering long-term economic growth are two sides of the same coin. Even the IMF, a bastion of free market thinking, has acknowledged that in rich as well as poor countries inequality is strongly correlated with shorter spells of economic expansion and thus less growth over time (International Monetary Fund, 2014). Similarly the Organization for Economic Cooperation and Development has stressed that high levels of pay inequality reduce growth prospects (OECD, 2014) and the Brookings Institution has concluded that globally, economic disparities contribute to cycles of poverty, disease and political turmoil (Brookings Institution, 2015). ‘Occupy’ movements and popular demonstrations are threatening social unrest.
The deleterious consequences of increased inequality reach beyond income. Access to employment, health services and education are all highly correlated with socio-economic status. The 2008 financial crisis and its aftermath have widened all of these gaps and induced deep distrust of ruling elites. Differences in income and wealth attributable to effort, skill or entrepreneurship are not resented. What most triggers public anger are distorted rules of the game, predatory economic behaviour or unethical business practices. The resulting public disaffection has opened a window of opportunity for evaluation.
A rising demand for independent evaluation
The Queen of England pointed to a deep crisis in economic theory when she asked London School of Economics scholars why they did not predict or even understand the 2008 crisis. The recognition that other disciplines must be brought to bear is spreading (Brett, 2009). Evaluation can help put together the right disciplinary mix as required for individual policy and program interventions.
The era of single economic narratives is over. The industrial democracies are no longer perceived as the beacon of sustained economic progress and social harmony. Just as centrally planned economic management approaches fell into disrepute following the implosion of the Soviet Union the ideological certainties of market fundamentalism have been swept away. Here again the sceptical, systematic, evidence-based approach of evaluators is prized.
No two countries can be expected to rely on the same mix of policies and institutions to achieve prosperity. The effectiveness of development strategies differs sharply among countries so that evaluation has a comparative advantage as a tailor made practice. It uniquely connects knowledge to policy on a case by case basis. For all these reasons the demand for evaluation has been growing and evaluators have an opportunity to make a difference and help restore the flagging momentum of democratic change.
The promise of democracy
The waves of popular protest that are rocking the world confirm the appeal of democratic governance. Even the most authoritarian governments claim to be democratic. Government accountability, transparency and participation of people in the decisions that affect their lives matter to citizens. In the wake of the Soviet Union’s implosion Francis Fukuyama went as far as to argue that Western liberal democracy constituted the end point of mankind’s ideological evolution and the final form of human government (Fukuyama, 1989).
Since World War II more countries have become democratic. Whereas in 1942 there were only 12 democracies 63 percent of the 195 United Nations members (i.e. 122 countries) are now classified by Freedom House as electoral democracies (Freedom House, 2014). Belief in democracy is associated with faith in progress and confidence in the future. Citizens in democracies feel in control of their destiny. Mature democracies are not likely to go to war with one another. 2
Despite the risks associated with indirect democracy (elitism; short-term thinking; tyranny of the majority, etc.) electoral democracy remains a popular form of government. Most prosperous and stable countries are representative democracies. 3 However, democracy requires more than free and fair elections. Other political rights (the right of association, freedom from corruption and domination by powerful groups, protection of ethnic and religious minorities, etc.) are critical ingredients of democratic governance.
Not all electoral democracies are liberal. According to Freedom House, 34 out of 122 electoral restrict freedom of speech, assembly or religion and/or take steps that restrict the press or constrain civil society organizations from engaging in advocacy. This means that 45 percent of United Nations member countries (88 countries) are liberal democracies. The balance is made up of 34 illiberal democracies and 73 autocracies: together they account for 60 percent of the world population.
The presence of independent oversight and investigative public bodies and the legal protections enjoyed by whistleblowers, activists and journalists are among the criteria used by Freedom House to classify countries as free, partly free or not free. Based on such considerations the state of independent evaluation in a country may well be indicative of its liberal democratic credentials. The converse proposition according to which evaluation can help improve the quality of democracy underlies the International Year of Evaluation initiative.
From its early days, evaluation was conceived as a way to help identify and where necessary fill the democratic deficit embedded in public policies and programmes: limited transparency; weak accountability, limited participation by disadvantaged groups, disregard of social equity, etc. The principled motivation and strong ethical stance of the evaluation pioneers remain the standard by which democratic evaluation should be judged today. Unwavering ethical commitment to freedom, equality and democracy was central to their vision.
Thus Ernest House championed evaluation in the public interest and identified ‘clientism’ as a threat to the integrity of the evaluation process (House, 1991). Michael Scriven proposed judicial and consumer models of independent evaluation (Scriven, 2002). Jennifer Greene boldly asserted that evaluation is advocacy interpreted as value commitment (Greene, 1997). Karen Kirkhart probed the cultural antecedents of exclusion and prejudice that evaluation should help overcome (Kirkhart, 2015) while Donna Mertens advocated a distinct transformative paradigm for evaluation grounded in critical social theory and participatory research (Mertens and Wilson, 2012).
In opposition to the value free stance still all too prevalent within the social sciences establishment it is time for a new generation of progressive evaluators to re-discover these values and reaffirm that evaluation is not evaluation unless it promotes the public interest. Accordingly they should face up to the role that evaluation is mandated to play in the political sphere. They should draw on all existing democratically oriented evaluation approaches specifically designed to serve social justice, equality and empowerment of the poor and disadvantaged. At the same time they should acknowledge that the operating context of contemporary evaluation is quite different from that which prevailed when democratic evaluation first emerged.
Is the democratic consensus unravelling?
Fresh doubts about democracy have recently mushroomed. They should inform shifts in the future policy directions of the evaluation discipline. The notion that democracies are most successful at triggering economic growth and sustaining prosperity is no longer widely held. The advent of market socialism in China and its remarkable success in generating rapid economic growth and poverty reduction (combined with the 2008 financial meltdown and the economic insecurity still haunting western economies) have generated unease about the notion that democracy and development necessarily converge.
We are midway through a century of high and accelerating growth in the developing world which translates into economic convergence with the advanced countries (Spence, 2011). Thirteen countries have managed to grow for 25 years or more at an average annual rate of 7 percent or more using export led strategies. 4 The list includes China where the Communist Party has sustained its resolve to prevent democratization following its deadly 1989 assault on protestors in Beijing’s Tiananmen Square. It also includes Hong Kong and Singapore classified by Freedom House as ‘partly free’, Oman where political parties are banned and no meaningful organised opposition exists as well as Thailand where democracy is currently under threat.
The global progress towards democracy has stalled. State capitalism combined with distinctive civilization characteristics is on the march. In parallel disenchantment with the neo-liberal policies that have characterized contemporary economic management has spread. The hollowing of the middle class and the persistence of high unemployment are undermining faith in the liberal democratic model. Extremist political parties in western countries are getting more adherents. It is now well established that inequality within the industrial democracies is extraordinarily high and getting worse (Olinto and Saavedra, 2012).
The trend is sparing no country and no region. By now, 40 percent of the world’s wealth is owned by the richest one percent of the population while the poorest half own only one percent. Over a third of the world’s wealth is owned by the richest 0.5 percent of the world population while more than two thirds of the world’s population (68%) share only 4 percent. The world’s three richest people own wealth that is equivalent to the combined GDP of the world’s poorest 48 countries. The world’s 1210 billionaires have a combined fortune of $4.5 trillion. This is equivalent to more than half of the net worth of 3 billion adults (Forbes, 2015).
For the eighth consecutive year Freedom House’s annual report has reported setbacks. Whereas in 2013 40 countries experienced greater freedoms 54 countries registered declines (Freedom House, 2014). The goal of modern authoritarianism is to dominate all branches of government as well as the civil society and the press. Its appeal lies in promises of rapid economic development articulated by leaders who are prone to depict legislative, judicial and civil society bodies as impediments to progress.
Are current evaluation models up to the task?
In authoritarian, illiberal and patrimonial contexts as well as within multinational corporations that hold sway in the globally interconnected market place how do existing evaluation models fare? Engaging critically with the ethical dilemmas of public life, restoring the pre-eminence of values in the evaluative process, supporting democratic debate through systematic collection and dispassionate interpretation of evidence could be transformative. From this standpoint, three categories of evaluation models in widespread use today fall short.
The first group emphasizes accountability and compliance. It examines whether public resources have been applied within existing rules and regulations towards agreed goals. The second group focuses on the pursuit of knowledge. The third group is dedicated to improved decision making. All these evaluation approaches have merit and they fulfil legitimate needs but as currently practiced none is intrinsically dedicated to challenging the status quo, promoting new public policy solutions or probing the political dynamics that lie behind policies and programs.
The major aim of accountability evaluation is compliance. Goal achievement is its privileged focus. Delivering on intended goals matters but whose goals? Which values are used to ascertain merit and worth? Who wins and who loses when particular evaluation methods are selected? Since goals tend to reflect the existing power structure and embody its mental constructs the accountability evaluation models rarely address issues of inequality, social inclusion or environmental sustainability.
When the pursuit of knowledge acquisition is the main goal, the evaluation approach is often akin to social science research. Here too knowledge oriented evaluation models have tended to focus on whether or not a policy or program works. In particular the evidence movement has generated considerable momentum behind attribution oriented evaluations. Attribution is of course a legitimate and critical concern but the way it is now frequently addressed evokes the value free scientific approach so that such questions as what knowledge is being sought, why and who benefits from it are frequently passed over.
Finally, the third and by far most influential group of evaluation models is decision oriented. Under this heading the utilization- focused evaluation model championed by Michael Quinn Patton has achieved extraordinary prominence (Patton, 2008). It is premised on the notion that evaluation has little use unless it generates concrete results. This instrumental approach risks de-emphasizing the manifold ways through which evaluation influences outcomes – including indirect and long-term impacts on stakeholders’ thinking and public opinion. But more fundamentally ‘being in bed’ with decision makers threatens the integrity of the evaluation process.
To be sure, utilization oriented evaluation approaches nurture positive working relationships with decision makers, engage them in designing the evaluation and secure from them a commitment to act on the results. They often yield substantial social benefits especially when stakeholders identify the goals of the evaluation, shape the approach used to carry it out, identify, collect and interpret the data required; etc. But when all is said and done the approach fits the definition of bureaucratic evaluation offered by Barry MacDonald (MacDonald, 1993: 133): [A]n unconditional service to those government agencies which have major control over the allocation of . . . resources. The evaluator accepts the values of those who hold office and offers information which will help them to achieve their policy objectives . . . He acts as a management consultant and his criterion of success is client satisfaction. He has no independence.
Lack of independence is the major limitation of the utilization-focused approach. It is also why the model is so popular with decision makers keen to use self-evaluation to achieve their corporate or political goals. But the approach threatens the credibility of the evaluation process. If evaluators judge their success by the actual use of their findings and recommendations they will try to please and they are unlikely to be objective. Arguably the widespread adoption of utilization-based evaluation models has facilitated client control and contributed to the social timidity of evaluation agendas.
To be sure most evaluators today struggle with courage and dignity to assert and protect their right to full freedom of inquiry, to maintain their objectivity and to resist vested interests’ attempts to control evaluative agendas and findings. But contractual obligations restrict their room for manoeuvre. The vast majority operate as advisers or friendly critics to decision makers who command the heights of society. Being fee dependent they are not prone to bite the hands that feed them given the adverse consequences on their livelihoods or their career that might ensue.
It is hard to avoid the conclusion that the main evaluation models that shape evaluation practice today are not geared to democratic renewal. Of course they all have useful features and fulfil valuable services. They will all continue to have adherents and to fulfil useful social functions. But their limitations should be acknowledged. Utilization focused evaluation helps decision making but it is subservient to the values of its sponsors and lacks independence. Knowledge acquisition and accountability models are independent but they are frequently value free. David Fetterman’s empowerment evaluation as well as participatory evaluation approaches used in knowledge acquisition and accountability models help to level the policy playing field but they ultimately lack independence (Fetterman, 1994) (Table 1).
Independence and transformative value among evaluation models.
A new approach is needed although evaluators who seek to promote liberal democratic ideals through their work can draw on the distinct strands of evaluation research and thinking most relevant to the challenges they face in today’s operating environment. As stressed by Stewart Donaldson clarity about the comparative advantage of individual theories and models is critical when selecting the mix of approaches relevant to a particular context (Donaldson, 2007).
A useful guiding framework is provided by Alkin’s theory tree (Alkin, 2012). It provides a convenient tool that distinguishes between three root systems and their associated clusters of evaluation models primarily focused on: (i) use; (ii) methods; and (iii) valuing. In the table below Alkin’s typology is modified through the addition of a social justice branch helpfully grafted on the theory tree by Donna Mertens (Table 2).
Alkins/Mertens classification of evaluation models.
The limits of democratic evaluation
In sum looking at the evaluation landscape today it is hard to avoid the conclusion that utilization oriented doctrines dominate the discipline. Evaluation has become a market good instead of the public good that should be its defining characteristic. A new progressive wave would fit the needs of the time by repositioning and reorienting the discipline in strategically relevant ways. Is the democratic evaluation tradition pioneered by Barry MacDonald the answer? It defined democratic evaluation as an information service to the community: Its basic value is an informed citizenry and the evaluator acts as broker in exchange of information between differing groups . . . He offers confidentiality to informants and gives them control over the use of the information. The report is non-recommendatory and the evaluator has no concept of evaluation misuse. The evaluator engages in periodic negotiation of his relationships with sponsors and programme participants. The criterion of success is the range of audiences served. The report aspires to best seller status. The key concepts of democratic evaluation are confidentiality, negotiation and accessibility. (MacDonald, 1976: 134)
Unfortunately a neutral brokering role is ill adapted to dysfunctional governance contexts. To be sure prominent democratic evaluators assert that this limited role does not preclude influence. They point to MacDonald’s mastery of evaluation politics and his remarkable success in speaking justice to power. They argue that it is all too easy to blame the authorizing environment for poor evaluators’ performance. Thus Helen Simons distinguishes between procedural neutrality and value neutrality. With respect to the latter she stresses that neutrality does not signify passive acceptance of individual stakeholders’ perspectives.
Case studies do show how a highly motivated and principled evaluator can stretch the limits inherent in evaluations funded by power holders. However, the very same case studies confirm that overcoming the constraints imposed by a neutral intermediation role in an undemocratic governance environment calls for unique skills and heroic efforts (Simons, 2010). Even in democratic settings the independence and integrity of evaluations are hard to protect as eloquently documented by MacDonald and Kushner (MacDonald and Kushner, 2005).
It is not accidental that the democratic evaluation model emerged in the United Kingdom at a time and in a sector (education) when in MacDonald’s own words it was sensible to ‘take seriously, rather than for granted, the public rhetoric of the liberal democratic state’: ‘good enough democracy’ is the foundation on which the evaluation model was erected and even within this context MacDonald recognized the enormous power of office holders and valued the ‘partial and qualified independence’ that academic freedom provides to university-based evaluators.
Enter deliberative democratic evaluation
To help promote the interests of the weakest and the least fortunate Ernest House and Kenneth Howe revised the traditional democratic evaluation model by steering clear of MacDonald’s detailed procedures relying instead on shared values and basic operating principles that ensure inclusion of all relevant stakeholders, promote dialogue with and among stakeholders and involve stakeholders in extended deliberation processes: The three key components are inclusion, dialogue and deliberation. Inclusion means working with underrepresented and powerless groups as key stakeholders of the evaluation, not just the sponsors and powerful stakeholders; extensive dialogue increases the chances of evaluators understanding stakeholders understanding each other; and extended deliberations is careful reasoned discussion of issues, values and findings of all concerned. (House, 2005)
Both for MacDonald and for House/Howe expert facilitation is the instrument of choice, an approach that falls short in authorizing environments that do not tolerate dissent and/or for assignments that are closely controlled by evaluation sponsors: the effectiveness of a facilitative approach hinges in large part on the quality of the governance framework within which it is embedded. Unfortunately evaluation having gone global is ill served by approaches and models that only work well in democratic environments. These account for less than half the United Nations membership and even within countries classified as free democracy is often plagued by gridlock and captured by vested interests.
House acknowledges that the model is an ideal that cannot ever be fully realized in a world where unexamined interests and values, power asymmetries and exclusion of weak segments of society constitute perennial threats to the validity and fairness of evaluation studies.
For him the democratic ‘evaluator is not a passive bystander, an innocent facilitator, or a philosopher king who makes decisions for others, but rather a conscientious professional who adheres to carefully considered principles’ (House and Howe, 1999: 111).
This implies a more activist stance. In particular the deliberative democratic evaluator does offer recommendations. However the revised approach still assumes, as does Barry MacDonald’s that the foundations of a democratic culture are present. Both the MacDonald and the House/Howe models work best with principled clients genuinely committed to the public interest, in authorizing environments where communicative rationality prevails and rational discourse buttressed by ethics influences decision making. Both suffer severe limitations where governance structures have been captured by private interests.
In a nutshell, existing democratic evaluation approaches are a ‘bridge too far’ in the very contexts where they are most needed. Given their neutral procedural stance they lack the leverage needed to induce responsibility of authority in environments where power asymmetries hinder evaluation independence. MacDonald’s avowed and explicit intent was to confront ‘the political stance of the evaluation specialist in the liberal democratic state (emphasis added)’ (MacDonald, 1978) while, according to House (personal communication): ‘there is no sense trying to conduct democratic evaluations in settings that are not democratic. It’s difficult enough without having the underlying culture working against you’.
What then is to be done where governance is undemocratic and/or illiberal? This question is especially relevant on the eve of the International Year of Evaluation since 107 out of 195 countries today are autocracies or illiberal democracies.
Progressive evaluation for the 21st century
Of all the existing models displayed in Table 1 above only social justice models combine equity orientation social and independence and within this category democratic evaluation models have pride of place. Within this category Barry MacDonald’s original model and House/Howe deliberative democratic evaluation models remain highly relevant to social change within democratic contexts. But serving the public interest and speaking truth to power in situations where democracy is absent, social inequities are rampant and/or governance has been captured calls for an alternative model: progressive evaluation (Table 3).
Democratic evaluation models.
The traditional and deliberative democratic evaluation models concentrate on the relationships between evaluators and stakeholders; the protection of informants’ anonymity; the creation of a level playing field for the exercise of voice and the interpretation of results. These ‘democracy in evaluation’ principles are highly valuable in all governance contexts. But on their own they cannot be relied upon to deliver results in undemocratic and illiberal regimes whether market oriented or state driven or both. Additional safeguards are required where the logic of dialogue, inclusion and deliberation has no traction.
In such authorizing environments independence and to the extent feasible contestability are key ingredients of evaluation credibility and effectiveness. Accordingly within highly hierarchical organizations or undemocratic political systems professional autonomy emerges as an imperative criterion of evaluation excellence. In other words, the MacDonald processes as well as the House/Howe principles are necessary ingredients of contemporary democratic evaluation. But they are insufficient in authorizing environments where participation is limited, democracy is absent or vested interests dominate.
In such contexts new principles and protocols are needed to protect the independence of evaluators and the integrity of evaluation processes. This is precisely why there is room for a new model (progressive evaluation) that embraces the vision of evaluation as a morally engaged, value driven occupation focused on democracy promotion. Just as the deliberative democratic approach it would amplify the voice of citizens and resist capture by private interests. But it would also protect its morally engaged stance by stout resistance to contractual arrangements that negate evaluation independence while striving to remain involved in the political process rather than being detached from it. 5
First, progressive evaluators would refuse to comply with terms of reference that are socially insensitive or infringe on the integrity and autonomy of the evaluation process. While they would make full use of facilitation techniques they would assume full ownership of evaluation products in contexts where democracy is absent or captured – just as Barry MacDonald’s autocratic evaluators do.
Second, keeping in mind that ‘he who pays the piper calls the tune’, progressive evaluators would seek and secure independent funding for their work and they would reject work assignments under which they would be called upon to report to decision makers in charge of the very policies or programmes being evaluated. This means that the model would be highly dependent on funding ‘without strings’ provided by like-minded individuals, organizations or foundations. Novel funding and sponsorship arrangements (e.g. independent think tanks, private donations, crowd funding, etc.) would be needed to break the chains of fee dependence that currently jeopardize the integrity of evaluation processes.
Third, progressive evaluators would not shy away from alliances with advocacy groups concerned with social justice and they would become intimately familiar with the ethical principles enunciated by contemporary moral philosophers. They would speak truth to power, advocate measures and recommend actions at the service of democratic ideals of freedom and equality rather than simply provide an information service to stakeholders.
Fourth, while they would aim at scrupulous objectivity and impartiality in line with time honoured evaluation guidelines progressive evaluators would seek inspiration from social justice oriented evaluation models and be highly sensitive to the social implications of current privilege and power arrangements and their frequent intrusion in the evaluation process (e.g. top down pseudo participatory methods). They would ensure that the voices of the weak and disadvantaged are actually heard through affirmative capacity building efforts aimed at stakeholder empowerment and at levelling the playing field of evaluative practices. They would reconsider existing metrics and give adequate priority to the assessment of the global dynamics that underlie current inequality trends.
In sum even in inauspicious authorizing environments progressive evaluation would be engaged in the policy-making process as the traditional MacDonald model prescribes and it would draw on the procedural neutrality, value driven principles and ethical canons of House/Howe’s deliberative democratic evaluation model. But it would break free of the neutral brokerage role associated with both of these models. Instead it would assert professional autonomy and make full use of social networking and the new information technologies to amplify the voices of the poor and involve them in the evaluative process. It would seek independent funding and rely on external sponsors to avoid evaluation capture.
Progressive evaluation may therefore be viewed as a variant, an offshoot or an update of the traditional and deliberative democratic evaluation models. It would position the evaluation function judiciously so as to take advantage of whatever separation of powers exists. For example progressive evaluation commissioned by the legislative or judicial branches would help promote the separation of powers without which citizens’ freedoms are at risk. Wherever possible it would make use of the nascent or latent influence of the civil society through alliances with organizations that operate at arm’s length from government and corporations.
Thus progressive democratic evaluation would assert its independence in shaping evaluation agendas and selecting evaluation methods. It would emphasize compliance with ethical standards, professional autonomy, analytical rigor and engagement with citizens and the civil society. It would reject the single technocratic and minimalist narrative about the role of evaluation in society currently associated with value free, narrowly conceived conceptions of experimentalism. It would embrace mixed methods, pluralistic modes of inquiry and inclusive evaluative processes.
Ultimately as evaluation capacity development initiatives bear fruit progressive evaluators would be entrusted with attesting to the quality of self-evaluation processes. This would create synergistic relationship between what MacDonald calls autocratic and bureaucratic evaluation in ways similar to the interaction between accounting and independent auditing. Evaluation professionalization and credentialing by enhancing evaluators’ autonomy would further contribute to achieving this state of affairs (Picciotto, 2011).
Finally progressive evaluation would operate at all levels of policy making – from the micro level to the macro level. In today’s globally interconnected world it would combine local knowledge with global savvy in order to grasp the political antecedents and social consequences of complex and often informal and undemocratic governance structures associated with the new global economic order (Stern, 2013). To this end it would draw on systems thinking (Williams, 2013) and policy coherence for development ideas (Picciotto, 2005). This will call for a major commitment to evaluation capacity building in developing countries, especially emerging market countries where most poor people live (Sumner, 2013).
Conclusion
The relevance of existing evaluation models has been gradually eroded by global economic and social trends. Shifting wealth patterns and capture of decision making by private interests have hindered the march of democracy worldwide. Inequality and social exclusion have risen in developed and developing countries alike. The centre of gravity of evaluation has migrated towards developmental states that have adopted state capitalism and unitary governance models at sharp variance with the democratic ideal. In order to promote social equity and the public interest current evaluation models leave a great deal to be desired. Existing democratic evaluation models need renewal.
It is not enough for democratic evaluators to provide neutral information services, broker debate and facilitate deliberative decision-making processes. The time has come to experiment with a more activist and independent evaluation model grounded in professional autonomy, reliant on independent funding sources and tailor made to diverse governance environments. Such a model would be designed to favour social equity, level the playing field of decision making and break the chains of fee dependence. It would be grounded in moral ethical precepts, own evaluation products and engage in the policy-making process through alliances with progressive forces. It might even pave the way for a democratic wave of evaluation diffusion in the 21st century.
Footnotes
Funding
This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.