Abstract
In an interconnected and complex world self contained ‘projects’ traditionally beloved of evaluators have become increasingly scarce! Markku Lehtonen is concerned with how we evaluate ‘megaprojects’ those large scale investments in infrastructure that so often shape our lives. These projects with changing goals, uncertain trajectories and permeable boundaries certainly challenge any standard view of ‘project’ characteristics. Lehtonen describes them as ‘organic open systems, coevolving with their context’ and requiring ‘novel evaluation approaches’. He understands megaprojects as networks that need to be mapped and analysed in each case paying particular attention to contextual factors, institutional and political as well as technological and socio-economic. Drawing on a review of literature to inform the evaluation of radioactive waste disposal in France, the author proposes an evaluation approach that acknowledges the potential of project uncertainty as a ‘source of learning, reflexivity and adaptive governance’. The author’s critique of existing evaluations of megaprojects is that they over-emphasise a particular understanding of accountability and downplay the importance of learning. In so doing he redefines rather than downplays the importance of accountability. At the heart of this redefinition is a notion of ‘network governance’ that unlike standard principal agent visions of accountability embodies ‘multiple accountability relationships’. Megaprojects, Lehtonen argues, are of their nature ‘‘heterarchical’, characterized by interdependence, lateral accountability, and organizational heterogeneity…’
Hélène Laperrière delves deep into the experience and consequences of the close engagement of ‘doers’ in front line work in difficult and often dangerous settings. In this she follows the Mintzberg and Srinivas distinction between ‘doers’ who are ‘local insiders’ and ‘helpers’ the cosmopolitan outsider. Based on her fieldwork with health-related projects in North Brazil the author is interested in what is and is not included in reports, what is censored out and why. The data for this article is ‘autoethnographic’. They derive from diaries, journals and reports prepared as part of ‘field action intended to provide help in understanding effective change’ around ‘leprosy and HIV/AIDS care and prevention’. Laperrière follows a field immersion approach, a ‘see–judge–act’ strategy that aims ‘to ensure a methodical progression from observation to evaluation – based, if possible, on collective assessments – in order to ground action decisions’. Laperrière is present successively in two roles: as a frontline worker and when she returns some years later as an evaluator. She documents her experience of how information collected about living conditions, everyday dangers from flooding and poor diet and other factors related to health such as crime, violence and corruption are ‘censored’ when report are sent to national and international agencies. She is advised that ‘context is only anecdotal and is best left out’. Universalistic and decontextualised data – such as statistical information on skin examinations is preferred. The author also documents her own ‘reticence’ to record and communicate certain kinds of information and analyses what underpins the impetus to self-censorship at different ‘moments’ in the fieldwork and evaluation process. Hélène Laperrière concludes by drawing together lessons for both researchers and evaluators who seek to ‘convey the meaning of local experiences to a public far removed from them in location’.
Are there distinctive evaluation traditions in different parts of the world? This is a question we in this journal have mused over on more than one occasion. Steffen Bohni Nielsen and Ditte Marie Winther explore this question in relation to the Nordic countries – Sweden, Norway, Denmark and Finland. Evaluators from these countries as well as external commentators have asserted that a Nordic tradition of evaluation does indeed exist, shaped perhaps by the relatively small population size of Nordic countries and a shared commitment to democratic institutions and social welfare. The authors test this assertion by analysing data on evaluation articles published in peer-reviewed, international evaluation journals. Comparisons are made throughout with the Netherlands as another relatively small country, with ‘elaborate welfare system’. The data is fascinating showing for example the preference of Nordic authors for European journals (including Evaluation) and the predominance of certain countries such as Sweden and of certain authors and institutions. However the authors conclude that ‘there seems to be little evidence supporting the notion of a shared tradition from this analysis of published articles.’ Bohni Nielsen and Winther base this conclusion on, for example, lack of cross-Nordic authorship, domain preferences and a focus on evaluation theory and methodology. They argue these patterns are not distinctive being very similar to articles published by evaluators from the Netherlands. The authors recognise the limitations of their analysis and suggest that greater attention in future to institutional contributions and funding; and to the ‘demand side’ – including budgeting, management and evaluation use – may allow for a more ‘nuanced’ conclusions. Probably a bigger question is how we conceptualise evaluation culture and whether patterns of publication are likely to be an adequate means to analyse this phenomenon – and what methods might be suitable? Contributions to this debate especially from other Nordic evaluators would be especially welcome!
Jill Anne Chouinard focuses on cross-cultural evaluation in which evaluators attempt to adapt their practice to be ‘commensurate with the culture, context and values of the program evaluation community’. For Chouinard evaluation as with any form of social inquiry is highly contextualised and relational: ‘culture and cultural context influence every dimension of evaluation’ including methodological choice and practice. The author takes an explicitly postmodern and critical standpoint in which there is no ‘single, monolithic criterion of truth that is applicable in all contexts and at all times’. There are instead ‘multiple, localized and partial truths’. In the absence of ‘objectivity’ a ‘postmodernist perspective provides a way for us to look at how we create truth, rather than merely providing another method that serves to reveal truth’. From this position, Chouinard is interested in how evaluators represent data and findings and indeed themselves in their texts; and quotes Denzin’s: ’representation is always self representation’. Whilst problems of representation are always present for evaluators Chouinard argues that this is even more critical when conducting cross-cultural evaluations. In order to illuminate this she discusses three ‘problematics’ that cross-cultural evaluators face: insider and outsider status; their identity and role; and the choice of methods. Her conclusions are radical. For example, she concludes that evaluators need to be concerned not only with how knowledge is constructed but even with ‘what constitutes a social category’ the stability and universality of which cannot be assumed across cultures. At the same time Chouinard argues that reflecting on the evaluator’s identity can itself be an ‘asset’ that enriches understanding within the dynamics of cross-cultural evaluation.
We all know that evaluative activity is uneven across organisations - some ministries and agencies are much more active commissioners and users of evaluation than others. However, according to Valérie Pattyn, attempts to explain why this is so have been inconclusive. Focusing on the Flanders administrative area of Belgium, Pattyn sets out to ‘investigate which organizations are now active in evaluations, and which conditions foster or impede policy evaluation activity.’ An exploratory phase enabled the selection of a sample of organisations that included organisations that were more and less active in evaluation. Pattyn like several recent authors in Evaluation (Verwij and Gerrits in 19.1; Blackman in 19.2; and Befani in 19.3) uses a new generation of case-based methods that are well-suited to deal with multiple explanations rather than single causes. She deploys two Boolean based methods: the Most Similar Different Outcome/Most Different Similar Outcome method (MSDO/MDSO) ‘to identify the conditions that have most potential to explain evaluation activity or evaluation inactivity’ and the ‘crisp set’ variant of Qualitative Comparative Analysis (QCA) ‘to detect the different combinations of conditions that promote or impede evaluation activity or evaluation inactivity’. All designs and methods that attempt to explain rely ultimately on some theoretical frame. The methods that Valérie Pattyn uses were built around a neo-institutional conceptual framework (ACI) that incorporates elements that emphasise both ‘structure’ and human ‘agency’.
Performance audits have become a major tool of national audit institutions in the wake of various attempts to ‘reform’ public services and public sector management. They are certainly designed to influence public management by making them more accountable. The question that Kristin Reichborn-Kjennerud raises is whether this kind of audit activity in Norway has had an impact on public policy. At a deeper level as the author notes is another question: whether evaluations for accountability purposes can ever contribute to improvement and learning. Reichborn-Kjennerud explores ‘conceptual impacts’ that might influence the way stakeholders think about policy. These impacts the author argues will rely on the use of different ‘institutional logics’, which may for example be professional, legal, managerial or democratic. Three different public debates are considered: those between the audit institution and ministers; those amongst members of parliamentary committees; and media debates involving the public following the publication of audit reports. These three levels of debate were considered in relation to two particular audit reports: one about weaknesses in university management; and the other about the disposal of real-estate and fixed assets by the Norwegian armed forces. The author concludes that the impact of audit reports on both public debates through the media and parliamentary committee debates is limited. Performance audits are more likely to influence policy through debates with implicated ministries. Whether they do exercise such influence depends on the particular institutional logics that the stakeholders bring to bear in the course of debates.