Evaluation

Embedded in the critical realist tradition, this article aims to explore the potentialities of the case study for evaluation purposes when complexity and specificity are moderate.Three objectives are pursued. First, it is stated that the focus of the evaluation effort is not necessarily the programme itself, but can be one premise upon which the programme is built. Second, taking into account Donald Campbell's contributions, it is asserted that, under favourable conditions, not only deterministic propositions, but also probabilistic ones can be rejected as highly improbable via case studies. Third, making use of the Context—Mechanism—Outcome model proposed by Pawson and Tilley, this article demonstrates the capacity of a critical case study to sustain theory building.

This article provides some reflective thoughts on current research on regulatory impact assessment (RIA) in Europe.The narrative draws on the author's fieldwork diaries. It starts from the methodological problems of identifying and analysing regulatory impact assessment in the UK, Denmark, the Netherlands and Sweden.This leads to more fundamental theoretical questions about research design, the relationship between supply and demand of research and, ultimately, the aims of academic research in this area.The conclusions draw lessons for the emerging research agenda on better regulation.The main problem is not whether `real' RIAs exist or not, since this would be equivalent to trying to fit the amoeba-like RIA `substance' into our abstract`forms'.The interesting problems are: what are the functions of different approaches to assessment, how are they used and how can they improve governance?

The objective was to assess the impact of new guidance on the conduct of narrative synthesis in systematic reviews of effectiveness, by means of a blinded comparison of guidance-led narrative synthesis against a meta-analysis of the same study data.The conclusions of the two syntheses were broadly similar. However, differences between the approaches meant that conclusions about the impact of moderators of effect appeared stronger when derived from the meta-analysis, whereas implications for future research appeared more extensive when derived from the narrative synthesis. These findings emphasize that a rigorously conducted narrative synthesis can add meaning and value to the findings of meta-analysis.The guidance framework provided a useful vehicle for structuring a narrative synthesis and increasing transparency and rigour of the process.While there may be risks with overinterpretation of study data, the framework, tools and techniques described in the guidance appear to increase the transparency and reproducibility of narrative synthesis.

Recent developments in innovation theory and policy have led policy makers to assign particular importance to supporting networks of cooperation among heterogeneous economic actors, especially in production systems composed of small and medium enterprises. Such innovative policies call for parallel innovations in policy analysis, monitoring and assessment. Our analysis of a policy experiment aimed at supporting innovation networks in the Italian region of Tuscany intends to address some issues connected with the design, monitoring and evaluation of such interventions. Combining tools from ethnographic research and social networks analysis, the article explores the structural elements of the policy programme, its macroscopic impact on the regional innovation system and the success of individual networks in attaining their specific objectives.This innovative approach allows us to derive some general methodological suggestions for the design and evaluation of similar programmes.

This article is a case study of the emergence of an evaluation culture in the public sector and particularly in education in Ireland. It suggests that the emergence of this culture was strongly influenced by external bodies, particularly the EU and, to a lesser but significant degree, the OECD. It is further argued that the continuation of systematic evaluation is still dependent on external forces, since a commitment to evaluation as a tool of governance has not taken hold among key policy-makers in Ireland. However it is postulated that, notwithstanding its arguably insecure foundations, evaluation practice has moved beyond the confines of externally funded EU programmes, which saw its first introduction into Ireland. In recent years a broad quality assurance agenda within the public service and to an extent beyond has emerged.The article concludes by making the point that an evaluation culture in a particular country is hugely contextualized and influenced by the constraints of existing ideologies and relationships between different interest groups.Thus, in Ireland, in line with the corporatist and partnership-driven approaches to economic policy and industrial relations which have been dominant in recent decades, the form of evaluation which has emerged is consensual, collaborative and negotiated.

Evidence-based evaluation of adult learning (training and capacity development) programs is essential for guiding investment decisions regarding such programs. But standard practice for evaluation of training and development programs relies on participants' self-reports regarding the quality, effectiveness, and impact of such training. More rigorous evaluation methods — including reliable measures of learning; evaluation designs that include multiple methods of data collection, scientific sampling, and randomized control trials (RCTs); and multivariate analyses — are rarely found in the training and development literature, which compromises the utility of these evaluations.This submission presents methods for increasing the rigor of evaluation of adult learning events, describes how these methods were put to use in one organization, and — drawing on these evaluations — identifies the features of effective programs.