Evaluation

In this article, we present an exemplar of the initial theory-building phase of theory-driven evaluation for the PARTNERS2 project, a collaborative care intervention for people with experience of psychosis in England. Initial theory-building involved analysis of the literature, interviews with key leaders and focus groups with service users. The initial programme theory was developed from these sources in an iterative process between researchers and stakeholders (service users, practitioners, commissioners) involving four activities: articulation of 442 explanatory statements systematically developed using realist methods; debate and consensus; communication; and interrogation. We refute two criticisms of theory-driven evaluation of complex interventions. We demonstrate how the process of initial theory-building made a meaningful contribution to our complex intervention in five ways. Although time-consuming, it allowed us to develop an internally coherent and well-documented intervention. This study and the lessons learnt provide a detailed resource for other researchers wishing to build theory for theory-driven evaluation.

Cultural historical activity and expansive learning theories were employed in a retrospective case study to investigate implementer learning and how knowledge is co-created in the early stage of an inter-agency partnership supported with developmental evaluation. Data collection methods were semi-structured qualitative interviews, evaluator field notes of implementer discursive interactions, evaluator log, and ethnographic program observations. Activity system analyses revealed a primary contraction between organizational independence and interdependence, which inhibited program implementation. Learning as negotiated work with conflicts manifesting from this contradiction evolved implementer perspectives and created a new model of interdependent implementer leadership practice. The mediating role of the developmental evaluator involved both reality testing and serving as a neutral mediating presence that supported implementers importing their resources for problem-solving as expansive learning.

Evaluation as a free-standing discipline arose out of the ashes of World War II, a time of optimism, when government turned to the academy to guide public policy. The evaluation pioneers shared a bracing vision: a search for truth in the public interest. Seventy years later the glitter has faded, and disenchantment has taken hold. Evaluation, a quintessential public good, has become a market good, and eminent evaluation thinkers are asking the same questions about evaluation that they have been routinely asking of others—with sobering results. Yet, countervailing currents and turbulent streams lie just below the surface. Once a tipping point is reached, a new wave of evaluation diffusion will begin to curl. What might it look like?

Evaluating complex interventions and policies is challenging. This is particularly true for the identification of unintended consequences, whether negative or positive. This article uses data from a workshop with policymakers and evaluators to explore the evaluation of unintended consequences. We identify three main challenges for policymakers and evaluators: being able to identify and evaluate unintended effects, to avoid creating unintended effects and being able to explain these effects. We discuss practical, political and ethical issues for each of these challenges and identify recommendations for evaluators who want to consider unintended consequences. First, use a broader range of methods to explore how policies play out; second, use theory to plan evaluations; and third, discuss both methods and theory with relevant stakeholders to make these as useful as possible. We offer novel insights into recent debates about theory-led and co-produced interventions and policies.

The core assumption of polycentric inspection is that when schools reach a certain quality threshold, they can further improve best through a coordinated, collaborative effort between clusters of schools and external agencies such as, for example, social services and training providers. The suggested role of an external agency with the respect and resources of the inspectorate is to provide stimulus and support to make the network effective. Using a bounded case study method, this research seeks to assess the potential of polycentric inspection as a tool for improving school effectiveness and outcomes. Evidence from this study suggests that this mode of evaluation has had a significant impact on improving schools, supporting teachers’ practice and, arguably, increasing student examination outcomes in the network examined. In consequence, it is suggested that these findings have wider implications for the changing conception of school evaluation and how improvement can be achieved in education.

The evaluation models described in the literature may be interpreted as prescriptive and uniform approaches to practice but, in the real world, practitioners are likely to blend aspects of different models to achieve multiple goals. Despite the commonality of pluralistic approaches in evaluation practice, literature on theoretical integration is sparse. This article guides readers through a theoretically integrative evaluation design process and explicates how different theories informed design decisions. The process integrates program theory–driven and utilization-focused evaluation with evaluability assessment and eclectically draws on principles, methods, and tools from other models. This integrative approach to evaluation aims to increase process use for intended users through shared decision-making, organizational learning, and capacity building while simultaneously producing a robust and relevant evaluation design suited to stakeholder needs and the evaluation context. The authors describe the process utilizing a case example to contribute to the literature on theoretically integrative evaluation practice.