Evaluation framework for the Joint Action PreventNCD: A comprehensive approach for a complex and multi-layered public health project

Abstract

Aims:

The evaluation of the Joint Action on Cancer and Other Non-Communicable Diseases (JA PreventNCD) is considered a relevant contribution to guiding future public health activities in European countries. It will assess the project outputs and evaluate their contribution to achieving the outcomes of the joint action (JA), as well as their impact at both the European Union (EU) and member state (MS) levels.

Methods:

The evaluation, informed by a logic model, assesses project-level outputs as well as outcomes and impacts at the MS and EU levels using mixed-methods approaches. This involves data collection and analysis to address the evaluation questions. Accompanying evaluation approaches will include monitoring selected implementation case studies and examining the influence of contextual factors across diverse settings.

Conclusions:

The evaluation will provide essential insights to guide the adaptation and scaling-up of interventions to prevent non-communicable diseases (NCD) across diverse settings and countries. These insights can assist decision-makers in designing and implementing future public health initiatives.

Keywords

Evaluation JA PreventNCD implementation case study knowledge translation capacity building

Background

Evaluating public health initiatives is essential for understanding their effectiveness in achieving desired health outcomes and for guiding improvements [1]. Many authors have suggested definitions for evaluation and proposed tools and approaches for assessing prevention programmes [2 –5]. Across these approaches, the core component of an evaluation is the collection of credible evidence on both the implementation and the effectiveness of a programme or policy. The focus of an evaluation should be guided by the objectives and logic model of the studied intervention, and should generate recommendations that enable stakeholders to act on the findings [2 –5].

Evaluation results are important for decision-makers for economic, ethical, and accountability reasons [6]. Therefore, the European Commission has a long tradition of evaluating, among others, their Framework Programmes for Research and Innovation [7]. European Union (EU) joint action (JA) plans are required to include arrangements for monitoring and evaluating their programmes [8]. Accordingly, when the JA PreventNCD started in 2024, a comprehensive evaluation concept was developed to better understand its effects across the various elements, tasks, and stages of the project.

The JA PreventNCD is a complex public health project funded for a four-year period from 2024 to 2027. It is structured into 10 work packages (WPs), with four cross-cutting WPs (coordination, communication, evaluation, and sustainability) and six technical WPs (regulation and taxation, healthy living environment, social inequalities, monitoring, health in all policies, identify individuals at risk). Each WP consists of numerous tasks, subtasks, and pilot projects, all with distinct aims and proposed outcomes. The project addresses health topics broadly comprising tobacco and alcohol consumption, physical activity, and dietary behaviour. The activities range from compiling reviews to conducting concrete social marketing or environmental interventions in various settings across 25 participating European countries [9].

The JA PreventNCD can be considered a complex initiative, comprising several sub-initiatives across different domains, each with its own objectives. This complexity requires an evaluation approach that captures perspectives and outcomes at the micro level (e.g., subtasks and pilot projects) [10, 11]. A narrow evaluation design would not reflect the multiple layers, varied contexts, and diversity of interventions within the JA. By tailoring the evaluation to the scale and scope of the JA, the evaluation results can be summarised into valuable insights to inform future large-scale public health initiatives.

State of the art in evaluation research

Several evaluation frameworks demonstrate the complexity of multi-component preventive programmes, such as the RE-AIM Framework [12], the organising framework proposed by Bauman and Nutbeam [4], the Health Promotion Evaluation Framework [2], and the guidance for evaluating complex interventions by the UK Medical Research Council [10]. Based on these frameworks, we identified a sequence of steps to follow when planning an evaluation:

Step 1. Describe the project and its context, clarifying the initiative’s aims and its linkage with other initiatives.

Step 2. Develop a logic model that links the elements of the interventions to the intended outcomes.

Step 3. Identify the relevant evaluation questions that are likely to concern all stakeholders.

Step 4. Develop indicators and measures for (a) process, programme fidelity, delivery of programme elements, and change mechanisms, and (b) outcomes and impact.

Step 5. Decide on the procedures and methods needed to obtain meaningful information.

Step 6. Collect data and gather evidence, develop case studies, and audit environmental changes and policy implementation.

Step 7. Analyse and interpret the data together with stakeholders.

Step 8. Generate conclusions, make recommendations, and disseminate the findings to funding agencies and other stakeholders.

A key challenge in large, multi-layered public health initiatives is choosing the appropriate outcome measure or evidence of change [10]. A logic model may identify possible outcomes and impacts across a range of domains. It is also advisable to work with involved stakeholders to assess which outcomes are most important. In complex interventions, it is recommended that outcome measures focus on changes within the system rather than changes in individuals. This could include the introduction of policies, changes in social norms, or normalisation of practice [10]. Evaluating outcomes and impact alone cannot fully explain the mechanisms behind an intervention’s success or failure [11]. Therefore, in addition to assessing the outcomes of an intervention or programme, it is also useful to investigate the fidelity and quality of implementation, mechanisms of change, and contextual factors. Implementation research offers support for investigating these processes and system changes, focusing on real world conditions where context plays a central role [13 –15]. Recent findings from the World Health Organization (WHO) highlight the role of social determinants of health and existing policy environments as critical contextual factors affecting NCD prevention outcomes [16].

When evaluating a complex initiative, there is no single ‘gold standard’ study design. A purely quantitative approach, using an experimental design, has not proven adequate for complex intervention research [10]. Qualitative and mixed-methods designs can help answer questions beyond effectiveness and provide contextualised insights into how an intervention induces change, including identifying barriers to delivery and change [10, 11]. This is especially important for evaluating large international NCD prevention programmes, as it provides essential insights to guide the adaptation and scaling-up of interventions across diverse settings and countries.

Aims of developing the evaluation concept

The goal was to develop an evaluation concept that addresses various requirements and needs. The concept should fulfil the following core conditions:

(a) The main evaluation objectives align with the main goals of the JA PreventNCD.

(b) The evaluation focuses on the overall JA PreventNCD, as well as on the WPs and tasks.

(c) The evaluation concept is based on a logic model of the JA PreventNCD. It includes outcome and impact indicators, while also considering more proximal indicators, such as output indicators, to better understand processes and resources.

(d) The evaluation accounts for the heterogeneity and variety of the JA’s topics and activities, identifying cross-cutting approaches and themes that are worth assessing.

(e) The evaluation analyses the JA’s capacity to achieve changes at both the WP participant and member state (MS) level, acknowledging international differences in political and healthcare systems.

(f) The chosen research methodology reflects the complexity of the initiative.

(g) The evaluation results contribute to informing decision makers on how public health activities can be planned and managed in the future.

Methods

Following the stepwise evaluation process outlined in the previous section, we describe the methods corresponding to each step. As data collection and analysis will occur over the next three years, we will focus on steps 1–5 of the evaluation design.

Project elements and activities (Steps 1 and 2)

We began developing the evaluation plan by analysing the JA PreventNCD, including its various activities, milestones, deliverables, and other intended outcomes (step 1). This included thoroughly reviewing the grant proposal documents for JA PreventNCD, searching the literature for evaluation frameworks of complex preventive interventions, and analysing the evaluation plans of other JAs [17, 18]. Then, we identified its connections with other initiatives, particularly the WHO Special Initiative on NCDs and Innovation [19], the EU NCD initiative ‘Healthier Together’ [20], and other (former) JAs concerned with preventing NCDs, such as the JA ‘Best ReMaP’ [21].

In addition, we engaged WP leaders during the development of the evaluation plan. For example, we hosted an initiating workshop in Berlin in April 2024, which brought together representatives from the project coordination and communication WPs. We also presented and discussed the evaluation ideas during online meetings with the Executive Committee (ExCo) in May and June 2024. By the end of June 2024, the draft evaluation plan was shared with all WP leaders, and all feedback and comments were incorporated into the document. The revised version was then circulated again for further input.

As part of the evaluation plan, we developed a logic model (step 2) that clusters activities in terms of inputs, outputs, outcomes, and impacts, both at the project and MS levels [22, 23]. By synthesising the elements of the WPs into a logic model, we were able to identify the overall mechanism of change and define the evaluation questions. The logic model is shown in Figure 1.

Figure 1.

Logic model for project elements and activities.

The main inputs of the JA (apart from the financial resources) are the experience, expertise, and commitment of the participants, e.g., the various entities and authorities actively involved in the tasks and subtasks of the technical WPs. Another key input is the cooperation among participants, i.e., the exchange of knowledge and working together.

Several outputs are planned within the WPs to support the achievement of the above-mentioned goals. We reviewed the activities and products planned in the various tasks and subtasks of the technical WPs and identified four output clusters: (1) Improving the knowledge base, e.g., systematic reviews or good practice examples; (2) Capacity building, e.g., toolkits and hands-on recommendations; (3) Prevention in practice, e.g., concrete setting-based interventions; and (4) Improving monitoring systems, e.g., through efforts to harmonise questionnaires.

The outcomes are defined in the goals of the JA PreventNCD, which form the basis of the grant agreement. These outcomes are: (a) improving the capacities of MSs to plan and implement NCD prevention policies and activities at the national/local level; (b) improving data and monitoring systems for NCDs and risk factors; (c) contributing to reduced social inequalities; and (d) engaging with and supporting key actors in NCD prevention. Furthermore, the evaluation will look at the unintended outcomes, which may be positive, offering unexpected benefits, or negative, potentially causing harm to those involved [24].

The impacts of the JA PreventNCD include sustainable changes at the civil society and policy level within the MSs, e.g., long-term achievements like reducing social inequalities and cancer prevalence.

Evaluation questions and indicators (Steps 3 and 4)

The logic model helped us identify relevant evaluation questions (step 3). For each domain of output, outcome, and impact evaluation, we develop a set of evidence-based indicators (step 4). The logic model clarifies the relevant stakeholders (e.g., WP leaders or MS representatives) who could best assess the different activities and their value. Table I provides an overview of the evaluation questions for each domain and the associated data collection methods.

Table I.

Levels of evaluation.

Type of evaluation	Element of evaluation	Description	Evaluation content/questions (selection)
Output evaluation	Self-assessment	Survey of WP leaders/task leaders on accomplishments of the respective WP and the effectiveness of the proposed tasks	What knowledge translation resources are being developed? What was the specific contribution of working together cross-country/joining forces within the WP? Challenges and added value of working jointly with other countries
	Peer-assessment	Focus group consultation with participating MSs on the developed resources, materials, and models	Adequacy and relevance of knowledge translation resources Acceptance and utilisation of resources/models by MSs Usefulness of resources/models in practice among stakeholders
	Quality assessment	Audit/review and appraisal of developed resources against quality criteria (e.g., maps, models, guidelines, toolboxes)	Usability and relevance of developed resources Rigour of development process
Outcome and impact evaluation	Self-assessment of implementation and impact	Surveys and/or interviews with JA participants of all MSs Assessment of the country level implementation process in participating MSs	Were countries able to plan and implement NCD prevention policies and activities at different levels?
	Stakeholder assessment of implementation and impact	Surveys and/or interviews with key stakeholders in selected countries Assessment of the country level implementation process in participating MSs	Were national data and monitoring systems improved? Did countries regularly engage with and support key actors in the field of NCDs?
	Change assessment	Interviews with MSs based on the MSC technique	What was the most significant national change that was put forward by participating in the JA?

WP: work package; MS: member state; JA: joint action; NCD: non-communicable diseases; MSC: most significant change.

It was decided that the data collection for process indicators at the project level will be carried out by the project coordination team. This includes monitoring the progress of activities, and the timeliness and achievement of internal milestones, to assess project fidelity. The evaluation team will then synthesise these results and provide key insights in two interim reports, due midway in 2025 and 2026, and one final report due at the end of the project in 2027.

Data collection methods (Step 5)

We then decided on a range of procedures and methods that would be appropriate for gathering output, outcome, and impact indicators (step 5). These indicators are specific and measurable characteristics – either quantitative or qualitative – that provide reliable means to assess tangible products or services as a result of project activities, reflect project related changes, and demonstrate progress towards outcomes and impacts [25]. The data collection methods vary depending on the different evaluation domains, with every data collection method outlined in Table I being conducted once.

The output evaluation assesses whether meaningful and useful materials, resources, and structures have been developed within the WPs and subsequently adopted by the participating MSs. A mixed-method approach will be used for this evaluation.

For self-assessment, all leaders will be asked to critically appraise the accomplishments of their respective WP and task. The first results of the self-assessment will help determine if there is added value of international collaboration in joint activities. The self-assessment will also provide a mapping of the knowledge translation resources being developed (e.g., best practices, toolboxes, guidelines).

For peer-assessment, we will conduct focus groups with participants from each WP to gain initial perceptions of the quality of the resources produced. When a critical mass of knowledge translation resources has been developed, we will conduct a quality assessment to evaluate the resources against specific quality criteria, e.g., rigour of the development process and the usability of the material [26].

Towards the end of the project, we will conduct an evaluation of the proposed outcomes and impacts (see Figure 1) to gain an understanding of the impact of the JA at both the MS and EU levels. For example, the JA aims to improve capacities for implementing prevention policies and activities, engage key actors (e.g., decision makers, patients’ groups) in NCD prevention, and enhance data systems for NCDs. To evaluate this, (semi-)standardised surveys and interviews will be conducted with both participating and external stakeholders, such as representatives from ministries or influential non-governmental organisations (NGOs) (stakeholder assessment).

Given that the starting points for prevention may differ significantly between participating MSs, a change assessment using the ‘most significant change’ (MSC) technique will be embedded in the evaluation [27]. The MSC technique will capture local knowledge and individual accomplishments in the form of narratives (change stories), which will be collected through semi-standardised interviews with all MSs. The accounts will undergo thematic analysis, allowing us to examine different types of changes and provide a more comprehensive picture of the impact.

Since the evaluation is primarily funded to cover the four-year duration of the JA, but the effects on NCD prevalence and related social inequalities are likely to emerge over a longer period, it is essential to establish measures now that will enable future assessment of these long-term impacts. Activities of other WPs, such as WP 7 (Social Inequalities) and WP 8 (Monitoring), aim at harmonising methodologies for the monitoring of health inequalities, risk factors, and outcomes relating to NCDs. By providing recommendations for improved systems for collection and analysis of data, the participating MSs can be enabled to monitor the development of NCDs as well as behavioural and social determinants, and thereby track potential long-term impacts of the JA beyond the initial funding period.

Accompanying evaluation approaches

Other WPs and stakeholders, such as the WHO Office for Europe, suggested that identifying promising examples of interventions early in the JA funding period, closely monitoring them, and sharing them with MSs as potential examples of good practice could be beneficial. Therefore, we decided to include ‘implementation case studies’ that will be identified together with WP leaders. These case studies will focus on the process of implementation and the influence of contextual factors [28] using a mixed-methods approach, including key informant interviews and document analysis. We aim to track implementation processes in a prospective manner. The criteria for eligibility include: (a) the expectation that concrete cancer and/or NCD policies are adopted and implemented during the JA funding period and (b) the availability of sufficient information and documents to support the case study.

In addition, the evaluation of the JA PreventNCD includes a component that will assess the coordination, management, and added value of activities that support collaboration within the project. This component will be outsourced to an independent external agency to ensure objectivity and avoid conflicts of interest. The evaluation should systematically collect information on: satisfaction with overall coordination; quality of selected workshops; usability of report templates; and effectiveness of project communication and dissemination on different platforms (e.g., website, social media, workshops). Structured interviews and/or standardised surveys will be conducted among WP leaders, members of newly established JA boards, and other relevant stakeholders.

Discussion

With its breadth and heterogeneity, the JA PreventNCD poses several challenges for evaluation. We addressed the heterogeneity of the JA by identifying clusters of activities across WPs and health topics: (1) Improving knowledge base; (2) Capacity building; (3) Putting prevention pilots into practice; and (4) Improving the monitoring systems of NCDs. We recognise that these clusters represent outputs of the JA and these alone cannot guarantee that real, lasting change will occur in the participating MSs. However, these activities provide key resources and assets that may serve as valuable support for participants, potentially enabling them to implement more effective health policies in their countries. Therefore, we decided to include these outputs as a main object of evaluation. This will complement the outcome evaluation, exploring whether changes in NCD prevention could be achieved in MSs.

Given the volume of pilot projects and interventions planned across various tasks and subtasks, we concluded that the project evaluation team would not be able to conduct evaluations of each individual measure. Instead, the overall evaluation of the JA will track and appraise changes at a higher level, focusing on accomplishments related to cross-cutting outputs and outcomes at both the WP (and task) level and MS level. We will, however, encourage participants to evaluate their own pilot projects, offer guidance on project evaluation through workshops, and include evaluation reports of pilot projects in our assessment.

The JA’s interventions and activities will not occur in isolation. Different contextual factors are expected to influence the implementation of activities and their potential to achieve desired outcomes [29]. The evaluation plan accounts for this by including two predominantly qualitative approaches: implementation case studies and the MSC technique. Using mixed-methods study designs aligns with recommendations for the evaluation of complex interventions and will enable us to create a comprehensive, adaptable evaluation that can respond to the complexities of the project’s operating environment.

The evaluation of outputs, outcomes, and impacts relies on data collected from WP (or task) leaders and from MSs. The accomplishments will therefore be reported from diverse perspectives. One of the main tasks of the evaluation will be to integrate different views and appraisals to form a complete and reliable picture of the contribution of the JA to NCD prevention in Europe. Additionally, the evaluation will investigate whether the JA’s efforts are coherent and interrelated with EU and WHO activities to avoid the risk of creating more fragmentation in the field.

As many of the reports on the specific objectives are due towards the end of the four-year funding period, some results may not be available when the final evaluation report is being prepared. In addition, key impacts – particularly those related to NCD prevalence and social inequalities – may not be ‘visible’ within the funding period but are likely to emerge in the months or years after the JA has ended. To address this, monitoring plans within WPs should include indicators that can be incorporated into routine data collection and broader surveillance systems to enable long-term tracking of outcomes. However, continued evaluation of these longer-term impacts of the JA PreventNCD will depend on securing additional funding to support the ongoing collection and analysis of key indicators.

A large, international project like the JA PreventNCD involves numerous partners and participants. Lessons learned from former JAs demonstrate challenges for the evaluation team to gather necessary information from project partners and stay in the loop. The evaluation process relies on input from project partners, particularly WP and task leaders. Due to limited resources, requests could not always be treated as a priority, often requiring multiple follow-ups and email exchanges to obtain the needed information [30]. While the evaluation framework will guide the evaluation questions and data collection methods, we recognise that the evaluation should remain flexible and adaptable to the dynamic nature of the project. We will regularly interact with relevant WPs and include their views in the evaluation design when appropriate. The evaluation process must actively engage key stakeholders at all stages, including representatives from WPs and MSs. Their participation is vital to ensure that the findings are both meaningful and relevant to the objectives and stakeholders of the JA. Engaging stakeholders throughout the process enhances the accuracy and comprehensiveness of data collected and helps ensure that results are interpreted and reported in a way that aligns with the JA’s goals.

Through rigorous evaluation of the project, we aim to ensure that the JA delivers high-quality outcomes and has a meaningful impact at both the MS and EU levels. By identifying barriers and facilitators to successful cooperation, projects, and policies, we intend to guide future public health initiatives across European countries, complementing the multiple activities and resources from other WPs, e.g., the WP on sustainability. The evaluation will generate insights to support this aim by establishing effective strategies and scalable interventions for settings with similar needs. Regular evaluation also promotes transparency, shared accountability, and collaboration, strengthening trust among stakeholders and supporting ongoing cooperation for public health improvement. Ultimately, these insights help assess whether public funds are being used effectively and whether the JA offers a valuable model for future large-scale public health initiatives.

Footnotes

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This project (GA – 101128023) is co-funded by the European Union. Views and opinions expressed are, however, those of the authors only and do not necessarily reflect those of the European Union or European Health and Digital Executive Agency (HaDEA). Neither the European Union nor HaDEA can be held responsible for them.

ORCID iDs

Julika Loss

Martin Thißen

Michael Lange

Simon Löbl

Anna Mbodj

Gert B.M. Mensink

Meg Yates

References

Piovani

Nikolopoulos

Bonovas

Non-communicable diseases: The invisible epidemic. J Clin Med 2022; 11: 5939.

Rootman

Goodstadt

Potvin

, et al. A framework for health promotion evaluation. WHO Reg Publ Eur Ser 2001; 92: 7–38.

Organisation for Economic Co-Operation and Development. Health at a glance 2011: OECD indicators. Paris: OECD Publishing, 2001.

Bauman

Nutbeam

Planning and evaluating population interventions to reduce noncommunicable disease risk – Reconciling complexity and scientific rigour?

Public Health Res Pract 2014; 25: e2511402.

Kidder

Luna

, et al. CDC program evaluation framework. Morbidity and Mortality Weekly Report. Report no. 73(6). Atlanta: U.S. Centers for Disease Control and Prevention, www.cdc.gov/mmwr/volumes/73/rr/pdfs/rr7306a1-H.pdf (2024, accessed 14 July 2025).

Waters

Doyle

Jackson

, et al. Evaluating the effectiveness of public health interventions: The role and activities of the Cochrane Collaboration. J Epidemiol Community Health 2006; 60: 285–289.

European Commission. A new horizon for Europe: Impact assessment of the 9th EU framework programme for research and innovation. Luxembourg: Directorate-General for Research and Innovation, https://op.europa.eu/en/publication-detail/-/publication/00d78651-a037-11e8-99ee-01aa75ed71a1 (2018, accessed 14 July 2025).

European Union. Guidance on joint action plans. Brussels: European Commission, https://ec.europa.eu/regional_policy/sources/guidance/guidance_jap_en.pdf (2015, accessed 14 July 2025).

Klepp

K-I

Helleve

Høstrup

, et al. Scaling up non-communicable disease prevention and health promotion across Europe: The Joint Action PreventNCD. Scand J Public Health 2025; 53(3S): 8–17.

10.

Skivington

Matthews

Simpson

, et al. A new framework for developing and evaluating complex interventions: Update of Medical Research Council guidance. BMJ 2021; 374: n2061.

11.

McGill

Marks

, et al. Qualitative process evaluation from a complex systems perspective: A systematic review and framework for public health evaluators. PLoS Med 2020; 17: e1003368.

12.

Gaglio

Shoup

Glasgow

RE.

The RE-AIM framework: A systematic review of use over time. Am J Public Health 2013; 103: e38–46.

13.

Damschroder

Aron

Keith

, et al. Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implementation Sci 2009; 4: 50.

14.

Peters

Taghreed

Olakunle

, et al. Implementation research: What it is and how to do it. BMJ 2013; 347: f6753.

15.

Damschroder

Reardon

Opra Widerquist

, et al. The updated Consolidated Framework for Implementation Research based on user feedback. Implementation Sci 2022; 17: 75.

16.

World Health Organization. Social determinants of health. Report by the Director-General. Geneva: Executive Board, 148^th session. Report no. EB148/24, https://apps.who.int/gb/ebwha/pdf_files/EB148/B148_24-en.pdf (2021, accessed 14 July 2025).

17.

Straarup

O’Donovan

Lambrou

, et al. The Joint Action on Tobacco Control: A cooperation project for strengthening tobacco control in Europe. Tob Prev Cessation 2022; 8: 26.

18.

Lindström

Wikström

Kivelä

, et al. Best-ReMaP D3.1. Evaluation Strategy. Finland: Finnish Institute for Health and Welfare. https://bestremap.eu/wp-content/uploads/2024/04/D3.1.pdf (2021, Accessed 14 July 2025).

19.

World Health Organization. Special initiative on NCDs and innovation (SNI), www.who.int/europe/teams/special-initiative-on-ncds-and-innovation/about-us (2024, accessed 30 January 2025).

20.

Michalis

Manalova

Hasnain

, et al. The EU ‘Healthier Together’ non-communicable diseases initiative – A commendable step in the right direction, but a more ambitious approach to mental health is needed. The Mental Health Advocacy Platform Statement. Brussels: Mental Healthy Advocacy Platform, https://eurohealthnet.eu/wp-content/uploads/publications/2022/220714_ncds-statement-mental-health-advocacy-platform.pdf (2022, accessed 14 July 2025).

21.

Gabrijelcic

JA Best-ReMaP recommendations. Presented at: Heads of Food Safety Agencies Plenary meeting; 14 November 2023, https://nijz.si/wp-content/uploads/2022/12/Best-ReMaP-Recommendations-PPT-Heads-of-Food-Safety-Agencies-Plenary-meeting.pdf (2023, accessed 14 July 2025).

22.

Freedman

Simmons

Lloyd

, et al. Public health training center evaluation: A framework for using logic models to improve practice and educate the public health workforce. Health Promot Pract 2014; 15: 80S–88S.

23.

Langley

Gillespie

Lewis

, et al. Developing logic models to inform public health policy outcome evaluation: An example from tobacco control. J Public Health (Oxf) 2021; 43: 639–646.

24.

Jabeen

Do we really care about unintended outcomes? An analysis of evaluation theory and practice. Eval Program Plann 2016; 55: 144–154.

25.

Kelly

Twohig

Woods

, et al. Reaching consensus on definitions for food and physical activity policies: Experience from the Policy Evaluation Network. Eur J Public Health 2022; 32: iv10–20.

26.

Godinho

Ansari

Guan

, et al. Toolkits for implementing and evaluating digital health: A systematic review of rigor and reporting. J Am Med Inform Assoc 2021; 28: 1298–1307.

27.

Davies

Dart

The ‘most significant change’ (MSC) technique: A guide to its use, www.mande.co.uk/wp-content/uploads/2005/MSCGuide.pdf (2005, accessed 14 July 2025).

28.

Peters

Tran

Taghreed

Implementation research in health: A practical guide. Geneva: Alliance for Health Policy and Systems Research, World Health Organization, https://iris.who.int/bitstream/handle/10665/91758/9789241506212_eng.pdf (2013, accessed 14 July 2025).

29.

World Health Organization. Implementing health in all policies: A pilot toolkit. Geneva: World Health Organization. https://iris.who.int/bitstream/handle/10665/366435/9789240057128-eng.pdf?sequence=1isAllowed=y (2022, accessed 14 July 2025).

30.

Kirchner

Pastler

Schroll

, et al. Joint Action on Tobacco Control (JATC) D3.2. Interim Evaluation Report. Vienna: The Austrian Agency for Health and Food Safety (AGES). AGES, https://jaotc.eu/wp-content/uploads/2019/09/WP3-D3.2-Interim-Evaluation-Report.pdf (2023, accessed 14 July 2025).