Challenges and strategies for implementers and evaluators working in conflict settings

Abstract

Practitioners implementing and evaluating humanitarian programs in conflict zones face unique challenges requiring context-sensitive guidance. Conflict zones—geographical areas with militarized and non-militarized violence, widespread political instability, and state-sanctioned intimidation and genocidal violence—are rife with logistical, methodological, and ethical challenges that impact implementation and evaluation. Although challenges have been well-documented, few solutions or evaluation frameworks exist to help evaluators in conflict zones confront these challenges. This article examines how the application of implementation science frameworks, specifically, Reach, Effectiveness, Adoption, Implementation, and Maintenance and the Consolidated Framework for Implementation Research, offers useful strategies to mitigate some challenges in conducting evaluations in conflict zones. Areas for future research include collaborative team approaches to using these frameworks, ethical guidance, and reporting and dissemination.

Keywords

conflict evaluation implementation science international development

Introduction

It is estimated that by 2030, two thirds of impoverished families will be located in conflict zones (Corral et al., 2020). Conflict zones—geographical areas with militarized and non-militarized violence, widespread political instability, and state-sanctioned intimidation and genocidal violence (Bush and Duggan, 2013)—are rife with logistical and methodological obstacles that impact implementation and evaluation of programs (Brusset et al., 2016; Duggan and Bush, 2014). Conflict zones are complex systems (Duggan and Bush, 2014; Phillips and De Wet, 2017; Ramalingam, 2013) where volatility, insecurity, and resource unpredictability make it difficult to implement humanitarian programs, assess the causality of one or multiple programs to outcome(s), or rely on fixed indicators (Duggan and Bush, 2014; Gates et al., 2021; Ridde et al., 2012). In addition to methodological and logistical challenges, conflict zones pose significant ethical challenges including safety and security concerns for locally based people and groups participating requiring guidance beyond “do no harm” (Aronsson and Hassnain, 2021; Duggan and Bush, 2014; Easterly and Pfutze, 2008). Although research extensively examines these and other challenges for evaluating in conflict zones (Augustinavicius et al., 2018; Bush and Duggan, 2013; Duggan and Bush, 2014; Easterly and Pfutze, 2008; Fitzpatrick, 2012; Makan-Lakha, 2016; Ridde et al., 2020), there is still a need for systemized frameworks, such as those used in implementation science, that can assist evaluators by providing common language and ways to link the outer context of an evaluation environment with the methods and intended outcomes of an intervention in a cyclical, rather than linear, manner.

This article examines how the application of implementation science frameworks, specifically, Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) and the Consolidated Framework for Implementation Research (CFIR), offers useful strategies to mitigate some challenges in conducting evaluations in conflict zones. We argue that evaluators working in conflict zones can learn from implementation scientists who have considered and documented how to navigate and assess dynamic humanitarian settings (Haque and Freeman, 2021). We describe some of the logistical, methodological, and ethical challenges in conflict zone evaluations, and how they can be mitigated through the application of implementation frameworks, while also acknowledging some limitations of these frameworks. We close with future directions for research on evaluating in conflict zones, including collaborative team approaches, ethical guidance, and reporting and dissemination, among others.

Implementation and evaluation challenges in conflict zones

Humanitarian agencies range from small grassroots initiatives to large multinational operations, and provide an extensive array of both specialized and generalized services for refugees, internally displaced people, and conflict-affected communities. They fund and run educational programs, public health programs (such as communicative and non-communicative disease control, sexual and reproductive health care, and sanitization programs), mental health and psychosocial support (MHPSS), nutritional aid and food delivery, and construction (Blanchet et al., 2017; Easterly and Pfutze, 2008).

Conflict zones are volatile, unpredictable, and resource-scarce (Bush and Duggan, 2013; Ridde et al., 2012), and humanitarian organizations may be seen as highly politicized (Fast et al., 2013). Evaluators in conflict-affected areas face a number of logistical, methodological, and ethical challenges that hinder the ability to conduct robust evaluations. These challenges require a deliberation of how, and in what ways, the role of the outer context can be factored into evaluation design. It is imperative that humanitarian stakeholders and evaluators be equipped with the theory, frameworks, and methods to assist conflict-affected families and evaluate services rigorously; however, conflict zones are an area of evaluation which require further examination and guidance. Prior research identifies a number of challenges. Challenges discussed below include safety and security of all stakeholders, navigating priorities of funders, standardizing terminology, and ensuring knowledge of a context that may be volatile and rapidly changing. Other operational challenges discussed below include access to program areas, tracking participants across intervention life cycle, engaging stakeholders, remote management, selecting appropriate indicators and tools of measurement, and establishing cause and effect in a complex, often volatile environment.

Conducting evaluations in conflict zones necessitates a standard of ethics that not only considers high-level concerns about the scope and purpose of the project but also recognizes and accounts for the potential of evaluation to exacerbate tensions and place staff and evaluands in dangerous circumstances during each stage of the project (Aronsson and Hassnain, 2021). It is difficult to collect data in areas that are conflict-affected or rebel-controlled. Participants, especially women, girls, or ethnic or religious minorities, may not be able to provide honest feedback for fear of their own safety, or may not be reachable by evaluators or program implementers (Cohen and Yaeger, 2021). Any missteps in elevating voices or framing an issue can have grave consequences, sometimes life or death. In addition, complexity and the vulnerability of program recipients in conflict zones means that erroneous findings or ineffective programs can lead to increased instability, corruption, and harm in the community (Autesserre, 2014).

There is a dearth of data measuring the extent of violence that impacts aid workers in conflict zones, but ensuring safety of all stakeholders is one of the biggest constraints for effective program implementation (Sarker et al., 2020) and securitization is of increasing concern (Bollettino, 2008; Duffield, 2012; Sarker et al., 2020). Targeted examples of violence include explosives placed outside of humanitarian buildings and kidnappings and ambushes on vehicles and convoys (Fast, 2007, 2010). In addition, places of humanitarian intervention, which often serve as home base for field staff, are highly susceptible to attack due to their consolidation of resources and perception of bias toward certain communities in the conflict (Sa’Da et al., 2013).

While elements like time and geography can affect evaluations in other non-conflict settings as well, the risk is much higher in conflict zones, and unintended issues that may be trivial can quickly spiral into devastating consequences (Bornstein, 2010; Duggan and Bush, 2014). Organizations increasingly thus push for increased remote management or very brief site visits, which places extra burden upon local staff who may not have the capacity, or the security, to step in and conduct evaluations (Chaudri et al., 2019; Guidero, 2022).

Accountability requirements can exacerbate power imbalances between funding or donor agencies and local organizations, staff and communities, and can narrow what gets evaluated and reported to short time frames. Evaluation in conflict zones historically emerged from international development work (Rossignoli et al., 2017), where in-house monitoring and evaluation are conducted within one department of the larger humanitarian organization running programming (Carden and Alkin, 2012). Aid workers, including evaluators, are accountable to not only the conflict-affected communities but also their own organizational hierarchies and programmatic funders (Ebrahim, 2016; Madianou et al., 2016).

Conflict zones create limitations in study designs, including study lengths and timeline expectations from intervention to evaluation. It is frequently more difficult to fully incorporate local stakeholders in program design or selection of indicators than it might be in other contexts (Autesserre, 2014), which can mean that especially vulnerable groups are left out of conversations, and outputs are misguided and mismatched to need (Stame et al., 2021) Safety concerns discussed above place pressure on programs to show results quickly by focusing on short-term outputs as opposed to using evaluation results to incorporate diverse stakeholders into research design, adapt interventions, and invest in long-term programming to make systemic changes (Makan-Lakha, 2016).

Debates about study design best practices within the fields of international development and humanitarian response have arisen; randomized control trials, often seen to be a gold standard design (Phillips and De Wet, 2017), are not designed for areas of rapid change (Brusset et al., 2016; Stern et al., 2012), and the complexity of a conflict zone means it can be difficult to isolate specific effects to show a linear outcome from an intervention (Makan-Lakha, 2016). This is made even more complicated when accounting for cultural and logistical limitations that evaluators face in trying to collect data. The use of standardized measures may not be relevant to, or difficult to validate in, specific situations (Makan-Lakha, 2016; Phillips and De Wet, 2017). Due to insecurity, access to program sites is often limited, which can prevent evaluators from conducting work onsite. In these cases, evaluators do not benefit from the salient cultural, historical, and political details that one can garner from direct interaction within a context (Duggan and Bush, 2014). This limits the ability to accurately assess the impact of a program with nuance, consideration of contextual barriers or facilitators, or identification of confounding factors that may impede mechanisms in the program’s theory of change (Duggan and Bush, 2014; Ling, 2012).

It is difficult for evaluators to establish a baseline and collect comparative data to observe changes pre- and post-implementation, or to collect data from a control group (Bozzoli et al., 2013; Chaudri et al., 2019). Attrition rates can be superficially elevated due to forced displacement (Bozzoli et al., 2013), and indicators of success often shift along with changing goals in response to a dynamic and fluid setting (Bozzoli et al., 2013). Therefore, evaluators must often contend with multiple theories of change and goals that have changed in scope and intention over the course of implementation.

Another challenge is the lack of common terminology or operationalization of concepts in humanitarian settings (Altare et al., 2022; Augustinavicius et al., 2018). This is particularly relevant in conflict zones where there is a plethora of stakeholders from many different countries (Kwan et al., 2019; Saez et al., 2021), and when operationalization and terminology directly impact measurement (Eyler, 2021). The lack of standardization in measurement in humanitarian settings inhibits comparability of findings and external validity (Altare et al., 2022), which may lead to further concerns about rigor and trustworthiness of findings. Ultimately, it remains challenging to assess both short- and long-term changes within interventions delivered in conflict settings (Bush and Duggan, 2013).

Evaluators and practitioners in conflict zones are making efforts to utilize innovative, participatory, social change–seeking methods, including short-term local consultancies (Ridde et al., 2012), incorporating focus groups and ethnographic qualitative methods (Rossignoli et al., 2017), and developing frameworks to distinguish evidence from rumor (Schon, 2021). Practitioners and evaluators in conflict zones have employed strategies such as remote management or delegating activities to local partners who are accepted by the local community and are cultural and linguistic insiders (Chaudri et al., 2019). While these strategies allow program implementation to continue in conflict zones, local partners frequently carry a heavy task burden with insufficient training and supervision (Bozzoli et al., 2013; Chaudri et al., 2019) and still face the same security concerns that all aid workers face.

The array of unique challenges calls for improved frameworks that address these challenges and allow evaluators to account for conflict-related factors and militarized contexts into the planning and conducting of the evaluation. Though challenges are immense, rigorous evaluation, and ultimately, establishment of an evidence base is imperative in humanitarian settings that are affected by conflict (Chaudri et al., 2019). The wide assortment of humanitarian interventions, from educational programming to disease control, can benefit from the utilization of frameworks that can be flexible, serve as templates, and provide guides for how to link outer and inner contexts of a setting.

Certainly, the humanitarian needs in such settings are grave, and therefore, the potential to meet basic needs and achieve impact is significant. Evaluation that produces reliable data allows practitioners to remain accountable to all stakeholders and to service communities most effectively and appropriately. These data can facilitate solutions that target the right people, address the right problems, consolidate the evidence base, and sustain impact even in a dynamic, conflict-affected environment.

In general, researchers across fields are striving to understand and adapt to environmental complexity, incorporating new methodologies, frameworks, and understandings into their work (Centre for the Evaluation of Complexity Across the Nexus (CECAN), 2021). Common themes have emerged, including non-linearity (i.e. that critical changes may happen rapidly or unexpectedly, as with a “tipping point”). The context may expose new behaviors that are difficult to predict, and actors (including participants, programmers, and evaluators) may adapt and change as the environment is changing or growing even more complex (Frelin, 2016).

Despite such efforts to account for complexity, many of the challenges in conflict zones are unique and require tailored solutions beyond a general accounting for complexity. Table 1 provides a summary of the aforementioned implementation and evaluation challenges that are specific to conflict settings.

Table 1.

Challenges for implementation and evaluation in conflict settings.

Identified challenge	Description
Safety and security	The safety of evaluands, enumerators, and evaluators is of utmost priority when conducting evaluation in conflict zones. Security risk in volatile regions may escalate quickly, and local staff carry extra burden of outer safety concerns and workplace stressors. Safety and security of international staff members is generally more discussed and prioritized in research and practice than local staff members.
Pressure and priorities from funders	Programs are under pressure to show results quickly and incentivized to focus on short-term outcomes rather than long-term outcomes. Evaluations often must prioritize reporting information that allows implementers to be accountable to funders rather than information that would allow for programmatic improvements.
Terminology and operationalization	Lack of common terminology and operationalization of concepts as a result of multiple stakeholders, including funders.
Knowledge of context	Evaluators do not have the time to conduct additional analyses, which prohibits them from becoming familiar with the outer context factors
Restricted access to areas in need/program sites	Evaluators often do not have access to the field due to security restrictions, which prohibits them from becoming familiar with the outer context factors or from collecting necessary data
Tracking program participants throughout the intervention lifecycle	High participant attrition rates, or changes between who was receiving services at the beginning of the program compared to the end, due to forced migration or other contextual factors can make participants very difficult to track
Engaging stakeholders	Differing values, opinions, priorities of stakeholders, lack of maintenance of shared databases
Remote management and reliance on local partners	Local partners may lack capacity to carry out evaluations independently. Remote management is often an effective implementation strategy in conflict zones but is ineffective when needing to conduct robust evaluations
Tools and methods	Data collection tools and methods that may be most appropriate to assess impact may not be feasible in conflict zones
Mapping theories of change and selecting indicators to be evaluated	Program goals and indicators of success often shift during implementation, and logic models and other tools are not equipped to incorporate changing indicators or program drift
Establishing cause and effect	Evaluators often cannot establish linear cause-and-effect relationships between implementation and impact indicators due to difficulties establishing a baseline or a control group or dynamic outer context factors

Frameworks to guide implementation and evaluation practice

Frameworks are needed which allow evaluators in conflict settings to consider the unique design of the program being evaluated, identify strategies that respond to methodological barriers and facilitators within the setting, account for contextual challenges, and consider ethical issues. To examine causal mechanisms and account for contextual factors that shape and affect intervention outcomes in conflict zones, evaluators must look at both impact and implementation (Ridde et al., 2020) rather than focusing on quantifying or measuring impact without adequately uncovering the complexity of barriers and facilitators to this impact (Ridde et al., 2020). Ridde et al. attribute this tendency to a lack of methodological diversity and reliance on quantitative rather than mixed-method approaches, and a lack of familiarity with theories and frameworks used in implementation science (Ridde et al., 2020). Thus, implementation science frameworks that have been utilized in conflict-affected settings may allow evaluators to mitigate many of the challenges faced.

Implementation science frameworks

Implementation science is “the scientific study of methods to promote the systematic uptake of research findings and other evidence-based programs into routine practice, and, hence, to improve the quality and effectiveness of health services” (Bauer et al., 2015: 3). In the field of implementation science, frameworks have emerged which assist in the developing and testing of methods to allow evidence-based interventions to be implemented and sustained across diverse and complex settings (Damschroder, 2020). Implementation may interact with complexity science when implementation and evaluation in complex settings requires thinking that acknowledges the dynamic, interconnected factors within the context (Braithwaite et al., 2018). Implementation frameworks have been applied in evidence-based program design and implementation across low-resource and conflict-affected settings (Damschroder and Lowery, 2013; Glasgow et al., 1999; Moullin et al., 2019), and can help the field make sense of contextual barriers and facilitators, and design appropriate methods and solutions (Moullin et al., 2020). Within these frameworks, implementation factors are embedded, interact with, and are influenced by multiple levels of contextual domains, including individuals and inner and outer intervention settings (Damschroder, 2020). Implementation frameworks can help enlighten how complex phenomena within intervention contexts are interrelated, and inform the identification and measurement of change mechanisms and outcomes (Moullin et al., 2019).

Frameworks hypothesize relationships between constructs, including mediators and moderators that may affect outcomes (Damschroder, 2020; Ridde et al., 2020). Implementation science frameworks can assist evaluators with tailoring data collection and design strategies in a manner that better fits the context, particularly in dynamic intervention settings, such as conflict zones. Some implementation scientists also dismiss the idea that “program drift”—changing goals or targets in the middle of the program—is always detrimental, and state that although contextual factors are frequently viewed as impediments to intervention implementation or evaluation, these factors can actually be studied and provide useful information about intervention fit (Chambers et al., 2013).

Evaluation frameworks

In the last several decades, evaluation approaches have expanded to take into consideration complexity through the incorporation of concepts, theories, and methods from the systems and complexity sciences (Barbrook et al., 2021; Gates et al., 2021; Marra, 2011; Midgley, 2016; Reynolds et al., 2016). For example, in the United Kingdom, the CECAN develops, tests, and disseminates research about complexity-informed approaches which they distinguish as including

ideas and methods from across the complexity sciences, including the study of complex adaptive systems and systems science and thinking, so that these various perspectives and tools can be brought to bear on the complex systems problems faced in evaluation. (Barbrook et al., 2021: 6)

Within humanitarian programming, Ramalingam’s (2013) Aid on the Edge of Chaos: Rethinking International Cooperation in a Complex World provides a significant example of applying complexity thinking to issues ranging from understanding problems, to implementing and managing responses, and with some attention to evaluation. Additional contributions include work distinguishing simple, complicated, and complex features of aspects of interventions and contexts (Rogers, 2008) and case examples and cross-case patterns, as in the special issue of New Directions for Evaluation (Gates et al., 2021). Within complexity-oriented approaches, there have been steady developments in realist evaluation—an approach grounded in critical realism that identifies configurations of contextual factors, generative causal mechanisms, and outcomes through a detailed process of building and testing an intervention’s theory of change (Pawson, 2013). A previous analysis of evaluation approaches in complex settings suggests that combining theories of change and realist evaluation approaches is an effective strategy (Dickinson, 2006). Realist evaluation frameworks help evaluators examine the underlying generative mechanisms that explain how outcomes were caused and how contextual factors influenced these outcomes (Berwick, 2008; Dickinson, 2006). They determine whether the intervention has been implemented as intended, and seek to answer the questions of “who, what, when, and where” (Centers for Disease Control and Prevention, 2014). Implementation science can build from realist evaluations as both fields are concerned with integrating the knowledge of context into program design and evaluation (Swindle, 2020).

While these developments around complexity and evaluation span multiple disciplines, areas of practice, and methodologies, evaluators working in conflict zones primarily look to humanitarian guidance. The Inter-Agency Standing Committee (IASC), a collaborative forum for humanitarian partners, presents guidelines stating that monitoring and evaluation is a minimum response in humanitarian settings (Augustinavicius et al., 2018; IASC, 2017). Several funding agencies have also released guidance and clarified minimum standards for conducting monitoring and evaluation in conflict zones (Organisation for Economic Cooperation and Development (OECD), 2012; United Nations Office for Disaster Risk Reduction, 2015; US Agency for International Development (USAID), 2006). Though these guidelines call for a thorough examination of the context and triangulation of data sources, programs funded through these agencies tend to follow a linear style of evaluation, such as logic models (Makan-Lakha, 2016; OECD, 2012; USAID, 2021). While logic models are commonly used and are helpful for selecting indicators and thinking through theories of change (Brusset et al., 2016; Payton et al., 2022), they do not address the logistical or methodological challenges in these dynamic evaluation environments. Many gaps in humanitarian guidance exist, and there are inadequate resources available to address the challenges that evaluators face when working in conflict zones (Bakrania et al., 2021). In sum, there is an opportunity to further operationalize contextual and implementation factors, particularly in relation to conflict settings (Fynn et al., 2020).

Two promising implementation-evaluation frameworks: RE-AIM and CFIR

In this article, two implementation science evaluation frameworks will be described and examples of studies in conflict zones that have applied these frameworks will be provided. The two frameworks include the RE-AIM framework and the CFIR. If applied to evaluations in conflict zones, implementation science frameworks may help evaluators in conflict zones account for the unique contextual challenges and effectively assess the impact and the process of an intervention despite a fluid outer context.

RE-AIM and CFIR have been used to assess the implementation of complex interventions, and are two of the most commonly used frameworks in low- and middle-income settings, including conflict zones (Glasgow et al., 2019; Means et al., 2020). These implementation science frameworks may help mitigate many of the challenges faced during evaluations, such as security challenges, navigating priorities from differing stakeholders, difficulties accessing data and communities, interchangeable terminology and resulting measurement challenges, and more. Because implementation science frameworks target both process and effectiveness outcomes (Rapport et al., 2018; Schliep et al., 2017), when used in collaboration with existing project logic models, implementation science frameworks can help teams assess outputs and both short- and long-term outcomes.

While implementation science frameworks have been applied most frequently in health services research in high-income settings, these frameworks have also been applied to interventions delivered in low-resource settings and fields including HIV/AIDS; nutrition; and water, sanitation, and hygiene (WASH) (Haque and Freeman, 2021).

RE-AIM

The RE-AIM framework is an evaluation framework that has been used widely over the past 20 years, including in low- and middle-income countries (LMICs), has been cited in over 28,000 publications (Glasgow et al., 2019: 3), and was originally developed to facilitate the translation of research into practice, particularly for health policy programs (Glasgow et al., 1999, 2019). The framework provides questions within various domains that guide assessments of both effectiveness and implementation, and emphasizes a systems-based and social-ecological approach (Glasgow et al., 1999). RE-AIM explicitly focuses on external validity issues by allowing researchers to account for contextual facilitators and barriers to intervention effectiveness and plan for broad, population-based impact (See Figure 1) (Glasgow et al., 2019).

Figure 1.

Reach, efficacy, adoption, implementation, and maintenance framework (Glasgow et al., 2019).

RE-AIM benefits and limitations in conflict evaluations

RE-AIM suggests that each of the five RE-AIM dimensions (reach, efficacy, adoption, implementation, and maintenance) plays a role in enabling an intervention to have its intended impact on the population that it is serving. RE-AIM can help evaluators to make sense of differing values, opinions, and priorities of stakeholders; avoid indicators that are most affected by high participant attrition rates; and consider sustainment of knowledge that ultimately equips local partners to conduct high-quality evaluations. Reach, the “R” in RE-AIM, asks researchers and evaluators to consider who the target population is and how they can best be reached through implementation. As evaluators in conflict zones grapple with high attrition rates, evaluators could partner with humanitarian agencies to select indicators that are less tied to specific individuals as the unit of analysis and instead consider community-level indicators. The “A” in RE-AIM, adoption, guides researchers and evaluators to consider the necessity of institutional and stakeholder support when implementing a program. In conflict zones, if relevant stakeholders are committing human rights violations and are incapable of providing support, evaluators could seek evidence of other types of plausible institutional support, such as community advisory boards, and feed this information into their evaluation findings. Finally, the “M” in RE-AIM, maintenance, suggests that evaluations must think about the long-term sustainment of interventions. If evaluations also included an assessment of the capacity of local partners in conflict zones and highlighted areas that they needed greater support in, humanitarian agencies could be better equipped to ensure that knowledge rests with local leadership in the long term. Indeed, local partners who are capable of carrying out rigorous evaluations could benefit from pre-existing contextual knowledge and greater site access with less security threats (Guidero, 2022).

One of the benefits of the RE-AIM framework is its flexibility and adaptability, allowing evaluators to deliberately account for safety of all stakeholders, short-term priorities, and methodological concerns in their designs. Evaluators using RE-AIM in community settings have suggested that the framework is most beneficial when integrated with contextual factors and facilitators and barriers to adoption, implementation, and maintenance. Some teams using RE-AIM to guide their evaluation reported that investing time up-front during the design phase resulted in a more efficient evaluation phase (Kwan et al., 2019). Taking time to operationalize and carefully select measures, developing relationships with project sites, and piloting data collection and tools ahead of time in a more accessible location each helped evaluators fine-tune data collection protocols and troubleshoot any potential issues or delays (Kwan et al., 2019).

Such planning processes or up-front investments in evaluation may ultimately shorten field visits and help circumvent safety issues that may arise due to longer evaluations in insecure areas. For example, an emerging area of research regarding RE-AIM is its iterative use of rapid, pragmatic assessments for evaluation (Harden et al., 2018). This may include early assessment of program enrollment, and if the demographics are skewed in any way, shifting program targets to reach more key segments of the population (such as ethnic minorities, genders, districts, etc.) (Harden et al., 2018). These rapid assessments allow for continual situational analysis of security, feasibility, and value-add of the project, and may allow for reduced time in the field and enhanced safety measures for participants and evaluation staff. Time in the field for evaluation may also be reduced by utilizing some secondary data from ongoing monitoring throughout project implementation, or by pragmatic selection of the most important indicators that align with project priorities rather than evaluation of multiple indicators (Harden et al., 2018). However, it is important to note that RE-AIM does not consider the full range of safety concerns, and RE-AIM should be used in tandem with current operational and ethical guidance and continual assessment of risks.

Previous applications in conflict settings

In the Democratic Republic of the Congo (DRC), RE-AIM has been utilized in the qualitative component of a mixed-methods evaluation of a diabetes mellitus (DM) health care program (Murphy et al., 2017). The area of the DRC in which the study takes place has been affected by persistent conflict since 2003, and contains 17 camps for internally displaced people. RE-AIM provided a framework for researchers to understand the challenges and successes that stakeholders across multiple levels (individual, health care provider, institutional) face, specifically around community awareness and sustainability of health care provision (Murphy et al., 2017). In this context, evaluators acknowledged that intervention impact is contingent upon RE-AIM’s five dimensions (reach, efficacy, adoption, implementation, and maintenance). The use of the RE-AIM framework helped evaluators frame their questions and seek to identify factors at each level of the ecology that supported or impeded the achievement of those dimensions (Murphy et al., 2017).

RE-AIM has also been used within a scoping review of studies of MHPSS services across a number of refugee camps (Cohen and Yaeger, 2021). RE-AIM provided a framework to understand how MHPSS adapted over time in settings that also were dynamic and changing, for example, how programs responded to changing populations within camps and adapted to the changing needs of refugee populations over time (Cohen and Yaeger, 2021). In many of the studies in the scoping review, program evaluation staff were lay people from within the refugee community who reported high levels of trauma exposure, post-traumatic stress, and suicidality, likely connected to their lived experience and their work (e.g. Bolton et al., 2014; Vijayakumar et al., 2017). The RE-AIM framework can provide a systematic way to assess and respond to both external and project-specific factors, including stressors and safety concerns, that may arise during the evaluation process that would impact the adoption and maintenance (the “A” and “M” of RE-AIM) of a project (Cohen and Yaeger, 2021).

Lessons learned for future applications in conflict settings

While the five dimensions of RE-AIM are relevant beyond the scope of conflict zones, evaluators working in conflict settings can especially benefit from the application of RE-AIM because its five dimensions are flexible and adaptable enough to fit complex, dynamic settings. For example, evaluations in both the DRC and in refugee camps were able to frame research questions and identify criteria to identify barriers and facilitators across each of the five dimensions, which provided a holistic picture of where and how the interventions were working, and not working as well, in conflict-affected settings. Furthermore, RE-AIM dimensions cover both process outcomes and effectiveness outcomes (Holtrop et al., 2021), both of which are important to assess in complex settings such as conflict zones (Landes et al., 2020; Ridde et al., 2020).

CFIR

The CFIR is another framework which, when applied to program evaluation, can be used to address challenges in conflict zones (see Figure 2) (Damschroder et al., 2009). Although it is an implementation, rather than an evaluation, framework, its ability to guide assessments of contextual determinants is useful for guiding evaluations and building knowledge (Damschroder et al., 2009, 2022: 4). The CFIR pulls from many theories and disciplines to create a framework that contains 39 constructs, grouped within five major domains, including outer setting, inner setting, characteristics of the innovation, characteristics of individuals, and process of conducting the intervention (Akinyemi et al., 2021). The CFIR allows evaluators to consider evaluation as a continual, iterative process. As seen in Figure 3, CFIR 2.0 uses right- and left-facing arrows to discuss temporality and demonstrate that specified outcomes are not linear, as antecedent and anticipated outcomes influence innovation outcomes, and in turn, innovation outcomes also affect antecedent and anticipated outcomes. The portrayal of the arrows in a loop portrays the complex dynamics of the implementation context, and how the different aspects of implementation can influence, and be influenced by, one another (Damschroder et al., 2022). In design phases, the CFIR’s domains provide guidance for approaching complex phenomena and considering potential implementation barriers and facilitators. Implementation scientists employing CFIR during intervention evaluation have been able to develop methods that enable the understanding and measuring of contextual factors (Damschroder et al., 2022).

Figure 2.

Consolidated Framework for Implementation Research (Damschroder et al., 2009).

Figure 3.

Addendum to consolidated framework for implementation research (Damschroder et al., 2022).

Recently, an addendum was published by the developers of the CFIR to clarify which outcomes CFIR determinants can influence, and the differentiation between determinants vs outcomes during coding and analysis (Damschroder et al., 2022). Although the CFIR is a deterministic framework, its capacity to inform evaluations allows users to test causal mechanisms and develop theories of change that link implementation determinants to implementation outcomes (Damschroder et al., 2022). After completing a literature review, Damschroder et al. added both implementation and innovation outcomes to the CFIR (Damschroder et al., 2022). Similar to RE-AIM, the implementation outcomes added to the CFIR are adoption, implementation, and sustainment.

CFIR benefits and limitations in conflict evaluations

The application of the CFIR allows evaluators to constantly re-assess acceptability, appropriateness, feasibility, and implementation climate, and implementation readiness throughout the course of the program lifecycle. This allows for continual assessment and quick adaptability, necessary in areas that are dynamic and dangerous. CFIR evades a linear cause-and-effect approach and instead focuses on the cycle of impact (assessing the program implementation itself), which then sets evaluators up to also evaluate effectiveness. CFIR could be used in collaboration with a logic model or integrated into existing logic models to make them less linear. For example, when creating a logic model framework for specific use in refugee settings, Payton et al. (2022) added new components to account for a bi-directional relationship between contextual factors and short- and long-term outcomes. If evaluators have adequately assessed implementation, they may begin data collection with a clear sense of the methods and approaches that are both feasible and useful for answering the research question in their specific setting. This aligns with the pragmatic, rapid assessments that are emerging through the continued use of RE-AIM (Harden et al., 2018). Furthermore, assessing implementation feasibility and implementation climate may guide evaluators to carefully select data collectors who are accepted within the local community by all stakeholders (Duffield, 2012; Fast et al., 2013), which may enhance security and ensure that program participants feel comfortable providing honest feedback.

Previous applications in conflict settings

Like RE-AIM, CFIR has been applied in dynamic and complex environments. It has been used to assess community engagement in polio eradication programs throughout Nigeria, including areas in the country experiencing high violence and instability from militants, insurgents, and bandits (Akinyemi et al., 2021). Using CFIR provided insight to researchers as to which of the five major domains, as well as specifics within each domain, were deemed to be the most significant and most challenging. This provided the data for subsequent conversations about what elements of the program may be modifiable and were of the highest priority (Akinyemi et al., 2021).

CFIR was also used to assess the effectiveness and feasibility of an ongoing hepatitis B immunization program throughout areas of rural Myanmar, in areas with historical and ongoing sectarian violence (Guan et al., 2021). The constructs and domains of CFIR guided the analysis of in-depth qualitative interviews, specifically to understand challenges and barriers to implementation. The differentiation between inner and outer settings provided guidance for recommendations, including how to include various health care stakeholders that, due to the nature of the conflict zone, have tense relationships with each other (Guan et al., 2021). Consideration of the outer setting may also lead evaluators to consider what the priorities are for various stakeholders, including funders, local governments, and participant communities (Guidero, 2022), which could help evaluators select specific indicators to include in their evaluations. However, further research is needed regarding the outer context of implementation and the relationship between the outer context and implementation and evaluation of interventions (Lengnick-Hall et al., 2021).

Lessons learned for future applications in conflict settings

CFIR has several components that fit such contexts well: the framework is unique in that it portrays non-linear relationships between factors within the implementation context, which makes it more applicable for evaluations in dynamic and fluid settings, such as conflict zones. Further, CFIR’s inner and outer settings can assist evaluation teams to assess security by considering how the inner setting of the evaluation would be influenced, or influenced, by the outer context. The inner and outer settings also frame evaluation findings with an ecological lens, which is critical for understanding barriers and facilitators to implementation effectiveness. Finally, like RE-AIM, CFIR’s outcomes assess both process and effectiveness outcomes.

Strategies from RE-AIM and CFIR frameworks to address implementation-evaluation challenges in conflict settings

As conflict-sensitive evaluation is in the early stages of its development (Hassnain et al., 2021), frameworks from the field of implementation science can provide valuable language and domains to streamline and synthesize the emerging guidance and tools for evaluators working in conflict zones. Implementation science frameworks can also help evaluators make sense of the various contextual factors and more accurately assess both program effectiveness and implementation despite a dynamic context. Table 2 portrays the documented challenges of conducting evaluations in conflict zones. The relevant humanitarian standards and guidance correspond to such challenges and highlight where the remaining gaps are.

Table 2.

Implementation-evaluation challenges, gaps, and guidance for conflict zones.

Identified challenge	Strategies from implementation science frameworks
Safety and security	The use of rapid, pragmatic assessments or iterative evaluations helps reduce time spent in the field and allows for continual review of risks for all stakeholders Any time spent in the field can be securitized through careful design, meaningful community partnerships, and using a strategy of acceptance.
Pressure and priorities from funders	Careful consideration of the “outer context” as delineated in the CFIR and selection of indicators that align with funding and other outer context priorities.
Terminology and operationalization	The field of implementation science, and therefore implementation science frameworks, uses common terminology, often referred to as “domains,” to discuss the common elements of program implementation and evaluation
Knowledge of context	Implementation science refers to context as the “outer setting,” and reliance on frameworks for program design and evaluation planning can help practitioners and evaluators understand and account for context from program design phases through program evaluation and closeout phases.
Restricted access to areas in need/program sites	In the long term, creative strategies to engage stakeholders and build capacity of local partners can bypass the need for access to program sites. In addition, continual review of outer context allows for careful selection of location so as to ensure feasibility of project.
Tracking program participants throughout the intervention lifecycle	Evaluators can partner with humanitarian agencies to select indicators that are less tied to specific individuals as the unit of analysis (thereby avoiding issues caused by high participant attrition rates) and instead consider community-level indicators.
Engaging stakeholders	If relevant stakeholders are committing human rights violations and are incapable of providing support, evaluators could seek evidence of other types of plausible institutional support, such as community advisory boards, and feed this information into their evaluation findings.
Remote management and reliance on local partners	If evaluations also included an assessment of the capacity of local partners in conflict zones and highlighted areas that they needed greater support in, humanitarian agencies could be better equipped to task-shift and ensure that knowledge rests with local leadership in the long term. Strategies such as acceptance with local communities and stakeholders, and the use of rapid, pragmatic evaluation methods, can help ensure the safety and success of local partners conducting evaluations.
Tools and methods	Implementation science frameworks provide location-flexible research questions to guide evaluations and help set up evaluators to use methods that assess both implementation and effectiveness.
Mapping theories of change and selecting indicators to be evaluated	The use of CFIR in program implementation and evaluation evades a linear cause-and-effect approach and instead focuses on the cycle of impact (assessing the program implementation itself), which then sets evaluators up to also evaluate effectiveness.

Note: CFIR = Consolidated Framework for Implementation Research.

Discussion

Evaluators working in conflict zones, and practitioners designing programs in these humanitarian settings, must think beyond linear cause-and-effect models and take into account the variety of interrelated factors by using an array of methods to continually evaluate their work (Duggan and Bush, 2014; Makan-Lakha, 2016; Ridde et al., 2020), which could include feedback loops, direction setting (Gates, 2016), and network (or stakeholder) analysis (Rosenthal et al., 2020). Implementation science frameworks can assist evaluation teams through the utilization of iterative, innovative models that can shorten evaluators’ time spent in unsafe conditions, appease stakeholders and funders who desire short- and long-term outcomes, and help evaluators navigate methodological challenges.

We see a number of implications for future evaluation design and future research, including collaboration and mutual learning opportunities, hybrid designs, and examination of remaining gaps, namely addressing security challenges and dissemination, to guide future research.

Collaborative team approaches in implementation science and evaluation

There are many stakeholders involved in evaluation, including but not limited to third-party evaluators, staff members of implementing humanitarian agencies, local informants and community advisors, and participants and community members (Mark et al., 2021; Tembo et al., 2021). Though not explicitly stated in the aforementioned implementation-evaluation frameworks, the broader field of implementation science has encouraged the use of collaborative team approaches in which stakeholders from across agencies and levels collaborate to implement, scale, and evaluate interventions (Hurlburt et al., 2014; Lengnick-Hall et al., 2021). Integrating frameworks like CFIR and RE-AIM can be most effective when evaluation teams are diverse, representative of communities, and when power is shared across levels of leadership (Lengnick-Hall et al., 2021). Collaborative team approaches can assist evaluation teams with selecting appropriate evaluation tools, coordinating monitoring and evaluation activities across the programming lifecycle, and sensitive and culturally appropriate reporting and dissemination strategies, which may include the use of indigenous evaluation tools and strategies (Bowman, 2020; Drawson et al., 2017). Participatory and collaborative team approaches are particularly relevant in conflict zones, where the dynamics of the conflict may exacerbate existing power dynamics between local and international staff, or between evaluands and evaluators (Chonka et al., 2022; Chouinard & Cousins, 2013; Ridde et al., 2012). In these contexts, it is important for coordination efforts to prioritize the safety of all stakeholders involved, and for trust to be built within and across agencies and groups of individuals. A platform of trust between all stakeholders involved can ultimately contribute to a culture where ethics are the foundation of the evaluation, and “doing good” and prioritizing safety become engrained (Aronsson and Hassnain, 2021).

Bridge siloes and collaborate across implementation and evaluation sciences

Typically, implementation scientists are involved in the program design, implementation, and evaluation phases of a program while evaluators often come in solely during evaluation and do not benefit from prior knowledge of the intervention setting or the ability to conduct an evaluation iteratively throughout the course of implementation. The field of implementation science intends to bridge the gap between research and practice by promoting widespread adoption of research findings into evidence-based practice (Olswang and Prelock, 2015), and the field of evaluation serves to bridge this gap as well (Urban and Trochim, 2009). Both fields are attempting to move toward more rigorous evaluation practices and to develop toolkits of best practices for navigating challenging contexts such as conflict zones, ultimately bridging the gap between research and practice (Moullin et al., 2020; Tabak et al., 2012; Urban and Trochim, 2009; Wanzer, 2021). Therefore, opportunities to share best practices, lessons learned, and research and evaluation findings can help bridge the gap and eliminate the siloes of implementation and evaluation science fields.

Hybrid implementation and evaluation frameworks and designs

Implementation science frameworks embed consideration of the outer context into all aspects of implementation. In other words, evaluators that may be unable to learn about the outer context through additional analyses and/or access to program sites could benefit if a program has already been innovatively designed in response to unique contextual factors. In this case, humanitarian agencies could set evaluators up for success by adopting implementation science frameworks during program inception phases as well as evaluation phases. Hybrid study designs can be an effective way for evaluators working in conflict zones to evaluate both implementation and effectiveness of interventions (Curran et al., 2012; Landes et al., 2020). There are multiple types of hybrid designs that allow for such a dual focus, and based on the research question and research priorities, implementers can select a design that shifts the evaluation focus more toward either effectiveness or implementation outcomes (Curran et al., 2012; Landes et al., 2020). Future research and evaluations should consider employing hybrid designs and reporting both implementation and effectiveness findings, and describing the relationship between implementation and effectiveness. If an evaluation is not designed to gather both types of data, then evaluators are apt to miss critical contextual factors that impact the success of the program being evaluated (Landes et al., 2020). Leaving out information regarding such crucial contextual factors is always deleterious in evaluations, but it is both easier to do and carries higher consequences for evaluators working in conflict zones. Indeed, the application of implementation science frameworks into humanitarian evaluation settings can guide evaluators in conflict zones to account for contextual factors and apply methods derived from complexity science to assess impact in the most effective manners.

Addressing security challenges

While implementation science frameworks offer pathways toward safer evaluations, they do not fully provide a solution for protecting the security of all stakeholders, especially local teams, working in conflict zones or challenges that arise from the diversity and multitude of humanitarian donors and their impact of evaluation design and selection of relevant outcomes. Much of the research regarding security of evaluators in humanitarian settings has focused on ex-patriates; however, local staff account for a majority of humanitarian employees (Guidero, 2022). Future research that continues to explore pathways toward mitigating security challenges must expand to include a focus on local staff and community members as well, as this will provide more holistic solutions for mitigating security challenges in evaluation. Furthermore, addressing complex security challenges will require the coordination of both programming and operations teams. Implementation science frameworks may be helpful for designing, implementing, and evaluating programs in conflict zones, but teams utilizing such frameworks and working in these areas should liaise with teams who are trained and equipped to monitor risk to all stakeholders. These teams should set and regularly update parameters regarding evaluation activities.

Dissemination strategies

While implementation science frameworks and their attention to the outer context may assist evaluators in indicator selection and security, there must also be long-term commitments to bridging the gap between research and practice and sharing terminology, tools, and standards. Such long-term commitments may entail the selection of research questions that are more applied (such as best methods for conducting needs assessments or best methods for monitoring and evaluation) and which contain answers that are more useful for humanitarian practice and allow them to build upon existing, local support (Tol et al., 2020). As humanitarian practice requires answers that are scalable and readily deployable (Tol et al., 2020), researchers and practitioners must also find more direct venues of communication rather than reliance on dissemination of findings in peer-reviewed journals. Continued collaboration between research and practice can ultimately lay the groundwork for more robust monitoring and evaluation. The application of implementation science frameworks can serve as a single step toward laying this groundwork and building common methods and language.

Conclusion

As conflict zones draw concerns globally, it is imperative for implementers and evaluators to adeptly work together to ensure that humanitarian programs are implemented with high quality, reducing methodological and logistical barriers, considering ethical issues, and that the impact of these interventions is properly assessed. The immense challenges have consequences for implementation and evaluation quality and, most importantly, the welfare of individuals living in conflict zones, whom humanitarian interventions seek to target and serve. While prior research has extensively documented challenges, including difficulties understanding and incorporating context or mapping and evaluating theories of change, strategies for confronting challenges are fragmented across different bodies of literature in both implementation science and evaluation fields. By delineating these challenges and synthesizing guidance from frameworks, with emphasis on RE-AIM and CFIR, this article offers guidance for implementers and evaluators working in conflict zones. Moreover, we draw on prior work (Augustinavicius et al., 2018; Ridde et al., 2020) to advance connections across these fields and research areas grounded in their shared commitments to better professional practice. Future research which aims to evaluate interventions delivered in conflict zones should utilize iterative, innovative models, guided by implementation science frameworks and hybrid designs focusing on implementation and effectiveness, to account for methodological and practical challenges. In addition, we suggest collaborative approaches for designing and evaluating interventions, which may also include reliance on other stakeholders such as operations teams, to prioritize safety of staff and participants and to guide dissemination of findings.

Footnotes

Acknowledgements

This article identifies some of the challenges in conflict settings to draw attention to this area and to spark further much-needed further research, particularly from evaluators with experience and lived experience in the Global South and in conflict-affected countries. The authors acknowledge that new perspectives are needed and acknowledge that those responsible for much of the work evaluating humanitarian programs in conflict zones have not had their experiences or voices adequately reflected in research.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Laura Bond

Emily F. Gates

Laura Bond is a doctoral candidate at the Boston College School of Social Work. Previously, she worked for a large international non-governmental agency in parts of Africa, Afghanistan, and India.

Elizabeth K. Klein is a doctoral candidate at Boston College School of Social Work. Previously, she worked as a social worker in Chicago Public Schools and with refugees in France and Thailand.

Emily F. Gates is an Assistant Professor of Measurement, Evaluation, Statistics, and Assessment in the Boston College Lynch School of Education and Human Development.

References

Akinyemi

Adebayo

Bassey

, et al. (2021) Assessing community engagement in Nigeria polio eradication initiative: Application of the Consolidated Framework for Implementation Research. BMJ Open 11(8): e048694.

Altare

Weiss

Ramadan

, et al. (2022) Measuring results of humanitarian action: Adapting public health indicators to different contexts. Conflict and Health 16(1): 54.

Aronsson

Hassnain

(2021) Evaluation and ethics in contexts of fragility, conflict and violence. In: Hassnain

Kelly

Somma

(eds) Ethics for Evaluation. London: Routledge, 80–100.

Augustinavicius

Greene

Lakin

, et al. (2018) Monitoring and evaluation of mental health and psychosocial support programs in humanitarian settings: A scoping review of terminology and focus. Conflict and Health 12: 9.

Autesserre

(2014) Peaceland: Conflict Resolution and the Everyday Politics of International Intervention. Cambridge: Cambridge University Press.

Bakrania

Balvin

Daidone

, et al. (2021) Impact evaluation in settings of fragility and humanitarian emergency. Innocenti Discussion Paper 2021-02. Florence: UNICEF Office of Research—Innocenti.

Barbrook-Johnson

Castellani

Hills

, et al. (2021) Policy evaluation for a complex world: Practical methods and reflections from the UK Centre for the Evaluation of Complexity across the Nexus. Evaluation 27(1): 4–17.

Bauer

Damschroder

Hagedorn

, et al. (2015) An introduction to implementation science for the non-specialist. BMC Psychology 3(1): 32.

Berwick

(2008) The science of improvement. Journal of the American Medical Association 299(10): 1182–4.

10.

Blanchet

Ramesh

Frison

, et al. (2017) Evidence on public health interventions in humanitarian crises. The Lancet 390(10109): 2287–96.

11.

Bollettino

(2008) Understanding the security management practices of humanitarian organizations. Disasters 32(2): 263–79.

12.

Bolton

Lee

Haroz

, et al. (2014) A transdiagnostic community-based mental health treatment for comorbid disorders: Development and outcomes of a randomized controlled trial among Burmese refugees in Thailand. PLoS Medicine 11(11): e1001757.

13.

Bornstein

(2010) Peace and conflict impact assessment (PCIA) in community development: A case study from Mozambique. Evaluation 16(2): 165–76.

14.

Bowman

(2020) Nation-to-nation in evaluation: Utilizing an indigenous evaluation model to frame systems and government evaluations. New Directions for Evaluation 2020(166): 101–18.

15.

Bozzoli

Bruck

Wald

(2013) Evaluating programs in conflict-affected areas. Households in conflict network. Working Paper No. 142. Brighton: The Institute of Development Studies.

16.

Braithwaite

Churruca

Long

, et al. (2018) When complexity science meets implementation science: A theoretical and empirical analysis of systems change. BMC Medicine 16(1): 63.

17.

Brusset

De Coning

Hughes

(eds) (2016) Complexity Thinking for Peacebuilding Practice and Evaluation. New York: Springer.

18.

Bush

Duggan

(2013) Evaluation in conflict zones: Methodological and ethical challenges. Journal of Peacebuilding & Development 8(2): 5–25.

19.

Carden

Alkin

(2012) Evaluation roots: An international perspective. Journal of Multidisciplinary Evaluation 8(17): 102–18.

20.

Centers for Disease Control and Prevention (2014) Types of evaluation. Available at: http://www.cdc.gov/std/program/pupestd.htm (accessed 21 April 2022).

21.

Centre for the Evaluation of Complexity Across the Nexus (2021) The complexity evaluation toolkit. Available at: https://mcconnellfoundation.ca/wp-content/uploads/2021/11/The-Complexity-Evaluation-Toolkit.pdf (accessed 2 July 2022).

22.

Chambers

Glasgow

Stange

(2013) The dynamic sustainability framework: Addressing the paradox of sustainment amid ongoing change. Implementation Science 8: 117.

23.

Chaudri

Cordes

Miller

, et al. (2019) Humanitarian health programming and monitoring in inaccessible conflict settings: A literature review. Journal of International Humanitarian Action 4: 9.

24.

Chonka

Edle Ali

Stuvøy

(2022) Eyes on the ground and eyes in the sky: Security narratives, participatory visual methods and knowledge production in “danger zones.” Security Dialogue 53(6): 567–88.

25.

Chouinard

Cousins

(2013) Participatory evaluation for development: Examining research-based knowledge from within the African context. African Evaluation Journal 1(1): 9.

26.

Cohen

Yaeger

(2021) Task-shifting for refugee mental health and psychosocial support: A scoping review of services in humanitarian settings through the lens of RE-AIM. Implementation Research and Practice 2: 1–13.

27.

Corral

Irwin

Krishnan

, et al. (2020) Fragility and Conflict: On the Front Lines of the Fight against Poverty. Washington, DC: World Bank.

28.

Curran

Bauer

Mittman

, et al. (2012) Effectiveness-implementation hybrid designs: Combining elements of clinical effectiveness and implementation research to enhance public health impact. Medical Care 50(3): 217–26.

29.

Damschroder

(2020) Clarity out of chaos: Use of theory in implementation research. Psychiatry Research 283: 112461.

30.

Damschroder

Lowery

(2013) Evaluation of a large-scale weight management program using the consolidated framework for implementation research (CFIR). Implementation Science 8: 51.

31.

Damschroder

Aron

Keith

, et al. (2009) Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implementation Science 4(1): 50.

32.

Damschroder

Reardon

Opra Widerquist

, et al. (2022) Conceptualizing outcomes for use with the consolidated framework for implementation research (CFIR): The CFIR Outcomes Addendum. Implementation Science 17(1): 7.

33.

Dickinson

(2006) The evaluation of health and social care partnerships: An analysis of approaches and synthesis for the future. Health & Social Care in the Community 14: 375–83.

34.

Drawson

Toombs

Mushquash

(2017) Indigenous research methods: A systematic review. International Indigenous Policy Journal 8(2). Available at: https://ojs.lib.uwo.ca/index.php/iipj/article/view/7515

35.

Duffield

(2012) Challenging environments: Danger, resilience and the aid industry. Security Dialogue 43(5): 475–92.

36.

Duggan

Bush

(2014) The ethical tipping points of evaluators in conflict zones. American Journal of Evaluation 35(4): 485–506.

37.

Easterly

Pfutze

(2008) Where does the money go? Best and worst practices in foreign aid. Journal of Economic Perspectives 22(2): 29–52.

38.

Ebrahim

(2016) The many faces of nonprofit accountability. In: Renz

Herman

(eds) The Jossey-Bass Handbook of Nonprofit Leadership and Management. Hoboken, NJ: John Wiley, 102–23.

39.

Eyler

(2021) Operationalization and measurement. In: Eyler

(ed.) Research Methods for Public Health. New York: Springer, 63–76.

40.

Fast

(2007) Characteristics, context and risk: NGO insecurity in conflict zones. Disasters 31(2): 130–54.

41.

Fast

(2010) Mind the gap: Documenting and explaining violence against aid workers. European Journal of International Relations 16(3): 365–89.

42.

Fast

Freeman

O’Neill

, et al. (2013) In acceptance we trust? Conceptualising acceptance as a viable approach to NGO security management. Disasters 37(2): 222–43.

43.

Fitzpatrick

(2012) Commentary: Collaborative evaluation within the larger evaluation context. Evaluation and Program Planning 35: 558–63.

44.

Frelin

(2016) Analysing complex conflicts: Requirements for complexity-aware conflict analysis. In: Brusset

De Coning

Hughe

(eds) Complexity Thinking for Peacebuilding Practice and Evaluation. London: Palgrave Macmillan, 207–37.

45.

Fynn

Hardeman

Milton

, et al. (2020) A scoping review of evaluation frameworks and their applicability to real-world physical activity and dietary change programme evaluation. BMC Public Health 20(1): 1000.

46.

Gates

(2016) Making sense of the emerging conversation in evaluation about systems thinking and complexity science. Evaluation and Program Planning 59: 62–73.

47.

Gates

Walton

Vidueira

, et al. (2021) Introducing systems-and complexity-informed evaluation. New Directions for Evaluation 2021(170): 13–25.

48.

Glasgow

Harden

Gaglio

, et al. (2019) RE-AIM planning and evaluation framework: Adapting to new science and practice with a 20-year review. Frontiers in Public Health 7: 64.

49.

Glasgow

Vogt

Boles

(1999) Evaluating the public health impact of health promotion interventions: The RE-AIM framework. American Journal of Public Health 89(9): 1322–7.

50.

Guan

Htut

Davison

, et al. (2021) Implementation of a neonatal hepatitis B immunization program in rural Karenni State, Myanmar: A mixed-methods study. PLoS One 16(12): e0261470.

51.

Guidero

(2022) Humanitarian (in) security: risk management in complex settings. Disasters 46(1): 162–84.

52.

Haque

Freeman

(2021) The applications of implementation science in water, sanitation, and hygiene (WASH) research and practice. Environmental Health Perspectives 129(6): 65002.

53.

Harden

Smith

Ory

, et al. (2018) RE-AIM in clinical, community, and corporate settings: Perspectives, strategies, and recommendations to enhance public health impact. Frontiers in Public Health 6: 71.

54.

Hassnain

Kelly

Somma

(eds) (2021) Evaluation in Contexts of Fragility, Conflict and Violence: Guidance from Global Evaluation Practitioners. Exeter: International Development Evaluation Association.

55.

Holtrop

Estabrooks

Gaglio

, et al. (2021) Understanding and applying the RE-AIM framework: Clarifications and resources. Journal of Clinical and Translational Science 5(1): e126.

56.

Hurlburt

Aarons

Fettes

, et al. (2014) Interagency Collaborative Team Model for Capacity Building to Scale-Up Evidence-Based Practice. Children and Youth Services Review 39: 160–168. https://doi.org/10.1016/j.childyouth.2013.10.005

57.

Inter-Agency Standing Committee (IASC) (2017) Guidance: Mental health and psychosocial support in emergency settings. Monitoring and evaluation. Available at: https://interagencystandingcommittee.org/system/files/2021-03/IASC%20Common%20Monitoring%20and%20Evaluation%20Framework%20for%20MHPSS%20Programmes%20in%20Emergency%20Settings_0.pdf (accessed 20 July 2022).

58.

Kwan

McGinnes

Ory

, et al. (2019) RE-AIM in the real world: Use of the RE-AIM framework for program planning and evaluation in clinical and community settings. Frontiers in Public Health 7: 345.

59.

Landes

McBain

Curran

(2020) Reprint of: An introduction to effectiveness-implementation hybrid designs. Psychiatry Research 283: 112630.

60.

Lengnick-Hall

Stadnick

Dickson

, et al. (2021) Forms and functions of bridging factors: Specifying the dynamic links between outer and inner contexts during implementation and sustainment. Implementation Science 16(1): 34.

61.

Ling

(2012) Evaluating complex and unfolding interventions in real time. Evaluation 18(1): 79–91.

62.

Madianou

Ong

Longboan

, et al. (2016) The appearance of accountability: Communication technologies and power asymmetries in humanitarian aid and disaster recovery. Journal of Communication 66(6): 960–81.

63.

Makan-Lakha

(2016) Monitoring and evaluation in peacebuilding: Reclaiming the lost opportunities for improving peacebuilding in practice. In: Brusset

De Coning

Hughe

(eds) Complexity Thinking for Peacebuilding Practice and Evaluation. London: Palgrave Macmillan, 177–204.

64.

Mark

Allen

Goodwin

(2021) Stakeholder involvement in evaluation: Does it affect observers’ perceptions of an evaluation? And which stakeholder group(s) do people think should to participate? Evaluation Review 45(3–4): 166–90.

65.

Marra

(2011) Some insights from complexity science for the evaluation of complex policies. In: Forss

Marra

Schwartz

(eds) Evaluating the Complex: Attribution, Contribution, and Beyond, vol. 18. New Brunswick, NJ: Transaction Publishers, 315–34.

66.

Means

Kemp

Gwayi-Chore

, et al. (2020) Evaluating and optimizing the consolidated framework for implementation research (CFIR) for use in low- and middle-income countries: A systematic review. Implementation Science 15(1): 17.

67.

Midgley

(2016) Four domains of complexity. Emergence: Complexity & Organization 18(2): 137–50.

68.

Moullin

Dickson

Stadnick

, et al. (2019) Systematic review of the Exploration, Preparation, Implementation, Sustainment (EPIS) framework. Implementation Science 14(1): 1.

69.

Moullin

Dickson

Stadnick

, et al. (2020) Ten recommendations for using implementation frameworks in research and practice. Implementation Science Communications 1(1): 42.

70.

Murphy

Biringanine

Roberts

, et al. (2017) Diabetes care in a complex humanitarian emergency setting: A qualitative evaluation. BMC Health Services Research 17(1): 431.

71.

Olswang

Prelock

(2015) Bridging the gap between research and practice: Implementation science. Journal of Speech, Language, and Hearing Research 58(6): S1818–26.

72.

Organisation for Economic Cooperation and Development (OECD) (2012) Evaluating peacebuilding activities in settings of conflict and fragility—Improving learning for results. DAC guidelines and reference series, OECD, Paris.

73.

Pawson

(2013) The Science of Evaluation: A Realist Manifesto. London: SAGE.

74.

Payton

Kumar

Kimball

, et al. (2022) A logic model framework for planning an International Refugee Health Research, Evaluation, and Ethics Committee. Health Promotion Practice 23: 852–60.

75.

Phillips

De Wet

(2017) Towards rigorous practice: A framework for assessing naturalistic evaluations in the development sector. Evaluation 23(1): 102–20.

76.

Ramalingam

(2013) Aid on the Edge of Chaos: Rethinking International Cooperation in a Complex World, 1st edn. Oxford: Oxford University Press.

77.

Rapport

Clay-Williams

Churruca

, et al. (2018) The struggle of translating science into action: Foundational concepts of implementation science. Journal of Evaluation in Clinical Practice 24(1): 117–26.

78.

Reynolds

Gates

Hummelbrunner

, et al. (2016) Towards systemic evaluation. Systems Research & Behavioural Science 33: 662–73. https://doi.org/10.1002/sres.2423

79.

Ridde

Goossens

Shakir

(2012) Short-term consultancy and collaborative evaluation in a post-conflict and humanitarian setting: Lessons from Afghanistan. Evaluation and Program Planning 35(1): 180–8.

80.

Ridde

Pérez

Robert

(2020) Using implementation science theories and frameworks in global health. BMJ Global Health 5(4): e002269.

81.

Rogers

(2008) Using programme theory to evaluate complicated and complex aspects of interventions. Evaluation 14(1): 29–48.

82.

Rosenthal

Arku

Baumgartner

, et al. (2020) Systems science approaches for global environmental health research: Enhancing intervention design and implementation for household air pollution (HAP) and water, sanitation, and hygiene (WASH) programs. Environmental Health Perspectives 128(10): 105001.

83.

Rossignoli

Giani

Di Iacovo

, et al. (2017) Enhancing participatory evaluation in a humanitarian aid project. Evaluation 23(2): 134–51.

84.

Sa’Da

Duroch

Taithe

(2013) Attacks on medical missions: Overview of a polymorphous reality: The case of Médecins Sans Frontières. International Review of the Red Cross 95(890): 309–30.

85.

Saez

Konyndyk

Worden

(2021) Financing the Humanitarian Public Good: Towards a More Effective Humanitarian Financing Model. Washington, DC: Center for Global Development.

86.

Sarker

Saha

Matin

, et al. (2020) Effective maternal, newborn and child health programming among Rohingya refugees in Cox’s Bazar, Bangladesh: Implementation challenges and potential solutions. PLoS One 15(3): e0230732.

87.

Schliep

Alonzo

Morris

(2017) Beyond RCTs: innovations in research design and methods to advance implementation science. Evidence-Based Communication Assessment and Intervention 11(3–4): 82–98.

88.

Schon

(2021) How narratives and evidence influence rumor belief in conflict zones: Evidence from Syria. Perspectives on Politics 19(2): 539–52.

89.

Stame

Hawkins

Van Den Berg

(2021) Evaluation contributing to the common good and to tackling bad: 259ethical processes in evaluation. In: Van Den Berg

Hawkins

Stame

(eds) Ethics for Evaluation. London: Routledge, 258–84.

90.

Stern

Stame

Mayne

, et al. (2012) Broadening the range of designs and methods for impact evaluations. Technical report. Available at: https://www.ids.ac.uk/publications/broadening-the-range-of-designs-and-methods-for-impact-evaluations/ (accessed 20 July 2022).

91.

Swindle

(2020) What’s in a name: Is implementation science repackaged process evaluation? The Society for Implementation Science in Nutrition. Available at: https://www.implementnutrition.org/impsci_and_process_eval/ (accessed 21 April 2022).

92.

Tabak

Khoong

Chambers

, et al. (2012) Bridging research and practice: Models for dissemination and implementation research. American Journal of Preventive Medicine 43(3): 337–50.

93.

Tembo

Hickey

Montenegro

, et al. (2021) Effective engagement and involvement with community stakeholders in the co-production of global health research. British Medical Journal 372: n178.

94.

Tol

Ager

Bizouerne

, et al. (2020) Improving mental health and psychosocial wellbeing in humanitarian settings: Reflections on research funded through R2HC. Conflict and Health 14(1): 71.

95.

United Nations Office for Disaster Risk Reduction (2015) Monitoring and evaluation framework. Available at: https://www.undrr.org/publication/monitoring-and-evaluation-framework (accessed 20 July 2022).

96.

Urban

Trochim

(2009) The role of evaluation in research-practice integration. American Journal of Evaluation 30(4): 538–53.

97.

US Agency for International Development (USAID) (2006) Monitoring & evaluation in post-conflict settings. Management Systems International, Washington, DC.

98.

US Agency for International Development (USAID) (2021) PMP: Performance management plan, June. Available at: https://www.usaid.gov/project-starter/program-cycle/pmp (accessed 20 July 2022).

99.

Vijayakumar

Mohanraj

Kumar

, et al. (2017) CASP—An intervention by community volunteers to reduce suicidal behaviour among refugees. International Journal of Social Psychiatry 63(7): 589–97.

100.

Wanzer

(2021) What is evaluation? Perspectives of how evaluation differs (or not) from research. American Journal of Evaluation 42(1): 28–46.