Predictors of alignment between aid donors’ and recipients’ interests in evaluating the effectiveness of the Shelter.Ukraine initiative: Contribution analysis

Abstract

This article evaluates the effectiveness of the Shelter.Ukraine Initiative, a civic response to the humanitarian crisis resulting from Russia’s full-scale invasion of Ukraine. The Initiative aims to address pressing needs through a local umbrella funding approach. Utilizing an evaluative framework that combines multiple approaches, including causal attribution techniques, contribution analysis, and regression analysis, we assess the effectiveness of the Shelter.Ukraine Initiative. We uncover influencing factors that contribute to the Initiative success. These include organizational ambidexterity, learning-by-doing, inter-organizational structure, and advanced technology for financial resource distribution. Our findings underscore the importance of reconfiguring the humanitarian aid architecture to enhance effectiveness. In addition, we suggest that the insights gained from our study can inform future evaluations of humanitarian aid initiatives. By incorporating similar evaluative frameworks and methodologies, researchers can assess the effectiveness of other initiatives and identify key factors contributing to success.

Keywords

aid donors aid recipients causal attribution cause-and-effect diagram contribution analysis effectiveness evaluation regression analysis

Introduction

The full-scale Russian invasion on Ukraine triggered a humanitarian catastrophe in the center of Europe unprecedented in decades. Humanitarian, non-governmental, and charitable organizations have been forced to rapidly increase their activities and provide support to millions of people whose lives have been destroyed by the war. Internally displaced persons in Ukraine have many problems that cannot be solved without the participation of the community, local authorities, volunteer, and social assistance.

In the first days of the full-scale war in Ukraine, the Shelter.Ukraine Initiative (Shelter Ukraine, 2022) was created, which is jointly implemented by non-governmental and charitable organizations. Its main goal is to help solve the humanitarian problems of the civilian population in Ukraine. The Initiative is led by the Valores Foundation (Poland), which is responsible for fundraising together with Ukrainian organizations SILab Ukraine, VPLYV Fund, and Teple Misto. This approach to local actors funding can most accurately be described as local umbrella funding (“trickle down”), which is one of the most suitable ones for responding to emergent local challenges in Ukraine. The Initiative activities covered all regions of Ukraine and were implemented directly through cooperation with volunteer initiatives, non-governmental organizations (NGOs), and local governments at the community level.

In times of crisis, it becomes especially important for civic and charitable organizations to correctly assess their capabilities and risks, decide with whom and how to work, and choose who to help. As recently stated by Antoniv et al. (2023), “given the scale of the conflict, the monitoring and evaluation sector is experiencing difficulties performing under the given circumstances, where decisions are to be made quickly ‘on the ground’” (p. 68). Despite the challenges mentioned above, our overall conclusion about the impact of the war on the development of the evaluation field in Ukraine is quite positive. Surprisingly, the war has served as an impetus for progress.

The purpose of our study is to identify predictors of alignment between aid donors’ and recipients’ interests by utilizing contribution analysis as a primary evaluative framework to assess the effectiveness of the Shelter.Ukraine Initiative. Further regression analysis allowed us to attribute the impact to specific components of the Initiative. This helped us understand which predictors were most influential in driving positive outcomes. This will allow to better understand the extent to which the urgent needs of beneficiaries in the humanitarian crisis are being met, as well as the possibility of developing new strategies and mechanisms for donors to help those in need.

Literature review

Previous research on contribution analysis for evaluating complex interventions

International assistance programs often involve complex interventions aimed at addressing multifaceted challenges such as poverty, inequality, conflict, and humanitarian crises. These interventions typically involve multiple components, stakeholders, and contextual factors, making it essential to evaluate their effectiveness comprehensively. The issue of evaluating the effectiveness of international assistance programs has been actively studied and discussed in the academic community over the past two decades. The reason for this is not so much the constant growth of such assistance and the desire to understand how effectively it is used and whether it achieves its goals. In fact, the nature and type of aid are changing—from the humanitarian and altruistic goal of providing food, clothing, and medical and hygiene products to significant financial injections to use such aid as a lever of influence on the country’s domestic and foreign policy. The policy of interaction between donors and recipients of aid is also changing. Nissanke (2008) notes that “the question as to how donor-recipient relationships should be configured to ensure more effective delivery of foreign aid has always constituted a critical issue for policy-makers and academics alike” (p. 22). However, notable changes in comprehension regarding enhancing aid efficacy have not resulted in a comparable restructuring of aid relationships.

Bigsten and Tengstam (2015) attempt to quantify the impact of improved donor coordination on aid effectiveness, especially in terms of the reduction in transaction costs that can be achieved through better coordination and concentration of aid, as well as the shift from project-based to programmatic approaches. The authors conclude that donor coordination can bring enormous potential benefits but warn that there may be serious political constraints to its implementation.

Sokol and Stiegert (2008) analyze technical assistance in the antitrust area but point out that this analysis can be useful for improving the effectiveness of assistance in other areas as well. Their generalization that agencies with a strong power base are better able to receive assistance is interesting for our study, and they advise donors to provide assistance to agencies with less power and to help strengthen their autonomy and authority.

Increased attention to the issue of aid effectiveness has sparked a flurry of empirical studies that try to measure the impact of aid flows on the macroeconomic performance of developing countries. Bourguignon and Platteau (2013) argue that these studies have produced mixed and even contradictory results, and one possible source of confusion is the fact that they are not based on solid theorizations. The authors attempt to address this gap by proposing a principal-agent model in which the donor monitors the use of aid and imposes sanctions in the event of fraud.

Evaluating the relevance, significance, effectiveness, and impact of aid efforts in fragile contexts is a complex task. The authors of the “Evaluation in Contexts of Fragility, Conflict and Violence” handbook (Hassnain et al., 2021) try to answer the difficult questions that monitoring and evaluation professionals face:

How can an evaluator assess the relevance of an aid effort in a fragile environment? How does an evaluator assess implementation efforts in a crisis and with undermined institutional capacity? How does one assess the effectiveness, efficiency, and efficacy of support while recognizing the potential for setbacks?

The authors believe that contribution analysis is one of the appropriate methods to evaluate interventions for several reasons. First, understanding the design and implementation of the intervention is crucial for developing an appropriate evaluation design. Contribution analysis allows for a nuanced understanding of how interventions are designed and implemented, which is essential for testing the assumptions underlying the theory of change and assessing how development interventions lead to change, especially in fragile, conflict, and violence (FCV) contexts.

While experimental and quasi-experimental designs are commonly used for impact evaluation in FCV contexts, they may not always be feasible or appropriate due to practical constraints or ethical considerations. In such cases, non-experimental design-impact-evaluation approaches, including contribution analysis, offer valuable alternatives. These approaches can integrate qualitative methods such as theory-based evaluation, process tracing, and qualitative comparative analysis, providing insights into the complex causal pathways through which interventions contribute to outcomes. In addition, newer methods for data collection and analysis, such as predictive analytics and satellite imagery, may complement contribution analysis in FCV contexts by providing additional layers of evidence and insight.

Contribution analysis (Mayne, 2012) has emerged as a valuable methodological approach for evaluating the effectiveness of various interventions, programs, and initiatives across diverse fields, including international development aid, public health, and social welfare (Delahais and Toulemonde, 2012). Contribution analysis is grounded in principles of causality and contribution, aiming to identify the extent to which an intervention or initiative contributes to observed outcomes. In a contribution analysis, “the evaluator seeks information about the linkage between the intervention and the noted outcomes, but also other potential causes surrounding the program that could have contributed to the outcomes instead” (Barrington and Triana-Tremain, 2022). Contribution analysis provides a robust framework for evaluating the effectiveness of interventions and understanding the complex causal dynamics at play. Wimbush et al. (2012) argue that it can be applied in the context of participatory strategic planning and performance monitoring, as well as impact evaluation. It offers valuable insights into not only the direct impact of the intervention but also the broader context in which it operates, ultimately informing decision-making, program improvement, and learning.

Forss et al. (2017) highlight the importance of asking the right questions in the contribution analysis of interventions. Traditional causality questions, which focus on attributing outcomes directly to interventions, may not always provide meaningful answers due to the complexity of factors influencing results. In many cases, it may be challenging or even impossible to establish a clear causal link between the intervention and the outcome. On the other hand, contribution questions offer a more nuanced approach to understanding intervention effectiveness. Rather than seeking a definitive causal relationship, these questions focus on assessing the extent to which the intervention has made a difference or contributed to observed outcomes. This shift in perspective acknowledges the multifaceted nature of interventions and the diverse array of factors that influence outcomes. By recognizing the limitations of traditional causality questions and embracing contribution questions, our study can gain deeper insights into how the Shelter.Ukraine Initiative has made a difference and contributed to observed outcomes.

Previous research has explored a range of methodological approaches and techniques for conducting contribution analysis. These include qualitative methods such as stakeholder interviews, document analysis, and participatory workshops, as well as quantitative methods such as regression analysis, propensity score matching, and counterfactual analysis. Researchers have also developed hybrid approaches that combine qualitative and quantitative methods to provide a more comprehensive understanding of causal pathways and mechanisms. From this point of view, it is important to understand the difference between predictors and influencing factors of interventions. While all predictors may have the potential to be influencing factors, not all predictors necessarily function as influencing factors in a given context. The importance of identifying and accounting for influencing factors is well established in the evaluation literature (Lemire et al., 2012). The distinction lies in how these variables interact with the mechanisms and contextual conditions to shape intervention outcomes.

Future research directions in contribution analysis include addressing methodological challenges, refining analytical techniques, enhancing the integration of qualitative and quantitative methods, and promoting transparency and rigor in evaluation practice.

Concepts and theories related to alignment of interests between aid donors and recipients

Private contributors have a crucial role in funding humanitarian interventions (Development Initiatives, 2022: 60). Funding from private donors is channeled primarily to NGOs. The rationales of donors in the realm of global humanitarian assistance are commonly depicted in the literature (Bandyopadhyay and Vermann, 2013) as twofold—humanitarian and strategic (economic and security). Among the approaches to the reasons for donor assistance identified by Sen (2018), a liberal approach is worthy of attention in our study which suggests that based on “enlightened self-interest” (p. 39), donors give foreign aid with the purpose of producing global public goods, stability, and alleviating problems stemming from global inequality. In addition, “NGOs may draw some comfort from the finding that the distribution of NGO aid is hardly affected by commercial and political self-interests that state agencies are often found to have when granting aid” (Thiele et al., 2008). Scott (2014) insists that “donors play a much bigger role than just funding” (p. 3): They also deliver various forms of assistance directly to affected individuals, exert influence on their partners and other significant stakeholders, and advocate at both global and country levels for humanitarian policy matters.

McKinley and Little (1979) were the first who developed the “donor interest vs recipient needs” model, which is still used as the mainstream model in studying foreign aid motivation. The donor interest model describes how aid is distributed in accordance with donor interests. The recipient need model shows that aid is distributed in accordance with economic and social welfare needs. Researchers concluded that the interests of donors, as opposed to the interests of recipients, are more likely to regulate the direction and scope of aid.

Contrary to the arguments of McKinley and Little (1979) about the prioritization of the interests of international aid donors, Büthe et al. (2012: 571) conclude that “the deeply rooted humanitarian discourse within and among aid NGOs drives their aid allocation,” and that “recipients’ humanitarian need is substantively and statistically the most significant determinant of private aid allocation.” And the best knowledge of the needs in communities is provided by local civil society organizations that have access, local knowledge, and connections and speak the same language as the recipients of aid. Banks et al. (2015) confirm this in their study, stating that “NGOs can build bridges between foreign organizations and local structures and processes, applying their knowledge of local contexts to strengthen their roles in empowerment and social transformation” (p. 707). Rahmani (2012), analyzing the effectiveness of international assistance for the reconstruction and development of Afghanistan since 1979, when Soviet troops invaded, emphasizes that “donors’ lack of understanding of the local context and their policies have contributed to local NGOs’ ineffectiveness.” Indeed, local civil societies should be the ones to answer questions about the most pressing and painful issues, as they have the permanent resources and authority to respond to the short- and long-term needs of their communities.

Research methodology

Our study utilized a combination of evaluative approaches, including contribution analysis and regression analysis, to assess the effectiveness of the Shelter.Ukraine Initiative. Contribution analysis helped in identifying the influencing factors that contributed to the success of the Shelter.Ukraine Initiative, as well as understanding the causal pathways through which these factors influenced outcomes. Regression analysis was used to identify predictors of success of the Initiative. By examining the significance of regression coefficients, we determine which factors were statistically associated with the Initiative’s success.

The evaluation of the effectiveness of the Shelter.Ukraine Initiative, conducted by the NGO Center for Social Audit in December 2022–January 2023, included an assessment of the degree to which the needs of target audiences were met, the timeliness and efficiency of assistance, the level of service delivery, the quality of goods in aid humanitarian kits, the psychological state of internally displaced persons, and cooperation with local partners. To obtain information about the outcomes, impact, and process of the Shelter.Ukraine Initiative, a dialogic evaluation approach was used, which involves obtaining evaluative judgments from different stakeholders’ perspectives.

As Rogers (2014) emphasizes, “one of the essential elements of an impact evaluation is that it not only measures or describes changes that have occurred but also seeks to understand the role of particular interventions in producing these changes” (p. 1). Therefore, the data analysis to assess the impact of the Initiative in terms of the role of donor-recipient relationships in aid effectiveness was conducted using a methodological innovation—an interest-based approach to stakeholder engagement which was recently proposed as one of the methods of rapid impact evaluation (Rowe, 2019: 4):

The underlying premise of interest-based approaches to stakeholders is that each interest has worldviews that shape and influence their assessments and that combining the worldviews of all interests who can influence the intervention and those affected by the intervention offsets potential bias from a narrower approach to stakeholders.

The process of implementing the Shelter.Ukraine Initiative is a four-level distribution of accumulated resources:

Valores Foundation (Poland) took the initiative in fundraising and collected international donations to its accounts;

Ukrainian organizations SILab Ukraine, Teple Misto, and VPLYV Fund took over the management of resource allocation at the local level;

Local civic or charitable organizations, volunteer initiatives, or local governments received financial support to equip centers for internally displaced persons or purchase food or hygiene products;

The final beneficiaries were the recipients of the aid—internally displaced persons.

The first two levels of distribution represent the donors of aid, while the other two represent the recipients.

A research design employed a mixed-method approach combining quantitative and qualitative methods. An online survey was conducted with 587 internally displaced persons who were beneficiaries of the Initiative, as well as 10 focus group interviews with representatives of 38 local partners through whom support was provided. In addition, 13 semi-structured in-depth interviews were conducted with representatives of aid donor organizations. Interviews were conducted over the Zoom platform, with an average duration of about 1 hour. All conducted interviews were transcribed. The data array with evaluative judgments of informants on the degree of success of their projects and activities, as well as on the intensity of influence of selected predictors on the outcome of the Initiative, contains 51 records. The database was created in the statistical program for social research SPSS 27.

Data analysis

Potential predictors identified from literature review and initial data analysis

To find the predictors for aligning the interests of the Initiative stakeholders to co-produce social value, we used the technology of causal attribution, which is defined as “ascription of a causal link between observed changes and a specific intervention” (OECD, 2007: 33). To undertake causal attribution for effectiveness evaluation of the Shelter.Ukraine Initiative, we used consistency of evidence with the causal relationship strategy. In the process of data collection and analysis, predictions were made based on the theory of change. The results of regression analysis were checked against expert predictions by interviewing key informants. All possible predictors of the Shelter.Ukraine Initiative were logically organized in a cause-and-effect diagram (Dahlgaard-Park, 2015). Furthermore, contribution analysis was used to identify non-alternative causal explanations from the spectrum of success prerequisites by asking key informants to explain the causal processes following their involvement and excluding explanations that are not specific and not crucial for the effectiveness of the Shelter.Ukraine Initiative.

Operational actors, donors, and policy-makers should “be held accountable for their contribution to humanitarian effectiveness” (Scott, 2014: i). The effectiveness of aid for internally displaced persons within the Shelter.Ukraine Initiative was evaluated through the perspectives of all stakeholders involved in its implementation. Data collected from final beneficiaries enabled the assessment of service provision levels, satisfaction of needs, alignment of expectations with received assistance, psychological support, and the enhancement of the condition of internally displaced persons. Cooperation experiences between local partners and Shelter.Ukraine Initiative representatives were positively evaluated, emphasizing the clarity and comprehensibility of application and cooperation conditions, the efficiency and flexibility of Initiative representatives, the innovative approach to assistance, and the potential of the Initiative to foster soft skills and exemplary models.

International donors perceive the Initiative’s effectiveness in meeting and surpassing fundraising and targeted spending goals, enhancing the institutional capacity of Ukrainian NGOs, adapting activities to the latest challenges of war and humanitarian crises, and establishing new mechanisms for aid provision. Ukrainian intermediaries SILab Ukraine, Teple Misto, and VPLYV Fund, acting as patronage organizations, consider parameters such as resource allocation quality, systemic monitoring, trust, and social networks of partners in evaluating the Initiative’s effectiveness.

In understanding the impact on outcomes, it is crucial to consider causal attributions as groups of properties divided into separate components. A comprehensive assessment of social intervention effectiveness requires obtaining stakeholders’ judgments on individual impact attributes, including an assessment of their importance. The interview guide was designed to capture a comprehensive range of factors influencing the success of the intervention, aligned with our theory of change. The qualitative data from these interviews were systematically coded and analyzed to identify recurring themes and patterns. By mapping these themes to the contribution analysis framework, we were able to distinguish between dispositional and situational factors, providing a structured cause-and-effect framework. This approach allowed us to pinpoint the key predictors of success, supported by robust qualitative evidence.

The study analyzed various predictors of the Shelter.Ukraine Initiative’s success, identifying key causes that led to specific measurable effects. Grouping the factors identified in the process of collecting evidence during the interview with informants allowed us to identify six content blocks (Figure 1):

Organizational factors (who provided the assistance);

Physical evidence of the results of the Initiative (what exactly describes the results of the assistance);

Human factors (for whom the assistance was provided);

Service factors (why the assistance was provided in this way);

Instrumental, procedural, communication factors (how the process of assistance was carried out);

Mother nature (why the assistance was needed) or factors that cannot be influenced.

Figure 1.

Cause-and-effect diagram of the Shelter.Ukraine Initiative’s effectiveness.

Quantitative analysis: Regression analysis to identify influencing factors of alignment

In this section, we employed multiple linear regression analysis to identify predictors that significantly contribute to the alignment of interests between aid donors and recipients in the context of the Shelter.Ukraine Initiative. Multiple linear regression was chosen due to its ability to model the relationship between multiple independent variables (predictors) and a single dependent variable (degree of success achieved by a project), allowing us to assess the relative importance of each predictor while controlling for other variables.

The dependent variable in our regression analysis was the estimated degree of achievement of the desired results by the implemented project, which can be interpreted as the degree of alignment of interests of donors and aid recipients in the context of the Shelter.Ukraine initiative. This score was obtained using a measurement instrument developed specifically for this study.

The 19 selected predictors served as independent variables in the regression model (Table 1). These predictors were selected based on theoretical considerations, previous research, and expert consultation. These predictors encompassed various dimensions, including organizational factors, service factors, instrumental factors, and procedural factors, as identified in our contribution analysis and empirical evidence.

Table 1.

Descriptive statistics of selected predictors.

	Predictors	Mean	Std. deviation	N
1	Correspondence of the Initiative to the purpose	8.02	1.157	51
2	Reliability of partners	8.22	1.316	51
3	Active exploration of recipients	7.43	1.540	51
4	Exploitation of previous contacts	7.80	1.233	51
5	Possibility of independent decision-making	6.86	1.149	51
6	Accumulated skills to work in war conditions	7.16	0.987	51
7	Adaptation of processes	7.55	1.101	51
8	Staff competence	8.16	0.903	51
9	Ability to meet needs of IDPs	7.14	1.721	51
10	Team communication	7.65	1.481	51
11	Involvement of local people	7.73	1.415	51
12	Learning-by-doing	8.57	1.005	51
13	Scaling up activities	5.53	1.347	51
14	Technology of financial distribution	7.88	1.437	51
15	Relevance and timeliness of assistance	8.57	1.100	51
16	Flexibility of the Initiative	8.20	1.000	51
17	Transparency, clarity, understandability of conditions	8.37	1.019	51
18	Organizational trust	8.67	0.864	51
19	Systemic monitoring	8.73	0.896	51

The regression coefficients associated with each predictor indicate the strength and direction of their influence on the alignment of interests, holding other variables constant. The contribution of each predictor is assessed while controlling for the effects of other predictors in the model. It provides a comprehensive understanding of how each predictor influences the dependent variable independently of other predictors. Selecting significant predictors based on the significance of regression coefficients involves examining the p-values associated with each coefficient in the regression model. Predictors with p-values below 0.05 are considered statistically significant and are retained in the model.

To build a parsimonious model that includes only the most important variables, we used the “forward” method in SPSS that systematically adds predictors based on specified criteria, helping to identify significant predictors and understand their contributions to the outcome variable (Table 2).

Table 2.

Variables entered the regression model.

Model	Variables entered	Method
1	Technology of financial distribution	Forward (Criterion: Probability-of-F-to-enter ⩽0.050)
2	Involvement of local people	Forward (Criterion: Probability-of-F-to-enter ⩽0.050)
3	Active exploration of recipients	Forward (Criterion: Probability-of-F-to-enter ⩽0.050)
4	Learning-by-doing	Forward (Criterion: Probability-of-F-to-enter ⩽0.050)
5	Exploitation of previous contacts	Forward (Criterion: Probability-of-F-to-enter ⩽0.050)
6	Ability to meet needs of IDPs	Forward (Criterion: Probability-of-F-to-enter ⩽0.050)

A multiple linear regression model was estimated to examine the relationship between the alignment of interests score and the selected predictors (Table 3).

Table 3.

Multiple linear regression model coefficients.^a

Model		Unstandardized coefficients		Standardized coefficients	t	Sig.
Model		B	Std. error	Beta	t	Sig.
1	(Constant)	0.749	0.454		1.648	0.106
1	Technology of financial distribution	0.437	0.057	0.740	7.711	0.000
2	(Constant)	0.114	0.449		0.253	0.801
	Technology of financial distribution	0.284	0.068	0.481	4.204	0.000
	Involvement of local people	0.239	0.069	0.398	3.476	0.001
3	(Constant)	−0.288	0.435		−0.662	0.511
	Technology of financial distribution	0.174	0.072	0.295	2.415	0.020
	Involvement of local people	0.241	0.063	0.401	3.798	0.000
	Active exploration of recipients	0.169	0.055	0.306	3.048	0.004
4	(Constant)	−1.437	0.660		−2.178	0.035
	Technology of financial distribution	0.188	0.069	0.318	2.706	0.010
	Involvement of local people	0.242	0.061	0.404	3.985	0.000
	Active exploration of recipients	0.128	0.056	0.232	2.289	0.027
	Learning-by-doing	0.155	0.069	0.183	2.248	0.029
5	(Constant)	−2.140	0.680		−3.145	0.003
	Technology of financial distribution	0.140	0.068	0.236	2.047	0.046
	Involvement of local people	0.235	0.057	0.392	4.093	0.000
	Active exploration of recipients	0.095	0.054	0.173	1.752	0.087
	Learning-by-doing	0.173	0.065	0.205	2.647	0.011
	Exploitation of previous contacts	0.157	0.061	0.228	2.570	0.014
6	(Constant)	−2.123	0.654		−3.245	0.002
	Technology of financial distribution	0.088	0.070	0.149	1.259	0.215
	Involvement of local people	0.206	0.057	0.343	3.622	0.001
	Active exploration of recipients	0.086	0.052	0.156	1.636	0.109
	Learning-by-doing	0.172	0.063	0.203	2.730	0.009
	Exploitation of previous contacts	0.154	0.059	0.224	2.627	0.012
	Ability to meet needs of IDPs	0.101	0.047	0.205	2.167	0.036

Dependent variable: Rate of success.

Our analysis revealed compelling insights into the predictors that significantly influence the success of the Shelter.Ukraine Initiative: technology of financial distribution, involvement of local people, active exploration of recipients, learning-by-doing, exploitation of previous contacts, ability to meet needs of internally displaced persons. Next, we are going to show that a qualitative interpretation of these predictors reduces their diversity to four main levels of influence.

Qualitative analysis to explore in-depth insights from survey and interviews

Our research into the efficacy of the Shelter.Ukraine Initiative reveals that international aid donors, spanning private, institutional, and organizational levels, were largely driven by emotions. Their objective was to swiftly provide assistance without protracted periods of cooperation, bureaucratic hurdles, or extensive paperwork, aiming to address the immediate needs of the most vulnerable individuals on the ground in Ukraine. International donors primarily sought to support organizations capable of delivering immediate assistance, well-acquainted with the local context, and adept at addressing the pressing needs of people in the vicinity without waiting for an emergency response. In addition, there was an alignment of donors’ interests with the goal of enhancing the institutional capacity of local organizations rather than depleting their potential and resources.

The Shelter.Ukraine Initiative not only fulfilled the humanitarian motives of donors but also had personal benefits for them. It elevated their status and reputation internationally, yielding certificates of organizational reliability for future fundraising activities. Furthermore, participants from donor organizations engaged in the Pro Bono Initiative, fostering teamwork and emphasizing relationships. For representatives of international donor organizations, this Initiative provided a departure from their routine activities, offering an opportunity to gain experience in supporting organizations beyond their usual scope. This less-tangible outcome served as a formative experience for these organizations, potentially prompting a re-evaluation of their flexibility and self-perception.

The analysis of interviews with representatives of donor organizations showed that a number of antecedents contributed to the involvement of donors and the implementation of the Initiative:

Donors are engaged—the Initiative creates opportunities to fulfill the tasks that donors have identified for themselves in the context of open Russian aggression against Ukraine;

Donors come to the Initiative because they believe in its vision and feel satisfied after giving—interest in working with reliable partners;

Donors are mission-driven—the alignment of the Initiative mission with donors’ priorities;

Donors understand their impact—correlation of the Initiative mission with the donor’s strategic directions.

At the same time, donors also noted a number of risks they faced at the beginning of their cooperation with the Shelter.Ukraine Initiative, including complications in long-term planning, lack of a mechanism for providing assistance developed over many years of experience, limitations of their own capabilities, and rigidity of the assistance procedure.

Under the Shelter.Ukraine Initiative, Ukrainian entities SILab Ukraine, Teple Misto, and VPLYV Fund endorsed 173 projects from local NGOs, charitable organizations, volunteer groups, or local governments. In our evaluation interviews, all management representatives from partner organizations highlighted the Initiative’s clear and comprehensible application and cooperation conditions. When discussing what sets the Shelter.Ukraine Initiative apart, organizations commended the effectiveness and flexibility demonstrated by the Initiative representatives.

Local Ukrainian civil society respondents asserted that the Shelter.Ukraine Initiative stands out as a model for other foundations, volunteer initiatives, and foreign organizations. They praised its organizational qualities, including high-quality communication, a collaborative atmosphere, transparency, robust social capital and networks, and systematic monitoring of internal processes preceding organizational and managerial decisions.

Acting as intermediaries between donors and local NGOs, SILab Ukraine, Teple Misto, and VPLYV Fund faced a managerial challenge in achieving project success, reinforcing local NGOs, and sustaining fundraising and services aligned with each organization’s competencies. Navigating the complexities of war, they skillfully utilized leadership and communication competencies, demonstrating organizational ambidexterity by balancing exploration of new opportunities with exploiting existing capabilities. This organizational agility emphasized their innovative activities, cross-functional teams, and close inter-organizational collaboration. For civil society organizations, innovation aligns with their core mission, benefiting both beneficiaries and the broader community.

The study of the Initiative’s effectiveness showed that IDPs positively assessed the services provided by local organizations to meet their needs: 84.8% positively assessed the food provided; 81.1% of respondents liked the hygiene products received; 72.3% positively assessed the assistance in the category of “daily necessities”; 71.2% liked the places for long-term residence; 69.8% positively assessed the hygiene products for children and infants; 68.1% positively assessed the places for temporary accommodation provided by the Shelter; 65.5% of respondents found the humanitarian aid in the form of clothing and footwear to be good; 62.9% liked the medicines and medical devices received; 62.9% gave high marks to baby food packages.

It is widely recognized that foreign aid has a positive impact on health and humanitarian needs. However, “the usefulness of aid will depend, first, on what types of aid are used in the given circumstances and, second, on how they are planned and implemented” (Park, 2019: 45). The vast majority of internally displaced persons (86.6%) are generally satisfied with the quality of services/goods they used/received under the Shelter.Ukraine Initiative. For 94.4% of respondents, their expectations regarding the assistance received were fully met.

Results

Causal attribution theory usually operates with two groups of attributes: dispositional, or internal, and situational, or external. According to this logic, all the predictors of the Initiative’s effectiveness can be divided into two macro groups: The dispositional ones include organizational conditions and results of assistance (who, what, for whom), and the situational ones include circumstances and processes in which assistance was provided (why, how, because of what). The configuration of relations between donors and recipients of a humanitarian aid has a significant impact on the effectiveness of the aid. But the configuration itself depends primarily on the prevalence of certain antecedents and enablers. The role played by each crucial predictor in the Initiative’s performance can be determined by contribution analysis—“a practical approach using logical argumentation to assess the extent to which an intervention was a contributory cause to observed changes in outcomes” (Wimbush et al., 2012: 312).

Our interpretation of the significance of the predictors of the regression analysis model in the above context is to reduce the six significant factors to four by eliminating one factor and combining the other two.

Since the interviews were conducted with NGOs and local volunteers who implemented the projects, their insights into the effectiveness of the projects, including their ability to meet the needs of IDPs, are invaluable. These stakeholders are directly involved in project planning, implementation, and monitoring, making their perspectives highly relevant for understanding project success. But precisely for this reason, we tend to consider this factor as one that automatically manifested its significance due to receiving information from those recipient organizations that received funding under the Initiative for the implementation of a targeted project aimed at supporting IDPs.

The factors of active exploration of recipients and exploitation of previous contacts can be combined into a broader category of “ambidexterity of organizations” based on their shared underlying principle of organizational adaptability and responsiveness. Such grouping provides a cohesive framework for understanding and promoting organizational adaptability, flexibility, and strategic alignment in the context of humanitarian aid initiatives like the Shelter.Ukraine Initiative. Our approach emphasizes the importance of balancing exploration and exploitation activities to achieve sustainable and impactful outcomes.

The use of innovative technology for fund accumulation and distribution emerged as a significant predictor of the Initiative’s success. Projects that effectively leveraged advanced financial distribution methods demonstrated higher levels of success, emphasizing the importance of streamlined processes and transparent fund utilization.

The active involvement of local communities and individuals in project implementation was found to be a significant predictor of success. Initiatives that prioritized community engagement and participation exhibited greater resilience and sustainability, underscoring the value of grassroots empowerment and collaboration.

The ability to learn from on-the-field experience and adapt strategies accordingly emerged as a significant predictor of success. Organizations that embraced a culture of continuous learning and improvement exhibited greater flexibility and resilience, enabling them to navigate uncertainties and optimize project outcomes over time.

The conducted contribution analysis of influencing factors eliminated the root causes of the effectiveness of the Shelter.Ukraine Initiative that have become mandatory (non-alternative) predictors of successful match between aid donors’ and recipients’ interests. Empirical evidence on achieved results and underlying assumptions allowed us to identify four crucial reasons that led to the achievement of the Initiative’s goal in the current circumstances: organizational factor—ambidexterity of civil society organizations (international donors and Ukrainian intermediaries); service factor—structure of relations (involvement of local people, joining efforts and learning from each other); instrumental factor—new technology of financial distribution; procedural factor—learning-by-doing.

Indeed, these factors exhibit paired interdependence. Ambidexterity, as a strategy for organizations navigating uncertainty, inherently involves learning-by-doing. This hands-on approach entails accumulating on-the-field experience, directly executing tasks, and transforming organizational behavior to align with the interests of all stakeholders. For civil society organizations, it also entails receiving feedback from those directly involved in organizing assistance and integrating these insights into future activities within a specific context. Our research indicates that, in these conditions, all civic, charitable, and volunteer organizations operate to some extent with limited foresight. Nevertheless, grassroots Ukrainian civic activists are better equipped to operate amid uncertainty. They possess extensive networking relationships, are unafraid to pose questions, and exhibit flexibility in responding to requests. Engaging with communities provides them with relevant experience, connections, and contacts, enabling them to align efforts with government agencies, local councils, partners, and donors. This practical approach involves collecting and verifying information, ensuring funds are directed toward their intended purpose.

The innovative structure of relations, resembling the classic umbrella model but with a notable shift in decision-making authority for spending to grassroots entities, necessitated the integration of advanced technology for fund accumulation and distribution. According to international aid donors, prioritizing social impact and, in emergency situations, survival led to the sidelining of various indicators and bureaucratic processes. Instead, the distribution of funds to target audiences is entrusted to grassroots partners on the ground. Ukrainian NGOs now collaborate more extensively, enabling broader coverage of audiences and territories. Flexibility, experience, the ability to facilitate dialogue between communities and government, and a prompt response to people’s needs emerge as crucial qualities for NGOs in challenging times.

Other influencing factors were assessed and either found to have made an insignificant contribution or their relative role in achieving the desired result was acknowledged.

These results highlight the importance of combining qualitative and quantitative approaches in contribution analysis. By integrating the ability to ask the right contribution questions with a robust mathematical mechanism for identifying influential factors, we were able to generate actionable insights that can inform decision-making, program design, and future research efforts. Our evaluative study contributes to the advancement of evaluation practice by offering a comprehensive, context-sensitive approach that integrates contribution analysis, combines qualitative and quantitative methods, emphasizes contextual factors, and identifies influential predictors. These contributions have implications for practitioners, policy-makers, and researchers seeking to design, implement, and evaluate interventions that make a meaningful difference in the lives of beneficiaries.

Discussion

The discussion of our results underscores several key findings and insights derived from our evaluative study of the Shelter.Ukraine Initiative.

First, we categorize the predictors into two overarching groups: dispositional and situational attributes. Dispositional attributes encompass organizational conditions and results of assistance, while situational attributes encompass the circumstances and processes surrounding aid provision. This distinction enables us to contextualize the predictors within a broader framework of causal attribution theory, elucidating how different factors contribute to the Initiative’s effectiveness.

Our interpretation of the regression analysis model has led us to consolidate the six significant predictors into four, by eliminating one factor and combining two others. This consolidation streamlines our understanding of the key drivers of the Initiative’s success, providing a clearer focus for future interventions and program improvements. The role played by each predictor can be discerned through contribution analysis, a practical approach that assesses the extent to which an intervention contributes to observed outcomes.

Our findings have significant implications for evaluation practice, particularly in the context of assessing humanitarian aid initiatives. The insights gleaned from our study offer valuable guidance for future evaluations, informing both the design and implementation of humanitarian interventions. Future research should explore additional predictors and consider contextual factors to further enhance our understanding of project success in humanitarian settings.

Limitations

The presence of statistical significance for variables in a regression analysis means that these variables have a statistically significant effect on the result. However, this does not necessarily mean that this influence is decisive. The decisive influence of variables depends on the context and objectives of the study. Sometimes even insignificant variables can have practical significance or be important for building a model. Thus, it is necessary to consider both the statistical significance and practical significance of variables when interpreting the results of regression analysis.

Another limitation may be related to the problem of endogeneity or endogenous variables in the regression model. When data on influencing factors were obtained only from those who were relevant to the outcome (as, e.g. in our case of surveys of opinions and expert assessments of donors and recipients of humanitarian assistance under the Initiative), there is a risk of endogeneity.

Endogeneity means that the explanatory variables (predictors) in the model may be correlated with the model error or with each other. This means that the variables that influence the outcome may have been distorted or skewed by the subjects whose scores are used as data. This may lead to incorrect conclusions about the influence of these variables on the outcome.

Conclusion

The evaluation of the civic initiative aimed at assisting internally displaced persons has enabled the identification of key influencing factors aligning the interests of donors and aid recipients. Summarizing the results, it can be noted that the factors that contributed to the effective implementation of the Shelter.Ukraine Initiative can be summarized as “going beyond” the traditional mechanisms of international humanitarian aid. This innovative approach, rooted in four pillars of success—organizational ambidexterity, learning-by-doing, an original inter-organizational structure, and advanced technology for financial resource allocation—proves more pronounced at the grassroots level and among direct implementers.

The invasion of the Ukrainian territory by Russian troops has underscored the international necessity to reconfigure the humanitarian aid architecture. International donors must adapt their organizational models to incorporate and bolster the role of local organizations and volunteers. Insights gained from the effectiveness evaluation of international aid programs should serve as a foundation for enhancing the methods and mechanisms of aid delivery, disseminating lessons learned to the widest possible range of stakeholders.

Against the background of the growing number of international aid programs and projects aimed at supporting Ukraine and its post-war reconstruction, it is necessary to note a number of difficulties faced by specialists in monitoring and evaluating these programs, including the case under study. The most significant challenges that arise in connection with evaluation in conditions of uncertainty and instability are as follows:

It has become more difficult to conduct surveys on time due to delays and interruptions in the schedule, anxiety, security risks, and power outages.

Access to respondents has become more difficult, and it is harder to contact them.

The results of the analysis have no basis for comparison.

The results of the analysis have low predictive power.

Remote work and the use of digital technologies during data collection have a negative impact on the reliability of the data collected.

The phenomena under study are difficult to assess in terms of specific indicators.

Respondents have become less willing to answer questions truthfully.

Cooperation with foreign consultants who do not understand the context of the evaluation is very difficult.

It is clear that the above changes in professional activities require additional training or adaptation to new realities. The nature of evaluation work and its content are changing under the influence of the war. For example, the increase in social interventions and initiatives in Ukraine provides ample opportunities to apply conceptually new approaches to evaluation and to test the effectiveness of these evaluation methods in practice. In addition, the approaches to data collection and analysis in the context of war have changed significantly. Methods of data collection tend to favor qualitative over quantitative approaches, moving away from strictly scientific methods and into alternative and applied categories. Digital research techniques have become predominant, and modern data-collection methods often violate human rights to anonymity and protection.

Despite the complexities and uncertainties associated with conducting evaluations in a war context, our study underscores the critical importance of such efforts. Evaluation, even at a time of war, provides invaluable insights that can improve the alignment between donors’ intentions and recipients’ needs. By systematically analyzing the factors that contribute to the success of humanitarian initiatives, we can ensure that aid is delivered more effectively and efficiently. This evaluation demonstrates that rigorous, evidence-based approaches can generate actionable insights that benefit both donors and beneficiaries. For donors, understanding which strategies are most effective ensures that resources are allocated wisely and have the maximum impact. For beneficiaries, the insights gained from such evaluations help ensure that their most urgent needs are met in a timely and effective manner.

Our study demonstrates the importance of adopting a comprehensive assessment framework that integrates both qualitative and quantitative methods. By combining contribution analysis with regression modeling, we were able to capture the multifaceted nature of the Initiative’s effectiveness, providing a nuanced understanding of the factors driving success. This approach ensures that evaluations consider a broad range of factors and perspectives, leading to more robust and actionable findings. Understanding predictors-of-interests alignment is essential for humanitarian aid as it enables aid organizations to allocate resources more efficiently, enhance sustainability, reduce dependency, and minimize conflicts and misunderstandings. By prioritizing initiatives that align with the interests and priorities of both donors and recipients, humanitarian organizations can maximize their impact and contribute to positive and sustainable outcomes for affected communities.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The study of the effectiveness of the Shelter.Ukraine Initiative was funded by the VPLYV Fund.

ORCID iDs

Oleg Mazuryk

Marharyta Shyrokova

Oleg Mazuryk, Doctor of Sociological Sciences, is Associate Professor; Head of the Social Structures and Social Relations Department, Faculty of Sociology, Taras Shevchenko National University of Kyiv; Head of BA program “Social Technologies”; Monitoring and evaluation expert; Head of the Ukrainian Evaluation Association; and Founder of NGO Center for Social Audit.

Marharyta Shyrokova, PhD in Economic Sociology, is Associate Professor of Social Structures and Social Relations Department, Faculty of Sociology, Taras Shevchenko National University of Kyiv, and is Member of NGO Center for Social Audit.

References

Antoniv

Kalyta

Kondratenko

, et al. (2023) Evaluation during war: Current realities and future possibilities of Ukrainian monitoring and evaluation. Evaluation 29(1): 67–72.

Bandyopadhyay

Vermann

(2013) Donor motives for foreign aid. Federal Reserve Bank of St. Louis Review 95: 327–36.

Banks

Hulme

Edwards

(2015) NGOs, states, and donors revisited: Still too close for comfort? World Development 66: 707–18.

Barrington

Triana-Tremain

(2022) Evaluation Time: A Practical Guide for Evaluation. Thousand Oaks, CA: Sage.

Bigsten

Tengstam

(2015) International coordination and the effectiveness of aid. World Development 69: 75–85. Available at: https://www.econstor.eu/bitstream/10419/80990/1/689698372.pdf

Bourguignon

Platteau

J-P

(2013) Optimal discipline in donor—recipient relationships—Reframing the aid effectiveness debate. HalSHS. Available at: https://shs.hal.science/halshs-00960570/

Büthe

Major

de Mello

Souza

(2012) The politics of private foreign aid: Humanitarian principles, economic development objectives, and organizational interests in NGO private aid allocation. International Organization 66(4): 571–607. Available at: http://www.jstor.org/stable/23279972

Dahlgaard-Park

(2015) Fishbone diagram. In: Dahlgaard-Park

(ed.) The Sage Encyclopedia of Quality and the Service Economy. Thousand Oaks, CA: Sage, 237–40.

Delahais

Toulemonde

(2012) Applying contribution analysis: Lessons from five years of practice. Evaluation 18(3): 281–93.

10.

Development Initiatives (2022) Global humanitarian assistance report 2022. Available at: https://devinit.org/documents/1221/GHA2022_Digital_v8_IdHI18g.pdf

11.

Forss

Marra

Schwartz

(2017) Evaluating the Complex: Attribution, Contribution and Beyond. New York: Routledge.

12.

Hassnain

Kelly

Somma

, et al. (2021) Evaluation in Contexts of Fragility, Conflict and Violence: Guidance from Global Evaluation Practitioners. Exeter: IDEAS. Available at: https://ideas-global.org/wp-content/uploads/2021/06/EvalFCV-Guide-web-A4-HR.pdf

13.

Lemire

Nielsen

Dybdal

(2012) Making contribution analysis work: A practical framework for handling influencing factors and alternative explanations. Evaluation 18(3): 294–309.

14.

Mayne

(2012) Contribution analysis: Coming of age? Evaluation 18(3): 270–80.

15.

McKinley

Little

(1979) The U.S. aid relationship: A test of the recipient need and the donor interest models. Political Studies 27(2): 236–50.

16.

Nissanke

(2008) Donor-recipient relationships in the aid effectiveness debate. In: Jerve

Shimomura

Hansen

(eds) Aid Relationships in Asia. London: Palgrave Macmillan, 22–40.

17.

OECD (2007) Glossary of key terms in evaluation and results based management. Available at: https://www.oecd.org/dac/evaluation/dcdndep/39249691.pdf

18.

Park

J-D

(2019) Assessing the role of foreign aid, donors and recipients. In: Park

J-D

(ed.) Re-Inventing Africa’s Development. Cham: Palgrave Macmillan, 37–60.

19.

Rahmani

(2012) Donors, beneficiaries, or NGOs: Whose needs come first? A dilemma in Afghanistan. Development in Practice 22(3): 295–304. Available at: http://www.jstor.org/stable/23214534

20.

Rogers

(2014) Overview: Strategies for causal attribution, methodological briefs. Impact Evaluation no. 6. Florence: UNICEF Office of Research. Available at: https://www.unicef-irc.org/publications/pdf/brief_6_overview_strategies_causal_attribution_eng.pdf

21.

Rowe

(2019) Rapid impact evaluation. Evaluation 25(4): 496–513.

22.

Scott

(2014) Imagining more effective humanitarian aid: A donor perspective. OECD Development Co-operation Working paper 18. Paris: OECD. Available at: https://www.oecd.org/dac/Imagining%20More%20Effective%20Humanitarian%20Aid_October%202014.pdf

23.

Sen

(2018) Strategic aid: Explaining the motives and choices of international donors. Dissertation, Georgia State University, Atlanta, GA.

24.

Shelter Ukraine (2022). Available at: https://www.silabua.com/en/post/our-shelter-ukraine-initiative-received-an-award-within-the-framework-of-the-national-competition

25.

Sokol

Stiegert

(2008) An empirical evaluation of long term advisors and short term interventions in technical assistance and capacity building. GCLC Working paper 02/08. Bruges: Global Competition Law Centre, College of Europe. Available at: https://www.coleurope.eu/sites/default/files/research-paper/gclc_wp_02-08_0.pdf

26.

Thiele

Koch

D-J

Nunnenkamp

, et al. (2008) NGO aid—Well targeted to the needy and deserving? The Centre for Economic Policy Research. Available at: https://cepr.org/voxeu/columns/ngo-aid-well-targeted-needy-and-deserving

27.

Wimbush

Montague

Mulherin

(2012) Applications of contribution analysis to outcome planning and impact evaluation. Evaluation 18(3): 310–29.