15. Evaluation

Introduction

In his summary of the conference in Phuket following the 26 December 2004 earthquake and tsunami in South East Asia, David Nebarro asked the question, “why have we not learned from what we have learned?” [1] For decades, the same errors and faulty interventions have been repeated during disaster after disaster [2]. The primary reasons for this failure to learn relate to that either no assessments have been performed, or those that have been conducted are inaccessible or not structured in a manner that facilitates their analysis. Also, conclusions may not have been in concert with other agendas of the organisation that provided the intervention. The only way to truly learn from what we have learned is to identify the needs, take appropriate actions (interventions) that are based on evidence (best practices), and then evaluate the final result related to the goal using structured, robust, and reproducible methods. Evidence is the basis of science; the only way to identify best practices is to evaluate what has been done and appropriately report the findings.

Thus, all relief and recovery responses (interventions) must be evaluated for the: (1) efficiency in reaching each of the objectives; (2) effectiveness of the processes used to reach the objectives; (3) the extent the reached objectives actually helped reaching the goal; and (4) other effects (anticipated and unanticipated) that influenced the goal negatively or positively.

Each specific project is designed to contribute to the stabilisation and/or the recovery of the affected population. Did it succeed? What benefits accrued to the society as a result of the project? What were the costs associated with the project? Which elements of the project fostered its success, and which of the processes or parts of the processes could be improved? The structure provided in these Guidelines should form the basis for all such evaluations. Furthermore, the findings must be generally available so that the lessons learned can be integrated into future projects. Some feel it is best that such evaluations be conducted by evaluators who are independent from the organisation that provided the project, but this philosophy is without general agreement (e.g. participating observation) [3].

Evaluations can be conducted on a project in terms of inputs, effects (outputs), outcomes/impacts on the affected population, and the resources consumed by the transformation process (human, materials, financial, environmental, and opportunity costs). Evaluations must be related to both the objectives and the goals of the intervention; cost-efficiency and effectiveness in achieving the objectives and cost–benefits (in achieving the goals) can be derived and should be compared with other interventions by the same or different providers. In addition, evaluators can examine one or more of the processes related to the intervention, different interventions aiming at the same objective and different interventions aiming at different objectives but the same goal.

For the most part, evaluations conducted to date have focused on the effects of the interventions executed using achievement indicators (i.e. the number of victims served, the numbers of immunisations provided, the numbers of boats replaced, the number of surgical procedures provided [4]. Little has been done in relation to the study of the processes that are associated with assessments, the identification of needs, the development of a strategic plan, the selection of interventions and other actions, operational planning, or the implementation of the interventions? We do not learn because we have not answered the questions, “(so) what?” and “why”? Studies should examine these processes with reference to the relative success (or failure) of a project in attaining the objectives for which it was selected and achieving all or part of the overarching goal.

Several additional questions are cogent. If the project was a success, which elements in the process used to transform assessments into outcomes were critical in attaining the desired goal outlined in the strategic plan? Was it the project at question that was responsible for the improvement (see chapter 6 in conceptual framework, best outcome without assistance) [5 pp102–12]. If other effects occurred with the intervention, what were the impacts of these other effects of the project on the affected society? Did these other effects add to the benefits accrued by the society or did they make things worse? Did the downside effects of the intervention on other functions have a greater negative impact than what was achieved through the function (dysfunction) addressed? Was the negative impact related to inappropriate implementation or to an inadequate, erroneous assessment? Did the intervention cost too much? Did the indicators selected really have the construct validity that truly reflected the effects of the intervention? Were all of the opportunity costs identified? Do the results of the evaluation indicate sufficient value that the intervention should be used again? Where in the array of processes outlined in the foregoing chapters, did a failure occur? These are but a few of the potential questions that should be answered by the conduct of evaluations.

Thus, evaluations conducted in regards to disasters or any crisis should be studied in much greater detail than generally is done today. Doing so will add evidence to the science of health disaster and allow us to really learn from what we have learned.

There always are at least two questions:

Thus, not only the production process associated with actions including interventions must be studied, but also the reasons for their success or failure must be isolated and remedied before the next time the intervention is being considered.

Outputs

The outputs of responses (actions/interventions) are products. Using the logic model [6, 7] (Figure 15.1), the outputs must be compared to the objectives for which the actions were implemented. Other effects of the transformation process may have greater significance on other functions also related to the outcome(s) for which the project was designed.

OPEN IN VIEWER

Figure 15.1.

Diagrammatic representation of cost-efficiency, cost-effectiveness, and cost–benefit. Each intervention must be carried out as efficiently as is possible. Then that intervention that proves more effective to deliver the wanted output (effect) is chosen. Still it remains to be evaluated to what extent the intervention really benefited the population (produced a positive outcome). This diagram has been expanded from chapter 6 of the conceptual framework [5] to distinguish between immediate outcomes and eventual later outcomes that may influence the overall long-term outcome positively or negatively. The use of antibiotics and development of multiresistance could serve as one example.

Outcomes

Outcomes are “the way something turns out” [8]; consequences [9]. It is frequently used interchangeably with impacts [10]. Generally, the effects of a response relative to a defined need are judged a “success” if they meet the objectives for which they were selected. However, it is important also to evaluate their contribution to the overall goal. If the objective of an immunisation campaign against measles was to vaccinate 10,000 children in the camp against measles, and at least 10,000 children were vaccinated, then the action did achieve the expected output, and it was a “success”. However, if the overall goal was to prevent an outbreak of measles in the camp by vaccinating 10,000 children, and an outbreak of measles did occur, then the desired impact of the intervention was not achieved [4]. While the programme met its objective (as indicated by the achievement objective), its achievement did not contribute to reaching the overarching goal. The intervention was not beneficial, and the chosen objective was not adequate to contribute to achieving the overarching goal of preventing an outbreak of measles in the camp. If an outbreak of measles did not occur, was it because the vaccination project was a “success”, or would it not have occurred even if the vaccination project had not been implemented? Did the project produce a benefit to the population at risk?

If the project could not meet its objective of vaccinating 10,000 children (achievement indicator) because the vaccine was administered only to 9000 children, the project did not reach its intended objective. Then a cogent question is, “why did it not meet its intended objective?” Perhaps, the objective, as defined in the operational plan, was too ambitious for the amount of available resources (the quantity of vaccine available, too few people to administer it, or too few unvaccinated children in the camp). Was the initial assessment incomplete or inaccurate and thus, the needs identified were incorrect? Was the plan flawed because of erroneous synthesis of the factors used to develop the strategic plan to meet the need?

In order to identify what prevented the project from reaching its objective, the processes used must be examined, and the critical points of failure must be identified. It is anticipated that the next time such a project is proposed, corrections of these critical points of failure will have been made based on these evaluations. A comprehensive evaluation must consider elements of the processes used as well as outcome indicators.

Benefits

Even though an intervention met its objective(s) and, therefore, was deemed a “success”, it may not have produced a benefit to the society for which it was provided. A benefit is whatever is for the good of a person or thing; a favourable or helpful factor or circumstance [5 p146]; benefit is a positive outcome [5 p102–12, 11, 12]. As noted, each intervention must address a specific need. Satisfaction of a need is the objective of all interventions, and achieving the objective should be associated with a benefit to the society. The determination of needs from the assessments is based on the knowledge of what benefit can be accrued from the implementation of an intervention to meet the defined need. Thus, the “value” of any output from the production process not only must be judged by whether it met all or part of the objectives for which the action/intervention was selected, but whether it had produced a benefit (positive outcome) to the affected society. If the outcome was beneficial, why was it a benefit? If the outputs did not result in a benefit to the society, what went astray? Where in the processes used was the critical point(s) of failure, and how can this process be modified so that the next time the intervention is considered, the project will fill a need and, thus, produce the desired benefits? Was it the wrong project due to incorrect assessments or interpretation of assessments? Who benefited and who did not?

Benefits can be associated to other effects as well as to those for which the intervention was selected. As noted, the benefits of the other effects may outweigh those for which the project was intended. On the other hand, negative impacts may outweigh the benefits produced. Where in the complex processes used in generating and executing the project was/were the critical points of failure?

Benefits may be difficult to quantify. Often the term outcome measures is used [10]. However, in these guidelines, the term indicators of benefit is used as the indicators may be qualitative, quantitative, or a combination of both. This is in line with the World Bank Independent Evaluation Group which uses the term outcome indicators [13]. Indicators must be chosen with care and their use validated.

It is worth mentioning that the World Bank operates with both impact evaluation and outcome evaluation. Outcome evaluation is more general and cannot sufficiently attribute improvement to the intervention. Impact evaluation, however is capable of identifying a link between improvement and intervention, if there is one [14].

Other effects

All actions and interventions have the potential to produce effects that are not related to the objective for which they were selected. The magnitude of these effects may be minimal or may have greater significance than the achievement of the designated outcome. These other effects must be accounted in the evaluation of the project. For example, were the side effects associated with the administration of the vaccine in the above hypothetical example minimal, or did its administration produce profound reactions at the injection site or even systemic manifestations? How many severe, local reactions occurred, and what were the consequences? Did some children die following the administration of the vaccine? If so, why? Vaccine-related encephalitis may be more common than previously anticipated [15]. The observed incidence of narcolepsy in children and adolescents in most Nordic countries as from 2010 has been clearly associated to the Pandemrix vaccine (after the pandemic H1N1 vaccination campaign late 2009 and early 2010) [16, 17].

Other effects need not always be negative. In fact, one or more of the other effects could be positive. If so, they must be documented and studied. The intervention may become useful for other purposes rather than for the ones for which it originally was selected.

Costs

Costs are a loss, sacrifice, and/or an expenditure of resources [5 pp109–11,148]. Thus, costs are the resources consumed by the transformation process. Costs are discussed here to emphasise the importance of considering costs as an important component of evaluations. Costs may be identified as economic, materials, human, opportunity, and/or environmental. Costs may be directly related to the damages caused by the event or indirectly as a result of dysfunctions caused by the damage. Indirect costs are not due to the energy released by the event. The indicators of cost are too many to delineate here. Importantly, the attempts to define costs always relate to the type and amount of the damage to structures and relate to the pre-event status or to the last assessment available.

Projects are encumbered by costs irrespective of their final output and outcome. Efficiency refers to the costs of any action/intervention [5 p106]. The greater the efficiency for a given output, the less is the cost of the process used to attain that output or the more that is attained for the same input. Two projects with the same outputs may have very different costs. The lower the costs, the greater the effectiveness in producing the specific output. Often, the term “cost-efficiency” is used, but strictly speaking, costs are imbedded in the term efficiency being measurable through the ratio of output to inputs [18].

The bottom line is the costs of the benefits accrued. Cost versus benefits not only should be estimated for the resources consumed in achieving the objective(s), but also should be estimated for the other effects of the intervention as well. There will always be opportunity costs. Much work remains to be done in identifying the costs and benefits of actions/interventions in health disasters.

As it is difficult, at least, to establish proper control groups for evaluation and research in a disaster setting, cost–benefit is difficult to calculate and also to document/prove if the ultimate best outcome is that nothing happens.

Uses of evaluations

Evaluations are used to enhance the quality of the interventions applied. Evaluations are a form of quality improvement and the findings feed directly into preparedness. They provide the evidence required to enhance preparedness. Evaluations should not be used as a basis for withdrawal of support for future projects. Donors should be aware that evaluations are for their benefit as well as for the providers of the interventions and the recipients of future interventions. Donors certainly are interested in having ever improving outcomes from their investments. Evaluations produce good returns for all involved.

Summary

Evaluations are the methods by which the science of health disaster management is built. The only way to evolve standards, best practices related to these standards, and competencies related to best practices, is through the conduct of evaluations. Evaluations are conducted to make things better for the next time (less damage and dysfunction, more rapid and complete recovery). They should be used for quality improvement of interventions – the next project will be even better. Evaluations provide the keys to preparedness.