American Journal of Evaluation

Evidence-based programs and grassroots programs are often adapted during implementation. Adaptations are often hidden, ignored, or punished. Although some adaptations stem from lack of organizational capacity, evaluators report other adaptations happen in good faith or are efforts to better fit the local context. Program implementers, facilitators who need to adapt during implementation, do not always report adaptations because they fear losing funding if the program is not implemented with fidelity. Program personnel including program evaluators need this information to improve effectiveness of programs, and to determine whether an adaptation is still consistent with the theory of change. Evaluators also need this information for generalizing results to varied settings and populations. Following the PRECEDE–PROCEED model, we recommend a hybrid approach to fidelity and adaptation. We argue in favor of advance planning to accommodate potential adaptations. Such planning also establishes evaluation criteria for determining whether adaptations are helpful, harmful, and appropriate to the context. We illustrate some types of adaptations that can occur, why they may be needed, and how to structure transparent reporting about adaptations to program developers and funding organizations.

This paper develops a novel approach called Outcome Trajectory Evaluation (OTE) in response to the long-causal-chain problem confronting the evaluation of research for development (R4D) projects. OTE strives to tackle four issues resulting from the common practice of evaluating R4D projects based on theory of change developed at the start. The approach was developed iteratively while conducting four evaluations of policy-related outcomes claimed by the CGIAR, a global R4D organization. The first step is to use a middle-range theory (MRT), based on “grand” social science theory, to help delineate and understand the trajectory that generated the set of outcomes being evaluated. The second step is to then identify project contribution to that trajectory. Other types of theory-driven evaluation are single step: they model how projects achieve outcomes without first considering the overarching causal mechanism—the outcome trajectory—from which the outcomes emerged. The use of an MRT allowed us to accrue learning from one evaluation to the next.

Organizational evaluation policies describe how evaluation practices should be structured and implemented. As such, they provide key insights into organizational priorities and values regarding evaluation. However, the link between evaluation policies and how evaluation policies translate into concrete practices has seldom been explored until now. Our study examines the implementation of two Canadian federal government evaluation policies over a 10-year timespan, through the secondary analysis of reports produced on behalf of governmental evaluation functions. Our findings show that some policy elements have been fully implemented, but the way in which these have been implemented varies between organizations. Further, we observed that the level of control of various organizational members responsible for implementing policy elements, as well as time, can influence implementation of certain policy requirements. We conclude by proposing further directions for research to examine the policy-practice link.

Principles focused evaluation (PFE) can complement existing formative and outcome evaluation plans by identifying Effectiveness Principles (EPs), an operationalization of values and standards that guide practitioners during program implementation. To date, however, few examples of PFE are available in the literature. This description of the application of PFE to the Washington State Rape Prevention and Education (RPE) sexual violence prevention program provides an example of how this flexible approach can augment an existing evaluation plan to distill shared evaluation components across different organizations implementing diverse prevention programming. Specifically, we describe the process used by a team of practitioners, funders, evaluation consultants and state-level sexual violence prevention technical assistance providers to identify EPs, operationalize indicators for each EP, and develop and test an EP measurement approach. In this process, the seven very different RPE-funded organizations, each serving a unique community, were able to identify and endorse shared, core EPs. This description illustrates PFE's promise for augmenting a shared evaluation approach and identifying common guiding tenets across uniquely situated organizations in a larger community of practice.

We developed principles drawn from empowerment and realist evaluation theories to guide a family youth alcohol and drug program evaluation, before reflecting on our experience to develop additional principles for future work. The research team included the program clinician and an independent researcher. A purposive sample of young people and one parent took part in interviews. The principles we applied include practical and transformative elements: a collaborative, multidisciplinary team; valuing client perspectives; diffusing power; the belief that programs aim to achieve positive results; capacity building; and sharing information for personal benefit. Additional principles were having young people on collaborative research teams, addressing organizational readiness to embed young people’s involvement in research, accounting for gatekeepers in recruitment processes, and using information communication technology. While principles should be flexibly applied at project level, they can guide project design and encourage the development of organizational systems that support collaborative inquiry.

Collective impact (CI) is a structured approach that helps drive multi-sector collaborations to address social problems through systems changes. While the CI approach is gaining popularity, practitioners experience challenges in evaluating its implementation and intended outcomes. We conducted a systematic scoping review to understand evaluation methods specific to CI initiatives, identify challenges or limitations with these evaluations, and provide recommendations for the design of CI evaluations. Eighteen studies met the inclusion criteria. Process evaluations were the most frequently used evaluation design. Most studies collected cross-sectional data to evaluate their efforts. The complexity of CI was most frequently cited as the greatest evaluation challenge. Study recommendations primarily focused on improvements during the evaluation planning phase. Taking careful consideration in the planning of CI evaluations, developing context-specific data collection methods, and communicating results intentionally and effectively could prove useful to sufficiently capture and assess this systems-level approach to address social problems.

Power has always been an element of the program evaluation field, however, it has traditionally been attributed a negative connotation. With that in mind, power is seen as a force possessed to some extent by different stakeholders that can be imposed over others, and by which certain views, ideas, and decisions can be adopted. Given that program evaluation has a political character in which power is always present, we seek to explore a different notion of this concept. In this paper, we intend to take advantage of Michel Foucault's ideas to explore power in the program evaluation field. We illustrate the introduction of these concepts in collaborative evaluations and in doing so, we also explore power as well as power-knowledge relations, adopting a transformative perspective on power. The introduction of these concepts in collaborative evaluation is useful in illustrating how they can be used in non-collaborative evaluations.

This study challenges persistent misrepresentations of evaluation as a value-neutral inquiry process by presenting an empirical study that deepens understanding of evaluators’ values and how they “show up” in evaluation practice. Through semistructured interviews and inductive analysis, we examined the values advanced by a sample of eight experienced evaluators. We surfaced and examined 12 values, which we organized into five clusters, that shaped the constitutive elements of the studies these evaluators conducted and guided how the evaluators positioned their work. Our findings provide empirical evidence about the role of values in evaluation practice and can support evaluators in reflecting on their own values and enacting their professional and ethical responsibilities to identify and articulate their values in the context of evaluation practice.

Evaluation scholars have committed decades of work to the development of evaluator competencies. The 2018 American Evaluation Association (AEA) Evaluator Competencies may be useful for evaluators to identify their strengths and weaknesses to improve their practice; however, a few empirically validated self-assessment tools based on the competencies exist. Two studies were conducted to develop a validated tool. The first study (N = 170) developed the Evaluator Competencies Assessment Tool (ECAT), a self-assessment tool based on the AEA, 2018 Evaluator Competencies. This study provided evidence for structural validity via confirmatory factor analysis. The second study (N = 142) reconfirmed structural validity with a new sample and examined variables that are associated with evaluator competencies through correlation and t-test analyses. Having a mentor, years of evaluation experience, age, evaluation training, and education level were positively related to evaluator competencies. The ECAT can be used to foster self-reflection for practitioners to improve evaluation competence.

This article compares the purposes and content of the four foundational documents of the American Evaluation Association (AEA): the Program Evaluation Standards, the AEA Public Statement on Cultural Competence in Evaluation, the AEA Evaluator Competencies, and the AEA Guiding Principles. This reflection on alignment is an early effort in the third step of professionalization: defining how to use and recognize evaluator competencies. The analysis intentionally focuses on content and reflects on the implications of the differences and similarities across documents. The comparison reveals important questions of interest at both the micro level (individual evaluator) and the macro level (evaluation). The article concludes with challenges, learnings, and proposed next steps of AEA's Professionalization and Competencies Working Group.

As part of its Guiding Principles for Evaluators, the American Evaluation Association (AEA) requires that evaluators develop cultural competencies. Using a successive-independent-samples design, the researchers sought to compare perceptions of cultural competence across a duration of 10 years. Qualitative data were collected via online surveying, which included 168 program evaluators in 2009 and 110 program evaluators in 2019. Content analysis was utilized, and content categories were identified and quantified for both data collections. The data reflect that, from 2009 to 2019, there has been an increased recognition of what cultural competence entails and a closer alignment between what the Guiding Principles for Evaluators promotes and what evaluators demonstrate. However, the data also indicate that perhaps preferences have evolved past the current cultural competence paradigm as well as the term “cultural competence” itself. These findings and implications are discussed in further detail.

Violence, marginalization, oppression, exploitation, erasure, and injustice are cornerstones of the Black experience in the United States. Despite seemingly insurmountable challenges, Black people have worked diligently and competently to earn spaces within the present-day professional arena. While the experiences of Black professionals have been investigated in multiple fields, little is known about Black evaluators. This study aimed to investigate Black evaluators’ experiences in an attempt to understand the intersectionality of their identities, roles, and practice and to ensure voices and contributions of Black evaluators are highlighted. We conducted 26 interviews with Black evaluators across academic, government, nonprofit, philanthropic, and private sectors. We found race and advocate for social justice are central to Black evaluators’ identities, roles, and practice. We developed a thematic framework as a result of our analysis and interpretations of the ways in which Black evaluators’ identities impacted their perceptions of their professional role and practice.

Evaluators often lament that the general public does not understand what we do. Yet, there is limited empirical research on what the general public does know—and think—about program evaluation. This article seeks to expand our understanding in this domain by capturing views about evaluation from a demographically representative sample of the U.S population. This article also explores different strategies for describing program evaluation to the general public. Using an experimental design, it builds on previous research by Mason and Hunt, testing a set of hypotheses about how to enhance communication about evaluation. Findings suggest that public understanding of evaluation is indeed low, although two specific communication strategies—using well-known examples of social programs and including a why statement that describes the purpose of evaluation—can strengthen understanding among members of the public.