Professionalizing Evaluation: A Time-Bound Comparison of the American Evaluation Association's Foundational Documents

Abstract

This article compares the purposes and content of the four foundational documents of the American Evaluation Association (AEA): the Program Evaluation Standards, the AEA Public Statement on Cultural Competence in Evaluation, the AEA Evaluator Competencies, and the AEA Guiding Principles. This reflection on alignment is an early effort in the third step of professionalization: defining how to use and recognize evaluator competencies. The analysis intentionally focuses on content and reflects on the implications of the differences and similarities across documents. The comparison reveals important questions of interest at both the micro level (individual evaluator) and the macro level (evaluation). The article concludes with challenges, learnings, and proposed next steps of AEA's Professionalization and Competencies Working Group.

Keywords

evaluator competencies professionalization evaluation foundations cultural competence ethical practice

Introduction

The purpose of this article is to compare the purposes and content of the four foundational documents of the American Evaluation Association (AEA): the Program Evaluation Standards (Yarbrough et al., 2011),¹ the AEA Public Statement on Cultural Competence in Evaluation (AEA, 2011), the AEA Evaluator Competencies (AEA, 2018a), and the AEA Guiding Principles (AEA, 2018b). The potential value of such a comparison became clear when the newly created AEA Professionalization and Competencies Working Group (P&CWG) began its discussions in 2021. The foundational documents resulted from AEA's commitment over many years to the evolving professionalization of the field, and while the earlier Competencies Task Force (CTF) carefully compared documents that existed at the time, it never completed a systematic, item-by-item comparison of their content, hence this article.

Table 1 details the activities of the CTF created in 2015 by the AEA Board.² This Task Force worked meticulously for three years. Four grounding characteristics guided the competency development process; it was purposefully pragmatic, inclusive, intentional, and dynamic. In addition—and importantly—its activities built on “the strong base of competency work and professionalization already in place and evolving globally” (Tucker et al., 2020, p. 33). The Task Force approached this work within a three-step framework of increasing professionalization (Davies & Brümmer, 2015):

Step 1: Reviewing foundational documents and reaching out globally to learn from others.

Step 2: Using the knowledge learned from reviewing and reaching out to develop and initially validate a set of AEA Evaluator Competencies grounded in member engagement.

Step 3: Defining how to use and recognize evaluator competencies.

Table 1.

Development Activities of the AEA Evaluator Competencies Task Force (2015–2020).

Year	Activities
2015	Studied competencies developed worldwide Conducted comparison of 11 sets of competencies along with alignment to AEA's Program Evaluation Standards, Guiding Principles, and Cultural Competence Statement Drafted initial set of competencies (Version 1) and sought feedback Developed plan for member input and engagement
2016	Analyzed member feedback from listening posts at the 2015 AEA Annual Conference Completed an update of draft competencies (Version 2) using feedback Secured IRB approval to document the work of the Competencies Task Force
2017	Prepared AEA member survey for launch in late Summer 2017 Conducted preliminary analysis of quantitative survey findings (n = 1221) in preparation for 2017 AEA Annual Conference sessions
2018	Continued analysis of survey results, including qualitative analysis Reviewed validation designs, ways to embed competencies in AEA events, and explored other uses Revised draft competencies (Version 3) with Fall 2017 survey data Submitted a proposal to AEA Board for competency approval along with a proposed process for the competencies’ revision/updating Received unanimous Board approval for the revised competencies in May 2018
2019	Based on feedback from reviewers of a New Directions for Evaluation proposal, revised proposal, and began writing Presented two sessions on professionalization and competencies at the 2019 AEA Annual Conference
2020	Completed the New Directions for Evaluation issue Transitioned the work of the Competencies Task Force to the new AEA Professionalization and Competencies Working Group

AEA = American Evaluation Association.

This article is an early effort in the third step of professionalization—reflecting on the alignment of our newly adopted competencies with other foundational AEA documents. Implementing Step 3 involved activating the P&CWG in 2021. It is significant to note the structural shift from a “task force” to a “working group.” The AEA Board manages task forces to develop policy, whereas the AEA Executive Director manages working groups with the intentional purpose of implementing board-adopted policy. In terms of competencies, the time had come to shift from competency development toward testing the professional utility of competencies inside and outside of AEA.

We recognize that the very act of seeking to increase professionalization raises potential concerns that require foregrounding. Five initial comments are in order.

First, the lack of a standardized understanding of the field or transdiscipline of evaluation unavoidably creates a challenge for its professionalization efforts (Clinton & Hattie, 2021; Reynolds & Schwandt, 2017). People engaged in dramatically different practices may all call themselves evaluators, so professionalization, including any competencies, standards, or principles, must create a large umbrella to encompass all.

Second, and a related point, we know that the set of program evaluation-oriented competencies the AEA Board approved in 2018 represents only the initial stage of AEA Evaluator Competencies development. Further refinement will likely result in distinct sets of competencies for evaluators in different jobs (e.g., quality assurance; monitoring; auditing; and monitoring, evaluation, and learning [MEL]), in different roles (e.g., internal vs. external or consultants vs. researchers), within different types of organizations (e.g., small, medium, large), within different subject areas (e.g., public health, international development, early childhood education), and at different stages of their career (e.g., novice, skilled, expert).

Third, any discussion of competencies needs to acknowledge what competencies are not (AEA, 2018c). No set of competencies will ever be perfect. They are not a means of blocking access to the field for qualified professionals, nor are they a credentialing or assessment system. Alone, they are not a way of addressing the systemic challenges of the field's professionalization.

Fourth, the appropriate role and inclusion of social justice content was a topic that the CTF embraced intentionally and one that demands further thoughtful discussion as professionalization continues moving forward. Therefore, we are looking forward to collaborations with AEA's Diversity, Equity and Inclusion Working Group formed in 2021.

Finally, because the Task Force that developed these competencies was part of the largest voluntary organization for professional evaluation (VOPE) in the world, we were and remain extremely aware of the negative potential for powerful organizations (e.g., Global North government and private funders financing large-scale projects in the Global South) to require their use without consideration of their appropriateness in context. As Morkel (2022) notes, “We must acknowledge how the persistent hegemony of powerful nations over less powerful ones influences the evaluation ‘industry’…” (para. 6)

A Proviso About the Different Origins of and Goals Across the Four Documents

Please note that the detailed comparison described in this article asserts no value judgment—either good or bad—in identifying any overlap or its absence. The comparison identifies areas where the four documents completely overlap, areas of some commonality, and areas that are unique without overlap. It intentionally focuses on content (the “what?”) and reflects on the implications of the differences and similarities across documents (the “so what?”). Given their different origins and goals, we are not saying that a lack of full alignment is fundamentally a bad thing. Rather, we believe that any agreement or lack thereof highlights important distinctions across the foundational documents that AEA should address.

Comparing the Four Foundational Documents

Initially, we made no assumptions about potential overlap. We recognized the importance of being systematic and therefore developed a process of comparison using the following steps. Three independent readers (this article's three authors) each brought experience and expertise to the process; two of us have been engaged in developing and studying competencies for over 20 years, and the third was an active member of the CTF and lead author of a chapter documenting the development process of the AEA Evaluator Competencies (Tucker et al., 2020). Our first task was to agree on a system to label the content of the AEA Public Statement on Cultural Competence in Evaluation (AEA, 2011) to make comparison possible, since this document is entirely in narrative.³

Next, we individually coded for alignment across the four documents item-by-item. The AEA Evaluator Competencies anchored our process because they are the focus of the new working group. We began by comparing them to the Program Evaluation Standards, then to the Cultural Competence Statement, and finally to the Guiding Principles.⁴ We then met several times via Zoom to put our analyses together (i.e., to discuss our individual coding and reach consensus on overlapping content across the four documents as well as to “agree to disagree”). Our criterion for consensus was that a minimum of two of us agreed on any claim; in most instances, all three of us agreed. The results of our comparison are presented in the following section.

It is important to understand that this comparison has an extremely short shelf life because evaluation content and contexts continue to evolve. Evaluation professionals must never consider any set of competencies, standards, or principles to be permanent. We find it helpful to envision these as written in pencil, ready for revision as needed. Think, for example, of how the COVID pandemic affected and likely permanently changed evaluation practice. In the two years since March 2020, virtually delivered evaluation has dramatically increased, and that is unlikely to decrease. Going forward, evaluators will probably need competencies related to this newly created context. But while the content of foundational documents may well change, the process of development and application likely does not. Discussion surfaces important concerns, enables formulation of content, and provides clarification of how foundations might be enacted for specific evaluators in the specific contexts in which they work (e.g., university settings, internal to organizations, large-scale projects, etc.). The point we emphasize relates ultimately to how practitioners can and should use competencies along with the other foundational documents in improving their practice.

Results of the Comparison

Purpose and Age of Foundational Documents

Comparing the four AEA foundational documents immediately revealed their different purposes, as well as the years in which they were updated or newly developed and adopted.

The Program Evaluation Standards were developed by the Joint Committee on Standards for Educational Evaluation and first published in 1981. The most current update is in its third edition, published over 10 years ago in 2011. Its stated purpose is to “identify and define evaluation quality and guide evaluators and evaluation users in the pursuit of evaluation quality” (Yarbrough et al., 2011, p. xxii).

The AEA Public Statement on Cultural Competence in Evaluation was developed by the Cultural Competence in Evaluation Task Force of AEA's Diversity Committee, approved through a vote by AEA membership in 2011, and is currently being updated. Its purpose is to articulate the core nature of cultural competence in evaluations and communicate expectations for culturally competent practice. Specifically, the document “affirms the significance of cultural competence in evaluation. It also informs the public of AEA's expectations concerning cultural competence in the conduct of evaluation” (AEA, 2011, p. 1).

The AEA Guiding Principles were originally developed by a Task Force created by the AEA Executive Board and ratified by AEA membership in 1994. Most recently, the Guiding Principles were updated and endorsed by a referendum of AEA membership in 2018. This document's purpose is to articulate principles that “reflect the core values of the American Evaluation Association (AEA) and are intended as a guide to the professional ethical conduct of evaluators” (AEA, 2018b, p. 2).

The AEA Evaluator Competencies were newly developed by the AEA Evaluator CTF during 2015 to 2018 and ultimately adopted by the AEA Board in May 2018. This document's purpose is to articulate “what makes evaluators distinct as practicing professionals” (AEA, 2018a, p. 1), which includes evaluator “knowledge, skills, and abilities deemed necessary for effective practice” (Stevahn et al., 2020, p. 76).

All of these foundational documents were developed over several years as rounds of input, vetting, feedback, and refinement occurred through processes intentionally designed by each group of developers to be expansive, representative, and inclusive. All of the documents also focus on grounding quality evaluation practice, yet each brings nuance to this purpose, differing slightly in orientation. The Program Evaluation Standards (Yarbrough et al., 2011), for example, most prominently address evaluations by articulating standards of quality. In contrast, the Guiding Principles (AEA, 2018b) and Evaluator Competencies (AEA, 2018a) focus on evaluators by articulating ethical values (principles) as well as knowledge, skills, and abilities (competencies) necessary for the conduct of quality professional practice. The Statement on Cultural Competence in Evaluation (AEA, 2011), however, addresses evaluators and evaluations in an integrative way. It recognizes that culture and its social structures, values, assumptions, perspectives, lived experiences, and so on—individual and collective—are inseparable from every aspect of evaluators and the evaluation choices they make, the contexts in which they conduct studies, and the processes and products that result. It is important to understand these distinctions in purpose across the four foundational documents when comparing and contrasting for alignments and gaps. Specifically, as we discovered, one should not assume that the same word or term in all four documents has the same meaning (e.g., context and culture) because the unique nuanced purpose of each document orients the meaning of its substance.

The age of these foundational documents also is noteworthy. The Program Evaluation Standards (Yarbrough et al., 2011) and Statement on Cultural Competence (AEA, 2011) are over a decade old, and time is passing quickly since adoption of the Guiding Principles (AEA, 2018b) and Evaluator Competencies (AEA, 2018a). In conducting the comparison, we noticed the age of the documents because the field of evaluation continuously evolves, and salient issues in any given era focus attention and direct energy in ways that influence and shape understanding. For example, in addition to context and culture, understandings of terms and definitions shift, such as those also pertaining to stakeholders and collaborative and participatory evaluation. New orientations to evaluation emerge, including Blue Marble Evaluation (Patton, 2020), Collaborative Approaches to Evaluation (Cousins, 2020), and Culturally Responsive Approaches to Evaluation (Chouinard & Cram, 2020) to name a few. Research also further refines, validates, or extends theoretical as well as practical evaluation concerns, such as the growing number of studies over the years on various dimensions of evaluator competencies (e.g., Clinton & Hattie, 2021; Davies, 2021; Diaz et al., 2020; Froncek et al., 2018; Garcia & Stevahn, 2020; King, 2020; Stevahn et al., 2005; Wilcox & King, 2014). In addition, national and global events catalyze societal shifts, center public focus, and prioritize pragmatic concerns in ways that affect evaluation. Consider, for example, the recently sharpened foci on climate change and environmental sustainability, Black Lives Matter, the COVID pandemic, and the invasion of Ukraine among others, that direct attention and shape understandings in ways that influence evaluation studies, evaluation research, evaluator education, and evaluation practice.

Societal shifts point to the importance of perceiving AEA's foundational documents as “living” and “organic,” made real by instituting regular cycles of review for reflection, discussion, and updating. Furthermore, coordinating this work to enable crosstalk among those leading and participating in such efforts could strategically, tactically, meaningfully, and broadly advance professionalization by intentionally fostering shared holistic understanding that promotes interconnected applications across diverse contexts, rather than treating each document as its own separate entity, disconnected from the others. We believe that identifying intersections among the four foundational documents in light of their unique purposes and translating this into concrete practice in one's own area of evaluation hold promise for the professionalization of evaluation.

Alignment Across Foundational Documents

Comparing items across each of the four foundational documents resulted in the identification of commonalities as well as gaps, suggesting varying degrees of alignment. The final comparison in Table 2 maps interconnections across all four foundational documents, showing how the AEA Evaluator Competencies (2018a) align with the Program Evaluation Standards (Yarbrough et al., 2011), Statement on Cultural Competence (AEA, 2011), and Guiding Principles (AEA, 2018b). In addition, Table 3 summarizes the number and percentages of items aligned. Specifically, of the 49 (100%) AEA Evaluator Competencies, 14 (28.6%) align with items across all the other three foundational documents, 16 (32.6%) align with items across any two of the other documents, 15 (30.6%) align with items across any one of the other documents, and 4 (8.2%) do not align with any items across the other documents.

Table 2.

Alignment Across AEA Foundational Documents.

	AEA Evaluator Competencies (2018)	Program Evaluation Standards (2011)	Cultural Competence in Evaluation (2011)	Guiding Principles (2018)
1 . 0	Professional Practice Domain focuses on what makes evaluators distinct as practicing professionals. Professional practice is grounded in AEA's foundational documents, including the Program Evaluation Standards, the AEA Guiding Principles, and the AEA Statement on Cultural Competence.
1.1	Acts ethically through evaluation practice that demonstrates integrity and respects people from different cultural backgrounds and indigenous groups.	X	X	X
1.2	Applies the foundational documents adopted by the AEA that ground evaluation practice.	X	X	X
1.3	Selects evaluation approaches and theories appropriately.		X
1.4	Uses systematic evidence to make evaluative judgments.	X		X
1.5	Reflects on evaluation formally or informally to improve practice.	X	X
1.6	Identifies personal areas of professional competence and needs for growth.
1.7	Pursues ongoing professional development to deepen reflective practice, stay current, and build connections.			X
1.8	Identifies how evaluation practice can promote social justice and the public good.		X	X
1.9	Advocates for the field of evaluation and its value.
2.0	Methodology Domain focuses on technical aspects of evidence-based, systematic inquiry for valued purposes. Methodology includes quantitative, qualitative, and mixed designs for learning, understanding, decision making, and judging.
2.1	Identifies evaluation purposes and needs.	X
2.2	Determines evaluation questions.
2.3	Designs credible and feasible evaluations that address identified purposes and questions.	X		X
2.4	Determines and justifies appropriate methods to answer evaluation questions, e.g., quantitative, qualitative, and mixed methods.		X
2.5	Identifies assumptions that underlie methodologies and program logic.	X		X
2.6	Conducts reviews of the literature when appropriate.
2.7	Identifies relevant sources of evidence and sampling procedures.		X
2.8	Involves stakeholders in designing, implementing, interpreting, and reporting evaluations as appropriate.	X	X	X
2.9	Uses program logic and program theory as appropriate.	X	X
2.10	Collects data using credible, feasible, and culturally appropriate procedures.	X	X	X
2.11	Analyzes data using credible, feasible, and culturally appropriate procedures.	X	X
2.12	Identifies strengths and limitations of the evaluation design and methods.	X		X
2.13	Interprets findings/results in context.	X	X	X
2.14	Uses evidence and interpretations to draw conclusions, making judgments and recommendations when appropriate.	X	X
3.0	Context Domain focuses on understanding the unique circumstances, multiple perspectives, and changing settings of evaluations and their users/stakeholders. Context involves site/location/environment, participants/stakeholders, organization/structure, culture/diversity, history/traditions, values/beliefs, politics/economics, power/privilege, and other characteristics.
3.1	Responds respectfully to the uniqueness of the evaluation context.	X	X	X
3.2	Engages a diverse range of users/stakeholders throughout the evaluation process.	X	X	X
3.3	Describes the program, including its basic purpose, components, and its functioning in broader contexts.	X
3.4	Attends to systems issues within the context.		X
3.5	Communicates evaluation processes and results in timely, appropriate, and effective ways.	X	X	X
3.6	Facilitates shared understanding of the program and its evaluation with stakeholders.	X	X	X
3.7	Clarifies diverse perspectives, stakeholder interests, and cultural assumptions.	X	X	X
3.8	Promotes evaluation use and influence in context.	X
4.0	Planning and Management Domain focuses on determining and monitoring work plans, timelines, resources, and other components needed to complete and deliver an evaluation study. Planning and management include networking, developing proposals, contracting, determining work assignments, monitoring progress, and fostering use.
4.1	Negotiates and manages a feasible evaluation plan, budget, resources, and timeline.	X		X
4.2	Addresses aspects of culture in planning and managing evaluations.	X	X	X
4.3	Manages and safeguards evaluation data.	X	X	X
4.4	Plans for evaluation use and influence.	X
4.5	Coordinates and supervises evaluation processes and products.	X
4.6	Documents evaluation processes and products.	X		X
4.7	Teams with others when appropriate.		X	X
4.8	Monitors evaluation progress and quality and makes adjustments when appropriate.	X		X
4.9	Works with stakeholders to build evaluation capacity when appropriate.	X
4.10	Uses technology appropriately to support and manage the evaluation.			X
5.0	Interpersonal Domain focuses on human relations and social interactions that ground evaluator effectiveness for professional practice throughout the evaluation. Interpersonal skills include cultural competence, communication, facilitation, and conflict resolution.
5.1	Fosters positive relationships for professional practice and evaluation use.		X
5.2	Listens to understand and engage different perspectives.		X	X
5.3	Facilitates shared decision making for evaluation.	X
5.4	Builds trust throughout the evaluation.		X
5.5	Attends to the ways power and privilege affect evaluation practice.		X	X
5.6	Communicates in meaningful ways that enhance the effectiveness of the evaluation.	X	X	X
5.7	Facilitates constructive and culturally responsive interaction throughout the evaluation.	X	X	X
5.8	Manages conflicts constructively.	X		X

Note. Alignment indicated by X.

AEA = American Evaluation Association.

Table 3.

Alignment of Evaluator Competencies (AEA, 2018a) and Other AEA Foundational Documents.

Competencies Frequency (percentage)	Aligned With Other Foundational Documents^a	Cumulative Frequency (percentage)
14 (28.6)	All 3	14 (28.6)
16 (32.6)	Any 2	30 (61.2)
15 (30.6)	Any 1	45 (91.8)
4 (8.2)	None	49 (100)

These include The Program Evaluation Standards (Yarbrough et al., 2011), Statement on Cultural Competence in Evaluation (AEA, 2011), and Guiding Principles (AEA, 2018b).

AEA = American Evaluation Association.

The following 14 AEA Evaluator Competencies aligned with similar items across all three other foundational documents:

1.0 Professional Practice Domain

1.1 Acts ethically through evaluation practice that demonstrates integrity and respects people from different cultural backgrounds and indigenous groups.

1.2 Applies the foundational documents adopted by the AEA that ground evaluation practice.

2.0 Methodology Domain

2.8 Involves stakeholders in designing, implementing, interpreting, and reporting evaluations as appropriate.

2.10 Collects data using credible, feasible, and culturally appropriate procedures.

2.13 Interprets findings/results in context.

3.0 Context Domain

3.1 Responds respectfully to the uniqueness of the evaluation context.

3.2 Engages a diverse range of users/stakeholders throughout the evaluation process.

3.5 Communicates evaluation processes and results in timely, appropriate, and effective ways.

3.6 Facilitates shared understanding of the program and its evaluation with stakeholders.

3.7 Clarifies diverse perspectives, stakeholder interests, and cultural assumptions.

4.0 Planning and Management Domain

4.2 Addresses aspects of culture in planning and managing evaluations.

4.3 Manages and safeguards evaluation data.

5.0 Interpersonal Domain

5.6 Communicates in meaningful ways that enhance the effectiveness of the evaluation.

5.7 Facilitates constructive and culturally responsive interaction throughout the evaluation.

Notice that competency alignments with the three other foundational documents occur across all five domains. Furthermore, this collection of competencies highlights a consistent concern for culture across all domains (specifically Competencies 1.1, 2.10, 3.7, 4.2, and 5.7), heavily emphasizes the importance of context (specifically Competencies 2.13 and 3.1, as well as 3.2, 3.5, 3.6, and 3.7 all within the Context Domain), and focuses on respecting, involving, and meaningfully communicating with people (especially Competencies 1.1, 2.8, 3.2, 3.5, 3.6, 3.7, 5.6, and 5.7).

In contrast, the following four AEA Evaluator Competencies did not align with any items across the other three foundational documents:

1.0 Professional Practice Domain

1.6 Identifies personal areas of professional competence and needs for growth.

1.9 Advocates for the field of evaluation and its value.

2.0 Methodology Domain

2.2 Determines evaluation questions.

2.6 Conducts reviews of the literature when appropriate.

This strikes us as a curious collection. Competency 1.6 (identifying personal competence and needs for growth) may be a precondition for thoughtfully making choices about pursuing professional development, yet, it is not mentioned in the other foundational documents, although the Guiding Principles do focus on the value of pursuing professional development. Competency 1.9 (advocating for the field of evaluation) assumes that an evaluation professional would promote or educate others about how evaluation studies can add value to organizations and individuals committed to developing and implementing effective decisions, policies, and practices, yet this is not in any of the other foundational documents. In fact, some argue that advocating for the field of evaluation is not a competency at all because this does not directly affect the ability of evaluators to conduct evaluations, nor does it impact the quality of the studies produced. Competency 2.2 (determining evaluation questions) is imperative to conducting evaluations yet is not addressed specifically in the other documents. Perhaps this is because it makes no sense to plan and carry out evaluations without questions, implicitly assumed to be part of framing studies as purposes, needs, and questions of interest are iteratively developed and ultimately together provide direction for implementing useful evaluation approaches and methodologies. Finally, Competency 2.6 (conducts literature reviews) may be viewed as less necessary for pragmatic reasons, such as working with shoe-string budgets for evaluation, dealing with time constraints, conducting small-scale or highly contextual studies, or facing other circumstances affecting the perceived benefits that a review of the literature might provide. Whatever are the reasons, these four Evaluator Competencies that do not align with any items in the other foundational documents point to gaps and raise questions about importance. In addition, concepts like evaluative thinking have emerged and grown in recognition (Buckley et al., 2015; Carden & Earl, 2007; Vo et al., 2018), yet are not represented in the competencies or the other documents.

Table 3 shows that, overall, a majority of the Evaluator Competencies, specifically 31 (63.2%), align with items in two or one of the other foundational documents, rather than aligning with items in all or none. Furthermore, cumulatively, 30 (61.2%) of the competencies align with items in all three or any two of the other documents, and 45 (91.8%) align with items in three, two, and one of the other documents. Some may view this as providing validity for the competencies and therefore affirming their content. As previously noted, the CTF considered 11 sets of evaluator competencies from around the world and engaged AEA membership in numerous rounds of feedback that eventually resulted in the version adopted.

On the other hand, 19 (38.8%) of the competencies align with items in only one and none of the other foundational documents, which some may view as a gap that needs bridging. Ideally, we believe that AEA should facilitate conversation about alignment—including what frequencies and percentages across the documents mean—in light of the content that overlaps or does not.

Simply put, any comprehensive analysis requires a thorough understanding of the content to interpret meaning. It is reasonable to conclude that, because of each document's different orientation toward enhancing evaluation, there will be some gaps. Yet, despite different orientations, it is also reasonable to believe that greater alignment of items across all four documents may enable evaluators to more holistically and cohesively articulate and map what grounds effective practice, therefore advancing professionalization.

Limitations and Implications of the Comparison

We recognize several limitations in conducting this analysis, and each suggests implications for moving forward. We highlight the following, keeping the goal of professionalization in mind.

It is challenging to compare documents developed for different purposes and adopted in different eras, suggesting the need for regular and coordinated updating. The AEA Board is responsible for updating three of the foundational policy documents and for participating in revisions of the Program Evaluation Standards conducted by the Joint Committee on Standards for Educational Evaluation. As previously noted, evaluation issues, orientations, learnings, understandings, and practices evolve over time. This affects how terms and concepts should be interpreted for comparison and, in turn, the extent to which foundations can usefully be applied in integrated ways. Therefore, coordination in developing standards, principles, and competencies is important because items that constitute these foundations are interdependent, each affecting the others.

Independent readers in content analysis provide some assurance of confidence in results; however, future comparisons need broad inclusive representation, as do document reviews and updates. The time-bound nature of this study necessitated acting quickly; the trade-off was that the three authors served as independent readers. In the future, we envision that representatives from groups that develop or update each of the foundational documents will participate in comparative analysis, sharing insights from each orientation and using replicable consensus strategy. In fact, we strongly recommend broad, diverse, inclusive, representative participation in future comparisons across the four foundational documents, as well as for all reviews and updates, because this further contributes to validity.

Information on the development of foundational documents appears in different places, formats, and degrees of detail, suggesting that clear, accurate, and easily accessible historical records are needed to systematically depict purposes, procedures, issues, assumptions, evidence collected, decisions made, and actions taken. Systematic records of purposes, processes, and rationales underlying the foundational documents would be helpful for review and update efforts, as well as for comparisons that can ground integrated understanding and use, thereby contributing to further professionalizing evaluation.

Challenges and Learnings

As described earlier, AEA's four core documents span multiple levels of interest and have different foci—ranging from individual evaluators to the AEA Board and management, AEA Topical Interest Group (TIG) and affiliate membership, and beyond to partners such as other VOPEs and commissioners. Responding to the challenge of making meaning across these documents created an unexpectedly engaging space for the authors to explore ways to align and strengthen our work with professionalization and competencies. The process of examining AEA foundational documents revealed important questions of interest at both the micro level (individual evaluator) and the macro level (evaluation, including AEA management, institutional members, evaluation development programs, and other evaluation partners).

More than an unanticipated exercise in professional development, this reflection involved evaluative thinking, which we believe is important for all practitioners. It has raised unresolved challenges with which AEA members have long grappled. Four challenges that emerged from our current analysis are relevant for moving professionalization and competency work forward:

Resolving multiple definitions and relationships among core concepts to build common cause, break down silos, and determine pathways to leverage common cause.

Grounding definitions, designs, engagement, decisions, and actions with accessible evidence/research related to current and future competencies and professionalization.

Leveraging and developing short-term and long-term policies about professional evaluation and professional evaluators in pragmatic and intentional ways to meet both tactical and strategic goals.

Engaging an inclusive representation of stakeholders to implement actions across AEA initiatives dealing with professionalization and competencies, an essential part of which will be developing an approach that meaningfully includes AEA members who work locally (e.g., within the United States and North America) and those who work globally (e.g., in international contexts or with transnational organizations).

Consensus Versus Silos

To approach the four challenges noted above, we first suggest aligning a myriad of values and operational definitions with potential deliverables. While a lack of consensus exists in terms of defining “evaluation” and “professionalization” within the AEA membership, operational definitions, and assumptions for “evaluation” are receiving renewed attention. Resolving the lack of alignment across present and future core documents requires transcending traditional hegemonies and silos such as those between practitioners and academics. Another resource to leverage common cause involves crossing boundaries between AEA's historical focus on individual members and the addition of systems-wise structures such as institutional membership. While the answers will shift over time and context, communication is key to reaching consensus within internal AEA audiences as well as responsibly connecting with our external audiences and partners such as commissioners, clients, and members of other VOPEs.

Investing in Evidence and Research

While in recent years the field has been active in generating questions related to professionalization, it has not been as active in generating actionable research to inform important decisions about it. Besides validity studies about the competencies, we need systematic studies and support strategies to expand the rigor, dissemination, and use of professionalization-related research. Collaborations with commissioners could develop additional support mechanisms that are not addressed in current funding schema. In the United States, the Foundations for Evidence-Based Policymaking Act of 2018 (that became Public Law in 2019)⁵ offers much promise to strengthen evaluation professionalization as well as individual evaluator development.

Policies Related to Evaluator and Evaluation Quality

AEA's Evaluation Policy Task Force has been instrumental in supporting the enactment of the Evidence-Based Policymaking Act in the U.S. government as well as breaking down federal agency silos with the help of other AEA-related groups such as the Washington Evaluators, an AEA affiliate. Collaborations by the P&CWG have begun in 2022 with both groups.

Evidence-based determination of evaluator and evaluation quality will need to be part of ongoing evaluation policy efforts. While the CTF positioned itself as not dealing with accreditation and certification, it did purposely frame the competencies in measurable terms, conscious that measurement would be important if such endeavors are ever pursued. In all likelihood AEA may want to consider different ways to structure competencies in addition to the current five domains. Other options could address job function, levels of expertise, and developmental levels (Rassel, 2018). Some options for near-term work involve situating AEA Evaluator Competencies within a larger framework of professionalization, using contexts and cross-cutting themes such as social justice embedded in the connections between and among competencies, uses, and commitments.

A future initiative by the AEA Executive and Board could align AEA task forces and working groups for regular accountability of processes and impacts within a collaborative timeline and cross-cutting charges. An important example would address the evaluator versus evaluation disjunction in future charges to AEA task forces and working groups, in addition to developing an updated competencies document using the Program Evaluation Standards format. Creating and maintaining an online electronic glossary of definitions and contextually nuanced evidence could help clarify how competencies interact with the other three foundational documents.

Engaging Inclusive Representation of Stakeholders

Although answers will shift over time and context, communication will be key to reaching consensus within internal as well as external AEA audiences such as commissioners, clients, and VOPE partners. While AEA member inclusion and engagement are valued across all four documents at a policy level, it is unclear how this aspiration can be systematically implemented. Although all TIGs were invited to participate in providing input and feedback to competencies development, the CTF succeeded in engaging only 50% of TIG leadership. Going forward, this will not be good enough. Mechanisms such as digital town halls have the potential to reach many people, but more diverse and regular, deeper engagement methods are also needed to achieve genuine representation. Next steps could continue to refine dispositional competencies, explicate the relationship of evaluative thinking across domains, and expand the competency structure beyond categorized individual evaluator competencies to address issues of teaming and complexity. In addition, groups of evaluators may want to engage in purposeful discussion of professionalization. The appendix in this article includes a sample activity involving the cross-cutting theme of social justice for such discussion.

We also believe that professionalization and competency work need to be both local and global to respond to present and future concerns involving health, environmental sustainability, human rights, and other major issues affecting our interdependent world. As mentioned earlier about silos and hegemonies, AEA cannot do this alone or generically and likely will need to develop subject-specific and context-specific competencies in collaboration with other professional organizations in the United States and beyond.

Conclusion and Next Steps Toward Professionalization

While this document comparison will become obsolete quickly, the review process has been valuable for helping AEA's P&CWG begin to build upon learnings from the CTF. Established by the AEA Board, the P&CWG has the following charges:

Define professionalization;

Discuss competencies and professionalization with members; and

Incorporate the 2018 AEA Evaluator Competencies into AEA programs/initiatives.

Since Fall 2021, the P&CWG has been working to grow evaluation professionalization with evidence, inclusivity, and a diverse community of practice. Competencies have been situated within the three-step approach to professionalization described earlier in this article. The working group consists of members who represent local, national, and international evaluation contexts from the Global North and South, including a range of practitioners, commissioners, and academics. Our focus will be on defining processes for gathering evidence of the utility of these competencies and identifying what may be missing. The P&CWG's ultimate goal is to optimize how evaluators can use competencies to support their individual professionalization as well as that of the field via the six tasks presented in Table 4. We intend to do this work dynamically through purposeful discussion among evaluators, making meaning in context, constantly raising questions about practice, and identifying needs for change both individually and as a profession. While aspirational, competencies can provide a common language and criteria for approaching operational definitions of evaluation practice and clarifying what makes evaluators unique professionally, as well as building consensus with shared experiences and research.

Table 4.

P&CWG Tasks and Activities Designed to Address Challenges Revealed in the Comparison.

P&CWG Tasks	Examples of Current Activities	Challenges
P&CWG Tasks	Examples of Current Activities	1	2	3	4
1. Institutionalize: Engage AEA Board to address P&C policy and support Evolve: Update competencies as evaluation practice changes over time	Collaborate with AEA Board; review core documents for alignment; P&CWG engage with other AEA structures such as TIGs, affiliates, work groups, and task forces	X		X
2. Use: Integrate competencies within AEA proposals, conferences, other professional development Align: Start with comparison of AEA's foundational documents	Write comparison AJE article about alignment, conduct pilots with TIGs, Evaluation Policy Task Force, and PDWG to use	X	X	X	X
3. Expand Beyond Competency Domains: Address job function, levels of expertise, contextual, and developmental levels	Support emerging evaluators, work with Eval4Action, discuss core competencies	X	X	X	X
4. Engage: Develop voluntary processes evaluators can use in specific contexts to self-reflect and engage with others; design materials that highlight and reinforce interconnections among AEA foundational documents	Facilitate AEA annual conference sessions (2021, 2022 + ), Town Halls; disseminate social justice and other modules in collaboration with PDWG	X			X
5. Research: Organize and support P&C research community	Create community of practice for P&C research, present research at AEA conferences and other organizations	X	X
6. Advocate: Advocate for professionalizing evaluation via global dialog Disseminate: Expand competencies documentation and disseminate	Begin joint work with VOPEs such as SAMEA, CES, UKES, IOCE	X			X

Note. Challenges are numbered as follows and addressed by tasks and activities indicated by X: 1 = Building consensus versus silos, 2 = Grounding work in research, 3 = Developing short- and long-term policies, 4 = Engaging inclusive representation of stakeholders.

AEA = American Evaluation Association; CES = Canadian Evaluation Society; IOCE = International Organization for Cooperation in Evaluation; P&C = professionalization and competencies; P&CWG = Professionalization and Competencies Working Group; PDWG = Professional Development Working Group; SAMEA = South African Monitoring and Evaluation Association; TIG = Topical Interest Group; UKES = United Kingdom Evaluation Society; VOPEs = voluntary organizations for professional evaluation.

Footnotes

APPENDIX: The Evolving Process of AEA Evaluator Competencies Development—Contributing to Social Justice: “Must” or “Maybe”?

A commitment to social justice and contributing to the public good are core values of the American Evaluation Association (AEA). Recognizing this, the Competencies Task Force positioned a competency explicating these values in the Professional Practice domain, even though we knew that some evaluators do not agree that these values are essential to competent evaluation practice. In fact, early into our work, we discovered that many members were not aware of AEA's core documents that explicate these values. Some respondents who wrote comments as part of the member survey in September 2017 took that position, either not understanding or disagreeing.

Consider the following sampling of 10 comments by respondents to the 2017 Draft Survey.

How do you think about social justice and the public good in your evaluation practice?

Does every evaluator need to “consider” issues of social justice? Why or why not? If so, under what conditions?

Does attention to social justice suggest an underlying “liberal agenda” that will drive conservative evaluators away?

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Laurie Stevahn

Jean A. King

Notes

References

American Evaluation Association (AEA) (2018a). AEA evaluator competencies. https://www.eval.org/About/Competencies-Standards

American Evaluation Association (AEA) (2018b). AEA guiding principles for evaluators. https://www.eval.org/About/Guiding-Principles

American Evaluation Association (AEA) (2018c). The 2018 evaluator competencies. Presented at the annual meeting of the American Evaluation Association, Cleveland, OH. https://www.eval.org/Portals/0/Docs/AEA%20Evaluator%20Competencies.pdf

American Evaluation Association (AEA) (2011). Public statement on cultural competence in evaluation. https://www.eval.org/About/Competencies-Standards/Cutural-Competence-Statement

Buckley

Archibald

Hargraves

Trochim

W. M.

(2015). Defining and teaching evaluative thinking: Insights from research on critical thinking. American Journal of Evaluation, 36(3), 375–388. https://doi.org/10.1177/1098214015581706

Carden

Earl

(2007). Infusing evaluative thinking as process use: The case of the International Development Research Centre (IDRC). In J. B. Cousins (Ed.), Process use in theory, research, and practice. New Directions for Evaluation, 116, 61–73. https://doi.org/10.1002/ev.243

Chouinard

J. A.

Cram

(2020). Culturally responsive approaches to evaluation. Sage.

Clinton

J. M.

Hattie

(2021). Cognitive complexity of evaluator competencies. Evaluation and Program Planning, 89, 1–8. https://doi.org/10.1016/j.evalprogplan.2021.102006

Cousins

J. B.

(Ed.). (2020). Collaborative approaches to evaluation: Principles in use. Sage.

10.

Davies

Brümmer

(2015). Professionalisation of evaluation. United Nations Evaluation Group.

11.

Davies

(2021). Establishing and developing professional evaluator dispositions. The Canadian Journal of Program Evaluation, 35(3), 330–349. https://journalhosting.ucalgary.ca/index.php/cjpe/article/view/71156 . https://doi.org/10.3138/cjpe.71156

12.

Diaz

Chaudhary

A. K.

Jayaratne

K. S. U.

Assan

(2020). Expanding evaluator competency research: Exploring competencies for program evaluation using the context of non-formal education. Evaluation and Program Planning, 79, 1–10. https://doi.org/10.1016/j.evalprogplan.2020.101790

13.

Foundations for Evidence-Based Policymaking Act of 2018, Pub. L. No. 115-435 (2019). https://www.congress.gov/bill/115th-congress/house-bill/4174/text

14.

Froncek

Mazziotta

Piper

Rohmann

(2018). Evaluator competencies in the context of diversity training: The practitioners’ point of view. Evaluation and Program Planning, 67, 122–128. https://doi.org/10.1016/j.evalprogplan.2018.01.004

15.

Garcia

G. L.

Stevahn

(2020). Situational awareness and interpersonal competence as evaluator competencies. American Journal of Evaluation, 41(1), 107–124. https://doi.org/10.1177/1098214018814941

16.

King

J. A.

(Ed.). (2020). The American Evaluation Association’s Program Evaluator Competencies . New Directions for Evaluation, 168, 1–172.

17.

Morkel

(2022). “Salmon Fishing in the Yemen”: Swimming upstream to transform evaluation systems. aea365. https://aea365.org/blog/international-evaluation-academy-iea-week-salmon-fishing-in-the-yemen-swimming-upstream-to-transform-evaluation-systems-by-candice-morkel/

18.

Patton

M. Q.

(2020). Blue marble evaluation: Premises and principles. The Guilford Press.

19.

Rassel

S. M.

(2018). The development and use of interpersonal competencies by evaluators (Unpublished doctoral dissertation). University of Minnesota, Minneapolis, MN.

20.

Reynolds

Schwandt

(2017). Evaluation as public work: An ethos for professional evaluation praxis. UK Evaluation Society Annual Conference: The Use and Usability of Evaluation: Demonstrating and Improving the Usefulness of Evaluation, London, UK.

21.

Stevahn

Berger

D. E.

Tucker

S. A.

Rodell

(2020). Using the 2018 AEA Evaluator Competencies for effective program evaluation practice. In J. A. King (Ed.), The American Evaluation Association’s Program Evaluator Competencies. New Directions for Evaluation, 168, 75–97. https://doi.org/10.1002/ev.20434

22.

Stevahn

King

J. A.

Ghere

Minnema

(2005). Establishing essential competencies for program evaluators. American Journal of Evaluation, 26(1), 43–59. https://doi.org/10.1177/1098214004273180

23.

Tucker

S. A.

Barela

Miller

R. L.

Podems

(2020). The story of the AEA Competencies Task Force (2015–2018). In J. A. King (Ed.), The American Evaluation Association’s Program Evaluator Competencies. New Directions for Evaluation, 168, 31–48. https://doi.org/10.1002/ev.20439

24.

A. T.

Schreiber

J. S.

Martin

(2018). Toward a conceptual understanding of evaluative thinking. In A. T. Vo & T. Archibald (Eds.), Evaluative Thinking. New Directions for Evaluation, 158, 29–47. https://doi.org/10.1002/ev.20324

25.

Wilcox

King

J. A.

(2014). A professional grounding and history of the development and formal use of evaluator competencies. The Canadian Journal of Program Evaluation, 28(3), 1–28. https://journalhosting.ucalgary.ca/index.php/cjpe/article/view/30949 . https://doi.org/10.3138/cjpe.0028.005

26.

Yarbrough

D. B.

Shulha

L. M.

Hopson

R. K.

Caruthers

F. A.

(2011). The program evaluation standards: A guide for evaluators and evaluation users (3rd ed.). Sage.