Abstract
Objective. This study aimed to understand the extent to which master of public health (MPH) graduates engage in evaluation on the job, to learn how MPH graduates implement evaluation, and to hear from MPH graduates about how their academic training prepared them for the evaluation work they perform. Methods. Using the Centers for Disease Control and Prevention’s Evaluation Framework, this convergent mixed-methods study included an online survey with 89 public health practitioners and follow-up interviews with 17 survey respondents. The study was performed in the United States during summer 2020. Results. In addition to participating in evaluation activities related to all six Centers for Disease Control and Prevention framework steps, MPH graduates engage in evaluation capacity building, evaluating for health equity and social justice, and funding activities. Participants noted a disconnect between academic preparation and community practice, were least confident in focusing the evaluation design, and most often used surveys to collect data. Conclusions. Public health practitioners commonly engage in evaluation activities but do not feel fully prepared to do so given their MPH training. Many opportunities exist to enhance graduate/postgraduate training through connecting public health with the broader professional practice of evaluation.
Keywords
Evaluation is an important component of public health practice. In 1994 and again in 2020, the Centers for Disease Control and Prevention (CDC) recognized evaluation as one of the 10 essential services of public health (CDC, 2017, 2020). In addition, schools and programs of public health integrate evaluation competencies into their curricula to meet the Council on Education for Public Health (CEPH) accreditation criteria, and public health practitioners are expected to possess evaluation-specific knowledge and skills including an understanding of evaluation methods and procedures (CEPH, 2016).
Despite the importance of evaluation within public health, we have limited insights into how public health practitioners engage in evaluation as part of their work and how well, or not, their academic training has prepared them for these tasks. Studies have documented the use of program evaluation within public health (Carman, 2007; Carman & Fredericks, 2010; Eriksson, 2000; Jordan et al., 2020; Lindholm & Rosén, 2000; Murphy & Mitchell, 2007; Northcross et al., 2015; Perkins, 2018) and demonstrate that organizations at all levels are motivated to conduct evaluation (Brown, 2018; Brownson et al., 2010; Hemingway, 2020). Additionally, a few studies examined how evaluation is reflected within master of public health (MPH) curricula at accredited schools and programs of public health (Fierro & Christie, 2011; Hobson et al., 2019). None of these empirical examinations take the next step by connecting evaluation training to evaluation practice.
In the current study, we begin filling this gap by gathering data about the extent to which practitioners feel prepared to engage in evaluation efforts post graduation. Specifically, the study objectives are
To understand the extent to which MPH graduates engage in evaluation on the job
To learn how MPH graduates are implementing evaluation in their jobs
To hear from MPH graduates about how well they feel their academic training prepared them for the evaluation work they perform
Methods
This study integrated qualitative and quantitative methods using a convergent mixed-methods approach (Creswell & Poth, 2018). Survey and interview questions were organized using the CDC’s Framework for Program Evaluation in Public Health (CDC Framework). The CDC Framework comprises six cyclical steps (Figure 1 and Table 1) for performing program evaluation and is intended to be used as a guide by public health professionals (CDC, 1999). We adopted the CDC Framework to organize our data collection efforts given that it is commonly used in public health and is familiar to MPH students (Freeman et al., 2020; Leeman et al., 2012).
Centers for Disease Control and Prevention Framework for program evaluation in public health with example activities.
Respondent Characteristics.
Note. MPH = master of public health; CHES = Certified Health Education Specialist.
Participants were able to select all that apply. bParticipants can be included in more than one category.
Quantitative Survey
Participants
Participants were recruited through public health organizations including the Society of Public Health Educators, state and regional public health associations, public health schools, and social media networks. A total of 41 participants (34% of all survey participants) were recruited via social media. These participants are excluded from response rate calculations as the denominator (population reached) is unknown. In addition to social media posts, email requests were sent to 105 public health organizations across the United States, and 17 interested groups confirmed they forwarded the study request directly to their members. Each email included a link specific to the organization to allow for calculation of response rates, which varied by organization. Response rates were calculated conservatively using the total membership of the organization, when known, even though it was difficult to determine the number of members that opened or viewed the message. The average response rate across all nonsocial media recruitment sources was less than 1% (range 0% to 8.80%). A total of 122 individuals started the survey, and 33 did not meet eligibility requirements (e.g., graduation from an accredited MPH program, minimum 1 year of work experience, and evaluation experience within a public health role), resulting in 89 responses.
Survey Instrument
A 23-item survey, distributed electronically via Qualtrics, provided participants a brief explanation of each step of the CDC Framework and asked them to approximate time spent on evaluation activities in their current position, their role in the activity, how well their MPH program prepared them for evaluation activities within each step, and their confidence with each step. We also asked their familiarity with the professional association for evaluators in the United States, exposure to evaluation in their coursework, graduation year, concentration, current employer type, and the total amount of time spent conducting evaluation.
Quantitative Data Analysis
R was used to calculate descriptive statistics for demographic variables and questions aligned with the CDC Framework steps.
Qualitative Interviews
Participants
Of the 89 participants who completed the survey, 47 (53%) were willing to be interviewed. Purposive sampling (Creswell & Poth, 2018) was used to select interview participants to obtain a heterogeneous sample of individuals representing a broad range of evaluation training, experience with evaluation, and employment types. Of the 22 participants invited, 17 (77%) participated in an interview.
Interview Instrument
Semistructured interviews were designed to learn about the participants’ experience and current work in evaluation. Using the CDC Framework, participants were asked to discuss specific examples of their evaluation experience and reflect on their academic preparation. Participants identified which evaluation skills they felt they did or did not have when they entered the workforce and shared what resources they used to strengthen their capacity.
Interviews were 1 hour in length and conducted via phone or video. All participants agreed to being recorded and answered all questions. Interview recordings were transcribed by a professional service, reviewed for accuracy, and stripped of identifying information prior to coding.
Qualitative Data Analysis
Two authors conducted thematic analysis of the interview transcripts with NVivo software using a conventional content analysis approach (Hsieh & Shannon, 2005). The initial codebook was generated from the interview protocol. Each transcript was reviewed and coded separately by both coders who met regularly to discuss findings and update the codebook with emerging codes. Comparison queries revealed consistency between coders with no noteworthy discrepancies.
Results
Respondent Characteristics and Engagement in Evaluation on the Job
Survey respondents and interviewees had all earned an MPH degree with the most common concentration being in social and behavioral sciences (46% of survey respondents; 41% of interviewees). As part of their MPH studies, most survey respondents (92%) completed at least one course with a partial focus on evaluation, and many (69%) completed a course focused specifically on evaluation. The majority (85%) of survey respondents pursued additional training post MPH—survey respondents and interviewees most frequently reported attending professional development workshops or conferences to enhance their evaluation capacity. Most interviewees worked in the nonprofit sector (53%), followed by government (24%), academic institution (12%), and health care (6%).
Survey respondents commonly contributed to evaluation tasks with only 7% indicating that they spend none of their time on evaluation and 33% reporting that they have contributed to more than twenty evaluation projects since completing their MPH degree. However, engaging in evaluation activities does not comprise a majority of survey respondents’ time—only 18% noted spending 50% or more of their time on evaluation. All interviewees indicated that evaluation is a critical function of public health practice and reported that evaluation was a common role for them, regardless of workplace setting, with 53% noting their role had a primary focus on evaluation.
Despite the prominence of evaluation in their work, half of the survey respondents (51%) had never heard of the primary professional association for evaluators in the United States—the American Evaluation Association (AEA). Of the 38 survey respondents (49%) who were familiar with AEA, the majority (66%) had never been a member, and roughly equivalent groups reported using (44%) or not using (40%) AEA resources. Of the 8 respondents who were current AEA members, all had pursued additional evaluation training beyond their MPH, two held an academic certificate in evaluation, and four had earned additional degrees (e.g., PhD, DrPH, other master’s degree).
How MPH Graduates Implement Evaluation in Their Jobs
CDC Evaluation Framework Steps
Survey data indicate that most participants spend time on evaluation activities related to all six CDC Framework steps, usually as a contributor or a leader of a team (see Table 2). Survey respondents spend more time on “Ensuring Use and Sharing Lessons Learned” (Step 6) than on any other step; 40% spend more than half of their time in a typical month on this step. Survey respondents spend the least amount of time on “Focusing the Evaluation Design” (Step 3), with 55% spending less than a quarter of their work time related to this step.
CDC Evaluation Framework Steps (N = 89).
Note. CDC = Centers for Disease Control and Prevention; MPH = master of public health.
Consistent with survey results, interviewees consistently placed the greatest emphasis on engaging stakeholders (Step 1) with statements such as “What you choose to evaluate should be relevant to the stakeholders and the engaged stakeholder is really important” [AS] and ensuring use and sharing lessons learned (Step 6) with statements such as “A big part of what we do is making sure those lessons don’t get lost” [MC].
Other Common Evaluation Tasks
Interviews suggest that evaluation activities extend beyond those captured in the CDC Framework steps, including evaluation capacity building, evaluating for health equity and social justice, and accounting for and acquiring external funding.
Evaluation capacity building
Interview participants commonly engage in activities to build evaluation capacity in their organization. In some cases, this is formally required by their role at the organization; they serve as an internal consultant who advises others on evaluation projects. One interviewee shared, “I’m often the only evaluator in the room or the only person who really understands evaluation. I have to . . . explain what I’m trying to do and make the case that this is important” [MC]. In other situations, the evaluation capacity-building role is less formal; the interviewee educates others on evaluation to be effective at their job. For instance, one participant explained their role as “helping other people gather and digest information about the work that they are doing” [KK].
Evaluation for health equity and social justice
Interviewees shared that health equity and social justice are important outcomes of program evaluation with sentiments such as “Evaluation can be a vehicle for change and social justice” [AS]. Health equity, cultural competence, and the importance of including the community voice emerged as important aspects of evaluation practice for many interviewees. They discussed the importance of tailoring evaluation methods for the populations they work with: “I work in a minority community, you can’t ask the same questions when it comes to certain drugs and alcohol. It didn’t translate in the same way, and they didn’t even have a Spanish version” [JH].
Accounting for and acquiring external funding
Nearly every interviewee discussed grant seeking, grant writing, and grant reporting activities and identified grant funding as a specific motivating factor for program evaluation. One participant explained the importance of evaluation as “the practice of measuring and documenting what you do . . . to make sure you’re doing the most good possible and to get more money” [CD]. Interviewees explained that many funders require program evaluation, and often evaluation is used as supportive evidence for acquiring new funding. They noted that funders set expectations for program evaluation and often provide evaluation resources to organizations that receive funding.
MPH Graduates’ Feelings Regarding Their Academic Preparation for Evaluation in Practice
CDC Evaluation Framework Steps
The greatest perceived weaknesses in MPH program preparation were related to engaging stakeholders (Step 1) and focusing the evaluation design (Step 3). Although 83% of respondents strongly agreed or agreed that they felt confident in their ability to conduct evaluation tasks related to engaging stakeholders (Step 1), 17% of them disagreed or strongly disagreed that their MPH program “prepared them well” to participate in this step. And while the majority (63%) of respondents felt confident in their ability to focus the evaluation design (Step 3), 26% of them disagreed or strongly disagreed that their MPH program “prepared them well” to do so (Table 2).
Step 1: Engaging stakeholders
Engaging stakeholders was often presented as one of the most important steps of evaluation. Although considered critical, consistent with survey findings, interview participants had mixed feelings about their confidence and preparedness to engage stakeholders. Most commonly, participants shared that their MPH training helped them understand why this step was important but did not prepare them with the specific skills essential to do so effectively.
Step 2: Describing the program
Participants were most confident in their skills regarding describing the program and generally agreed that they had been prepared to do so in their MPH program. One participant explained that it was their responsibility to “put the program in pictures, draw it, or . . . simplify it enough that different audiences can understand it” [AZ]. Logic models, flow charts, and narratives were used to gain clarity among stakeholders and to describe programs to funders.
Step 3: Focusing the evaluation design
Participants were least confident in their ability to focus on the evaluation design and felt least prepared by their MPH program to participate in these tasks. One participant expressed relief at working in a “larger organization” with established evaluation practices, because “you don’t have to create that from scratch. In my case, I didn’t get practical experience or even experience in the classroom, actually developing [an] evaluation” [AU].
A number of interviewees felt intimidated or unprepared for the technical components of evaluation. Other participants expressed concern that they were not prepared academically to conduct what they perceived as more complex or challenging evaluations such as “economic evaluations, policy evaluations, evaluations where you’re . . . supposedly trying to compare large geographic areas or states, or something like this” [AS].
Step 4: Gathering credible evidence
Participants described a variety of activities related to gathering data and discussed using both qualitative and quantitative methods. However, surveys were—by far—the most common method discussed. Reasons for heavy reliance on surveys included comfort and familiarity with the method and relative ease of implementation. Interviewees frequently mentioned difficulty obtaining data related to limited resources: “Everyone is stretched for time and each institution’s systems are so different . . . it’s very difficult to get the data we needed . . . in a comparable enough format” [CD]. Some participants felt that the MPH program did not prepare them with the skills needed such as “the realities of collecting data. Getting responses or even getting participants for these activities” [PX]. A number of participants reported exposure to new methodological approaches (windshield surveys, dot surveys, mixed methods) after graduating from the MPH program, through work experience or advanced education.
Step 5: Justifying conclusions
Survey results indicate that study participants are involved in justifying conclusions, but no clear patterns of behavior were readily discernable in the data. Nor were clear approaches noted in the interviews. Some participants discussed the importance of strong methods to justify findings; some described conducting this step on their own, while others sought to “facilitate more collaborative analysis and interpretation” [KK] with stakeholders to substantiate conclusions; some worked with program staff, while others thought it was more important to engage a wide range of stakeholders.
Step 6: Ensuring use and sharing lessons learned
While interviewees considered this step an important part of the evaluation process, most expressed frustration regarding this step with comments such as “I think as evaluators, it is easy to realize the value or appreciate the value of the work that we’re doing. It’s another thing to have a sense for what other folks see that value as being” [KK]. This sense of disconnect may be a result of the limited training that they received in this area: “I did not receive training [on ensuring use]. And I think that has been really challenging” [LB]. Many shared similar feelings and indicated that their MPH program did not provide enough training on practical tools for ensuring use and sharing lessons learned.
Disconnect Between Academic and Community Evaluation
Throughout the interviews, participants differentiated between the application of evaluation in academic settings and in the community. Many participants described the difference between their academic training and practical experience: “It felt more academic based—my training—and everything that I’m doing is applied research and evaluation” [JS]. “I mean I took those courses on . . . how to do randomized control trials, qualitative evaluations, and design surveys. . . . I don’t have enough time to do it as well as I should” [LR].
While many interviewees acknowledged the disconnect, most indicated that they were learning to adjust with statements such as “No community-based organization could do what [academic research expects] . . . so after you work in communities, you just learn how to adjust it” [AZ] or “I’m going to get out there and make it work. . . . From my education background, I understood the importance of it . . . [but] nobody ever talks about how this was going to look” [JH]. At times, this disconnect between academic training and practical experience emerged as frustration: “I came to evaluation . . . as an accidental type evaluator who stumbled into it. . . . I was more treading water than really swimming anywhere” [KK].
Discussion
Evaluation is a central component of public health practice. This study, considered with other empirical examinations on the evaluation-specific content of MPH curricula (Fierro & Christie, 2011; Hobson et al., 2019), provide insights about how prepared MPH graduates are to engage in evaluation practice on the job. Specifically, we add to the literature base by exploring the workplace experience of graduates—increasing our collective understanding of the prevalence of evaluation in graduates’ daily workplace activities, the types of evaluation activities they engage in, their role in those activities, and their perspectives about how well their MPH training prepared them for the realities of evaluation practice.
It is important to recognize the study sample composition and size as it may affect interpretation. The large majority of survey respondents reported completing one or more courses with a primary or partial focus on evaluation as part of their MPH degree. This is likely due to the high proportion of respondents who concentrated on social and behavioral sciences for their MPH. Other empirical studies have found roughly equivalent exposure to evaluation in MPH coursework for this concentration area while also recognizing significantly less exposure to evaluation coursework during the MPH for other concentration areas (Fierro & Christie, 2011). Thus, the findings from the current research study may indicate a “best case scenario” with respect to academic preparation in evaluation.
Study participants commonly engage in evaluation, though evaluation activities typically comprise less than 50% of their work time. When they are engaged in evaluation activities, it is most frequently as a contributor, though almost a third of respondents indicated leading evaluation activities. Participants most commonly indicated spending time on tasks related to Step 1 and Step 6, but they also reported that they did not feel well prepared to carry out these steps. Focusing on Step 3 presented a specific challenge for MPH graduates; participants indicated not feeling as confident in this step as they did with others. Interviewees reported using a narrow set of study designs and data collection methods, namely, pre-post designs and surveys. Some expressed concerns and frustration that they did not have a broader repertoire of designs and methods to draw on from their MPH training. Respondents also indicated engaging in a variety of evaluation tasks that go beyond the CDC Framework—they build evaluation capacity within their organizations, use evaluation to advance social justice and health equity, and account for and acquire external funding through evaluation.
The insights from this small sample offer a glimpse into the practice of evaluation among MPH graduates in light of their academic preparation, and our study confirms some previous findings. Fierro and Christie (2011) and Hobson et al. (2019) both draw conclusions about an apparent disconnect between evaluation within the discipline of public health and the broader field of professional evaluation as represented by the AEA. Slightly more than half of survey respondents had never heard of AEA, and among those who were familiar with it, almost two thirds have never been a member. Furthermore, interviewees rarely to never discussed using approaches common within the evaluation field such as empowerment evaluation, theory-driven evaluation, or transformative evaluation. Though respondents frequently mentioned the use of evaluation in promoting social justice and health equity, they never reported drawing on the well-published literature about culturally responsive evaluation nor AEA’s (2011) Public Statement on Cultural Competence in Evaluation.
Public Health Implications
The demand for evaluation is strong, particularly within public health (Kinarsky, 2018; Lemire et al., 2018). As a result, many public health professionals are conducting program evaluation to meet funding requirements (Perkins, 2018). Additionally, program evaluation is an essential component of building an evidence base for effective public health interventions. Therefore, it is critical that public health professionals commissioning, participating in, or leading evaluations are adequately equipped to perform well in these roles. Our study suggests that there is room for growth.
Although the CEPH accreditation criteria does include competencies regarding conducting program evaluation, connecting the public health community with the broader professional practice of evaluation is important to advance public health evaluation practice. One connection includes consulting the AEA Evaluator Competencies (King & Stevahn, 2020) to identify competencies that could be better integrated into MPH curricula—the Council of Education for Public Health, schools and programs of public health, and faculty members can all help bolster curriculum through this action. Another option includes familiarizing MPH faculty, students, and alumni with AEA and the many resources it offers such as professional practice guidelines (AEA guiding principles, AEA statement on cultural competence in evaluation), trainings (professional development workshops, e-trainings), evaluation-specific journals (American Journal of Evaluation, New Directions for Evaluation), and the annual conference. Leaders of public health professional organizations representing public health practitioners broadly (e.g., the American Public Health Association), local and state health officials (e.g., the Association of State and Territorial Health Officials, the National Association of County and City Health Officials), and specific public health disciplines (e.g., the Council of State and Territorial Epidemiologists, the Society for Public Health Education) all have a role to play in this alongside leaders within the AEA.
Footnotes
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors received no financial support for the research, authorship, and/or publication of this article.
