Evaluation Capacity Building

Introduction

Evaluation is “increasingly a core component of social work education/curricula at all levels” (Wretman, 2014, p. 501). This is reflected in the length of the section devoted to “evaluation” in the accreditation standards of the Council on Social Work Education as they are now (CSWE, 2008) compared to a first draft of the new standards, expected to become effective in 2015 (CSWE, 2013). Whereas the former devotes to evaluation one short sentence stating, “Social workers critically analyze, monitor and evaluate interventions” (CSWE, 2008, p. 7) the draft for the new document states,

Social workers understand that evaluation is an ongoing component of the dynamic and interactive process of social work practice with, and on behalf of, individuals, families, groups, organizations and communities. Social workers recognize the importance of evaluating outcomes to advance practice, policy and service delivery effectiveness. Social workers understand methods for evaluating outcomes and practice effectiveness. Social workers select and use appropriate methods for evaluation of outcomes; critically analyze, monitor and evaluate intervention processes and outcomes; and apply evaluation findings to improve practice effectiveness. (CSWE, 2013, p. 7)

The growing emphasis on evaluation in social work education coalesces with recent developments in the interdisciplinary field of program evaluation, namely, a growing interest in evaluation capacity building (ECB). It is increasingly being recognized that program evaluation should not be the exclusive domain of evaluation specialists but should ideally involve the management and staff of organizations that are being evaluated (Wharton & Kazi, 2012). ECB can be seen as a component of “empowerment evaluation” (Fetterman, 2001) although, as Patton (1997) claims, “Capacity-Building might be thought of as a necessary but not sufficient condition for an evaluation to be considered empowering” (p. 155).

Over the last decade, there has been a plethora of publications defining ECB, critically examining the idea, and reporting on ECB projects (Labin, Duffy, Meyers, Wandersman, & Lesesne, 2012). This literature has seen a shift from an emphasis on change at the organizational level (Stockdill, Baizerman, & Compton, 2002), to a recognition that the enhancement of evaluation capacity in organizations requires change in the individuals that make up the organization. Recent definitions stress the human capital aspect of ECB (Labin, 2014; Nielsen, Lemire, & Skov, 2011; Preskill, 2014; Preskill & Boyle, 2008; Wandersman, 2014). Labin et al. (2012, p. 308) define ECB as “an intentional process to increase individual motivation, knowledge and skills and to enhance a group or organization’s ability to conduct or use evaluation.” In their systematic review of empirical studies of ECB, Labin and her associates (2012) found that 77% of the reviewed programs had a training component, and 92% reported outcomes at the individual level.

ECB efforts directed at individuals usually take place in the context of in-service training. This may take the form of short intensive workshops (Adams & Dickinson, 2010; Karisson, Beijer, Eriksson, & Leissner, 2008) or of long-term efforts to educate the field personnel of agencies (Arnold, 2006; Garcia-Iriarte, Suarez-Balkazar, Taylor-Ritzier, & Luna, 2011). The possibility that evaluation courses that are conducted in academia as part of degree granting programs can enhance the evaluation capacity of organizations is rarely discussed. Labin et al. (2012, pp. 313–314), excluded publications that they described as “classroom-based cases” or “articles describing evaluation courses” from their review of the literature.

Conversely, Morris (1994, p. 54) argued for “single evaluation courses” as a means of preparing “educated consumers” of evaluation as well as “household handymen” who do not practice evaluation as a profession but contribute to the evaluation capacity of their agencies. Such single courses would be most effective if they involve “hands-on” experience of the students, and if embedded in a curriculum containing complementary offerings of courses in research methods, statistics, and other relevant subject matters (see also Campbell & Tatro, 1998).

In this article, we ask whether, and to what extent, an evaluation course conducted within an academic setting, attended by practicing social workers, can contribute to the evaluation capacity of both the students and their agencies. We are also interested in the participants’ perspective regarding possible variables that might affect such a contribution, based on their personal experience and in-depth knowledge of the social work arena. The case in point is a seminar on program evaluation conducted within a postgraduate social work course, leading to a master of social work (MSW) degree.

The Present Study

Tel-Hai College is located in northern Israel, close to the Lebanese border and relatively far from any major urban center. It serves an area that is mostly rural, with several small and midsized towns. The college has been offering an undergraduate social work degree since 1996; and in 2008, it introduced a postgraduate program, leading to the MSW degree. An undergraduate degree in social work and at least 2 years of professional experience ¹ are required for admission to this program. About 30–40 students are admitted each year; all of them currently employed in a variety of social work agencies. Some are engaged in direct service to clients, others in supervision or management.

A central component of the MSW course is a yearlong seminar on social program evaluation. Students are required to evaluate, or at least develop an evaluation plan, for a program or project conducted within their agency—not necessarily one which they themselves are involved in. Students can do the work either alone or in pairs. A few students were permitted to join a classmate working in an agency other than their own.

Many of the programs selected by the students as objects of evaluation were therapeutic or educational groups—for example, a parenting group for prisoners with a history of family violence, an empowerment group for wives of alcoholics, an outdoor therapy program for delinquent youths, or a course on disaster preparedness for the staff of a social agency. Other programs were aimed at changing the modus operandi of an agency—such as the introduction of a periodic assessment tool to a therapeutic community for drug addicts or a program to enhance awareness of elder abuse among the staff of a hospital. Some students decided to evaluate a specific outcome of a program with multiple goals—for example, the effect of an after-school program on school attendance.

In the first semester, the students were required to produce seven working papers on various aspects of the evaluation—such as the logic model of the program, stakeholders and their involvement in the evaluation, a review of previous evaluations of similar programs (methods and findings), implementation monitoring, measurement or assessment of outcomes, and measures of efficiency.

In the second semester, the students presented their evaluation plans to the class. The other students and the instructor acted as consultants, offering constructive criticism and suggesting alternative ways of evaluating the programs. At the end of the academic year, the students handed in their seminar papers, either reporting on an evaluation that they had conducted or proposing a detailed evaluation plan.

Informal feedback from the students to this seminar was very positive. This was reflected in verbal comments during and after the seminar and in anonymous feedback questionnaires routinely administered by the college at the end of each semester. Students reported that the seven working papers provided them with an opportunity to gain hands-on experience in a real situation. They appreciated the extensive discussion of logic models, which they considered to be of great value to their practice. Some students told us that the work they had done had helped to improve the program and to enhance their interest, and the interest of their agency, in program evaluation. This anecdotal evidence called for an exploratory evaluation of the seminar and its impact on the students and their agencies.

Method

Results

As mentioned earlier, the aim of this exploratory study was to ask whether, and to what extent, an evaluation course in an academic setting, attended by practicing social workers, could contribute to the evaluation capacity of their agencies and to their own interest and knowledge in program evaluation. We also set out to explore possible factors that may affect this possible contribution.

Most of the students (76%) had had no previous acquaintance with evaluation and 16% had had only some passive exposure. Only four respondents had previously participated in an evaluation. Similarly, most of the organizations (61%) had no previous experience in evaluation. Only nine of the programs had previously been evaluated. In most cases (57%), the students only developed an evaluation plan; in some instances (13%) they conducted a small pilot or partial evaluation; and 17 respondents (30%) actually performed an evaluation.

Despite the seminar’s emphasis on stakeholder involvement, not all of the students had involved others in the evaluation. Forty-two percent had sought and received approval from their superiors, 33% had consulted superiors, or colleagues at the start, 52% consulted as they went along, and 44% presented the results of their work to their superiors or colleagues. Fifty-six percent felt supported by their organization. A majority of the respondents (61%) felt that the work they had done at the seminar had an impact on the program they evaluated. As one participant wrote: “The fact that the program was being evaluated made the program clearer and better constructed, with accurately defined goals.”

Almost all the respondents indicated that the seminar enhanced their interest in evaluation considerably (59%) or to a certain extent (36%). As one participant shared: “I had no prior knowledge of evaluation. Today, it seems unprofessional to me to run programs without evaluation.” Another respondent wrote,

The seminar was an important turning point for me, in terms of reflection on the processes in social work, and in changing my personal stance to one that favors clear, measurable goals. The seminar also contributed to my ability to construct other programs, even when evaluation was not applied to them.

The gap between the effect on the participants and the effect on the agencies’ leadership and staff can be probably partially understood through the following remark written by one of the participants:

The seminar was very interesting and challenging, I enjoyed the process…I would like to be involved in more evaluations…Unfortunately, at my place of work this subject is not high on the priority list—it’s not even at the bottom of it…

Qualitative data offered by participants reflects a broad range of potential factors that might mediate the seminar’s influence on the evaluation capacity of agencies. These include factors related to the worker, program, organization, and to the service. As one participant commented, “Organizations are complex entities: there are many factors and powers operating within and without.” Factors reported by participants are displayed in Table 1.

OPEN IN VIEWER

Table 1.

Factors Influencing the Seminar’s Ability to Affect Agency’s Evaluation Capacity, As Seen by Respondents.

Level	Factors
Individual	Current involvement in the organization (e.g., do they still work in the same organization) Degree of sharing with stakeholders (e.g., only the basic concept, not the methodology) Workload (e.g., overload) Position and seniority in the organization (e.g., individuals in management positions are seen as more influencing their agencies) Level of motivation, interest and understanding of evaluation Ability to share their newfound knowledge (ability to do “PR”)
Program	Worker’s level of involvement in program being evaluated (less involvement or no involvement negatively affects ECB) Whether or not the evaluated program is still being executed by the organization Management’s level of interest in said program
Organization	Resources: time, manpower, budget Organization’s awareness of evaluation and interest in it Organization’s openness and willingness to make changes Organization’s priorities (e.g., does it give its workers time to conduct evaluations) Attitude of managers and supervisors toward evaluation in general, and willingness to deal with adverse findings Organization’s pace of work (e.g., overload, resulting in only the most pressing and immediate problems being attended to) Other workers’ knowledge and abilities related to evaluation Current organizational atmosphere (e.g., crisis, due to change in leadership, internal conflicts or intrigues) Place of social work within the organization (as a primary or secondary service) Donors’ demands and expectations (in NGOs)
Service as a whole (with regard to organizations that are a local branch of a national service)	Service sector (private, public, NGO) Institutionalization of evaluation (e.g., is there a person in charge of evaluations in the service) Service’s rules and regulations pertaining to evaluation Degree of hierarchy and bureaucracy of the service Size of service (larger services seen as harder to influence) Age of service (newer services portrayed as more inclined to evaluation) Coalitions with other organizations (e.g., with philanthropic foundations that insist on evaluation) Degree of privatization of the service (privatized services demand evidence of effectiveness)

Note. ECB = evaluation capacity building; NGO = nongovernmental organization.

This being an exploratory study, no hypotheses were formally proposed. However, we were interested to learn which, if any, aspects of the seminar were related to enhancement of the respondents’ or the agency’s evaluation capacity. Cross-tabulations showed some interesting and plausible relationships, but with such a small sample, only a few of these were statistically significant. Results that were found to be statistically significant, or at least close to significance, are presented in Table 2.

OPEN IN VIEWER

Table 2.

Predictors of Enhanced Evaluation Interest and Capacity (Percentages Reporting Enhanced Interest or Capacity and χ2).

Variable	Respondent		Agency
	Enhanced interest	Enhanced capacity	Enhanced interest	Enhanced capacity
Type of paper
Performed evaluation	59%	71%	47%	31%
Planned evaluation	59%	34%	36%	31%
χ2(1,56)	0.001	6.269	0.529	0.001
P	0.992	0.012	0.467	0.972
Agency’s previous evaluation experience
Some	48%	40%	25%	15%
None	67%	52%	50%	32%
χ2(1,56)	1.929	0.663	3.194	4.299
P	0.165	0.416	0.074	0.038
Management and colleagues support
None	42%	46%	8%	8%
Some	67%	42%	38%	38%
Strong agency support	57%	50%	63%	35%
χ2(1,56)	2.083	0.288	8.518	3.539
P	0.353	0.866	0.014	0.170
Did the evaluation affect the program?
Yes	74%	47%	44%	42%
No	36%	43%	32%	14%
χ2(1,56)	7.623	0.092	0.781	5.123
P	0.006	0.761	0.377	0.024
Consulted superiors and colleagues?
Consulted	69%	53%	43%	34%
Did not consult	41%	35%	38%	29%
χ2(1,56)	3.567	1.417	0.130	0.124
P	0.059	0.234	0.718	0.725
Presented to superiors and colleagues?
Presented	59%	48%	55%	41%
Did not present	61%	46%	23%	22%
χ2(1,56)	0.027	0.040	5.437	2.249
p	.870	.842	.020	.134

Respondents who had not only developed an evaluation plan but actually performed the evaluation reported a greater effect on their own evaluation capacity: 71% of those who actually performed the evaluation—as opposed to 34% of those who only planned one—felt that the experience enhanced their ability to evaluate. No similar effect was found with regard to the agency’s evaluation capacity.

When the agency had had no prior experience with evaluation, the perceived impact on the organization’s capacity was stronger than in the case of agencies that had prior experience. One participant remarked that “In my agency there’s a great deal of emphasis on program evaluation, and since they do it anyway, there was no interest in mine.”

When the respondents described the organizational environment as supportive of their evaluation effort, they also reported that their involvement was positively correlated with the interest of the leadership and staff in evaluation. Of those who had strong support, 63%—versus only 8% of those who had no support—reported enhanced interest.

Positive relationships were also found between perceived impact of the evaluation on the evaluated program and the respondent’s interest in evaluation, as well as between perceived impact of the evaluation on the program and the agency’s evaluation capacity. This was reflected in comments such as “In my case it can be said unequivocally that without the evaluation, the program would have died in its early stages. The evaluation helped the director of the organization establish procedures and protocols aimed at strengthening the program.” On the other hand, one respondent claimed that “I don’t think that the evaluation had enough visibility at the agency to create more initiatives of evaluation.”

Finally, it was found that 55% of those who presented the results of their work to their colleagues and superiors (in the form of either an evaluation plan or a completed evaluation)—versus 23% of those who did not—reported higher levels of interest among the agency’s leadership and personnel in evaluation.

Discussion and Applications to Social Work

The results point out to several interesting possible links between ECB and different aspects related to the students’ work in the seminar. It is important to note, though, that the seminar described in this article was not designed with ECB in mind. The exploration of the relationship between planning or performing an evaluation through the seminar and the agencies’ evaluation capacity was an afterthought, prompted by informal feedback from students.

Garcia-Iriarte, Suarez-Balkazar, Taylor-Ritzier, and Luna (2011) propose a catalyst-for-change approach to ECB. They argue that it may be efficient and effective to train just one person—somebody in a leadership position, or at least with leadership potential—to serve as a change agent. In this article, we pose the question whether practitioners who attend a postgraduate seminar on program evaluation can serve as change agents to enhance their agencies’ evaluation capacity? The survey’s results indicate that most of the students felt that the seminar enhanced their own evaluation capacity, and a sizable minority reported a positive effect on their agencies’ evaluation capacity. This relationship was strongest for agencies without prior experience in evaluation. The results also point out that carrying out an evaluation (as opposed to just planning one) is related to higher self-perceived ability to perform evaluations. Thus, we would suggest that if Garcia-Iriarte et al.’s approach is adopted—“hands-on” training which involves performing an evaluation might prove more effective than planning one.

The perceived impact of the course on the agency’s interest and capacity was greater when the students felt supported by their organization, when they had consulted their superiors and colleagues, and presented their evaluation plan or findings to them. This suggests that if ECB becomes an explicit goal of the seminar, greater emphasis must be placed on stakeholder involvement.

The involvement of relevant stakeholders—especially agency management and personnel—in the evaluations planned and conducted by the students will need to become a course requirement. This is consistent with Stockdill et al.’s (2002) assertion that “to reach a point at which quality evaluation and its appropriate use are ordinary and ongoing practices, ECB practitioners must pay attention to all relevant systems and players as stakeholders” (p. 17).

A more direct approach to the involvement of the agency’s administration is proposed by Adams and Dickinson (2010). This involves a series of workshops for organization staff, and ongoing consultancy to address the need to impart evaluation knowledge and skills to individuals and to work toward change at the organizational level. In our case, some degree of direct engagement by the seminar instructor with management of the agencies represented in class may enhance the impact of the seminar on the agencies’ evaluation capacity. Consequently, if the seminar is to have an impact on ECB, additional resources are needed (Preskill & Boyle, 2008; Stockdill et al., 2002).

Many of the comments written by respondents, and summarized in Table 1, refer to organization-level variables, such as level of bureaucratization and managers’ attitude toward evaluation. In the past, it has been suggested that these be treated as organizational variables to be considered before starting an ECB process, because of their potential influence on it (Preskill & Torres, 1999). However, some of the factors mentioned as influencing the seminar’s ability to affect the agency’s evaluation capacity may be unique to social work settings—for example, whether social work is a primary or secondary service within the organization. One point made by many participants was the pace of work, which was described as hectic, allowing them to deal with only the most pressing and immediate problems, thus not having time to evaluate. This ties in with a comment made by Leviton (2014) in a recent forum on ECB, to the effect that “time and resources are precious for most programs…Evaluation is competing for these commodities with many other considerations” (p. 90). Future research on the influence of classroom-based seminars on ECB could entail an exploration of these qualitatively reported factors as possible mediators of such an influence.

At this point, several limitations of this study should be mentioned. An evaluation based on feedback from participants may well be positively biased. A further source of bias is the fact that less than half of the intended sample responded to the questionnaires. In the future, it may be possible to overcome this bias by a survey directed at the management of the agencies involved. A larger sample and the addition of questions based on the statements collated in Table 1 may also provide a more comprehensive view of the seminar’s effects. Also, this study’s exploratory design does not enable us to infer causality.

Can the findings of this study be generalized to other countries? Virtually, all MSW students in Israeli institutions of higher education are experienced social workers, currently employed in the country’s social service system. The participants of this survey joined the MSW program with a strong desire to apply their learning to their current practice. This may have affected their willingness to embrace theories and practices of program evaluation. Furthermore, some of them had influential positions within their agencies, which may have enhanced the seminar’s contribution to ECB. This is not the case in North America and other parts of the world, where the MSW is often a first professional degree, and therefore most of the students have no prior or current experience. However, even if the experience described in this article is not directly relevant to MSW programs in the United States and other parts of the world, it may still be relevant to continuing education courses offered within academic settings, as well as to professional third degree programs in Social Work and in other professions.