Abstract
Programs are the primary unit of action undertaken by any organization. Through their programs, organizations provide direct services to people in need, offer solutions to socially rooted problems, and engage with communities, among other functions. The impact and significance of organizational programs may be sometimes overemphasized with anecdotal evidence about the effectiveness of an organization in meeting its service delivery goals. A corrective to this situation, of course, is improved methods of program evaluation that can be implemented across a range of program types within organizations. The Handbook of Practical Program Evaluation offers a means by which program managers and evaluators can begin to think about conceptualizing and undertaking such an evaluation process.
Now in its third edition, the Handbook of Practical Program Evaluation is an edited collection divided into four sections (28 chapters across approximately 700 pages): part 1—Evaluation planning and design; part 2—Practical data collection procedures; part 3—Data analysis; and part 4—Use of evaluation. The chapters follow in a useful manner from research study design through to dissemination and, as an edited collection, the chapters are well organized and are all written in a similar manner with their respective focus. Many of the chapters are authored by leaders in their fields of study who likely have sufficient insight about program evaluation and research methods. A review of an earlier edition of the text commented on the absence of theory related to program evaluation and organizations (Schröter, 2006), and this remains consistent for the third edition. The content of the third edition is similar to that of the second edition, but some important additions have been made in some key research areas—in particular, qualitative approaches and the range of research study designs that might be considered for program evaluation. With regard to the later, whole chapters have been included that focus on conducting case studies and comparison group designs, along with conducting multisite evaluations. In all, this edition of the book (in comparison with the two earlier editions) provides a more comprehensive overview of a range of research designs, methods, and issues associated with program evaluation. This edition also offers an “Instructor’s Guide” that is available online and includes class exercises, key terms, and questions for further study.
The first section includes chapters on engaging with stakeholders, using logic models, and conducting exploratory evaluation and covers a range of other topics related to research study designs, including comparison group designs, randomized controlled trials, and monitoring program outcomes. The second section focuses on data-collection methods, with chapters themed around surveys, records and documentation, and qualitative interviewing, among others. The third section provides a brief overview of analysis methods, including qualitative and statistical analysis. This section also includes chapters on meta-analyses and systematic reviews and cost-effectiveness. The final section focuses on the use of evaluation with chapters on report writing and problems with evaluation studies, along with a focus on current trends in evaluation studies and challenges that evaluators might encounter. Together, the book provides an extensive volume of knowledge for more experienced program evaluators.
In the first chapter, the editors describe program evaluation as “the application of systemic methods to address questions about program operations and results” (p. 5). This broad definition remains consistent throughout each chapter in the text. However, with such a general definition, it is not entirely clear for the reader what exactly the difference is between general social science research about programs and their subsequent outcomes and a program evaluation study. This all-encompassing definition may have been a useful approach, aiding the editors in reaching a wider readership within the nonprofit and public sector. At the same time, it might also be useful for the inexperienced program evaluator to have a more substantive introductory chapter that begins with a more focused description of evaluation—what it is, what it means, and the social context in which it is being undertaken. There remains a sociopolitical and political-economic context in which the domain of program evaluation has emerged in service-based settings, but this book provides little discussion of this context, instead favoring an applied approach. As such, it does not offer a critical analysis of the evaluation process.
Many of the chapters could also be used as academically oriented stand-alone chapters on data collection, data analysis, and dissemination, for example, in a textbook designed for a social science graduate educational program. Although a positive for the potential graduate student audience, this higher level of complexity might have a negative impact on whether the book can meet its stated objective to “strengthen program managers’ and staff members’ abilities to meet the increasing demand for evaluation information” (p. 6). Furthermore, while a focus on developing and designing program evaluations is necessary for managers and program staff, the book fails to provide the context in which program evaluation has emerged and exists in direct service-based organizations. Aspects of the intra- and interorganizational context might directly affect the success of organizational managers or program staff in undertaking evaluations; such factors might include budgetary restrictions, relationships with funders, and the day-to-day functioning of their program. These challenging experiences will likewise have direct implications on the systematic research process that is required for useful program evaluations—a focus that remains underexplored in this text.
As the book was designed to address practical implementation, it is important to highlight the extent to which it meets this standard of practicality. A disconnect between scholarly output and direct practice within an organizational context can be problematic when attempting to apply methods of program evaluation. The editors describe that they use the term practical program evaluation “because most of the procedures presented here are intended for application at reasonable cost and without extensive involvement of outside experts” (p. 6). The term practical is being used as a synonym for feasible. The feasibility of program evaluation is an important issue to recognize for organizations, many of which are experiencing budgetary constraints along with funding structures based on direct billable hours, thus making it difficult to even consider undertaking an evaluation study. However, even this notion of practicality as a guiding principle of the book is missing from many of the chapters. Furthermore, feasibility is not the only issue of practicality to consider. For example, practicality might also include the expertise of program managers, the nature of day-to-day service delivery within organizations, and interorganizational dynamics between organizations and funders. With regard to the later, many direct service delivery organizations belong to a particular sector with a common public sector funder. A definition of practical program evaluation might also need to consider the relationship dynamics and nature of collaboration within a sector and the subsequent impact on successful program evaluation.
As stated earlier, finding ways to address the challenges organizations experience when conducting program evaluation and adapting these research methods accordingly would have been a welcome addition to this definition of “practical program evaluation.” One solution might have been to include in the book more in-depth case examples demonstrating how the evaluation process might unfold in an organization and within particular program types or, alternatively, make available online as supplemental material. Some space could have been afforded to these applied examples of successful program evaluations within organizations. For instance, case studies that the authors know about or have participated in could have been integrated within the sections on developing and designing an evaluation study. Reference to a range of program types could be helpful for program managers in different areas of practice and in different organization types (whether human services, membership organizations, educational institutions, etc.). Keeping in mind these ideas about practicality and the stated target audience of the book, some of the chapters—such as the meta-analysis and cost-effectiveness—might be important for professional program evaluators to understand, but—due to their complexity—may have less applicability for program managers trying to undertake a more “practical” approach to program evaluation. Also, the chapters in the final section of the text could have been combined, as they were somewhat redundant in the focus on evaluation dissemination and problems emerging with evaluation.
As it stands, to decipher the technical research jargon and to follow many of the evaluation processes outlined in this text, many program managers and staff persons reading this book would require a substantive knowledge base of mainstream social science research methods. This is a knowledge base—depending on the area of practice and budgetary size of the organization—that they may not have. That said, this third edition of the handbook will likely remain a useful text for the more advanced program evaluator, educator of program evaluation, and social science researcher working with organizations in evaluating their programs.