Abstract
Welcome to the June 2021 issue of the Evaluation Journal of Australasia (EJA) that brings a strong advocacy feel from the social justice and capacity-building perspectives. This dominant feature in the issue brings to light that social justice and capacity building are quite often the taken-for-granted aspects that do not receive the focus they warrant. Readers of this issue will find that in the papers presented, empowerment evaluation approaches remedy this situation when accountability and internal evaluation unite. Accountability is mapped through an organisation’s mission and vision and enacted by its leadership that then drives and dictates the performance of its functions in the best interests of those it serves. As the papers in this issue reveal, internal evaluation has inherent powers and potential for emancipatory and enabling outcomes that serve as an underlying source of empowerment to those that are impacted by the programmes being delivered.
Fetterman (2005) provides relevant and useful support to further introduce the significant evaluative notions and sentiments of the papers in this issue. He refers to 10 guiding principles of empowerment evaluation that encourage and promote internal accountability and contends that these principles are sustainable and prevail even after external evaluators and funders have been and gone. These guiding principles of empowerment evaluation are as follows (Fetterman, 2005, pp. 43–50):
Improvement: Commitment to improve the lives of people through improving programmes;
Community ownership: The community and its members operate and shape the programme activities;
Inclusion: Invite and encourage participation from key stakeholders – include rather than exclude;
Democratic participation: Determining how the pertinent stakeholder groups will interact and make decisions – ensuring that everyone has a vote;
Social justice: Respecting people, protecting an individual’s pride, honouring the individual struggles people are facing and keeping the focus on social justice, equity and fairness towards a just society;
Community knowledge: Respecting the invaluable knowledge of local community members;
Evidence-based strategies: Utilising useful information arising from track records and external credibility to allow local communities to build on the ideas and models that offer adaptability;
Capacity building: Programme staff and participants learning how to conduct their own evaluations – building skills and evaluative capacity;
Organisational learning: Enhancing an organisation’s receptivity to new adaptive strategies and responsiveness to environmental changes – self-reflection and continually evaluating performance; and
Accountability: Employees and staff members are individually accountable and accountable as a group or community of learners – individuals hold one another accountable for promises and commitments.
These principles are individually common knowledge for most evaluators, but as stated above they can also lose their significance as they become taken for granted. The overlap and relationship between these principles can often hide the individual importance. Therefore, I have chosen to highlight them as a refresher and introduction to the great papers to follow in this issue. Similar principles, from an array of literature by different authors, are discussed and practised in the case studies presented in this issue that contribute informative methodologies towards a pathway of empowerment evaluation that breaks down the power dynamics and values and promotes the voice of the otherwise oppressed.
Two papers in particular convey evaluative approaches that value the voices of women and children by breaking down the power imbalance through designing evaluation methodologies that align closely with the aims and objectives of service providers and thereby respecting diversity and the needs of their clients. A further paper in this issue provides an insight about the perspectives of leaders of philanthropic foundations in relation to how they evaluate grant applications and what factors influence their perspectives. The strength of all three papers is how the development of the internal evaluation process is essentially customised to the mission and aims of the organisations involved and their respective programme delivery.
The first paper for this issue is titled ‘Valuing beneficiary voice: Involving children living in out-of-home care in program evaluation’ by Ruth L. Knight and Kylie L. Kingston. These authors provide a detailed step-by-step process of the design and development of a survey instrument for children in out-of-home care, which potentially has wider application for mentoring and educational programmes. The importance of listening to children using a relational approach is validated through evidence gained that children, in this study, are willing and able to provide feedback about their relationships with adults, which provides useful information about their well-being. The authors demonstrate their care and thoroughness in the development of the instrument and outline how they navigated complex challenges along the way. The underlying process and resulting survey instrument highlight the importance of evaluation design that connects with those who should be front and centre of programme evaluation to enable their voice to be heard and valued. The concept of accountability is utilised as the theoretical framework for this study and is seamlessly woven into the evaluative process, highlighting the cohesive nature of both accountability and evaluation. Therefore, the internal evaluation process in this case study manifests itself as intrinsically linked to the organisation’s primary accountability, that is, to its clients and related activities.
The second paper for this issue explores the strengths, benefits and challenges in using feminist participatory action research (FPAR) to evaluate a women’s leadership programme. Titled ‘We are women! We are ready! Amplifying women’s voices through feminist participatory action research’ by Tracy McDiarmid, Alejandra Pineda and Amanda Scothern, this paper utilises a case study to demonstrate the co-designed evaluative approach. This approach is aimed a understanding how a women’s leadership programme created positive change for participants. The authors explain what a feminist approach is and critically review participatory action research as part of the substantiation of adopting the FPAR approach and the benefits and limitations therein. The strength of this story, similar to Knight and Kingston’s paper discussed above, is the methodological approach used to inform transformative change in power relationships. According to the authors, the utilisation of FPAR had further significant outcomes in terms of increased ownership of evaluative process and outputs by co-evaluation partners, which led to strengthening of confidence, skills and evaluative experiences for local stakeholders to engage in their own evaluative work. Other important factors such as trust, participation and voice became a systematic part of the evaluative activities. By adopting a feminist approach to evaluation, a wider diversity of experience and insight was achieved to give rise to local leadership and ownership of knowledge creation processes.
In the third paper for this issue, authors Alexandra Williamson and Kylie L. Kingston also focus on internal evaluation, moreover evaluative culture within philanthropic foundations. In their paper titled ‘Performance measurement evaluation and accountability in public philanthropic foundations’, the perspectives of senior leaders from 28 philanthropic foundations were analysed to understand what is considered important when evaluating funding applications and deciding on who to distribute funds to. The authors use a theoretical framework of accountability, which enables a broader lens to view accountability and evaluation. This framework was found to align with the philanthropic principles of gifting and generosity. The study was based on a critical realist ontological approach to derive four key themes from their analysis of interview data, representing the influences for selecting prospective recipients. The four themes are motivation, values, criteria and processes of evaluation. The critical contribution of this study arises from the raising of awareness and understanding of how foundations of this nature use evaluation in their work for public benefit purposes.
In this issue, Rick Cummings is our profiled evaluator for the Evaluator Perspective and also our book reviewer. The EJA really appreciates Rick’s generosity and support to enrich this issue. It is always a pleasure to read the incredible career paths and evaluation journeys of the many esteemed evaluators who are great contributors to the Australian Evaluation Society (AES). Rick, like many other fellows of the AES, kind of fell into evaluation and discovered how interesting and important it was. As Rick’s story unfolds, you cannot help noticing that he attributes his success to the great evaluators who supported him and helped him develop professionally, the same evaluators that many of our previous Evaluator Perspective subjects have also referred to as the major influences in their evaluation practice. Rick is a shining example of the high standard of the AES fellows and his story is a valuable contribution to this issue.
And finally, our book review by Rick Cummings for this issue is based on the book titled Changing Bureaucracies: Adapting to Uncertainty, and How Evaluation Can Help, edited by Burt Perrin and Tony Tirrell. Rick promotes this book as a must read and relevant for a wide audience of readers who are interested in the relationship between evaluation and bureaucracy, the challenges that evaluators face in the evaluation of bureaucracies and the potential role for evaluation to support the change process. Rick provides a great overview of the book outlining the book’s 17 authors who are all experts in evaluation with vast experience working with a large range of bureaucracies including government agencies and large international aid agencies. With the current complexity of public policy and programmes and changes afoot, Rick suggests that this is a timely book, to understand not only bureaucracies but also the inherent issues within the context of evaluation.
As many of our EJA readers and AES members will have seen, a Call for COVID-19 evaluation–related papers has been circulating since November 2020. We will continue to receive submissions for some time yet, so if you have not already submitted a paper on COVID-19, there is still time. This will mean that over the next year, the EJA quarterly issues will have an array of interesting papers highlighting evaluation during the COVID-19 era. See link for the COVID-19 Call for Papers at EJA_papers_COVID19.pdf (aes.asn.au)
As always, we are open to all evaluation-related papers and look forward to receiving your submission soon.