Abstract
This article examines the evaluation process and approach undertaken for a recent 3-year Integrated Justice Practice project. Three key approaches underpinned the evaluation framework or program logic: participatory evaluation, action research, and continuous reflective practice. The project involved an evaluation of community agencies working in complex settings, within a human service delivery context. The mix of processes encouraged these agencies to own the evaluation through providing clarity and grounded information about what works, how, and what does not work and why, so as to improve both service delivery and community understanding, and to affect policy and funding settings. The discussion is situated within several theories of ‘participatory evaluation’ – meaning that the views of service receivers and providers were included both in the research and in its design. These perspectives were essential because input from young people about how legal services support them, and from providers about the policies services adopt is rare. The services and their partners reported that the evaluation process had been ‘transformative’, with each identifying changes in practice. It’s also edifying for the evaluators, revealing that cultural competency, trust, respect and safety are critical elements when engaging with young people with unresolved legal issues, including family violence.
Keywords
Introduction
This article explores an evaluation process and approach undertaken in a recent 3-year research and evaluation project which came to be described as an Integrated Justice Practice (IJP). 1 The discussion considers the evaluation approaches undertaken within a specific service project and aims to identify learnings about evaluation processes and conduct. In undertaking this research and evaluation project (hereinafter referred to as the IJP evaluation), the authors adopted three key approaches which underpinned the framework and project logic: participatory evaluation, action research, and continuous reflective practice. These approaches all sit within the overall framework of participatory evaluation, which was chosen because of its value in working to represent the voices of those with lived experience. These approaches also aligned with the philosophy and values of the authors and the manner in which they conduct their evaluation practice, and additionally, with the values of the four partner agencies in the modest service project being evaluated.
The authors argue that an ethical, value-based approach to evaluation (Gullickson, 2018) helps ensure the process is informed by lived experience. Participatory evaluation is particularly important for members of the community who have been excluded from research processes, or have historically remained at the fringes of research, for example, Aboriginal and Torres Strait Islander peoples. In this project, input from people with lived experience has been sought by (1) including participatory opportunities in the evaluation design and (2) reflection and dialogue between the evaluators, community affected, service providers and organisational management, at the beginning, throughout, and at the end of the project. These methods mean the lessons learned from the exercise have the opportunity to be more transformative for the services being evaluated, partners, practitioners, funders, the evaluators, policy makers and, importantly, the community being serviced. Given the audience for this publication, this article will focus on the conduct and process of the IJP evaluation. A future article is planned which will examine the findings of the IJP evaluation itself.
Background to the evaluation
The project being evaluated, the ‘Invisible Hurdles project’ (IH project), aimed to produce ‘better outcomes for young people experiencing family violence in north-east Victoria’. 2 A key objective of that project is to build stronger partnerships between legal and non-legal service providers, and for the partners to create better referral pathways to ‘joined-up’ services that are available to young persons aged 15–25 years experiencing family violence. It is a project of Hume Riverina Community Legal Service (HRCLS) and was funded by the Victorian Legal Services Board and Commissioner (VLSB + C) Grants program for a period from the end of December 2015 to June 2018. Funding has recently been continued based on the findings of the IJP evaluation.
In the IJP approach to services delivery, a lawyer works collaboratively on site and alongside non-legal agencies where young people are likely to turn for help. The IJP evaluation was embedded in service operations and commenced at the end of December 2015 and continued until June 2018. The modest IH project funds a community lawyer from HRCLS, a free legal advisory, to people living in north-east Victoria and the Southern Riverina of NSW with project partners. A lawyer spends a day a week visiting multiple locations across north-east Victoria and the southern Riverina in various service settings, working with other non-legal professionals who currently reach excluded young people. This model of service delivery contrasts with more traditional lawyering models where the lawyer sits in a legal office waiting for people to make appointments and referrals. The project partner agencies are the Albury Wodonga Aboriginal Health Service (AWAHS) – an Aboriginal community controlled health service; the Wodonga Flexible Learning Centre – an educational centre which re-engages vulnerable young people (160 students) between 15 and 19 years of age with a focus on students struggling with attending school for a variety of reasons, and the North East Support and Action for Youth Inc. (NESAY) – a service providing support to young people and their families with the aim of addressing the issues of education and employment, family relationships, independent living skills and housing.
An evaluation was needed to inform service development, to understand the needs of young clients of the service, and to assess new ways of providing legal services beyond the traditional approach of legal appointments (as noted above). Research indicates that the traditional appointment and referral approach is not working, and that those experiencing disadvantage are more likely to turn to other helping professions (Coumarelos et al., 2012). The IJP evaluation aimed to explore whether embedding a lawyer with other services could build awareness of legal options, build visibility and increase trust and engagement with services and the young people they assist. It also sought to identify what elements might assist in reaching young people, engaging and collaborating with them, and building their capacity and empowerment. The aims were to assess the IJP as a different model and how, or if, it enhanced engagement with young people and their non-legal professional supports, all of whom had not previously been inclined to engage with lawyers or seek legal solutions and options. The authors sought to achieve these aims by also adopting a continuous reflection, learning, development and improvement approach, and action research evaluation approach (discussed further in section ‘Theoretical framework and operationalising the approaches’).
Through partial funding from the Legal Services Board, the authors’ university was commissioned to engage the author researchers (who also have worked as legal practitioners as part of their academic role) to conduct this research and evaluation. Given the small amount of monies available for this work and given its social justice aims, the university provided significant ‘in-kind’ assistance. Funding was quarantined to cover out-of-pocket expenses such as travel, vouchers for young participants and accommodation. As a result, much of the authors’ research labour was provided free of charge. They were mindful of the need to ensure that the research and evaluation process and conduct of the service project were culturally appropriate and inclusive (Australian Government National Health and Medical Research Council, 2005); however, they did not have the funds to recruit an Aboriginal researcher.
There was also a social research aspect to the project. As a brief context for this article, legal problems have a detrimental impact on the health, safety and wellbeing of many children and young people experiencing disadvantage, whom recent research identifies as specifically having difficulty accessing a lawyer (Law Council of Australia, 2018, p. 19). Coumarelos et al., found that 15- to 17-year-olds also had elevated levels of employment and housing problems, while 18- to 24-year-olds had peak levels of employment problems and elevated levels of consumer, credit/debt, family, government and health problems (Coumarelos et al., 2012; Dignan, 2006). In the authors’ social research component of the IJP evaluation, they found the lives of young people were impacted by complex family, individual and systemic influences. These were unpacked and explored, and the findings were explicitly detailed in the Final Report for the project (Curran & Taylor-Barnett, 2018) so as to inform replicable models and policy settings but are not within the scope of the current discussion.
Methodology
An Advisory Group for the project was established in early February 2016 and an evaluation framework and program logic was settled in April 2016. It included the views of the four partner agencies and staff, and fortunately, an Aboriginal advisor as well as an Aboriginal Elder. The brief for the evaluation included the making of recommendations, not only for the partnership and the IH project, but also to inform policy and general approaches for service delivery more broadly.
The idea of having a multi-method triangulated approach was to test and verify the results across the various data collection tools to see if they remained consistent. As will be discussed later in more detail, the questions were framed and informed by previous research and literature (Ball, Wong, & Victorian Legal Services Board Curran, 2016; Curran, 2015, 2017a, 2017b, 2017c, 2017d; Sanderson, Edwards, Williams, & Curran, 2017). The tools and associated interview questions and prompts were discussed with the Advisory Group, an Aboriginal Advisor, the initial service provider, and a Focus Group of young people who added their insights and refined the tools to ensure they were appropriate and relevant before roll-out. The tools were then submitted through the various human ethics processes required by the university and each agency.
Although not ideal, the authors had to gain information secondhand from their legal and non-legal professional supports through semi-structured interviews and professional journals about service intervention impacts over the life of the project. The authors examined data for shifts in engagement of young people with lawyers and the legal system. The non-legal professionals also looked out for changes in their own professional practice over the life of the project and how they found themselves interacting with lawyers around issues that had a legal dimension.
The questions in the tools were designed to overcome the ethical approval limitation on the evaluation which prohibited direct questioning of young people using the service. This was a critical limitation, as client legal privilege operated and interviewing young people accessing the legal help would have been a direct breach of client legal privilege.
The tools elicited qualitative and quantitative data. Curran has written elsewhere of some of the dangers of relying on quantitative data alone, and also about the value of Australian legal assistance services collecting qualitative data alongside quantitative data to identify the reasons underlying the statistics (Curran, 2013). Drawing on both types of data enables the complexities facing clients and the factors around barriers and breakthroughs to be examined.
The authors collected data using the following tools:
A Focus Group mid-way through the project with young people (5) from the Flexible Learning Centre (FLC) discussing a scenario. This helped inform and shape benchmarks as to what an effective service looked like from the standpoint of young people. Although three partners advertised for participants for a Focus Group planned for each agency, through posters and Facebook posts (with prior ethics approval), only one agency was able to host a Focus Group. The partners explained this was because the Focus Groups presented the same dilemma as the purpose of the service project – namely, to engage hard to reach young people and the ‘drop-in’ nature of NESAY and AWHAS.
Guided professional journals kept by the lawyers (4) over a month in each of the two field trips (totalling 8 journals). The lawyer(s) used some guiding questions as a framework and reflected on relationship building with and between the three services and young people, trust, barriers, breakthroughs, complexities of clients, and systemic issues (de-identified).
A 30-minute interview with non-legal professionals from the three partner agencies (totalling 24 over mid and final field trips) who have engaged with the IH project. Additional questions occurred in the final field trip to enable reflection over the life of the IJP evaluation from the point of view of a specific de-identified client, based on the professional’s impressions of their client about Social Determinants of Health (SDH) impacts of this IH project. This included analysis of some perception-based questions on client stress and anxiety levels and improvements.
A 30-minute interview with the project lawyers (4) in each of the two field trips (totalling 6 interviews).
A ‘Professional Development Evaluation Tool Pre-and Post’ (the final field trip) of sequenced professional training provided by the IH project to the non-legal professional partners and administered by HRCLS.
A 30-minute interview with managers (totalling 5) of the four partner agencies in each of the two field trips.
A 30-minute interview with an identified key external relationship holder in the final field trip in February 2018. This was to gauge external perceptions of the project given the passage of time.
HRCLS also collected routine service data on an in-house database for its other accountabilities. This aggregated de-identified data was provided to the evaluation team by HRCLS and included the following:
Number of clients seen (101). All noted they would not have otherwise seen a lawyer; The nature and number of client matters (198); Referrals to and from the services of the IH project (50) were underestimated based on qualitative data (including the date, a brief description and the professional role, for example, maternal and child health nurse and agency). Secondary consultations (288 in the last year of the project) were significantly underestimated based on qualitative data (including the number, nature of the query, date, a brief description and the professional role and agency). The last two listed items were for additional data requested by the authors for the IJP evaluation. Lawyers who recorded the data admitted in the final field trip that they had not included all consultations as they had interpreted the definition more narrowly than the project definition in the Information and Consent Forms they had signed prior to the interview.
Proxies were developed as benchmarks that, if present, suggest a positive outcome and impact of the service. Proxies as measures/indicators of effectiveness and impact of the project were discussed and co-designed through discussion and input from partner participants and staff (using [Curran]’s previous work and literature reviews as a starting point for discussion). These proxies also emerged as critical elements to be measured in the Focus Group:
Engagement of client/patient/professional, staff and organisation;
Collaboration of client/patient/professional, staff and organisation;
Capacity of client/patient/professional, staff and organisation;
Empowerment of client/patient/professional, staff and organisation – which involves giving room for the voices of clients/patients/professionals and staff, and improved advocacy for clients/patients/ professionals, towards systemic change.
Informed by research and participatory discussion with the Advisory Group, Aboriginal advisor, and professional staff, further areas were identified and included for measurement:
Reach;
Previous exposure to lawyers/legal system of young people or family members;
Levels of stress and anxiety (SDH);
Levels of hope (SDH);
Breakthroughs and why;
Legal secondary consultations – frequency, downstream use and utility;
Reciprocity between services in expertise sharing and referrals;
An open question as to what works and what does not work and why;
Impact of service intervention;
Early intervention and prevention of problem escalation (SDH);
Changes in practice (SDH);
Case studies.
A Collaborative Measurement Tool developed by Curran (2017b) and Curran and Taylor-Barnett (2018) based in part on a VicHealth model (2011), was used with data from the final field trip to assess how services were working together over the life of the project. This analysis revealed a movement from service start-up of working in silos, to more collaboration and ‘seamless interactions’ which connected young people across the services for their different needs, resulting in an increase in young people and non-legal professionals trusting lawyers.
The data responses and participant feedback also led to questioning of the professionals around the impact of the project, both personally and professionally, including changes in practice and shifts in ways of working together over time. These reflections were then used to benchmark against the indicators in the Collaborative Measurement Tool.
Theoretical framework and operationalising the approaches
Ebrahim and Rangan (2010) note that ‘organizational efforts beyond . . . scope are a misallocation of scarce resources’ (p. 4) and discuss the need to gather data that is meaningful and has purpose, otherwise it becomes a time and resource intensive but useless exercise. They also warn that any attempt to measure outcomes must be aligned with an agency’s strategy and mission, and the systems and measurements which support such alignment. These key points were highly relevant to the IJP evaluation given, as noted above, the limited resources for the evaluation and limited funds for the partner services in the community service setting.
Adopting the three evaluation approaches provided a means to ‘check in’ with the services about the evaluation process and gain their input on the methodology undertaken. This collaboration identified ways to minimise the burden on organisations and participants and encouraged the partners and participants to feel that they owned the evaluation process and the findings as they had co-designed it.
Participatory evaluation
Stack et al. (2018) note that this is a ‘consultative, iterative approach, emphasizing stakeholder engagement’ (p. 122). They adopted this approach in their program evaluation to ‘mitigate the effects of challenges of responding to the variable nature of the program and multiple stakeholders with a vested interest in its implementation’ (Stack et al., 2018, p. 122). They further note that continual reflection and quality improvement allowed these challenges to be addressed and continue to allow lessons learned from earlier evaluations to inform the design of future studies (Stack et al., 2018, pp. 122, 129–130). The approach of the IJP evaluation was very similar but, rather than a government-wide project, the authors’ evaluation was a small ‘not for profit’ sector project across four different partner agencies in three regional settings in NSW and Victoria. Similarly, it was conducted at multiple time points across the life of the reform program, enabling refinements to occur through monitoring and understanding the impact of the new approach to service delivery − namely, using IJP with non-legal agencies working with a justice agency, in three different sites.
Stack et al. conclude, [t]his highly collaborative approach was critical to reducing the risk of duplication and over-burdening stakeholders, while ensuring the program-level evaluations collect all relevant activity-level information . . . Engaging stakeholders early in the design process establishes evaluation credibility and fosters buy-in. In this way, stakeholders observe their insights being incorporated into the design, conduct and findings of the evaluation, pre-disposing them to a sense of shared commitment to the evaluation findings and recommendations. (Stack et al., 2018, p. 129)
Rogers et al. also highlight the benefits of the participatory approach, in a way that the authors’ experience also demonstrated, namely, ‘[t]he value of participating for the community members and partner organizations is shared and the benefits and implications for participants and the evaluator’ (Rogers et al., 2018, p. 79). Rogers et al. (2018) urge practitioners and evaluators to ‘consider participatory ways of working with communities to support community directed action and social change’ (p. 79). The authors’ evaluation, like that of Rogers et al., had multiple audiences, multiple purposes for the evaluation . . . (implementation, outcome, and improvement), and cross-cultural dynamics. In Rogers’ study, the evaluator had to draw upon extensive experience working in culturally diverse settings, a wide range of data collection methods from a generalist skill set, high-level adult teaching techniques, and appropriate rapport building strategies. They share how an evaluation was undertaken using a developmental approach in combination with empowerment evaluation theory, similar to the current authors, and engaged Aboriginal communities in remote and urban contexts (Rogers et al., 2018, p. 80).
Similarly, the IJP evaluation was participatory in that it was embedded from the commencement of the service, starting with a dialogue between the partners, the Advisory Group (including an Aboriginal advisor) and practitioners as to what was important both for effective service to young people and for an evaluation of that service.
A Focus Group with young people one third of the way into the project, was pivotal in developing benchmarks around what would be effective in engaging young people. The authors felt it was critical to find a way of enabling the partner agencies of the evaluated service that was still in the early stages of development to hear the voice of and include the young people who were service users. The ethics approval process required Focus Groups with only young people who were not currently using the IH project service, to ascertain their views as to what an effective service would look like, using a scenario as a discussion point. Their input also assisted the evaluators in formulating additional reflective questions for the final field trip.
The discussion with non-legal professional staff sought their views on how to enhance the evaluation process and provide their input (which was subject to later ethics approval requirements), and how to tailor the evaluation to best suit the young people the service targeted.
The next section further explores how, throughout the evaluation the views of participants in the research, the Advisory Group (including Aboriginal advisor and service providers) the agency management and staff were included in the evaluation and resulted in recalibrations to the service delivery and evaluation. For example, as mentioned above, a request from a participant after the first field trip that they be given opportunity at the end of the project (in the last field trip) to reflect on the shifts and changes in practice from service start-up to the third year. This request resulted in the adaptation of the Collaborative Measurement Tool and interview questions and subsequent ethics variation to accommodate this.
Action research
Participatory action research has been described as a reflective process of progressive problem solving (Leering, 2014), led by individuals working as part of a ‘community of practice’ to improve the way they address issues and solve problems (Dick, 2009/2011). Other studies suggest that changes in practice are impacts as well (Tobin-Tyler, 2012; Triado, White, & Brown, 2013; World Health Organisation and Commission on Social Determinants of Health, 2008). Using a participatory action research approach means not only using literature informing the project but also collaboration in design by participants, including service users and professionals. Accordingly, there is collaboration in the design with those who provide the services (Sommerlad, 2004, 2008) within a model of continuous learning, development and improvement (Jayatilleke & Mackie, 2013). The authors adopted this approach for the IJP evaluation, building on previous evaluation practice of other services undertaken by Curran since 2011.
Rogers et al. (2018) note that the origin of action research is often attributed to the American psychologist Kurt Lewin who described the methodology in the 1940s as ‘a comparative research on the conditions and effects of various forms of social action, and research leading to social action’ (Lewin, 1946, p. 35). That process involves iterative planning, action, and analysis of that action. More recent descriptions of action research (McCutcheon & Jung, 1990) emphasise that the methodology is undertaken by the subjects of the research, that they undertake it as a collective, and that their purpose is understanding and therefore improvement of their practice or situation.
As Rogers et al. note, ‘participatory evaluation can be embedded in programs to support good governance and facilitate informed decision making in Aboriginal communities in remote and urban contexts . . . complex community development programs used participatory processes to build evaluation capacity and solve problems’ (Rogers et al., 2018, p. 78). In the IJP evaluation, the authors had an Aboriginal advisor on the Advisory Group. In addition, an Aboriginal Elder advised at each stage of the project specifically on the approach taken, the tools, the draft interim report and the final report. These discussions guided the authors in how they questioned, approached and dealt with data in a context of the impacts of service approaches on community, the importance of dialogue and inclusion, and the significance of past and present colonial practices on engagement and trust. In the IJP evaluation, one of the partner agencies was an Aboriginal community-controlled health service, and participatory evaluation enabled the partners to solve problems that arose in the course of the IH project. Aboriginal participants offered insightful feedback about the contextual realities of the evaluation, which fed back into decision making by the managers of the partner agencies regarding the future of the IH project. The authors’ findings from this input helped to develop recommendations specifically around the role of trust and need for community cultural recognition and cultural safety, respect and dignity in service delivery (Curran & Taylor-Barnett, 2018).
The IJP evaluation, after the initial participatory input, undertook two field trips a year apart. This timeframe allowed preliminary exploration and time for the partner agencies to reflect and shape the service delivery, having been informed about its progress and any key impediments and positives. In this way, it was research in action which provided opportunity for continuous reflection and improvement. The evaluators shared feedback through an interim report with preliminary findings, as well as a debrief after the first field trip. This again encouraged participation as well as reflection and learning that could shape the project along the way, taking on broad elements that could make the service more effective in engaging with young people and services. As an illustration, in the first field trip, it was made clear by Aboriginal research participants that the lawyer tended to sit in an office when based at AWAHS and that this was perceived as a form of ‘arrogance’ and was unhelpful to engagement. This showed that it was easy to slip back into traditional modes of lawyering. After a detailed discussion with participants about what the lawyer could do to overcome this perception, the lawyer took steps to be more visible, and to attend functions and events such as NAIDOC week and Reconciliation events which are so significant to Aboriginal communities, and to better value informal exchanges. The following field trip followed a similar structure, with a debrief and provision of a draft final report for comment and discussion before being finalised. In returning to issues raised in the first field trip, it was clear that the lessons learned had led to changes in practice that enhanced engagement and trust with Aboriginal clients and Aboriginal community members.
Continuous reflection, learning, development and improvement
From the outset, the program logic for the IJP evaluation was not about measuring outputs or client numbers. The authors were of the view that if young people were not being currently reached for their legal problems, then social research could explore the unresolved problems, seek to uncover the complexity, and work to engage and counteract poor experiences of the law and legal system. This had been identified as a feature in Curran’s (2012) previous research. By embedding the evaluation from service start-up and doing it in stages, rather than at the end only, the authors structured opportunities to learn, and to recalibrate – not just for the evaluation team, but to enable managers, the organisations, professional staff (including teachers, lawyers, doctors, nurses, social workers, counsellors, Aboriginal workforce and reception staff, and so on) to think about their mode of operation and practices with young people. Also critical were the opportunities for the different professionals to each take time to reflect upon and learn about their own professional approaches and ethical duties which could, at times, seem to be at odds. This ‘unpacking’ helped staff to find ways they could work together mindfully and accommodate different practice.
Leering (2017) has observed that to be prepared to face myriad interconnected and complex challenges, twenty-first century legal professionals need a reflective legal education that emphasises self-assessment and self-efficacy, develops metacognition, supports and diversifies lifelong learning, promotes innovative and critical thinking, strengthens legal professionalism, and builds a stronger aptitude for problem-solving. She sets out a ‘working conceptualization for reflective practice’ (p. 49). This approach was integrated into the IJP evaluation, culminating in the final field trip with its reflective questions in the interview tool and in the professional journals kept by research participants. This approach aimed to adopt, through the evaluation, a framework for participants to develop strategies to improve and deepen their professional learning, especially from experience. As Leering (2017) observes, [a] strong reflective capacity can provoke innovative thinking and inspire change through critical and collective reflection – on whether the status quo is acceptable, on what works and on what is possible – which can in turn generate new insights and possibilities, challenge assumptions and mental models, and interrogate current practices. (p. 51)
The data at the end of the project revealed that all participants noted changes in practice and ways of collaborating across and between the agencies. Crockett and Curran (2016) have previously noted some of the international research (Commission for Social Care Inspection, 2006; Kilian, 2009) reveals that the ‘qualities about which service users are particularly concerned’ include the following: choice, flexibility, information, being like other people, respect and being heard, fairness and no discrimination, cost and value, and safety. Other qualities include the following: responsiveness, empathy, involvement, accessibility, listening carefully, being kept up to date, discreet atmosphere, explaining things clearly and ‘in a way I understood so I knew what to do and what was going to happen’, helpfulness of staff, and provided with relevant information in a timely way. Crockett and Curran (2016) conclude that [t]he approach also encourages staff to reflect on the way they provide services and what characterises service quality and effectiveness, that is, reflective practice. By collecting both quantitative and qualitative data from interviews and professional journals with guiding questions the authors were better able to unpick and identify the policy and legislative settings. They also identify the impact of complex clients on the services work and identify what worked and did not work and why – features which the authors consider essential in human service research evaluation such as the provision of legal assistance services.
The data from the IJP evaluation revealed that the qualities young people found critical were similar to the qualities discussed above (Crockett & Curran, 2016). Finding ways of enabling participants to include the young people that the service targeted, and seek their views on what would engage them – and also provide opportunities for staff of all the three partner agencies to reflect on what these young people said, as well as incorporate their vast experience as professionals working with young people in a specialised way – was something the HRCLS had not done prior to this project. The research and evaluation has also assisted HRCLS to learn about what an effective youth service looks like. The data from the final field trip revealed that HRCLS, as a service agency, had rethought its whole service delivery approach to young people and other clients as a result of the IJP evaluation.
Discussion
Participation and ownership
In the authors’ experience, partner input in the design and implementation of the evaluation was key. The intention of enlisting the help of the four project partners at the initial stakeholder and Advisory Group meetings was to check that the IJP evaluation measures were relevant, useable, culturally and age appropriate, respectful, realistic and practicable. This actioned the participatory evaluation approach underpinning the program logic, and gave participants the opportunity to ask questions and provide input about:
The appropriate data to collect throughout the project (keeping in mind the need for a low burden on staff, given existing caseloads and commitments and the stretch of community service agencies);
How to best evaluate the project;
How to engage the target young people client group;
How the evaluation might obtain valuable input from the client group of their experience of the IH project, mindful of ethics considerations;
Whether staff and management would be able to participate in a professional interview and complete Professional Development Evaluation Surveys after they undertook any training;
Utilising the agencies’ expertise in working with young people and how the IH project should be framed;
The expertise of AWAHS and the Advisory Group on what measures were needed to make the study and the proposed service culturally aware and sensitive, especially given a focus on Aboriginal young people and their Aboriginal agency;
Their understanding of their own organisations’ ethics processes which might be required in addition to the University Human Ethics Committee Approval needed.
By seeking views from partners and research participants, the co-design of the evaluation occurred. This collaborative process also ensured a voice for service providers and young people (albeit limited due to ethics constraints) which helped to ensure the evaluation was authentic, that it resonated with the participants, and that they gained ownership in the evaluation process itself.
Complementary and safe
The three approaches for the evaluation overlap are complementary and support each other. However, they also take time, planning and tweaking through adjustment as they inform the evaluation as it progresses, and can therefore involve intense periods of ‘to-ing and fro-ing’. For the IJP evaluation, these approaches were critical in working through emerging issues and seeking to continuously improve the service engagement, to enable it to better reach young people with legal problems. In particular, to enhance their safety as, so often once explored, the legal problems revealed situations of family violence that were going unreported with legal options being overlooked.
Some of the evaluation findings included that young people experiencing family violence are unlikely to reveal it but are likely to disclose other issues first to test out a lawyer before opening up. In some instances, family violence was only disclosed after 3 months of contact. Young Aboriginal people with a lived experience of the Stolen Generations perpetrated by the government of the day are distrustful of the legal system. Many Aboriginal families see the second round of the Stolen Generations embodied in child protection. Young people will wait and observe how the lawyer interacts with others first, sometimes for up to 6 months, before feeling they can approach a lawyer or their support worker about a legal issue.
The IJP evaluation took time to build momentum. This is consistent with the evaluations of other Health Justice Partnerships around the world (Tobin-Tyler, 2012). Young people will only open up if they feel safe, the lawyer is approachable, non-hierarchical, non-judgemental and speaks simply. Over the life of the IJP evaluation, the data revealed that with a visible lawyer on site, available and approachable, more young people were availing themselves of legal help, either directly or through a trusted non-legal professional than would have been the case without the IH project. Quick access to secondary consultations with the lawyers built the capacity of non-legal professionals to respond in a timely way with improved decision-making and reduced the professionals’ sense of anxiety as their confidence in the lawyer and the lawyer’s advice grew over the life of the IH project.
Conclusion
It may seem that having several approaches in one evaluation is complex, yet there are many interconnections and the service setting where the evaluation took place is also complex. In the authors’ view, such participatory evaluative approaches assisted in developing a context for and ways to deliver quality, effective and relevant human services. A combined approach also can inform replicable models of care and the changes in practice that can shape policy and decision making, including service funding. The authors’ found that the approaches adopted have also had internal impacts on the project, which means the project can, as Fleming (2018) describes it, have ‘increased organisational effectiveness, and thereby justify the resources it consumes . . . embodied, power-conscious and collective learning approaches and strategies’.
The authors agree with the conclusion of Stack el al. that [t]his highly collaborative approach was critical to reducing the risk of duplication and over-burdening stakeholders, while ensuring the program-level evaluations collect all relevant activity-level information . . . Engaging stakeholders early in the design process establishes evaluation credibility and fosters buy-in. In this way, stakeholders observe their insights being incorporated into the design, conduct and findings of the evaluation, pre-disposing them to a sense of shared commitment to the evaluation findings and recommendations. (Stack et al., 2018, p. 129)
The authors argue that including the views of those individuals often overlooked or missed in evaluation design enhances opportunities to learn. It also helps to appropriately target scarce resources and make agencies more efficient and effective, and accordingly refine their chances of making a positive and more sustainable impact. Without such participatory, inclusive and reflective approaches, some of these opportunities might otherwise be missed, and there is a greater risk that evaluations will be too remote from the actual exigencies of service delivery. We hope that other evaluators will find this article useful.
Footnotes
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The author(s) received part financial support for the conduct of the evaluation for the Victorian Legal Services Board and Commissioner through the Hume Riverina Community Legal Service grant but no funding for the research, authorship, and/or publication of this article.