Aboriginal Family Planning Circle evaluation: Empowering Aboriginal communities in evaluating and future-proofing Aboriginal-led community programmes

Introduction

The impacts of colonisation continue to have a profound impact on Aboriginal families and communities. The removal of Aboriginal children from their families, collectively known as the ‘Stolen Generations’, has deprived many of the family ties and cultural identity, and contributed to a deep mistrust of government institutions. The unresolved trauma of the Stolen Generations continues to be passed onto one generation after another, with one consequence being that Aboriginal children continue to be over-represented in the child protection system (Blignault & Williams, 2017).

In 2017/2018, it was estimated that 38.9% of the 17,387 children in out-of-home care (OOHC) in New South Wales (NSW) were of Aboriginal origin. Aboriginal children (71.2 per 1,000) in NSW were over 10 times more likely to be in OOHC than their non-Aboriginal counterparts (6.4 per 1,000), with this trend expected to increase over time (Productivity Commission, 2019). The independent Tune Review in 2015 acknowledged that the OOHC system is ineffective and is ‘failing to improve long term outcomes for children and arrest the devastating cycles of intergenerational abuse . . . particularly for Aboriginal children, young people and families’ (NSW Government, 2016, p. 3). The Review also found that around $300 million (AUD) per year was being spent on family support programmes without evidence that they work (McFarlane, 2016). Peak Aboriginal organisations have continued to advocate for early intervention family support service models which keep Aboriginal children with immediate families or in kinship arrangements (AbSec, 2019), but most of these Aboriginal-led initiatives are also not evaluated (Williams, 2018).

This article explores two external evaluations undertaken by WESTIR Limited (WESTIR), a non-Aboriginal community-based research service in Western Sydney, on an Aboriginal-led family support programme called the Aboriginal Family Planning Circle (AFPC). Commissioned by AFPC staff, the purpose of the first evaluation in 2015 was to provide a comprehensive assessment of how effective the AFPC programme had been for participants and identify programme aspects that could be improved or developed. WESTIR was again approached by AFPC staff in 2017 to undertake a second evaluation in 2017, with the purpose of examining whether the AFPC programme had continued to meet its objectives and address some knowledge gaps identified by the funding body, particularly the estimated savings and return on investment that the programme created for the OOHC system in NSW. This article describes the AFPC model and the evaluation process, as well as the results and recommendations from both evaluations. It will describe the lessons learned from the evaluations and emphasise the need to adequately resource Aboriginal programme evaluations in the community services sector now and in the future.

History of the AFPC

The AFPC programme was established in 2006 by the Aboriginal Family Workers Support Group working in the Blacktown Local Government Area (LGA). The AFPC programme works with Aboriginal families in Western Sydney to address their complex issues and reduce the risk of having their children put into OOHC. The issues addressed by the programme include drug and alcohol addiction, housing and financial instability, domestic violence and intergenerational trauma.

The main features of the AFPC programme include referral intake and assessment, initial engagement, service delivery, exit planning and training and measurement. A description of each stage is shown in Figure 1. The AFPC programme differs from other Aboriginal family support models as it emphasises a person-centred approach, localised governance and the involvement of the community in co-designing goals and solutions to family issues and needs. The programme also works with Aboriginal families across the child protection continuum, from early intervention to prevent child removal to working with families to secure child restoration.

OPEN IN VIEWER

Figure 1.

Aboriginal Family Planning Circle programme model.

The AFPC programme was initially a volunteer-run, community-based initiative, but funding was sought from the then NSW Department of Family and Community Services (FACS; now NSW Department of Communities and Justice [DCJ]) when it outgrew its original capacity. From 2015 to 2017, the funding was used to appoint an AFPC Coordinator at the FACS Aboriginal Intensive Family Based Service office in Rooty Hill and expand service delivery to both Blacktown and Penrith LGAs. In 2018, with the support of the Aboriginal Family Worker Support Group, the AFPC programme was moved to Marrin Weejali Aboriginal Corporation, an Aboriginal-registered non-government organisation in Blacktown LGA. Despite no additional government funding, the AFPC programme continues to support more families than outlined in their funding agreement.

Evaluation methods

WESTIR was approached by AFPC staff to undertake an initial evaluation of the AFPC programme in 2015. The purpose of the first evaluation was to provide a comprehensive assessment of how effective the AFPC programme had been for participants and identify programme aspects that could be improved or developed.

A cross-sectional, qualitative framework was adopted to collect data for the first evaluation during August and September 2015. This approach was chosen because it allowed participants to represent their experiences in their own words. Semi-structured 1 hr interviews were conducted by WESTIR researchers with seven clients, mostly parents and kinship carers of Aboriginal origin (n = 7). Aboriginal and non-Aboriginal service providers involved with the AFPC programme were also invited to participate through semi-structured interviews, focus groups and written responses (n = 15). All interviewees were asked about aspects of the AFPC programme, including initial involvement, general impressions, programme strengths and outcomes, and areas for improvement. Interviewees were selected by AFPC staff through convenience sampling and included those who had past and current involvement with the AFPC programme. All interviews and focus groups were recorded, transcribed and thematically coded, with data analysis used to formulate evaluation recommendations. The final report of the first evaluation was provided to AFPC staff and the Aboriginal Family Support Worker Group in January 2016.

Following a review of the first evaluation report, DCJ senior management requested evidence on the economic value of the AFPC programme. As a result, WESTIR was again approached by AFPC staff to undertake a second AFPC evaluation between March and June 2017. The purpose of a second evaluation was to examine whether the AFPC programme had continued to meet its objectives and address some knowledge gaps identified by the funding body, particularly the estimated savings and return on investment that the programme created for the OOHC system in NSW.

The second evaluation adopted a mixed-methods framework, which was undertaken between March and July 2017. The qualitative method from the first evaluation was adopted to undertake and analyse additional semi-structured interviews with five service workers (including the AFPC Coordinator; n = 5) and six former and current clients (n = 6). Analysis of programme records and service provider entry surveys were also undertaken but are not discussed in this article as links between programme outcomes and preventing children going into OOHC could not be established at that time.

The second evaluation also included a return on investment analysis. The savings of the AFPC programme were calculated by multiplying the number of children who were restored or prevented from going into OOHC due to the AFPC programme in 2015/2016 with the Productivity Commission’s (2017) average cost of keeping a child in OOHC in NSW in 2015/2016. Figures on the annual funding received by the AFPC programme in 2015/2016 were also sourced from AFPC staff. A basic return on investment calculation was produced by subtracting the annual programme funding from the annual programme savings, and then dividing it by the annual programme funding. The final report of the second evaluation, with updated programme recommendations, was presented to AFPC staff and the Aboriginal Family Worker Support Group in July 2017.

The evaluation frameworks for both AFPC evaluations were developed collaboratively between non-Aboriginal WESTIR researchers and Aboriginal-identifying AFPC staff. The collaboration with AFPC staff, who were often both service workers and members of the local Aboriginal community, helped ensure culturally responsive methods and practices were utilised even when the evaluators were of non-Aboriginal origin. Both evaluations received ethics clearance through WESTIR’s internal approval procedures.

Evaluation results

Evaluation recommendations

The recommendations from both AFPC evaluations were largely similar and included retaining the AFPC Coordinator position, increasing programme funding, training more AFPC facilitators, encouraging more community presence, developing referral pathways and post-programme support, establishing a robust data collection system and developing clearer communication around programme requirements. The recommendations have been valuable in driving positive changes in the AFPC programme. This includes the establishment of pregnancy family meetings with Western Sydney Area Health Service and DCJ representatives to prevent an unborn child entering the AFPC programme or from being removed at birth. The AFPC programme has also connected with local men’s groups to better address issues such as domestic violence and has transitioned to an online data collection system for faster and more informed programme reporting. The implementation of recommendations has largely occurred after the AFPC programme moved to Marrin Weejali, with the Aboriginal organisational setting enabling implementation through factors such as strong Aboriginal community control and leadership, closer and more efficient service delivery to the community and better defined relationships with government agencies (Morley, 2015).

Lessons from the AFPC evaluation

A number of lessons were learned from the AFPC evaluations. The use of culturally responsive frameworks for both evaluations highlighted the importance of incorporating Aboriginal culture in all evaluation stages as a means of redistributing decision-making power back to these historically marginalised communities (Hood et al., 2015). The AFPC evaluation frameworks aimed to be culturally responsive in practice by honouring strengths-based principles such as self-determination and integrating Aboriginal voices, knowledge and experiences (Rogers et al., 2018). The partnership between non-Aboriginal WESTIR researchers and Aboriginal-identifying AFPC staff in co-designing and implementing the evaluation frameworks was a practical way in being culturally responsive. The adoption of this ‘two-way approach’, where Aboriginal practices and knowledge was combined with non-Aboriginal ‘ways of doing’, has been known to balance decision-maker needs with people’s voice, lessen bias and misinterpretation and make research safer for Aboriginal participants (Rossingh & Yunupingu, 2016). Due to the pro bono nature of the AFPC evaluations, there were not enough resources to ensure the evaluations were wholly administered by the local Aboriginal community. This suggests that more funding is needed to ensure community input into local Aboriginal programme evaluations, including the training and engagement of Aboriginal evaluators (Williams, 2018).

The concept of ‘outsider’ evaluators has been widely discussed in the literature, with many Aboriginal communities being exploited by short-term, non-Aboriginal evaluators who arrive with a predetermined agenda, extract data with little regard for Indigenous culture and then leave without implementing recommendations in the community (Muir & Dean, 2017; Scougall, 2006). The non-Aboriginal evaluators of the AFPC programme were mindful of their positions as short-term ‘outsiders’ and sought strategies to mitigate power imbalances, including attending cultural awareness training before the evaluations. During the evaluations, it was also imperative to have AFPC staff broker relationships between the non-Aboriginal evaluators and Aboriginal participants. The assistance of programme staff, often called local ‘sponsors’, increased trust and communication for all involved, provided guidance around cultural norms and practices and ensured that Aboriginal participants had ownership over the process (Price et al., 2012). In addition, most interviews were conducted at an Aboriginal service so that Aboriginal participants could participate in a culturally safe space.

The evaluations also showed the importance of communicating evaluation findings to relevant stakeholders. The presentation of evaluation results to AFPC and Aboriginal Family Worker Support Group stakeholders reinforced the notion of evaluators ‘reporting back’ to solidify a sense of community ownership over the process and show a genuine desire for the work to be used in informing community outcomes (Sutton et al., 2016). The evaluation reports were key in advocating for continued government funding, with evidence-based recommendations helping to justify the ongoing need for the AFPC Coordinator position. The overall experience of AFPC evaluators echoes the ongoing politicisation of Aboriginal funding decisions and the value of evidence-based evaluation in challenging the dominance of government interests in this space (McCausland, 2019).

The evaluations illustrated methodological issues that continue to emerge in Aboriginal evaluations. The AFPC evaluations illustrated the strength of the mixed-methods approach, with qualitative data providing depth and context to the programme’s story and quantitative data providing objective indicators against which change can be measured. The second evaluation highlighted the ongoing direction of government agencies to demonstrate the economic value of Aboriginal programmes. This aspect of the evaluation proved difficult for the AFPC evaluators as economic evaluation methods are specialised skills. The experience prompted the evaluators to pursue further training in value-based evaluation methods, such as Social Return on Investment, to ensure these skills are available for future evaluations. The second evaluation also emphasised the need to establish robust data collection systems to provide a solid evidence base and justify the continued funding of the AFPC programme. Research generally supports robust data collection for Aboriginal programmes and services, but it must acknowledge that one size does not fit all (Markiewicz, 2012), embrace Aboriginal knowledge (Hurworth & Harvey, 2012) and allow Aboriginal control so it is responsive to local needs (Muir & Dean, 2017).

As a whole, the evaluation findings highlight the ongoing need to recognise Aboriginal-led programme models and to adequately resource Aboriginal programme evaluation in the community services sector. In NSW, current state government reforms prioritise Aboriginal children and families in service delivery and evaluation rhetoric, and government agencies such as DCJ have adopted international case management models such as Family Group Conferencing when working with Aboriginal families (FACS, 2019; NSW Government, 2018a, 2018b). The Family Group Conferencing model, originally from New Zealand, is a three-stage process of information sharing, family time and action plan creation, which shares similar features to the AFPC programme. The AFPC evaluations do, however, reveal the limitations of the widely used model, including the prioritisation of government requirements within the co-design process, the inadequate resourcing for case management and ongoing evaluation, and the limited evidence of being culturally appropriate in the Australian context (Connolly, 2004). The model’s limitations reiterate the need for greater political recognition and funding of community-initiated Aboriginal models like AFPC programme so the needs of Aboriginal families are effectively met within their own communities.

Conclusion

The AFPC evaluations demonstrate the value of Aboriginal-led early intervention programme models in addressing the complex needs of participating families and preventing Aboriginal over-representation in the child protection system. This case study shows that culturally responsive evaluations can provide an avenue for Aboriginal communities to advocate for the continued funding of their programmes, particularly in an ever-changing, resource-constrained political environment. Despite the progress in Aboriginal evaluation practice, the AFPC evaluations highlight the ongoing challenges experienced by evaluators, including a lack of dedicated funding, the reliance on non-Aboriginal ‘outsider’ evaluators, using evidence to challenge political agendas, and the expectation to demonstrate the economic value of community-based programmes. These challenges strengthen the call for governments to adequately resource Aboriginal programme evaluation in the community services sector so that Aboriginal communities can evaluate and future-proof their programmes now and in the future.