Abstract
The Aboriginal Family Planning Circle (AFPC) programme is an Aboriginal-led community programme, which works with Aboriginal families in Western Sydney to address their complex needs and reduce the risk of having their children put into out-of-home care (OOHC). This article explores two external evaluations undertaken by WESTIR Limited (WESTIR), a non-Aboriginal research service, on the AFPC programme. The purpose of the first evaluation was to provide an assessment of how effective the AFPC programme had been for participants and identify programme aspects that could be improved or developed. The second evaluation was undertaken to examine whether the AFPC programme had continued to meet its objectives and address some knowledge gaps, particularly the estimated savings and return on investment that the programme created for the OOHC system in New South Wales (NSW). The evaluations used qualitative and quantitative techniques, including interviews, focus groups and a return on investment analysis. This article outlines methods, results and recommendations from both evaluations, along with lessons learned to better inform evaluation practice. This case study shows that culturally responsive evaluations can provide an avenue for Aboriginal communities to advocate for the continued funding of their programmes. It also emphasises the need to adequately resource Aboriginal programme evaluations in the community services sector now and in the future.
Introduction
The impacts of colonisation continue to have a profound impact on Aboriginal families and communities. The removal of Aboriginal children from their families, collectively known as the ‘Stolen Generations’, has deprived many of the family ties and cultural identity, and contributed to a deep mistrust of government institutions. The unresolved trauma of the Stolen Generations continues to be passed onto one generation after another, with one consequence being that Aboriginal children continue to be over-represented in the child protection system (Blignault & Williams, 2017).
In 2017/2018, it was estimated that 38.9% of the 17,387 children in out-of-home care (OOHC) in New South Wales (NSW) were of Aboriginal origin. Aboriginal children (71.2 per 1,000) in NSW were over 10 times more likely to be in OOHC than their non-Aboriginal counterparts (6.4 per 1,000), with this trend expected to increase over time (Productivity Commission, 2019). The independent Tune Review in 2015 acknowledged that the OOHC system is ineffective and is ‘failing to improve long term outcomes for children and arrest the devastating cycles of intergenerational abuse . . . particularly for Aboriginal children, young people and families’ (NSW Government, 2016, p. 3). The Review also found that around $300 million (AUD) per year was being spent on family support programmes without evidence that they work (McFarlane, 2016). Peak Aboriginal organisations have continued to advocate for early intervention family support service models which keep Aboriginal children with immediate families or in kinship arrangements (AbSec, 2019), but most of these Aboriginal-led initiatives are also not evaluated (Williams, 2018).
This article explores two external evaluations undertaken by WESTIR Limited (WESTIR), a non-Aboriginal community-based research service in Western Sydney, on an Aboriginal-led family support programme called the Aboriginal Family Planning Circle (AFPC). Commissioned by AFPC staff, the purpose of the first evaluation in 2015 was to provide a comprehensive assessment of how effective the AFPC programme had been for participants and identify programme aspects that could be improved or developed. WESTIR was again approached by AFPC staff in 2017 to undertake a second evaluation in 2017, with the purpose of examining whether the AFPC programme had continued to meet its objectives and address some knowledge gaps identified by the funding body, particularly the estimated savings and return on investment that the programme created for the OOHC system in NSW. This article describes the AFPC model and the evaluation process, as well as the results and recommendations from both evaluations. It will describe the lessons learned from the evaluations and emphasise the need to adequately resource Aboriginal programme evaluations in the community services sector now and in the future.
History of the AFPC
The AFPC programme was established in 2006 by the Aboriginal Family Workers Support Group working in the Blacktown Local Government Area (LGA). The AFPC programme works with Aboriginal families in Western Sydney to address their complex issues and reduce the risk of having their children put into OOHC. The issues addressed by the programme include drug and alcohol addiction, housing and financial instability, domestic violence and intergenerational trauma.
The main features of the AFPC programme include referral intake and assessment, initial engagement, service delivery, exit planning and training and measurement. A description of each stage is shown in Figure 1. The AFPC programme differs from other Aboriginal family support models as it emphasises a person-centred approach, localised governance and the involvement of the community in co-designing goals and solutions to family issues and needs. The programme also works with Aboriginal families across the child protection continuum, from early intervention to prevent child removal to working with families to secure child restoration.

Aboriginal Family Planning Circle programme model.
The AFPC programme was initially a volunteer-run, community-based initiative, but funding was sought from the then NSW Department of Family and Community Services (FACS; now NSW Department of Communities and Justice [DCJ]) when it outgrew its original capacity. From 2015 to 2017, the funding was used to appoint an AFPC Coordinator at the FACS Aboriginal Intensive Family Based Service office in Rooty Hill and expand service delivery to both Blacktown and Penrith LGAs. In 2018, with the support of the Aboriginal Family Worker Support Group, the AFPC programme was moved to Marrin Weejali Aboriginal Corporation, an Aboriginal-registered non-government organisation in Blacktown LGA. Despite no additional government funding, the AFPC programme continues to support more families than outlined in their funding agreement.
Evaluation methods
WESTIR was approached by AFPC staff to undertake an initial evaluation of the AFPC programme in 2015. The purpose of the first evaluation was to provide a comprehensive assessment of how effective the AFPC programme had been for participants and identify programme aspects that could be improved or developed.
A cross-sectional, qualitative framework was adopted to collect data for the first evaluation during August and September 2015. This approach was chosen because it allowed participants to represent their experiences in their own words. Semi-structured 1 hr interviews were conducted by WESTIR researchers with seven clients, mostly parents and kinship carers of Aboriginal origin (n = 7). Aboriginal and non-Aboriginal service providers involved with the AFPC programme were also invited to participate through semi-structured interviews, focus groups and written responses (n = 15). All interviewees were asked about aspects of the AFPC programme, including initial involvement, general impressions, programme strengths and outcomes, and areas for improvement. Interviewees were selected by AFPC staff through convenience sampling and included those who had past and current involvement with the AFPC programme. All interviews and focus groups were recorded, transcribed and thematically coded, with data analysis used to formulate evaluation recommendations. The final report of the first evaluation was provided to AFPC staff and the Aboriginal Family Support Worker Group in January 2016.
Following a review of the first evaluation report, DCJ senior management requested evidence on the economic value of the AFPC programme. As a result, WESTIR was again approached by AFPC staff to undertake a second AFPC evaluation between March and June 2017. The purpose of a second evaluation was to examine whether the AFPC programme had continued to meet its objectives and address some knowledge gaps identified by the funding body, particularly the estimated savings and return on investment that the programme created for the OOHC system in NSW.
The second evaluation adopted a mixed-methods framework, which was undertaken between March and July 2017. The qualitative method from the first evaluation was adopted to undertake and analyse additional semi-structured interviews with five service workers (including the AFPC Coordinator; n = 5) and six former and current clients (n = 6). Analysis of programme records and service provider entry surveys were also undertaken but are not discussed in this article as links between programme outcomes and preventing children going into OOHC could not be established at that time.
The second evaluation also included a return on investment analysis. The savings of the AFPC programme were calculated by multiplying the number of children who were restored or prevented from going into OOHC due to the AFPC programme in 2015/2016 with the Productivity Commission’s (2017) average cost of keeping a child in OOHC in NSW in 2015/2016. Figures on the annual funding received by the AFPC programme in 2015/2016 were also sourced from AFPC staff. A basic return on investment calculation was produced by subtracting the annual programme funding from the annual programme savings, and then dividing it by the annual programme funding. The final report of the second evaluation, with updated programme recommendations, was presented to AFPC staff and the Aboriginal Family Worker Support Group in July 2017.
The evaluation frameworks for both AFPC evaluations were developed collaboratively between non-Aboriginal WESTIR researchers and Aboriginal-identifying AFPC staff. The collaboration with AFPC staff, who were often both service workers and members of the local Aboriginal community, helped ensure culturally responsive methods and practices were utilised even when the evaluators were of non-Aboriginal origin. Both evaluations received ethics clearance through WESTIR’s internal approval procedures.
Evaluation results
First evaluation
Analysis of seven client interviews found that there were many positive experiences with the AFPC programme. All interviewed clients (n = 7) believed that as an Aboriginal-led programme, the AFPC programme gave them the motivation to seek help and empowered them to change their lives with the support of service providers and the local community. Over half of the interviewed clients (n = 4) stated that the AFPC had created a culturally safe space to develop more balanced relationships with government services. The interviewed clients identified a range of tangible outcomes as a result of participating in the AFPC programme, including securing housing (n = 3), addressing at-risk behaviours (n = 2) and retaining or resuming custody of their children (n = 5). The AFPC programme gave many of the interviewed clients the confidence to maintain the positive changes over time (n = 5) and at least one client (n = 1) felt the programme gave them hope for the future. Despite these positive perceptions, just over half of the interviewed clients (n = 4) suggested the programme could be improved by creating more post-programme support such as follow-ups, support groups or options to re-enter the programme when needed.
A total of 15 service workers also spoke about the strengths of the AFPC programme. Over half of interviewed service workers (n = 8) believed that the programme was effective because it implemented culturally appropriate practices and facilitated self-determination. Just over a quarter of interviewed service workers (n = 4) believed that the programme’s client-centred approach was crucial to achieving client outcomes. Most service providers (n = 10) also spoke about aspects of the programme that benefitted their own professional practice, including more effective communication and support between Aboriginal and non-Aboriginal services. While many service providers saw tangible outcomes for their clients (n = 8), some acknowledged that the programme’s effectiveness was affected by lack of funding (n = 2), insufficient number of facilitators (n = 2), the need for more early intervention referrals (n = 2) and conflicting expectations and time frames between stakeholders (n = 3).
Second evaluation
Analysis of six client interviews in the second evaluation continued to clarify aspects of the AFPC programme that were working and those that were not. All interviewed clients (n = 6) believed the AFPC programme had empowered them make the changes necessary to provide a safe home for their children. At least half of the interviewed clients (n = 3) spoke about the positive impact that the AFPC Coordinator had provided in terms of support, but the same number (n = 3) continued to speak about the need to provide ongoing support after they had finished with the programme.
Interviews with five service workers further clarified the benefits of the AFPC model, with most (n = 4) still believing the strengths-based, culturally appropriate approach was empowering clients to take control over their lives and reduce the mistrust that often occurred with government agencies. Most service workers (n = 4) spoke about how the AFPC model brought service providers together to address client needs and keep all parties accountable, but a couple of interviewees (n = 2) noted that the strength of the multi-stakeholder collaboration was diminished when service providers pulled out of meetings or the programme altogether. Service workers also spoke openly about programme challenges, such as meeting mandatory child protection requirements in a voluntary process (n = 3) and dealing with unacceptable client behaviour at AFPC meetings (n = 2). Most interviewees (n = 3) could see that the AFPC Coordinator had provided structure to the programme as it expanded but were concerned that uncertainty around the position’s ongoing funding would have a continual impact on service delivery.
The return on investment analysis showed that two children were restored and five children remained at home due to AFPC programme participation in 2015/2016. Based on the average annual cost of keeping a child in OOHC in NSW for that financial year (Aus $57,800 per child), it was estimated that the AFPC programme saved the NSW Government approximately Aus $404,600 in 2015/2016. The only funding received for the AFPC programme during that financial year was the salary of the AFPC Coordinator (Aus $95,039 per annum), equating to approximately a 326% return on investment for the NSW Government in 2015/2016.
The evaluation acknowledged that there were limitations to the estimated savings and return on investment calculations, providing only an overall indication of the programme’s economic impact. The average cost per child in OOHC was overstated because more children were in care during the year than at that point in time. However, the estimated savings calculated most likely underestimated the total time a child spends in OOHC, other cost of living factors and savings from preventive and restorative activities. Similarly, return on investment calculations did not consider the in-kind contribution of service providers or other positive participant outcomes such as repayment of social security debt. The evaluation recommended that based on these limitations, other value-based methods be explored in the future to provide a more accurate picture of the financial returns created by the AFPC programme.
Evaluation recommendations
The recommendations from both AFPC evaluations were largely similar and included retaining the AFPC Coordinator position, increasing programme funding, training more AFPC facilitators, encouraging more community presence, developing referral pathways and post-programme support, establishing a robust data collection system and developing clearer communication around programme requirements. The recommendations have been valuable in driving positive changes in the AFPC programme. This includes the establishment of pregnancy family meetings with Western Sydney Area Health Service and DCJ representatives to prevent an unborn child entering the AFPC programme or from being removed at birth. The AFPC programme has also connected with local men’s groups to better address issues such as domestic violence and has transitioned to an online data collection system for faster and more informed programme reporting. The implementation of recommendations has largely occurred after the AFPC programme moved to Marrin Weejali, with the Aboriginal organisational setting enabling implementation through factors such as strong Aboriginal community control and leadership, closer and more efficient service delivery to the community and better defined relationships with government agencies (Morley, 2015).
Lessons from the AFPC evaluation
A number of lessons were learned from the AFPC evaluations. The use of culturally responsive frameworks for both evaluations highlighted the importance of incorporating Aboriginal culture in all evaluation stages as a means of redistributing decision-making power back to these historically marginalised communities (Hood et al., 2015). The AFPC evaluation frameworks aimed to be culturally responsive in practice by honouring strengths-based principles such as self-determination and integrating Aboriginal voices, knowledge and experiences (Rogers et al., 2018). The partnership between non-Aboriginal WESTIR researchers and Aboriginal-identifying AFPC staff in co-designing and implementing the evaluation frameworks was a practical way in being culturally responsive. The adoption of this ‘two-way approach’, where Aboriginal practices and knowledge was combined with non-Aboriginal ‘ways of doing’, has been known to balance decision-maker needs with people’s voice, lessen bias and misinterpretation and make research safer for Aboriginal participants (Rossingh & Yunupingu, 2016). Due to the pro bono nature of the AFPC evaluations, there were not enough resources to ensure the evaluations were wholly administered by the local Aboriginal community. This suggests that more funding is needed to ensure community input into local Aboriginal programme evaluations, including the training and engagement of Aboriginal evaluators (Williams, 2018).
The concept of ‘outsider’ evaluators has been widely discussed in the literature, with many Aboriginal communities being exploited by short-term, non-Aboriginal evaluators who arrive with a predetermined agenda, extract data with little regard for Indigenous culture and then leave without implementing recommendations in the community (Muir & Dean, 2017; Scougall, 2006). The non-Aboriginal evaluators of the AFPC programme were mindful of their positions as short-term ‘outsiders’ and sought strategies to mitigate power imbalances, including attending cultural awareness training before the evaluations. During the evaluations, it was also imperative to have AFPC staff broker relationships between the non-Aboriginal evaluators and Aboriginal participants. The assistance of programme staff, often called local ‘sponsors’, increased trust and communication for all involved, provided guidance around cultural norms and practices and ensured that Aboriginal participants had ownership over the process (Price et al., 2012). In addition, most interviews were conducted at an Aboriginal service so that Aboriginal participants could participate in a culturally safe space.
The evaluations also showed the importance of communicating evaluation findings to relevant stakeholders. The presentation of evaluation results to AFPC and Aboriginal Family Worker Support Group stakeholders reinforced the notion of evaluators ‘reporting back’ to solidify a sense of community ownership over the process and show a genuine desire for the work to be used in informing community outcomes (Sutton et al., 2016). The evaluation reports were key in advocating for continued government funding, with evidence-based recommendations helping to justify the ongoing need for the AFPC Coordinator position. The overall experience of AFPC evaluators echoes the ongoing politicisation of Aboriginal funding decisions and the value of evidence-based evaluation in challenging the dominance of government interests in this space (McCausland, 2019).
The evaluations illustrated methodological issues that continue to emerge in Aboriginal evaluations. The AFPC evaluations illustrated the strength of the mixed-methods approach, with qualitative data providing depth and context to the programme’s story and quantitative data providing objective indicators against which change can be measured. The second evaluation highlighted the ongoing direction of government agencies to demonstrate the economic value of Aboriginal programmes. This aspect of the evaluation proved difficult for the AFPC evaluators as economic evaluation methods are specialised skills. The experience prompted the evaluators to pursue further training in value-based evaluation methods, such as Social Return on Investment, to ensure these skills are available for future evaluations. The second evaluation also emphasised the need to establish robust data collection systems to provide a solid evidence base and justify the continued funding of the AFPC programme. Research generally supports robust data collection for Aboriginal programmes and services, but it must acknowledge that one size does not fit all (Markiewicz, 2012), embrace Aboriginal knowledge (Hurworth & Harvey, 2012) and allow Aboriginal control so it is responsive to local needs (Muir & Dean, 2017).
As a whole, the evaluation findings highlight the ongoing need to recognise Aboriginal-led programme models and to adequately resource Aboriginal programme evaluation in the community services sector. In NSW, current state government reforms prioritise Aboriginal children and families in service delivery and evaluation rhetoric, and government agencies such as DCJ have adopted international case management models such as Family Group Conferencing when working with Aboriginal families (FACS, 2019; NSW Government, 2018a, 2018b). The Family Group Conferencing model, originally from New Zealand, is a three-stage process of information sharing, family time and action plan creation, which shares similar features to the AFPC programme. The AFPC evaluations do, however, reveal the limitations of the widely used model, including the prioritisation of government requirements within the co-design process, the inadequate resourcing for case management and ongoing evaluation, and the limited evidence of being culturally appropriate in the Australian context (Connolly, 2004). The model’s limitations reiterate the need for greater political recognition and funding of community-initiated Aboriginal models like AFPC programme so the needs of Aboriginal families are effectively met within their own communities.
Conclusion
The AFPC evaluations demonstrate the value of Aboriginal-led early intervention programme models in addressing the complex needs of participating families and preventing Aboriginal over-representation in the child protection system. This case study shows that culturally responsive evaluations can provide an avenue for Aboriginal communities to advocate for the continued funding of their programmes, particularly in an ever-changing, resource-constrained political environment. Despite the progress in Aboriginal evaluation practice, the AFPC evaluations highlight the ongoing challenges experienced by evaluators, including a lack of dedicated funding, the reliance on non-Aboriginal ‘outsider’ evaluators, using evidence to challenge political agendas, and the expectation to demonstrate the economic value of community-based programmes. These challenges strengthen the call for governments to adequately resource Aboriginal programme evaluation in the community services sector so that Aboriginal communities can evaluate and future-proof their programmes now and in the future.
Footnotes
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The Aboriginal Family Planning Circle (AFPC) programme receives funding from the NSW Department of Communities and Justice (DCJ) and WESTIR Limited is partly funded by DCJ. The views and opinions expressed in this article are those of the authors and do not reflect the views of DCJ.
Data Accessibility Statement
Full reports of the Aboriginal Family Planning Circle (AFPC) evaluations can be requested from Marrin Weejali Aboriginal Corporation. All data collected for the AFPC evaluations are owned by the AFPC programme.
