A Study of Evaluation Influence in Two Child Protection Programs

Evaluation Influence

Early studies of evaluation impact tended to focus on whether findings were used immediately and directly in decisions about a program (Alkin et al., 1979). By this measure, evaluations were found to be rarely used (Alkin et al., 1974; Caplan et al., 1975), although decision-makers reported the information from evaluations to be valuable (e.g., Weiss & Bucuvalas, 1980). This observation resulted in the identification of several types of evaluation use (Johnson, 1998), the number of which has grown over time to keep up with developments in evaluation theory and practice (Patton, 2008).

Critical of conceptual frameworks for evaluation use that were “simultaneously impoverished and overgrown” (Mark & Henry, 2004, p. 57) evaluation researchers proposed evaluation influence as an alternative. Evaluation influence draws on existing change mechanisms from social psychology to describe the full impact of an evaluation (Mark, 2011), with an aim to organize a systematic body of evidence drawing on existing change mechanisms to inform evaluation practice (Mark, 2008). Responding to criticisms of previous typologies of use that lacked a moral or ethical purpose, Mark and Henry (2012) characterize evaluation influence as including all effects that lead towards or away from social betterment.

Mark and Henry’s (2004) evaluation influence framework (see Table 1) provides a set of definitions for 28 mechanisms across individual, interpersonal, and collective levels, and across types of change (i.e., general, attitudinal, motivational, and behavioral). Each mechanism brings with it an existing body of literature from social psychology (Mark, 2011; Mark & Henry, 2004). The authors suggest that these mechanisms interact and accrete, pointing researchers toward the study of the connection between mechanisms of change, for instance, from the attitudinal to the behavioral, and from the individual level to the collective level. This set of definitions and the metaphor of interconnected pathways of influence was intended as the foundation for the evidence-based study of evaluation influence, paralleling the study of research use (e.g., Cherney et al., 2015; Holzer et al., 2007).

OPEN IN VIEWER

Table 1.

Definitions of Mechanisms in the Evaluation Influence Framework (Adapted From Mark & Henry, 2004, p. 41).

Level of Analysis	Mechanisms	Description
Individual mechanisms
General	Elaboration	Thinking about an issue in response to evaluation information
	Heuristics	Formation or change of attitudes without thinking comprehensively about information, or reliance on quick answers from experiential learning and intuition
	Priming	Makes an idea or concept central to the judgment of an issue
	Skill acquisition	Development of skills from participation in an evaluation
Cognitive/affective	Salience	Increasing the importance of an issue to an individual; A priority shift
	Opinion/attitude valence	Change in the attitudes about a program or the underlying assumptions of the program through evaluation
Motivational	Personal goals and aspirations	Change to the goals a person aims to achieve in their work related to evaluation
Interpersonal mechanisms
General	Justification	Evaluation findings used to support an existing position in attempting to influence others
	Persuasion	Drawing on information from the evaluation to change the attitudes of others
	Change agent	A person who actively pursues change because of some change in awareness from the evaluation
	Minority-opinion influence	Individuals with a minority opinion bring about change through evaluation information
Cognitive/affective	Local descriptive norms	Change to the attitudes and values at the local level
Motivational	Injunctive norms	Change to the shared standards of appropriate behaviors influenced by the evaluation
	Social reward	Social inducement to adopt a behavior related to the evaluation
	Exchange	Actual inducement to adopt a behavior
Behavioral	Collaborative change in practice	Change in practice as part of an interaction or persuasive process
Collective mechanisms
General	Ritualism	Outwards conformity to norms without the inner commitment to the requisite social values and roles
	Legislative hearings	Evaluation is drawn on as part of the discussion and proposal of legislation
	Coalition forming	The formation and alignment of groups sharing a position on an issue related to the evaluation
	Drafting legislation	Evaluation is drawn on in the drafting of legislation
	Standard setting	Evaluation results in the setting or changing of standards
	Policy consideration	Evaluation enters consideration in the policy-making process
Cognitive/affective	Agenda setting	An issue emerging from an evaluation shows up on a meeting agenda
	Policy-oriented learning	Evaluation leads to an increased understanding at the policy-making level of the organization
Motivational	Structural incentives	Incentives are devised and implemented that affect motivations related to some issue from the evaluation
	Market forces	Incentives related to the functioning of markets change motivations related to some issue from the evaluation
Behavioral	Program continuation, cessation or change	Evaluation has an impact on decisions to continue, shut-down, expand or alter some aspect of the program
	Policy change	Like policy-oriented learning but resulting in policy change
	Diffusion	Change resulting from other evaluations

Herbert’s (2014) review of the evaluation influence research found a developing body of literature examining how evaluations had influenced change in social programs. Overwhelmingly the review found retrospective case studies that applied the evaluation influence framework to make some inferences about why influence occurred as it did (e.g., Fjellstrom, 2007; Gildemyn, 2014; Oliver, 2008). Herbert (2014) observed that many studies lacked the detail to be replicable, that most studies involved evaluation influence being researched by the same people that carried out the evaluation under study (e.g., Diaz-Puente et al., 2009; Diaz-Puente et al., 2008; Weiss et al., 2005), and that studies primarily relied on self-report from either organizational respondents (e.g., Fjellstrom, 2007) or official records (Gildemyn, 2014; Oliver, 2008).

An additional critique of the evaluation influence literature is the lack of attention to including practice level respondents to report on if an evaluation changed the way practitioners think about or undertake their work with clients. Despite often perceiving program evaluation as not engaging with questions relevant to practice (Herbert, 2015), practitioners are critical end-users of evaluation, with the capacity to resist or support the implementation of evaluation recommendations (Houlbrook, 2011).

Evaluation influence is part of a long history of research into evaluation use, positioned around mostly descriptive theories of use (King & Alkin, 2018). Høydal (2018) argues for a theoretical approach that straddles the pragmatic outlook of theories of influence (Mark & Henry, 2004), and a more sociological view of evaluation as cultural expression referred to as The Evaluation Society (Dahler-Larsen, 2012). Theories of influence present frameworks that describe the logical effects of evaluation, framed around aspirations of evaluation contributing to a better society (Henry, 2003). The Evaluation Society (Dahler-Larsen, 2012) characterizes evaluation as a ritual of modernity, which drives social change through constitutive effects—observing that the world changes as a result of being measured. Høydal (2018) points to the compatibility between these theoretical positions across interest in the effects of evaluation on world-views and social identities, and interest in the effects of evaluation toward social betterment in the context of a social program. Examining the observed effects of an evaluation provides an opportunity to examine the second-order or constitutive effects of evaluation.

Relative to other areas of social policy (e.g., international development and housing), the child protection field is relatively new and lags other pediatric fields in terms of research evidence and the translation of research and evaluation findings (Lewig et al., 2006). Child protection practitioners may have less experience with and fewer preconceptions about program evaluation, or perceive evaluation as something that is more relevant to service funders (Herbert, 2015). A view of how evaluation influence plays out in this context presents as uniquely nascent—allowing for observation of second-order effects of evaluation where participants have limited impressions of the value or relevance of evaluation.

This research aimed to identify mechanisms of evaluation influence in two child protection programs to examine the extent to which evaluation findings are translated into reported changes at the individual, interpersonal, and collective levels (Mark & Henry, 2004), including reported changes that resulted from the experience of being evaluated. Addressing Herbert’s (2014) criticisms of existing studies, this study set out to: (a) include a replicable method; (b) study influence in programs the researcher was not involved in evaluating; (c) combine both organizational respondents and documents; and (d) include both service managers and practitioners as organizational respondents. The study will address the following research question:

• What mechanisms of evaluation influence can be observed following the evaluation of a child protection program?

By examining the relationship between how evaluations were undertaken and the reported influence across two case studies this study will add to the empirical literature on how evaluation influence plays out in different program contexts. The study will also draw on these patterns of influence in context to consider how evaluation practice could have differed to prompt more meaningful practice level change in programs.

Organization/Program Selection

To identify suitable programs for the study, a sample of child protection programs that had been evaluated in the past 2 years was created using email responses from five large non-government organizations in a large Australian city. Information related to these evaluations were examined using selection criteria: (a) programs with an intended outcome of preventing child abuse and neglect; (b) ongoing programs with the capacity to foster practice improvement; (c) programs evaluated externally; and (d) that the evaluation had a stated goal of fostering internal learning and improvement within the program. The two programs were selected based on fulfilling these criteria and the willingness of organizations to support the research. From herein, these are referred to as Case Study A, and Case Study B. As the research focused on changes to the agencies delivering services, the funding agency and evaluators were not included in the research.

Data Collection

Within the two selected programs, information about the influence of the evaluations was obtained through a combination of interviews with key informants and organizational documents.

As discussed above, this study had a deliberate strategy to recruit informants with knowledge about changes in the program at both a practice (e.g., social workers, psychologists, and practice managers) and at a policy level (e.g., area managers and research and policy managers). Participation involved a semi-structured interview that covered:

1. Role and background of the person interviewed;

Organizational documents were sought to complement the perspectives of the interview respondents. This included evaluation reports, slides from presentations about the evaluation, and key actions to be taken by agencies, changed practice guidelines, and response documents.

Procedure

Following ethics approval, the researcher obtained permission from each of the agencies to undertake the study and secured cooperation from the manager responsible for the evaluation within each of the agencies. These evaluation managers identified and contacted practice managers and practitioners within their agencies that had knowledge of the evaluation and its influence over time. The evaluation managers made initial contact with respondents, provided an outline of the study supplied by the researcher, and secured permission for the researcher to contact respondents. From this point, the evaluation managers had no knowledge about the participation of workers or any of the information provided by workers as participant details were made anonymous in the final report and subsequent publications. The evaluation managers also provided the researcher any documents that pertained to the evaluation, or the effects of the evaluation.

Once permission was granted for individuals to be contacted, the researcher contacted each of the potential respondents by email to invite them to participate in the study, either at their office or by telephone. Participants included the managers of program sites, team leaders (practice supervisors), and direct human service practitioners, although these categories often overlapped.

Across the two case studies, all the potential respondents contacted by the managers participated in the research, except two in Case Study A who had since left the agency. Case Study A resulted in thirteen interviews with seven managers and six practitioners. Case Study B resulted in seven interviews with one manager and five practitioners. The length of interviews varied from 15 minutes to over an hour, with an average length of 43 minutes. All interviews were recorded with the permission of the participants, and then sent to a professional transcription service. Transcripts of the interviews were emailed to the participants prior to publication, allowing them to review and potentially veto parts of the interview.

The interviews were complemented by the collection of documents related to the evaluation. In Case Study A, there were 12 documents, which included the evaluation reports, editions of the program casework manual from before and after the evaluation, service provision guidelines, organizational templates, policies and procedures, and presentations by the agency about the evaluation. In Case Study B, two documents were included in the analysis, the evaluation report and an internal agency report about responding to the findings of the evaluation.

The researcher considered several important ethical issues in the planning of the study, with several challenges related to undertaking research in organizational settings. As initial contact with participants was made by managers in the organizations, it was important to maintain anonymity, so that participants could not receive any risk or benefit related to participating. As participation involved individuals potentially criticizing their own organization or the government funder of their organization, participants were given the opportunity to obtain a copy of their transcript and omit or revise any information.

Analysis of Data

The interviews and the accompanying documents were coded to identify all instances of evaluation influence. Using NVIVO software, all places in the interview transcripts and in the documents that suggested the evaluation may have changed something were coded (Van Manen, 1990). These were roughly categorized into events, so that multiple accounts of the same events across the transcripts and documents could be compared and cross-checked. Using the definitions described in Table 1, adapted from Mark and Henry (2004), these events were coded as evaluation influence mechanisms, firstly into different groups of mechanisms (individual, interpersonal, and organization), and then into more specific mechanisms (e.g., elaboration and agenda setting). The analysis allowed for the possibility of identifying mechanisms that did not fit into the existing framework; however, all the identified impacts fit into the existing definitions. The analysis document was then used to write up the case studies describing the influence of the evaluation in these two programs—acknowledging that the participant perspectives and the organizational documents are not a perfect record of evaluation influence. Observations of how evaluation practices and the evaluation context resulted in influence events are inevitably shaped by the data collection procedures.

Case Study A

Case Study A was an evaluation of a large early intervention child protection program funded by a state government and delivered by a mix of non-government and state government service branches. The program involved the combination of service elements (e.g., case management, child care, home visiting, and parenting programs) to respond to children with exposure to domestic violence, parental drug and alcohol abuse, parental mental health issues, and other risk factors for child abuse and neglect.

The case study included thirteen interviews conducted over the course of a year, complemented by a set of organizational documents including evaluation tenders and reports. Participants included the program management team (n = 7), and six team leaders (practice supervisors) and caseworkers who worked in the program during the evaluation. The participants were based at sites in a metropolitan area (n = 9) and regional areas (n = 4). The final report of the evaluation had been completed around a year before the research began, with some ongoing work within the organization to implement the findings.

The evaluation was conducted across the multiple government and non-government agencies involved in the delivery of the program. The evaluation had a stated aim to assist service providers to identify ways to improve service delivery, although the primary aim of the evaluation was to inform government decision-making about the program. This involved a family survey designed to measure parental capacity and family functioning before and after the intervention, along with a data set of client demographics and output information. As the evaluation was operating across many agencies, responsibility for the dissemination of the findings primarily relied on the research and evaluation teams within the participating agencies. The data provided was aggregated across sites, so this feedback was at a whole of program level. The evaluation had been completed at the time of data collection, with the final report released around a year before this study commenced.

Case Study B is a collection of complementary services around an intensive reunification program for mothers whose children have been removed from their care. The evaluation examined the effect of integrating complementary programs and services at a single site. The service provider commissioned the evaluation to examine the quality and effect of this integration, through formative and summative approaches. The evaluation was ongoing at the time of the research; this case study represents the experience of participants at the mid-point of an evaluation, following the release of an interim report. Interviews were conducted with the casework team (n = 7) based at the program’s single site, and the manager responsible for the contracting of the evaluation within the agency.

Analysis of Evaluation Influence

In Case Study A, the evaluation was associated with considerable collective level influence (see Table 2), with participants linking the evaluation to major changes in the way the program was delivered at a state level. Individually, some practitioners reported an increased receptivity to evaluation, while others reported the opposite due to cynicism about the state government’s influence over the evaluation. Within the agency, the evaluation was linked to efforts to address client disengagement from the program. The evaluation also had enduring impacts in prompting a response to clients that were more high risk than the intended target population, and with changing the suite of services that formed the program. Major changes to the program were primarily driven outside of the agency delivering the program, with the evaluation influencing decision-making at a state government level from the perspective of the participants.

OPEN IN VIEWER

Table 2.

Mechanisms of Evaluation Influence Identified in Case Study A.

Individual
Level of Analysis	Mechanisms	Description
General	Heuristics	• The evaluation confirmed practitioners’ expectations that the program was working;• The evaluation fostered a sense that the program had a future because it had been evaluated
Cognitive/affective	Priming	• Practitioners talked about getting a sense of how their role connected to the rest of the program, a sense of the logic of the program as a whole
	Salience	• Some of the evaluation information highlighted characteristics of clients that practitioners reported being more responsive to and proactive in identifying
	Opinion/attitude valence	• Changed attitudes about evaluation being negative, that it can be done in a way that supports practice and reflection;• Engendered resentment towards the state government child protection agency as practitioners did not think the comparison part of the evaluation properly represented the difference between their work
Interpersonal
General	Change agent	• One of the managers actively engaged in attempting to foster a culture of evaluation among practitioners, to have them be excited and critical about evaluation
Cognitive/affective	Local descriptive norms	• A culture of evaluation and change practice developed around the organization affecting attitudes about the role of evaluation in practice
Motivational	Injunctive norms	• The evaluation affected the delivery of the program to be more standardized, meaning that if particular checklist requirements were not fulfilled a service was not counted as delivered;• Managers were influenced to be aware of and informed by the data in the way they ran their sites, to understand the implications and be responsive to issues;• Practitioners were encouraged to think about and use data in their practice through managers, supervision, and participation in s structured process for actioning evaluation findings
Collective
General	Policy consideration	• The evaluation data was used to support tendering for grants;• Key operational issues were highlighted (i.e. early exit of aboriginal families);
Cognitive/affective	Agenda setting	• Raised awareness of the differences between the context of program sites, which led to discussions at program managers meetings;• There was a standing item for discussing the evaluation in the program mangers meetings
	Policy-oriented learning (internal)	• Discussion of the issue of early exiting identified from the evaluation led to some discussion around if engagement times could be shortened, or if clients could still benefit from a shorter intervention;• The use of case plan goals as a metric in the evaluation led to efforts to refine and improve the writing of case plan goals;• Led to consideration of the impact of having higher level clients than the program specified on practitioners and how to foster resilience;• Recognition of some of the challenges of the rural and regional sites and how service delivery may differ
	Policy-oriented learning (external)	• Evaluation data was included in a submission in response to the Wood report, highlighting problems with the state government side of the program
Behavioral	Program continuation, Cessation or change	• Some of the parenting programs across different sites were dropped or changed;• Reduction in the required time for clients to be engaged in the program from 2 years to 1 year
	Policy change (internal)	• Creation of a research translation position, and the development of organizational evaluation capacity through additional positions in the team;• The social research team within the agency took ownership of implementing the evaluation from the operational side;• Development of an organization evaluation policy formed from the experience of participating in and responding to the evaluation;• Development of a structured process for responding to evaluation findings
	Policy change (external)	• The state government began taking on a more intensive and high-risk caseload because of the evaluation;• Change in the guidelines around the 80/20 split, to a 60/40 split, to eventually becoming uncapped;• Change to the program being 100% lead agency delivered

The evaluation of Case Study B was ongoing at the time of the research; although two interim evaluation reports had been released, culminating in several significant collective level changes. The collection of impacts and the application of evaluation influence mechanisms were much more straightforward than in the previous case, as the evaluation was commissioned internally, the program was based at one site, and involved a relatively small number of staff. While the evaluation reports did include summative results, it had a significant formative component focusing on how the integrated service model operated. Overall, the analysis of evaluation influence (see Table 3) highlighted changes at the individual and collective levels; changes to practitioner perceptions of how the program worked, attitudes about participating in evaluation, and the adoption of the integrated approach across the organization.

OPEN IN VIEWER

Table 3.

Mechanisms of Evaluation Influence Identified in Case Study B.

Individual
Level of Analysis	Mechanisms	Description
General	Heuristics	• The evaluation confirmed practitioners' expectations that the network approach was working and that there were benefits from co-locating;• Provided a sense that the program (and their jobs) was secure and was likely to continue because the program had been evaluated
	Skill acquisition	• Improved the capacity of individual practitioners to work with and reflect on evaluation data in their work
	Salience	• The evaluation caused practitioners to think about the community connector role as being front and center in the network
Cognitive/affective	Opinion/attitude valance	• The positive experience of the evaluation process, and the initial feedback of findings led to an openness to work with evaluation;• Disappointment with not being consulted before the second phase of the evaluation before it was presented resulted in a sense of a betrayal of trust. It reduced their receptiveness to working with evaluation
Collective
General	Policy consideration	• The evaluation data was used to support funding applications for other integrated centers
Cognitive/affective	Agenda setting	• Meetings were occurring to work through the implications of the evaluation for the future expansion of networks;• The evaluation was credited with bringing researchers and practitioners together to collaborate on funding models and ways to improve program delivery;• The developmental measures did not show significant change, which led to a re-thinking of the logic of the program that relied on the connection between parental capacity and children’s development
	Policy-oriented learning	• The integrated service model was in the process of being adopted across the organization The description of the model in the report was used in other sites;• At an organizational level it reinforced that the program was being delivered to the right group;• The evaluation highlighted the advantages of the network approach in creating a sense of belonging and place, and that there was value in being able to move around the network;• Evaluation raised awareness of the local context, allowing the program to be more responsive;• Served as a pilot for a national research grant, tested out some of the developmental measures, and developed the relationship between the agency and the university-based evaluators
Behavioral	Policy change	• The community connector role was intended to be based at the center, but based on the recommendations of the evaluation, the role has continued to work in the community;• Community connector roles have been built into other programs and integrated services;• The integrated service model has been adopted and rolled out across the organization

Following previous analyses of evaluation influence in case studies (e.g., Diaz-Puente et al., 2009; Fjellstrom, 2007), the findings are presented in individual, interpersonal, and collective categories, reflecting the expectation that individual level changes are the antecedents to other forms of change (Henry & Mark, 2003).

Individual Level Change

Across both case studies, the evaluation had the effect of reinforcing perceptions that the program was working well and that service users were experiencing the benefits of the programs. These instances were coded as heuristics, which Fleming (2011) describes as a collection of short-cuts to internalizing new information. For example, a participant said that: “...we know anecdotally and through our own information that it’s having an impact, so to have something official by academics that’s been rigorously assessed… that’s a terrific thing, and that the program will continue and possibly be extended is only good” #6 – Case Study A. In both case studies, the evaluations supported participants’ existing impression that the program was effective, but also provided a sense that the program was likely to be continued over the long term, that “in that shifting economic climate where there are decisions made about what services are successful and what services might be closed… the fact that we had that opportunity to get evaluated and it showed significant benefits…” #19 – Case Study B.

Salience was identified in both case studies; the increased importance of an issue that affects the way a program is understood (Henry & Mark, 2003). In Case Study A, evaluation information on the prevalence of domestic violence among clients made some staff more responsive to social cues with their clients: “...she’s here and I know there’s a 50% chance there’s domestic violence [referring to demographic information about the clients collected for the evaluation], so I just need to keep my eyes and ears open and just give an opportunity or ask the point-blank question is this happening for you and how do we talk about that?” #4–Case Study A. In Case Study B, salience related to the role of the evaluation in highlighting to workers the importance of an outreach worker to the performance of the program, which led to further influence mechanisms as noted below.

Both case studies also included mechanisms of Opinion/Attitude Valence, or the change of beliefs and attitudes about a program through a thoughtful process of reflection and learning (Fleming, 2011; Henry & Mark, 2003; Oliver, 2008). In both, this change of beliefs concerned second-order effects from the experience of participating in an evaluation, with both positive and negative aspects (Dahler-Larsen, 2012). Participants in Case Study B reported an overwhelmingly positive experience of the evaluation: “they were very very good, the researchers who came in, they were very good at engaging the mums, they were very very respectful” #15 – Case Study B. This had the effect of increasing the willingness of practitioners to collaborate with evaluators. Similarly, in Case Study A one participant highlighted that “you can have evaluation and have it mean that you are curious about your work and that its okay to have some difficulties in your work and for that to be on display if you like” #5 – Case Study A. Program staff discussed evaluation findings and were involved in problem solving issues identified by the evaluation. Participation in these processes seemed to change attitudes about the relevance of evaluation to practice. Each of the case studies included some participant experiences that raised reservations about evaluation and evaluators:

I attended a conference and there was a discussion about this wonderful program called [Case Study B]. It was all up on the big screen with the outcomes and what have you and it’s–that’s about me and people I work with and I’ve never seen that information. I guess in some ways that made me a little more hesitant to have a discussion because there was stuff that was up… #19 – Case Study B.

Both of these examples in the cases reflected concerns about the genuineness of the consultation and engagement between practitioners and the evaluators; in Case Study A, this focused on participant concerns about the influence of the evaluation funder on the findings; in Case Study B, all participants talked about the loss of trust was lost due to the failure to present findings to the program before presenting to a broader audience.

Examples of Priming were identified in Case Study A, where participants reported an improved understanding of a concept changed their judgment about an issue (Oliver, 2008). Participants indicated that learning about how other program sites operated helped them to reflect on how the program worked: “...I do think it’s created a more thoughtful approach to our work, and I suppose a sense of model fidelity in terms of providing how the four different sections interrelated with one another to create an outcome for a client” #4 – Case Study A. Case Study B also included an example of skill acquisition, that the evaluation and the agency’s workshopping of findings was seen by some as developing evaluative capacity, particularly in having practitioners deal with feedback about the service, and their capacity to understand and work with evaluation data.

Interpersonal Level Change

Interpersonal level change was only identified in Case Study A, reflecting the efforts of the agency to facilitate discussions and respond to the evaluation findings. One of the participants identified as a change agent, someone who because of the evaluation focused on changing how things were done in the organization (Mark & Henry, 2004). This person was proactive about building a culture of evaluation in their program site, working to engage staff by promoting an open dialog about the evaluation: “I was trying to be a person who said, so if we had some questions or worries can we put them on the table and can you answer them, so that the staff could see we were having an open dialog, we could get an open response and feel okay about going forward” #4 – Case Study A.

Some participants felt that the evaluation played a role in developing a culture of evaluation and change practice around the organization. This seems to be akin to the mechanism local descriptive norms, which Oliver (2008) describes as evaluation affecting the social behavior and interaction of stakeholders. The evaluation led to regular discussions between staff to reflect on the significance of the data for their practice. The influence of evaluation on social norms matches Dahler-Larsen’s (2012) description of constitutive effects, influence stemming from an evaluation having occurred.

Through injunctive social norms, which Oliver (2008) describes as agreed upon values about how to conduct oneself in a setting, the evaluation appeared to influence the delivery of the program. Some practitioners viewed the evaluation as reductive and were critical of practice having to be done in a more standardized way, that “the rules and regulations they’ve wanted … has impacted on the delivery...you have to do it this way for the evaluation” #5 – Case Study A. Injunctive norms were also changed in encouraging both managers and practitioners to reflect on evaluation information:

…there’s an expectation that you’ll not only read it, but know the implications for your service, you’ll have a program improvement plan, you’ll have your own program improvement plan to help increase your retention rates… so a sense that you would need to respond to what… the evaluation data and indeed throughput data was telling you #4 – Case Study A.

There was also a reported change in the injunctive norms for practitioners, the expectation that practitioners thought about and were aware of the implications of evaluation findings for practice: “… what I liked about it was it brought it back to “hey this is what you’re doing… this is the things of what we’re seeing” I guess it just made us think more about our practice and tried to encourage us to be more purposeful in what we’re doing…” #9 Case Study A.

Collective Level Change

Each of the case studies identified the evaluations as entering into policy consideration, with both agencies using the evaluation to support funding applications. In Case Study B, this involved targeted efforts to obtain funding for an outreach worker at other program sites, based on the evaluation highlighting the importance of the role. In Case Study A, the evaluation informed policy consideration in dealing with early exits and client engagement, participants described that “families [were] coming in and once their crisis had been resolved they were quick to leave the program… we had intended families to stay with us for about 2 years…” #2 Case Study A. Processes (described below) were put in place to address this.

The evaluation led to agenda setting; in Case Study A, some participants noted that the evaluation put data behind some of the perceived differences between sites, which resulted in productive discussions at the management level. There was a standing agenda item at these meetings about the evaluation, with managers expected to talk through the implications of the data for their service sites. In Case Study B, the evaluation was attributed with prompting a reconsideration of the program due to the lack of significant changes in the outcomes for children:

…the children's findings in phase two… weren't statistically significant… I guess what we were doing as a reference group, as a researcher was to go back to the literature and finding out about we don't actually know much about how parent outcomes translate to children, and we really just assume that happens #13 Case Study B.

The evaluation did however highlight positive shifts in parental capacity, relationships between parents and children, and levels of community support, which informed work within the agency of changing the expected outcomes for the program.

The evaluation led to policy-oriented learning, which Oliver (2008) defines as an increased understanding at the policy-making level of the organization; in Case Study A, the evaluation was attributed with several examples of this. The evaluation data identified that for some families the length of the program was unnecessary, which prompted a more flexible approach for families exiting the program when they had finished their case plans. The evaluation report and several participants noted that the client group were much higher on the spectrum of risk than the program was rated for, which raised questions of the relevance of the research evidence informing the program, and challenges for staff: “…we recruited staff at this level thinking that this program was going to be an early intervention program… whereas in fact what we were seeing was families very close to the tip of child protection, very complex issues surrounding them” #2 Case Study A. This led to some reflection within the agency about their responsibilities as an employer, and how to equip staff with the resilience and judgment needed to work with this different target group. The evaluation also prompted consideration about the fidelity of the program in a regional setting:

… we’re still thinking we can do home visits in the way we assume home visits should work, so by and large we’re still trying to do those things, have we compensated enough for say rural areas, where it’s not that feasible unless you live in the town to use childcare as a mechanism, have we really worked out what to do about that, I’m not sure about that #5 Case Study A.

Many examples of policy-oriented learning were identified in Case Study B. The evaluation found benefits from the integrated program model, and accordingly, the agency adopted the approach across other programs, placing a greater emphasis on partnerships and network building. A participant praised the evaluation as laying out the operation of Case Study B in a clear way that could be applied to other networks. The evaluation data provided an improved understanding of the local community, allowing the program to be more responsive to community needs.

Beyond policy learning, the evaluation resulted in several examples of policy change linked to the evaluations. In Case Study A, the evaluation led to an increase in the agency’s evaluation capacity through the hiring of evaluation staff and the creation of a research translation position. This increased capacity led to changes in the way evaluation information was disseminated. From surveys of the organization’s evaluation capacity, it was found that staff felt that evaluation recommendations were generally not being implemented:

We weren’t getting very positive feedback that evaluation recommendations were going anywhere or improving practice, and we had a bit of a debate with my boss about what role if any should we have in that, and I didn’t really think that it was a role for us, but she was of the opinion that if we don’t do it then nobody will do it… just leaving it to the operational side of the organization… was obviously not working #1 Case Study A.

This then led to the development of workshops with managers and practitioners to help respond to evaluation findings. The experience of the evaluation was highly formative in the creation of the agency’s evaluation policy, an element of which was the use of a structured template for practitioners to reflect on and respond to evaluation findings. This was used within the agency to facilitate collaborative problem solving and share practices across sites.

Case Study A also included policy changes that impacted the program and agency but were driven from outside the agency—the influence of the evaluation on these changes was more speculative on the part of the participants compared to the internal policy changes. Some participants thought the evaluation had been highly influential at the state government level, while others thought that the evaluation had been used to support existing decisions made about the program: “It seemed to me that they were trying to argue a case and look for any support for that case rather than just say what the data said” #3 Case Study A. This concerned a series of changes that led towards the program being wholly delivered by non-government agencies, in part prompted by the complexity of the cases being managed.

Case Study B had some minor policy changes, which reflected the different phase of the evaluation relative to Case Study A. The evaluation was reported to have shaped the role of the community connector, in that the evaluation very early on identified that the role was central to the functioning of the network. The evaluation recommended retaining the outreach worker role in the community to sustain referrals and to raise the profile of the service. Similar roles had been built into other integrated services across the agency because of this evaluation finding: “…we applied for… an integrated center in [another location] … and we built a community connector role into that…” #13 Case Study B.

There were some examples of program continuation, cessation, or change in Case Study A. The evaluation identified the limited evidence base and efficacy of some of the parenting programs being delivered, as a result these were changed: “…I do think the parenting program issue that came out in the evaluation, that group parenting programs weren’t particularly successful, and I think they’ve disappeared virtually straight away” #5 Case Study A. Also, a result of the evaluation recommendations the service provision guidelines changed engagement times from 2 years to 1 year.

Limitations

As research and evaluation managers within each of the agencies provided the documents and participants, a key limitation of this research is the potential for selective inclusion of sources flattering to the agencies involved. While this does not appear to be the case given the critical perspectives expressed by participants, obtaining participants without the research and evaluation managers as intermediary may have resulted in different accounts of the evaluation and its influence. Similarly, expanding the case studies out to include the evaluators and the evaluation funders may have provided a clearer sense of how evaluation influence played out.