Abstract
Since the introduction of the National Mental Health strategy in 1992, significant service reform has taken place. This has not only included structural reform of mental health services but also a push towards improved service quality, introduction of innovative models of service delivery and a greater demand for accountability and service monitoring [1]. The Second National Mental Health Plan has as one of its key themes the further development of effective, quality service delivery [2]. Strategies to achieve this goal include development of standards of care, service evaluation and introduction of outcome measures.
In a recent thought-provoking article, Adler and Mathieson [3] highlight that the recent service reforms require a concurrent change in the training and continuing education of psychiatrists. A review of the current Royal Australian and New Zealand College of Psychiatrists’ training guidelines confirms that ‘trainees should have skills relevant to effective delivery of mental health services’ [4]. However, trainee psychiatrists generally have no formal training in service evaluation, leadership, management or accountability [3]. Any such training only occurs if provided by individuals with an interest in the field. In practice this means only a small proportion of registrars have direct experience of these issues before taking on consultant psychiatrist positions. Adler proposes a thorough review of training with an increased emphasis on continuing education and a culture of learning [3]. This article describes the recent experience of conducting a qualitative service evaluation project and the attitude of registrars to this process. It highlights some of the practical issues involved with adopting Adler's recommendations.
LITERATURE REVIEW
Whilst there is plentiful literature pertaining to service evaluation, there is little published work on clinicians’ attitudes to their service being evaluated. At present there is a significant gap between the beliefs and attitudes about measuring health care, and our ability and desire to change it [5]. The majority of outcome measurement to date has occurred by outsiders, whereas those with the power to implement change usually work within the organisation. This presents a gap between the evaluation process and action as a result of it. It has been documented that the effects of health service reform reduce enthusiasm for work among medical specialists in New Zealand, Australia and the United Kingdom [6]. This loss of enthusiasm can be counter-balanced by involving the specialists in the managerial process of reform and evaluation [6].
Berwick suggests that clinicians should study measurement tools, systems theory and leadership skills in order to participate in improvement strategies [7]. Methods for creating a culture of cooperation and learning include: ‘developing a shared purpose; creating a safe environment; encouraging diverse viewpoints; learning to negotiate agreement and insisting on equity in applying rules’ [8]. Berwick suggests that health providers need to learn to ‘use measurement for improvement, not just measurement for judgement’ [9]. He argues that clinicians are more able to accept prudent local changes in practice then sweeping changes [10].
Studies in the USA demonstrate that clinicians go through a lengthy learning process during service evaluation [11]. Initially they demonstrate anxiety and suspicion regarding data pertaining to their utilisation rates. With time, active involvement and ongoing education, clinicians come to accept and learn from the evaluation process. A valuable component of quality improvement is therefore to bring clinicians into an organisation capable of nurturing trust, respect and review [11]. Strategies to encourage this process include recruiting those with an interest in service-related research, allowing room for individuals to conduct evaluation in specific fields of interest and providing effective role models and social supports to allow development of ideas [12].
It has been suggested that a qualitative approach to organisational research may enable an improved understanding of attitudes, opinions and feelings through provision of more descriptive data than can be obtained quantitatively [13]. The majority of organisational research examples that invoke clinician opinions and engagement utilise a participatory action research model. Participatory action research implies that individuals involved in the organisation under study participate with the researchers throughout the research process, including at the stages of project planning, implementation and discussion of results [14].
Anticipated benefits of such a methodology include an emphasis on valuing the opinions of the subjects and allowing them to act as agents of change, converting research recommendations into new policy and clinical practice [15]. Thus participatory action research provides a method for total quality management, moving the focus of control from outside the individual to within [16].
This paper discusses a service evaluation project from the perspective of these ideas.
ORGANISATIONAL CONTEXT
The South Eastern Sydney Area Health Service is one of the metropolitan health services in Sydney, NSW. The mental health program comprises four semiautonomous operational units covering separate geographical catchment districts. Each of these operational units consists of an integrated inpatient psychiatry unit and a community-based mental health team. All components of the mental health program have accreditation for training with the Royal Australian and New Zealand College of Psychiatrists.
PROJECT OUTLINE
The project commenced in response to increasing difficulties of managing demand for acute inpatient beds within the Area. Anecdotal tales of psychiatry registrars struggling for hours to find an appropriate inpatient facility for a patient presenting acutely were commonplace. It was hypothesised that examination of the factors influencing bed demand could lead to more systematic approaches to addressing the situation.
The project set out to find an instrument that would increase the objectivity of decision making about admission and therefore provide management with information about factors that could be addressed to improve the situation. In the course of obtaining this information we discovered some interesting issues related to registrar engagement in evaluation.
METHOD
A steering committee comprising service directors, academic and clinical psychiatrists, a health economist and a policy analyst was formed. This committee designed a multidimensional decision analysis tool that examined factors which might make a contribution to decisions to admit/not admit at the point of acute presentation to psychiatric services. These factors included patient and illness characteristics, functionality, service availability, registrar experience and environmental support.
Derivation of the anchor points for a dimensional measure of illness, risk, functionality and service availability factors occurred by modification of an American care utilisation tool called LOCUS (17). Other factors were agreed upon by consensus of steering committee experience.
During five randomly selected weeks over a nine month period, data on all acute psychiatric presentations to the four mental health services were collected according to the agreed categories of the decision analysis tool. From the total number of acute psychiatric presentations for the time period, two cohorts of patients were identified: those who were admitted and those not admitted.
Data were obtained from clinical file audit and from telephone interviews with the assessing registrar, which took place as soon as possible to the date on which the acute clinical assessment had occurred. The psychiatry registrar interviews took approximately 15 minutes.
Prior to commencement of the project, registrars were notified via their service directors of the service's willingness to participate. Registrars were individually contacted in writing to explain the project and at the time of the interview they were reassured of confidentiality.
RESULTS
The analysis of the samples of clients admitted and not admitted is the subject of another report. Here we describe registrar attitudes to being involved in the evaluation process. It was noted that registrars were reluctant participants in the process of evaluation. We identified two major themes underlying this reluctance. These were registrar defensiveness about being evaluated and a general lack of interest in evaluation per se.
Reluctance to participate was demonstrated by a range of responses from guarded co-operation to passive refusal by evading the interview process. In one case over 10 phonecalls were made to engage a registrar in the process. When an interview was finally arranged, the registrar did not attend. Those registrars who did agree to participate questioned the motives of the project, believing that as individuals they were under scrutiny when the reality was that the process of service delivery was the subject of the study. Even when this had been carefully explained, the sense was of guarded answers, protecting themselves as individuals and justifying their decisions in clinical terms rather than revealing concerns about the structure of local policies and organisational structure. Some registrars commented that the research would not be used to bring about any beneficial changes and that it was more likely to be used as a vehicle for criticism of their decision making.
Several registrars expressed the belief that any service evaluation is merely an academic tool, of no relevance to their day-to-day practice. Expressions that any form of research was an unnecessary and unwelcome intrusion on day-to-day activities were common. The need to ‘get on with the routine work’ was a recurrent theme.
When comments were made on the service, there was a sense that such comments were in vain and would be unlikely to bring about any significant practice or organisational change. Some suggested that the project data would be used to reduce resources that were already perceived as scarce, for example beds. Others advised that the money used to fund evaluation would be better spent on providing greater staff numbers.
DISCUSSION
General comments
It should be emphasised that this service evaluation project did not set out to determine the attitude of registrars to service evaluation. The findings are entirely a subjective report of the experience of one of the evaluators (PH-J). Thus, there are some serious methodological flaws from the perspective of determining attitude. We therefore admit that we are merely inferring attitude from a range of verbal and behavioural responses to being requested to participate in service evaluation.
Engagement
The level of defensiveness of the registrars to evaluation suggests that the methodology utilised was not sufficiently engaging. A degree of anxiety and resistance to the process could have been predicted [11]. Strategies known to engage clinicians in the process of service evaluation and quality improvement will need to be incorporated in future similar projects involving medical staff. These include involvement of medical opinion leaders, ensuring education is the focus, establishing open communication and being prepared to deal with resistance [11]. We will need to explore whether the time taken to conduct the interviews was sufficient to allow the registrars to understand and question the process and, in addition, whether conducting interviews by telephone is adequate for this process.
We recognise that future service evaluation projects involving medical staff will need to have a stronger focus on collaboration and engagement at the outset. Meeting the registrars in person, rather than relying on correspondence to convey information regarding projects, and including small group discussions are options to be considered. Group discussions have been described as providing the security associated with being with others when sensitive material in being discussed [18]. In addition it is anticipated that the interaction between group members may stimulate discussion and reflection.
Project Planning
It is important to note that whilst planners may recognise that service evaluation requires a collaborative input, this project demonstrates that it is not always possible to identify with whom to consult. During the planning stages of this project, service directors, consultant psychiatrists and academics were involved but the registrars received only information on the project. During the project itself the value of the contribution of registrars was recognised. This will need to be taken into account when planning the next phase of the project, thus demonstrating the use of a participatory action research model in service evaluation.
Registrar Training
The general lack of interest expressed by registrars implies not only that this particular methodology was insufficiently engaging but also that evaluation is not currently incorporated into current models of service delivery or training. Registrar training is focussed on developing clinical practitioners and underemphasises a range of research techniques and quality improvement skills. Since emphasising quality and effectiveness is an impetus of the Second National Mental Health Plan it would seem essential for registrars to develop interest in, and understanding of, quality issues. The authors contend that achievement of this goal will require definite intervention approaches. Merely exposing registrars to service evaluation in action is unlikely to generate their interest in quality issues.
CONCLUSION
The changing model of mental health care delivery as outlined in the Second National Mental Health Plan requires a change in the way psychiatrists function. Evaluation of their role and ability to incorporate quality improvement into routine clinical service delivery will be required. As Adler has advised, the culture of psychiatry education will need review and change. A current challenge to mental health services is to ensure that psychiatrists are skilled and interested in conveying to their registrars (in their clinical teaching and role modelling) the importance of service evaluation and quality improvement.