Abstract
In the last decade, a new international emphasis on the ongoing measurement of ‘outcome’ in routine mental health services has emerged. A standard definition of outcome in mental health is a measurable change in the health of an individual, or group of people or population, which is attributable to an intervention or series of interventions. 1 In New South Wales, the Mental Health Outcomes and Assessment Tools (MH-OAT) were implemented in July 2001. The MH-OAT consists of a set of standardized clinical measures to be used within mental health services, allowing for the documentation of clinical practice at different time points in the cycle of care. 2
MH-OAT appears to be the product of the clinical governance movement. It has evolved for reasons including making the mental health profession more accountable in light of reduced public confidence in the medical profession's ability to self-regulate because of medical errors probably occurring more frequently than is acceptable. 3 , 4 Also, continued funding of mental health services is cited as a prominent reason for the need for outcome measures so as to justify the effectiveness of the work done in mental health settings, as well as the increasing pressure from consumer organizations for issues of choice, evidence-based quality and value to be considered. 5
USING MH-OAT: TWO STUDIES
The focus of the current paper is on the use of MH-OAT within two community child and family mental health service teams in Sydney, Australia. (The general term ‘MH-OAT’ will be used throughout this paper in reference to the outcome measures used within the child and adolescent services; these measures are sometimes referred to as MH-OATCA). The MH-OAT data collection for these two services commenced in mid-2002. Typically, data from the MH-OAT measures are collected on admission, every 13 weeks (reviews), and on discharge. The measures used on each occasion are as follows: Health of the Nation Outcomes Scales for Children and Adolescents (HoNOSCA), Children's Global Assessment Scale, ICD-10 Factors Influencing Health Status, and Strengths and Difficulties Questionnaire (SDQ). The first three measures are clini-cian-rated while SDQ is a client-rated questionnaire.
STUDY 1
Given the extent of data collected via MH-OAT, we aimed to describe the client profile for our service by analysing the SDQ data that were collected over nearly a 2 year period (August 2002–May 2004). For this period, there were admission SDQ data on just 36 clients and only one client had corresponding SDQ admission and discharge data. With these small numbers, it was not possible to conduct any type of descriptive profiling that would be representative of the children who access the service. The admission SDQ figures represented 15% of all client admissions during this period for this service. Fuller analyses explored the compliance rate with the clinician-rated MH-OAT measure, the HoNOSCA. There were a total of 171 clinician-rated admissions, which represented 70% of all client admissions during this period. Further, 71 of these clients were still current, leaving 100 clients who should have had discharge clinician-rated data, but only 53 did. This figure was originally 80. However, upon investigating the actual HoNOSCA data, 27 of these who were registered on MH-OAT as a discharge (suggesting MH-OAT discharge data were available) had inadequate data (e.g. too much missing data (defined as 4 or more out of the 15 questions, but was usually the entire set of questions) and/or the clinician had stated that the answer to the items was ‘not known/unable to rate’, again this was usually for the entire set of questions). Thus, approximately half of those clients who had clinician-rated admissions data also had clinician-rated discharge data, and 30% of all client admissions for this service had clinicianrated admission and discharge data. (This last figure takes into account the deduction of the 71 clients from the total client admissions (i.e. 246) as they were still being seen by the service).
These results raise a number of issues. It appears that clinicians are often not completing the clinicianrated measures (to a lesser degree on admission and a significantly greater degree on discharge), and rarely were they requesting and/or following up clients to complete the SDQ on both admission and discharge. Clearly, such low completion rates of required Department of Health data beg the question, why? Also, was this situation an isolated incidence within the child and family mental health services? A second study conducted by another service, independently of ours, provides some insight to these questions.
STUDY 2
With the advent of the NSW Health Department Circular 2004/30 which stated that ‘all public Mental Health Services are required to adopt the standardized MH-OAT clinical assessment protocols and modules from the date of this Circular’, a small email survey was conducted with 23 clinicians. The survey focused on the clinicians’ current use of MH-OAT and consisted of the following four questions: (i) Do you regularly review the progress of clients? (ii) Do you use the MH-OAT standard measures to inform this process? (iii) If not, what do you use? (iv) Do you follow the MH-OAT 13 week schedule?
Twelve out of the 23 clinicians responded to the survey (52% return rate), with all stating that they regularly reviewed the progress of their clients. With one exception, there was an emphatic ‘no’ (usually replied to in upper case and bold) in answer to the question ‘Do you use the MH-OAT standard measures to inform this process?’. All clinicians described the case review meeting, the individual managementplanning process or the clinical supervision process as the vehicle for reviewing progress. Apart from the one person who used MH-OAT, the remainder utilized clinical notes and the Connor Rating Scale as the data used to inform the process. With regard to the final question, ‘Do you follow the MH-OAT 13 week schedule?’, one clinician stated that they tried but they found it difficult to do so. The remaining 11 did not follow the 13 week review schedule. The results of this survey suggest that while clinicians are reviewing the progress of their clients, they rarely use the MHOAT data for this purpose and they do not regularly complete the MH-OAT measures. This is supported by the findings from the first study.
It should be noted that the pattern of involvement with families at this service is not conducive to a 13 week review cycle. A family will make initial contact and be seen within a month of that contact and then not seen for 2–3 months or longer (but there will be phone contact). After this time, there will be an intensive 1 week treatment episode, which in MH-OAT terms is an admission, then a period of less intense contact followed by at least one more episode of intense involvement.
LOW COMPLIANCE RATES
These results demonstrate that clinicians on the two child and family mental health service teams did not regularly complete the MH-OAT measures. Further, the second study highlighted that MH-OAT outcome data were rarely used to evaluate the progress of clients. This shows that low compliance in using MH-OAT is probably not an isolated event and the survey gives some clues as to possible explanations for this.
Callaly et al. note that, for mental health professionals, the clinical governance movement, of which MHOAT is a part, presents a reduction in their clinical autonomy. 3 This may go some way in explaining the clinicians’ inconsistent use of MH-OAT in these two studies as being representative of a type of ‘passive resistance’ to its implementation.
It is also possible that some clinicians may be aware that the HoNOS has been shown to have poor reliability and validity. 6 – 8 Consequently, they may question the value of using the HoNOSCA, and the MHOAT measures generally, in clinical practice.
A separate, yet related point to validity is the ‘applicability’ and ‘utility’ of a measure. Andrews et al. refer to applicability of a measure as the degree to which it addresses dimensions of importance to the rater. 9 Stein, in referring to the work of Fernstein, 10 explains that for the clinician, the utility of a measure is assessed on its ability to contribute meaningfully to the care of the client:
… unless a clinician believes that an intervention would directly help the patient in the consulting room, or, at the very least, assist in the diagnostic process, the intervention will not be undertaken. Time spent between clinicians and patients is highly valued by both parties, and may even be highly charged emotionally with its sole purpose of trying to help the patient. To complete a rating scale which has an ulterior motive such as assisting a purchaser, or helping to gather national statistics, is to act on behalf of third parties. It would therefore serve as an intrusion into the clinician-patient relationship and would not be tolerated… 11
The general low compliance of completing the MHOAT measures within the two services studied may, in part, be explained by the possibility that clinicians perceive the applicability and utility of these measures as very limited in directly helping their clients. This possibility is strengthened by the finding from the second study that MH-OAT outcome data were rarely used to evaluate the progress of clients. Indeed, this finding echos previous results 12 that HoNOS ratings contributed little to client care plans developed by clinicians, and were completed almost as an afterthought in the care plan review meetings.
Blisker and Goldner also highlight another potential problem with the current MH-OAT system. They state that relying on treatment providers as the source of data violates a basic precept of outcome measurement design:
… the practitioner who is responsible for delivery of a clinical intervention, and thus has a stake in the outcome of the intervention, is not in a position to objectively assess the outcome. The practitioner should be regarded as a biased observer of client status… A treatment effectiveness study in which the treatment provider was also the outcome rater would almost certainly be discarded from systematic review as being compromised… Outcome data are increasingly valued in mental health systems to guide funding and other decision-making; thus programmes and providers have more incentive to produce questionable data. 13
The findings from the current studies, along with Blisker and Goldner's comments, 13 raise questions about whether the current MH-OAT system is the best method of assessing outcomes. In their paper covering this issue, Blisker and Goldner present two possible alternatives. 13 Their first suggestion, ‘selective independent rating’, uses independent observers to investigate a restricted number of programmes or treatments during specified periods. This process is not undertaken routinely, which reduces cost. The second suggestion is ‘assessment of fidelity’ (the degree to which programmes or treatments adhere to established evidence-based practices). It is argued that since it has been previously demonstrated that such practices produce the best outcomes for clients, a programme or treatment implementing these practices in the right way for the appropriate clients should result in the best outcome. Blisker and Goldner claim that ‘assessment of fidelity is less costly than routine outcome measurement: intervention protocols, systematic review of clinical records, and treatment observation for representative samples of patients provide the necessary data’. 13
CONCLUSION
The ‘outcome measurement revolution’ in mental health has arrived, representing a marked and permanent change in the mental health system. 4 However, within this revolution of permanent change, there needs to be accommodation for the evolution of the outcome measurement process, through critiquing, consulting and development. To suggest at this stage that we have arrived at the right method, in ‘clinical governance speak’, ‘to assure and improve the quality of clinical services’, would be premature.
The current research has served to provide some insight into the largely unexplored area of the execution of MH-OAT in everyday clinical practice. It also highlights the need for further research with clinicians to more fully understand their experiences with MH-OAT and empirically delineate their concerns with the aim of improving the outcome measurement process with their supportive involvement.