Abstract
In recent years, evaluation has emerged as the basis for good clinical practice and health service management. As well as being an important activity in its own right, it is also a vital component in health service accreditation. In Australia, the Australian Council of Health Care Standards, Evaluation and Quality Improvement Program 3 (EQuIP 3), has determined that, to be accredited, health-care services would need to achieve a rating of moderate achievement for all the mandatory standards.[1] To achieve this rating, health-care services, including mental health services, are required to demonstrate evidence of evaluation.
Whatever the impetus, the focus on evaluation augurs well for opportunities for improvement in clinical practice and mental health service delivery. The drivers for evaluation have provided legitimacy for clinicians to make the time to reflect on their clinical practice and for managers to support them in these activities.
DEFINITION
Evaluation is defined in the Oxford Dictionaryas the process for assessing or appraising.[2] In today's health environment, evaluation is about assessing outcomes, in particular health outcomes, consumer outcomes and service outcomes. It is important to note that outcomes are implicitly conditioned by value systems as reflected in practice standards. The two most influential value systems in mental health in Australia are found in the National Standards for Mental Health Services (NSMHS) and the Australian Council for Health Care Standards (ACHS) Evaluation and Quality Improvement Program 3 (EQuIP3).
THE PURPOSE OF EVALUATION
The clinician can choose to evaluate to identify the outcomes of his or her own practice or the work of the team or service in order to effect system changes, or change existing treatment and intervention options being offered to consumers of mental health services. In essence, evaluation provides a means of knowing what works and what does not work when providing care. Clinicians may also wish to verify what is generally accepted but not evident in the published work or, alternatively, to test what is in the reports.[3]
Clinicians who manage mental health service delivery can also undertake evaluation to better understand compliance with the service delivery outcomes as required within a Health Service Agreement with the funding body and for the purpose of clinical governance. Clinical governance can be defined as a framework through which health organizations are accountable for continually improving the quality of their services and safeguarding high standards of care by creating an environment in which excellence in clinical care will flourish.[4] The accountabilities for clinical governance has led all health organizations to better define their evaluation processes.
The ultimate purpose of evaluation is the pursuit of quality as a universal activity by clinicians and health-care organizations in order to improve the care of consumers of mental health services.
AN EVALUATION FRAMEWORK
A useful evaluation framework is one derived from the National Health Performance Committee (NHPC).[5] The National Health Performance Framework (NHPF) uses nine descriptive domains and adopts a systems approach to evaluation. The nine domains are effective, appropriate, efficient, responsive, accessible, safe, continuous, capable and sustainable. The framework and descriptors are included in Table 1.
Evaluating health system performance
A systems approach to using the NHPF in evaluation
The NHPF has been primarily designed to evaluate health service systems and adopts a whole-of-service evaluation approach to evaluate existing or new services.
A systems approach is best undertaken through a mapping exercise of how consumers travel through the system of care, often referred to as ‘continuum of care’.
The continuum of care is described in terms of access, entry, assessment, care planning, care implementation, care evaluation, separation, exit and re-entry. Using the nine domains, it is relatively easy to define areas for evaluation and monitoring. For example, in the area of entry, the service structure within a public mental health service would be triage/duty. Table 2 illustrates examples of questions that could be asked to evaluate triage/duty. The last column indicates outcomes that can be expected.
Evaluation questions for triage
The greater knowledge gained from evaluation can galvanize activity, particularly when the outcomes have not been achieved. In this example, one could undertake a process review of how decisions are made, as well as a structurereview to ensure that a senior clinician is available to support the triage function. By extending this framework across the continuum of care, a systems approach to evaluation can be achieved.
Using the NHPC framework to evaluate treatment
As mentioned previously, the NHPC framework has been primarily designed to evaluate health service systems. Nonetheless, it serves as a useful way to look at the evaluation questions that could emerge at a clinical level as well. A practical application of the framework to clinical practice can be demonstrated as follows:
Scenario: Your service has been given limited access to a new atypical depot medication, Drug X, the cost of which is not subsidized. As a clinician, you can prescribe for only five patients. You have decided that it is important to ensure that the five patients will be those who will receive the maximum benefit of the drug.
You have decided to look at the patients who would meet the inclusion criteria and are also frequent users of mental health services, for example, those with frequent need for clinical appointments to monitor mental state and compliance, and who have had four or more acute hospital admissions and used an average of 60 bed days over 12 months. Using the framework discussed here, you can choose to ask evaluation questions across the nine domains. The basic rule around evaluation questions is that consideration must be given to measurable outcomes, as shown in Table 3.
Measurable outcomes
Evaluation tools
Indicators
Indicators are measurements of performance to indicate progress towards service outcomes and hence service performance. Indicators are defined in numerator and denominator terms. It is important to determine targets and thresholds because they indicate expectations of performance. For example, when assessing responsiveness in care, one may choose to define a triage or duty indicatoras follows.
Numerator:The number of urgent referrals where the patients are seen (assessed) within 2 hours in the time period under study.
Denominator:The number of urgent referrals in the time period under study.
Target: 100%.
Indicator data should be routinely reported in order to identify trends. It is possible to select several indicators, and the value or worth of each could be determined using the criteria set out in Table 4.
National Health Performance Committee indicator selection criteria[5]
Audit tools/surveys
These are generally paper-based or electronic-based questionnaires. The content would vary depending on the type of audit or survey. Audits and surveys can be done retrospectively or prospectively. Audit tools can be designed to capture individual activity or multiple activities. Because there may be inconsistencies in practice within a record of activity, it is important to have areas for comment. The significance of audit cannot be underestimated because it captures the details of practice that cannot be captured by indicators alone. Audit tools are particularly useful when evaluating compliance with clinical or treatment guidelines and policy.[6] Surveys are conducted primarily to get feedback and opinions, and can be both qualitative and quantitative. They are most frequently used with staff and clients of mental health services.
Gap analysis against a set of standards
Evaluation can also be undertaken through a gap analysis against a set of standards. If the standard is to ensure that client needs for quality and safe care with desirable outcomes are addressed through the planning, delivery and evaluation of care, list how this is being done and identify the gaps. The process for identifying the gap can be done by looking at examples of activity that would reflect the intent of the standard. To achieve the standards described here, the examples of how it can be done could include: (i) multiservice case reviews for complex needs consumers in place; and (ii) an audit of prescribing practices resulting in significant reduction of use of typical depot medication.
The gap in this example could be: establishment of an ongoing group psychosocial rehabilitation programme for people who do not access psychiatric disability support services.
An important feature of gap analysis is to ensure that it is tabulated in a way that holds individuals accountable and has a time frame for completion.
CONCLUSION
Evaluation as a prevailing culture is a hallmark of excellence in care[7]. Evaluation is not a complex process. What is required is a willingness to be accountable for outcomes of care. It does not require complex data systems but openness to scrutiny through reflective practices and a commitment to continue to explore the possibilities of making things better.
The process for evaluation needs to be part of clinical practice, clinical care and service delivery. It is not undertaken instead of clinical work, but is part of the measure of the value and worth of clinical services.