Consumer and Service Determinants of Completion of a Consumer Self-Rating Outcome Measure

METHOD

The data come from the routinely collected outcomes measurement that commenced in four Victorian mental health agencies in mid-2000. At that time, staff in the adult services were trained, specialized software was provided, and clinical staff began using the suite of measures, which consisted of the Health of the Nation Outcome Scales (HoNOS), 10 the short (16-item) form of the Life Skills Profile (LSP) 11 and the Focus of Care, 11 which are all clinician-completed measures, and the Behaviour and Symptom Identification Scale (BASIS), 12 which is the consumer self-rating measure adopted in Victoria. The HoNOS consists of 12 items rated 0 (no problem) to 4 (severe problem) in the areas of problem behaviours, impairments, psychiatric symptoms, and social problems. The LSP consists of 16 items rated 0 (no problem) to 3 (extreme problem) in the areas of withdrawal, self-care, compliance, and antisocial behaviour. The BASIS consists of 32 items rated 0 (no difficulty) to 4 (extreme difficulty) in the areas of relation to self and others, daily living and role functioning, depression and anxiety, impulsive and addictive behaviour, and psychosis.

Measures were to be completed at certain points in the process of care. In acute inpatient settings, only the HoNOS was to be completed at admission and discharge. In non-residential settings (i.e. in the community) all measures apart from the LSP were to be used at the start of an episode of care, and all three measures were to be used at a minimum of 3 monthly intervals, and at the closure of the episode. The two clinician-completed measures do not require face-to-face contact with the consumer, and no specific instructions were given to staff concerning the degree of personal presence required when completing them. As for the consumer self-rated measure, however, clinicians were advised in training that where possible they should be with the consumer when he or she completes the BASIS.

While clinical staff were expected to complete their measures at the required times, consumer self-rating was necessarily subject to different conditions. First, the BASIS was entirely voluntary. Second, clinical staff were expected to use their clinical judgement in whether and how to invite consumers to complete it. It was recognized that under certain circumstances (e.g. consumer very unwell or hostile) it would be inappropriate to offer the BASIS.

Outcomes data were entered locally into specialized software. 13 ,14 For the purposes of analysis, the local data files from the four agencies were collated centrally. The aggregated data consisted of 22 928 records of assessment on one of the three main instruments. One agency had been using the HoNOS alone since 1996; of all the recorded administrations of a measure, 6566, or 28.6%, were of administrations prior to 1 July 2000. These have been excluded for the purpose of the present study.

At the individual consumer level, the available data items are age at assessment, sex, principal and secondary psychiatric diagnosis, case manager, and treating psychiatrist. For each assessment, the available data items are date of assessment, type of assessment (HoNOS, LSP or BASIS), clinician identifier, and the individual item ratings, subscale scores and total scores of the instrument. For each clinician the only information that was available was sex and discipline.

In order to reduce the potential influence of extraneous contextual factors, only assessments conducted at case review in the community will be considered. These are, in any case, the most numerous in the current data.

RESULTS

Of the 6114 consumers with recorded outcomes assessment data, 3351 had assessments conducted in the context of case review. The mean number of administrations of any of the three instruments per consumer was 5.1. A total of 1864 (55.6%) consumers were male, and the mean age was 39.8 years (SD = 14.9). Psychiatric diagnoses were recorded according to the International Classification of Diseases. 15 The numbers of consumers with principal psychiatric diagnoses in the most numerous classes are shown in Table 1; a further 76 consumers had principal diagnoses in other classes, and in 424 cases the diagnosis was not recorded.

The four mental health agencies will be denoted by letter codes A-D. The numbers and percentages of consumers across the four agencies was as follows: A, 1238 (36.9%); B, 1337 (39.9%); C, 496 (14.8%); and D, 280 (8.4%).

Of the 3351 patients with recorded assessments at case review, 2222 (66.3%) did not complete a BASIS and 1129 (33.7%) did. There was no difference between the sexes. However, the mean age of those completing (35.9 years) was significantly lower than those not completing (41.8 years; (t = 10.9, df = 3349, p < 0.0001). This effect was not uniform across agencies; the difference was highly significant in agencies A and B but not significant in agencies C and D.

For consumers with HoNOS and/or LSP assessments at review, the mean of those assessments was computed. The mean of the HoNOSs of those consumers who did not complete a BASIS (11.1) was significantly higher than that of those who did (8.8; t = 10.1, df = 3011, p < 0.0001). A similar picture was evident with the LSP, where the mean of the LSPs of those consumers who did not complete a BASIS (13.6) was significantly higher than that of those who did (11.0; t = 8.0, df = 1976, p < 0.0001). Scores on all HoNOS and LSP subscales were also significantly different in the same direction.

Next, we look at the association with diagnosis. Table 1 shows the percentage of consumers who did and did not complete a BASIS within the most numerous principal psychiatric diagnoses.

Table 1 shows that the rate of completion of BASIS varied considerably by diagnosis (χ 2 = 102.7, df = 5, p < 0.001). Only two of 85 consumers with organic diagnoses completed a BASIS. Rates of completion were also comparatively low for consumers with substance abuse and anxiety disorders. The completion rate of consumers with mood and personality disorders was around 30%, and for consumers with non-organic psychoses (mainly schizophrenia) it was 39%.

The aforementioned results show that age, HoNOS total and subscale scores, and LSP total and subscale scores were separately associated with the likelihood of completing a BASIS at review. However, all these variables are correlated to some degree. 16 In order to assess their independent effects simultaneously, a further analysis (logistic regression) was conducted. For this we included as independent variables age, the four HoNOS subscales and the four LSP subscales. This analysis showed that higher rates of BASIS completion were significantly associated with lower age, higher scores on HoNOS symptoms subscale, lower scores on HoNOS behavioural and social sub-scales, and lower scores on LSP withdrawal and compliance. In a further analysis in which the diagnostic information from Table 2 was added, none of the principal diagnoses significantly increased the capacity to predict BASIS completion.

Table 2 shows the numbers and percentages of consumers with review assessments who did and did not have a BASIS, broken down by agency.

It may be seen that the percentage of patients whose review assessments included a BASIS ranged from a low of 8% to a high of 60%. The association between agency and completion is highly significant (χ 2 = 647.8, df = 3, p < 0.0001).

For the purposes of examining case manager associations, a subset of the data was formed. As mentioned earlier, clinicians were expected to consider offering the BASIS to the consumer at case reviews, when they themselves were completing the HoNOS and LSP. Thus, it is reasonable to use a HoNOS administration at case review as the potential trigger for a BASIS assessment. The relevant data set therefore consisted of HoNOS assessments conducted in the community in the context of a case review. For each of these assessments, the information relating to any LSP or BASIS conducted within 2 days was appended. This resulted in a data file of a little over 3000 HoNOS assessments on a little under 2000 consumers. Examination revealed that, where a consumer had more than one HoNOS assessment, there was little difference between them in whether or not they were accompanied by a BASIS. Thus, only the earliest HoNOS assessment was retained, and a few additional assessments were discarded because the identity of the clinician was not recorded. The final data set consisted of assessments conducted on 1853 consumers by 220 clinicians in four agencies.

The 1853 consumers comprised 1061 men (57.3%) and their mean age was 38.0 years (SD = 11.7). A total of 1227 had a principal psychiatric diagnosis of non-organic psychosis (66.2%), another 435 had mood disorder diagnoses (23.5%), 81 had anxiety or neurotic disorders (4.4%), 71 had personality disorders (3.8%), and 39 had substance abuse disorders (2.1%). The 220 clinicians consisted of 105 men (47.7%); 156 (70.9%) were nurses, 21 (9.5%) were social workers, 18 (8.2%) were occupational therapists, 16 (7.3%) were psychologists and nine (4.1%) were medical staff.

Table 3 shows the numbers of clinicians and HoNOS assessments conducted in the community in the context of a case review, as well as the number and percentage of those assessments that were accompanied by a contemporaneous BASIS assessment, broken down by agency.

Table 3 shows that, overall, just over one-quarter of HoNOS assessments conducted at case review in the community were accompanied by a BASIS with a date not more than 2 days away. However, as in Table 2, there is wide variation between agencies, ranging from 3% to more than 50%, and the association between agency and completion is again highly significant (χ 2 = 417.4, df = 3, p < 0.0001).

Clinicians themselves varied greatly in the proportion of assessments they conducted that were accompanied by a BASIS. As Figure 1 shows, more than half had no recorded BASIS assessments; for another approximately 10%, about half of their assessments were accompanied by a BASIS, and all of the assessments of a further 10% of clinicians were so accompanied.

OPEN IN VIEWER

Figure 1:

Percentage of clinicians for whom given percentages of their Health of the Nation Outcome Scale (HoNOS) assessments were accompanied by a Behaviour and Symptom Identification Scale (BASIS) self-rating.

Given the differences between agencies already noted, it is unsurprising to find that clinicians with different rates of completion are unequally distributed between agencies. Table 4 shows this.

Agencies A and C, with the lowest overall BASIS completion rates, had a majority of clinicians none of whose assessments were accompanied by a BASIS; in each of these agencies there were only approximately 3% of clinicians, all of whose assessments were accompanied by a BASIS. In contrast, in agencies B and D there were only 19% and 36% of clinicians, respectively, with no BASIS assessments, and 15% and 23% of clinicians, respectively, all of whose assessments were accompanied by a BASIS.

In order to discern the concurrent contributions of different factors to the likelihood of a consumer having a BASIS when there is a recorded HoNOS at review in the community, a logistic regression was undertaken. The dependent variable is whether or not a BASIS was present, and the independent variables were age, consumer sex, clinician sex, clinician discipline, the four HoNOS subscale scores, the four LSP subscale scores, and agency. Agency was represented by three ‘dummy’ variables, with agency A as the baseline category. Clinician discipline (nurse, medical, psychologist, occupational therapist and social worker) was represented by four dummy variables, with nurse being the baseline category. Because it was likely that the chances of a consumer having a BASIS depended in part on who the clinician was, the model was further elaborated by specifying that the data were clustered on clinician. This clustering process ensures correct estimates of standard errors, and hence interpretations of significance (Stata User's Guide, section 23.14 17 ).

In the logistic regression analysis described, higher likelihood of a BASIS was significantly associated with: agencies B, C and D (compared to the base agency, A; all p $0.003), lower age (p = 0.01) and lower scores on the compliance subscale of the LSP (p < 0.001). The other variables in the model, namely consumer and clinician sex, all HoNOS subscales, the other LSP subscales, and clinician discipline, were not significant.

DISCUSSION

The strongest association with BASIS completion was not a consumer characteristic, but which agency was involved in the consumer's care. Although data from only four agencies were available, the differences in BASIS completion rates between them were large, ranging from less than 10% to more than 50%. The information available does not enable any conclusive statement as to the reason for this, but a strong possibility is that it is the culture, including the expectations, of the service system that is a powerful factor influencing the uptake of consumer self-rating.

Only three consumer characteristics, lower age, diagnosis and higher compliance, were consistently related to BASIS completion. It may be that younger consumers are more likely to adopt an innovation in practice, which is what routine consumer self-rating is in most contemporary mental health settings. The rate of BASIS uptake in consumers with organic diagnoses was 2%; not included in these analyses were consumers with learning disability, for whom there was not a single instance of BASIS completion in any setting. Clearly, some minimum degree of cognitive capacity is required to respond to a systematic enquiry into one's health status. Apart from the question of capacity, it has been noted that cognitive impairment may adversely affect the validity of selfratings. 18 Goldman et al. noted that ‘… the accuracy of self-reports varies a great deal among chronic conditions’ (p. 152) and found that cognitive function was a significant correlate of accuracy of selfreport of hypertension. 19 The compliance result is consistent with the finding of Trauer and Callaly, reported in a previous section, of a link between noncompletion and involuntary legal status, 8 because non-compliance with treatment is one factor in conferring involuntary legal status. Clinicians were expected to use their commonsense in whether they offered the BASIS to a consumer, and it is understandable that they should be less likely to offer it in the face of general non-compliance.

In the present paper, we have looked at the joint contribution of consumer and service factors in the completion of a consumer self-rating measure. However, the interpretation of raw completion rates is hampered by at least two considerations. First, there was no information on whether the clinician-rated instruments were completed in the presence of the consumer, in which case it would presumably have been easier to offer the self-rating instrument. Second, there was no information as to whether non-completion occurred because the clinician did not invite a self-rating, or if an invitation was made but refused. It would be most helpful in monitoring the implementation of this or any other self-assessment instrument to have the reason for non-completion routinely recorded. Some jurisdictions have made provision for such information to be recorded, typically by including two questions for the responsible clinician on the standard documentation. The first question is ‘Was the self-rating instrument offered to the consumer?’ and the second presents several choices to indicate the reason for non-offer or non-completion, such as being incapable or refusing. To the extent that services seek to maximize self-rating participation rates, there is good reason to include such items in their documentation, and subsequently entering the responses into the information system. This will allow continuous monitoring of this aspect of routine outcome measurement and provide a basis for remedial action to be taken as necessary.

Across all four agencies that provided data for the present study, just over half of all clinicians had no recorded assessments that were accompanied by a BASIS. While there were wide variations between agencies in this clinician-level effect, even in the two more productive agencies there were sizeable minorities of clinicians with no BASIS-associated assessments. To this extent, much of the potential benefits of consumer self-rating are not being realized. For example, Eisen et al. showed that those consumers who completed a BASIS and then discussed their results with their psychiatric resident, felt that they had been treated with greater respect and dignity than those who received treatment as usual from their resident or psychiatrist. 20 The present results may be interpreted as indicating that the practice of consumer self-rating as part of routine outcome measurement is patchy and not yet firmly established as part of the culture of the service organization. While many instances of non-completion may be attributed to consumer factors such as cognitive limitation and non-compliance, service-level factors such as service culture and clinician practice appear to be equally important.

One method of increasing self-rating participation levels that is worthy of further research is that of computer administration. Allenby et al. asked patients to report their psychosocial function using touch screen technology in an ambulatory cancer clinic, and found that ‘although over half the patients had no prior computer experience, nearly all found the touch screen easy to use and the instructions easy to understand’. 21 Bliven et al. asked patients attending a cardiology clinic to complete a quality of life and functional status measure in both paper-and-pencil and electronic formats. 22 They found that ‘Computer literacy, educational level, age, sex, and race were not significantly associated with the ability to successfully complete the computer-based questionnaire. 82% of patients preferred the computer-assisted administration to paper, and 89% reported that they would feel comfortable using the software in the future without any technical assistance’. In the mental health area, Greist et al. compared interviewer- and computer-administered versions of the Diagnostic Interview Schedule and found that most patients liked the computer interview better and found it less embarrassing, 23 while Epstein et al. found that respondents from the general population reported more mental health problems with ‘computer-assisted self-interviewing’ than with an interviewer-administered paper-and-pencil method. 24 Recently, the computer and paper versions of the instrument that has been widely adopted as the self-report outcome measure in child and adolescent mental health, the Strengths and Difficulties Questionnaire (SDQ), 25 were compared; trends towards better reliability and user satisfaction were observed for the computer version. There is thus considerable promising evidence in computerized self-assessment in health and specifically mental health. To the extent that much of the value in consumer self-assessment lies in the promotion of beneficial dialogue with the clinician, 9 services will still need to ensure that this information, however it is gathered, is utilized in the ongoing process of care.