Abstract
Recovery from serious mental illness refers to the attainment of a meaningful, productive, and satisfying life, regardless of the presence or absence of reoccurring symptoms. A lack of psychometrically sound instruments has thwarted attempts to measure recovery. With the goal of addressing this need, we administered the Recovery Assessment Scale (RAS) to 92 people with serious mental illness attending a Clubhouse program. Clubhouses are community-based psychosocial rehabilitation programs for people living with mental illness. Rasch analysis enabled close examination of the RAS’s internal validity and reliability. Through iterative analyses, we made enhancements to the instrument where possible. The preponderance of evidence suggests that the modified RAS forms a unidimensional construct; however, the instrument remains far from a gold standard. Occupational therapy is well suited to take leadership in further development of this instrument.
Recovery from serious mental illness, which includes schizophrenia, was described as early as the 1900s (Warner, 2004). Only recently, however, with expanding numbers of large-cohort, longitudinal studies and parallel publication of numerous personal testimonies, has the reality of recovery begun to drive systemic change. Pessimistic prognoses have slowly given way to mounting evidence for recovery. Slade, Amering, and Oades (2008) gave an overview of these large-cohort, longitudinal studies, which have collectively found that as many as one-third of people are likely to recover completely and that another third are likely to improve such that their illness no longer affects their daily life. Although many studies have confirmed these more optimistic prognoses, the criteria for being “recovered” are not uniform. Attempts to reach consensus on a definition of recovery have fueled ongoing debate; lack of consensus thwarts efforts to measure recovery.
Empirical, longitudinal studies have tended to consider recovery in terms of symptom remission or eradication and return to before-illness level of functioning. This medically oriented definition of recovery has been termed recovered from (Davidson & Roe, 2007). However, the degree and duration of symptom absence necessary for one to be considered recovered from serious mental illness vary among studies.
Consumer voices have highlighted inter- and intrapersonal processes involved in recovery. Central to consumers’ definitions are hope, empowerment, self-efficacy, and establishment of relationships based on interdependency and trust. Consumers have also emphasized the importance of engagement in meaningful activity (Anthony, 1993; Bellack, 2006; Onken, Craig, Ridgeway, Ralph, & Cook, 2007), a factor that resonates with occupational therapy’s understanding of the relationship of occupation to health (Aubin, Hachey, & Mercier, 1999; Eklund & Leufstadius, 2006; Goldberg, Britnell, & Goldberg, 2002). These parameters have been termed in recovery (Davidson & Roe, 2007) and refer to a state in which symptoms may still be present but the person has recovered a dignified, full, and self-determined life.
Occupational therapists have highlighted the synergy between recovery principles and principles central to occupational therapy (Gruhl, 2005; Krupa & Clark, 2004). Gruhl (2005) and Krupa and Clark (2004) have challenged occupational therapists to take a leading role in facilitating the translation of recovery principles into systemic change in mental health service delivery.
Recent attempts to connect the recovery-from (clinical) orientation with the in-recovery (personal) perspective have suggested that considering multiple domains of recovery might be useful. Slade and colleagues (2008) suggested that clinical and personal recovery be considered separately. Other researchers have suggested four separate recovery domains: (1) clinical or symptom, (2) personal, (3) functional, and (4) social (Lloyd, Waghorn, & Williams, 2008).
The terms clinical recovery and symptom recovery are used synonymously in the literature to describe amelioration or successful management of symptoms (Mueser et al., 2002). Personal recovery refers to internal processes, facilitated by empowering and inclusive environments, through which consumers establish or reestablish a sense of hope, purpose, and self-efficacy (Deegan, 2003). Functional recovery refers to doing or engaging in meaningful roles and occupations. Social recovery incorporates community inclusion and establishment of meaningful, supportive, and satisfying relationships and social networks (Corrigan & Phelan, 2004; Davidson et al., 2001). Whether recovery is a unidimensional construct or a multidimensional construct with several domains of recovery occurring simultaneously has not been discussed in the literature and is a matter for investigation.
Internationally, occupational therapists and others working in mental health face two equally powerful directives: adoption of a recovery orientation and establishment of effective outcome measurement practices. Consequently, psychometrically sound self-report instruments are needed to better describe pathways to recovery and measure progress along those pathways. Although many instruments (several of which have been developed by occupational therapists) measure symptoms and functioning, few attempt to measure the personal recovery domain (e.g., hope, self-efficacy; Campbell-Orde, Chamberlin, Carpenter, & Leff, 2005). Although one might argue that occupation-oriented instruments capture the functional domain of recovery, psychometrically sound instruments to provide evidence regarding personal recovery are scarce, and occupational therapists are as likely as other professionals to resort to medically oriented, symptom-focused measures when exploring personal or intrapersonal and interpersonal aspects.
In recent years, a few instruments emphasizing the personal recovery domain have emerged (Campbell-Orde et al., 2005; Ralph, Kidder, & Phillips, 2000). The Recovery Assessment Scale (RAS; Giffort, Schmook, Woody, Vollendorf, & Gervain, 1995) is one such instrument. Although the RAS emphasizes personal recovery, it has items relating to other domains (i.e., symptom or clinical recovery, social recovery and, to a lesser extent, functional recovery).
A criticism of the current suite of recovery instruments, including the RAS, is that they have not been tested for their ability to discriminate among people at different points of recovery (Campbell-Orde et al., 2005). To date, psychometric evaluation of the RAS has been conducted within a classical test theory framework (Corrigan, Giffort, Leary, & Okeke, 1999; Corrigan, Salzer, Ralph, Sangster, & Keck, 2004; Flinn, 2005; McNaught, Caputi, Oades, & Deane, 2007). Item scores are summed to obtain a total score even though testing to date has questioned the construct validity of data gathered with the RAS. Unidimensionality is required for summing scores (Wright & Linacre, 1989).
Corrigan and colleagues (2004) found that only 24 of 41 RAS items loaded on five factors. Factor loadings for the 24 items ranged from 0.53 to 0.87. This proposed 24-item test seems to suggest that the remaining 17 items are outside the recovery construct. A closer examination of those items, however, revealed that several are related to symptom management and a sense of control over the recovery journey and therefore reflect central tenets of the recovery literature. Moreover, even with a reduced number of items, Corrigan and colleagues flagged concerns regarding conceptual overlap and the small number of items within some factors. For example, three items within one factor (“I know when to ask for help,” “I am willing to ask for help,” “I ask for help when I need it”) might be more reflective of item overlap than of a genuine factor. Clearly, more research is required.
Over recent years, awareness has been growing in the health sciences of modern test theory approaches, such as the Rasch measurement model (Andrich, 2004; Bezruczko, 2005). Occupational therapists are increasingly using Rasch analysis in developing and examining measurement instruments (Fisher, Bryze, & Atchison, 2000). A Rasch approach assumes that the construct under study is unidimensional and then looks for evidence to disprove that assumption.
The purpose of this study was to use Rasch analysis to examine evidence for reliability and internal validity of data collected with the RAS from a sample of community-dwelling people with mental illness. Such research is important in the quest for robust instruments to measure mental health recovery (Campbell-Orde et al., 2005).
Method
Ethics approval was obtained from the University of Sydney Human Ethics Committee and the Northern Sydney and Central Coast Area Health Services’ Human Ethics Committee.
Setting
This study took place in an Australian Clubhouse program. Clubhouses are community-based psychosocial rehabilitation services with >300 programs operating in 27 countries (Macias, Jackson, Schroeder, & Wang, 1999; see the International Centre for Clubhouse Development Web site, www.iccd.org/about.html, for more information). A Clubhouse is a restorative community space where people with mental illness can achieve or regain the confidence and skills required to lead productive and socially satisfying lives (Henry, Barreira, Banks, Brown, & McKay, 2001; Townsend, Birch, Langley, & Langille, 2000).
Participants
All members attending the Pioneer Clubhouse were invited to participate (N = 139). We approached members directly and through announcements made at weekly Clubhouse meetings. Potential participants were excluded if they had inadequate English comprehension to complete questionnaires or were in acute inpatient care.
In total, 92 members volunteered, and all gave written consent. The participants were 31 women and 61 men with primary diagnoses of schizophrenia or schizoaffective disorders (n = 51), bipolar disorders (n = 27), depression (n = 11), or anxiety disorders (n = 3) ranging from age 18 to 68 (mean [M] = 39 yr, standard deviation [SD] = 6.92). We obtained diagnoses from the Clubhouse database, updated by current referral letters from treating medical practitioners.
Instrument
The RAS is a self-report instrument with 41 items scored on a 5-point Likert scale. Item statements were developed from analysis of consumer testimonies relating to recovery. Respondents report the degree to which they agree (e.g., 1 = strongly disagree, 5 = strongly agree) with each statement. Satisfactory test–retest reliability (r = .88) and internal consistency (Cronbach’s α = .93) have been reported (Corrigan et al., 1999). The original 41 RAS items are listed in abbreviated form in Tables 1 and 2 for ease of reference.
Recovery Assessment Scale Item Measure and Fit Statistics: Final Model
Note. Items are ordered by level of difficulty, with the hardest item first and easiest item last. SE = standard error; MnSQ = mean square.
Items Removed During Analysis
Note. Items were removed because of poor fit statistics (i.e., >1.5 mean square), except for Item 31; Item 31 was removed because of redundancy (see Item 32 in Table 1).
Procedure
To facilitate the response of participants for whom poor literacy or other factors (e.g., decreased attention) might create barriers, we administered the RAS using individual interviews. Interviews took place in a quiet Clubhouse location. One of four trained interviewers read the items to each respondent. Reading questions to participants has been recommended in previous studies (Corrigan et al., 1999). A structured guide enabled interviewers to provide uniform clarification if needed. Participants recorded their responses privately. Completing the instrument took 5–10 min.
Analysis
Raw data were subjected to Rasch analysis using Winsteps Version 3.68.2 (http://Winsteps.com, Chicago). Rasch analysis (Smith & Smith, 2004; Wright & Stone, 1979) and the rating scale model (Andrich, 1999) are detailed fully elsewhere. We provide only a brief description here.
Rasch measurement assumes that all people are more likely to have high scores on easy items and that people who are more recovered are more likely than people who are less recovered to have higher scores on hard items (Smith & Smith, 2004; Wright & Stone, 1979). During Rasch analysis, ordinal-level raw data are transformed to interval-level data called measures; all item and people measures are placed on a single hierarchy according to relative difficulty of items and level of recovery of people. We used Andrich’s (1999) rating scale model because of the items’ Likert response structure.
Analysis followed an iterative process involving several steps. Throughout our analyses, we examined the following sources of evidence, although we did not examine all at each step (Linacre, 2005).
Direction of Correlations Between Items and the Measure.
A negative correlation between an item and the measure means that the item is not part of the construct. Positive correlations were therefore required.
Rating Scale Category Structure.
If the 5-point rating scale is used as desired (i.e., people scoring 1 on any item would have an average instrument measure lower than those scoring 2 and so on), the scale categories would progress in an orderly fashion. Ideally, distances between each scale point would be similar, and all categories would be used regularly (n > 10). In the situation in which this source of evidence was not met, we did not proceed. Instead, we collapsed rating categories to fix the problem, reran the analysis, and then rechecked criteria.
Goodness-of-Fit Statistics.
We expected that the goodness-of-fit statistics for each item, expressed as mean square (MnSQ) values (ratio of observed scores to expected scores), would be <1.5 (Wright & Linacre, 1994). A MnSQ >1.5 suggests that people respond to the item in unpredictable ways because the item is interpreted differently or does not contribute to the construct under examination. We examined items failing to meet this criterion for their clarity and value to the construct. Items deemed not to offer useful information to the construct were removed, and we reran the analysis. This process continued until remaining items appeared to be a good reflection of the construct.
Item Bias.
We computed differential item function analyses for all relevant person factors (i.e., age, gender, diagnosis) to test that they did not affect item responses. If >5% of t tests were significant or a meaningful pattern was identified, then we assumed the item hierarchy differed between subgroups (e.g., men and women).
Construct Representation.
We examined the item hierarchy for acceptable spread of items across the range of difficulty. Attention was paid to areas along the hierarchy where the construct was either underrepresented, reducing instrument precision (items missing), or overrepresented (i.e., more than one item at the same level of difficulty), suggesting redundancy (Baghaei, 2008). Additionally, we examined the overall match between range of difficulty of items and range of relative recovery of the sample to ensure that they were similar. Item measures are set to M = 50 (SD = 10). People measures are not anchored; an M near 50 indicates a match between people’s ability and scale difficulty.
Logic of the Hierarchy.
We examined the ordering of items in the context of the recovery literature. That is, does the difficulty of items progress in a way that is in keeping with collective knowledge?
Principal Components Analysis.
We conducted principal components analysis of residuals.1 Significant amounts of unexplained variance (>40%) suggest the possibility of additional dimensions, particularly when the percentage of variance explained by the first contrast is >5% and eigenvalue units are >3 (Tennant & Pallant, 2006). We examined the data for patterns suggestive of a meaningful second factor when the first contrast approached these values. We further examined dimensionality through differential person analysis and a series of t tests comparing person estimates on items identified in the first contrast. Substantial numbers (>5%) of statistically significant t tests also suggest additional dimensions.
Correlations Among Residuals.
We also examined correlations among residuals for evidence of local dependence in the data, which is a violation of a Rasch assumption and another indicator of additional dimensions. We performed all possible correlations; any >.30 were considered to reflect local dependence.
Internal Reliability.
We examined both the people separation index and a reliability index statistic (Cronbach’s α equivalent). We required the instrument to separate people into two or more groups of ability. We also required a reliability index >.80 (Fisher, 1992).
Results
Winsteps analysis revealed several significant problems with the data yielded by the RAS. Thus, following the guidelines described in the Analysis section, we undertook a series of analyses to revise the scale and items and produce the best possible instrument. Results from each phase informed the analysis that followed.
Item Correlation
All items had a positive correlation with the overall measure. Correlations ranged from .45 to .76.
Rating Scale Category Structure
In the initial analysis (5-point scale), the scale structure for >40% of items was disordered, and the distance between categories for many items was unsatisfactory (Figure 1). We would expect that for each item (x axis), scoring categories 1–5 would appear in order from left (1) to right (5). This order would mean that participants who rated themselves higher on a particular item had higher overall scores than participants who rated themselves lower on the item. Moreover, scoring categories 1, 2, and 3 on all items had a low probability of being selected. Combining these adjacent scoring categories resulted in a 3-point (1–3 = disagree, 4 = agree, and 5 = strongly agree) scale, resolving disordering and low probability and, less satisfactorily, separation between categories. Figure 1 also illustrates the scale structure after collapsing categories.
Average person measures for observed categories of sample items (A) before and (B) after collapsing categories.
Goodness-of-Fit Statistics
Using data from the 3-point scale, MnSQ goodness-of-fit statistics for 9 items were outside the acceptable range (see Table 2). The poor fit of data from these items closely reflects results of earlier studies (Corrigan et al., 2004; McNaught et al., 2007). The problem appears to relate to framing of item statements or failure to fit the recovery construct. Given that all items with poorly fitting data were situated at the same level of difficulty (item numbers on the same horizontal line in Figure 2) as other, better fitting items, we removed them. We also removed Item 31 (“I am willing to ask for help”), which was situated at the same level of difficulty as Item 32 (“I ask for help when I need it”) and appeared to be redundant. In total, we removed 10 items, creating a 31-item assessment.
Hierarchy of people and items.
After adaptations were made to the scale structure and some items were removed, we again subjected the data to Rasch analysis and reexamined the evidence regarding internal validity and reliability. Data from all items had acceptable fit (MnSQ <1.5; Wright & Linacre, 1994; see Table 1).
Item Bias
Differential item functioning analyses computed for age, diagnosis, and gender revealed that, with two exceptions, all ts were <2. The oldest group (≥ age 55) had significantly higher scores on Item 5, “I have a purpose in life” (t = 2.5, df = 5), and men measured higher on Item 29, “Symptoms occur for shorter periods” (t = 2.36, df = 2).
Construct Representation
The mean person measure dropped from 63.55 to 54.37, indicating a reasonably good targeting of items to people. Figure 2 maps people’s self-rated recovery level against item difficulty. Items at the bottom of the map are easier for people to affirm; the most challenging item appears at the top. People at the top of the map awarded themselves higher scores, indicating greater self-reported recovery. Figure 2 shows that there is a dearth of items with which to precisely measure relatively more recovered people.
Logic of the Hierarchy
The hierarchy of items was examined against the literature and appeared logical. Easiest items related to awareness of what helped recovery. Hardest items related to symptoms having lesser impact on daily living. Items are ordered by level of difficulty in Table 1.
Principal Components Analysis of Residuals and Correlations Among Residuals
A principal components analysis of residuals revealed that the amount of variance left unexplained by the measures was relatively high (54.5%), a result that appears to be because both person and item distributions are noticeably central. Person and item distributions need to be diverse to explain >50% of data variance (Linacre, 2006). The first contrast explained 4.8% of the variance. The strength of this contrast was 3.1 eigenvalue units. Neither variance explained by the measures nor percentage of variance explained by the first contrast changed in response to alterations to the assessment. Examination of people and items represented in the first contrast revealed that approximately 10% of the sample received unexpectedly high scores on relatively hard items (e.g., symptom control) and unexpectedly low scores on generally easier items (e.g., social connectedness). Comparison of overall person estimates on the two sets of items reflected in the first contrast, using differential item functioning analysis and t tests, revealed that scores of 16.3% of people differed significantly on the two sets of items. This higher-than-desirable value provides some evidence of multidimensionality of RAS items. Eight of the 961 correlations (<1%) were >.3; all were <.5.
Internal Reliability
The people separation index was 3.93, indicating that the revised instrument was able to separate people into approximately four levels of recovery. The reliability index (Cronbach’s α equivalent) of .94 was excellent.
Discussion
Occupational therapists and others working in mental health have a tremendous need for assessments that measure recovery. The RAS shows great potential in this area. However, previous difficulty with establishing validity using factor analysis (Corrigan et al., 2004) led us to examine the RAS with Rasch analysis. Rasch allowed us to (1) examine the possibility that RAS items actually form a unidimensional construct of recovery and (2) derive information about the internal validity and reliability of data collected with it. Although we did not find a definitive answer about the former, we gained substantial information that enabled us to modify the instrument to enhance internal validity and reliability of data collected with it. Because we wanted to explore unidimensionality, we analyzed data from all the items together. We might have completed separate analyses on each of the item subsets associated with the five factors (subscales) identified by Corrigan et al. (2004). However, the subscales are small (some had as few as 3 items), and the items composing them were sometimes repetitive. Thus, we chose not to take that path.
Using Rasch analysis, we found that a 31-item RAS, with a 3-point rating scale, has acceptable evidence for internal reliability and validity. However, the instrument remains far from a gold standard. Although a significant number of items appeared to be dichotomous, with only the strongly agree category separating from other scoring categories (see, e.g., Items 16 and 28 in Figure 1), we felt it would be illogical to collapse categories with contradicting labels (i.e., disagree with agree). Thus, we retained three scoring categories. Further improvement of the instrument might be made if a dichotomous scoring option were offered; this option should be tested in future research.
Many items remain at the same level of difficulty, and they do not add to measurement precision. Removing some items might make the RAS shorter and, thus, quicker to complete. However, because the items measure different aspects of recovery conceptually, we chose to retain them to maximize utility of the RAS in practice settings for intervention planning.
No assessment ever represents a completely unidimensional construct. The question is always, To what degree it is multidimensional (Smith & Smith, 2004)? In this case, the preponderance of evidence suggests that the modified RAS is measuring a single construct that one can call recovery. Although some findings have suggested that the applicability of the hierarchy to all people is in question, most people in this sample shared a single pattern in their recovery journey. The typical recovery hierarchy reported in these data began with awareness and knowledge of the importance of friendship, healthy habits, and fun as the easiest items. It progressed to having a goal and future orientation, then trusting and relying on others. Further along the hierarchy came general optimism, hope, self-love and acceptance, and a sense of purpose in life. Achievements relating to sense of self-efficacy and the belief that one can succeed follow. At the pinnacle of the construct is successful symptom management and symptoms having a lesser effect on daily living.
Most people recover a sense of social connectedness, hope, and optimism before establishing a stronger sense of self-identity, empowerment, and a sense of control over their lives. For other people, however, recovery seems to occur in the reverse order: They recover in terms of clinical symptoms and control over those symptoms before establishing a sense of social connectedness, optimism, and hope for the future. This variation reflects literature generated by mental health consumers emphasizing the individually unique journey toward recovery (Bellack, 2006).
Although several items were at the same level of difficulty, the most difficult end of the hierarchy had a dearth of items, indicating underrepresentation of the construct. Participants who reached the ceiling of this instrument were still actively attending the rehabilitation service, and we therefore surmise that they would not consider themselves at the end of their recovery journey. Construct underrepresentation is a threat to the construct validity of an instrument (Baghaei, 2008). An alternative possibility requiring further exploration is that this instrument measures only the in-recovery construct and that 10% of our sample was indeed in recovery.
Examination of items in light of the existing recovery literature suggested that although they address the personal and clinical domains of recovery, they do not cover functional or social domains adequately. Additional items addressing level and quality of engagement in personally meaningful occupations and measuring degree of social connectedness might be more difficult than current items reflecting predominantly personal recovery. Such additional items could address the need for items at the higher end of the hierarchy and would also produce a more holistic instrument that is in keeping with occupational therapy’s understanding of the occupation–wellness relationship.
Occupational therapists have demonstrated that engagement in valued occupations enhances mental health, well-being, and recovery (Eklund & Leufstadius, 2006). Occupational therapists are well suited to take a leadership role in further development of the RAS and to play a more active role than has occurred to date in the endeavor to establish a gold-standard measure of mental health recovery.
Limitations
Participants were limited to active members of a Clubhouse program in Sydney, New South Wales, Australia. Findings might not be able to be generalized beyond this population of community-dwelling Australians living with mental illness. Clubhouse members attend the program of their own volition and might be predictably further along in their recovery journey than many other people with serious mental illness.
Implications for Future Study
Future study is needed to examine whether current enhancements to the measurement properties of the RAS hold true when used with people at various stages of recovery. Examination of explained and unexplained variance with a less homogeneous sample would also add confidence that the scale measures a unidimensional construct. Finally, the proposed 3-point category structure needs to be tested with a larger sample size.
Footnotes
Acknowledgments
We thank the Pioneer Clubhouse community for ongoing collegial contributions to the project.