Construct Validity and Reliability of the Comprehensive Occupational Therapy Evaluation Scale (COTES) in People With Schizophrenia

Abstract

OBJECTIVE. We evaluated the construct validity (i.e., unidimensionality and convergent validity) and Rasch reliability of the 20-item Comprehensive Occupational Therapy Evaluation Scale (COTES) in people with schizophrenia.

METHOD. Retrospective chart review was used to collect COTES data from 505 inpatients with schizophrenia. For construct validity, we first examined unidimensionality of each of the three COTES subscales using Rasch analysis. After unidimensionality was supported, we examined convergent validity using Pearson’s r and Rasch reliability of the individual subscales.

RESULTS. After deleting two misfitting items, the remaining items (i.e., the COTES–18) showed unidimensionality. Infit and outfit mean squares were 0.73–1.25. Moderate correlations were found among the three COTES–18 subscales (rs = .57–.71). The Rasch reliabilities of the three subscales were .83–.92.

CONCLUSION. The COTES–18 has sufficient construct validity and reliability to assess three specific dimensions of behavior affecting occupational performance in people with schizophrenia.

Schizophrenia is a chronic mental disorder. As a result of the disorder’s psychiatric symptoms, behaviors such as untidy appearance, disorganized thoughts, poor social skills, lack of motivation, and short duration of work (Eklund & Leufstadius, 2007) are present, thereby affecting occupational performance. These impaired behaviors can be observed in activities of daily living, social functions, and work in the community. Improving behaviors that affect occupational performance is an important goal for occupational therapy in clinical and research settings. Clinicians and researchers must be able to measure these behaviors to design plans for interventions.

The Comprehensive Occupational Therapy Evaluation Scale (COTES), designed for people with mental illness, is one of the most widely used measures to assess behaviors that influence occupational performance in people with schizophrenia in Taiwan (Chen et al., 2006; Hsiao et al., 2000). It is a revised version of Brayman’s Occupational Therapy Evaluation Scale (Kau et al., 1981), which was culturally modified for use in Asian countries (e.g., Taiwan). The COTES was developed on the basis of occupational therapy theories, such as Azima and Azima’s (1959) Dynamic Theory of Occupational Therapy, Fidler and Fidler’s (1963) Object Relations Theory, and Mosey’s (1971) frames of reference.

The COTES contains three subscales: General Behavior, Social Behavior, and Work Behavior. The COTES is an observational measure that can be applied in acute psychiatric settings; it can be administered during a group activity, which can reduce burden on examiners, and it assesses multidimensional aspects of behaviors that affect occupational performance, which is useful in developing an occupational profile and determining clients’ strengths and weaknesses.

Two important psychometric properties of a measure are construct validity (e.g., unidimensionality, convergent validity) and reliability. Psychometric properties of a measure are sample dependent. When the measure is applied to a specific group (e.g., people with schizophrenia), psychometric properties of the measure need to be examined in that group (Chiu et al., 2014; Chiu, Lee, Lai, et al., 2015). For clinical applications of the COTES in people with schizophrenia, it is fundamental to examine construct validity and reliability to explain the underlying constructs and determine the degree of measurement precision.

Rasch analysis has been used for evaluating a measure’s reliability because it offers three advantages: (1) Scores of the measure can be transformed into interval scores; (2) items fit the assumptions of the Rasch model, demonstrating unidimensionality; and (3) Rasch reliability can be examined at the unidimensional or multidimensional level. Rasch reliability using the multidimensional approach takes into consideration the correlations among the subscales, which can increase the reliability estimates (Hsiao et al., 2015). For the COTES, unidimensionality using Rasch analysis has been examined in people with mental disorders (Wang et al., 2012) but not specifically in people with schizophrenia. Convergent validity and Rasch reliability of the COTES have not been examined in people with schizophrenia exclusively.

Therefore, the purposes of this study were to examine the construct validity (i.e., unidimensionality and convergent validity) and Rasch reliability of the COTES in people with schizophrenia. We first examined unidimensionality of the three individual COTES subscales. After unidimensionality was supported, we examined the convergent validity and Rasch reliability using unidimensional and multidimensional approaches.

Method

Participants

This retrospective study of 505 Taiwanese inpatients with schizophrenia took place in an inpatient psychiatric center in northern Taiwan. Data were collected from participants’ occupational therapy charts. The COTES was administered to participants from April 2005 to August 2014. Because participants did not work while living in the psychiatric center, the COTES Work Behavior subscale was assessed when they performed tasks during occupational therapy.

The inclusion criterion for data selection was a diagnosis of schizophrenia based on the Diagnostic and Statistical Manual of Mental Disorders (4th ed., text rev.; American Psychiatric Association, 2000). Patients who had a history of severe brain injury or who had a diagnosis of mental retardation or substance abuse were excluded. This study was approved by the psychiatric center’s institutional review board (TCHIRB-1010811).

The age range of participants was 20–76 yr, and about half (49.7%) were male. The mean length of time they received mental health services was 17.1 yr. The mean scores on the COTES were 23.4, 14.3, and 20.8 for the General Behavior, Social Behavior, and Work Behavior subscales, respectively. Table 1 lists further details of participant demographic characteristics.

Table 1.

Patient Characteristics (N = 505)

Characteristic	n (%) or M (SD)
Gender
Male	251 (49.7)
Female	254 (50.3)
Age, yr	41.8 (12.1)
Onset age, yr	24.9 (9.3)
No. of admissions	5.8 (5.1)
Duration of mental health services, yr	17.1 (11.0)
Duration of untreated psychosis, yr	2.0 (4.9)
Duration of drug treatment, yr	15.6 (11.4)
Length of time from diagnosis to assessment, yr	12.3 (10.6)
Education
Elementary school	28 (5.6)
Junior high school	84 (16.6)
Senior high school	275 (54.4)
College and above	118 (23.4)
Type of antipsychotic a
First generation	151 (29.9)
Second generation	418 (82.8)
COTES subscale scores
General Behavior	23.4 (3.9)
General Behavior (6 items) b	19.2 (3.5)
Social Behavior	14.3 (2.9)
Work Behavior	20.8 (5.0)
Work Behavior (7 items) c	17.7 (4.3)

Note. COTES = Comprehensive Occupational Therapy Evaluation Scale; M = mean; SD = standard deviation.

12.7% of patients were taking two types of antipsychotics.

Attendance item deleted from subscale.

Motivation item deleted from subscale.

Procedure

Secondary data from the COTES were collected by 29 occupational therapists during their weekly duties. These therapists were licensed and trained to administer the COTES (e.g., familiar with test items and scoring). In this study, research assistants (not the aforementioned occupational therapists) reviewed the occupational therapy charts and selected the eligible participants. One research assistant recorded participants’ COTES and demographic data, and another research assistant proofread the records to ensure the accuracy of the data recorded. The initial COTES assessment for each eligible participant was used for data analysis in this study.

Instrument

The 20-item COTES rates patients’ performance in work, leisure, and daily activities. The General Behavior subscale has seven items: attendance, appearance, activity level, emotional disturbance, nonproductive behavior, physiological needs, and impairment of speech. The Social Behavior subscale has five items: sociability, impression of others, acceptance of opinions, role in groups, and self-assertion. The Work Behavior subscale has eight items: motivation, duration, responsibility, frustration tolerance, self-expectation, comprehension, technique, and fine motor.

The COTES uses a 5-point rating scale (1–5) for each item except for five items (i.e., appearance, nonproductive behavior, motivation, comprehension, and technique). These five items allow raters to rate patients’ performance using intervals of 0.5 between the integers. The score ranges of the General Behavior, Social Behavior, and Work Behavior subscales are 7–35, 5–25, and 8–40, respectively. A higher score indicates better performance (Kau et al., 1981).

Data Analysis

Construct Validity.

Unidimensional Rasch analysis was conducted using Winsteps (Beaverton, OR) to examine the unidimensionality of each COTES subscale. We transformed the five items with intervals of 0.5 into a 9-point rating scale (1–9). Rasch analysis with the partial credit model was performed because the COTES items have different rating scales (i.e., 5-point and 9-point rating scales). We used infit and outfit statistics to determine whether item responses fit the expectations. Items with an infit or outfit mean square (MnSq) less than 0.6 or greater than 1.4 indicated a misfit (Chiu, Lee, Kuo, et al., 2015). The misfit items of a subscale were deleted, and then Rasch analysis was reconducted to ensure the unidimensionality of the subscale. Moreover, we conducted principal-components analysis (PCA) on the standardized residuals to confirm unidimensionality. The eigenvalue of the residual variance in the first contrast was calculated. The criterion of unidimensionality was eigenvalue <3.0 (Smith, 2002).

For convergent validity, we examined the correlations among the three subscales using Pearson’s r. Correlations among subscales were defined as strong, r ≥.75, or moderate, .40 ≤ r < .75 (Hsueh et al., 2012). Moderate to strong correlations among the three subscales should be observed because the subscales were developed to assess different aspects of behaviors that affect occupational performance.

Reliability.

After the unidimensionality of the three subscales was confirmed, we calculated Rasch reliability using unidimensional and multidimensional Rasch analyses. The multidimensional analysis was performed using ConQuest software (Australian Council for Educational Research, Camberwell, Victoria, Australia). The standard of reliability was ≥.70 for group comparisons and ≥.90 for individual comparisons (Chiu et al., 2014). Theoretically, multidimensional Rasch analysis can improve the precision of estimating participants’ abilities by considering the correlations between subscales (Hsiao et al., 2015).

The reliabilities of the three subscales in the multidimensional approach should be greater than those in the unidimensional approach. We used the Spearman–Brown prophecy formula to calculate the increment in the number of items (i.e., subscale length) for unidimensional reliability to reach multidimensional reliability (Hsiao et al., 2015). In addition, we estimated the percentage of reliability improvement using the following formula: (the difference between unidimensional and multidimensional reliabilities/multidimensional reliability) × 100.

Results

Construct Validity

Two misfitting items were demonstrated in Rasch analysis. In the General Behavior subscale, the infit and outfit MnSqs of the attendance item were 1.64 and 2.08, respectively. In the Work Behavior subscale, the infit and outfit MnSqs of the motivation item were 1.64 and 1.73, respectively. We removed these two items and reconducted the Rasch analysis. The remaining items of the General Behavior and Work Behavior subscales fit the expectations of the Rasch model (the ranges of the infit and outfit MnSqs were 0.84–1.20 and 0.73–1.25, respectively; Table 2). The Social Behavior subscale items met the criteria of infit and outfit MnSqs (0.81–1.18). In addition, the residuals in PCA of the subscales in the now 18-item COTES (COTES–18) showed that the eigenvalues of the first contrast were 1.6, 1.6, and 2.7 in the General Behavior, Social Behavior, and Work Behavior subscales, respectively. For convergent validity, our results showed moderate correlations among the three COTES–18 subscales: r = .71 between General Behavior and Social Behavior, r = .57 between General Behavior and Work Behavior, and r = .59 between Social Behavior and Work Behavior.

Table 2.

Rasch Estimates of Difficulty and Infit and Outfit Mean Squares

Subscale and Item	Difficulty Logit a	Infit MnSq	Outfit MnSq
General Behavior
Appearance	−0.19	1.09	1.07
Activity level	0.79	1.20	1.15
Emotional disturbance	0.83	0.94	0.94
Nonproductive behavior	−0.66	0.88	0.84
Physiological needs	−1.43	0.95	0.94
Impairment of speech	0.66	0.92	0.93
Social Behavior
Sociability	1.27	1.13	1.15
Impression of others	0.20	1.11	1.18
Acceptance of opinions	0.86	0.85	0.81
Role in groups	−1.71	0.85	0.92
Self-assertion	−0.63	0.99	0.96
Work Behavior
Duration	0.96	1.25	1.24
Responsibility	−0.70	1.06	1.04
Frustration tolerance	−0.20	0.95	0.90
Self-expectation	−0.42	0.90	0.88
Comprehension	−0.91	1.09	1.11
Technique	0.12	0.91	0.92
Fine motor	1.16	0.75	0.73

Note. MnSq = mean square.

Mean value of four or eight adjacent response categories in item difficulty.

Reliability

The multidimensional reliability of the three COTES–18 subscales was .88–.92 (Table 3). The unidimensional reliability of the subscales was .83–.90. Regarding the results using the Spearman–Brown prophecy formula, the three subscales had to be increased by 27.8%–50.2% of the number of items for the unidimensional approach to achieve the same reliability level as the multidimensional approach. For example, the length of the General Behavior subscale needed to be increased from six to nine items to achieve the reliability level of the multidimensional approach. The reliability improvement from unidimensional to multidimensional approaches was 2.2%–5.7%.

Table 3.

Rasch Reliability and the Effect of the Multidimensional Approach on the Increment in Subscale Length

Subscale	Unidimensional Reliability	Multidimensional Reliability	Increment in Subscale Length, %	Reliability Improvement, a %
General Behavior	.83	.88	50.2	5.7
Social Behavior	.85	.88	29.4	3.4
Work Behavior	.90	.92	27.8	2.2

Improvement from unidimensional to multidimensional reliability.

Discussion

Rasch analysis was used to examine the unidimensionality of the three COTES subscales in patients with schizophrenia. We deleted two items (one item each from the General Behavior and Work Behavior subscales), thereby supporting the unidimensionality of the three individual subscales from the results of the infit and outfit MnSqs and PCA in residuals. Our findings on unidimensionality were different from those in a previous study (Wang et al., 2012). In the previous study, one item (i.e., sociability) was removed in the Social Behavior subscale and the remaining items fit the model’s expectations. However, in that study, unidimensionality was not supported in the other two subscales.

A possible reason for the different results between our study and the previous study was the difference in the samples. The sample in the previous study was mixed, including patients with schizophrenia, mood disorder, and other mental disorders. The sample in our study was exclusively patients with schizophrenia. These inconsistent results among different populations indicate that our findings on unidimensionality should not be generalized to people with the other mental disorders.

The COTES–18 showed satisfactory unidimensionality for each subscale, indicating that each subscale measures a single construct. For clinical meaning of unidimensionality, the items’ scores in each subscale can be summed up to reflect the subscale-specific function. Clinicians and researchers can use the scores of the three subscales to understand subscale-specific functions of people with schizophrenia and follow up on progress of behaviors that affect occupational performance.

Convergent validity is defined as the degree to which the constructs that should be theoretically related correlate in reality (Chiu et al., 2014). Moderate correlations were found among the three COTES–18 subscales, which supports the theoretical framework (i.e., these subscales assess behaviors that affect occupational performance). The results of these correlations are similar to those in the previous COTES study (Kau et al., 1981). According to our findings, the COTES–18 has adequate convergent validity in people with schizophrenia.

Rasch reliability for the three COTES–18 subscales using the multidimensional approach was higher than that of the unidimensional approach. Future studies may increase the number of subscale items to increase the unidimensional reliability (i.e., achieve the same reliability level as the multidimensional approach). In clinical use, higher reliability means that clinicians and researchers can more precisely assess the abilities of people with schizophrenia. However, the increments are small (<6.0%). Possible reasons are that moderate correlations were found among the three subscales and that the number of COTES–18 subscales is small.

For the multidimensional approach, the higher the correlations among the subscales and the greater the number of subscales, the greater the precision will be (Wang & Chen, 2004). In the unidimensional approach, the Work Behavior subscale had high reliability (.90), necessary for individual comparisons of test scores (e.g., clinical application). The reliabilities of the other two subscales (.83–.85) are suitable for group comparisons of test scores. Therefore, the COTES–18 can be used to precisely assess behaviors that affect occupational performance in people with schizophrenia.

We deleted two misfitting items (i.e., the attendance item in the General Behavior subscale and the motivation item in the Work Behavior subscale). The possible reason for the misfitting of the attendance item is the concept of scale descriptors (frequency vs. capacity; Chien et al., 2005). The attendance item rates frequency of attending activities of occupational therapy. However, other items of the General Behavior subscale rate the capacity for participating in the activities. The reason for the misfitting of the motivation item may be a characteristic of our sample (i.e., inpatients in the acute phase). Inpatients in the acute phase have severe symptoms that are not associated with motivation (Fervaha et al., 2018; Kim et al., 2016). In this study, the motivation item showed relatively low correlation with the Work Behavior subscale compared with other items. In the acute phase, the motivation item may be excluded for people with schizophrenia. Further studies may reexamine the unidimensionality after revising the scale descriptor of the attendance item and recruiting participants from different phases.

Study Limitations and Future Research

Three limitations of this study are noted. First, we used secondary data from occupational therapy records. We could not ensure the fidelity of the ratings and the interrater reliability among the 29 occupational therapists. The quality of our data and consistency of ratings could cause concerns. Future research may include a prospective study with a few trained raters who reach interrater reliability (e.g., intraclass correlation coefficient >.80) to cross-validate our findings.

Second, our data were collected from inpatients with schizophrenia and evaluated using only Rasch analysis, restricting generalization to other populations (e.g., outpatients and healthy people). Future studies are warranted to recruit other populations to evaluate unidimensionality using confirmatory factor analysis and to identify the cutoff score using the receiver operating characteristic curve to discriminate between patients and healthy people.

Third, we could not collect certain information for some participants, such as whether they had early psychosis or established schizophrenia and the duration of untreated illness. The results of psychometric validations in participants with early psychosis could differ from results in those with established schizophrenia.

Implications for Occupational Therapy Practice

The results of this study have the following implications for occupational therapy practice:

The COTES–18 includes three subscales that provide a profile of clients’ behaviors that affect their occupational performance.

The three subscales showed a unidimensional construct, and subscale scores are suitable to represent subscale-specific functions in both clinical and research settings.

Good construct validity and Rasch reliability indicate that the COTES–18 appears to be useful for assessing behaviors that affect occupational performance, designing intervention programs, and following clients’ progress.

Conclusion

In this study, we deleted two items from the COTES, after which each of the three subscales showed a unidimensional construct. The COTES–18 had satisfactory convergent validity. Rasch reliability of the three subscales was sufficient in both unidimensional and multidimensional approaches for group or individual comparisons. Therefore, the COTES–18 appears to be a valid and reliable tool that can be used in occupational therapy practice to interpret the behaviors that affect engagement in occupation in clients with schizophrenia.

The COTES–18 provides structures for observing diverse patient behaviors that are addressed in treatment for mental illness. The observations made using the COTES–18 include a wide range of behaviors involving client factors and performance skills that affect the ability to effectively react to the demands of various occupations. Because the COTES–18 is a behavior rating scale, it can contribute bottom-up information to the treatment planning process. Moreover, it can be administered while observing a group of clients in an activity. It can be complementary to measures of occupational performance used in psychiatric settings such as the Kohlman Evaluation of Living Skills (Gary, 2011), the Independent Living Scales (Revheim & Medalia, 2004), and the Texas Functional Living Scale (Cullum et al., 2009). It is therefore worthy of consideration for use in occupational therapy with people with schizophrenia and other mental health disorders.

Footnotes

Acknowledgments

We are grateful to all participants for their involvement. This study was supported by a research grant from the Taipei City Hospital (TPCH-103-063), Taipei, Taiwan. Shu-Chun Lee and Ching-Lin Hsieh contributed equally to this study.

References

American Psychiatric Association. (2000). Diagnostic and statistical manual of mental disorders (4th ed., text rev.). Washington, DC: Author.

Azima

, & Azima

F. J.

(1959). Outline of a dynamic theory of occupational therapy. American Journal of Occupational Therapy, 13, 215–221.

Chen

F.-L.

Y.-J.

Shen

W. W.

, & Chen

C. C.

(2006). Improving psychiatric patients’ satisfaction by using a client-centered goal formulation structure. Journal of Taiwan Occupational Therapy Research and Practice, 2, 82–89.

Chien

C.-W.

Wang

J.-D.

Yao

K.-P. G.

C.-A.

, & Hsieh

C.-L.

(2005). Minnan version of determining scale descriptors of the interview questionnaire of WHOQOL. Formosan Journal of Medicine, 9, 584–594.

Chiu

E.-C.

Hsueh

I.-P.

Hsieh

C.-H.

, & Hsieh

C.-L.

(2014). Tests of data quality, scaling assumptions, reliability, and construct validity of the SF-36 health survey in people who abuse heroin. Journal of the Formosan Medical Association, 113, 234–241. https://doi.org/10.1016/j.jfma.2012.05.010

Chiu

E.-C.

Lee

S.-C.

Kuo

C.-J.

Lung

F.-W.

Hsueh

I.-P.

, & Hsieh

C.-L.

(2015). Development of a performance-based measure of executive functions in patients with schizophrenia. PLoS One, 10, e0142790. https://doi.org/10.1371/journal.pone.0142790

Chiu

E.-C.

Lee

Lai

K.-Y.

Kuo

C.-J.

Lee

S.-C.

, & Hsieh

C.-L.

(2015). Construct validity of the Chinese version of the Activities of Daily Living Rating Scale III in patients with schizophrenia. PLoS One, 10, e0130702. https://doi.org/10.1371/journal.pone.0130702

Cullum

Saine

, & Weiner

M. F.

(2009). Texas Functional Living Scale. London: Pearson.

Eklund

, & Leufstadius

(2007). Relationships between occupational factors and health and well-being in individuals with persistent mental illness living in the community. Canadian Journal of Occupational Therapy, 74, 303–313. https://doi.org/10.1177/000841740707400403

10.

Fervaha

Takeuchi

Foussias

Hahn

M. K.

Agid

, & Remington

(2018). Achievement motivation in early schizophrenia: Relationship with symptoms, cognition and functional outcome. Early Intervention in Psychiatry, 12, 1038–1044. https://doi.org/10.1111/eip.12405

11.

Fidler

, & Fidler

(1963). Occupational therapy: A communication process in psychiatry. New York: Macmillan.

12.

Gary

K. W.

(2011). Kohlman Evaluation of Living Skills. In Kreutzer

J. S.

DeLuca

, & Caplan

(Eds.), Encyclopedia of clinical neuropsychology (pp. 1410–1411). New York: Springer. https://doi.org/10.1007/978-0-387-79948-3_1841

13.

Hsiao

S.-L.

Pan

A.-W.

Chung

L.-I.

, & Lu

S.-J.

(2000). The use of evaluation tools in mental health occupational therapy: A national survey. Journal of Taiwan Occupational Therapy Association, 18, 19–32.

14.

Hsiao

Y.-Y.

Shih

C.-L.

W.-H.

Hsieh

C.-H.

, & Hsieh

C.-L.

(2015). Examining unidimensionality and improving reliability for the eight subscales of the SF-36 in opioid-dependent patients using Rasch analysis. Quality of Life Research, 24, 279–285. https://doi.org/10.1007/s11136-014-0771-z

15.

Hsueh

I.-P.

Wang

C.-H.

Liou

T.-H.

Lin

C.-H.

, & Hsieh

C.-L.

(2012). Test-retest reliability and validity of the comprehensive activities of daily living measure in patients with stroke. Journal of Rehabilitation Medicine, 44, 637–641. https://doi.org/10.2340/16501977-1004

16.

Kau

L.-J.

Yeh

E.-K.

Lin

M.-Y.

Chu

T.-F.

Chou

M.-H.

, & Li

S.-I.

(1981). A study with the revised Chinese version of Brayman’s Comprehensive Occupational Therapy Evaluation Scale. Chinese Journal of Psychology, 23, 1–7.

17.

Kim

J.-S.

Jang

S.-K.

Park

S.-C.

J.-S.

Park

J.-K.

Lee

J. S.

, … Lee

S.-H.

(2016). Measuring negative symptoms in patients with schizophrenia: Reliability and validity of the Korean version of the Motivation and Pleasure Scale–Self-Report. Neuropsychiatric Disease and Treatment, 12, 1167–1172. https://doi.org/10.2147/NDT.S107775

18.

Mosey

A. C.

(1971). Three frames of reference for mental health. Thorofare, NJ: Slack.

19.

Revheim

, & Medalia

(2004). The independent living scales as a measure of functional outcome for schizophrenia. Psychiatric Services, 55, 1052–1054. https://doi.org/10.1176/appi.ps.55.9.1052

20.

Smith

E. V. J.

Jr . (2002). Detecting and evaluating the impact of multidimensionality using item fit statistics and principal component analysis of residuals. Journal of Applied Measurement, 3, 205–231.

21.

Wang

I.-T.

Pan

A.-W.

Chung

Chen

T.-J.

Liu

L.-T.

, & Chen

Y.-L.

(2012). Psychometric properties of the Comprehensive Occupational Therapy Evaluation Scale for persons with mental illness. Formosan Journal of Medicine, 16, 121–128.

22.

Wang

W.-C.

, & Chen

P.-H.

(2004). Implementation and measurement efficiency of multidimensional computerized adaptive testing. Applied Psychological Measurement, 28, 295–316. https://doi.org/10.1177/0146621604265938