Abstract
The authors’ review of the evidence using the COSMIN Risk of Bias checklist suggests that the Canadian Occupational Performance Measure lacks high-quality validation.
Patient-reported outcome measures (PROMs) are important in evaluating the value of health care. Engaging in a critical self-analysis of an illness experience helps clients identify and quantify changes in their health (Lloyd et al., 2019; Sloan et al., 2007), and capturing this information assists health care organizations in determining whether health resources are efficiently allocated (Black, 2013). Several PROMs have been developed for assessing people who have undergone rehabilitation. A systematic review by Chen et al. (2013) identified several PROMs that are routinely used in clinical practice or research with rehabilitation clients; however, the quality of evidence these measures provide has not been established. To implement evidence-based practice, researchers must evaluate the measurement properties of PROMs, such as reliability, validity, and responsiveness, so that practitioners can select appropriate measures for the targeted client population. The COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) initiative developed guidelines to assess the measurement properties of PROMs (Mokkink et al., 2006, 2010); these guidelines were revised in 2018 (Mokkink, De Vet, et al., 2018; Terwee et al., 2018).
The widely used Canadian Occupational Performance Measure (COPM) is a PROM occupational therapy practitioners use to assess clients’ occupational performance and satisfaction and the effectiveness of intervention (Law et al., 1994). The COPM was designed to capture a client’s self-perceived performance level and level of satisfaction with their performance over time (Law et al., 1998). The COPM is classified as an “individualized” PROM; it allows clients to select their own personal concerns rather than choosing from a predetermined list of concerns in a questionnaire (Deshpande et al., 2011).
In the COPM semistructured interview, the practitioner first asks the client about three areas of occupational performance—self-care, productivity, and leisure—taking into account the client’s developmental stage and life circumstances. The client then identifies specific occupational problems, rates the importance of each occupation in their life using a 10-point scale (Law et al., 1998), and selects up to five most pressing problems. The client rates each of the identified problems, again using a 10-point scale, on the basis of their current performance and satisfaction with their performance; these ratings are used to calculate their scores on the COPM Performance (COPM–P) and COPM Satisfaction (COPM–S) subscales. Originally published in 1991, the COPM has been used in approximately 40 countries and translated into more than 35 languages (Law et al., 1998). It is sufficiently flexible for use in various clinical settings with no limitations on target population. It has been a primary outcome measure in many randomized controlled trials (Sturkenboom et al., 2014) and is a gold-standard outcome measure in global clinical research.
The measurement properties of the COPM have been examined in a variety of settings. In 2004, Carswell et al. reviewed the use of the COPM in client-centered research and practice and found that the COPM was reliable, valid, and responsive. However, the authors of this review did not apply rigorous systematic methodology, and an updated review is needed. To judge the quality of the COPM, occupational therapy practitioners can benefit from a summary of the results of the previous studies that is guided by strict criteria. Our primary aim in this systematic review was to identify, critically appraise, and synthesize all available studies examining the psychometric properties of the COPM using the updated COSMIN methodology.
Method
Design
We conducted and are reporting this systematic review according to the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P) guidelines (Moher et al., 2015). We used the standards for assessing systematic reviews set forth in AMSTAR (A MeaSurement Tool to Assess systematic Reviews; Shea et al., 2007).
Study Eligibility Criteria
Studies were included in this systematic review if they evaluated at least one measurement property that appears on the COSMIN Risk of Bias checklist, which includes content validity, structural validity, internal consistency, cross- cultural validity, reliability (i.e., test–retest reliability), measurement error, criterion validity, hypothesis testing for construct validity, and responsiveness; we also included development studies for the COPM. We excluded abstracts, dissertations and theses, unpublished data, supplementary files, commentaries, and studies in which quantification of COPM properties was not the primary goal. We did not restrict database searches by date or publication language.
Literature Search Strategy
We conducted comprehensive searches of MEDLINE, Web of Science, Scopus, OTseeker, and the Cochrane Library from their inception to June 6, 2019. The search terms were “Canadian Occupational Performance Measure” OR “COPM.” We also manually searched the reference lists of all included studies and systematic reviews. Studies published in languages other than English were translated using Google Translate (Google, Mountain View, CA).
Study Selection
One author (Kanta Ohno) screened the titles and abstracts for eligibility, and two authors (Kanta Ohno and Kounosuke Tomori) retrieved the full text of all relevant and potentially relevant articles. They resolved disagreements at each level of screening through consensus discussion and the involvement of a third author if required. We ensured that data from duplicate publications were entered only once in our review, and we tabulated the excluded studies and the reasons for excluding them.
Data Extraction
For each article, we extracted information on sample size, target population, participant age and sex, country and language, and measurement property assessed and associated results. When data were missing from the full-text article, we contacted the authors for clarification and further details.
Ratings of Individual Studies
Methodological Quality of the Study
Two authors, Kanta Ohno and Kounosuke Tomori, reviewed the full-text articles and assessed the studies using the updated COSMIN Risk of Bias checklist (Mokkink, Prinsen, et al., 2018). We rated the methodological quality of each study as very good, adequate, doubtful, or inadequate.
Quality of Each Measurement Property
For studies that examined multiple measurement properties, we examined each property separately. We determined the rating for each property using the “worst score counts” principle (i.e., by taking the lowest rating) and developed narrative summaries of the measurement property types, measurement coefficients, and study samples. Following the updated COSMIN criteria, we rated the quality of each measurement property as sufficient, insufficient, inconsistent, or indeterminate. Explanations for inconsistent results require further exploration (Prinsen et al., 2018; Terwee et al., 2007) and are beyond the scope of the current study. Considerations in assessing the measurement properties were as follows:
Content validity includes three aspects: (1) relevance (all items in a PROM should be relevant to the construct of interest within a specific population and context of use), (2) comprehensiveness (no key aspects of the construct should be missing), and (3) comprehensibility (clients should understand the items as intended; Terwee et al., 2018).
Construct validity includes convergent validity and discriminant validity. For convergent validity, r was expected to be >.50 for correlations between the comparator instrument and the COPM on similar constructs (i.e., occupational performance). For discriminant validity, r was expected to be <.50 for correlations with the comparator instrument and the COPM on different constructs. We rated construct validity as sufficient if ≥75% of the results were in accordance with the hypotheses, insufficient if ≥75% of the results were not in accordance with the hypotheses, and indeterminate if no hypotheses were defined. For the other measurement properties, we rated the results according to the consensus-based criteria proposed by Prinsen et al. (2018).
Quality of the Evidence
We graded the quality of evidence supporting each measurement property as high, moderate, low, or very low using a modified version of the Grading of Recommendations Assessment, Development, and Evaluation approach (Terwee et al., 2018), taking into account risk of bias (methodological quality of the study), inconsistency of results across studies, and imprecision (total sample size of the included studies). Indirectness (evidence from different populations) was not applicable in this review because the COPM was developed without a specific target population or context of use.
Pooled Ratings
Finally, we rated each measurement property in two ways: (1) the overall rating, a summary of the quality of the measurement property based on evidence from all available studies (sufficient, insufficient, inconsistent, or indeterminate) and (2) the quality of evidence available for each property (high, moderate, low, or very low). Two authors (Kanta Ohno and Kounosuke Tomori) independently performed the analysis from data extraction to grading of the quality of evidence, and three authors (Kanta Ohno, Kounosuke Tomori, and Ayana Yaguchi) conferred to produce the final overall ratings of the quality of each measurement property and the supporting evidence.
Results
The literature search yielded 1,611 publications. After removing duplicates, we screened 766 studies on the basis of their titles and abstracts and eliminated irrelevant studies. Thirty-five studies remained, of which 2 were excluded after full-text screening, and we identified 2 additional studies in a manual search of reference lists, for a final total of 35 studies (Figure 1).
Flow diagram of the study selection process for the systematic review.
Of the 35 studies, 32 examined measurement properties and are summarized in Table A.1 in the Appendix, and 3 were development studies (Law, 1987; Law et al., 1994; Pollock et al., 1990). Participants’ ages varied; 3 studies involved children with disabilities and their parents (Cusick et al., 2006, 2007; Verkerk et al., 2006). With regard to diagnoses, 14 studies investigated a particular diagnosis (e.g., stroke, rheumatoid arthritis, spinal cord injury, schizophrenia), 4 studies covered multiple diagnoses (e.g., neurological dysfunction and psychiatric disorders), 10 studies did not specify diagnoses (i.e., participants had various diagnoses), and 4 studies analyzed pediatric clients who had occupational problems. The studies were conducted in Europe (n = 16), Asia (n = 4), Oceania (n = 4), the United States (n = 2), and South America (n = 1) as well as in Canada (n = 5), where the COPM was developed.
COSMIN Risk-of-Bias Assessment
Table A.2 in the Appendix shows the overall ratings for each measurement property and the quality of the supporting evidence. No studies assessed the structural validity of the COPM. The quality of the three studies that reported on the development of the COPM (Law, 1987; Law et al., 1994; Pollock et al., 1990) was rated as doubtful because clients were not involved in the concept elicitation process. In addition, the pilot tests did not include questions about the relevance and comprehensiveness of the COPM.
Content Validity
Only 1 study asked clients about the relevance, comprehensiveness, and comprehensibility of the COPM; the quality of this evidence was inadequate (Larsen et al., 2019). Two studies asked health professionals about the relevance, comprehensiveness, and comprehensibility of the COPM (Chan & Lee, 1997; Larsen et al., 2019). These studies provide very low quality evidence for inconsistent relevance, low-quality evidence for inconsistent comprehensiveness, and moderate-quality evidence for sufficient comprehensibility. Overall, low-quality evidence indicates that the content validity of the COPM is inconsistent.
Internal Consistency
Two studies of doubtful quality assessed the internal consistency of the COPM’s three item categories (self-care, productivity, and leisure), both reporting Cronbach’s αs of >.70 (Cusick et al., 2007; Spadaro et al., 2010). Because neither study assessed the unidimensionality of the COPM, these two studies do not provide evidence in support of internal consistency according to recent COSMIN guidelines. Therefore, low-quality evidence with serious risk of bias and imprecision indicates that the COPM has indeterminate internal consistency.
Cross-Cultural Validity
In 1 study, the COPM was translated into Dutch from English (Larsen et al., 2019). The authors conducted posttranslational pilot testing but did not validate the translation using exploratory factor analysis, regression analysis, or item response theory. On the basis of the COSMIN guidelines, very low quality evidence with serious risk of bias indicates that the cross-cultural validity of the COPM is indeterminate.
Reliability
Eight studies assessed reliability (i.e., test–retest reliability), and 6 provided intraclass correlation coefficients (ICCs). Four studies indicated sufficient reliability (Kjeken et al., 2005; Pan et al., 2003; Sewell & Singh, 2016; Spadaro et al., 2010), and 2 studies indicated insufficient reliability (Chaves et al., 2016; Eyssen et al., 2005); however, the results showing insufficient reliability (.66 for COPM–P and .67 for COPM–S in Chaves et al.; .67 for COPM–P and .69 for COPM–S in Eyssen et al.) were close to the COSMIN-specified sufficiency threshold of >.70. One study found Spearman’s correlation coefficients of .89 for COPM–P and .88 for COPM–S (Cup et al., 2003); another found Pearson’s correlation coefficients of .80 for COPM–P and .84 for COPM–S (Atashi et al., 2010). Overall, moderate-quality evidence indicates that the COPM has inconsistent reliability because of serious inconsistency in some studies.
Measurement Error
Three studies assessed measurement error (Eyssen et al., 2005; Kjeken et al., 2005; Verkerk et al., 2006). All 3 calculated limits of agreement (LoA) but did not validate minimally important change (MIC). If LoA and MIC for the same population are not defined, the quality of evidence is indeterminate, according to the COSMIN guidelines. Therefore, moderate-quality evidence indicates indeterminate measurement error for the COPM.
Criterion Validity
Criterion validity was reported in 13 studies (Boyer et al., 2000; Carpenter et al., 2001; Donnelly et al., 2004; Harper et al., 2006; Hill et al., 2014; Kjeken et al., 2004; McColl et al., 2000; Ripat et al., 2001; Rochman et al., 2008; Spadaro et al., 2010; van de Ven-Stevens et al., 2015; Verkerk et al., 2006; Walsh et al., 2004). Participant diagnoses in studies that specified them included rheumatoid arthritis (Ripat et al., 2001), schizophrenia (Boyer et al., 2000), chronic pain (Carpenter et al., 2001), and stroke (Hill et al., 2014). Twenty-three comparator instruments were used as the gold standard for criterion validity; for example, the FIM® was used in a study of people with spinal cord injury (Donnelly et al., 2004), and the Disabilities of the Arm, Shoulder, and Hand questionnaire was used in a study of people with hand injury (van de Ven-Stevens et al., 2015).
We rated 11 studies as having very good methodological quality and 2 as having inadequate quality because of issues related to correlation methods or noncalculation of areas under the curve (Hill et al., 2014; Verkerk et al., 2006). Only 1 study found a correlation coefficient of .70 (Rochman et al., 2008). Therefore, high-quality evidence indicates that the COPM has insufficient criterion validity.
Hypothesis Testing for Construct Validity
In 11 studies, the quality of the construct validity was very good; however, the comparator measures and study samples varied, as did the criterion validity. Eight studies assessed convergent validity (Cusick et al., 2007; Dedding et al., 2004; Kjeken et al., 2004; McColl et al., 2000; Nieuwenhuizen et al., 2014; Roe et al., 2020; Spadaro et al., 2010; Thyer et al., 2018), 2 assessed discriminant validity (Cup et al., 2003; Tuntland et al., 2016), and 1 assessed both convergent and discriminant validity (Chan & Lee, 1997). All hypotheses for convergent validity were supported, whereas none of the hypotheses for discriminant validity were supported. The evidence for inconsistent construct validity is of moderate quality because results varied considerably across the 11 studies.
Responsiveness
Ten studies analyzed responsiveness by comparing the COPM scores of patients at admission, before discharge, and at discharge (Chen et al., 2002; Cusick et al., 2007; Eyssen et al., 2011; Kjeken et al., 2004; Nieuwenhuizen et al., 2014; Roe et al., 2020; Spadaro et al., 2010; Tuntland et al., 2016; Walsh et al., 2004; Wressle et al., 1999). Moderate-quality evidence indicates inconsistent responsiveness for the COPM.
Minimally Important Change
The COPM manual states that MIC for both COPM–P and COPM–S is 2.0 points. However, the value of the MIC cut point is questionable because the manual does not describe the method for determining it in detail (Law et al., 1998). Previous studies reported MICs for COPM–P and COPM–S of 1.37 and 1.90 points in outpatients with various diagnoses (Eyssen et al., 2011) and 1.37 and 1.90 points in home-dwelling older adults (Tuntland et al., 2016).
Discussion
The COPM has been used as a patient-reported outcome measure to assess clients with a variety of conditions in different clinical settings and countries. However, the quality of the COPM has never been reviewed with rigorous methodology. We evaluated the available evidence for the COPM’s measurement properties using the updated COSMIN methodology and found little high-quality evidence. The results of 35 studies indicate that overall, the COPM’s measurement properties are inconsistent.
Validity of the COPM
High-quality evidence from 13 studies assessing validity indicate that the COPM’s criterion validity is insufficient. Only 1 of the studies found a correlation coefficient that exceeded the COSMIN-specified threshold of r > .70; the others did not meet this criterion for either the COPM–P or COPM–S. The COSMIN developers defined criterion validity as “the degree to which the scores of a PROM are an adequate reflection of a gold standard” (Mokkink, Prinsen, et al., 2018). The methods used in the studies to investigate criterion validity were most likely unsuitable for the COPM because items in the self-care, productivity, and leisure subcategories, although comprehensive, are unique to each client and because the interactive process between practitioner and client of selecting items may result in changes to the items over the course of intervention. In this systematic review, 23 comparator instruments were used as the gold standard for analysis, and most of these instruments focused on functional impairment and activity limitation separately.
The COSMIN guidelines, which are used in the development PROMs, highlight the importance of content validity and of asking clients and practitioners for their input early in the development process (Terwee et al., 2018). Unlike instruments that have fixed items, the COPM has each client generate items in collaboration with the practitioner, and COPM items are thus potentially unique to each client. Four studies investigated whether the items identified by the COPM classified into the self-care, productivity, and leisure domains (Eyssen et al., 2005; Kjeken et al., 2005; Law et al., 1994; Verkerk et al., 2006). However, these studies did not analyze client-chosen occupations that reflected the client’s true intention. Although the COPM interview process would appear to ensure the relevance, comprehensiveness, and comprehensibility of the items, observational and qualitative studies of goal setting have shown that perceptual gaps exist between clients and practitioners. For example, in a qualitative analysis of perceptions of goal setting in clients with stroke, Brown et al. (2014) found that some did not clearly express their desires during the client-centered goal-setting process, and Saito et al. (2021) reported that the ratio of concordance between the rehabilitation goals of independent occupational therapists and of clients was only 21%. Therefore, further content validity studies are needed to evaluate whether COPM items match the client’s true priorities in the goal-selection process.
Reliability and Responsiveness of the COPM
Reliability is the extent to which scores on a given construct are stable between repeated measurements (Prinsen et al., 2016, 2018). Six studies tested the reliability (test–retest reliability) of the COPM using ICCs. Four found ICCs that met the COSMIN criterion of ≥.70, and the other two found ICCs of .66–.69 (Prinsen et al., 2016, 2018). Hence, the COPM’s reliability has shown almost high quality for repeated measurements if the client’s situation remains stable.
The COSMIN guidelines define responsiveness as “the ability of a PROM to detect change over time in the construct to be measured” (Mokkink, Prinsen, et al., 2018, p. 60). Only 2 of 10 studies showed results for responsiveness that met the COSMIN criteria. Thus, the COPM may not be able to detect change in clients between pre- and postintervention. In addition, any changes in the client’s items and priorities during the intervention process may cause problems for the use of the COPM as an outcome measure.
The concept of response shift in PROMs has attracted attention to the potential consequences of changes in the meaning or priority clients attribute to items or component domains. Response shift may be caused by three factors: (1) changes in the instrument’s internal standards (i.e., recalibration), (2) changes in the respondent’s priorities (i.e., reprioritization), and (3) changes in the definition of the targeted construct (i.e., reconceptualization; Sprangers & Schwartz, 1999). Neither the COPM manual nor the articles included in this review mentioned response shift or described steps to resolve related issues. We were unable to determine whether response shift affected the results of this study; further study of the reliability and responsiveness of the COPM as an outcome measure is needed that considers the effect of response shift on the instrument’s ability to detect change between pre- and postintervention.
Study Limitations
A limitation of this systematic review is the potential for incomplete retrieval of relevant studies. The COPM is used worldwide, and the results of studies in languages other than English may have added additional insights. Moreover, studies published after June 6, 2019, are not included in this review.
Implications for Occupational Therapy Practice
This systematic review summarizes the quality of evidence for the measurement properties of the COPM. The primary implications for occupational therapy practice and research are as follows: Despite use of the COPM with a broad range of target populations and settings, no studies have provided high-quality evidence of sufficient measurement properties. Because no studies examined the validity of the item selection process, clinicians should carefully consider whether the item identified by the COPM is an occupation that the patient truly wants to achieve. When using the COPM as an outcome measure, clinicians should consider that the COPM may not be able to detect change over time.
Conclusion
High-quality validation of the COPM is lacking. Further examination of its measurement properties using updated relevant guidelines is required.
Footnotes
*
Indicates studies included in the scoping review.
Acknowledgment
This systematic review was prospectively registered with the International Prospective Register of Systematic Reviews (CRD4 2019138103).
Appendix. Supplemental Tables
Summary of Findings
| Measurement Property | No. of Studies (No. of Participants) | Summary of Pooled Results | Overall Rating of Measurement Property | Quality of Evidence |
|---|---|---|---|---|
| Content validity | 4 (343 patients, 15 occupational therapists, 42 parents) | Both the development study and the content validity did not examine all of the COSMIN criteria and indicated indeterminate quality. | Inconsistent | Low: very serious risk of bias (multiple studies of inadequate quality) |
| Relevance | 1 of 4 interviewed both of the patients and the professionals for relevance. | Inconsistent | Very low: extremely serious risk of bias (only one study of inadequate quality and one of doubtful quality) | |
| Comprehensiveness | 1 of 4 interviewed both of the patients and the professionals for comprehensiveness | Inconsistent | Low: very serious risk of bias (multiple studies of inadequate quality) | |
| Comprehensibility | 1 of 4 indicated adequate quality using face-to-face interview | Sufficient | Moderate: serious risk of bias (only 1 study of adequate quality) | |
| Structural validity | 0 (0) | NA | NA | NA |
| Internal consistency | 2 (72) | Cronbach’s αs: COPM–P total, .78–.86; COPM–S total, .79–.88 | Indeterminate | Low: serious risk of bias (multiple studies of doubtful quality), serious imprecision |
| Cross-cultural validity | 1 (37 patients, 15 occupational therapists) | No multiple group factor analysis or different item functioning analysis performed | Indeterminate | Very low: very serious risk of bias (only 1 study of doubtful quality), serious imprecision |
| Reliability | 6 (324) | Intraclass correlation coefficients: COPM–P, .66–.92; COPM–S, .67–.93 | Inconsistent | Moderate: 1 very good and multiple adequate studies, inconsistent results |
| 1 (60) | Spearman’s correlation coefficients: COPM–P, .89; COPM–S, .88 | |||
| 1 (26) | Pearson’s correlation coefficients: COPM–P, .80; COPM–S, .84 | |||
| Measurement error | 1 (40) | Limits of agreement, patients with ankylosing spondylitis: COPM–P, –1.82 to +1.12 (SD = 1.47); COPM–S, –1.62 to +1.98 (SD = 1.80) | Indeterminate | Moderate: 1 very good study, serious inconsistency |
| 1 (80) | Limits of agreement, parents of children with various diagnoses: COPM–P, –2.4 to +2.3 (SD = 1.2); COPM–S, –2.3 to +2.6 (SD = 1.2) | |||
| 1 (95) | Limits of agreement, patients with various diagnoses: COPM–P, –2.5 to +2.4 (SD = 1.2); COPM–S, –2.3 to +2.7 (SD = 1.4) | |||
| Criterion validity | 13 (814) | 1 of 61 comparator instruments was correlated with COPM–P ≥.70 (22 of 61 NS) | Insufficient | High: multiple very good studies |
| 1 of 55 comparator instruments was correlated with COPM–S ≥.70 (16 of 55 NS) | ||||
| Hypothesis testing for construct validity | 9 (508) | Quantitative pooling for convergent validity (pooled correlation coefficients): 0 of 9 hypotheses supported | Inconsistent | Moderate: multiple very good studies, serious inconsistency |
| 3 (290) | Quantitative pooling for discriminant validity (pooled correlation coefficients): 3 of 3 hypotheses supported | |||
| Responsiveness | 10 (784) | 2 of 9 hypotheses supported (1 study did not define a hypothesis) | Inconsistent | Moderate: multiple very good studies, serious inconsistency |
Note. COPM–P = Canadian Occupational Performance Measure, Participation subscale; COPM–S = Canadian Occupational Performance Measure, Satisfaction subscale; NA = not available; NR = not reported; NS = not significant.
Citation: Ohno, K., Tomori, K., Sawada, T., Seike, Y., Yaguchi, A., & Kobayashi, R. (2021). Measurement properties of the Canadian Occupational Performance Measure: A systematic review (Table A.2). American Journal of Occupational Therapy, 75, 7506205100. https://doi.org/10.5014/ajot.2021.041699