Abstract
The results of this study highlight the need to establish more accurate measurement methods to reduce the impact of response shifts on scores from the Canadian Occupational Performance Measure.
The concepts of patient-centered treatment (Stewart, 2001) and patient engagement (Miller, 2016; Murali & Deao, 2019) are gaining in popularity and replacing physician-led health care. Patients are encouraged not only to receive medical care but also to actively participate in their medical care. Therefore, patient-reported outcome measures (PROMs), defined as instruments that address “any aspect of a patient’s health status that comes directly from the patient without the interpretation of the [patient’s] responses by a physician or anyone else” (FDA Center for Drug and Evaluation Research et al., 2006) have become a critical end-point indicator of the effectiveness of medical treatment (Ahmed et al., 2012; Deshpande et al., 2011; Sawatzky et al., 2017).
The Canadian Occupational Performance Measure (COPM) is a well-known PROM commonly used in occupational therapy practice and research worldwide (Law et al., 1990; Law, Baptiste, et al., 1998; McColl et al., 2000). It is based on the Canadian Model of Occupational Performance and has been expanded to include the Canadian Model of Occupational Performance and Engagement, both of which are grounded in client-centered theory (Law et al., 1990). In client-centered theory, emphasis is placed on maintaining a perspective centered around the client throughout the practice process (Kyler, 2008; Law, Darrah, et al., 1998; Townsend et al., 1990). However, clinicians and researchers must be concerned with interpreting COPM results as a PROM. PROMs can be classified into several types on the basis of their measurement of the concept of interest (Deshpande et al., 2011). The COPM is classified as “individualized,” allowing respondents to select their own personal interests, unlike measures that use predetermined personal interests in the list of questionnaire items (Deshpande et al., 2011). In the COPM procedure, the client is first asked about their occupational performance in three areas: self-care, productivity, and leisure (Law et al., 1990). However, clients have the autonomy to freely define specific names for questionnaire items. Thus, there is a potential for the client’s perception to evolve over time. Moreover, the possibility of risks to the accuracy of COPM scoring because of information bias has also been reported in the assessment of temporal points. Sawada et al. (2022) explored the impact of information bias on COPM scores through a qualitative study of subacute rehabilitation hospital inpatients that generated potential information biases in the process of selecting and identifying occupations and accurately quantifying the respondents’ perceptions of scoring. In other words, it is thought that changes in the Internal criteria of respondents’ self-perceptions can easily occur on the COPM because of its structural characteristics, in which questionnaire items are not specified.
To advance evidence-based practice, it is essential for practitioners and researchers to meticulously evaluate the measurement properties of PROMs, including reliability, validity, and responsiveness. The cOnsensus-based Standards for the selection of health Measurement iNstruments (COSMIN) initiative serves as a pivotal guideline in this pursuit (Mokkink et al., 2006, 2010). Of note is that the COSMIN initiative dedicated to refining the assessment of PROMs underwent significant enhancements in 2018, solidifying its status as a cornerstone resource in occupational therapy (Mokkink et al., 2018a, 2018b; Terwee et al., 2018). A systematic review conducted by Ohno, Tomori, Sawada, Seike, et al. (2021) found that none of the measurement properties met the COSMIN criteria for the COPM. In particular, only 2 of 10 studies met the COSMIN criteria for responsiveness (Ohno, Tomori, Sawada, Seike, et al., 2021), defined as “the ability of a PROM to detect change over time in the construct to be measured” (Mokkink et al., 2018b, p. 12; Mokkink et al., 2018a ; Prinsen et al., 2018; Terwee et al., 2018). The longitudinal changes in PROMs are based on the assumption that respondents’ interpretations of constructs are stable over time (Sajobi et al., 2018). Therefore, the COPM may not adequately reflect changes in the respondent’s condition over time.
A person’s self-assessment of their health status and quality of life (QOL) may not be the same at different points in time, and this may affect PROM results (Schwartz et al., 2006; Sprangers & Schwartz, 1999). This is phenomenon is known as a response shift (RS), and it is an evaluation bias specific to PROMs (Schwartz et al., 2006; Sprangers & Schwartz, 1999). Sprangers and Schwartz (1999) identified three types of RS: recalibration, reprioritization, and reconceptualization. Recalibration refers to a change in an individual’s internal standards of measurement, reprioritization refers to a change in the person’s values (changes in the importance of the items or component domains that constitute the QOL concept), and reconceptualization refers to a change in the definition of the targeted construct (redefinition QOL; Sprangers & Schwartz, 1999).
The concept of RS was first proposed in the area of management science in the 1970s (Golembiewski et al., 1976; Howard et al., 1979), followed by a report on RS in the related area of health care in 1991 (Breetvelt & Van Dam, 1991). The theoretical model of RS occurring in response to catalysts was described by Sprangers et al. in 1999. Of note is that the COPM, in its manual (Law, Baptiste, et al., 1998) and development study (Law, 1987, 1994), did not initially address the presence of RS or provide established methods to mitigate its effects. Therefore, this study was positioned as a pilot investigation; we sought to meticulously examine, using a convergent mixed-methods approach, changes in the perceptions of occupational performance over time (i.e., RS) in adult patients with neurological and musculoskeletal disorders who had been admitted to a subacute rehabilitation hospital. The objective of Study 1 was to comprehensively understand the occurrence, classification, and structure of RS in the COPM (QUAL). In Study 2, we aimed to assess the impact of RS on COPM scores (QUAN).
Method
Study Design and Ethics
In this study, we took a convergent mixed-methods research (QUAL + QUAN) approach to qualitatively and quantitatively reveal RS effects in the COPM. The qualitative analysis phase (Study 1) consisted of inductive content analysis using semistructured individual interviews. Study 1 followed the Consolidated Criteria for Reporting Qualitative Research (Tong et al., 2007). The quantitative research phase (Study 2) included the calculation of descriptive statistics of COPM scores before and after RS correction. Ethical approval was obtained from the Tokyo University of Technology Ethics Committee (E19HS-006). All patients were provided with a detailed explanation of the research objectives and procedures before providing their consent. Consent was obtained through the patient’s voluntary signature on the provided consent form. After the research explanation, a period of 1 to 2 days was allowed for the consent-acquisition process. It is important to note that patients had the right to withdraw their consent at any point during the study. All data were stored securely, and patient confidentiality was maintained. To ensure patient comfort, the interviews were conducted in a private setting, such as a private room or ward cafeteria, where there were no other people around. Interviews were recorded using an integrated circuit recorder.
Participants
We enrolled adult inpatients at a 150-bed subacute rehabilitation hospital in Japan between May 2020 and September 2020. In Japanese subacute rehabilitation hospitals, the mean onset-admission interval is 31.5 days (SD = 18.6), the mean length of stay is 75.9 days (SD = 46.1), and 69.1% of inpatients are discharged to their homes (Miyai et al., 2011). The inclusion criteria were as follows: (1) admitted to the subacute rehabilitation hospital because of stroke, orthopedic disease, or disuse syndrome (a condition that is caused by a lack of physical activity, secondary to pneumonia and coronary occlusion); (2) received client-centered occupational therapy from occupational therapists; and (3) age ≥18 yr; and (d) Mini-Mental State Examination (MMSE; Folstein et al., 1975) score ≥20. Patients with severe cognitive deficits due to dementia or aphasia were excluded from the study. In general, an MMSE score <24 indicates the presence of cognitive impairment. However, in rehabilitation practice occupational therapy interviews are conducted even with individuals exhibiting mild cognitive impairments (Dick et al., 1984). Therefore, for the comprehensive criteria of this study we set a minimum MMSE score of 20 and included patients we deemed capable of practical communication.
The inclusion criteria for the occupational therapists were as follows: (1) had a minimum of at least 1 yr of full-time clinical experience as an occupational therapist; (2) had read the COPM manual (Law, Baptiste, et al., 1998); and (3) had completed a total of 15 COPM pre–post administrations with patients. Occupational therapists who met the inclusion criteria received lectures on the Canadian Model of Occupational Performance and Engagement, and scoring and interpretation of the COPM was conducted by Kanta Ohno.
The number of participants was not predetermined; however, sampling was conducted sequentially and purposively until theoretical saturation was reached. In this study, theoretical saturation was defined as a state in which no new codes could be extracted from the interview transcripts when the thematic content (hereafter referred to as “codes”) continued to be examined.
Data Collection: Studies 1 and 2
Ohno interviewed the patients within 24 hr of the completion of the COPM by the occupational therapist in charge of the patients (Time 1 [T1]). For the interviews, we used an interview guide developed with reference to previous studies of goal-setting methodologies (O’Neill & Harris, 1982) and previous studies (Ock et al., 2016) that had aimed to qualitatively capture changes in respondents’ perceptions of the occupations identified in the COPM and to understand respondents’ self-perceptions in detail.
The interview guide consisted of five “Ws” and “How,” which are defined as follows: (1) “What is the name of the occupation (title)?,” (2) “Who does the occupation (respondent)?,” (3) “When do you do the occupation (duration/frequency)?,” (4) “Where do you do the occupation (place)?,” (5) “Why do you do the occupation (reason/purpose)?,” and (6) “How do you perform the occupation (methods/procedure/means)?” (Figure 1). For example, the occupation of shopping (What) is recorded as “I want to go shopping alone (Who) to buy groceries (Why) at a supermarket that is a 10-minute walk from my house (Where), once every two days (When) by bicycle (How).” This interview guide was used to identify patients’ perceptional changes in their occupational performance (RSs). Therefore, it cannot be assumed that all items based on the five Ws and How were documented in the COPM data sheet by the occupational therapist who conducted the assessment. To ensure the interrater reliability of this unique interviewing method, all interviewers received comprehensive training from Ohno regarding the interviewing approach. In addition, interviews were conducted using a sheet that listed the five Ws and How.
Data collection protocols.
Several studies have adopted various RS detection methods, such as the then-test, structural equation modeling (SEM), group-based trajectory analysis of residuals, and relative importance analysis (Sajobi et al., 2018). In particular, SEM is an accurate statistical method for detecting the RS effects of PROMs (Sajobi et al., 2018). However, it is difficult to adapt SEM because the COPM’s items are determined by the respondent (i.e., classified as “individualized”; Deshpande et al., 2011). In the current study, we adapted the original method using the five Ws and How as an interview guide.
At the time of reassessment (Time 2 [T2]), as well as the initial assessment, an interview guide (five Ws and How) was used to confirm the patients’ perceptions of each of the identified occupations. Interviews were conducted twice for each patient in this study (T1 and T2). The same interviewer conducted the initial and reassessment interviews.
We assessed patients’ level of independence in activities of daily living using the FIMTM at the initial assessment (T1). 1 The FIM is an 18-item assessment tool designed to quantify an individual’s level of care-related disability. Each item is scored on a scale that ranges from 1 (indicating the need for complete assistance) to 7 (representing complete independence), resulting in a total score ranging from 7 to 126. The FIM yields three distinct scores: (1) a total score (FIM–Total: all 18 items); (2) a motor score (FIM–Motor, covering activities such as eating, grooming, bathing, upper and lower body dressing, toileting, bladder and bowel management, and various transfers); and (3) a cognitive score (FIM–Cognitive, comprising tasks related to auditory comprehension, verbal expression, social interaction, problem solving, and memory; Ottenbacher et al., 1996).
Data Analysis
Study 1: Qualitative Analysis Phase
We conducted an inductive content analysis to analyze the data obtained from the interviews (Braun & Clarke, 2006; Salsbury et al., 2018). Verbatim transcripts were prepared from recorded data from the initial assessment (T1) and reassessment interviews (T2).
We compared each of the five Ws and How items for each identified occupation between the initial assessment (T1) and reassessment (T2), and if even one item was changed, we determined that an RS had occurred. For example, during the initial assessment (T1) the patient may have responded to the interview about the occupation of shopping (What) as follows: “I want to go shopping alone (Who) to buy groceries (Why) at a supermarket that is a 10-minute walk from my house (Where), once every two days (When), and by bicycle (How).” When the reassessment interview (T2) response changed to “I want to go shopping with my husband (Who) to buy groceries (Why) at a supermarket that is a 10-minute walk from my house (Where), once a week (When) and walking with a cane (How),” we determined that an RS occurred with the three items of Who, When, and How (T1 vs. T2).
The RSs that occurred were categorized according to the five Ws and How and assigned a code reflecting the content. We then created primary categories by integrating the primary categories on the basis of the intrinsic similarity of the codes. We created secondary categories by integrating primary categories on the basis of semantic content similarity. In the analysis process, semantic content similarity entails integrating primary categories on the basis of their underlying meaning or content while ensuring mutual exclusivity (Pope & Mays, 1995). For example, subcategories such as “Adding walking aid” and “Adding alternative means” may emerge. These subcategories are then consolidated under the overarching category of “Adding (How),” ensuring that each category is distinct and nonoverlapping and collectively addressing the concept of substitution in the rehabilitation process.
Data analysis was conducted independently by Ohno and Tomori, who are faculty members in an occupational therapy education program and have published articles on COPM and goal-setting (Ohno, Tomori, Sawada, & Kobayashi, 2021; Ohno, Tomori, Sawada, Seike, et al., 2021; Saito et al., 2019; Sawada et al., 2020, 2022). The research team critically examined the analysis and made repeated revisions to ensure study validity. In cases where the two authors did not agree, discussions were conducted by three members, including a third party (Sawada) who was neutral but trained in qualitative research methodologies. This additional perspective contributed to a more comprehensive understanding of the data and played a key role in reaching the theoretical saturation (Kreslake et al., 2016).
Study 2: Quantitative Analysis Phase
To calculate the impact of RS on COPM scores, we asked patients to re-rate COPM scores for occupations in which any of the five Ws and How changed between the initial assessment (T1) and the reassessment (T2). Patients were given feedback on their perception of the occupation at the initial assessment (T1) by Ohno and were asked to re-rate the COPM–Performance (COPM–P) and COPM–Satisfaction (COPM–S) scores. We defined the score corrected for RS effects as the T2′ score (i.e., RS–adjusted score).
Hence, the reassessment score (T2) is the patient’s internal criterion score that has changed for whom the researcher has not provided feedback on the patient’s perception at the time of the initial assessment, indicating that a RS has occurred and influenced the measurement score. In Study 2 (QUAN), we analyzed the quantitative change that an RS gives to the COPM by calculating the difference between T2 and T2′ (T2′ minus T2). We performed this analysis for the COPM–P and COPM–S, respectively.
Results
Demographic Data
Nineteen patients were enrolled in this study during the 6-mo recruitment period. Their demographic characteristics are presented in Table 1. Nine patients were male (47.4%), and the mean age was 65.3 yr (SD = 18.9). The diagnostic categories were stroke (n = 7, 36.8%), orthopedic disease (including hip fractures and spinal cord injury; n = 10, 52.6%), and disuse syndrome (a condition caused by a lack of physical activity, secondary to pneumonia and coronary occlusion; n = 2, 10.5%). Their mean score on the MMSE was 26.9 (SD = 3.4), and the average score for the FIM–Total items was 96.6 (SD = 13.8; FIM–Motor items: M = 66.2, SD = 11.9; FIM–Cognitive items: M = 30.4, SD = 5.00). Occupational therapists who implemented the COPM had a mean of 3.9 yr experience (SD = 2.2). The mean period from the initial assessment (T1) to reassessment (T2) was 29.7 days (SD = 10.9, range = 12–52; Table 1).
Participant Characteristics and Identified Occupations in the Canadian Occupational Performance Measure
Note. N = 19. COPM = Canadian Occupational Performance Measure; Max = maximum; Min = minimum; MMSE: Mini-Mental State Examination.
Study 1: Qualitative Analysis Phase
RS Occurrence
Among the 19 patients, 90 occupational performance problems were identified through the COPM interview at the initial assessment. The distributions of the three dimensions are shown in Table 1. The most frequently identified occupation was related to productivity activities (n = 45, 50.0%). This was followed by self-care (n = 30, 33.3%) and leisure (n = 15, 16.7%). The most dominant occupational category was household arrangement (n = 36, 40.0%). When we compared the initial assessment (T1) and reassessment (T2) of the occupations identified on the COPM, we noted that 46 (51.1%) occupations showed a change in at least one of the five Ws and How items. Of the 19 patients, all five Ws and How items remained unchanged in the occupations identified in the COPM for only 1 patient (5.3%). In other words, 18 of the 19 patients’ perceptions of occupations identified in the COPM were altered by the RS effect. The percentages of RSs occurring for each domain of occupation was 53.3% (16 of 30) for self-care, 51.1% (23 of 45) for productive activities, and 46.7% (7 of 15) for leisure.
Categories of the RSs
A total of 82 codes were extracted from 46 occupations in which the change in perception was due to RS effects. The numbers of codes by item according to the five Ws and How were as follows: 10 for Who (subject), 10 for Why (reason/purpose), 12 for Where (place), 19 for When (duration/frequency), and 31 for How (methods/procedure/means). These codes underwent a process of repeated categorization, ultimately resulting in their classification into five overarching concepts: Replacing, Adding, Reducing, Unspecified, and Embodiment (Table 2). We meticulously examined each of these categories was until we noted they formed mutually exclusive relationships (Table 2). These concepts further encompassed a total of 29 subcategories (Table 2).
Concepts and Categories of Response Shift
Note. Text in italics denotes response shifts. T1 = Time 1; T2 = Time 2.
Replacing.
Replacing was defined as “a change in which an element is replaced by a different element over time.” It included five subcategories: (1) place (Where), (2) destination (Where), (3) purpose (Why), (4) walking aid (How), and (5) object (How). Some examples are as follows: ▪ “I want to be able to go to the bathroom . . .” □ “. . . at the hospital (T1)” replacing “. . . at home (T2)” (Where: place). ▪ “I want to be able to go to the bathroom. . .” □ “. . . using a wheelchair (T1)” replacing “. . . using a cane (T2)” (How: walking aid). ▪ “I want to be able to change into . . .” □ “. . . my pajamas (T1)” replacing “. . . everyday clothes (T2)” (How: object).
Adding.
Adding was defined as “a change in which an element that was not present at the initial assessment becomes incidental to the reassessment.” There were seven subcategories: (1) caregiver (Who), (2) partner (Who), (3) frequency (When), (4) destination (Where), (5) purpose (Why), (6) walking aid (How), and (7) alternative means (How). Examples of this concept include the following: ▪ “I want to be able to . . .” □ “. . . cook alone (T1)” replacing “. . . cook with home helpers (T2)” (Who: caregiver). ▪ “I want to be able to go out by train . . .” □ “. . . once a month (T1)” replacing “. . . twice a month (T2)” (When: frequency). ▪ “I want to be able to go out to visit . . .” □ “. . . a doctor (T1)” replacing “. . . a doctor and go shopping (T2)” (Why: purpose).
Reducing.
Reducing was defined as “a change in which an element set at the first time is deleted or reduced in the reassessment.” Eight subcategories were included: (1) companion (Who), (2) frequency (When), (3) place (Where), (4) destination (Where), (5) purpose (Why), (6) walking aid (How), (7) equipment (How), and (8) job content (How). Some examples are as follows: ▪ “I want to be able to . . .” □ “. . . go to pharmacies and shopping streets (T1)” replacing “. . . go to a shopping street (T2)” (Where: destination). ▪ “I want to be able to . . .” □ “. . . go to the hospital while walking to maintain my health (T1)” replacing “. . . go to the hospital (T2)” (Why: purpose). ▪ “I want to be able to keep my garden . . .” □ “. . . using a stepladder and a ladder (T1)” replacing “. . . without using a stepladder or a ladder because it is dangerous for me (T2)” (How: equipment).
Unspecified.
Unspecified was defined as “elements that are not specifically identified and self-perception is inconsistent from one measurement period to the next measurement period.” It included four types: (1) companion (Who), (2) frequency (When), (3) destination (Where), and (4) purpose (Why). Some examples are as follows: ▪ “I want to be able to go out . . .” □ “. . . by train with someone in my family (T1)” replacing “. . . by train alone or with someone in my family (T2)” (Who: companion). ▪ “I want to be able to . . .” □ “. . . go shopping about every other day (T1)” replacing “. . . go shopping once every other 3 or 4 days (T2)” (Where: frequency). ▪ “I want to be able to . . .” □ “I have no particular reason, but I want to be able to go out at my own pace (T1)” replacing “If I am forced to say, I want to be able to go out with my wife to maintain my physical strength (T2)” (Why: purpose).
Embodiment.
Embodiment was defined as “a change in which an element that was abstract the first time becomes concrete with reassessment.” The following five subcategories were included: (1) caregiver (Who), (2) frequency (When), (3) place (Where), (4) destination (Where), and (5) purpose (Why). The following are examples: ▪ “I want to be able to . . .” □ “. . . go to the hospital alone or with my son (T1)” replacing “. . . go to the hospital with my wife (T2)” (Who: caregiver). ▪ “I want to be able to . . .” □ “. . . walk around without deciding on my destination (T1)” replacing “. . . go to a park (T2)” (Where: destination). ▪ “I want to be able to . . .” □ “. . . walk[,] because I cannot do anything without walking (T1)” replacing “. . . walk [so I can go] shopping (T2)” (Why: purpose).
Study 2: Quantitative Analysis Phase
Of the 90 occupations identified by all patients, we investigated 46 with a confirmed RS in Study 1 (QUAL) for the extent of the RS effect. At the reassessment (T2), patients were asked to re-rate the COPM scores for occupations in which any of the five Ws and How changed between the initial assessment (T1) and reassessment (T2), that is, the T2′ score. Figure 2 is a histogram of the differences between T2 and T2′ for the COPM–P and COPM–S. The most frequent differences were ±0 points for both the COPM–P and COPM–S, and 21 of 46 (45.7%) and 19 of 46 (41.3%), respectively. The maximum score was –6 points (1 of 46, 2.2%) for the COPM–P, –7 points (1 of 46, 2.2%), and +7 points (1 of 46, 2.2%) for the COPM–S. The percentages of all occupations with a difference of <2 points (+1, ±0, –1 points) and the minimal clinically important difference (MCID) described in the COPM manual (Law, Baptiste, et al., 1998) were 73.6% (34 of 46) for the COPM–P and 69.6% (32 of 46) for the COPM–S. In other words, the percentages of occupations for which the change in score due to correcting the change in perception because of an RS exceeded the MCID (±2 points) were 26.1% (12 of 46) for the COPM–P and 30.4% (14 of 46) for the COPM–S.
Difference scores of occupations that showed a response shift.
Discussion
In this study, we qualitatively and quantitatively investigated changes in patients’ perceptions of the occupations identified in the COPM over time due to the effects of RS. In Study 1 (QUAL), we classified them into five RS concepts. To rigorously analyze our data, we applied the concept of semantic content similarity. This method involves integrating primary categories on the basis of their underlying meaning or content while ensuring that they are mutually exclusive. This approach enabled us to capture subtle changes in patients’ perspectives throughout the rehabilitation process, ensuring the integrity of our categorization. In Study 2 (QUAN), the extent of the RS effects on the COPM was over the MCID (12 of 46 [26.1%] and 14 of 46 [30.4%] for the COPM–P and COPM–S, respectively). These results indicate that researchers and clinicians should consider the impact of RS effects when using the COPM as an outcome measure. In the following paragraphs, we discuss the occurrence and impact of RSs in the COPM and how to avoid RS effects.
The COPM allows respondents to determine the assessment items according to their personal interests (Deshpande et al., 2011; Fitzpatrick et al., 1998), and this characteristic has led to innovative advancements in client-centered occupational therapy practice and research. However, this feature is considered to put the COPM at risk for changes in internal criteria over time (recalibration) because it is dependent on the values and preferences of the respondent at each assessment point. In fact, Eyssen et al. (2005) reported that the consistency ratio of identified occupations using the COPM, in which two different occupational therapists interviewed the same patient 1 wk apart, was only 66%. Verkerk et al. (2006), who investigated the consistency ratio of occupations using the same method as Eyssen et al., observed a consistency rate of 74.0%. These studies show that patients’ perceptions of occupational performance problems can fluctuate easily and are unstable over time. The COPM also has several concerns regarding its scoring system. Sawada et al. (2022) explored the impact of information bias on COPM scores through a qualitative study of subacute rehabilitation hospital inpatients that generated potential information biases in the process of selecting and identifying occupations and accurately quantifying their perceptions of scoring. Their study showed the changes in internal criteria over time in the COPM. Moreover, the results of the present study provide further evidence that the COPM exhibits various forms of RS effects, suggesting that people’s internal standards for occupational performance can undergo considerable change. Even after identifying and sharing occupational goals, clinicians and researchers must remain vigilant about the potential for shifts in perception. However, to the best of our knowledge this is the first study to examine the effect of RSs on COPM scores between pre- and postintervention.
When interpreting the results of this study, it is important to consider the method used to detect the RS effect. Among the most appropriate methods for detecting RS is PROMs (Sajobi et al., 2018). However, Visser et al. (2005) simultaneously adapted three RS detection methods (the then-test, the anchor-recalibration approach, and SEM) to estimate the presence and extent of RS effects and reported conflicting results regarding the presence of RSs. These discrepancies in RS detection are caused by the fact that each method is based on different statistical assumptions and the robustness of the method with regard to data analysis conditions (Sajobi et al., 2018). In this study, we applied a unique approach using the five Ws and How interview guides, taking advantage of the flexibility in item selection characteristics of the COPM. Hence, it is necessary to consider the fact that in this study we adopted a nongeneralized method, and thus the results may contain statistical bias.
The percentages of occupations in which the effect of RS on the COPM score was greater than the MCID (2 points; Law, Baptiste, et al., 1998) were 26.1% (12 of 46) and 30.4% (14 of 46) for the COPM–P and COPM–S, respectively. As a result, when the COPM is used as an outcome measure there is a risk that the change in scores before and after the intervention are not due to the treatment but are contaminated with assessment bias caused by changes in the respondent’s internal criteria. Therefore, clinicians and researchers should be aware of RS to accurately interpret COPM scores.
Of the five concepts of RSs integrated in this study, Replacing, Adding, and Reducing are all elemental changes in the meaning and form of occupation. This indicates that people may be unable to recall and identify their perceptions of occupations in detail over time. Moreover, Unspecified and Embodiment mean that the internal criteria of occupations are not consistent between the initial assessment and reassessment because of abstract perceptions of the occupations. Rehabilitation patients find it difficult to consider reasonable goals at the start of intervention because of a lack of information and knowledge (Southwell et al., 2022; Stenner et al., 2016). According to client-centered theory (Kyler, 2008; Law et al., 1995; Townsend et al., 1990), it is important for the therapist to accept the uncertainty of perception by RS effects and to work collaboratively in response to change. However, if the COPM is used as an outcome measure to examine the effects of interventions, then clinicians and researchers need to minimize the influence of RS effects on the interpretation of COPM scores.
The Specific, Measurable, Achievable, Relevant, and Time-bound (SMART) framework was developed to establish clients’ systematic and individualized goals (Bovend’Eerdt et al., 2009; Schut & Stam, 1994). Bexelius et al. (2018) evaluated the quality of pediatric rehabilitation goals through a retrospective analysis using SMART criteria. On the one hand, all of the goals (161 goals, extracted from 42 children) were linked to the Activity/Participation domain, and the quality of the Specific and Relevant domains was sufficiently high (Bexelius et al., 2018). On the other hand, the Measurable domain had the highest percentage of inadequacy, with 20% of the goals having unclear scaling and 5% of the goals having no defined scaling (Bexelius et al., 2018). Even if patients and therapists identify their own meaningful occupations through the COPM, there is a risk that the achievement criteria can easily become abstract, because details regarding the occupations are rarely discussed. Therefore, we propose that the use of systematic interview frameworks, such as the five Ws and How and the Goal Attainment Scale (Turner-Stokes, 2009), can enhance the accuracy of COPM measurement (Reedman et al., 2021; Spooren et al., 2011). At the beginning of occupational therapy, clear intention-sharing enables prompt recognition of any changes in the patient’s understanding of occupations or setting goals during the subsequent rehabilitation process. This, in turn, allows for immediate adjustments in occupational therapy practice or length of stay according to the patient’s preferences.
It also is important to recognize that RS may not be limited solely to the COPM or the field of occupational therapy. This phenomenon could potentially be extended to other assessment tools that are based on client-centered care principles. Understanding the implications of RSs across various assessment tools beyond the confines of our study is crucial for advancing client-centered care and ensuring the accuracy of outcomes after treatment interventions.
Limitations
This study has two limitations. First, it is important to note that this was a pilot study with a small sample size. The limited sample size resulted in restricted variation in personal characteristics or occupational areas, thus constraining the generalizability of our findings. Therefore, further large-scale validations are required. Moreover, we conducted data collection during the early stages of the coronavirus disease 2019 pandemic. The unique circumstances, characterized by stringent societal restrictions on outings and close interactions with others, may have potentially influenced goal-setting behaviors in the context of rehabilitation. It is essential to consider this historical background when interpreting the outcomes of our pilot study.
Implications for Occupational Therapy Practice
We investigated the RS effects of the COPM on inpatients in subacute rehabilitation hospitals using convergent mixed-methods research. The primary implications for occupational therapy practice and research are as follows: ▪ The RS effects of the COPM were classified into five categories: Replacing, Adding, Reducing, Unspecified, and Embodiment. ▪ In 18 of the 19 patients in the study, at least one occupation changed in the patient’s perception over time, because of a RS. ▪ To use the COPM as an outcome measure, it is necessary to identify patients’ perceptions of occupation in more detail to reduce the impact of RSs.
These findings have notable implications for the understanding, choice, and use of PROMs. First, the recognition that patients’ internal criteria are changeable underscores the dynamic nature of their experiences and preferences over time. Considering that people may not be fully informed about future developments in occupational therapy, it is possible that this could lead to variations in the scoring outcomes. In light of this, we suggest that conducting goal-setting and scoring after clients have had the opportunity to gain experience with the tasks might enhance the accuracy and meaningfulness of assessment outcomes.
Conclusion
Our aim in this study was to explore the effects of RS on the COPM for adult inpatients. A comparison of patients’ perceptions of the occupations identified in the COPM between the initial assessment and reassessment detected five types of RSs. Moreover, because the risk of RSs affecting COPM scores may exceed that of the MCID, it is important to identify more specific perceptions when using the COPM.
Footnotes
Acknowledgments
We thank all of the participants for their cooperation in this study.
1
FIMTM is a trademark of the Uniform Data System for Medical Rehabilitation, a division of UB Foundation Activities, Inc.