Interrater Reliability and Clinical Utility of the Personal Care Participation Assessment and Resource Tool (PC–PART) in an Inpatient Rehabilitation Setting

Abstract

OBJECTIVE. We examined the interrater reliability and clinical utility of the Personal Care Participation Assessment and Resource Tool (PC–PART) in a rehabilitation setting.

METHOD. Ninety-six patients were recruited from the caseload of four occupational therapists. Patients were assessed on admission. Clinical utility was based on time to complete assessments.

RESULTS. Interrater reliability for the total score was very high (intraclass correlation coefficient = .91). Limits of agreement for the program indicated aggregate data from individual scores were within 1.3 units (range = −1.3 to 0.5) but individual scores would be within 10 units (range = −9.3 to 0.2). The PC–PART took 27 min to complete.

CONCLUSION. The PC–PART may have sufficient interrater reliability and clinical utility to evaluate program outcomes. The limits of agreement for rating individuals were relatively large, suggesting that it may be difficult to use the PC–PART to make clinical inferences about an individual patient.

Keywords

outcome assessment (health care)rehabilitation reproducibility of results self care

The use of validated and reliable tools with robust psychometric properties is important in clinical practice and a necessary prerequisite to demonstrate effectiveness (Fricke & Darzins, 2006). Health care is concerned with providing the highest quality care within increasing financial constraints. As a result, health professionals need to measure clinically relevant outcomes to provide evidence of the effectiveness of their interventions, and service managers require information to substantiate programs and justify funding. In rehabilitation settings, the drive to measure and describe outcomes is reinforced by the high level of therapeutic intervention and multidisciplinary resources provided.

Occupational therapy interventions ultimately support patients’ health, well-being, and level of occupational engagement (Townsend & Polatajko, 2007). Commonly, interventions take the form of retraining in activities of daily living, treatments to reduce impairment and improve functioning, education, assessment of the home environment, and provision of aids and equipment to support discharge home (Fricke, 1993). Hemmingsson and Jonsson (2005) described the occupation-focused perspective as looking at the ordinary things that people routinely do in their daily lives and their impact on health and development. Within rehabilitation, the client-centered approach and occupation-focused process often includes reassessment and negotiation of goals with the discharge aims in mind. The effects and outcomes of intervention and programs, however, are largely unmeasured.

Unsworth (2000) and others (e.g., Bowman & Llewellyn, 2002; Fricke & Darzins, 2006) have urged occupational therapists to document outcomes and measure the effectiveness of their interventions through the systematic use of standardized tools. Occupational therapy professionals, however, have struggled to meet the challenge of measuring outcomes, although clinicians and service managers are increasingly aware of the need to justify their services by demonstrating the effect of therapy and the need to continually improve the quality of care (Unsworth, 2000). Despite the acknowledged need to measure, the use of standardized outcome measures is low (Stapleton & McBrearty, 2009). Therapists have argued that many outcome measures are not holistic or sensitive to change (Stapleton & McBrearty, 2009). Clinical pressures, including competing demands for their time, a lack of training, and limited perceived value, may also contribute to the low use of outcome measures by occupational therapists (Bowman & Llewellyn, 2002). In many cases occupational therapists allow service managers and wider program initiatives to dictate how their interventions are measured (Stapleton & McBrearty, 2009).

Personal Care Participation Assessment and Resource Tool

Outcome measures for occupational therapy must include those that reflect participation of the individual, shaped by environmental and personal factors, because enabling participation is the primary focus for occupational therapy intervention. The Personal Care Participation Assessment and Resource Tool (PC–PART), formerly the Handicap Assessment and Resource Tool, was designed to measure participation restriction (Darzins, 2004). The PC–PART developers described the strong association between participation restrictions and discharge planning as it relates to the individual’s ability to live within the community (Vertesi, Darzins, Lowe, McEvoy, & Edwards, 2000).

Whether the PC–PART measures what it purports to measure with low amounts of error is important. Without evidence of the psychometric properties of the PC–PART, it is difficult to support its use. This notion is especially important when a consensus about the best measures to use in rehabilitation is lacking.

A recent systematic review of the measurement properties of the PC–PART provides evidence of the validity of the tool and identifies the need for further research of its reliability and practicality. Darzins, Imms, and Di Stefano (2013) examined 7 studies that considered at least one measurement property of the tool. They concluded that although evidence supported the content and face validity of the PC–PART, only preliminary evidence supported interrater reliability and clinical utility of the tool in acute and subacute settings. For example, the amount of time required to administer the tool emerged as a theme.

The purpose of this study was to add to the existing body of knowledge on the psychometric properties of the PC–PART. The specific aim was to evaluate interrater reliability in an inpatient rehabilitation population. A secondary aim was to provide information to evaluate reported clinician concerns about the clinical use of the PC–PART as measured by time required to complete the assessment (Barbara & Whiteford, 2005). Knowledge gained from this study will allow decisions to be made about the use of the PC–PART in rehabilitation and identify opportunities for future research.

Method

Research Design

In this study we used a reliability design to establish the level of agreement between the routine assessment of ward occupational therapists and a research occupational therapist using the PC–PART in a rehabilitation setting. Interrater reliability studies are commonly used to provide data about the reproducibility of an assessment and a rigorous assessment of reliability because they include sources of error from different therapists and from being assessed a second time when there is no expectation of change. We also assessed aspects of clinical use of the PC–PART to assist in determining the feasibility of using the PC–PART within this clinical setting. Full ethics approval was obtained from both health service (HEC#23/1011) and University Human Research Ethics (FHEC#10/258) Committees. All participants provided written informed consent, or consent was gained from a caregiver for participants without cognitive capacity.

Setting

This study was conducted in rehabilitation wards of a publicly funded hospital in an outer metropolitan region of Australia. After admission to hospital for an acute condition, patients are typically admitted to inpatient rehabilitation wards and assessed for their potential to return to independent living in the community. The rehabilitation setting provides a high level of therapeutic intervention and multidisciplinary management.

At the time of data collection, the rehabilitation wards had a total of 45 beds. The patient group was predominantly English speaking. The vast majority were adults age 65 yr or older. Patients are typically admitted with neurological, musculoskeletal, or cardiorespiratory conditions. The PC–PART had been used routinely at this site since July 2010.

Sample Size Estimate

We aimed to recruit a sample of 100 patients and 4 treating occupational therapists in addition to the research therapist, for a total of 5 raters. This sample size was calculated as being sufficient to demonstrate an anticipated intraclass correlation coefficient (ICC) of .90 for the total score, with standard error set at .025 (Streiner & Norman, 2008). The literature suggests a range of acceptable reliability coefficients of .70 to .90, depending on whether the scale is to be used in research or clinical practice (Nunnally, 1978). Because the aim of the study was to inform clinical practice, the ICC for the sample size estimation was set at the higher level.

Participants

Patient Participants.

Patients were recruited from a consecutive series of admissions to the rehabilitation wards. All patients admitted to these wards were screened for inclusion in the study. Patients were eligible for inclusion if they were age 18 yr or older and admitted for rehabilitation. Patients with or without cognitive capacity who gave consent (or whose caregiver gave consent) to the study were eligible. Patients with cognitive impairment were specifically identified as eligible for inclusion because they are commonly part of the general rehabilitation population. For example, a study conducted at the same site showed that 26% (117 of 453) of patients admitted to rehabilitation had impairment of cognition (Brusco, Shields, Taylor, & Paratz, 2007). Cognition was assessed using Mini-Mental State Examination (MMSE) scores (Folstein, Folstein, & McHugh, 1975).

Therapist Participants.

All occupational therapists working on the rehabilitation wards were eligible for inclusion in the study if they were trained or willing to train in the use of the PC–PART. Therapists were withdrawn from the study if they stopped working within the rehabilitation unit or had contributed fewer than 10 PC–PART assessments.

Procedure

Assessments of patients were completed by the ward occupational therapists and a research occupational therapist rater. All assessments occurred in a setting with privacy and where it was expected that there would be few distractions.

The Instrument.

The PC–PART is a 43-item scale, divided into seven personal care domains: clothing, hygiene, nutrition, mobility, safety, residence, and supports (Vertesi et al., 2000). The tool measures participation restriction in each of these areas. Restrictions in these seven areas are considered to be relevant for discharge planning because they are likely to influence the ability of the individual to live in the community. The tool is designed to be administered using patient interview, key informant interview, and task observation, if needed, with each item categorized as ok by self, ok with help, or not ok. The number of participation restrictions (i.e., those scored not ok) is scored globally, and the sum total out of 43 documented (score range = 0–43). Data are recorded on the standardized PC–PART worksheet. For the purpose of the study, individual item scores (1 = not ok, 0 = ok by self or ok with help) and the number of restrictions identified within each domain were recorded: clothing (range = 0–5), hygiene (range = 0–8), nutrition (range = 0–8), mobility (range = 0–9), safety (range = 0–6), residence (range = 0–5), and supports (range = 0–2).

The usual occupational therapy assessment consists of the completion of a nonstandardized template of questions. The occupational therapist records demographic information and information about the social situation and supports of the patient. It guides the therapist through a review of the patient’s premorbid and current level of function in domestic, personal, and community activities of daily living and generates flags for more detailed assessment of cognition and upper limb function as needed. It is completed using the medical notes, feedback from significant others, and direct interview and observation. The occupational therapist completes a problem list and sets goals with the patient to guide the rehabilitation plan.

Clinical utility of the PC–PART was assessed by measuring the amount of time taken to score the assessment. This specifically detailed time in minutes was spent on occupational therapy assessment, key informants, and the PC–PART as defined below:

Assessment: Time in minutes to complete routine occupational therapy initial evaluation. Evaluation included face-to-face time with the patient and time spent obtaining information from medical records to complete the evaluation.

PC–PART: Time in minutes spent specifically conducting the PC–PART face to face with the patient. This component included any time required to obtain information from medical records that was required only to complete the PC–PART and was in addition to routine occupational therapy assessment.

Key informant: Time in minutes spent obtaining information from a family, caregiver, or other health professional to inform both the occupational therapy initial assessment and PC–PART. This component included all time spent gathering information from the responsible person.

Therapist Training.

The PC–PART was used routinely by occupational therapists at the study site. Because it was anticipated that the eligible occupational therapists would have varying experience in using the PC–PART, all participating occupational therapists were provided with standardized education in the use of the PC–PART before commencement of data collection. Training consisted of a 1-hr session with an occupational therapist experienced in use of the PC–PART. In addition, the PC–PART manual (Darzins, 2004) and DVD (Francis, 2004) were made available to all participating therapists. On the basis of prior experience of using the PC–PART in the facility, a “frequently asked questions” sheet was developed and provided to participant therapists. No further verification of therapist training or reliability was assessed before data collection.

Data Collection.

After recruitment to the study, all patients underwent occupational therapy evaluation, including assessment with PC–PART, by their treating therapist. Initial evaluations by occupational therapists were routinely completed within 3 days of admission. The research therapist then completed a second assessment to rate the PC–PART. The research therapist was blinded to the rating of the treating therapist. This second research assessment occurred within 1 working day of the treating therapist’s assessment. The two ratings were recorded on separate PC–PART worksheets. Time spent administering the tool was recorded by the treating and research therapists.

In addition to PC–PART data, demographic information, including the patient’s age, sex, and primary admitting diagnosis, were collected from the patient’s medical file. Specific demographic characteristics of the therapist participants were not collected, because the small number of occupational therapists within the service allowed them to be easily identified.

Data Analysis.

We calculated Cohen’s κ; percentage agreement; ICC (Model 2,1), representing relative agreement; and limits of agreement, representing absolute agreement. This analysis occurred between the research therapist and each treating therapist and between the research therapist and all therapists combined, where N = 96. To determine the extent of agreement between categories, a generalized Cohen’s κ coefficient was used (Cohen, 1960). Kappa examines interrater reliability between categorical data with chance considered. The ICC gives a relative index of the ratio of variance between subjects to the variance between subjects plus error variance. The ICC model (2, 1) takes account of variance between each rater’s scores and was applicable because each participant was measured on each occasion (Shrout & Fleiss, 1979). A 95% confidence interval (CI) for aggregate data of individual scores was obtained from the difference between means (Ms) of paired scores and is equivalent to the 95% CI for the paired t test (Altman, 1991):

where mean difference (Md) of retest minus test scores and standard deviation (SD) of the difference between each rater's scores. To interpret individual scores, we used 95% CIs with limits of agreement to establish the level of agreement between two raters’ scores; analysis was conducted at the item level, at the domain level, and with the overall score. To determine these 95% CIs for individuals, N = 1 is substituted into the above equation. Clinical utility was analyzed descriptively on the basis of the time taken to complete assessments. There were no missing data. IBM SPSS Statistics (SPSS, Inc., Chicago) Version 18 was used for analysis.

Results

Along with the occupational therapy researcher, an additional 6 occupational therapists were recruited to the study from a total of 7 eligible occupational therapists. Two therapists withdrew because of changes in their area of work. The occupational therapist participants included men and women and had experience ranging from new graduate to senior clinician grades with a mean of 7 yr experience (SD = 2.0).

During the study period, 106 participants were recruited from a consecutive series of admissions to the rehabilitation wards. Of the 106 who consented, 6 patients were withdrawn after consent but before commencement of data collection because the occupational therapy researcher was on leave, 2 patients were withdrawn because they no longer wished to participate, and 2 were excluded from analysis because their treating occupational therapist was withdrawn. Of the total sample of 96 patient participants, 54 (56%) were women and 42 (44%) were men with mean age of 73.1 yr (SD = 12.1). The mean MMSE score was 26.6 (SD = 4.2). Of the sample, 14% had a cognitive impairment (MMSE ≤ 24); of the 14%, 5 patients scored 19 on the MMSE, and their inclusion into the study required the consent of a person responsible for decisions related to their health care. The most common diagnoses of the recruited patients were stroke (n = 27, 28%), postfracture (n = 25, 26%), and post hip- and knee-joint replacement (n = 21, 22%; Table 1). The demographic characteristics of the patients assessed by each treating therapist were similar.

Table 1.

Demographics of Patient Participants

Measure	T2 (n = 27)	T3 (n = 27)	T4 (n = 22)	T5 (n = 20)	Total (N = 96)	p
MMSE, M (SD)	24.7 (5.3), n = 23	26.9 (4.2), n = 27	26.7 (3.6), n = 20	28.5 (1.9), n = 19	26.6 (4.2), n = 89	.45
Age, yr, M (SD)	74.8 (9.5)	74.7 (9.5)	68.6 (16.0)	73.6 (3.2)	73.1 (12.1)	.26
Consent, provided by patient/provided by other, n	24/3	26/1	21/1	20/0	91/5	.37
Gender, female/male, n	14/3	17/10	10/12	13/7	54/42	.50
Admitting health condition, n (%)						.18
Stroke	4 (15)	3 (11)	10 (45)	10 (50)	27 (28)
Fracture	9 (33)	8 (30)	5 (23)	3 (15)	25 (26)
Hip–knee replacement	5 (19)	9 (33)	3 (13.5)	4 (20)	21 (22)
Other ortho	6 (22)	4 (15)	3 (13.5)	1 (5)	14 (15)
Othera	3 (11)	3 (11)	1 (5)	2 (10)	9 (9)

Note. M = mean; MMSE = Mini-Mental State Examination; ortho = orthopedic; SD = standard deviation; T2–T5 = Treating Therapists 2–5.

Includes cardiac, pulmonary, and spinal conditions.

Absolute Agreement Between the Research Therapist and Combined Treating Therapists for Domain and Total Scores

The absolute agreement between ratings for the domain and total scores is shown in Table 2. For all domains except residence, no systematic difference in scores was found, as indicated by the 95% CI of the mean difference. The individual absolute agreement for each domain and the total score were wide relative to the range of the PC–PART scale. In contrast, the group limits of agreement were narrow for both domain and the total score.

Table 2.

Absolute Agreement Between Research Therapist and Combined Treating Therapists for PC–PART Domain and Total Score

Domain (Score Range)	M (SD) T1	M (SD) T2–T5	M difference	SD difference	95% CI: Group	95% CI: Individual
Clothing (0–5)	1.5 (1.4)	1.4 (1.4)	0.3	1.1	[−0.04, 0.4]	[−2.0, 2.3]
Hygiene (0–8)	2.3 (2.2)	2.5 (2.2)	−0.2	1.7	[−0.5, 0.2]	[−3.5, 3.2]
Nutrition (0–8)	1.1 (1.2)	1.3 (1.4)	−0.2	1.1	[−0.4, 0.05]	[−2.3, 2.0]
Mobility (0–9)	4.4 (1.6)	4.1 (2.1)	0.3	1.4	[0.0, 0.6]	[2.5, 3.0]
Safety (0–6)	1.0 (1.1)	1.3 (1.2)	−0.3	0.9	[−0.4, −0.08]	[−3.1, 2.6]
Residence (0–5)	0.5 (0.3)	0.3 (0.6)	−0.2	0.6	[−0.3, −0.10]	[−1.4, 1.0]
Supports (0–2)	0.8 (0.9)	0.8 (0.9)	−0.08	0.7	[−0.2, 0.06]	[−1.5, 1.3]
Total (0–43)	11.2 (6.3)	11.6 (8.0)	−0.4	4.4	[−1.3, 0.5]	[−9.6, 8.2]

Note. CI = confidence interval; M = mean; PC–PART = Personal Care Participation Assessment and Resource Tool; SD = standard deviation; T1 = research therapist; T2–T5 = Treating Therapists 2–5.

Relative Reliability for Domain and Total Scores

Intraclass correlation coefficients were calculated for each pair of therapists by domain. Table 3 displays the data, demonstrating the overall agreement for total PC–PART score (ICC = .91). The data show good to very good levels (ICC > .61) for each domain except residence, for which the level of agreement was poor. Some variability was found in estimates of agreement between each pair of therapists within domains; however, because of the overlapping CIs it could not be concluded that agreement was lower between any pair of therapists. We found more consistency in total domain ICCs.

Table 3.

Relative Reliability (ICC 2,1) for PC–PART Total and Domain Scores

Domain	ICC [95% CI]
Domain	T1 and T2 (n = 27)	T1 and T3 (n = 27)	T1 and T4 (n = 22)	T1 and T5 (n = 20)	All (N = 96)
Clothing	.88 [0.74, 0.95]	.58 [0.19, 0.81]	.97 [0.93, 0.99]	.71 [0.26, 0.88]	.84 [0.75, 0.89]
Hygiene	.91 [0.79, 0.96]	.70 [0.34, 0.86]	.98 [0.95, 0.99]	.99 [−0.04, 0.84]	.82 [0.73, 0.88]
Nutrition	.69 [0.31, 0.86]	.80 [0.56, 0.91]	.77 [0.44, 0.90]	.67 [0.17, 0.87]	.77 [0.66, 0.85]
Mobility	.87 [0.72, 0.94]	.74 [0.42, 0.88]	.89 [0.73, 0.95]	.86 [0.62, 0.94]	.84 [0.76, 0.89]
Safety	.90 [0.77, 0.95]	.31 [−0.52, 0.68]	.83 [0.60, 0.93]	.80 [0.50, 0.92]	.82 [0.73, 0.88]
Residence	.49 [−0.11, 0.77]	.00 [−1.19, 0.54]	.00 [−1.41, 0.59]	.42 [−0.47, 0.77]	.38 [0.07, 0.59]
Supports	.71 [0.36, 0.87]	.78 [0.52, 0.90]	.76 [0.43, 0.90]	.70 [0.24, 0.88]	.80 [0.71, 0.87]
Total	.91 [0.85, 0.95]	.87 [0.78, 0.93]	.91 [0.84 to 0.96]	.86 [0.75, 0.94]	.91 [0.88, 0.93]

Note. CI = confidence interval; ICC = intraclass correlation coefficient; PC–PART = Personal Care Participation Assessment and Resource Tool; T1 = research therapist; T2–T5 = Treating Therapists 2–5.

Interrater Agreement of Item Scores

The majority of the 43 items of the PC–PART had moderate to very good levels of interrater agreement: 31 of the items scored >85% agreement. Kappa scores, however, could not be calculated for 49 of the 172 pairs of items (43 items × 4 pairs), because every patient was rated the same by the raters for these items. For example, for meal planning, money management, home security, and temperature, both items were scored as ok by self by both therapists for all the patients rated. Overall κs for the items clothing selection appropriate for environment; alcohol and substance misuse; smoking; and shopping for personal needs, household items, and so forth were poor. The levels of agreement did not appear to vary substantially in each pair of therapists.

Clinical Utility.

On average it took 20–35 min for the occupational therapists to complete the PC–PART (M = 26.5, SD = 10.96), although the time ranged from 8 to 60 min (Table 4). The treating therapists spent a mean of 66 min (SD = 19.4, range = 30–120 min) in total completing the occupational therapy assessment, which included using the PC–PART. Very little time was spent with key informants (M = 3.2 min, SD = 6.2, range = 0–30 min). There was no evidence of differences between the research therapist and the treating therapists in the amount of time it took to administer the PC–PART (all ps > .05)

Table 4.

Time for PC–PART

Variable	M (SD) [range]
Variable	T1 (N = 96)	T2 (n = 27)	T3 (n = 27)	T4 (n = 22)	T5 (n = 20)	All (N = 96)
Assessment time	NA	33.0 (13.2) [15−60]	43.8 (11.7) [2−60]	33.0 (9.1) [20−60]	49.3 (7.3) [40−60]	39.4 (12.7) [2−60]
PC–PART time	18.5 (7.5) [9−55]	20.2 (6.0) [10−30]	26.3 (10.2) [15−55]	15.8 (6.0) [8−30]	31.5 (7.8) [20−45]	23.3 (9.5) [8−55]
PC–PART key informant	3.2 (4.9) [0−25]	0.6 (2.9) [0−15]	4.0 (5.1) [0−25]	5.0 (6.0) [0−15]	3.8 (9.5) [0−30]	3.2 (6.2) [0−30]
Clinical total	NA	53.7 (15.1) [30−90]	74.1 (12.9) [55−107]	53.8 (14.0) [30−85]	84.6 (17.0) [65−120]	65.9 (19.4) [30−120]
PC–PART total	21.7 (7.6) [9−55]	20.8 (6.2) [10−30]	30.3 (10.6) [16−56]	20.8 (8.0) [8−35]	35.3 (11.9) [20−60]	26.5 (10.96) [8−60]

Note. There were no missing time data. M = mean; PC–PART = Personal Care Participation Assessment and Resource Tool; SD = standard deviation; T1 = research therapist; T2–T5 = Treating Therapists 2–5; NA = not applicable.

Discussion

This study provides evidence that the PC–PART may be a useful tool for managers and researchers to evaluate program outcomes during rehabilitation. This was demonstrated by the interrater agreement for PC–PART domain and total scores. The results suggest that mean differences for the overall PC–PART score ranged from −1.3 to 0.5, with 95% confidence. In other words, if one therapist rated the average number of restrictions in a group of patients as 11, then one could be 95% confident that a second therapist rating the same group of patients would rate them as having somewhere between 9.7 and 11.5 restrictions. Relative to the scale width (0–43), this result suggests a high degree of absolute reliability in applying the PC–PART to groups of patients. The results suggest that the PC–PART is adequately reliable for use in the evaluation of participation profile using aggregate data from individual patients, because it is able to pick up small changes in groups in the inpatient rehabilitation setting. These changes are small relative to the amount of change that can be expected over an episode of inpatient rehabilitation (Peiris, Sheilds, Brusco, Watts, & Taylor, 2013).

Although group-level reliability was high in this study, the interrater reliability of the PC–PART was not as high when evaluating individual patients. The limits of agreement in this study ranged from −9.6 to 8.2 units. The overall PC–PART rating of a patient by one therapist could be as many as 10 points different from the rating of another therapist, even if the patient was not expected to have changed. Therefore, if one therapist rated a patient as having 20 restrictions, one could be 95% confident that a second therapist rating the same patient would rate him or her as having somewhere between 10 and 29 restrictions. How far apart the limits of agreement can be without causing difficulties is a question of judgment. Relative to the scale range (0–43), however, limits of agreement of 10 are considered relatively large and indicative of a moderate degree of measurement error.

Another consideration is how much change can be expected in the scale. A distinction exists between monitoring change (which would be informed by a retest reliability design) and evaluating agreement between two raters; however, both have error contributions from two sets of measurement. The results of this study suggest that the limits of agreement were relatively large and may make it difficult to be confident of the true number of restrictions when different raters are used and, hence, to make clinical inferences about an individual patient in rehabilitation.

The apparent discrepancy of having a scale with high overall ICC (.91) and relatively wide limits of agreement demonstrates the value of including relative and absolute measures of reliability (Keating & Matyas, 1998). It has been argued that relying on relative reliability alone by reporting only ICCs can lead to a misleading appraisal of the reliability of a scale.

The PC–PART demonstrated evidence of clinical utility with time for assessment typically being less than 30 min. Barbara and Whiteford (2005) interviewed occupational therapists who reported that the PC–PART took longer than a standard initial assessment process. Their study, however, did not have quantifiable evidence of the time taken to administer the tool. Previous studies have used the PC–PART as an assessment tool by the interdisciplinary team, other allied health disciplines, and managers for program evaluation in rehabilitation (Darzins, Smith, & Bremnar, 2002; Smith et al., 2001; Turner, Fricke, & Darzins, 2009). The current study demonstrated that the amount of time required for an occupational therapist to gather this breadth of information about the participation profile of patients was not onerous, taking less than 27 min in addition to standard assessment time. The majority of the items in the PC–PART are highly relevant to occupational therapy practice and usual clinical assessment. Therefore, the additional time taken by occupational therapists to complete the PC–PART may be less than if the tool were used by the multidisciplinary team or another profession.

The perception of the lack of clinical utility of outcome measures can influence their use by clinicians and service managers. The use of the PC–PART has been limited in part because of the notion that it is time consuming. Stapleton and McBrearty (2009) surveyed 109 occupational therapists to explore their use of standardized assessment and outcome measures. Their study found that barriers to use of outcome measures included time restraints, the measure not specifically assessing occupational therapy intervention, and the tools not being client centered enough or sensitive enough to change. In many instances the perception of a lack of clinical utility may persist because of a lack of evidence to dispel this notion.

Of note, 94% of therapists in Stapleton and McBrearty (2009) felt that more emphasis should be placed on incorporating outcome measures into clinical practice despite barriers. Clinicians need to understand and see the value of outcome measures. Data collected should be used in a meaningful and transparent way if clinicians are to support the use of the PC–PART. Where programs are evaluated by service managers, the purpose of the outcome measure must be clearly defined and conveyed to clinicians. If this occurs, then the barrier to its use will be lessened. This study demonstrates that the time to complete a PC–PART is reasonable and should not be viewed as a barrier to its use.

Sources of Error

The results of this study demonstrate that the PC–PART may have sufficient reliability for some purposes, such as evaluating group programs, but insufficient reliability for other purposes, such as evaluating the number of restrictions in a client. The relatively large limits of agreement for evaluating clients suggest that sources of error that could be contributing to these results should be explored further. We noted several potential sources of error in this study, which may have affected interrater reliability. These sources of error can be categorized as related to (1) the outcome measure and (2) the patient participants.

Outcome Measures.

The PC–PART synthesizes information from a variety of sources, and therapists using the PC–PART are encouraged to gather the information from all of these sources, if required. The manual advises the rater to ask questions in the standardized phrasing provided. This variety of sources means that it is highly unlikely that two therapists will gather the information in an identical manner. In addition, the phrasing of some of the items leaves enough ambiguity around what is to be asked and assessed by the PC–PART to contribute to a lack of agreement.

Patient Participants.

The study protocol required the two ratings to take place within 1 working day. This requirement was met in all cases. This period of time was chosen to reduce the likelihood of clinical change in the sample population but allow sufficient time for usual clinical assessment. With 44% of participants assessed on the same day, there is a chance that patients remembered the questions and reduced the researcher’s need to seek further evidence through direct observation and key informants. It is also possible that patients within rehabilitation could make gains in functional ability between the two ratings. In particular, this methodological issue might have affected 43% of the ratings done the next day and 13% of assessments with a weekend in between ratings. However, we found no systematic differences between the two ratings for the total score.

Although it is difficult to determine contribution to error, elements of research design such as ensuring the second assessment was completed within a day of the first assessment and standardizing clinician training suggest that factors related to administering the PC–PART and collecting information from a variety of sources could have contributed to unpredictable variability in the measurement.

Strengths and Limitations

One measure of strength of this study is the extent to which it meets the COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) checklist for reliability studies (Mokkink et al., 2010). The COSMIN checklist was developed using a Delphi study with the aim of creating a checklist to evaluate studies on measurement properties. This study meets all criteria proposed by Mokkink et al. as design requirements and statistical methods for the appraisal of reliability measurement. One of the identified strengths of this study is the sample size. With four pairs of raters and a large number of variables, it was necessary to have a large patient sample to be able to generalize the results of the study (Streiner & Norman, 2008). Our sample of 96 participants was sufficient to allow confidence in making assertions about the reliability of the PC–PART.

Our study had several limitations, such as the information collected about the therapist raters. Kottner et al. (2011) described the need to fully identify the therapist population, detailing therapist qualifications, clinical background, knowledge, degree of expertise, and training. They reported that these attributes play a pivotal role in reliability and agreement estimates. A fuller description of our therapist participants would have allowed for the identification of individual therapists in this small service. It was decided that maintaining the anonymity of the therapists would facilitate recruitment of therapists.

In addition, the time measurement used in the study was important to understand its clinical utility in rehabilitation. However, similarities between the usual occupational therapy assessment and PC–PART may have made the coding of time allocation difficult for the treating therapist. In particular, it may have been difficult to discern time spent gathering information that would have been part of normal assessment but also would assist in rating the PC–PART. For example, if the therapist completed a personal care assessment with a patient that took 30 min and then completed the PC–PART, he or she could have regarded that time as direct observation for the bathing and bath transfer items or as usual clinical assessment. However, the research therapist took an average of 27 min to complete the PC–PART. Finally, it also may have been helpful for all therapists to rate using the PC–PART on the initial encounter before any other occupational therapy assessments or for all therapists to complete the occupational therapy evaluation as well as the PC–PART.

Further Development of the PC–PART

Despite high levels of interrater reliability and evidence of clinical utility, we found scope for further development of the tool. κ was unable to be calculated for 28% of the paired ratings at the item level because there was no variability in these scores between patients. In all instances, one or both therapists rated this item as ok or ok with help for all participants. One interpretation is that therapists agreed, which is positive. The lack of between-patient variability, however, suggests the potential redundancy of items. If all 96 patients rate the same on a scale, then it needs to be questioned whether that item adds further information for discrimination or monitoring. With limited variability in rating items between patients in the residence domain and low levels of agreement between the raters (ICC = .38), consideration ought to be given to whether the domain provides the therapist with clinically useful information. A case for an abridged version of the PC–PART could be made, which may increase the acceptability of the tool in rehabilitation. A revised version of the PC–PART would, however, need to undergo additional scrutiny of its measurement properties and clinical utility.

In addition, a standardized method of training should be considered to supplement the DVD and user manual. The researchers in this study held a 1-hr training session and developed a quick reference tool to support the use of the PC–PART. Further development of education and training materials, which includes practice scoring, may influence reliability.

Implications for Occupational Therapy Practice

High levels of interrater reliability are important because they allow clinicians to have confidence that the PC–PART measure is consistent each time it is used. A high level of reliability is considered to be a necessary characteristic of a clinical measure. High levels of interrater reliability within inpatient rehabilitation are particularly important because patients may be treated by several clinicians during their episode of care, and the assessment findings of colleagues are relied on to reduce unnecessary duplication. If the PC–PART is used within services as a tool by occupational therapists to plan and evaluate their treatment or by managers as a means of assessing program outcomes, it is essential to know that the ratings by different therapists are repeatable and thus reliable. In summary,

Measurement of participation in rehabilitation is important in occupational therapy practice.

The PC–PART has sufficient interrater reliability to be considered as a measure of participation restriction for researchers and managers to measure program outcomes in rehabilitation.

Conclusion

The PC–PART as a measure of participation restriction has a sound theoretical basis that is relevant to occupational therapy practice. This study adds to the literature by demonstrating that the PC–PART has sufficient interrater reliability to evaluate group programs and has quantified that approximately 27 min is required to complete a PC–PART assessment. Despite its sound theoretical basis and the preliminary evidence of reliability and utility, further research on its psychometric properties is required before its routine use in rehabilitation can be recommended.

Footnotes

Acknowledgments

We thank the occupational therapy staff and patients of the Angliss Hospital for their contribution to this study.

References

Altman

D. G.

(1991). Practical statistics for medical research. London: Chapman & Hall.

Barbara

, & Whiteford

(2005). Clinical utility of the Handicap Assessment and Resource Tool: An investigation of its use with the aged people in hospital. Australian Occupational Therapy Journal, 52, 17–25. http://dx.doi.org/10.1111/j.1440-1630.2005.00476.x

Bowman

, & Llewellyn

(2002). Clinical outcomes research from the occupational therapist’s perspective. Occupational Therapy International, 9, 145–166. http://dx.doi.org/10.1002/oti.162

Brusco

N. K.

Shields

Taylor

N. F.

, & Paratz

(2007). A Saturday physiotherapy service may decrease length of stay in patients undergoing rehabilitation in hospital: A randomised controlled trial. Australian Journal of Physiotherapy, 53, 75–81. http://dx.doi.org/10.1016/S0004-9514(07)70039-9

Cohen

(1960). A coefficient of agreement for nominal scales. Educational and Physiological Measurement, 20, 37–46.

Darzins

(2004). PC–PART: Personal Care Participation and Resource Tool user's manual. Melbourne: PART Group.

Darzins

Imms

, & Di Stefano

(2013). Measurement properties of the Personal Care Participation and Resource Tool (PC–PART): A systematic review. Disability and Rehabilitation, 35, 265–281. http://dx.doi.org/10.3109/09638288.2012.690819

Darzins

Smith

, & Bremnar

(2002). Outcome measures in rehabilitation Phase 2. Report to the Department of Human Services. Retrieved from http://www.health.vic.gov.au/subacute/outcomefinal.pdf

Folstein

M. F.

Folstein

S. E.

, & McHugh

P. R.

(1975). “Mini-Mental State”: A practical method for grading the cognitive state of patients for the clinician. Journal of Psychiatric Research, 12, 189–198. http://dx.doi.org/10.1016/0022-3956(75)90026-6

10.

Francis

(2004). How to use the PC–PART to guide in-patient rehabilitation [videorecording]. Norwood, South Australia, Australia: John Francis Productions, Darzins Consulting.

11.

Fricke

(1993). Measuring outcomes in rehabilitation. British Journal of Occupational Therapy, 56, 217–221.

12.

Fricke

, & Darzins

(2006). Measuring occupational performance: Supporting best practice in occupational therapy (2nd ed.). Australian Occupational Therapy Journal, 53, 246. http://dx.doi.org/10.1111/j.1440-1630.2006.00589.x

13.

Hemmingsson

, & Jonsson

(2005). The Issue Is—An occupational perspective on the concept of participation in the International Classification of Functioning, Disability and Health—Some critical remarks. American Journal of Occupational Therapy, 59, 569–576. http://dx.doi.org/10.5014/ajot.59.5.569

14.

Keating

, & Matyas

(1998). Unreliable inferences from reliable measurements. Australian Journal of Physiotherapy, 44, 5–10.

15.

Kottner

Audigé

Brorson

Donner

Gajewski

B. J.

Hróbjartsson

, … Streiner

D. L.

(2011). Guidelines for reporting reliability and agreement studies (GRRAS) were proposed. International Journal of Nursing Studies, 48, 661–671. http://dx.doi.org/10.1016/j.ijnurstu.2011.01.016

16.

Mokkink

L. B.

Terwee

C. B.

Patrick

D. L.

Alonso

Stratford

P. W.

Knol

D. L.

, … de Vet

H. C. W.

(2010). The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: An international Delphi study. Quality of Life Research, 19, 539–549. http://dx.doi.org/10.1007/s11136-010-9606-8

17.

Nunnally

(1978). Psychometric methods. New York: McGraw Hill.

18.

Peiris

C. L.

Sheilds

Brusco

N. K.

Watts

J. J.

, & Taylor

N. F.

(2013). Additional Saturday rehabilitation improves functional independence and quality of life and reduces length of stay: A randomized control trial. BMC Medicine, 11, 198. http://dx.doi.org/10.1186/1741-7015-11-198

19.

Shrout

P. E.

, & Fleiss

J. L.

(1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420–428. http://dx.doi.org/10.1037/0033-2909.86.2.420

20.

Smith

Darzins

Steel

Murray

Osborne

, & Gilsenan

(2001). Outcome measures in rehabilitation (project report of National Ageing Research Institute). Retrieved from http://www.health.vic.gov.au/subacute/outcome_phase1.pdf

21.

Stapleton

, & McBrearty

(2009). Use of standardised assessments and outcome measures among a sample of Irish occupational therapists working with adults with physical disabilities. British Journal of Occupational Therapy, 72, 55–64.

22.

Streiner

D. L.

, & Norman

G. R.

(2008). Health measurement scales: A practical guide to their development and use. New York: Oxford University Press.

23.

Townsend

E. A.

, & Polatajko

H. J.

(2007). Enabling occupation II: Advancing an occupational therapy vision for health, well-being, and justice through occupation. Ottawa, Ontario: CAOT Publications ACE.

24.

Turner

Fricke

, & Darzins

(2009). Interrater reliability of the Personal Care Participation Assessment and Resource Tool (PC–PART) in a rehabilitation setting. Australian Occupational Therapy Journal, 56, 132–139. http://dx.doi.org/10.1111/j.1440-1630.2008.00750.x

25.

Unsworth

(2000). Measuring the outcome of occupational therapy: Tools and resources. Australian Occupational Therapy Journal, 47, 147–158. http://dx.doi.org/10.1046/j.1440-1630.2000.00239.x

26.

Vertesi

Darzins

Lowe

McEvoy

, & Edwards

(2000). Development of the Handicap Assessment and Resource Tool (HART). Canadian Journal of Occupational Therapy, 67, 120–127. http://dx.doi.org/10.1177/000841740006700305