Validity of a New Tool for Assessing the Discrepancy Between Observed and Self-Reported Quality of Activities of Daily Living (ADL) Task Performance

SIGNIFICANCE AND INNOVATION: This study presents preliminary validity and reliability evidence for the Assessment of Compared Qualities—Occupational Performance (ACQ–OP), an innovative new tool that can be used in research and practice to measure the extent of the discrepancy between two perspectives: a person’s self-report and direct observation. Although the earlier versions were focused on awareness, not occupation, the ACQ–OP is occupation focused and better supports occupation-centered practice.

APPROACH: Hypotheses for this study included the following:

(1) The ACQ–OP rating scale demonstrates sound psychometric properties (i.e., ordered category measures and goodness of fit of the rating scale categories; MnSq ≤ 1.5).

(2) The ACQ–OP items and activities of daily living (ADL) tasks define a single unidimensional construct (i.e., goodness of fit; MnSq ≤ 1.4 and/or z < 2).

(3) The ACQ–OP tasks/items separate people by differing levels of discrepancy between self-reported and observed quality of ADL task performance (separation index ≥ 2; separation reliability ≥ .80).

(4) The ACQ–OP measures are reliable (i.e., SE ≤ 0.4).

Developing a new instrument requires evaluating preliminary data to ensure that the new tool demonstrates sound psychometric qualities. Rasch measurement methods are preferred for generating such evidence.

METHOD: This study used a cross-sectional design. Recruitment of participants was based on convenience sampling procedures. The participants were 1,226 persons, aged 5 to 97 yr (M ≥ 50, SD = 23), who were both well and who had a variety of diagnoses (e.g., neurological, developmental) and who were from Scandinavia, Asia, North America, and Slovenia. Of the participants, slightly more than 50% were women, 21% maintained independent living, and 79% needed minimal (38%) to substantial (41%) assistance for community living. All data were gathered in clinical and community settings.

When the ACQ–OP is administered, the person is first evaluated using the Assessment of Motor and Process Skills (AMPS). More specifically, the person is observed performing two prioritized ADL tasks. Immediately following each AMPS observation, the ACQ–OP interview is administered by asking 11 open-ended questions designed to encourage the person to reflect on and report any perceived problems with ADL task performance. Following the ACQ–OP interview, the occupational therapist scores the AMPS and compares the AMPS results to the responses that the person gave during the ACQ–OP interview. The 11 ACQ–OP items are then scored, using a four-category rating scale, on the basis of the extent of discrepancy between the two perspectives, observed and reported.

Each person’s raw ACQ–OP scores were analyzed using many-facet Rasch (MFR) analysis that adjusted each person’s ACQ–OP measure to account for rater severity and challenge of the observed ADL task. MFR analyses routinely generate the statistics specified in the research questions described earlier.

RESULTS: The MFR analysis revealed ordered category measures, and all categories demonstrated acceptable fit. Ninety-two of 95 tasks (97%) and all items demonstrated acceptable goodness of fit. The ACQ–OP tasks and items separated people into at least three different levels (separation index = 2.9; separation reliability = .86). The SE was 0.3.

CONCLUSION: The results support validity and good reliability of the ACQ–OP for use in clinical practice and research in a variety of world regions.