Abstract
The study findings reveal a gap between patient-reported outcomes and performance-based assessments in the context of medication management at home.
Medication management is a complex health management occupation (American Occupational Therapy Association [AOTA], 2020b). Clients who do not complete this occupation at home correctly and consistently over time experience greater rates of morbidity, mortality, hospitalizations, and health care spending (Bitton et al., 2013; Burnier & Egan, 2019; Cutler et al., 2018). Given the severity of the consequences, occupational therapists across the continuum of care evaluate the performance of medication management to ensure that clients have the capacity for a successful and safe discharge home.
Occupational therapy scholars have developed several standardized medication management assessments to help practitioners across clinical settings measure occupational performance in a rigorous and quantifiable manner (e.g., Baum & Wolf, 2013; Boone & Wolf, 2021; Chisholm et al., 2014; Rogers, 1984). Although the rigor, standardization, and reproducibility of the assessments are strengths, it is unclear how these assessments translate into performance in the client’s lived environments (Payne, 2002).
Glass (1998), in his tenses of function model, considers the role of the environment as he defines three different types of performance: ▪ Hypothetical performance describes what one “can do” and is measured through patient-reported outcome measures (PROMs). ▪ Experimental performance describes what one “could do” and is measured using a performance-based assessment. ▪ Enacted performance describes what people “actually do” in their home environment and is measured by frequency and/or duration of an activity. Glass argues that the three tenses of function are distinct and that people may demonstrate different levels of performance on the basis of the supports and barriers present in the environment.
Although clients’ performance may change the basis of environment, the majority of occupational therapy practitioners in the United States are clinically based (AOTA, 2020a). This means that the practitioner must use available standardized assessments to infer clients’ performance at home and prepare treatment and discharge plans accordingly. The tenses of function model suggests that PROMs and performance-based assessments may not capture enacted performance at home over time. Clarity on how assessments translate between the tenses of function is important to inform occupational therapy practitioners as they make treatment plans and discharge recommendations.
The purpose of this study was to explore the relationship between hypothetical and experimental performance compared with enacted performance in the home, specifically for medication management and adherence. Medication management is a complex health management task that includes communicating with the prescriber, filling medication at the pharmacy, interpreting medication information, and taking medication as prescribed (AOTA, 2020b, p. 32). Medication adherence refers to how well a person takes their medication as prescribed (World Health Organization, 2003).
Practitioners use standardized assessments to identify people who are at risk for low adherence to reduce the likelihood of its negative effects. Specifically within medication management and adherence, by measuring hypothetical performance, PROMs are used to investigate a person’s perceptions of how well they take their medications (Kwan et al., 2020; Nassar et al., 2022). Performance-based assessments are used to investigate how well a person performs tasks associated with taking medication, such as sorting pills into a pillbox according to the directions and measuring experimental performance (Badawoud et al., 2020; Lam & Fresco, 2015; Stirratt et al., 2015). There is also a highly accurate measure of enacted performance that uses pill bottles with embedded electronic monitors to help practitioners understand how well people actually take their medication as prescribed over time (Choudhry et al., 2022; Vrijens et al., 2017; Williams et al., 2013). The availability of assessments across tenses of function make medication management and adherence an optimal area for exploring the relationship between types of performance.
To understand the relationship between the tenses of function in the context of medication management and adherence, we sought to complete two aims. First, we sought to quantify concurrent validity across medication management and adherence assessments from different tenses of function. Second, we sought to understand the predictive validity of PROMs (measuring hypothetical performance) and performance-based assessments (measuring experimental performance) on medication adherence, as measured by electronic monitoring (enacted performance).
Method
Design
Because of the lack of preexisting information about how assessments translate across the tenses of function, researchers used an exploratory approach, administering a series of PROMs, performance-based assessments, and objective observation measures. The Washington University in St. Louis’s Institutional Review Board reviewed and approved all research methods.
Participants
Researchers recruited participants from a registry maintained by the university associated with its network of physicians. The inclusion criteria required participants to be adults (≥18 yr), prescribed a daily antihypertensive medication, free of significant cognitive impairment, independent in daily living activities, and independent in medication management.
Measures
The authors selected two PROMs and two performance-based measures. They selected the Hill-Bone Compliance to High Blood Pressure Therapy Scale and the Hill-Bone Medication Adherence Reasons Scale (MARS) because of their wide use and established psychometrics (Kim et al., 2000; Nassar et al., 2022; Unni et al., 2014, 2019). Researchers also selected the Performance Assessment of Self-Care Skills (PASS) and the Executive Function Performance Test–Enhanced (EFPT–e) because of their wide use by occupational therapy practitioners in considering medication management and adherence and established psychometric properties (Baum & Wolf, 2013; Boone & Wolf, 2019, 2021; Chisholm et al., 2014; Holm et al., 2008; Rogers, 1984; Schwartz & Richard, 2019).
Measures of Hypothetical Performance (Can Do)
The Hill-Bone Compliance to High Blood Pressure Therapy Scale is a nine-item self-report survey designed to measure behaviors that are associated with adherence (Kim et al., 2000). Participants indicated their adherence with taking medication on a 4-point Likert-type scale ranging from 1 (all of the time) to 4 (none of the time). Scores range from 9 to 36, with higher scores indicating better adherence. The assessment demonstrates acceptable reliability with Cronbach’s α ranging from .68 to .84 and an intraclass correlation coefficient (ICC) of .78 (Denguir et al., 2019; Krousel-Wood et al., 2005).
The MARS is a 20-item, self-report survey designed to measure the reasons for nonadherence (Unni et al., 2014, 2019). Participants reported both the frequency and reasons for nonadherence on a 7-point scale that records the number of days, ranging from 0 (this was not a reason for missing the medicine) to 7 (missed taking the medicine on all 7 days due to this reason). Responses were summed, with scores ranging from 0 to 140; higher scores indicated worse adherence. The assessment demonstrates acceptable reliability, with Cronbach’s α ranging from .85 to .95 (Unni et al., 2014).
Measures of Experimental Performance (Could Do)
The PASS is a performance-based assessment designed to objectively measure occupational performance in daily living activities (Chisholm et al., 2014; Rogers, 1984). The Medication Management subtest asks participants to review two medication bottles and sort the medications into a pillbox. The assessment administrator provides increasing levels of cues that are needed to help the participant complete the task correctly. Participants are scored on independence, safety, and adequacy. Although, clinically, these three constructs are interpreted separately, we summed the scores to quantify the participant’s total performance, which is consistent with similar studies (Grenier et al., 2022). This also made the score comparable with that from the other performance-based test used in this study. Lower scores indicate worse performance (indicating that more cues were needed). The PASS has acceptable test–retest reliability (rs = .82–.92) and interrater reliability (90%–96%; Holm et al., 2008). In terms of validity, the PASS can discriminate between known groups such as adults with versus without depression or heart failure (Holm et al., 2008). The PASS was also able to differentiate adults with depression who were assigned to different levels of care (Holm et al., 2008).
The EFPT–e is a performance-based assessment designed to measure executive function in the context of performing a task (Baum & Wolf, 2013; Boone & Wolf, 2019, 2021). In the EFPT–e medication management subtask, participants review five medication bottles and labels, ignore three distractors, and sort the needed medications into a pillbox. The participant then describes how to take the medications according to the initial instructions. The assessment administrator provides increasing levels of cues needed to help the participant complete the task correctly. The assessment administrator scores participants on the type and frequency of cues received in the areas of initiation, organization, sequencing, judgment, and completion, which are summed for a total score. Higher scores indicate worse performance (indicating that more cues were needed). Researchers also record time to complete the EFPT–e. The EFPT–e has acceptable reliability (ICCs = .90–.98; Boone & Wolf, 2021). In terms of validity, the assessment was able to differentiate between known groups with versus without cancer (Boone & Wolf, 2021).
Measure of Enacted Performance (Actually Do)
The Medication Event Monitoring System (MEMS) measured medication adherence over 30 days at home. Researchers placed an electronic cap on a participant’s assigned pill bottle filled with their antihypertensive medications. The electronic cap recorded each time the medication bottle was opened. All participants used the electronic monitoring for one antihypertensive medication taken daily. For Aim 1 (concurrent validity), medication adherence was evaluated as a continuous variable. For Aim 2 (predictive validity), adherence scores were bifurcated into high adherence (≥90%) and low adherence (<90%), because this threshold is associated with meaningful differences in clinical outcomes (Baumgartner et al., 2018; Dalli et al., 2021; Yang et al., 2017). Although adherence assessment has strengths and weaknesses, experts consider electronic monitoring to be the most accurate, and it is the reference standard for validating other measures (Lam & Fresco, 2015; Vrijens et al., 2017). Participants who used pillboxes in the study were told to put their antihypertensive medication in the MEMS pill bottle and store the pill bottle on top of the pillbox. Participants who indicated a concern about deviating from their regular habits and routines were instructed to leave their medications in their pillbox and place the MEMS pill bottle on top of the pillbox, opening and closing the MEMS cap whenever the antihypertensive medication was taken.
Training and Fidelity
Three researchers, two experienced occupational therapists and an occupational therapy graduate student, implemented the PASS and the EFPT–e. Researchers read the respective manuals, watched official training videos, and practiced the assessments with an experienced colleague. Researchers met weekly and discussed questions and issues around study implementation with the goal of enhancing consistency between researchers. To monitor fidelity, researchers recorded the administration of the PASS and EFPT–e. A second trained researcher reviewed 20% of the (randomly selected) sessions for fidelity to the EFPT–e and PASS procedures, using a checklist that was based on the administration manual. The fidelity checklists can be found in the Supplemental Material (available online with this article at https://research.aota.org/ajot). The second trained researcher also independently scored the EFPT–e and PASS. Discrepancies in scoring triggered a meeting to review scoring.
Data Collection
Researchers used the Research Electronic Data Capture (REDCap) tool to support the data collection process. Researchers screened potential participants on the phone. If participants met inclusion criteria, the researcher and participant reviewed the informed consent. Participants who chose to continue in the study completed a demographic assessment and the PROMs. Then, participants met with a trained researcher to complete the performance-based assessments of medication management. Researchers administered most assessments in a university research lab setting. Participants without access to transportation were eligible to conduct the study by means of a home visit. In the home visit, a researcher administered the clinic version of the assessments using the same protocol. Next, the researcher instructed the participant on how to use the electronic medication cap to track their antihypertensive medication adherence. Participants then used the cap for 30 days. On completion, participants mailed the cap back to the research team.
Data Analysis
Researchers exported data directly from REDCap to SPSS (Version 27) for data analysis. Researchers began with descriptive statistics and visually inspecting the data points. For Aim 1 (concurrent validity), researchers used Spearman’s rank-order correlations to evaluate relationships with the Hill-Bone Medication Adherence Scale, MARS, EFPT–e, and PASS scores and Pearson’s correlation to evaluate the relationship with time to complete the EFPT–e. For Research Question 2 (predictive validity), researchers conducted a receiver-operating characteristic (ROC) analysis on how well the Hill-Bone Medication Adherence Scale, MARS, EFPT–e score, EFPT–e time to complete, and PASS score predicted a 30-day adherence rate of ≥90%, indicating high adherence. Participants with less than 50% adherence were deemed to be outliers and were dropped from analyses. To determine outliers, we visualized all variables on box plots, histograms, and scatterplots. Participants with very low adherence were found to be outliers across visualizations. This affected 5 participants whose adherence ranged from 17% to 41%. We believe that these participants did not use the supplied electronic monitoring caps.
Results
Of the people who clicked on the online study flyer, 67% proceeded to screening. Of the people who were screened, 50% were found eligible and consented to participate in the study. Of those who enrolled, there was a 16% attrition rate. Sixty people on the cardiovascular disease continuum from hypertension to moderate stroke completed the study. The demographics (Table 1) were consistent with the census demographics from the metropolitan area in Missouri where the participants were recruited. Scores across assessment type are presented in Table 2.
Participant Demographics
Note. Participants’ mean age was 64.70 yr (SD = 11.33). GED = general equivalency diploma.
Range and Mean Scores on Medication Management Measures
Note. EFPT–e = Executive Function Performance Test–Enhanced; PASS = Performance Assessment of Self-Care Skills, Medication Management subtest.
Concurrent Validity
Table 3 presents correlations with medication adherence as measured by MEMS cap. Additionally, the Hill-Bone Medication Adherence Scale and the MARS had a correlation of −.52 (p < .01), and the EFPT–e and the PASS total score had a correlation of −.44 (p < .01).
Correlations Between Performance-Based Measures of Medication Management and Medication Adherence
Note. EFPT–e = Executive Function Performance Test–Enhanced; PASS = Performance Assessment of Self-Care Skills, Medication Management subtest.
p < .05.
Predictive Validity
Table 4 displays the predictive validity of hypothetical and experimental performance measures to enacted performance measures. The area under the curve ranged from .58 to .67.
Predictive Validity as Measured by Area Under the Curve
Note. EFPT–e = Executive Function Performance Test–Enhanced; PASS = Performance Assessment of Self-Care Skills, Medication Management subtest.
Discussion
Standardized assessments help occupational therapy practitioners to engage in treatment and discharge planning in a rigorous and consistent manner. It is unclear, however, how well standardized assessments that are completed in one setting (i.e., the clinic) translate to performance in another setting (i.e., the home). Therefore, the purpose of this study was to quantify the concurrent and predictive validity of PROMs and performance-based assessments to performance at home using a series of medication management and adherence assessments.
The PROMs and performance-based measures have previously shown the ability to discriminate between two known groups, establishing validity (Boone & Wolf, 2021; Holm et al., 2008). Subsequently, occupational therapy practitioners and scholars hypothesized that PROMs and performance-based assessments would translate to performance at home. The tenses of function model, however, predicted different levels of performance across contexts. The results of this study indicated poor concurrent and predictive validity of medication management and adherence assessments across the tenses of function, aligning with Glass’s tenses of function model. In terms of concurrent validity between hypothetical (“can-do”) or experimental (“could-do”) and enacted (“actually do”) assessments, the correlations were weak, indicating little to no relationship. For predictive validity, the area under the curve ranged from .58 to .67. An area under of the curve of .65 indicates that there is a 65% chance that the assessment will correctly predict whether a person will have high medication adherence. In diagnostic medicine, the guidelines for assessment adoption require an area under the curve of .70 or greater (Mandrekar, 2010). These findings indicate that both surveys and performance-based assessments alone lack the precision needed to predict a client’s medication adherence at home over time.
The broader literature is consistent with the findings from this study. When the assessments were developed, concurrent validity was measured as the relationship between similar assessments within the same functional category (Baum & Wolf, 2013; Boone & Wolf, 2021; Holm et al., 2008; Kim et al., 2000; Unni et al., 2019). In this study, the relationships of the greatest strength were between assessments of the same category (i.e., the two PROMs or the two performance-based assessments). For PROMs, a systematic review found 36 PROMs, of which only 13 used MEMS in the validation process (Nguyen et al., 2014). Of those 13, only 11 demonstrated a significant correlation with electronic monitoring. Consistent with that finding, in this study, the Hill-Bone Medication Adherence Scale had a significant but weak correlation with electronic monitoring. In terms of performance-based measures, Grenier et al. (2022) examined the ability of the PASS Medication Management subtest to predict adverse events on hospital discharge. Grenier et al. (2022) found an AUC of .71, which just exceeds the .64 found in this study, as well as the .70 threshold for clinical usefulness.
Our findings and the broader literature suggest three reasons why assessments are failing to translate between tenses of function. First, standardized assessments frequently do not consider the environmental barriers and facilitators affecting a person’s independence with medication management and adherence, which can limit the generalizability of findings from the clinic to the home environment. This is consistent with the findings of Provencher et al. (2012), who found differences between task performance in the clinic and at home.
Second, as suggested by Glass (1998), it is possible that assessments across tenses of function are measuring different things. Specifically, the assessment developers report that the Hill-Bone Medication Adherence Scale measures “behaviors” that are associated with adherence, the MARS measures reasons for nonadherence, the EFPT–e measures “cognitive function,” and the PASS measures the level of assistance needed to complete “functional tasks” (Baum & Wolf, 2013; Chisholm et al., 2014; Holm et al., 2008; Kim et al., 2000; Unni et al., 2019). Although each assessment is reported to measure different things, practitioners use these tools interchangeably to better understand a person’s medication management and adherence (Nassar et al., 2022; Schwartz & Richard, 2019). Additionally, the broader adherence literature suggests that a specific construct, such as executive function, can be used to identify people at risk for low adherence (Insel et al., 2006). The findings of this study, however, conversely suggest that practitioners cannot use these assessments interchangeably and that performance on one construct of medication management and adherence is not translating to performance at home over time.
Finally, social desirability bias—the tendency to respond in a pleasing way (Villaseñor et al., 2017)—can impair how standardized assessments translate from the clinic to the home. In the context of medication management and adherence, people know that it is desirable to health care professionals for patients take their medications. Subsequently this means that, on PROMs, people may overestimate their medication adherence and be more thorough on performance-based assessments than in real-life situations (Wilson et al., 2009). Although there are measures of social desirability that researchers can deploy, this adds to an already high burden of assessments for clinicians and researchers.
Limitations and Future Research
Limitations constrain generalization of this study. Although we were able to attain a diverse medium-sized sample, many of the statistical analyses failed to meet the threshold for significance. Findings from this study warrant future research with larger sample sizes. It remains problematic, however, that the PROMs and performance-based assessments were unable to be used in a clinically meaningful way across a sample of 60 participants. Second, it is possible, as with any study with electronic medication adherence monitors, that the participants changed their behavior, because they recognized that their adherence was being watched (i.e., the Hawthorne effect). Therefore, these assessments may perform differently outside the context of a research study. Finally, this study investigated the performance of a few assessments related to medication management and adherence. In typical clinical practice, determinations are made on the basis of a full evaluation, including chart review, performance in other occupational areas, understanding of what is important to the client, and barriers and facilitators in the environment. Future work should consider the use of a composite measure that combines factors that affect medication adherence and incorporates practitioners’ clinical reasoning.
Implications for Occupational Therapy Practice
Findings from this study indicate that, in the context of medication management and adherence, a single PROM or performance-based assessment demonstrates poor content and predictive validity. Despite the negative results of the study, the findings have the following implications for occupational therapy practice: ▪ Practitioners should use valid and reliable standardized assessments in occupational therapy practice. ▪ Practitioners may continue to use the assessments tested in this study to identify deficits or occupational performance breakdowns that affect the performance of medication management and adherence activities. ▪ Practitioners should remain mindful of the limitations of the assessments tested in this study and not overinterpret the assessments to assume that performance in clinical settings will reflect performance at home. ▪ Practitioners should implement a full evaluation composed of several assessments, the clinical reasoning of the practitioner, and consideration of a client’s preferences and goals to guide clinical decision-making.
Conclusion
The purpose of this study was to understand how PROMs and performance-based assessments translate to performance at home over time in the context of medication management and adherence. We explored the differences between what clients “think they can do” on a PROM, “could do” on a performance-based assessment, and “actually do” at home over time. The findings of poor concurrent and predictive validity suggest that the PROMs and performance-based assessments in medication management and adherence tested in this study are not related to how well a client will manage their medication at home over time. Subsequently, new measures are needed to improve identification of people at risk for low adherence. Occupational therapy practitioners should continue to use a battery of assessments, clinical reasoning, and a client’s goals and preferences to inform treatment and discharge planning. To demonstrate value to clients and payers, it is essential for tools in health care to have strong psychometrics related to meaningful outcomes at home.
Supplemental Material
Supplementary material for Performance of Medication Tasks: Relationship Among Patient-Reported Outcomes, Performance-Based Assessments, and Objective Assessments
Supplementary material, sj-pdf-1-aot-10.5014_ajot.2024.050500.pdf for Performance of Medication Tasks: Relationship Among Patient-Reported Outcomes, Performance-Based Assessments, and Objective Assessments by Jaclyn K. Schwartz, Katherine Aylmer, Samara Green, Sami Tayeb, Timothy J. Wolf, Elizabeth Unni and Emily Somerville in The American Journal of Occupational Therapy
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
