Abstract
The study findings suggest that the Activity Measure for Post-Acute Care Applied Cognitive Inpatient Short Form “6-Clicks” can reliably be used to identify impairments, optimize patients’ function, and facilitate safe discharge planning in the acute care setting.
Patients admitted to acute care hospitals often experience cognitive impairment that can be devastating and life-altering. Although up to 50% of adults experience delirium, characterized as an acute change in cognition, altered arousal levels, and attentional deficits, and 80% of adults experience new or worsening cognitive impairment, which is broadly defined as a deficiency in thought process, knowledge, or judgment, many cases often go undetected during and after a hospital stay (Amini et al., 2019; Casey et al., 2023; Fuchs et al., 2020; Geense et al., 2021; Gonzalez Kelso & Tadi, 2022; Jackson et al., 2016; Xu et al., 2022). Cognitive impairment can result in decreased independence in activities of daily living (ADLs) such as bathing and dressing, as well as instrumental activities of daily living (IADLs) such as medication management, meal preparation, bill paying, and returning to work (Bardesono et al., 2022; Casey et al., 2023; Ernst et al., 2020; Jackson et al., 2016). Previous research found that cognitive impairments have also been associated with increased length of hospital stay, complex discharge planning needs, risk of hospital readmission, inflated cost of care, and adverse effects on mental health after discharge (Honarmand et al., 2020; Xu et al., 2022). It is critical that cognitive impairment is recognized and addressed promptly in the hospital setting.
Occupational therapy is a skilled rehabilitative service that uses a systematic approach to integrate meaningful activities into daily care in acute care hospitals (Pozzi et al., 2020; Xu et al., 2022). Occupational therapy practitioners play a critical role in the screening, identification, and treatment of cognitive impairment, yet they only evaluate a small fraction of all hospitalized patients (Casey et al., 2023; Pozzi et al., 2020). A promising approach is to form a collaborative partnership with nursing staff for cognitive screening purposes. Nurses, who dedicate extensive time to providing direct bedside care from the onset of a patient’s hospitalization, are in an optimal position to conduct such screenings. Moreover, they already factor in cognitive considerations for certain patient groups and within various risk assessments, such as the Johns Hopkins Fall Risk Assessment Tool, the Short Orientation–Memory–Concentration Test, and the 4AT (Ihle-Hansen et al., 2023; Gonzalez Kelso & Tadi, 2022). Yet, before implementation of a standardized approach for cognitive impairment screening across all patient demographics, it is essential to establish the consistency of evaluations between therapists—including occupational therapy practitioners and speech-language pathologists—and nursing staff. At present, research into the consistency of these assessments between such health care providers is lacking. Confirming interrater reliability among health care providers on a cognitive screen ensures consistent and accurate assessment, fostering collaborative and cohesive patient care. There is currently little research exploring the interrater reliability between therapists and nurses on tools screening cognition, such as the Activity Measure for Post-Acute Care Applied Cognitive Inpatient Short Form “6-Clicks” (AM-PAC ACISF).
Evaluating cognitive impairment is not routinely practiced in acute hospital settings (Xiong et al., 2024). Instituting a universal screening protocol for all patients could facilitate more focused assessments and interventions. Such widespread screening necessitates a tool that is concise, valid, and reliable. A holistic strategy to manage cognitive impairment must encompass initial screening, accurate identification, and subsequent treatment (Casey et al., 2023; Pozzi et al., 2020). Initiating cognitive function screening in the dynamic environment of acute care hospitals provides a quick indicator of a person’s cognitive function and should be feasible in this setting (Bardesono et al., 2022; Casey et al., 2023). This initial screening serves as a precursor to more detailed evaluations, guiding the subsequent care pathway. Should the screening reveal cognitive concerns, occupational therapy practitioners may conduct in-depth assessments, gaining insights into specific cognitive deficits to tailor personalized treatment plans (Casey et al., 2023; Pozzi et al., 2020). Standardized screening also could enhance collaborative care discussions among the interdisciplinary health care team. Toward these goals, the AM-PAC ACISF is a brief, standardized screening tool that is reliable, valid, and easy to use by occupational therapy practitioners in the acute hospital (Casey et al., 2023). The AM-PAC ACISF is part of the entire AM-PAC, a comprehensive measurement system for patients with various diagnoses used across practice settings to assess activity, mobility, and cognition (Jette et al., 2014).
Standardized interdisciplinary screening for patient limitations with mobility and activity has led to favorable patient-centered outcomes, such as identifying appropriate discharge disposition and limiting readmissions (Hoyer et al., 2018; Jackson et al., 2016; Jette et al., 2014; Young et al., 2020). Establishing a standardized screening process for cognition is essential for early identification of cognitive impairment, optimization of patient functioning, and facilitation of safe discharge planning. Therefore, the purpose of this study was to examine the nursing–therapist interrater reliability of the AM-PAC ACISF in the acute care hospital setting.
Method
Participants
For this study, we obtained approval from the Johns Hopkins University institutional review board. Consent from participants was not required because the data were collected as part of usual care. The study was conducted at an academic medical institution on four acute inpatient hospital units treating patients with general medicine, cardiac, and surgical diagnoses. Patients were included in this study if they were assigned to a study nurse on the day of data collection and if they met the inclusion criteria of being English-speaking and at least 18 yr of age. Patients were excluded if they were pregnant, imprisoned or under arrest, blind, deaf, or refused nursing care or the presence of the observing therapist.
The clinician participants in this study included six nurses, five occupational therapy practitioners, and one speech-language pathologist. The occupational therapy practitioners had varying years of experience, ranging from less than 1 yr to 16 yr, with a median of 4 yr. Four of the occupational therapy practitioners held a clinical doctorate. The speech-language pathologist had 7 yr of experience and received training at the master’s level. The nurses had experience ranging from 3 yr to 13 yr, with a median of 6.25 yr. One nurse held a master’s degree in nursing, whereas the remaining nurses all had a bachelor of science degree.
Cognitive Screening Instrument
The AM-PAC ACISF has six items scored through direct observation or proxy judgment of a person’s ability to complete distinct functional cognitive tasks. The six different cognitive tasks include following a speech or presentation, taking medication, understanding ordinary conversation, remembering a list of four to five errands, remembering where things were placed or put away, and taking care of complicated tasks. The clinician scores each item on a 4-point ordinal scale based on their direct observation or judgment. A score of 1 indicates that the patient is unable to complete the activity without assistance or cues, whereas a score of 4 indicates that the patient can complete the cognitive task independently without additional time or effort. This yields a raw score that ranges from 6 to 24, with a higher score indicating less functional cognitive impairment. In this study, we converted the raw score into a standardized T score that was used in all analyses. The standardized score had a population mean of 50 and a standard deviation of 10 (AM-PAC, n.d.).
Study Procedures
The principal investigator and coinvestigator (Kelly Casey and Erin Sim) recruited nurses to participate in the study by reaching out to nursing managers and advertising during nursing staff huddles. All study clinicians received training on how to administer the AM-PAC ACISF. Nurses participated in a 13-min virtual presentation (live or recorded) during which the study objectives, the AM-PAC ACISF, and confirmation of interest were reviewed. An additional 9 min were kept open for questions. The presentation discussed the scoring of AM-PAC ACISF, addressed frequent questions and concerns, and discussed case studies based on the presenters’ clinical experiences. The course presenters were Casey and Sim, who were also occupational therapy practitioners with extensive experience in administering the AM-PAC ACISF. Study therapists (occupational therapy practitioners and speech-language pathologists) reviewed the AM-PAC ACISF with study nurses for approximately 5 min, answering any questions and reviewing scoring guidelines, on the first day of data collection. Study therapists were familiar with the AM-PAC ACISF administration and scoring; however, all study therapists met with the principal and coinvestigators before data collection to review the tool and study procedures.
We conducted data collection over 15 separate sessions from October 2021 to February 2022 based on the mutual availability of therapist and nurse participants. A study nurse was paired with a study therapist on the data collection day. The patients already assigned to that nurse for regular work that day were the patients included in the study. The study therapist reviewed AM-PAC ACISF scoring with the nurse and answered all questions before seeing any patients. Then the study therapist observed the nurse in real time completing their routine morning patient care. After leaving the patient’s room, the therapist and the nurse independently scored the AM-PAC ACISF based on these observations. The nurses were instructed to not discuss the patient, their observations, or the scoring. The study therapist also recorded demographic information obtained from the patient chart. This process was repeated for all the patients on the nurse’s caseload for that day during a 2-hr period.
Data Analysis
We calculated interrater agreement for each item of the AM-PAC ACISF with a linearly weighted κ using the therapists’ scores. The internal consistency for each item of the AM-PAC ACISF was evaluated using the Cronbach α statistic separately for nurses and therapists. We compared interrater reliability of total AM-PAC ACISF scores between therapists and nurses with the intraclass correlation coefficient (ICC) calculated using a two-way random-effects ICC model. On the basis of a prior similar study (Casey et al., 2023), we estimated that a sample size of 50 patients was needed to have a power of 0.80 and a minimum interrater ICC of 0.85.
Results
Interrater Reliability
A total of 50 patients whose ages ranged from 29 to 94 yr with an average of 63 yr (Table 1) were assessed. The most common patient diagnoses were cardiac, followed by general medicine or other, and then respiratory or COVID (Table 1). The mean AM-PAC ACISF score (standardized T score) across all patients was 41.8 (SD = 17.93) for therapists and 42.73 (SD = 17.98) for nurses. For the AM-PAC ACISF, the minimum and maximum observed scores for both therapists and nurses were 6 and 24. Interrater reliability (ICC) for AM-PAC ACISF total scores comparing nurses with therapists was 0.88, with a 95% confidence interval from 0.79 to 0.93 (Table 2).
Characteristics of Patient Participants (N = 50)
AM-PAC ACISF Item Agreement Between Nurses and Therapists
Note. Linear weighting was used to compare nurses’ scores with therapists’ scores (occupational therapy practitioners and speech-language pathologist) for each item. AM-PAC ACISF = Activity Measure for Post-Acute Care Applied Cognitive Inpatient Short Form.
The internal consistency reliability of the AM-PAC ACISF scores was 0.97 (95% CI [0.96, 0.98]) recorded by therapists (occupational therapy practitioners and speech-language-pathologist) and was 0.96 (95% CI [0.94, 0.98]) recorded by nurses. The linearly weighted κ values for item agreement ranged from 0.35 (95% CI [0.18, 0.53]) for the question of how much difficulty the patient has understanding familiar people during ordinary conversation to 0.67 (95% CI [0.52, 0.80]) for the question of how much difficulty the patient currently has remembering a list of four or five errands without writing it down (Table 2).
Discussion
In this prospective study, we evaluated the interrater reliability of the AM-PAC ACISF, comparing scores recorded by therapists (occupational therapy practitioners or speech-language pathologists) and nurses. We observed good interrater reliability between therapists and nurses. With good interrater reliability, the AM-PAC ACISF can serve as a common language for cognitive impairment screening in the hospital and guide discussions about patient cognition in the acute care setting. These results align with prior studies that used the AM-PAC to measure inpatient mobility, which similarly demonstrated high reliability across different professional disciplines, including physical therapy and nursing. This measure is currently being used for clinical discussions pertaining to patient mobility (Hoyer et al., 2018).
Prior research suggests that a raw score below 22 on the AM-PAC ACISF is indicative of cognitive impairment requiring further evaluation and intervention (Casey et al., 2023). Verifying that nurses can effectively use the AM-PAC ACISF for cognitive impairment screening enables its use in guiding discussions among health care professionals about patients with cognitive impairment. This can facilitate timely consultations with speech-language pathologists or occupational therapy practitioners, aiming to address cognitive issues and enhance patients’ functional independence.
Training individuals to use the AM-PAC ACISF was relatively quick and easy. Training for study nurses was completed within 30 min, including addressing questions. The training was conducted online and recorded for individuals who could not attend live sessions. Considering the strong interrater reliability demonstrated, these training methods could be suitably replicated for broader training initiatives ahead of widespread system implementation. The scoring process is notably efficient, requiring less than 1 min to complete (Casey et al., 2023). This efficiency suggests that incorporating the AM-PAC ACISF into nursing documentation would not be overly burdensome. Furthermore, it offers the advantage of early detection and ongoing monitoring of cognitive impairment during a patient’s hospital stay, facilitating earlier consultations with relevant professionals such as occupational therapy practitioners or speech-language pathologists.
Using nurses for cognitive screening in patient care presents numerous benefits and aids in standardizing discussions regarding cognition in acute care settings. Previous research has asserted that cognitive impairment can be missed in the acute hospital (Jackson et al., 2016). Implementing more standardized screening methods could aid in detecting previously overlooked cognitive impairments (Bickel et al., 2018). Recognizing these impairments is crucial, because they can affect a patient’s capability to perform key tasks such as adhering to health care appointments, managing medications, and safely navigating their environment (Amini et al., 2019; Ando et al., 2018; Becker et al., 2021; Ernst et al., 2020; Geense et al., 2021; Hopkins et al., 2017; Vannorsdall et al., 2022; Wilcox et al., 2021).
Because the AM-PAC ACISF is efficient and effective from an interdisciplinary perspective, it can be used in a standardized manner to screen for cognitive impairment. The AM-PAC ACISF is appropriate for screening all patients in acute care for cognitive impairment, including those with delirium, because the purpose of the screening tool is to identify any patient’s ability to complete distinct functional cognitive tasks. With interdisciplinary use, the AM-PAC ACISF can be used to discuss patients’ rehabilitation needs during multidisciplinary rounds to identify cognitive changes and the need for rehabilitation consults, as well as increase the focus on and understanding of cognitive impairment in acute care. Furthermore, it can be used in other care settings (e.g., home care, outpatient rehabilitation), with the ability for the standardized score to be used across the continuum (Casey et al., 2023).
In conjunction with standardized, interdisciplinary use, future research could examine how patient cognition affects discharge disposition, such as exploring the predictive nature of the AM-PAC Basic Mobility or the AM-PAC Daily Activity Inpatient Short Forms on discharge disposition (Harry et al., 2021; Young et al., 2020). Additional research could use the AM-PAC ACISF as a measurement tool to examine the impact of occupational therapy interventions on functional cognition.
Study Limitations
Data collection for this study occurred during the COVID-19 pandemic, and we therefore experienced several challenges. During the recruitment and data collection phase, several nurses who were trained or were interested in participating in the study either left their positions or transitioned to a different role or shift that excluded them from participation. Given the staffing challenges faced during the pandemic, a convenience sampling method was used. We advertised the study on units with interested leaders, and the nursing administration provided us with the names of interested nurses. With this convenience sampling method, the nurses recruited may have been more engaged than the average bedside nurse, which could potentially affect generalizability. In addition, data collection was scheduled based on the mutual availability of a study nurse and a study therapist, meaning that each unit was not represented equally. This affected the variety of the patients included in the study. Despite this, there was a diverse range of patient diagnoses spanning medical and surgical services.
Implications for Occupational Therapy Practice
When admitted to the hospital, patients can experience a cognitive decline for myriad reasons. Impaired cognition correlates with post-discharge issues, including decreased independence with ADLs and IADLs, risk of hospital readmission, increased costs of care, and adverse effects on mental health (Casey et al., 2023; Ernst et al., 2020; Honarmand et al., 2020; Jackson et al., 2016; Xu et al., 2022); therefore, addressing cognitive impairment is an important priority. Occupational therapy practitioners can evaluate and treat these impairments when consulted. In addition, this study has the following implications for occupational therapy practice: Nursing and therapy staff had good interrater reliability on scoring the AM-PAC ACISF. As such, this tool could be used as a common language for early identification of cognitive impairment to target appropriate occupational therapy consults. With the increased frequency of standardized measurement, there is an opportunity for greater attention, focus, and understanding from the interdisciplinary team (occupational therapy practitioners, speech-language pathologists, nurses, social workers, etc.) of the importance of addressing hospitalized patients’ cognitive impairments. Systematic screening of cognition could help early identification of patients whose cognitive status changes during hospitalization. Identification of cognitive impairment using the AM-PAC ACISF can be an important part of occupational therapy evaluation; however, cognitive screening tools should not be used in isolation. The results of the AM-PAC ACISF can contribute to the creation of specific occupational therapy goals and client-centered, occupation-based interventions to address specific cognitive impairments. Conducting early screenings for cognitive impairment during a hospital stay is a crucial measure for preventing long-term functional impairment and helps to facilitate a swift return to engaging in meaningful occupations (Casey et al., 2023). Further research is needed to determine the impact of interprofessional, standardized functional cognitive screening on patient outcomes and system benefits. Occupational therapy practitioners should be involved in this research to solidify the profession’s role in assessing and treating functional cognition.
Conclusion
The study’s finding suggests that the AM-PAC ACISF is a reliable tool for use by interdisciplinary teams in the acute care setting. It fosters a standardized approach for the early identification of cognitive impairment, establishing a common language among clinicians and guiding clinical discussions regarding patient cognition. Early detection of cognitive impairment is essential for optimizing patients’ function and occupational engagement, facilitating timely occupational therapy interventions, and supporting safe discharge planning. The AM-PAC ACISF can play a critical role in improving communication and decision-making regarding cognitive assessments in the acute care environment.
Footnotes
Acknowledgments
We acknowledge the Johns Hopkins Hospital nurses, occupational therapy practitioners, and speech-language pathologists who participated in this study as data collectors.