Abstract
This study tested and compared the psychometrics and feasibility of three cognitive screens and concludes that each tool demonstrated acceptable reliability and construct validity in acute care hospitals, with the Activity Measure for Post-Acute Care “6-Clicks” Applied Cognitive Inpatient Short Form (AM-PAC ACISF) overall having the optimum mix of performance and feasibility.
Cognitive impairment is associated with poor hospital outcomes and is common among adults admitted to acute care hospitals. For example, Cameron et al. (2017) estimated that 25% to 80% of patients with heart failure have cognitive impairment. Cognitive impairment contributes to impairments in activities of daily living (ADLs) and instrumental activities of daily living, including medication management, return to work, driving, and social participation and leisure (Petersen et al., 2014; Read et al., 2020). Cognitive impairment and delirium also have been linked to discharge to postacute care settings rather than home, increased risk of hospital readmission, and increased need for assistance with ADLs (Jackson et al., 2016). However, cognitive impairment is often unidentified or undiagnosed in acute care hospital settings (Amini et al., 2019). Cognitive impairment is essential to identify during the hospital stay so that providers can promptly begin addressing functional deficits and safe discharge planning (Pritchard et al., 2019; Rogers et al., 2017).
Health care has undergone a dramatic shift in focus, from volume to quality and value, since passage of the Patient Protection and Affordable Care Act in 2010 (Pub. L. 111-148; Pritchard et al., 2019). This shift has expanded the opportunities for occupational therapy practitioners in acute care hospitals to provide quality care (Pritchard et al., 2019). A study illustrating the distinct value of occupational therapy was conducted by Rogers et al. (2017), who examined spending across 7,000 medical units and revealed that occupational therapy was the only category associated with fewer readmissions, an important quality indicator.
One of the essential occupational therapy roles in acute care is screening for, evaluating, and treating cognitive impairment. Cognitive screening tools give a broad overview of a person’s cognitive functioning and can be administered in a shorter time frame than cognitive assessments, which give more in-depth results on specific cognitive impairments but take longer to administer. Identifying an optimal screen to assess for cognitive deficits in the fast-paced acute care hospital setting is critical not just for occupational therapy practitioners; screening for cognitive impairment is the responsibility of the entire interdisciplinary team throughout the patient’s hospital stay.
The Montreal Cognitive Assessment (MoCA) is commonly used to identify cognitive impairment and is often considered the gold standard (Cameron et al., 2017; Nasreddine et al., 2005); however, it has not been fully tested in acute care settings. Additionally, the MoCA can take more than 20 min (Saczynski et al., 2015) and requires training and certification to administer, presenting challenges for widespread use in acute care hospitals. Several other screening tools assess cognitive impairment, but they have not been compared to the MoCA or to each other. For this study, we selected two screens on the basis of convenience and their specific attributes: the Brief Cognitive Assessment Tool Short Form (BCAT–SF) and the Activity Measure for Post-Acute Care (AM-PAC) “6-Clicks” Applied Cognitive Inpatient Short Form (ACISF). Although each of these screens assesses different domains of cognition, we sought to examine screens for general cognitive deficiencies, and all three can serve that purpose.
The BCAT–SF was designed as a cognitive screening tool for frontline providers. It is very brief (3–4 min), easily administered, cost-effective, and psychometrically robust (Mansbach & MacDougall, 2012). The AM-PAC ACISF is part of a larger instrument designed to examine Applied Cognition, Daily Activities, and Basic Mobility (Haley et al., 2004). Like the BCAT–SF, the AM-PAC ACISF is brief, easily administered, and psychometrically sound (Andres et al., 2003; Haley et al., 2004). In addition, the AM-PAC mobility and activity short forms were already being used by therapists and nurses at the Johns Hopkins Hospital, where this study was conducted.
Although all these tools have known reliability and validity, limited psychometric testing has been conducted with acute care hospital patients and providers. Therefore, the purpose of this study was to test the reliability, construct validity, and clinical usability of the BCAT–SF and AM-PAC ACISF and compare them to those of the MoCA with adults and therapists in an acute care hospital.
Method
Participants
This study was approved by the Johns Hopkins institutional review board (IRB00176092). Patients on the cardiac surgery and cardiac medicine units at the Johns Hopkins Hospital were recruited, and they or their active health care agents provided oral consent. A sample size of 50 was determined to sufficiently power the analysis and to account for attrition (Cohen, 1992). The inclusion criteria were adults ages 18 to 90 yr who were admitted to the hospital and had a physical therapy, occupational therapy, or speech- language pathology order. Excluded were patients who did not speak English, were pregnant, were prisoners, and had blindness or deafness. One speech-language pathologist and four occupational therapists were selected by convenience; their years of experience ranged from 2 to 15.
Instruments
The MoCA is a 16-item screen with a maximum score of 30; higher scores indicate better cognitive function in the areas of visual–spatial and executive functioning, naming, memory, attention, language, abstraction, delayed recall, and orientation (Ciesielska et al., 2016). The MoCA is administered verbally, and patients respond either verbally or in writing. A score ≥26 indicates normal cognitive functioning (Davis et al., 2015). The MoCA can distinguish normal cognitive function from mild cognitive impairment or mild Alzheimer’s disease and has a test–retest reliability coefficient of .92, an internal consistency coefficient of .83, sensitivity of 100%, and specificity of 87% (Nasreddine et al., 2005). A 1-hr training and certification module is required to administer the MoCA.
The BCAT–SF is a quick six-item screen with a maximum score of 21; higher scores indicate better cognitive function in the areas of orientation, immediate verbal recall, immediate story recall, executive function, delayed story recall, and story recognition. The BCAT–SF is administered verbally, and patients respond verbally. A score ≥19 indicates normal cognitive function. It has good reliability, with an internal consistency coefficient of .86 and a test–retest reliability coefficient of .98 (Mansbach & MacDougall, 2012).
The AM-PAC ACISF is a six-item screen that assesses how much difficulty a person has completing six cognitive tasks: following a speech or presentation, understanding ordinary conversation, taking medications, remembering where things were placed or put away, remembering a list of four to five errands, and taking care of complicated tasks. Each item is scored using an ordinal scale from 0 (unable to do the task) to 4 (no difficulty completing the task), for a possible total score of 24; higher scores indicate better cognitive function. In a sample of patients in postacute care, Andres et al. (2003) found a test–retest reliability coefficient of .91 and an interrater reliability coefficient of .68. Designed as a patient-report measure, the AM-PAC ACISF can also be administered by a clinician or caregiver proxy. In the current study, a therapist scored the items on the basis of clinical observations of the patient (Jette et al., 2020). At the time of the study, no cutoff score identifying cognitive impairment had been determined.
One frequently asked question regarding administration and scoring of the AM-PAC ACISF is “What if the patient has not done the activity?” According to the AM-PAC Short Forms Manual 4.0 (Jette et al., 2020), items can be completed if the response can be reliably estimated. The AM-PAC ACISF asks how much difficulty a patient would have completing a task, such as remembering to take medications at the appropriate time. It is scored by four levels: none, meaning a patient does not experience any problems completing the activity; a little, meaning the patient can do the activity without help from another person but requires more time and effort; a lot, meaning the patient can do the activity without help from another person but requires a lot more time and effort; or unable, meaning the patient cannot complete the task at all or can only complete it with help from another person. It is scored on the basis of the amount of difficulty a patient experiences or is anticipated to experience in completing the identified task. If at any point it is estimated that the patient would need help or cues from another person, they are scored as unable.
Usability Assessments
Participants rated the burden, difficulty, and usefulness of the MoCA and BCAT–SF on a short, three-item survey using a 5-point Likert scale with both numbers and faces; patients did not rate the AM-PAC ACISF because it was scored by the therapists. The therapists also rated the same three questions related to burden, difficulty, and usefulness of each screen in a brief three-item survey using the same 5-point Likert scale. Burden, difficulty, and usefulness were left undefined for each patient and therapist to determine. Therapists also reported the time required to complete each screen with each participant.
Procedures
Participants were selected from among those with at least one therapy order (physical therapy, occupational therapy, speech-language pathology) on two inpatient acute care cardiac units: the cardiovascular progressive care unit (CVPCU) and the progressive cardiac care unit (PCCU). Participant names were sorted alphabetically using the electronic medical record and selected for testing using a pregenerated list of random numbers (e.g., 3 was the third patient alphabetically).
Four acute care occupational therapists and one speech-language pathologist collected data for this study. The therapists received training to administer each screen in a standardized manner, according to the specific verbiage provided by the screen developers (Jette et al., 2020; Nasreddine, 2010; Mansbach & MacDougall, 2012). The therapists were provided with a folder for each participant containing patient information and the screens in the order they were to be given. One of the four therapists was assigned to administer the screens and another to silently observe; both therapists scored each screen.
Participants were allowed to use glasses or hearing aids if needed. A sign was placed on the door indicating that testing was taking place and to enter only if absolutely necessary. If family members were present, they were instructed to remain quiet or to leave the room. The patient was able to participate either in bed with the head of bed elevated or in a chair. The bedside table was placed in front of the patient for written components. Participants first completed a verbal questionnaire pertaining to their prior level of function, including baseline functional level with ADLs and medication management, and their highest level of education; the therapist then recorded the patient’s age, sex, diagnosis, and medical record number from the medical record.
A random number generator was used to assign each patient a 1 or a 2, and testing was done in the following order: (1) BCAT–SF, patient-reported usability of the BCAT–SF, MoCA, patient-reported usability of the MoCA, and AM-PAC ACISF, or (2) MoCA, patient-reported usability of the MoCA, BCAT–SF, patient-reported usability of the BCAT–SF, and AM-PAC ACISF. Administration order was randomized to reduce potential confounding from fatigue or practice effects. To score the AM-PAC ACISF by proxy with consistent information, therapists always scored it last after observing the patient completing the BCAT–SF and the MoCA. Therapists provided no additional prompting to the patient during administration. Both therapists recorded the total time each screen took to administer and score. Participants completed the usability surveys for the BCAT–SF and MoCA by rating the difficulty, burden, and usefulness of each screen. After completion of testing, therapists informed participants that if the screens indicated that any cognitive impairments were present, the physician and therapy team would discuss them with patients as appropriate.
When participant testing was complete, the therapists completed their own usability survey for each screen. They returned the patient folders with completed forms to a study team member for data entry. Individual forms were scanned into a secure electronic storage system, and data were then manually entered into an electronic spreadsheet. Data validation was conducted by two study team members who reviewed paper copies of the screens and compared them with spreadsheet data. After data validation, all paper copies were shredded.
Data Analysis
Descriptive statistics, including frequencies and percentages for categorical variables and means and standard deviations for continuous variables, were calculated to summarize patient characteristics overall and by cognition status, patient survey data, screen administration time, and therapist survey data. For interrater reliability, intraclass correlation coefficients (ICCs) were calculated from the results of a linear random effects regression model. ICCs were calculated for all three screens, with 95% confidence intervals derived using a nonparametric bootstrap (Altman & Bland, 1983) in which participants were resampled with replacement. ICCs were high for all three instruments, so the average of the two raters’ results for each screen was used for subsequent analysis.
Construct validity was demonstrated using Pearson correlations for the BCAT–SF and AM-PAC ACISF with the MoCA. Clinical utility of the BCAT–SF and AM-PAC ACISF was evaluated by calculating their sensitivity and specificity using the MoCA as the gold standard. Because there was no a priori cutoff value for AM-PAC ACISF scores to define impaired cognition, sensitivity and specificity were calculated for a range of cutoff points to determine which value best distinguished between impaired and normal cognitive functioning. Two-category and three-category comparisons were evaluated with t tests and general linear models, respectively. All analyses were conducted using SAS Version 9.4 (SAS Institute, Cary, NC).
Results
Fifty participants (17 women, 33 men) completed the study. Participant ages ranged from 33 to 90 yr, with an average of 66.4 yr (Table 1). Of the 50 participants, 62% had cognitive impairment according to the BCAT–SF and MoCA. In addition, 13% of participants with cognitive impairment had only a physical therapy order and no occupational therapy or speech-language pathology order.
Participant Characteristics
Note. ADLs = activities of daily living.
Montreal Cognitive Assessment score <26.
Montreal Cognitive Assessment score ≥26.
Psychometric and Clinical Usability Results (N = 50)
Note. — = reference; CI = confidence interval; ICC = intraclass correlation coefficient.
Mean values of two raters.
We found a significant relationship between cognition status and education level. Participants with a high school education or less had lower scores (i.e., greater impairment) than those with at least some college on the MoCA (M = 17.0, SD = 4.9, vs. M = 21.8, SD = 4.9; p = .001), the BCAT–SF (M = 14.9, SD = 3.9, vs. M = 17.9, SD = 3.1; p = .004), and the AM-PAC ACISF (M = 12.1, SD = 4.0, vs. M = 16.5, SD = 4.9; p = .002). The relationship between cognition and age was as expected: The higher the age, the lower the cognition status. No significant relationships were found between cognition and gender, primary medical diagnosis, or self-reported prior level of ADL function.
Interrater reliability was high (.86–.98) for all three screens, so therapists’ scores were averaged for each patient for the MoCA (M = 19.8, SD = 5.4), the BCAT–SF (M = 16.6, SD = 3.7), and the AM-PAC ACISF (M = 14.7, SD = 5.0; Table 2). Scores on the BCAT–SF (r = .76) and AM-PAC ACISF (r = .79) were highly correlated with MoCA scores (p < .001), demonstrating construct validity. Using the MoCA as the gold standard, the AM-PAC ACISF cutoff indicating cognitive impairment was <22. For identifying cognitive impairment, the BCAT–SF and the AM-PAC ACISF had 100% sensitivity, whereas the AM-PAC ACISF had higher specificity (98%) than the BCAT–SF (74%; Table 2).
We found no significant differences in patient- perceived levels of burden, difficulty, or usefulness between the BCAT–SF and the MoCA. Average administration times were 13.3 min for the MoCA, 5.0 min for the BCAT–SF, and 1.0 min for the AM-PAC ACISF (p < .001; Table 3). We calculated time to complete the screens as the amount of time spent beyond routine care. For example, the AM-PAC ACISF requires observation of the patient during any functional cognitive task in the course of routine care, and this tool took 1 min longer than routine care to score. Likewise, the MoCA and the BCAT–SF took 13.3 and 5.0 min, respectively, beyond routine care to administer and score. The therapists considered the MoCA to be more useful yet more burdensome than the BCAT–SF and the AM-PAC ACISF, and they found none of the screens to be difficult (Table 3).
Screening Measures: Therapist-Reported Time to Administer and Usability
Overall and pairwise p < .001.
How useful was this assessment? Range = 1 (useless) to 5 (very useful); overall p = .039; BCAT–SF and AM-PAC ACISF, p > .05; MoCA and BCAT–SF, p = .019; MoCA and AM-PAC ACISF, p = .039.
How burdensome was this assessment? Range = 1 (very burdensome) to 5 (not at all); overall p = .016; BCAT–SF and AM-PAC ACISF, p > .05; MoCA and BCAT–SF, p = .022; MoCA and AM-PAC ACISF, p = .007.
How difficult was this assessment to administer? Range = 1 (very difficult) to 5 (very easy), overall p = .152; all pairwise ps > .05.
Discussion
The BCAT–SF, AM-PAC ACISF, and MoCA all demonstrated acceptable interrater reliability, validity, and clinical usability. Correlations above .70 between the MoCA and the BCAT and AM-PAC ACISF were expected and indicate that these screens measure similar cognitive constructs (Cohen et al., 2002). The AM-PAC ACISF required significantly less time than the other screens to administer while maintaining high levels of sensitivity and specificity; high sensitivity is important when screening a large population for cognitive impairment. The AM-PAC ACISF was the least burdensome to administer, as reported by the therapists. Participants also reported generally positive feelings about the burden, usefulness, and difficulty of the BCAT–SF and MoCA; the therapist completed the AM-PAC ACISF, so patient burden was eliminated. We were able to establish a cutoff score of 22 for the AM-PAC ACISF to distinguish patients with and without cognitive impairment. This cutoff score may be useful for intervention planning and establishing patient-centered goals.
This study found a prevalence of cognitive impairment of 62% in a sample of hospitalized patients with therapy orders, consistent with a previous report of 66% in patients over age 65 yr (Boustani et al., 2010). Of the 50 participants in our study, 13% had orders for physical therapy but not for occupational therapy or speech-language pathology; this is of concern because occupational therapy and speech-language pathology practitioners are typically better trained than physical therapists to address cognitive deficits that influence functional performance and safe discharge. For patients referred to any therapy discipline, cognitive screening can capture impairments and inform consultation with other appropriate disciplines.
The AM-PAC ACISF appears to be a quick, valid, and reliable screening tool for cognitive impairment that fits well in the fast-paced acute care setting. Although all screens examined in this study were reliable, the low burden of administering the AM-PAC ACISF makes it the most usable in acute care hospitals. For the MoCA and BCAT–SF, the examiner incorporates specific cognitive tasks into their questions, whereas for the AM-PAC ACISF, the examiner observes any cognitive tasks during usual care to score the items by proxy. Thus, the AM-PAC ACISF adds just one additional minute to usual care, whereas the other two screens must be conducted apart from and in addition to usual care activities. To use the AM-PAC ACISF during routine evaluation and treatment, occupational therapy practitioners must be able to gather information through observation to accurately score the screen by proxy.
If a practitioner is unable to observe usual care interactions with the patient, the MoCA and BCAT–SF may be more appropriate. In addition, it may be valuable to have other health care providers, such as nurses, administer the AM-PAC ACISF as a screen for all patients, using the results to generate a consult for occupational therapy and speech-language pathology when the score identifies cognitive impairment.
Limitations
Our sample was taken from two acute care hospital units with primarily cardiac patients at a large academic medical center. It is unknown whether the proportion of patients with cognitive impairment on these units is similar to the proportion in other patient populations; any differences in proportion may limit the generalizability of our results. The patients in our sample had already had at least one therapy order; their level of cognitive impairment may have been different from that of patients without a therapy order. Additionally, participant-reported ratings of the screens’ usability might have been influenced by the therapist’s presence. However, we actively solicited critical feedback, and patients’ responses indicated that they were not inhibited or otherwise influenced by the therapist’s presence.
Implications for Occupational Therapy Practice
The results of this study have the following implications for occupational therapy practice: ▪ Occupational therapy practitioners have the training and skills needed to screen acute care patients for cognitive impairment and assess the impact on occupational performance across the continuum of care. Evaluation and treatment of cognitive impairments are a key component of the Occupational Therapy Practice Framework: Domain and Process (American Occupational Therapy Association, 2020). ▪ Multidisciplinary use of the AM-PAC ACISF can facilitate appropriate orders by occupational therapy practitioners and other professionals to address cognitive impairment in acute care patients. ▪ The AM-PAC ACISF can help occupational therapy practitioners identify general areas of cognitive impairment affecting functional performance and can guide the use of additional assessments. In addition, AM-PAC ACISF results can contribute to the creation of specific goals and client-centered, occupation-based interventions to address specific cognitive impairments. ▪ The AM-PAC was created with item response theory methods, and the items in the ACISF come from a larger item bank. AM-PAC ACISF scores thus can be compared across administrations throughout the continuum of care.
Conclusion
Cognitive changes are too often overlooked or misunderstood in fast-paced acute care hospital settings. Consistent use of the AM-PAC ACISF can help clinicians identify cognitive impairment and provide client-centered intervention. Screening for cognitive impairment early in the hospital stay is one step in preventing potentially long-standing functional impairments and providing a quicker return to participation in meaningful occupations.
Clinicians charged with screening patients for cognitive impairment must do so accurately and consistently, using tools that are administered and scored correctly. Proxy judgments of observed behaviors by examiners who are not trained entail the risk of error, omission, and misunderstanding. Practitioners must review the AM-PAC ACISF training manual in detail and abide by recommendations for activities and specific behaviors to observe to recognize any impairments. It is our hope that future implementation of cognitive screens such as the AM-PAC ACISF will help improve the identification of cognitive impairment in acute care hospital settings.