Abstract
Performance of complex everyday activities is crucial to independent community living. These activities are often referred to as instrumental activities of daily living (IADLs), and they include shopping, meal preparation, financial management, medication management, time management, and community mobility skills. Cognitive skills are essential for successful completion of IADLs because such tasks require executive functions such as complex attention, working memory, and problem solving. Most IADLs require clients to draw on accumulated knowledge and performance skills to perform a task in a complex environment. Wesson and colleagues (2016) referred to this integration of cognitive and physical performance skills needed for independence in IADLs as functional cognition. As implied in this definition, assessment of functional cognition is focused on the performance level instead of the impairment level, where cognitive assessments (e.g., neurocognitive assessments) are typically directed. Therefore, occupational therapists who assess functional cognition and other health care professionals should use a performance-based test (PBT).
Occupational therapists have used PBTs for many years to assess the ability of adults across diagnostic groups to live independently in the community (Moore et al., 2007; Poulin et al., 2013; Wesson et al., 2016). PBTs have greater ecological validity (representative of everyday life) and are thought to more accurately assess the cognitive and performance skills needed to successfully function in real-world activities compared with self- or informant-report measures (Morrison et al., 2015; Schmitter-Edgecombe et al., 2011). PBTs of functional cognition are designed to assess the client at his or her performance level across several functional domains, including cognition, in real-world contexts. In general, these tasks have a greater cognitive than physical emphasis and require the integration and sequencing of multiple action steps for goal completion. Many functional cognitive performance–based tests exist, including, but not limited to, the Executive Function Performance Test (EFPT; Baum et al., 2003), the Assessment of Motor and Process Skills (AMPS; Fisher & Bray Jones, 2010), the Performance Assessment of Self-care Skills (PASS; Rogers & Holm, 1989), and the Complex Task Performance Assessment (Wolf et al., 2008, 2015). However, these instruments were developed to be administered primarily in outpatient settings by trained occupational therapists, leaving few existing options to assess functional cognition in other contexts; this has created a concern for the health care system that the Centers for Medicare & Medicaid Services (CMS) is currently addressing (Skidmore, 2017).
In response to the requirements of the Improving Medicare Post-Acute Care Transformation (IMPACT) Act of 2014 (Pub. L. 113-185), CMS has been reviewing the assessments used to evaluate Medicare and Medicaid beneficiaries. CMS has contracted with the RAND Corporation to identify and test cognitive measures for inclusion in the standardized assessment battery required by the IMPACT Act. Two measures, the Confusion Assessment Method (CAM; Inouye et al., 1990) and the Brief Interview for Mental Status (BIMS; Saliba et al., 2012), which are currently used in the Minimum Data Set and will be included in the new CMS assessment protocol. The CAM assesses delirium, and the BIMS assesses mental status.
The American Occupational Therapy Association’s (AOTA’s) comments on the proposed assessment battery prompted the RAND expert panel to explore the concept of functional cognition for inclusion in patient assessment in postacute care (PAC) settings. AOTA advocates for the screening of clients in PAC settings for deficits in functional cognition that may affect essential IADLs. Given that unrecognized impairments in functional cognition may be associated with impairments in IADL performance, identifying clients in need of assessment and intervention may reduce hospital readmission (Anderson & Birge 2016; Deschodt et al., 2015), emergency department utilization, and caregiver burden. The challenge lies with determining the best way to screen for functional cognitive deficits in PAC settings.
In both acute hospital and long-term care environments, clients may not perform complex IADLs or sustain complex behaviors, making the observation of these spontaneously occurring behaviors impossible. Informant reports may introduce bias and underestimate the need for assistance (Schmitter-Edgecombe et al., 2011). Self-report measures have been shown to underestimate difficulties, especially in clients with new-onset conditions or in those lacking self-awareness (Morrison et al., 2015). Even for clients referred to occupational therapy, observation of naturalistic or contrived complex functional performance (i.e., IADLs) may be time consuming and unlikely given shortened lengths of stay.
Although evidence indicates that measures of functional cognition more accurately predict capacity in real-world functional performance than traditional neuropsychological measures (Jekel et al., 2015; Wesson et al., 2016), we were unable to locate a reliable and valid brief screening measure that can be used in various health care disciplines to screen functional cognition across health care treatment settings. The purpose of this study was to develop and examine the preliminary reliability, validity, and clinical utility of the Menu Task (MT) as a functional cognition screening measure.
Method
Research Design
This cross-sectional study included three phases of MT development and initial evaluation. The institutional review boards of the University of Wisconsin–Madison and the University of Missouri–Columbia approved the study. All participants provided written informed consent.
Participants and Recruitment
Two convenience samples were recruited for this study. A sample of healthy community-dwelling adults (community sample) was recruited in Madison, Wisconsin (n = 130), using study flyers posted on university, library, and grocery store bulletin boards and distributed at senior centers and congregate housing complexes. In addition, we used in-person recruitment, which included researchers attending community events and word-of-mouth referrals from study participants.
A second sample of participants, hospitalized for elective orthopedic surgery (n = 60), was recruited from the University of Missouri Orthopedic Institute in Columbia, Missouri (hospital sample). Potential participants scheduled for elective orthopedic surgery were contacted by phone before the date of their surgery and screened for eligibility. Those interested were approached on the day of surgery to provide informed consent before any sedating medication had been administered. Consenting participants were recontacted after surgery during their hospital stay to complete the study procedures. Inclusion criteria for both samples were age 55 or older, living in the community, and willingness to be tested in English; for the hospital sample, participants had to be referred for elective orthopedic surgery requiring a hospital stay and be independent in activities of daily living (ADLs) before they were admitted for surgery (Barthel Index [BI] scores >95; Mahoney & Barthel, 1965).
Menu Task Development
The MT is intended to be a brief, reliable, and valid screening measure of functional cognition that can be easily administered by a member of any health care discipline across both acute and postacute treatment settings. The assessment is modeled on the Multiple Errands Test–Revised (MET–R; Morrison et al., 2013) in that the participant is asked to perform a number of selection tasks while adhering to a set of rules. To meet the criteria for inclusion in the CMS assessment battery, the MT needed to have the following characteristics:
Simple to learn
Quick to administer
Simple to score
Sensitive to mild and moderate levels of impaired functional cognition that may result in increased resource utilization and failure in community settings
Able to be administered without equipment.
We used Anastasi and Urbina’s (1997) definition of content validity as we developed and refined the MT. Its content validity is based on “the systematic examination of the test content to determine whether it covers a representative sample of the behavior domain to be measured” (Anastasi & Urbina, 1997, p. 225). In this case, we used the principles of PBTs and a review of the items and instructions used in the MET–R and the EFPT to identify the essential elements of the MT as a screening measure of functional cognition. The first phase of this process focused on the comparison of MT scores to neuropsychological tests and self-report measures of ADL and IADL function.
Measures
Two considerations informed the selection of study measures used to determine the validity of the MT. First, we included two scales proposed for use in the new CMS assessment battery: the BIMS and the Montreal Cognitive Assessment (MoCA; Nasreddine et al., 2005). Second, we selected scales to assess aspects of executive function (Trail Making Tests A and B [Trails A and B]; Reitan, 1992) ADL and IADL measures (BI and Nottingham Extended Activities of Daily Living Scale [NEADLS]; Nouri & Lincoln, 1987). These measures are described in Table 1.
Study Measures
Note. ADLs = activities of daily living; Trails = Trail Making Test.
Administration and Scoring of the Menu Task
We administered the MT assessment protocol in hospital, university, and community settings. Occupational therapy graduate students and other research staff from the University of Wisconsin and the University of Missouri were trained to administer the test battery and performed all testing. Testing was conducted in quiet settings, with participants seated at a table or bedside for the Missouri Orthopedic Institute participants. Participants completed a brief questionnaire documenting demographic information and a brief health history. The test battery was administered in the following order: MT, MoCA, NEADLS, BIMS, and Trails A and B. Videos presenting test administration and scoring are available on YouTube (see Marks, 2016a, 2016b).
Statistical Analyses
Data from each sample were analyzed separately first and then combined for additional analyses. We computed descriptive analyses for continuous data and frequency distributions for noncontinuous data. The internal consistency of the MT was evaluated using Cronbach’s (1951) α statistic. We also calculated simple bivariate Pearson product–moment correlations.
Published criterion scores for the BIMS, MoCA, and NEADLS were used to create “impaired” and “not-impaired” groups for each measure. A receiver operating characteristic (ROC) curve analysis was used to explore discriminant validity and empirically establish the cutoff (criterion) score for the MT using the MoCA as the criterion measure. The area under the curve (AUC) statistic was used as the statistical index in combination with the specificity and sensitivity coefficients to determine the optimal score. Student’s t tests were used to compare MT scores between impaired and not-impaired groups for each scale. IBM SPSS Statistics (Version 23; IBM Corp., Armonk, NY) was used for all statistical analyses.
Results
The MT was administered to 190 participants age 55 or older across two settings. Sixty participants (mean [M] age = 65.21 yr, standard deviation [SD] = 6.79) were tested at the University of Missouri Orthopedic Institute after elective orthopedic surgery, and 130 community residents (M age = 73.56, SD = 11.21) were recruited and tested at the University of Wisconsin and in community settings in Madison. In Table 2 we present participants’ demographic characteristics and scores for each of the study measures by sample. The two samples differed significantly on all demographic variables. The hospital sample was significantly younger and less educated, reported fewer chronic health conditions, and had significantly higher (better) scores on the MoCA than the community sample.
Demographic Characteristics and Scores on Study Measures, by Group
Note. — = not tested; BIMS = Brief Interview of Mental Status; M = mean; MoCA = Montreal Cognitive Assessment; NEADLS = Nottingham Extended Activities of Daily Living Scale; SD = standard deviation; Trails = Trail Making Test.
The published criterion scores for each neurocognitive test and measure were used to separate participants into the impaired and not-impaired groups. The results are presented in Table 2. The community sample had higher levels of impairment on each assessment than the younger, healthier hospital sample. In each group, fewer participants indicated impairment on the BIMS (11% of the community sample and 2% of the hospital sample) than on Trails A and B or the MoCA, suggesting that screening with the BIMS alone would underestimate the number of participants with cognitive impairments, thus affecting occupational therapists’ ability to help them manage the IADL demands of independent community living.
Reliability
In the first phase of the study, before we assessed the participants, we established interrater reliability on the MT by testing 10 participants recruited from the community. We compared scores of two independent raters for all MT items. The scores for each interviewer were identical, resulting in intraclass correlations of 1 for all test items. We used Cronbach’s α based on the combined data from both samples to estimate the internal consistency of the MT. The overall α coefficient was .70, indicating adequate internal consistency.
Feasibility
We evaluated the feasibility of implementing the MT in health care settings by examining the ease of administration in each participant’s hospital room and by recording the time needed to administer and score the task. The average time for the MT administration and scoring was less than 4 min (see Table 2). All hospitalized participants completed the testing, and no problems in administration were identified by the testers.
Validity
Consistent with the methods of Campbell and Fiske (1959), we hypothesized that the MT total score would more highly correlate with the MoCA and Trails B (because these tests assess aspects of executive function) than with the BIMS and Trails A (because these tests assess memory, orientation, and visual–motor speed, but not executive function). The results provide preliminary support for the construct validity of the MT. MT total scores were more highly correlated with the MoCA (r = .56, p < .001) and Trails B (r = −.54, p < .001) scores than with the BIMS (r = .34, p < .01) or Trails A (r = −.35, p < .01) scores. As expected, age was negatively correlated with performance on the MT (r = −.48, p < .01): Older participants tended to make more errors than did younger participants. Education was not associated with performance on the MT (r = .05, not significant), suggesting that the MT is an appropriate screening measure for clients across a range of educational achievement.
To more fully examine the discriminant validity of the MT, we computed an ROC analysis to define a criterion, or cutoff score, using the MoCA as the dependent measure (Zweig & Campbell, 1993). In this analysis we also wanted to reflect the likely clinical use of the MT, in that CMS uses a hierarchical approach to assessment. People with delirium who perform poorly on the CAM are not tested on the other measures. Similarly, clients whose BIMS scores indicate impairment would not proceed to further testing with the MT. Thus, the 12 participants in our samples whose BIMS indicated impairment were excluded from the ROC analysis, yielding a sample size of 168 for this analysis. In this sample, the AUC was .78, and the analysis indicated that the optimal criterion score was between 5.5 and 6.5. Using an MT score of 6, a sensitivity of .80 and specificity of .65 were obtained for classifying participants as impaired on the basis of their MoCA score. On the basis of a total score of 6 or less, 25% of the community sample and 23% of the hospital sample were classified as impaired on the MT.
We determined the discriminant validity of the MT by comparing the scores on each study measure between groups classified as impaired and not impaired on the MT. Independent-sample t tests computed separately for each sample (community and hospital) compared age, education, BIMS, MoCA, Trails A and B, NEADLS, and BI scores by group (Table 3).
Independent-Sample t Tests Comparing Participants Classified as Impaired and Not Impaired on the Menu Task, by Sample
Note. — = not tested; BIMS = Brief Interview of Mental Status; M = mean; MoCA = Montreal Cognitive Assessment; NEADLS = Nottingham Extended Activities of Daily Living Scale; SD = standard deviation; Trails = Trail Making Test.
The data for each sample were analyzed separately. Within each sample (community and hospital), we created two groups—impaired or not impaired on the MT—and then determined whether there were significant differences between the impaired and not-impaired groups on demographic, neuropsychological, and functional variables. In the community sample, participants deemed impaired according to the MT were significantly older (p < .0000) and less independent in ADLs and IADLs as measured by the NEADLS (p < .001) than those who were not impaired according to the MT. The community sample impaired group also performed significantly more poorly on the BIMS, MoCA, and Trails A and B.
In the hospital sample we found fewer significant differences between the impaired and not-impaired groups on the MT. Hospitalized participants deemed impaired on the basis of their MT scores were significantly younger and had significantly poorer scores on the BIMS and the MoCA. There were no significant differences between the MT groups on Trails A and B and the BI in the hospital sample. These findings support the discriminant validity of the MT. More robust between-group differences were observed in the community sample, the members of which were older and less healthy than the hospital sample.
Discussion
According to Hennekens and Buring (1987), an ideal screening test is inexpensive, is easy to administer, and imposes minimal burden on the clients to be screened. In addition, the results of the screening test should be valid, reliable, and reproducible. We sought to create and test a brief screening tool designed to identify people who are unimpaired on measures of delirium or dementia but possibly in need of further occupational therapy evaluation and treatment. Our findings support the reliability and construct and discriminant validity of the MT, which was created to address gaps identified by AOTA and a group of occupational therapy experts in the assessment battery under consideration by CMS. Our goal was to create and validate a brief performance-based screening measure to assess aspects of executive functioning needed for independence in community-based IADLs. Although the actual task in the MT appears to be very simple, the novel set of rules presented in the instructions require the person to adjust familiar behaviors and suppress habitual responses. These task demands provide an opportunity to observe the functional cognition skills needed for independence in IADLs (Wesson et al., 2016) and to identify clients in need of more comprehensive evaluation and treatment.
Our findings also support the feasibility and clinical utility of the MT, which is easy to administer and score in both hospital and community settings. On average, participants in both settings were able to complete the task in less than 4 min. The scoring is simple and unambiguous, and we achieved 100% interrater reliability in both settings. We also observed good internal consistency in both groups of participants.
We found that MT total scores were more highly correlated with scores on the MoCA and Trails B, which measure executive function, than with scores on the BIMS and Trails A, which assess orientation, attention, and short-term memory, thus providing evidence for construct validity. The ROC analysis defined a cutoff score of 6 or less on the MT as the criterion for impaired function. Using this score, we also computed sensitivity and specificity. We observed a sensitivity of .80 and specificity of .65, which we believe to be satisfactory given the intended use of this test.
In general, screening tests prioritize sensitivity over specificity, thus decreasing the likelihood of false negatives (Hennekens & Buring, 1987). Sensitivity is preferred over specificity in cases where the consequences of a false positive screen are determined to be less problematic or hazardous than a false negative screen. In other words, the consequences of not identifying a person with functional deficits in need of further occupational therapy evaluation (false negative) are greater than identifying a person who is functionally independent but flagged as impaired on the MT (false positive) and subsequently found to be independent after a more comprehensive occupational therapy evaluation.
The AUC statistic, computed as part of ROC analysis, provides an additional indicator of validity because it measures the performance of a test over its whole range; the larger the area, the better the performance (Zweig & Campbell, 1993). The AUC of .78 in this analysis suggests that the MT performed well compared with the MoCA. Further evidence of the validity of the MT is supported by the significant between-group differences for participants identified as impaired or not impaired on the MT. In the community sample, individuals classified as impaired on the basis of their MT score performed significantly less well on all other study neurocognitive and functional measures. This was not the case for the hospital sample. In this group, only the MoCA scores were significantly different. These results are consistent with the differences between the two samples. The hospital sample was younger, was more highly educated, and reported fewer chronic health conditions.
Limitations and Future Research
This study has a number of important limitations. The sample size was relatively small and was limited in racial and ethnic diversity because we recruited convenience samples in relatively small, and predominantly White, Midwestern cities. Participants in both the community and hospital samples were not preselected for the presence of cognitive impairment. Participants who had been hospitalized for elective orthopedic surgery are not representative of all older people and Medicare beneficiaries with disabilities. The hospital sample was recruited to provide preliminary data needed to determine the feasibility of administering the MT in an inpatient setting. As a result, the number of participants with cognitive impairments (and percentage of those with cognitive impairments in the sample) was relatively low, and considerably lower than the proportion of clients expected to have cognitive deficits in the population of interest to CMS (i.e., clients who may be admitted to skilled nursing facilities, long-term care hospitals, or inpatient rehabilitation facilities or treated by home health agencies). We used the MoCA as a gold standard and the NEADLS to measure ADL and IADL ability in our community sample. Future researchers should evaluate the MT against other performance-based screening measures and other PBTs of known validity, as well as against client-observed IADL performance.
Implications for Occupational Therapy Practice
The relationships we found between the MT and the NEADLS, a commonly used measure of neurocognitive function, suggest that the MT has the capacity to address the gap in the proposed CMS assessment battery for Medicare recipients across postacute care settings. AOTA and the occupational therapy experts advising CMS have stressed the importance of a brief performance-based screening tool to identify people who need more comprehensive occupational therapy evaluation. The implementation of a functional cognition screening tool as part of the required CMS assessment protocol should greatly increase the number of patients referred for occupational therapy.
The difficulty in assessing cognition-related IADL skills in acute and postacute settings has been well documented (Bottari & Dawson, 2011; Edwards et al., 2006). In addition, self-report and proxy measures have been shown to underestimate impairment, in particular in people with limited insight or with new-onset conditions (Morrison et al., 2015). Fortunately, a variety of PBTs, such as the PASS, AMPS, and EFPT, are available to further evaluate patients who perform poorly on a screening measure such as the MT (Poulin et al., 2013). In addition to improving participation and quality of life, the identification and treatment of functional cognitive deficits may help reduce hospital readmission (Anderson & Birge, 2016; Deschodt et al., 2015), excess emergency department utilization, and undue caregiver burden.