The reliability of the Montreal Cognitive Assessment using telehealth in a rural setting with veterans

Abstract

Background

Telehealth neuropsychological services can increase the availability of specialised care for individuals in rural areas where barriers to these services are faced. As this practice becomes more commonplace, the reliability and validity of neuropsychological assessment administered by telehealth continues to be established. The Montreal Cognitive Assessment, a screener for general neurocognitive dysfunction, may be particularly useful since this measure can be given by telehealth with minimal adaptation.

Methods

Veterans from a rural area of the country who were referred to an outpatient neuropsychology clinic were administered the Montreal Cognitive Assessment either in-person or by telehealth by a clinician. A second clinician observed the administration in-person or by telehealth and independently scored the each participant’s performance. The inter-rater reliabilities across conditions were compared to assess for differences between in-person and telehealth consultations.

Results

The inter-rater reliability of the Montreal Cognitive Assessment across the three conditions of interest was acceptably high and values ranged from r = 0.88 to r = 0.98. Reliability correlations were compared and no significant differences among the conditions were observed (p’s > 0.10). Beyond reliability, univariate comparison of the absolute mean differences of clinician scores showed no significant differences among the actual raw scores of the three conditions tested, indicating good accuracy (p = 0.56).

Conclusions

The inter-rater reliabilities of Montreal Cognitive Assessment scores across conditions were all acceptably high, and administration of the Montreal Cognitive Assessment using telehealth technology did not significantly alter the total scores. Overall, the lack of significant differences suggests that administering the Montreal Cognitive Assessment by telehealth is reliable, accurate and well received by participants.

Keywords

Telehealth telegeriatrics

Introduction

Elderly individuals living in rural areas face additional barriers to accessing essential medical care and social services, and this has become an increasing priority nationally.^1,2 The elderly population in the USA (individuals over the age of 65 years) is not uniformly distributed, with older adults being most concentrated in rural areas,³ and the age distribution of a location tends to become older as rurality increases.⁴ A large portion of individuals living in rural areas are faced with a range of disabilities that are particularly prevalent in, and specific to, geriatric populations. Even normative physical changes associated with aging, such as impaired hearing, loss of vision, decreased mobility and other self-care issues, can increase the difficulty of attaining appropriate physical and mental health care in a rural setting.

Minimising and eliminating obstacles obstructing access to care is particularly important for individuals in rural areas with possible neurodegenerative disorders who are part of the ever increasing geriatric population. Driving restrictions, reliance on family and friends for assistance, restricted income, unfamiliarity with newer healthcare models, physical weakness and other medical diagnoses all increase the difficulty of obtaining needed care. These difficulties are even more pronounced when trying to access specialised services such as neuropsychological testing services which tend to be located in larger populous centres. Less than 25% of patients with dementia or probable dementia are diagnosed by their primary care physicians,⁵ so improving access to specialised services may lead to earlier identification of dementia and improved clinical outcomes.

Access to quality care and specialised services is a particularly relevant issue for veterans.⁶ Twenty-four percent of living veterans are in rural areas, and 36% of the veterans living in rural areas utilise the Veterans Affairs (VA) healthcare system. Veterans living in rural areas are at a higher risk for facing additional barriers to accessing adequate health care since they are also more likely to have long-standing service connected disabilities.⁷ To help facilitate and address the specific needs of underserved veterans living in rural areas, the Office of Rural Health was established in 2007 to support a wide range of projects such as telemedicine, rural facilities and expansion of services into rural areas. The improvement of neuropsychological assessments administered by telehealth is critical since there is a large increase in neuropsychology requests now that Vietnam War veterans are experiencing dementing disorders along with the substantial increase in blast-related head injuries in younger veterans as a result of the Operation Enduring Freedom, Operation Iraqi Freedom, and Operation New Dawn conflicts.

The feasibility, acceptance and use of telehealth services continues to develop across a wide range of healthcare disciplines and specific disorders such as Parkinson’s disease.^8–10 The expansion of telehealth has not been limited to one aspect of care but includes all stages of care including interview, examination, intervention and rehabilitation. These efforts also span across multiple fields such as telepsychiatry, teleneurology and telemental health. Cullum et al.⁸ noted that many of these services rely inherently on verbal and visual communication and are particularly well suited for telehealth applications. Many forms of psychotherapy, diagnostic interviewing and case management appear to be appropriate for use over video teleconferencing equipment.^11–13 The specific use of video teleconferencing to complete remote neuropsychological assessment is becoming increasingly feasible with growing acceptance among providers and patients.^6,11,14,15 Patient acceptance and satisfaction of teleneuropsychology services are often as high as 98% with a large majority of participants reporting having no preference between in-person versus video assessment modalities.¹⁶

Establishing reliability and validity is a critical component for the appropriate use of teleneuropsychology as an empirically supported practice. Studies assessing the reliability and validity of neurocognitive screening measures have had positive results.^8,17–19 A systematic meta-analysis of remote neuropsychological testing looked at 12 studies that used common tests and concluded that videoconference administration did not clearly lead to inferior performance compared to on-site administration.²⁰ The only test where there was a consistent difference between remote and on-site observed scores was for the Boston Naming Test, and even this effect was small (one-tenth of a standard deviation (SD)). This systematic review highlights the relative lack of investigations into the use of video conferencing for neuropsychological assessment and provides a clear rational for the need of additional studies.

Neurocognitive screeners have received the most attention due to their brevity and significant impact on clinical decision making. The Mini-Mental State Exam (MMSE) has the most support with the first comparisons of face-to-face versus telehealth administration dating back to the late 1990s.^21,22 Evidence suggesting that the administration of MMSE using telehealth is reliable and valid across a diverse set of populations and settings continues to grow.^8,18,23 The reported intraclass correlations between face-to-face and telehealth administration have exceeded 0.85 across multiple studies in addition to high intra-rater agreement.^8,24 One of the well-documented weaknesses of the MMSE is it’s poor sensitivity for mild cognitive impairment.^24–27 To overcome this weakness in the MMSE several other neuropsychological measures have been evaluated for suitability for telehealth administration.^8,28

The Montreal Cognitive Assessment (MoCA)²⁹ is a widely used neurocognitive screener that has a higher sensitivity to mild cognitive impairment compared to the MMSE. It assesses several cognitive domains including Visuospatial/Executive, Naming, Immediate and Delayed Memory, Attention, Language, Abstraction, and Orientation. Despite the popularity of this neuropsychological screener in person, very few investigations have looked the reliability and validity characteristics when administered by telehealth. Wong et al.³⁰ used a modified MoCA profile that excluded all of the visual components so that it could be administered remotely by phone to patients post ischaemic stroke/transient ischaemic attack. Participants were given the full MoCA in person as part of an annual assessment and then given the abbreviated telephone-based MoCA four weeks later. Face-to-face and telephone scores were highly correlated (r = 0.87), and total scores were on average only 1.8 points higher on the abbreviated protocol compared to the full version administered in person. Lindauer et al.³¹ is the only known study that specifically gave the MoCA, along with other neurocognitive tests, via video telehealth. A test-retest design was used where participants were administered the MoCA at two time-points with a two-week interval in between in-person and telehealth administrations. Consistent with many telehealth studies, they made minimal modifications to accommodate the telehealth administration such as enlarged stimuli. Like the MMSE, the MoCA had very strong intraclass correlation coefficient (ICC) of 0.93 suggesting good reliability.

The limitation of the test-retest methodology is that it introduces practice effects and time influences into the method which can potentially influence overall results. This study used multiple raters simultaneously so that the participant would only be administered the MoCA at one time. The results of this study can be used in further demonstrating the reliability and accuracy of the MoCA. Further demonstrating the appropriateness of telehealth administration of the MoCA by telehealth will give mental health professionals additional options when choosing the best measure to give when completing telehealth assessments. Overall, this study represents another important step forward to support the practice of neuropsychological evaluation using telehealth technology.

Method

Participants

Seventeen participants (age, mean (SD) = 62.8 (14.50) years, 15 male/2 female) who were referred to an outpatient neuropsychology clinic for wide range of cognitive concerns were recruited for this project. All but one participant identified themselves as Caucasian. The mean length of education was 13.7 (1.8) years ranging from 11–18 years.

Procedure

Patients were approached after completing their scheduled appointment in a general neuropsychology clinic and asked if they were interested in participating in this study. Individuals who expressed interest were informed that the clinic was validating a new way to administer a neurocognitive screener to improve access to neuropsychological assessment in rural areas and that they would be asked to complete a short cognitive screener either in person or using telehealth. Participants had the opportunity to take a short break while materials were prepared to help minimise fatigue. Individuals who agreed to participate completed the informed consent process approved by the institutional review board (IRB) and were randomly assigned to one of three conditions (see Figure 1) and then administered the geriatric version of the MoCA with enlarged visual stimuli.

Condition 1: this condition represented the control group where the participant was administered the MoCA face-to-face by a clinician physically present in the room. A second rater, also physically present in the room, observed the administration and scored the participants’ performances independently.

Condition 2: the participant was administered the MoCA face-to-face by a physically present clinician; however, this time the second rater scored the participants’ responses independently from a different room using a video-based telehealth system.

Condition 3: the roles of the clinicians from Condition 2 were reversed so that the clinician physically present in the room assumed the role of the independent rater, and the clinician in the remote location was administering the MoCA. Thus, in Condition 3, the participant was administered the MoCA by telehealth while the second rater in the room independently scored the administration. The rater physically in the room with the participant helped facilitate the administration by placing the MoCA stimuli in front of the participant and then holding the participants written responses up to the camera. They were not allowed to converse or engage with the participant in any other way. All instructions and participant questions were conducted using the telehealth system. Raters were not allowed to discuss the participant responses or collaborate in any other way.

Figure 1.

Description of the three assigned conditions.

Videoconferencing equipment and setup

For this current project, the 384 kbit/s Internet connection available for use at our facility was used to connect two endpoints of two non-adjacent rooms within the facility. When the MoCA was administered by telehealth, the administrator sat at their work station 20 in away from a computer monitor mounted Cisco TelePresence Precision HD USB Web camera utilising Cisco Jabber Video for Telepresence Software. Participants in the non-adjacent room were seated approximately 72 in in front of a 32 in LCD television monitor equipped with a Cisco Telepresence System SX20 Quick Set with Precision HD 1080p that ran proprietary VA software.

Analysis

All analyses were completed using IBM SPSS Statistics version 21.0. The mean difference between the overall MoCA score across the three conditions was computed and the absolute agreement was calculated using ICCs.

Results

All 17 participants completed the full protocol and there were no missing data. There was a good range of ages present in the sample which generally reflected the population of patients being referred for outpatient neuropsychological evaluations at this site (Table 1). All but one participant had a high school education or higher. Consistent with the population seen in this clinic, the sample was predominantly male (88.2%) and White (94.1%). Just over one-third of the participants were diagnosed with major neurocognitive disorder (35.3%), just less than one-third with minor neurocognitive disorder (29.4%), and almost one-third with no cognitive impairment (29.4%). Participants were assigned to one of three possible conditions, and the groups did not significantly differ based on age, level of education, race, gender or diagnosis (all p’s > 0.05). Across the full sample, the mean (SD) MoCA score, based on the administrators' ratings, was 23.18 (3.80). MoCA scores did not differ significantly across the three conditions (F(14) = 0.46, p = 0.64).

Table 1.

Participant characteristics.

Demographics
Age (years)
Mean ± SD	62.8 ± 14.6
Median, range	66, 34–86
Male (%)	88.2% (n = 15)
Education (years)
Mean ± SD	13.7 ± 1.8
Median, range	14.0, 11–18
Ethnicity (% White)	94.1% (n = 16)
Diagnosis
Deferred (%)	5.9% (n = 1)
No cognitive impairment (%)	29.4% (n = 5)
Mild neurocognitive disorder (%)	29.4% (n = 5)
Major neurocognitive disorder (%)	35.3% (n = 6)

SD: standard deviation.

Each of the three conditions showed very high (>0.98) ICC scores which indicates high absolute agreement (Table 2). The inter-rater reliability of the MoCA across the three conditions of interest was acceptably high and ranged from r = 0.97 to r = 0.98. Reliability correlations were compared using Fisher r-to-z transformations, and no significant differences among the conditions were observed (p’s > 0.10). Univariate comparison of the absolute mean differences in clinician scores showed no significant differences among the actual raw scores of the three conditions tested indicating accuracy (F(14) = 0.01, p = 0.56) (Figure 2, Table 3).

Table 2.

Mean (SD) Montreal Cognitive Assessment (MoCA) score agreement by condition.

	Administrator	Observer	Pearson correlation	ICC
Condition 1 (n = 6)	24.33 (3.3)	24.50 (3.6)	0.980	0.990
Condition 2 (n = 6)	22.17 (4.3)	22.67 (4.7)	0.977	0.986
Condition 3 (n = 5)	23.00 (3.8)	23.60 (3.8)	0.977	0.984
All conditions (n = 17)	23.18 (3.8)	23.59 (3.9)	0.975	0.985

ICC: intraclass correlation coefficient.

Figure 2.

Scatterplot of MoCA scores separated by condition.

Table 3.

Montreal Cognitive Assessment (MoCA) score agreement by diagnosis.

	Administrator	Observer	Pearson correlation	ICC
No cognitive impairment (%)	26.40 (2.8)	26.60 (3.1)	0.990	0.994
Mild neurocognitive disorder (%)	22.40 (2.8)	23.00 (3.2)	0.933	0.996
Major neurocognitive disorder (%)	21.17 (4.1)	21.50 (4.4)	0.972	0.986

ICC: intraclass correlation coefficient.

As expected the mean MoCA scored decreased as the severity of neurocognitive diagnosis increased. The highest agreement was observed within individuals not diagnosed with cognitive impairment (r = 0.99), and individuals in the mild cognitive impairment range had the lowest inter-rater agreement (r = 0.93). However, there was no meaningful decline in agreement based on severity of the diagnosis and the ICCs remained very high across all groups.

Discussion

With 24% of all veterans (5.3 m) living in rural areas, it is essential for specialty medical services within the VA to continue developing and improving ways to best serve these individuals. The number of individuals living with neurocognitive disorders is expected to rise to unprecedented levels in the next decade. Telehealth applications for neuropsychology services within the VA network are already numerous with the potential to significantly improve access, cost of care, efficiency and diagnostic accuracy far into the future. Establishing which tests are reliable and valid to give over telehealth and establishing what conditions should be met to engage in the practice of teleneuropsychology is an important step in this process. Ongoing research on telehealth-based neuropsychological assessment helps ensure patient and provider safety alike.

The results of this study are consistent with the original hypothesis that administering a cognitive screening tool, the MoCA, using videoconferencing equipment, would not have a significant negative impact on the reliability or validity of the test. When two raters independently scored the performance of participants, there was very high agreement across all conditions. No meaningful decrease in the absolute agreement between raters was observed when the test was either observed remotely or administered by telehealth. No decease in the overall performance of the participants was seen when the MoCA was administered by telehealth which supports not only the reliability but also the validity of giving this neurocognitive screener by telehealth. These findings are consistent with the ever growing body of evidence showing high agreement between tests given face-to-face and remotely.^8,32 The agreement is marginally higher in the current study, but it is suspected that this reflects the minimal amount of facilitation that the raters in the room provided to each participant. Even this slight provision of assistance could minimise measurement error by helping the participants feel more comfortable, stay oriented to the task and ensure that written responses were clearly seen over the videoconference system. This study adds to the growing body of research suggesting that tele-neuropsychological assessment is an appropriate way to increase the availability of neuropsychological assessment services to individuals who are unable to otherwise access this type of specialised service.

This study indicates that teleneuropsychological assessment for individuals diagnosed with either major or mild neurocognitive impairments is feasible, which is consistent with the use of telemedicine in other clinical populations.^9,10 These participants were able to successfully complete the testing by telehealth and their scores appeared as reliable and accurate as when the test was given in-person. This may have significant impact going forward since individuals experiencing cognitive decline are those most likely to face the largest barriers to accessing speciality care services. While not formally measured, participants with major neurocognitive impairments reported that they felt comfortable interacting with an evaluator by telehealth and felt that the actual setup and execution of the testing was fluid and not aversive. This is important and beneficial, considering the role of remote testing as part of more encompassing treatment protocols for individuals experiencing cognitive decline. It is becoming increasingly clear that the use of the MoCA, or other brief cognitive screeners, can be incorporated into the larger effort to provide both initial evaluation and sustained clinical intervention for patients using video telehealth.³³ Several participants from rural areas commented on their desire to utilise telehealth services through their local community based outpatient clinics in the near future.

For this study, the alternate rater in Condition 3 was allowed to help manipulate the test stimuli to orient the participant to an area of the sheet and to help the administrator get a clear view of the participants written responses. This approach has been utilised to differing degrees in other studies that have included telehealth clinical technicians.^12,29 This may have helped improve agreement. This approach was chosen at the onset of the study since the VA often has telehealth technicians present with patients to help administer these types of procedures, overcome technical difficulties, and to improve the overall experience of the patient. In many ways, having the observer present and slightly active during the procedure is a more accurate representation of how this procedure could be used within the VA healthcare system and helps determine the positive effects of having a second professional present with the patient when remote testing is being completed.

Limitations to this study include the small sample size and cross-sectional design. Despite the small sample size and cross-sectional design, the conclusions are unlikely to significantly change even if more participants were added to the study. The highly select sample characteristics of the participants for this study inherently decreases the external validity of the findings. Additional large studies that address the influence of education, more severe cognitive impairments and older age would be particularly useful in the future.

Conclusions

The intraclass coefficients and inter-rater reliabilities of the neuropsychological screener scores across all conditions were all acceptably high, and administration of the MoCA using telehealth technology did not significantly alter the total scores. The reliability of the MoCA when administered and scored using videoconferencing equipment did not decrease in the subset of participants with the highest level of cognitive difficulties. Overall, the lack of significant differences suggests that administering the MoCA neuropsychological by telehealth is reliable, accurate and well received by participants. Additional research to further assess the performance of this measure over telehealth in different populations will continue to be useful.

Footnotes

Declaration of Conflicting Interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: N DeYoung was the Neuropsychology Fellow at the Salem Veteran Affairs Medical Center at the time of the study. BV Shenal is the director of the Center for Neurocognitive Services at the Salem Veteran Affairs Medical Center.

Funding

The author(s) received no financial support for the research, authorship and/or publication of this article.

References

Douthit

Kiv

Dwolatzky

et al.

Exposing some important barriers to health care access in the rural USA. Public Health 2015; 129: 611–620.

Weinhold

Gurtner

. Understanding shortages of sufficient health care in rural areas. Health Policy 2014; 118: 201–214.

Mather M, Jacobsen LA and Pollard KM. Aging in the United States. Population Bulletin, December 2015, http://www.prb.org/pdf16/aging-us-population-bulletin.pdf.

Meit M, Knudson A, Gilbert T, et al. The 2014 update of the rural-urban chartbook. Rural Health Reform Policy Research Center, October 2014, https://ruralhealth.und.edu/projects/health-reform-policy-research-center/pdf/2014-rural-urban-chartbook-update.pdf.

Lin

O’Connor

Rossom

et al.

Screening for cognitive impairment in older adults: A systematic review for the US Preventive Services Task Force. Ann Intern Med 2013; 159: 601–612.

Yoder

Turner

Assessment via telemental health technology. In: Bush

(ed). Psychological assessment of veterans, New York: Oxford, 2014, pp. 159–173.

VHA Office of Rural Health. Information about the VHA Office of Rural Health and Rural Veterans, www.ruralhealth.va.gov/docs/factsheets/ORH_General_FactSheet_2014.pdf (2014, accessed 22 February 2017).

Cullum

Hynan

Grosch

et al.

Teleneuropsychology: Evidence for video teleconference-based neuropsychological assessment. J Int Neuropsychol Soc 2014; 20: 1028–1033.

Shores

Ryan-Dykes

Williams

et al.

Identifying undiagnosed dementia in residential care veterans: Comparing telemedicine to in-person clinical examination. Int J Geriatr Psychiatry 2004; 19: 101–108.

10.

Dorsey

Deuel

Voss

et al.

Increasing access to specialty care: A pilot, randomized controlled trial of telemedicine for Parkinson’s Disease. J Clin Mov Disord 2010; 25: 1652–1659.

11.

Abolahli

Scoglio

Killoran

et al.

Potential reliability and validity of a modified version of the Unified Parkinson’s Disease Rating Scale that could be administered remotely. Parkinsonism Relat Disord 2013; 19: 218–221.

12.

Tyrrell

Couturier

Montani

et al.

Teleconsultation in psychology: The use of videolinks for interviewing and assessing elderly patients. Age Ageing 2001; 30: 191–195.

13.

Capner

. Videoconferencing in the provision of psychological services at a distance. J Telemed Telecare 2000; 6: 311–319.

14.

Harrell

Wilkins

Connor

et al.

Telemedicine and the evaluation of cognitive impairment: The additive value of neuropsychological assessment. J Am Med Dir Assocn 2014; 15: 600–606.

15.

Luxton

Pruitt

Osenbach

. Best practices for remote psychological assessment via telehealth technologies. Prof Psychol Res Pr 2014; 45: 27–35.

16.

Parikh

Grosch

Graham

et al.

Consumer acceptability of brief videoconference-based neuropsychological assessment in older individuals with and without cognitive impairment. Clin Neuropsychol 2013; 27: 808–817.

17.

Padala

Nabholz

Padala

et al.

Televideo Clock Drawing Test to reach rural veterans: Feasibility in community-based outpatient clinics in the Department of Veterans Affairs. J Am Geriatr Soc 2015; 63: 205–206.

18.

Parsons TD. Telemedicine, mobile, and internet-based neurocognitive assessment. In: Clinical neuropsychology and technology. Switzerland: Springer International Publishing, 2016, pp. 99–111.

19.

Grosch

Gottlieb

Cullum

. Initial practice recommendations for teleneuropsychology. Clin Neuropsychol 2011; 25: 1119–1133.

20.

Brearly TW, Shura RD, Martindale SL, et al. Neuropsychological test administration by videoconference: A systematic review and meta-analysis. Neuropsychol Rev. Epub ahead of print 16 June 2017. DOI: 10.1007/s11065-017-9349-1.

21.

Montani

Billaud

Tyrrell

Fluchaire

et al.

Psychological impact of a remote psychometric consultation with hospitalized elderly people. J Telemed Telecare 1997; 3: 140–145.

22.

Ball

Puffett

. The assessment of cognitive function in the elderly using videoconferencing. J Telemed Telecare 1998; 4: 36–88.

23.

Grosch

Weiner

Hynan

et al.

Video teleconference-based neurocognitive screening in geropsychiatry. Psychiatry Res 2015; 225: 734–735.

24.

Loh

Donaldson

Flicker

et al.

Development of a telemedicine protocol for the diagnosis of Alzheimer's disease. J Telemed Telecare 2007; 13: 90–94.

25.

Ihl

Frölich

Dierks

et al.

Differential validity of psychometric tests in dementia of the Alzheimer type. Psychiatry Res 1992; 44: 93–106.

26.

Tombaugh

McIntyre

. The Mini-Mental State Examination: A comprehensive review. J Am Geriatr Soc 1992; 40: 922–935.

27.

Wind

Schelevis

Van Staveren

et al.

Limitations of the Mini-Mental State Examination in diagnosing dementia in general practice. Int J Geriatr Psychiatry 1997; 12: 101–108.

28.

Galusha-Glasscock

Horton

Weiner

et al.

Video teleconference administration of the Repeatable Battery for the Assessment of Neuropsychological Status. Arch Clin Neuropsychol 2016; 31: 8–11.

29.

Nasreddine

Philips

Bedirian

et al.

The Montreal Cognitive Assessment, MoCA: A brief screening tool for mild cognitive impairment. J Am Geriatr Soc 2005; 52: 695–699.

30.

Wong

Nyenhuis

Black

et al.

Montreal Cognitive Assessment 5-minute protocol is a brief, valid, reliable, and feasible cognitive screen for telephone administration. Stroke 2015; 46: 1059–1064.

31.

Lindauer A, Seelye A, Lyons B, et al. Dementia care comes home: Patient and caregiver assessment via telemedicine. Gerontologist 2017; 57: e85–e93.

32.

Castanho

Amorim

Moreria

et al.

Assessing cognitive function in older adults using a videoconference approach. EBioMedicine 2016; 11: 278–284.

33.

Dang

Gomez-Orozco

van Zuilen

et al.

Providing dementia consultations to veterans using clinical video telehealth: Results from a clinical demonstration project. Telemed J E Health 2018; 24: 1–7.