Diagnostic Performance of an Eye-Tracking Assisted Visual Inference Language Test in the Assessment of Cognitive Decline due to Alzheimer’s Disease

Population and sample

Participants were recruited from the outpatient service of a memory clinic in a tertiary hospital in Sao Paulo, Brazil. The convenience sample comprised older adults diagnosed with normal cognition, MCI, and mild dementia due to AD. We initially screened 350 older adults with cognitive complaints from June 2019 until May 2020. After initial triage, 268 clinical and cognitive assessments were performed. Then, after exclusion criteria were applied, the final sample comprised 95 older adults with varying degrees of cognitive impairment. These were 18 with mild dementia due to AD (AD group), 49 with mild cognitive impairment (MCI group), and 28 cognitively unimpaired (control group). The study was approved by the local Ethics Committee, and all participants signed informed consent.

Inclusion and exclusion criteria

Participants with or without cognitive complaints who were sixty years or older, with normal or corrected-to-normal vision, were included. Laboratory tests (blood count, biochemistry, serum levels of glucose, thyroid hormones, folic acid, vitamin B12, lipid profile, and immune tests for syphilis) and neuroimaging scans such as computerized tomography (CT) or magnetic resonance imaging (MRI) were obtained for all participants for the diagnostic procedure and to exclude medical conditions that could be responsible for the cognitive symptoms. For the group with AD dementia, we included only patients with mild dementia, established according to the Clinical Dementia Rating scale (CDR) (CDR = 1) [13]. Patients with moderate or severe dementia due to AD (CDR = 2 or CDR = 3), non-AD dementia, visual or oculomotor disabilities, monocular vision, illiterates, and those with other neurological or psychiatric comorbidities were excluded from the study.

Cognitive screening, neuropsychological, language, and functional assessment

A physician (either a neurologist or a geriatric psychiatrist) performed a cognitive screening protocol at baseline, which included the Brazilian versions of the Montreal Cognitive Assessment (MoCA) [14], the Geriatric Depression Scale (GDS) [15], and the Scale of Instrumental Activities of Daily Living (IADL) [16]. A comprehensive neuropsychological examination was then performed for all participants which included the Rivermead Behavioral Memory Test [17, 18]; the Trail Making Test [19, 20]; the Digit Span test [21, 22]; the FAS-COWA test [19]; the Rey Auditory Verbal Learning Test (RAVLT) [23, 24], the Stroop test [19, 20], and the Rey-Osterrieth Complex Figure test [25, 26]. Additionally, the subjects underwent a language assessment that covered all linguistic domains (phonology, semantics, syntax, and pragmatics): An Object and Action Naming Battery (OANB) [27]; Test for Reception of Grammar –2 (TROG-2) [28], The Token Test [29]; The Northwestern Anagram Test (NAT) [30]; La Gestion De L'Implicite (The Implicit Management Test – IMT) [31]; The Cookie Theft Picture [32]; and the Semantics Association Test [33].

Diagnostic assessment

Participants were evaluated following the assessment protocol consisting of cognitive screening, clinical, neuropsychological, and evaluation by a language pathologist. A multidisciplinary consensus was held to establish diagnoses considering all available clinical information. The team included psychiatrists and neurologists, neuropsychologists, and language pathologists. The diagnosis of MCI was defined using the Mayo Clinic guidelines [34], and the diagnosis of probable AD was made according to the National Institute on Aging – Alzheimer’s Association (NIA-AA) [35].

Apparatus and calibration

We used a Tobii TX300 eye tracker to record eye movements. The apparatus for this study was used in other projects by our group [4, 5], where additional information can be found. In brief, the system sampling capacity was 300 Hz, with a processing latency of 1–3.3 ms. The participants were seated 65 cm from the 23” screen presenting the stimuli (total visual angle of 10.1 deg.). Head movements were not restrained: participants’ head movement could vary freely by±37 cm along the horizontal, and by±17 cm vertically. We conducted an eye movement register and the following data filtering using the Tobii Studio Software. Previously existing images were presented in black and white on a white background and were formatted to fit the computer screen. To guarantee accuracy in registering participants’ visual search, we have performed a calibration of the eye-tracking system before the task presentation. A nine-point (3x3 greed) calibration procedure was accomplished by having the participant fixate nine points at specific locations on the screen by following a moving marker (red circle) that appeared across the nine points. The calibration occurred until the equipment accurately mapped the participant’s gaze on the screen. We excluded participants to whom the equipment could not be calibrated (under 85% of corresponding visual search map). This same procedure was done for previous studies from our group [4, 5].

Testing and scoring procedures

The ability to perform visual inferences was assessed using a computational version of the instrument “300 exercices de compréhension d’inférences logique et pragmatique et de chaînes causales” (“300 Exercises of Comprehension of Logical and Pragmatic Inferences and Causal Chains”) [36]. For this study, only the pictorial stimuli were employed, consisting of 13 black-and-white pictures with tracing and textures that were homogeneous. Two pictures were used only as an example, and the other 11 were used as the experimental tasks. All stages of the test required inferential abilities at different levels of complexity.

Task description

The participant should observe pictures (a total of 13 pictures) presented separately on the first screen (challenge stimulus), e.g., a scene in which a boy crying with a toothache approaches his mother (Fig. 1, top, screen A). At this moment, all eye-tracking parameters were recorded. The participant should then decide which would be the best outcome for the previous scene, as portrayed in pictures 1, 2, or 3 (Fig. 1, bottom, screen B). All eye-tracking parameters were again recorded. To minimize memory bias, the challenge stimulus from screen A is presented again on screen B. In this example, the correct answer corresponds to the scene that shows the mother taking the boy to the dentist (Fig. 1, bottom, screen B, picture 2). The total duration of the test was recorded for each participant. All pictures were presented consecutively, during one session.

OPEN IN VIEWER

Fig. 1

Example of an inference process test. The participant should observe a scene: a boy crying with a toothache approaches his mother (screen A, top); the participant should then decide which was the best outcome for the story (screen B, bottom); the correct answer is one of the three pictures presented (the scene showing the mother taking the boy to the dentist) (screen B, picture 2).

The task required two types of inferential processing: a) logical inferences, i.e., to find the logical cause or consequence of a given situation, and b) pragmatic inferences, i.e., to determine which situation, among the alternatives, is most likely to occur. The stimuli allowed for only one correct answer and no time limit was imposed for performing the task. Scoring was done as follows: 0 – failed inference realization comprehension; and 1 – correct inference realization. The maximum score was 6 for logical inferences, 5 for pragmatic inferences, and 11 for total (logical and pragmatic) inferences. Concurrently with the assessment of inferences based on pictorial stimuli, eye movement behavior was collected.

Subsequently, for data analysis, areas of interest were manually defined as the spatial delimitation in which fixations and ocular visits occurred on each screen. These areas corresponded to the perception of the image as a whole and the subcomponents considered crucial for decision-making and inference realization. Thus, the following areas were defined (Fig. 2):

OPEN IN VIEWER

Fig. 2

Interest areas were manually defined as the spatial delimitation in which fixations and ocular visits occurred. These areas corresponded to the perception of the image as a whole and the subcomponents considered crucial for decision-making and understanding of inferences.

In screen A: challenge stimulus A – the whole image presented in screen A; target A – part of the image presented in screen A defined as crucial for inference comprehension.

In screen B: challenge stimulus B – the same image presented in screen A, but in the answer’s screen; correct target B – part of the image from the correct answer presented in screen B defined as crucial for inference comprehension; correct response stimulus B – the whole image of the correct answer presented in screen B; distractor stimulus B – the whole incorrect pictures presented in screen B.

The following eye-tracking parameters were captured: time to first fixation, number of fixations, number of visits, and total fixation duration. Together these variables comprise the visual search map. These parameters are defined elsewhere [5].

Statistical analyses

Numerical variables were described as mean and standard deviation, whereas categorical ones as absolute and relative frequencies. Importantly, in the present study, the same subjects performed the same task (inferential task) under multiple experimental conditions (different pictures) which generated correlated responses that needed to be taken into account. Thus, we performed a non-parametric repeated measures ANOVA model to verify the verbal answers of each group for each inference type, i.e., logic inference or pragmatic inference [37]. Additionally, eye tracking variables were assessed with linear mixed models (LMM) or their generalized version with gamma and inverse normal families. Each parameter was assessed with a distinct model. The outcome of each model was the parameter of interest (e.g., number of fixations) and the predictor was the group. Pairwise comparisons of groups on variables with strong evidence of difference were computed with estimated marginal means, and Tukey’s method was used to deal with the multiple inference problem. We performed the pairwise comparison of group means to estimate the means of each group (control, MCI, and AD) for a given parameter within its model, followed by a comparison of those means. Eye-tracking variables were then used as predictors of correct verbal answers in a generalized linear mixed model for the binomial family. Noteworthy, non-parametric repeated measures ANOVA and LMM tests deal with the problem of correlated observations in which longitudinal data fall into. However, this was a cross-sectional study with repeated measures over multiple experimental conditions. Models were set as group comparisons including a random intercept for the subject that accounts for correlated observations within each participant; this is equivalent to setting a compound symmetry correlation structure for each subject. The different experimental conditions are not of interest, hence they are being averaged out and the groups compared over all the conditions for each parameter of interest.

Finally, we applied Spearman’s rank correlation test to verify the presence of correlations between the visual search map parameters and the neuropsychological assessment parameters.

Residual and random effects normality were assessed with QQ plots. A plot of fitted values against the squared root of absolute values of standardized residuals was constructed to evaluate homoscedasticity. When any of those assumptions were not met, alternatives in the generalized linear mixed model were tried. Only when those failed the non parametric approach was used. All analyses were conducted on statistical package R 4.2.1 (http://www.r-project.org/).

Overview

A total of 95 individuals were included in the study and distributed in three diagnostic groups, i.e., cognitively healthy subjects or controls (n = 28), MCI (n = 49), and mild dementia due to AD (n = 18). All participants completed the test.

Clinical and sociodemographic data

Table 1 summarizes the clinical and sociodemographic data of the sample. There were no statistically significant differences in mean age across diagnostic groups (p = 0.094), nor in the number of males and females. Individuals with AD dementia had fewer years of education than controls and MCI subjects (p < 0.001). As expected, AD subjects had lower scores on the cognitive screening test (MoCA), and MCI subjects had intermediate results. As for the GDS, controls scored lower than subjects with MCI and AD. Subjects with AD also had higher scores in functional assessment (Lawton & Brody), indicating, as expected, a worse performance as compared to MCI and controlgroups.

Verbal answers for inference type between diagnostic groups

First, we investigated if there was a difference between logic inference and pragmatic inference across diagnostic groups. No evidence of interaction or main effect of inference were found, only main effect of group (Inference, statistic = 1.64, df = 1, p = 0.200; Group:inference, statistic = 2.37, df = 1, p = 0.094; Group, statistic = 26.04, df = 1, p < 0.001; non-parametric repeated measures ANOVA) (Fig. 3). Differences in verbal answers were statistically significant across all diagnostic groups (Control-MCI, logic: mean = 0.952, 95% CI = 0.887–0.983 versus mean = 0.918, 95% CI = 0.868–0.950; pragmatic: mean = 0.936, 95% CI = 0.859–0.974 versus mean = 0.872, 95% CI = 0.821–0.908, p = 0.008; Control-AD, logic: mean = 0.952, 95% CI = 0.887–0.983 versus mean = 0.694, 95% CI = 0.591–0.773; pragmatic: mean = 0.936, 95% CI = 0.859–0.974 versus mean = 0.744, 95% CI = 0.645–0.829, p < 0.001; MCI-AD, logic: mean = 0.918, 95% CI = 0.868–0.950 versus mean = 0.694, 95% CI = 0.591–0.773; pragmatic: mean = 0.872, 95% CI = 0.821–0.908 versus mean = 0.744, 95% CI = 0.645–0.829, p < 0.001) (Fig. 3).

OPEN IN VIEWER

Fig. 3

Verbal answers depending on inference type between diagnostic groups. No evidence of interaction or main effect of inference were found, only main effect of group (top graph; Inference, statistic = 1.64, df = 1, p = 0.200; Group:inference, statistic = 2.37, df = 1, p = 0.094; Group, statistic = 26.04, df = 1, p < 0.001, respectively; non-parametric repeated measures ANOVA). Differences in verbal answers were significant across all diagnostic groups (bottom table; Control-MCI p = 0.008; Control-AD p < 0.001; MCI-AD p < 0.001).

Visual search map across diagnostic groups

There were statistically significant differences between groups regarding duration of the test. The total duration of the test was higher in the AD group, followed by the MCI, and control groups (Table 2).

Comparing the visual search map across diagnostic groups, there were statistically significant group differences only for the answer screen (screen B). The following parameters: number of visits to the challenge stimulus, number of visits to the correct response stimulus, number of fixations on distractors, and time to first fixation on the correct response stimulus (Table 3) were capable of distinguishing diagnostic groups. Pairwise post-hoc analyses revealed that AD subjects had more visits to the challenge stimulus (Control-MCI, –0.108, SE = 0.149, p = 0.748; Control-AD, –0.622, SE = 0.190, p = 0.004; MCI-AD, –0.514, SE = 0.173, p = 0.011), more visits to the correct response stimulus (Control-MCI, –0.417, SE = 0.300, p = 0.350; Control-AD, –1.363, SE = 0.383, p = 0.002; MCI-AD, –0.946, SE = 0.349, p = 0.022), more fixations on distractors (Control-MCI, –1.640, SE = 0.919, p = 0.180; Control-AD, –4.580, SE = 1.172, p = 0.001; MCI-AD, –2.940, SE = 1.070, p = 0.020), and a longer time to first fixation on the correct response stimulus (Control-MCI, –0.108, SE = 0.149, p = 0.748; Control-AD, –0.622, SE = 0.190, p = 0.004; MCI-AD, –0.514, SE = 0.173, p = 0.011) than controls and MCI subjects. No differences in these parameters were found between controls and subjects with MCI.

We applied generalized linear mixed models for the binomial family to predict the odds of a correct answer based on the visual search map. Table 4 portrays the results from these models. The odds ratio (OR) expresses how much an increase of one unit of each parameter changes the odds of correct response. We observed that the parameters time to first fixation, number of visits, and fixation duration on target stimulus in screen A (target A) significantly contributed to the odds of correct answers in the inference comprehension task (screen B). Each second in the time to first fixation on the target (target A) increased the odds of a correct answer by 2.17 times. Each visit to the target (target A) increased the odds of a correct answer by 38%. Each second of the fixation duration increased the odds of a correct answer by 56% (Table 4). The number of fixations on the correct response stimulus on screen B increased the odds of making correct answers in the respective task (each fixation on the stimulus increases the odds of a correct answer by 20%), while the number of fixations on the distractor (distractor stimulus B) reduced the odds of a correct answer (each fixation on the distractor reduces the odds of a correct answer by 15%) (Table 4).

Correlations between oculomotor parameters and neuropsychological assessment

All visual search map parameters except for the number of visits and the number of fixations to target A had a significant correlation with one or more cognitive screening and neuropsychological tests (Table 5).

Type of inference and verbal answer

The understanding of inferences is considered a high-demanding task that recruits multiple cognitive functions such as language, memory, attention, and executive function. Therefore, it could be affected even in the early stages of MCI. In this study, the determinant of differences in verbal answers was the diagnostic group. Controls had a higher number of correct answers followed by MCI and AD subjects, subsequently. Different from expected, the type of inference, i.e., logic or pragmatic, did not discriminate diagnostic groups as per verbal answers. The cognitive demand in pragmatic inferences is considered higher than that of logical inferences. Our findings suggest that the level of cognitive decline seen in MCI subjects is sufficient to affect even the simpler type of visual inference.

Studies on inferences and cognitive decline focus mainly on textual inferences, with a lack of studies evaluating the inferential process through images. Schmitter-Edgecombe and Creamer verified that subjects with amnestic MCI produced fewer inferences in a story comprehension task than controls and had more difficulties explaining story events and using preliminary text information to support inference generation [38]. Similarly, Gaudreau et al. found that MCI participants were impaired in identifying ironic or sincere stories that required mental inference capacities, compared to control subjects [39]. Two studies evaluated a different cohort of MCI individuals in a textual inference task and also found that MCI patients had difficulty understanding inferences compared to controls [40, 41]. The results presented in our study are analogous to the findings found in the literature focused on text inferences [38–41].

Visual search map

As previously stated, the sum of the oculomotor parameters is defined as the visual search map. Although the parameters analyzed in screen A are significantly relevant for the correct understanding of inferences, the main differences found in this study were related to the visual search map on the “answers screen”, or screen B. The number of visits to the challenge stimulus, number of visits to the correct response stimulus, number of fixations on distractors, and time to first fixation on the correct response stimulus were particularly affected by group differences. Ultimately, the visual search map allowed the identification of the main visual aspects related to a better performance in the pictoric inference comprehension task. These findings could have both diagnostic and therapeutic implications.

In our study, screen A was composed only of a single stimulus, minimizing the odds for errors related to the subject’s distraction and inattention. Additionally, the subjects had no additional task but to scrutinize the image in screen A, which probably contributed to the absence of significant differences in the visual search map among the diagnostic groups. A previous study from our group found that the analyzed parameters were not able to discriminate between diagnostic groups in the familiarization phase when the subjects had only to look at the image presented on the screen [5]. Conversely, screen B presented the target stimulus and two distracting stimuli in addition to the challenge stimulus. Together, these findings suggest that increasing cognitive demand in a given task may account for differences in performance in subjects with different degrees and types of cognitive impairment.

It is known that the performance of an inference comprehension task consists of at least 4 steps: 1) attention to clues; 2) selection of relevant clues; 3) integration of relevant clues and finally; 4) association of explicit cues with previous experience. The inferential process follows an organized and stepwise order that includes different cognitive domains initiating with a semantic encoding, then the creation or recovery of mental models [42], defined by implicit and explicit stimulus information culturally and socially determined, ultimately leading to the comprehension of the visual stimuli [43]. This complex process involves diverse cognitive skills such as attention, sustained attention, memory, working memory, and theory of mind [44]. The maintenance of the attentional level is a basic condition for all cognitive tasks and a failure at this point can influence all later stages in a sequence of cognitive integration [45]. From a clinical point of view, our findings of an increase in target fixation time, an increase in the number of visits to the target, as well as an increase in the fixation duration have led us to hypothesize that patients whose attentional processing is adequate are in advantage in understanding visual inferences when compared to patients with attentional impairment. Furthermore, these results suggest that the differences among diagnostic groups are related to impairment in late and more complex processes of inferential processing, i.e., integration of relevant cues and association of explicit cues with previous experience (which is performed on screen B, where more than one stimulus is presented simultaneously, requiring that individuals divide their attention while selecting the appropriate answer). Camargo et al. found a similar pattern in their study, i.e., subjects with AD made longer fixations and more visits on distractors, more frequent fixations, and more visits to the target areas than other groups during the response phase [5].

Neuropsychological assessment

Most visual search map parameters were correlated with one or more cognitive screening and neuropsychological tests. The MoCA test showed a negative correlation with the following ET parameters, all from screen B tasks: number of visits to the challenge stimulus, number of visits to the correct response stimulus, and number of fixations on the correct response stimulus and the distracting stimuli. The pattern found for these parameters indicates that subjects expended a greater cognitive effort to perform the task. This pattern would lead to a lower score on the cognitive screening test. Tadokoro et al. used one of the ET parameters (fixation duration on predefined areas of interest) to assess tasks of visual working memory, attention, calculation, and visuospatial function in a group of control patients and patients with MCI and AD [46]. The authors found that the total eye tracking score significantly decreased in the MCI group and that this result correlated positively with the Mini-Mental State Examination (MMSE) score.

The parameters number of visits to the challenge stimulus, number of visits to the correct response stimulus, number of fixations on the correct response stimulus, and number of fixations on the distracting stimuli (all on screen B) also negatively correlated with the OANB, a test of oral naming of nouns and verbs involving visual processing. According to Spezzano et al., the naming task requires the recovery of phonological and semantic information, organized in a memory system that is accessed given the particularities of a given stimulus and involves the identification of the represented object [47]. All these steps are necessary for understanding inferences through visual stimuli.

The trail-making test (TMT) is a test that assesses attention, sequencing, mental flexibility, visual searching, and motor function. Likewise, the Stroop test, in addition to assessing such skills, also investigates executive function and inhibitory control. The main variable of interest, in both tests, is the total time to complete the test. Thus, we hypothesized that a worse TMT performance would relate to an increase in the number of fixations and visits not only in the correct target stimulus but also in the whole visual stimulus. A previous report from our group [4] aimed to determine whether older adults with MCI or incipient dementia due to AD displayed subtle signs of impairment in executive functioning as compared to healthy controls, through the analysis of eye movement. The results revealed that subjects with AD and MCI had a greater impact on the frequency and latency of eye movement metrics, showing a greater executive decline than controls. Additionally, the results were correlated with cognitive tests that evaluated inhibitory control, attention, working memory, and self-monitoring, including the TMT and Stroop.

Finally, as previously mentioned, the IMT evaluates the ability to comprehend inferences during reading activities and is designed to assess adult individuals with cognitive and/or communication complaints [31]. As expected, we found a correlation between the results of both tests that require the involvement of the same cognitive abilities, visual search map and attention, in addition to language. Importantly, although ET parameters and traditional cognitive assessments have previously shown positive correlations, there is a lack of studies further exploring these associations and the potential role of ET as an alternative to more traditional neuropsychological testing.

Underlying neuropathology

Both cerebral hemispheres are involved on some level in the inferential process. However, it has been reported that the right hemisphere is related to the ability to use context to generate inferences, thus being predominantly responsible for the inferential ability [48, 49]. Studies with individuals with right hemisphere damage show conflicting results with some revealing loss of ability to generate inferences and to predict upcoming events [50] while others maintain that these patients are still able to generate inferences [51]. These conflicting results may be interpreted as each hemisphere contributing differently to the generation of inferences depending on the context and background needed for the process, i.e., if familiar or unfamiliar situations, for instance [49]. Aspects related to the visual stimulus, e.g., its complexity also plays an important role in determining the brain regions involved in the inferential process [52]. Moreover, bilateral large-scale cognitive networks, including areas of visual processing, have also been implicated in inferential processing [53].

Limitations

Our study has some limitations. One aspect not investigated by our protocol is the role of top-down mediation on oculomotor parameters during a task. Oculomotor behavior during different visual tasks is known to be influenced by top-down factors that direct fixations and visits toward task-relevant areas of interest [54]. This aspect was investigated by Castelhano et al. using instructions– searching for a particular aspect or remembering characteristics of an image– that influenced specific oculomotor parameters, such as fixation duration and saccade amplitudes [55]. These findings were similar to those described by Mills et al., with spatial (e.g., amplitude) and temporal characteristics (e.g., duration) being found to be influenced by the task instructions [56]. In our study, the same instruction was given to all participants across the whole experiment. However, the top-down mediation could be a confounding factor in the study of cognitive decline in different stages of the AD continuum. There is evidence that top-down attentional control is impaired in individuals with MCI [57].

In general, one main difficulty faced when studying visual inferences is the ability to discriminate the cognitive demand involved (e.g., inferential, visuoperceptual, attentional, or working memory). These aspects have a non-neglectable effect on the interpretation of results. We partially addressed this matter by trying to isolate the inferential process domain when we added the challenge stimulus image to screen B, presented initially on screen A, reducing the subject’s need to demand working memory capacity for the task.

Although the number of subjects included was sufficient for reliable results, further studies with larger sample sizes are needed before these findings can be generalized. Our main focus was to determine the validity of the test developed by our team. The relatively reduced sample size also impacted our choice to not distinguish MCI subtypes (amnestic and non-amnestic), albeit recognizing that the visual search impairment present in amnestic MCI has been implicated as an early diagnostic marker in the AD continuum [3]. Also, non-AD MCI patients may have distinct visual search patterns, representing a bias in our results. However, there is a lack of studies investigating oculomotor behavior in MCI of all types, with most of them showing conflicting findings related to saccadic movements [58–61]. We believe that our present findings will contribute to the literature in the field.

Furthermore, cultural and academic backgrounds together with schooling play an important role in an individual’s ability to process images presented in an inference task [52, 62]. This interference cannot be neglected. A previous study from our group revealed that lower-educated subjects performed worse in an inferential task than higher-educated ones for both visually simple and complex pictures [63]. In our study, subjects with AD dementia had fewer years of education than controls and subjects with MCI. To avoid at least in part the effect of schooling on the ability to make inferences, we used only visually simple and inferentially simple images, a concept defined previously [63].