Reliability and Feasibility of the Memory Associative Test TMA-93

INTRODUCTION

In cognition, binding is the function that supports the integration of multiple elements together [1–3]. Errors in conjunctive binding (the integration of features within an object) and also in relational binding or associative memory (the ability to remember novel associations between words or pictures) have been reported in Alzheimer’s disease (AD) at an early phase [4]. Conjunctive binding is supported by the entorhinal and perirhinal cortex and seems more sensitive than the relational one to early AD [5, 6]. The “Short-Term Memory Binding Test” is the most used tool for examining conjunctive binding. This test, one of the most promising neuropsychological tools, has been incorporated into trials to predict who among those with mild cognitive impairment will go on to develop AD [7]. There is also evidence that relational binding, that relies on the hippocampus, parahippocampal cortex, and default mode network regions (posterior cingulate cortex/precuneus/lateral parietal and medial frontal cortex) [5], declines in the prodromal stages of late-onset sporadic AD [8–10]. Even more, asymptomatic individuals with greater amyloid-β burden on amyloid imaging have shown abnormal scores on relational binding tests when the performance on other standardized episodic memory test is still preserved [11]. In neuropsychology, the relational binding ability can be examined by different tests. The “Wechsler Memory Scale” (WMS) assesses binding through learning and recall of paired associated words [12]. This WMS subtest discerns between easy (i.e., North/South) and complex associations (i.e., School/Cellar) [12]. The “Memory Binding Test” (MBT) examines associative memory through the recall of pairs of items that belong to the same semantic category (i.e., flea/ant = insects) but presented in two different lists of words [13]. The “Face Name Associative Memory Exam” is a cross-modal associative test based on a functional magnetic resonance imaging (fMRI) task that pairs pictures of unfamiliar faces with common first names [14].

Testing relational binding only by images rather than words could be more feasible for low educated individuals. The “Memory Associative Test of the district of Seine-Saint-Denis” (TMA-93) was recently developed in France for the early diagnosis of AD among low educated immigrants [15]. Briefly, during the encoding phase, the patient is shown ten pairs of drawings of common and easy to recognize objects from daily life that are semantically related (Fig. 1A). Only one of the two items is shown in the recall phase, and the patient is asked to recall the missing item (Fig. 1B) [15]. In the original paper, the test demonstrated high diagnostic accuracy for discriminating AD patients from healthy controls in a sample of immigrant residents from a district in Paris (France) [15]. In that study, the cutoff of 24 of 30 showed a sensitivity of 88% and a specificity of 97% for distinguishing AD patients from healthy controls [15]. A posterior validation study in older educationally-diverse Spanish people demonstrated that the test is so sensitive as the picture version of the Free and Cued Selective Reminding Test (FCSRT) in discriminating between amnestic mild cognitive impairment (aMCI) patients and healthy controls (HCs) [16]. In that study, the receiver operating characteristic (ROC) curve analysis determined an optimal area under the ROC curve (AUC) of 0.97 (95% CI, 0.89–1.00, p < 0.001) to distinguish between aMCI patients and HCs [16].

OPEN IN VIEWER

Fig. 1

Pairs of semantically-related drawings of the TMA-93. In the codification phase, the semantically-related drawings are presented in pairs (A). In the recall phase, the subject has to recall the missing object (B).

On the other hand, the most used memory tests are based on learning and recalling two paragraphs or a list of words and often include a final step of facilitation with verbal cues or recognition among distractors that have to be administered 15–30 min later [17, 18]. These tests take too long time to be used in busy primary care and general neurology outpatient settings with limited face-to-face time per patient. On the contrary, the TMA-93 is a relatively short test that may be more suitable in that context.

These potential uses and advantages of the TMA-93 encourage the completion of the development of the test. There are no previous studies focused on the reliability or feasibility of the TMA-93. There is a need to validate the test-retest reliability of binding tasks to detect and monitor AD-related populations [4]. Tests providing such reliability will be appropriate for use in longitudinal research. On the other hand, feasibility has been considered a crucial prerequisite by a consensus document on neuropsychological assessment [19]. The aim here was to study the reliability (the internal consistency and the inter-rater and test-retest reliability) of the TMA-93 and its feasibility (by recording the percentage of participants who completed the test, and by timing the administration time).

METHODS

RESULTS

Socio-demographics characteristics and neuropsychological background for the extension of the cross-sectional study focused on internal consistency are shown in Table 1. For the total sample (n = 75), 46 participants were females. Their average age was 74.6 (SD = 5.9, range = 51–84). Regarding educational attainment, 31 individuals (41.3%) had not completed primary studies (Table 1). There were no significant differences in age, gender, or educational attainment between aMCI and HCs groups (Table 1). aMCI patients scored significantly lower than HCs on Phototest, DMS48, and TMA-93 (Table 1). Internal consistency was “optimal” (Cronbach’s alpha = 0.936). Split-half reliability was also high (Spearman-Brown coefficient = 0.911). Corrected item-total score correlations ranged from 0.661 for the pair “hand-watch” to 0.837 for the pair “flower-sun” (Table 2). There was no redundancy of any item as the Cronbach’s alpha did not increase at deleting anyone (Table 2).

Socio-demographics characteristics and neuropsychological background for the test-retest reliability study are shown in Table 3. Their average age was 64.8 (SD = 8.9, range = 50–86). 13 subjects (25.5 %) had not completed primary studies, and 30 (58.8%) were females (Table 3). Test-retest reliability for the TMA-93 total score was “good” [ICC = 0.802 (CI 95%  = 0.653–0.887)]. The “total score time 2 minus total score time 1” variable showed a non-normal, right asymmetric, and leptokurtic distribution (median = 0, IQR = 0–1, Range = –3–3). There were four atypical observations: two of them scored three points higher at the retest and the remaining two scored two and three points lower, respectively (Fig. 2). We analyzed the TMA-93 total score at time 2 by the TMA-93 total score at time 1: the variability was greater for scores below 28, and some practice effect could be detected in the range 27–29 (Fig. 3).

OPEN IN VIEWER

Fig. 2

Boxplot chart showing the distribution of the “total score time 2 minus total score time 1” variable. There are four outliers.

OPEN IN VIEWER

Fig. 3

Scatterplot with Time 1 performance on the x axis and Time 2 performance on the y axis. The variability in the measure was greater for scores below 28, and some practice effect could be detected in the range 27–29.

Table 4 shows the characteristics of the sample for the inter-rater reliability and feasibility study. Their average age was 68.7 (SD = 7.2, range = 55–81). 16 subjects (32.7%) had not completed primary studies, and 32 (65.3%) were females. There were statistically significant differences in the TMA-93 scores across the three diagnostic groups (Table 4). The inter-rater reliability was “optimal” for the TMA-93 total score [ICC = 0.999, (CI 95%  = 0.999–1)], number of intrusions [ICC = 0.985 (CI 95%  = 0.974–0.992)], and number of errors [ICC = 0.996 (CI 95%  = 0.993–0.998)]. The inter-rater reliability for the number of perseverations was “good” [ICC = 0.853 (CI 95%  = 0.738–0.918)].

All participants, including mild AD dementia patients, completed the test. There were statistically significant differences in the TMA-93 duration across the three diagnostic groups (Table 4). Post-hoc multiple comparison analyses revealed that the duration of the administration (in minutes) was significantly lower in healthy controls (median = 2.2, IQR = 2.0–4.0, range = 1.5–5.5) than in aMCI (median = 6.2, IQR = 4.7–7.8, range = 2.3–11.7, p < 0.05) and mild AD dementia patients (median = 7.5, IQR = 5.9–9.4, range = 5.0–17.2, p < 0.001). The aMCI and mild AD dementia groups did not differ significantly (p = 0.337). There were two outliers from the mild dementia group, with an administration time longer than 15 min (Fig. 4). There were no statistically significant differences in the TMA-93 administration time by educational attainment in the inter-rater reliability study (<first grade: median = 6.28, IQR = 2.94–9.00, range = 1.82–17.25; first grade: median = 5.18, IQR = 2.63–7.58, range = 1.85–11.78; >first grade: median = 4.58, IQR = 2.43–7.10, range = 1.53–11.33; p = 0.399) (Fig. 5). To better analyze the educational attainment effect on the administration time, we went back to the test-retest study and evaluated differences in administration time by educational attainment among the HCs at test 1. Again, there were no significant differences (<first grade: median = 2.47, IQR = 1.77–3.40, range = 1.37–4.59; first grade: median = 2.23, IQR = 1.44–2.46, range = 1.38–3.16;>first grade: median = 2,26, IQR = 1.88–3.15, range = 1.46–4.11; p = 0.352) (Fig. 6).

OPEN IN VIEWER

Fig. 4

Boxplot chart depicting the differences in “Administration Time” (in minutes) among the three diagnostic groups of the inter-rater reliability and feasibility study.

OPEN IN VIEWER

Fig. 5

Boxplot chart depicting no significant differences in “Administration Time” (in minutes) by educational attainment in the inter-rater reliability and feasibility study.

OPEN IN VIEWER

Fig. 6

Boxplot chart depicting no significant differences in “Administration Time” (in minutes) by educational attainment in the test-retest reliability study at time 1.

DISCUSSION

To our knowledge, this is the first study focused on the reliability and feasibility of the TMA-93, the French visual relational binding test [15]. The test has already demonstrated high diagnostic accuracy in validation studies [15, 16] and has normative studies from French and Spanish populations [24, 38].

Internal consistency among the 10 pairs of semantically-related drawings of real-life objects that compose the TMA-93 was “optimal” (Cronbach’s alpha = 0.936). This result means the 10 items of the test are highly correlated each other and measure the construct of interest, visual relational binding, in a similar way [39]. By comparison, an “acceptable” internal consistency has been reported for the FCSRT, a standard memory test (Cronbach’s alpha = 0.810) [40].

“Corrected Item-Total Correlation” is the correlation of the item designed with the summated score for all other items. A rule-of-thumb states that this value should be at least 0.40 to rule out item redundancy [35]. Every item of the TMA-93 fulfilled the rule. In the same way, the Cronbach’s alpha did not increase at deleting any of the ten pairs, so, again, redundancy of any item could not be demonstrated.

Split-half testing is another measure of internal consistency. This method measures the extent to which all parts of the test contribute equally to what is being measured. We found a strong correlation between the two virtual halves of the TMA-93, indicating that HCs and aMCI patients performed equally well (or as poorly) on both halves of the test.

The TMA-93 showed a “good” test-retest reliability [ICC = 0.802 (CI 95%  = 0.653–0.887)]. By comparison, this reliability is similar to that reported for the “Mini-Mental State Examination” (MMSE) (0.80) [41] and suggests stability in performance over time. The test-retest reliability studies’ design varies by the time considered between test 1 and test 2, and by the selection of participants (only HCs or mixed sample of HCs and patients). Here, we considered 2–4 months for administering the retest and only HCs. The time interval seems to be short enough to prevent the effect of an eventual cognitive impairment on the sample, particularly from participants with lower scores, and long enough to prevent a practice effect. With a similar design, the MBT demonstrated ICC values ranged from 0.64 to 0.76 [42]. Analyzing the distribution of the “total score time 2 minus total score time 1” variable, there were four atypical observations that probably precluded this reliability could be upgraded to “optimal”. Two of them scored 3 points more at the retest. On the opposite side, two outliers scored 2 and 3 points less, respectively. The former could be explained by practice effect and the latter by cognitive decline, but a more global explanation could be that binding is somewhat changeable and dynamic, making it difficult for a test to achieve an “optimal” test-retest reliability [43]. The variability in the measure were greater for scores below 28 at time 1. The test-retest reliability could be supported by scores above 28 at time 1 and, thus, overestimated due to ceiling effect. To clarify this issue, future test-retest reliability TMA-93 studies should recruit enough HCs scoring below 28 at time 1 and consider participants’ AD biomarker status to understand eventual score changes over time.

Inter-rater reliability of the TMA-93 was “optimal” for the total score and the number of errors and intrusions, and “good” for the number of perseverations. We noted that the administration and scoring are relatively simple, but that classifying the incorrect responses in errors, intrusions or perseverations can lead to disagreements between examiners and requires some training. Individually, perseverations—scored as the number of times that an error (a response that corresponds to a different drawing pair) is repeated—were the main source of disagreement between examiners.

Regarding TMA-93 feasibility, all participants, including mild AD dementia patients, were able to complete the test. Participants’ task-tolerability was good, including that of those who scored the minimum (4 out of 30) or whose administration time was the longest (17.2 min). There were significant differences in administration time by diagnosis: cognitively impaired patients spent more time on recalling the missing drawing, made more mistakes, and usually needed the maximum of three memory trials.

The average time required to complete the test was 2–3 min for HCs, 6 min for aMCI patients, and 7 min for mild AD dementia patients, so this test is relatively short despite being a specific memory test and not a brief cognitive screening test as MMSE or MoCA. By comparison, the time of passing the MMSE in cognitively impaired patients is, on average, 4 min 51 s [44]. Busy primary care and general neurology outpatient settings with limited face-to-face time per patient need a short but specific memory test. The TMA-93 could fill the gap. The test runs with a ceiling effect in HCs and is highly discriminative for diagnosing patients with aMCI or mild dementia [16]. However, a floor effect should be expected in patients with moderate dementia and may could be already present in some patients with mild dementia, here represented by the outliers for whom the administration of the test took longer than 15 min. The target of the TMA-93 are mainly patients with memory complaints and no functional impairment when total scores on MMSE or MoCA are around the cutoffs and are not conclusive [24]. Studies comparing the diagnostic accuracy and feasibility of the TMA-93 against screeners, as MMSE or MoCA, in settings with limited face-to-face time per patient are needed.

The samples here tested were composed of a relatively high percentage of low-educated participants. Lack of education remains a limitation in many elderly Spanish people since they had limited primary school access in the aftermath of the Spanish Civil War (1936-1939). Although the situation has significantly improved in recent years, 59% of the population over 65 years of age in Spain did not complete primary studies [45]. Low-education is also a limitation for people in many developing countries in the world. In most developed countries, multicultural individuals with a different primary language, not proficient in the host country one, also have this limitation. The neuropsychological examination must comply with this handicap. Here, the TMA-93 was again demonstrated feasible to be administered to low-educated individuals. In fact, there were no significant differences in administration time by educational attainment. Despite this feasibility, the TMA-93 total score should be expected lower in low-educated individuals. Feasibility does not mean that the test is free of educational bias. Associative learning is also trained and acquired at school and, accordingly, normative studies show lower TMA-93 total score in less educated groups [24, 38].

In addition to optimal diagnostic accuracy previously reported for the TMA-93, the good reliability and feasibility here demonstrated encourages the completion of the test's development. The next steps will be phase II and III validation studies, including AD biomarkers and comparing the diagnostic accuracy of the test with that of the standard memory instruments on samples organized by educational attainment.

CONFLICT OF INTEREST

The authors have no conflict of interest to report.