Abstract
Brief cognitive assessments are increasingly emphasized in MS treatment studies and clinical care. While much is known about the reliability of several widely-used neuropsychological tests, interpretation of the changes in individual patients is inadequate. The FDA offers guidance on the issue, as related to patient-reported outcomes. Unfortunately, cognitive ability is only weakly correlated with the frequency and severity of self-reported cognitive problems. In this review, we critically examined the psychometrics of neuropsychological testing in MS, emphasizing statistical and anchor-based approaches to interpreting clinically meaningful change.
We suggest that there are two paths forward that should be currently pursued. First, to employ co-primary outcomes, including a brief cognitive test and a clinician or observer’s impression on a scale of change, where successful treatment would require showing significant improvement in both measures. Secondly, to work toward showing that when reliable brief cognitive tests are employed, increments of statistically-relevant change would correlate with changes in clinically-relevant anchors (such as vocational disability or clinical relapses with cognitive impairment). The latter goal will allow a more parsimonious and scientifically efficient approach of utilizing only the brief cognitive test as a primary outcome. While some progress has been made in this direction, more research is needed. We are of the opinion that data from both the statistical and clinically meaningful approaches will be necessary to develop valid definitions of meaningful change on cognitive outcome measures, and that it would be best to pursue research using tests that already have well-established reliability and validity.
Keywords
Cognitive dysfunction is common in multiple sclerosis (MS). 1 –3 Advances in immunomodulatory therapy have chiefly been shown to impact physical functioning, but there is comparatively little evidence about their impact on cognitive impairment. As prognosis improves, patients are increasingly interested in understanding this area of more limited information. As researchers progress down the path of understanding medication’s effects on neuropsychological (NP) status, it is imperative that the psychometric underpinnings of cognitive outcome measures be fully understood. We require confidence in the reliability, validity, and interpretability of future study results.
Psychometrics of candidate cognitive outcomes
The US Food and Drug Administration (FDA) has published a guidance on patient-reported outcomes (PROs) that emphasizes psychometric standards. 4 While PROs are distinct from NP measures, the psychometric principles discussed in this Guidance are relevant to our discussion.
Reliability is the degree to which the patient’s test result is the same across time points, examiners, and varying test formats, absent any change in clinical status. In short, reliability refers to freedom from measurement error. Reliability of cognitive outcomes used in MS studies has frequently not been thoroughly evaluated; however, it is important because an error-prone outcome measure can hinder detection of a treatment effect, thus increasing the likelihood of a type 2 error. Fortunately, there is recent support for the reliability of some commonly-used NP tests in MS patients. 5
Interpretability, or validity, comes in many forms that for our purposes can be reduced to two general concepts. The first is construct or concept validity, or the nature of the meaning ascribed to a test score. If we are using a test of information processing speed, for example, the test should be able to discriminate MS patients from healthy controls, should correlate with established measures of processing speed, and predict any problems that are known to be related to processing speed, such as vocational disability. Consensus has developed regarding the most conceptually valid neuropsychological tests for MS research. 5 –8
The other principle we label quantitative interpretability: The metric of a test should be adequately understood in order to enable interpretation of values obtained (e.g. high average, abnormal) and the incremental changes in individual patients (e.g. meaningful improvement or worsening). The FDA Guidance emphasizes that evidence of treatment benefit is not a purely statistical issue, but rather that it is important to also be able to interpret the observed treatment effect as clinically meaningful. The guidance does not discuss in detail how the interpretability of a PRO (or by implication a NP test) is established. One possible approach is the use of a responder definition: Identification of a score difference that can be interpreted as a treatment benefit (i.e. clinically meaningful). Development of patient assessment tools in many fields has often considered the smallest change in the measurement that would warrant considering modifying patient management, the minimally clinically important difference (MCID). 9 The FDA Guidance also indicates that while group data may be used to help develop thresholds for a MCID, ultimately, the clinical meaningfulness of a score difference is something that should be judged relative to an individual patient, not as the mean score of a treatment group.
For many outcome assessments used in clinical trials, “face validity” (i.e. the self-apparent value to a patient of what is being directly measured) has been relied upon to establish clinical meaning. Obviously, when there is strong face validity, less additional evidence of interpretability is needed. Some components of the Expanded Disability Status Scale (EDSS) 10 are face valid in that the gait impairment elicited during the test is readily observable and clearly impacts the patient. As such, the EDSS is accepted by the FDA as an efficacy endpoint in clinical trials of treatments for MS. The face validity of NP tests is considerably poorer, necessitating more information that will substantiate not only the construct/concept validity of a test, but also its quantitative interpretability, as related to clinically meaningful outcomes.
Quantitative interpretability: A statistical approach
NP results, like most patient assessments, encompass variability around the conceptual true score for an individual. Variability can arise from many sources, including variation in the patient’s motivation and subtleties in how the test is administered by study site staff; in fact, the amount of variability from these error-related sources may change during a clinical trial at successive time-points. In a clinical study, there might also be gain in the true score at sequential time-points, from sources other than the effects of treatment, such as an increasing familiarity with the test, which is the so-called practice effect.
One approach to demarcating a significant change in score might address this variability by determining the amount of change (improvement or decline) that exceeds the variability in the measure, thus representing a statistically true change in NP score status. This variability could be assessed by evaluating the relationship of intra-patient test and re-test scores, under conditions when the patient’s underlying true test score has not changed. Determination of a test-retest reliability coefficient (more generally termed the correlation coefficient) in a patient group comparable to the patients planned for study would enable calculation of a statistically reliable change index (RCI), 11 the amount of change in an individual patient’s score needed to justify concluding that it is statistically significantly different than the pre-treatment score (Table 1). An alternative approach would be to calculate the test-retest difference for each patient, and from these values, the mean and standard deviation of the test-retest differences. A confidence interval (e.g. 90% CI) could then be drawn around the mean, to describe a “limit of agreement,” 12 –14 which like the RCI reveals the amount of change needed to conclude that a statistically relevant change has occurred. Both these approaches can guide the understanding of what differences reflect “true” change in an outcome; however, because they consider only the statistical characteristics of the measure, there is no information about the amount of change that has actual meaning to a patient’s daily life.
Hypothetical data to illustrate statistically significant change.
Notes: In the hypothetical data above, the effect size d is 0.4 in treated patients and 0.1 in controls (mean difference divided by each group’s SD). Our objective was to identify individuals who responded to treatment on a statistical basis. The test-retest r (Pearson’s correlation coefficient for this case) is used to form a CI for identifying a change that is beyond that seen in the placebo group (where it is assumed no patient had a true change in neurological function). The r and SD enable calculation of the SEm (formula is SEm=SD*sqrt(1-r)), and in turn the SEdiff. The RCI adds to the SEdiff-based CI the mean shift observed in the placebo group (potentially a learning effect), to become the RCI that can be applied to the individual patients in the treated group, to designate which patients have had a change in score that is statistically likely to be, at least in part, from a true drug effect. Higher r values or smaller SEdiff values produce a narrower CI (e.g. the 80%CI shown). The data in the different Placebo Control rows illustrate how changes in the variability (SD) and test reliability markedly impact the CI. Reliable tests have less error, and in turn more confidence, that smaller changes from baseline reflect a non-error difference, e.g. treatment effect.
CI: confidence interval; Post TRT: post-treatment; r: reliability coefficient; RCI: reliable change index; SeDIFF: standard error of the difference between test and re-test; SEm: standard error of the measure; sqrt: square root.
Quantitative interpretability and clinically meaningful change
The statistical approach described may be sufficient when the outcome measure is clearly face valid overall, as the measure may also intrinsically enable clinical interpretation of scores or score changes; however, many NP measures lack sufficient face validity to provide intrinsic quantitative interpretability. A measure often used in clinical trials where the experimental therapy may improve walking ability is the 25-foot walk (T25FW) 15 time, which seems to have face validity as an outcome in that it is a direct assessment of the construct under investigation. Yet, even in this case, it can be questioned whether a particular change in speed is relevant. Is it important, for example, if after treatment a patient can walk 25 feet in 4.25 rather than 5.00 seconds?
The question is critical for any outcome measure, but particularly so when the outcome lacks face validity. For example, the Symbol Digit Modalities Test (SDMT) 16 has superior reliability in MS studies of cognition, 3 but the clinical meaning of any particular change in score (number of correct symbol-number pairings within 90 seconds) is not yet well understood. Indeed, on the surface, the task, which is timed processing of symbol/number pairings, is not clearly related to everyday activities. The anchor-based approach to establishing quantitative interpretability relies on a comparison of the measurement outcome to a face-valid patient rating or judgment of meaningfulness to the patient. The rating should be external to the psychometric measure, but in the same content domain or construct. The FDA Guidance 4 recommends that an anchor should be easier to interpret, or more concrete, than the outcome measure.
This consideration brings out the idea that the patient’s perspective on change is the most relevant and meaningful. Given the low or questionable face validity of many NP measures, we are therefore tasked with finding valid anchors for gauging the meaningfulness of incremental gains or declines in MS studies.
Lessons learned from studies of motor function in MS
The concerns about the clinical interpretation of specific amounts of score changes or differences is not limited to NP tests. Two motor function tests are commonly used in evaluating MS patients: Timed 25-Foot Walk test (T25FW) 15 and the Nine-Hole Peg Test (9HPT). 17 Several investigator groups have worked on the issue of what constitutes a clinically perceptible change in these measures. Some studies 18,19 examined changes in T25FW and 9HPT over time, along with changes in the Guy’s Neurological Disability Scale (GNDS), 20 a rating scale evaluating 12 different domains. The comparisons for each test were made in a grouped manner; evaluating the mean GNDS change in patients categorized as: 20% or more improvement, less than 20% change in either direction, or 20% or more worsening. A separate study used a patient global assessment of change after steroid treatment as the anchor. 21 These studies recognized the need for an assessment external to the test to serve as an anchor with intrinsic clinical meaning. These studies suggest that changes of 20% or more in tests may indicate perceptible effects in patient function; however, these studies are unable to provide more definitive evidence, because of the structure of the analysis and the type of anchor used. Clinically perceptible change is an individual patient event, but these studies did not analyze the data in a patient-by-patient manner, to determine the likelihood of perceptible change when test scores change by 20%. Analysis of the mean GNDS change in three categorized groupings of patients does not distinguish whether the perceptible change occurs with 10%, 20%, or 30% changes in the test scores. Also, all three of these studies were limited in the strength of their conclusions because the external anchor was not an evaluation of the same concept that T25FW or 9HPT measure. The change in GNDS explicitly, and the patient global assessment implicitly, describe the patient’s perceived change across many domains of function.
Kaufman and colleagues 22 applied T25FW prior to, and during, MS relapses and used the patient’s perception of change in their walking ability as the external anchor. These data were analyzed on a within-patient comparison of the amount of change seen in T25FW testing versus the patient’s perception of change in walking ability. Because the anchor was of the same domain as the test, and the comparison was set up in a within-patient manner, this study provides better evidence that it is a 20% change in T25FW results that is clinically perceptible to patients, while lesser amounts are not.
Schwid et al. 23 used these tools as well, but focused on determining a statistically reliable change size. The report emphasizes the distinction between the statistical and clinically meaningful change approaches, which was to be the focus of succeeding work.
In summary, there has been some previous success in evaluating meaningful change in tests of motor function, at least with respect to the T25FW. These studies illustrate an experimental model to make progress. A standard of 20% change, established as reliable based on statistical grounds, was associated with an external anchor of the patient’s perception of gait deterioration (a face valid rating). Both the statistical reliability determination and the clinical meaning anchor comparison are important to assure that an assessment is useful in clinical trials. It is clear that the meaningfulness question is difficult to address for motor function; it is especially so for cognition, although this research shows that it is likely an achievable task.
Self-reports are not valid anchors for cognitive outcomes
The MS Neuropsychological Screening Questionnaire (MSNQ) was originally developed to screen for cognitive disorders, 24,25 and secondarily, to assess the patient and caregiver perspectives on NP status. Prospective research, 24,26 confirmed by others, 27,28 shows that while the former goal was not met, the latter was achieved. Specifically, it appears that the degree of correlation between self-reported problems with cognition and performance-based metrics is small, tending to be statistically significant in groups of patients with a more progressive course and with greater degrees of impairment, but not significant in more mildly-affected patients. 27,30 There is better agreement between NP testing and informant-reported MSNQ data, although not to the degree that one can use the MSNQ to screen for NP impairment with sufficient sensitivity and specificity. While the MSNQ is considered reliable, providing interesting data with regard to indifference and depression pseudo-dementia, 26 several studies show that self-reported scales focusing on cognition are mostly related to depression, not cognitive capacity. 28,29,31 Thus, it would be difficult, if not impossible, to supplant a performance-based measure of cognition with a self-reported assessment, nor to use self-assessments as anchors for interpretability.
Other methods for achieving cognitive outcomes interpretation: Co-primary outcomes
A co-primary approach to the primary efficacy evidence is used in drug treatment studies in some clinical areas where no optimal study endpoint had been identified. The co-primary method employs the best of what psychometric testing offers, as well as the less quantitative but more face valid measures. In brief, we accept the weaknesses of each approach and require that statistically significant change be observed on each measure.
Treatment efficacy in Alzheimer’s disease (AD) is usually demonstrated through use of such co-primary endpoints: a psychometrically-developed adequate cognition-specific test (e.g. ADAS-cog) and a non-specific co-primary rating such as the Clinical Global Impression of Change (CGIC). The latter is performed by clinicians blinded to patient treatment, who develop a subjective assessment of response to treatment via interview and other gross examination techniques. The CGIC is less specific than the NP test, because many factors other than cognitive capacity contribute to the rating, but it assures that if a change in cognitive performance is demonstrated during therapy, there is also a meaningful benefit to patients. This approach has seldom been applied in MS studies of treatments regarding cognition.
A form of the CGIC was used in some recent trials of symptomatic therapies for cognitive dysfunction. 32 –34 In Krupp et al. (2004), 32 the CGIC was completed by a single physician and defined as no change, worse, or improved. In a more recent study, 33 the CGIC was enhanced by use of the MSNQ; and was the only outcome showing a trend toward significance that favored the treated group. The co-primary endpoint approach is not optimal; however, it may be recommended when the interpretation of a change in the psychometric, domain-specific test is difficult. To the best of our knowledge, there is no single version of the CGIC that is recognized as optimally reliable or valid for MS patients.
Another approach is to use performance-based activities of daily living (ADL) scales as co-primary outcomes, rather than a GCIC. Examples are the Timed Instrumental Activities of Daily Living Test (TIADL) 35 and the Executive Function Performance Test (EFPT). 36 These performance tests have greater face validity than the more commonly-used NP tests such as the SDMT. For these, patients are asked to perform activities of daily living (e.g. paying a bill by writing a check or cooking a simple meal) in a standardized, controlled setting. In MS, the TIADL 37 and EFPT 38 are correlated with conventional NP testing and they can discriminate patients from controls with medium effects sizes (e.g. d = 0.2 for medication management to 0.6 for bill paying). 39 Recent work has extended this approach to making purchases using the internet. 40 Limitations of the method are the requirements for setting up the standardized environments in which to conduct tasks, and the questionable fit with the subject’s real daily activities of participants (i.e. their face validity is not, in our opinion, absolutely clear).
Current status of cognitive outcomes interpretation: Correlates of change scores
The need for a co-primary outcome may be obviated if an increment of change is known to be associated with a clinically-meaningful anchor. Thus, there is great value to having an especially reliable and valid test that is easily administered, and on close examination, is shown to be associated with concrete, face valid anchors that are relevant to cognition. While such work is in its infancy, some preliminary progress toward this goal has been made.
In one study, 41 97 employed patients were assessed using a consensus standard neuropsychological battery. At the follow-up after roughly three and one-half years, it was found that 45% of the patients had experienced a decline in employment status. Changes in SDMT and the California Verbal Learning Test’s second edition (CVLT2) 42 best distinguish patients that are judged to have stable employment from those patients experiencing a loss of employment, after controlling for demographic and other disease course variables. An SDMT decline of 4 points maximally discriminated the groups.
Another potentially face valid anchor is a clinically observable decline in neurological status, eg a MS relapse. In a 48-week open-label, safety-extension study of natalizumab (STRATA), MS patients were screened for PML using monthly administration of the MSNQ, SDMT and a depression test. Relapses occurred in 53 of 660 patients and their SDMTs were available for the pre and post-relapse. The data from relapsing patients was compared with SDMT scores from 115 MS patients without relapses, who were matched in age, gender, baseline SDMT and time from study initiation. It was found that SDMT is adversely impacted at the time of relapse, but shows a nearly complete return to the control level afterward (Figure 1). These results confirm the involvement of cognitive function during some MS relapses and suggest that a meaningful decrease in an individual’s SDMT may be on the order of the 5% group mean decrease observed in this study.
SDMT performance at monthly time points, reprinted with permission from Morrow SA, Jurgensen S, Forrestal, et al.: Effects of acute relapses on neuropsychological status in multiple sclerosis patients. Journal of Neurology 2011; 258: 1603–1608. Scores are for before (pre) and after (post) the documented case relapses (n = 53) and at equivalent time points in the controls (n = 115). These data may show that a clinically meaningful change represented by the test score decline and rebound are timed to a readily discernible clinical event. However, the neurological basis of the event requires clarification in a group of patients undergoing mental status changes during the relapse.
A problem with this study is that the clinical status of the patients in relapse was not recorded in such a way as to be certain as to the neurological symptoms at the time of testing, so that the relapse-related perceptible cognitive change status is unknown. In a second study now underway, we (RHBB and colleagues) have collected data from patients undergoing a cognitive relapse and their matched MS controls who were undergoing repeat testing while they were clinically stable. Three months later, the exam was repeated. All patients have a baseline neuropsychological examination already available via archived data. Our study will determine the amount of NP score decline at the point of clinically-evident decline in mental status, plus the degree of rebound after 3 months.
Conclusions and future directions
In MS, cognitive impairment is clinically relevant, but there are few, if any, tests that possess all the important attributes of a good endpoint for clinical trials; i.e. being reliable, conceptually valid, and quantitatively interpretable, so as to be able to establish that there is a clinically meaningful change. Some NP tests have a strong psychometric foundation; their reliability coefficients are good to excellent, especially where processing speed and memory are emphasized. 43 With good reliability, any small changes in a patient’s score can be statistically significant, but that change as defined by statistical significance does not establish the clinical meaningfulness of such change. NP test measurements are also associated with the capacity for employment and for instrumental activities of daily living; 44,45 however, the relationship with vocational disability may not apply to any patients choosing not to participate in the work force. It has not been as yet adequately studied in a patient-by-patient manner, to determine the likelihood of a perceptible change when the test scores have changed by specific amounts.
Some studies have begun to show a path forward, in seeking to achieve clinical interpretability of change on NP testing. This work has been done with a single, mental processing speed test and it will be important to be able to replicate the results in the domain of memory, which is often defective in MS patients. 1 It is an area of recent treatment investigation. 33,46 We are of the opinion that data from both the statistical and clinically-meaningful approaches are needed, and therefore that it will be best to start with the NP tests that have well-established reliability. Better anchors may also be needed. Also, vocational status assessment has been implemented in a coarse dichotomous manner, but a more fine-grained approach (e.g. number of hours worked per week, frequency of work errors, etc.) may permit finer discrimination of patients who are benefitting from treatment versus the non-responders. Other measures of employment status, such as the formal annual performance appraisals used in many workplaces, might also be considered. Other cognition-related aspects of a patient’s life may also offer anchors for establishing the quantitative interpretability of NP tests. If more finely graduated, but still face-valid, anchors can be applied, smaller increments of change in NP tests could be shown to be clinically meaningful. With progress in the area of NP testing, we hope that investigators will eventually be able to measure the effects of either existing or new treatments on cognitive function, and patients will be able to consider the potential benefits when weighing their choices of whether, or how, to treat their MS.
Footnotes
Funding statement
This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.
Conflict of interest
The authors declare no conflicts of interest.
Disclaimer
The views expressed in this article by author MKW are the author’s, and do not represent an official FDA position.