Advances in the Evaluation of Longitudinal Construct Validity of Clinical Outcome Assessments

Abstract

Historically, the assessment of longitudinal construct validity in the field of psychosocial measurement involved defining hypotheses and calculating correlation coefficients using scores based on 2 measures at 2 or more time points. In the context of patient-reported outcomes, this evolved into sensitivity to change and responsiveness, including the computation of effect size estimates of change, standardized response means, and indices such as Guyatt’s statistic. Cross-sectional analyses or analyses based on 2 time points have been the standard practice. Evolving conceptualizations have incorporated more than 2 time points and have included depictions of individual trajectories of change in multiple measures, structural equation models, and mixed modeling techniques. The focus of this article is on methods to evaluate longitudinal construct validity. We describe a sample of these methods and provide considerations and recommendations for designing a thoughtful longitudinal construct validity evaluation of clinical outcome assessments.

Keywords

longitudinal construct validity validation responsiveness psychometrics clinical outcome assessment

Introduction

While the traditional textbook definition of validity is “the extent to which an assessment measures what it is supposed to measure,”^1

–5 a comprehensive definition of validity refers to the extent that the proposed uses and interpretations of assessment scores are supported by theory and evidence.^6

–13 Validity is not simply yes or no—it is whether a particular concept is adequately measured by an assessment and, more importantly, whether the scores derived from the assessment can support inferences and decisions that are expected to be made using the assessment. As stated in the FDA’s guidance on patient-reported outcome (PRO) measures, “the adequacy of an instrument’s development and testing is specific to its intended application in terms of population, condition, and other aspects of the measurement context for which the instrument was developed.”¹⁴ Indeed, a medical product labeling claim based on a clinical outcome assessment (COA) must be “consistent with the instrument’s documented measurement capability … to ensure the conclusions drawn using instrument scores are valid.”¹⁴

It has also long been stated that “validation is never finished.”¹⁵ It is an “unending”¹² or at least an “ongoing”^16,17 and “continuing” process.¹⁰ What is intended by these assertions is that validity cannot be definitively established or proven—much like a hypothesis cannot be proven—and an assessment is never conclusively validated. Instead, analyses provide and accumulate evidence for the validity and interpretation of the scores in a particular setting and within a given patient population or context of use.

For example, COAs used to support endpoints in clinical trials or observational studies that are considered to validly measure the physical functioning of the general adult population in the United States are not necessarily valid to evaluate changes in the physical functioning of elderly heart failure patients in the context of a multicenter international clinical trial. Nor are those COA scores automatically valid measures for comparing postsurgical differences in physical functioning between groups of patients receiving different types of knee implants. As another example, COA scores used by clinicians in practice for the purpose of tracking the worsening of physical symptoms in patients with a particular neuromuscular disorder will not necessarily support a claim of treatment benefit in medical product labeling or even symptom improvement in a clinical trial.

The validity of an assessment can be divided into numerous types, including face validity, content validity, criterion validity, and construct validity.^12,18,19 In the United States, regulatory review of COA scores proposed as key endpoints is often led by the Food and Drug Administration (FDA) COA staff (formerly Study Endpoints and Labeling Development [SEALD]). Guidance by this team has focused primarily on content validity and construct validity: Content validity as supported through qualitative input from patients during the development of a COA to provide evidence that the measure captures the aspects that are relevant to capture; construct validity as primarily supported through quantitative results that COA scores are consistent with a priori hypotheses (including hypotheses describing the internal COA relationships among item scores and external relationships among items and scale scores with scores from other related and unrelated measures).^7,12,14,19 Although the FDA guidance specifies PROs in its title, the methods and principles of the PRO guidance are to be applied to the evaluation of all COAs, including observer-reported outcomes (ObsROs) and clinician-reported outcomes (ClinROs).

The focus of this article is on construct validity, specifically the extension of the evaluation of construct validity beyond a cross-sectional assessment to incorporate data from multiple time points: a longitudinal assessment of construct validity. Numerous statistical techniques have been developed to simultaneously model multiple outcomes collected across multiple time points in the study of change and factors that impact change.^20

–26 Our overall objective is to describe methods, including a sample of emerging techniques, used for evaluating the construct validity of COAs over time with considerations and recommendations for designing a thoughtful COA longitudinal construct validity evaluation.

Construct Validity Methods

As there are multiple ways to classify aspects of validity, there are multiple ways to classify construct validity. For the purpose of this paper, any psychometric assessment that begins with a hypothesis about how COA item or scale scores behave is construct validity. Using this definition, evidence for a COA’s construct validity can be gained through the use of three key types of analyses: correlational, known groups, and responsiveness analyses.

While correlational analyses are considered the primary method, we have included known groups and responsiveness within construct validity because results from these methods can provide important information about relationships among scores. Again, the essential requirement for the application of these evaluations is the development of hypotheses based on the constructs intended to be measured by the target COA and the constructs or groups defined by the available external measures or criteria. We contend that results reported without this theoretical step are not evaluations of construct validity.

Correlational Analyses

In all construct validity applications, hypotheses are proposed regarding the relationships between the constructs intended for measurement by the COA and available external measures. These hypotheses should indicate the direction (positive or negative) of the construct relationships and the magnitude (either in strength or relative to other planned correlations). Typically, correlation coefficients are computed between the COA and external measures to assess the strength and direction of the hypothesized associations. Correlational analyses treat the 2 outcome measures symmetrically. Other regression-based methods to test associations require the COA to be predicted by an external measure, and there are several excellent references describing the extension of these methods in both cross-sectional and longitudinal frameworks.^1,27
–29 However, we often compute correlation coefficients and employ Cohen’s³⁰ correlation rule of thumb to describe the strength of expected correlations (strong: r ≥ 0.50; moderate: 0.10 ≥ r < 0.50; weak: r < 0.10), but in some cases we may be able to hypothesize only in terms of relative strength.

An optimal correlational analysis will include convergent (similar) and divergent (dissimilar) construct validity hypotheses, with internal hypotheses based on the item scores of the COA and external hypotheses constructed for the COA scores with scores from other measures. The pattern can be formalized and tested as in the context of a multitrait-multimethod matrix.^12,31 Alternatively, Zou³² provides an illustration of statistically testing individual correlation values against a prespecified value (eg, H₀: ρ = 0.5) rather than relying on the traditional null (H₀: ρ = 0). Cross-sectional or longitudinal construct validity is supported when the observed correlation values are as hypothesized (in terms of magnitude, sign, and pattern).

Reeve and colleagues report the results of a survey of members of the International Society for Quality of Life Research (ISOQOL) that sought to establish minimum standards for PRO measures.⁵ Fifty-five percent would require that a measure have evidence supporting its construct validity, including “documentation of empirical findings that support predefined hypotheses on the expected associations among measures similar or dissimilar to the measured PRO”; 44% considered this “desirable but not required.”⁵

Cross-sectional

After hypotheses have been finalized, the computations can be as simple as a series of Pearson or Spearman correlation coefficients between the target COA and another measure that can be computed using data from a single time point or a series of single time points. These snapshots provide evidence (or lack of evidence) in support of the hypothesized relationships. Comparisons cannot be statistically tested for independent correlation coefficients across time and measures unless modeled simultaneously.

Longitudinal

We can capitalize on the nature of longitudinal COA data for a statistically testable evaluation of construct validity over time. The simplest case is the computation of a correlation using change scores from 2 time points; for increased complexity, change scores across more than 2 time points can be evaluated (Figure 1 shows individual trajectories of several individuals for a single COA).

With this approach, hypotheses can account for the time-varying nature of the relationships, thereby providing a context for testing differences in the relationships over time. For example, if an established treatment is included in the study, it may be hypothesized that the relationship between a COA and a related clinical variable’s change will be more strongly associated after several weeks of treatment than during the study run-in period.

There are numerous approaches and software programs for simultaneously modeling multiple dependent variables (in our application, the target COA and external measures). Mixed models with both fixed- and random-effects structures present one mathematical framework to estimate the correlations jointly. Results from these models can provide cross-sectional and longitudinal information, including statistical comparisons, supporting the construct validity of the target COA.

One approach presented in the statistics literature is to fit the generalized linear mixed model (GLMM) to accommodate both continuous and discrete dependent variables.^20,33,34 Faes and colleagues³⁵ describe 2 ways to jointly model and test the correlation between 2 dependent variables, which can be implemented in SAS PROC GLIMMIX: (1) in the residual error term using an unstructured correlation matrix (ie, each pair of outcomes has its own correlation coefficient)^35,36 or (2) through a shared random effect.

Another approach used extensively in the social sciences is to simultaneously fit these curves (Figure 1) in the structural equation modeling (SEM) framework.^{23,24,37
–39} Within the SEM framework, these models are typically referred to as latent growth models (LGMs). Essentially, LGMs are a special case or mapping of the linear (or the generalized linear) mixed models and provide an estimate of the between-person differences in within-person change.⁴⁰ Dependent LGMs can be implemented in SEM software such as MPlus. (See Box 1.)

Box 1. Description of the LGM

In growth-mixture models, the fixed effect represents the mean trajectory pooled across all persons in the sample, and the random effect represents the variance of the individual trajectories around the mean trajectory. In the LGM framework, growth parameters are modeled as a set of latent constructs. This requires that each loading of the observed variables (eg, the time-varying COA observations) and latent growth parameters (eg, intercept and slope) be fixed to a particular value. Typically, the loadings on the latent slopes are fixed to polynomial regression coefficients (eg, linear, quadratic, cubic), and the loadings on the latent intercepts are fixed to 1.

The following equations present an example parameterization of an unconditional (ie, no covariates) model for a COA and an external measure (EM) in an LGM framework. In the example, i refers to an individual the COA was measured at four time points and the EM was measured at three time points, which do not have to be the same time points. Additionally, the model assumes a linear function of time. Figure 2 depicts this model using a path diagram.

Y_{i 1} = B C O A_{0 i} * 1 + B C O A_{1 i} t_{1} + ε_{i 1}

Y_{i 2} = B C O A_{0 i} * 1 + B C O A_{1 i} t_{2} + ε_{i 2}

Y_{i 3} = B C O A_{0 i} * 1 + B C O A_{1 i} t_{3} + ε_{i 3}

Y_{i 4} = B C O A_{0 i} * 1 + B C O A_{1 i} t_{4} + ε_{i 4}

Y_{i 5} = B E M_{0 i} * 1 + B E M_{1 i} t_{5} + ε_{i 5}

Y_{i 6} = B E M_{0 i} * 1 + B E M_{1 i} t_{6} + ε_{i 6}

Y_{i 7} = B E M_{0 i} * 1 + B E M_{1 i} t_{7} + ε_{i 7}

[\begin{matrix} \begin{matrix} B C O A_{0 i} \\ B C O A_{1 i} \end{matrix} \\ \begin{matrix} B E M_{0 i} \\ B E M_{1 i} \end{matrix} \end{matrix}] \sim N ([\begin{matrix} μ_{B C O A_{0}} \\ μ_{B C O A_{1}} \\ μ_{B E M_{0}} \\ μ_{B E M_{1}} \end{matrix}], [\begin{matrix} σ_{B C O A_{0}}^{2} & - \\ σ_{B C O A_{1} B C O A_{0}} & σ_{B C O A_{1}}^{2} \\ \begin{matrix} σ_{B E M_{0} B C O A_{0}} \\ σ_{B E M_{1} B C O A_{0}} \end{matrix} & \begin{matrix} σ_{B E M_{0} B C O A_{1}} \\ σ_{B E M_{1} B C O A_{1}} \end{matrix} \end{matrix} \begin{matrix} - & - \\ - & - \\ \begin{matrix} σ_{B E M_{0}}^{2} \\ σ_{B E M_{0} B E M_{1}} \end{matrix} & \begin{matrix} - \\ σ_{B E M_{1}}^{2} \end{matrix} \end{matrix}])

ε_{i j} \sim N (0, σ_{ε}^{2})

In this example, the first dependent variable, the COA, was assessed at 4 time points (referred to as {Y_i1, Y_i2, Y_i3, Y_i4} at times {t₁, t₂, t₃, t₄} in Equations 1a –1d, also shown in the path diagram in Figure 2). These are regressed onto a latent intercept (BCOA_0i), a latent slope (BCOA₁ _i), and the corresponding residual error {∊_i1, ∊_i2, ∊_i3, ∊_i4}. The second dependent variable, the EM, was assessed at 3 points (referred to as {Y_i5, Y_i6, Y_i7} at times {t₅, t₆, t₇} in Equations 1e –1g in Figure 2), and regressed onto a latent intercept (BEM_0i), a latent slope (BEM₁ _i), and the corresponding residual error {∊_i5, ∊_i6, ∊_i7}. The BCOA and BEM latent intercepts and slopes are assumed to be multivariate normally distributed, with mean structure and variance-covariance structure shown in Equation 2. The residual terms can be assumed independent and homoscedastic as in Equation 3 or may be parameterized to allow the residual terms of the BCOA and BEM to correlate. The longitudinal construct validity correlation between 2 dependent variables can be conceived as the correlation between their 2 latent slopes, $ρ_{{BEM}_{1} {BCOA}_{1}}$ , in Figure 2.

In exchange for the ability to statistically test the comparisons based on the a priori hypotheses, these techniques can require a sizeable number of observations (in order for the modeling algorithm to converge) and assume data that are missing at random.^20,24 Alternatively, when appropriate to ignore the time-varying nature of the relationships or to provide a signal of construct validity, the correlation values can be computed using change from baseline to the postbaseline treatment period average, for example, when the treatment response is immediate and consistent across time.

Known-Groups Analysis

Known-groups analyses are used to evaluate the discriminating ability of a COA by classifying study subjects into groups based on some characteristic with a hypothesized relationship to the construct and testing group mean differences.^{1,10,41
–43} An optimal analysis will include hypotheses based on classifications that are strongly related to the COA’s construct. Evidence is weaker when group mean differences fail to achieve statistical significance (if hypothesized as strong) and weaker still when differences are in the wrong direction. Known-group differences for other measures provide an immediate context for the results computed for the target COA.

Reeve and colleagues’ survey of measurement professionals showed that only 41% would require known-groups analyses for support of construct validity while 57% felt it was “desirable but not required.”⁵

Cross-sectional

Group-level score comparisons based on data from a single time point or a series of single time points can be conducted using t tests, analyses of variance (ANOVAs), or nonparametric tests depending on the nature of the COA scores and group classification variables.

Longitudinal

Group-level score comparisons can be conducted using repeated-measures ANOVA (when the known groups are constant) or the mixed-model framework (using the GLMM or the SEM framework when the known groups vary over time).^18,20,24,44 As with cross-sectional correlational analyses, the size and statistical significance support construct validity in the longitudinal case.

Ability to Detect Change

Necessary evidence for the longitudinal validity of a COA is its responsiveness—that is, its ability to detect change when change is expected (ie, the ability of a COA to measure a change in a clinical state that is valuable and meaningful to the patient or clinician).^1,2,43 The PRO Guidance favors information as to whether “there is evidence that PRO scores are affected by changes that are not specific to the concept of interest, the PRO instrument’s validity may be questioned.”¹⁴ Evidence that the patient experience of a concept has changed in tandem with the COA of interest supports the COA’s responsiveness and its successful use in clinical trials. The PRO Guidance does not cite best methods for quantifying a COA’s ability to detect change.¹⁴

An optimal hypothesis focuses on change over a period of time that includes a strong and relevant intervention (eg, an established therapy) or a duration that has been established as long enough for meaningful disease progression.² The nature of the medical condition must be taken into consideration when developing the responsiveness hypotheses. For example, in disorders such as oncology and muscle wasting, the treatment goals surround slowing or halting deterioration rather than facilitating improvements. Identifying a COA capable of detecting even the smallest deterioration is key in these types of conditions to understanding the disease and treatment impacts. Gathering evidence to support responsiveness can be more difficult in such circumstances compared with conditions in which quick and strong improvements can be achieved through established interventions. Computation of responsiveness statistics for other measures provides an immediate context for the size of the responsiveness statistic computed for the target COA.

The results of the ISOQOL survey by Reeve and colleagues⁵ showed that a majority (57%) agreed that a measure to be used in longitudinal research should have evidence of responsiveness, and 42% considered this “desirable but not required.” A second question about responsiveness in the ISOQOL survey asked whether cross-sectional data were acceptable: “if a PRO measure has cross-sectional data that provide sufficient evidence in regard to the reliability, content validity, and construct validity but has no data yet on responsiveness over time, would you accept use of the PRO measure to provide valid data over time in a longitudinal study if no other PRO measure was available?” The use of cross-sectional data in lieu of longitudinal data was endorsed by 65% of the respondents, while 32% of respondents would require prior evidence of responsiveness.⁵ Interestingly, in their review of sample sizes used to validate PRO measures, Anthoine and colleagues⁴⁵ reported that responsiveness was examined in only about 10% of the studies reviewed, typically via paired t test.

Cross-sectional

By definition, the evaluation of responsiveness requires at least 2 time points. In the absence of 2 time points, some researchers have suggested that a known-groups classification at a single time to compute a responsiveness statistic based on the assumption that the amount of cross-sectional difference between the 2 groups may in some circumstances be a suitable proxy for change over time.^46,47

Longitudinal

There is a large and extended family of responsiveness statistics, and a number of excellent sources describe and review measures of responsiveness.^{4,46

–51} While Terwee and colleagues⁵¹ identified more than 30 different statistics in the literature for quantifying a COA’s ability to detect change, many of these take the general form of a difference (between time points or groups) in the numerator and a measure of variability (eg, standard deviation [SD] or standard error [SE]) in the denominator, resulting in a difference or change value expressed, respectively, in standard deviation units or t test statistics, both of which are 2 types of standardized effect sizes.

From these various reviews, there are numerous ratios between a point estimate and a measure of variability for quantifying responsiveness. The more common and popular responsiveness statistics are the paired (or repeated measures) t test, effect size estimate of change ([Mean_follow-up – Mean_baseline] ÷ SD_baseline), standardized response mean ([Mean_follow-up – Mean_baseline] ÷ SD_change), standardized mean change difference ([Mean change_group1 – Mean change_group2] ÷ SD_{baseline pooled}), and Guyatt’s statistic⁵² ([Mean_{treatment group at follow-up} – Mean_{treatment group at baseline}] ÷ SD_{change in control}). In addition to evaluating responsiveness using the family of statistics described in this section, the correlation between change in a COA and change in an external measure is a measure of responsiveness.

In a nutshell, these statistics provide a comparison of change based on a prespecified unit. The unit should be chosen based on the ultimate use of the COA. Deyo and colleagues⁴⁷ prefer a variant of Guyatt’s statistic, Norman and colleagues⁵⁰ recommend Cohen’s d and the standardized response mean (SRM), Zou⁴⁹ recommends the SRM, and Terwee and colleagues⁵¹ recommend change correlations. Finally, the responsiveness statistic value for the target COA should exceed the observed statistic for another measure selected as a construct validity comparator, if available.²⁰

Cohen³⁰ provided multiple rules of thumb for interpreting effect sizes, including correlations (described earlier). Cohen’s rule of thumb regarding the interpretation of noncorrelation effect sizes is often-repeated and much-maligned: effect sizes of about 0.20 are small, those of about 0.50 are moderate or medium, and those ≥0.80 represent large effects. These simple guidelines are echoed by Kazis and colleagues,⁵³ Cappelleri and colleagues,⁵⁴ and others^55,56: 0.2-0.49 are small, 0.5-0.79 moderate, and ≥0.80 are large. These are only rough guidelines and do not supplant a thorough familiarity with typical effect size results in the area of COA research and development. It is particularly important to compute effect size statistics for other measures included in the study to provide an immediate context for the size of the responsiveness statistic computed for the COA of interest. Note that disease-specific measures are generally expected to be more responsive than generic measures.

Discussion

Before using a COA to support decision making, research should evaluate the validity of the COA for the given population and intended purpose. If the planned assessment is related to change in the COA scores, it is imperative that the COA be shown to be a valid measure of change. Therefore, longitudinal evidence should be available. We have described a sample of methods for evaluating aspects of a COA’s longitudinal construct validity.

Beyond these, numerous additional methods offer some information about construct validity (eg, longitudinal item response theory). The selection of an assessment based on modeling is often highly dependent on the researcher’s training and experience. Biostatisticians may prefer to parameterize using mixed models, education researchers may prefer hierarchical linear models, and psychologists may focus on LGMs. As the numerical techniques and software packages evolve, even the subtle variations in terminology and parameterization across these models can make it difficult to bridge the differences and offer straightforward recommendations for how to apply these techniques to support longitudinal construct validity. For the interested researcher, we recommend Fitzmaurice and colleagues,^20,34 Davidian and Giltinan,²¹ Grimm,²² Stull,²³ Preacher,²⁴ Duncan and Duncan,²⁵ Bollen and Curran,³⁷ and Singer and Willet²⁶ as good resources to help build the technical background necessary to choose among approaches.

Standard practice should involve a thoughtful assessment with theory-driven hypotheses (ranked by importance or priority) using correlational analyses, known-groups analyses, and an evaluation of the measure’s ability to detect change.¹⁹ The number and quality of the hypotheses (eg, availability of external measures that are highly relevant to the measured construct) and consistency of findings across methods can provide a substantial case for the COA’s longitudinal construct validity.⁵ If a study includes few relevant measures, the strength of any inferences is diminished. Additional challenges for the collection of construct validity evidence include sample size, study design, and the level of missing data.⁵⁷ The PRO guidance emphasizes the COA’s fit for purpose, but psychometric evaluations using cross-sectional data pose a complication for small samples encountered in rare and orphan diseases. Longitudinal designs offer an opportunity to assess the psychometric properties of COAs for rare/orphan diseases. There is an important need for the COA literature to address small-sample longitudinal modeling, in particular, exploration of questionnaire structure, methods of scaling, and measures of goodness of fit.

In summary, although cross-sectional evaluations of construct validity provide some evidence in support of a COA intended to measure change (eg, a COA designed to show the benefit of an intervention), adding the complexity of the longitudinal data through multiple approaches may provide stronger evidence and should be considered when designing the evaluation. Selecting multiple external measures that assess both strongly similar and strongly dissimilar concepts will provide a stronger base of evidence, rather than using a limited set of external measures and potentially “double dipping.”

COAs allow us to measure patient-centered outcomes in a consistent way across studies, samples, and, where appropriate, across diseases. Because of the importance of identifying and developing appropriate items to measure these constructs, the development of a COA requires a large investment of time and resources. The subsequent pretesting and psychometric evaluation provide quantitative evidence to indicate whether the COA development was successful for its intended use. Evidence in support of construct validity—incorporating both cross-sectional data and longitudinal data—is a key component of the psychometric evaluation.

Footnotes

Acknowledgments

The authors acknowledge Sheri Fehnel, Donald Stull, and Lee Bennett for their review and technical contributions to this paper.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was performed and funded by RTI Health Solutions. All authors are employees of RTI Health Solutions.

References

Cappelleri

Zou

Bushmakin

Alvir

JMJ

Alemayehu

Symonds

. Patient-Reported Outcomes Measurement, Implementation, and Interpretation. London: Chapman & Hall/CRC; 2014.

Hays

Revicki

. Reliability and validity (including responsiveness). In: Fayers

Hays

, eds. Assessing Quality of Life in Clinical Trials. 2nd ed. New York: Oxford University Press; 2005:25–40.

Hays

Anderson

Revicki

. Assessing reliability and validity of measurement in clinical trials. In: Staquet

Hays

Fayers

, eds. Quality of Life Assessment in Clinical Trials. New York: Oxford University Press; 1998:169–182.

Husted

Cook

Farewell

Gladman

. Methods for assessing responsiveness: a critical review and recommendations. J Clin Epidemiol. 2000;53:459–468.

Reeve

Wyrwich

. ISOQOL recommends minimum standards for patient-reported outcome measures used in patient-centered outcomes and comparative effectiveness research. Qual Life Res. 2013;22:1889–1905.

AERA, APA, NCME. Standards for Educational and Psychological Testing. Washington, DC: AERA; 2014.

Borsboom

Mellenbergh

van Heerden

. The concept of validity. Psychol Bull. 2004;111:1061–1071.

Cronbach

Meehl

. Construct validity in psychological tests. Psychol Bull. 1955;52:281–302.

De Vet

HCW

Terwee

Mokkink

Knol

. Measurement in Medicine. New York: Cambridge University Press; 2011.

10.

McDowell

Newell

. Measuring Health—A Guide to Rating Scales and Questionnaires. 2nd ed. New York: Oxford University Press; 1996.

11.

Messick

. Validity. In: Linn

, ed. Educational Measurement. 3rd ed. New York: American Council on Education, Macmillan; 1989:150–170.

12.

Nunnally

Bernstein

. Psychometric Theory. 3rd ed. New York: McGraw-Hill; 1994.

13.

Wainer

Braun

, eds. Test Validity. Hillsdale, NJ: Lawrence Erlbaum; 1988.

14.

Food and Drug Administration. Guidance for industry. Patient-reported outcome measures: use in medical product development to support labeling claims. http://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/UCM193282.pdf. Published December 2009. Accessed December 31, 2013.

15.

Cronbach

. Five perspectives on validity argument. In: Wainer

Braun

, eds. Test Validity. Hillsdale, NJ: Lawrence Erlbaum; 1988:3–17.

16.

Scientific Advisory Committee of the Medical Outcomes Trust. Assessing health status and quality-of-life instruments: attributes and review criteria. Qual Life Res. 2002;11:193–205.

17.

Suen

. Principles of Test Theories. Hillsdale, NJ: Lawrence Erlbaum; 1990.

18.

Cappelleri

Bushmakin

Harness

Mamolo

. Psychometric validation of the physician global assessment scale for assessing severity of psoriasis disease activity. Qual Life Res. 2013;22:2489–2499.

19.

Mokkink

Terwee

Patrick

. The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient-reported outcomes. J Clin Epidemiol. 2010;63:737–745.

20.

Fitzmaurice

Davidian

Verbeke

Molenberghs

, eds. Longitudinal Data Analysis. Boca Raton, FL: Chapman & Hall/CRC; 2009.

21.

Davidian

Giltinan

. Nonlinear Models for Repeated Measurement Data. Boca Raton, FL: Chapman & Hall/CRC; 1995.

22.

Grimm

. Multivariate longitudinal methods for studying developmental relationships between depression and academic achievement. Int J Behav Dev. 2007;31:328–339.

23.

Stull

. Analyzing growth and change: latent variable growth curve modeling with an application to clinical trials. Qual Life Res. 2008;17:47–59.

24.

Preacher

. Latent Growth Curve Modeling. Thousand Oaks, CA: Sage; 2008.

25.

Duncan

. The ABC’s of LGM: an introductory guide to latent variable growth curve modeling. Soc Personal Psychol Compass. 2009;3:979–991.

26.

Singer

Willett

. Applied Longitudinal Data Analysis: Modeling Change and Event Occurrence. New York: Oxford University Press; 2003.

27.

Fairclough

. Design and Analysis of Quality of Life Studies in Clinical Trials. 2nd ed. Boca Raton, FL: Chapman & Hall/CRC; 2010.

28.

Cappelleri

Bushmakin

. Interpretation of patient-reported outcomes. Stat Methods Med Res. 2014;23:460–483.

29.

Mallinckrodt

Lane

Schnell

Peng

Mancuso

. Recommendations for the primary analysis of continuous endpoints in longitudinal clinical trials. Drug Inform J. 2008;42:305–319.

30.

Cohen

. Statistical Power Analysis for the Behavioral Sciences. 2nd ed. Hillsdale, NJ: Lawrence Erlbaum; 1988.

31.

Khanna

Maranian

Rothrock

. Feasibility and construct validity of PROMIS and legacy instruments in an academic scleroderma clinic—analysis from the UCLA scleroderma quality of life study. Value Health. 2012;15:128–134.

32.

Zou

. Sample size formulas for estimating intraclass correlation coefficients with precision and assurance. Stat Med. 2012;31:3972–3981.

33.

Nelder

Wedderburn

RWM

. Generalized linear model. J Royal Stat Soc A. 1972;135:370–384.

34.

Fitzmaurice

Laird

Ware

. Applied Longitudinal Analysis. 2nd ed. Hoboken, NJ: John Wiley; 2011.

35.

Faes

Geys

Catalan

. Joint models for continuous and discrete longitudinal data. In: Fitzmaurice

Davidian

Verbeke

Molenberghs

, eds. Longitudinal Data Analysis. Boca Raton FL: Chapman & Hall/CRC; 2009.

36.

Odom

McLeod

Sherif

Nelson

McSorley

. Methods to assess the association of patient-reported outcomes and clinical endpoints. Presented at: 20th Annual International Meeting of the International Society for Pharmacoeconomics and Outcomes Research; May 18, 2015; Philadelphia, PA.

37.

Bollen

Curran

. Latent Curve Models: A Structural Equation Perspective. New York, NY: John Wiley; 2006.

38.

Meredith

Tisak

. Latent curve analysis. Psychometrika. 1990;55:107–122.

39.

Swartz

Schwartz

Basch

. The king’s foot of patient reported outcomes: current practices and new developments for the measurement of change. Qual Life Res. 2011;20:1159–1167.

40.

Curran

Obeidat

Losardo

. Twelve frequently asked questions about growth curve modeling. J Cogn Dev. 2010;11:121–136.

41.

DeVellis

. Scale Development—Theory and Applications. 2nd ed. Thousand Oaks, CA: Sage; 2003.

42.

Terwee

Bot

SDM

de Boer

. Quality criteria were proposed for measurement properties of health status measures. J Clin Epidemiol. 2007;60:34–42.

43.

Fayers

Machin

. Quality of Life: The Assessment, Analysis and Interpretation of Patient-Reported Outcomes. 2nd ed. Chichester, England: John Wiley; 2007.

44.

Brussoni

Kruse

Walker

. Validity and reliability of the EQ-5D-3L™ among a paediatric injury population. Health Qual Life Outcomes. 2013;11:157. doi:10.1186/1477-7525-11-157.

45.

Anthoine

Moret

Regnault

Sebille

Hardouin

. Sample size used to validate a scale: a review of publications on newly developed patient-reported outcome measures. HQOL Outcomes. 2014;12:176. doi:10.1186/s12955-014-0176-2.

46.

Beaton

Bombardier

Katz

Wright

. A taxonomy of responsiveness. J Clin Epidemiol. 2001;54:1204–1217.

47.

Deyo

Diehr

Patrick

. Reproducibility and responsiveness of health status measures. Control Clin Trials. 1991;12:142S–158S.

48.

Liang

. Longitudinal construct validity: establishment of clinical meaning in patient evaluative instruments. Med Care. 2000;38(9 suppl):II84–II90.

49.

Zou

. Quantifying responsiveness of quality of life measures without an external criterion. Qual Life Res. 2005;14:1545–1552.

50.

Norman

Wyrwich

Patrick

. The mathematical relationship among different forms of responsiveness coefficients. Qual Life Res. 2007;16:815–822.

51.

Terwee

Dekker

Wiersinga

Prummel

Bossuyt

PMM

. On assessing responsiveness of health-related quality of life instruments: guidelines for instrument evaluation. Qual Life Res. 2003;12:349–362.

52.

Guyatt

Walter

Norman

. Measuring change over time: assessing the usefulness of evaluative instruments. J Chron Dis. 1987;40:171–178.

53.

Kazis

Anderson

Meenan

. Effect sizes for interpreting changes in health status. Med Care. 1989;27(suppl):S178–S189.

54.

Cappelleri

Bushmakin

McDermott

Sadosky

Petrie

Martin

. Psychometric properties of a single-item scale to assess sleep quality among individuals with fibromyalgia. Health Qual Life Outcomes. 2009;7:54–60.

55.

Nelson

Williams

VSL

Fehnel

. Psychometric validation of patient-reported outcome measures assessing chronic constipation. Clin Exp Gastroenterol. 2014;7:385–394.

56.

Williams

VSL

Morlock

Feltner

. Psychometric evaluation of a visual analog scale for the assessment of anxiety. Health Qual Life Outcomes. 2010;8:57.

57.

Chen

McLeod

Nelson

Williams

Fehnel

. Quantitative challenges facing patient-centered outcomes research. Expert Rev Pharmacoecon Outcomes Res. 2014;14:379–386.