Abstract
When establishing telehealth services, clinicians need to be confident that the examinations, assessments and clinical decisions that they make while using technology are equivalent to conventional best practice. Method-comparison studies are ideally suited to answering these questions, however there is a lack of consistency in the telehealth literature in the study methodologies and data analysis techniques used. Methodologies should closely match clinical practice to maximise external validity and data analysis techniques should match the data types generated in order to be clinically meaningful. In this article we discuss the design, analysis and interpretation of method-comparison studies in the context of telehealth research.
Introduction
With the ability to provide quality care at a distance, telehealth has rapidly become a key strategy for many healthcare providers. In addition to considering the evidence base for the clinical effectiveness of telehealth, clinicians need to be confident that the examinations, assessments and clinical decisions that they make while using this model of service delivery are similar to conventional practice. This becomes increasingly important as technology moves into the homecare setting where often the only point of contact with the patient may be via telehealth. Method-comparison studies are ideally suited to address these questions. The previous article in this research methods series has discussed the important topic of properly constructing a research question and choosing an appropriate methodology to answer it. This article will serve to further explore and illustrate research designs that are appropriate to validate new assessments and measurements performed via telehealth.
Researchers and clinicians who set out to establish the validity and reliability of a telehealth assessment are often faced with many questions such as:
What should the assessments be compared to? How accurate does the assessment need to be for it to be acceptable? How many assessments should be performed in the study and on how many different patients? What is the best way to analyse the results? And how should the results be interpreted from a clinical perspective?
Assessments that are performed via telehealth tend to fall into two categories: either the examination of individual items (e.g. measurement of blood pressure, blood glucose, muscle strength, joint range of motion, functional assessment) which can be used to establish health status or treatment outcomes; or the assessment may be diagnostic in nature (for example, to determine the presence or absence of a condition or disease). In this article, we will discuss the design, analysis and interpretation of method-comparison studies with reference to these two categories of telehealth assessment.
Understanding the general methodology
Two different overarching methodologies for method-comparison studies have been commonly used: equivalence studies and non-inferiority studies. In equivalence studies, we are interested in whether the new assessment does not differ from the conventional (usually in-person) assessment in either direction by a pre-specified amount (i.e. a two-sided test). In an equivalence trial the new assessment method will be selected regardless of whether it is better or worse than an existing assessment as long as the difference falls within the predefined zone of allowable difference (and meets other criteria such as cost effective and stakeholder satisfaction). 1
Commonly in telehealth, the existing model of care (e.g. specialist assessment in tertiary hospital for cognitive impairment) will not be replaced, but rather the telehealth option will be used for people who cannot access conventional services. In this case, the question is whether the telehealth assessment is ‘as good’ as or rather ‘not inferior’ to conventional practice. It is irrelevant whether the telehealth assessment is in fact better than the existing model of service, as long as it is not worse, the telehealth assessment can be implemented in jurisdictions where it will be useful. This type of study is called ‘non-inferiority’, because the focus is only on whether the new assessment is not worse than usual care (i.e. a one-sided test). For these reasons, non-inferiority designs are more prevalent than equivalence studies in the telehealth literature. Figure 1 demonstrates the subtle but important difference between these two study types.
Boundaries of equivalence and non-inferiority trials. The symbol δ represents the pre-specified equivalence/non-inferiority margin.
Getting the design right
In method-comparison studies, each assessment/activity is performed at least twice with one performed via each method of assessment (telehealth and conventional). This enables the concurrent validity of the telehealth assessment to be established by comparing the results to the conventional assessment. Care must be taken in the planning of these assessments and the design of the study to make sure that the telehealth assessment has the best chance of being equivalent/non-inferior. Factors which need to be considered include, but are not limited to: the setting, timing considerations, technology selection, defining the comparator, choosing raters and choosing the number of assessments and participants to include. A brief discussion of each of these items follows.
Selecting the setting
It should be obvious that the setting in which the telehealth assessments are performed can have a large impact on the quality of the assessment results. Consider an assessment performed via videoconferencing into a telehealth studio where the lighting, acoustics and technology have been carefully selected, compared to an assessment performed into the home via a low-cost tablet computer. When designing the method-comparison study, consideration should be given to how the assessment will be performed in clinical practice and this should be recreated as closely as possible. If multiple settings are used in practice, the validity of the assessment in each environment should be separately established.
Often, the setting also dictates the support structures that are available at the end of the consultation. Hospital or clinic-based end-points may have telehealth support staff who can assist whereas homecare consultation may not. Including or excluding support staff during the method-comparison study should be carefully considered depending on the intended telehealth service.
Similar considerations should be given to the conventional assessment: these assessments should be conducted in a setting that is consistent with usual practice with all conventional equipment available. It should be noted that the method-comparison study only establishes the validity of the telehealth assessment as it was performed in the study and any variation should be avoided when translating the assessment to the clinical environment.
Timing considerations
In method-comparison studies, we are primarily interested in determining if two different methods of assessing a patient agree. It is therefore important that the underlying element that they are assessing is relatively stable between the two assessments. This is critical as any real change in the patient will appear as variance between the two assessment methods in the analysis. This is not a problem in conditions which are relatively stable over time and are unlikely to change. For example, the detection of melanoma mentioned previously is unlikely to change in the time that is required to assess the patient in both environments. It does, however, become problematic for assessments which fluctuate over time. Consider, for example, the evaluation of the motor symptoms associated with Parkinson’s disease which may vary on factors such as the time of the day or the therapeutic level of medications. While limiting the time between assessments may minimise these factors, there may also be undesirable effects such as fatiguing the patient, introducing test-retest bias and exacerbating symptoms such as pain for the second assessment. Any of these factors can produce a real change in the patient which will be interpreted as variance between the two assessment methods. This problem has been widely encountered in the literature.2,3 A careful consideration of these many factors is required to minimise the possibility of a real change in the patient between the two assessment points.
A methodology used less often in telehealth method-comparison studies is that of simultaneous telehealth and conventional assessment. 4 The primary advantage of this methodology is that there is no potential for variance in the underlying patient state, and therefore any discrepancy is attributable to the method of assessment. Using the example of a motor assessment in a patient with Parkinson’s disease, both the telehealth assessor and the conventional assessor would simultaneously observe the patient performing an assessment task (such as a functional activity) and simultaneously score the item. The disadvantage of this methodology is that the assessment needs to be led by one of the practitioners with the other practitioner passively observing and scoring. This calls into question the validity of the assessment as it varies in terms of the setting and the clinical protocol that would naturalistically be performed. One strategy that can partially mitigate this problem is to randomise which practitioner leads the assessment so that approximately half of the assessments are led by each assessment method.
Technology selection
It should be obvious that all technology used in a telehealth method-comparison study should match the equipment that is planned for the telehealth service. However, a commonly observed flaw in telehealth method-comparison trials is the use of a non-naturalistic network to connect the equipment in the trial (e.g. a high speed local area network). For real-time telehealth interactions, it should be noted that the performance of any telehealth equipment is governed by the underlying quality of the network and factors such as the speed of the connection (bandwidth), the delay (latency), the change in the delay over time (jitter) and the stability of the network (dropped packets and congestion) all play a critical role in the quality of the conference. For this reason, an equivalent network to that which will host the telehealth service should be used in method-comparison studies to maximise the validity of the trial.
Defining the comparator
It is important to choose the best available commonly used conventional assessment method as the comparator in a telehealth method-comparison study. Selecting a ‘gold standard’ assessment method, with the best possible psychometric properties, gives the telehealth assessment the best possible chance of reaching equivalence/non-inferiority. If a conventional assessment with poor psychometric properties such as low reliability is selected, the assessment of agreement between the methods is both meaningless and untrustworthy.
Choosing raters
When designing a method-comparison study it is important to carefully chose the number of raters in order to minimise bias and maximise the validity of the study. Although there are a number of factors which may impact on a rater's performance of an assessment (such as training and level of experience), it is generally accepted that a different rater should perform the conventional and telehealth assessment. The bias which is associated with a single rater performing both assessments is difficult, if not impossible, to overcome and often involves the introduction of a lengthy time period between assessment. This time period is used to limit ‘bleed through’ which relates to a rater's ability to remember the initial assessment which subsequently influences the independence of the second assessment. The validity of the trial is strengthened if multiple raters are used within each method, if raters have a varied levels of experience (consider counterbalancing the experience level of raters between the two methods), if raters are randomly allocated to assessment methods (so they have an equal chance of being allocated to either method for each participant) and if the order of assessment is randomised (so there is an equal chance of the conventional or telehealth assessment being performed first).
Number of assessments
The number of participants that are required in a method-comparison trial in order for the results to be robust and meaningful is a key factor to consider when designing the trial. The greater the number of paired observations in the trial, the more likely the sample finding will reflect the population at large and the more confidence that can be placed on the outcome. The number of participants that are required depends on the design of the study (including the number of assessors), the outcome measure that is being compared and level of power (probability of finding a difference in the sample when a difference exists in the population). Sample size considerations and methods for the calculation of sample sizes in method-comparisons are described extensively elsewhere and the reader is referred to these works for discrete data 5 and for continuous data. 6 In preparing your study design, you may need to do some pilot work to establish required information about the study population and the prevalence of the disease or outcome measure to assist in the sample size calculations.
Participant selection
An advantage of a large sample size in method-comparison studies is that the sample will be more representative of the underlying population. Method-comparison studies should include participants with a wide range of possible presentations to ensure that the telehealth assessment has external validity. For example, if the study was to investigate the equivalence of measuring joint angle via telehealth, then the full range of possible joint angles should be represented in the sample. Failure to do so will result in the telehealth tool only being valid for the ranges investigated. Stratified sampling can be a useful strategy to ensure that a diverse sample of participants are recruited.
Reliability of the assessment
The reliability of the new telehealth assessment is an import concept which must be mentioned in the context of method-comparison trials. This is important because if one method has poor reliability (considerable variation in repeated measures), the agreement between the two methods is likely to be poor and the results will be meaningless. For this reason, it is important to establish the reliability of the telehealth assessment either prior to, or as part of the method comparison study.
7
The most straightforward way of establishing reliability is to administer the assessment on two or more occasions via the same method. With consideration given to test-retest bias, this assessment can be conducted by the same rater (intra-rater reliability) or by multiple raters (inter-rater reliability). As previously mentioned, it is important to consider and limit any participant variance between assessments where possible as any real change in the parameter being measured will be reflected in reduced reliability values. Figure 2 presents a common research design which incorporates a reliability element into a method-comparison study.
Methods-comparison study with a reliability component for the telehealth assessment. In this design, the patient is assessed four times by three different raters.
Comparing the results
Performing the appropriate analysis on the paired sets of data that are produced in method-comparison studies is critical to the proper interpretation of the results. It is important to understand that we are interested in seeing if the telehealth method produces results which are sufficiently similar to the conventional assessments. This is different from many conventional research trials which are interested in investigating if an intervention produces sufficiently different results when compared to a control. For this reason, tests of difference, such as t-tests that use a p value of <0.05 to indicate that there is less than a 5% chance that the methods are different by chance, are not appropriate. Rather, we are interested in testing if there is less than 5% chance that the methods do not agree. This is a subtle but important difference. Another common mistake seen in method-comparison studies is the use of correlation measures such as the Pearson product-moment correlation coefficients. While this measure indicates the linear relationship between the two continuous variables of interest, it does not provide information on the magnitude of the difference between the two measurements. For example, consider a mobile Sphygmomanometer that routinely measures blood pressure 20 mm Hg higher than a conventional device. In this situation the two measurements may be highly related in that a change in one is reflected in a change in the other (results in good correlation), but the large magnitude of difference between the devices is not detected with this analysis. This is problematic when evaluating the validity of a measurement tool and therefore correlation coefficients should not be used in isolation. A consideration of the appropriate analysis for different data types generated in method-comparison studies follows, however the reader is directed to an article by Watson and Petrie (2010) for a more comprehensive overview. 8
Continuous and interval outcomes
Continuous outcomes are outcomes that can take any value between a certain set of real numbers. Examples of continuous data include a person’s height or the time taken to complete an assessment item. A seminal article that discusses the appropriate analysis for the measurement of agreement when using two different methods of examination is presented by Bland and Altman. 9 This article reasons that some lack of agreement between the two methods of measurement is inevitable, but if the two measures are sufficiently similar then the difference is unlikely to change clinical interpretation, so the new method could be used in place of the existing method. They propose a ‘limits-of-agreement’ method which calculates a range of differences within which 95% of the differences between the two methods will lie. Once calculated, this range can be compared against what is likely to be clinically relevant for the measurement under examination and decisions about equivalence can be made. Selecting a clinically relevant value is not straight forward however, and should be based on sound clinical or biological factors. One method is to base this value on the minimal clinically important difference (MCID) of the outcome, if available in the literature. The MCID is defined as the smallest change in a treatment outcome that a patient would identify as important. 10 As the assessment needs to be sensitive enough to detect the MCID in practice, it is not unreasonable to set the clinically relevant value to a fraction of the MCID (such as half of the MCID value for example). Regardless of the value selected, it is important that the clinically relevant margin is selected a priori in the study design phase. Practical guides on performing ‘limits-of-agreement’ analyses on clinical data are provided elsewhere. 11
Interval scale variables (with equidistant intervals between two consecutive ratings) such as Celsius temperature, are often considered as continuous data in method-comparison studies. Another statistic that is popular and appropriate for the comparison of both continuous and interval data is the intra-class correlation coefficient (ICC). 12 The ICC is a general method of agreement or consensus which can be used on normally distributed data. While it is a form of correlation, ICC has the advantage of adjusting for the effect of the scale of the measurement and hence it overcomes the issues mentioned above. An example of a study using ICC to compare in-person and telephone-based pill counts is referenced here. 13
Nominal and ordinal outcomes
Agreement between outcomes with these data types is quite commonly reported descriptively using a simple percentage of agreement. For example, a paper by Maa et al. (2014), compared a telehealth protocol for assessing eye disease to conventional assessments and reported that 100% agreement existed for the detection of cataracts, 96% agreement for macular degeneration and 87% for glaucoma. 14 Percentage agreement is easy for clinicians to understand. However, the interpretation of the percentage value is not always straightforward as it does not take into account the agreement that would be expected purely by chance. Cohen’s Kappa statistics on the other hand provide a measure of agreement which is beyond that expected by chance, and value ranges have been reported in the literature to assist in the interpretation of the result. 15 The weighted Kappa statistic is recommended for measuring agreement in ordinal outcomes as it penalises disagreement in terms of the seriousness of the disagreement. For example, a disagreement of one point on a seven-point Likert scale is not penalised to the same extent as a three-point difference on the same scale.
Binary outcome
Agreement in binary outcomes, such as whether the patient has a certain condition or disease or not, can be examined using similar techniques of percentage agreement and Cohen’s Kappa (unweighted) as mentioned above. For diagnostic tests, we can go further to describe the sensitivity and specificity of the telehealth assessment. Sensitivity measures the proportion of true positives that are correctly identified via the telehealth assessment or, put more simply, sensitivity is a measure of how well the telehealth assessment can detect the presence of the condition. Specificity, on the other hand, measures the proportion of true negatives that are correctly identified, or put more simply, how well the telehealth assessment can correctly identify those who do not have the condition. Both specificity and sensitivity analyses, however, have limitations and many advocate for the use of likelihood ratios which overcome many of these limitations. A study by Kim et al., who investigated the effectiveness of tele-mentored ultrasonography between emergency medicine residents and remote experts for the diagnosis of acute appendicitis is a good illustration of the use of these measures in a telehealth context. 16
Conclusions
Method-comparison studies are ideally suited to examine the validity of assessments performed via telehealth. However, careful consideration needs to be given to design elements and data analysis techniques to ensure that high-quality, clinically meaningful scientific evidence results.
Footnotes
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Centre of Research Excellence in Telehealth funded by the National Health and Medical Research Council (NHMRC; grant number APP1061183).