Abstract
How the absence of a physical examination in telehealth visits influences clinical decision-making in telehealth visits is not known. The aim of this study was to examine whether provision of patient self-collected vital signs influences telehealth management decisions for acute respiratory tract infections (ARIs). A consecutive sample of adult patients seeking care for ARIs completed a simulated telehealth visit either before or after their scheduled in-person evaluation. Patients were given instructions and devices to collect their vital signs before the telehealth visit. We compared the telehealth physician diagnostic and treatment recommendations with those made by the in-person treating physician. Forty-five patients completed the study. Compared to the in-person management decisions, telehealth evaluations yielded perfect agreement for when patients needed hospital admission. However, there was significant disagreement around the need for chest imaging and antibiotic treatment, with modest differences following the provision of patient self-collected vital signs.
Introduction
Even before the COVID-19 pandemic, telehealth (i.e., remote interactions with health care providers using communication technology, such as video or telephone) was rapidly emerging as an alternative to traditional, in-person care for acute respiratory tract infections (ARIs), because it offered convenience and efficiency and could help reduce emergency department (ED) overcrowding. Although ARIs are often managed using evidence-based algorithms that frequently incorporate vital sign information, there is little research regarding the validity of using these algorithms in remote patients. We have previously demonstrated that patients can accurately self-collect heart rate and temperature using commonly available home devices such as a digital thermometer and pulse oximeter. 1
Therefore, we conducted a pilot study of patients visiting a hospital-based urgent care clinic (UCC) comparing diagnostic and treatment recommendations made during an in-person versus simulated telehealth visit, the latter providing recommendations both before and after receiving patient self-collected vital sign information. Our primary hypothesis was that the level of agreement between the recommendations from in-person and telehealth evaluations would be high and would be further enhanced by incorporating patient self-collected vital signs into the telehealth evaluation.
Methods
This pilot study was an analysis of agreement rates between in-person evaluations and simulated telehealth evaluations of a consecutive sample of patients with ARIs.
STUDY SITES
The study was conducted in an UCC within the Massachusetts General Hospital, which has approximately 3,500 in-person visits annually.
STUDY SUBJECTS
Eligible participants included adults, age ≥20 years, scheduled for an urgent evaluation with a chief complaint of less than 2 weeks of an illness with cough as a new or worsening feature. Urgent evaluations were conducted by staff physicians and nurse practitioners at the clinic. Exclusion criteria included any immunocompromising condition (including current use of immunosuppressing medications), previous evaluation (in-person or telehealth) for the same illness, and non-English language.
ENROLLMENT AND DATA COLLECTION
With prior approval from clinic staff, a trained research coordinator approached a consecutive sample of patients at the time of their scheduled urgent visit and obtained verbal consent for study participation. Depending on the clinic schedule, participants could elect to complete the study before or after the scheduled in-person evaluation. Participants were given an electronic home thermometer (Vicks SpeedRead V912US Digital Thermometer®) and a home pulse oximeter (Zacurate ProSeries 500DL Fingertip Pulse Oximeter®) along with a simple instruction guide for how to collect heart rate (via pulse oximeter), oral temperature, and oxygen saturation (via pulse oximeter). This instruction guide was used in a prior study assessing the accuracy of patient-measured vital signs. 1 After reviewing the instruction sheet, participants collected their vital signs and manually entered the values onto an electronic data form (REDCap) on a tablet.
Immediately after each participant collected and entered their vital signs, the RC initiated a simulated telehealth visit using a tablet PC with a secure intranet connection using the Zoom platform (Zoom Video Communications, San Jose, CA). The telehealth visit was conducted by emergency medicine physicians (N = 4), none of whom worked in the UCC but worked in the same hospital ED. These telehealth physicians did not have access to the patient’s chart in the electronic health record and were not able to review prior records, add new documentation, or place orders.
The telehealth physicians were not given a specific script to follow for the visit. Rather, they were asked to conduct their standard evaluation for patients with acute cough illness. At the start of the visit, the telehealth physicians confirmed that this was a simulated visit, no actual treatment decisions would be provided to the patient, and the results of the visit would not be shared with the treatment team at the clinic.
At the conclusion of the telehealth visit, the telehealth physicians entered their diagnostic and treatment recommendations through an online survey on REDCap, which used a 5-point Likert scale to assess their agreement with a series of management decisions including: (1) The patient needs to be admitted to the hospital; (2) The patient needs a chest X-ray; and (3) The patient needs antibiotic treatment. The telehealth physician also entered a diagnosis through a text field. These same data were collected twice: before and after the telehealth physician was provided the results of the patient self-collected vital signs.
The research coordinator also reviewed the medical record of each participant to collect the management decisions of the in-person clinician including: (1) Whether the patient was triaged to the ED for hospital admission; (2) Whether a chest X-ray was ordered; and (3) Whether an antibiotic was prescribed. The visit diagnosis was also abstracted from the record.
DATA ANALYSIS
Diagnoses entered by both the telehealth and in-person clinicians were independently coded into a limited number of categories (e.g., acute upper respiratory infection, pneumonia, sinusitis), and an overall agreement rate was calculated. For each of the three management decisions (hospital admission, chest imaging, and antibiotic treatment), we calculated the agreement between the in-person evaluation and the telehealth evaluation separately for both positive and negative decisions as well as the overall agreement combining both positive and negative decisions. We also calculated kappa statistics to summarize the agreement beyond chance. We made these calculations for the telehealth recommendations both before and after they were provided with patient self-collected vital signs. For all telehealth recommendations, we categorized strongly agree and agree recommendations as a positive recommendation and neutral, disagree, and strongly disagree as negative recommendations. A sensitivity analysis that looked at strongly agree responses alone showed similar results. For all measures, we calculated 95% confidence limits.
We calculated a preliminary sample size of 50 study subjects to determine an overall agreement rate with a 95% confidence interval of ±15%, assuming an estimated agreement rate of 84%.
The study was approved by the Mass General Brigham Institutional Review Board.
Results
From December 2023 through March 2024, we enrolled 45 subjects from the UCC. The average age was 62 years. A majority (69%) were female patients, 73% were non-Hispanic White, 16% were non-Hispanic Black, and 4% were Hispanic or Latino. Most telehealth evaluations (87%) were conducted after the in-person evaluation. All patients who consented to participate in the study completed both visits. In general, the telehealth evaluations that occurred after the in-person evaluation yielded similar results compared to the telehealth evaluations that occurred before the in-person evaluation (data not shown).
The most common diagnoses documented after the in-person evaluation were viral upper respiratory infection/upper respiratory infection (67%), followed by pneumonia (11%) and sinusitis (4%). There was a 78% agreement between the in-person and telehealth evaluations on diagnosis, which was the same both before and after sharing patient self-collected vital signs for the telehealth evaluation.
Table 1 summarizes the agreement between in-person versus telehealth management decisions, the latter both before and after receiving data on self-collected vital signs. None of the patients were recommended for hospitalization by either the in-person or telehealth evaluations (100% agreement). Thirty-eight percent of the patients (95% CI: 24 − 53%) received a chest X-ray. Among the patients who had a chest X-ray ordered by the in-person clinician, 53% (95% CI: 28 − 77%) also had a chest X-ray recommended by the telehealth clinician. After vital signs were provided for the telehealth evaluation, the percentage of these patients for whom the telehealth provider recommended a chest X-ray rose to 76% (95% CI: 50 − 93%). Among the patients who did not have a chest X-ray ordered by the in-person clinician, 71% (95% CI: 51 − 87%) did not have a recommendation for a chest X-ray by the telehealth clinician. This percentage fell to 64% (95% CI: 44 − 81%) after vital sign information was provided for the telehealth evaluation. Overall agreement between in-person and telehealth clinicians was poor, with kappa statistics ranging from 0.24 to 0.38 before and after provision of vital signs, respectively.
Agreement Between In-Person and Telehealth Recommendations for Patients with ARIs
Kappa cannot be calculated when 3 of 4 cells are zeros.
Overall concordance = number of visits where both in-person and telehealth clinician recommended action + number of visits where both in-person and telehealth clinician did not recommend action divided by all visits.
Twenty-nine percent of the patients (95% CI: 16 − 44%) were prescribed an antibiotic by the in-person clinician. The telehealth clinician did not indicate they would prescribe antibiotics to any of the patients, and these recommendations did not change with the provision of patient self-collected vital signs.
Discussion
This pilot study demonstrated that in-person versus telehealth evaluations of patients with ARIs yield different management decisions in some areas. In this relatively small sample, the paucity of patients with severe symptoms meant that none of the patients required hospitalization, and the telehealth evaluations were in 100% agreement with this management decision. In contrast, chest imaging and antibiotic prescriptions were ordered by the in-person clinician for a significant number of patients, and the evaluations by the telehealth clinicians yielded different recommendations. For example, even though antibiotics were prescribed for 29% of patients after the in-person evaluation, none of these patients were recommended for antibiotics based on the telehealth evaluation, even after self-collected vital sign data were provided. Of the 38% of visits where a chest X-ray was ordered by the in-person clinician, only about half of the time did the telehealth clinician recommend chest imaging, though this increased to 77% of those visits after self-collected vital sign data were provided to the telehealth clinician, suggesting that access to vital signs during a telehealth/remote visit influences clinical decision-making for test ordering.
In this study, we treated the in-person visit as a gold standard to evaluate the quality of the telehealth evaluations. However, the in-person evaluation is an imperfect gold standard. Specifically, different clinicians with different clinical training backgrounds conducted the in-person versus telehealth visits, which could have impacted the differences in recommendations. Moreover, the telehealth clinicians were aware that the visit was for research purposes only and could have felt less pressure to recommend further interventions, such as ordering antibiotics, because there would be no actual impact on patient outcomes. The telehealth clinicians did not have access to the electronic medical record of the patients, which could have also influenced decision-making by the in-person clinicians. We recognize these limitations as part of the pilot nature of the work.
Prior research has demonstrated that telehealth visits for ARIs are not more likely to result in antibiotic prescriptions for patients with acute respiratory infections compared to in-person visits, though these comparisons are for distinct populations of patients.2–4 Our results contribute to this literature by suggesting that within the same patient population, telehealth versus in-person evaluations yield different management decisions.
We have previously shown that vital sign data incorporated into decision support tools can improve the quality of management decisions for patients with ARIs, 5 and we have demonstrated that patients can accurately self-collect some of these measures to support telehealth evaluations. 1 However, in this study, the provision of patient self-collected vital sign data did not substantially improve agreement between in-person and telehealth evaluations. More chest radiographs were recommended to be ordered when vital sign data were provided, but that increase was for patients who both did and did not receive chest imaging at the in-person evaluation. Overall, this study provides limited evidence that having patients self-collect their vital sign data could significantly improve medical decision-making during telehealth evaluations.
There are limitations to our study. This study was performed at a single site, and other urgent care or virtual urgent care programs may have different available resources. Second, we do not have access to the baseline prescribing patterns of the in-person clinicians or telehealth physicians. It is possible that the in-person clinicians in this study tend to prescribe antibiotics for ARI encounters more frequently than the study telehealth physicians, especially since the two groups of participants had different clinical training backgrounds.
In summary, in this preliminary study, simulated telehealth evaluations of patients with ARIs yielded perfect agreement regarding the need for hospitalization but poor agreement regarding chest X-ray and antibiotic management compared to actual in-person evaluations. The addition of patient-measured vital signs for the telehealth physicians did not significantly increase the agreement between the in-person and telehealth clinicians. Whether similar differences would persist in comparing actual decisions in both settings and whether one mode of evaluation yields an overall higher quality of decisions and patient outcomes will be the focus of future investigations.
Authors’ Contributions
J.P.M.: Funding acquisition; methodology; project administration; supervision; writing (original draft). R.G. and T.J.J.: Methodology; supervision; writing (review and editing). P.B., S.D., S.W.L., B.A.W., and S.O.: Data curation; writing (review and editing). Y.C.: Formal analysis; methodology; writing (review and editing). B.A.P.: Methodology; project administration; supervision; writing (review and editing); M.D.T.: Data curation; project administration; writing (review and editing). E.M.H.: Funding acquisition; methodology; project administration; writing (review and editing).
Footnotes
Acknowledgments
Dr. Metlay had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Author Disclosure Statement
No competing financial interests exist.
Funding Information
This project was supported by a grant from the CRICO Risk Management Foundation. The funder had no role in the design and conduct of the study, analysis of data, and creation and final approval of the article.