Five analytic challenges in working with electronic health records data to support clinical trials with some solutions

Abstract

Electronic health records data are becoming a key data resource in clinical research. Owing to issues of data efficiency, electronic health records data are being used for clinical trials. This includes both large-scale pragmatic trails and smaller—more focused—point-of-care trials. While electronic health records data open up a number of scientific opportunities, they also present a number of analytic challenges. This article discusses five particular challenges related to organizing electronic health records data for analytic purposes. These are as follows: (1) data are not organized for research purposes, (2) data are both densely and irregularly observed, (3) we don’t have all data elements we may want or need, (4) data are both cross-sectional and longitudinal, and (5) data may be informatively observed. While laying out these challenges, the article notes how many of these challenges can be addressed by careful and thoughtful study design as well as by integration of clinicians and informaticians into the analytic team.

Keywords

Electronic health records selection bias study design

Electronic health records (EHR) data are a ubiquitous source of real-world data used for clinical research. They are currently being used for studies ranging from clinical risk prediction¹ to comparative effectiveness² to population health.³ The appeals of EHR data are numerous. For a single health system, they consist of readily available data on a potentially large number of patients with clinical variables captured over a variety of fields. Finally, as a naturally collected data source, they are relatively cheap to use for research and evaluation purposes.

As the 2019 Conference on Statistical Issues in Clinical Trials highlighted, EHR data are increasingly being used for clinical trials. This includes both small single health system–based trials⁴ and large pragmatic multisite trials.⁵ A particular appeal, with regard to clinical trials, is that unlike some clinical trial samples, the source population from an EHR is highly representative of an identifiable patient population.⁶ There are a number of ways that EHR data can be used to support various stages of clinical trials (Table 1). At the outset, EHR data can be used to inform the design of clinical trials by helping to select study end points or ideal study population.⁷ The second is through participant recruitment where EHR systems can be used to detect patients eligible for point-of-care studies.⁸ Another is through the use of pragmatic trials.⁹ In these studies, the trialist relies on the naturally collected EHR data to adjudicate patient characteristics and outcomes¹⁰ as opposed to purposefully collected clinical research forms. The fourth is through the translation of clinical trials results to clinically meaningful target populations.¹¹ Finally, EHR data can be used for adverse event reporting and post-market surveillance.¹²

Table 1.

Uses of EHR data to support clinical trials.

Trial support	EHR data needs
Trial design	Retrospective assessment of target population
Trial recruitment	Identification of patient population
Abstraction of clinical end points (pragmatic trials)	Automatic capture of relevant data elements
Trial results generalization	Patient population to use treatment
Post-trial surveillance	Monitoring of patients receiving treatment of interest

EHR: electronic health records.

Ultimately, EHR data allow researchers to design cheaper and more efficient studies.¹³ While the appeals for using EHR data to support clinical trails are numerous, because the data are not collected for research purposes–but to support clinical care and billing–there are important considerations that statisticians and analysts need to consider when working with EHR data. This article discusses five particular challenges analysts may encounter in organizing EHR data for analytic purposes–both for prospective and retrospective studies–and some design considerations that one can employ to address these challenges (Table 2).

Table 2.

Data challenges and design solutions.

Data challenge	Design solution
Data are not organized for research purposes	Use of data models and DataMarts
Data may be measured repeatedly, infrequently, or not at all	Flexible analytic models
We don’t have everything we want	Defining regular patients
Data are both longitudinal and cross-sectional	Definitions of study eligibility
The information collected is informative	Design with patient selection in mind

Challenge 1: data are not organized for research purposes

One of the biggest challenges with EHR data is structural: they are not organized for research purposes. In their rawest form, EHR data are typically stored in large relational databases that can have thousands of tables. For example, EPIC,¹⁴ the largest EHR vendor for academic medical centers, has in its base structure, Chronicles, approximately 95,000 data elements. These data are stored nightly into a data warehouse, Clarity, which contains ∼12,000 tables consisting of ∼125,000 columns. Even the simplified version of the EPIC data structure, Caboodle, contains 19 tables and 76 dimensions.¹⁵ This complicated form is designed to maintain the granularity of the data. As such, it is often impossible for statisticians to work with the source data. Instead, one typically needs to partner with an informaticist that has a deep understanding of the back-end system. Once the needed elements are identified—typically in conjunction with the clinical team—the informaticist can write code that pulls the required data.

Once the appropriate data elements are pulled, one still has to do significant data processing. In most analyses, it is necessary to determine which conditions a patient has—for example to indicate study eligibility. However, within the EHR, there is not a single field that indicates a patient’s conditions. While there are designations for “problem lists,” these are often not very reliable.¹⁶ Instead, it is necessary to use the observed data to construct computable phenotypes. Computable phenotypes are Boolean definitions—usually based on a combination of diagnosis codes, testing results, and/or medications—that are used to indicate whether a patient has the condition of interest. For example, a computable phenotype for diabetes may include the presence of a diabetes diagnosis code OR the presence of diabetes medication OR a hemoglobin A1c test > 6.5%.¹⁷ The type of definition used will impact the sensitivity and specificity of the final phenotype.¹⁸ While in the author’s experience the specific phenotype itself does not typically alter model inference, if there is differential amounts of information known about different patients, bias can be induced.¹⁹ Given the number of potential phenotype definitions, it is not generally feasible to compare inference when all different definitions are used. However, it may be advisable to perform sensitivity analyses for a few different versions of the primary outcome and/or exposure. Often times one wants a more specific definition when it comes to exposure or outcome assessment while a more sensitive definition can be used for comorbidity assessment. Overall, the research team a priori defines the primary phenotype definition to be used throughout the analysis.

Solution: use of data models and DataMarts

There are many decisions that need to be made when setting up analytic data. One way that this is often resolved is through the use of Common Data Models (CDMs). CDMs are a set of “rules” that indicate which data elements need to be extracted from the health record and how they ought to be handled. Typical CDMs include the PCORnet CDM²⁰ and the Observational Medical Outcomes Partnership CDM.²¹ While CDMs can be used in a variety of research settings, they provide the greatest benefits when working across multiple centers. Since different sites will often have different and unique back-end EHR systems, a CDM provides a uniform set of standards for how data need to be handled.

The data model can also help define how medications and diagnosis codes may be grouped together. Many data elements are represented in overly granular form. For example, it may not be necessary to know precisely the type of antibiotic a patient received, just simply that they received an antibiotic. Ontologies such as RxNorm and Clinical Classification Software provide a hierarchical system to allow research to “roll” the data elements to the proper level of granularity.

Once the relevant data elements are defined, it is often helpful to organize the data into research DataMarts-focused relational databases. DataMarts can be useful for organizing retrospective data as well as prospectively to track patients enrolled in a study. One particular appeal is that depending on the research environment, the DataMarts can be designed to automatically refresh as patients have more encounters with the EHR system.

Challenge 2: data may be measured repeatedly, infrequently, or not at all

One of the appeals of EHR data is the ability to view data over time. However, this also means that it is typical for many data elements to be observed multiple times, infrequently, or not at all. For example, depending on the length and type of follow-up, it is typical for an individual to have multiple vital signs measurements over time. Conversely, for laboratory values, it is typical for some patients to have multiple measurements while others have none. Moreover, the amount of data a patient has is typically informative of a patient’s health status.²² Therefore, it is not reasonable to assume the uncollected measurements are missing-at-random,²³ that is, missingness is conditional on observed covariates. As such typical strategies to resolve missingness—like multiple imputation—may not be valid.²⁴

Solution: flexible analytic models

Depending on the data structure and the particular question, different analytic designs may be necessary. For a study with multiple repeated exposures (e.g. blood pressure) and single time-to-event outcome (e.g. heart disease), a time-varying covariate Cox Model²⁵ would be appropriate. For questions with a repeated outcome, a longitudinal mixed model may be used.²⁶ When an exposure variable is measured repeatedly, it is unclear whether it is best to model the variable trajectory (e.g. via joint models),²⁷ simply the most recent measurement or a summary measurement (e.g. median, min/max). Work within the context of predictive models suggest simpler summarizations do just as well as more complex models,^28,29 though more work is needed within the inferential context.

When data are irregularly collected, and the collection mechanism is informative, the absence of the measurement needs to be directly accounted for in the analytic model. This can be achieved by including a count for the number of times a laboratory is measured or by generating a categorical variable that contains levels: “normal,”“high,”“low,” and “not-collected.” All else being equal, such categorization is less efficient than using the continuous variable and can lead to loss of information.³⁰ However, such categorization captures the informativeness of the collected data while retaining the non-linear relationship that many clinical variables have on outcomes, presenting a worthwhile trade-off.

Challenge 3: we don’t have everything we want

EHR data are appealing because of the depth of information they provide on a patient. However, it is important to be cognizant of which data elements are not well captured. Köpcke et al.³¹ noted how many of the components needed to run a clinical trial—particularly data around health status and consent—are not contained within EHRs. Similarly, the general analytic assumption is that the absence of a data element (e.g. a medication prescription, diagnosis code) implies the absence of the event.²⁴ However, this is not always the case—particularly for patients who make seek care at multiple facilities.³² A particular weakness in most EHR systems are indicators of social health: socio-economic status, alcohol use, nutrition, and patient-reported outcomes. Depending on the study, these data elements may need to be collected separately or captured via proxy variables. For example, a proxy for individual socio-economic status can be captured based on knowledge of one’s address and characterizing the socio-economic status of their neighborhood.³³ Moreover, it may be desirable to link EHR data with external data such as environmental and geospatial data.³⁴

A more challenging problem is clinical information that are not captured. Since EHR data are derived from the perspective of the health system (i.e. provider), encounters outside the health system will not be observed. This is one of the biggest differences between EHR data and administrative/billing data. Administrative data, which are collected by the payer, typically follows a patient across different providers. The lack of comprehensive patient encounters is a particular challenge if the health system from which data are derived is primarily a referral center or is in a locale with competition from other systems. This means that a patient may have encounters—indicative of either a changing exposure or outcome—that is not captured in the EHR being analyzed. Moreover, there is typically no indication that these encounters may have occurred—they are truly unobserved.

Solution: defining regular patients

The primary solution to this is to design one’s study to include patients that are regular users of the health system of interest. It is important to note that this will change the target population from which inference is derived. Moreover, it won’t resolve all missed encounter. However, reframing the analytic question around a well-defined patient population will increase the overall capture, on a per-person basis. In practice, for retrospective analyses, this typically requires including “burn-in” periods where a patient is required to have a certain number of visits to the health system before they are eligible for study. It is also possible to use the granular data to require a patient has, for example, a primary care provider within the health system. A more complete, albeit more challenging, solution is to link administrative data with EHR data.³⁵ In addition, linking in external data sources can help mitigate some missing data challenges.²⁴ Ultimately, it is important to assess the quality and fit-for-use of the data being used.³⁶ Sometimes it is important to acknowledge that the data one has is not well suited for answering the desired questions.

Challenge 4: data are both longitudinal and cross-sectional

A challenging aspect of working with EHR data is that the data share characteristics of both cross-sectional and longitudinal data.

EHR data as cross-sectional data

Typically, when working with EHR data one has a limited and often arbitrary temporal selection of data. The beginning date is often based on when the medical system switched on to an integrated EHR platform, while the end date will be based on the most recently available data. This means that the analyst often does not have a complete history or follow-up on patients. This is particularly challenging if she or he wants to know how long a patient has had a given disease to define study eligibility.

EHR data as longitudinal data

At the same time, as discussed above, patients are observed at multiple points over time. This allows one to perform time-to-event and trajectory-based analyses. However, this longitudinal cohort is also dynamic. Patients can enter and leave the catchment of the health system that underlies the record system. This makes defining exposure history challenging because it is not always obvious if a patient lacks an encounter because she or he had no health problems or just recently moved into the catchment.

Solution: definitions of study eligibility

As above, the development of eligibility rules, via burn-in periods, is important. If we want to define the incidence of disease or medication usage, we may want to require a 2-year history where there is no indication of that disease or medication. Borrowing from traditional cohort studies, nested case–control designs become very useful.³⁷ Here, one can identify people when they have an outcome of interest and then sample a control who had not developed the outcome yet. Another approach is to identify an arbitrary landmark time point (e.g. 1 January of a given year). This cross-sectional point creates a notion of a uniform time 0 as well as a common window across which to define patients’ clinical history. If there is a particular intervention point from which to evaluate, one can also employ “new-user” designs.³⁸ Employing traditional case–control designs, while occasionally appealing, is more challenging as it can induce selection bias if controls are not properly defined.³⁹

Challenge 5: the information collected is informative

While the previous challenge concerns data we do not see, we also have to be wary of data we do observe. Since patients are engaging with the health system when they are sick, and doctors are taking measurements on factors for which they are concerned, most measurements can be considered informative. This has been termed “informed-presence.” Related to data Missing Not At Random, informed presence emphasizes the fact that what we observe is informative. Informed presence of data is best expressed by the notion that, with EHR data, we have more information about sicker patients.²² As such it is important to always consider why certain data elements are being observed. There are a few ways that informed presence can manifest itself:

It is easier to generate computable phenotypes for sicker patients.⁴⁰ The fact that we have more data on sicker patients is in itself not surprising. However, what this means is that we also have non-differential misclassification where it is easier to phenotype sicker patients.

The type of care a person seeks is informative. The same data collected on the same person in the emergency department is often significantly different than when it’s collected in the outpatient setting.³²

The type of facility a patient seeks care at is informative.³² Health systems often consist of a mixture of primary care, community, and tertiary care hospitals. An individual may use a mixture of these facilities. While these encounters will all be captured within the health record, it is important to be aware that the types of services a patient receives at each facility will differ. For example, one is more likely to have the data elements to phenotype a patient for potential cancer at a tertiary care center than a community center.

Non-local or referral patients are often qualitatively different from local patients.³² Patients that seek care out of their typical locale often have more severe and complicated disease than community patients. Recognizing this potential admixture is important in cohort definitions.

If patients seek care only when they are sick, their biomarker values may be elevated.⁴¹ This can generate feedback loops that induce associations.

The presence of these informative data, which generally can be thought of as a selection process,⁴² can impact study inference.

Solution: design with patient selection in mind

First and foremost, it is important to understand the process dictating how patient data are observed and selected into the analytic dataset. This is where clinical partners are most useful since they have a deeper understanding of why patients may or may not have a given encounter and why some information may or may not be collected. While we cannot avoid all aspects of informative selection, many of these effects can be mitigated by careful design. For example, including patients that are local ensures that there is no bias due to admixture. Similarly, mainly using data from outpatient settings avoids the adverse selection effects of being in the emergency department. In a related way, primarily using outpatient data increases the likelihood that some encounters are pre-scheduled and therefore not driven by a patient’s underlying health status. Incorporating such data into the analytic model can mitigate overall bias.⁴¹ Additionally, since many forms of informed presence are driven by the amount of information we have on a patient, analytically conditioning on the number of encounter—that is., the amount of information we have on a patient—can also mitigate the biasing effects of the differential misclassification.^40,41 Finally, work has shown that mixed models tend to perform well in these settings as the more important slope parameters are less prone to bias.^43,44 When one wants to directly model the informative visit process, more sophisticated joint models have been proposed.^45,46

Summary

EHR data have become an important resource for clinical research and are increasingly finding their way to support clinical trials. Not only do they provide cheap access to granular patient data, they also ensure that a truly relevant clinical population is being observed. However, since EHR systems were not designed for research purposes, they present many analytic challenges including setting up the data, accounting for what data were not collected, handling of temporality, and resolving informative visit processes. Thoughtful study design can mitigate many of these challenges. As discussed, fully understanding these challenges requires insight into the analytic design, the source of the data, and the nature of the clinical encounter. As such, resolving these challenges requires the joint contribution of statisticians, informaticians, and clinicians.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Benjamin A Goldstein

References

Goldstein

Navar

Pencina

, et al. Opportunities and challenges in developing risk prediction models with electronic health records data: a systematic review. J Am Med Inform Assoc 2017; 24(1): 198–208.

Setji

Page

Pagidipati

, et al. Differences in achieving HBA1C goals among patients seen by endocrinologists and primary care providers. Endocr Pract 2019; 25(5): 461–469.

Casey

Schwartz

Stewart

, et al. Using electronic health records for population health research: a review of methods and applications. Annu Rev Public Health 2016; 37: 61–81.

Hooper

Weavind

Wheeler

, et al. Randomized trial of automated, electronic monitoring to facilitate early detection of sepsis in the intensive care unit. Crit Care Med 2012; 40(7): 2096–2101.

Johnston

Jones

Hernandez

. The ADAPTABLE trial and aspirin dosing in secondary prevention for patients with coronary artery disease. Curr Cardiol Rep 2016; 18(8): 81.

Weng

Ryan

, et al. A distribution-based method for assessing the differences between clinical trial target populations and patient populations in electronic health records. Appl Clin Inform 2014; 5(2): 463–479.

Weng

. Optimizing clinical research participant selection with informatics. Trends Pharmacol Sci 2015; 36(11): 706–709.

Lai

Afseth

. A review of the impact of utilising electronic medical records for clinical research recruitment. Clin Trials 2019; 16(2): 194–203.

Thorpe

Zwarenstein

Oxman

, et al. A pragmatic-explanatory continuum indicator summary (PRECIS): a tool to help trial designers. J Clin Epidemiol 2009; 62(5): 464–475.

10.

Bruland

McGilchrist

Zapletal

, et al. Common data elements for secondary use of electronic health record data for clinical trial execution and serious adverse event reporting. BMC Med Res Methodol 2016; 16(1): 159.

11.

Goldstein

Phelan

Pagidipati

, et al. An outcome model approach to transporting a randomized controlled trial results to a target population. J Am Med Inform Assoc 2019; 26(5): 429–437.

12.

de Hoon

SEM

Hek

van Dijk

, et al. Adverse events recording in electronic health record systems in primary care. BMC Med Inform Decis Mak 2017; 17(1): 163.

13.

Mc Cord

Ewald

Ladanie

, et al. Current use and costs of electronic health records for clinical trial research: a descriptive study. CMAJ Open 2019; 7(1): E23–E32.

14.

EPIC. Verona, WI, https://www.epic.com/ (accessed 18 February 2020).

15.

Epic Date Model, http://healthitmhealth.com/tag/epic-data-model/ (accessed 11 February 2020).

16.

Wright

McCoy

Hickman

, et al. Problem list completeness in electronic health records: a multi-site study and assessment of success factors. Int J Med Inform 2015; 84(10): 784–790.

17.

Richesson

Rusincovitch

Wixted

, et al. A comparison of phenotype definitions for diabetes mellitus. J Am Med Inform Assoc 2013; 20(e2): e319–e326.

18.

Spratt

Pereira

Granger

, et al. Assessing electronic health record phenotypes against gold-standard diagnostic criteria for diabetes mellitus. J Am Med Inform Assoc 2017; 24(e1): e121–e128.

19.

Chen

Wang

Chubak

, et al. Inflation of type I error rates due to differential misclassification in EHR-derived outcomes: empirical illustration using breast cancer recurrence. Pharmacoepidemiol Drug Saf 2019; 28(2): 264–268.

20.

PCORnet Common Data Model (CDM) Specification), https://pcornet.org/wp-content/uploads/2019/09/PCORnet-Common-Data-Model-v51-2019_09_12.pdf (accessed 11 February 2020).

21.

Observational Medical Outcomes Partnership (OMOP) Common Data Mode, http://omop.org (accessed 11 February 2020).

22.

Weiskopf

Rusanov

Weng

. Sick patients have more data: the non-random completeness of electronic health records. AMIA Annu Symp Proc 2013; 2013: 1472–1477.

23.

Little

RJA

Rubin

. Statistical analysis with missing data. 2nd ed. Hoboken, NJ: John Wiley & Sons, 2002.

24.

Wells

Chagin

Nowacki

, et al. Strategies for handling missing data in electronic health record derived data. EGEMS Wash DC 2013; 1(3): 1035.

25.

Therneau

Grambsch

. Modeling survival data: extending the Cox model. New York: Springer, 2000.

26.

Fitzmaurice

Laird

Ware

. Applied longitudinal analysis (Wiley Series in Probability and Statistics). 2nd ed. Hoboken, NJ: Wiley, 2011, p. 701.

27.

Rizopoulos

. Joint models for longitudinal and time-to-event data: with applications in R (Chapman & Hall/CRC Biostatistics Series). Boca Raton: CRC Press, 2012, p. 261.

28.

Goldstein

Pomann

Winkelmayer

, et al. A comparison of risk prediction methods using repeated observations: an application to electronic health records for hemodialysis. Stat Med 2017; 36(17): 2750–2763.

29.

Sweeting

Barrett

Thompson

, et al. The use of repeated blood pressure measures for cardiovascular risk prediction: a comparison of statistical models in the ARIC study. Stat Med 2017; 36(28): 4514–4528.

30.

Collins

Ogundimu

Cook

, et al. Quantifying the impact of different approaches for handling continuous predictors on the performance of a prognostic model. Stat Med 2016; 35(23): 4124–4135.

31.

Köpcke

Trinczek

Majeed

, et al. Evaluation of data completeness in the electronic health record for the purpose of patient recruitment into clinical trials: a retrospective analysis of element presence. BMC Med Inform Decis Mak 2013; 13: 37.

32.

Phelan

Bhavsar

Goldstein

. Illustrating informed presence bias in electronic health records data: how patient interactions with a health system can impact inference. EGEMS Wash DC 2017; 5(1): 22.

33.

Bhavsar

Gao

Phelan

, et al. Value of neighborhood socioeconomic status in predicting risk of outcomes in studies that use electronic health record data. JAMA Netw Open 2018; 1(5): e182716.

34.

Xie

Greenblatt

Levy

, et al. Enhancing electronic health record data with geospatial information. AMIA Jt Summits Transl Sci Proc 2017; 2017: 123–132.

35.

Mitiku

Ivers

, et al. Evaluation of electronic medical record administrative data linked database (EMRALD). Am J Manag Care 2014; 20(1): e15–e21.

36.

Weiskopf

Bakken

Hripcsak

, et al. A data quality assessment guideline for electronic health record data reuse. EGEMS Wash DC 2017; 5(1): 14.

37.

Goldstein

Assimes

Winkelmayer

, et al. Detecting clinically meaningful biomarkers with repeated measurements: an illustration with electronic health records. Biometrics 2015; 71(2): 478–486.

38.

Ray

. Evaluating medication effects outside of clinical trials: new-user designs. Am J Epidemiol 2003; 158(9): 915–920.

39.

Schuemie

Ryan

Man

KKC

, et al. A plea to stop using the case-control design in retrospective database studies. Stat Med 2019; 38(22): 4199–4208.

40.

Goldstein

Bhavsar

Phelan

, et al. Controlling for informed presence bias due to the number of health encounters in an electronic health record. Am J Epidemiol 2016; 184(11): 847–855.

41.

Goldstein

Phelan

Pagidipati

, et al. How and when informative visit processes can bias inference when using electronic health records data for clinical research. J Am Med Inform Assoc 2019; 26(12): 1609–1617.

42.

Haneuse

Daniels

. A general framework for considering selection bias in EHR-based studies: what data are observed and why? EGEMS Wash DC 2016; 4(1): 1203.

43.

Neuhaus

McCulloch

Boylan

. Analysis of longitudinal data from outcome-dependent visit processes: failure of proposed methods in realistic settings and potential improvements. Stat Med 2018; 37(29): 4457–4471.

44.

McCulloch

Neuhaus

Olin

. Biased and unbiased estimation in longitudinal studies with informative visit processes. Biometrics 2016; 72(4): 1315–1324.

45.

Cook

Lawless

. Independence conditions and the analysis of life history studies with intermittent observation. Biostatistics 2019; 2019: kxz047.

46.

Gasparini

Abrams

Barrett

, et al. Mixed-effects models for health care longitudinal data with an informative visiting process: a Monte Carlo simulation study. Stat Neerl 2019; 74(1): 5–23.