Abstract
The wide adoption of electronic health records (EHRs) since 2015, in combination with the regulatory requirements for data sharing, has created an environment to support the creation of population-scale clinical databases. Historically, many large health care databases were restricted primarily to billing data, such as the MEDPAR database from Medicare Part A claims. More recently, databases using clinical data from EHRs such as Epic’s Cosmos, PCORnet, TriNetX, and others combine billing and clinical data to create clinically rich datasets useful in observational research, such as cohorts and case–control studies. These databases can include diagnoses, medications, encounters, immunizations, laboratory results, procedures, vital signs, social determinants of health, patient histories, and even genetic data. The size of these databases now offers unique opportunities to study associations across many diseases that can help shed light on possible disease pathophysiology and potential treatments.
It is important, however, to understand the benefits and risks of doing research using these large datasets. One of the first areas of concern is that the size of these datasets allows for the detection of statistically significant but clinically insignificant differences. It is crucial, then, to establish a threshold of clinical significance, also called a smallest effect size of interest (SESOI), instead of just looking at a P value, which quantifies how compatible the observed data are with the null hypothesis. In his 1988 book, Jacob Cohen proposed conventional benchmarks for effect sizes, with a Cohen’s d of 0.2 generally considered a small effect. 1 Although clinical significance depends on the specific clinical context, a Cohen’s d of 0.2 is often used as a reasonable starting point when defining a SESOI. Assuming 80% power and a two-sided α of 0.05, a Cohen’s d of 0.2 can be detected with approximately 400 subjects per comparison group. Figure 1 shows the relationship between sample size and Cohen’s d, odds ratios, and risk ratios (for rare outcomes). Given the scale of databases such as Cosmos, even rare diseases and uncommon exposures may become amenable to adequately powered observational analyses.
Database size and smallest detectable effect size. Assumes baseline risk of 1% in the unexposed group. Calculations are based on normal approximation for continuous outcomes and standard formulas for binary outcomes. Actual power may vary based on study design, outcome prevalence, effect size definition, and analytical approach. Power calculations performed using standard formulas.1,2
A key to responsibly using these datasets for observational research is to understand potential biases and confounders in the data. Because these data are collected primarily for clinical rather than research purposes, they may contain biases related to documentation practices, health care utilization, coding variation, payor sources, and incomplete follow-up. Temporal factors are also important to look at, as data completeness may be lower in earlier periods surrounding initial EHR implementation. Longitudinal completeness is highly dependent on ongoing engagement with contributing health care systems and should be assessed through approaches such as minimum encounter thresholds, as patients with limited follow-up may have substantially incomplete clinical data. Both Cosmos and TriNetX offer some information, although transparency and standardization of completeness metrics vary across platforms. A listing of advantages and limitations of using large clinical datasets is shown in Table 1.
Benefits and Limitations of Using Large Clinical Datasets for Research
As with any observational research, confounders can strongly influence results. Identification of confounders, colliders, and mediators is crucial. Using tools such as a directed acyclic graph (DAG) can help identify which variables should be included in the regression modeling and which should be adjusted for, excluded, or modeled appropriately. Even after multivariable regression, residual bias may remain because treatment assignment is not random. One approach to reducing measured confounding is the use of propensity-score methods. In large observational datasets, inverse probability of treatment weighting (IPTW) is particularly valuable because it can improve comparability between exposure groups while preserving the full study population. The effectiveness of IPTW is commonly assessed using standardized mean differences (SMDs) of measured covariates before and after weighting. SMD values below 0.05 generally indicate excellent balance between groups for the variables included in the propensity model, increasing confidence that observed differences in outcomes are less likely to be attributable to measured confounding. However, as with all observational methods, IPTW cannot account for unmeasured or unknown confounders.
Data from clinical databases often contain clinical errors in documentation, including implausible values such as ages >120 or body mass index <10 kg/m2. It is important to clean the data of likely data errors before analysis. Potential data-quality outliers can be identified using a combination of distribution-based methods and clinically informed thresholds. Distribution-based approaches, such as identifying observations exceeding ±3 standard deviations from the mean, may be useful for approximately normally distributed variables, whereas clinically defined plausibility limits, such as gestational age <20 weeks, are often preferable for skewed clinical measures. Sensitivity analyses may help assess whether exclusion of implausible observations materially influences study findings. Data missingness is another issue to consider with observational data, as this may not occur at random and can introduce additional bias when key variables are incompletely captured.
Benchmarking against established epidemiological and demographic literature is also important to assess external validity. When possible, baseline cohort characteristics should be compared with published reference populations, findings should be replicated across independent datasets, and results should be interpreted in the context of biological plausibility.
The temptation to draw causal inferences from observational data without randomization is strong, but careful methodological safeguards are necessary to reduce the risk of inaccurate conclusions. Target trial emulation (TTE) is a framework where investigators first specify the protocol of the hypothetical randomized trial they would ideally conduct and then design the observational study to emulate that trial as closely as possible. Key components include explicit specification of eligibility criteria, treatment strategies, assignment time (time zero), follow-up, outcomes, censoring rules, and analytic methods. TTE, combined with DAG-informed confounder selection, prespecified thresholds for clinical significance, propensity-score methods such as IPTW, sensitivity analyses, focus on data integrity, and consideration of biological plausibility, can strengthen causal inference from observational data in situations for which randomized controlled trials would be impractical, prohibitively expensive, or ethically challenging.
Large clinical databases provide unprecedented opportunities to study disease epidemiology, treatment effectiveness, and health care delivery at the population scale. However, the validity of findings depends on careful attention to study design, data quality, confounding, clinical significance, and causal inference. A thorough understanding of both the strengths and limitations of these resources is essential for investigators, reviewers, and readers seeking to interpret observational research responsibly.