Assessing quality and agreement of structured data in automatic versus manual abstraction of the electronic health record for a clinical epidemiology study

Abstract

Objective

We evaluate data agreement between an electronic health record (EHR) sample abstracted by automated characterization with a standard abstracted by manual review.

Study Design and Setting

We obtain data for an epidemiology cohort study using standard manual abstraction of the EHR and automated identification of the same patients using a structured algorithm to query the EHR. Summary measures of agreement (e.g., Cohen’s kappa) are reported for 12 variables commonly used in epidemiological studies.

Results

Best agreement between abstraction methods is observed among demographic characteristics such as age, sex, and race, and for positive history of disease. Poor agreement is found in missing data and negative history, suggesting potential impact for researchers using automated EHR characterization. EHR data quality depends upon providers, who may be influenced by both institutional and federal government documentation guidelines.

Conclusion

Automated EHR abstraction discrepancies may decrease power and increase bias; therefore, caution is warranted when selecting variables from EHRs for epidemiological study using an automated characterization approach. Validation of automated methods must also continue to advance in sophistication with other technologies, such as machine learning and natural language processing, to extract non-structured data from the EHR, for application to EHR characterization for clinical epidemiology.

Keywords

Unstructured data extraction automated data extraction structured query electronic health record characterization epidemiology investigative techniques epidemiologic methods data collection data accuracy negative results health information management medical records systems computerized reproducibility of results automation health services administration organization and administration meaningful use colorectal neoplasms

Highlights

•

Discrepancies may appear when comparing automatic versus manual EHR characterization.

•

Discrepancies may introduce bias to measures of association such as odds ratios.

•

Poor agreement for unstructured (free-form) EHR data may explain variable quality.

•

Automatically abstracted EHR data may require validation with manual methods.

•

Policy affecting clinician EHR editing may improve underlying data quality.

Introduction

Background and significance

Use of electronic health record (EHR) data in clinical epidemiological research has grown explosively.¹ Transition from paper charts to EHR systems enabled electronic search and abstraction of clinical data for manual chart review studies.^2,3 Increasingly, EHR characterization involves automated search algorithms employing International Classification of Diseases (ICD) codes, billing claims and patient demographics, or other characteristics from structured (template-like) vs unstructured (free-form) EHR sources. Accuracy of data obtained through automated EHR characterization versus manual searches has been studied in diverse contexts, with heterogeneity of the EHR and variable agreement between abstraction methods being common themes.^4-6

Manual EHR abstraction, while considered a gold standard,⁷ compares to automated abstraction in its susceptibility to quality issues due to clinical or administrative coding errors and also missing data. However, despite common limitations due to underlying data quality, potential advantages of automated, standardized search techniques are numerous, including more timely results, reduced labor intensity, reduced likelihood of data entry errors, unbiased queries, larger sample sizes, and capacity to define search parameter ranges precisely.⁸ Manual chart review may have additional limitations of lack of rigorous quality control and increased subjectivity in collection techniques.^9-11

Study setting and aims

The setting of this study is a single academic health center, the Medical University of South Carolina (MUSC). Like multiple other academic health centers, MUSC maintains EHR warehouses, which are populated by clinical data (The automated chart review method) and which may be queried for research.^{12, 13} To improve accuracy of automated searching algorithms, it is crucial to understand the contexts in which EHR searches produce reliable data. We compute measures of agreement for 12 variables commonly assessed in clinical epidemiology studies; we evaluate the impact of agreement measures on the attenuation in true odds ratios due to error; and finally, we analyze how measures of agreement between automated and manual methods improved due to national policy changes.

Materials and methods

Study population and patient privacy and protection

The Medical University of South Carolina (MUSC) Institutional Review Board (IRB# Pro00007139) approved all activities. The study population comprised patients with histologically confirmed advanced adenomatous colorectal polyp diagnosed between October 2012 and 30 June 2016. All cases were selected from the MUSC pathology laboratory information system CoPath (Cerner Corporation, Kansas City, MO). Specifically, we identified patients who underwent a sigmoidoscopy or colonoscopy with polypectomy of $\geq 1$ advanced lesion(s) between October 2012 and May 2016. For each case, a pathologist reviewed the slide to confirm that the tissue was colorectal in origin with a diagnosis of conventional or serrated dysplasia.

Data collection

The manual chart abstraction (MCA)

A trained data abstractor blind to the study intent performed the MCA according to a short procedural manual. No sources of information within the EHR were explicitly excluded; patient histories, progress notes, and pathology reports were all searched to obtain the defined demographic and medical criteria (see Supplementary Table A). Study data were stored at MUSC using REDCap (Research Electronic Data Capture).¹⁴ We reviewed patient encounter dates, problem lists (summary of all active diagnoses), histories (medical and family), progress notes, procedure notes, office notes, narrative and impression documents, and pathology results. Additionally, the EHR of each patient was manually queried using text search within patient records using these terms: colon, colectomy, flexible sigmoidoscopy, colonoscopy, diabetes, BMI, hypertension, hyperlipidemia, smoking, and cancer (Supplementary Table A). Manual review directly searched EHR records for medical history since 2004; however, manual chart review captured both smoking status and BMI only if assessed at a contact date within ±30 days of the procedure date, a cutoff constructed for increased relevance to colorectal cancer prognosis.

The automated chart review method

International Classification of Disease (ICD) codes and current procedural terminology (CPT) codes were used to define a structured query language (SQL) for EHR characterization. In this work, EHR characterization refers to the development and use of algorithms needed to capture an accurate digital impression of observed patient characteristics documented within the EHR.^15-17 Algorithms for automated chart review were defined to match date ranges and subjects from the MCA. Patient medical record numbers (MRNs) from the MCA and query parameters were supplied to an honest broker who performed the automated EHR characterization within the Research Data Warehouse (RDW) at MUSC.¹⁸ The RDW supports longitudinal research by sourcing data from the EHR, legacy systems, and patient registries on campus. Contact dates were used to limit medical history to May 2012, when the first outpatient contacts were documented in the EHR. However, history assessed at those contact dates could reflect, for example, a colonoscopy performed prior to May 2012. ICD codes C18, C20, Z85, and C34 were queried for cancer history, and numerous ICD codes such as I10-15, E78, and E08 were queried to identify patients with a history of hypertension, hypelipidemia, or diabetes. In addition to ICD codes, CPT codes were used to identify family history of cancer among patient and provider records (Supplementary Table A).

Statistics

Percent agreements were calculated for the data obtained using each of the two abstraction methods: the numerator is the number of patients who were classified identically by both sources and the denominator is the total number of cases evaluated by both sources. The summary agreement for each variable is the total number of agreeing results divided by the total number of results.

Cohen’s kappas with their 95% confidence intervals (CIs) were calculated to assess agreement for each categorical variable.¹⁹ Landis and Koch²⁰ cutoffs were used to interpret kappas: a score of 0.00–0.20 indicates slight agreement, 0.21–0.40 is fair, 0.41–0.60 is moderate, 0.61–0.80 is good, and 0.81–1.00 is very good. For continuous variables, the coefficients of Pearson product-moment correlations (PCCs) were calculated as a measure of interclass reliability.²¹ Pearson correlations have been interpreted similarly: negligible (0.00–0.30), low (0.30–0.50), moderate (0.50–0.70), high (0.70–0.90), and very high (0.90–1.00) correlation.²² Bland-Altman mean-difference plots were also included to improve interpretation of agreement for the continuous variables age and BMI.²³ We employed Pearson correlations, Bland-Altman plots, and Cohen’s kappa methods to compare agreement trends over time for smoking history.

Finally, we estimated the potential impact of the rates of agreement over a range of true odds ratios for colorectal polyps and comorbidities using methodology from White et al.²¹ The attenuation factor for kappa assumes non-differential measurement error and uncorrelated errors between the manual and automated techniques. For the reliability estimates (i.e., PCC), we assumed a parallel test methodology and assessed the attenuation in ORs using the formula $O R_{0} = O R_{T}^{ρ^{2}}$ , where ρ is the correlation coefficient. For kappa, we used the formula $O R_{0} = O R_{T}^{\sqrt{k}}$ , where κ is the kappa.

Analyses were performed in SAS v 9.4 (Cary, NC), with the exception of Figure 4, prepared in R (Vienna, Austria) using the tidyverse ggplot2 package.²⁴

Results

Patient characteristics

In Table 1, we describe prevalence of patient characteristics for comparison of manual and automated reviews. We report mean and standard deviation for continuous variables Age and BMI, and percent of total subjects (103) for other variables.

Table 1.

Comparison of patients in manual and automated reviews.

Variable compared: Manual versus automated review	Manual (n=103)	Automated (n=103)
Age, yrs. Mean (SD.)	63.95 (10.12)	63.97 (10.25)
Race, n, (%) African American Caucasian American Missing	49 (48) 54 (52) 0 (0)	49 (48) 53 (51) 1 (1)
Sex, n, (%) Male Female Missing	60 (58) 43 (42) 0 (0)	58 (56) 44 (43) 1 (1)
Body Mass index (BMI), mean (SD.)	29.47 (5.91)	29.29 (5.90)
BMI – Availability, n, (%) BMI found within ±30d BMI not found within ±30d Missing	78 (76) 20 (19) 5 (5)	72 (70) 30 (29) 1 (1)
Smoking history, n, (%) Ever (current or former) Never Missing	59 (57) 40 (39) 4 (4)	51 (50) 39 (38) 13 (13)
Patient history of any cancer, n, (%) Yes No Missing	18 (17) 74 (72) 11 (11)	27 (26) 13 (13) 63 (61)
Family history of any cancer, n, (%) Yes No Missing	38 (37) 43 (42) 22 (21)	33 (32) 6 (6) 64 (62)
Patient history of CRC, n, (%) Yes No Missing	5 (5) 86 (83) 12 (12)	15 (15) 10 (10) 78 (76)
Family history of CRC, n, (%) Yes No Missing	12 (12) 65 (63) 26 (25)	8 (8) 31 (30) 64 (62)
Patient history, hypertension, n, (%) Yes No Known unknown^a Missing	76 (74) 22 (22) — 5 (5)	37 (36) — 40 (39) 26 (25)
Patient history, hyperlipidemia, n, (%) Yes No Known unknown^a Missing	59 (57) 36 (35) — 8 (8)	25 (25) — 40 (39) 37 (36)
Patient history, diabetes, n, (%) Yes No Known unknown^a Missing	31 (30) 63 (61) — 9 (9)	16 (16) — 40 (39) 47 (46)

^aThe automated process searched a chart and found no record of these variables but also did not find documentation of negative history.

Patient history of cancer excepted, point estimates reveal that automated compared to manual review detected fewer cases of chronic diseases (hypertension, hyperlipidemia, and diabetes). Automated review also detected lower absolute percentages of patients with family history of colorectal cancer. Manual chart review was missing 0–25% of records, whereas automated EHR characterization was missing 1–76% of records.

Measures of agreement

We estimated measures of agreement between abstraction methods for categorical variables using Cohen’s kappa and for continuous variables using Pearson interclass correlations (PCCs) (Table 2).

Table 2.

Measures of Agreement for individual responses: percent agreement, Cohen’s Kappa and Pearson Correlation Coefficient (PCC).

Variable assessed for agreement	Percent agreement, automated versus Manual	Measure of agreement, Cohen’s kappa (95% CI)	Pearson correlation coefficient (PCC)
Age, years (n=102)	N/A	N/A	0.997 (95% CI: 0.996, 0.998)
Race African American Caucasian American Missing Total	49/49 (100%) 53/54 (98%) 0/1 (0%) 102/103 (99%)	1.000 (1.000, 1.000) 1.000 (1.000, 1.000) N/A^c —	—
Sex Male Female Missing Total	58/60 (97%) 43/44 (98%) 0/1 (0%) 101/103 (98%)	0.980 (0.941, 1.000) 0.980 (0.941, 1.000) N/A^c —	—
BMI (n=66) ^a	N/A	—	0.997 (95% CI: 0.996, 0.998)
BMI – Availability^b BMI found within ±30days^d BMI not found within ±30d Missing Total	66/84 (79%) 14/36 (39%) 0/6 (0%) 80/103 (78%)	0.561 (0.382, 0.739) — — —	—
Smoking History Ever (current or former) Never Missing Total	51/59 (86%) 36/43 (84%) 3/14 (21%) 90/103 (87%)	0.826 (0.719, 0.933) 0.856 (0.754, 0.959) 0.291 (0.015, 0.566) —	—
Patient history of any Cancer Yes No Missing Total	15/30 (50%) 8/79 (10%) 8/66 (12%) 31/103 (30%)	0.578 (0.392, 0.765) -0.039 (−0.135, 0.056) 0.042 (−0.052, 0.136) —	—
Family history of any Cancer Yes No Missing Total	32/39 (82%) 5/44 (11%) 20/66 (30%) 57/103 (55%)	0.850 (0.743, 0.957) 0.113 (−0.001, 0.228) 0.216 (0.095, 0.337) —	—
Patient history of CRC Yes No Missing Total	4/16 (25%) 9/87 (10%) 11/79 (14%) 24/103 (23%)	0.354 (0.083, 0.623) 0.016 (−0.031, 0.063) 0.053 (−0.008, 0.114)	—
Family history of CRC Yes No Missing Total	8/12 (67%) 23/73 (32%) 21/69 (30%) 52/103 (50%)	0.779 (0.573, 0.986) 0.121 (−0.028, 0.270) 0.168 (0.032, 0.304) —	—
Patient history, Hypertension Yes No Missing Total	35/78 (45%) — 1/30 (3%) 36/103 (35%)	0.264 (0.136, 0.391) N/A^e 0.056 (0.006, 0.105) —	—
Patient history, Hyperlipidemia Yes No Missing Total	23/62 (37%) — 2/43 (5%) 25/103 (24%)	0.294 (0.152, 0.436) N/A^e 0.028 (−0.025, 1.081) —	—
Patient history, Diabetes Yes No Missing Total	15/32 (47%) — 3/53 (6%) 18/103 (17%)	0.545 (0.366, 0.724) N/A^e 0.010 (−0.035, 0.055) —	—

^aBMI Agreement – Of 103 patients, 66 had BMI records available within 30 days of the index procedure date for both manual and automated review.

^bBMI availability – The existence of BMI values in the Epic EHR.

^c1 patient was missing a value for race and sex in electronic review because of a change in ID number that was not carried forward in the automated review process.

^dBMI values outside the 30-day window were counted as missing for Kappa calculations.

^eElectronic search did not return data on patients with negative history of hypertension, hyperlipidemia or diabetes.

Personal characteristics

Agreement for age, race, and sex is “very good” or “very high.” Of 103 total subjects, 93 (90%) agreed between methods (Table 2). For age, automated results agree exactly with 90% of the manual review results (Figure 1). Agreement between methods for race and sex was nearly perfect (Tables 1 and 2 and Supplementary TableB2).

Figure 1.

Overall Age: Bland-Altman Mean-Difference Plot. 1-Year differences represent that Epic rounds when a patient’s birthday is upcoming; manual chart review did not round to nearest birthday. A 7-year difference outlier represents a manual transcription error.

Body mass index (BMI)

Of 66 patients whose BMI values were available from both automated and manual review within 30 days of the index procedure date, agreement was very good (PCC = 0.997, 95% CI: 0.996, 0.998). Manual review returned BMI values for 11 patients within the 30-day window that were not found by automated EHR characterization. Conversely, BMI values obtained through the automated approach within the 30-day window detected values for 6 patients that were not found by manual chart review. No systematic explanations for these differences were apparent, apart from the existence of multiple BMI values in most patients' charts (Figure 2).

Figure 2.

Overall BMI: Bland-Altman Mean-Difference Plot.

Smoking

Overall, agreement between the automated and manual search methods for ever smokers (current and former smokers) and never smokers was good, with Cohen’s kappa >0.8. The sources show poor agreement (kappa = 0.291) on the question of which patients are missing data on smoking; the methods agreed on 3 out of 14 total missing records identified by both processes.

Cancer history

Overall agreement (average of positive, negative, and missing) between manual and automated reviews was poor for cancer history, from 23% overall agreement for patient history of colorectal cancer to 55% overall agreement for family history of any cancer. The greatest agreement between methods for cancer history was for positive family history of any cancer, at 82%, exemplifying the trend that positive history of cancer showed better agreement compared to negative history or missing history. Of note, raw results from the automated EHR characterization returned some patients with both a positive and negative history of cancer. Whereas a patient might be positive for colorectal cancer, the same patient could also be listed as negative for “cancer,” as well as negative for many other specific cancers. Thus, depending on which results were viewed after automated characterization, contradictory observations could be made about the cancer history of a patient. We assumed that indication of a positive history for a specific cancer superseded a general negative for cancer.

Chronic disease history

Statistical agreement for chronic diseases (diabetes, hypertension, and hyperlipidemia) is poor, with positive history of each disease agreeing in 37–47% of cases. Discrepancies were greatest for negative and missing history of these diseases, with agreement as low as 3% on missing data for hypertension. The automated query returned minimal data for negative history of disease, and thus agreement with manual review is very poor for negative history.

Time trends in agreement

Table 3 shows age, smoking history, and family history of cancer in 2013 vs 2016. Pearson correlation coefficient (PCC) with a 95% confidence interval (CI) is reported for age, while Cohen’s kappa with a 95% CI is reported for both smoking and family history of cancer.

Table 3.

Time trends in automated versus manual chart review agreement for age, smoking and family history: 2013 and 2016.

Variable compared: Automated versus manual	2013 comparison (N=31)	2016 comparison (N=18)
Age	PCC (95% CI) 0.992 (95% CI: 0.983, 0.996)	PCC (95% CI) 1.000 (1.000, 1.000)
Smoking History Ever (current or former) Never Missing	Cohen’s kappa (95% CI) 0.733 (0.497, 0.968) 0.801 (0.591, 1.000) 0.367 (0.048, 0.687)	Cohen’s kappa (95% CI) 1.000 (1.000, 1.000) 1.000 (1.000, 1.000) N/A^a
Family history of cancer Yes No Missing	Cohen’s kappa (95% CI) 0.803 (0.593, 1.000) 0.299 (−0.037, 0.636) 0.251 (−0.057, 0.558)	Cohen’s kappa (95% CI) 0.889 (0.679, 1.000) 0.169 (−0.133, 0.471) 0.222 (−0.061, 0.505)

^aNo missing data for 2016 smoking.

Agreement for age remained very high over time in 2013 compared to 2016 (Table 3 and Figure 3). From Table 3, agreement in smoking history improved significantly in 2016 compared to 2013, but was unchanged for family history of cancer over the same time points. Unlike smoking history, from 2013 to 2016, no statistically significant change was observed in agreement for family history of cancer. The confidence intervals, however, have shrunk in 2016.

Figure 3.

Age in 2013: Bland-Altman Mean-Difference Plot.

Attenuation of odd ratios

We estimated a range of attenuated values, from an approximate 5% attenuation to ORs for demographic characteristics age, race, and sex, to a greater than 90% attenuation in the OR for negative patient history of colorectal cancer (Figure 4). Our data suggest potential attenuation of 11–58% in ORs for positive patient or family history of cancers and chronic disease.

Figure 4.

Attenuation of Odds Ratios at Several Values of Cohen’s Kappa. True OR increases with Kappa (2.1.4, Statistical Methods).

Discussion

Discrepancies between automated and manual methods may have minimal or profound impact upon the reliability of odds ratios (ORs) (Figure 4). We estimated an approximate 5% attenuation to ORs for age, race, and sex, and greater than a 90% attenuation to the OR for negative patient history of colorectal cancer. We estimated attenuation of 11%–58% in ORs for positive patient or family history of cancers and chronic disease. A recent study²⁵ examining the risk of venous thromboembolism in elderly, hospitalized internal medicine patients compared automated and manual abstraction techniques, finding that automatically abstracted data may be inconsistent with manual abstraction. For example, the study found that sensitivity of the automated tool to detect acute conditions (such as myocardial infarction) is “suboptimal,” but provides better agreement in sensitivity for chronic conditions, such as obesity.²⁵ The authors attributed the discrepancy to their EHR search algorithm, which depended largely on specific ICD codes and structured data fields.²⁶ Another recent study compared obstetrical EHR data obtained through automated versus manual review and also found a range of reliability.²⁷ The authors found good agreement for mode of delivery (i.e. vaginal vs. Caesarian), but only fair agreement for the acute conditions of vertex presentation and post-partum hemorrhage. They explain that some discrepancy could arise from the extent of interpretation made by a clinician to determine a qualitative finding such as “hemorrhage.” They further described discrepancies in reliability as a “serious limitation” to the utility of automated chart reviews and recommended that measures of reliability be reported with automatically abstracted data.²⁷

Both these examples and our original findings suggest that automated chart review is able to capture some EHR variables more reliably. Variables could be divided into many conceptual classes, such as acute versus chronic,²⁵ active problem versus historical, severity versus non-severity of condition, personal versus family history, or positive versus negative history.²⁸ However, not all variation in agreement between automated and manual methods is predicted by these classes. In our results and in the examples from Pavon et al.²⁵ and Altman et al.,²⁷ variability seems to be best explained by the definition of structured versus unstructured data fields in the EHR, and by how meticulously and consistently the structured fields are populated by accurate ICD codes.

“Meaningful Use” and policy

EHR data quality depends upon providers, who may be influenced by both institutional EHR quality standards and Centers for Medicare and Medicaid Services (CMS) guidelines for documentation.^29,30 CMS requires of its reimbursed clinicians that “specific abnormal and relevant negative findings of the examination of the affected or symptomatic body area(s) or organ system(s) should be documented” in the medical record. Additionally, “pertinent negatives” should be documented for the history of the present illness.³¹ That is, pertinent negative findings are not explicitly defined by CMS, and thus compliance with CMS guidelines does not necessarily require uniformity in documentation of negative findings. Furthermore, it has been shown that clinicians may avoid obtaining or documenting a full family history for some visits.³² Although policymakers should avoid excessive documentation requirements on clinicians, the effectiveness of meaningful use criteria for improving clinical data quality has been demonstrated, such as at Intermountain Healthcare.³³

Data quality over time

In assembling data for longitudinal cohorts, an important feature to consider is consistency in the reliability of exposures over time.³⁴ Noted increased agreement of automated EHR characterization and manual abstraction of smoking history between 2013 and 2016 likely reflects initiatives at our institution in the intervening period investigating techniques to help inpatients to quit smoking. These studies coincided with improved completion of smoking data in the EHR system, which was adopted for inpatients at MUSC during that time.^35-37 In addition to these initiatives at MUSC, the CMS criteria for meaningful use were in place prior to 2013, and documentation of patient smoking status, in addition to age, race, sex, ethnicity, and BMI, were already of high quality.³⁰ In contrast to the significant improvement observed in the quality of smoking data over time as shown in Table 3, no statistically significant improvement occurred over the same interval for Age (Figure 3) or Family History of Cancer, although the point estimate of agreement for positive family history of cancer appears to be higher in 2016 with more narrow confidence intervals. The greater point estimate could be interpreted as improved agreement, but the point estimate for negative family history of cancer was lower in 2016. We conclude that age, previously a meaningful use variable recorded in a structured data field, remained at very good quality, while family history of cancer, recorded with inconsistent ICD codes or in unstructured fields in the EHR, remained at lower quality.

Limitations

Study data were obtained from a single academic health center using the Epic EHR. Manual extraction is limited by the accuracy of a human reviewer. However, we further acknowledge the limitation that a single human extractor performed the manual chart abstraction, whereas this process is more typically completed by multiple extractors. Because the study aims to identify agreement between the manual and automated sources, “Missing” was treated as an additional response category to allow meaningful interpretation. Future comparisons of manual and automated techniques should include validity controls that estimate the impact of systematic error.

Conclusion

Future work to validate these methods must keep pace with application of automated technologies to clinical epidemiological data. Use of natural language processing and machine learning is increasing in application to phenotyping of electronic health records for research.^38-40 Our results indicated poor quality among chronic disease variables. Future studies may detect more chronic diseases from clinical narratives using natural language processing methodologies.

Supplemental Material

sj-pdf-1-rmm-10.1177_26320843211061287 – Supplemental Material for Assessing quality and agreement of structured data in automatic versus manual abstraction of the electronic health record for a clinical epidemiology study

Supplemental Material, sj-pdf-1-rmm-10.1177_26320843211061287 for Assessing quality and agreement of structured data in automatic versus manual abstraction of the electronic health record for a clinical epidemiology study by J Grant Brazeal, Alexander V Alekseyenko, Hong Li, Mario Fugal, Katie Kirchoff, Courtney Marsh, David N Lewin, Jennifer Wu, Jihad Obeid and Kristin Wallace in Research Methods in Medicine & Health Sciences

Footnotes

Acknowledgments

Many thanks to Kathleen Cartmell, Graham Warren, Jonathan Beall and Michael Cummings for their professional consultation.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was partly funded by grants from the National Institutes of Health, National Cancer Institute (R03 CA156668, K07 CA151864, U54 CA210962) and National Library of Medicine (R01 LM012517); the Biostatistics Shared Resource, Hollings Cancer Center, Medical University of South Carolina (P30 CA138313); South Carolina Clinical and Translational Research (SCTR) Institute NIH Grant Number UL1 TR001450.

ORCID iDs

Joseph Grant Brazeal

Jihad Obeid

Supplemental Material

Supplementary Material for this article is available online.

References

Casey

Schwartz

Stewart

, et al. Using electronic health records for population health research: a review of methods and applications. Annu Rev Public Health 2016; 37: 61–81. DOI: 10.1146/annurev-publhealth-032315-021353.

Melton

Moeller

, et al. Accelerating chart review using automated methods on electronic health record data for postoperative complications. AMIA Annu Symp Proceedings. AMIA Symp 2017; 2016: 1822–1831.

C-I

Sohn

Rolfes

, et al. Application of a natural language processing algorithm to asthma ascertainment. an automated chart review. Am J Respir Crit Care Med 2017; 196: 430–437. DOI: 10.1164/rccm.201610-2006OC.

Devine

Van Eaton

Zadworny

, et al. Automating electronic clinical data capture for quality improvement and research: The CERTAIN validation project of real world evidence. eGEMs (Generating Evidence & Methods to Improve Patient Outcomes) 2018; 6: 8. DOI: 10.5334/egems.211.

Leung

Raulli

A-O

, et al. Assessment of the impact of EHR heterogeneity for clinical research through a case study of silent brain infarction. BMC Med Inform Decis Making 2020; 20: 60. DOI: 10.1186/s12911-020-1072-9.

Liede

Hernandez

Roth

, et al. Validation of international classification of diseases coding for bone metastases in electronic health records using technology-enabled abstraction. Clin Epidemiology 2015; 7: 441–448. DOI: 10.2147/CLEP.S92209.

Kahn

Eliason

Bathurst

. Quantifying clinical data quality using relative gold standards. AMIA Annu Symp Proceedings. AMIA Symp 2010; 2010: 356–360.

Newgard

Zive

Jui

, et al. Electronic versus manual data processing: evaluating the use of electronic health records in out-of-hospital clinical research. Acad Emerg Med 2012; 19: 217–227. DOI: 10.1111/j.1553-2712.2011.01275.x.

Delgado-Rodriguez

Llorca

. Bias. J Epidemiol Community Health 2004; 58: 635–641. DOI: 10.1136/jech.2003.008466.

10.

Khajouei

Abbasi

Mirzaee

. Errors and causes of communication failures from hospital information systems to electronic health record: A record-review study. Int J Med Inform 2018; 119: 47–53, DOI: 10.1016/j.ijmedinf.2018.09.004.

11.

Vassar

Matthew

. The retrospective chart review: important methodological considerations. J Educ Eval Health Professions 2013; 10: 12. DOI: 10.3352/jeehp.2013.10.12.

12.

Hulsen

Jamuar

Moody

, et al. From big data to precision medicine. Review. Front Med 2019; 6: 34. DOI: 10.3389/fmed.2019.00034.

13.

Obeid

Beskow

Rape

, et al. A survey of practices for the use of electronic health records to support research recruitment. J Clin Translational Sci 2017; 1: 246–252. DOI: 10.1017/cts.2017.301.

14.

Harris

Taylor

Thielke

, et al. Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform 2009; 42: 377–381.

15.

Borthwick

Smelser

Bock

, et al. ePhenotyping for Abdominal Aortic Aneurysm in the Electronic Medical Records and Genomics (eMERGE) Network: Algorithm Development and Konstanz Information Miner Workflow. 2016/04/08. Int Journal Biomedical Data Mining 2015; 4.

16.

Hripcsak

Albers

. Next-generation phenotyping of electronic health records. 2012/09/06. J Am Med Inform Assoc 2013; 20: 117–121. DOI: 10.1136/amiajnl-2012-001145.

17.

Pathak

Kho

Denny

. Electronic health records-driven phenotyping: challenges, recent advances, and perspectives. J Am Med Inform Assoc 2013; 20: e206–e211. DOI: 10.1136/amiajnl-2013-002428.

18.

MUSC Bioinformatics Center . Research Data Overview—Research Data Warehouse (RDW), http://academicdepartments.musc.edu/bmic/ResearchDataOverview.html (accessed Accessed Sept 2020).

19.

Hallgren

. Computing inter-rater reliability for observational data: an overview and tutorial. Tutorials Quantitative Methods Psychol 2012; 8: 23–34.

20.

Landis

Koch

. The measurement of observer agreement for categorical data. Biometrics 1977; 33: 159–174. DOI: 1977/03/01.

21.

Armstrong

White

Saracci

. Principles of Exposure Measurement in Epidemiology. Oxford: Oxford University Press, 1992, p. 351.

22.

Mukaka

. Statistics corner: A guide to appropriate use of correlation coefficient in medical research. Malawi Medical Journal : The Journal Med Assoc Malawi 2012; 24: 69–71. DOI: 2013/05/03.

23.

Altman

Bland

. Measurement in medicine: the analysis of method comparison studies. The Statistician 1983; 32: 307–317. DOI: 10.2307/2987937.

24.

Wickham

Averick

Bryan

, et al. Welcome to the Tidyverse. J Open Source Softw 2019; 4: 1686–6. DOI: 10.21105/joss.01686.

25.

Pavon

Sloane

Pieper

, et al. Automated versus manual data extraction of the padua prediction score for venous thromboembolism risk in hospitalized older adults. Appl Clin Inform 2018; 09: 743–751. DOI: 10.1055/s-0038-1670678.

26.

Bush

Kuelbs

Ryu

, et al. Structured data entry in the electronic medical record: perspectives of pediatric specialty physicians and surgeons. J Medical Systems 2017; 41: 75. DOI: 10.1007/s10916-017-0716-5. 2017/03/21.

27.

Altman

Colorafi

Daratha

. The reliability of electronic health record data used for obstetrical research. Appl Clin Inform 2018; 09: 156–162. DOI: 10.1055/s-0038-1627475.

28.

Ceusters

Elkin

Smith

. Negative findings in electronic health records and biomedical ontologies: a realist approach. Int Journal Medical Informatics 2007; 76(Suppl 3): S326–S333. DOI: 10.1016/j.ijmedinf.2007.02.003.

29.

The Joint Commission. Safe Use of Health Information Technology. Sentinel Event Alert 2015.

30.

Blumenthal

Tavenner

. The “meaningful use” regulation for electronic health records. New Engl J Med 2010; 363: 501–504. DOI: 10.1056/NEJMp1006114.

31.

Centers for Medicare and Medicaid Services . Evaluation and Management Services Guide, 2017, https://www.cms.gov/Outreach-and-Education/Medicare-Learning-Network-MLN/MLNProducts/Downloads/eval-mgmt-serv-guide-ICN006764.pdf (accessed September 2020).

32.

Tarini

McInerney

. Family history in primary care pediatrics. Pediatrics 2013; 132: S203–S210. DOI: 10.1542/peds.2013-1032D.

33.

Bowes

3rd . Impacts of EHR certification and meaningful use implementation on an integrated delivery network. AMIA Annu Symp Proceedings. AMIA Symp 2014; 2014: 325–332.

34.

Vandenbroucke

Pearce

. Point: incident exposures, prevalent exposures, and causal inference: does limiting studies to persons who are followed from first exposure onward damage epidemiology?. 2015/10/26. Am J Epidemiol 2015; 182: 826–833. DOI: 10.1093/aje/kwv225.

35.

Buchanan

Nahhas

Guille

, et al. Tobacco use prevalence and outcomes among perinatal patients assessed through an "opt-out" cessation and follow-up clinical program. Matern Child Health J 2017; 21: 1790–1797. DOI: 10.1007/s10995-017-2309-9.

36.

Cartmell

Dooley

Mueller

, et al. Effect of an evidence-based inpatient tobacco dependence treatment service on 30-, 90-, and 180-day hospital readmission rates. Med Care 2018; 56: 358–363. DOI: 10.1097/MLR.0000000000000884.

37.

Nahhas

Wilson

Talbot

, et al. Feasibility of implementing a hospital-based "opt-out" tobacco-cessation service. Nicotine Tob Res 2017; 19: ntw312–943. 2016/12/09. DOI: 10.1093/ntr/ntw312

38.

Koleck

Dreisbach

Bourne

, et al. Natural language processing of symptoms documented in free-text narratives of electronic health records: a systematic review. J Am Med Inform Assoc 2019; 26: 364–379. DOI: 10.1093/jamia/ocy173.

39.

Spasic

Nenadic

. Clinical text data in machine learning: systematic review. JMIR Med Inform 2020; 8: e17984. DOI: 10.2196/17984.

40.

Van Vleck

Chan

Coca

, et al. Augmented intelligence with natural language processing applied to electronic health records for identifying patients with non-alcoholic fatty liver disease at risk for disease progression. Int J Med Inform 2019; 129: 334–341. DOI: 10.1016/j.ijmedinf.2019.06.028.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.19 MB