Optimizing Heparin Quality Assurance Utilizing Electronic Data Abstraction

Introduction

Unfractionated heparin (UFH) is a widely used anticoagulant in hospitalized patients. ¹ While use is integral to prevention of thromboembolic events, the drug carries significant risks due to complex dosing and bleeding complications. Consistent with The Joint Commission’s National Patient Safety Goal (NPSG) 03.05.01, which focuses on “reducing the likelihood of patient harm associated with the use of anticoagulant therapy,’ adoption of stringent monitoring and continuous quality assurance is essential to minimize adverse events and ensure patient safety.^2-6 While quality assurance measures are essential for optimizing the use of UFH, obtaining the necessary data is associated with a significant cost. In 2016, the overall cost of reporting quality metrics reached $14.5 billion. ⁵ As health care costs continue to rise, the need for efficient solutions becomes increasingly critical. The majority of time spent on quality data reporting in health care is related to data collection.^5,7 One study found that physician practices dedicate a substantial portion of their workweek – approximately 15 hours per physician – solely to collection of data to report quality measures. ⁵ This raises concerns about the optimal use of health care professionals’ time and highlights the need for a more streamlined and cost-effective approach.

Reducing time invested in data collection while maintaining accuracy is crucial to mitigating costs associated with monitoring quality assurance measures and capturing additional data. Few studies have examined electronic versus manual data collection methods. One retrospective study evaluating antibiotic use across 30 facilities found a high level of agreement between these methods for initial severity measures (agreement = 86-89% with a kappa = 0.5-0.82). ⁸ However, these findings may not be generalizable to other medication types, such as anticoagulants. Furthermore, 2 studies in obstetrics which evaluated agreement with electronic data collection observed that results varied depending on the specific data point evaluated.^9,10

At present, no studies have investigated the differences between electronic and manual data collection for evaluating anticoagulation therapy. The goal of this study is to compare anticoagulation quality assurance measures for heparin collected through manual versus electronic data abstraction and to evaluate the level of agreement between the 2 approaches.

Methods

This is a single-center, observational cohort study conducted at Ascension St. John Hospital, a 714-bed hospital in Detroit, Michigan. Patients who received intravenous UFH targeting the institutional aPTT therapeutic range from September 1, 2023 to November 20, 2023 were evaluated for inclusion. Patients treated for less than 24 hours were excluded. The study was conducted in accordance with ethical guidelines and received approval from the Institutional Review Board.

All patients admitted to Ascension St. John Hospital receive heparin management through the pharmacy anticoagulation service. Prescribers are responsible for placing an initial order for “heparin managed by pharmacy,” which delegates authority to the pharmacist for further management including dosing, laboratory monitoring, and periprocedural management. Throughout the hospital admission, pharmacists manually record laboratory data, dose changes, bleeding, and thrombotic events on paper anticoagulation management forms. Data from anticoagulation management forms are manually reviewed for reporting of quality assurance indicators by the pharmacy department on a quarterly basis. Data collection for quality assurance was transitioned to an electronic process in 2022. This process involved querying data from the institutional data warehouse, further organized with rules in Google Sheets for all outcomes (Appendix 1). Laboratory data were recorded based on collection times, while medication administration times were obtained from barcode scanning data. Clinical outcomes were electronically extracted using International Classification of Diseases (ICD) codes for systemic embolism, venous thromboembolism (VTE), stroke, and major bleeding events.

The primary outcome of the study was to assess the level of agreement between electronic and manual data collection methods regarding the percentage of patients achieving therapeutic aPTT within 24 hours. Secondary outcomes included the agreement on the percentage of patients achieving either a therapeutic or supratherapeutic aPTT within 24 hours, the average time to therapeutic and supratherapeutic aPTT, and the total time spent on data collection per month. In addition, agreement on patients who received an initial bolus of heparin as documented in the medical records, and those who received subsequent heparin boluses in response to low aPTT values, according to the documentation, were evaluated. Lastly, agreement on clinical outcomes including new VTE, stroke, systemic embolism, and major bleeding was evaluated. In cases of disagreement with the primary outcome, the cases were re-evaluated to determine whether manual or electronic data were accurate.

Results

Between September and November 2023, a total of 288 patients on anticoagulation with heparin for at least 24 hours. Patients were receiving UFH for the following indications: VTE (35.8%), atrial fibrillation (32.3%), acute coronary syndrome (30.9%), mechanical valve (3.5%), and other (6.6%). The average time to achieve a therapeutic PTT was 1.13 ± 0.75 days, while time to either therapeutic or supratherapeutic PTT was 0.82 ± 0.67 days.

For the primary outcome, agreement of patients achieving therapeutic aPTT within 24 hours, the kappa value was 0.86 indicating excellent agreement (Table 1). Manual data collection indicated 44.1% of patients were therapeutic within 24 hours, whereas electronic data collection indicated 46.9% of patients were therapeutic in 24 hours. The time to therapeutic aPTT differed by 0.05 days (95% CI: 0.09 to 0.01, P = 0.04) between the 2 methods. Bland-Altman analysis showed similar results (Figure 1). The method of data collection only explained 0.6% of the mean difference between groups. For achieving either therapeutic or supratherapeutic aPTT within 24 hours, there was substantial agreement with a kappa value of 0.69. Manual data collection indicated 61.5% of patients were therapeutic within 24 hours, whereas electronic data collection indicated 73.3% of patients were therapeutic in 24 hours. The time to achieve either therapeutic or supratherapeutic PTT levels demonstrated no significant difference of 0.02 days between methods (95% CI: −0.84 to 0.04, P=0.51), with data collection type explaining only 1.4% of the mean difference. There were 10 cases of disagreement between electronic and manual data collection. A closer examination of these cases revealed that 7 discrepancies were due to errors with the electronic rule, which stemmed from heparin orders being stopped and restarted. The remaining 3 cases were attributed to manual collection errors.

OPEN IN VIEWER

Table 1.

Assessment of Reliability by Outcome.

Outcome	Kappa	Strength of agreement
Therapeutic aPTT within 24 hours	0.86	Excellent agreement
Therapeutic or supratherapeutic aPTT within 24 hours	0.69	Substantial agreement
Receipt of initial heparin bolus	0.6	Moderate agreement
Receipt of heparin bolus for subtherapeutic aPTT	0.13	Poor agreement
New diagnosis—CVA	0	Poor agreement
New diagnosis—VTE	-0.01	Poor agreement

OPEN IN VIEWER

Figure 1.

Bland-Altman analysis of the difference in time to therapeutic PTT.

When assessing whether a patient received an initial bolus of heparin, the kappa value was 0.6, indicating moderate agreement. However, for identifying patients who received subsequent heparin boluses in response to a low aPTT value, the kappa value was 0.13, indicating poor agreement. Clinical outcomes showed even lower agreement, with new VTE cases exhibiting a kappa value of −0.01, suggesting poor agreement. For patients with a new stroke, the kappa value was 0 also indicating poor agreement. The lack of new major bleeding events and new systemic embolisms in both groups prevented the calculation of the kappa value for these outcomes due to lack of events. In addition, time spent performing data collection and manipulation was significantly longer in the manual compared with the electronic group (15.3 hours vs 1.3 hours, P < 0.01) for each month of data, respectively.

Discussion

Our study shows that the reliability of electronic data collection varies by data point. Excellent agreement was found for therapeutic (or supratherapeutic) within 24 hours, indicating high reliability for this metric. While the absolute time difference for time to therapeutic aPTT of 0.05 days was statistically significant, this lacks clinical significance.

Moderate agreement was observed for the initial heparin bolus, suggesting some reliability with room for improvement. Discrepancies were noted for subsequent heparin boluses, with poor agreement indicating significant issues between manual and electronic data extraction. We believe this discrepancy was influenced by cases where no initial bolus was given because patients were receiving anticoagulation prior to admission. In these cases, the subsequent bolus was identified as the initial bolus; no subsequent boluses were administered due to either the first PTT being at or above goal or heparin was discontinued. Furthermore, we relied on nursing documentation for administration times, which may have inaccuracies. Normalizing the bolus dose based on patient weight could help decrease errors, but further verification would be required due to potential inaccuracies with documentation of patient weight. Misclassification of a 50 units/kg subsequent bolus versus a 60 units/kg initial bolus for our acute coronary syndrome protocol is possible. Given the low reliability of this endpoint, manual abstraction remains necessary until electronic methods are improved.

Assessment of clinical outcomes showed poor agreement between electronic and manual data capture. Specifically, there was poor agreement in identifying new VTE cases and new stroke cases, and kappa values could not be calculated for new major bleeds and new systemic embolism due to zero occurrences in both groups The discrepancy in identifying new VTE and stroke cases was due to ICD misclassification, where historical events were incorrectly coded as acute occurrences. These findings highlight significant limitations in electronic data capture for clinical outcomes, which aligns with prior reports noting low reliability of ICD codes for these events. ¹¹ The results emphasize the importance of manual validation of these critical endpoints to ensure accurate reporting and patient safety.

The nearly perfect agreement in measuring therapeutic aPTT within 24 hours suggests that electronic data collection for this metric is highly reliable. This finding supports the transition to electronic data abstraction for aPTT within 24 hours, offering a way to streamline monitoring and reduce manual workload without compromising accuracy. The substantial kappa value for capturing therapeutic or supratherapeutic aPTT within 24 hours confirms the reliability of electronic abstraction for this metric. However, the slightly lower agreement compared with the last metric of therapeutic aPTT within 24 hours suggests some caution is warranted. This result indicates that while the electronic method is generally reliable, further refinement and validation are needed to ensure optimal accuracy.

Our findings align with other studies that have examined the reliability and agreement between electronic and manual data extraction for quality assurance across various medical fields. One study which explored the reliability of electronic medical record data for obstetrical research identified challenges with consistency and completeness with electronic extraction. While there was almost perfect agreement on extracting mode of delivery, the study only found fair agreement for identifying postpartum hemorrhage and vertex presentation. ⁷ Similarly, research on antimicrobial prescribing quality metrics for hospitalized veterans revealed discrepancies between extraction methods for certain metrics. While they found high consistency in the percentage of patients receiving anti-MRSA coverage and duration of total treatment, there was less consistency in identifying excessive duration of antibiotic therapy and the number of days to clinical stability. ⁸ These studies, along with our study, highlight the ongoing challenge of achieving reliable and efficient quality assurance, further proving a need for a more standardized and reliable process.

While electronic data collection shows promise for improving the efficacy and accuracy of heparin quality assurance data, it is not a universal solution. The reliability of each metric must be individually assessed to determine its suitability for electronic collection. Manual data collection, which inherently involves clinical judgment and reasoning, may capture nuanced insights that electronic methods may not fully capture. This may explain the differences observed in certain metrics, particularly clinical outcomes. Therefore, while collecting data electronically can streamline processes, it is important to recognize the value of clinical insights provided by manual methods. A balanced approach, combining both electronic and manual data collection, may be necessary to ensure accuracy in capturing patient information while maintaining overall efficiency.

At our institution, these findings led to a transition from an intern-based manual data collection process to a hybrid approach that integrates both electronic and manual methods. We use the percent therapeutic obtainment and time to therapeutic data to evaluate monthly trends for all heparin patients. For clinical outcomes, we transitioned to an electronic form which allows pharmacists to document major bleeding and thrombotic events when they occur. These changes have allowed us to eliminate all intern hours previously related to data collection, enabling them to focus on other areas of need.

This study has several limitations that should be considered. First, the study was conducted at a single institution which may limit the generalizability to other institutions in the setting of different patient populations, protocols, and electronic health record systems. Second, the study focused exclusively on heparin, limiting its applicability to other anticoagulants and to all other medications. Finally, the results showed lack of agreement on clinical outcomes, possibly due to the infrequency of these events, highlighting the need for a study that overlooks data for a longer period of time. These limitations stress the importance of multi-center studies and ongoing evaluation of electronic data collection systems to ensure reliability and accuracy across various clinical contexts.

Conclusion

Our study demonstrated that the reliability of electronic data collection varies depending on the specific data point evaluated. For clinical outcomes, manual data collection remains the gold standard. However, a hybrid approach combining electronic and manual methods can maintain reliability while decreasing the time required for data collection.