Comparing the Predictive Effects of Patient Medication Adherence Indices in Electronic Health Record and Claims-Based Risk Stratification Models

Data source

This 2-year retrospective cohort study (2012–2013) used data provided by Health Partners, an integrated health care provider and health insurance company located in Bloomington, Minnesota. Health Partners insures 1.8 million beneficiaries and delivers health care services through 1800+ physicians, 8 hospitals, 55 primary care clinics, and 22 urgent care locations.

Study data contained both structured EHR and claims data linked by unique patient identifiers. EHR data contained patient demographic characteristics at their first encounter, outpatient diagnoses as International Classification of Diseases, Ninth Revision, Clinical Modification codes, medication as National Drug Code (NDC), and details of each filled prescription. EHR records of patient encounters that occurred outside of the Health Partner's care network were not accessible for this study, although if claimable, such encounters were represented in the claims data.

Study cohort

The original data included 114,665 patients in EHRs and 97,575 in claims. The study cohort was limited to patients with continuous enrollment with Health Partner's insurance plan in both 2012 and 2013. The aim was to measure outpatient medication adherence; thus, only outpatient records were extracted from EHRs and claims. Those who were missing outpatient visits, absent of outpatient prescriptions in EHRs or prescription filled records in claims, or older than age 65 (because of missing Medicare insurance claims) were excluded from the study sample. After applying the exclusion criteria, the sample size was reduced to 70,054 patients who had both EHR and claims data (Supplementary Fig S1).

Risk adjusters

The researchers aimed to assess multiple medication-derived risk adjusters at the patient level. First, 2 simple measures of medication count were derived. The unique yearly medication (Rx) count of NDCs was calculated for patient claims, EHRs, and the merged data set, respectively. Researchers then adopted the Johns Hopkins Adjusted Clinical Group (ACG) system (version 11) to convert NDCs into one of 62 prescribed medication-defined morbidity groups (RxMGs). ²⁵ RxMGs were created on account of the product active ingredient, route of administration, therapeutic use, and mechanisms of action, and thus, prevent the issues of overcounting the active ingredient, which may arise when using NDCs. ²⁵ The unique number of RxMGs also was calculated for the 3 data streams of EHRs, claims, and both.

Three medication adherence indices (ie, MRCI, MPR, PFR) were calculated using the medication data. MRCI measures the complexity of medication management for patients, containing major characteristics of a prescription. ^{6 –8} MRCI relies on EHR data and consists of 3 weighted elements of dosage form, dosing frequency, and additional drug administration instructions. ^4,6,10 MPR, a common measure of medication adherence using claims data, is generated from drug dispensing records. ^5,6 MPR indicates the proportion of times when patients with an adequate amount of medication follow their refill schedule. MPR is calculated as the sum of the days' supply for all fills of a drug divided by the number of days in a particular period. ⁵ Three MPR markers were created subsequently to indicate good, middle, and poor adherence levels, using thresholds of 0, .5, .8, and 1.0. ⁵

PFR is developed by mapping the date of a drug prescribed in the EHR to the date of prescription filled in claims in a given time period and averaging overall prescriptions. Thirty-day PFR was calculated to indicate the proportion of prescribed medications filled within 30 days. ^6,7 Additional details about the inclusion and exclusion criteria of medications, temporal limits, and exact methodology used to calculate MRCI, MPR, and PFR were published previously. ⁶

To simulate a real-world setting, such as a provider not having access to claims data for all patients, the researchers extracted relevant predictors for risk modeling based on the underlying data source used for the analysis. ^11,24 For example, when training the predictive models using EHR data, only EHR data were used to construct predictors such as diagnosis and MRCI. In contrast, when using claims data for modeling, only claims data were used to extract diagnosis and calculate MPR. And, when both data sources of EHRs and claims were used, then the combination of both underlying data sources were used to extract diagnosis and calculate PFR. ¹¹

Outcomes and predictive models

Three base models were constructed to predict 2 outcomes in 2 temporal patterns using 3 distinctive data sources. Temporal patterns included prospective (ie, predicting next year's outcome) and concurrent (ie, predicting same-year outcome) time frames. Data sources used to train models included insurance claims only, EHR only, and a merged data set of both. Two outcomes included any hospitalization in the year and total annual health care cost (extracted using claims). A binary marker was created for any hospitalization. Annual health care total cost was truncated at the top 1%. ¹¹

To represent a comparable and comprehensive set of utilization outcomes, claims data were used to measure utilization across all model. Claims were used for outcomes because EHR data only include utilization occurring within the provider's network, while claims data contain utilization that has occurred across all health care providers. ¹¹ Additionally, comparing the performance of different models requires predicting the same outcome. ¹¹ The 2012 independent variables were used to explain/predict outcomes in 2012 (concurrent) or 2013 (prospective).

To assess the added value of each medication adherence index, 3 models with increasing levels of complexity (1 to 3) were used. Base model 1 only included patient demographics (age and sex). Base model 2 included patient demographics plus the Charlson index. Seventeen Charlson comorbidity categories were summed up to indicate a weighted individual morbidity score. ²⁶ Base model 3 included a diagnosis-based risk score derived from the ACG System (ACG-Dx score). ²⁵ The ACG-Dx score is comparable to the Charlson index; however, the ACG-Dx score represents a more complex diagnosis-based measure of morbidity. ^7,11,25

Statistical analysis

Patients' demographic characteristics, medication, and diagnosis histories in 2012, and annual health care costs and utilization rates in 2012 and 2013 were analyzed. The researchers tested the contribution of MRCI, MPR, 30-day PFR, Rx count, and RxMG count to model predictions by adding them to the 3 base models. Logistic regression was performed to predict the binary outcome of hospitalization, and linear regression was chosen for total cost estimation as a continuous outcome.

Area Under the Curve (AUC) and adjusted R² (controlling for the number of variables input into the model) were calculated to measure the performance of logistic and linear regressions. The complete data set was split into a 1:1 training and testing set to conduct a 200-time bootstrapping to validate model performance. The statistics of integrated discrimination improvement (IDI) and improved R² were calculated for the enhanced model in contrast to the corresponding base models to reveal performance improvement. The 95% confidence intervals were derived to accommodate improvement uncertainty based on 200-time bootstrapping results for each measurement statistics (ie, AUC, IDI, R², improved R²). All statistical analyses were conducted using R version 3.5.1 (R Foundation for Statistical Computing, Vienna, Austria). ²⁷ The Institutional Review Board of Johns Hopkins Bloomberg School of Public Health approved this study.

Population specification

The mean age of study subjects was 38 years, and more than half of them were female (Table 1). Around 4% of the patients were hospitalized in 2012, and the average annual cost was $5600. The unique Rx count per patient, average MRCI, mean MPR, and mean 30-day PFR, calculated using EHR and/or claims data, are shown in Table 1. Twenty-two percent of the patients adhered well (MPR ≥.8) to their prescriptions in 2012.

Base model comparisons

Base model 1 demonstrated the same prediction performance for hospitalization and total cost using EHR or claims as demographic data were the same across both data sources (Table 2). Base model 2 (Charlson index) improved the model performance (AUC: .639 vs. .653, R²: .076 vs. .104; Tables 2 and 3). Base model 3 (ACG-Dx) further increased the predictive power of the models (Tables 2 and 3). Generally, claims-derived base models (except for base model 1) were superior to EHR-derived models in predicting both hospitalization and total cost of care (Figure 1).

OPEN IN VIEWER

FIG. 1.

Comparing performance of prospective predictive models of hospitalization (left) and total cost (right) after adding various medication indices: (a & b) model 1: age + sex; (c & d) model 2: age + sex + Charlson index; and (e & f) model 3: age + sex + ACG-Dx. * significant model performance improvement (ie, the estimated value of AUC or adjusted R² of the enhanced model was larger than the 95% confidence interval upper tier of the base model). Three grouped bars of each medication adherence index (eg, MPR of (a)): models trained using (left bar) merged EHR and claims data; (middle bar) claims data only; and (right bar) EHR data only. All bars: (bottom color) prediction without the medication adherence index; and, (top color) improvement in prediction after adding the medication adherence index. ACG-Dx, ACG-software diagnosis model; AUC, area under the curve; EHR, electronic health record; MPR, medication possession ratio; MRCI, medication regimen complexity index; PFR 30 days, prescription fill rate at 30 days; Rx, medications; and RxMG, ACG-software medication-defined morbidity groups.

Medication adherence indices and prospective predictions

When using claims data to predict next year's hospitalization, adding PFR to base model 1 boosted predictions slightly (AUC: .608, IDI: 6.1%), but the enhancement was negligible for the morbidity base models 2 and 3 (Table 2). MPR enhanced base model 1 and base model 2 considerably, but it had minimal impact on base model 3. MRCI improved base model 1 and base model 2 but the improvement was much smaller for base model 3. Rx count and RxMG counts showed larger boosts to all base models including the complex base model 3 (Table 2).

When using claims data to predict next year's total cost (Table 2), PFR or MPR could hardly impact any of the base models (ie, improved-R ² ranged between .000 and .011). MRCI improved total cost forecasting for all base models with less improvement as the complexity of the model increased (ie, improved-R ² of .075, .035, and .018 for base models 1, 2 and 3). Adding Rx or RxMG counts improved all base models with improved-R ² ranging between .026 and .156 (Table 2).

When using EHR data to predict next year's outcomes (Table 3), medication adherence indices improved the performance of base models analogous to claims models; however, the improvements were usually larger than claims models. PFR showed low added values to all base models (ie, IDIs ranging from .3% to 6.1% for hospitalization; and R² did not improve for cost). MPR showed some value for base models 1 and 2 but minimized improvement for base model 3 (Table 3). Similarly, MRCI improved base models 1 and 2; however, it improved base model 3 minimally. Rx and RxMG counts showed high improvements for all base models (eg, IDIs: 5.2% and 12.0% for base model 3). Using a mix of EHR and claims data to predict prospective outcomes yielded similar results (Figure 1; Supplementary Table S1).

Medication adherence indices and concurrent predictions

Overall, concurrent predictions outperformed prospective predictions (Figure 1; Supplementary Fig S2). In claims-derived models (Supplementary Table S2), PFR performed weakly in predicting hospitalization with the highest IDI at 16.4% across all base models. PFR also did not add any meaningful improvements in predicting total cost (ie, maximum improved-R ² of .001). MPR performed better in describing the same-year hospitalization after being added to base models 1 (ie, AUC: .683, IDI: 310.2%; Supplementary Table S2) and 2 (ie, AUC: .738, IDI: 31.0%), but added minimally to base model 3 (ie, AUC:.980; IDI:.5%). Adding MPR to base models to predict total cost provided similar results, with base model 1 improving the most and base model 3 improving the least.

MRCI improved concurrent hospital predictions in base models 1 and 2 (ie, IDIs: 759.6% and 39.5%) but not base model 3 (ie, IDI: .6%). MRCI modestly improved concurrent cost predictions (ie, improved R² ranging from .136 to .032). Rx and RxMG counts showed stronger prediction improvements for concurrent hospitalization and total cost, with base model 3 receiving the least boost (ie, IDI: 1.5% and .4%; improved R²: .062 and .029). Adherence measures of MPR and PFR in the concurrent year were nearly incapable of explaining variances of total cost (Supplementary Table S2).

Comparable to claims-derived models, medication adherence measures modestly improved the performance of EHR-derived models in explaining concurrent outcomes (Supplementary Table S3). But using a mix of EHR and claims data did not result in considerably different model performance improvements after adding the medication adherence indices (Supplementary Tables S1 and S4).

As a robustness check, the research team also explored the added value of medication adherence indices as a combination when added to the predictive models of utilization (ie, adding all 3 indices of MPR, MRCI, and PFR simultaneously to each model). The prediction improvement of the combination of indices was equivalent to the largest boost observed among the individual indices (results not shown), hence not showing any advantage compared to individual indices such as MRCI, which had the best performance improvement among the 3 indices.

Model improvement

This study confirmed that MRCI and MPR measures could improve utilization risk stratification models, but their effects require further validation. Consistent with previous studies, ^14,33 present study results also demonstrated that MRCI was not as predictive as the simple medication counts (eg, RxMG count). The additional characteristics of prescriptions used to calculate MRCI (eg, dosage of medications) seemed to be overshadowed by the count of medications, which also is used to construct the MRCI index. MPR, a claims-based adherence measure, has traditionally been used for risk stratification purposes ^34,35 ; however, the present study showed that additional evidence is needed to explain the value of MPR in predicting cost and utilization on a population level.

In contrast to MRCI and MPR, PFR showed little to no effect in improving the predictive power of any of the base predictive models of hospitalization and cost. In some instances, PFR also degraded the performance of such models. Nonetheless, despite the negative results, PFR provides a unique insight into the actual behavior of patients filling certain prescribed medications but not others. ^6,7 Moreover, given the loose correlation between PFR and other medication-derived indices, ^6,7 the team considers PFR as a distinctive dimension of medication adherence for population health management efforts. Subsequent studies should consolidate the methods used to measure PFR, and examine the value of a stable PFR index in specific subpopulations who might benefit the most from medication adherence interventions.

Data quality challenges

Both insurance claims and EHR data suffer from data quality challenges. ^31,36,37 Missing data and inconsistent terminologies used to code medications may affect the validity of MPR, MRCI, and PFR measures. To address the coding issues, this study was extended to examine if the added value of MRCI or MPR in improving the predictive models of utilization was affected by the underlying mapping of medication terminologies. Using different medication coding terminologies (eg, NDC vs. internal medication IDs) resulted in slightly different improvements in predicting hospitalization and total cost (Supplementary Tables S5 and S6).

Moreover, most studies using MPR and MRCI measures do not clarify the approaches, including mapping terminologies, used to calculate them, which makes it difficult to compare such measures across different populations. ³⁸ For example, MPR values are strongly affected by defining the underlying coding terminology and measuring period associated with refills. ⁵ Although MRCI seemed to not rely much on the mapping terminology, ^4,10 the weights assigned to each MRCI component could drift the score. Weighting the components often requires input from pharmacists to interpret medication instructions, which will increase the variability of measuring MRCI. Future studies should investigate how different terminologies and methods used for calculating medication adherence affect model predictions.

The research team also suspects that issues with coding terminology might have contributed to the minimal impact of PFR on utilization prediction models. PFR index is calculated using both EHR and claims data (ie, prescription data vs. filling data). Given the inconsistencies in encoding medications across EHRs (eg, using internal Rx codes instead of standardized RxNorm codes) and matching them to medication fills listed in claims (eg, NDC codes), PFR may vary depending on the underlying coding terminologies and mapping methods used to construct the index. ⁶ For example, the team observed the average 30-day PFR of the population at 81% in 2012, a relatively higher proportion compared to other studies (.67–.76^6–8). However, when repeating PFR calculations for 2013, the proportion dropped to .72 for the same population. The inconsistency of the PFR calculations across years, and the potential impact of terminology mapping issues, can be one of reasons that PFR did not improve predictive models of utilization and cost. ^{6 –8}

Limitations

This study offers novel insight into using medication adherence measures for risk stratification; however, the study has several limitations that require further exploration. First, the findings should not be generalized to all populations. For example, the study excluded older adults because of the unavailability of Medicare claims data. Given the higher number of chronic conditions, more complex medication regimens, and higher rates of utilization among elderly patients, the predictive power of medication adherence indices demands further investigation among Medicare patients. Second, the team had access to only 2 years of data; hence, results may have not revealed the long-term effects of medication nonadherence on health care utilization. Future research should evaluate the temporal changes of medication adherence indices to enrich model forecasting. Third, the team used specific terminologies used to encode medication data and generate MPR, MRCI, and PFR across all populations. Upcoming studies should explore the sensitivity of methods and terminologies used to calculate medication adherence indices on utilization prediction within specific subpopulations that may have different medication adherence trends (eg, having a specific chronic disease, belonging to a unique age group such as the pediatric population). Fourth, linked EHR and claims data were used to calculate medication adherence indices. However, availability of linked EHR and claims data for large populations of patients is limited. ¹¹ Fifth, the team only evaluated the impact of medication adherence indices on predictive models of utilization using the Charlson and ACG-Dx scores. Other studies may evaluate the added value of such indices using other predictive scores of hospitalization and total cost. Sixth, this research does not consider if the change in medication adherence is related to patient incompliance to treatment or improving clinical outcomes and thus not needing the medication anymore. Future research should attempt to distinguish and separate different underlying reasons for changes in medication adherence. Finally, the team only used structured medication data of outpatient settings to calculate MPR, MRCI, and PFR. Future studies should evaluate other aspects of medication adherence (eg, patient, disease, health system-related factors) in other health care settings (eg, inpatient) while using various types of data (eg, unstructured data). For example, adherence information may be stored in unstructured EHR data (eg, clinical notes), hence requiring text mining methods to extract adherence-related factors for target populations (eg, elderly) before calculating an adherence index. ^39,40