Bolstering Advance Care Planning Measurement Using Natural Language Processing

In Research…

…the method chosen to measure ACP documentation may influence the estimated effectiveness of an intervention. Large-scale ACP measurement tends to rely upon standardized ACP data (e.g., documents such as health care proxy forms). Although completing such forms is an important part of ACP, in isolation—as mere datapoints within the larger context of a trial—they may provide a limited view of the care a patient received.

“Forms or Free-Text?: Measuring Advance Care Planning Activity Using Electronic Health Records,” a 2023 study recently published in Journal of Pain and Symptom Management, illustrates how the methods chosen to measure ACP can impact the preference accuracy and occurrence ¹³ of ACP within the patient population. This study, a secondary analysis of data collected in a multisite pragmatic trial focused on improving ACP among seriously ill patients ¹⁴ (ACP: Promoting Effective and Aligned Communication for the Elderly [UG3AG060626]), evaluated all ACP documentation in the EHR of 435 patients.

ACP was captured in two ways: (1) review of standardized ACP documents (e.g., MOLST, HCP, Living Wills) and (2) NLP-assisted chart review of documented ACP discussions (defined as documentation of at least one conversation about goals-of-care conversations, hospice use, palliative care involvement, and/or limitations of life-sustaining treatments) within the free text of clinical notes.^9,15

In widening the scope of ACP measurement to include NLP-assisted chart review, the clinical notes revealed that significantly more patients had engaged in ACP than would have been detected by standardized data alone (NLP found ACP documentation among 55% of patients as opposed to 43% detected through standardized documents). ¹³ Furthermore, this analysis discovered a substantial number of patients (16.6%) only had ACP documents that were incorrectly completed or left incomplete, highlighting potential limitations associated with relying on only standardized documents to measure ACP. ¹⁵

The capacity for NLP to capture previously inaccessible free-text documentation highlights new opportunities for improving research, and overcoming these identified limitations, allowing us to better measure dynamic ACP activity.

In Quality Improvement…

…operational data support high-quality care by giving insights into care practices and delivery. Such data can be used to drive incentives, report back to payers, or inform quality improvement projects. NLP can capture the in-the-moment ACP conversations recorded in the notes, a data source previously inaccessible in rapid evaluations of ACP. With these data in hand, hospitals and systems can better understand institutional ACP practices and identify potential areas for clinical improvement.

The promise of NLP for clinical settings is showcased by on ongoing project at Dana-Farber Cancer Institute (DFCI). Patients with serious illnesses at DFCI can reach a poor prognosis, a clinical designation made by the patient's care team. ¹⁶ Once patients have reached this point, their clinicians will receive nudges reminding them to engage the patients in ACP and document it in the EHR.

As a part of an effort to understand and improve this practice, data from a sample of 91 patients from 12 oncology disease centers at DFCI were analyzed to determine the frequency of goals-of-care conversations before their clinician receiving a nudge to document. Goals-of-care conversations were measured through the completion of a standardized structured serious illness conversation EHR tab and through NLP-assisted chart review of clinical notes.

Among the 91 patients, 62 had at least one goals-of-care conversation documented either in the EHR tab or in the free-text of clinical notes. Twenty-one patients had ACP documentation in both the tab and NLP-assisted chart review, 5 patients had documentation only through the tab, and 36 patients had documentation solely identified by NLP-assisted chart review.

A major benefit of the ACP tab is that the information is readily available for clinicians caring for patients, for example, in the emergency room, making it easier to provide ongoing goals-of-care conversations across care settings. However, being able to identify the 36 patients who had ACP documented in free-text clinical notes, but not in the ACP tab, identifies a different quality improvement target than cases who do not have any conversation documented. Future study should identify novel ways to efficiently use NLP at the point of care to find ACP documentation in unstructured EHR fields such as clinical notes.

In Clinical Settings…

…successful ACP is characterized by engaged informed participation from patients and families. The data used to benchmark the success of ACP-focused interventions are often a product of clinicians' activities, such as documentation or scanning ACP-related forms into the EHR. Though these data are valuable, they may not be fully representative of a patient's experience of engaging in ACP. Rather, integrating patient-reported ACP outcomes alongside evidence of clinician documentation is one way of better understanding how patients perceive and participate in ACP.

The results of a recent randomized trial ¹⁷ focused on improving serious illness conversations among patients with metastatic breast cancer showcased the power of using NLP to investigate clinician ACP documentation alongside patient-reported outcomes. In the context of end-of-life care, the trial collected patient responses to the question: “Have you and your doctor discussed any particular wishes you have about the care you would want to receive if you were dying? (Yes/No)” and identified goals-of-care conversation documentation in clinical notes through NLP-assisted chart review.

Collecting both these data simultaneously provided an opportunity to compare NLP-detected clinical documentation of goals-of-care discussions with patient-reported engagement in goals-focused end-of-life conversations, a comparison that had the power to reveal latent perspectives embedded in patient- and clinician-centered data. Among intervention patients, only 38% reported having had a conversation about wishes for care with their oncologist. Yet, 67% of patients were found to have documentation of an end-of-life care discussion in their chart. This 29% difference in clinician documentation and patient-reported discussions points to differences in patient and clinician understandings and perceptions of ACP communication.

Some of the differences may be due to survey methodology. For example, in this study, despite receiving the survey containing ACP-relevant questions every four weeks, the respondents may have only answered the survey once, at a time point when no goals-of-care conversation had yet been held. However, having both timely patient and clinician data together allows us to understand the complexity of ACP and the limitations of our measures and proxies, inform care in a nuanced way, and identify questions about potentially different definitions or thresholds regarding these conversations to inform ongoing improvements in care. NLP methods allow us to capture the medical record documentation in a timely manner to inform integrated assessments of ACP.

Conclusion

Together, these three example applications show the importance of integrating NLP into ACP measurement. They show how NLP-assisted chart review can provide an efficient approach to measure ACP outcomes accurately and more comprehensively, how it can be used to enhance ACP quality improvement, and how it can help garner important insights about patient perceptions of ACP. Rule-based NLP, such as that used in the example applications, is far from a perfect way of measuring or understanding all that ACP encompasses. Given that it relies on text documentation, NLP is limited by what is written in a patient's chart, the quality of that documentation, its authorship reflecting clinician rather than patient testimony, and how human users of NLP-assisted chart review software utilize the tool.

Inherent limitations to rule-based NLP also include that searches only take place regarding defined targets (e.g., domains relevant to ACP are defined by the study team; different clinicians/researchers may disagree on what is important to include), and that keyword libraries of search terms are not exhaustive. Even still, NLP offers a notable opportunity for clinicians and researchers seeking a more nuanced understanding of ACP's issues and incongruences. In the near future, NLP could be applied to audio recordings of conversations between patients and their clinicians.

As speech-to-text models improve and audio data become increasingly available with the rise of telehealth, NLP may soon become an option for recording and analyzing in-the-moment ACP conversations, bypassing the testimonial limitations of clinician-authored clinical notes (though transcription biases will be the important limitation to consider). Ultimately, NLP is a technology that continues to grow as the data and context evolve around it.