Enhancing Clinical Note Generation with ICD-10,Clinical Ontology Knowledge Graphs,and Chain-of-Thought Prompting Using GPT-4

Abstract

In the past decade, a surge in the amount of electronic health record (EHR) data in the United States has been attributed to a favorable policy environment created by the Health Information Technology for Economic and Clinical Health (HITECH) Act of 2009 and the 21st Century Cures Act of 2016. Clinical notes for patients’ assessments, diagnoses, and treatments are captured in these EHRs in free-form text by physicians, who spend a considerable amount of time entering and editing them. Manually writing clinical notes takes a considerable amount of a doctor’s valuable time, increasing the patient’s waiting time and possibly delaying diagnoses. Large language models (LLMs) possess the ability to generate news articles that closely resemble human-written ones. We investigate the usage of Chain-of-Thought (CoT) prompt engineering to improve the LLM’s response in clinical note generation. In our prompts, we use as input International Classification of Diseases codes and basic patient information. We investigate a strategy that combines the traditional CoT with semantic search results to improve the quality of generated clinical notes. Additionally, we infuse a knowledge graph built from clinical ontology to further enrich the domain-specific knowledge of generated clinical notes. We test our prompting technique on six clinical cases from the CodiEsp test dataset using GPT-4, and our results show that it outperformed the clinical notes generated by standard one-shot prompts.

Keywords

Chain-of-Thought clinical note generation generative AI International Classification of Diseases (ICD) codes knowledge graph large language models

1. INTRODUCTION

In the past decade, there has been a surge in the amount of electronic health record (EHR) data in the United States. In 2008, only 42% of physicians had access to an EHR (Office of the National Coordinator for Health Information Technology, 2025). This figure has now risen to 88% as reported in 2021 (Office of the National Coordinator for Health Information Technology, 2025). This increase may be attributed to a favorable policy environment created by the Health Information Technology for Economic and Clinical Health (HITECH) Act of 2009 (Burde, 2011) and the 21st Century Cures Act of 2016 (U.S. Food and Drug Administration, 2020).

Clinical notes for patients’ assessments, diagnoses, and treatments are captured in these EHRs in free-form text by physicians, who spend a considerable amount of time entering them into computers. These notes offer valuable insights based on real-time observed data, which have been shown to enhance the predictive capabilities of medical decision-making models (Classen et al., 2018).

Despite the rich information contained in these notes, it is likely that some details are excluded from publicly available large language models (LLMs) due to restrictions on access to their content, a consequence of the Health Insurance Portability and Accountability Act (HIPAA) of 1996 (Moore and Frye, 2019). HIPAA plays a crucial role in safeguarding the privacy and security of patients’ protected health information (PHI) in the context of clinical notes. One key outcome of HIPAA is the standardization of clinical text de-identification, which allows sensitive patient information to be stripped from records so the data can be safely reused. This de-identification process itself is a complex natural language processing (NLP) task, often requiring substantial time (Negash et al., 2023). As a result, datasets used for pretraining LLMs are typically assumed to exclude PHI.

Despite these technological and regulatory advancements, the effectiveness of clinical note documentation has not kept pace (Arndt et al., 2017; Gellert et al., 2015; Kroth et al., 2019). Among the factors contributing to this stagnation is physician burnout, a growing concern that impacts both provider well-being and patient care. Physicians often experience emotional exhaustion, reduced motivation, and a sense of detachment from their patients, largely due to the demanding and high-pressure nature of their work. One significant driver of this burnout is the administrative burden associated with EHRs, particularly the excessive time spent on data entry and clinical documentation (Arndt et al., 2017; Gellert et al., 2015; Kroth et al., 2019; Liu, 2018).

The medical scribe industry has emerged to handle the burdensome documentation tasks behind the scenes (Gellert et al., 2015). However, relying on nonprofessional scribes presents challenges, as these unlicensed individuals, working under clinician supervision, may lack the medical expertise necessary for accurate EHR documentation, reflecting broader gaps in understanding of their impact and effectiveness (Ash et al., 2021). To address the physician burnout challenge, we focus our attention on LLMs given that remarkable progress has been made in recent years with some observations suggesting that they exhibit more powerful reasoning abilities as the model size increases (Brown et al., 2020). Clinicians show growing interest in adopting LLMs for medical question answering, clinical note summarization, diagnostic assistance, and clinical note documentation, appreciating their potential to reduce documentation burden and improve efficiency, as highlighted in recent reviews (Busch et al., 2025; Maity and Saikia, 2025; Vrdoljak et al., 2025).

In this article, we investigate the use of LLMs to generate medical notes. We begin by leveraging International Classification of Diseases (ICD) codes as initial prompts for the LLM. To enhance the quality and relevance of the generated notes, we apply Chain-of-Thought (CoT) prompting, using example clinical cases to guide GPT-4’s reasoning process. In addition, we explore the integration of SNOMED CT OWL expressions as a clinical ontology knowledge graph (KG) prompt. The KG offers a structured, machine-readable representation of medical concepts and their interrelationships, such as disease hierarchies, symptom-diagnosis associations, treatment pathways, and comorbid conditions. We evaluated the performance of GPT-4 in generating the patient’s current history of present illness (HPI) based on specific task instructions, using diagnosis codes and relevant patient information as input. To benchmark our approach, we apply the CoT prompting technique to six clinical cases from the CodiEsp test dataset.

2. RELATED WORK

The rapid advancements in LLMs have greatly enhanced their ability to comprehend patterns and relationships between words and phrases more effectively by developing a general understanding of grammar, syntax, and semantic relationships to generate text, bringing their output closer to human-level quality in areas of news compositions, story generation, and code generation (Wu et al., 2023). LLMs like GPT-3 (Brown et al., 2020) and GPT-4 (OpenAI, 2023) have demonstrated impressive performance on downstream NLP tasks, even in zero-shot and few-shot settings. With its substantial capacity, it possesses the ability to generate news articles that closely resemble human-written ones, making it difficult to distinguish between the two (Brown et al., 2020). This poses a particular challenge in detecting LLM-generated text, which is crucial for ensuring responsible AI governance (Wu et al., 2023). GPT-4 is said to adhere more closely to guardrails, ensuring a higher level of responsible text generation.

Prompt engineering (or In-Context Prompting) (Wei et al., 2022; Ye et al., 2023) emerged as a recent field focused on crafting and refining prompts to effectively harness techniques aimed at interacting with LLMs to guide their behavior toward specific goals, without making changes to the model weights. Since the recent releases of LLMs, Google researchers recently revolutionized a prompting strategy for solving word problems across five different LLMs (Zhang et al., 2023). Several prompt engineering techniques (Li et al., 2022; Sahoo et al., 2024) have emerged and significantly improved the performance of LLMs on many natural language generation tasks. Recent studies, such as CoT (Wei et al., 2022; Zhang et al., 2023), Tree-of-Thought (ToT) (Long, 2023; Yao et al., 2023a), and Graph-of-Thought (GoT) (Besta et al., 2024; Yao et al., 2023b), have been shown to improve the reasoning and accuracy performance of LLMs by providing rationales for a given word or phrase (Besta et al., 2024; Long, 2023; Wei et al., 2022; Yao et al., 2023a, 2023b; Zhang et al., 2023). Although self-verification (Weng et al., 2023) and self-consistency (Wang et al., 2022) have enhanced performance in CoT prompting, recent prompting techniques such as ToT (Long, 2023; Yao et al., 2023a) and GoT (Besta et al., 2024; Yao et al., 2023b) have shown improvement, though their effectiveness is still being assessed. CoT (Li’evin et al., 2022) has demonstrated that LLMs are capable of reasoning through multiple-choice questions on medical board exams. For our purposes, can it reason about ICD codes along with some patient information to generate clinical notes?

Previous endeavors have demonstrated that employing an attention mechanism in a multi-label classification task can effectively yield ICD codes from clinical notes (Makohon and Li, 2021) and shows that numerous prior research endeavors have revolved around classifying ICD codes using clinical notes as their primary input data (Li et al., 2019; Niu et al., 2023; Wu et al., 2022). Our work in this article is to reverse this process by generating comprehensive clinical notes, guided by provided ICD codes and supplemented basic patient information using instructional prompting techniques. In a recent study (Lee and Lindsey, 2024), LLMs were investigated using zero-shot prompting to predict ICD-10 codes. ICD-10 codes were provided in their prompt: “Predict these ICD-10 codes to the best of your ability” without any patient information or a clear instruction task to generate clinical notes. Based on their outputs, the LLMs’ outputs predicted just the ICD code titles, not actual patient clinical notes.

Additionally, recent studies have explored the use of LLMs for generating clinical notes with the use of prompt engineering. These include leveraging LLMs to convert transcribed interactions into structured notes through structured prompting and integration of supplementary data for improved quality (Biswas and Talukdar, 2024), developing a specialized medical LLM to understand and summarize medical conversations using zero-shot prompts for note generation (Yuan et al., 2024), and providing rapid access to medical information via a chatbot that utilizes a predefined system prompt to perform contextual searches and Retrieval-Augmented Generation (RAG) techniques (Leong et al., 2024). Some of the challenges in clinical note generation through use of LLMs are captured in this study (Wang et al., 2024), which highlights the feasibility of training efficient open-source LLMs for clinical note generation, with opportunities for further exploration in domain adaptation, data selection, and reinforcement learning from human feedback (RLHF). RLHF helps align LLMs with human preferences and can be applied in two ways: outcome-supervised, which focuses on improving the overall quality of the text, and process-supervised, which provides more detailed guidance on specific text components, such as reasoning steps, as seen in approaches like InstructGPT (Li et al., 2022).

We conduct experiments on the closed-source GPT-4 using semantic searches and the CoT prompting technique to query similar clinical cases based on the given ICD codes or text references. We further explore the integration of a clinical ontology KG into the prompts to enrich the LLM input with well-defined medical relationships and constraints. To the best of our knowledge, we are the first to perform experiments of this kind using diagnosis codes (ICD codes) as input along with basic patient information to generate clinical notes using LLMs and CoT prompting instructions. We seek to answer our research question: Can LLMs reason about ICD codes to guide the generation of clinical notes using instruction prompting?

3. METHODOLOGY

This article explores a method for guiding the generation of clinical notes using an LLM while providing patient information, ICD codes, and task instructions as input, as well as medical relationships between ICD codes and constraints derived from a KG, utilizing CoT with examples of clinical note diagnoses returned by semantic search.

3.1. Semantic search and clinical cases

CodiEsp, introduced during the CodiEsp track for CLEF eHealth 2020, is recognized as a gold-standard annotated data source (Miranda-Escalada1 et al., 2020). The dataset comprised 1 000 clinical cases, where the clinical notes are translated from Spanish to English. It encompasses both ICD-10-CM and PCS codes, distributed across three randomly sampled datasets: the Training set contains 500 clinical cases. The Development (validation) and the Test set each contain 250 clinical cases. The text-reference column consists of text used during the annotated process using the Brat visualization tool (Stenetorp et al., 2012). Hereafter, we will refer to the text-reference column as the Text Reference.

We combine CodiEsp’s training and validation datasets (750) for our semantic search embedding query, while reserving the test dataset (250) for selecting six clinical case samples for evaluating ground truth against generative text. We converted the texts in the combined dataset of 750 clinical cases into numerical vector representations with OpenAI’s text embedding model (text-embedding-3-small). Our objective is to leverage the embedding-driven retrieval to tap into the rich semantic features present in other clinical cases. This is achieved through the use of query ICD codes or text references, which facilitates the provision of clinical case examples for use as “thoughts” in our CoT prompt. As we will demonstrate, these semantic searches provide an efficient approach to identifying examples resembling the examples in the prompts. The embedded query (ICD code or text reference) is used to pinpoint the most relevant clinical cases by assessing their proximity within the embedding space, utilizing document similarity to rank and present the top-n most suitable clinical cases. For each query, the cosine similarity is used to identify the top-n most similar clinical cases. Figure 1 shows an illustration of the semantic search query used to search for similarities within the embedding space. The top illustration shows the ICD code query. The bottom illustration shows the text reference query. The violet highlight depicts the type of inputs for semantic search queries using examples. The query result provides the top-10 similar clinical cases for use as a prompt thought.

FIG. 1.

An illustration of the semantic search query used to search for similarities within the embedding space.

We randomly selected six clinical cases with less than 200 words from the test dataset that contain 1 or more ICD codes or text references. The breakdown of the CodiEsp samples along with their ICD codes, text references, and word count for each clinical case are shown in Tables 1 and 2.

Table 1.

Clinical Case Samples (Counts)

Clinical case	CodiEsp ArticleID	ICD code	Text reference	Clinical note
A	S0213-12852003000600002-1	2	2	99
B	S1130-05582017000100031-1	1	1	113
C	S1130-01082008001000008-1	4	5	128
D	S1130-01082009000500011-1	9	9	107
E	S1130-01082008000100009-1	10	11	107
F	S1130-01082006001000017-1	9	9	90

ICD, International Classification of Diseases.

3.2. Knowledge graph and clinical cases

We utilized the SNOMED CT to ICD-10-CM map (National Library of Medicine, 2021) as the foundation for constructing our KG prompt. To enrich this mapping, we curated data from the SNOMED CT OWL ontology reference set (National Library of Medicine, 2019), which provides the OWL expressions and the accompanying description file (National Library of Medicine, 2019), which provides human-readable terms that convey the intended meaning of each concept. By integrating SNOMED CT concepts with their corresponding ICD-10-CM codes, OWL expressions, and descriptive labels, we established clinically meaningful links between the ICD codes, ontological definitions, and semantic representations.

Figure 2 illustrates the OWL expression derived from SNOMED CT and corresponds to the ICD-10 code K08.109 (Complete loss of teeth, unspecified cause) within Clinical Case B. It is used to formalize the concept Edentulous (finding) (278650002) using a combination of ontological relationships that capture its clinical semantics. Specifically, the expression indicates the absence (Absent [2667000]) of the observable entity Tooth presence (278652005), located at the finding site All teeth (1162715001). These attributes are grouped under the Role group (609096000), reflecting a normalized structure for reasoning over clinical findings.

FIG. 2.

An example of the OWL expression prompt for CoT KG prompting. CoT, Chain-of-Thought; KG, knowledge graph.

The OWL axioms capture logical relationships and attributes among SNOMED CT concepts, enabling more advanced, ontology-driven reasoning. This alignment results in a structured and semantically rich prompt for GPT-4, facilitating the generation of a clinically accurate and logically coherent KG that connects diagnoses, procedures, and broader biomedical relationships. From the listing of ICD-10 codes within our curated data, we were able to match the ICD code K08.109 (Complete loss of teeth, unspecified cause, unspecified class) to the only ICD code associated with Clinical Case B in Table 2.

Table 2.

Clinical Case Samples (Details)

Clinical case	ICD code	Text reference	Age	Gender
A	R52, K08.89	Pain, toothache	56	Female
B	K08.109	Edentulism	54	Female
C	E11.9 M79.81 R05 T14.8	Abdominal wall hematoma, cough, DM ID, hematoma, right anterior rectal hematoma	61	Male
D	B00.9 B99.9 C15.9 F17.210 K22.2 R13.10 R50.9 R60.9 R63.4	Dysphagia, edema, esophageal carcinoma, fever, herpes simplex, infection, partial narrowing of the esophageal lumen, smoker, weightloss	67	Male
E	B99.9 C81.90 G93.40 K72.00 K72.90 K92.1 K92.2 R58 R69 R74.0	Acute liver failure, digestive bleeding, disease, elevated transaminases, encephalopathy, hemorrhage, hepatic encephalopathy, Hodgkin’s disease, infection, liver failure, manes	16	Male
F	I10 K42.9 K56.60 K92.2 N80.5 R10.32 R10.814 R19.8 R58	AHT, colon endometriosis, colon hemorrhage, hemorrhage, pain in FII, pain on deep palpation in FII, rectal urgency, sigma level stricture imaging, umbilical hernia	42	Female

3.3. Prompt format

Our standard prompt, referred to as the baseline, is formatted as task instructions with one-shot prompt to generate the HPI clinical notes based on the given ICD codes shown in Figure 3. Our CoT semantic search (CoT prompting) is formatted as instructions to guide the output of the language model by controlling its generated text, shown in Figure 4. In the CoT instruction prompt, each experiment contains the ground-truth clinical case’s ICD codes along with basic patient information.

FIG. 3.

Baseline (one-shot) prompt.

FIG. 4.

CoT Prompt (leveraging the ICD code semantic search query). ICD, International Classification of Diseases.

In addition, the top-10 similar clinical cases are provided by semantic search query (based on the ground-truth ICD codes or text references), which uses contextualized word embeddings and the cosine similarity function to find related clinical cases, and are provided as prompts. These inputs act as rationales, enabling the LLM to learn and generate the intended clinical notes based on the provided ICD codes.

Our CoT prompting takes inputs, such as task instruction, ICD codes for the diagnosis and/or procedure, examples of similar clinical cases using the semantic search (ICD codes or text references) query, and basic patient information (age and gender). Both CoT KG (Fig. 5) and CoT Semantic Search KG (Fig. 6) prompting techniques utilize the SNOMED CT KG derived from OWL expressions, as illustrated in Figure 2. The key difference between the two lies in the use of semantic search: CoT KG does not incorporate semantic search, whereas CoT SS KG does.

FIG. 5.

CoT Prompt (leveraging the knowledge graph).

FIG. 6.

CoT Prompt (leveraging the semantic search query and knowledge graph).

3.4. Metrics

Cosine distance is the complement of cosine similarity, which measures the angular difference between two vector representations in a multidimensional space. It is a mathematical function that quantifies the degree of dissimilarity between two vectors based on their orientation rather than their magnitude. The cosine distance formula is defined as:

Cosine Distance = 1 - \frac{A \cdot B}{‖ A ‖ ‖ B ‖} = 1 - \frac{\sum_{i = 1}^{n} A_{i} B_{i}}{\sqrt{\sum_{i = 1}^{n} A_{i}^{2}} \sqrt{\sum_{i = 1}^{n} B_{i}^{2}}},

(1)

where

A \cdot B

is the dot product of the sentence vectors

A

and

B

‖ A ‖

and

‖ B ‖

are the magnitudes (norms) of the vectors, and the result gives a measure of the angular distance between the vectors.

Transformer-based models, such as Bidirectional Encoder Representations from Transformers (BERT) (Devlin et al., 2019), capture both syntactic and semantic relationships between words by generating contextualized word embeddings. To assess the similarity between machine-generated and ground-truth documents, we use BERT. Both documents are processed through the bert-large-cased model to obtain embeddings, which are then used to calculate sentence similarities. The two steps are the following. 1.

Using the special “classification” [CLS] token of each sentence. The [CLS] token in BERT serves as a holistic representation of the input sequence. The output of [CLS] is inferred by all other words in this sentence. This implies that the [CLS] contains all information in other words, which makes [CLS] a representation for sentence-level classification.

Calculating the MEAN of the sentence embeddings provides a way to quantify the overall cosine distance between the sentences based on semantic meaning.

3.5. Experiments

For our experiments, we use OpenAI’s GPT-4 (gpt-4) model for all experiments, with the following parameters: seed (123), temperature (0), top_p (0.000001), frequency_penalty (0), and presence_penalty (0). We establish a baseline for our results using the standard one-shot prompt and compare it against the results from our CoT prompts, which utilize a semantic search query based on the provided ICD codes or text references from ground-truth clinical case samples. Basic patient information is provided as supplementary prompts. Our semantic search query introduces an extra prompt, which includes the top-10 most similar clinical cases based on the given ICD codes or text references.

For each clinical case sample, we collect results from 100 API calls to GPT-4, with each call treated as an independent interaction. This ensures there is no memory or history from previous interactions, making each response independent. The clinical case’s top-10 relatedness scores from the semantic search query are presented in Table 3. These scores are calculated using cosine distance, which evaluates spatial proximity to identify the top-10 most similar clinical cases based on the provided ICD codes or text references.

Table 3.
Semantic Search Query (Top-10 Relatedness Scores)

Case# ICD code relatedness Text reference relatedness

A 0.762, 0.754, 0.729, 0.724, 0.718, 0.715, 0.708, 0.695, 0.687, 0.683 0.563, 0.483, 0.478, 0.469, 0.462, 0.458, 0.433, 0.431, 0.429, 0.417

B 0.720, 0.717, 0.711, 0.701, 0.677, 0.672, 0.653, 0.644, 0.635, 0.630 0.469, 0.434, 0.411, 0.387, 0.380, 0.361, 0.359, 0.338, 0.336, 0.334

C 0.812, 0.780, 0.775, 0.768, 0.768, 0.760, 0.759, 0.754, 0.754, 0.751 0.601, 0.553, 0.545, 0.522, 0.521, 0.520, 0.520, 0.517, 0.512, 0.511

D 0.803, 0.802, 0.798, 0.796, 0.787, 0.783, 0.783, 0.782, 0.782, 0.782 0.630, 0.613, 0.606, 0.568, 0.565, 0.551, 0.547, 0.537, 0.524, 0.522

E 0.846, 0.807, 0.805, 0.795, 0.786, 0.774, 0.770, 0.769, 0.767, 0.757 0.637, 0.632, 0.623, 0.613, 0.610, 0.609, 0.601, 0.598, 0.595, 0.594

F 0.797, 0.775, 0.764, 0.763, 0.759, 0.752, 0.744, 0.741, 0.739, 0.739 0.654, 0.646, 0.645, 0.629, 0.627, 0.625, 0.624, 0.617, 0.615, 0.610

Case#	ICD code relatedness	Text reference relatedness
A	0.762, 0.754, 0.729, 0.724, 0.718, 0.715, 0.708, 0.695, 0.687, 0.683	0.563, 0.483, 0.478, 0.469, 0.462, 0.458, 0.433, 0.431, 0.429, 0.417
B	0.720, 0.717, 0.711, 0.701, 0.677, 0.672, 0.653, 0.644, 0.635, 0.630	0.469, 0.434, 0.411, 0.387, 0.380, 0.361, 0.359, 0.338, 0.336, 0.334
C	0.812, 0.780, 0.775, 0.768, 0.768, 0.760, 0.759, 0.754, 0.754, 0.751	0.601, 0.553, 0.545, 0.522, 0.521, 0.520, 0.520, 0.517, 0.512, 0.511
D	0.803, 0.802, 0.798, 0.796, 0.787, 0.783, 0.783, 0.782, 0.782, 0.782	0.630, 0.613, 0.606, 0.568, 0.565, 0.551, 0.547, 0.537, 0.524, 0.522
E	0.846, 0.807, 0.805, 0.795, 0.786, 0.774, 0.770, 0.769, 0.767, 0.757	0.637, 0.632, 0.623, 0.613, 0.610, 0.609, 0.601, 0.598, 0.595, 0.594
F	0.797, 0.775, 0.764, 0.763, 0.759, 0.752, 0.744, 0.741, 0.739, 0.739	0.654, 0.646, 0.645, 0.629, 0.627, 0.625, 0.624, 0.617, 0.615, 0.610

4. RESULTS AND DISCUSSION

We evaluate our prompting technique using cosine distance as the primary metric, comparing the generated text against the ground truth. We utilized Kernel Density Estimation (KDE) plots to visualize the distribution of our results and support our discussion, box plots to show the comparison of our CoT prompting against the various CoT KG approaches, and Uniform Manifold Approximation and Projection (UMAP) (McInnes et al., 2018) plots to show a visual comparison of noun distributions between Ground Truth and various generation methods.

4.1. KDE results and discussion

We present the results using KDE plots, which include Bootstrap Confidence Intervals (BCIs) for both sentence-level [CLS] and mean scores. BCIs were used to estimate the uncertainty of results by resampling the data with replacement, allowing for robust, distribution-free confidence estimates. This provides a more reliable measure of variability, especially when parametric assumptions may not hold. KDE plots offer a smooth, continuous representation of the underlying data distribution, allowing for clearer identification of patterns, peaks, and differences between groups. Unlike histograms, which can be sensitive to bin size and may obscure subtle features in the data, KDE provides a more refined view that enhances interpretability. This visualization enables a direct comparison between our CoT prompting approach and the baseline one-shot prompt. The distributional analysis shows that CoT prompting, enhanced with semantic search queries such as ICD codes and relevant text references, consistently improves the model’s ability to capture underlying clinical reasoning. This includes better representation of structured information from ICD codes and contextual patient details, leading to superior performance compared with the baseline.

The KDE with BCI plots (Figs. 7–10) reveal a leftward shift in the peaks for the CoT semantic search prompting technique, indicating that its distribution has lower values compared with the baseline prompts. This shift highlights notable differences in semantic alignment with ground-truth clinical cases. The inclusion of BCIs provides statistical validation, enhancing the robustness and interpretability of these findings. However, as shown in Figure 10, we observe that in clinical case A, the baseline prompt works better than the CoT text reference prompt. This is due to the presence of two text references (pain, toothache). In contrast, in Figure 8, the two ICD codes (K08.89, R52) in clinical case A perform better than the baseline prompt.

FIG. 7.

Illustration of the six clinical cases using KDE with BCIs to compare the baseline and CoT ICD code semantic search, based on cosine distance score of sentence-level [CLS]. BCI, Bootstrap Confidence Interval; KDE, Kernel Density Estimation.

FIG. 8.

Illustration of the six clinical cases using KDE with BCIs to compare the baseline and CoT text reference semantic search, based on cosine distance score of sentence-level [CLS].

FIG. 9.

Illustration of the six clinical cases using KDE with BCIs to compare the baseline and CoT ICD code semantic search, based on cosine distance score of sentence-level MEAN.

FIG. 10.

Illustration of the six clinical cases using KDE with BCIs to compare the baseline and CoT text reference semantic search, based on cosine distance score of sentence-level MEAN.

One limitation we observed with CoT prompting is its sensitivity to ambiguous or underspecified clinical terms, such as “pain” or “toothache.” These terms are semantically broad and often lack contextual qualifiers like location, severity, or duration, which are essential for generating accurate and clinically useful notes. For example, the ICD-10 code R52 (Pain, unspecified) is generally considered too broad for clinical documentation because it fails to capture the underlying source or etiology of the pain and may even raise compliance concerns. In contrast, when the record specifies dental pain or pain due to a tooth disorder, K08.89 (Other specified disorders of teeth and supporting structures) is the more clinically meaningful choice, making R52 redundant unless no other ICD code provides greater specificity. In such edge cases, CoT prompting can either propagate the ambiguity of the input or reflect limitations in the annotated data, defaulting to overly broad codes when faced with vague terms.

4.2. CoT KG results and discussion

To summarize and compare the performance across the baseline and CoT prompting methods, we use box plots to visualize the results collected from 250 API calls to GPT-4 for our CoT KG setup. These plots provide a concise representation of the output distribution, highlighting key statistical features such as the median, interquartile range, and potential outliers, thereby facilitating an informative comparison of variability and central tendency across prompting methods.

In Figure 11, the results of [CLS] (left) show that CoT prompting yields the lowest cosine distances, while CoT KG leads to the highest distances with the greatest variance. In the [MEAN] results (right), CoT prompting achieves the lowest distances, and all prompts exhibit tighter distributions and improved semantic similarity. These findings suggest that, although the ontology-based KG introduces greater output diversity, it does not significantly enhance the precision or semantic accuracy of the generated medical notes.

FIG. 11.

Illustration of the KG results against the baseline and CoT prompting for sentence-level CLS and MEAN.

4.3. Part-of-speech tagging (nouns and PROPN) results and discussion

To better understand why CoT prompting outperforms both CoT KG approaches in aligning ground-truth and generated outputs, we applied UMAP (McInnes et al., 2018) along with the transformer-based model SciBERT (Beltagy et al., 2019) to visualize noun embeddings across different prompting strategies. For each of five sampled instances, we generated a separate plot comparing all four prompting techniques. Each technique’s visualization was based on a single randomly selected run from its 250 generations. The UMAP dimensionality reduction technique allows us to project high-dimensional token-level embeddings into a 2D-space, facilitating direct visual comparison of distributions. We focused specifically on part-of-speech (PoS) tagging (NOUN and PROPN), as they often carry the most critical common and proper noun meaning within the tokens. The UMAP parameters—n_neighbors (15), min_dist (0.1), and random_state (1)—were used to emphasize local structure and ensure reproducibility.

In Figures 12–16, the illustrations show the UMAP distribution of NOUN and PROPN (proper noun) tokens from ground-truth and machine-generated outputs, where each subplot shows one representative result for a different prompt technique. In each figure, the Baseline output displays broader dispersion and limited overlap between generated and ground-truth tokens, indicating greater semantic divergence. In contrast, the CoT-based prompting techniques demonstrate tighter clustering and increased overlap, suggesting improved semantic alignment of the knowledge. Like the Baseline, the CoT KG plots show moderate overlap but exhibit some spatial drift between token clusters, which may reflect subtle content hallucination shifts. The CoT prompting technique appears to leverage the clinical knowledge from the semantic search examples.

FIG. 12.

UMAP token projection for sample instance #1. UMAP, Uniform Manifold Approximation and Projection.

4.4. Clinical case example

Figure 17 shows that some words are present in the ground truth but missing from the generated text. It also reveals that ICD-10 codes K08.89 (pain) and R52 (toothache) in Table 2 are linked to terms like “oral,” “cavity,” and “discomfort,” which appear in the generated text, all related to oral health. Both texts include “A 54-year-old female,” confirming alignment in basic patient information. Figure 18 presents a pairwise token-level cosine similarity analysis between the ground-truth text and generated text, with similarity distinguished by height and color. Each bar represents the cosine similarity of a token from the ground truth (x-axis) with its corresponding token in the generated output. Bar colors indicate similarity levels: blue for high similarity, gray for moderate similarity, and red for low similarity. Higher bars reflect a stronger lexical or semantic alignment between the ground-truth and the generated text. Several tokens such as “woman,” “mild,” and “pain” fall within moderate to high similarity (0.3–0.6), suggesting that the model captured clinical terms. Tokens, such as “inspection,” “crown,” and “exam” in the (0.2–0.4) range, indicate partial alignment. The red bars (right) correspond to lower similarity scores, indicating a greater deviation from the ground-truth text.

4.5. Results summary

Overall, by using visual comparisons, such as overlaying KDE plots for all clinical cases and prompting techniques, we were able to assess the extent of the observed shifts. These plots, along with the cosine distance measurements, confirm that the differences in semantic similarity performance between the techniques are both statistically significant and practically meaningful. For instance, clinical cases A and B exhibit a significant leftward shift in the peaks (Figs. 7 and 10) for the ICD code semantic search, indicating that their distributions have lower values compared with the baseline. The ICD code prompting technique appears to outperform the Text Reference, likely due to higher relatedness scores for the clinical case examples in Table 3. From observation, these visual insights combined with the precision offered by the BCIs show that both CoT prompting technique results distributions differ from the baseline standard prompt, which suggests that this technique can guide the generation of clinical notes through instruction prompts using similar clinical cases.

5. CONCLUSION AND FUTURE WORK

Our study constructs the HPI clinical notes using CoT prompting with ICD codes, clinical case examples, clinical ontology KG, and basic patient information. Experiments were conducted across various clinical cases (clinical cases with 1 ICD code, 2 ICD codes, and several cases with multiple ICD codes), comparing results obtained from a baseline one-shot prompt and two CoT prompting templates. Through cosine distance analysis, we compared the generated text with ground-truth text, addressing whether LLMs can effectively reason about ICD codes to produce clinical notes using CoT prompting.

Our analysis concludes that the GPT-4 LLM is capable of reasoning about ICD codes using our CoT semantic search prompting techniques over the baseline one-shot prompt to produce clinical notes. Also, comparing an EHR to a single ground truth may not be effective, as different doctors write EHRs differently. For this reason, human evaluation, either through expert review, structured validation protocols, or human-in-the-loop feedback, should complement automatic metrics to ensure clinical relevance, factual accuracy, and real-world utility. This perspective is supported by recent work. Tam et al. (2024) propose a framework for human evaluation of LLMs in health care, highlighting that automatic metrics alone cannot capture critical dimensions such as safety and clinical appropriateness. Similarly, Chiu and Chung (2025) outline a protocol for human evaluation of generative AI chatbots in clinical consultations, stressing the importance of domain-specific, structured human validation. Designing a human-in-the-loop evaluation phase to complement our current quantitative analyses will be one of our future directions.

Although our results show that the clinical ontology KG can improve over the baseline, it does not further contribute to more precise generation of medical notes when CoT with semantic search is incorporated, mainly due to the increased variability of output generated. How to effectively incorporate clinical ontology KG into prompting and how to optimally extract and align model outputs with respect to the KG will be the key focus of our future research. The code and materials used for model replication are available at our GitHub repository (https://github.com/Moh-najafi14/COT-KG).

To enhance the prompting techniques, we propose some follow-up studies not only in areas of using other instruction prompting techniques, but in these areas as well to enrich LLMs reasoning: 1.

CoT Prompting using Patient’s Past Medical History

The Medical Information Mart for Intensive Care (MIMIC-III) (Johnson et al., 2016) offers clinical data on 30-day ICU readmissions, allowing past medical history and admission notes to guide model predictions for future visits.

Fine-Tune an LLM to become more biased toward the Physician’s output

LLMs are prone to biases from training data, but they can be fine-tuned for individual physicians using personalized data. This adjustment, achieved through instruction prompting, allows the model to better meet specific needs. Physician notes can be extracted for this fine-tuning from the MIMIC-III dataset.

Use RAG in conjunction with CoT prompting

RAG enables retrieval of relevant information, such as patient data from medical databases, to inform the instruction prompting process. Our semantic search embeddings identify the most relevant documents based on query similarity (e.g., patients with similar ICD codes). This helps guide LLM text generation, while RAG minimizes “hallucinations” by feeding relevant facts into the model, improving the accuracy and relevance of clinical note generation.

Better Integrating Disease and Clinical Ontologies as KG Reasoners for Clinical Inference

Incorporating structured and standardized representations of clinical and biomedical knowledge through disease ontologies (e.g., SNOMED CT [National Library of Medicine, 2021, 2019], MONDO [Vasilevsky et al., 2022]) has the potential to enhance an LLM’s reasoning by providing a formal framework of relational disease concepts. We plan to investigate more effective methods for embedding ontology KGs into CoT prompting, such as translating them into natural language descriptions or designing more precise graph-based representations, enabling the LLMs to better infer latent relationships between conditions, symptoms, and comorbidities for clinical note generation. Furthermore, optimally aligning model outputs with patient-specific details and related conditions in the medical notes could further improve the quality of the generated content.

FIG. 13.

UMAP token projection for sample instance #2.

FIG. 14.

UMAP token projection for sample instance #3.

FIG. 15.

UMAP token projection for sample instance #4.

FIG. 16.

UMAP token projection for sample instance #5.

FIG. 17.

A sample from clinical case A showing ground-truth and the machine-generated text.

FIG. 18.

Pairwise token-level cosine similarity between ground-truth and generated text from clinical case A.

AUTHORS’ CONTRIBUTIONS

I.M.: Conceptualization, methodology, software, data curation, formal analysis, writing—original draft preparation, visualization. M.N.: Software, writing—reviewing and editing. J.W.: Methodology, writing—reviewing and editing, funding acquisition. M.B.: Writing—reviewing and editing. Y.L.: Supervision, project administration, conceptualization, methodology, writing—reviewing and editing, funding acquisition.

Footnotes

AUTHOR DISCLOSURE STATEMENT

No competing financial interests exist.

FUNDING INFORMATION

The project is partially supported by the AI Initiative funded by Old Dominion University.

References

Arndt

, Beasley

, Watkinson

, et al. Tethered to the EHR: Primary care physician workload assessment using EHR event log data and time-motion observations. Ann Fam Med, 2017; 15(5):419–426; doi: 10.1370/afm.2121

Ash

, Corby

, Mohan

, et al. Safe use of the EHR by medical scribes: A qualitative study. J Am Med Inform Assoc, 2021; 28(2):294–302; doi: 10.1093/jamia/ocaa199

Beltagy

, Lo

, Cohan

. SciBERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019; pp. 3615–3620.

Besta

, Blach

, Kubicek

, et al. Graph of thoughts: Solving elaborate problems with large language models. AAAI, 2024; 38(16):17682–17690; doi: 10.1609/aaai.v38i16.29720

Biswas

, Talukdar

. Intelligent clinical documentation: Harnessing generative AI for patient-centric clinical note generation. ArXiv, 2024. arXiv:2405.18346.

Brown

, Mann

, Ryder

, et al. Language models are few-shot learners. In: NeurIPS. Curran Associates Inc.: Vancouver, Canada; 2020.

Burde

. Health law the HITECH act - An overview. Virtual Mentor, 2011; 13(3):172–175; doi: 10.1001/virtualmentor.2011.13.3.hlaw1-1103

Busch

, Hoffmann

, Rueger

, et al. Current applications and challenges in large language models for patient care: A systematic review. Commun Med (Lond), 2025; 5(1):26; doi: 10.1038/s43856-024-00717-2

Chiu

, Chung

. Protocol for human evaluation of generative artificial intelligence chatbots in clinical consultations. PLoS One, 2025; 20(3):e0300487; doi: 10.1371/journal.pone.0300487

10.

Classen

, Li

, Miller

, et al. An electronic health record–based real-time analytics program for patient safety surveillance and improvement. Health Aff (Millwood), 2018; 37(11):1805–1812; doi: 10.1377/hlthaff.2018.0728

11.

Devlin

, Chang

, Lee

, et al. Bert: Pre-training of deep bidirectional transformers for language understanding. In: NAACL: HLT; Minneapolis, Minnesota; 2019; pp. 4171–4186.

12.

Gellert

, Ramirez

, Webster

. The rise of the medical scribe industry: Implications for the advancement of electronic health records. JAMA, 2015; 313(13):1315–1316; doi: 10.1001/jama.2014.17128

13.

Johnson

, Pollard

, Shen

, et al. MIMIC-III, a freely accessible critical care database. Sci Data, 2016; 3:160035; doi: 10.1038/sdata.2016.35

14.

Kroth

, Morioka-Douglas

, Veres

, et al. Association of electronic health record design and use factors with clinician stress and burnout. JAMA Netw Open, 2019; 2(8):e199609; doi: 10.1001/jamanetworkopen.2019.9609

15.

Lee

, Lindsey

. Do large language models understand medical codes? ArXiv, 2024 arXiv:2403.10822v2

16.

Leong

, Gao

, Ji

, et al. A gen AI framework for medical note generation. ArXiv, 2024 arXiv:2410.01841.

17.

, Tang

, Zhao

, et al. Pre-trained language models for text generation: A survey. ACM Comput Surv, 2022; 56:1–39.

18.

, Fei

, Zeng

, et al. Automated icd-9 coding via a deep learning approach. IEEE/ACM Trans Comput Biol Bioinform, 2019; 16(4):1193–1202; doi: 10.1109/TCBB.2018.2817488

19.

Li’evin

, Hother

, Winther

. Can large language models reason about medical questions? Patterns, 2022; 5.

20.

Liu

. Learning to write notes in electronic health records, arXiv, 2018. Available from: arXiv:1808.02622

21.

Long

. Large language model guided tree-of-thought. ArXiv, 2023 arXiv:2305.08291.

22.

Maity

, Saikia

. Large language models in healthcare and medical applications: A review. Bioengineering, 2025; 12(6):631; doi: 10.3390/bioengineering12060631

23.

Makohon

, Li

. Multi-label classification of icd-10 coding clinical notes using mimic codiesp. In: 2021 IEEE EMBS International Conference on Biomedical and Health Informatics (BHI). IEEE; 2021; pp. 1–4; doi: 10.1109/BHI50953.2021.9508541

24.

McInnes

, Healy

, Saul

, et al. UMAP: Uniform manifold approximation and projection. JOSS, 2018; 3(29):861; doi: 10.21105/joss.00861

25.

Miranda-Escalada1

, Gonzalez-Agirre1

, Armengol-Estap

, et al. ′ Overview of automatic clinical coding: Annotations, guidelines, and solutions for non-English clinical cases at codiesp track of clef ehealth 2020. In: Conference and Labs of the Evaluation Forum (CLEF), 2020.

26.

Moore

, Frye

. Review of HIPAA, part 1: History, protected health information, and privacy and security rules. J Nucl Med Technol, 2019; 47(4):269–272; doi: 10.2967/jnmt.119.227819

27.

National Library of Medicine. SNOMED CT. U.S. Department of Health and Human Services, 2019. Available from: https://www.nlm.nih.gov/healthit/snomedct/index.html [Last accessed: May 31, 2025].

28.

National Library of Medicine. SNOMED CT to ICD-10-CM Map. U.S. Department of Health and Human Services; 2021. Available from: https://www.nlm.nih.gov/research/umls/mapping_projects/snomedct_to_icd10cm.html [Last accessed: May 31, 2025].

29.

Negash

, Katz

, Neilson

, et al. De-identification of free text data containing personal health information: A scoping review of reviews. Int J Popul Data Sci, 2023; 8(1):2153; doi: 10.23889/ijpds.v8i1.2153

30.

Nelson

. Copying and pasting patient treatment notes. Virtual Mentor, 2011; 13(3):144–147; doi: 10.1001/virtualmentor.2011.13.3.ccas1-1103

31.

Niu

, Wu

, Li

, et al. Retrieve and rerank for automated ICD coding via contrastive learning. J Biomed Inform, 2023; 143:104396; doi: 10.1016/j.jbi.2023.104396

32.

Office of the National Coordinator for Health Information Technology. Office-based physician electronic health record adoption, health it quick-stat #50. 2025. Available from: https://www.healthit.gov/data/quickstats/office-based-physician-electronic-health-record-adoption

33.

OpenAI. GPT-4 technical report. ArXiv, 2023; doi: 10.48550/arXiv.2303.08774arXiv:2303.08774

34.

Sahoo

, Singh

, Saha

, et al. A systematic survey of prompt engineering in large language models: Techniques and applications. ArXiv, 2024 arXiv:2402.07927.

35.

Stenetorp

, Pyysalo

, Topić

, et al. brat: A web-based tool for NLP-assisted text annotation. In: Proceedings of the Demonstrations at the 13th Conference of the European Chapter of the Association for Computational Linguistics, 2012.

36.

Tam

TYC

, Sivarajkumar

, Kapoor

, et al. A framework for human evaluation of large language models in healthcare derived from literature review. NPJ Digit Med, 2024; 7(1):258; doi: 10.1038/s41746-024-01258-7

37.

U.S. Food & Drug Administration. 21st century cures act. 2020. Available from: https://www.fda.gov/regulatory-information/selected-amendments-fdc-act/21st-century-cures-act

38.

Vasilevsky

, Matentzoglu

, Toro

, et al. Mondo: Unifying diseases for the world, by the world. medRxiv, 2022; doi: 10.1101/2022.04.13.22273750

39.

Vrdoljak

, Boban

, Vilović

, et al. A review of large language models in medical education, clinical decision support, and healthcare administration. Healthcare, 2025; 13(6):603; doi: 10.3390/healthcare13060603

40.

Wang

, Gao

, Liu

, et al. Towards adapting open-source large language models for cost-effective, expert-level clinical note generation with on-policy reinforcement learning. ArXiv, 2024 arXiv:2405.00715.

41.

Wang

, Wei

, Schuurmans

, et al. Self-consistency improves chain of thought reasoning in language models. ArXiv, 2022.

42.

Wei

, Wang

, Schuurmans

, et al. Chain-of-thought prompting elicits reasoning in large language models. In: Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ‘22, 2022.

43.

Weng

, Zhu

, Xia

, et al. Large language models are better reasoners with self-verification. In: Findings of the ACL: EMNLP, ACL; 2023; pp. 2550–2575.

44.

, Yang

, Zhan

, et al. A survey on LLM-generated text detection: Necessity, methods, and future directions. ArXiv, 2023 arXiv:2310.14724.

45.

, Zeng

, Yu

, et al. A pseudo label-wise attention network for automatic ICD coding. IEEE J Biomed Health Inform, 2022; 26(10):5201–5212; doi: 10.1109/JBHI.2022.3193291

46.

Yao

, Yu

, Zhao

, et al. Tree of thoughts: Deliberate problem solving with large language models. In: NeurIPS; ACM; 2023a.

47.

Yao

, Li

, Zhao

. Beyond chain-of-thought, effective graph-of-thought reasoning in large language models. ArXiv, 2023b arXiv:2305.16582.

48.

, Hwang

, Yang

, et al. Investigating the effectiveness of task-agnostic prefix prompt for instruction following. ArXiv, 2023 arXiv:2302.14691.

49.

Yuan

, Rastogi

, Naik

, et al. A continued pretrained LLM approach for automatic medical note generation. In: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics, 2024; pp. 565–571.

50.

Zhang

, Zhang

, Li

, et al. Multimodal chain-of-thought reasoning in language models. ArXiv, 2023 arXiv:2302.00923.