Abstract
Background
Aneurysmal subarachnoid hemorrhage results in significant mortality and disability, which is worsened by the development of delayed cerebral ischemia. Tests to identify patients with delayed cerebral ischemia prospectively are of high interest.
Objective
We created a machine learning system based on clinical variables to predict delayed cerebral ischemia in aneurysmal subarachnoid hemorrhage patients. We also determined which variables have the most impact on delayed cerebral ischemia prediction using SHapley Additive exPlanations method.
Methods
500 aneurysmal subarachnoid hemorrhage patients were identified and 369 met inclusion criteria: 70 patients developed delayed cerebral ischemia (delayed cerebral ischemia+) and 299 did not (delayed cerebral ischemia−). The algorithm was trained based upon age, sex, hypertension (HTN), diabetes, hyperlipidemia, congestive heart failure, coronary artery disease, smoking history, family history of aneurysm, Fisher Grade, Hunt and Hess score, and external ventricular drain placement. Random Forest was selected for this project, and prediction outcome of the algorithm was delayed cerebral ischemia+. SHapley Additive exPlanations was used to visualize each feature's contribution to the model prediction.
Results
The Random Forest machine learning algorithm predicted delayed cerebral ischemia: accuracy 80.65% (95% CI: 72.62–88.68), area under the curve 0.780 (95% CI: 0.696–0.864), sensitivity 12.5% (95% CI: −3.7 to 28.7), specificity 94.81% (95% CI: 89.85–99.77), PPV 33.3% (95% CI: −4.39 to 71.05), and NPV 84.1% (95% CI: 76.38–91.82). SHapley Additive exPlanations value demonstrated Age, external ventricular drain placement, Fisher Grade, and Hunt and Hess score, and HTN had the highest predictive values for delayed cerebral ischemia. Lower age, absence of hypertension, higher Hunt and Hess score, higher Fisher Grade, and external ventricular drain placement increased risk of delayed cerebral ischemia.
Conclusion
Machine learning models based upon clinical variables predict delayed cerebral ischemia with high specificity and good accuracy.
Introduction
Cerebral aneurysm rupture results in aneurysmal subarachnoid hemorrhage (aSAH) and causes death in 40%–50% and significant morbidity among survivors.1,2 Much of the morbidity and mortality of aSAH is due to delayed cerebral ischemia (DCI), which occurs six to nine days after aSAH.3,4 DCI is defined as a decline in arousal or the development of sustained focal neurologic deficits that are not attributable to other causes, such as fever, hydrocephalus, or seizure. 5 DCI is closely coupled to the development of cerebral artery vasospasm, which is a major contributor to DCI symptoms. Given the clinical impact of DCI, significant effort and expense are harnessed to identify and prevent this complication of cerebral aneurysm rupture.
Early detection of DCI is facilitated by frequent neurological assessments in the intensive care unit, daily transcranial Doppler (TCD) studies, periodic screening by computed tomography angiography (CTA), computed tomography perfusion (CTP), or even digital subtraction angiography (DSA). 6 However, the sensitivity of these techniques for DCI prediction is poor, and CTA, CTP, and DSA are largely confirmatory studies for the presence of vasospasm and DCI.5,6 A sensitive and accurate prospective test that predicts DCI development would allow for tailored monitoring of patients at high risk of DCI and de-escalation of care and significant cost savings in those at low risk of DCI development.
Machine learning (ML) algorithms are a form of artificial intelligence that can analyze datasets to discern multivariable patterns that may accurately predict clinical outcome. By processing a large amount of data, ML may assist clinical decision-making, and has the potential to offer superior predictions of outcomes. Indeed, ML has offered preliminary promise as a predictor of DCI based upon clinical data in previous studies.7–10
In this study, we developed an Explainable ML algorithm based upon a SHapley Additive exPlanations (SHAP) approach to accurately predict DCI in patients with aSAH.
Material and methods
Study population and outcome definition
We performed a retrospective cohort study of consecutive aSAH patients who were treated at Stanford University Medical Center from January 2006 to September 2019. Subjects for inclusion were identified through a database of all patients who underwent cerebral aneurysm treatment at our institution over this time period. This study was approved by our institutional internal review board, which waived the need for patient consent.
Inclusion criteria were: the presence of clinical variables for modeling (see below), baseline non-contrast CT and CTA available, and record of DCI development and treatment. We identified 500 consecutive aSAH patients, and 131 patients were excluded due to missing prespecified variables for the analysis. The diagram outlining patient selection is shown in Supplemental Figure 1.
Patients were dichotomized into those who developed DCI (DCI+) and those who did not (DCI−). DCI was defined as new neurologic deterioration that was not evident after aneurysm treatment and that could not be attributed to aneurysm re-bleeding, hydrocephalus, infection, seizure, hyponatremia or other metabolic abnormality. 11 The development of a new focal neurologic deficit, a decreased level of consciousness, and/or a decline on the Glasgow Coma Scale of ≥2 points that was sustained for ≥1 h was considered to represent a new neurologic deficit or deterioration.
Clinical and imaging data acquisition and preparation
Clinical data were obtained from the electronic medical record. We pre-specified the following variables for our analysis: sex, age, hypertension, hyperlipidemia, diabetes mellitus, coronary artery disease, smoking history, family history of cerebral aneurysm, Hunt and Hess Score on presentation, modified Fisher Grade on presentation head CT, and whether an extra-ventricular drain (EVD) was placed. All patients were treated with oral nimodipine after admission.
Imaging data were obtained for all subjects included in the study from the institutional radiology Picture Archiving and Communication System. Baseline non-contrast head CT and CTA images were anonymized and imported into a Horos Database (Horos Project). Non-contrast CT images were reviewed to assign a modified Fisher Grade score, and CTA images were reviewed to confirm the presence of a ruptured cerebral aneurysm by two neurointerventionalists (ES, 4 years of experience; BP 3 years of experience).
Machine learning algorithm selection and training
All analyses were developed in Python (version 3.8) within Jupyter Notebook. Before training the model, a Pearson correlation matrix of DCI features was analyzed and visualized to investigate the potential correlations between different features (Supplemental Figure 2). Patient age was standardized prior to training the algorithm using Scikit Learn's standard scaler.
The primary prediction outcome of the ML algorithm was DCI development (DCI+). To develop the algorithm, the cohorts were randomly split into training set (75%, 276 cases, 222 DCI− and 54 DCI+) and testing set (25%, 93 cases, 77 DCI− and 16 DCI+). The percentage of DCI+ and DCI− cases are about the same in both the training and testing datasets with the following breakdown: training: 19.6% DCI+ and 80.4% DCI−, testing:17.2% DCI+ and 82.8 DCI−. The ML algorithm was trained based upon age, sex, hypertension (HTN), diabetes, hyperlipidemia, congestive heart failure, CAD, smoking history, family history of aneurysm, modified Fisher Grade, Hunt and Hess score, and EVD placement.
To find the best algorithm for this particular application, the training set was used to train several ML models, such as XGBoost, Random Forest, decision tree, support vector machines, and logistic regression. A nested cross-validation strategy was then employed to avoid overfitting the model and to improve its generalizability. The training dataset was divided into five folds, and each fold was further split for training and validating the algorithm. Using cross-validation combined with grid search to optimize the hyperparameters, we compared all models against each another. We used the average of accuracies of the cross-validations to compare the hyperparameter combinations in the algorithm's grid search. Random Forest, the ML model with the best average performance over all the cross-validation sets, was selected as the ML algorithm of choice for this study.
SHAP was implemented to explain the DCI predictions by the ML model from the input clinical variables. The Shapley value for each feature (age, sex, HTN, etc.) represents the average marginal contribution of the feature to the output (DCI+ or DCI−) considering all possible feature combinations. A matrix of SHAP values was generated to provide a clear representation of each feature's individualized contribution to the model's predictions.
Statistical analysis
Two-tailed t-test for continuous variables and chi-square for binary variables were used to calculate the p-value and determine the differences in the clinical variables. Wilcox Rank-Sum test was used for the analysis of difference between Hunt and Hess and Fisher Grade values between DCI+ and DCI− groups. All statistical analyses were performed using SciPy, which is a free and open-source Python library for various statistical computations. Statistical significance was set at
Results
We identified 500 consecutive aSAH patients, and 131 patients were excluded due to missing prespecified variables for the analysis. Of the remaining 369 patients, 68.6% (253/339) were female, and the mean age was 51.6 ± 12.5 y. There were 299 patients who did not develop DCI (DCI−) and 70 patients who developed DCI (DCI+). There were significant differences between DCI+ and DIC− patients with respect to smoking status, Hunt and Hess Score, Fisher Grade, and EVD placement (Table 1). In DCI+ patients, the median of Hunt and Hess score and Fisher Grade were 3 and 3, respectively, while the same parameters in DCI− group were 2 and 3. 47.14% of DCI+ patients have a history of smoking and 71.43% had EVD placement, while 27.76% of DCI− patients smoke and 47.5% had EVD placement. Hunt and Hess Score, Fisher Grade, and EVD placement were all highly positively correlated (Supplemental Figure 2). While, on the other hand, Fisher Grade and family history of aneurysm have a slightly negative correlation.
Summary of clinical variables of DCI+ vs. DCI− Patients.
Note: Numbers within parenthesis are percentages of total patients within each group. Age is reported as the mean plus or minus the standard deviation. Hunt&Hess and Fisher Grade are reported by their medians, with the first and third quartiles of their distribution in the parenthesis. DCI: delayed cerebral ischemia; EVD: external ventricular drain.
P-value calculation: Two-tailed t-test for continuous variables chi-square for binary variables.
P-value calculation: Two-tailed t-test for binary variables.
Wilcox Rank-Sum test was used for the analysis of the difference between Hunt and Hess and Fisher Grade values between DCI+ and DCI− groups.
The ML algorithm was trained with cross-validation, and Random Forest had the best performance with an average validation accuracy of 77.5% (95% CI:0.726–0.824). XGBoost had the second-best performance with an average accuracy of 77.2% (95% CI:0.723–0.821). The performance of support vector machines and logistic regression were comparable, with the average accuracies below 75.0 Therefore, we applied the Random Forest model to the ML testing stage and SHAP approach.
In the testing dataset, the Random Forest ML algorithm predicted DCI with: Accuracy 80.65% (95% CI: 72.62–88.68), area under the curve (AUC) 0.780 (95% CI: 0.696–0.864), sensitivity 12.5% (95% CI: −3.7 to 28.7), specificity 94.81% (95% CI: 89.85 to 99.77), positive predictive value 33.3% (95% CI: −4.39 to 71.05), and negative predictive value 84.1% (95% CI: 76.38 to 91.82). The performance of the model is further represented by the confusion matrices in Figure 1 and AUC curves in Supplemental Figure 3. The SHAP method was used to determine which variables were the strongest predictors of DCI. We sorted the importance of the features in the prediction of DCI using the SHAP approach (Figure 2). The topmost important variables in terms of predictive values for DCI were higher modified Fisher Grade, younger age, higher Hunt and Hess score, absence of history of hypertension, and EVD placement. Female sex, smoking history, and presence of hyperlipidemia were also associated with an increased risk of DCI, but to a lesser extent. SHAP value also revealed the feature importance of the prediction on individual cases (Figure 3).
Confusion matrix of predictions: Training (a) and testing (b) results. The matrix in (a) shows the results of training the algorithm on the portion of the training dataset. The algorithm showed a 77.17% accuracy while training it. (b) shows the confusion matrix on the testing portion of the dataset. The algorithm showed an 80.65% accuracy on the test dataset.
Feature importance for the prediction of DCI. These figures show the importance of various features in DCI+ outcome prediction by the algorithm. (a) The features that contribute the most to a DCI+ outcome are shown. Fisher grade, age, and Hunt and Hess are shown to contribute the most to positive DCI outcomes. (b) The value of features and their contributions to DCI+ outcome are further depicted. Each dot on the figure represents a datapoint, and its color corresponds to the raw values (not the SHAP values) of the datapoint. The redder the dot, the higher its raw value is while the bluer the dot, the smaller the value. For instance, an older patient's datapoint on the age row will be redder than a younger patient's datapoint, which will be closer to being blue. Furthermore, the location of each dot on the SHAP value axis indicates its contribution to the outcome. If a datapoint is on the positive side of the SHAP value axis, it positively contributes to the DCI+ outcome and vice versa. Putting it all together, if a datapoint is red and is on the positive side of SHAP scale, it means that the larger that variable is, the more it contributes to a DCI+ outcome. On the other hand, if a datapoint has a red color but is on the negative part of the SHAP scale, it indicates that the higher that variable is, the more it negatively contributes to the outcome. For instance, lack of HTN positively contributes to DCI+ while the presence of it reduces the odds of DCI+. Moreover, smoking, which has many red dots in the positive SHAP value, contributes to a DCI+ prediction. SHAP: SHapley Additive exPlanations; DCI: delayed cerebral ischemia; HTN: hypertension.
Predicted possibility of DCI in three separate cases. In case 1, the predicted probability is 0.11, which is less than the baseline value of 0.2. This means that the algorithm is predicting a low probability of DCI. The width of each box represents its contribution to the final result. The pink variables increase the probability while the blue variables decrease it. In case one, for instance, the lack of EVD placement and positive HTN substantially lower the odds of DCI. In case 2, the predicted probability is 0.37, which means the algorithm is predicting a higher chance of DCI. Hunt and Hess score has the widest pink box, therefore, it is contributing the most to a positive DCI prediction. In case 3, the algorithm has a strong prediction for DCI. All positive pink variables are pointing to a positive DCI prediction while the blue variables are not contributing much to the outcome. DCI: delayed cerebral ischemia; EVD: external ventricular drain; HTN: hypertension.
Discussion
In this study, we found that our Random Forest ML algorithm had strong accuracy for the prediction of DCI in patients with aSAH. The ML algorithm demonstrated a 94.81% specificity, 12.5% sensitivity, 33.1% positive predictive value, and 84.1% negative predictive value, which suggests that the algorithm may be useful to identify patients at an increased risk of DCI in whom close monitoring is warranted. Considering that DCI is a preventable cause of mortality and poor neurological outcome in aSAH patients, tests to identify patients at risk of DCI are of high interest. 12 Our study has important implications for future artificial intelligence research for the evaluation of aSAH patients and DCI prediction.
The morbidity and mortality that occurs secondary to the initial cerebral aneurysm rupture remain very difficult to manage for physicians and medical teams. By contrast, the impact of DCI may be mitigated through hemodynamic modulation, nimodipine administration, and endovascular vasospasm treatment.5,6 Thus, the ability to identify patients at risk of DCI has important treatment implications. Current screening tests for DCI are largely driven by non-invasive imaging techniques. Daily TCDs are perhaps the most commonly used screening tool, but the sensitivity of this technique is relatively poor and ranges from 48% to 87%. 13 Importantly, TCD sensitivity depends on specific artery location; for the anterior cerebral artery it may be as low as 18%. 14 Specificity of TCDs can range from 65% to 99% and is also dependent on the involved artery. 14 Computed tomography and magnetic resonance imaging techniques have improved sensitivity, but these tests cannot be performed daily and are generally used as a tool to triage patients for endovascular vasospasm treatment in an effort to prevent DCI.15,16 Considering these technical limitations, new accurate screening tests that identify patients at risk of DCI are sorely needed.
Prior ML studies have shown AUC for DCI prediction to range from 0.63 to 0.89.9,13–15 The results of our study found an AUC of 0.780, which is comparable with these studies. We hypothesize that additional clinical and imaging characteristics, such as baseline non-contrast head CT evaluation, of aSAH patients may have importance for DCI prediction and may improve the performance of our ML algorithm, particularly with respect to sensitivity. We also hypothesize that ML algorithms that have a high negative predictive value, such as ours, have the potential to significantly reduce hospitalization costs by allowing for shorter intensive care unit stays and reduced TCD screening for DCI. Future studies are warranted to test these hypotheses.
There are several limitations to our study. The retrospective design, exclusion of patients with missing clinical data, and relatively small sample size of our study may introduce bias. The retrospective design also limits the inclusion of additional data points that may influence the model's performance further. The single-center design may limit the generalizability of our findings. Lastly, the unbalance nature of our dataset (70 DCI+, 299 DCI−) likely placed limits on the performance of our model. Future larger, prospective multicenter studies are warranted to obviate these limitations.
Conclusions
ML models based on clinical variables predict DCI with high specificity and good accuracy.
Supplemental Material
sj-docx-1-ine-10.1177_15910199231170411 - Supplemental material for Prediction of delayed cerebral ischemia after cerebral aneurysm rupture using explainable machine learning approach*
Supplemental material, sj-docx-1-ine-10.1177_15910199231170411 for Prediction of delayed cerebral ischemia after cerebral aneurysm rupture using explainable machine learning approach* by Reza M Taghavi, Guangming Zhu, Max Wintermark, Gabriella M Kuraitis, Eric S Sussman, Benjamin Pulli, Brook Biniam, Sophie Ostmeier and Gary K Steinberg, Jeremy J Heit in Interventional Neuroradiology
Footnotes
Acknowledgments
This work was supported by a generous donation from the CYNL Foundation to Dr. Heit.
Author contribution statement
RMT and JJH conceived the study and performed data collection and had substantial input to the primary manuscript in revisions. RMT, SO, and BB performed coding and data analysis. All other authors contributed significantly to the revisions.
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Supplemental material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
